|
2 | 2 |
|
3 | 3 | > A vendor-neutral .NET reader for [Apache Iceberg](https://iceberg.apache.org) tables. |
4 | 4 |
|
5 | | -[](https://github.com/andreabozzo/IcebergSharp/actions/workflows/ci.yml) |
| 5 | +[](https://github.com/AndreaBozzo/IcebergSharp/actions/workflows/ci.yml) |
| 6 | +[](https://github.com/AndreaBozzo/IcebergSharp/actions/workflows/integration.yml) |
6 | 7 | [](#license) |
7 | 8 | [](#requirements) |
| 9 | +[](https://iceberg.apache.org/spec/) |
8 | 10 |
|
9 | | -IcebergSharp lets .NET applications read Apache Iceberg tables directly from a REST |
10 | | -catalog (Apache Polaris, Project Nessie, Snowflake Open Catalog, AWS Glue, lakekeeper, |
11 | | -Unity Catalog) without going through Spark, Trino, or any JVM service. |
| 11 | +IcebergSharp lets .NET applications read [Apache Iceberg](https://iceberg.apache.org) |
| 12 | +tables directly from a REST catalog (Apache Polaris, Project Nessie, Snowflake Open |
| 13 | +Catalog, AWS Glue, lakekeeper, Unity Catalog) without going through Spark, Trino, or |
| 14 | +any JVM service. |
12 | 15 |
|
13 | | -The library handles the things that make Iceberg interesting and hard to get right: |
| 16 | +It handles the things that make Iceberg interesting and hard to get right: |
14 | 17 | field-id resolution for schema evolution, partition pruning at the manifest level, |
15 | 18 | column-stats pruning at the data-file level, and time travel via snapshots. |
16 | 19 |
|
| 20 | +### My idea behind IcebergSharp |
| 21 | + |
| 22 | +Today, .NET teams that want to read Iceberg either spin up a JVM service (Spark |
| 23 | +Connect, Trino) and pay the latency / ops cost, or go through a query engine that |
| 24 | +doesn't expose Iceberg's metadata to them. There's no native client that gives a |
| 25 | +.NET app the same first-class access that `pyiceberg` gives Python or |
| 26 | +`iceberg-rust` gives Rust. IcebergSharp aims to be that client: read-only, no JVM, |
| 27 | +no embedded query engine — just metadata and Arrow batches you can hand to |
| 28 | +DuckDB.NET, ML.NET, or Power BI. |
| 29 | + |
17 | 30 | > **Status:** Phase 0 — repository scaffolding only. No public API yet. See the |
18 | 31 | > [roadmap](#roadmap) for what is coming and when. |
19 | 32 |
|
20 | 33 | --- |
21 | 34 |
|
22 | 35 | ## Scope (what this library does and does NOT do) |
23 | 36 |
|
24 | | -**In scope for v1:** |
| 37 | +**In scope for v1** — read-only Iceberg, focused on what analytical workloads actually |
| 38 | +need: |
25 | 39 |
|
26 | | -- Read-only access to Iceberg tables. |
27 | | -- REST Catalog client (spec-compliant, with dynamic endpoint discovery). |
28 | | -- AWS Glue Iceberg REST endpoint (SigV4 auth). |
29 | | -- Local, S3, and Azure Data Lake Storage Gen2 file IO. |
| 40 | +- Spec-compliant REST Catalog client with dynamic endpoint discovery; AWS Glue SigV4. |
| 41 | +- File IO for local, S3, and ADLS Gen2. |
30 | 42 | - Parquet data files with field-id resolution. |
31 | 43 | - Partition pruning, column-stats pruning, projection pushdown. |
32 | | -- Time travel: scan any historical snapshot by id or timestamp. |
33 | | -- Schema evolution: add column, drop column, rename column, type promotion. |
34 | | -- Apache Arrow output adapter for zero-copy interop with DuckDB.NET / Polars.NET / ML.NET. |
35 | | - |
36 | | -**Out of scope for v1 (explicit non-goals):** |
37 | | - |
38 | | -- ❌ **No write path.** No `CREATE TABLE`, no `INSERT`, no commits, no snapshot creation, |
39 | | - no manifest writing. Writing Iceberg correctly is ~70% of the engineering effort and |
40 | | - ~90% of the bugs in existing implementations. The goal of v1 is "read-only, done well." |
41 | | -- ❌ **No merge-on-read / delete files.** v1 reads copy-on-write tables only. Position |
42 | | - and equality delete files are skipped with a clear warning. Most analytical workloads |
43 | | - on Iceberg are still COW-dominant. |
44 | | -- ❌ **No Hive Metastore.** REST catalogs only. If you are on HMS, point an Iceberg REST |
45 | | - adapter at it and use IcebergSharp against the adapter. |
46 | | -- ❌ **No bundled SQL engine.** IcebergSharp exposes `IAsyncEnumerable<RecordBatch>` and |
47 | | - Arrow streams; bring your own query layer (DuckDB.NET, Microsoft.Data.Analysis, etc.). |
48 | | - |
49 | | -These boundaries are load-bearing. Re-discussing them risks turning the project into a |
50 | | -half-finished Iceberg engine instead of a focused, reliable reader. |
| 44 | +- Time travel by snapshot id or timestamp. |
| 45 | +- Schema evolution: add / drop / rename column, type promotion. |
| 46 | +- Apache Arrow output for zero-copy interop with DuckDB.NET / Polars.NET / ML.NET. |
| 47 | + |
| 48 | +**Out of scope for v1** (load-bearing boundaries — re-discussing them risks turning |
| 49 | +the project into a half-finished engine instead of a focused reader): |
| 50 | + |
| 51 | +- ❌ **No write path.** No `CREATE TABLE`, `INSERT`, commits, or manifest writing. |
| 52 | + Writing Iceberg correctly is ~70% of the engineering effort and ~90% of the bugs |
| 53 | + in existing implementations. v1 is "read-only, done well." |
| 54 | +- ❌ **No merge-on-read / delete files.** COW tables only; delete files are skipped |
| 55 | + with a warning. Most analytical Iceberg workloads remain COW-dominant. |
| 56 | +- ❌ **No Hive Metastore.** REST catalogs only — point an Iceberg REST adapter at HMS. |
| 57 | +- ❌ **No bundled SQL engine.** You get `IAsyncEnumerable<RecordBatch>` and Arrow |
| 58 | + streams; bring your own query layer. |
51 | 59 |
|
52 | 60 | --- |
53 | 61 |
|
@@ -161,8 +169,8 @@ Catalog |
161 | 169 | └─ DataFile[] (Parquet w/ field-id annotations) |
162 | 170 | ``` |
163 | 171 |
|
164 | | -The reader's job is: given `catalog.LoadTable(...)`, walk this tree, apply partition + |
165 | | -stats pruning, and stream rows from Parquet with correct field-id resolution. |
| 172 | +Given `catalog.LoadTable(...)`, the reader walks this tree, prunes by partition and |
| 173 | +column stats, and streams Parquet rows with field-id resolution. |
166 | 174 |
|
167 | 175 | --- |
168 | 176 |
|
|
0 commit comments