@knockdata/objectexplorer 0.4.7 → 0.5.0

package/demo/hudi/README.md

@@ -0,0 +1,92 @@
+# Hudi reference tables
+
+Five Apache Hudi tables, each one holding a single thing a reader has to get right. They exist
+because `server/src/duckdb` reads Hudi itself, and these are what say it reads it correctly.
+
+```
+hudi/
+	sales      insert, modify, delete — the file group is rewritten by every commit
+	partition  a file group per partition, and a commit that touches only one of them
+	replace    a replacecommit, which retires whole file groups at once
+	inflight   a commit that never completed, whose parquet is on disk anyway
+	mor        merge on read: a base file, and log files holding everything since
+```
+
+Regenerate with `uv run objectexplorer/reference/data/generate-hudi.py`. Instant times and file ids
+are constants rather than fresh uuids, so regenerating changes nothing and
+`server/test/hudi-golden.json` stays valid.
+
+## The names are the metadata
+
+Delta keeps a log and Iceberg keeps a chain. Hudi writes most of the answer into the file names:
+
+```
+sales/
+	.hoodie/
+		20260817120000000.commit           insert
+		20260817120100000.commit           modify
+		20260817120200000.commit           delete
+		hoodie.properties                  the table's type, key and version
+	b1a2…e001-0_0-1-0_20260817120000000.parquet   4 rows, written by the first commit
+	b1a2…e001-0_0-1-0_20260817120100000.parquet   4 rows, written by the second
+	b1a2…e001-0_0-1-0_20260817120200000.parquet   2 rows, written by the third — live
+```
+
+A base file is `<fileId>_<writeToken>_<instant>.parquet`. Every file with the same file id is one
+**file group** — the same rows rewritten — and the newest instant among them is the one the table
+is made of. The other two are older versions of it, sitting right there, which is why:
+
+```sql
+SELECT * FROM 'hudi/sales';             -- 2 rows, the table
+SELECT * FROM 'hudi/sales/*.parquet';   -- 10 rows, three versions stacked
+```
+
+Ten is 4 + 4 + 2. The `id = 2` row appears three times at two amounts, and the deleted `north` rows
+come back. Nothing in the parquet files says which of them is current.
+
+That first query has no function around the path on purpose. `server/src/duckdb/sql-expand.js` sees
+a `.hoodie` in the folder and rewrites the query into the `read_parquet` it should have been.
+
+`.hoodie` is what says which instants finished, and that is not decoration. `inflight` holds a write
+that died: its parquet is on disk, its name looks like every other base file's, and only the
+missing `20260817120100000.commit` says those four rows are not in the table.
+
+## Merge on read
+
+`mor` is the other half of Hudi. A change does not rewrite the base file — it is appended to a log
+file beside it, in Hudi's own block format, and the rows the table has are the base file merged with
+everything the logs said afterwards:
+
+| block | instant | what it says |
+|-----------------|-------------------|--------------------------------------------------|
+| avro data | 20260817120100000 | `id = 2` now costs 95 |
+| delete | 20260817120200000 | `id = 3` is gone |
+| parquet data | 20260817120300000 | `id = 4` now costs 999 |
+| avro data | 20260817120400000 | `id = 5` exists |
+| command | 20260817120500000 | ...no it does not: rolls back 20260817120400000 |
+
+Three rows come out of four on disk. A reader that only read the base file would answer with the
+row that was deleted, the two rows that were changed, and none of what the logs added — and a
+reader that read the blocks without honouring the rollback would invent a fifth row that no writer
+ever committed.
+
+The blocks are read by `server/src/duckdb/hudi-log.js`. An avro data block holds raw records with
+the schema up in the block header, so it is wrapped back into the Avro container file it came out
+of and duckdb reads it (`hudi-avro.js`) — the same move the Iceberg reader makes for a manifest.
+
+## Written by hand
+
+Hudi writes with Spark and there is no JVM here, so the generator writes the bytes itself. What
+makes that trustworthy is hudi-rs, the reader behind the `hudi` package on PyPI: it opens every one
+of these and its row counts are what the generator prints.
+
+It gets through four of the five. hudi-rs 0.4.0 opens `HoodieDeleteRecord.avsc` from disk at run
+time and its wheel ships without it, so `mor` is the one it declines — the same way `deltalake`
+declines the Delta deletion vector fixture. The blocks either side of that delete are hudi-rs's own
+answer, the delete block is written the way its source parses one, and `server/test/hudi.test.js`
+reads all of it back.
+
+Two small liberties, neither of which any reader can see: a rolled-back instant would also leave a
+`.rollback` file in `.hoodie`, which is Avro rather than json and says nothing the command block in
+the log does not; and there is no `.hoodie/metadata` table, which is an index of the listing a
+reader can do for itself.

package/demo/iceberg/README.md

@@ -0,0 +1,80 @@
+# Iceberg reference tables
+
+Seven Iceberg tables, each one holding a single thing a reader has to get right. They exist because
+`server/src/duckdb` reads Iceberg itself now, and these are what say it reads it correctly.
+
+```
+iceberg/
+	sales       an append, then a delete that rewrote the file it touched
+	partition   identity partitioning, so a manifest carries the partition value
+	evolved     a renamed column and an added one, so the files disagree about their schema
+	hadoop      v1.metadata.json + version-hint.text, the layout with no catalog behind it
+	position    merge-on-read position deletes
+	equality    merge-on-read equality deletes
+	vector      a v3 deletion vector, in a Puffin file
+```
+
+Regenerate with `uv run objectexplorer/reference/data/generate-iceberg.py`. UUIDs and snapshot ids
+change every time; shapes do not.
+
+## The chain
+
+Delta keeps a log. Iceberg keeps a chain, and a reader has to walk all of it:
+
+```
+metadata/v1.metadata.json      the schema, the partition specs, and every snapshot
+	└── snapshot                   the current one, named by current-snapshot-id
+		└── snap-<id>-….avro       the manifest list: which manifests this snapshot is made of
+			└── <uuid>-m0.avro     a manifest: which data files, and which delete files
+				└── data/….parquet the rows
+```
+
+Nothing in `data/` says which snapshot it belongs to, so — exactly as with Delta — a wildcard over
+`data/*.parquet` answers with every version of the table at once:
+
+```sql
+SELECT * FROM 'iceberg/sales';             -- 2 rows, the table
+SELECT * FROM 'iceberg/sales/*.parquet';   -- 6 rows, both snapshots stacked
+```
+
+The first query has no function around the path on purpose. `server/src/duckdb/sql-expand.js` sees a
+folder with a `metadata/` in it, walks the chain, and rewrites the query. There is no
+`iceberg_scan()` any more, and no iceberg extension in the engine either.
+
+## Three ways to delete a row
+
+Iceberg can delete a row without rewriting the file it is in — *merge on read* — and it has three
+spellings for it, which is why three of these fixtures exist.
+
+| fixture | how the delete is recorded | what a reader must do |
+|------------|--------------------------------------------|--------------------------------------------|
+| `sales` | the file is rewritten without the row | nothing; just read the current files |
+| `position` | a parquet of `(file_path, pos)` | drop those row numbers of that file |
+| `equality` | a parquet of values, e.g. `id` | drop rows matching those values |
+| `vector` | a roaring bitmap inside a Puffin file | decode it, drop those row numbers |
+
+Which deletes apply to which data file is decided by sequence number, and the two rules differ by
+one character — a difference that is not cosmetic:
+
+- a **position** delete applies when `data.sequence_number <= delete.sequence_number`, so a delete
+  written in the same commit as its data file *does* apply;
+- an **equality** delete applies when `data.sequence_number < delete.sequence_number`, so one
+  written in that same commit does *not*.
+
+`position`, `equality` and `vector` are written by hand, because pyiceberg only ever rewrites whole
+files. The manifests are real Avro written with `fastavro` (reusing the schema pyiceberg wrote), the
+Puffin blob is built to the layout `delta-kernel-rs`'s Iceberg counterpart reads, and each one was
+only trusted once duckdb's iceberg extension read it and agreed.
+
+## Two things that bite
+
+**Absolute paths.** Iceberg writes fully-qualified paths into its metadata and manifests, so a table
+copied anywhere — another bucket, another machine, a git checkout — names files that are not there.
+The generator rewrites them to wherever the fixture landed, and the reader resolves every recorded
+path against the folder it was handed anyway. duckdb-iceberg calls this `allow_moved_paths`; for a
+file browser it is not optional.
+
+**Snapshot ids are int64.** They are routinely larger than `2^53`, so `JSON.parse` rounds them and
+two snapshots can collide into one number. `position` and `vector` have snapshots exactly one apart
+for precisely this reason — before the reader quoted those ids, it picked the older snapshot and
+quietly returned the undeleted rows.

package/demo/iceberg/equality/metadata/00000-3898835a-27a8-4802-9fd8-25bf74eb05e9.metadata.json

@@ -0,0 +1 @@
+{"location":"file:///Users/rock/git/rock2/objectexplorer/reference/data/iceberg/equality","table-uuid":"98a7b63e-3fff-42e4-9244-27f8e01b8e55","last-updated-ms":1786970727979,"last-column-id":4,"schemas":[{"type":"struct","fields":[{"id":1,"name":"id","type":"long","required":false},{"id":2,"name":"region","type":"string","required":false},{"id":3,"name":"product","type":"string","required":false},{"id":4,"name":"amount","type":"double","required":false}],"schema-id":0,"identifier-field-ids":[]}],"current-schema-id":0,"partition-specs":[{"spec-id":0,"fields":[]}],"default-spec-id":0,"last-partition-id":999,"properties":{},"snapshots":[],"snapshot-log":[],"metadata-log":[],"sort-orders":[{"order-id":0,"fields":[]}],"default-sort-order-id":0,"refs":{},"statistics":[],"partition-statistics":[],"format-version":2,"last-sequence-number":0}

package/demo/iceberg/equality/metadata/00001-bff789d6-a3b8-40b0-adf6-ab42edc90384.metadata.json

@@ -0,0 +1 @@
+{"location":"file:///Users/rock/git/rock2/objectexplorer/reference/data/iceberg/equality","table-uuid":"98a7b63e-3fff-42e4-9244-27f8e01b8e55","last-updated-ms":1786970727982,"last-column-id":4,"schemas":[{"type":"struct","fields":[{"id":1,"name":"id","type":"long","required":false},{"id":2,"name":"region","type":"string","required":false},{"id":3,"name":"product","type":"string","required":false},{"id":4,"name":"amount","type":"double","required":false}],"schema-id":0,"identifier-field-ids":[]}],"current-schema-id":0,"partition-specs":[{"spec-id":0,"fields":[]}],"default-spec-id":0,"last-partition-id":999,"properties":{},"current-snapshot-id":6727406315757390781,"snapshots":[{"snapshot-id":6727406315757390781,"sequence-number":1,"timestamp-ms":1786970727982,"manifest-list":"file:///Users/rock/git/rock2/objectexplorer/reference/data/iceberg/equality/metadata/snap-6727406315757390781-0-6762229c-bca3-4c19-8088-f5e8fd37939e.avro","summary":{"operation":"append","added-files-size":"1716","added-data-files":"1","added-records":"4","total-data-files":"1","total-delete-files":"0","total-records":"4","total-files-size":"1716","total-position-deletes":"0","total-equality-deletes":"0"},"schema-id":0}],"snapshot-log":[{"snapshot-id":6727406315757390781,"timestamp-ms":1786970727982}],"metadata-log":[{"metadata-file":"file:///Users/rock/git/rock2/objectexplorer/reference/data/iceberg/equality/metadata/00000-3898835a-27a8-4802-9fd8-25bf74eb05e9.metadata.json","timestamp-ms":1786970727979}],"sort-orders":[{"order-id":0,"fields":[]}],"default-sort-order-id":0,"refs":{"main":{"snapshot-id":6727406315757390781,"type":"branch"}},"statistics":[],"partition-statistics":[],"format-version":2,"last-sequence-number":1}

package/demo/iceberg/equality/metadata/00002-delete.metadata.json

@@ -0,0 +1,115 @@
+{
+  "location": "file:///Users/rock/git/rock2/objectexplorer/reference/data/iceberg/equality",
+  "table-uuid": "98a7b63e-3fff-42e4-9244-27f8e01b8e55",
+  "last-updated-ms": 1786970727982,
+  "last-column-id": 4,
+  "schemas": [
+    {
+      "type": "struct",
+      "fields": [
+        {
+          "id": 1,
+          "name": "id",
+          "type": "long",
+          "required": false
+        },
+        {
+          "id": 2,
+          "name": "region",
+          "type": "string",
+          "required": false
+        },
+        {
+          "id": 3,
+          "name": "product",
+          "type": "string",
+          "required": false
+        },
+        {
+          "id": 4,
+          "name": "amount",
+          "type": "double",
+          "required": false
+        }
+      ],
+      "schema-id": 0,
+      "identifier-field-ids": []
+    }
+  ],
+  "current-schema-id": 0,
+  "partition-specs": [
+    {
+      "spec-id": 0,
+      "fields": []
+    }
+  ],
+  "default-spec-id": 0,
+  "last-partition-id": 999,
+  "properties": {},
+  "current-snapshot-id": 6727406315757390782,
+  "snapshots": [
+    {
+      "snapshot-id": 6727406315757390781,
+      "sequence-number": 1,
+      "timestamp-ms": 1786970727982,
+      "manifest-list": "file:///Users/rock/git/rock2/objectexplorer/reference/data/iceberg/equality/metadata/snap-6727406315757390781-0-6762229c-bca3-4c19-8088-f5e8fd37939e.avro",
+      "summary": {
+        "operation": "append",
+        "added-files-size": "1716",
+        "added-data-files": "1",
+        "added-records": "4",
+        "total-data-files": "1",
+        "total-delete-files": "0",
+        "total-records": "4",
+        "total-files-size": "1716",
+        "total-position-deletes": "0",
+        "total-equality-deletes": "0"
+      },
+      "schema-id": 0
+    },
+    {
+      "snapshot-id": 6727406315757390782,
+      "parent-snapshot-id": 6727406315757390781,
+      "sequence-number": 2,
+      "timestamp-ms": 1786970728982,
+      "manifest-list": "file:///Users/rock/git/rock2/objectexplorer/reference/data/iceberg/equality/metadata/snap-6727406315757390782-0-delete.avro",
+      "summary": {
+        "operation": "delete"
+      },
+      "schema-id": 0
+    }
+  ],
+  "snapshot-log": [
+    {
+      "snapshot-id": 6727406315757390781,
+      "timestamp-ms": 1786970727982
+    },
+    {
+      "snapshot-id": 6727406315757390782,
+      "timestamp-ms": 1786970728982
+    }
+  ],
+  "metadata-log": [
+    {
+      "metadata-file": "file:///Users/rock/git/rock2/objectexplorer/reference/data/iceberg/equality/metadata/00000-3898835a-27a8-4802-9fd8-25bf74eb05e9.metadata.json",
+      "timestamp-ms": 1786970727979
+    }
+  ],
+  "sort-orders": [
+    {
+      "order-id": 0,
+      "fields": []
+    }
+  ],
+  "default-sort-order-id": 0,
+  "refs": {
+    "main": {
+      "snapshot-id": 6727406315757390781,
+      "type": "branch"
+    }
+  },
+  "statistics": [],
+  "partition-statistics": [],
+  "format-version": 2,
+  "last-sequence-number": 2
+}

package/demo/iceberg/evolved/metadata/00000-7b1ee4a6-d23e-4597-a952-1faeae2c6631.metadata.json

@@ -0,0 +1 @@
+{"location":"file:///Users/rock/git/rock2/objectexplorer/reference/data/iceberg/evolved","table-uuid":"e8bd35bb-21a6-4cbe-9d12-c3313b48a1f7","last-updated-ms":1786970727942,"last-column-id":4,"schemas":[{"type":"struct","fields":[{"id":1,"name":"id","type":"long","required":false},{"id":2,"name":"region","type":"string","required":false},{"id":3,"name":"product","type":"string","required":false},{"id":4,"name":"amount","type":"double","required":false}],"schema-id":0,"identifier-field-ids":[]}],"current-schema-id":0,"partition-specs":[{"spec-id":0,"fields":[]}],"default-spec-id":0,"last-partition-id":999,"properties":{},"snapshots":[],"snapshot-log":[],"metadata-log":[],"sort-orders":[{"order-id":0,"fields":[]}],"default-sort-order-id":0,"refs":{},"statistics":[],"partition-statistics":[],"format-version":2,"last-sequence-number":0}

package/demo/iceberg/evolved/metadata/00001-f82a6028-170f-4c99-9f69-dc9c5db7706f.metadata.json

@@ -0,0 +1 @@
+{"location":"file:///Users/rock/git/rock2/objectexplorer/reference/data/iceberg/evolved","table-uuid":"e8bd35bb-21a6-4cbe-9d12-c3313b48a1f7","last-updated-ms":1786970727945,"last-column-id":4,"schemas":[{"type":"struct","fields":[{"id":1,"name":"id","type":"long","required":false},{"id":2,"name":"region","type":"string","required":false},{"id":3,"name":"product","type":"string","required":false},{"id":4,"name":"amount","type":"double","required":false}],"schema-id":0,"identifier-field-ids":[]}],"current-schema-id":0,"partition-specs":[{"spec-id":0,"fields":[]}],"default-spec-id":0,"last-partition-id":999,"properties":{},"current-snapshot-id":1074794464454607928,"snapshots":[{"snapshot-id":1074794464454607928,"sequence-number":1,"timestamp-ms":1786970727945,"manifest-list":"file:///Users/rock/git/rock2/objectexplorer/reference/data/iceberg/evolved/metadata/snap-1074794464454607928-0-f612a2cc-fd42-4fe7-8f30-849c886ed42f.avro","summary":{"operation":"append","added-files-size":"1716","added-data-files":"1","added-records":"4","total-data-files":"1","total-delete-files":"0","total-records":"4","total-files-size":"1716","total-position-deletes":"0","total-equality-deletes":"0"},"schema-id":0}],"snapshot-log":[{"snapshot-id":1074794464454607928,"timestamp-ms":1786970727945}],"metadata-log":[{"metadata-file":"file:///Users/rock/git/rock2/objectexplorer/reference/data/iceberg/evolved/metadata/00000-7b1ee4a6-d23e-4597-a952-1faeae2c6631.metadata.json","timestamp-ms":1786970727942}],"sort-orders":[{"order-id":0,"fields":[]}],"default-sort-order-id":0,"refs":{"main":{"snapshot-id":1074794464454607928,"type":"branch"}},"statistics":[],"partition-statistics":[],"format-version":2,"last-sequence-number":1}

package/demo/iceberg/evolved/metadata/00002-5751d450-f81a-453c-b1c1-7c0ecb3b847a.metadata.json

@@ -0,0 +1 @@
+{"location":"file:///Users/rock/git/rock2/objectexplorer/reference/data/iceberg/evolved","table-uuid":"e8bd35bb-21a6-4cbe-9d12-c3313b48a1f7","last-updated-ms":1786970727948,"last-column-id":5,"schemas":[{"type":"struct","fields":[{"id":1,"name":"id","type":"long","required":false},{"id":2,"name":"region","type":"string","required":false},{"id":3,"name":"product","type":"string","required":false},{"id":4,"name":"amount","type":"double","required":false}],"schema-id":0,"identifier-field-ids":[]},{"type":"struct","fields":[{"id":1,"name":"id","type":"long","required":false},{"id":2,"name":"region","type":"string","required":false},{"id":3,"name":"product","type":"string","required":false},{"id":4,"name":"total","type":"double","required":false},{"id":5,"name":"currency","type":"string","required":false}],"schema-id":1,"identifier-field-ids":[]}],"current-schema-id":1,"partition-specs":[{"spec-id":0,"fields":[]}],"default-spec-id":0,"last-partition-id":999,"properties":{},"current-snapshot-id":1074794464454607928,"snapshots":[{"snapshot-id":1074794464454607928,"sequence-number":1,"timestamp-ms":1786970727945,"manifest-list":"file:///Users/rock/git/rock2/objectexplorer/reference/data/iceberg/evolved/metadata/snap-1074794464454607928-0-f612a2cc-fd42-4fe7-8f30-849c886ed42f.avro","summary":{"operation":"append","added-files-size":"1716","added-data-files":"1","added-records":"4","total-data-files":"1","total-delete-files":"0","total-records":"4","total-files-size":"1716","total-position-deletes":"0","total-equality-deletes":"0"},"schema-id":0}],"snapshot-log":[{"snapshot-id":1074794464454607928,"timestamp-ms":1786970727945}],"metadata-log":[{"metadata-file":"file:///Users/rock/git/rock2/objectexplorer/reference/data/iceberg/evolved/metadata/00000-7b1ee4a6-d23e-4597-a952-1faeae2c6631.metadata.json","timestamp-ms":1786970727942},{"metadata-file":"file:///Users/rock/git/rock2/objectexplorer/reference/data/iceberg/evolved/metadata/00001-f82a6028-170f-4c99-9f69-dc9c5db7706f.metadata.json","timestamp-ms":1786970727945}],"sort-orders":[{"order-id":0,"fields":[]}],"default-sort-order-id":0,"refs":{"main":{"snapshot-id":1074794464454607928,"type":"branch"}},"statistics":[],"partition-statistics":[],"format-version":2,"last-sequence-number":1}

package/demo/iceberg/evolved/metadata/00003-b392ee5d-33c3-4c03-9448-941481157646.metadata.json

@@ -0,0 +1 @@
+{"location":"file:///Users/rock/git/rock2/objectexplorer/reference/data/iceberg/evolved","table-uuid":"e8bd35bb-21a6-4cbe-9d12-c3313b48a1f7","last-updated-ms":1786970727951,"last-column-id":5,"schemas":[{"type":"struct","fields":[{"id":1,"name":"id","type":"long","required":false},{"id":2,"name":"region","type":"string","required":false},{"id":3,"name":"product","type":"string","required":false},{"id":4,"name":"amount","type":"double","required":false}],"schema-id":0,"identifier-field-ids":[]},{"type":"struct","fields":[{"id":1,"name":"id","type":"long","required":false},{"id":2,"name":"region","type":"string","required":false},{"id":3,"name":"product","type":"string","required":false},{"id":4,"name":"total","type":"double","required":false},{"id":5,"name":"currency","type":"string","required":false}],"schema-id":1,"identifier-field-ids":[]}],"current-schema-id":1,"partition-specs":[{"spec-id":0,"fields":[]}],"default-spec-id":0,"last-partition-id":999,"properties":{},"current-snapshot-id":4829795160162008264,"snapshots":[{"snapshot-id":1074794464454607928,"sequence-number":1,"timestamp-ms":1786970727945,"manifest-list":"file:///Users/rock/git/rock2/objectexplorer/reference/data/iceberg/evolved/metadata/snap-1074794464454607928-0-f612a2cc-fd42-4fe7-8f30-849c886ed42f.avro","summary":{"operation":"append","added-files-size":"1716","added-data-files":"1","added-records":"4","total-data-files":"1","total-delete-files":"0","total-records":"4","total-files-size":"1716","total-position-deletes":"0","total-equality-deletes":"0"},"schema-id":0},{"snapshot-id":4829795160162008264,"parent-snapshot-id":1074794464454607928,"sequence-number":2,"timestamp-ms":1786970727951,"manifest-list":"file:///Users/rock/git/rock2/objectexplorer/reference/data/iceberg/evolved/metadata/snap-4829795160162008264-0-257f0a82-aa89-49ca-ab08-2e15d7a6ce61.avro","summary":{"operation":"append","added-files-size":"1961","added-data-files":"1","added-records":"1","total-data-files":"2","total-delete-files":"0","total-records":"5","total-files-size":"3677","total-position-deletes":"0","total-equality-deletes":"0"},"schema-id":1}],"snapshot-log":[{"snapshot-id":1074794464454607928,"timestamp-ms":1786970727945},{"snapshot-id":4829795160162008264,"timestamp-ms":1786970727951}],"metadata-log":[{"metadata-file":"file:///Users/rock/git/rock2/objectexplorer/reference/data/iceberg/evolved/metadata/00000-7b1ee4a6-d23e-4597-a952-1faeae2c6631.metadata.json","timestamp-ms":1786970727942},{"metadata-file":"file:///Users/rock/git/rock2/objectexplorer/reference/data/iceberg/evolved/metadata/00001-f82a6028-170f-4c99-9f69-dc9c5db7706f.metadata.json","timestamp-ms":1786970727945},{"metadata-file":"file:///Users/rock/git/rock2/objectexplorer/reference/data/iceberg/evolved/metadata/00002-5751d450-f81a-453c-b1c1-7c0ecb3b847a.metadata.json","timestamp-ms":1786970727948}],"sort-orders":[{"order-id":0,"fields":[]}],"default-sort-order-id":0,"refs":{"main":{"snapshot-id":4829795160162008264,"type":"branch"}},"statistics":[],"partition-statistics":[],"format-version":2,"last-sequence-number":2}

package/demo/iceberg/hadoop/metadata/v1.metadata.json

@@ -0,0 +1 @@
+{"location":"file:///Users/rock/git/rock2/objectexplorer/reference/data/iceberg/hadoop","table-uuid":"5286ed49-1d5b-4399-9cc1-12959a24df75","last-updated-ms":1786970727961,"last-column-id":4,"schemas":[{"type":"struct","fields":[{"id":1,"name":"id","type":"long","required":false},{"id":2,"name":"region","type":"string","required":false},{"id":3,"name":"product","type":"string","required":false},{"id":4,"name":"amount","type":"double","required":false}],"schema-id":0,"identifier-field-ids":[]}],"current-schema-id":0,"partition-specs":[{"spec-id":0,"fields":[]}],"default-spec-id":0,"last-partition-id":999,"properties":{},"current-snapshot-id":1143458465259875935,"snapshots":[{"snapshot-id":1143458465259875935,"sequence-number":1,"timestamp-ms":1786970727961,"manifest-list":"file:///Users/rock/git/rock2/objectexplorer/reference/data/iceberg/hadoop/metadata/snap-1143458465259875935-0-01fd755b-8ea2-4aed-9a21-07e07fcf34bc.avro","summary":{"operation":"append","added-files-size":"1716","added-data-files":"1","added-records":"4","total-data-files":"1","total-delete-files":"0","total-records":"4","total-files-size":"1716","total-position-deletes":"0","total-equality-deletes":"0"},"schema-id":0}],"snapshot-log":[{"snapshot-id":1143458465259875935,"timestamp-ms":1786970727961}],"metadata-log":[{"metadata-file":"file:///Users/rock/git/rock2/objectexplorer/reference/data/iceberg/hadoop/metadata/00000-5f9520ae-7bee-483f-8ff2-9f40a0fcbcc3.metadata.json","timestamp-ms":1786970727958}],"sort-orders":[{"order-id":0,"fields":[]}],"default-sort-order-id":0,"refs":{"main":{"snapshot-id":1143458465259875935,"type":"branch"}},"statistics":[],"partition-statistics":[],"format-version":2,"last-sequence-number":1}

package/demo/iceberg/hadoop/metadata/version-hint.text

@@ -0,0 +1 @@
+1

package/demo/iceberg/partition/metadata/00000-cde33df7-2164-4f3a-ac15-10aab9c841ac.metadata.json

@@ -0,0 +1 @@
+{"location":"file:///Users/rock/git/rock2/objectexplorer/reference/data/iceberg/partition","table-uuid":"4ac926be-c3af-4ec5-87a7-bd63348d769c","last-updated-ms":1786970727927,"last-column-id":4,"schemas":[{"type":"struct","fields":[{"id":1,"name":"id","type":"long","required":false},{"id":2,"name":"region","type":"string","required":false},{"id":3,"name":"product","type":"string","required":false},{"id":4,"name":"amount","type":"double","required":false}],"schema-id":0,"identifier-field-ids":[]}],"current-schema-id":0,"partition-specs":[{"spec-id":0,"fields":[]}],"default-spec-id":0,"last-partition-id":999,"properties":{},"snapshots":[],"snapshot-log":[],"metadata-log":[],"sort-orders":[{"order-id":0,"fields":[]}],"default-sort-order-id":0,"refs":{},"statistics":[],"partition-statistics":[],"format-version":2,"last-sequence-number":0}

package/demo/iceberg/partition/metadata/00001-49f1d5a2-0111-4c83-b71c-6a84ffc1eeb4.metadata.json

@@ -0,0 +1 @@
+{"location":"file:///Users/rock/git/rock2/objectexplorer/reference/data/iceberg/partition","table-uuid":"4ac926be-c3af-4ec5-87a7-bd63348d769c","last-updated-ms":1786970727929,"last-column-id":4,"schemas":[{"type":"struct","fields":[{"id":1,"name":"id","type":"long","required":false},{"id":2,"name":"region","type":"string","required":false},{"id":3,"name":"product","type":"string","required":false},{"id":4,"name":"amount","type":"double","required":false}],"schema-id":0,"identifier-field-ids":[]}],"current-schema-id":0,"partition-specs":[{"spec-id":0,"fields":[]},{"spec-id":1,"fields":[{"source-id":2,"field-id":1000,"transform":"identity","name":"region_part"}]}],"default-spec-id":1,"last-partition-id":1000,"properties":{},"snapshots":[],"snapshot-log":[],"metadata-log":[{"metadata-file":"file:///Users/rock/git/rock2/objectexplorer/reference/data/iceberg/partition/metadata/00000-cde33df7-2164-4f3a-ac15-10aab9c841ac.metadata.json","timestamp-ms":1786970727927}],"sort-orders":[{"order-id":0,"fields":[]}],"default-sort-order-id":0,"refs":{},"statistics":[],"partition-statistics":[],"format-version":2,"last-sequence-number":0}

package/demo/iceberg/partition/metadata/00002-455a595a-b069-4002-b1f6-bd89611f483c.metadata.json

@@ -0,0 +1 @@
+{"location":"file:///Users/rock/git/rock2/objectexplorer/reference/data/iceberg/partition","table-uuid":"4ac926be-c3af-4ec5-87a7-bd63348d769c","last-updated-ms":1786970727936,"last-column-id":4,"schemas":[{"type":"struct","fields":[{"id":1,"name":"id","type":"long","required":false},{"id":2,"name":"region","type":"string","required":false},{"id":3,"name":"product","type":"string","required":false},{"id":4,"name":"amount","type":"double","required":false}],"schema-id":0,"identifier-field-ids":[]}],"current-schema-id":0,"partition-specs":[{"spec-id":0,"fields":[]},{"spec-id":1,"fields":[{"source-id":2,"field-id":1000,"transform":"identity","name":"region_part"}]}],"default-spec-id":1,"last-partition-id":1000,"properties":{},"current-snapshot-id":5440112098897409547,"snapshots":[{"snapshot-id":5440112098897409547,"sequence-number":1,"timestamp-ms":1786970727936,"manifest-list":"file:///Users/rock/git/rock2/objectexplorer/reference/data/iceberg/partition/metadata/snap-5440112098897409547-0-2d48a1ff-55ba-4bae-9c6b-8732d8323bd7.avro","summary":{"operation":"append","added-files-size":"4932","added-data-files":"3","added-records":"4","changed-partition-count":"3","total-data-files":"3","total-delete-files":"0","total-records":"4","total-files-size":"4932","total-position-deletes":"0","total-equality-deletes":"0"},"schema-id":0}],"snapshot-log":[{"snapshot-id":5440112098897409547,"timestamp-ms":1786970727936}],"metadata-log":[{"metadata-file":"file:///Users/rock/git/rock2/objectexplorer/reference/data/iceberg/partition/metadata/00000-cde33df7-2164-4f3a-ac15-10aab9c841ac.metadata.json","timestamp-ms":1786970727927},{"metadata-file":"file:///Users/rock/git/rock2/objectexplorer/reference/data/iceberg/partition/metadata/00001-49f1d5a2-0111-4c83-b71c-6a84ffc1eeb4.metadata.json","timestamp-ms":1786970727929}],"sort-orders":[{"order-id":0,"fields":[]}],"default-sort-order-id":0,"refs":{"main":{"snapshot-id":5440112098897409547,"type":"branch"}},"statistics":[],"partition-statistics":[],"format-version":2,"last-sequence-number":1}

package/demo/iceberg/position/metadata/00000-0c9f7b09-cbc3-4933-b0a8-01d8a809916e.metadata.json

@@ -0,0 +1 @@
+{"location":"file:///Users/rock/git/rock2/objectexplorer/reference/data/iceberg/position","table-uuid":"415dd6b0-de3c-455d-a85d-78a650391aad","last-updated-ms":1786970727968,"last-column-id":4,"schemas":[{"type":"struct","fields":[{"id":1,"name":"id","type":"long","required":false},{"id":2,"name":"region","type":"string","required":false},{"id":3,"name":"product","type":"string","required":false},{"id":4,"name":"amount","type":"double","required":false}],"schema-id":0,"identifier-field-ids":[]}],"current-schema-id":0,"partition-specs":[{"spec-id":0,"fields":[]}],"default-spec-id":0,"last-partition-id":999,"properties":{},"snapshots":[],"snapshot-log":[],"metadata-log":[],"sort-orders":[{"order-id":0,"fields":[]}],"default-sort-order-id":0,"refs":{},"statistics":[],"partition-statistics":[],"format-version":2,"last-sequence-number":0}

package/demo/iceberg/position/metadata/00001-8fc1224a-057c-4848-9c74-9c4e592c3545.metadata.json

@@ -0,0 +1 @@
+{"location":"file:///Users/rock/git/rock2/objectexplorer/reference/data/iceberg/position","table-uuid":"415dd6b0-de3c-455d-a85d-78a650391aad","last-updated-ms":1786970727972,"last-column-id":4,"schemas":[{"type":"struct","fields":[{"id":1,"name":"id","type":"long","required":false},{"id":2,"name":"region","type":"string","required":false},{"id":3,"name":"product","type":"string","required":false},{"id":4,"name":"amount","type":"double","required":false}],"schema-id":0,"identifier-field-ids":[]}],"current-schema-id":0,"partition-specs":[{"spec-id":0,"fields":[]}],"default-spec-id":0,"last-partition-id":999,"properties":{},"current-snapshot-id":8554796374254190464,"snapshots":[{"snapshot-id":8554796374254190464,"sequence-number":1,"timestamp-ms":1786970727972,"manifest-list":"file:///Users/rock/git/rock2/objectexplorer/reference/data/iceberg/position/metadata/snap-8554796374254190464-0-8d1a982c-c1b7-47d9-a584-de646ada36b6.avro","summary":{"operation":"append","added-files-size":"1716","added-data-files":"1","added-records":"4","total-data-files":"1","total-delete-files":"0","total-records":"4","total-files-size":"1716","total-position-deletes":"0","total-equality-deletes":"0"},"schema-id":0}],"snapshot-log":[{"snapshot-id":8554796374254190464,"timestamp-ms":1786970727972}],"metadata-log":[{"metadata-file":"file:///Users/rock/git/rock2/objectexplorer/reference/data/iceberg/position/metadata/00000-0c9f7b09-cbc3-4933-b0a8-01d8a809916e.metadata.json","timestamp-ms":1786970727968}],"sort-orders":[{"order-id":0,"fields":[]}],"default-sort-order-id":0,"refs":{"main":{"snapshot-id":8554796374254190464,"type":"branch"}},"statistics":[],"partition-statistics":[],"format-version":2,"last-sequence-number":1}

package/demo/iceberg/position/metadata/00002-delete.metadata.json

@@ -0,0 +1,115 @@
+{
+  "location": "file:///Users/rock/git/rock2/objectexplorer/reference/data/iceberg/position",
+  "table-uuid": "415dd6b0-de3c-455d-a85d-78a650391aad",
+  "last-updated-ms": 1786970727972,
+  "last-column-id": 4,
+  "schemas": [
+    {
+      "type": "struct",
+      "fields": [
+        {
+          "id": 1,
+          "name": "id",
+          "type": "long",
+          "required": false
+        },
+        {
+          "id": 2,
+          "name": "region",
+          "type": "string",
+          "required": false
+        },
+        {
+          "id": 3,
+          "name": "product",
+          "type": "string",
+          "required": false
+        },
+        {
+          "id": 4,
+          "name": "amount",
+          "type": "double",
+          "required": false
+        }
+      ],
+      "schema-id": 0,
+      "identifier-field-ids": []
+    }
+  ],
+  "current-schema-id": 0,
+  "partition-specs": [
+    {
+      "spec-id": 0,
+      "fields": []
+    }
+  ],
+  "default-spec-id": 0,
+  "last-partition-id": 999,
+  "properties": {},
+  "current-snapshot-id": 8554796374254190465,
+  "snapshots": [
+    {
+      "snapshot-id": 8554796374254190464,
+      "sequence-number": 1,
+      "timestamp-ms": 1786970727972,
+      "manifest-list": "file:///Users/rock/git/rock2/objectexplorer/reference/data/iceberg/position/metadata/snap-8554796374254190464-0-8d1a982c-c1b7-47d9-a584-de646ada36b6.avro",
+      "summary": {
+        "operation": "append",
+        "added-files-size": "1716",
+        "added-data-files": "1",
+        "added-records": "4",
+        "total-data-files": "1",
+        "total-delete-files": "0",
+        "total-records": "4",
+        "total-files-size": "1716",
+        "total-position-deletes": "0",
+        "total-equality-deletes": "0"
+      },
+      "schema-id": 0
+    },
+    {
+      "snapshot-id": 8554796374254190465,
+      "parent-snapshot-id": 8554796374254190464,
+      "sequence-number": 2,
+      "timestamp-ms": 1786970728972,
+      "manifest-list": "file:///Users/rock/git/rock2/objectexplorer/reference/data/iceberg/position/metadata/snap-8554796374254190465-0-delete.avro",
+      "summary": {
+        "operation": "delete"
+      },
+      "schema-id": 0
+    }
+  ],
+  "snapshot-log": [
+    {
+      "snapshot-id": 8554796374254190464,
+      "timestamp-ms": 1786970727972
+    },
+    {
+      "snapshot-id": 8554796374254190465,
+      "timestamp-ms": 1786970728972
+    }
+  ],
+  "metadata-log": [
+    {
+      "metadata-file": "file:///Users/rock/git/rock2/objectexplorer/reference/data/iceberg/position/metadata/00000-0c9f7b09-cbc3-4933-b0a8-01d8a809916e.metadata.json",
+      "timestamp-ms": 1786970727968
+    }
+  ],
+  "sort-orders": [
+    {
+      "order-id": 0,
+      "fields": []
+    }
+  ],
+  "default-sort-order-id": 0,
+  "refs": {
+    "main": {
+      "snapshot-id": 8554796374254190464,
+      "type": "branch"
+    }
+  },
+  "statistics": [],
+  "partition-statistics": [],
+  "format-version": 2,
+  "last-sequence-number": 2
+}

package/demo/iceberg/sales/metadata/00000-82dd52ff-dd85-4033-b71a-c16358a1926e.metadata.json

@@ -0,0 +1 @@
+{"location":"file:///Users/rock/git/rock2/objectexplorer/reference/data/iceberg/sales","table-uuid":"4db7af6d-b75e-4376-9a52-9a026c046de5","last-updated-ms":1786970727853,"last-column-id":4,"schemas":[{"type":"struct","fields":[{"id":1,"name":"id","type":"long","required":false},{"id":2,"name":"region","type":"string","required":false},{"id":3,"name":"product","type":"string","required":false},{"id":4,"name":"amount","type":"double","required":false}],"schema-id":0,"identifier-field-ids":[]}],"current-schema-id":0,"partition-specs":[{"spec-id":0,"fields":[]}],"default-spec-id":0,"last-partition-id":999,"properties":{},"snapshots":[],"snapshot-log":[],"metadata-log":[],"sort-orders":[{"order-id":0,"fields":[]}],"default-sort-order-id":0,"refs":{},"statistics":[],"partition-statistics":[],"format-version":2,"last-sequence-number":0}

package/demo/iceberg/sales/metadata/00001-357f9784-8695-4b85-8f06-fd34b1785c06.metadata.json

@@ -0,0 +1 @@
+{"location":"file:///Users/rock/git/rock2/objectexplorer/reference/data/iceberg/sales","table-uuid":"4db7af6d-b75e-4376-9a52-9a026c046de5","last-updated-ms":1786970727885,"last-column-id":4,"schemas":[{"type":"struct","fields":[{"id":1,"name":"id","type":"long","required":false},{"id":2,"name":"region","type":"string","required":false},{"id":3,"name":"product","type":"string","required":false},{"id":4,"name":"amount","type":"double","required":false}],"schema-id":0,"identifier-field-ids":[]}],"current-schema-id":0,"partition-specs":[{"spec-id":0,"fields":[]}],"default-spec-id":0,"last-partition-id":999,"properties":{},"current-snapshot-id":7501977550537779287,"snapshots":[{"snapshot-id":7501977550537779287,"sequence-number":1,"timestamp-ms":1786970727885,"manifest-list":"file:///Users/rock/git/rock2/objectexplorer/reference/data/iceberg/sales/metadata/snap-7501977550537779287-0-16494fa7-6997-413a-83c6-bfc6d53f3da5.avro","summary":{"operation":"append","added-files-size":"1716","added-data-files":"1","added-records":"4","total-data-files":"1","total-delete-files":"0","total-records":"4","total-files-size":"1716","total-position-deletes":"0","total-equality-deletes":"0"},"schema-id":0}],"snapshot-log":[{"snapshot-id":7501977550537779287,"timestamp-ms":1786970727885}],"metadata-log":[{"metadata-file":"file:///Users/rock/git/rock2/objectexplorer/reference/data/iceberg/sales/metadata/00000-82dd52ff-dd85-4033-b71a-c16358a1926e.metadata.json","timestamp-ms":1786970727853}],"sort-orders":[{"order-id":0,"fields":[]}],"default-sort-order-id":0,"refs":{"main":{"snapshot-id":7501977550537779287,"type":"branch"}},"statistics":[],"partition-statistics":[],"format-version":2,"last-sequence-number":1}