@sixb/ducklake 0.1.8 → 0.2.0

package/README.md

@@ -86,9 +86,15 @@ Or run it directly against the provider:
 const report = await lakeStorage.runMaintenance({
   dryRun: true,
   expireOlderThan: "7 days",
+  protect: [{ datasetId: "invoices", versionId: "ducklake:42" }],
 })
 ```
 
+Expiring a snapshot removes the version it holds. Two kinds of versions are kept whatever their
+age, and the snapshots holding them are reported as `keptSnapshots`: each dataset's latest
+version, so a dataset idle longer than the retention stays readable, and the versions passed as
+`protect`. `sixb lake cleanup` protects the version each projection applied.
+
 ### Read and Write Concurrency
 
 Large reads stream through dedicated connections in the same DuckDB engine, sharing its memory
@@ -193,12 +199,36 @@ Explicit `expectedLatestVersionId` guards are never relaxed.
 See [source ordering](../../docs/datasets/source-ordering.md).
 
 Local catalogs serialize commits within one process. PostgreSQL catalogs support concurrent
-writer processes; known snapshot-ID conflicts are retried from the retained staging table.
-Other provider/connection failures propagate without an automatic retry.
+writer processes. DuckLake commits a transaction on top of versions committed since it began
+unless one of them altered the table or deleted rows it touches, so a write that missed a
+concurrent version could keep that version's rows but record what it read before them. Such a
+version claims its table, an idempotent comment update that makes any later concurrent write
+conflict:
+
+- a version whose lineage carries dependencies on other datasets, or is unknown, so no later
+  write records a lineage without them;
+- a keyed append or merge, because DuckLake has no primary key and two writes of one new key
+  would both commit;
+- a write into an empty dataset, because a snapshot that also read it empty would keep its rows;
+- a version that changes no rows (new provenance), which also gives it a DuckLake snapshot.
+
+Each claim bumps DuckLake's catalog-wide schema version, which every connection then reloads. On
+PostgreSQL with two writers appending 100 rows each, claiming every commit cut throughput by a
+quarter to a third in our measurements, writers to two different datasets included. Snapshots that
+replace rows conflict through their deletes and do not claim, nor do unkeyed appends to a dataset
+that holds only its own columns, such as a sync target: those commit concurrently.
+
+A version that changes no rows, a schema change, a merge that only advances source ordering, and
+an unkeyed append guarded on `expectedLatestVersionId` also fail on a concurrent commit to any
+dataset of the catalog. The writer that loses reruns its whole commit from the retained staging
+table: up to 16 attempts, the first rerun at once, later ones after a random wait in a window
+that doubles from 20 ms up to 1 s (at most 9.3 s of waiting in all). A merge's
+`commit({ signal })` also stops the wait, and a writer that runs out of attempts fails with
+`LakeConcurrencyError`. Other provider/connection failures propagate without an automatic retry.
 
 SQLite reads commit metadata before the write transaction, then verifies the native snapshot
-has not changed. Preparation retries up to three times on a race; data writes are not replayed.
-The catalog's existing journal mode is preserved.
+has not changed; a race reruns the commit the same way, and data writes are applied once per
+attempt. The catalog's existing journal mode is preserved.
 
 Snapshot, append, and SQL-transform writes to keyed datasets also enforce unique keys so a later
 merge never starts from an ambiguous baseline. Application authors normally use
@@ -284,6 +314,31 @@ new rows to the existing target. Omitting a source `versionId` resolves the late
 committed version at execution time; providing a `versionId` pins that source for
 repeatable transforms.
 
+A source can pick the columns a transform may read with `columns`. The relation then exposes only
+those columns, in that order, and referencing any other column fails:
+
+```ts
+sources: {
+  orders: { dataset: rawOrdersDataset, columns: ["customerId", "total"] },
+}
+```
+
+The committed version records its lineage in `DatasetVersion.lineage`: which columns each output
+column was computed from, and which ones decide the result's rows and their order, traced through
+what the source versions read record themselves. An append also keeps the lineage of the rows it
+keeps. `analyze(...)` returns the SQL's own lineage, by source name, without reading any data or
+requiring a committed source version. The provider binds the SQL in an isolated DuckDB instance
+against empty tables shaped like the sources. SQL it cannot analyze exactly, such as `PIVOT`, still
+executes. When its parse tree reads nothing but the sources and calls only functions by names DuckDB
+defines, every column the sources expose is recorded as deciding its rows, so each output column
+carries the markings of all of them. A `PIVOT` needs an explicit `IN` list for that.
+
+Lineage only covers the sources. SQL that may read anything else, such as a lake table by its name,
+a table function that reads files or other tables (`read_parquet`, `read_csv`, `query_table`), or a
+macro the lake's setup defines, records no lineage for a pipeline version, and an unknown lineage
+for any other one. Either way, the version is classified with every marking. Read data only through
+the sources.
+
 DuckDB promotes `sum(BIGINT)` results to `HUGEINT`. When the matching target column is a Sixb
 `int64`, the executor applies a checked conversion back to `BIGINT` before comparing or writing
 rows. Values outside the signed 64-bit range fail the transform instead of being truncated.

package/dist/ducklake-storage.d.ts

@@ -14,6 +14,7 @@ export declare class DuckLakeStorage implements LakeStorageWithSql<"duckdb"> {
         readonly id: "ducklake";
         readonly version: "1.0";
     };
+    readonly recordsLineage = true;
     readonly sql: DuckLakeSqlExecutor;
     private readonly connections;
     private readonly datasets;

package/dist/ducklake-storage.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"ducklake-storage.d.ts","sourceRoot":"","sources":["../src/ducklake-storage.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,iBAAiB,EAAE,UAAU,EAAE,kBAAkB,EAAE,MAAM,YAAY,CAAA;AACnF,OAAO,KAAK,EACV,sBAAsB,EACtB,sBAAsB,EACtB,mBAAmB,EACnB,cAAc,EACd,cAAc,EACd,gBAAgB,EAChB,gBAAgB,EAChB,mBAAmB,EACnB,4BAA4B,EAC5B,uBAAuB,EACvB,oBAAoB,EACrB,MAAM,yBAAyB,CAAA;AAOhC,OAAO,EAAE,mBAAmB,EAAE,MAAM,kCAAkC,CAAA;AAEtE,OAAO,KAAK,EACV,0BAA0B,EAC1B,yBAAyB,EACzB,sBAAsB,EACvB,MAAM,SAAS,CAAA;AAEhB;;;;;;GAMG;AACH,qBAAa,eAAgB,YAAW,kBAAkB,CAAC,QAAQ,CAAC;IAClE,QAAQ,CAAC,QAAQ;;;MAA8C;IAC/D,QAAQ,CAAC,GAAG,EAAE,mBAAmB,CAAA;IAEjC,OAAO,CAAC,QAAQ,CAAC,WAAW,CAA2B;IACvD,OAAO,CAAC,QAAQ,CAAC,QAAQ,CAAwB;IACjD,OAAO,CAAC,QAAQ,CAAC,WAAW,CAAqB;IACjD,OAAO,CAAC,QAAQ,CAAC,IAAI,CAAmB;IACxC,OAAO,CAAC,QAAQ,CAAC,OAAO,CAAsB;IAC9C,OAAO,CAAC,QAAQ,CAAC,cAAc,CAAwB;IACvD,OAAO,CAAC,QAAQ,CAAC,MAAM,CAA0B;gBAErC,OAAO,EAAE,sBAAsB;IAwCrC,aAAa,CAAC,UAAU,EAAE,iBAAiB,GAAG,OAAO,CAAC,iBAAiB,CAAC;IAIxE,kCAAkC,CACtC,WAAW,EAAE,SAAS,iBAAiB,EAAE,GACxC,OAAO,CAAC,IAAI,CAAC;IAIV,UAAU,CAAC,SAAS,EAAE,MAAM,GAAG,OAAO,CAAC,iBAAiB,GAAG,IAAI,CAAC;IAIhE,YAAY,IAAI,OAAO,CAAC,SAAS,iBAAiB,EAAE,CAAC;IAIrD,uBAAuB,CAC3B,UAAU,EAAE,SAAS,MAAM,EAAE,GAC5B,OAAO,CAAC,SAAS,mBAAmB,EAAE,CAAC;IAIpC,YAAY,CAAC,SAAS,EAAE,MAAM,EAAE,KAAK,CAAC,EAAE,MAAM,GAAG,OAAO,CAAC,SAAS,cAAc,EAAE,CAAC;IAInF,UAAU,CAAC,KAAK,EAAE,sBAAsB,GAAG,OAAO,CAAC,gBAAgB,CAAC;IAIpE,UAAU,CAAC,KAAK,EAAE,sBAAsB,GAAG,OAAO,CAAC,gBAAgB,CAAC;IAIpE,gBAAgB,CAAC,SAAS,EAAE,MAAM,GAAG,OAAO,CAAC,cAAc,GAAG,IAAI,CAAC;IAInE,UAAU,CAAC,SAAS,EAAE,MAAM,EAAE,SAAS,EAAE,MAAM,GAAG,OAAO,CAAC,cAAc,GAAG,IAAI,CAAC;IAIhF,uBAAuB,CAC3B,KAAK,EAAE,4BAA4B,GAClC,OAAO,CAAC,mBAAmB,GAAG,IAAI,CAAC;IAItC,QAAQ,CAAC,KAAK,EAAE,oBAAoB,GAAG,aAAa,CAAC,UAAU,CAAC;IAIhE,WAAW,CAAC,KAAK,EAAE,uBAAuB,GAAG,OAAO,CAAC,cAAc,GAAG,IAAI,CAAC;IAIrE,cAAc,CAAC,OAAO,CAAC,EAAE,0BAA0B,GAAG,OAAO,CAAC,yBAAyB,CAAC;IAI9F;;;OAGG;IACG,KAAK,IAAI,OAAO,CAAC,IAAI,CAAC;CAG7B"}
+{"version":3,"file":"ducklake-storage.d.ts","sourceRoot":"","sources":["../src/ducklake-storage.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,iBAAiB,EAAE,UAAU,EAAE,kBAAkB,EAAE,MAAM,YAAY,CAAA;AACnF,OAAO,KAAK,EACV,sBAAsB,EACtB,sBAAsB,EACtB,mBAAmB,EACnB,cAAc,EACd,cAAc,EACd,gBAAgB,EAChB,gBAAgB,EAChB,mBAAmB,EACnB,4BAA4B,EAC5B,uBAAuB,EACvB,oBAAoB,EACrB,MAAM,yBAAyB,CAAA;AAOhC,OAAO,EAAE,mBAAmB,EAAE,MAAM,kCAAkC,CAAA;AAEtE,OAAO,KAAK,EACV,0BAA0B,EAC1B,yBAAyB,EACzB,sBAAsB,EACvB,MAAM,SAAS,CAAA;AAEhB;;;;;;GAMG;AACH,qBAAa,eAAgB,YAAW,kBAAkB,CAAC,QAAQ,CAAC;IAClE,QAAQ,CAAC,QAAQ;;;MAA8C;IAC/D,QAAQ,CAAC,cAAc,QAAO;IAC9B,QAAQ,CAAC,GAAG,EAAE,mBAAmB,CAAA;IAEjC,OAAO,CAAC,QAAQ,CAAC,WAAW,CAA2B;IACvD,OAAO,CAAC,QAAQ,CAAC,QAAQ,CAAwB;IACjD,OAAO,CAAC,QAAQ,CAAC,WAAW,CAAqB;IACjD,OAAO,CAAC,QAAQ,CAAC,IAAI,CAAmB;IACxC,OAAO,CAAC,QAAQ,CAAC,OAAO,CAAsB;IAC9C,OAAO,CAAC,QAAQ,CAAC,cAAc,CAAwB;IACvD,OAAO,CAAC,QAAQ,CAAC,MAAM,CAA0B;gBAErC,OAAO,EAAE,sBAAsB;IA4CrC,aAAa,CAAC,UAAU,EAAE,iBAAiB,GAAG,OAAO,CAAC,iBAAiB,CAAC;IAMxE,kCAAkC,CACtC,WAAW,EAAE,SAAS,iBAAiB,EAAE,GACxC,OAAO,CAAC,IAAI,CAAC;IAIV,UAAU,CAAC,SAAS,EAAE,MAAM,GAAG,OAAO,CAAC,iBAAiB,GAAG,IAAI,CAAC;IAIhE,YAAY,IAAI,OAAO,CAAC,SAAS,iBAAiB,EAAE,CAAC;IAIrD,uBAAuB,CAC3B,UAAU,EAAE,SAAS,MAAM,EAAE,GAC5B,OAAO,CAAC,SAAS,mBAAmB,EAAE,CAAC;IAIpC,YAAY,CAAC,SAAS,EAAE,MAAM,EAAE,KAAK,CAAC,EAAE,MAAM,GAAG,OAAO,CAAC,SAAS,cAAc,EAAE,CAAC;IAInF,UAAU,CAAC,KAAK,EAAE,sBAAsB,GAAG,OAAO,CAAC,gBAAgB,CAAC;IAIpE,UAAU,CAAC,KAAK,EAAE,sBAAsB,GAAG,OAAO,CAAC,gBAAgB,CAAC;IAIpE,gBAAgB,CAAC,SAAS,EAAE,MAAM,GAAG,OAAO,CAAC,cAAc,GAAG,IAAI,CAAC;IAInE,UAAU,CAAC,SAAS,EAAE,MAAM,EAAE,SAAS,EAAE,MAAM,GAAG,OAAO,CAAC,cAAc,GAAG,IAAI,CAAC;IAIhF,uBAAuB,CAC3B,KAAK,EAAE,4BAA4B,GAClC,OAAO,CAAC,mBAAmB,GAAG,IAAI,CAAC;IAItC,QAAQ,CAAC,KAAK,EAAE,oBAAoB,GAAG,aAAa,CAAC,UAAU,CAAC;IAIhE,WAAW,CAAC,KAAK,EAAE,uBAAuB,GAAG,OAAO,CAAC,cAAc,GAAG,IAAI,CAAC;IAIrE,cAAc,CAAC,OAAO,CAAC,EAAE,0BAA0B,GAAG,OAAO,CAAC,yBAAyB,CAAC;IAI9F;;;OAGG;IACG,KAAK,IAAI,OAAO,CAAC,IAAI,CAAC;CAG7B"}