@eltjs/elt-sqlite 0.1.3 → 0.3.0

package/README.md

@@ -33,12 +33,15 @@ Inferred columns map flat JSON Schema fields:
 Tables:
 
 - SQLite makes strict tables. SQL constraints that exist stay in force.
-- Each copy adds `loaded_at`, the UTC start time of the load. A field with that name gets a different column name.
+- Each copy adds `loaded_at`, the UTC start time of the load that wrote the values of the row. A field with that name gets a different column name.
+- Each copy also adds `committed_in`, the number of the [commit](#commit-numbers-and-positions) that last wrote the row, and `_elt_row_hash`, the SHA-256 of the values of its columns. A [file in the SQLite file](#file-bytes-in-the-database) counts by the SHA-256 of its bytes, not by its id. A field with one of these names also gets a different column name. So does a field with the name `_rowid_`: SQLite reads that name as the id of each row, which an overwrite and a reload use.
+- When a deduplicating load, an overwrite or a reload brings a row with the same values as a stored row, it leaves the stored row as it is, with its `loaded_at` and its `committed_in`. A deduplicating load finds the stored row by its key. An overwrite and a reload find it by [difference](#commit-numbers-and-positions), also in a table with no key.
 - Deduplication upserts on a unique index that is named after the table and the key (`_elt_dedup_<hash>`). SQLite makes the index one time, and makes it again only when the key changes. It uses the native [UPSERT with a cursor comparison](https://www.sqlite.org/lang_upsert.html).
 - Deduplication also checks the stored types of the key and cursor columns, and refuses null keys and cursors.
 - A change to a normal `append` or `overwrite` removes the index of the mode. Explicit constraints stay.
 - A table of append history with repeated keys must be replaced with `overwrite_dedup` before incremental deduplication can start.
 - When the stored definition of a table does not agree with the columns of the stream, the load [evolves it](../../elt/README.md#shape-changes) before it reads. SQLite cannot change the type or the constraints of a column in place. Thus, in one transaction, the rows move, column by name, into a table of the new definition. That table takes the name of the table, with its index and its reader view. A value that its type or a `CHECK` refuses fails the load, and the table stays as it was.
+- A row that has a value in a column that the table drops, or that it keeps as a different type, loses or changes that value. Thus the evolve takes a commit number of the load, the [history](#commit-numbers-and-positions) keeps each such row as it was, and the row gets the number in `committed_in`. Its `_elt_row_hash` counted the old value, so it becomes null: a null hash matches no record, and the next commit that writes the row gives it a new version. A column that the table adds is null in each row, and a null value does not count in `_elt_row_hash`. Thus it changes no row.
 - An overwrite reloads such a table into a hidden table `_elt_next_<hash>`. The hidden table replaces the table and its reader view when the stream ends ([resets](../../elt/README.md#resets)).
 - The writer of each target is in the reserved table `_elt_writers` ([target ownership](../../elt/README.md#target-ownership)).
 
@@ -54,12 +57,15 @@ Counts:
 
 SQLite stores an original file in chunks, because one BLOB has a limit of 1,000,000,000 bytes (`SQLITE_MAX_LENGTH` in the build of Node), and a value of the full file would stay in memory.
 
-- The `bytes` column holds an `INTEGER` file id.
+- The `bytes` column holds an `INTEGER` file id. Each load gives each file a new id.
 - A table `_elt_files_<hash>` for each column holds `(file, n, bytes)` rows of up to 4 MiB. `n` counts from 0.
+- The ids of a chunk table only go up. The reserved table `_elt_file_ids` keeps the highest id that each chunk table gave, as `sqlite_sequence` keeps the highest rowid of a table with `AUTOINCREMENT`. Thus a new file never takes the id of a file that a row of the [history](#commit-numbers-and-positions) refers to, also when the chunk table was dropped and made again.
 - The hash is the first 40 hex characters of the SHA-256 of `JSON.stringify([table, column])`, with the two names in ASCII lower case. Thus table `a_b` with column `c` and table `a` with column `b_c` keep their chunks apart.
 - The description of the column names its chunk table.
 - An empty file has one empty chunk. A file that is not available gives a null value.
-- Triggers remove the chunks of a row in the same transaction when the row is deleted, overwritten or replaced. A record that a deduplication guard refuses stores no chunks.
+- The chunks of a file stay while a row of the table or of its history refers to the file. Thus a [reading](#commit-numbers-and-positions) as of a position reads the bytes of each file that the position shows. When the [retention](#retention) deletes the last history row that refers to a file, the same commit deletes its chunks.
+- A commit deletes the chunks of each file that it staged and that no row of the table or of its history refers to. Thus a record that a deduplication guard refuses stores no chunks. The commit that resets a partition, the commit at the end of a reload, and a clear delete the chunks of each file that no row of the table or of its history refers to.
+- When a row with a file has the same values as a stored row, also the same file bytes, the stored row stays with its file id. The commit deletes the chunks of the new copy of the file.
 
 Read a file back in order, with the chunk table that the description names:
 
@@ -81,14 +87,58 @@ Checked on 2026-09-24 (macOS 26.6.2, Node.js 26.8.1): a file of 1.5 GB loaded as
 - SQLite takes no lock other than its transactions. Loads into one file run side by side. A [checkpoint store](#checkpoints) keeps two runs of one copy apart.
 - No transaction stays open between commits, as in Postgres.
 - `prepare` only refuses and stages. It refuses the target of a different writer, a reader view that is not its own, and stored rows that a deduplicating load cannot keep.
-- Each commit is one write transaction. The first commit also makes the table, the index of its mode, its reader view, its descriptions and its writer.
+- Each commit is one write transaction. The first commit also makes the table, the index of its mode, its reader view, its descriptions and its writer. A full refresh commits one time, when its stream ends. Thus an overwrite replaces the rows of the table in that one commit, and a reader never sees a part of it.
 - Other writers of the file wait only for a commit. A stream that fails before its first commit leaves nothing.
 - Each stream stages its operations in a `TEMP` table that only its connection sees. The commit of a different stream never publishes them, and a crash leaves nothing.
 - A commit merges the staged operations of one stream into its table. The result is the same as when the operations apply one at a time: a staged `DELETE` removes its key, only the records after the last `DELETE` of a key count, `replace` keeps the last record, and `cursor_newer` keeps the first record with the greatest cursor.
-- Original files stream into a `TEMP` chunk table that only the connection sees, and they get to the store in the commit of their rows. After each merge, SQLite deletes the chunks that no row refers to. A clear empties the store.
-- A file column that the target drops takes its chunk table with it when the table evolves or a reload replaces it.
+- Original files stream into a `TEMP` chunk table that only the connection sees, and they get to the store in the commit of their rows.
+- A file column that the target drops takes its chunk table with it when the table evolves or a reload replaces it. A reading as of a position reads only the columns of the table, so it does not read that column.
 - `:memory:` is possible only for a full refresh. The database of the run is gone when its handle closes, so it cannot keep incremental progress.
 
+## Commit numbers and positions
+
+Each commit in the SQLite file has a number. A reader that compares the file at two times can find the positions where no pass is half done, as a Kafka consumer reads up to the last stable offset.
+
+- The table `_elt_commits` has one row for each commit: its `number`, its `load` and its `committed_at`. SQLite writes one transaction at a time, so the numbers follow the order of the commits, also across connections.
+- The table `_elt_loads` has one row for each load: its `connector` (the name of the connection) or its `dataset` (the name of a [computed dataset](#declare-a-computed-dataset)), its `opened_at`, and its `closed_at`, which is null while the load is open. A load of a connection opens when a pass starts to load, and closes when the pass ends. A load of a computed dataset opens when a refresh starts, and closes when the refresh ends, also when a stop or a failure ends it.
+- A connection runs one pass at a time, and a computed dataset runs one refresh at a time. Thus when a load opens, it closes each load of the same connection or dataset that is still open: the process of that pass or refresh ended first. A closed load cannot commit. A load with no commit changes no position.
+- A commit into a [hidden table](./CONTEXT.md) shows nothing to readers. When the stream ends, one more commit gives its rows to the table, and readers see them from that commit on.
+- An overwrite and a reload replace the rows of the table by [difference](./CONTEXT.md). The difference compares the rows as multisets, so it needs no key. On each side, each row gets a number k among the rows with the same `_elt_row_hash`, and the k-th copy of some values matches the k-th copy of the same values on the other side:
+  - A row with a match stays as it is, with its `committed_in` and its `loaded_at`.
+  - A row of the table with no match goes into the history and leaves the table.
+  - A new row with no match enters the table with the number of the commit.
+- An overwrite applies the difference in its one commit, to the records that it staged. A reload applies it, to the rows of its hidden table, in the commit at the end of the stream. The table keeps its indexes, its triggers and its reader view. When the table has a different shape than the hidden table, the reload puts the hidden table in its place, and every row of the table goes into the history.
+- A file in the SQLite file counts in `_elt_row_hash` by the SHA-256 of its bytes, which the load calculates while it stages the chunks. Each load gives the file a new id, but a row whose file has the same bytes matches. The row keeps its file id, and the commit deletes the chunks of the new copy.
+- The table `_elt_epoch` keeps the epoch, an id that names the numbering. A file that is made again, or tables that a different version declared in a different way, start a new epoch.
+- `destination.positions(after?)` gives the positions after the position `after`, in ascending order, or all positions. A position is `<epoch>:<number>`. It names the last commit of a closed load, when no other load has commits before and after it. With no `after`, it gives each position from [min valid](#retention) on. When `after` is of a different epoch, is after the newest commit, or is older than min valid, `positions` gives `{ expired }` with the reason. It opens the file read-only.
+- A clear that empties a table is a load of one commit. It does not close the open loads of its connection.
+- Each table has a [history](./CONTEXT.md) `_elt_history_<hash>`, which the table `_elt_histories` names. Before a commit changes or removes a row, it copies the row into the history, with `replaced_in`, its own number. This occurs for a deduplicated row with different values, a `DELETE`, the reset of a partition, a clear, a row that an overwrite or a reload does not bring again, the rows of a table that a reload of a different shape replaces, a row that loses or changes a value when its table evolves, and an output row of a computed dataset that a refresh changes or deletes. A commit into a hidden table copies nothing. The history has each column that the table had, and it does not drop a column. A field with the name `replaced_in` gets a different column name.
+- A file that a row of the history refers to stays, in the SQLite file and in a [file storage](../../elt/README.md#local-files): the values that the destination gives the file storage to keep include the values of the history.
+- The history gets the columns of the table when it is made. SQLite cannot add a column with the name `rowid` or `oid` to a table. Thus when a stream adds a field with one of these names later, a commit that keeps a row in the history fails. Delete the SQLite file to load it again.
+- The reset of a partition copies each row of the partition into the history. The rows that the source sends again after the reset can come in later commits, for example after a checkpoint of each page. Thus no commit can know which rows of the partition left.
+- `destination.reading(position?)` gives a [reading](./CONTEXT.md) of the file as of the position, or as of the newest position: `{ sql, position, at }`, where `sql` is a read-only `DatabaseSync` in one read transaction and `at` is the time of the commit. Dispose the reading to close it. A position of a different epoch, after the newest commit, or older than [min valid](#retention), gives `{ expired }`. A commit number that `positions` does not list, for example a commit inside a pass, makes `reading` throw a `TypeError`, as a string that is not a position does: a reading there would show a part of a pass. A position that `positions` listed stays a position, because no later load can have commits before and after it.
+- As of a position before the newest commit, each table, its reader view and each view of the file show the rows that the commit of the position left. A column that a later commit added is `NULL`. The sync history keeps no versions, and a read of it fails. A stored view that names a table as `main.<table>` makes the reading fail, because that name shows the table as it is now.
+
+- `asOfSelect(database, table, position, { schema })` gives the `SELECT` that reads one table as of a position: the rows of the table with `committed_in ≤ P`, and the rows of its history with `committed_in ≤ P < replaced_in`. A reading uses the same `SELECT` for its `TEMP` views. `schema` names the SQLite file where `database` attached it, `main` by default. Thus a reader can copy a table as of a position into a database of its own. It checks the position as `reading` does, so run the `SELECT` in the same read transaction. See [Read the file as of a position](./docs/recipes/read-the-file-as-of-a-position.md).
+
+### Retention
+
+The history keeps each position readable for the retention of the destination, as Delta Lake keeps its log:
+
+```ts
+new SQLiteDestination({
+  path: 'import.sqlite',
+  retention: Temporal.Duration.from({ days: 30 }), // the default
+});
+```
+
+- In each commit, let W be the newest position whose commit is older than now minus the retention. The commit deletes each history row with `replaced_in ≤ W`, and the chunks of each file that only those rows refer to. It sets min valid, the column `min_valid` of `_elt_epoch`, to W. This is the `MIN_VALID_VERSION` of SQL Server change tracking.
+- Min valid is a position, and it never passes the newest position. Thus a pass that stays open longer than the retention does not expire the position before it.
+- A position older than min valid is expired: `reading`, `positions` and `asOfSelect` give `{ expired }` with a reason that names the retention.
+- A file in a [file storage](../../elt/README.md#local-files) that only the deleted history rows refer to goes when the stream of its table next commits.
+- `_elt_commits` keeps its rows.
+- A retention that is less than zero throws a `RangeError`. A retention in months or years throws a `RangeError` of `Temporal`, because their length depends on the date.
+
 ## Checkpoints
 
 `SQLiteCheckpointStore({ path })` keeps the checkpoints of incremental copies in a table `checkpoints` (`id`, `binding`, `reloading`, `state`) in its own file. Use it with SQLite destinations.
@@ -112,7 +162,7 @@ Checked on 2026-09-24 (macOS 26.6.2, Node.js 26.8.1): a file of 1.5 GB loaded as
 
 ## Reserved names
 
-The destination and its sync history own each table, index and view whose name starts with `_elt_`, in any case. Examples are `_elt_writers`, `_elt_descriptions`, the chunk tables `_elt_files_<hash>` and the sync history's `_elt_sync_attempts`. A stream, a view or a computed dataset never has such a name, and `publishSQLiteViews` refuses one. A reader that follows the data of a file, such as a changefeed, skips these tables with `isReserved(name)`:
+The destination and its sync history own each table, index and view whose name starts with `_elt_`, in any case. Examples are `_elt_writers`, `_elt_descriptions`, `_elt_commits`, the chunk tables `_elt_files_<hash>` and the sync history's `_elt_sync_attempts`. A stream, a view or a computed dataset never has such a name, and `publishSQLiteViews` refuses one. A reader that follows the data of a file, such as a changefeed, skips these tables with `isReserved(name)`:
 
 ```ts
 import { isReserved } from '@eltjs/elt-sqlite';
@@ -194,10 +244,18 @@ At each refresh, the computed dataset compares the inputs with the output rows.
 - Each call of the rule commits as it settles, so readers see the output rows as they land. See [ADR 0003](./docs/adr/0003-each-rule-call-commits-as-it-settles.md).
 - When the declaration of the raw table changes, the refresh makes the table again. See [ADR 0004](./docs/adr/0004-a-changed-declaration-makes-the-table-again.md).
 
+A refresh is a load in `_elt_loads`, so a [reading](#commit-numbers-and-positions) shows the computed dataset as of a position, as it shows a table that a copy loads:
+
+- Each transaction of the refresh that changes a row takes one commit number. A transaction that changes no row takes none. No commit inside a refresh that is open is a position.
+- The raw table has the columns `committed_in` and `_elt_row_hash` after the columns of the rule. The view `<name>` does not show them. `_elt_row_hash` is the SHA-256 of the values of the other columns, as for a copy.
+- When the rule makes a row again with the same values, for example a `failed` row that fails again with the same error, the row stays as it is, with its `committed_in`.
+- Before a commit changes or deletes an output row, it copies the row into the [history](./CONTEXT.md) of the raw table. When the declaration changes, the commit that makes the table again copies each row of the old table into the history. A column that the old table did not have is `NULL` as of a position before.
+
 ## When something fails
 
 - **A rule failure** gives an output row with the status `failed` and the reason in `error`. The run continues, and the refresh after the next pass tries the input again.
-- **A dataset failure**, for example a query that SQLite refuses, rejects the run, or it stops the watch.
+- **A dataset failure**, for example a query that SQLite refuses, rejects the run, or it stops the watch, with a `DerivedDatasetError` that names the dataset.
+- **A table that is gone**, for example one that a person dropped by hand, is no failure. A computed dataset that reads it waits and keeps its output rows. When the copy loads the table again, the dataset asks the rule only for inputs that changed.
 - **A copy that failed** keeps its last committed rows. The derived datasets read those rows.
 
 ## Recipes
@@ -205,6 +263,7 @@ At each refresh, the computed dataset compares the inputs with the output rows.
 One use case each, with a full program that you can run:
 
 - [Search the text of stored files with SQL](./docs/recipes/search-the-text-of-stored-files.md)
+- [Read the file as of a position](./docs/recipes/read-the-file-as-of-a-position.md)
 
 ## Decisions
 
@@ -214,6 +273,9 @@ One use case each, with a full program that you can run:
 - [ADR 0004](./docs/adr/0004-a-changed-declaration-makes-the-table-again.md): a changed declaration makes the table again.
 - [ADR 0005](./docs/adr/0005-each-checkpointed-copy-has-its-own-lock.md): each checkpointed copy has its own lock.
 - [ADR 0006](./docs/adr/0006-a-sole-writer-fails-the-passes-it-finds-unfinished.md): a sole writer fails the passes that it finds unfinished.
+- [ADR 0007](./docs/adr/0007-each-pass-tries-an-input-at-most-once.md): each pass tries an input at most once.
+- [ADR 0008](./docs/adr/0008-a-file-that-several-connections-load-shows-one-pass.md): a file that several connections load shows one pass.
+- [ADR 0009](./docs/adr/0009-the-sqlite-file-numbers-its-commits-and-keeps-a-history-of-each-row.md): the SQLite file numbers its commits and keeps a history of each row.
 
 ## Development
 
@@ -231,7 +293,10 @@ The files of the destination:
 src/
   sqlite-destination.ts      SQLiteDestination: tables and the load
   open-destination.ts        openDestination: the open of each connection that writes the file
-  sqlite-writer.ts           the stage, the commit, evolve and reload of a table
+  sqlite-writer.ts           the stage, the commit, evolve, reload and the difference of a table
+  sqlite-loads.ts            SQLiteLoad: the loads and commit numbers of a file, its positions and its retention
+  sqlite-history.ts          SQLiteHistory: the history of a table, and the rows that the retention deletes
+  sqlite-as-of.ts            readings and asOfSelect: the file as of a position
   sqlite-checkpoint-store.ts SQLiteCheckpointStore
   sqlite-sync-history.ts     SQLiteSyncHistory
   sqlite-catalog.ts          installSQLiteCatalog
@@ -246,4 +311,5 @@ src/
   sqlite-views.ts            publishSQLiteViews: makes described views and checks their columns
   view.test.ts               the tests of View
   computed-dataset.test.ts   the tests of ComputedDataset, and of the refreshes of a run and a watch
+  computed-as-of.test.ts     the tests of the commits, the positions and the history of a refresh
 ```

package/dist/computed-dataset.d.ts

@@ -1,5 +1,5 @@
 import type { SQLInputValue } from 'node:sqlite';
-import { type Connection, DerivedDataset, type RefreshTrigger, type Rule } from '@eltjs/elt';
+import { type Connection, DerivedDataset, type PipelineClock, type RefreshTrigger, type Rule } from '@eltjs/elt';
 import { SQLiteDestination } from './sqlite-destination.ts';
 import type { SQLiteTable } from './sqlite-table.ts';
 import { View } from './view.ts';
@@ -14,8 +14,9 @@ export declare class ComputedDataset<Input, Output extends SQLInputValue> extend
         compute: Rule<Input, Output>;
     });
     validate(connections: readonly Connection<SQLiteTable>[]): void;
-    refresh({ connections, signal, trigger, }: {
+    refresh({ connections, clock, signal, trigger, }: {
         readonly connections: readonly Connection<SQLiteTable>[];
+        readonly clock: PipelineClock;
         readonly signal?: AbortSignal;
         readonly trigger: RefreshTrigger;
     }): Promise<void>;

package/dist/computed-dataset.js

@@ -3,8 +3,11 @@ import { Declaration } from '@zukhruf/sqlite-drift';
 import { open } from '@zukhruf/sqlite-open';
 import { transaction } from '@zukhruf/sqlite-transaction';
 import { DerivedDataset, } from '@eltjs/elt';
-import { busyTimeout, openDestination } from './open-destination.js';
+import { busyTimeout } from './open-destination.js';
 import { SQLiteDestination } from './sqlite-destination.js';
+import { SQLiteHistory, rowHash } from './sqlite-history.js';
+import { identifiers } from './sqlite-identifiers.js';
+import { SQLiteLoad } from './sqlite-loads.js';
 import { publishSQLiteViews } from './sqlite-views.js';
 import { View } from './view.js';
 const quote = (name) => `"${name.replaceAll('"', '""')}"`;
@@ -25,6 +28,10 @@ const engineColumns = {
 // A rule's failure on one input is that row's; it never fails the load.
 // Readers query the view name, over the table raw_<name>, which a refresh
 // replaces when its declaration changes, since it is rebuilt from the inputs.
+// A refresh is a load of the file: each of its transactions that changes a
+// row is a numbered commit, each row names the commit that last wrote it,
+// and the history keeps each row a commit replaced or removed, so the view
+// reads as of a position as the copies' tables do.
 export class ComputedDataset extends DerivedDataset {
     name;
     #into;
@@ -33,6 +40,7 @@ export class ComputedDataset extends DerivedDataset {
     #rule;
     #table;
     #declaration;
+    #history;
     // Every column of an output row, in table order.
     #columns;
     constructor({ into, name, description, from, compute, }) {
@@ -43,6 +51,10 @@ export class ComputedDataset extends DerivedDataset {
         this.#from = Object.freeze([...from]);
         this.#rule = compute;
         this.#table = `raw_${name}`;
+        this.#history = new SQLiteHistory({
+            name: this.#table,
+            location: identifiers.key(this.#table),
+        });
         this.#columns = Object.freeze([
             compute.key.name,
             ...Object.keys(engineColumns),
@@ -57,6 +69,8 @@ export class ComputedDataset extends DerivedDataset {
                 `"status" TEXT NOT NULL CHECK ("status" IN ('computed', 'empty', 'failed'))`,
                 '"error" TEXT',
                 ...ruleColumns,
+                '"committed_in" INTEGER NOT NULL',
+                '"_elt_row_hash" TEXT NOT NULL',
             ].join(', ')}) STRICT`,
         ], { rows: 'replace' });
         Object.freeze(this);
@@ -76,12 +90,35 @@ export class ComputedDataset extends DerivedDataset {
             if (made.includes(name.toLowerCase()))
                 throw new TypeError(`Computed dataset ${this.name} needs ${name}, which a copy of the pipeline makes`);
     }
-    async refresh({ connections, signal, trigger, }) {
+    async refresh({ connections, clock, signal, trigger, }) {
         const inputs = this.#inputs(connections);
-        using database = openDestination(this.#into.path);
+        if (inputs === null)
+            return;
+        // Open from the start, so it closes a refresh that a process ended; with
+        // no commit it is no span, and leaves the positions as they are.
+        using load = SQLiteLoad.open(this.#into.path, { dataset: this.name }, { clock, retention: this.#into.retention });
+        const { database } = load;
         const key = this.#rule.key.name;
+        const table = quote(this.#table);
+        const byKey = `FROM ${table} AS "_elt_old" WHERE "_elt_old".${quote(key)} = ?`;
+        // The number of a transaction's commit, taken at its first change, so a
+        // transaction that changes no row takes none.
+        const commits = () => {
+            let number;
+            return () => (number ??= load.commit());
+        };
+        const written = [...this.#columns, 'committed_in', '_elt_row_hash'].map(quote);
         const stored = transaction(database, () => {
+            const commit = commits();
+            // A new declaration replaces the table, and the history keeps the
+            // rows it drops.
+            if (this.#declaration.drift(database) === 'stale' &&
+                database.prepare(`SELECT 1 FROM ${table} LIMIT 1`).get() !== undefined) {
+                this.#history.create(database, written);
+                this.#history.keep(database, commit(), `FROM ${table} AS "_elt_old"`);
+            }
             this.#declaration.apply(database);
+            this.#history.create(database, written);
             publishSQLiteViews(database, {
                 views: [
                     {
@@ -92,18 +129,20 @@ export class ComputedDataset extends DerivedDataset {
                             ...engineColumns,
                             ...Object.fromEntries(Object.entries(this.#rule.output).map(([column, { description }]) => [column, description])),
                         },
-                        query: `SELECT ${this.#columns.map(quote).join(', ')} FROM ${quote(this.#table)}`,
+                        query: `SELECT ${this.#columns.map(quote).join(', ')} FROM ${table}`,
                     },
                 ],
             });
             const rows = new Map();
-            const remove = database.prepare(`DELETE FROM ${quote(this.#table)} WHERE ${quote(key)} = ?`);
+            const remove = database.prepare(`DELETE FROM ${table} WHERE ${quote(key)} = ?`);
             for (const row of database
-                .prepare(`SELECT ${quote(key)} AS "key", "input_hash", "rule", "status" FROM ${quote(this.#table)}`)
+                .prepare(`SELECT ${quote(key)} AS "key", "input_hash", "rule", "status" FROM ${table}`)
                 .all()) {
                 const id = String(row.key);
-                if (!inputs.has(id))
+                if (!inputs.has(id)) {
+                    this.#history.keep(database, commit(), byKey, [id]);
                     remove.run(id);
+                }
                 else
                     rows.set(id, {
                         inputHash: String(row.input_hash),
@@ -113,8 +152,33 @@ export class ComputedDataset extends DerivedDataset {
             }
             return rows;
         }, { mode: 'immediate' });
-        const write = database.prepare(`INSERT OR REPLACE INTO ${quote(this.#table)} (${this.#columns.map(quote).join(', ')}) VALUES (${this.#columns.map(() => '?').join(', ')})`);
-        const store = (id, inputHash, rule, status, error, values) => write.run(id, inputHash, rule, status, error, ...Object.keys(this.#rule.output).map((column) => values[column] ?? null));
+        const version = database.prepare(`SELECT "_elt_row_hash" AS "hash" FROM ${table} WHERE ${quote(key)} = ?`);
+        const write = database.prepare(`INSERT INTO ${table} (${written.join(', ')}) VALUES (${written.map(() => '?').join(', ')}) ON CONFLICT (${quote(key)}) DO UPDATE SET ${written
+            .slice(1)
+            .map((column) => `${column} = excluded.${column}`)
+            .join(', ')}`);
+        const hashed = this.#columns.map((name) => ({ name }));
+        // Writes a row in the commit that commit numbers, unless the stored row
+        // has the same values: then it keeps its commit. The history keeps the
+        // row it replaces.
+        const store = (commit, id, inputHash, rule, status, error, values) => {
+            const row = [
+                id,
+                inputHash,
+                rule,
+                status,
+                error,
+                ...Object.keys(this.#rule.output).map((column) => values[column] ?? null),
+            ];
+            const hash = rowHash(hashed, row);
+            const before = version.get(id)?.hash;
+            if (before === hash)
+                return;
+            const number = commit();
+            if (before !== undefined)
+                this.#history.keep(database, number, byKey, [id]);
+            write.run(...row, number, hash);
+        };
         const due = [];
         for (const [id, input] of inputs) {
             if (signal?.aborted)
@@ -128,7 +192,7 @@ export class ComputedDataset extends DerivedDataset {
                     continue;
                 if (before?.rule === 'unsupported' && before.inputHash === inputHash)
                     continue;
-                store(id, inputHash, 'unsupported', 'empty', null, {});
+                transaction(database, () => store(commits(), id, inputHash, 'unsupported', 'empty', null, {}), { mode: 'immediate' });
                 continue;
             }
             if (before !== undefined &&
@@ -168,16 +232,17 @@ export class ComputedDataset extends DerivedDataset {
                 }
                 // Each call commits on its own, so readers see rows as they land.
                 transaction(database, () => {
+                    const commit = commits();
                     batch.forEach(({ id, inputHash, rule }, index) => {
                         const result = results[index];
                         if (result === undefined || 'gone' in result)
                             return;
                         if ('value' in result)
-                            store(id, inputHash, rule, 'computed', null, result.value);
+                            store(commit, id, inputHash, rule, 'computed', null, result.value);
                         else if ('error' in result)
-                            store(id, inputHash, rule, 'failed', result.error, {});
+                            store(commit, id, inputHash, rule, 'failed', result.error, {});
                         else
-                            store(id, inputHash, rule, 'empty', null, {});
+                            store(commit, id, inputHash, rule, 'empty', null, {});
                     });
                 }, { mode: 'immediate' });
             }
@@ -188,6 +253,10 @@ export class ComputedDataset extends DerivedDataset {
     // Each input by key, from the rows of each relation of from that exists
     // yet: what the copies loaded, or what a view before this dataset gives. A
     // key read twice keeps its first input.
+    // null while a relation reads a table that is gone, such as one a person
+    // dropped by hand: its copy loads it again, and the dataset waits with what it
+    // made, as a view waits. Read as no rows, it would delete every output, and the
+    // reload would pay the rule for each again.
     #inputs(connections) {
         const inputs = new Map();
         for (const target of this.#from) {
@@ -204,9 +273,16 @@ export class ComputedDataset extends DerivedDataset {
                 .prepare('SELECT 1 FROM sqlite_schema WHERE lower("name") = lower(?)')
                 .get(relation) === undefined)
                 continue;
-            for (const row of database
-                .prepare(`SELECT * FROM ${quote(relation)}`)
-                .all()) {
+            let rows;
+            try {
+                rows = database.prepare(`SELECT * FROM ${quote(relation)}`).all();
+            }
+            catch (error) {
+                if (Error.isError(error) && error.message.startsWith('no such table: '))
+                    return null;
+                throw error;
+            }
+            for (const row of rows) {
                 const read = this.#rule.read(row);
                 if (read !== null && !inputs.has(read.key))
                     inputs.set(read.key, read.input);

package/dist/index.d.ts

@@ -5,6 +5,7 @@ export { SQLiteCheckpointStore } from './sqlite-checkpoint-store.ts';
 export { SQLiteColumn } from './sqlite-column.ts';
 export { SQLiteColumns } from './sqlite-columns.ts';
 export { SQLiteDestination } from './sqlite-destination.ts';
+export { type SQLiteReading, asOfSelect } from './sqlite-as-of.ts';
 export { type PassState, type PassStatus, readPassStatus, type StreamPassStatus, } from './sqlite-passes.ts';
 export { SQLiteSyncHistory } from './sqlite-sync-history.ts';
 export { SQLiteTable } from './sqlite-table.ts';

package/dist/index.js

@@ -5,6 +5,7 @@ export { SQLiteCheckpointStore } from './sqlite-checkpoint-store.js';
 export { SQLiteColumn } from './sqlite-column.js';
 export { SQLiteColumns } from './sqlite-columns.js';
 export { SQLiteDestination } from './sqlite-destination.js';
+export { asOfSelect } from './sqlite-as-of.js';
 export { readPassStatus, } from './sqlite-passes.js';
 export { SQLiteSyncHistory } from './sqlite-sync-history.js';
 export { SQLiteTable } from './sqlite-table.js';

package/dist/sqlite-append-writer.d.ts

@@ -1,8 +1,9 @@
 import type { DatabaseSync } from 'node:sqlite';
 import type { CopyConfiguration } from '@eltjs/elt';
+import type { SQLiteDestination } from './sqlite-destination.ts';
 import type { SQLiteTable } from './sqlite-table.ts';
-import { SQLiteWriter } from './sqlite-writer.ts';
+import { SQLiteWriter, type Stamp } from './sqlite-writer.ts';
 export declare class SQLiteAppendWriter extends SQLiteWriter {
-    constructor(configuration: CopyConfiguration, path: string, table: SQLiteTable);
-    protected merge(database: DatabaseSync, stage: string, loadedAt: string, into: string): void;
+    constructor(configuration: CopyConfiguration, destination: SQLiteDestination, table: SQLiteTable);
+    protected merge(database: DatabaseSync, stage: string, stamp: Stamp, into: string): void;
 }

package/dist/sqlite-append-writer.js

@@ -1,10 +1,10 @@
 import { SQLiteWriter } from './sqlite-writer.js';
 export class SQLiteAppendWriter extends SQLiteWriter {
-    constructor(configuration, path, table) {
-        super(configuration, path, table);
+    constructor(configuration, destination, table) {
+        super(configuration, destination, table);
         Object.freeze(this);
     }
-    merge(database, stage, loadedAt, into) {
-        this.append(database, stage, loadedAt, into);
+    merge(database, stage, stamp, into) {
+        this.append(database, stage, stamp, into);
     }
 }

package/dist/sqlite-as-of.d.ts

@@ -0,0 +1,29 @@
+import { type DatabaseSync } from 'node:sqlite';
+/**
+ * The SQLite file as of one position: `sql` reads in one read transaction,
+ * and sees each target as it was at that position. Dispose it to close it.
+ */
+export type SQLiteReading = {
+    readonly sql: DatabaseSync;
+    readonly position: string;
+    readonly at: Temporal.Instant;
+} & Disposable;
+export declare function reader(path: string): DatabaseSync | null;
+export declare function readAsOf(path: string, position: string | undefined): SQLiteReading | {
+    readonly expired: string;
+};
+/**
+ * The SELECT that reads one target of the file as of a position, for a reader
+ * that copies the target, such as into a database of its own: the current
+ * rows that the commit of the position had made, and the history rows that it
+ * had made and a later commit replaced. `schema` names the file where
+ * `database` attached it, main by default. Run the SELECT in the read
+ * transaction that this checks the position in, since a later commit can
+ * expire the position. A position of another epoch, past the newest commit,
+ * or older than the retention, is expired.
+ */
+export declare function asOfSelect(database: DatabaseSync, target: string, position: string, { schema }?: {
+    readonly schema?: string;
+}): string | {
+    readonly expired: string;
+};