@eltjs/elt-sqlite 0.3.2 → 0.3.4

package/README.md

@@ -35,6 +35,7 @@ Tables:
 - SQLite makes strict tables. SQL constraints that exist stay in force.
 - Each copy adds `loaded_at`, the UTC start time of the load that wrote the values of the row. A field with that name gets a different column name.
 - Each copy also adds `committed_in`, the number of the [commit](#commit-numbers-and-positions) that last wrote the row, and `_elt_row_hash`, the SHA-256 of the values of its columns. A [file in the SQLite file](#file-bytes-in-the-database) counts by the SHA-256 of its bytes, not by its id. A field with one of these names also gets a different column name. So does a field with the name `_rowid_`: SQLite reads that name as the id of each row, which an overwrite and a reload use.
+- The external schema of a stream, which a reader sees, is its columns and `loaded_at`. A reader view, each view of a [reading](#commit-numbers-and-positions), `asOfSelect` and `asOfDefinitions` show exactly these columns. `committed_in`, `_elt_row_hash`, `replaced_in` and `_rowid_` are internal: only the SQL of the destination reads them, for the history, the retention, the readings and the difference. Thus a column that the destination adds for itself changes nothing that a reader sees, as SQL Server hides the period columns of a temporal table that it marks `HIDDEN`. SQLite cannot hide a column of a table, so a raw table itself shows `committed_in` and `_elt_row_hash`. Readers query the reader view.
 - When a deduplicating load, an overwrite or a reload brings a row with the same values as a stored row, it leaves the stored row as it is, with its `loaded_at` and its `committed_in`. A deduplicating load finds the stored row by its key. An overwrite and a reload find it by [difference](#commit-numbers-and-positions), also in a table with no key.
 - Deduplication upserts on a unique index that is named after the table and the key (`_elt_dedup_<hash>`). SQLite makes the index one time, and makes it again only when the key changes. It uses the native [UPSERT with a cursor comparison](https://www.sqlite.org/lang_upsert.html).
 - Deduplication also checks the stored types of the key and cursor columns, and refuses null keys and cursors.
@@ -104,7 +105,7 @@ Each commit in the SQLite file has a number. A reader that compares the file at
 - A connection runs one pass at a time, and a computed dataset runs one refresh at a time. Thus when a load opens, it closes each load of the same connection or dataset that is still open: the process of that pass or refresh ended first. A closed load cannot commit. A load with no commit changes no position.
 - A commit into a [hidden table](./CONTEXT.md) shows nothing to readers. When the stream ends, one more commit gives its rows to the table, and readers see them from that commit on.
 - An overwrite and a reload replace the rows of the table by [difference](./CONTEXT.md). The difference compares the rows as multisets, so it needs no key. On each side, each row gets a number k among the rows with the same `_elt_row_hash`, and the k-th copy of some values matches the k-th copy of the same values on the other side:
-  - A row with a match stays as it is, with its `committed_in` and its `loaded_at`.
+  - A row with a match stays as it is, with its commit and its `loaded_at`.
   - A row of the table with no match goes into the history and leaves the table.
   - A new row with no match enters the table with the number of the commit.
 - An overwrite applies the difference in its one commit, to the records that it staged. A reload applies it, to the rows of its hidden table, in the commit at the end of the stream. The table keeps its indexes, its triggers and its reader view. When the table has a different shape than the hidden table, the reload puts the hidden table in its place, and every row of the table goes into the history.
@@ -118,10 +119,11 @@ Each commit in the SQLite file has a number. A reader that compares the file at
 - The reset of a partition copies each row of the partition into the history. The rows that the source sends again after the reset can come in later commits, for example after a checkpoint of each page. Thus no commit can know which rows of the partition left.
 - `destination.reading(position?)` gives a [reading](./CONTEXT.md) of the file as of the position, or as of the newest position: `{ sql, position, at }`, where `sql` is a read-only `DatabaseSync` in one read transaction and `at` is the time of the commit. Dispose the reading to close it. A position of a different epoch, after the newest commit, or older than [min valid](#retention), gives `{ expired }`. A commit number that `positions` does not list, for example a commit inside a pass, makes `reading` throw a `TypeError`, as a string that is not a position does: a reading there would show a part of a pass. A position that `positions` listed stays a position, because no later load can have commits before and after it.
 - As of a position before the newest commit, each table, its reader view and each view of the file show the rows that the commit of the position left. A column that a later commit added is `NULL`. The sync history keeps no versions, and a read of it fails. A stored view that names a table as `main.<table>` makes the reading fail, because that name shows the table as it is now.
+- At each position, also the newest, each table of a reading shows the external schema of its stream: its columns and `loaded_at`, or the columns of a computed dataset. Thus a reading shows the same columns at each position. A stored view that reads `committed_in` or `_elt_row_hash` of a table fails when it is read as of a position before the newest commit.
 
-- `asOfSelect(database, table, position, { schema })` gives the `SELECT` that reads one table as of a position: the rows of the table with `committed_in ≤ P`, and the rows of its history with `committed_in ≤ P < replaced_in`. A reading uses the same `SELECT` for its `TEMP` views. `schema` names the SQLite file where `database` attached it, `main` by default. Thus a reader can copy a table as of a position into a database of its own. It checks the position as `reading` does, so run the `SELECT` in the same read transaction. See [Read the file as of a position](./docs/recipes/read-the-file-as-of-a-position.md).
+- `asOfSelect(database, table, position, { schema })` gives the `SELECT` that reads one table as of a position: the rows of the table with `committed_in ≤ P`, and the rows of its history with `committed_in ≤ P < replaced_in`, with the columns of the external schema. A reading uses the same `SELECT` for its `TEMP` views. `schema` names the SQLite file where `database` attached it, `main` by default. Thus a reader can copy a table as of a position into a database of its own. It checks the position as `reading` does, so run the `SELECT` in the same read transaction. See [Read the file as of a position](./docs/recipes/read-the-file-as-of-a-position.md).
 - `asOfDefinitions(database, { schema, position, prefix })` gives the `CREATE TEMP VIEW` statements of the [views of a position](./CONTEXT.md) of the file that `database` attached as `schema`, or `{ expired }` as `asOfSelect` does. Thus one connection reads several files as of a position each, with no copy. Each table, reader view, view and table of the sync history is the view `<prefix>_<name>`, for example `mail_messages`:
-  - A table reads as `asOfSelect` reads it, without `_elt_row_hash`.
+  - A table reads as `asOfSelect` reads it.
   - A stored view names the views of the position in place of its tables and views. SQLite's own `ALTER TABLE … RENAME` maps the names, in a database of its own with a table of the same columns for each name. A stored view that reads a table with no view of the position, for example `catalog`, fails when it is read.
   - A table of the sync history fails when it is read, and so does each view over it.
   - A view reads the file when it is read, so a later commit changes nothing in it. When the retention deletes history that the position reads, or the file is made again, each read of a table fails with the reason: in the same statement, the view compares the epoch and min valid of the file with the position.
@@ -253,9 +255,9 @@ At each refresh, the computed dataset compares the inputs with the output rows.
 A refresh is a load in `_elt_loads`, so a [reading](#commit-numbers-and-positions) shows the computed dataset as of a position, as it shows a table that a copy loads:
 
 - Each transaction of the refresh that changes a row takes one commit number. A transaction that changes no row takes none. No commit inside a refresh that is open is a position.
-- The raw table has the columns `committed_in` and `_elt_row_hash` after the columns of the rule. The view `<name>` does not show them. `_elt_row_hash` is the SHA-256 of the values of the other columns, as for a copy.
+- The raw table has the columns `committed_in` and `_elt_row_hash` after the columns of the rule. A computed dataset has no `loaded_at`. Its external schema is the key, the engine columns and the columns of the rule: the view `<name>`, the views of a reading, `asOfSelect` and `asOfDefinitions` show only these. `_elt_row_hash` is the SHA-256 of the values of the other columns, as for a copy.
 - A rule column cannot have the name of the key, of an engine column, of `committed_in` or of `_elt_row_hash`, in any ASCII case, as SQLite compares column names. The constructor of `ComputedDataset` refuses such a rule with a `TypeError` that names the column and the dataset.
-- When the rule makes a row again with the same values, for example a `failed` row that fails again with the same error, the row stays as it is, with its `committed_in`.
+- When the rule makes a row again with the same values, for example a `failed` row that fails again with the same error, the row stays as it is, with its commit.
 - Before a commit changes or deletes an output row, it copies the row into the [history](./CONTEXT.md) of the raw table. When the declaration changes, the commit that makes the table again copies each row of the old table into the history. A column that the old table did not have is `NULL` as of a position before.
 
 ## When something fails

package/dist/sqlite-as-of.d.ts

@@ -32,7 +32,9 @@ export declare function readAsOf(path: string, position: string | undefined): SQ
  * The SELECT that reads one target of the file as of a position, for a reader
  * that copies the target, such as into a database of its own: the current
  * rows that the commit of the position had made, and the history rows that it
- * had made and a later commit replaced. `schema` names the file where
+ * had made and a later commit replaced, with the columns of the stream and
+ * `loaded_at`, or of a computed dataset, and none of the internal columns
+ * that only the destination reads. `schema` names the file where
  * `database` attached it, main by default. Run the SELECT in the read
  * transaction that this checks the position in, since a later commit can
  * expire the position. A position of another epoch, past the newest commit,

package/dist/sqlite-as-of.js

@@ -2,10 +2,9 @@ import { constants } from 'node:sqlite';
 import { UnavailableError, open } from '@zukhruf/sqlite-open';
 import { busyTimeout } from './open-destination.js';
 import { histories } from './sqlite-history.js';
-import { identifiers, quote } from './sqlite-identifiers.js';
+import { identifiers, internalColumns, quote } from './sqlite-identifiers.js';
 import { committedAt, expiry, parsePosition, readNumbering, } from './sqlite-loads.js';
 import { attempts, checks, coverage } from './sqlite-sync-history-schema.js';
-import { version as rowHash } from './sqlite-writer.js';
 /**
  * The SQL functions that views of a position call to fail, by name: a host
  * registers each one on the connection that reads the views, as
@@ -69,8 +68,7 @@ export function readAsOf(path, position) {
                 ? `${path} has no position yet`
                 : `Position ${position} names no commit of ${path}`,
         };
-    if (number < numbered.newest)
-        shadow(database, number);
+    shadow(database, number, number < numbered.newest);
     stack.move();
     return {
         sql: database,
@@ -85,14 +83,20 @@ function refuseInsidePass(numbered, { epoch, number }, file) {
     if (!numbered?.ends.includes(number))
         throw new TypeError(`${epoch}:${number} is not a position of ${file}: a reading of a commit inside a pass would show a part of that pass. Read a position that positions() lists.`);
 }
-// Shadows, for this connection, each name a reader queries with a TEMP view
-// of what it showed as of commit number. A TEMP view takes the place of a
-// table or view of main for an unqualified name, and SQLite resolves the
+// Shadows, for this connection, each target with a TEMP view of what it
+// showed as of commit number, with the columns a reader sees, so a reading
+// shows the same columns at each position. Before the newest commit, each
+// other name a reader queries is shadowed too. A TEMP view takes the place of
+// a table or view of main for an unqualified name, and SQLite resolves the
 // names of a view when it is read, so the views can be made in any order.
-function shadow(database, number) {
+function shadow(database, number, past) {
     const targets = versioned(database, 'main');
     for (const version of targets)
         database.exec(`CREATE TEMP VIEW ${quote(version.target)} AS ${selectAsOf(database, version, number, 'main')}`);
+    // At the newest commit, the views and the sync history read the file as
+    // it is.
+    if (!past)
+        return;
     database.function('_elt_unversioned', asOfFunctions._elt_unversioned);
     for (const view of syncHistoryViews(database, 'main', (table) => table))
         database.exec(view);
@@ -105,7 +109,9 @@ function shadow(database, number) {
  * The SELECT that reads one target of the file as of a position, for a reader
  * that copies the target, such as into a database of its own: the current
  * rows that the commit of the position had made, and the history rows that it
- * had made and a later commit replaced. `schema` names the file where
+ * had made and a later commit replaced, with the columns of the stream and
+ * `loaded_at`, or of a computed dataset, and none of the internal columns
+ * that only the destination reads. `schema` names the file where
  * `database` attached it, main by default. Run the SELECT in the read
  * transaction that this checks the position in, since a later commit can
  * expire the position. A position of another epoch, past the newest commit,
@@ -131,13 +137,13 @@ export function readable(database, position, schema) {
     return asked;
 }
 // A target as of commit number: its current rows that the commit had made,
-// and the history rows that the commit had made and a later commit replaced.
-// A column that the history lacks reads as NULL there. A guarded select is
-// for a view that outlives the read transaction: each read first checks, in
-// the same statement, that the file still reads the position exactly, and
-// it leaves out the row hash of the writers.
+// and the history rows that the commit had made and a later commit replaced,
+// with the columns a reader sees, none of the internal ones. A column
+// that the history lacks reads as NULL there. A guarded select is for a view
+// that outlives the read transaction: each read first checks, in the same
+// statement, that the file still reads the position exactly.
 export function selectAsOf(database, { target, history }, number, schema, guard) {
-    const columns = columnsOf(database, quote(target), schema).filter((column) => guard === undefined || quote(column) !== rowHash);
+    const columns = columnsOf(database, quote(target), schema).filter((column) => !internalColumns.includes(identifiers.key(column)));
     const kept = new Set(columnsOf(database, quote(history), schema).map((column) => identifiers.key(column)));
     const current = columns.map((column) => `"r".${quote(column)}`).join(', ');
     const replaced = columns

package/dist/sqlite-identifiers.d.ts

@@ -1,4 +1,5 @@
 import { Identifiers } from '@eltjs/elt';
+export declare const internalColumns: readonly string[];
 declare class SQLiteIdentifiers extends Identifiers {
     protected readonly maxBytes: number;
     protected readonly ownColumns: string[];

package/dist/sqlite-identifiers.js

@@ -1,17 +1,20 @@
 import { Identifiers } from '@eltjs/elt';
+// The columns that only the destination's own SQL reads: the commit that last
+// wrote a row and the hash of its values in each table, the commit that
+// replaced a row in each history, and _rowid_, the id SQLite gives each row,
+// which a difference needs, as long as no column takes the name. No reader
+// view, view of a position or SELECT as of a position shows them.
+export const internalColumns = [
+    'committed_in',
+    '_elt_row_hash',
+    'replaced_in',
+    '_rowid_',
+];
 // SQLite holds a name of any length and compares two without ASCII case: it
 // folds A-Z only, so `É` and `é` stay two names.
 class SQLiteIdentifiers extends Identifiers {
     maxBytes = Infinity;
-    // _rowid_ reads the id SQLite gives each row, which a difference needs, as
-    // long as no column takes the name.
-    ownColumns = [
-        'loaded_at',
-        'committed_in',
-        '_elt_row_hash',
-        'replaced_in',
-        '_rowid_',
-    ];
+    ownColumns = ['loaded_at', ...internalColumns];
     key(name) {
         return name.replaceAll(/[A-Z]/g, (letter) => letter.toLowerCase());
     }

package/dist/sqlite-writer.d.ts

@@ -30,7 +30,7 @@ export declare abstract class SQLiteWriter extends Writer {
     protected get replaces(): boolean;
     protected kept(_stage: string): string;
     private get hash();
-    protected get fields(): string[];
+    protected get shown(): string[];
     protected get written(): string[];
     protected encode(record: unknown): SQLInputValue[];
     protected deletionKeys(_key: Readonly<Record<string, KeyValue>>): readonly [readonly SQLiteColumn[], SQLInputValue[]];

package/dist/sqlite-writer.js

@@ -119,18 +119,18 @@ export class SQLiteWriter extends Writer {
     get hash() {
         return hash('sha256', this.table.location);
     }
-    // What a reader reads of each row.
-    get fields() {
+    // What a reader sees of each row, its reader view's columns: the stream's
+    // columns and loaded_at, none of the internal ones.
+    get shown() {
         return [
             ...this.table.columns.map((column) => column.quotedName),
             '"loaded_at"',
-            '"committed_in"',
         ];
     }
-    // What a commit writes into each row: its fields, and the hash of its
-    // values.
+    // What a commit writes into each row: what a reader sees, the commit's
+    // number, and the hash of its values.
     get written() {
-        return [...this.fields, version];
+        return [...this.shown, '"committed_in"', version];
     }
     encode(record) {
         return this.table.columns.map((column) => column.encode(record));
@@ -228,27 +228,31 @@ export class SQLiteWriter extends Writer {
         }
     }
     // What the table, its reader view and their columns say to readers,
-    // rewritten with every load so they follow the stream's schema.
+    // rewritten with every load so they follow the stream's schema. The reader
+    // view describes only the columns it shows.
     describe(database) {
         const dataTypes = new Map([
             ...this.table.columns.map(({ name, dataType }) => [name, dataType]),
             ['loaded_at', 'timestamp'],
             ['committed_in', 'integer'],
         ]);
-        const columns = Object.fromEntries(Object.entries({
-            ...this.#description.columns,
-            loaded_at: 'Start time of the load that wrote these values of the row. A deduplicating load, an overwrite or a reload that brings the same values leaves the row, and its loaded_at, as they are. Not the source modification time or the most recent successful sync.',
-            committed_in: 'Number of the commit that last wrote this row: the row of _elt_commits with this number tells when, and in which load.',
-        }).map(([name, description]) => [
+        const described = (descriptions) => Object.fromEntries(Object.entries(descriptions).map(([name, description]) => [
             name,
             { description, dataType: dataTypes.get(name) },
         ]));
-        describe(database, this.table.name, this.#description.table, columns);
+        const shown = {
+            ...this.#description.columns,
+            loaded_at: 'Start time of the load that wrote these values of the row. A deduplicating load, an overwrite or a reload that brings the same values leaves the row, and its loaded_at, as they are. Not the source modification time or the most recent successful sync.',
+        };
+        describe(database, this.table.name, this.#description.table, described({
+            ...shown,
+            committed_in: 'Number of the commit that last wrote this row: the row of _elt_commits with this number tells when, and in which load.',
+        }));
         if (this.table.readerView !== undefined)
-            describe(database, this.table.readerView, this.#description.table, columns);
+            describe(database, this.table.readerView, this.#description.table, described(shown));
     }
     #viewDefinition(view) {
-        return `CREATE VIEW ${quote(view)} AS SELECT ${this.fields.join(', ')} FROM ${this.table.quotedName}`;
+        return `CREATE VIEW ${quote(view)} AS SELECT ${this.shown.join(', ')} FROM ${this.table.quotedName}`;
     }
     // Refuses, before anything is read, a view of other columns or another
     // table, rather than adopting it. A stale table's view still selects the

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@eltjs/elt-sqlite",
-  "version": "0.3.2",
+  "version": "0.3.4",
   "description": "The SQLite destination of elt: each stream is a raw table in one SQLite file, with a reader view over it and a catalog of each table, view and column.",
   "license": "Apache-2.0",
   "repository": {
@@ -39,7 +39,7 @@
     "node": ">=26"
   },
   "dependencies": {
-    "@eltjs/elt": "^0.3.2",
+    "@eltjs/elt": "^0.3.4",
     "@zukhruf/mutex": "^0.8.10",
     "@zukhruf/sqlite-drift": "^0.8.10",
     "@zukhruf/sqlite-open": "^0.8.10",