dbgate-sqlite-dumper 0.1.0

package/docs/dump-api.md

@@ -0,0 +1,253 @@
+# Dump API
+
+```ts
+dumpSqlite(connection, options, output, onProgress?, signal?): Promise<DumpResult>
+```
+
+Runs a complete dump — acquire a handle, open a read snapshot, introspect, plan, render,
+stream rows — writing plain SQL in the native `.dump` layout to `output`.
+
+```ts
+import { createWriteStream } from 'node:fs';
+import { dumpSqlite } from 'dbgate-sqlite-dumper';
+import { connectBetterSqlite3 } from 'dbgate-sqlite-dumper/better-sqlite3';
+
+const { connection, close } = await connectBetterSqlite3('shop.db', { readonly: true });
+try {
+  const result = await dumpSqlite(connection, {}, createWriteStream('shop.sql'));
+  console.log(`${result.rowsExported} rows in ${result.statementsWritten} statements`);
+} finally {
+  await close();
+}
+```
+
+## `output`
+
+Any `Writable`. Never ended or closed by this package — the caller owns its lifecycle.
+
+A dump is written incrementally, honouring backpressure, so a multi-gigabyte database
+dumps in constant memory. `output` may receive `Buffer` chunks as well as text: a `TEXT`
+value holding bytes that are not valid UTF-8 is written raw, exactly as the native `.dump`
+writes it, and cannot be routed through a JavaScript string without corruption.
+
+## `DumpSqliteOptions`
+
+| Option        | Default      | Meaning                                         |
+| ------------- | ------------ | ----------------------------------------------- |
+| `mode`        | `'full'`     | `'full'`, `'schema-only'`, `'data-only'`.       |
+| `schemaName`  | `'main'`     | `main`, `temp`, or an attached database's name. |
+| `selection`   | everything   | Per-name object filters; see below.             |
+| `objectKinds` | all `true`   | Which kinds participate at all.                 |
+| `render`      | see below    | Output shape.                                   |
+| `dataExport`  | see below    | Row rendering and batching.                     |
+| `consistency` | `'snapshot'` | How a consistent view is obtained.              |
+
+The dump itself never names the schema (`CREATE TABLE t`, not `CREATE TABLE aux.t`), so a
+dump of an attached database restores into whatever database it is run against — like a
+native `.dump`.
+
+### `mode`
+
+- **`'full'`** — schema and rows: the native `.dump`.
+- **`'data-only'`** — `INSERT` statements only, no frame: the native `.dump --data-only`.
+  See [data-only dumps](#data-only-dumps).
+- **`'schema-only'`** — every `CREATE`, no rows, no `AUTOINCREMENT` counters, no statistics.
+  Virtual tables are created with their own `CREATE VIRTUAL TABLE`, so their module sets up
+  its shadow tables, which are therefore left out.
+
+### `selection`
+
+Names are exact identifiers, never patterns, compared the way SQLite compares them (ASCII
+letters case-insensitively, nothing else folded).
+
+```ts
+selection: {
+  tables: ['orders', 'order_lines'],   // omit for all
+  excludeTables: ['orders_archive'],   // applied after `tables`
+  views: [...], excludeViews: [...],
+  triggers: [...], excludeTriggers: [...],
+  excludeIndexes: [...],               // indexes otherwise follow their table
+
+  // Structure dumped, rows skipped — for a large log or cache table you still
+  // want recreated.
+  dataExcludedTables: ['request_log'],
+}
+```
+
+Selecting a virtual table selects its shadow tables (`docs` brings `docs_data`,
+`docs_idx`, ...). A trigger whose table is not selected is dropped rather than orphaned,
+and reported as `trigger-table-not-selected`. In a partial dump, the `sqlite_sequence` and
+statistics rows are restricted to the selected tables, and only their counters are reset.
+
+### `objectKinds`
+
+```ts
+objectKinds: {
+  includeTables: true,
+  includeViews: true,
+  includeIndexes: true,
+  includeTriggers: true,
+  includeVirtualTables: true,   // and their shadow tables
+  includeSystemTables: true,    // sqlite_sequence, sqlite_stat*; false = native --nosys
+}
+```
+
+### `render` — output shape
+
+| Option                     | Default   | Meaning                                                            |
+| -------------------------- | --------- | ------------------------------------------------------------------ |
+| `includeSessionGuards`     | `true`    | `PRAGMA foreign_keys=OFF;` / `BEGIN TRANSACTION;` … `COMMIT;`      |
+| `includeHeaderComments`    | `true`    | The defensive-mode warning line, when there are virtual tables.    |
+| `addDropStatements`        | `false`   | `DROP … IF EXISTS` before each `CREATE`.                           |
+| `includeDatabaseSettings`  | `false`   | `PRAGMA user_version` / `application_id` before `COMMIT`.          |
+| `legacySchemaTableName`    | `false`   | Write `sqlite_master` instead of `sqlite_schema` (targets < 3.33). |
+| `unsupportedFeaturePolicy` | `'error'` | `'warn-omit'` skips an entry that cannot be rendered instead.      |
+
+Notes on the ones that carry weight:
+
+- **`includeSessionGuards: false`** produces a `session-guards-disabled` warning. Without
+  `foreign_keys=OFF`, tables that reference each other may not restore in creation order;
+  without the transaction, every `INSERT` is its own transaction and its own disk sync.
+- **`includeDatabaseSettings`** is off to match the native `.dump`, which does not carry
+  `user_version`. Applications often track their schema migrations there, so a non-zero
+  value that is not being dumped is reported as `user-version-not-dumped`.
+
+### `dataExport` — row rendering
+
+| Option                | Default | Meaning                                                                 |
+| --------------------- | ------- | ----------------------------------------------------------------------- |
+| `preserveRowids`      | `false` | `INSERT INTO t(rowid,…)` for hidden rowids; native `--preserve-rowids`. |
+| `rawNewlines`         | `false` | Raw newlines in literals instead of `replace()`; native `--newlines`.   |
+| `extendedInsert`      | `false` | Multi-row `INSERT … VALUES (…),(…)`.                                    |
+| `maxRowsPerStatement` | `500`   | With `extendedInsert`.                                                  |
+| `maxStatementBytes`   | 1 MiB   | With `extendedInsert`; a single larger row is still emitted alone.      |
+| `streamBatchSize`     | —       | Hint passed to `connection.stream()`.                                   |
+
+- **`preserveRowids`**: a table without an `INTEGER PRIMARY KEY` has a hidden rowid, and
+  without this option a restore renumbers its rows from 1. That matters when anything
+  refers to rowids — an external-content FTS index, an application storing them.
+- **`extendedInsert`** makes the file smaller and the restore faster, at the cost of native
+  byte identity. One row per statement is not slow in SQLite — the whole restore is one
+  transaction — so the default follows the native layout.
+
+Rows are read in the table's natural order, with no `ORDER BY`, exactly like the native
+`.dump`: rowid order for an ordinary table, primary-key order for a `WITHOUT ROWID` one. Two
+dumps of an unchanged database are identical.
+
+### `consistency`
+
+- **`'snapshot'`** (default) — the whole dump runs inside one read transaction (a
+  `SAVEPOINT`, so it nests inside a transaction the caller already has open, and reads
+  what that transaction sees). In WAL mode that is a true snapshot and writers carry on; in
+  rollback-journal mode it holds a `SHARED` lock that keeps writers out until the dump
+  ends. It is the same mechanism the native `.dump` uses.
+- **`'none'`** — no transaction; only safe when nothing else writes to the database.
+
+## Data-only dumps
+
+`.dump --data-only` — and `mode: 'data-only'`, which reproduces it — has two quirks that
+matter when loading the rows into an existing schema:
+
+- it carries each virtual table's rows twice: through the table, and as its shadow tables'
+  rows, which collide with the target table's own shadow rows;
+- it inserts into `sqlite_sequence` and `sqlite_stat*` without preparing them, leaving
+  duplicate counters, or failing where the target was never analyzed.
+
+For loading rows into an existing schema, this option set avoids both:
+
+```ts
+await dumpSqlite(
+  connection,
+  {
+    mode: 'data-only',
+    objectKinds: { includeSystemTables: false },
+    selection: {
+      dataExcludedTables: ['docs_data', 'docs_idx', 'docs_content', 'docs_docsize', 'docs_config'],
+    },
+  },
+  output,
+);
+
+// …and restore with
+await restoreSqlDump({
+  connection: target,
+  source,
+  options: { transaction: 'wrap', disableForeignKeys: true },
+});
+```
+
+Create triggers after the load, or they fire for every restored row.
+
+## `DumpResult`
+
+```ts
+{
+  bytesWritten: number;
+  rowsExported: number;
+  statementsWritten: number;
+  renderedDumpIds: readonly string[];
+  skippedDumpIds: readonly string[];
+  warnings: readonly SqliteDiagnostic[];
+  cancelled: boolean;
+}
+```
+
+`cancelled` is `true` when the `AbortSignal` fired; the dump is truncated but nothing
+throws, and the read transaction is ended cleanly.
+
+## Progress
+
+Phases, in order: `connecting`, `starting-snapshot`, `introspecting`, `detecting-version`,
+`planning-archive`, `rendering-schema`, `exporting-data`, `finalizing`. Rendering events
+carry `section`, `objectsProcessed`/`objectsTotal` and `objectName`; data events carry
+`exportState` (`started`/`progress`/`finished`/`failed`/`cancelled`), `tableName`,
+`rowsExported` and `bytesWritten`, every 1000 rows.
+
+## Cancellation
+
+```ts
+const controller = new AbortController();
+const result = await dumpSqlite(connection, {}, output, undefined, controller.signal);
+if (result.cancelled) console.warn('dump truncated');
+```
+
+Cancellation takes effect between rows. Buffered output is deliberately **not** flushed: a
+dump cut off at an arbitrary row should look truncated, not complete.
+
+## Composing the stages
+
+```ts
+import {
+  introspectSqlite,
+  inspectDumpArchive,
+  renderPlainSql,
+  BufferDumpWriter,
+} from 'dbgate-sqlite-dumper';
+
+const { database, version } = await introspectSqlite(connection);
+
+// Inspect the plan without rendering anything.
+const archive = inspectDumpArchive(database, { mode: 'schema-only' });
+for (const entry of archive.entries) {
+  console.log(entry.sequenceNumber, entry.section, entry.objectType, entry.name);
+}
+
+// Render schema only, into memory, with no database access.
+const writer = new BufferDumpWriter();
+await renderPlainSql({ database, archive, writer, mode: 'schema-only', sourceVersion: version });
+console.log(writer.toString());
+```
+
+`exportTableDataAsInserts` streams one table's rows on its own:
+
+```ts
+import { exportTableDataAsInserts } from 'dbgate-sqlite-dumper';
+
+await exportTableDataAsInserts({
+  connection,
+  schemaName: 'main',
+  table: database.tables.find(table => table.name === 'orders')!,
+  writer,
+  options: { extendedInsert: true },
+});
+```

package/docs/known-limitations.md

@@ -0,0 +1,88 @@
+# Known limitations
+
+What this package does not do, and why. The items in the first section are also returned by
+`unsupportedFeatureDiagnostics()`, so a UI can list them without hardcoding the text.
+
+## Not dumped
+
+### File-level settings
+
+`page_size`, `auto_vacuum`, `journal_mode`, `encoding` and the like describe how the
+database _file_ is laid out, not what it contains, and most can only be chosen before the
+first table exists. The native `.dump` does not carry them either. Set them on the target
+before restoring if they matter; the dump restores correctly under any of them.
+
+`user_version` and `application_id` are the exception: they are carried with
+`render.includeDatabaseSettings`, and reported as a warning when they are non-zero and not
+carried.
+
+### The temp schema and attached databases
+
+A dump covers one schema — `main` by default. Temporary objects belong to one connection;
+each attached database is a separate file. Dump an attached database on its own with
+`schemaName`.
+
+### Application-defined functions, collations and modules
+
+User-defined SQL functions, collating sequences and virtual-table modules are registered by
+the application (or loaded as extensions), not stored in the database. Schema that uses
+them is dumped verbatim, and the restoring application must register them first.
+`checkTargetCompatibility` reports custom collations used by indexes, and — when the
+target's module list is known — missing virtual-table modules.
+
+### Encryption
+
+An encrypted database (SQLCipher, SEE) dumps as plain SQL once it is opened with its key.
+The dump is not encrypted, and no key is ever written into it or into any diagnostic.
+
+## Behaviours to know about
+
+### `REAL` digits depend on the SQLite release
+
+See [native-compatibility.md](native-compatibility.md#the-one-exception-real-digit-tails).
+The value never changes; the spelling past the 17th digit can.
+
+### `sqlite_stat4` across builds
+
+`sqlite_stat4` exists only on SQLite built with `SQLITE_ENABLE_STAT4` (such as
+`better-sqlite3`'s). A dump that carries its rows cannot be restored on a build without
+it — by the native `.dump` either — because the `ANALYZE sqlite_schema;` that prepares the
+table does not create it there. `preflightRestore` reports `statistics-table-unsupported`;
+dump with `objectKinds.includeSystemTables: false`, or drop the table first.
+
+Conversely, restoring through a STAT4 build creates an empty `sqlite_stat4` the source did
+not have. It is harmless, and `ANALYZE` repopulates it.
+
+### Data-only dumps
+
+A data-only dump reproduces `.dump --data-only`, including two quirks that matter when
+loading the rows into an existing schema (virtual-table rows carried twice; system tables
+not prepared). [dump-api.md](dump-api.md#data-only-dumps) gives the option set that avoids
+both.
+
+### Hidden rowids are renumbered by default
+
+As natively; see [supported-objects.md](supported-objects.md#hidden-rowids).
+
+### Virtual-table dumps need defensive mode off to restore
+
+Inherent to how the native `.dump` recreates virtual tables. The restore lifts
+`SQLITE_DBCONFIG_DEFENSIVE` itself through adapters that can (the bundled one can); with an
+adapter that cannot, the restore succeeds only on a handle not in defensive mode.
+
+### Cancellation granularity
+
+`better-sqlite3` runs each statement synchronously, so cancellation takes effect between
+rows and between statements, never inside one.
+
+### Memory
+
+A dump holds one statement and a 64 KiB output buffer; a restore holds one statement's
+text. A single enormous value — a 500 MB blob — is held whole, once, because one row cannot
+be split.
+
+### Scripts the parser refuses
+
+Dot-commands that change the database (`.read`, `.import`, `.open`, …), identifiers
+containing bytes that are not valid UTF-8, and input ending inside a string or quoted
+identifier. See [restore-api.md](restore-api.md#errors).

package/docs/native-compatibility.md

@@ -0,0 +1,124 @@
+# Native compatibility
+
+The promise, in both directions:
+
+1. **A dump written by this package restores with the native shell**:
+   `sqlite3 target.db < dump.sql`.
+2. **A dump written by the native `.dump` restores with this package**:
+   `restoreSqlDump({ connection, source })`.
+
+Both are tested on every run against the real `sqlite3` shell (3.45 on Ubuntu 24.04 in CI),
+ending with a deep comparison of the restored database's schema and every row against the
+source. See [round-trip-testing.md](round-trip-testing.md).
+
+## Byte identity
+
+With default options, `dumpSqlite` writes the same bytes as `sqlite3 db .dump`, and each
+native switch has an equivalent that is tested the same way:
+
+| Native                    | This package                                      |
+| ------------------------- | ------------------------------------------------- |
+| `.dump`                   | `{}`                                              |
+| `.dump --data-only`       | `{ mode: 'data-only' }`                           |
+| `.dump --preserve-rowids` | `{ dataExport: { preserveRowids: true } }`        |
+| `.dump --newlines`        | `{ dataExport: { rawNewlines: true } }`           |
+| `.dump --nosys`           | `{ objectKinds: { includeSystemTables: false } }` |
+
+### The one exception: `REAL` digit tails
+
+The shell writes a non-integral `REAL` with `sqlite3_snprintf("%!.20g")`, and this package
+has SQLite evaluate `printf('%!.20g', value)` for it — the same formatter. But the digits
+that formatter produces past the 17th significant digit changed between SQLite releases
+(3.45 writes `0.100000000000000005`, 3.53 writes `0.1000000000000000056`). A driver that
+bundles its own SQLite — as `better-sqlite3` does — therefore writes the digits of _its_
+release.
+
+Those digits carry no information: 17 significant digits already identify a double
+exactly, and both spellings parse back to the identical value. The interop tests compare
+every line byte for byte, and compare a differing line again with each `REAL` literal
+parsed as a double; nothing else is allowed to differ. When the driver and the shell embed
+the same SQLite release, the output is byte-identical without exception.
+
+## What is reproduced, and why
+
+Every rule below is ported from `shell.c`, because each one changes bytes in the output.
+
+- **The frame.** `PRAGMA foreign_keys=OFF;` and `BEGIN TRANSACTION;` first, `COMMIT;`
+  last. A data-only dump has neither, as natively.
+- **Creation order.** Tables in `sqlite_schema` rowid order (`sqlite_sequence` moved last),
+  each followed by its rows; then indexes, triggers and views in rowid order. Creation order
+  is a valid dependency order for everything SQLite checks at `CREATE` time, and
+  `foreign_keys=OFF` covers foreign keys, circular ones included.
+- **Verbatim DDL.** SQLite stores every object's `CREATE` statement exactly as written; that
+  text is what is emitted, never a reconstruction.
+- **`printSchemaLine()`.** A table whose name is quoted with `'` or `"` becomes
+  `CREATE TABLE IF NOT EXISTS`; DDL ending in a `--` comment or an unterminated `/*` gets a
+  newline or `*/` before its `;`, chosen with `sqlite3_complete()` exactly as the shell does.
+- **Index, trigger and view terminators.** The statement's `;` goes on a line of its own if
+  its text contains `--` anywhere.
+- **Identifier quoting.** Table and column names are quoted in `INSERT` only when not a
+  plain identifier or when a keyword (`quoteChar()`, with SQLite's 147 keywords).
+- **Values.** Integers as decimal; `REAL` as `%lld.0` when integral and in range, else
+  `%!.20g`, with infinities as `9.0e+999`; blobs as lower-case `X'...'`; text quoted with
+  `'` doubled, and — by default — newlines and carriage returns rewritten as
+  `replace('...\n...','\n',char(10))`, with a placeholder guaranteed not to occur in the
+  text (`unused_string()`).
+- **Virtual tables.** Written as `INSERT INTO sqlite_schema(...)` under
+  `PRAGMA writable_schema=ON`, not as `CREATE VIRTUAL TABLE`, so the module's constructor
+  does not run and the shadow tables — which the dump then fills with the source's exact
+  contents — are not rebuilt. The whole dump is preceded by
+  `/* WARNING: Script requires that SQLITE_DBCONFIG_DEFENSIVE be disabled */`.
+- **System tables.** `DELETE FROM sqlite_sequence;` before the `AUTOINCREMENT` counters,
+  `ANALYZE sqlite_schema;` before each statistics table's rows.
+
+## Native quirks reproduced deliberately
+
+These are faithful to the shell rather than "fixed", because a dump that differs from the
+native one for no stated reason is harder to trust than one with a documented quirk. Each is
+reported as a warning when it applies.
+
+- **`.dump --data-only` selects virtual tables' rows twice** — through the table itself,
+  and again as the shadow tables' rows — because the shell checks for data-only before it
+  checks for a virtual table. Restoring such a dump into a database where the virtual table
+  exists conflicts with its own shadow rows (`data-only-virtual-table`).
+- **`.dump --data-only` inserts into system tables without preparing them**: the
+  `sqlite_sequence` rows are not preceded by a `DELETE`, and the statistics rows by no
+  `ANALYZE` (`data-only-system-table`).
+- **`-0.0` is written as `0.0`**, because the shell's integral test is true for it.
+
+[dump-api.md](dump-api.md#data-only-dumps) gives the option set that avoids the first two
+when loading rows into an existing schema.
+
+## Deliberate deviations
+
+Only where the native behaviour loses data, and never for data the shell can represent:
+
+- **`NUL` characters in text are kept.** The shell handles values as C strings and drops
+  everything after the first `NUL`; this package writes `'a'||char(0)||'b'`, which the
+  shell restores correctly. Text without `NUL` — all text in practice — is unaffected.
+- **A schema-only dump** has no native `.dump` equivalent. It creates virtual tables with
+  their own `CREATE VIRTUAL TABLE` (as `.schema` shows them) and leaves their shadow tables
+  to the module, since no shadow rows follow.
+- **Partial dumps** (a `selection`) keep only the selected tables' `sqlite_sequence` and
+  statistics rows, and reset only their counters
+  (`DELETE FROM sqlite_sequence WHERE lower(name) IN (...)`). The native `.dump PATTERN`
+  omits the counters of the selected tables entirely.
+
+## Restoring native dumps
+
+The restore side reproduces how the shell _reads_ a script, which is what makes any
+`sqlite3` script restorable, not just `.dump` output:
+
+- statement boundaries from `sqlite3_complete()`, so trigger bodies stay whole;
+- `GO` and `/` lines as terminators, `#` lines at column 0 as comments, dot-commands at
+  column 0 as shell commands;
+- a `\r` immediately before `\n` dropped everywhere, as the shell's line reader does;
+- a leading UTF-8 byte-order mark ignored;
+- a final statement without `;` still executed.
+
+And it accounts for running inside an application instead of a process that exits:
+
+- the defensive mode a virtual-table dump requires is lifted for the restore and put back;
+- the schema is reloaded afterwards, so restored virtual tables work on the same handle;
+- `foreign_keys` is put back to its previous value (a native dump never re-enables it);
+- a transaction left open by a failing or truncated script is rolled back.