dbgate-sqlite-dumper 0.1.0 → 0.1.1

package/docs/better-sqlite3-adapter.md

@@ -1,83 +1,83 @@
-# The `better-sqlite3` adapter
-
-```ts
-import { fromBetterSqlite3, connectBetterSqlite3 } from 'dbgate-sqlite-dumper/better-sqlite3';
-```
-
-The only module that knows `better-sqlite3` exists, reachable only through this separate
-entry point. `better-sqlite3` is an optional peer dependency: the type import costs nothing
-at runtime and the value import is dynamic, so the core package — and this module — load
-without it installed. `better-sqlite3` is the driver DbGate's own SQLite plugin uses.
-
-## Ownership
-
-| Function                                | Who closes the database                   |
-| --------------------------------------- | ----------------------------------------- |
-| `fromBetterSqlite3(database)`           | The caller. The database is **borrowed**. |
-| `connectBetterSqlite3(filename, opts?)` | The returned `close()`. Idempotent.       |
-
-`connectBetterSqlite3` passes `opts` straight to the `Database` constructor, so
-`{ readonly: true }` for a dump source, `{ fileMustExist: true }` and the rest all work.
-
-## Behaviour
-
-- **Integers are read as `bigint`** (`safeIntegers(true)`) for every catalog query, so no
-  value is ever rounded. Row data does not depend on this: it is read through SQL that has
-  SQLite render every value as text or bytes (see
-  [supported-data-types.md](supported-data-types.md)).
-- **`stream()` steps the statement with `iterate()`**, one row at a time, yielding between
-  rows — constant memory for a table of any size. While a stream is open, the database
-  cannot run another statement; this package never asks it to.
-- **`execute()` reports a statement's own row count** (`changes`), 0 for DDL — not the
-  count left over from the previous `INSERT`. Text holding several statements, which the
-  parser does not produce but other callers might, falls back to `exec()`.
-- **`isInTransaction()`** is `database.inTransaction`, which is what lets a restore roll
-  back exactly the transaction the script opened.
-- **`setDefensive()`** is `unsafeMode()`, which is how `better-sqlite3` exposes
-  `SQLITE_DBCONFIG_DEFENSIVE` — on by default in `better-sqlite3`. A restore lifts it only
-  for a dump that writes `sqlite_schema` (a virtual-table dump), and puts it back. Unsafe
-  mode also relaxes the driver's own guard against writing while iterating; this package
-  never does that.
-- **`cancel()` is a no-op.** `better-sqlite3` runs each statement synchronously to
-  completion, so cancellation takes effect between rows and between statements — prompt,
-  since every statement a dump or a restore runs is short.
-
-## Foreign keys
-
-`better-sqlite3` turns `foreign_keys` **on** for every database it opens; the `sqlite3`
-shell leaves it off. A full dump turns enforcement off itself, but a data-only dump does
-not, so loading one whose tables reference each other needs `disableForeignKeys: true`
-(see [restore-api.md](restore-api.md)).
-
-## Statistics tables
-
-`better-sqlite3` is built with `SQLITE_ENABLE_STAT4`, so `ANALYZE` — including the
-`ANALYZE sqlite_schema;` a dump runs — creates a `sqlite_stat4` table. A dump of a database
-that has one does not restore on a build without STAT4 (the native `.dump` of it fails the
-same way); `preflightRestore` reports it as `statistics-table-unsupported`.
-
-## Other drivers
-
-Anything that can run SQL can be adapted by implementing `SqliteConnection`:
-
-```ts
-interface SqliteConnection {
-  query(
-    query: { sql: string; parameters?: readonly unknown[] },
-    signal?,
-  ): Promise<{ rows: readonly Row[] }>;
-  stream(query, options?): AsyncIterable<Row>;
-  execute?(sql: string, signal?): Promise<{ changes: number }>; // run text verbatim, no parameter parsing
-  describeError?(error): { code?: string; errno?: number; message: string } | undefined;
-  isInTransaction?(): boolean;
-  setDefensive?(enabled: boolean): Promise<void>;
-  cancel(): Promise<void>;
-}
-```
-
-`query` and `stream` return rows as objects keyed by column name. Values may be any of
-SQLite's storage classes as the driver represents them; the core only relies on text,
-bytes and `null`, which every driver delivers exactly. The optional members degrade
-gracefully: without `execute`, statements go through `query`; without `isInTransaction`,
-restore tracks the script's own `BEGIN`/`COMMIT`; without `setDefensive`, a virtual-table
-dump restores only onto a handle that is not in defensive mode, and preflight says so.
+# The `better-sqlite3` adapter
+
+```ts
+import { fromBetterSqlite3, connectBetterSqlite3 } from 'dbgate-sqlite-dumper/better-sqlite3';
+```
+
+The only module that knows `better-sqlite3` exists, reachable only through this separate
+entry point. `better-sqlite3` is an optional peer dependency: the type import costs nothing
+at runtime and the value import is dynamic, so the core package — and this module — load
+without it installed. `better-sqlite3` is the driver DbGate's own SQLite plugin uses.
+
+## Ownership
+
+| Function                                | Who closes the database                   |
+| --------------------------------------- | ----------------------------------------- |
+| `fromBetterSqlite3(database)`           | The caller. The database is **borrowed**. |
+| `connectBetterSqlite3(filename, opts?)` | The returned `close()`. Idempotent.       |
+
+`connectBetterSqlite3` passes `opts` straight to the `Database` constructor, so
+`{ readonly: true }` for a dump source, `{ fileMustExist: true }` and the rest all work.
+
+## Behaviour
+
+- **Integers are read as `bigint`** (`safeIntegers(true)`) for every catalog query, so no
+  value is ever rounded. Row data does not depend on this: it is read through SQL that has
+  SQLite render every value as text or bytes (see
+  [supported-data-types.md](supported-data-types.md)).
+- **`stream()` steps the statement with `iterate()`**, one row at a time, yielding between
+  rows — constant memory for a table of any size. While a stream is open, the database
+  cannot run another statement; this package never asks it to.
+- **`execute()` reports a statement's own row count** (`changes`), 0 for DDL — not the
+  count left over from the previous `INSERT`. Text holding several statements, which the
+  parser does not produce but other callers might, falls back to `exec()`.
+- **`isInTransaction()`** is `database.inTransaction`, which is what lets a restore roll
+  back exactly the transaction the script opened.
+- **`setDefensive()`** is `unsafeMode()`, which is how `better-sqlite3` exposes
+  `SQLITE_DBCONFIG_DEFENSIVE` — on by default in `better-sqlite3`. A restore lifts it only
+  for a dump that writes `sqlite_schema` (a virtual-table dump), and puts it back. Unsafe
+  mode also relaxes the driver's own guard against writing while iterating; this package
+  never does that.
+- **`cancel()` is a no-op.** `better-sqlite3` runs each statement synchronously to
+  completion, so cancellation takes effect between rows and between statements — prompt,
+  since every statement a dump or a restore runs is short.
+
+## Foreign keys
+
+`better-sqlite3` turns `foreign_keys` **on** for every database it opens; the `sqlite3`
+shell leaves it off. A full dump turns enforcement off itself, but a data-only dump does
+not, so loading one whose tables reference each other needs `disableForeignKeys: true`
+(see [restore-api.md](restore-api.md)).
+
+## Statistics tables
+
+`better-sqlite3` is built with `SQLITE_ENABLE_STAT4`, so `ANALYZE` — including the
+`ANALYZE sqlite_schema;` a dump runs — creates a `sqlite_stat4` table. A dump of a database
+that has one does not restore on a build without STAT4 (the native `.dump` of it fails the
+same way); `preflightRestore` reports it as `statistics-table-unsupported`.
+
+## Other drivers
+
+Anything that can run SQL can be adapted by implementing `SqliteConnection`:
+
+```ts
+interface SqliteConnection {
+  query(
+    query: { sql: string; parameters?: readonly unknown[] },
+    signal?,
+  ): Promise<{ rows: readonly Row[] }>;
+  stream(query, options?): AsyncIterable<Row>;
+  execute?(sql: string, signal?): Promise<{ changes: number }>; // run text verbatim, no parameter parsing
+  describeError?(error): { code?: string; errno?: number; message: string } | undefined;
+  isInTransaction?(): boolean;
+  setDefensive?(enabled: boolean): Promise<void>;
+  cancel(): Promise<void>;
+}
+```
+
+`query` and `stream` return rows as objects keyed by column name. Values may be any of
+SQLite's storage classes as the driver represents them; the core only relies on text,
+bytes and `null`, which every driver delivers exactly. The optional members degrade
+gracefully: without `execute`, statements go through `query`; without `isInTransaction`,
+restore tracks the script's own `BEGIN`/`COMMIT`; without `setDefensive`, a virtual-table
+dump restores only onto a handle that is not in defensive mode, and preflight says so.

package/docs/d1-adapter.md

@@ -0,0 +1,98 @@
+# The Cloudflare D1 adapter
+
+```ts
+import { fromD1Http, fromD1Binding } from 'dbgate-sqlite-dumper/d1';
+```
+
+Dumps a [Cloudflare D1](https://developers.cloudflare.com/d1/) database into the same
+plain-SQL file the native `sqlite3 .dump` would write for a local copy of it. The adapter
+has no dependencies: it uses the global `fetch` (or the one you pass it).
+
+**Dump only.** Restoring into D1 is not supported: D1 refuses the `BEGIN TRANSACTION` /
+`COMMIT` and `PRAGMA writable_schema` statements a dump script relies on. To load a dump
+into D1, use `wrangler d1 execute <db> --remote --file=dump.sql`, after removing those
+statements as Cloudflare's import guide describes. A D1 dump restores into an ordinary
+SQLite database with `restoreSqlDump` or the `sqlite3` shell as usual.
+
+## Over the REST API
+
+```ts
+import fs from 'node:fs';
+import { dumpSqlite } from 'dbgate-sqlite-dumper';
+import { fromD1Http } from 'dbgate-sqlite-dumper/d1';
+
+const connection = fromD1Http({
+  accountId: process.env.CLOUDFLARE_ACCOUNT_ID,
+  databaseId: 'c8a3...', // the UUID `wrangler d1 list` shows
+  apiToken: process.env.CLOUDFLARE_API_TOKEN, // needs D1 Read
+});
+const result = await dumpSqlite(connection, {}, fs.createWriteStream('backup.sql'));
+```
+
+| Option       | Default                                | Meaning                                  |
+| ------------ | -------------------------------------- | ---------------------------------------- |
+| `accountId`  | required                               | Cloudflare account ID                    |
+| `databaseId` | required                               | The database's UUID                      |
+| `apiToken`   | required                               | API token with `D1 Read` (or `D1 Edit`)  |
+| `apiBaseUrl` | `https://api.cloudflare.com/client/v4` | For a proxy or a test server             |
+| `pageSize`   | `1000`                                 | Rows per request when table data is read |
+| `fetch`      | `globalThis.fetch`                     | Any `fetch`-compatible function          |
+
+Statements go to the `/raw` query endpoint. The catalog of every table is fetched in
+`batch` requests of up to 50 statements, so introspection takes a handful of round trips
+whatever the number of tables. The token is sent only in the `Authorization` header and is
+never part of an error message. Errors are `D1Error`s carrying the message Cloudflare
+returned, the HTTP `status`, and the `SQLITE_*` code when the message names one.
+
+`connection.cancel()` (and the dump's `AbortSignal`) aborts the request in flight.
+
+## Inside a Worker
+
+```ts
+import { dumpSqlite, BufferDumpWriter } from 'dbgate-sqlite-dumper';
+import { fromD1Binding } from 'dbgate-sqlite-dumper/d1';
+
+const connection = fromD1Binding(env.DB, { pageSize: 1000 });
+```
+
+Uses `prepare().bind().raw({ columnNames: true })` and `batch()`. The package uses
+`Buffer`, so the Worker needs the `nodejs_compat` compatibility flag.
+
+## How D1's restrictions are handled
+
+The adapter declares what D1 does not allow as `connection.features`
+(`d1ConnectionFeatures()`), and the core reads the database with queries D1 does accept.
+None of this changes the dump; each is tested against an emulation of D1 that refuses what
+D1 refuses, and must produce a dump byte-identical to the one of a local copy.
+
+| D1 restriction                                        | What the dump does instead                                                                                                                           |
+| ----------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
+| No `BEGIN` / `SAVEPOINT`                              | Reads without a snapshot, and reports `snapshot-unavailable` (see below)                                                                             |
+| No table-valued `pragma_xxx()` functions              | Uses the `PRAGMA table_info('t')` statement forms                                                                                                    |
+| Possibly no `table_xinfo` / `index_xinfo`             | Falls back to `table_info` / `index_info` the first time an extended pragma is refused                                                               |
+| One database, `main`                                  | Names no schema in any statement                                                                                                                     |
+| JSON results: no bytes, integers as doubles           | Text and blobs are fetched as `hex()`; numbers already arrive as SQLite-rendered text                                                                |
+| Whole results, no streaming                           | Table data is read in pages: `WHERE rowid > :last ORDER BY rowid LIMIT n`, or on the primary key of a `WITHOUT ROWID` table — an index seek per page |
+| Reserved `_cf_` tables in `sqlite_schema`, unreadable | Left out, with their `sqlite_sequence` and `sqlite_stat*` rows; reported as `reserved-objects-skipped`                                               |
+
+### Consistency
+
+D1 has no read transactions a client can hold across requests, so a D1 dump is not a
+snapshot: a write committed while the dump runs may be partly included. The result carries
+a `snapshot-unavailable` warning. For a consistent copy, dump while nothing writes, or
+create a [Time Travel](https://developers.cloudflare.com/d1/reference/time-travel/)
+bookmark first and dump a database restored from it.
+
+### Row order of `WITHOUT ROWID` tables
+
+Pages follow the primary-key index, so rows come out in the order the native `.dump`
+writes them. When D1 refuses `index_xinfo`, the key's direction and collation are not
+known; pages then follow the declared key, ascending — the same rows, possibly in a
+different order for a key declared `DESC` or with its own `COLLATE`.
+
+## Other engines
+
+The features are not specific to D1. Any adapter for an engine that speaks SQLite's SQL
+with similar restrictions — over HTTP, or behind an authorizer — can declare the
+same `SqliteConnectionFeatures` and get the same treatment. See
+[architecture.md](architecture.md) for the `SqliteConnection` contract.