@procedures/sqlite 0.3.0 → 0.5.0

package/CHANGELOG.md

@@ -1,5 +1,62 @@
 # @procedures/sqlite
 
+## 0.5.0
+
+### Minor Changes
+
+- 89481b0: `SqliteStorage` has two engines. The new option `engine` is `"auto"` (the default), `"node"` or `"sqljs"`.
+  
+  - The engine `node` is `node:sqlite` of Node.js 22.13 and later. SQLite writes only the changed pages of the file. Before, sql.js wrote the whole file after each write, so a write to a large collection was slow.
+  - In the nightly benchmark, 1,000 writes one after the other to a file of 1,000 items take 141 ms with `node` and 1,246 ms with `sqljs`.
+  - With `node`, the file is in WAL mode with a busy timeout (the new option `busyTimeout`, 5 seconds by default). Two processes can use one file: a write waits for the lock of the other process, and the versions of the change log agree.
+  - The engine `sqljs` is sql.js, as before. The engine `auto` uses it when the import of `node:sqlite` fails, for example on Node.js 22.12 and before.
+  - The two engines use one file format: a file of one engine opens with the other, with its items and its versions.
+  - The guarantees stay for each engine:
+    - Each write runs in one transaction.
+    - The storages of one file in one process share one database.
+    - The change log and its versions stay valid after a restart.
+    - A storage that only reads does not make the file.
+    - The contract of `close()` does not change.
+  - The storages of one file in one process share one engine. A storage with the other engine refuses the open database.
+  - sql.js refuses a file with a write-ahead log of the engine `node` that holds changes.
+  - New: `getEngine()`, `getIds()` (the ids without the values), `isNodeSqliteAvailable()`, and `stats.engine` in `getMetadata()`.
+  - `getChangesSince()` reads in one read transaction, and it reads the epoch from the file at each call.
+- bc74091: After `close()`, each operation of `SqliteStorage` and `MongoStorage` rejects with an error that says "closed", and a second `close()` does nothing. Before, `SqliteStorage.flush()` and an empty batch resolved after `close()`, and `MongoStorage` kept working.
+
+### Patch Changes
+
+- Updated dependencies [0f75644]
+- Updated dependencies [09e5566]
+- Updated dependencies [ab77f75]
+- Updated dependencies [a47bee6]
+- Updated dependencies [bc74091]
+  - @procedures/client@0.5.0
+
+## 0.4.0
+
+### Minor Changes
+
+- cc3f5f6: The SQLite procedures have the namespace `sqlite`, not the general namespace `db`: `sqlite.query`, `sqlite.execute`, `sqlite.tables`, `sqlite.schema`, `sqlite.transaction`, `sqlite.migrate` and `sqlite.backup`. The `logs.*` procedures keep their names.
+  
+  The old names stay for one release as deprecated aliases (`db.query` and the others). Each alias has the same handler, the same schemas and the same effects. Its description starts with `Deprecated: use sqlite.<name>`, and its metadata has `deprecated: true`. The next release removes the aliases.
+  
+  The description of the `dbPath` field does not show the home folder of the computer any more. It says `~/logs/cli/cli.db`.
+
+### Patch Changes
+
+- Updated dependencies [d919678]
+- Updated dependencies [057d69a]
+- Updated dependencies [b2a1a1e]
+- Updated dependencies [057d69a]
+- Updated dependencies [057d69a]
+- Updated dependencies [057d69a]
+- Updated dependencies [057d69a]
+- Updated dependencies [057d69a]
+- Updated dependencies [afd6a19]
+- Updated dependencies [057d69a]
+- Updated dependencies [4e751da]
+  - @procedures/client@0.4.0
+
 ## 0.3.0
 
 ### Minor Changes

package/README.md

@@ -18,9 +18,17 @@ SQLite database operations via client procedures. Execute queries and store stru
 - **A collection storage**: `SqliteStorage` keeps a collection in a SQLite table, with a change log
 
 The package has 9 procedures. Its register entry (`import "@procedures/sqlite/register"`) registers them:
-- **Database**: `db.query`, `db.execute`, `db.tables`, `db.schema`, `db.transaction`, `db.migrate`, `db.backup`
+- **Database**: `sqlite.query`, `sqlite.execute`, `sqlite.tables`, `sqlite.schema`, `sqlite.transaction`, `sqlite.migrate`, `sqlite.backup`
 - **Logs**: `logs.store`, `logs.query`
 
+### The old names `db.*`
+
+In version 0.3.0 and before, the database procedures had the namespace `db`. The general name `db` did not tell the database. Since version 0.4.0, the namespace is `sqlite`.
+
+The old names stay for one release as deprecated aliases: `db.query`, `db.execute`, `db.tables`, `db.schema`, `db.transaction`, `db.migrate` and `db.backup`. Each alias has the same handler, the same schemas and the same effects as its new procedure. Its description starts with `Deprecated: use sqlite.<name>`. The next release removes the aliases.
+
+The MCP server (`bundle-mcp`) does not expose the aliases. Its tools are `sqlite.query`, `sqlite.execute`, `sqlite.tables` and `sqlite.schema`. The `logs.*` procedures keep their names.
+
 ## Architecture
 
 ```mermaid
@@ -34,7 +42,7 @@ graph TB
         Register["register.ts<br/>(Auto-registers procedures)"]
 
         subgraph "Procedure Categories"
-            DB["Database Procedures<br/>db.query, db.execute, db.tables, db.schema,<br/>db.transaction, db.migrate, db.backup"]
+            DB["Database Procedures<br/>sqlite.query, sqlite.execute, sqlite.tables, sqlite.schema,<br/>sqlite.transaction, sqlite.migrate, sqlite.backup"]
             Logs["Log Procedures<br/>logs.store, logs.query"]
         end
 
@@ -148,7 +156,7 @@ You can override this path in each procedure call:
 
 ```typescript
 await client.exec(
-  ["db", "query"],
+  ["sqlite", "query"],
   {
     sql: "SELECT * FROM logs",
     dbPath: "./custom/path/database.db"
@@ -160,13 +168,13 @@ await client.exec(
 
 ### Database Procedures
 
-#### `db.query`
+#### `sqlite.query`
 
 Execute a SQL SELECT query and return rows.
 
 ```typescript
 const result = await client.exec(
-  ["db", "query"],
+  ["sqlite", "query"],
   {
     sql: "SELECT * FROM logs WHERE level = ? ORDER BY timestamp DESC",
     params: ["error"]
@@ -197,7 +205,7 @@ interface DbQueryOutput<T = Record<string, unknown>> {
 **Example:**
 ```typescript
 const result = await client.exec(
-  ["db", "query"],
+  ["sqlite", "query"],
   {
     sql: `
       SELECT level, COUNT(*) as count
@@ -212,13 +220,13 @@ const result = await client.exec(
 // result.rows = [{ level: "info", count: 45 }, { level: "error", count: 2 }]
 ```
 
-#### `db.execute`
+#### `sqlite.execute`
 
 Execute a SQL statement (INSERT, UPDATE, DELETE) and return the number of affected rows.
 
 ```typescript
 const result = await client.exec(
-  ["db", "execute"],
+  ["sqlite", "execute"],
   {
     sql: "DELETE FROM logs WHERE timestamp < ?",
     params: ["2024-01-01"]
@@ -248,7 +256,7 @@ interface DbExecuteOutput {
 ```typescript
 // INSERT
 await client.exec(
-  ["db", "execute"],
+  ["sqlite", "execute"],
   {
     sql: "INSERT INTO users (name, email) VALUES (?, ?)",
     params: ["Alice", "alice@example.com"]
@@ -257,7 +265,7 @@ await client.exec(
 
 // UPDATE
 await client.exec(
-  ["db", "execute"],
+  ["sqlite", "execute"],
   {
     sql: "UPDATE users SET active = ? WHERE id = ?",
     params: [true, 123]
@@ -266,7 +274,7 @@ await client.exec(
 
 // DELETE
 await client.exec(
-  ["db", "execute"],
+  ["sqlite", "execute"],
   {
     sql: "DELETE FROM users WHERE active = ?",
     params: [false]
@@ -274,29 +282,29 @@ await client.exec(
 );
 ```
 
-#### `db.tables`
+#### `sqlite.tables`
 
 The procedure lists the tables and the views of a database, without the internal `sqlite_` tables. Each item has the `name`, the `type` and the `sql` that made it. With `views: false`, it gives only the tables. The procedure only reads. A missing file is an error (`ENOENT`), and the procedure does not make it.
 
 ```typescript
-const { tables } = await client.exec(["db", "tables"], { dbPath: "./app.db" });
+const { tables } = await client.exec(["sqlite", "tables"], { dbPath: "./app.db" });
 ```
 
-#### `db.schema`
+#### `sqlite.schema`
 
 The procedure gives the `columns`, the `indexes` and the `foreignKeys` of a table, or the columns of a view. The name of the table goes to SQLite as a bound value, not as SQL text. The procedure only reads.
 
 ```typescript
-const schema = await client.exec(["db", "schema"], { dbPath: "./app.db", table: "users" });
+const schema = await client.exec(["sqlite", "schema"], { dbPath: "./app.db", table: "users" });
 // schema.columns: [{ name: "id", type: "INTEGER", notNull: false, defaultValue: null, primaryKey: 1 }, ...]
 ```
 
-#### `db.transaction`
+#### `sqlite.transaction`
 
 The procedure puts a list of statements in one transaction. When one statement fails, it rolls back the transaction, and the file stays as it was. The `params` of a statement bind to one statement. Without `params`, one `sql` can hold several statements. A statement that gives rows (`SELECT`, or `RETURNING`) gives its `rows` in its result. The procedure refuses `BEGIN`, `COMMIT`, `ROLLBACK`, `SAVEPOINT` and `RELEASE`, because it controls the transaction itself.
 
 ```typescript
-const { results, changes } = await client.exec(["db", "transaction"], {
+const { results, changes } = await client.exec(["sqlite", "transaction"], {
   dbPath: "./app.db",
   statements: [
     { sql: "UPDATE accounts SET balance = balance - ? WHERE id = ?", params: [10, 1] },
@@ -306,7 +314,7 @@ const { results, changes } = await client.exec(["db", "transaction"], {
 });
 ```
 
-#### `db.migrate`
+#### `sqlite.migrate`
 
 The procedure applies the numbered migrations that the database does not have, in the order of their numbers. The migrations come from a `folder` or from a list (`migrations`). In a folder, each migration is a file `<number>_<name>.sql`, for example `001_create_users.sql`. The procedure ignores the files that do not end in `.sql`.
 
@@ -317,20 +325,20 @@ The procedure applies the numbered migrations that the database does not have, i
 - A migration must not hold `BEGIN`, `COMMIT` or `ROLLBACK`.
 
 ```typescript
-const { applied, skipped, version } = await client.exec(["db", "migrate"], {
+const { applied, skipped, version } = await client.exec(["sqlite", "migrate"], {
   dbPath: "./app.db",
   folder: "./migrations",
 });
 ```
 
-#### `db.backup`
+#### `sqlite.backup`
 
 The procedure copies a database to a new file (`to`). It does not replace a file, and it does not write the source. With `vacuum: true`, it compacts the copy.
 
 This package uses SQLite in WebAssembly (sql.js). The database is in the memory of the module. `VACUUM INTO` and the backup API of SQLite write into the file system of the module, not to the disk. Thus the procedure writes the image of the database (its `export()`). It takes the image in the file queue of the database, where no other call of the process changes it. It writes the image to a file next to the target, checks it with `PRAGMA integrity_check`, and then renames it.
 
 ```typescript
-const { path, bytes } = await client.exec(["db", "backup"], { dbPath: "./app.db", to: "./backups/app-2026-10-16.db" });
+const { path, bytes } = await client.exec(["sqlite", "backup"], { dbPath: "./app.db", to: "./backups/app-2026-10-16.db" });
 ```
 
 ### Log Procedures
@@ -496,11 +504,38 @@ const users = new SqliteStorage<User>({ path: "./data/app.db", table: "users" })
 await users.set("u1", { name: "Ada" });
 await users.setBatch([["u2", { name: "Bob" }], ["u3", { name: "Cy" }]]);
 const changes = await users.getChangesSince(); // { version, full: true, changed, deleted: [] }
+await users.getEngine(); // "node" on Node.js 22.13 or later
 await users.close();
 ```
 
 The entry `@procedures/sqlite/storage` has the storage alone. The package root exports it too.
 
+### The engines
+
+The storage has two SQLite engines. The option `engine` selects one.
+
+| `engine` | The engine | The file write |
+|---|---|---|
+| `node` | `node:sqlite` of Node.js, without a native build | SQLite writes only the changed pages. The file is in WAL mode. |
+| `sqljs` | sql.js (SQLite in WebAssembly), through `@mark1russell7/docker-sqlite` | The storage keeps the database in memory, and it writes the whole file after a write. |
+| `auto` (the default) | `node` when Node.js has `node:sqlite`, and else `sqljs` | |
+
+- Node.js 22.13 and later have `node:sqlite` without a flag. Node.js 22.5 to 22.12 have it only with the flag `--experimental-sqlite`. Node.js 22.0 to 22.4 do not have it.
+- Node.js 22 shows the warning `ExperimentalWarning: SQLite is an experimental feature` at the first open of the engine `node`. Node.js 24 and 26 do not show it.
+- Node.js 26.11 renamed the class `DatabaseSync` to `Database`. The storage uses `Database` when the module has it, and else `DatabaseSync`.
+- With `auto`, the storage uses `sqljs` when the import of `node:sqlite` fails. With `node`, the open fails then, and each operation gives the error.
+- The two engines use the same file format. A file that one engine writes opens with the other engine, with its items and its versions.
+- `getEngine()` gives the engine of the storage. `getMetadata()` gives it in `stats.engine`. `isNodeSqliteAvailable()` tells if this Node.js has `node:sqlite`.
+
+The nightly benchmarks (`bench/storage.bench.ts`) measure the two engines in a file. These are the mean times of one CI run with Node.js 26 on Linux:
+
+| The benchmark | `node` | `sqljs` |
+|---|---|---|
+| 1,000 calls of `set()`, one after the other, in a table of 1,000 items | 141 ms | 1,246 ms |
+| One `getAll()` of a table of 10,000 items | 20 ms | 28 ms |
+
+The engine `sqljs` writes the whole file at each write, so its write time grows with the size of the file.
+
 ### The table and the values
 
 - The option `path` is a database file, or `:memory:` for a database that lives as long as the storage.
@@ -508,6 +543,7 @@ The entry `@procedures/sqlite/storage` has the storage alone. The package root e
 - The table has the columns `id` and `value`. The value is the JSON text of the item.
 - A value without JSON text, for example `undefined` or a `BigInt`, is a `TypeError`. The storage writes nothing then.
 - A read gives a new copy of the item.
+- `getIds()` gives the ids in the order of the items, and it does not read the values. A full sync of `HybridStorage` uses it.
 - The order of the items is the order of the first write of each id, as in a `Map`.
 
 ### The change log and the versions
@@ -516,17 +552,40 @@ The entry `@procedures/sqlite/storage` has the storage alone. The package root e
 - A version is `<epoch>:<seq>`, as in `InMemoryStorage`. The table `_storage_versions` keeps the epoch, so a version stays valid after a restart.
 - The table `_storage_log` keeps the last change of each id. The option `maxTombstones` (10000 by default) limits the deletes that it keeps.
 - A version that is older than the dropped deletes gives all items (`full: true`).
+- A delta follows the order of the last change of each id, as in each other storage. A full result follows the order of the items.
 - Write the table only through the storage. A change from other SQL is not in the change log.
 
 ### The transactions and the file
 
 - Each write runs in a transaction with the change log. A batch that fails changes nothing.
-- The storage keeps the database in memory, and it writes the file after each write. The write ends after the file write.
-- When several writes wait, one file write holds their changes.
-- A failed file write fails the write. The change stays in memory, and `flush()` or the next write writes it.
-- A storage that only reads does not make the file.
+- A write ends after its change is in the file.
+- A storage that only reads does not make the file. The storage makes the file, and its folder, at the first write that changes an item.
+- `close()` writes the file and releases the database. After it, each operation rejects, also `flush()` and an empty batch. A second `close()` does nothing.
 - The storages of one file in one process share one database. Thus the storages of two tables in one file keep their items.
-- Another process, and the procedures `db.query` and `db.execute`, do not share this database. Do not write the file from them while a storage has it open.
+- The first storage of a file sets the engine of the shared database. A later storage with the engine `auto` uses it. A later storage with the other engine refuses it.
+
+The engine `node`:
+
+- The commit of a transaction writes the change to the file. `flush()` has nothing to write.
+- The file is in WAL mode, with `synchronous = NORMAL`. After a crash, the file stays consistent. A power loss can lose the last commits.
+- A write that cannot make the file fails, and it changes nothing.
+- A read-only file opens. The reads work, and each write fails.
+
+The engine `sqljs`:
+
+- The storage keeps the database in memory, and it writes the whole file after each write. When several writes wait, one file write holds their changes.
+- A failed file write fails the write. The change stays in memory, and `flush()` or the next write writes it.
+- The storage refuses a file with a write-ahead log (`<file>-wal`) of the engine `node`. The log can hold changes that are not in the file. Open the file with the engine `node` one time, or close the process that has it open.
+
+### Two processes on one file
+
+- With the engine `node` in each process, two processes can use one file at the same time.
+- SQLite locks the file for each write. A write waits for the lock of another process for `busyTimeout` (5 seconds by default). After it, the write fails with `database is locked`.
+- In WAL mode, a read does not wait for a write of another process.
+- Each write reads the version counter in its transaction, so the versions of the two processes agree. `getChangesSince()` reads in one read transaction.
+- A storage that opened before the file existed opens the file when another process makes it.
+- The engine `sqljs` does not coordinate with another process. Its whole-file write replaces the writes of the other process. Do not use it on a file that another process writes.
+- The procedures `sqlite.query` and `sqlite.execute` use sql.js, and they do not share the database of the storages. Do not write the file from them while a storage has it open.
 
 ### The durable local copy of `HybridStorage`
 
@@ -556,7 +615,7 @@ const client = new Client(new LocalTransport({ registry: PROCEDURE_REGISTRY }));
 
 // Create a table
 await client.exec(
-  ["db", "execute"],
+  ["sqlite", "execute"],
   {
     sql: `
       CREATE TABLE IF NOT EXISTS users (
@@ -573,7 +632,7 @@ await client.exec(
 
 // Insert data
 await client.exec(
-  ["db", "execute"],
+  ["sqlite", "execute"],
   {
     sql: "INSERT INTO users (name, email) VALUES (?, ?)",
     params: ["Alice Johnson", "alice@example.com"],
@@ -583,7 +642,7 @@ await client.exec(
 
 // Query data
 const result = await client.exec(
-  ["db", "query"],
+  ["sqlite", "query"],
   {
     sql: "SELECT * FROM users WHERE active = ? ORDER BY created_at DESC",
     params: [1],
@@ -747,7 +806,7 @@ interface User {
 }
 
 const result = await client.call<DbQueryOutput<User>>(
-  ["db", "query"],
+  ["sqlite", "query"],
   {
     sql: "SELECT * FROM users WHERE active = ?",
     params: [true],
@@ -827,11 +886,13 @@ export type {
 client-sqlite/
 ├── src/
 │   ├── types.ts              # Type definitions
-│   ├── module.ts             # The procedures as data, and the db.query, db.execute and logs.* handlers
+│   ├── module.ts             # The procedures as data, and the sqlite.query, sqlite.execute and logs.* handlers
 │   ├── register.ts           # The ./register entry: registers the module
 │   ├── storage.ts            # SqliteStorage, the collection storage (the ./storage entry)
-│   ├── procedures/db/        # db.tables, db.schema, db.transaction, db.migrate, db.backup
+│   ├── engines.ts            # The engines of SqliteStorage: node:sqlite and sql.js
+│   ├── procedures/db/        # sqlite.tables, sqlite.schema, sqlite.transaction, sqlite.migrate, sqlite.backup
 │   └── index.ts              # Main entry point
+├── bench/                    # The benchmarks of the engines (the nightly workflow runs them)
 ├── dist/                     # Compiled output
 ├── package.json
 ├── tsconfig.json