@procedures/sqlite 0.4.0 → 0.6.0

package/CHANGELOG.md

@@ -1,5 +1,47 @@
 # @procedures/sqlite
 
+## 0.6.0
+
+### Minor Changes
+
+- 72a8c65: The deprecated `db.*` aliases are gone: `db.query`, `db.execute`, `db.tables`, `db.schema`, `db.transaction`, `db.migrate` and `db.backup`. Use `sqlite.query` and the other `sqlite.*` names. Version 0.4.0 renamed them and kept the old names for one release.
+
+### Patch Changes
+
+- @procedures/client@0.6.0
+
+## 0.5.0
+
+### Minor Changes
+
+- 89481b0: `SqliteStorage` has two engines. The new option `engine` is `"auto"` (the default), `"node"` or `"sqljs"`.
+  
+  - The engine `node` is `node:sqlite` of Node.js 22.13 and later. SQLite writes only the changed pages of the file. Before, sql.js wrote the whole file after each write, so a write to a large collection was slow.
+  - In the nightly benchmark, 1,000 writes one after the other to a file of 1,000 items take 141 ms with `node` and 1,246 ms with `sqljs`.
+  - With `node`, the file is in WAL mode with a busy timeout (the new option `busyTimeout`, 5 seconds by default). Two processes can use one file: a write waits for the lock of the other process, and the versions of the change log agree.
+  - The engine `sqljs` is sql.js, as before. The engine `auto` uses it when the import of `node:sqlite` fails, for example on Node.js 22.12 and before.
+  - The two engines use one file format: a file of one engine opens with the other, with its items and its versions.
+  - The guarantees stay for each engine:
+    - Each write runs in one transaction.
+    - The storages of one file in one process share one database.
+    - The change log and its versions stay valid after a restart.
+    - A storage that only reads does not make the file.
+    - The contract of `close()` does not change.
+  - The storages of one file in one process share one engine. A storage with the other engine refuses the open database.
+  - sql.js refuses a file with a write-ahead log of the engine `node` that holds changes.
+  - New: `getEngine()`, `getIds()` (the ids without the values), `isNodeSqliteAvailable()`, and `stats.engine` in `getMetadata()`.
+  - `getChangesSince()` reads in one read transaction, and it reads the epoch from the file at each call.
+- bc74091: After `close()`, each operation of `SqliteStorage` and `MongoStorage` rejects with an error that says "closed", and a second `close()` does nothing. Before, `SqliteStorage.flush()` and an empty batch resolved after `close()`, and `MongoStorage` kept working.
+
+### Patch Changes
+
+- Updated dependencies [0f75644]
+- Updated dependencies [09e5566]
+- Updated dependencies [ab77f75]
+- Updated dependencies [a47bee6]
+- Updated dependencies [bc74091]
+  - @procedures/client@0.5.0
+
 ## 0.4.0
 
 ### Minor Changes

package/README.md

@@ -25,9 +25,9 @@ The package has 9 procedures. Its register entry (`import "@procedures/sqlite/re
 
 In version 0.3.0 and before, the database procedures had the namespace `db`. The general name `db` did not tell the database. Since version 0.4.0, the namespace is `sqlite`.
 
-The old names stay for one release as deprecated aliases: `db.query`, `db.execute`, `db.tables`, `db.schema`, `db.transaction`, `db.migrate` and `db.backup`. Each alias has the same handler, the same schemas and the same effects as its new procedure. Its description starts with `Deprecated: use sqlite.<name>`. The next release removes the aliases.
+Version 0.4.0 kept the old names for one release as deprecated aliases. Version 0.6.0 removed them: `db.query`, `db.execute`, `db.tables`, `db.schema`, `db.transaction`, `db.migrate` and `db.backup` give `NOT_FOUND`. Use the `sqlite.*` names.
 
-The MCP server (`bundle-mcp`) does not expose the aliases. Its tools are `sqlite.query`, `sqlite.execute`, `sqlite.tables` and `sqlite.schema`. The `logs.*` procedures keep their names.
+The tools of the MCP server (`bundle-mcp`) are `sqlite.query`, `sqlite.execute`, `sqlite.tables` and `sqlite.schema`. The `logs.*` procedures keep their names.
 
 ## Architecture
 
@@ -504,11 +504,38 @@ const users = new SqliteStorage<User>({ path: "./data/app.db", table: "users" })
 await users.set("u1", { name: "Ada" });
 await users.setBatch([["u2", { name: "Bob" }], ["u3", { name: "Cy" }]]);
 const changes = await users.getChangesSince(); // { version, full: true, changed, deleted: [] }
+await users.getEngine(); // "node" on Node.js 22.13 or later
 await users.close();
 ```
 
 The entry `@procedures/sqlite/storage` has the storage alone. The package root exports it too.
 
+### The engines
+
+The storage has two SQLite engines. The option `engine` selects one.
+
+| `engine` | The engine | The file write |
+|---|---|---|
+| `node` | `node:sqlite` of Node.js, without a native build | SQLite writes only the changed pages. The file is in WAL mode. |
+| `sqljs` | sql.js (SQLite in WebAssembly), through `@mark1russell7/docker-sqlite` | The storage keeps the database in memory, and it writes the whole file after a write. |
+| `auto` (the default) | `node` when Node.js has `node:sqlite`, and else `sqljs` | |
+
+- Node.js 22.13 and later have `node:sqlite` without a flag. Node.js 22.5 to 22.12 have it only with the flag `--experimental-sqlite`. Node.js 22.0 to 22.4 do not have it.
+- Node.js 22 shows the warning `ExperimentalWarning: SQLite is an experimental feature` at the first open of the engine `node`. Node.js 24 and 26 do not show it.
+- Node.js 26.11 renamed the class `DatabaseSync` to `Database`. The storage uses `Database` when the module has it, and else `DatabaseSync`.
+- With `auto`, the storage uses `sqljs` when the import of `node:sqlite` fails. With `node`, the open fails then, and each operation gives the error.
+- The two engines use the same file format. A file that one engine writes opens with the other engine, with its items and its versions.
+- `getEngine()` gives the engine of the storage. `getMetadata()` gives it in `stats.engine`. `isNodeSqliteAvailable()` tells if this Node.js has `node:sqlite`.
+
+The nightly benchmarks (`bench/storage.bench.ts`) measure the two engines in a file. These are the mean times of one CI run with Node.js 26 on Linux:
+
+| The benchmark | `node` | `sqljs` |
+|---|---|---|
+| 1,000 calls of `set()`, one after the other, in a table of 1,000 items | 141 ms | 1,246 ms |
+| One `getAll()` of a table of 10,000 items | 20 ms | 28 ms |
+
+The engine `sqljs` writes the whole file at each write, so its write time grows with the size of the file.
+
 ### The table and the values
 
 - The option `path` is a database file, or `:memory:` for a database that lives as long as the storage.
@@ -516,6 +543,7 @@ The entry `@procedures/sqlite/storage` has the storage alone. The package root e
 - The table has the columns `id` and `value`. The value is the JSON text of the item.
 - A value without JSON text, for example `undefined` or a `BigInt`, is a `TypeError`. The storage writes nothing then.
 - A read gives a new copy of the item.
+- `getIds()` gives the ids in the order of the items, and it does not read the values. A full sync of `HybridStorage` uses it.
 - The order of the items is the order of the first write of each id, as in a `Map`.
 
 ### The change log and the versions
@@ -524,17 +552,40 @@ The entry `@procedures/sqlite/storage` has the storage alone. The package root e
 - A version is `<epoch>:<seq>`, as in `InMemoryStorage`. The table `_storage_versions` keeps the epoch, so a version stays valid after a restart.
 - The table `_storage_log` keeps the last change of each id. The option `maxTombstones` (10000 by default) limits the deletes that it keeps.
 - A version that is older than the dropped deletes gives all items (`full: true`).
+- A delta follows the order of the last change of each id, as in each other storage. A full result follows the order of the items.
 - Write the table only through the storage. A change from other SQL is not in the change log.
 
 ### The transactions and the file
 
 - Each write runs in a transaction with the change log. A batch that fails changes nothing.
-- The storage keeps the database in memory, and it writes the file after each write. The write ends after the file write.
-- When several writes wait, one file write holds their changes.
-- A failed file write fails the write. The change stays in memory, and `flush()` or the next write writes it.
-- A storage that only reads does not make the file.
+- A write ends after its change is in the file.
+- A storage that only reads does not make the file. The storage makes the file, and its folder, at the first write that changes an item.
+- `close()` writes the file and releases the database. After it, each operation rejects, also `flush()` and an empty batch. A second `close()` does nothing.
 - The storages of one file in one process share one database. Thus the storages of two tables in one file keep their items.
-- Another process, and the procedures `sqlite.query` and `sqlite.execute`, do not share this database. Do not write the file from them while a storage has it open.
+- The first storage of a file sets the engine of the shared database. A later storage with the engine `auto` uses it. A later storage with the other engine refuses it.
+
+The engine `node`:
+
+- The commit of a transaction writes the change to the file. `flush()` has nothing to write.
+- The file is in WAL mode, with `synchronous = NORMAL`. After a crash, the file stays consistent. A power loss can lose the last commits.
+- A write that cannot make the file fails, and it changes nothing.
+- A read-only file opens. The reads work, and each write fails.
+
+The engine `sqljs`:
+
+- The storage keeps the database in memory, and it writes the whole file after each write. When several writes wait, one file write holds their changes.
+- A failed file write fails the write. The change stays in memory, and `flush()` or the next write writes it.
+- The storage refuses a file with a write-ahead log (`<file>-wal`) of the engine `node`. The log can hold changes that are not in the file. Open the file with the engine `node` one time, or close the process that has it open.
+
+### Two processes on one file
+
+- With the engine `node` in each process, two processes can use one file at the same time.
+- SQLite locks the file for each write. A write waits for the lock of another process for `busyTimeout` (5 seconds by default). After it, the write fails with `database is locked`.
+- In WAL mode, a read does not wait for a write of another process.
+- Each write reads the version counter in its transaction, so the versions of the two processes agree. `getChangesSince()` reads in one read transaction.
+- A storage that opened before the file existed opens the file when another process makes it.
+- The engine `sqljs` does not coordinate with another process. Its whole-file write replaces the writes of the other process. Do not use it on a file that another process writes.
+- The procedures `sqlite.query` and `sqlite.execute` use sql.js, and they do not share the database of the storages. Do not write the file from them while a storage has it open.
 
 ### The durable local copy of `HybridStorage`
 
@@ -838,8 +889,10 @@ client-sqlite/
 │   ├── module.ts             # The procedures as data, and the sqlite.query, sqlite.execute and logs.* handlers
 │   ├── register.ts           # The ./register entry: registers the module
 │   ├── storage.ts            # SqliteStorage, the collection storage (the ./storage entry)
+│   ├── engines.ts            # The engines of SqliteStorage: node:sqlite and sql.js
 │   ├── procedures/db/        # sqlite.tables, sqlite.schema, sqlite.transaction, sqlite.migrate, sqlite.backup
 │   └── index.ts              # Main entry point
+├── bench/                    # The benchmarks of the engines (the nightly workflow runs them)
 ├── dist/                     # Compiled output
 ├── package.json
 ├── tsconfig.json