@procedures/sqlite 0.2.3 → 0.4.0

package/CHANGELOG.md

@@ -1,5 +1,79 @@
 # @procedures/sqlite
 
+## 0.4.0
+
+### Minor Changes
+
+- cc3f5f6: The SQLite procedures have the namespace `sqlite`, not the general namespace `db`: `sqlite.query`, `sqlite.execute`, `sqlite.tables`, `sqlite.schema`, `sqlite.transaction`, `sqlite.migrate` and `sqlite.backup`. The `logs.*` procedures keep their names.
+  
+  The old names stay for one release as deprecated aliases (`db.query` and the others). Each alias has the same handler, the same schemas and the same effects. Its description starts with `Deprecated: use sqlite.<name>`, and its metadata has `deprecated: true`. The next release removes the aliases.
+  
+  The description of the `dbPath` field does not show the home folder of the computer any more. It says `~/logs/cli/cli.db`.
+
+### Patch Changes
+
+- Updated dependencies [d919678]
+- Updated dependencies [057d69a]
+- Updated dependencies [b2a1a1e]
+- Updated dependencies [057d69a]
+- Updated dependencies [057d69a]
+- Updated dependencies [057d69a]
+- Updated dependencies [057d69a]
+- Updated dependencies [057d69a]
+- Updated dependencies [afd6a19]
+- Updated dependencies [057d69a]
+- Updated dependencies [4e751da]
+  - @procedures/client@0.4.0
+
+## 0.3.0
+
+### Minor Changes
+
+- 89d8a4a: Three procedures only read now (CAP-1, CAP-2, CAP-3, roadmap R7), so a read-only session (`mark mcp --read-only`) serves them. `db.query` refuses a statement that changes data (`PRAGMA query_only`), and the file keeps its bytes. `mongo.documents.aggregate` refuses `$out` and `$merge`, also in the pipelines of `$facet`, `$lookup` and `$unionWith`. `logs.query` gives no entries for a missing file or table, and does not make them.
+- 94dd2a0: Five database procedures (roadmap R44). `db.tables` lists the tables and views, and `db.schema` gives the columns, indexes and foreign keys of a table. Both only read, and a missing file is an error. `db.query` stays read only.
+  
+  `db.transaction` puts a list of statements in one transaction, and rolls it back when one fails. `db.migrate` applies numbered migrations from a folder or a list, in one transaction, with a table of the applied migrations and their sha256. A second call applies nothing. `db.backup` copies a database to a new file and checks the copy. It does not replace a file. With `vacuum: true`, it compacts the copy.
+- 4ef0d44: The new collection storage `SqliteStorage` (roadmap R45) keeps the items of one collection in one table of a SQLite database: a file, or `:memory:`. It implements `CollectionStorage`, so a `ProcedureServer`, an `ApiStorage` server and a `HybridStorage` use it as they use `InMemoryStorage`. The new entry `@procedures/sqlite/storage` has it alone, and the package root exports it too.
+  
+  The table has the columns `id` and `value` (the JSON text of the item). The name of the table starts with a letter and has only letters, digits and underscores. `getChangesSince()` reads a change log in the tables `_storage_log` and `_storage_versions`, with versions `<epoch>:<seq>` as `InMemoryStorage` has. The epoch stays in the file, so a version stays valid after a restart. The option `maxTombstones` limits the deletes that the log keeps.
+  
+  Each write runs in a transaction, so a batch that fails changes nothing. The storage keeps the database in memory and writes the file after each write. One file write can hold the changes of several writes. The storages of one file in one process share one database. A storage that only reads does not make the file.
+
+### Patch Changes
+
+- Updated dependencies [ddea529]
+- Updated dependencies [b2a3e3a]
+- Updated dependencies [a8c47f1]
+- Updated dependencies [381ce3a]
+- Updated dependencies [0c627f8]
+- Updated dependencies [ae7f05f]
+- Updated dependencies [7873940]
+- Updated dependencies [a70d445]
+- Updated dependencies [bc70357]
+- Updated dependencies [07cc998]
+- Updated dependencies [a6b01a9]
+- Updated dependencies [c799cdd]
+- Updated dependencies [540f8d9]
+- Updated dependencies [b6daaec]
+- Updated dependencies [5b9264e]
+- Updated dependencies [956e88b]
+- Updated dependencies [ff62281]
+- Updated dependencies [d81d0ed]
+- Updated dependencies [aa67cc7]
+- Updated dependencies [061fa22]
+- Updated dependencies [cfb9d69]
+- Updated dependencies [43f7de5]
+- Updated dependencies [b07bb50]
+- Updated dependencies [4a3bfa1]
+- Updated dependencies [bd0b626]
+- Updated dependencies [a6be873]
+- Updated dependencies [fb30718]
+- Updated dependencies [c5528e4]
+- Updated dependencies [fee5d9c]
+- Updated dependencies [21f7743]
+- Updated dependencies [31a0b46]
+  - @procedures/client@0.3.0
+
 ## 0.2.3
 
 ### Patch Changes

package/README.md

@@ -15,11 +15,20 @@ SQLite database operations via client procedures. Execute queries and store stru
 - **Logging procedures**: Store and query structured application logs
 - **Migration support**: Schema versioning and migration utilities
 - **Default log database**: Pre-configured logging at `~/logs/cli/cli.db`
+- **A collection storage**: `SqliteStorage` keeps a collection in a SQLite table, with a change log
 
-The package has 4 procedures. Its register entry (`import "@procedures/sqlite/register"`) registers them:
-- **Database**: `db.query`, `db.execute`
+The package has 9 procedures. Its register entry (`import "@procedures/sqlite/register"`) registers them:
+- **Database**: `sqlite.query`, `sqlite.execute`, `sqlite.tables`, `sqlite.schema`, `sqlite.transaction`, `sqlite.migrate`, `sqlite.backup`
 - **Logs**: `logs.store`, `logs.query`
 
+### The old names `db.*`
+
+In version 0.3.0 and before, the database procedures had the namespace `db`. The general name `db` did not tell the database. Since version 0.4.0, the namespace is `sqlite`.
+
+The old names stay for one release as deprecated aliases: `db.query`, `db.execute`, `db.tables`, `db.schema`, `db.transaction`, `db.migrate` and `db.backup`. Each alias has the same handler, the same schemas and the same effects as its new procedure. Its description starts with `Deprecated: use sqlite.<name>`. The next release removes the aliases.
+
+The MCP server (`bundle-mcp`) does not expose the aliases. Its tools are `sqlite.query`, `sqlite.execute`, `sqlite.tables` and `sqlite.schema`. The `logs.*` procedures keep their names.
+
 ## Architecture
 
 ```mermaid
@@ -33,7 +42,7 @@ graph TB
         Register["register.ts<br/>(Auto-registers procedures)"]
 
         subgraph "Procedure Categories"
-            DB["Database Procedures<br/>db.query, db.execute"]
+            DB["Database Procedures<br/>sqlite.query, sqlite.execute, sqlite.tables, sqlite.schema,<br/>sqlite.transaction, sqlite.migrate, sqlite.backup"]
             Logs["Log Procedures<br/>logs.store, logs.query"]
         end
 
@@ -147,7 +156,7 @@ You can override this path in each procedure call:
 
 ```typescript
 await client.exec(
-  ["db", "query"],
+  ["sqlite", "query"],
   {
     sql: "SELECT * FROM logs",
     dbPath: "./custom/path/database.db"
@@ -159,13 +168,13 @@ await client.exec(
 
 ### Database Procedures
 
-#### `db.query`
+#### `sqlite.query`
 
 Execute a SQL SELECT query and return rows.
 
 ```typescript
 const result = await client.exec(
-  ["db", "query"],
+  ["sqlite", "query"],
   {
     sql: "SELECT * FROM logs WHERE level = ? ORDER BY timestamp DESC",
     params: ["error"]
@@ -196,7 +205,7 @@ interface DbQueryOutput<T = Record<string, unknown>> {
 **Example:**
 ```typescript
 const result = await client.exec(
-  ["db", "query"],
+  ["sqlite", "query"],
   {
     sql: `
       SELECT level, COUNT(*) as count
@@ -211,13 +220,13 @@ const result = await client.exec(
 // result.rows = [{ level: "info", count: 45 }, { level: "error", count: 2 }]
 ```
 
-#### `db.execute`
+#### `sqlite.execute`
 
 Execute a SQL statement (INSERT, UPDATE, DELETE) and return the number of affected rows.
 
 ```typescript
 const result = await client.exec(
-  ["db", "execute"],
+  ["sqlite", "execute"],
   {
     sql: "DELETE FROM logs WHERE timestamp < ?",
     params: ["2024-01-01"]
@@ -247,7 +256,7 @@ interface DbExecuteOutput {
 ```typescript
 // INSERT
 await client.exec(
-  ["db", "execute"],
+  ["sqlite", "execute"],
   {
     sql: "INSERT INTO users (name, email) VALUES (?, ?)",
     params: ["Alice", "alice@example.com"]
@@ -256,7 +265,7 @@ await client.exec(
 
 // UPDATE
 await client.exec(
-  ["db", "execute"],
+  ["sqlite", "execute"],
   {
     sql: "UPDATE users SET active = ? WHERE id = ?",
     params: [true, 123]
@@ -265,7 +274,7 @@ await client.exec(
 
 // DELETE
 await client.exec(
-  ["db", "execute"],
+  ["sqlite", "execute"],
   {
     sql: "DELETE FROM users WHERE active = ?",
     params: [false]
@@ -273,6 +282,65 @@ await client.exec(
 );
 ```
 
+#### `sqlite.tables`
+
+The procedure lists the tables and the views of a database, without the internal `sqlite_` tables. Each item has the `name`, the `type` and the `sql` that made it. With `views: false`, it gives only the tables. The procedure only reads. A missing file is an error (`ENOENT`), and the procedure does not make it.
+
+```typescript
+const { tables } = await client.exec(["sqlite", "tables"], { dbPath: "./app.db" });
+```
+
+#### `sqlite.schema`
+
+The procedure gives the `columns`, the `indexes` and the `foreignKeys` of a table, or the columns of a view. The name of the table goes to SQLite as a bound value, not as SQL text. The procedure only reads.
+
+```typescript
+const schema = await client.exec(["sqlite", "schema"], { dbPath: "./app.db", table: "users" });
+// schema.columns: [{ name: "id", type: "INTEGER", notNull: false, defaultValue: null, primaryKey: 1 }, ...]
+```
+
+#### `sqlite.transaction`
+
+The procedure puts a list of statements in one transaction. When one statement fails, it rolls back the transaction, and the file stays as it was. The `params` of a statement bind to one statement. Without `params`, one `sql` can hold several statements. A statement that gives rows (`SELECT`, or `RETURNING`) gives its `rows` in its result. The procedure refuses `BEGIN`, `COMMIT`, `ROLLBACK`, `SAVEPOINT` and `RELEASE`, because it controls the transaction itself.
+
+```typescript
+const { results, changes } = await client.exec(["sqlite", "transaction"], {
+  dbPath: "./app.db",
+  statements: [
+    { sql: "UPDATE accounts SET balance = balance - ? WHERE id = ?", params: [10, 1] },
+    { sql: "UPDATE accounts SET balance = balance + ? WHERE id = ?", params: [10, 2] },
+    { sql: "SELECT id, balance FROM accounts" },
+  ],
+});
+```
+
+#### `sqlite.migrate`
+
+The procedure applies the numbered migrations that the database does not have, in the order of their numbers. The migrations come from a `folder` or from a list (`migrations`). In a folder, each migration is a file `<number>_<name>.sql`, for example `001_create_users.sql`. The procedure ignores the files that do not end in `.sql`.
+
+- The table `schema_migrations` (or the `table` of the input) records each applied migration and the sha256 of its SQL.
+- The pending migrations are in one transaction. When one fails, the procedure applies none of them.
+- A second call applies nothing, and does not write the file.
+- When the SQL of an applied migration changes, the procedure refuses the call. The sha256 reads CRLF line ends as LF.
+- A migration must not hold `BEGIN`, `COMMIT` or `ROLLBACK`.
+
+```typescript
+const { applied, skipped, version } = await client.exec(["sqlite", "migrate"], {
+  dbPath: "./app.db",
+  folder: "./migrations",
+});
+```
+
+#### `sqlite.backup`
+
+The procedure copies a database to a new file (`to`). It does not replace a file, and it does not write the source. With `vacuum: true`, it compacts the copy.
+
+This package uses SQLite in WebAssembly (sql.js). The database is in the memory of the module. `VACUUM INTO` and the backup API of SQLite write into the file system of the module, not to the disk. Thus the procedure writes the image of the database (its `export()`). It takes the image in the file queue of the database, where no other call of the process changes it. It writes the image to a file next to the target, checks it with `PRAGMA integrity_check`, and then renames it.
+
+```typescript
+const { path, bytes } = await client.exec(["sqlite", "backup"], { dbPath: "./app.db", to: "./backups/app-2026-10-16.db" });
+```
+
 ### Log Procedures
 
 #### `logs.store`
@@ -425,6 +493,65 @@ const { logs } = await client.exec(
 );
 ```
 
+## The collection storage
+
+`SqliteStorage` keeps the items of one collection in one table of a SQLite database. It implements `CollectionStorage` of `@procedures/client`. A `ProcedureServer`, an `ApiStorage` server and a `HybridStorage` use it as they use `InMemoryStorage`.
+
+```ts
+import { SqliteStorage } from "@procedures/sqlite/storage";
+
+const users = new SqliteStorage<User>({ path: "./data/app.db", table: "users" });
+await users.set("u1", { name: "Ada" });
+await users.setBatch([["u2", { name: "Bob" }], ["u3", { name: "Cy" }]]);
+const changes = await users.getChangesSince(); // { version, full: true, changed, deleted: [] }
+await users.close();
+```
+
+The entry `@procedures/sqlite/storage` has the storage alone. The package root exports it too.
+
+### The table and the values
+
+- The option `path` is a database file, or `:memory:` for a database that lives as long as the storage.
+- The option `table` is the name of the table. It starts with a letter, and it has only letters, digits and underscores. The prefix `sqlite_` is not permitted.
+- The table has the columns `id` and `value`. The value is the JSON text of the item.
+- A value without JSON text, for example `undefined` or a `BigInt`, is a `TypeError`. The storage writes nothing then.
+- A read gives a new copy of the item.
+- The order of the items is the order of the first write of each id, as in a `Map`.
+
+### The change log and the versions
+
+- `getChangesSince(version)` gives the items that changed and the ids that were deleted after a version. `HybridStorage` uses it for a delta sync.
+- A version is `<epoch>:<seq>`, as in `InMemoryStorage`. The table `_storage_versions` keeps the epoch, so a version stays valid after a restart.
+- The table `_storage_log` keeps the last change of each id. The option `maxTombstones` (10000 by default) limits the deletes that it keeps.
+- A version that is older than the dropped deletes gives all items (`full: true`).
+- Write the table only through the storage. A change from other SQL is not in the change log.
+
+### The transactions and the file
+
+- Each write runs in a transaction with the change log. A batch that fails changes nothing.
+- The storage keeps the database in memory, and it writes the file after each write. The write ends after the file write.
+- When several writes wait, one file write holds their changes.
+- A failed file write fails the write. The change stays in memory, and `flush()` or the next write writes it.
+- A storage that only reads does not make the file.
+- The storages of one file in one process share one database. Thus the storages of two tables in one file keep their items.
+- Another process, and the procedures `sqlite.query` and `sqlite.execute`, do not share this database. Do not write the file from them while a storage has it open.
+
+### The durable local copy of `HybridStorage`
+
+`HybridStorage` of `@procedures/store` takes the storage of its local copy in the option `local`. With a `SqliteStorage`, the local copy stays after a restart. A new storage in `local-only` mode then reads the items of the last run, without the remote.
+
+```ts
+import { HybridStorage } from "@procedures/store";
+import { FileQueueStore } from "@procedures/store/file-queue";
+import { SqliteStorage } from "@procedures/sqlite/storage";
+
+const users = new HybridStorage(remote, {
+  mode: "write-back",
+  local: new SqliteStorage({ path: "./users.db", table: "users" }),
+  queueStore: new FileQueueStore("./users.queue.jsonl"),
+});
+```
+
 ## Usage Examples
 
 ### Basic Database Operations
@@ -437,7 +564,7 @@ const client = new Client(new LocalTransport({ registry: PROCEDURE_REGISTRY }));
 
 // Create a table
 await client.exec(
-  ["db", "execute"],
+  ["sqlite", "execute"],
   {
     sql: `
       CREATE TABLE IF NOT EXISTS users (
@@ -454,7 +581,7 @@ await client.exec(
 
 // Insert data
 await client.exec(
-  ["db", "execute"],
+  ["sqlite", "execute"],
   {
     sql: "INSERT INTO users (name, email) VALUES (?, ?)",
     params: ["Alice Johnson", "alice@example.com"],
@@ -464,7 +591,7 @@ await client.exec(
 
 // Query data
 const result = await client.exec(
-  ["db", "query"],
+  ["sqlite", "query"],
   {
     sql: "SELECT * FROM users WHERE active = ? ORDER BY created_at DESC",
     params: [1],
@@ -628,7 +755,7 @@ interface User {
 }
 
 const result = await client.call<DbQueryOutput<User>>(
-  ["db", "query"],
+  ["sqlite", "query"],
   {
     sql: "SELECT * FROM users WHERE active = ?",
     params: [true],
@@ -708,7 +835,10 @@ export type {
 client-sqlite/
 ├── src/
 │   ├── types.ts              # Type definitions
-│   ├── register.ts           # Auto-registration + procedure implementations
+│   ├── module.ts             # The procedures as data, and the sqlite.query, sqlite.execute and logs.* handlers
+│   ├── register.ts           # The ./register entry: registers the module
+│   ├── storage.ts            # SqliteStorage, the collection storage (the ./storage entry)
+│   ├── procedures/db/        # sqlite.tables, sqlite.schema, sqlite.transaction, sqlite.migrate, sqlite.backup
 │   └── index.ts              # Main entry point
 ├── dist/                     # Compiled output
 ├── package.json