@quereus/store 4.4.0 → 4.5.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (33) hide show
  1. package/README.md +512 -486
  2. package/dist/src/common/cached-kv-store.js +4 -0
  3. package/dist/src/common/cached-kv-store.js.map +1 -1
  4. package/dist/src/common/encoding.d.ts +32 -1
  5. package/dist/src/common/encoding.d.ts.map +1 -1
  6. package/dist/src/common/encoding.js +42 -10
  7. package/dist/src/common/encoding.js.map +1 -1
  8. package/dist/src/common/index.d.ts +3 -2
  9. package/dist/src/common/index.d.ts.map +1 -1
  10. package/dist/src/common/index.js +3 -1
  11. package/dist/src/common/index.js.map +1 -1
  12. package/dist/src/common/json-key.d.ts +96 -0
  13. package/dist/src/common/json-key.d.ts.map +1 -0
  14. package/dist/src/common/json-key.js +204 -0
  15. package/dist/src/common/json-key.js.map +1 -0
  16. package/dist/src/common/key-builder.d.ts +41 -5
  17. package/dist/src/common/key-builder.d.ts.map +1 -1
  18. package/dist/src/common/key-builder.js +60 -21
  19. package/dist/src/common/key-builder.js.map +1 -1
  20. package/dist/src/common/kv-store.d.ts +12 -2
  21. package/dist/src/common/kv-store.d.ts.map +1 -1
  22. package/dist/src/common/store-module.d.ts +64 -15
  23. package/dist/src/common/store-module.d.ts.map +1 -1
  24. package/dist/src/common/store-module.js +439 -133
  25. package/dist/src/common/store-module.js.map +1 -1
  26. package/dist/src/common/store-table.d.ts +226 -12
  27. package/dist/src/common/store-table.d.ts.map +1 -1
  28. package/dist/src/common/store-table.js +454 -37
  29. package/dist/src/common/store-table.js.map +1 -1
  30. package/dist/src/testing/kv-conformance.d.ts.map +1 -1
  31. package/dist/src/testing/kv-conformance.js +32 -0
  32. package/dist/src/testing/kv-conformance.js.map +1 -1
  33. package/package.json +6 -6
package/README.md CHANGED
@@ -1,486 +1,512 @@
1
- # @quereus/store
2
-
3
- Abstract key-value storage module for [Quereus](https://github.com/gotchoices/quereus). Provides platform-agnostic interfaces and a generic `StoreModule` virtual table implementation.
4
-
5
- ## Architecture
6
-
7
- This package provides the **abstract layer** that separates virtual table logic from platform-specific storage:
8
-
9
- ```
10
- @quereus/store (this package)
11
- ├── KVStore interface - Abstract key-value store
12
- ├── KVStoreProvider interface - Store factory/management
13
- ├── StoreModule - Generic VirtualTableModule
14
- ├── StoreTable - Generic virtual table implementation
15
- ├── StoreConnection - Generic transaction support
16
- └── Common utilities - Encoding, serialization, events
17
-
18
- @quereus/plugin-leveldb (Node.js) @quereus/plugin-indexeddb (Browser)
19
- ├── LevelDBStore ├── IndexedDBStore
20
- ├── LevelDBProvider ├── IndexedDBProvider
21
- └── Plugin registration ├── IndexedDBManager
22
- └── CrossTabSync
23
- ```
24
-
25
- This architecture enables:
26
- - **Platform portability** - Same SQL tables work across Node.js, browsers, and mobile
27
- - **Custom storage backends** - Implement `KVStore` for IndexedDB, LevelDB, LMDB, or other "NoSQL" stores
28
- - **Dependency injection** - Use `KVStoreProvider` for store management
29
-
30
- ## Storage Architecture
31
-
32
- The store module uses separate logical stores for different data types:
33
-
34
- **Store Naming Convention:**
35
- - `{schema}.{table}` - Data store (row data)
36
- - `{schema}.{table}_idx_{indexName}` - Index stores (one per secondary index)
37
- - `{prefix}.__stats__` - Unified stats store (row counts for all tables)
38
- - `__catalog__` - Catalog store (DDL metadata)
39
-
40
- **Key Formats:**
41
- - **Data keys**: Encoded primary key (no prefix)
42
- - **Index keys**: Encoded index columns + encoded PK
43
- - **Index values**: The row's encoded **data key**. A secondary-index scan resolves each
44
- matched entry back to its base row with one data-store read at this key, rather than
45
- decoding the index key's PK suffix (that suffix is encoded lossily for a NOCASE/RTRIM
46
- PK column, so it is not recoverable to SQL values). Entries are not covering — the row
47
- itself always lives in the data store.
48
- - **Catalog keys**:
49
- - Tables: `{schema}.{table}` as a string (the `CREATE TABLE` bundle, with its index DDL and any exposed-implicit-index tag DDL)
50
- - Views: `\x00view\x00{schema}.{view}` (reserved-prefix; `generateViewDDL`)
51
- - Materialized views: `\x00mview\x00{schema}.{mv}` (reserved-prefix; `generateMaterializedViewDDL`)
52
-
53
- This design eliminates redundant prefixes and groups related stores together by table name. The leading-`0x00` view/MV prefixes never collide with an unprefixed table key, so a view/MV may safely share a name with a table; a full catalog scan returns all three kinds intermixed and rehydrate classifies each by its key prefix.
54
-
55
- **Catalog DDL is re-persisted on catalog-only mutations.** `ALTER … SET TAGS` (and the programmatic `setTableTags` / `setColumnTags` / `setConstraintTags` / `setViewTags` / `setMaterializedViewTags`), plus `CREATE`/`DROP VIEW` and `CREATE`/`DROP MATERIALIZED VIEW`, never reach `module.alterTable`/`module.destroy`. The module subscribes to the engine's schema-change events (`table_modified`, the `view_*` events, and the `materialized_view_*` events) and writes the matching `__catalog__` entry when its `generate*DDL` output changes — table / column / constraint / **index** / **view** / **materialized-view** tags, and view/MV lifecycle, all survive close → reopen. A table's bundle is its `CREATE TABLE` DDL, one `CREATE [UNIQUE] INDEX` line per secondary index, and one trailing `alter index … set tags (…)` line per *exposed implicit index* carrying user tags (an exposed implicit index is never materialized in the store's *engine-facing* schema — only in its internal enforcement schema, see the implicit-index note below — so its `UniqueConstraintSchema.exposedIndexTags` has no `CREATE INDEX` line to ride; the alter line re-applies silently on import). These async writes are serialized and drained by `closeAll()` (or the `whenCatalogPersisted()` barrier) before the provider closes. On reopen, `rehydrateCatalog` classifies entries by key prefix and imports them in phases — tables → views → materialized views, all through the engine's `importCatalog` (MVs re-materialize silently via the shared create core, dependency-ordered for MV-over-MV by fixpoint retry). See [`docs/schema.md`](../../docs/schema.md#view-and-materialized-view-persistence) for the full design.
56
-
57
- **How a UNIQUE constraint is enforced.** For each row written, the store looks for a conflicting row through the cheapest sound route available:
58
-
59
- 1. **A linked row-time covering materialized view** — its backing table answers the uniqueness question.
60
- 2. **A physical secondary index realizing the constraint** — one bounded seek into the index store. Available for **every** non-derived `UNIQUE` (see the implicit-index note below), for a constraint that came from a `CREATE UNIQUE INDEX` (it names its own index), and for any *full* (non-partial) index whose columns match the constraint's. The index need not itself be UNIQUE.
61
- 3. **A full scan of the data store** — always correct, and O(rows) per row written.
62
-
63
- Route 2 turns a bulk insert from O(n²) into roughly O(n log n). It is skipped for a constraint the index cannot answer soundly:
64
-
65
- - A **partial** index cannot serve a constraint it does not derive from: it physically omits its out-of-scope rows, so a seek would miss a conflict among them.
66
- - Index-column bytes are encoded under the **table key collation** `K` (the `collation` module option, default `NOCASE`), not under the constraint's enforcement collation `C`. A seek returns exactly the rows `K`-equal to the new row, so it is a sound *superset* of the true conflict set only when `K` is coarser-than-or-equal-to `C` (a column that can never hold text; `C == K`; or `K = NOCASE` over `C = BINARY`). When `K` is finer — `K = BINARY` over `C = NOCASE`, or `K = NOCASE` over `C = RTRIM` — a seek would under-fetch and silently accept a real duplicate, so the constraint falls back to the full scan. "Can never hold text" is judged by physical representation, not by declared type name: an `ANY` or `JSON` column stores a string as a string, so neither is exempt.
67
-
68
- Whichever route runs, the conflicting row is re-validated identically: the row being written is excluded by primary key, each constrained column is compared under its enforcement collation (a `CREATE UNIQUE INDEX … (col COLLATE x)` enforces `x`, else the column's declared collation), and a partial constraint's predicate must hold on the candidate.
69
-
70
- Because route 2 trusts the index store to hold an entry for every live row, `CREATE INDEX` populates the new index from the table's **effective** rows — committed rows merged with the open transaction's pending writes. A row inserted earlier in that transaction is therefore indexed, and participates in `CREATE UNIQUE INDEX`'s duplicate check, rather than being invisible to every later seek. Index entries are written outside the transaction coordinator, so a later `ROLLBACK` leaves entries for rows that no longer exist; every reader resolves an index entry to its live row and drops it when the row is gone or no longer matches, so a stale entry can never manufacture a result.
71
-
72
- **Implicit per-constraint index (`_uc_*`).** Every non-derived `UNIQUE` — declared inline at `CREATE TABLE` or added by `ALTER TABLE … ADD CONSTRAINT` — is backed by a hidden secondary index named `<constraint name>` or, when unnamed, `_uc_<columns>` (the same convention the memory backend and the engine's `implicitIndexName` use). This is what makes route 2 reach a plain `UNIQUE`, so a bulk load no longer degrades to O(n²). The index is:
73
-
74
- - **Kept out of the engine.** It lives only in the StoreTable's *enforcement* schema, never in the engine-registered schema, so the read-query planner does not see it (a plain `UNIQUE` gets no read-side plan from it — matching the memory backend) and it is never written to the catalog as a `CREATE INDEX`.
75
- - **Derived on open.** Reconstructing a StoreTable re-materializes the schema entry from `uniqueConstraints`; the physical index store persists on disk under its deterministic name and is reopened lazily. (A store written *before* this feature has no `_uc_*` store on disk — backwards compatibility is waived project-wide; reopening such a database would need the index rebuilt.)
76
- - **Reconciled across ALTER.** `ADD CONSTRAINT` builds the physical store from the existing rows (after the existing-row duplicate check passes); `DROP CONSTRAINT` tears it down (so a later re-`ADD` cannot reopen stale entries); `RENAME CONSTRAINT`, and a column rename that changes an unnamed constraint's implicit name, move the store; a PK / collation / data-type `ALTER` re-encodes it via the same rebuild that re-encodes explicit indexes.
77
- - **Relocated / reclaimed with the table.** `DROP TABLE` deletes the `_uc_*` store alongside the data + explicit-index stores; `RENAME TABLE` relocates it under the new name. Both resolve the physical store list from the *enforcement* schema (not the engine-facing `.indexes`, which omits `_uc_*`), so the implicit store is never stranded on drop nor left behind on rename (which would leave the renamed table seeking an empty store and accepting a duplicate).
78
- - **Always materialized, even alongside an explicit index.** When a user also declares a matching `CREATE INDEX`, both are maintained (their key bytes are byte-identical, so this is redundant, never wrong). Reusing the explicit index instead is a deferred optimization — see `tickets/backlog/debt-store-implicit-unique-index-reuse`.
79
-
80
- ## Installation
81
-
82
- ```bash
83
- npm install @quereus/store
84
- ```
85
-
86
- For platform-specific implementations:
87
- ```bash
88
- # Node.js
89
- npm install @quereus/plugin-leveldb
90
-
91
- # Browser
92
- npm install @quereus/plugin-indexeddb
93
- ```
94
-
95
- ## Usage
96
-
97
- ### With a Provider
98
-
99
- ```typescript
100
- import { Database } from '@quereus/quereus';
101
- import { StoreModule } from '@quereus/store';
102
- import { createLevelDBProvider } from '@quereus/plugin-leveldb';
103
- // OR: import { createIndexedDBProvider } from '@quereus/plugin-indexeddb';
104
-
105
- const db = new Database();
106
-
107
- // Create provider for your platform
108
- const provider = createLevelDBProvider({ basePath: './data' });
109
-
110
- // Create the generic store module with your provider
111
- const storeModule = new StoreModule(provider);
112
- db.registerModule('store', storeModule);
113
-
114
- // Use it in SQL
115
- await db.exec(`
116
- create table users (id integer primary key, name text)
117
- using store
118
- `);
119
- ```
120
-
121
- ### Custom Storage Backend
122
-
123
- Implement `KVStore` and `KVStoreProvider` to create custom storage backends:
124
-
125
- ```typescript
126
- import type { KVStore, KVStoreProvider } from '@quereus/store';
127
-
128
- class MyCustomStore implements KVStore {
129
- async get(key: Uint8Array) { /* ... */ }
130
- async put(key: Uint8Array, value: Uint8Array) { /* ... */ }
131
- async delete(key: Uint8Array) { /* ... */ }
132
- async has(key: Uint8Array) { /* ... */ }
133
- iterate(options?: IterateOptions) { /* ... */ }
134
- batch() { /* ... */ }
135
- async close() { /* ... */ }
136
- async approximateCount(options?: IterateOptions) { /* ... */ }
137
- }
138
-
139
- class MyCustomProvider implements KVStoreProvider {
140
- async getStore(schemaName: string, tableName: string) {
141
- return new MyCustomStore(/* ... */);
142
- }
143
- async getIndexStore(schemaName: string, tableName: string, indexName: string) {
144
- return new MyCustomStore(/* ... */);
145
- }
146
- async getStatsStore(schemaName: string, tableName: string) {
147
- return new MyCustomStore(/* ... */);
148
- }
149
- async getCatalogStore() { /* ... */ }
150
- async closeStore(schemaName: string, tableName: string) { /* ... */ }
151
- async closeIndexStore(schemaName: string, tableName: string, indexName: string) { /* ... */ }
152
- async closeAll() { /* ... */ }
153
- }
154
-
155
- // Use it with StoreModule
156
- const provider = new MyCustomProvider();
157
- const module = new StoreModule(provider);
158
- db.registerModule('store', module);
159
- ```
160
-
161
- **Validate a new backend against the shared conformance suite.** `@quereus/store/testing`
162
- exports `runKVStoreConformance(name, makeBackend)` — one parameterized battery of
163
- behavioral tests written against the `KVStore` contract (point ops, ordering, range
164
- iteration, streaming across page boundaries, batch semantics, optional persistence, and
165
- cross-backend encoded-key ordering). Wire a tiny lifecycle adapter and run it under Mocha
166
- so any drift from the contract fails a test:
167
-
168
- ```typescript
169
- import { runKVStoreConformance } from '@quereus/store/testing';
170
-
171
- runKVStoreConformance('MyCustomStore', () => ({
172
- open: async () => new MyCustomStore(/* ... */),
173
- // Omit `reopen` for a non-persistent backend; supply it (reopen the SAME keyspace
174
- // without wiping) to also exercise the persistence tier.
175
- teardown: async () => { /* close handles, remove backing storage */ },
176
- }));
177
- ```
178
-
179
- See `test/kv-conformance.spec.ts` (in-memory), and the LevelDB / IndexedDB plugins'
180
- `test/conformance.spec.ts` for worked adapters.
181
-
182
- ## KVStore Interface
183
-
184
- The `KVStore` interface is the foundation for all storage backends:
185
-
186
- ```typescript
187
- interface KVStore {
188
- get(key: Uint8Array): Promise<Uint8Array | undefined>;
189
- put(key: Uint8Array, value: Uint8Array): Promise<void>;
190
- delete(key: Uint8Array): Promise<void>;
191
- has(key: Uint8Array): Promise<boolean>;
192
- iterate(options?: IterateOptions): AsyncIterable<KVEntry>;
193
- batch(): WriteBatch;
194
- close(): Promise<void>;
195
- approximateCount(options?: IterateOptions): Promise<number>;
196
- }
197
-
198
- interface KVStoreProvider {
199
- // Get data store for a table
200
- getStore(schemaName: string, tableName: string): Promise<KVStore>;
201
-
202
- // Get index store for a secondary index
203
- getIndexStore(schemaName: string, tableName: string, indexName: string): Promise<KVStore>;
204
-
205
- // Get stats store for table statistics
206
- getStatsStore(schemaName: string, tableName: string): Promise<KVStore>;
207
-
208
- // Get catalog store for DDL metadata
209
- getCatalogStore(): Promise<KVStore>;
210
-
211
- // Close specific stores
212
- closeStore(schemaName: string, tableName: string): Promise<void>;
213
- closeIndexStore(schemaName: string, tableName: string, indexName: string): Promise<void>;
214
- closeAll(): Promise<void>;
215
-
216
- // Optional: Delete stores. `indexNames` is the table's exact secondary-index
217
- // names (from the schema) — build index store names from it via
218
- // buildIndexStoreName instead of prefix-scanning `{table}_idx_`, which would
219
- // also match a sibling table literally named `{table}_idx_<x>`.
220
- deleteIndexStore?(schemaName: string, tableName: string, indexName: string): Promise<void>;
221
- deleteTableStores?(schemaName: string, tableName: string, indexNames: readonly string[]): Promise<void>;
222
-
223
- // Optional: Relocate a table's data + index stores for ALTER TABLE ... RENAME TO
224
- // (`indexNames` carries the same authoritative, exact index list).
225
- renameTableStores?(schemaName: string, oldName: string, newName: string, indexNames: readonly string[]): Promise<void>;
226
- }
227
- ```
228
-
229
- ## Module Capabilities
230
-
231
- The `StoreModule` reports its capabilities via `getCapabilities()`:
232
-
233
- ```typescript
234
- const storeModule = new StoreModule(provider);
235
- const caps = storeModule.getCapabilities();
236
-
237
- // {
238
- // isolation: false, // Store module does NOT provide transaction isolation
239
- // savepoints: true, // Coordinator-buffered ops support savepoints within a transaction
240
- // persistent: true, // Data persists across restarts
241
- // secondaryIndexes: true,// Supports secondary indexes
242
- // rangeScans: true // Supports range scans
243
- // }
244
- ```
245
-
246
- **Important:** The base `StoreModule` does not provide transaction isolation:
247
- - No snapshot isolation: between connections, reads see only committed data, and concurrent readers may observe partial writes
248
- - Within a transaction, reads through the table's shared coordinator DO see that transaction's own pending writes (read-your-own-writes). This extends to DML's own internal reads: the insert PK-conflict probe, the update/delete old-image reads, and the update PK-change conflict probe all read through the pending merge, so an INSERT/UPDATE/DELETE against a row written earlier in the same transaction sees that pending row — it raises a UNIQUE conflict (or evicts under `OR REPLACE`), cleans up secondary-index entries, tracks the correct row-count delta, and emits events carrying the pending `oldRow`
249
- - Row-validating DDL reads the same effective stream: `create index` / `create unique index` populate from it (see above), and `alter table … add constraint … unique` plus the `set collate` re-validation of a covering non-PK UNIQUE scan it too — so a duplicate inserted earlier in the still-open transaction is rejected rather than surviving to commit
250
- - Savepoints (create / release / rollback-to) work within a transaction via the coordinator's buffered op log
251
- - Caveat: a DDL-commit operation (`replaceContents` / `renameTable`, e.g. `refresh materialized view` or `alter table … rename`) commits the coordinator mid-transaction, clearing the savepoint stack. A later `rollback to` / `release` targeting a now-vanished savepoint degrades to a no-op (warn-and-return) rather than throwing; the committed DDL and everything before it stays committed
252
-
253
- ## Atomic multi-store commit (module-wide, cross-table)
254
-
255
- A single `TransactionCoordinator` is shared by **every table of one storage
256
- module** — it is the unit of cross-table atomicity. Every buffered op is
257
- addressed by its explicit target `KVStore` handle (data ops, secondary-index
258
- ops, and backing-host writes alike), so a transaction touching tables A and B
259
- accumulates all of their stores' ops in one coordinator. Because the engine
260
- commits virtual-table connections **sequentially** and the coordinator's
261
- `commit()`/`rollback()` are **idempotent**, the first connection to commit
262
- flushes **every** touched store of **every** table the transaction wrote; the
263
- remaining connections no-op.
264
-
265
- `TransactionCoordinator.commit()` thus writes each table's data store and each of
266
- its secondary-index stores. By default it writes **one `KVStore.batch()` per
267
- store, sequentially** — a crash between those batches can leave tables/indexes
268
- divergent on disk, with no automatic healing (no worse than the prior per-table
269
- commits, which were already non-atomic across tables).
270
-
271
- A provider whose stores share a single durable commit domain can close that
272
- window by implementing the optional `KVStoreProvider.beginAtomicBatch()`:
273
-
274
- ```typescript
275
- interface AtomicBatch {
276
- put(store: KVStore, key: Uint8Array, value: Uint8Array): void;
277
- delete(store: KVStore, key: Uint8Array): void;
278
- write(): Promise<void>; // one durable, all-or-nothing physical commit
279
- clear(): void;
280
- }
281
-
282
- interface KVStoreProvider {
283
- // ...
284
- // Open a batch spanning this provider's stores, or undefined when the provider
285
- // has no shared commit domain (the coordinator then falls back to per-store batch()).
286
- beginAtomicBatch?(): AtomicBatch | undefined;
287
- }
288
- ```
289
-
290
- The batch addresses stores by **`KVStore` handle**, so it composes with the
291
- coordinator's existing per-store bucketing without a name lookup. When present,
292
- `commit()` queues every pending op — every store of **every table** in the
293
- transaction — into one `AtomicBatch` and issues a single `write()`; all of those
294
- tables commit or roll back together. When absent — or when the factory returns
295
- `undefined` — behavior is byte-identical to the per-store loop, so providers
296
- without a shared domain are unaffected.
297
-
298
- The capability surface spans **multiple stores of one provider** (every store of
299
- every table the module owns), giving full module-wide cross-table atomicity with
300
- no interface change. The
301
- [`@quereus/plugin-indexeddb`](../quereus-plugin-indexeddb) provider implements it
302
- over its single IndexedDB database (multiple object stores, one
303
- `db.transaction(...,'readwrite')`), invalidating each touched store's read cache
304
- after a successful write so read-your-own-writes survives the cache.
305
-
306
- ## Materialized-View Backing Host
307
-
308
- The store module implements the engine's backing-host capability
309
- (`StoreBackingHost`), so `create materialized view mv using store as <body>`
310
- places the MV's backing table in persistent storage. Maintenance writes ride
311
- the module's shared `TransactionCoordinator`'s pending state (committing/rolling
312
- back in lockstep with the source write — and, since the coordinator is
313
- module-wide, in the same all-or-nothing batch as a write to a same-module
314
- source), mid-transaction reads of the MV see pending
315
- maintenance through the read-your-own-writes merge, and the backing's text
316
- primary-key columns are keyed under the store's `collation` arg (default
317
- `NOCASE` — pass `using store(collation = 'BINARY')` for byte-exact keys). The
318
- isolation wrapper forwards the capability automatically. See
319
- [`docs/mv-backing-host.md` § The store host](../../docs/mv-backing-host.md#the-store-host-using-store).
320
-
321
- ## External Row-Write Entry Point
322
-
323
- `StoreTable.applyExternalRowChanges(ops)` applies trusted, externally-originated
324
- row writes (e.g. inbound replication) directly to a **source** table's committed
325
- storage — table-owned data-key put/delete, **secondary-index maintenance**, and
326
- stats tracking — and returns the effective `BackingRowChange[]` (the shape
327
- `Database.ingestExternalRowChanges` consumes). It is the index-maintaining
328
- sibling of the backing host (whose MV backing tables carry no indexes): a caller
329
- writing the data `KVStore` directly would silently skip index and stats upkeep.
330
-
331
- Resolve the table with `StoreModule.getTableForExternalWrite(db, schema, table)`
332
- (same ownership/wrapper resolution as `getBackingHost`), read a row's current
333
- image with `StoreTable.readRowByPk(pk)`, then apply one `ExternalRowOp` per row:
334
-
335
- ```typescript
336
- const table = storeModule.getTableForExternalWrite(db, 'main', 'users');
337
- if (table) {
338
- const changes = await table.applyExternalRowChanges([
339
- { op: 'upsert', row: [1, 'alice'] }, // full row, schema column order
340
- { op: 'delete', pk: [2] }, // PK values, PK-definition order
341
- ]);
342
- }
343
- ```
344
-
345
- Deliberately emits **no** module data events (the caller owns emission and the
346
- `remote` flag), opens **no** coordinator transaction (writes commit at once,
347
- last-writer-wins against any pending local transaction), and runs **no**
348
- constraint validation (the origin is trusted). No-ops are suppressed: a delete
349
- of an absent key and a value-identical upsert (byte-faithful) write nothing and
350
- report nothing.
351
-
352
- ## Transaction Isolation
353
-
354
- To add full ACID transaction semantics with snapshot isolation, wrap the store module with the `IsolationModule`:
355
-
356
- ```typescript
357
- import { Database, MemoryTableModule } from '@quereus/quereus';
358
- import { IsolationModule } from '@quereus/isolation';
359
- import { StoreModule, createIsolatedStoreModule } from '@quereus/store';
360
- import { createLevelDBProvider } from '@quereus/plugin-leveldb';
361
-
362
- const db = new Database();
363
- const provider = createLevelDBProvider({ basePath: './data' });
364
-
365
- // Option 1: Use the convenience function
366
- const isolatedModule = createIsolatedStoreModule({ provider });
367
- db.registerModule('store', isolatedModule);
368
-
369
- // Option 2: Manual wrapping for more control
370
- const storeModule = new StoreModule(provider);
371
- const isolatedModule = new IsolationModule({
372
- underlying: storeModule,
373
- overlay: new MemoryTableModule(),
374
- });
375
- db.registerModule('store', isolatedModule);
376
-
377
- // Now transactions have full isolation
378
- await db.exec('BEGIN');
379
- await db.exec(`INSERT INTO users VALUES (1, 'Alice')`);
380
-
381
- // Read-your-own-writes: sees uncommitted insert
382
- const user = await db.get('SELECT * FROM users WHERE id = 1');
383
- console.log(user.name); // 'Alice'
384
-
385
- await db.exec('COMMIT'); // Or ROLLBACK to discard
386
- ```
387
-
388
- The isolation layer provides:
389
- - **Read-your-own-writes** within transactions
390
- - **Snapshot isolation** for consistent reads
391
- - **Savepoint support** via the overlay module
392
-
393
- ### Checking for Isolation Support
394
-
395
- ```typescript
396
- import { hasIsolation } from '@quereus/store';
397
-
398
- const storeModule = new StoreModule(provider);
399
- console.log(hasIsolation(storeModule)); // false
400
-
401
- const isolatedModule = createIsolatedStoreModule({ provider });
402
- console.log(hasIsolation(isolatedModule)); // true
403
- ```
404
-
405
- ## API
406
-
407
- ### Core Exports
408
-
409
- | Export | Description |
410
- |--------|-------------|
411
- | `KVStore` | Key-value store interface (type) |
412
- | `KVStoreProvider` | Store factory interface (type) |
413
- | `WriteBatch` | Batch write interface (type) |
414
- | `AtomicBatch` | Cross-store all-or-nothing batch from `KVStoreProvider.beginAtomicBatch` (type) |
415
- | `IterateOptions` | Iteration options (type) |
416
- | `StoreModule` | Generic VirtualTableModule |
417
- | `StoreTable` | Virtual table implementation (incl. `applyExternalRowChanges` / `readRowByPk` for externally-applied source writes) |
418
- | `ExternalRowOp` | One externally-applied row op (`upsert`/`delete`) for `StoreTable.applyExternalRowChanges` (type) |
419
- | `resolvePkKeyCollations` | Per-PK-column key collations (pass to `buildDataKey`/`buildIndexKey` to match `StoreTable`'s key bytes) |
420
- | `StoreConnection` | Transaction connection |
421
- | `TransactionCoordinator` | Transaction management |
422
- | `StoreEventEmitter` | Event system for data/schema changes |
423
-
424
- ### Isolation Layer Utilities
425
-
426
- | Export | Description |
427
- |--------|-------------|
428
- | `createIsolatedStoreModule` | Create store module with isolation layer |
429
- | `hasIsolation` | Check if a module has isolation capability |
430
- | `IsolatedStoreModuleConfig` | Configuration for isolated store module |
431
-
432
- ### Caching
433
-
434
- | Export | Description |
435
- |--------|-------------|
436
- | `CachedKVStore` | Read-through LRU cache wrapper for any `KVStore` |
437
- | `CacheOptions` | Configuration for cache (maxEntries, maxBytes, enabled) |
438
-
439
- ### Encoding Utilities
440
-
441
- | Export | Description |
442
- |--------|-------------|
443
- | `encodeValue` | Encode a SQL value to sortable bytes |
444
- | `decodeValue` | Decode bytes back to SQL value |
445
- | `encodeCompositeKey` | Encode multiple values as composite key |
446
- | `decodeCompositeKey` | Decode composite key to values |
447
- | `BUILTIN_KEY_NORMALIZER_RESOLVER` | Built-ins-only key-normalizer resolver (`EncodeOptions.normalizers` default) |
448
-
449
- ### Serialization Utilities
450
-
451
- | Export | Description |
452
- |--------|-------------|
453
- | `serializeRow` | Serialize a row to bytes |
454
- | `deserializeRow` | Deserialize bytes to row |
455
- | `serializeValue` | Serialize a single value |
456
- | `deserializeValue` | Deserialize a single value |
457
-
458
- ### Key Building
459
-
460
- | Export | Description |
461
- |--------|-------------|
462
- | `buildDataStoreName` | Build store name for table data |
463
- | `buildIndexStoreName` | Build store name for an index |
464
- | `buildStatsStoreName` | Build store name for table stats |
465
- | `buildDataKey` | Build key for row data (encoded PK) |
466
- | `buildIndexKey` | Build key for index entry |
467
- | `buildCatalogKey` | Build key for a table's catalog entry (`{schema}.{table}`) |
468
- | `buildViewCatalogKey` | Build key for a view's catalog entry (reserved `\x00view\x00` prefix) |
469
- | `buildMaterializedViewCatalogKey` | Build key for an MV's catalog entry (reserved `\x00mview\x00` prefix) |
470
- | `classifyCatalogKey` | Classify a loaded catalog key as `'table'` / `'view'` / `'materializedView'` |
471
- | `buildFullScanBounds` | Build bounds for full table scan |
472
- | `buildIndexPrefixBounds` | Build bounds for index prefix scan |
473
- | `buildPkPrefixBounds` | Build bounds for a data-store PK prefix range (per-column DESC + key collations) |
474
- | `buildCatalogScanBounds` | Build bounds for catalog scan |
475
- | `CATALOG_STORE_NAME` | Reserved catalog store name constant |
476
- | `STORE_SUFFIX` | Store name suffixes (INDEX, STATS) |
477
-
478
- ## Related Packages
479
-
480
- - [`@quereus/plugin-leveldb`](../quereus-plugin-leveldb/) - LevelDB implementation for Node.js
481
- - [`@quereus/plugin-indexeddb`](../quereus-plugin-indexeddb/) - IndexedDB implementation for browsers
482
- - [`@quereus/sync`](../quereus-sync/) - CRDT sync layer
483
-
484
- ## License
485
-
486
- MIT
1
+ # @quereus/store
2
+
3
+ > **Stability: Beta** — complete and tested, but the surface is still being shaped; a
4
+ > breaking change may land in a minor release. Its on-disk key encoding is not frozen, so
5
+ > a format change can make an existing database unreadable without a migration step. See
6
+ > [Stability Tiers](../../docs/stability.md#tiers).
7
+
8
+ Abstract key-value storage module for [Quereus](https://github.com/gotchoices/quereus). Provides platform-agnostic interfaces and a generic `StoreModule` virtual table implementation.
9
+
10
+ ## Architecture
11
+
12
+ This package provides the **abstract layer** that separates virtual table logic from platform-specific storage:
13
+
14
+ ```
15
+ @quereus/store (this package)
16
+ ├── KVStore interface - Abstract key-value store
17
+ ├── KVStoreProvider interface - Store factory/management
18
+ ├── StoreModule - Generic VirtualTableModule
19
+ ├── StoreTable - Generic virtual table implementation
20
+ ├── StoreConnection - Generic transaction support
21
+ └── Common utilities - Encoding, serialization, events
22
+
23
+ @quereus/plugin-leveldb (Node.js) @quereus/plugin-indexeddb (Browser)
24
+ ├── LevelDBStore ├── IndexedDBStore
25
+ ├── LevelDBProvider ├── IndexedDBProvider
26
+ └── Plugin registration ├── IndexedDBManager
27
+ └── CrossTabSync
28
+ ```
29
+
30
+ This architecture enables:
31
+ - **Platform portability** - Same SQL tables work across Node.js, browsers, and mobile
32
+ - **Custom storage backends** - Implement `KVStore` for IndexedDB, LevelDB, LMDB, or other "NoSQL" stores
33
+ - **Dependency injection** - Use `KVStoreProvider` for store management
34
+
35
+ ## Storage Architecture
36
+
37
+ The store module uses separate logical stores for different data types:
38
+
39
+ **Store Naming Convention:**
40
+ - `{schema}.{table}` - Data store (row data)
41
+ - `{schema}.{table}_idx_{indexName}` - Index stores (one per secondary index)
42
+ - `{prefix}.__stats__` - Unified stats store (row counts for all tables)
43
+ - `__catalog__` - Catalog store (DDL metadata)
44
+
45
+ **Key Formats:**
46
+ - **Data keys**: Encoded primary key (no prefix)
47
+ - **Index keys**: Encoded index columns + encoded PK
48
+ - **Index values**: The row's encoded **data key**. A secondary-index scan resolves each
49
+ matched entry back to its base row with one data-store read at this key, rather than
50
+ decoding the index key's PK suffix (that suffix is encoded lossily for a NOCASE/RTRIM
51
+ PK column, so it is not recoverable to SQL values). Entries are not covering — the row
52
+ itself always lives in the data store.
53
+ - **Catalog keys**:
54
+ - Tables: `{schema}.{table}` as a string (the `CREATE TABLE` bundle, with its index DDL and any exposed-implicit-index tag DDL)
55
+ - Views: `\x00view\x00{schema}.{view}` (reserved-prefix; `generateViewDDL`)
56
+ - Materialized views: `\x00mview\x00{schema}.{mv}` (reserved-prefix; `generateMaterializedViewDDL`)
57
+
58
+ This design eliminates redundant prefixes and groups related stores together by table name. The leading-`0x00` view/MV prefixes never collide with an unprefixed table key, so a view/MV may safely share a name with a table; a full catalog scan returns all three kinds intermixed and rehydrate classifies each by its key prefix.
59
+
60
+ **`IN`-list index seeks ("multi-seek").** An `IN`-list on an indexed column
61
+ (`where v in (1, 2, 3)`, including parameter-bound lists) is served from the index as one
62
+ deduplicated, key-ordered point seek per distinct list value, instead of a full scan with
63
+ a residual filter. `NULL` list values match nothing and are skipped, duplicate values
64
+ yield their rows once, and a composite index serves the cross-product of per-column lists
65
+ (`a in (1,2) and b in (10,20)` is four seeks). Very large lists (over 1000 seek keys) and
66
+ lists on semantically-ordered column types (TIMESPAN, JSON) fall back to the scan path —
67
+ still correct, just not accelerated. `IN` on the primary key currently scans (see backlog
68
+ `feat-store-pk-in-list-multiseek`). Index choice among several usable indexes is
69
+ first-match, not cheapest — see backlog `bug-store-index-choice-ignores-cost`.
70
+
71
+ The same multi-seek path also serves an `IN` whose values only exist once the query runs
72
+ — `where v in (select … )`, which the engine may materialize into a set and hand down as
73
+ a seek (its `KeySetSemiJoin`). **Nothing in this module distinguishes the two.** The
74
+ engine stamps a `FilterInfo` byte-identical in shape to a literal list's, so every gate
75
+ above applies unchanged: the 1000-key cap, the semantic-ordering decline, the key-collation
76
+ guard, the partial-index exclusion, and the primary-key arm's refusal of any `IN`. At plan
77
+ time such a set describes itself only by a ceiling (`PredicateConstraint.runtimeSet.maxCount`),
78
+ which is what the cost and cap arithmetic judges. The engine also re-checks every row the
79
+ seek returns, so an over-fetching window costs only time.
80
+
81
+ **Catalog DDL is re-persisted on catalog-only mutations.** `ALTER … SET TAGS` (and the programmatic `setTableTags` / `setColumnTags` / `setConstraintTags` / `setViewTags` / `setMaterializedViewTags`), plus `CREATE`/`DROP VIEW` and `CREATE`/`DROP MATERIALIZED VIEW`, never reach `module.alterTable`/`module.destroy`. The module subscribes to the engine's schema-change events (`table_modified`, the `view_*` events, and the `materialized_view_*` events) and writes the matching `__catalog__` entry when its `generate*DDL` output changes — table / column / constraint / **index** / **view** / **materialized-view** tags, and view/MV lifecycle, all survive close → reopen. A table's bundle is its `CREATE TABLE` DDL, one `CREATE [UNIQUE] INDEX` line per secondary index, and one trailing `alter index … set tags (…)` line per *exposed implicit index* carrying user tags (an exposed implicit index is never materialized in the store's *engine-facing* schema — only in its internal enforcement schema, see the implicit-index note below — so its `UniqueConstraintSchema.exposedIndexTags` has no `CREATE INDEX` line to ride; the alter line re-applies silently on import). These async writes are serialized and drained by `closeAll()` (or the `whenCatalogPersisted()` barrier) before the provider closes. On reopen, `rehydrateCatalog` classifies entries by key prefix and imports them in phases — tables → views → materialized views, all through the engine's `importCatalog` (MVs re-materialize silently via the shared create core, dependency-ordered for MV-over-MV by fixpoint retry). See [`docs/schema.md`](../../docs/schema.md#view-and-materialized-view-persistence) for the full design.
82
+
83
+ **How a UNIQUE constraint is enforced.** For each row written, the store looks for a conflicting row through the cheapest sound route available:
84
+
85
+ 1. **A linked row-time covering materialized view** — its backing table answers the uniqueness question.
86
+ 2. **A physical secondary index realizing the constraint** — one bounded seek into the index store. Available for **every** non-derived `UNIQUE` (see the implicit-index note below), for a constraint that came from a `CREATE UNIQUE INDEX` (it names its own index), and for any *full* (non-partial) index whose columns match the constraint's. The index need not itself be UNIQUE.
87
+ 3. **A full scan of the data store** — always correct, and O(rows) per row written.
88
+
89
+ Route 2 turns a bulk insert from O(n²) into roughly O(n log n). It is skipped for a constraint the index cannot answer soundly:
90
+
91
+ - A **partial** index cannot serve a constraint it does not derive from: it physically omits its out-of-scope rows, so a seek would miss a conflict among them.
92
+ - Index-column bytes are encoded under the **table key collation** `K` (the `collation` module option, default `NOCASE`), not under the constraint's enforcement collation `C`. A seek returns exactly the rows `K`-equal to the new row, so it is a sound *superset* of the true conflict set only when `K` is coarser-than-or-equal-to `C` (a column that can never hold text; `C == K`; or `K = NOCASE` over `C = BINARY`). When `K` is finer — `K = BINARY` over `C = NOCASE`, or `K = NOCASE` over `C = RTRIM` — a seek would under-fetch and silently accept a real duplicate, so the constraint falls back to the full scan. "Can never hold text" is judged by physical representation, not by declared type name: an `ANY` or `JSON` column stores a string as a string, so neither is exempt.
93
+
94
+ Whichever route runs, the conflicting row is re-validated identically: the row being written is excluded by primary key, each constrained column is compared under its enforcement collation (a `CREATE UNIQUE INDEX … (col COLLATE x)` enforces `x`, else the column's declared collation), and a partial constraint's predicate must hold on the candidate.
95
+
96
+ Because route 2 trusts the index store to hold an entry for every live row, `CREATE INDEX` populates the new index from the table's **effective** rows — committed rows merged with the open transaction's pending writes. A row inserted earlier in that transaction is therefore indexed, and participates in `CREATE UNIQUE INDEX`'s duplicate check, rather than being invisible to every later seek. Index entries are written outside the transaction coordinator, so a later `ROLLBACK` leaves entries for rows that no longer exist; every reader resolves an index entry to its live row and drops it when the row is gone or no longer matches, so a stale entry can never manufacture a result.
97
+
98
+ **Implicit per-constraint index (`_uc_*`).** Every non-derived `UNIQUE` — declared inline at `CREATE TABLE` or added by `ALTER TABLE … ADD CONSTRAINT` — is backed by a hidden secondary index named `<constraint name>` or, when unnamed, `_uc_<columns>` (the same convention the memory backend and the engine's `implicitIndexName` use). This is what makes route 2 reach a plain `UNIQUE`, so a bulk load no longer degrades to O(n²). The index is:
99
+
100
+ - **Kept out of the engine.** It lives only in the StoreTable's *enforcement* schema, never in the engine-registered schema, so the read-query planner does not see it (a plain `UNIQUE` gets no read-side plan from it — matching the memory backend) and it is never written to the catalog as a `CREATE INDEX`.
101
+ - **Derived on open.** Reconstructing a StoreTable re-materializes the schema entry from `uniqueConstraints`; the physical index store persists on disk under its deterministic name and is reopened lazily. (A store written *before* this feature has no `_uc_*` store on disk — backwards compatibility is waived project-wide; reopening such a database would need the index rebuilt.)
102
+ - **Reconciled across ALTER.** `ADD CONSTRAINT` builds the physical store from the existing rows (after the existing-row duplicate check passes); `DROP CONSTRAINT` tears it down (so a later re-`ADD` cannot reopen stale entries); `RENAME CONSTRAINT`, and a column rename that changes an unnamed constraint's implicit name, move the store; a PK / collation / data-type `ALTER` re-encodes it via the same rebuild that re-encodes explicit indexes.
103
+ - **Relocated / reclaimed with the table.** `DROP TABLE` deletes the `_uc_*` store alongside the data + explicit-index stores; `RENAME TABLE` relocates it under the new name. Both resolve the physical store list from the *enforcement* schema (not the engine-facing `.indexes`, which omits `_uc_*`), so the implicit store is never stranded on drop nor left behind on rename (which would leave the renamed table seeking an empty store and accepting a duplicate).
104
+ - **Always materialized, even alongside an explicit index.** When a user also declares a matching `CREATE INDEX`, both are maintained (their key bytes are byte-identical, so this is redundant, never wrong). Reusing the explicit index instead is a deferred optimization — see `tickets/backlog/debt-store-implicit-unique-index-reuse`.
105
+
106
+ ## Installation
107
+
108
+ ```bash
109
+ npm install @quereus/store
110
+ ```
111
+
112
+ For platform-specific implementations:
113
+ ```bash
114
+ # Node.js
115
+ npm install @quereus/plugin-leveldb
116
+
117
+ # Browser
118
+ npm install @quereus/plugin-indexeddb
119
+ ```
120
+
121
+ ## Usage
122
+
123
+ ### With a Provider
124
+
125
+ ```typescript
126
+ import { Database } from '@quereus/quereus';
127
+ import { StoreModule } from '@quereus/store';
128
+ import { createLevelDBProvider } from '@quereus/plugin-leveldb';
129
+ // OR: import { createIndexedDBProvider } from '@quereus/plugin-indexeddb';
130
+
131
+ const db = new Database();
132
+
133
+ // Create provider for your platform
134
+ const provider = createLevelDBProvider({ basePath: './data' });
135
+
136
+ // Create the generic store module with your provider
137
+ const storeModule = new StoreModule(provider);
138
+ db.registerModule('store', storeModule);
139
+
140
+ // Use it in SQL
141
+ await db.exec(`
142
+ create table users (id integer primary key, name text)
143
+ using store
144
+ `);
145
+ ```
146
+
147
+ ### Custom Storage Backend
148
+
149
+ Implement `KVStore` and `KVStoreProvider` to create custom storage backends:
150
+
151
+ ```typescript
152
+ import type { KVStore, KVStoreProvider } from '@quereus/store';
153
+
154
+ class MyCustomStore implements KVStore {
155
+ async get(key: Uint8Array) { /* ... */ }
156
+ async put(key: Uint8Array, value: Uint8Array) { /* ... */ }
157
+ async delete(key: Uint8Array) { /* ... */ }
158
+ async has(key: Uint8Array) { /* ... */ }
159
+ iterate(options?: IterateOptions) { /* ... */ }
160
+ batch() { /* ... */ }
161
+ async close() { /* ... */ }
162
+ async approximateCount(options?: IterateOptions) { /* ... */ }
163
+ }
164
+
165
+ class MyCustomProvider implements KVStoreProvider {
166
+ async getStore(schemaName: string, tableName: string) {
167
+ return new MyCustomStore(/* ... */);
168
+ }
169
+ async getIndexStore(schemaName: string, tableName: string, indexName: string) {
170
+ return new MyCustomStore(/* ... */);
171
+ }
172
+ async getStatsStore(schemaName: string, tableName: string) {
173
+ return new MyCustomStore(/* ... */);
174
+ }
175
+ async getCatalogStore() { /* ... */ }
176
+ async closeStore(schemaName: string, tableName: string) { /* ... */ }
177
+ async closeIndexStore(schemaName: string, tableName: string, indexName: string) { /* ... */ }
178
+ async closeAll() { /* ... */ }
179
+ }
180
+
181
+ // Use it with StoreModule
182
+ const provider = new MyCustomProvider();
183
+ const module = new StoreModule(provider);
184
+ db.registerModule('store', module);
185
+ ```
186
+
187
+ **Validate a new backend against the shared conformance suite.** `@quereus/store/testing`
188
+ exports `runKVStoreConformance(name, makeBackend)` — one parameterized battery of
189
+ behavioral tests written against the `KVStore` contract (point ops, ordering, range
190
+ iteration, streaming across page boundaries, batch semantics, optional persistence, and
191
+ cross-backend encoded-key ordering). Wire a tiny lifecycle adapter and run it under Mocha
192
+ so any drift from the contract fails a test:
193
+
194
+ ```typescript
195
+ import { runKVStoreConformance } from '@quereus/store/testing';
196
+
197
+ runKVStoreConformance('MyCustomStore', () => ({
198
+ open: async () => new MyCustomStore(/* ... */),
199
+ // Omit `reopen` for a non-persistent backend; supply it (reopen the SAME keyspace
200
+ // without wiping) to also exercise the persistence tier.
201
+ teardown: async () => { /* close handles, remove backing storage */ },
202
+ }));
203
+ ```
204
+
205
+ See `test/kv-conformance.spec.ts` (in-memory), and the LevelDB / IndexedDB plugins'
206
+ `test/conformance.spec.ts` for worked adapters.
207
+
208
+ ## KVStore Interface
209
+
210
+ The `KVStore` interface is the foundation for all storage backends:
211
+
212
+ ```typescript
213
+ interface KVStore {
214
+ get(key: Uint8Array): Promise<Uint8Array | undefined>;
215
+ put(key: Uint8Array, value: Uint8Array): Promise<void>;
216
+ delete(key: Uint8Array): Promise<void>;
217
+ has(key: Uint8Array): Promise<boolean>;
218
+ iterate(options?: IterateOptions): AsyncIterable<KVEntry>;
219
+ batch(): WriteBatch;
220
+ close(): Promise<void>;
221
+ approximateCount(options?: IterateOptions): Promise<number>;
222
+ }
223
+
224
+ interface KVStoreProvider {
225
+ // Get data store for a table
226
+ getStore(schemaName: string, tableName: string): Promise<KVStore>;
227
+
228
+ // Get index store for a secondary index
229
+ getIndexStore(schemaName: string, tableName: string, indexName: string): Promise<KVStore>;
230
+
231
+ // Get stats store for table statistics
232
+ getStatsStore(schemaName: string, tableName: string): Promise<KVStore>;
233
+
234
+ // Get catalog store for DDL metadata
235
+ getCatalogStore(): Promise<KVStore>;
236
+
237
+ // Close specific stores
238
+ closeStore(schemaName: string, tableName: string): Promise<void>;
239
+ closeIndexStore(schemaName: string, tableName: string, indexName: string): Promise<void>;
240
+ closeAll(): Promise<void>;
241
+
242
+ // Optional: Delete stores. `indexNames` is the table's exact secondary-index
243
+ // names (from the schema) — build index store names from it via
244
+ // buildIndexStoreName instead of prefix-scanning `{table}_idx_`, which would
245
+ // also match a sibling table literally named `{table}_idx_<x>`.
246
+ deleteIndexStore?(schemaName: string, tableName: string, indexName: string): Promise<void>;
247
+ deleteTableStores?(schemaName: string, tableName: string, indexNames: readonly string[]): Promise<void>;
248
+
249
+ // Optional: Relocate a table's data + index stores for ALTER TABLE ... RENAME TO
250
+ // (`indexNames` carries the same authoritative, exact index list).
251
+ renameTableStores?(schemaName: string, oldName: string, newName: string, indexNames: readonly string[]): Promise<void>;
252
+ }
253
+ ```
254
+
255
+ ## Module Capabilities
256
+
257
+ The `StoreModule` reports its capabilities via `getCapabilities()`:
258
+
259
+ ```typescript
260
+ const storeModule = new StoreModule(provider);
261
+ const caps = storeModule.getCapabilities();
262
+
263
+ // {
264
+ // isolation: false, // Store module does NOT provide transaction isolation
265
+ // savepoints: true, // Coordinator-buffered ops support savepoints within a transaction
266
+ // persistent: true, // Data persists across restarts
267
+ // secondaryIndexes: true,// Supports secondary indexes
268
+ // rangeScans: true // Supports range scans
269
+ // }
270
+ ```
271
+
272
+ **Important:** The base `StoreModule` does not provide transaction isolation:
273
+ - No snapshot isolation: between connections, reads see only committed data, and concurrent readers may observe partial writes
274
+ - Within a transaction, reads through the table's shared coordinator DO see that transaction's own pending writes (read-your-own-writes). This extends to DML's own internal reads: the insert PK-conflict probe, the update/delete old-image reads, and the update PK-change conflict probe all read through the pending merge, so an INSERT/UPDATE/DELETE against a row written earlier in the same transaction sees that pending row — it raises a UNIQUE conflict (or evicts under `OR REPLACE`), cleans up secondary-index entries, tracks the correct row-count delta, and emits events carrying the pending `oldRow`
275
+ - Row-validating DDL reads the same effective stream: `create index` / `create unique index` populate from it (see above), and `alter table … add constraint … unique` plus the `set collate` re-validation of a covering non-PK UNIQUE scan it too — so a duplicate inserted earlier in the still-open transaction is rejected rather than surviving to commit
276
+ - Savepoints (create / release / rollback-to) work within a transaction via the coordinator's buffered op log
277
+ - Caveat: a DDL-commit operation (`replaceContents` / `renameTable`, e.g. `refresh materialized view` or `alter table … rename`) commits the coordinator mid-transaction, clearing the savepoint stack. A later `rollback to` / `release` targeting a now-vanished savepoint degrades to a no-op (warn-and-return) rather than throwing; the committed DDL and everything before it stays committed
278
+
279
+ ## Atomic multi-store commit (module-wide, cross-table)
280
+
281
+ A single `TransactionCoordinator` is shared by **every table of one storage
282
+ module** — it is the unit of cross-table atomicity. Every buffered op is
283
+ addressed by its explicit target `KVStore` handle (data ops, secondary-index
284
+ ops, and backing-host writes alike), so a transaction touching tables A and B
285
+ accumulates all of their stores' ops in one coordinator. Because the engine
286
+ commits virtual-table connections **sequentially** and the coordinator's
287
+ `commit()`/`rollback()` are **idempotent**, the first connection to commit
288
+ flushes **every** touched store of **every** table the transaction wrote; the
289
+ remaining connections no-op.
290
+
291
+ `TransactionCoordinator.commit()` thus writes each table's data store and each of
292
+ its secondary-index stores. By default it writes **one `KVStore.batch()` per
293
+ store, sequentially** — a crash between those batches can leave tables/indexes
294
+ divergent on disk, with no automatic healing (no worse than the prior per-table
295
+ commits, which were already non-atomic across tables).
296
+
297
+ A provider whose stores share a single durable commit domain can close that
298
+ window by implementing the optional `KVStoreProvider.beginAtomicBatch()`:
299
+
300
+ ```typescript
301
+ interface AtomicBatch {
302
+ put(store: KVStore, key: Uint8Array, value: Uint8Array): void;
303
+ delete(store: KVStore, key: Uint8Array): void;
304
+ write(): Promise<void>; // one durable, all-or-nothing physical commit
305
+ clear(): void;
306
+ }
307
+
308
+ interface KVStoreProvider {
309
+ // ...
310
+ // Open a batch spanning this provider's stores, or undefined when the provider
311
+ // has no shared commit domain (the coordinator then falls back to per-store batch()).
312
+ beginAtomicBatch?(): AtomicBatch | undefined;
313
+ }
314
+ ```
315
+
316
+ The batch addresses stores by **`KVStore` handle**, so it composes with the
317
+ coordinator's existing per-store bucketing without a name lookup. When present,
318
+ `commit()` queues every pending op — every store of **every table** in the
319
+ transaction — into one `AtomicBatch` and issues a single `write()`; all of those
320
+ tables commit or roll back together. When absent — or when the factory returns
321
+ `undefined` — behavior is byte-identical to the per-store loop, so providers
322
+ without a shared domain are unaffected.
323
+
324
+ The capability surface spans **multiple stores of one provider** (every store of
325
+ every table the module owns), giving full module-wide cross-table atomicity with
326
+ no interface change. The
327
+ [`@quereus/plugin-indexeddb`](../quereus-plugin-indexeddb) provider implements it
328
+ over its single IndexedDB database (multiple object stores, one
329
+ `db.transaction(...,'readwrite')`), invalidating each touched store's read cache
330
+ after a successful write so read-your-own-writes survives the cache.
331
+
332
+ ## Materialized-View Backing Host
333
+
334
+ The store module implements the engine's backing-host capability
335
+ (`StoreBackingHost`), so `create materialized view mv using store as <body>`
336
+ places the MV's backing table in persistent storage. Maintenance writes ride
337
+ the module's shared `TransactionCoordinator`'s pending state (committing/rolling
338
+ back in lockstep with the source write — and, since the coordinator is
339
+ module-wide, in the same all-or-nothing batch as a write to a same-module
340
+ source), mid-transaction reads of the MV see pending
341
+ maintenance through the read-your-own-writes merge, and the backing's text
342
+ primary-key columns are keyed under the store's `collation` arg (default
343
+ `NOCASE` — pass `using store(collation = 'BINARY')` for byte-exact keys). The
344
+ isolation wrapper forwards the capability automatically. See
345
+ [`docs/mv-backing-host.md` § The store host](../../docs/mv-backing-host.md#the-store-host-using-store).
346
+
347
+ ## External Row-Write Entry Point
348
+
349
+ `StoreTable.applyExternalRowChanges(ops)` applies trusted, externally-originated
350
+ row writes (e.g. inbound replication) directly to a **source** table's committed
351
+ storage — table-owned data-key put/delete, **secondary-index maintenance**, and
352
+ stats tracking — and returns the effective `BackingRowChange[]` (the shape
353
+ `Database.ingestExternalRowChanges` consumes). It is the index-maintaining
354
+ sibling of the backing host (whose MV backing tables carry no indexes): a caller
355
+ writing the data `KVStore` directly would silently skip index and stats upkeep.
356
+
357
+ Resolve the table with `StoreModule.getTableForExternalWrite(db, schema, table)`
358
+ (same ownership/wrapper resolution as `getBackingHost`), read a row's current
359
+ image with `StoreTable.readRowByPk(pk)`, then apply one `ExternalRowOp` per row:
360
+
361
+ ```typescript
362
+ const table = storeModule.getTableForExternalWrite(db, 'main', 'users');
363
+ if (table) {
364
+ const changes = await table.applyExternalRowChanges([
365
+ { op: 'upsert', row: [1, 'alice'] }, // full row, schema column order
366
+ { op: 'delete', pk: [2] }, // PK values, PK-definition order
367
+ ]);
368
+ }
369
+ ```
370
+
371
+ Deliberately emits **no** module data events (the caller owns emission and the
372
+ `remote` flag), opens **no** coordinator transaction (writes commit at once,
373
+ last-writer-wins against any pending local transaction), and runs **no**
374
+ constraint validation (the origin is trusted). No-ops are suppressed: a delete
375
+ of an absent key and a value-identical upsert (byte-faithful) write nothing and
376
+ report nothing.
377
+
378
+ ## Transaction Isolation
379
+
380
+ To add full ACID transaction semantics with snapshot isolation, wrap the store module with the `IsolationModule`:
381
+
382
+ ```typescript
383
+ import { Database, MemoryTableModule } from '@quereus/quereus';
384
+ import { IsolationModule } from '@quereus/isolation';
385
+ import { StoreModule, createIsolatedStoreModule } from '@quereus/store';
386
+ import { createLevelDBProvider } from '@quereus/plugin-leveldb';
387
+
388
+ const db = new Database();
389
+ const provider = createLevelDBProvider({ basePath: './data' });
390
+
391
+ // Option 1: Use the convenience function
392
+ const isolatedModule = createIsolatedStoreModule({ provider });
393
+ db.registerModule('store', isolatedModule);
394
+
395
+ // Option 2: Manual wrapping for more control
396
+ const storeModule = new StoreModule(provider);
397
+ const isolatedModule = new IsolationModule({
398
+ underlying: storeModule,
399
+ overlay: new MemoryTableModule(),
400
+ });
401
+ db.registerModule('store', isolatedModule);
402
+
403
+ // Now transactions have full isolation
404
+ await db.exec('BEGIN');
405
+ await db.exec(`INSERT INTO users VALUES (1, 'Alice')`);
406
+
407
+ // Read-your-own-writes: sees uncommitted insert
408
+ const user = await db.get('SELECT * FROM users WHERE id = 1');
409
+ console.log(user.name); // 'Alice'
410
+
411
+ await db.exec('COMMIT'); // Or ROLLBACK to discard
412
+ ```
413
+
414
+ The isolation layer provides:
415
+ - **Read-your-own-writes** within transactions
416
+ - **Snapshot isolation** for consistent reads
417
+ - **Savepoint support** via the overlay module
418
+
419
+ ### Checking for Isolation Support
420
+
421
+ ```typescript
422
+ import { hasIsolation } from '@quereus/store';
423
+
424
+ const storeModule = new StoreModule(provider);
425
+ console.log(hasIsolation(storeModule)); // false
426
+
427
+ const isolatedModule = createIsolatedStoreModule({ provider });
428
+ console.log(hasIsolation(isolatedModule)); // true
429
+ ```
430
+
431
+ ## API
432
+
433
+ ### Core Exports
434
+
435
+ | Export | Description |
436
+ |--------|-------------|
437
+ | `KVStore` | Key-value store interface (type) |
438
+ | `KVStoreProvider` | Store factory interface (type) |
439
+ | `WriteBatch` | Batch write interface (type) |
440
+ | `AtomicBatch` | Cross-store all-or-nothing batch from `KVStoreProvider.beginAtomicBatch` (type) |
441
+ | `IterateOptions` | Iteration options (type) |
442
+ | `StoreModule` | Generic VirtualTableModule |
443
+ | `StoreTable` | Virtual table implementation (incl. `applyExternalRowChanges` / `readRowByPk` for externally-applied source writes) |
444
+ | `ExternalRowOp` | One externally-applied row op (`upsert`/`delete`) for `StoreTable.applyExternalRowChanges` (type) |
445
+ | `resolvePkKeyCollations` | Per-PK-column key collations (pass to `buildDataKey`/`buildIndexKey` to match `StoreTable`'s key bytes) |
446
+ | `StoreConnection` | Transaction connection |
447
+ | `TransactionCoordinator` | Transaction management |
448
+ | `StoreEventEmitter` | Event system for data/schema changes |
449
+
450
+ ### Isolation Layer Utilities
451
+
452
+ | Export | Description |
453
+ |--------|-------------|
454
+ | `createIsolatedStoreModule` | Create store module with isolation layer |
455
+ | `hasIsolation` | Check if a module has isolation capability |
456
+ | `IsolatedStoreModuleConfig` | Configuration for isolated store module |
457
+
458
+ ### Caching
459
+
460
+ | Export | Description |
461
+ |--------|-------------|
462
+ | `CachedKVStore` | Read-through LRU cache wrapper for any `KVStore` |
463
+ | `CacheOptions` | Configuration for cache (maxEntries, maxBytes, enabled) |
464
+
465
+ ### Encoding Utilities
466
+
467
+ | Export | Description |
468
+ |--------|-------------|
469
+ | `encodeValue` | Encode a SQL value to sortable bytes |
470
+ | `decodeValue` | Decode bytes back to SQL value |
471
+ | `encodeCompositeKey` | Encode multiple values as composite key |
472
+ | `decodeCompositeKey` | Decode composite key to values |
473
+ | `BUILTIN_KEY_NORMALIZER_RESOLVER` | Built-ins-only key-normalizer resolver (`EncodeOptions.normalizers` default) |
474
+
475
+ ### Serialization Utilities
476
+
477
+ | Export | Description |
478
+ |--------|-------------|
479
+ | `serializeRow` | Serialize a row to bytes |
480
+ | `deserializeRow` | Deserialize bytes to row |
481
+ | `serializeValue` | Serialize a single value |
482
+ | `deserializeValue` | Deserialize a single value |
483
+
484
+ ### Key Building
485
+
486
+ | Export | Description |
487
+ |--------|-------------|
488
+ | `buildDataStoreName` | Build store name for table data |
489
+ | `buildIndexStoreName` | Build store name for an index |
490
+ | `buildStatsStoreName` | Build store name for table stats |
491
+ | `buildDataKey` | Build key for row data (encoded PK) |
492
+ | `buildIndexKey` | Build key for index entry |
493
+ | `buildCatalogKey` | Build key for a table's catalog entry (`{schema}.{table}`) |
494
+ | `buildViewCatalogKey` | Build key for a view's catalog entry (reserved `\x00view\x00` prefix) |
495
+ | `buildMaterializedViewCatalogKey` | Build key for an MV's catalog entry (reserved `\x00mview\x00` prefix) |
496
+ | `classifyCatalogKey` | Classify a loaded catalog key as `'table'` / `'view'` / `'materializedView'` |
497
+ | `buildFullScanBounds` | Build bounds for full table scan |
498
+ | `buildIndexPrefixBounds` | Build bounds for index prefix scan |
499
+ | `buildPkPrefixBounds` | Build bounds for a data-store PK prefix range (per-column DESC + key collations) |
500
+ | `buildCatalogScanBounds` | Build bounds for catalog scan |
501
+ | `CATALOG_STORE_NAME` | Reserved catalog store name constant |
502
+ | `STORE_SUFFIX` | Store name suffixes (INDEX, STATS) |
503
+
504
+ ## Related Packages
505
+
506
+ - [`@quereus/plugin-leveldb`](../quereus-plugin-leveldb/) - LevelDB implementation for Node.js
507
+ - [`@quereus/plugin-indexeddb`](../quereus-plugin-indexeddb/) - IndexedDB implementation for browsers
508
+ - [`@quereus/sync`](../quereus-sync/) - CRDT sync layer
509
+
510
+ ## License
511
+
512
+ MIT