@quereus/store 4.14.0 → 4.16.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +761 -702
- package/dist/src/common/cost-profile.d.ts +32 -0
- package/dist/src/common/cost-profile.d.ts.map +1 -1
- package/dist/src/common/cost-profile.js +30 -0
- package/dist/src/common/cost-profile.js.map +1 -1
- package/dist/src/common/events.d.ts +18 -28
- package/dist/src/common/events.d.ts.map +1 -1
- package/dist/src/common/events.js.map +1 -1
- package/dist/src/common/index.d.ts +1 -0
- package/dist/src/common/index.d.ts.map +1 -1
- package/dist/src/common/index.js +5 -0
- package/dist/src/common/index.js.map +1 -1
- package/dist/src/common/memory-store.d.ts +9 -0
- package/dist/src/common/memory-store.d.ts.map +1 -1
- package/dist/src/common/memory-store.js +11 -0
- package/dist/src/common/memory-store.js.map +1 -1
- package/dist/src/common/pk-key-resolution.d.ts +34 -0
- package/dist/src/common/pk-key-resolution.d.ts.map +1 -1
- package/dist/src/common/pk-key-resolution.js +47 -0
- package/dist/src/common/pk-key-resolution.js.map +1 -1
- package/dist/src/common/store-module-access-plan.d.ts +8 -0
- package/dist/src/common/store-module-access-plan.d.ts.map +1 -1
- package/dist/src/common/store-module-access-plan.js +362 -19
- package/dist/src/common/store-module-access-plan.js.map +1 -1
- package/dist/src/common/store-module-alter.d.ts.map +1 -1
- package/dist/src/common/store-module-alter.js +2 -3
- package/dist/src/common/store-module-alter.js.map +1 -1
- package/dist/src/common/store-module-schema-sync.d.ts.map +1 -1
- package/dist/src/common/store-module-schema-sync.js +6 -0
- package/dist/src/common/store-module-schema-sync.js.map +1 -1
- package/dist/src/common/store-table-scan.d.ts.map +1 -1
- package/dist/src/common/store-table-scan.js +19 -1
- package/dist/src/common/store-table-scan.js.map +1 -1
- package/dist/src/testing/index.d.ts +12 -0
- package/dist/src/testing/index.d.ts.map +1 -1
- package/dist/src/testing/index.js +12 -0
- package/dist/src/testing/index.js.map +1 -1
- package/dist/src/testing/kv-counting-store.d.ts +96 -0
- package/dist/src/testing/kv-counting-store.d.ts.map +1 -0
- package/dist/src/testing/kv-counting-store.js +226 -0
- package/dist/src/testing/kv-counting-store.js.map +1 -0
- package/dist/src/testing/kv-delegating-store.d.ts +28 -0
- package/dist/src/testing/kv-delegating-store.d.ts.map +1 -0
- package/dist/src/testing/kv-delegating-store.js +47 -0
- package/dist/src/testing/kv-delegating-store.js.map +1 -0
- package/dist/src/testing/memory-provider.d.ts +51 -0
- package/dist/src/testing/memory-provider.d.ts.map +1 -0
- package/dist/src/testing/memory-provider.js +124 -0
- package/dist/src/testing/memory-provider.js.map +1 -0
- package/package.json +5 -5
package/README.md
CHANGED
|
@@ -1,702 +1,761 @@
|
|
|
1
|
-
# @quereus/store
|
|
2
|
-
|
|
3
|
-
> **Stability: Beta** — complete and tested, but the surface is still being shaped; a
|
|
4
|
-
> breaking change may land in a minor release. Its on-disk key encoding is not frozen, so
|
|
5
|
-
> a format change can make an existing database unreadable without a migration step. See
|
|
6
|
-
> [Stability Tiers](../../docs/stability.md#tiers).
|
|
7
|
-
|
|
8
|
-
Abstract key-value storage module for [Quereus](https://github.com/gotchoices/quereus). Provides platform-agnostic interfaces and a generic `StoreModule` virtual table implementation.
|
|
9
|
-
|
|
10
|
-
## Architecture
|
|
11
|
-
|
|
12
|
-
This package provides the **abstract layer** that separates virtual table logic from platform-specific storage:
|
|
13
|
-
|
|
14
|
-
```
|
|
15
|
-
@quereus/store (this package)
|
|
16
|
-
├── KVStore interface - Abstract key-value store
|
|
17
|
-
├── KVStoreProvider interface - Store factory/management
|
|
18
|
-
├── StoreModule - Generic VirtualTableModule
|
|
19
|
-
├── StoreTable - Generic virtual table implementation
|
|
20
|
-
├── StoreConnection - Generic transaction support
|
|
21
|
-
└── Common utilities - Encoding, serialization, events
|
|
22
|
-
|
|
23
|
-
@quereus/plugin-leveldb (Node.js) @quereus/plugin-indexeddb (Browser)
|
|
24
|
-
├── LevelDBStore ├── IndexedDBStore
|
|
25
|
-
├── LevelDBProvider ├── IndexedDBProvider
|
|
26
|
-
└── Plugin registration ├── IndexedDBManager
|
|
27
|
-
└── CrossTabSync
|
|
28
|
-
```
|
|
29
|
-
|
|
30
|
-
This architecture enables:
|
|
31
|
-
- **Platform portability** - Same SQL tables work across Node.js, browsers, and mobile
|
|
32
|
-
- **Custom storage backends** - Implement `KVStore` for IndexedDB, LevelDB, LMDB, or other "NoSQL" stores
|
|
33
|
-
- **Dependency injection** - Use `KVStoreProvider` for store management
|
|
34
|
-
|
|
35
|
-
## Storage Architecture
|
|
36
|
-
|
|
37
|
-
The store module uses separate logical stores for different data types:
|
|
38
|
-
|
|
39
|
-
**Store Naming Convention:**
|
|
40
|
-
- `{schema}.{table}` - Data store (row data)
|
|
41
|
-
- `{schema}.{table}_idx_{indexName}` - Index stores (one per secondary index)
|
|
42
|
-
- `{prefix}.__stats__` - Unified stats store (row counts for all tables)
|
|
43
|
-
- `__catalog__` - Catalog store (DDL metadata)
|
|
44
|
-
|
|
45
|
-
**Key Formats:**
|
|
46
|
-
- **Data keys**: Encoded primary key (no prefix)
|
|
47
|
-
- **Index keys**: Encoded index columns + encoded PK
|
|
48
|
-
- **Index values**: The row's encoded **data key**. A secondary-index scan resolves each
|
|
49
|
-
matched entry back to its base row by reading this key, rather than decoding the index
|
|
50
|
-
key's PK suffix (that suffix is encoded lossily for a NOCASE/RTRIM PK column, so it is
|
|
51
|
-
not recoverable to SQL values). Entries are not covering — the row itself always lives
|
|
52
|
-
in the data store. The resolution is **batched**: the scan collects surviving index
|
|
53
|
-
entries (across every window of an `IN`-list seek, not just one) up to
|
|
54
|
-
`ROW_RESOLUTION_BATCH` = 256 and resolves each batch with a single `KVStore.getMany`,
|
|
55
|
-
so an indexed read of *N* rows costs about *N*/256 data-store round trips rather than
|
|
56
|
-
*N* serialized point reads — on IndexedDB, one readonly transaction per batch instead
|
|
57
|
-
of one per row. The trade is early termination: a `limit 1` collects one batch rather
|
|
58
|
-
than one row (the same trade `iterate`'s pager already makes).
|
|
59
|
-
- **Catalog keys**:
|
|
60
|
-
- Tables: `{schema}.{table}` as a string (the `CREATE TABLE` bundle, with its index DDL and any exposed-implicit-index tag DDL)
|
|
61
|
-
- Views: `\x00view\x00{schema}.{view}` (reserved-prefix; `generateViewDDL`)
|
|
62
|
-
- Materialized views: `\x00mview\x00{schema}.{mv}` (reserved-prefix; `generateMaterializedViewDDL`)
|
|
63
|
-
|
|
64
|
-
This design eliminates redundant prefixes and groups related stores together by table name. The leading-`0x00` view/MV prefixes never collide with an unprefixed table key, so a view/MV may safely share a name with a table; a full catalog scan returns all three kinds intermixed and rehydrate classifies each by its key prefix.
|
|
65
|
-
|
|
66
|
-
**Which predicate shapes become a seek.** Everything else is answered by a full scan plus
|
|
67
|
-
a re-check of the pushed predicate, which is always correct — the shapes below are the ones
|
|
68
|
-
that also *narrow the bytes read*:
|
|
69
|
-
|
|
70
|
-
| Shape | Served by |
|
|
71
|
-
|-------|-----------|
|
|
72
|
-
| Equality on every primary-key column | one data-store point read |
|
|
73
|
-
| `IN` on every primary-key column | one data-store point read per distinct key tuple (`plan=5`, see below) |
|
|
74
|
-
| `<` / `<=` / `>` / `>=` on the **leading** primary-key column | one data-store byte window |
|
|
75
|
-
| Equality on a contiguous **leading prefix** of a secondary index | one index byte window (`plan=2`) |
|
|
76
|
-
| `IN` on a contiguous leading prefix of a secondary index | one index window per distinct list value (`plan=5`, see below) |
|
|
77
|
-
| `<` / `<=` / `>` / `>=` on the **leading** column of a secondary index | one index byte window (`plan=3`) |
|
|
78
|
-
| Equality on a **strict** leading prefix of a secondary index **plus** a bound on the very next index column | one index byte window covering just the bounded slice inside the prefix (`plan=7`) |
|
|
79
|
-
|
|
80
|
-
The last row is what makes `where entity = ? and date between ? and ?` over an index on
|
|
81
|
-
`(entity, date)` read only the dated slice rather than every row of the entity. It needs the
|
|
82
|
-
prefix pinned to a **single** value per column — a multi-value `IN` prefix keeps the
|
|
83
|
-
multi-seek above and leaves the bound as a residual — and it gives up the narrowing (falling
|
|
84
|
-
back to the plain prefix window, answer unchanged) when the bounded column's key bytes do not
|
|
85
|
-
reproduce its collation's order, or when its bound value has no faithful key position under a
|
|
86
|
-
semantically-ordered type. A bound on a *later* index column than the one right after the
|
|
87
|
-
prefix is not seeked at all. No ordering is advertised for this path, so an `ORDER BY` on the
|
|
88
|
-
bounded column keeps its `Sort`.
|
|
89
|
-
|
|
90
|
-
**A seek priced above a full scan is dropped.** A secondary-index seek reads two things per
|
|
91
|
-
matched row — the index entry, then the row it names — while a scan reads one, so the module
|
|
92
|
-
charges that second read and then compares the arm against its own sequential-scan cost,
|
|
93
|
-
returning the scan when the seek prices higher. The predicate stays as a residual either
|
|
94
|
-
way, so the rows are identical and only the speed differs. The module keeps no per-column
|
|
95
|
-
statistics, so how many rows a predicate matches is guessed as a fixed fraction of the table
|
|
96
|
-
(10% for an equality, 30% for a range); that makes the comparison per-*index-shape* rather
|
|
97
|
-
than per-query, and today it only fires where the guess is the whole table. Multi-seeks are
|
|
98
|
-
exempt from the comparison — the engine reads their cost as a curve to decide its own key-set
|
|
99
|
-
rewrites, and answering it with a scan would switch that feature off.
|
|
100
|
-
|
|
101
|
-
**Backend cost profile.** How expensive those reads actually are depends on the backend: a
|
|
102
|
-
point read is nearly free on LevelDB (in-process, block-cached) and is a separate request
|
|
103
|
-
across an IPC boundary on IndexedDB. A provider can therefore declare a `costProfile`
|
|
104
|
-
(`KVCostProfile`, in `src/common/cost-profile.ts`), whose unit is **one row read
|
|
105
|
-
sequentially during a full scan = 1.0**:
|
|
106
|
-
|
|
107
|
-
| knob | what it prices | parity default | declared by IndexedDB |
|
|
108
|
-
|---|---|---|---|
|
|
109
|
-
| `pointRead` | resolving one secondary-index entry to the row it names | 1.0 | 3.0 |
|
|
110
|
-
| `seekPositioning` | one key of a multi-seek — positioning the window *and* reading its row | 0.5 | 5.0 |
|
|
111
|
-
|
|
112
|
-
Both fields are optional and an omitted field takes its parity default, which reproduces the
|
|
113
|
-
module's pre-profile constants exactly — **a provider that declares nothing plans exactly as
|
|
114
|
-
it always did.** A field declared non-finite or non-positive falls back to its parity value
|
|
115
|
-
with a warning rather than breaking planning. Only LevelDB's numbers are unmeasured (it
|
|
116
|
-
declares nothing, deliberately); IndexedDB's come from the benchmark in
|
|
117
|
-
`packages/quereus-plugin-indexeddb/bench/README.md`. The profile is expressed as a *ratio*,
|
|
118
|
-
not milliseconds, because these costs are request-latency dominated — a slower device scales
|
|
119
|
-
the seek path and the scan baseline together.
|
|
120
|
-
|
|
121
|
-
`seekPositioning` is the knob that moves plan choice today: it is what the engine's key-set
|
|
122
|
-
rewrite reads when it interpolates the key count at which a seek overtakes the plan it would
|
|
123
|
-
displace, so an expensive-seek backend stops firing that rewrite on small tables.
|
|
124
|
-
`pointRead` changes what an index arm *advertises* and how two indexes rank against each
|
|
125
|
-
other — but **the seek-vs-scan comparison above is deliberately still priced at the parity
|
|
126
|
-
1.0.** Because the match-count guess is a fixed fraction of the table, that comparison is
|
|
127
|
-
arm-disabling rather than arm-tuning: past `pointRead` 2.83 the range arm would be switched
|
|
128
|
-
off for every query on every table, off a guess no measurement can resolve. When real
|
|
129
|
-
per-column statistics replace the guess, the comparison will use the declared cost instead.
|
|
130
|
-
|
|
131
|
-
**`IN`-list index seeks ("multi-seek").** An `IN`-list on an indexed column
|
|
132
|
-
(`where v in (1, 2, 3)`, including parameter-bound lists) is served from the index as one
|
|
133
|
-
deduplicated, key-ordered point seek per distinct list value, instead of a full scan with
|
|
134
|
-
a residual filter. `NULL` list values match nothing and are skipped, duplicate values
|
|
135
|
-
yield their rows once, and a composite index serves the cross-product of per-column lists
|
|
136
|
-
(`a in (1,2) and b in (10,20)` is four seeks). Very large lists (over 1000 seek keys) and
|
|
137
|
-
lists on semantically-ordered column types (TIMESPAN, JSON) fall back to the scan path —
|
|
138
|
-
still correct, just not accelerated. When several secondary indexes can serve the same
|
|
139
|
-
predicate, the lowest-cost seek wins (equal costs keep the first-declared index), so
|
|
140
|
-
declaration order no longer decides whether a query does one seek or hundreds. The primary-key
|
|
141
|
-
arms still take precedence over any secondary index whenever they apply.
|
|
142
|
-
|
|
143
|
-
An `IN` covering the **whole primary key** takes the same path against the data store
|
|
144
|
-
itself: `where pk in (1, 2, 3)`, or `a in (…) and b in (…)` on a composite key, becomes one
|
|
145
|
-
deduplicated point read per distinct key tuple rather than a scan. Every gate above applies
|
|
146
|
-
(the 1000-key cap, the semantic-ordering decline, NULL-skip and dedup), and a key only
|
|
147
|
-
*partly* pinned — `a in (…)` alone on a `(a, b)` key — still scans with the predicate as a
|
|
148
|
-
residual. Unlike a secondary multi-seek, this one **advertises primary-key order**: the point
|
|
149
|
-
reads are emitted ascending by encoded data key, which is primary-key order (a `DESC` key
|
|
150
|
-
member's inversion is baked into its bytes), so `where pk in (…) order by pk` needs no `Sort`.
|
|
151
|
-
|
|
152
|
-
The same multi-seek path also serves an `IN` whose values only exist once the query runs
|
|
153
|
-
— `where v in (select … )`, which the engine may materialize into a set and hand down as
|
|
154
|
-
a seek (its `KeySetSemiJoin`). **Nothing in this module distinguishes the two.** The
|
|
155
|
-
engine stamps a `FilterInfo` byte-identical in shape to a literal list's, so every gate
|
|
156
|
-
above applies unchanged: the 1000-key cap, the semantic-ordering decline, the key-collation
|
|
157
|
-
guard, and the partial-index exclusion — on the primary-key arm as well as the secondary
|
|
158
|
-
one, so `where pk in (select …)` is served as a `_primary_` multi-seek too. At plan
|
|
159
|
-
time such a set describes itself only by a ceiling (`PredicateConstraint.runtimeSet.maxCount`),
|
|
160
|
-
which is what the cost and cap arithmetic judges. The engine also re-checks every row the
|
|
161
|
-
seek returns, so an over-fetching window costs only time.
|
|
162
|
-
|
|
163
|
-
**Catalog DDL is re-persisted on catalog-only mutations.** `ALTER … SET TAGS` (and the programmatic `setTableTags` / `setColumnTags` / `setConstraintTags` / `setViewTags` / `setMaterializedViewTags`), plus `CREATE`/`DROP VIEW` and `CREATE`/`DROP MATERIALIZED VIEW`, never reach `module.alterTable`/`module.destroy`. The module subscribes to the engine's schema-change events (`table_modified`, the `view_*` events, and the `materialized_view_*` events) and writes the matching `__catalog__` entry when its `generate*DDL` output changes — table / column / constraint / **index** / **view** / **materialized-view** tags, and view/MV lifecycle, all survive close → reopen. A table's bundle is its `CREATE TABLE` DDL, one `CREATE [UNIQUE] INDEX` line per secondary index, and one trailing `alter index … set tags (…)` line per *exposed implicit index* carrying user tags (an exposed implicit index is never materialized in the store's *engine-facing* schema — only in its internal enforcement schema, see the implicit-index note below — so its `UniqueConstraintSchema.exposedIndexTags` has no `CREATE INDEX` line to ride; the alter line re-applies silently on import). These async writes are serialized and drained by `closeAll()` (or the `whenCatalogPersisted()` barrier) before the provider closes. On reopen, `rehydrateCatalog` classifies entries by key prefix and imports them in phases — tables → views → materialized views, all through the engine's `importCatalog` (MVs re-materialize silently via the shared create core, dependency-ordered for MV-over-MV by fixpoint retry). See [`docs/view-persistence.md`](../../docs/view-persistence.md#view-and-materialized-view-persistence) for the full design.
|
|
164
|
-
|
|
165
|
-
**How a UNIQUE constraint is enforced.** For each row written, the store looks for a conflicting row through the cheapest sound route available:
|
|
166
|
-
|
|
167
|
-
1. **A linked row-time covering materialized view** — its backing table answers the uniqueness question.
|
|
168
|
-
2. **A physical secondary index realizing the constraint** — one bounded seek into the index store. Available for **every** non-derived `UNIQUE` (see the implicit-index note below), for a constraint that came from a `CREATE UNIQUE INDEX` (it names its own index), and for any *full* (non-partial) index whose columns match the constraint's. The index need not itself be UNIQUE.
|
|
169
|
-
3. **A full scan of the data store** — always correct, and O(rows) per row written.
|
|
170
|
-
|
|
171
|
-
Route 2 turns a bulk insert from O(n²) into roughly O(n log n). It is skipped for a constraint the index cannot answer soundly:
|
|
172
|
-
|
|
173
|
-
- A **partial** index cannot serve a constraint it does not derive from: it physically omits its out-of-scope rows, so a seek would miss a conflict among them.
|
|
174
|
-
- Index-column bytes are encoded under each index column's **own** key collation (the index's per-column `COLLATE`, else the table column's declared collation), and the constraint re-validates candidates under its enforcement collation `C` (the same resolution). For every index a designed path can hand the check the two are identical by construction, so the seek window is exactly the `C`-equal set. The check is kept because one undesigned path can still disagree — a user index whose name collides with the `_uc_*` a constraint would have minted lets a same-columns index with different collations be selected — and there a seek would under-fetch and silently accept a real duplicate, so the constraint falls back to the full scan. The **table key collation** `K` (the `collation` module option, default `NOCASE`) is not part of this decision; it is the fallback for an undecorated *primary-key* member only. "Can never hold text" (exempt — key bytes are type-native) is judged by physical representation, not by declared type name: an `ANY` or `JSON` column stores a string as a string, so neither is exempt — and because their `compare` ignores collation, they key hard-`BINARY`, so an `ANY`/`JSON` column carrying a declared `COLLATE` is exactly the shape that declines.
|
|
175
|
-
|
|
176
|
-
Whichever route runs, the conflicting row is re-validated identically: the row being written is excluded by primary key, each constrained column is compared under its enforcement collation (a `CREATE UNIQUE INDEX … (col COLLATE x)` enforces `x`, else the column's declared collation), and a partial constraint's predicate must hold on the candidate.
|
|
177
|
-
|
|
178
|
-
Because route 2 trusts the index store to hold an entry for every live row, `CREATE INDEX` populates the new index from the table's **effective** rows — committed rows merged with the open transaction's pending writes. A row inserted earlier in that transaction is therefore indexed, and participates in `CREATE UNIQUE INDEX`'s duplicate check, rather than being invisible to every later seek. Index entries are written outside the transaction coordinator, so a later `ROLLBACK` leaves entries for rows that no longer exist; every reader resolves an index entry to its live row and drops it when the row is gone or no longer matches, so a stale entry can never manufacture a result.
|
|
179
|
-
|
|
180
|
-
**Implicit per-constraint index (`_uc_*`).** Every non-derived `UNIQUE` — declared inline at `CREATE TABLE` or added by `ALTER TABLE … ADD CONSTRAINT` — is backed by a hidden secondary index named `<constraint name>` or, when unnamed, `_uc_<columns>` (the same convention the memory backend and the engine's `implicitIndexName` use). This is what makes route 2 reach a plain `UNIQUE`, so a bulk load no longer degrades to O(n²). The index is:
|
|
181
|
-
|
|
182
|
-
- **Kept out of the engine.** It lives only in the StoreTable's *enforcement* schema, never in the engine-registered schema, so the read-query planner does not see it (a plain `UNIQUE` gets no read-side plan from it — matching the memory backend) and it is never written to the catalog as a `CREATE INDEX`.
|
|
183
|
-
- **Derived on open.** Reconstructing a StoreTable re-materializes the schema entry from `uniqueConstraints`; the physical index store persists on disk under its deterministic name and is reopened lazily. (A store written *before* this feature has no `_uc_*` store on disk — backwards compatibility is waived project-wide; reopening such a database would need the index rebuilt.)
|
|
184
|
-
- **Reconciled across ALTER and CREATE / DROP INDEX.** `ADD CONSTRAINT` builds the physical store from the existing rows (after the existing-row duplicate check passes); `DROP CONSTRAINT` tears it down (so a later re-`ADD` cannot reopen stale entries); `RENAME CONSTRAINT`, and a column rename that changes an unnamed constraint's implicit name, move the store; a PK / collation / data-type `ALTER` re-encodes it via the same rebuild that re-encodes explicit indexes.
|
|
185
|
-
- **Relocated / reclaimed with the table.** `DROP TABLE` deletes the `_uc_*` store alongside the data + explicit-index stores; `RENAME TABLE` relocates it under the new name. Both resolve the physical store list from the *enforcement* schema (not the engine-facing `.indexes`, which omits `_uc_*`), so the implicit store is never stranded on drop nor left behind on rename (which would leave the renamed table seeking an empty store and accepting a duplicate).
|
|
186
|
-
- **Skipped when an explicit index already realizes the constraint.** A `UNIQUE` whose columns are covered — in the same order — by a full (non-partial) `CREATE INDEX` whose per-column collations match the declared column collations gets no `_uc_*` at all: that index enforces it, so a write maintains one structure instead of two. Reuse is refused for a partial index, a partial `UNIQUE`, an index over different or reordered columns, and a collation-mismatched index; those still build the `_uc_*` and both structures coexist. The decision is recomputed whenever the schema changes and is never persisted, so `CREATE INDEX` over a constrained column tears the redundant `_uc_*` store down and `DROP INDEX` rebuilds it from the live rows.
|
|
187
|
-
- **DDL-committing when it tears a store down.** Deleting a `_uc_*` store (or an explicit index store) discards a KVStore handle the transaction coordinator may hold buffered writes against, so those DDL statements flush the module's pending transaction first — the same `ddlTransactionality: 'auto-commit'` posture the row-rewriting `ALTER` arms take. A `CREATE INDEX` that retires no `_uc_*` does not flush.
|
|
188
|
-
|
|
189
|
-
## Installation
|
|
190
|
-
|
|
191
|
-
```bash
|
|
192
|
-
npm install @quereus/store
|
|
193
|
-
```
|
|
194
|
-
|
|
195
|
-
For platform-specific implementations:
|
|
196
|
-
```bash
|
|
197
|
-
# Node.js
|
|
198
|
-
npm install @quereus/plugin-leveldb
|
|
199
|
-
|
|
200
|
-
# Browser
|
|
201
|
-
npm install @quereus/plugin-indexeddb
|
|
202
|
-
```
|
|
203
|
-
|
|
204
|
-
## Usage
|
|
205
|
-
|
|
206
|
-
### With a Provider
|
|
207
|
-
|
|
208
|
-
```typescript
|
|
209
|
-
import { Database } from '@quereus/quereus';
|
|
210
|
-
import { StoreModule } from '@quereus/store';
|
|
211
|
-
import { createLevelDBProvider } from '@quereus/plugin-leveldb';
|
|
212
|
-
// OR: import { createIndexedDBProvider } from '@quereus/plugin-indexeddb';
|
|
213
|
-
|
|
214
|
-
const db = new Database();
|
|
215
|
-
|
|
216
|
-
// Create provider for your platform
|
|
217
|
-
const provider = createLevelDBProvider({ basePath: './data' });
|
|
218
|
-
|
|
219
|
-
// Create the generic store module with your provider
|
|
220
|
-
const storeModule = new StoreModule(provider);
|
|
221
|
-
db.registerModule('store', storeModule);
|
|
222
|
-
|
|
223
|
-
// Use it in SQL
|
|
224
|
-
await db.exec(`
|
|
225
|
-
create table users (id integer primary key, name text)
|
|
226
|
-
using store
|
|
227
|
-
`);
|
|
228
|
-
```
|
|
229
|
-
|
|
230
|
-
### Custom Storage Backend
|
|
231
|
-
|
|
232
|
-
Implement `KVStore` and `KVStoreProvider` to create custom storage backends:
|
|
233
|
-
|
|
234
|
-
```typescript
|
|
235
|
-
import { defaultGetMany, type KVStore, type KVStoreProvider } from '@quereus/store';
|
|
236
|
-
|
|
237
|
-
class MyCustomStore implements KVStore {
|
|
238
|
-
async get(key: Uint8Array) { /* ... */ }
|
|
239
|
-
// Positional batch point-read. Override it when the backend has a native multi-get or a
|
|
240
|
-
// way to issue every request on one round trip; otherwise `defaultGetMany(this, keys)`.
|
|
241
|
-
getMany(keys: readonly Uint8Array[]) { return defaultGetMany(this, keys); }
|
|
242
|
-
async put(key: Uint8Array, value: Uint8Array) { /* ... */ }
|
|
243
|
-
async delete(key: Uint8Array) { /* ... */ }
|
|
244
|
-
async has(key: Uint8Array) { /* ... */ }
|
|
245
|
-
iterate(options?: IterateOptions) { /* ... */ }
|
|
246
|
-
batch() { /* ... */ }
|
|
247
|
-
async close() { /* ... */ }
|
|
248
|
-
async approximateCount(options?: IterateOptions) { /* ... */ }
|
|
249
|
-
}
|
|
250
|
-
|
|
251
|
-
class MyCustomProvider implements KVStoreProvider {
|
|
252
|
-
async getStore(schemaName: string, tableName: string) {
|
|
253
|
-
return new MyCustomStore(/* ... */);
|
|
254
|
-
}
|
|
255
|
-
async getIndexStore(schemaName: string, tableName: string, indexName: string) {
|
|
256
|
-
return new MyCustomStore(/* ... */);
|
|
257
|
-
}
|
|
258
|
-
async getStatsStore(schemaName: string, tableName: string) {
|
|
259
|
-
return new MyCustomStore(/* ... */);
|
|
260
|
-
}
|
|
261
|
-
async getCatalogStore() { /* ... */ }
|
|
262
|
-
async closeStore(schemaName: string, tableName: string) { /* ... */ }
|
|
263
|
-
async closeIndexStore(schemaName: string, tableName: string, indexName: string) { /* ... */ }
|
|
264
|
-
async closeAll() { /* ... */ }
|
|
265
|
-
}
|
|
266
|
-
|
|
267
|
-
// Use it with StoreModule
|
|
268
|
-
const provider = new MyCustomProvider();
|
|
269
|
-
const module = new StoreModule(provider);
|
|
270
|
-
db.registerModule('store', module);
|
|
271
|
-
```
|
|
272
|
-
|
|
273
|
-
**Validate a new backend against the shared conformance suite.** `@quereus/store/testing`
|
|
274
|
-
exports `runKVStoreConformance(name, makeBackend)` — one parameterized battery of
|
|
275
|
-
behavioral tests written against the `KVStore` contract (point ops, ordering, range
|
|
276
|
-
iteration, streaming across page boundaries, batch semantics, optional persistence,
|
|
277
|
-
cross-backend encoded-key ordering, bounded iteration, and the batch point-read). Wire a
|
|
278
|
-
tiny lifecycle adapter and run it under Mocha so any drift from the contract fails a test:
|
|
279
|
-
|
|
280
|
-
```typescript
|
|
281
|
-
import { runKVStoreConformance } from '@quereus/store/testing';
|
|
282
|
-
|
|
283
|
-
runKVStoreConformance('MyCustomStore', () => ({
|
|
284
|
-
open: async () => new MyCustomStore(/* ... */),
|
|
285
|
-
// Omit `reopen` for a non-persistent backend; supply it (reopen the SAME keyspace
|
|
286
|
-
// without wiping) to also exercise the persistence tier.
|
|
287
|
-
teardown: async () => { /* close handles, remove backing storage */ },
|
|
288
|
-
// Optional. The battery holds only a KVStore handle and cannot see what the backend
|
|
289
|
-
// reads underneath, so supply a counter over whatever the store reads FROM plus the
|
|
290
|
-
// backend's batch size, and the bounded-iteration tier also measures reads. Omit it
|
|
291
|
-
// and that tier still checks that abandoning an iteration releases resources.
|
|
292
|
-
readMeter: { entriesRead: () => myBackingReads, maxReadAhead: 1 },
|
|
293
|
-
// Also optional, and a DIFFERENT counter: `readMeter` counts entries an iteration
|
|
294
|
-
// yields, so it cannot see a point read. Supply this — trips to backing storage, one
|
|
295
|
-
// per `get`, one per native multi-get — and the batch point-read tier additionally
|
|
296
|
-
// asserts that `getMany` over K keys costs exactly ONE trip. Omit it on a backend whose
|
|
297
|
-
// `getMany` is the shared fallback; that tier's correctness cases still run.
|
|
298
|
-
pointReadMeter: { roundTrips: () => myBackingRoundTrips },
|
|
299
|
-
}));
|
|
300
|
-
```
|
|
301
|
-
|
|
302
|
-
See `test/kv-conformance.spec.ts` (in-memory), and the `test/conformance.spec.ts` of each
|
|
303
|
-
backend plugin — LevelDB, IndexedDB, React Native LevelDB, NativeScript SQLite — for worked
|
|
304
|
-
adapters. The two mobile plugins run against an in-process stand-in for their native module
|
|
305
|
-
(a mock LevelDB, and better-sqlite3 behind the NativeScript SQLite interface), which is what
|
|
306
|
-
makes them testable off-device; the NativeScript one uses a temp file database so it also
|
|
307
|
-
drives the persistence tier.
|
|
308
|
-
|
|
309
|
-
**If your provider implements `beginAtomicBatch`, also run the provider suite.** The same
|
|
310
|
-
entry point exports `runKVProviderConformance(name, makeProviderBackend)` — the battery for
|
|
311
|
-
the provider-level contract that ops queued across SEVERAL of the provider's stores commit
|
|
312
|
-
in one durable, all-or-nothing physical write (multi-store commit landing only in each op's
|
|
313
|
-
own store, nothing visible until `write()`, mixed put + delete, a delete of a missing key,
|
|
314
|
-
same-key last-op-wins, `clear()` discarding, the empty `write()` no-op, and `MISUSE` on a
|
|
315
|
-
store handle from a different provider):
|
|
316
|
-
|
|
317
|
-
```typescript
|
|
318
|
-
import { runKVProviderConformance } from '@quereus/store/testing';
|
|
319
|
-
|
|
320
|
-
runKVProviderConformance('MyCustomProvider atomic batch', () => ({
|
|
321
|
-
open: async () => new MyCustomProvider(/* fresh, empty keyspace */),
|
|
322
|
-
// A SECOND provider over a DIFFERENT keyspace — source of the foreign handle that
|
|
323
|
-
// must be rejected with MISUSE.
|
|
324
|
-
openForeign: async () => new MyCustomProvider(/* another keyspace */),
|
|
325
|
-
teardown: async () => { /* close both, remove backing storage */ },
|
|
326
|
-
}));
|
|
327
|
-
```
|
|
328
|
-
|
|
329
|
-
The LevelDB / IndexedDB plugins' `test/atomic-batch.spec.ts` run it, keeping only their
|
|
330
|
-
backend-specific cases (LevelDB's "no batch before the shared root is open", IndexedDB's
|
|
331
|
-
post-write read-cache invalidation) alongside.
|
|
332
|
-
|
|
333
|
-
**Every provider runs the store-name distinctness battery.** `runStoreNameDistinctness(name,
|
|
334
|
-
makeNamingBackend)` holds a provider to the one naming rule the engine depends on: two
|
|
335
|
-
logical stores it considers distinct never share one physical store. A provider must build
|
|
336
|
-
its names with `buildDataStoreName` / `buildIndexStoreName` and escape them into its own
|
|
337
|
-
namespace injectively; a name the namespace cannot represent must be *rejected*, never folded
|
|
338
|
-
onto another. The battery asserts this behaviorally — it opens a corpus of adversarial
|
|
339
|
-
`(schema, table)` and `(schema, table, index)` names, writes a distinct marker into each under
|
|
340
|
-
one shared key, and checks each still reads back its own — plus that two spellings of one
|
|
341
|
-
case-insensitive identifier land on the SAME store, and that a user table named `__stats__` or
|
|
342
|
-
`__catalog__` stays off the reserved stores:
|
|
343
|
-
|
|
344
|
-
```typescript
|
|
345
|
-
import { runStoreNameDistinctness } from '@quereus/store/testing';
|
|
346
|
-
|
|
347
|
-
runStoreNameDistinctness('MyCustomProvider store names', () => ({
|
|
348
|
-
open: async () => new MyCustomProvider(/* fresh, empty keyspace */),
|
|
349
|
-
teardown: async () => { /* close the provider, remove backing storage */ },
|
|
350
|
-
}));
|
|
351
|
-
```
|
|
352
|
-
|
|
353
|
-
All four backend plugins register it in their `test/conformance.spec.ts`. LevelDB and
|
|
354
|
-
IndexedDB are its reference implementations, so a failure there means the battery is wrong
|
|
355
|
-
rather than the plugin. `assertStoreNamesDistinct` — its core assertion — is exported
|
|
356
|
-
standalone; `test/store-name-distinctness.spec.ts` drives it against provider doubles built
|
|
357
|
-
to fold, truncate, and over-reject, so the guard itself is watched failing.
|
|
358
|
-
|
|
359
|
-
**Every provider runs the store-reclaim battery too.** `runStoreReclaimConformance(name,
|
|
360
|
-
makeReclaimBackend)` holds a provider to the other rule the engine depends on: when it is
|
|
361
|
-
told to delete a table's stores, those stores must be *erased*, not merely closed. Store
|
|
362
|
-
handles are re-opened lazily by name, so a provider that only drops its cached handle leaves
|
|
363
|
-
the bytes in place and the next table created under that name comes back holding the dropped
|
|
364
|
-
table's rows — a defect two of the four shipped providers had. The battery deletes and
|
|
365
|
-
re-opens by the same names, checking each store reads back empty through both a point `get`
|
|
366
|
-
and a scan, that a store re-created under a dropped name holds only its own new rows, that
|
|
367
|
-
`deleteIndexStore` touches only that one index, that sibling tables (including one literally
|
|
368
|
-
named `{table}_idx_<x>`), an index store the caller did not name, and the reserved
|
|
369
|
-
`__stats__` / `__catalog__` stores all survive, that deleting a store that was never created
|
|
370
|
-
is a no-op rather than a throw, that a store whose handle the caller already closed is still
|
|
371
|
-
erased (the order every `drop index` takes), and that a store holding more rows than a
|
|
372
|
-
provider erases in one pass comes back empty to the last key:
|
|
373
|
-
|
|
374
|
-
```typescript
|
|
375
|
-
import { runStoreReclaimConformance } from '@quereus/store/testing';
|
|
376
|
-
|
|
377
|
-
runStoreReclaimConformance('MyCustomProvider store reclaim', () => ({
|
|
378
|
-
open: async () => new MyCustomProvider(/* fresh, empty keyspace */),
|
|
379
|
-
teardown: async () => { /* close the provider, remove backing storage */ },
|
|
380
|
-
}));
|
|
381
|
-
```
|
|
382
|
-
|
|
383
|
-
It takes the same `KVProviderLifecycle` adapter as the naming battery, so a plugin's spec
|
|
384
|
-
builds one closure and registers both. Both delete hooks are optional on `KVStoreProvider`;
|
|
385
|
-
a provider that omits one is skipped for those cases, and the skip is printed rather than
|
|
386
|
-
silent. Reclaiming bytes *on disk* is deliberately not asserted — a provider may leave an
|
|
387
|
-
emptied database directory or an unshrunk file behind and still be correct.
|
|
388
|
-
|
|
389
|
-
|
|
390
|
-
|
|
391
|
-
|
|
392
|
-
|
|
393
|
-
|
|
394
|
-
|
|
395
|
-
|
|
396
|
-
|
|
397
|
-
|
|
398
|
-
|
|
399
|
-
|
|
400
|
-
|
|
401
|
-
|
|
402
|
-
|
|
403
|
-
|
|
404
|
-
|
|
405
|
-
|
|
406
|
-
|
|
407
|
-
|
|
408
|
-
|
|
409
|
-
|
|
410
|
-
|
|
411
|
-
|
|
412
|
-
|
|
413
|
-
|
|
414
|
-
|
|
415
|
-
|
|
416
|
-
|
|
417
|
-
|
|
418
|
-
|
|
419
|
-
|
|
420
|
-
|
|
421
|
-
|
|
422
|
-
|
|
423
|
-
|
|
424
|
-
|
|
425
|
-
|
|
426
|
-
|
|
427
|
-
|
|
428
|
-
|
|
429
|
-
|
|
430
|
-
|
|
431
|
-
|
|
432
|
-
|
|
433
|
-
|
|
434
|
-
|
|
435
|
-
|
|
436
|
-
|
|
437
|
-
|
|
438
|
-
|
|
439
|
-
|
|
440
|
-
|
|
441
|
-
|
|
442
|
-
|
|
443
|
-
|
|
444
|
-
|
|
445
|
-
|
|
446
|
-
|
|
447
|
-
|
|
448
|
-
|
|
449
|
-
|
|
450
|
-
|
|
451
|
-
|
|
452
|
-
|
|
453
|
-
|
|
454
|
-
|
|
455
|
-
//
|
|
456
|
-
//
|
|
457
|
-
|
|
458
|
-
|
|
459
|
-
|
|
460
|
-
|
|
461
|
-
|
|
462
|
-
|
|
463
|
-
|
|
464
|
-
|
|
465
|
-
|
|
466
|
-
|
|
467
|
-
|
|
468
|
-
|
|
469
|
-
|
|
470
|
-
|
|
471
|
-
|
|
472
|
-
|
|
473
|
-
|
|
474
|
-
|
|
475
|
-
|
|
476
|
-
|
|
477
|
-
|
|
478
|
-
|
|
479
|
-
|
|
480
|
-
|
|
481
|
-
|
|
482
|
-
|
|
483
|
-
|
|
484
|
-
|
|
485
|
-
|
|
486
|
-
|
|
487
|
-
|
|
488
|
-
|
|
489
|
-
|
|
490
|
-
|
|
491
|
-
|
|
492
|
-
|
|
493
|
-
|
|
494
|
-
|
|
495
|
-
|
|
496
|
-
|
|
497
|
-
|
|
498
|
-
|
|
499
|
-
|
|
500
|
-
|
|
501
|
-
|
|
502
|
-
|
|
503
|
-
|
|
504
|
-
|
|
505
|
-
|
|
506
|
-
|
|
507
|
-
|
|
508
|
-
|
|
509
|
-
|
|
510
|
-
|
|
511
|
-
|
|
512
|
-
|
|
513
|
-
|
|
514
|
-
|
|
515
|
-
|
|
516
|
-
|
|
517
|
-
|
|
518
|
-
`
|
|
519
|
-
|
|
520
|
-
|
|
521
|
-
|
|
522
|
-
|
|
523
|
-
|
|
524
|
-
(
|
|
525
|
-
|
|
526
|
-
|
|
527
|
-
|
|
528
|
-
|
|
529
|
-
|
|
530
|
-
|
|
531
|
-
|
|
532
|
-
|
|
533
|
-
|
|
534
|
-
|
|
535
|
-
|
|
536
|
-
|
|
537
|
-
|
|
538
|
-
|
|
539
|
-
|
|
540
|
-
|
|
541
|
-
|
|
542
|
-
|
|
543
|
-
|
|
544
|
-
|
|
545
|
-
|
|
546
|
-
|
|
547
|
-
|
|
548
|
-
|
|
549
|
-
|
|
550
|
-
|
|
551
|
-
|
|
552
|
-
|
|
553
|
-
|
|
554
|
-
|
|
555
|
-
|
|
556
|
-
|
|
557
|
-
|
|
558
|
-
|
|
559
|
-
|
|
560
|
-
|
|
561
|
-
|
|
562
|
-
|
|
563
|
-
|
|
564
|
-
|
|
565
|
-
|
|
566
|
-
|
|
567
|
-
|
|
568
|
-
|
|
569
|
-
|
|
570
|
-
|
|
571
|
-
|
|
572
|
-
|
|
573
|
-
|
|
574
|
-
|
|
575
|
-
|
|
576
|
-
|
|
577
|
-
|
|
578
|
-
|
|
579
|
-
|
|
580
|
-
|
|
581
|
-
|
|
582
|
-
|
|
583
|
-
|
|
584
|
-
|
|
585
|
-
|
|
586
|
-
|
|
587
|
-
|
|
588
|
-
|
|
589
|
-
|
|
590
|
-
|
|
591
|
-
|
|
592
|
-
|
|
593
|
-
|
|
594
|
-
|
|
595
|
-
|
|
596
|
-
|
|
597
|
-
|
|
598
|
-
|
|
599
|
-
|
|
600
|
-
|
|
601
|
-
|
|
602
|
-
|
|
603
|
-
|
|
604
|
-
|
|
605
|
-
|
|
606
|
-
|
|
607
|
-
|
|
608
|
-
|
|
609
|
-
|
|
610
|
-
|
|
611
|
-
|
|
612
|
-
|
|
613
|
-
|
|
614
|
-
|
|
615
|
-
|
|
616
|
-
|
|
617
|
-
|
|
618
|
-
|
|
619
|
-
|
|
620
|
-
|
|
621
|
-
|
|
622
|
-
|
|
623
|
-
|
|
624
|
-
|
|
625
|
-
|
|
626
|
-
|
|
627
|
-
|
|
628
|
-
|
|
629
|
-
|
|
630
|
-
|
|
631
|
-
|
|
632
|
-
|
|
633
|
-
|
|
634
|
-
|
|
635
|
-
|
|
636
|
-
|
|
637
|
-
|
|
638
|
-
|
|
639
|
-
|
|
640
|
-
|
|
641
|
-
|
|
642
|
-
|
|
643
|
-
|
|
644
|
-
|
|
645
|
-
|
|
646
|
-
|
|
647
|
-
|
|
648
|
-
|
|
649
|
-
|
|
650
|
-
|
|
651
|
-
|
|
652
|
-
|
|
653
|
-
|
|
654
|
-
|
|
655
|
-
|
|
656
|
-
|
|
657
|
-
|
|
658
|
-
|
|
659
|
-
|
|
660
|
-
|
|
661
|
-
|
|
662
|
-
|
|
663
|
-
|
|
664
|
-
|
|
665
|
-
|
|
666
|
-
|
|
667
|
-
|
|
668
|
-
|
|
669
|
-
|
|
670
|
-
|
|
671
|
-
|
|
672
|
-
|
|
673
|
-
|
|
674
|
-
|
|
675
|
-
|
|
676
|
-
|
|
677
|
-
|
|
678
|
-
|
|
679
|
-
|
|
680
|
-
|
|
681
|
-
|
|
682
|
-
|
|
683
|
-
|
|
|
684
|
-
|
|
685
|
-
| `
|
|
686
|
-
| `
|
|
687
|
-
| `
|
|
688
|
-
| `
|
|
689
|
-
| `
|
|
690
|
-
| `
|
|
691
|
-
| `
|
|
692
|
-
| `
|
|
693
|
-
|
|
694
|
-
|
|
695
|
-
|
|
696
|
-
|
|
697
|
-
|
|
698
|
-
|
|
699
|
-
|
|
700
|
-
|
|
701
|
-
|
|
702
|
-
|
|
1
|
+
# @quereus/store
|
|
2
|
+
|
|
3
|
+
> **Stability: Beta** — complete and tested, but the surface is still being shaped; a
|
|
4
|
+
> breaking change may land in a minor release. Its on-disk key encoding is not frozen, so
|
|
5
|
+
> a format change can make an existing database unreadable without a migration step. See
|
|
6
|
+
> [Stability Tiers](../../docs/stability.md#tiers).
|
|
7
|
+
|
|
8
|
+
Abstract key-value storage module for [Quereus](https://github.com/gotchoices/quereus). Provides platform-agnostic interfaces and a generic `StoreModule` virtual table implementation.
|
|
9
|
+
|
|
10
|
+
## Architecture
|
|
11
|
+
|
|
12
|
+
This package provides the **abstract layer** that separates virtual table logic from platform-specific storage:
|
|
13
|
+
|
|
14
|
+
```
|
|
15
|
+
@quereus/store (this package)
|
|
16
|
+
├── KVStore interface - Abstract key-value store
|
|
17
|
+
├── KVStoreProvider interface - Store factory/management
|
|
18
|
+
├── StoreModule - Generic VirtualTableModule
|
|
19
|
+
├── StoreTable - Generic virtual table implementation
|
|
20
|
+
├── StoreConnection - Generic transaction support
|
|
21
|
+
└── Common utilities - Encoding, serialization, events
|
|
22
|
+
|
|
23
|
+
@quereus/plugin-leveldb (Node.js) @quereus/plugin-indexeddb (Browser)
|
|
24
|
+
├── LevelDBStore ├── IndexedDBStore
|
|
25
|
+
├── LevelDBProvider ├── IndexedDBProvider
|
|
26
|
+
└── Plugin registration ├── IndexedDBManager
|
|
27
|
+
└── CrossTabSync
|
|
28
|
+
```
|
|
29
|
+
|
|
30
|
+
This architecture enables:
|
|
31
|
+
- **Platform portability** - Same SQL tables work across Node.js, browsers, and mobile
|
|
32
|
+
- **Custom storage backends** - Implement `KVStore` for IndexedDB, LevelDB, LMDB, or other "NoSQL" stores
|
|
33
|
+
- **Dependency injection** - Use `KVStoreProvider` for store management
|
|
34
|
+
|
|
35
|
+
## Storage Architecture
|
|
36
|
+
|
|
37
|
+
The store module uses separate logical stores for different data types:
|
|
38
|
+
|
|
39
|
+
**Store Naming Convention:**
|
|
40
|
+
- `{schema}.{table}` - Data store (row data)
|
|
41
|
+
- `{schema}.{table}_idx_{indexName}` - Index stores (one per secondary index)
|
|
42
|
+
- `{prefix}.__stats__` - Unified stats store (row counts for all tables)
|
|
43
|
+
- `__catalog__` - Catalog store (DDL metadata)
|
|
44
|
+
|
|
45
|
+
**Key Formats:**
|
|
46
|
+
- **Data keys**: Encoded primary key (no prefix)
|
|
47
|
+
- **Index keys**: Encoded index columns + encoded PK
|
|
48
|
+
- **Index values**: The row's encoded **data key**. A secondary-index scan resolves each
|
|
49
|
+
matched entry back to its base row by reading this key, rather than decoding the index
|
|
50
|
+
key's PK suffix (that suffix is encoded lossily for a NOCASE/RTRIM PK column, so it is
|
|
51
|
+
not recoverable to SQL values). Entries are not covering — the row itself always lives
|
|
52
|
+
in the data store. The resolution is **batched**: the scan collects surviving index
|
|
53
|
+
entries (across every window of an `IN`-list seek, not just one) up to
|
|
54
|
+
`ROW_RESOLUTION_BATCH` = 256 and resolves each batch with a single `KVStore.getMany`,
|
|
55
|
+
so an indexed read of *N* rows costs about *N*/256 data-store round trips rather than
|
|
56
|
+
*N* serialized point reads — on IndexedDB, one readonly transaction per batch instead
|
|
57
|
+
of one per row. The trade is early termination: a `limit 1` collects one batch rather
|
|
58
|
+
than one row (the same trade `iterate`'s pager already makes).
|
|
59
|
+
- **Catalog keys**:
|
|
60
|
+
- Tables: `{schema}.{table}` as a string (the `CREATE TABLE` bundle, with its index DDL and any exposed-implicit-index tag DDL)
|
|
61
|
+
- Views: `\x00view\x00{schema}.{view}` (reserved-prefix; `generateViewDDL`)
|
|
62
|
+
- Materialized views: `\x00mview\x00{schema}.{mv}` (reserved-prefix; `generateMaterializedViewDDL`)
|
|
63
|
+
|
|
64
|
+
This design eliminates redundant prefixes and groups related stores together by table name. The leading-`0x00` view/MV prefixes never collide with an unprefixed table key, so a view/MV may safely share a name with a table; a full catalog scan returns all three kinds intermixed and rehydrate classifies each by its key prefix.
|
|
65
|
+
|
|
66
|
+
**Which predicate shapes become a seek.** Everything else is answered by a full scan plus
|
|
67
|
+
a re-check of the pushed predicate, which is always correct — the shapes below are the ones
|
|
68
|
+
that also *narrow the bytes read*:
|
|
69
|
+
|
|
70
|
+
| Shape | Served by |
|
|
71
|
+
|-------|-----------|
|
|
72
|
+
| Equality on every primary-key column | one data-store point read |
|
|
73
|
+
| `IN` on every primary-key column | one data-store point read per distinct key tuple (`plan=5`, see below) |
|
|
74
|
+
| `<` / `<=` / `>` / `>=` on the **leading** primary-key column | one data-store byte window |
|
|
75
|
+
| Equality on a contiguous **leading prefix** of a secondary index | one index byte window (`plan=2`) |
|
|
76
|
+
| `IN` on a contiguous leading prefix of a secondary index | one index window per distinct list value (`plan=5`, see below) |
|
|
77
|
+
| `<` / `<=` / `>` / `>=` on the **leading** column of a secondary index | one index byte window (`plan=3`) |
|
|
78
|
+
| Equality on a **strict** leading prefix of a secondary index **plus** a bound on the very next index column | one index byte window covering just the bounded slice inside the prefix (`plan=7`) |
|
|
79
|
+
|
|
80
|
+
The last row is what makes `where entity = ? and date between ? and ?` over an index on
|
|
81
|
+
`(entity, date)` read only the dated slice rather than every row of the entity. It needs the
|
|
82
|
+
prefix pinned to a **single** value per column — a multi-value `IN` prefix keeps the
|
|
83
|
+
multi-seek above and leaves the bound as a residual — and it gives up the narrowing (falling
|
|
84
|
+
back to the plain prefix window, answer unchanged) when the bounded column's key bytes do not
|
|
85
|
+
reproduce its collation's order, or when its bound value has no faithful key position under a
|
|
86
|
+
semantically-ordered type. A bound on a *later* index column than the one right after the
|
|
87
|
+
prefix is not seeked at all. No ordering is advertised for this path, so an `ORDER BY` on the
|
|
88
|
+
bounded column keeps its `Sort`.
|
|
89
|
+
|
|
90
|
+
**A seek priced above a full scan is dropped.** A secondary-index seek reads two things per
|
|
91
|
+
matched row — the index entry, then the row it names — while a scan reads one, so the module
|
|
92
|
+
charges that second read and then compares the arm against its own sequential-scan cost,
|
|
93
|
+
returning the scan when the seek prices higher. The predicate stays as a residual either
|
|
94
|
+
way, so the rows are identical and only the speed differs. The module keeps no per-column
|
|
95
|
+
statistics, so how many rows a predicate matches is guessed as a fixed fraction of the table
|
|
96
|
+
(10% for an equality, 30% for a range); that makes the comparison per-*index-shape* rather
|
|
97
|
+
than per-query, and today it only fires where the guess is the whole table. Multi-seeks are
|
|
98
|
+
exempt from the comparison — the engine reads their cost as a curve to decide its own key-set
|
|
99
|
+
rewrites, and answering it with a scan would switch that feature off.
|
|
100
|
+
|
|
101
|
+
**Backend cost profile.** How expensive those reads actually are depends on the backend: a
|
|
102
|
+
point read is nearly free on LevelDB (in-process, block-cached) and is a separate request
|
|
103
|
+
across an IPC boundary on IndexedDB. A provider can therefore declare a `costProfile`
|
|
104
|
+
(`KVCostProfile`, in `src/common/cost-profile.ts`), whose unit is **one row read
|
|
105
|
+
sequentially during a full scan = 1.0**:
|
|
106
|
+
|
|
107
|
+
| knob | what it prices | parity default | declared by IndexedDB |
|
|
108
|
+
|---|---|---|---|
|
|
109
|
+
| `pointRead` | resolving one secondary-index entry to the row it names | 1.0 | 3.0 |
|
|
110
|
+
| `seekPositioning` | one key of a multi-seek — positioning the window *and* reading its row | 0.5 | 5.0 |
|
|
111
|
+
|
|
112
|
+
Both fields are optional and an omitted field takes its parity default, which reproduces the
|
|
113
|
+
module's pre-profile constants exactly — **a provider that declares nothing plans exactly as
|
|
114
|
+
it always did.** A field declared non-finite or non-positive falls back to its parity value
|
|
115
|
+
with a warning rather than breaking planning. Only LevelDB's numbers are unmeasured (it
|
|
116
|
+
declares nothing, deliberately); IndexedDB's come from the benchmark in
|
|
117
|
+
`packages/quereus-plugin-indexeddb/bench/README.md`. The profile is expressed as a *ratio*,
|
|
118
|
+
not milliseconds, because these costs are request-latency dominated — a slower device scales
|
|
119
|
+
the seek path and the scan baseline together.
|
|
120
|
+
|
|
121
|
+
`seekPositioning` is the knob that moves plan choice today: it is what the engine's key-set
|
|
122
|
+
rewrite reads when it interpolates the key count at which a seek overtakes the plan it would
|
|
123
|
+
displace, so an expensive-seek backend stops firing that rewrite on small tables.
|
|
124
|
+
`pointRead` changes what an index arm *advertises* and how two indexes rank against each
|
|
125
|
+
other — but **the seek-vs-scan comparison above is deliberately still priced at the parity
|
|
126
|
+
1.0.** Because the match-count guess is a fixed fraction of the table, that comparison is
|
|
127
|
+
arm-disabling rather than arm-tuning: past `pointRead` 2.83 the range arm would be switched
|
|
128
|
+
off for every query on every table, off a guess no measurement can resolve. When real
|
|
129
|
+
per-column statistics replace the guess, the comparison will use the declared cost instead.
|
|
130
|
+
|
|
131
|
+
**`IN`-list index seeks ("multi-seek").** An `IN`-list on an indexed column
|
|
132
|
+
(`where v in (1, 2, 3)`, including parameter-bound lists) is served from the index as one
|
|
133
|
+
deduplicated, key-ordered point seek per distinct list value, instead of a full scan with
|
|
134
|
+
a residual filter. `NULL` list values match nothing and are skipped, duplicate values
|
|
135
|
+
yield their rows once, and a composite index serves the cross-product of per-column lists
|
|
136
|
+
(`a in (1,2) and b in (10,20)` is four seeks). Very large lists (over 1000 seek keys) and
|
|
137
|
+
lists on semantically-ordered column types (TIMESPAN, JSON) fall back to the scan path —
|
|
138
|
+
still correct, just not accelerated. When several secondary indexes can serve the same
|
|
139
|
+
predicate, the lowest-cost seek wins (equal costs keep the first-declared index), so
|
|
140
|
+
declaration order no longer decides whether a query does one seek or hundreds. The primary-key
|
|
141
|
+
arms still take precedence over any secondary index whenever they apply.
|
|
142
|
+
|
|
143
|
+
An `IN` covering the **whole primary key** takes the same path against the data store
|
|
144
|
+
itself: `where pk in (1, 2, 3)`, or `a in (…) and b in (…)` on a composite key, becomes one
|
|
145
|
+
deduplicated point read per distinct key tuple rather than a scan. Every gate above applies
|
|
146
|
+
(the 1000-key cap, the semantic-ordering decline, NULL-skip and dedup), and a key only
|
|
147
|
+
*partly* pinned — `a in (…)` alone on a `(a, b)` key — still scans with the predicate as a
|
|
148
|
+
residual. Unlike a secondary multi-seek, this one **advertises primary-key order**: the point
|
|
149
|
+
reads are emitted ascending by encoded data key, which is primary-key order (a `DESC` key
|
|
150
|
+
member's inversion is baked into its bytes), so `where pk in (…) order by pk` needs no `Sort`.
|
|
151
|
+
|
|
152
|
+
The same multi-seek path also serves an `IN` whose values only exist once the query runs
|
|
153
|
+
— `where v in (select … )`, which the engine may materialize into a set and hand down as
|
|
154
|
+
a seek (its `KeySetSemiJoin`). **Nothing in this module distinguishes the two.** The
|
|
155
|
+
engine stamps a `FilterInfo` byte-identical in shape to a literal list's, so every gate
|
|
156
|
+
above applies unchanged: the 1000-key cap, the semantic-ordering decline, the key-collation
|
|
157
|
+
guard, and the partial-index exclusion — on the primary-key arm as well as the secondary
|
|
158
|
+
one, so `where pk in (select …)` is served as a `_primary_` multi-seek too. At plan
|
|
159
|
+
time such a set describes itself only by a ceiling (`PredicateConstraint.runtimeSet.maxCount`),
|
|
160
|
+
which is what the cost and cap arithmetic judges. The engine also re-checks every row the
|
|
161
|
+
seek returns, so an over-fetching window costs only time.
|
|
162
|
+
|
|
163
|
+
**Catalog DDL is re-persisted on catalog-only mutations.** `ALTER … SET TAGS` (and the programmatic `setTableTags` / `setColumnTags` / `setConstraintTags` / `setViewTags` / `setMaterializedViewTags`), plus `CREATE`/`DROP VIEW` and `CREATE`/`DROP MATERIALIZED VIEW`, never reach `module.alterTable`/`module.destroy`. The module subscribes to the engine's schema-change events (`table_modified`, the `view_*` events, and the `materialized_view_*` events) and writes the matching `__catalog__` entry when its `generate*DDL` output changes — table / column / constraint / **index** / **view** / **materialized-view** tags, and view/MV lifecycle, all survive close → reopen. A table's bundle is its `CREATE TABLE` DDL, one `CREATE [UNIQUE] INDEX` line per secondary index, and one trailing `alter index … set tags (…)` line per *exposed implicit index* carrying user tags (an exposed implicit index is never materialized in the store's *engine-facing* schema — only in its internal enforcement schema, see the implicit-index note below — so its `UniqueConstraintSchema.exposedIndexTags` has no `CREATE INDEX` line to ride; the alter line re-applies silently on import). These async writes are serialized and drained by `closeAll()` (or the `whenCatalogPersisted()` barrier) before the provider closes. On reopen, `rehydrateCatalog` classifies entries by key prefix and imports them in phases — tables → views → materialized views, all through the engine's `importCatalog` (MVs re-materialize silently via the shared create core, dependency-ordered for MV-over-MV by fixpoint retry). See [`docs/view-persistence.md`](../../docs/view-persistence.md#view-and-materialized-view-persistence) for the full design.
|
|
164
|
+
|
|
165
|
+
**How a UNIQUE constraint is enforced.** For each row written, the store looks for a conflicting row through the cheapest sound route available:
|
|
166
|
+
|
|
167
|
+
1. **A linked row-time covering materialized view** — its backing table answers the uniqueness question.
|
|
168
|
+
2. **A physical secondary index realizing the constraint** — one bounded seek into the index store. Available for **every** non-derived `UNIQUE` (see the implicit-index note below), for a constraint that came from a `CREATE UNIQUE INDEX` (it names its own index), and for any *full* (non-partial) index whose columns match the constraint's. The index need not itself be UNIQUE.
|
|
169
|
+
3. **A full scan of the data store** — always correct, and O(rows) per row written.
|
|
170
|
+
|
|
171
|
+
Route 2 turns a bulk insert from O(n²) into roughly O(n log n). It is skipped for a constraint the index cannot answer soundly:
|
|
172
|
+
|
|
173
|
+
- A **partial** index cannot serve a constraint it does not derive from: it physically omits its out-of-scope rows, so a seek would miss a conflict among them.
|
|
174
|
+
- Index-column bytes are encoded under each index column's **own** key collation (the index's per-column `COLLATE`, else the table column's declared collation), and the constraint re-validates candidates under its enforcement collation `C` (the same resolution). For every index a designed path can hand the check the two are identical by construction, so the seek window is exactly the `C`-equal set. The check is kept because one undesigned path can still disagree — a user index whose name collides with the `_uc_*` a constraint would have minted lets a same-columns index with different collations be selected — and there a seek would under-fetch and silently accept a real duplicate, so the constraint falls back to the full scan. The **table key collation** `K` (the `collation` module option, default `NOCASE`) is not part of this decision; it is the fallback for an undecorated *primary-key* member only. "Can never hold text" (exempt — key bytes are type-native) is judged by physical representation, not by declared type name: an `ANY` or `JSON` column stores a string as a string, so neither is exempt — and because their `compare` ignores collation, they key hard-`BINARY`, so an `ANY`/`JSON` column carrying a declared `COLLATE` is exactly the shape that declines.
|
|
175
|
+
|
|
176
|
+
Whichever route runs, the conflicting row is re-validated identically: the row being written is excluded by primary key, each constrained column is compared under its enforcement collation (a `CREATE UNIQUE INDEX … (col COLLATE x)` enforces `x`, else the column's declared collation), and a partial constraint's predicate must hold on the candidate.
|
|
177
|
+
|
|
178
|
+
Because route 2 trusts the index store to hold an entry for every live row, `CREATE INDEX` populates the new index from the table's **effective** rows — committed rows merged with the open transaction's pending writes. A row inserted earlier in that transaction is therefore indexed, and participates in `CREATE UNIQUE INDEX`'s duplicate check, rather than being invisible to every later seek. Index entries are written outside the transaction coordinator, so a later `ROLLBACK` leaves entries for rows that no longer exist; every reader resolves an index entry to its live row and drops it when the row is gone or no longer matches, so a stale entry can never manufacture a result.
|
|
179
|
+
|
|
180
|
+
**Implicit per-constraint index (`_uc_*`).** Every non-derived `UNIQUE` — declared inline at `CREATE TABLE` or added by `ALTER TABLE … ADD CONSTRAINT` — is backed by a hidden secondary index named `<constraint name>` or, when unnamed, `_uc_<columns>` (the same convention the memory backend and the engine's `implicitIndexName` use). This is what makes route 2 reach a plain `UNIQUE`, so a bulk load no longer degrades to O(n²). The index is:
|
|
181
|
+
|
|
182
|
+
- **Kept out of the engine.** It lives only in the StoreTable's *enforcement* schema, never in the engine-registered schema, so the read-query planner does not see it (a plain `UNIQUE` gets no read-side plan from it — matching the memory backend) and it is never written to the catalog as a `CREATE INDEX`.
|
|
183
|
+
- **Derived on open.** Reconstructing a StoreTable re-materializes the schema entry from `uniqueConstraints`; the physical index store persists on disk under its deterministic name and is reopened lazily. (A store written *before* this feature has no `_uc_*` store on disk — backwards compatibility is waived project-wide; reopening such a database would need the index rebuilt.)
|
|
184
|
+
- **Reconciled across ALTER and CREATE / DROP INDEX.** `ADD CONSTRAINT` builds the physical store from the existing rows (after the existing-row duplicate check passes); `DROP CONSTRAINT` tears it down (so a later re-`ADD` cannot reopen stale entries); `RENAME CONSTRAINT`, and a column rename that changes an unnamed constraint's implicit name, move the store; a PK / collation / data-type `ALTER` re-encodes it via the same rebuild that re-encodes explicit indexes.
|
|
185
|
+
- **Relocated / reclaimed with the table.** `DROP TABLE` deletes the `_uc_*` store alongside the data + explicit-index stores; `RENAME TABLE` relocates it under the new name. Both resolve the physical store list from the *enforcement* schema (not the engine-facing `.indexes`, which omits `_uc_*`), so the implicit store is never stranded on drop nor left behind on rename (which would leave the renamed table seeking an empty store and accepting a duplicate).
|
|
186
|
+
- **Skipped when an explicit index already realizes the constraint.** A `UNIQUE` whose columns are covered — in the same order — by a full (non-partial) `CREATE INDEX` whose per-column collations match the declared column collations gets no `_uc_*` at all: that index enforces it, so a write maintains one structure instead of two. Reuse is refused for a partial index, a partial `UNIQUE`, an index over different or reordered columns, and a collation-mismatched index; those still build the `_uc_*` and both structures coexist. The decision is recomputed whenever the schema changes and is never persisted, so `CREATE INDEX` over a constrained column tears the redundant `_uc_*` store down and `DROP INDEX` rebuilds it from the live rows.
|
|
187
|
+
- **DDL-committing when it tears a store down.** Deleting a `_uc_*` store (or an explicit index store) discards a KVStore handle the transaction coordinator may hold buffered writes against, so those DDL statements flush the module's pending transaction first — the same `ddlTransactionality: 'auto-commit'` posture the row-rewriting `ALTER` arms take. A `CREATE INDEX` that retires no `_uc_*` does not flush.
|
|
188
|
+
|
|
189
|
+
## Installation
|
|
190
|
+
|
|
191
|
+
```bash
|
|
192
|
+
npm install @quereus/store
|
|
193
|
+
```
|
|
194
|
+
|
|
195
|
+
For platform-specific implementations:
|
|
196
|
+
```bash
|
|
197
|
+
# Node.js
|
|
198
|
+
npm install @quereus/plugin-leveldb
|
|
199
|
+
|
|
200
|
+
# Browser
|
|
201
|
+
npm install @quereus/plugin-indexeddb
|
|
202
|
+
```
|
|
203
|
+
|
|
204
|
+
## Usage
|
|
205
|
+
|
|
206
|
+
### With a Provider
|
|
207
|
+
|
|
208
|
+
```typescript
|
|
209
|
+
import { Database } from '@quereus/quereus';
|
|
210
|
+
import { StoreModule } from '@quereus/store';
|
|
211
|
+
import { createLevelDBProvider } from '@quereus/plugin-leveldb';
|
|
212
|
+
// OR: import { createIndexedDBProvider } from '@quereus/plugin-indexeddb';
|
|
213
|
+
|
|
214
|
+
const db = new Database();
|
|
215
|
+
|
|
216
|
+
// Create provider for your platform
|
|
217
|
+
const provider = createLevelDBProvider({ basePath: './data' });
|
|
218
|
+
|
|
219
|
+
// Create the generic store module with your provider
|
|
220
|
+
const storeModule = new StoreModule(provider);
|
|
221
|
+
db.registerModule('store', storeModule);
|
|
222
|
+
|
|
223
|
+
// Use it in SQL
|
|
224
|
+
await db.exec(`
|
|
225
|
+
create table users (id integer primary key, name text)
|
|
226
|
+
using store
|
|
227
|
+
`);
|
|
228
|
+
```
|
|
229
|
+
|
|
230
|
+
### Custom Storage Backend
|
|
231
|
+
|
|
232
|
+
Implement `KVStore` and `KVStoreProvider` to create custom storage backends:
|
|
233
|
+
|
|
234
|
+
```typescript
|
|
235
|
+
import { defaultGetMany, type KVStore, type KVStoreProvider } from '@quereus/store';
|
|
236
|
+
|
|
237
|
+
class MyCustomStore implements KVStore {
|
|
238
|
+
async get(key: Uint8Array) { /* ... */ }
|
|
239
|
+
// Positional batch point-read. Override it when the backend has a native multi-get or a
|
|
240
|
+
// way to issue every request on one round trip; otherwise `defaultGetMany(this, keys)`.
|
|
241
|
+
getMany(keys: readonly Uint8Array[]) { return defaultGetMany(this, keys); }
|
|
242
|
+
async put(key: Uint8Array, value: Uint8Array) { /* ... */ }
|
|
243
|
+
async delete(key: Uint8Array) { /* ... */ }
|
|
244
|
+
async has(key: Uint8Array) { /* ... */ }
|
|
245
|
+
iterate(options?: IterateOptions) { /* ... */ }
|
|
246
|
+
batch() { /* ... */ }
|
|
247
|
+
async close() { /* ... */ }
|
|
248
|
+
async approximateCount(options?: IterateOptions) { /* ... */ }
|
|
249
|
+
}
|
|
250
|
+
|
|
251
|
+
class MyCustomProvider implements KVStoreProvider {
|
|
252
|
+
async getStore(schemaName: string, tableName: string) {
|
|
253
|
+
return new MyCustomStore(/* ... */);
|
|
254
|
+
}
|
|
255
|
+
async getIndexStore(schemaName: string, tableName: string, indexName: string) {
|
|
256
|
+
return new MyCustomStore(/* ... */);
|
|
257
|
+
}
|
|
258
|
+
async getStatsStore(schemaName: string, tableName: string) {
|
|
259
|
+
return new MyCustomStore(/* ... */);
|
|
260
|
+
}
|
|
261
|
+
async getCatalogStore() { /* ... */ }
|
|
262
|
+
async closeStore(schemaName: string, tableName: string) { /* ... */ }
|
|
263
|
+
async closeIndexStore(schemaName: string, tableName: string, indexName: string) { /* ... */ }
|
|
264
|
+
async closeAll() { /* ... */ }
|
|
265
|
+
}
|
|
266
|
+
|
|
267
|
+
// Use it with StoreModule
|
|
268
|
+
const provider = new MyCustomProvider();
|
|
269
|
+
const module = new StoreModule(provider);
|
|
270
|
+
db.registerModule('store', module);
|
|
271
|
+
```
|
|
272
|
+
|
|
273
|
+
**Validate a new backend against the shared conformance suite.** `@quereus/store/testing`
|
|
274
|
+
exports `runKVStoreConformance(name, makeBackend)` — one parameterized battery of
|
|
275
|
+
behavioral tests written against the `KVStore` contract (point ops, ordering, range
|
|
276
|
+
iteration, streaming across page boundaries, batch semantics, optional persistence,
|
|
277
|
+
cross-backend encoded-key ordering, bounded iteration, and the batch point-read). Wire a
|
|
278
|
+
tiny lifecycle adapter and run it under Mocha so any drift from the contract fails a test:
|
|
279
|
+
|
|
280
|
+
```typescript
|
|
281
|
+
import { runKVStoreConformance } from '@quereus/store/testing';
|
|
282
|
+
|
|
283
|
+
runKVStoreConformance('MyCustomStore', () => ({
|
|
284
|
+
open: async () => new MyCustomStore(/* ... */),
|
|
285
|
+
// Omit `reopen` for a non-persistent backend; supply it (reopen the SAME keyspace
|
|
286
|
+
// without wiping) to also exercise the persistence tier.
|
|
287
|
+
teardown: async () => { /* close handles, remove backing storage */ },
|
|
288
|
+
// Optional. The battery holds only a KVStore handle and cannot see what the backend
|
|
289
|
+
// reads underneath, so supply a counter over whatever the store reads FROM plus the
|
|
290
|
+
// backend's batch size, and the bounded-iteration tier also measures reads. Omit it
|
|
291
|
+
// and that tier still checks that abandoning an iteration releases resources.
|
|
292
|
+
readMeter: { entriesRead: () => myBackingReads, maxReadAhead: 1 },
|
|
293
|
+
// Also optional, and a DIFFERENT counter: `readMeter` counts entries an iteration
|
|
294
|
+
// yields, so it cannot see a point read. Supply this — trips to backing storage, one
|
|
295
|
+
// per `get`, one per native multi-get — and the batch point-read tier additionally
|
|
296
|
+
// asserts that `getMany` over K keys costs exactly ONE trip. Omit it on a backend whose
|
|
297
|
+
// `getMany` is the shared fallback; that tier's correctness cases still run.
|
|
298
|
+
pointReadMeter: { roundTrips: () => myBackingRoundTrips },
|
|
299
|
+
}));
|
|
300
|
+
```
|
|
301
|
+
|
|
302
|
+
See `test/kv-conformance.spec.ts` (in-memory), and the `test/conformance.spec.ts` of each
|
|
303
|
+
backend plugin — LevelDB, IndexedDB, React Native LevelDB, NativeScript SQLite — for worked
|
|
304
|
+
adapters. The two mobile plugins run against an in-process stand-in for their native module
|
|
305
|
+
(a mock LevelDB, and better-sqlite3 behind the NativeScript SQLite interface), which is what
|
|
306
|
+
makes them testable off-device; the NativeScript one uses a temp file database so it also
|
|
307
|
+
drives the persistence tier.
|
|
308
|
+
|
|
309
|
+
**If your provider implements `beginAtomicBatch`, also run the provider suite.** The same
|
|
310
|
+
entry point exports `runKVProviderConformance(name, makeProviderBackend)` — the battery for
|
|
311
|
+
the provider-level contract that ops queued across SEVERAL of the provider's stores commit
|
|
312
|
+
in one durable, all-or-nothing physical write (multi-store commit landing only in each op's
|
|
313
|
+
own store, nothing visible until `write()`, mixed put + delete, a delete of a missing key,
|
|
314
|
+
same-key last-op-wins, `clear()` discarding, the empty `write()` no-op, and `MISUSE` on a
|
|
315
|
+
store handle from a different provider):
|
|
316
|
+
|
|
317
|
+
```typescript
|
|
318
|
+
import { runKVProviderConformance } from '@quereus/store/testing';
|
|
319
|
+
|
|
320
|
+
runKVProviderConformance('MyCustomProvider atomic batch', () => ({
|
|
321
|
+
open: async () => new MyCustomProvider(/* fresh, empty keyspace */),
|
|
322
|
+
// A SECOND provider over a DIFFERENT keyspace — source of the foreign handle that
|
|
323
|
+
// must be rejected with MISUSE.
|
|
324
|
+
openForeign: async () => new MyCustomProvider(/* another keyspace */),
|
|
325
|
+
teardown: async () => { /* close both, remove backing storage */ },
|
|
326
|
+
}));
|
|
327
|
+
```
|
|
328
|
+
|
|
329
|
+
The LevelDB / IndexedDB plugins' `test/atomic-batch.spec.ts` run it, keeping only their
|
|
330
|
+
backend-specific cases (LevelDB's "no batch before the shared root is open", IndexedDB's
|
|
331
|
+
post-write read-cache invalidation) alongside.
|
|
332
|
+
|
|
333
|
+
**Every provider runs the store-name distinctness battery.** `runStoreNameDistinctness(name,
|
|
334
|
+
makeNamingBackend)` holds a provider to the one naming rule the engine depends on: two
|
|
335
|
+
logical stores it considers distinct never share one physical store. A provider must build
|
|
336
|
+
its names with `buildDataStoreName` / `buildIndexStoreName` and escape them into its own
|
|
337
|
+
namespace injectively; a name the namespace cannot represent must be *rejected*, never folded
|
|
338
|
+
onto another. The battery asserts this behaviorally — it opens a corpus of adversarial
|
|
339
|
+
`(schema, table)` and `(schema, table, index)` names, writes a distinct marker into each under
|
|
340
|
+
one shared key, and checks each still reads back its own — plus that two spellings of one
|
|
341
|
+
case-insensitive identifier land on the SAME store, and that a user table named `__stats__` or
|
|
342
|
+
`__catalog__` stays off the reserved stores:
|
|
343
|
+
|
|
344
|
+
```typescript
|
|
345
|
+
import { runStoreNameDistinctness } from '@quereus/store/testing';
|
|
346
|
+
|
|
347
|
+
runStoreNameDistinctness('MyCustomProvider store names', () => ({
|
|
348
|
+
open: async () => new MyCustomProvider(/* fresh, empty keyspace */),
|
|
349
|
+
teardown: async () => { /* close the provider, remove backing storage */ },
|
|
350
|
+
}));
|
|
351
|
+
```
|
|
352
|
+
|
|
353
|
+
All four backend plugins register it in their `test/conformance.spec.ts`. LevelDB and
|
|
354
|
+
IndexedDB are its reference implementations, so a failure there means the battery is wrong
|
|
355
|
+
rather than the plugin. `assertStoreNamesDistinct` — its core assertion — is exported
|
|
356
|
+
standalone; `test/store-name-distinctness.spec.ts` drives it against provider doubles built
|
|
357
|
+
to fold, truncate, and over-reject, so the guard itself is watched failing.
|
|
358
|
+
|
|
359
|
+
**Every provider runs the store-reclaim battery too.** `runStoreReclaimConformance(name,
|
|
360
|
+
makeReclaimBackend)` holds a provider to the other rule the engine depends on: when it is
|
|
361
|
+
told to delete a table's stores, those stores must be *erased*, not merely closed. Store
|
|
362
|
+
handles are re-opened lazily by name, so a provider that only drops its cached handle leaves
|
|
363
|
+
the bytes in place and the next table created under that name comes back holding the dropped
|
|
364
|
+
table's rows — a defect two of the four shipped providers had. The battery deletes and
|
|
365
|
+
re-opens by the same names, checking each store reads back empty through both a point `get`
|
|
366
|
+
and a scan, that a store re-created under a dropped name holds only its own new rows, that
|
|
367
|
+
`deleteIndexStore` touches only that one index, that sibling tables (including one literally
|
|
368
|
+
named `{table}_idx_<x>`), an index store the caller did not name, and the reserved
|
|
369
|
+
`__stats__` / `__catalog__` stores all survive, that deleting a store that was never created
|
|
370
|
+
is a no-op rather than a throw, that a store whose handle the caller already closed is still
|
|
371
|
+
erased (the order every `drop index` takes), and that a store holding more rows than a
|
|
372
|
+
provider erases in one pass comes back empty to the last key:
|
|
373
|
+
|
|
374
|
+
```typescript
|
|
375
|
+
import { runStoreReclaimConformance } from '@quereus/store/testing';
|
|
376
|
+
|
|
377
|
+
runStoreReclaimConformance('MyCustomProvider store reclaim', () => ({
|
|
378
|
+
open: async () => new MyCustomProvider(/* fresh, empty keyspace */),
|
|
379
|
+
teardown: async () => { /* close the provider, remove backing storage */ },
|
|
380
|
+
}));
|
|
381
|
+
```
|
|
382
|
+
|
|
383
|
+
It takes the same `KVProviderLifecycle` adapter as the naming battery, so a plugin's spec
|
|
384
|
+
builds one closure and registers both. Both delete hooks are optional on `KVStoreProvider`;
|
|
385
|
+
a provider that omits one is skipped for those cases, and the skip is printed rather than
|
|
386
|
+
silent. Reclaiming bytes *on disk* is deliberately not asserted — a provider may leave an
|
|
387
|
+
emptied database directory or an unshrunk file behind and still be correct.
|
|
388
|
+
|
|
389
|
+
**Counting test doubles, for asserting on storage traffic.** `@quereus/store/testing` also
|
|
390
|
+
exports the doubles this package's own specs use to prove the engine reads and writes what
|
|
391
|
+
it claims to: `DelegatingKVStore` (forwards every method to an inner store — the base to
|
|
392
|
+
subclass when modelling ONE method's behavior), `CountingKVStore` (tallies reads —
|
|
393
|
+
`iterate()` entries, `get()` calls, `getMany()` round trips plus the keys they carried —
|
|
394
|
+
and writes — point `put`/`delete` issued outside any batch, `WriteBatch.write()` round
|
|
395
|
+
trips, and the operations those carried; `reset()` zeroes all eight), and
|
|
396
|
+
`createCountingProvider(map, scope?)`, an all-in-memory `KVStoreProvider` whose stores are
|
|
397
|
+
counting stores, recorded into a map the caller reads counters back out of:
|
|
398
|
+
|
|
399
|
+
```typescript
|
|
400
|
+
import { CountingKVStore, createCountingProvider } from '@quereus/store/testing';
|
|
401
|
+
|
|
402
|
+
const stores = new Map<string, CountingKVStore>();
|
|
403
|
+
// 'data' (default) counts only each table's data store; 'all' counts index, stats, and
|
|
404
|
+
// catalog stores too. Keys are BUILT store names — 'main.t', 'main.t_idx_ix_v', …
|
|
405
|
+
const provider = createCountingProvider(stores, 'all');
|
|
406
|
+
db.registerModule('store', new StoreModule(provider));
|
|
407
|
+
|
|
408
|
+
await db.exec(`select v from t where id = 7`);
|
|
409
|
+
expect(stores.get('main.t')!.iterateEntryCount).to.be.at.most(1); // sought, not scanned
|
|
410
|
+
```
|
|
411
|
+
|
|
412
|
+
`CountingKVStore.getMany` deliberately answers through its own counted `get`, so a batch of
|
|
413
|
+
N keys advances `getCount` by N and `getManyCalls` by one. That measures batching AT THE
|
|
414
|
+
CALL BOUNDARY — did the caller issue one `getMany` or N `get`s — which is what the engine's
|
|
415
|
+
specs are proving; it does NOT measure a real backend's own native multi-get, and layering
|
|
416
|
+
it over one suppresses that batch. See the class doc comment before reusing it for that.
|
|
417
|
+
|
|
418
|
+
The write counters have the same call-boundary meaning, and one rule worth knowing:
|
|
419
|
+
operations belong to the `write()` that carried them, not to the moment they were queued —
|
|
420
|
+
a batch that is `clear()`ed or abandoned without committing contributes nothing. Only the
|
|
421
|
+
per-store `WriteBatch` path is counted; a provider with a shared atomic commit domain (this
|
|
422
|
+
one has none) would commit through `AtomicBatch`, which this double does not wrap.
|
|
423
|
+
|
|
424
|
+
**A plain in-memory provider, for specs that need a real backend with no counting attached.**
|
|
425
|
+
`createInMemoryProvider(options?)` builds an all-`InMemoryKVStore` `KVStoreProvider` — the
|
|
426
|
+
one shared implementation to reach for instead of pasting yet another private copy. (The
|
|
427
|
+
twenty-odd copies already living in this package's specs have not been swept onto it yet;
|
|
428
|
+
that sweep is its own backlog item.) Unlike `createCountingProvider`, it implements
|
|
429
|
+
`deleteIndexStore` and `deleteTableStores`, and both actually ERASE the named store (a live
|
|
430
|
+
handle still open on it sees the erase too), so a spec exercising `drop index` / `drop table`
|
|
431
|
+
reclaim can use it directly:
|
|
432
|
+
|
|
433
|
+
```typescript
|
|
434
|
+
import { createInMemoryProvider } from '@quereus/store/testing';
|
|
435
|
+
|
|
436
|
+
const provider = createInMemoryProvider(); // no costProfile property at all
|
|
437
|
+
// const provider = createInMemoryProvider({ costProfile: { pointRead: 2 } });
|
|
438
|
+
db.registerModule('store', new StoreModule(provider));
|
|
439
|
+
```
|
|
440
|
+
|
|
441
|
+
`renameTableStores` is not implemented; `StoreModule` falls back to its generic copy for a
|
|
442
|
+
provider without it. Neither is `beginAtomicBatch` — separate `InMemoryKVStore`s share no
|
|
443
|
+
commit domain, so there is nothing to make atomic. That means `StoreModuleBase` takes its
|
|
444
|
+
per-store commit path here, not the single-physical-write path a LevelDB-family provider
|
|
445
|
+
gets; a benchmark comparing this provider against LevelDB is comparing two commit paths,
|
|
446
|
+
not one path over two backends.
|
|
447
|
+
|
|
448
|
+
## KVStore Interface
|
|
449
|
+
|
|
450
|
+
The `KVStore` interface is the foundation for all storage backends:
|
|
451
|
+
|
|
452
|
+
```typescript
|
|
453
|
+
interface KVStore {
|
|
454
|
+
get(key: Uint8Array): Promise<Uint8Array | undefined>;
|
|
455
|
+
// Positional: result[i] belongs to keys[i], `undefined` for an absent key, the array is
|
|
456
|
+
// always keys.length long, and a repeated key is answered at every position it occupies
|
|
457
|
+
// in its own buffer. No internal paging — the CALLER sizes the batch.
|
|
458
|
+
getMany(keys: readonly Uint8Array[]): Promise<(Uint8Array | undefined)[]>;
|
|
459
|
+
put(key: Uint8Array, value: Uint8Array): Promise<void>;
|
|
460
|
+
delete(key: Uint8Array): Promise<void>;
|
|
461
|
+
has(key: Uint8Array): Promise<boolean>;
|
|
462
|
+
iterate(options?: IterateOptions): AsyncIterable<KVEntry>;
|
|
463
|
+
batch(): WriteBatch;
|
|
464
|
+
close(): Promise<void>;
|
|
465
|
+
approximateCount(options?: IterateOptions): Promise<number>;
|
|
466
|
+
}
|
|
467
|
+
|
|
468
|
+
interface KVStoreProvider {
|
|
469
|
+
// Get data store for a table
|
|
470
|
+
getStore(schemaName: string, tableName: string): Promise<KVStore>;
|
|
471
|
+
|
|
472
|
+
// Get index store for a secondary index
|
|
473
|
+
getIndexStore(schemaName: string, tableName: string, indexName: string): Promise<KVStore>;
|
|
474
|
+
|
|
475
|
+
// Get stats store for table statistics
|
|
476
|
+
getStatsStore(schemaName: string, tableName: string): Promise<KVStore>;
|
|
477
|
+
|
|
478
|
+
// Get catalog store for DDL metadata
|
|
479
|
+
getCatalogStore(): Promise<KVStore>;
|
|
480
|
+
|
|
481
|
+
// Close specific stores
|
|
482
|
+
closeStore(schemaName: string, tableName: string): Promise<void>;
|
|
483
|
+
closeIndexStore(schemaName: string, tableName: string, indexName: string): Promise<void>;
|
|
484
|
+
closeAll(): Promise<void>;
|
|
485
|
+
|
|
486
|
+
// Optional: Delete stores. `indexNames` is the table's exact secondary-index
|
|
487
|
+
// names (from the schema) — build index store names from it via
|
|
488
|
+
// buildIndexStoreName instead of prefix-scanning `{table}_idx_`, which would
|
|
489
|
+
// also match a sibling table literally named `{table}_idx_<x>`.
|
|
490
|
+
deleteIndexStore?(schemaName: string, tableName: string, indexName: string): Promise<void>;
|
|
491
|
+
deleteTableStores?(schemaName: string, tableName: string, indexNames: readonly string[]): Promise<void>;
|
|
492
|
+
|
|
493
|
+
// Optional: Relocate a table's data + index stores for ALTER TABLE ... RENAME TO
|
|
494
|
+
// (`indexNames` carries the same authoritative, exact index list). Omit it and the
|
|
495
|
+
// module copies every entry through getStore/getIndexStore instead — correct on any
|
|
496
|
+
// provider, but O(table size); implement this for a native move.
|
|
497
|
+
renameTableStores?(schemaName: string, oldName: string, newName: string, indexNames: readonly string[]): Promise<void>;
|
|
498
|
+
}
|
|
499
|
+
```
|
|
500
|
+
|
|
501
|
+
## Module Capabilities
|
|
502
|
+
|
|
503
|
+
The `StoreModule` reports its capabilities via `getCapabilities()`:
|
|
504
|
+
|
|
505
|
+
```typescript
|
|
506
|
+
const storeModule = new StoreModule(provider);
|
|
507
|
+
const caps = storeModule.getCapabilities();
|
|
508
|
+
|
|
509
|
+
// {
|
|
510
|
+
// isolation: false, // Store module does NOT provide transaction isolation
|
|
511
|
+
// savepoints: true, // Coordinator-buffered ops support savepoints within a transaction
|
|
512
|
+
// persistent: true, // Data persists across restarts
|
|
513
|
+
// secondaryIndexes: true,// Supports secondary indexes
|
|
514
|
+
// rangeScans: true // Supports range scans
|
|
515
|
+
// }
|
|
516
|
+
```
|
|
517
|
+
|
|
518
|
+
**Important:** The base `StoreModule` does not provide transaction isolation:
|
|
519
|
+
- No snapshot isolation: between connections, reads see only committed data, and concurrent readers may observe partial writes
|
|
520
|
+
- Within a transaction, reads through the table's shared coordinator DO see that transaction's own pending writes (read-your-own-writes). This extends to DML's own internal reads: the insert PK-conflict probe, the update/delete old-image reads, and the update PK-change conflict probe all read through the pending merge, so an INSERT/UPDATE/DELETE against a row written earlier in the same transaction sees that pending row — it raises a UNIQUE conflict (or evicts under `OR REPLACE`), cleans up secondary-index entries, tracks the correct row-count delta, and emits events carrying the pending `oldRow`
|
|
521
|
+
- Row-validating DDL reads the same effective stream: `create index` / `create unique index` populate from it (see above), and `alter table … add constraint … unique` plus the `set collate` re-validation of a covering non-PK UNIQUE scan it too — so a duplicate inserted earlier in the still-open transaction is rejected rather than surviving to commit
|
|
522
|
+
- A **schema-only** DDL statement that fails is a clean no-op: `create index` / `drop index` and every `alter table` path that rewrites no row data (`add` / `drop` / `rename constraint`, `rename column`, and the `alter column` shapes with no value rewrite, key re-encode or index rebuild — `drop not null`, `set default`, a `set not null` over rows holding no NULL, a `set collate` on an unindexed non-PK column) put the connected table's cached schema back when the catalog write throws. Without that undo the module's in-memory schema stayed one statement ahead of both the engine and the catalog for the rest of the session, and the *next* successful catalog write — from any later statement — made the refused change durable
|
|
523
|
+
- Not so for the **row-rewriting** paths (`add` / `drop column`, `alter primary key`, and the `alter column` shapes that convert values, re-key, or rebuild indexes): by the time they persist they have already re-encoded the store outside any transaction, so restoring the old schema would misread the new bytes. An IO error between the rewrite and the catalog write leaves the two diverged until the statement is re-run — an accepted tradeoff whose real fix is one durable marker covering the whole physical rewrite
|
|
524
|
+
- Savepoints (create / release / rollback-to) work within a transaction via the coordinator's buffered op log
|
|
525
|
+
- Caveat: a DDL-commit operation (`replaceContents` / `renameTable`, e.g. `refresh materialized view` or `alter table … rename`) commits the coordinator mid-transaction, clearing the savepoint stack. A later `rollback to` / `release` targeting a now-vanished savepoint degrades to a no-op (warn-and-return) rather than throwing; the committed DDL and everything before it stays committed
|
|
526
|
+
|
|
527
|
+
## Atomic multi-store commit (module-wide, cross-table)
|
|
528
|
+
|
|
529
|
+
A single `TransactionCoordinator` is shared by **every table of one storage
|
|
530
|
+
module** — it is the unit of cross-table atomicity. Every buffered op is
|
|
531
|
+
addressed by its explicit target `KVStore` handle (data ops, secondary-index
|
|
532
|
+
ops, and backing-host writes alike), so a transaction touching tables A and B
|
|
533
|
+
accumulates all of their stores' ops in one coordinator. Because the engine
|
|
534
|
+
commits virtual-table connections **sequentially** and the coordinator's
|
|
535
|
+
`commit()`/`rollback()` are **idempotent**, the first connection to commit
|
|
536
|
+
flushes **every** touched store of **every** table the transaction wrote; the
|
|
537
|
+
remaining connections no-op.
|
|
538
|
+
|
|
539
|
+
`TransactionCoordinator.commit()` thus writes each table's data store and each of
|
|
540
|
+
its secondary-index stores. By default it writes **one `KVStore.batch()` per
|
|
541
|
+
store, sequentially** — a crash between those batches can leave tables/indexes
|
|
542
|
+
divergent on disk, with no automatic healing (no worse than the prior per-table
|
|
543
|
+
commits, which were already non-atomic across tables).
|
|
544
|
+
|
|
545
|
+
A provider whose stores share a single durable commit domain can close that
|
|
546
|
+
window by implementing the optional `KVStoreProvider.beginAtomicBatch()`:
|
|
547
|
+
|
|
548
|
+
```typescript
|
|
549
|
+
interface AtomicBatch {
|
|
550
|
+
put(store: KVStore, key: Uint8Array, value: Uint8Array): void;
|
|
551
|
+
delete(store: KVStore, key: Uint8Array): void;
|
|
552
|
+
write(): Promise<void>; // one durable, all-or-nothing physical commit
|
|
553
|
+
clear(): void;
|
|
554
|
+
}
|
|
555
|
+
|
|
556
|
+
interface KVStoreProvider {
|
|
557
|
+
// ...
|
|
558
|
+
// Open a batch spanning this provider's stores, or undefined when the provider
|
|
559
|
+
// has no shared commit domain (the coordinator then falls back to per-store batch()).
|
|
560
|
+
beginAtomicBatch?(): AtomicBatch | undefined;
|
|
561
|
+
}
|
|
562
|
+
```
|
|
563
|
+
|
|
564
|
+
The batch addresses stores by **`KVStore` handle**, so it composes with the
|
|
565
|
+
coordinator's existing per-store bucketing without a name lookup. When present,
|
|
566
|
+
`commit()` queues every pending op — every store of **every table** in the
|
|
567
|
+
transaction — into one `AtomicBatch` and issues a single `write()`; all of those
|
|
568
|
+
tables commit or roll back together. When absent — or when the factory returns
|
|
569
|
+
`undefined` — behavior is byte-identical to the per-store loop, so providers
|
|
570
|
+
without a shared domain are unaffected.
|
|
571
|
+
|
|
572
|
+
The capability surface spans **multiple stores of one provider** (every store of
|
|
573
|
+
every table the module owns), giving full module-wide cross-table atomicity with
|
|
574
|
+
no interface change. The
|
|
575
|
+
[`@quereus/plugin-indexeddb`](../quereus-plugin-indexeddb) provider implements it
|
|
576
|
+
over its single IndexedDB database (multiple object stores, one
|
|
577
|
+
`db.transaction(...,'readwrite')`), invalidating each touched store's read cache
|
|
578
|
+
after a successful write so read-your-own-writes survives the cache.
|
|
579
|
+
|
|
580
|
+
## Materialized-View Backing Host
|
|
581
|
+
|
|
582
|
+
The store module implements the engine's backing-host capability
|
|
583
|
+
(`StoreBackingHost`), so `create materialized view mv using store as <body>`
|
|
584
|
+
places the MV's backing table in persistent storage. Maintenance writes ride
|
|
585
|
+
the module's shared `TransactionCoordinator`'s pending state (committing/rolling
|
|
586
|
+
back in lockstep with the source write — and, since the coordinator is
|
|
587
|
+
module-wide, in the same all-or-nothing batch as a write to a same-module
|
|
588
|
+
source), mid-transaction reads of the MV see pending
|
|
589
|
+
maintenance through the read-your-own-writes merge, and the backing's text
|
|
590
|
+
primary-key columns are keyed under the store's `collation` arg (default
|
|
591
|
+
`NOCASE` — pass `using store(collation = 'BINARY')` for byte-exact keys). The
|
|
592
|
+
isolation wrapper forwards the capability automatically. See
|
|
593
|
+
[`docs/mv-backing-host.md` § The store host](../../docs/mv-backing-host.md#the-store-host-using-store).
|
|
594
|
+
|
|
595
|
+
## External Row-Write Entry Point
|
|
596
|
+
|
|
597
|
+
`StoreTable.applyExternalRowChanges(ops)` applies trusted, externally-originated
|
|
598
|
+
row writes (e.g. inbound replication) directly to a **source** table's committed
|
|
599
|
+
storage — table-owned data-key put/delete, **secondary-index maintenance**, and
|
|
600
|
+
stats tracking — and returns the effective `BackingRowChange[]` (the shape
|
|
601
|
+
`Database.ingestExternalRowChanges` consumes). It is the index-maintaining
|
|
602
|
+
sibling of the backing host (whose MV backing tables carry no indexes): a caller
|
|
603
|
+
writing the data `KVStore` directly would silently skip index and stats upkeep.
|
|
604
|
+
|
|
605
|
+
Resolve the table with `StoreModule.getTableForExternalWrite(db, schema, table)`
|
|
606
|
+
(same ownership/wrapper resolution as `getBackingHost`), read a row's current
|
|
607
|
+
image with `StoreTable.readRowByPk(pk)`, then apply one `ExternalRowOp` per row:
|
|
608
|
+
|
|
609
|
+
```typescript
|
|
610
|
+
const table = storeModule.getTableForExternalWrite(db, 'main', 'users');
|
|
611
|
+
if (table) {
|
|
612
|
+
const changes = await table.applyExternalRowChanges([
|
|
613
|
+
{ op: 'upsert', row: [1, 'alice'] }, // full row, schema column order
|
|
614
|
+
{ op: 'delete', pk: [2] }, // PK values, PK-definition order
|
|
615
|
+
]);
|
|
616
|
+
}
|
|
617
|
+
```
|
|
618
|
+
|
|
619
|
+
Deliberately emits **no** module data events (the caller owns emission and the
|
|
620
|
+
`remote` flag), opens **no** coordinator transaction (writes commit at once,
|
|
621
|
+
last-writer-wins against any pending local transaction), and runs **no**
|
|
622
|
+
constraint validation (the origin is trusted). No-ops are suppressed: a delete
|
|
623
|
+
of an absent key and a value-identical upsert (byte-faithful) write nothing and
|
|
624
|
+
report nothing.
|
|
625
|
+
|
|
626
|
+
## Transaction Isolation
|
|
627
|
+
|
|
628
|
+
To add full ACID transaction semantics with snapshot isolation, wrap the store module with the `IsolationModule`:
|
|
629
|
+
|
|
630
|
+
```typescript
|
|
631
|
+
import { Database, MemoryTableModule } from '@quereus/quereus';
|
|
632
|
+
import { IsolationModule } from '@quereus/isolation';
|
|
633
|
+
import { StoreModule, createIsolatedStoreModule } from '@quereus/store';
|
|
634
|
+
import { createLevelDBProvider } from '@quereus/plugin-leveldb';
|
|
635
|
+
|
|
636
|
+
const db = new Database();
|
|
637
|
+
const provider = createLevelDBProvider({ basePath: './data' });
|
|
638
|
+
|
|
639
|
+
// Option 1: Use the convenience function
|
|
640
|
+
const isolatedModule = createIsolatedStoreModule({ provider });
|
|
641
|
+
db.registerModule('store', isolatedModule);
|
|
642
|
+
|
|
643
|
+
// Option 2: Manual wrapping for more control
|
|
644
|
+
const storeModule = new StoreModule(provider);
|
|
645
|
+
const isolatedModule = new IsolationModule({
|
|
646
|
+
underlying: storeModule,
|
|
647
|
+
overlay: new MemoryTableModule(),
|
|
648
|
+
});
|
|
649
|
+
db.registerModule('store', isolatedModule);
|
|
650
|
+
|
|
651
|
+
// Now transactions have full isolation
|
|
652
|
+
await db.exec('BEGIN');
|
|
653
|
+
await db.exec(`INSERT INTO users VALUES (1, 'Alice')`);
|
|
654
|
+
|
|
655
|
+
// Read-your-own-writes: sees uncommitted insert
|
|
656
|
+
const user = await db.get('SELECT * FROM users WHERE id = 1');
|
|
657
|
+
console.log(user.name); // 'Alice'
|
|
658
|
+
|
|
659
|
+
await db.exec('COMMIT'); // Or ROLLBACK to discard
|
|
660
|
+
```
|
|
661
|
+
|
|
662
|
+
The isolation layer provides:
|
|
663
|
+
- **Read-your-own-writes** within transactions
|
|
664
|
+
- **Snapshot isolation** for consistent reads
|
|
665
|
+
- **Savepoint support** via the overlay module
|
|
666
|
+
|
|
667
|
+
### Checking for Isolation Support
|
|
668
|
+
|
|
669
|
+
```typescript
|
|
670
|
+
import { hasIsolation } from '@quereus/store';
|
|
671
|
+
|
|
672
|
+
const storeModule = new StoreModule(provider);
|
|
673
|
+
console.log(hasIsolation(storeModule)); // false
|
|
674
|
+
|
|
675
|
+
const isolatedModule = createIsolatedStoreModule({ provider });
|
|
676
|
+
console.log(hasIsolation(isolatedModule)); // true
|
|
677
|
+
```
|
|
678
|
+
|
|
679
|
+
## API
|
|
680
|
+
|
|
681
|
+
### Core Exports
|
|
682
|
+
|
|
683
|
+
| Export | Description |
|
|
684
|
+
|--------|-------------|
|
|
685
|
+
| `KVStore` | Key-value store interface (type) |
|
|
686
|
+
| `KVStoreProvider` | Store factory interface (type) |
|
|
687
|
+
| `WriteBatch` | Batch write interface (type) |
|
|
688
|
+
| `AtomicBatch` | Cross-store all-or-nothing batch from `KVStoreProvider.beginAtomicBatch` (type) |
|
|
689
|
+
| `IterateOptions` | Iteration options (type) |
|
|
690
|
+
| `StoreModule` | Generic VirtualTableModule |
|
|
691
|
+
| `StoreTable` | Virtual table implementation (incl. `applyExternalRowChanges` / `readRowByPk` for externally-applied source writes) |
|
|
692
|
+
| `ExternalRowOp` | One externally-applied row op (`upsert`/`delete`) for `StoreTable.applyExternalRowChanges` (type) |
|
|
693
|
+
| `resolvePkKeyCollations` | Per-PK-column key collations (pass to `buildDataKey`/`buildIndexKey` to match `StoreTable`'s key bytes) |
|
|
694
|
+
| `resolveIndexKeyCollations` | Per-index-column key collations (the index half of `buildIndexKey`: index COLLATE ?? column collation ?? BINARY) |
|
|
695
|
+
| `StoreConnection` | Transaction connection |
|
|
696
|
+
| `TransactionCoordinator` | Transaction management |
|
|
697
|
+
| `StoreEventEmitter` | Event system for data/schema changes |
|
|
698
|
+
|
|
699
|
+
### Isolation Layer Utilities
|
|
700
|
+
|
|
701
|
+
| Export | Description |
|
|
702
|
+
|--------|-------------|
|
|
703
|
+
| `createIsolatedStoreModule` | Create store module with isolation layer |
|
|
704
|
+
| `hasIsolation` | Check if a module has isolation capability |
|
|
705
|
+
| `IsolatedStoreModuleConfig` | Configuration for isolated store module |
|
|
706
|
+
|
|
707
|
+
### Caching
|
|
708
|
+
|
|
709
|
+
| Export | Description |
|
|
710
|
+
|--------|-------------|
|
|
711
|
+
| `CachedKVStore` | Read-through LRU cache wrapper for any `KVStore` |
|
|
712
|
+
| `CacheOptions` | Configuration for cache (maxEntries, maxBytes, enabled) |
|
|
713
|
+
|
|
714
|
+
### Encoding Utilities
|
|
715
|
+
|
|
716
|
+
| Export | Description |
|
|
717
|
+
|--------|-------------|
|
|
718
|
+
| `encodeValue` | Encode a SQL value to sortable bytes |
|
|
719
|
+
| `decodeValue` | Decode bytes back to SQL value |
|
|
720
|
+
| `encodeCompositeKey` | Encode multiple values as composite key |
|
|
721
|
+
| `decodeCompositeKey` | Decode composite key to values |
|
|
722
|
+
| `BUILTIN_KEY_NORMALIZER_RESOLVER` | Built-ins-only key-normalizer resolver (`EncodeOptions.normalizers` default) |
|
|
723
|
+
|
|
724
|
+
### Serialization Utilities
|
|
725
|
+
|
|
726
|
+
| Export | Description |
|
|
727
|
+
|--------|-------------|
|
|
728
|
+
| `serializeRow` | Serialize a row to bytes |
|
|
729
|
+
| `deserializeRow` | Deserialize bytes to row |
|
|
730
|
+
| `serializeValue` | Serialize a single value |
|
|
731
|
+
| `deserializeValue` | Deserialize a single value |
|
|
732
|
+
|
|
733
|
+
### Key Building
|
|
734
|
+
|
|
735
|
+
| Export | Description |
|
|
736
|
+
|--------|-------------|
|
|
737
|
+
| `buildDataStoreName` | Build store name for table data |
|
|
738
|
+
| `buildIndexStoreName` | Build store name for an index |
|
|
739
|
+
| `buildStatsStoreName` | Build store name for table stats |
|
|
740
|
+
| `buildDataKey` | Build key for row data (encoded PK) |
|
|
741
|
+
| `buildIndexKey` | Build key for index entry |
|
|
742
|
+
| `buildCatalogKey` | Build key for a table's catalog entry (`{schema}.{table}`) |
|
|
743
|
+
| `buildViewCatalogKey` | Build key for a view's catalog entry (reserved `\x00view\x00` prefix) |
|
|
744
|
+
| `buildMaterializedViewCatalogKey` | Build key for an MV's catalog entry (reserved `\x00mview\x00` prefix) |
|
|
745
|
+
| `classifyCatalogKey` | Classify a loaded catalog key as `'table'` / `'view'` / `'materializedView'` |
|
|
746
|
+
| `buildFullScanBounds` | Build bounds for full table scan |
|
|
747
|
+
| `buildIndexPrefixBounds` | Build bounds for index prefix scan |
|
|
748
|
+
| `buildPkPrefixBounds` | Build bounds for a data-store PK prefix range (per-column DESC + key collations) |
|
|
749
|
+
| `buildCatalogScanBounds` | Build bounds for catalog scan |
|
|
750
|
+
| `CATALOG_STORE_NAME` | Reserved catalog store name constant |
|
|
751
|
+
| `STORE_SUFFIX` | Store name suffixes (INDEX, STATS) |
|
|
752
|
+
|
|
753
|
+
## Related Packages
|
|
754
|
+
|
|
755
|
+
- [`@quereus/plugin-leveldb`](../quereus-plugin-leveldb/) - LevelDB implementation for Node.js
|
|
756
|
+
- [`@quereus/plugin-indexeddb`](../quereus-plugin-indexeddb/) - IndexedDB implementation for browsers
|
|
757
|
+
- [`@quereus/sync`](../quereus-sync/) - CRDT sync layer
|
|
758
|
+
|
|
759
|
+
## License
|
|
760
|
+
|
|
761
|
+
MIT
|