@crvouga/sqlite-mem 0.1.0 → 1.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENTS.md +154 -0
- package/COMPATIBILITY-AUDIT.md +7 -9
- package/COMPATIBILITY.md +5 -4
- package/README.md +166 -16
- package/compat/coverage.json +797 -512
- package/compat/requirements.json +1 -1
- package/dist/api/database.d.ts +97 -12
- package/dist/api/statement.d.ts +65 -18
- package/dist/ast/nodes.d.ts +11 -2
- package/dist/constraints/check.d.ts +5 -6
- package/dist/errors/index.d.ts +36 -3
- package/dist/executor/attach.d.ts +3 -4
- package/dist/executor/ddl.d.ts +3 -4
- package/dist/executor/dml.d.ts +4 -4
- package/dist/executor/env.d.ts +9 -11
- package/dist/executor/execute.d.ts +3 -4
- package/dist/executor/pragma.d.ts +3 -4
- package/dist/executor/result.d.ts +10 -6
- package/dist/executor/select.d.ts +4 -5
- package/dist/executor/simple-select.d.ts +3 -4
- package/dist/executor/triggers.d.ts +6 -7
- package/dist/executor/vtable.d.ts +5 -6
- package/dist/expressions/context.d.ts +5 -5
- package/dist/expressions/equals.d.ts +3 -0
- package/dist/expressions/eval.d.ts +9 -4
- package/dist/expressions/index.d.ts +3 -4
- package/dist/expressions/like.d.ts +13 -3
- package/dist/functions/aggregate.d.ts +1 -2
- package/dist/functions/datetime.d.ts +1 -2
- package/dist/functions/extensions.d.ts +1 -2
- package/dist/functions/index.d.ts +8 -9
- package/dist/functions/json.d.ts +3 -4
- package/dist/functions/math.d.ts +1 -2
- package/dist/functions/registry.d.ts +5 -6
- package/dist/functions/scalar.d.ts +2 -3
- package/dist/functions/table-valued.d.ts +3 -4
- package/dist/functions/window.d.ts +1 -2
- package/dist/index.d.ts +27 -12
- package/dist/index.js +986 -203
- package/dist/index.js.map +4 -4
- package/dist/indexes/index.d.ts +17 -4
- package/dist/json/index.d.ts +7 -8
- package/dist/json/jsonb.d.ts +1 -2
- package/dist/json/ops.d.ts +8 -3
- package/dist/json/parse.d.ts +2 -3
- package/dist/json/path.d.ts +1 -2
- package/dist/json/stringify.d.ts +1 -2
- package/dist/json/tvf.d.ts +1 -2
- package/dist/json/types.d.ts +0 -1
- package/dist/lexer/tokenize.d.ts +7 -1
- package/dist/parser/index.d.ts +11 -5
- package/dist/parser/parser.d.ts +3 -3
- package/dist/planner/access.d.ts +13 -6
- package/dist/planner/index.d.ts +1 -2
- package/dist/runtime/clock.d.ts +8 -2
- package/dist/runtime/index.d.ts +3 -4
- package/dist/runtime/options.d.ts +4 -6
- package/dist/runtime/prng.d.ts +12 -1
- package/dist/schema/catalog.d.ts +2 -3
- package/dist/serialization/codec.d.ts +17 -2
- package/dist/serialization/index.d.ts +1 -2
- package/dist/storage/database-state.d.ts +16 -8
- package/dist/storage/index.d.ts +6 -7
- package/dist/storage/row.d.ts +1 -2
- package/dist/storage/table.d.ts +8 -4
- package/dist/transactions/manager.d.ts +2 -3
- package/dist/types/collation.d.ts +1 -2
- package/dist/types/strict.d.ts +6 -0
- package/dist/types/value.d.ts +30 -2
- package/dist/unstable.d.ts +20 -0
- package/dist/unstable.js +8734 -0
- package/dist/unstable.js.map +7 -0
- package/dist/vtable/fts/options.d.ts +0 -1
- package/dist/vtable/fts/porter.d.ts +0 -1
- package/dist/vtable/fts/query.d.ts +0 -1
- package/dist/vtable/fts/table.d.ts +5 -6
- package/dist/vtable/fts/tokenize.d.ts +1 -2
- package/dist/vtable/fts5.d.ts +1 -2
- package/dist/vtable/index.d.ts +2 -3
- package/dist/vtable/modules.d.ts +2 -3
- package/package.json +19 -10
- package/compat/gate-report.json +0 -424
- package/dist/api/database.d.ts.map +0 -1
- package/dist/api/statement.d.ts.map +0 -1
- package/dist/ast/nodes.d.ts.map +0 -1
- package/dist/constraints/check.d.ts.map +0 -1
- package/dist/errors/index.d.ts.map +0 -1
- package/dist/executor/attach.d.ts.map +0 -1
- package/dist/executor/ddl.d.ts.map +0 -1
- package/dist/executor/dml.d.ts.map +0 -1
- package/dist/executor/env.d.ts.map +0 -1
- package/dist/executor/execute.d.ts.map +0 -1
- package/dist/executor/pragma.d.ts.map +0 -1
- package/dist/executor/result.d.ts.map +0 -1
- package/dist/executor/select.d.ts.map +0 -1
- package/dist/executor/simple-select.d.ts.map +0 -1
- package/dist/executor/triggers.d.ts.map +0 -1
- package/dist/executor/vtable.d.ts.map +0 -1
- package/dist/expressions/context.d.ts.map +0 -1
- package/dist/expressions/eval.d.ts.map +0 -1
- package/dist/expressions/index.d.ts.map +0 -1
- package/dist/expressions/like.d.ts.map +0 -1
- package/dist/functions/aggregate.d.ts.map +0 -1
- package/dist/functions/datetime.d.ts.map +0 -1
- package/dist/functions/extensions.d.ts.map +0 -1
- package/dist/functions/index.d.ts.map +0 -1
- package/dist/functions/json.d.ts.map +0 -1
- package/dist/functions/math.d.ts.map +0 -1
- package/dist/functions/registry.d.ts.map +0 -1
- package/dist/functions/scalar.d.ts.map +0 -1
- package/dist/functions/table-valued.d.ts.map +0 -1
- package/dist/functions/window.d.ts.map +0 -1
- package/dist/index.d.ts.map +0 -1
- package/dist/indexes/index.d.ts.map +0 -1
- package/dist/json/index.d.ts.map +0 -1
- package/dist/json/jsonb.d.ts.map +0 -1
- package/dist/json/ops.d.ts.map +0 -1
- package/dist/json/parse.d.ts.map +0 -1
- package/dist/json/path.d.ts.map +0 -1
- package/dist/json/stringify.d.ts.map +0 -1
- package/dist/json/tvf.d.ts.map +0 -1
- package/dist/json/types.d.ts.map +0 -1
- package/dist/lexer/tokenize.d.ts.map +0 -1
- package/dist/parser/index.d.ts.map +0 -1
- package/dist/parser/parser.d.ts.map +0 -1
- package/dist/planner/access.d.ts.map +0 -1
- package/dist/planner/index.d.ts.map +0 -1
- package/dist/runtime/clock.d.ts.map +0 -1
- package/dist/runtime/index.d.ts.map +0 -1
- package/dist/runtime/options.d.ts.map +0 -1
- package/dist/runtime/prng.d.ts.map +0 -1
- package/dist/schema/catalog.d.ts.map +0 -1
- package/dist/serialization/codec.d.ts.map +0 -1
- package/dist/serialization/index.d.ts.map +0 -1
- package/dist/storage/database-state.d.ts.map +0 -1
- package/dist/storage/index.d.ts.map +0 -1
- package/dist/storage/row.d.ts.map +0 -1
- package/dist/storage/table.d.ts.map +0 -1
- package/dist/transactions/manager.d.ts.map +0 -1
- package/dist/types/collation.d.ts.map +0 -1
- package/dist/types/value.d.ts.map +0 -1
- package/dist/vtable/fts/options.d.ts.map +0 -1
- package/dist/vtable/fts/porter.d.ts.map +0 -1
- package/dist/vtable/fts/query.d.ts.map +0 -1
- package/dist/vtable/fts/table.d.ts.map +0 -1
- package/dist/vtable/fts/tokenize.d.ts.map +0 -1
- package/dist/vtable/fts5.d.ts.map +0 -1
- package/dist/vtable/index.d.ts.map +0 -1
- package/dist/vtable/modules.d.ts.map +0 -1
package/AGENTS.md
ADDED
|
@@ -0,0 +1,154 @@
|
|
|
1
|
+
# AGENTS.md — contributing to sqlite-mem
|
|
2
|
+
|
|
3
|
+
Guidance for humans and coding agents editing this repository. For install and consumer API, see [README.md](README.md). For the feature matrix, see [COMPATIBILITY.md](COMPATIBILITY.md).
|
|
4
|
+
|
|
5
|
+
## Mission and non-goals
|
|
6
|
+
|
|
7
|
+
**Mission:** full SQLite3 **SQL dialect** behavioral parity vs **SQLite 3.51.0** (`bun:sqlite`). Same statements → same observable results (rows, errors, counters), proven by differential contracts and a fail-closed gate.
|
|
8
|
+
|
|
9
|
+
**Allowed intentional differences:**
|
|
10
|
+
|
|
11
|
+
1. Custom snapshot codec (`SQLM`), not on-disk `.sqlite` files
|
|
12
|
+
2. Deterministic `random()` / `randomblob()` and fixed `'now'` by default (injectable)
|
|
13
|
+
3. `NOT APPLICABLE` items: C API, VFS/pager/WAL, file locking, URI filenames
|
|
14
|
+
|
|
15
|
+
**Non-goals:** shipping a SQLite file format, native/WASM bindings, or matching better-sqlite3’s full Node API surface.
|
|
16
|
+
|
|
17
|
+
## SQL pipeline
|
|
18
|
+
|
|
19
|
+
```
|
|
20
|
+
SQL string
|
|
21
|
+
→ tokenize() src/lexer/tokenize.ts
|
|
22
|
+
→ parse() src/parser/index.ts → parser.ts
|
|
23
|
+
→ Statement[] AST src/ast/nodes.ts
|
|
24
|
+
→ Statement.execute() src/api/statement.ts
|
|
25
|
+
→ executeStatement() src/executor/execute.ts
|
|
26
|
+
→ per-stmt executor select / dml / ddl / …
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
Public entry points: `Database.exec` / `query` / `prepare` in [`src/api/database.ts`](src/api/database.ts).
|
|
30
|
+
|
|
31
|
+
**Planner note:** [`src/planner/index.ts`](src/planner/index.ts) `preparePlan` is an **identity stub** (`Plan === Statement`) and is unused. Runtime index/PK access paths live in [`src/planner/access.ts`](src/planner/access.ts) (`tryIndexedTableRows`, `lookupTableRows`, `tryJoinProbe`), called from the executor.
|
|
32
|
+
|
|
33
|
+
SELECT has a **fast path** ([`src/executor/simple-select.ts`](src/executor/simple-select.ts)) and a **full path** ([`src/executor/select.ts`](src/executor/select.ts)). Fast paths return `null` when the shape is unsupported; then the full path runs. Changing join/WHERE/expression semantics in only one path will silently diverge.
|
|
34
|
+
|
|
35
|
+
## `src/` map
|
|
36
|
+
|
|
37
|
+
| Directory | Role |
|
|
38
|
+
| --- | --- |
|
|
39
|
+
| `api/` | Public `Database` / `Statement` facade |
|
|
40
|
+
| `ast/` | Discriminated-union AST (`nodes.ts`) |
|
|
41
|
+
| `lexer/` | Tokenizer |
|
|
42
|
+
| `parser/` | Recursive-descent parser |
|
|
43
|
+
| `planner/` | Access-path helpers (not a logical plan IR) |
|
|
44
|
+
| `executor/` | Statement dispatch, SELECT, DML, DDL, PRAGMA, triggers, ATTACH |
|
|
45
|
+
| `expressions/` | `evalExpr`, LIKE/GLOB, eval context |
|
|
46
|
+
| `functions/` | Scalar / aggregate / window / datetime / JSON / math / TVF registries |
|
|
47
|
+
| `types/` | Engine `SqlValue`, affinity, collation |
|
|
48
|
+
| `storage/` | In-memory tables, rows, `DatabaseState` |
|
|
49
|
+
| `indexes/` | Equality index store |
|
|
50
|
+
| `schema/` | `sqlite_master` / catalog |
|
|
51
|
+
| `constraints/` | NOT NULL / PK / UNIQUE / CHECK |
|
|
52
|
+
| `transactions/` | BEGIN / COMMIT / SAVEPOINT (clones state + PRNG) |
|
|
53
|
+
| `runtime/` | Clock, PRNG, `DatabaseOptions` |
|
|
54
|
+
| `serialization/` | `SQLM` snapshot codec |
|
|
55
|
+
| `json/` | JSON1 / JSONB internals |
|
|
56
|
+
| `vtable/` | FTS5 and other virtual-table modules |
|
|
57
|
+
| `errors/` | `SqliteError`, `unsupported()` |
|
|
58
|
+
|
|
59
|
+
Hot / large files: `parser/parser.ts`, `executor/select.ts`, `executor/dml.ts`.
|
|
60
|
+
|
|
61
|
+
## Critical conventions
|
|
62
|
+
|
|
63
|
+
- **AST `type` tags** are snake_case (`"create_table"`, `"drop_index"`). TypeScript interfaces are PascalCase (`CreateTableStmt`).
|
|
64
|
+
- **Identifiers** are case-folded (`toLowerCase`) as Map keys in storage and function registries.
|
|
65
|
+
- **Engine `SqlValue`** ([`src/types/value.ts`](src/types/value.ts)) may include `SqlReal` / `SqlJsonText`. **Harness `SqlValue`** ([`tests/harness/types.ts`](tests/harness/types.ts)) is the normalized compare type — do not confuse them.
|
|
66
|
+
- **API `Statement`** vs **AST `Statement`**: the API class aliases the AST union as `AstStatement`.
|
|
67
|
+
- **`Database`** (API) vs **`DatabaseState`** (engine storage).
|
|
68
|
+
- Throw **`SqliteError`** with an `ErrorCategory`. Missing SQL must fail loud via `unsupported()` — the inventory gate fails if the oracle exposes an unimplemented builtin/module.
|
|
69
|
+
- Fast-path helpers (`tryExecuteSimpleSelect`, `tryFastInsert`, `tryIndexedTableRows`, …): return `null` → fall through. Update **both** paths when semantics change.
|
|
70
|
+
- TypeScript: `strict` + `noUncheckedIndexedAccess`. Imports use `.ts` extensions. Biome: 2-space, double quotes, 120 columns.
|
|
71
|
+
- IEEE `-0` is canonicalized to `+0` on bind, affinity, and arithmetic. Keep determinism invariants (see README).
|
|
72
|
+
|
|
73
|
+
## Change checklists
|
|
74
|
+
|
|
75
|
+
### New SQL statement
|
|
76
|
+
|
|
77
|
+
1. Add union member + interface in [`src/ast/nodes.ts`](src/ast/nodes.ts)
|
|
78
|
+
2. Parse in [`src/parser/parser.ts`](src/parser/parser.ts) (`parseStatement` dispatch)
|
|
79
|
+
3. Handle in [`src/executor/execute.ts`](src/executor/execute.ts) (+ `ddl.ts` / `dml.ts` / `vtable.ts` as needed)
|
|
80
|
+
4. Mutate `DatabaseState` if schema changes
|
|
81
|
+
5. Add differential contract under `tests/contract/<area>/`
|
|
82
|
+
6. Optionally add evidence paths in `SOURCE_SEED` in [`scripts/sqlite-requirements.ts`](scripts/sqlite-requirements.ts), then `bun run requirements`
|
|
83
|
+
|
|
84
|
+
### New SQL function
|
|
85
|
+
|
|
86
|
+
1. Implement and register in the right map under `src/functions/*` (`getScalarFunctions()`, datetime, JSON, etc.) so [`scripts/sqlite-inventory.ts`](scripts/sqlite-inventory.ts) sees it
|
|
87
|
+
2. Contract tests under `tests/contract/functions/` (and related areas)
|
|
88
|
+
3. Run `bun run inventory` / `bun run test:sqlite-compat` — oracle builtins must not be missing
|
|
89
|
+
|
|
90
|
+
### New contract test
|
|
91
|
+
|
|
92
|
+
1. Prefer helpers in [`tests/contract/helpers.ts`](tests/contract/helpers.ts):
|
|
93
|
+
- `parity` — query both engines
|
|
94
|
+
- `execParity` — writes
|
|
95
|
+
- `sequenceParity` — multi-step (optional final-state compare)
|
|
96
|
+
- `errorParity` / `queryErrorParity` — both must fail
|
|
97
|
+
- `ftsRankParity` — REAL epsilon compare
|
|
98
|
+
2. Or `matrixBoth` + `expectParity` from `tests/harness/`
|
|
99
|
+
3. **Do not** treat isolated internal unit tests as SQLite proof. The differential suite is authoritative.
|
|
100
|
+
4. Gate: `bun run test:sqlite-compat`
|
|
101
|
+
|
|
102
|
+
## Test layout
|
|
103
|
+
|
|
104
|
+
| Path | Role |
|
|
105
|
+
| --- | --- |
|
|
106
|
+
| `tests/contract/` | Differential SQL vs `bun:sqlite` (**authoritative**) |
|
|
107
|
+
| `tests/fuzz/` | fast-check property tests (seeded); same two backends |
|
|
108
|
+
| `tests/harness/` | Compare/normalize helpers + harness unit tests |
|
|
109
|
+
| `tests/adapters/` | Wrappers for sqlite-mem and `bun:sqlite` |
|
|
110
|
+
|
|
111
|
+
Examples of public API usage: `tests/contract/api/`, `tests/contract/parameters/`, `tests/contract/determinism/`, `examples/react-vite`.
|
|
112
|
+
|
|
113
|
+
### Fuzz replay
|
|
114
|
+
|
|
115
|
+
Default seed `0x5a17e0e1`. On failure the seed is printed:
|
|
116
|
+
|
|
117
|
+
```bash
|
|
118
|
+
bun test tests/fuzz
|
|
119
|
+
SQLITE_MEM_FUZZ_SEED=12345 bun test tests/fuzz
|
|
120
|
+
SQLITE_MEM_FUZZ_SEED=12345 SQLITE_MEM_FUZZ_PATH='0:1' bun test tests/fuzz
|
|
121
|
+
```
|
|
122
|
+
|
|
123
|
+
## Compat system
|
|
124
|
+
|
|
125
|
+
| Command | Role |
|
|
126
|
+
| --- | --- |
|
|
127
|
+
| `bun run test:sqlite-compat` | Requirements + fail-closed gate + contract/fuzz/harness |
|
|
128
|
+
| `bun run inventory` | Oracle `pragma_function_list` / modules vs memory registries |
|
|
129
|
+
| `bun run requirements` | Refresh sqlite.org requirements → `compat/requirements.json` + `compat/coverage.json` |
|
|
130
|
+
| `bun run fts-surface` | FTS oracle surface → `compat/fts-oracle-surface.json` |
|
|
131
|
+
|
|
132
|
+
Statuses: **VERIFIED** / **PARTIALLY VERIFIED** / **UNSUPPORTED** / **NOT APPLICABLE**. Do not market PARTIAL as complete. Coverage evidence is directory paths (e.g. `tests/contract/joins/`), not automatic from test filenames.
|
|
133
|
+
|
|
134
|
+
Details: [COMPATIBILITY.md](COMPATIBILITY.md), audit: [COMPATIBILITY-AUDIT.md](COMPATIBILITY-AUDIT.md).
|
|
135
|
+
|
|
136
|
+
## Local gates
|
|
137
|
+
|
|
138
|
+
Requires [Bun](https://bun.sh).
|
|
139
|
+
|
|
140
|
+
```bash
|
|
141
|
+
bun install
|
|
142
|
+
bun run ci:local # same gates as GitHub Actions CI (except publish)
|
|
143
|
+
bun run check # format + lint + typecheck + sqlite-compat suite
|
|
144
|
+
bun run format
|
|
145
|
+
bun run lint
|
|
146
|
+
bun run typecheck
|
|
147
|
+
bun run test:sqlite-compat
|
|
148
|
+
bun test # contract + fuzz + harness
|
|
149
|
+
bun run build
|
|
150
|
+
```
|
|
151
|
+
|
|
152
|
+
## PR and commits
|
|
153
|
+
|
|
154
|
+
Use [Conventional Commits](https://www.conventionalcommits.org/) for commits and PR titles (enforced in CI). Prefer squash merges with a conventional title. Releasing is automated via semantic-release — see [README.md](README.md#releasing).
|
package/COMPATIBILITY-AUDIT.md
CHANGED
|
@@ -44,14 +44,13 @@ JSON / JSONB:
|
|
|
44
44
|
VERIFIED
|
|
45
45
|
|
|
46
46
|
Aggregates / windows:
|
|
47
|
-
VERIFIED — includes string_agg, ntile, cume_dist, percent_rank
|
|
48
|
-
(window EXCLUDE still thinner)
|
|
47
|
+
VERIFIED — includes string_agg, ntile, cume_dist, percent_rank, EXCLUDE
|
|
49
48
|
|
|
50
49
|
CTEs / transactions / savepoints / constraints / FK / triggers:
|
|
51
|
-
VERIFIED (deferred FK
|
|
50
|
+
VERIFIED (deferred FK + composite FK)
|
|
52
51
|
|
|
53
52
|
Indexes / views / generated / STRICT / WITHOUT ROWID:
|
|
54
|
-
VERIFIED /
|
|
53
|
+
VERIFIED — partial/expression indexes, STRICT tables, leftmost prefix
|
|
55
54
|
|
|
56
55
|
PRAGMAs:
|
|
57
56
|
PARTIALLY VERIFIED — schema/FK; storage pragmas N/A or :memory: no-op
|
|
@@ -67,11 +66,11 @@ ANALYZE / REINDEX / VACUUM:
|
|
|
67
66
|
VERIFIED (:memory: observable parity)
|
|
68
67
|
|
|
69
68
|
Prepared statements / errors / snapshot:
|
|
70
|
-
|
|
69
|
+
VERIFIED — schema invalidation re-prepares; SQLM logical round-trip VERIFIED
|
|
71
70
|
|
|
72
|
-
Differential tests:
|
|
73
|
-
Total:
|
|
74
|
-
Passed:
|
|
71
|
+
Differential tests:
|
|
72
|
+
Total: 727 under `bun test` (contract + fuzz + harness)
|
|
73
|
+
Passed: 727
|
|
75
74
|
Failed: 0
|
|
76
75
|
|
|
77
76
|
Stateful / fuzz:
|
|
@@ -91,7 +90,6 @@ New incompatibilities found & fixed (this pass):
|
|
|
91
90
|
Remaining known differences:
|
|
92
91
|
Custom SQLM snapshots; deterministic random()/'now';
|
|
93
92
|
EXPLAIN/INDEXED BY stubs/no-ops; some PRAGMA storage no-ops;
|
|
94
|
-
Window EXCLUDE / deferred FK / STRICT / prepared invalidation thinner;
|
|
95
93
|
BigInt beyond Number.MAX_SAFE_INTEGER without bun safeIntegers;
|
|
96
94
|
NOT APPLICABLE C API / on-disk / VFS surfaces.
|
|
97
95
|
|
package/COMPATIBILITY.md
CHANGED
|
@@ -34,10 +34,11 @@ Oracle builtins (math, string extras, uuid, ieee754, …) and modules (FTS3/4/5,
|
|
|
34
34
|
| Area | Status | Notes |
|
|
35
35
|
| --- | --- | --- |
|
|
36
36
|
| Core DML / SELECT / joins / CTE / UPSERT / RETURNING | VERIFIED | Contract + fuzz |
|
|
37
|
+
| STRICT tables / indexes | VERIFIED | STRICT types; partial + expression indexes; leftmost prefix |
|
|
37
38
|
| Expressions / operators / `->` `->>` / row values | VERIFIED | Row-value + precedence contracts |
|
|
38
39
|
| Affinity / NULL / COLLATE | VERIFIED | |
|
|
39
|
-
| Constraints / FK / triggers / views / ATTACH | VERIFIED | Deferred FK
|
|
40
|
-
| Windows (incl. ntile/cume_dist/percent_rank) | VERIFIED | EXCLUDE
|
|
40
|
+
| Constraints / FK / triggers / views / ATTACH | VERIFIED | Deferred FK, composite FK |
|
|
41
|
+
| Windows (incl. ntile/cume_dist/percent_rank) | VERIFIED | EXCLUDE NO OTHERS/CURRENT ROW/GROUP/TIES |
|
|
41
42
|
| JSON1 / JSONB / TVFs | VERIFIED | |
|
|
42
43
|
| Math / string / date extras / uuid / ieee754 | VERIFIED | Scope-3 inventory |
|
|
43
44
|
| FTS3 / FTS4 / FTS5 + MATCH | PARTIALLY VERIFIED | Differential FTS suite + fuzz vs 3.51.0; see FTS matrix below. Shadow-table change counters intentionally diverge. |
|
|
@@ -76,7 +77,7 @@ Reference: **SQLite 3.51.0** (`bun:sqlite`). Inventory: `bun run scripts/fts-ora
|
|
|
76
77
|
| RTREE / dbstat / bytecode / tables_used | VERIFIED | dbstat synthetic pages; bytecode empty cursor |
|
|
77
78
|
| ANALYZE / REINDEX / VACUUM | VERIFIED | `:memory:` observable parity |
|
|
78
79
|
| EXPLAIN / INDEXED BY | PARTIALLY VERIFIED | Stub shapes / no-ops |
|
|
79
|
-
| Prepared stmt schema invalidation |
|
|
80
|
+
| Prepared stmt schema invalidation | VERIFIED | Re-prepare after ALTER/DROP; `tests/contract/api/schema-invalidation.test.ts` |
|
|
80
81
|
| On-disk file format / C API | NOT APPLICABLE | |
|
|
81
82
|
|
|
82
83
|
## How to verify
|
|
@@ -85,7 +86,7 @@ Reference: **SQLite 3.51.0** (`bun:sqlite`). Inventory: `bun run scripts/fts-ora
|
|
|
85
86
|
bun run test:sqlite-compat # requirements + gate + contract/fuzz/harness
|
|
86
87
|
bun run inventory # oracle function/module inventory
|
|
87
88
|
bun run requirements # refresh SQLite.org requirements + coverage
|
|
88
|
-
bun run
|
|
89
|
+
bun run build && bun run verify-package # ESM browser build + isomorphic pack gates
|
|
89
90
|
```
|
|
90
91
|
|
|
91
92
|
Do not treat isolated unit tests of internal modules as proof of SQLite compatibility. The differential matrix runner is authoritative for SQL behavior; `test:sqlite-compat` is the release gate.
|
package/README.md
CHANGED
|
@@ -5,10 +5,22 @@ Pure TypeScript, completely in-memory SQLite implementation aiming for **full SQ
|
|
|
5
5
|
- Runs in modern browsers and Node.js / Bun
|
|
6
6
|
- **Zero** WASM, native bindings, workers, or filesystem dependencies
|
|
7
7
|
- Entire database stored in memory
|
|
8
|
+
- **Synchronous** ESM-only API (no Promises, no `require`)
|
|
8
9
|
- **Verified against SQLite 3.51.0** (`bun:sqlite`) via differential contracts + fail-closed gate
|
|
9
10
|
- Intentional differences: deterministic `random()` / `'now'` by default, and a custom snapshot format (not `.sqlite` files)
|
|
10
11
|
|
|
11
|
-
See [COMPATIBILITY.md](COMPATIBILITY.md) for the matrix and [COMPATIBILITY-AUDIT.md](COMPATIBILITY-AUDIT.md) for the audit report.
|
|
12
|
+
See [COMPATIBILITY.md](COMPATIBILITY.md) for the matrix and [COMPATIBILITY-AUDIT.md](COMPATIBILITY-AUDIT.md) for the audit report. Agents contributing to this repo: start with [AGENTS.md](AGENTS.md).
|
|
13
|
+
|
|
14
|
+
## Documentation
|
|
15
|
+
|
|
16
|
+
| Doc | For |
|
|
17
|
+
| --- | --- |
|
|
18
|
+
| [README.md](README.md) | Install, API, pitfalls (this file) |
|
|
19
|
+
| [AGENTS.md](AGENTS.md) | Architecture, how to change code, test/compat gates |
|
|
20
|
+
| [COMPATIBILITY.md](COMPATIBILITY.md) | Feature matrix + verify commands |
|
|
21
|
+
| [COMPATIBILITY-AUDIT.md](COMPATIBILITY-AUDIT.md) | Audit evidence |
|
|
22
|
+
| [docs/SECRETS.md](docs/SECRETS.md) | npm / CI publish setup |
|
|
23
|
+
| [benchmarks/PERFORMANCE.md](benchmarks/PERFORMANCE.md) | Performance notes |
|
|
12
24
|
|
|
13
25
|
## Install
|
|
14
26
|
|
|
@@ -18,6 +30,8 @@ bun add @crvouga/sqlite-mem
|
|
|
18
30
|
npm install @crvouga/sqlite-mem
|
|
19
31
|
```
|
|
20
32
|
|
|
33
|
+
Requires Node.js ≥ 20 or Bun ≥ 1.1. The published package is **ESM only** (`import` from `@crvouga/sqlite-mem`).
|
|
34
|
+
|
|
21
35
|
## Usage
|
|
22
36
|
|
|
23
37
|
```ts
|
|
@@ -32,7 +46,7 @@ db.exec(`
|
|
|
32
46
|
)
|
|
33
47
|
`);
|
|
34
48
|
|
|
35
|
-
db.
|
|
49
|
+
db.prepare(`INSERT INTO users (name) VALUES (?)`).run("Alice");
|
|
36
50
|
|
|
37
51
|
const users = db.query<{ id: number; name: string }>(`SELECT * FROM users`);
|
|
38
52
|
console.log(users);
|
|
@@ -42,36 +56,132 @@ const db2 = new Database();
|
|
|
42
56
|
db2.restore(snap);
|
|
43
57
|
```
|
|
44
58
|
|
|
59
|
+
All methods are **synchronous** — do not `await` them. Browser and Node/Bun share the same in-memory JS surface (no filesystem; `ATTACH` opens a new empty in-memory schema, not a file).
|
|
60
|
+
|
|
61
|
+
## Example
|
|
62
|
+
|
|
63
|
+
A React + Vite SQL playground lives in [`examples/react-vite`](examples/react-vite):
|
|
64
|
+
|
|
65
|
+
```bash
|
|
66
|
+
cd examples/react-vite
|
|
67
|
+
bun install
|
|
68
|
+
bun run dev
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
From the repo root after that install: `bun run example`.
|
|
72
|
+
|
|
45
73
|
## API
|
|
46
74
|
|
|
47
75
|
```ts
|
|
76
|
+
import { Database, SqliteError } from "@crvouga/sqlite-mem";
|
|
77
|
+
|
|
78
|
+
interface DatabaseOptions {
|
|
79
|
+
seed?: number | bigint; // default 1 — PRNG for random() / randomblob()
|
|
80
|
+
now?: Date | (() => Date); // default 2000-01-01T00:00:00.000Z
|
|
81
|
+
}
|
|
82
|
+
|
|
48
83
|
interface Database {
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
query<T = Record<string, unknown>>(sql: string, params?: unknown[]): T[];
|
|
84
|
+
constructor(options?: DatabaseOptions);
|
|
85
|
+
exec(sql: string): void;
|
|
86
|
+
query<T = QueryRow>(sql: string, params?: BindValue[]): T[];
|
|
53
87
|
prepare(sql: string): Statement;
|
|
54
88
|
transaction<T>(fn: () => T): T;
|
|
55
89
|
snapshot(): Uint8Array;
|
|
56
90
|
restore(snapshot: Uint8Array): void;
|
|
57
91
|
close(): void;
|
|
92
|
+
[Symbol.dispose]?(): void; // alias for close() when Symbol.dispose exists
|
|
93
|
+
readonly changes: number;
|
|
94
|
+
readonly lastInsertRowid: number | bigint;
|
|
58
95
|
}
|
|
59
96
|
|
|
60
97
|
interface Statement {
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
|
|
98
|
+
run(...params: BindValue[]): RunResult;
|
|
99
|
+
all<T = QueryRow>(...params: BindValue[]): T[];
|
|
100
|
+
get<T = QueryRow>(...params: BindValue[]): T | undefined;
|
|
101
|
+
result(...params: BindValue[]): ResultSet; // includes columns + values when zero rows
|
|
102
|
+
}
|
|
103
|
+
|
|
104
|
+
interface RunResult {
|
|
105
|
+
changes: number;
|
|
106
|
+
lastInsertRowid: number | bigint;
|
|
107
|
+
}
|
|
108
|
+
|
|
109
|
+
interface ResultSet {
|
|
110
|
+
columns: string[];
|
|
111
|
+
rows: QueryRow[];
|
|
112
|
+
values: QueryValue[][]; // always present (empty array for zero rows)
|
|
113
|
+
changes: number;
|
|
114
|
+
lastInsertRowid: number | bigint;
|
|
65
115
|
}
|
|
116
|
+
|
|
117
|
+
class SqliteError extends Error {
|
|
118
|
+
readonly category: ErrorCategory; // syntax, no_such_table, constraint_unique, misuse, …
|
|
119
|
+
readonly sqliteCode: string; // always set; default "SQLITE_ERROR"
|
|
120
|
+
readonly code: string; // === sqliteCode (Node err.code convention)
|
|
121
|
+
}
|
|
122
|
+
```
|
|
123
|
+
|
|
124
|
+
Stick to `Database`, `Statement`, and `SqliteError` for application code. Advanced internals (`parse`, `tokenize`, `evalExpr`, snapshot codec pieces, `SqlValue` utilities, `Prng`, …) are available only from `@crvouga/sqlite-mem/unstable` and are **exempt from semver**.
|
|
125
|
+
|
|
126
|
+
### Method semantics
|
|
127
|
+
|
|
128
|
+
| Method | Behavior |
|
|
129
|
+
| --- | --- |
|
|
130
|
+
| `exec(sql)` | Runs all semicolon-separated statements; **discards** row results (`void`). Does **not** accept bind parameters. Read `db.changes` / `db.lastInsertRowid` afterward if needed (counters reflect the **most recent** completed statement, matching SQLite). |
|
|
131
|
+
| `query(sql, params?)` | **Single statement only** (trailing `;` is fine). Returns all rows. Multi-statement scripts throw `misuse`. |
|
|
132
|
+
| `prepare(sql)` | **Single statement only**. Parses immediately; AST is reused. Pass binds as rest args to `run` / `all` / `get` / `result` on each call. |
|
|
133
|
+
| `transaction(fn)` | If idle: `BEGIN` → `fn()` → `COMMIT`, or `ROLLBACK` + rethrow. If already in a transaction: nested savepoint. Nested SQL `BEGIN` still errors. `close()` inside `fn` throws `misuse`. |
|
|
134
|
+
| `snapshot` / `restore` | Custom binary format (see below). |
|
|
135
|
+
| `close()` | Idempotent; rolls back an open SQL transaction; further ops throw `misuse`. Also available as `[Symbol.dispose]` when supported. |
|
|
136
|
+
|
|
137
|
+
SQL `BEGIN` / `COMMIT` / `ROLLBACK` / `SAVEPOINT` / `RELEASE` are first-class. Empty / comment-only SQL on `prepare` / `query` throws `misuse` (`empty statement`), matching SQLite prepare failure.
|
|
138
|
+
|
|
139
|
+
### Parameter binding
|
|
140
|
+
|
|
141
|
+
Supported styles: `?`, `?NNN`, `:name`, `@name`, `$name`.
|
|
142
|
+
|
|
143
|
+
- The JS API takes **rest args** (or a positional array into `query`) only — there is **no** sticky `bind()` and **no** `bind({ name: value })`.
|
|
144
|
+
- Named parameters occupy slots in **first-occurrence order**; repeated names share one slot.
|
|
145
|
+
- Prefixes are part of the name: `@x`, `$x`, and `:x` are **three different** parameters.
|
|
146
|
+
- Names are lowercased for lookup (`:Left` ≡ `:left`).
|
|
147
|
+
- Bindable: `null`, `string`, finite `number`, `bigint`, `boolean` → `0`/`1`, `Uint8Array` / `ArrayBuffer`.
|
|
148
|
+
- Rejected (`misuse`): `DataView`, typed-array views other than `Uint8Array`, `SharedArrayBuffer` / SAB-backed buffers.
|
|
149
|
+
- Rejected (`datatype_mismatch`): `undefined`, `Date`, plain objects, `NaN` / `Infinity`.
|
|
150
|
+
|
|
151
|
+
```ts
|
|
152
|
+
db.query(`SELECT ? AS a, :name AS b`, [1, "Alice"]);
|
|
153
|
+
db.prepare(`SELECT @id AS id`).get(42);
|
|
66
154
|
```
|
|
67
155
|
|
|
156
|
+
### Returned JavaScript types
|
|
157
|
+
|
|
158
|
+
| SQL storage | JS value | Notes |
|
|
159
|
+
| --- | --- | --- |
|
|
160
|
+
| NULL | `null` | Never `undefined` |
|
|
161
|
+
| INTEGER | `number` or `bigint` | `bigint` when outside `Number.MAX_SAFE_INTEGER` |
|
|
162
|
+
| REAL | `number` | Including integer-valued reals (`1.0` → `1`); use SQL `typeof()` to distinguish from INTEGER |
|
|
163
|
+
| TEXT | `string` | JSON subtype unwrapped to string |
|
|
164
|
+
| BLOB | `Uint8Array` | |
|
|
165
|
+
|
|
166
|
+
Duplicate column names collapse in row objects (last write wins). Use `stmt.result().values` for positional cells.
|
|
167
|
+
|
|
168
|
+
### Snapshots
|
|
169
|
+
|
|
170
|
+
- Format magic `SQLM` followed by an explicit little-endian format-version `u32` — **not** a portable `.sqlite` file and not loadable by the SQLite CLI.
|
|
171
|
+
- Round-trips ordinary tables, views, indexes, change counters, PRNG state, and clock.
|
|
172
|
+
- **Not** encoded: triggers, ATTACH’d schemas, virtual tables (FTS / RTREE / …), `userVersion`.
|
|
173
|
+
- Cannot `restore()` while a transaction is open.
|
|
174
|
+
- `restore()` replaces `now` with a fixed clock from the snapshot (a live `() => Date` is overwritten).
|
|
175
|
+
- Equivalent databases produce byte-identical snapshots (schema/rows sorted) **within a single library version**.
|
|
176
|
+
- **Compatibility policy:** newer library versions can always restore older snapshots; older libraries cannot restore newer format versions (`snapshot_version` / `SQLITE_FORMAT`). Corrupt magic yields a distinct error.
|
|
177
|
+
|
|
68
178
|
## Determinism
|
|
69
179
|
|
|
70
180
|
The engine is deterministic by default. Invariants:
|
|
71
181
|
|
|
72
182
|
| Source | Default | Override / notes |
|
|
73
183
|
| --- | --- | --- |
|
|
74
|
-
| `random()` / `randomblob()` | Seeded xorshift64* (`seed: 1`) | `new Database({ seed })`
|
|
184
|
+
| `random()` / `randomblob()` | Seeded xorshift64* (`seed: 1`) | `new Database({ seed })` |
|
|
75
185
|
| `date('now')` / friends | Fixed `2000-01-01T00:00:00.000Z` | `new Database({ now: Date \| (() => Date) })` |
|
|
76
186
|
| Table scans | Rowid order | Same order after `snapshot`/`restore` |
|
|
77
187
|
| Snapshots | Sorted schema/rows + PRNG state + clock | Restored into PRNG and `now` |
|
|
@@ -86,12 +196,53 @@ SQLITE_MEM_FUZZ_SEED=12345 bun test tests/fuzz
|
|
|
86
196
|
SQLITE_MEM_FUZZ_SEED=12345 SQLITE_MEM_FUZZ_PATH='0:1' bun test tests/fuzz # exact replay
|
|
87
197
|
```
|
|
88
198
|
|
|
199
|
+
## Stability policy
|
|
200
|
+
|
|
201
|
+
The exports of the main entry (`@crvouga/sqlite-mem`) are **frozen**:
|
|
202
|
+
|
|
203
|
+
- **Never** outside a major: removals, renames, signature changes, or changes to documented behavior of the stable surface.
|
|
204
|
+
- **Allowed in minors:** additions (new methods, new optional `DatabaseOptions` fields, new `ErrorCategory` values). Consumers that `switch` on `category` must include a default case — new categories may appear without a major bump.
|
|
205
|
+
- **`@crvouga/sqlite-mem/unstable`** is exempt from semver and may change or disappear in any release.
|
|
206
|
+
- **Snapshots:** newer library → can restore older blobs; older library → cannot restore newer format versions; byte-identical snapshot guarantee holds only within one library version.
|
|
207
|
+
|
|
208
|
+
## Compatibility notes for integrators
|
|
209
|
+
|
|
210
|
+
Goal: drop-in SQL behavior vs SQLite **3.51.0**. Full matrix: [COMPATIBILITY.md](COMPATIBILITY.md).
|
|
211
|
+
|
|
212
|
+
**Intentional differences:** custom `SQLM` snapshots; seeded `random()` / fixed `'now'`; no C API / on-disk DB / VFS.
|
|
213
|
+
|
|
214
|
+
**Know these thin or partial areas** (do not assume full oracle fidelity):
|
|
215
|
+
|
|
216
|
+
- FTS3/4/5 — largely implemented; shadow-table change counters intentionally diverge; some edges partial
|
|
217
|
+
- `EXPLAIN` / `EXPLAIN QUERY PLAN` — stub shapes, not real bytecode
|
|
218
|
+
- `INDEXED BY` / `NOT INDEXED` — parsed and discarded
|
|
219
|
+
- Unknown `PRAGMA` succeeds with an empty result (SQLite-like); storage/journal/WAL pragmas N/A or no-op
|
|
220
|
+
|
|
221
|
+
## Common pitfalls
|
|
222
|
+
|
|
223
|
+
1. **Do not `await`** — the API is sync.
|
|
224
|
+
2. **No named-object binds and no sticky `bind()`** — pass positional rest args / arrays in declaration order to `query` / `run` / `all` / `get` / `result`.
|
|
225
|
+
3. **`query` / `prepare` are single-statement only** — multi-statement scripts belong in `exec()` (which does not take bind parameters).
|
|
226
|
+
4. **`exec` returns `void` and takes no params** — use `db.prepare(…).run(…)` or `db.query(…)` for binds; use `db.changes` / `stmt.run()` for counters.
|
|
227
|
+
5. **`'now'` is not wall-clock** unless you pass `{ now: () => new Date() }`. Default is year 2000.
|
|
228
|
+
6. **`random()` is seeded**, not OS entropy; snapshots restore the PRNG.
|
|
229
|
+
7. **Snapshots are not `.sqlite` files** and do not round-trip FTS / triggers / ATTACH.
|
|
230
|
+
8. **No better-sqlite3 extras** — no `iterate`, `pluck`/`raw`, `safeIntegers` option, `pragma()` helper, `loadExtension`, or SQLite-file `serialize()`.
|
|
231
|
+
9. **Do not bind `Date` objects** — store unixepoch integers or ISO text. Do not bind `DataView` / non-`Uint8Array` typed arrays.
|
|
232
|
+
10. **Do not use `Number.isInteger` for SQL REAL vs INTEGER** — use SQL `typeof()`.
|
|
233
|
+
11. **Do not import `@crvouga/sqlite-mem/unstable` in application code** unless you accept breakage in any release.
|
|
234
|
+
|
|
235
|
+
Working examples beyond this README: `examples/react-vite`, `tests/contract/api/`, `tests/contract/parameters/`, `tests/browser/run.ts`.
|
|
236
|
+
|
|
89
237
|
## Development
|
|
90
238
|
|
|
91
|
-
Requires [Bun](https://bun.sh).
|
|
239
|
+
Requires [Bun](https://bun.sh). For architecture, change checklists, and how to add contract tests, see **[AGENTS.md](AGENTS.md)**.
|
|
240
|
+
|
|
241
|
+
Parity is proven only by differential contracts against real SQLite (`bun:sqlite`). Isolated internal unit tests are not SQLite compatibility proof.
|
|
92
242
|
|
|
93
243
|
```bash
|
|
94
244
|
bun install
|
|
245
|
+
bun run ci:local # same gates as GitHub Actions CI (except publish)
|
|
95
246
|
bun run check # format + lint + typecheck + sqlite-compat suite
|
|
96
247
|
bun run format # write Biome formatting
|
|
97
248
|
bun run lint # Biome lint
|
|
@@ -99,10 +250,10 @@ bun run typecheck
|
|
|
99
250
|
bun run test:sqlite-compat # requirements + inventory gate + differential suite
|
|
100
251
|
bun test # contract + fuzz + harness
|
|
101
252
|
bun run build
|
|
102
|
-
bun run test:browser # Playwright smoke (Chrome/Firefox/Safari)
|
|
253
|
+
bun run test:browser # Playwright smoke (Chrome/Firefox/Safari; CI uses Chromium)
|
|
103
254
|
```
|
|
104
255
|
|
|
105
|
-
|
|
256
|
+
See [COMPATIBILITY.md](./COMPATIBILITY.md).
|
|
106
257
|
|
|
107
258
|
## Releasing
|
|
108
259
|
|
|
@@ -137,8 +288,7 @@ PR titles must also follow Conventional Commits (enforced in CI). Prefer squash
|
|
|
137
288
|
Local checks:
|
|
138
289
|
|
|
139
290
|
```bash
|
|
140
|
-
|
|
141
|
-
bun run build && bun run verify-package
|
|
291
|
+
bun run ci:local # commitlint + quality + tests + browser + benchmarks
|
|
142
292
|
# dry-run needs a GitHub token for API calls; CI publish uses Trusted Publishing (no NPM_TOKEN)
|
|
143
293
|
bun run release:dry-run
|
|
144
294
|
```
|