@noorm/broccolidb 2.0.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (56) hide show
  1. package/LICENSE +21 -0
  2. package/README.md +219 -0
  3. package/dist/TokenCompressionService.d.ts +59 -0
  4. package/dist/TokenCompressionService.d.ts.map +1 -0
  5. package/dist/TokenCompressionService.js +179 -0
  6. package/dist/TokenCompressionService.js.map +1 -0
  7. package/dist/broccolidb-aggregation.d.ts +14 -0
  8. package/dist/broccolidb-aggregation.d.ts.map +1 -0
  9. package/dist/broccolidb-aggregation.js +158 -0
  10. package/dist/broccolidb-aggregation.js.map +1 -0
  11. package/dist/broccolidb-cas.d.ts +56 -0
  12. package/dist/broccolidb-cas.d.ts.map +1 -0
  13. package/dist/broccolidb-cas.js +285 -0
  14. package/dist/broccolidb-cas.js.map +1 -0
  15. package/dist/broccolidb-kernel.d.ts +63 -0
  16. package/dist/broccolidb-kernel.d.ts.map +1 -0
  17. package/dist/broccolidb-kernel.js +287 -0
  18. package/dist/broccolidb-kernel.js.map +1 -0
  19. package/dist/broccolidb-mutex.d.ts +37 -0
  20. package/dist/broccolidb-mutex.d.ts.map +1 -0
  21. package/dist/broccolidb-mutex.js +121 -0
  22. package/dist/broccolidb-mutex.js.map +1 -0
  23. package/dist/broccolidb-natural-query.d.ts +13 -0
  24. package/dist/broccolidb-natural-query.d.ts.map +1 -0
  25. package/dist/broccolidb-natural-query.js +188 -0
  26. package/dist/broccolidb-natural-query.js.map +1 -0
  27. package/dist/broccolidb-table.d.ts +62 -0
  28. package/dist/broccolidb-table.d.ts.map +1 -0
  29. package/dist/broccolidb-table.js +893 -0
  30. package/dist/broccolidb-table.js.map +1 -0
  31. package/dist/broccolidb-wal.d.ts +49 -0
  32. package/dist/broccolidb-wal.d.ts.map +1 -0
  33. package/dist/broccolidb-wal.js +168 -0
  34. package/dist/broccolidb-wal.js.map +1 -0
  35. package/dist/broccolidb.contracts.d.ts +232 -0
  36. package/dist/broccolidb.contracts.d.ts.map +1 -0
  37. package/dist/broccolidb.contracts.js +7 -0
  38. package/dist/broccolidb.contracts.js.map +1 -0
  39. package/dist/index.d.ts +17 -0
  40. package/dist/index.d.ts.map +1 -0
  41. package/dist/index.js +17 -0
  42. package/dist/index.js.map +1 -0
  43. package/docs/API.md +208 -0
  44. package/docs/ARCHITECTURE.md +160 -0
  45. package/docs/BRIEF.md +56 -0
  46. package/docs/CONTRIBUTING.md +82 -0
  47. package/docs/GLOSSARY.md +23 -0
  48. package/docs/OPERATIONS.md +176 -0
  49. package/docs/PHILOSOPHY.md +91 -0
  50. package/docs/README.md +116 -0
  51. package/docs/RELEASE_NOTES.md +34 -0
  52. package/docs/TROUBLESHOOTING.md +145 -0
  53. package/docs/adr/ADR-001-portable-inmemory-kernel.md +74 -0
  54. package/docs/adr/README.md +38 -0
  55. package/docs/adr/TEMPLATE.md +35 -0
  56. package/package.json +43 -0
@@ -0,0 +1,176 @@
1
+ # Operations guide
2
+
3
+ BroccoliDB is embedded in the host process. There is no daemon to start, but
4
+ there is still an explicit lifecycle and a durable state directory to operate.
5
+
6
+ ## Lifecycle
7
+
8
+ ```ts
9
+ const db = new BroccoliDatabaseKernel({
10
+ workspaceRoot: "/var/lib/my-app/state",
11
+ walDebounceMs: 20,
12
+ })
13
+
14
+ await db.start()
15
+ try {
16
+ // register tables, read state, and perform mutations
17
+ await db.transaction(async () => {
18
+ // related writes share one process-local lock
19
+ })
20
+ await db.flush()
21
+ } finally {
22
+ await db.stop()
23
+ }
24
+ ```
25
+
26
+ Rules:
27
+
28
+ 1. Start once before reading or writing.
29
+ 2. Keep one kernel owner per workspace root in a process.
30
+ 3. Use `transaction()` for related async mutations.
31
+ 4. Use `flush()` before reporting a durable handoff to another component.
32
+ 5. Use `checkpoint()` before long imports, migrations performed by the host, or
33
+ risky workflows.
34
+ 6. Always call `stop()` during normal shutdown.
35
+
36
+ `start()` and `stop()` are idempotent at the service level. A process crash can
37
+ still occur between an in-memory mutation and its asynchronous WAL append;
38
+ choose an explicit flush or transaction boundary when that window matters.
39
+
40
+ ## Workspace selection
41
+
42
+ `workspaceRoot` defaults to `process.cwd()`. For applications with multiple
43
+ projects, use an explicit absolute or application-resolved path so state does
44
+ not accidentally follow the launch directory.
45
+
46
+ ```ts
47
+ const db = new BroccoliDatabaseKernel({
48
+ workspaceRoot: path.join(appDataDir, "broccolidb"),
49
+ })
50
+ ```
51
+
52
+ Do not point two independent processes at the same workspace unless the host
53
+ provides an external single-writer or lease protocol. The built-in mutex only
54
+ coordinates async work inside one Node.js process.
55
+
56
+ ## On-disk layout
57
+
58
+ | Path | Purpose | Safe to edit manually? |
59
+ |---|---|---|
60
+ | `.broccolidb/wal.log` | Pending and historical mutation frames | No |
61
+ | `.broccolidb/wal.log.old` | Previous WAL after checkpoint rotation | No |
62
+ | `.broccolidb/checkpoint.db` | Latest base table snapshot | No |
63
+ | `.broccolidb/checkpoints/<id>.json` | Named checkpoint history | No |
64
+ | `.broccolidb/cas/blobs/<shard>/<hash>` | CAS payloads | No |
65
+ | `.broccolidb/cas/corrupt/` | Quarantined payloads and manifest | Preserve for investigation |
66
+
67
+ The directory is application state and should be included in the host's backup
68
+ policy. It is ignored by the package repository's `.gitignore` because it is
69
+ runtime data, not source.
70
+
71
+ ## Backup and restore
72
+
73
+ ### Consistent backup
74
+
75
+ 1. Stop the owning application, or quiesce all writes.
76
+ 2. Await `db.flush()` and preferably `db.checkpoint("backup-<label>")`.
77
+ 3. Copy the complete `.broccolidb/` directory to the backup target.
78
+ 4. Record the package version and Node.js version with the backup.
79
+ 5. Restart the application.
80
+
81
+ Copying only `checkpoint.db` omits checkpoint history, WAL state, and CAS blobs.
82
+ Copying only the CAS directory omits table records and references.
83
+
84
+ ### Restore
85
+
86
+ 1. Stop the application.
87
+ 2. Preserve the current state directory for forensic comparison.
88
+ 3. Restore the complete backup directory into the configured workspace root.
89
+ 4. Start the same or a compatible package version.
90
+ 5. Run `health()` and verify representative table and blob reads.
91
+
92
+ Do not delete an invalid WAL before preserving it. A checksum failure is useful
93
+ evidence about an interrupted write, manual mutation, or storage problem.
94
+
95
+ ## Checkpoints and rollback
96
+
97
+ Use checkpoints as application-level restore points:
98
+
99
+ ```ts
100
+ const beforeUpgrade = await db.checkpoint("before-upgrade")
101
+
102
+ try {
103
+ await runHostUpgrade(db)
104
+ } catch (error) {
105
+ const restored = await db.rollback(beforeUpgrade.checkpointId)
106
+ if (!restored) throw new Error("Upgrade failed and rollback was unavailable", { cause: error })
107
+ }
108
+ ```
109
+
110
+ `listCheckpoints()` reports records loaded or created in the current process.
111
+ After a restart, history files are available to `rollback(id)` when the ID is
112
+ known. Keep checkpoint IDs with the host's operation record if they are needed
113
+ for later recovery.
114
+
115
+ ## Health and observability
116
+
117
+ ```ts
118
+ const report = await db.health()
119
+
120
+ if (report.status !== "HEALTHY") {
121
+ console.warn(report.actionableRecommendations)
122
+ }
123
+ ```
124
+
125
+ The report covers:
126
+
127
+ - state-directory writability and estimated CAS disk usage;
128
+ - CAS blob counts, quarantine counts, and reported compression savings;
129
+ - WAL frame totals, buffered frame count, and last sync time;
130
+ - table count, record count, and current index-parity status.
131
+
132
+ The report is a fast operational probe. It does not replace a backup restore
133
+ test, a full application invariant check, or cross-process coordination.
134
+
135
+ ## WAL maintenance
136
+
137
+ The default WAL debounce is 20 ms. Increase it only when the host accepts a
138
+ larger durability window; decrease it when write visibility matters more than
139
+ batching. `checkpoint()` flushes before rotation and leaves a new checkpoint
140
+ marker in the WAL.
141
+
142
+ If WAL growth is persistent:
143
+
144
+ 1. Confirm the process is calling `flush()` or `stop()`.
145
+ 2. Inspect `health().pillars.walJournal`.
146
+ 3. Create a checkpoint after quiescing writes.
147
+ 4. Preserve `wal.log` and `wal.log.old` before manual intervention.
148
+
149
+ ## CAS maintenance
150
+
151
+ Store a blob by retaining its returned hash in a record, conventionally as a
152
+ string such as `CAS:<hash>`:
153
+
154
+ ```ts
155
+ const hash = await db.storeBlob(Buffer.from(payload))
156
+ table.put("document-1", { id: "document-1", blob: `CAS:${hash}` })
157
+ const bytes = await db.readBlob(hash)
158
+ ```
159
+
160
+ `gc()` scans current table values for `CAS:` references and removes all other
161
+ blob files. Only call it when all durable references are represented in the
162
+ currently loaded tables. External manifests, pending imports, and backups are
163
+ not discovered automatically.
164
+
165
+ When CAS verification fails, the payload is moved under `cas/corrupt/` and an
166
+ entry is appended to `manifest.jsonl`. Preserve the quarantine directory before
167
+ attempting a restore.
168
+
169
+ ## Security and data handling
170
+
171
+ - Choose a state directory with permissions appropriate for the records stored.
172
+ - Do not put secrets in natural-language queries, error messages, or checkpoint
173
+ labels; labels are persisted in checkpoint history.
174
+ - Treat WAL, checkpoint, CAS, and quarantine files as sensitive application data.
175
+ - Validate record content before persistence; BroccoliDB does not encrypt files.
176
+ - Use an external lock/lease when more than one process can write the same root.
@@ -0,0 +1,91 @@
1
+ # Design philosophy
2
+
3
+ BroccoliDB is intentionally smaller than a general-purpose database. Its design
4
+ optimizes for an embeddable application boundary where the owner can see the
5
+ tables, choose the workspace path, decide when to flush, and recover from
6
+ ordinary files.
7
+
8
+ ## Principles
9
+
10
+ ### 1. Memory is the hot path; durability is explicit
11
+
12
+ Table reads and indexes stay in memory. The WAL, checkpoint, and CAS layers are
13
+ explicit durability mechanisms rather than hidden database behavior. Callers
14
+ should know which boundary they have crossed:
15
+
16
+ | Boundary | Meaning |
17
+ |---|---|
18
+ | `put()`/`delete()` returned | The in-memory table changed; the WAL append was scheduled |
19
+ | `await db.flush()` | Buffered WAL frames were written |
20
+ | `await db.transaction(fn)` returned | The callback completed and the kernel flushed its WAL buffer |
21
+ | `await db.checkpoint()` returned | A base snapshot and checkpoint history were written and the WAL was rotated |
22
+ | `await db.stop()` returned | Kernel subsystems were flushed and stopped |
23
+
24
+ ### 2. Files should be boring
25
+
26
+ The durable format uses JSON, JSONL, SHA-256, temporary files, and rename. This
27
+ makes state inspectable and transferable without a native reader. The trade-off
28
+ is that applications should not expect SQL joins, page-level indexes, or a
29
+ server-grade transaction log.
30
+
31
+ ### 3. Integrity belongs at the boundary
32
+
33
+ WAL frames carry checksums and previous-frame metadata. CAS payloads are verified
34
+ against their requested SHA-256 address on read. Checkpoint snapshots include a
35
+ hash. Corrupt data should fail visibly or be quarantined; it should not silently
36
+ be treated as valid state.
37
+
38
+ ### 4. Contracts are more stable than implementation details
39
+
40
+ Consumers program against `IDbTable`, `IBroccoliDatabaseKernel`, query types,
41
+ aggregate types, and change events. Internal index maps and file helpers may
42
+ change, but public signatures and documented lifecycle semantics require review.
43
+
44
+ ### 5. Portability beats incidental compatibility
45
+
46
+ BroccoliDB does not preserve the removed SQLite package's SQL or driver API. The
47
+ supported path is the standalone `@noorm/broccolidb` package. Adapters should
48
+ translate application queries into the typed table/query contracts instead of
49
+ reintroducing a private compatibility layer.
50
+
51
+ ### 6. Operational behavior must be legible
52
+
53
+ Health reports, checkpoint records, WAL metrics, CAS stats, and explicit error
54
+ types are preferable to opaque magic. The health report is a diagnostic probe,
55
+ not a substitute for an application-specific audit or backup test.
56
+
57
+ ## Rejected alternatives
58
+
59
+ ### Native SQLite binding as the package core
60
+
61
+ Rejected for this package because it adds native installation and ABI concerns.
62
+ Applications that need SQL should choose a dedicated SQL database; BroccoliDB is
63
+ the portable table substrate.
64
+
65
+ ### A remote database service
66
+
67
+ Rejected because the package is designed for local, offline, process-owned state
68
+ and should not introduce networking, credentials, or service discovery.
69
+
70
+ ### Implicit persistence on every mutation
71
+
72
+ Rejected because it couples latency and correctness decisions to an invisible
73
+ policy. Micro-batching plus explicit `flush()`, `transaction()`, and
74
+ `checkpoint()` calls make the boundary visible to the caller.
75
+
76
+ ### Treating token estimates as billing truth
77
+
78
+ Rejected in `TokenCompressionService`. The four-characters-per-token estimate is
79
+ a fast budget signal; provider usage accounting remains the provider's concern.
80
+
81
+ ## Boundaries
82
+
83
+ - The async mutex is process-local and re-entrant through async context.
84
+ - The kernel does not coordinate independent Node processes writing the same
85
+ workspace.
86
+ - WAL replay validates frame checksums and reconstructs table mutations; it is
87
+ not a general migration engine.
88
+ - CAS garbage collection is only safe when the referenced-hash set accurately
89
+ reflects application records.
90
+ - Health probes report the implementation's current checks; they are not a
91
+ full filesystem scrub or cross-process consistency protocol.
package/docs/README.md ADDED
@@ -0,0 +1,116 @@
1
+ # BroccoliDB documentation
2
+
3
+ This is the documentation map for the standalone `@noorm/broccolidb` package.
4
+ The package docs follow a layered structure used by mature storage systems:
5
+
6
+ **Concepts → How it works → Reference → Operations → Decisions**
7
+
8
+ The implementation and tests are authoritative when documentation disagrees.
9
+ This map explains where a claim belongs and gives each reader a shortest useful
10
+ path through the package.
11
+
12
+ ## Reading paths by role
13
+
14
+ | Role | Start here | Then read |
15
+ |---|---|---|
16
+ | **Application developer** | [Quick start](../README.md#quick-start) | [API reference](API.md) · [Operations](OPERATIONS.md) |
17
+ | **Architect / reviewer** | [Brief](BRIEF.md) | [Architecture](ARCHITECTURE.md) · [Philosophy](PHILOSOPHY.md) · [ADR-001](adr/ADR-001-portable-inmemory-kernel.md) |
18
+ | **Operator / support** | [Operations](OPERATIONS.md) | [Troubleshooting](TROUBLESHOOTING.md) · [Glossary](GLOSSARY.md) |
19
+ | **Contributor** | [Contributing](CONTRIBUTING.md) | [Architecture](ARCHITECTURE.md) · [API reference](API.md) · [ADR process](adr/README.md) |
20
+ | **Auditor / security reviewer** | [Boundaries](PHILOSOPHY.md#boundaries) | [Operations](OPERATIONS.md#integrity-and-recovery) · [Architecture](ARCHITECTURE.md#failure-model) |
21
+ | **Release owner** | [Release notes](RELEASE_NOTES.md) | [Contributing](CONTRIBUTING.md#release-checklist) · [ADR index](adr/README.md) |
22
+
23
+ ## Document catalog
24
+
25
+ ### Concepts — why
26
+
27
+ | Document | Purpose |
28
+ |---|---|
29
+ | [Brief](BRIEF.md) | One-page problem statement, solution, guarantees, and non-goals |
30
+ | [Philosophy](PHILOSOPHY.md) | Design principles, explicit trade-offs, and rejected alternatives |
31
+ | [Glossary](GLOSSARY.md) | Canonical terms for tables, WAL, CAS, checkpoints, and recovery |
32
+
33
+ ### How it works — what happens
34
+
35
+ | Document | Purpose |
36
+ |---|---|
37
+ | [Architecture](ARCHITECTURE.md) | Runtime layers, mutation flow, startup, checkpointing, rollback, and failure model |
38
+ | [Operations](OPERATIONS.md) | Filesystem layout, lifecycle rules, backup/restore, integrity, and garbage collection |
39
+
40
+ ### Reference — what to call
41
+
42
+ | Document | Purpose |
43
+ |---|---|
44
+ | [API reference](API.md) | Public exports, kernel/table methods, query operators, indexes, aggregation, and events |
45
+ | [Package README](../README.md) | Install, minimal example, portability statement, and command index |
46
+ | [Source entry point](../src/index.ts) | Export surface — the source of truth for package imports |
47
+ | [Contracts](../src/broccolidb.contracts.ts) | Type-level API and serialized record contracts |
48
+
49
+ ### Operations — how to run and debug
50
+
51
+ | Document | Purpose |
52
+ |---|---|
53
+ | [Operations guide](OPERATIONS.md) | Start/stop, durability boundaries, backup, recovery, health, and CAS GC |
54
+ | [Troubleshooting](TROUBLESHOOTING.md) | Symptom → likely cause → action runbooks |
55
+ | [Release notes](RELEASE_NOTES.md) | Supported release history and compatibility policy |
56
+
57
+ ### Decisions — why the shape is stable
58
+
59
+ | Document | Purpose |
60
+ |---|---|
61
+ | [ADR index](adr/README.md) | Decision inventory and writing rules |
62
+ | [ADR-001](adr/ADR-001-portable-inmemory-kernel.md) | Why BroccoliDB is a portable in-memory kernel with explicit file durability |
63
+
64
+ ## Documentation conventions
65
+
66
+ 1. **Describe the current package.** Do not document the removed in-repository
67
+ SQLite implementation or imply that it remains supported.
68
+ 2. **Use contract language.** Distinguish what is guaranteed after an in-memory
69
+ mutation, after `flush()`, after `checkpoint()`, and after `stop()`.
70
+ 3. **Name the source of truth.** Public exports live in `src/index.ts`; type
71
+ contracts live in `src/broccolidb.contracts.ts`; behavior is verified by
72
+ `test/`.
73
+ 4. **Keep examples executable.** Examples should use the public package import,
74
+ explicit lifecycle calls, and a temporary or application-owned workspace.
75
+ 5. **Document failure boundaries.** Say what is process-local, what is on disk,
76
+ what is recoverable, and what requires an external coordination system.
77
+ 6. **Prefer stable vocabulary.** Say table, record, WAL frame, checkpoint, CAS
78
+ blob, replay, rollback, flush, and health report consistently.
79
+ 7. **Update docs with contracts.** Export changes, file-format changes, and
80
+ durability changes require API/operations updates plus an ADR or release note.
81
+
82
+ ## Source-of-truth matrix
83
+
84
+ | Question | Source of truth |
85
+ |---|---|
86
+ | What can consumers import? | `src/index.ts` and generated `dist/index.d.ts` |
87
+ | What does a method accept/return? | `src/broccolidb.contracts.ts` and class signatures |
88
+ | How does a write reach disk? | `src/broccolidb-kernel.ts` and `src/broccolidb-wal.ts` |
89
+ | How are blobs addressed and verified? | `src/broccolidb-cas.ts` |
90
+ | What behavior is protected? | `test/*.test.ts` |
91
+ | Is the package publishable? | `package.json`, `npm pack --dry-run`, and `npm run docs:check` |
92
+
93
+ ## Status
94
+
95
+ | Metric | Current value |
96
+ |---|---|
97
+ | Package | `@noorm/broccolidb@2.0.1` |
98
+ | API status | Standalone supported package; modern ESM surface |
99
+ | Runtime dependencies | 0 |
100
+ | Node.js | `>=18` |
101
+ | Public entry point | `src/index.ts` / `dist/index.js` |
102
+ | Persistence | In-memory tables + optional filesystem WAL/checkpoints/CAS |
103
+ | License | MIT |
104
+
105
+ ## Quick links
106
+
107
+ ```bash
108
+ # Build and test the implementation
109
+ npm test
110
+
111
+ # Validate docs and relative Markdown links
112
+ npm run docs:check
113
+
114
+ # Inspect the publishable artifact without creating it
115
+ npm pack --dry-run
116
+ ```
@@ -0,0 +1,34 @@
1
+ # Release notes
2
+
3
+ ## 2.0.1 — first npm publication
4
+
5
+ This patch release publishes the standalone package under a new version because
6
+ the previously unused `2.0.0` version had already been unpublished on npm and
7
+ cannot be reused by the registry.
8
+
9
+ The package contents and supported API are unchanged from the standalone
10
+ release described below.
11
+
12
+ ## 2.0.0 — standalone portable package
13
+
14
+ This release is the supported standalone BroccoliDB package.
15
+
16
+ ### Included
17
+
18
+ - in-memory typed tables with equality, sorted, composite, and prefix indexes;
19
+ - operator filters, boolean query clauses, fluent queries, aggregation, CDC,
20
+ TTL expiration, and deterministic natural-language parsing;
21
+ - checksum-linked micro-batched WAL and restart replay;
22
+ - atomic JSON checkpoints, named history, and rollback;
23
+ - SHA-256 CAS storage with optional Brotli compression and corruption quarantine;
24
+ - re-entrant async mutex and kernel transaction boundary;
25
+ - dependency-free runtime package for Node.js `>=18`.
26
+
27
+ ### Compatibility policy
28
+
29
+ The standalone package is the supported implementation. The removed SQLite
30
+ implementation is not a supported compatibility target, and this package does
31
+ not promise SQL, Kysely, or `better-sqlite3` API compatibility.
32
+
33
+ Changes to exported contracts or durable file formats require a release-note
34
+ entry and an architecture decision record.
@@ -0,0 +1,145 @@
1
+ # Troubleshooting
2
+
3
+ Use the symptom → likely cause → action flow below. Preserve `.broccolidb/`
4
+ before destructive recovery work.
5
+
6
+ ## Startup and recovery
7
+
8
+ ### `start()` throws `WalIntegrityError`
9
+
10
+ **Likely causes:** a truncated JSONL write, hand-edited WAL, filesystem
11
+ corruption, or a package/version mismatch that changed frame serialization.
12
+
13
+ **Action:** stop all writers, copy the entire `.broccolidb/` directory, inspect
14
+ the failing line and checksum message, and restore the last known-good backup if
15
+ the WAL cannot be repaired. Do not delete the WAL as a first response.
16
+
17
+ ### Data is missing after a process crash
18
+
19
+ **Likely cause:** a mutation changed memory but its asynchronous WAL append had
20
+ not been flushed before termination.
21
+
22
+ **Action:** use `transaction()`, `flush()`, or `checkpoint()` at the host's
23
+ durability boundary. BroccoliDB cannot recover a frame that never reached the
24
+ WAL.
25
+
26
+ ### Startup is slow
27
+
28
+ **Likely causes:** a large WAL tail, a large checkpoint, or too many records
29
+ being rebuilt into indexes.
30
+
31
+ **Action:** measure startup with the same workload, inspect WAL metrics, quiesce
32
+ writes, and create a checkpoint. Keep tables bounded or partition application
33
+ state when a single in-memory table becomes too large.
34
+
35
+ ## Tables and queries
36
+
37
+ ### A query is correct but slower than expected
38
+
39
+ **Likely cause:** no matching index exists or the predicate cannot use the
40
+ available index shape.
41
+
42
+ **Action:** call `select().where(...).explain()`, inspect `scanStrategy`, and add
43
+ an equality, sorted, composite, or prefix index that matches the hot predicate.
44
+ Do not assume an index changes semantics; it changes candidate selection.
45
+
46
+ ### An index appears stale
47
+
48
+ **Likely causes:** records were restored with a custom snapshot, a host kept a
49
+ second table reference, or application code mutated an object after passing it
50
+ to the table.
51
+
52
+ **Action:** treat records as immutable application values, use table mutation
53
+ methods, and rebuild the table/indexes through a controlled restore if needed.
54
+
55
+ ### TTL behavior is surprising
56
+
57
+ **Likely cause:** TTL timers are process-local and depend on the process staying
58
+ alive; an expired record is not a durable scheduler job.
59
+
60
+ **Action:** use TTL for local cache-like expiration. For business deadlines,
61
+ persist an explicit expiration field and run a host-owned reconciliation pass.
62
+
63
+ ## Checkpoints and rollback
64
+
65
+ ### `rollback(id)` returns `false`
66
+
67
+ **Likely causes:** the ID is unknown, history was not copied with the backup, or
68
+ the checkpoint file is unreadable.
69
+
70
+ **Action:** call `listCheckpoints()`, verify
71
+ `.broccolidb/checkpoints/<id>.json`, and restore the complete state directory if
72
+ the file is missing.
73
+
74
+ ### Rollback did not remove a table created later
75
+
76
+ **Likely cause:** rollback rebuilds tables represented by the checkpoint; it is
77
+ not a schema registry that automatically deletes every table absent from a
78
+ historical snapshot.
79
+
80
+ **Action:** explicitly clear or recreate application tables as part of the
81
+ rollback procedure, then checkpoint the resulting state.
82
+
83
+ ## CAS
84
+
85
+ ### `readBlob(hash)` returns `null`
86
+
87
+ **Likely cause:** the hash is absent from the configured workspace or the host
88
+ opened a different `workspaceRoot`.
89
+
90
+ **Action:** log the resolved root, compare `cas.getBaseDir()`, and restore the
91
+ CAS directory from the matching backup.
92
+
93
+ ### `StorageIntegrityError` is raised
94
+
95
+ **Likely cause:** a payload failed Brotli decompression or SHA-256 verification.
96
+
97
+ **Action:** preserve `cas/corrupt/manifest.jsonl`, inspect the quarantined blob,
98
+ and restore a known-good backup. Do not overwrite the quarantine entry before
99
+ capturing it.
100
+
101
+ ### `gc()` deleted data needed by the application
102
+
103
+ **Likely cause:** the reference was not stored in a currently loaded table as a
104
+ `CAS:<hash>` string.
105
+
106
+ **Action:** restore from backup and make references explicit before rerunning GC.
107
+ `gc()` is conservative only with respect to the references it can see.
108
+
109
+ ## Locking and concurrency
110
+
111
+ ### `DeadlockTimeoutError`
112
+
113
+ **Likely causes:** a callback awaited a long-running operation while holding the
114
+ lock, nested operations are waiting on another resource, or two application
115
+ components use locks in inconsistent order.
116
+
117
+ **Action:** keep transaction callbacks short, avoid external network calls while
118
+ holding the kernel lock, and review lock ordering. The mutex is re-entrant for
119
+ the same async context, but it is not a general deadlock solver.
120
+
121
+ ### Two processes see divergent state
122
+
123
+ **Likely cause:** both processes wrote the same `.broccolidb/` directory. The
124
+ package does not provide cross-process fencing.
125
+
126
+ **Action:** assign one writer, put an external lease around the workspace, or
127
+ choose a database designed for multi-process coordination.
128
+
129
+ ## Packaging and imports
130
+
131
+ ### `ERR_MODULE_NOT_FOUND` for internal imports
132
+
133
+ **Likely cause:** the package was copied without building `dist/`, or a consumer
134
+ is bypassing the package export and importing source files incorrectly.
135
+
136
+ **Action:** run `npm run build`, import from `@noorm/broccolidb`, and inspect the
137
+ `files` list with `npm pack --dry-run`.
138
+
139
+ ### A native SQLite package appears in the dependency tree
140
+
141
+ **Likely cause:** the host application installed an unrelated database package
142
+ or a stale lockfile was copied into the package.
143
+
144
+ **Action:** run `npm ls --omit=dev --depth=0`, inspect the host's dependency
145
+ graph, and keep BroccoliDB's runtime `dependencies` object empty.
@@ -0,0 +1,74 @@
1
+ # ADR-001: Portable in-memory kernel with explicit filesystem durability
2
+
3
+ - **Status:** Accepted
4
+ - **Date:** 2026-09-20
5
+ - **Owners:** BroccoliDB maintainers
6
+ - **Scope:** package architecture, runtime dependencies, and persistence boundary
7
+
8
+ ## Context
9
+
10
+ BroccoliDB is embedded in applications that need fast local tables, indexes, and
11
+ recoverable state. The previous SQLite-oriented implementation introduced a
12
+ native database dependency and a portability burden for consumers that only
13
+ needed table operations and local durability.
14
+
15
+ The standalone package must work across supported Node.js environments without a
16
+ native compiler, Electron ABI rebuild, database server, or SQL-specific adapter.
17
+
18
+ ## Decision
19
+
20
+ Use an in-memory table kernel as the supported implementation and compose
21
+ durability from ordinary filesystem primitives:
22
+
23
+ - typed `Map`-backed tables and secondary indexes for the hot path;
24
+ - a micro-batched JSONL WAL with checksum metadata for mutation replay;
25
+ - atomic JSON checkpoints and named checkpoint history for restore points;
26
+ - SHA-256 content-addressable files with optional Brotli compression for blobs;
27
+ - a process-local re-entrant async mutex for kernel transactions;
28
+ - zero production dependencies and an ESM Node.js `>=18` package surface.
29
+
30
+ The public contract is the package export from `src/index.ts` and the interfaces
31
+ in `src/broccolidb.contracts.ts`. The removed SQLite implementation is not a
32
+ compatibility target.
33
+
34
+ ## Alternatives considered
35
+
36
+ ### Keep `better-sqlite3` in the package
37
+
38
+ Rejected. It creates native installation/ABI coupling and makes a small embedded
39
+ table substrate harder to transfer between environments.
40
+
41
+ ### Use Kysely over a database driver
42
+
43
+ Rejected. A query builder does not remove the underlying driver and would force
44
+ the package to preserve SQL/dialect semantics it does not need.
45
+
46
+ ### Use an external service
47
+
48
+ Rejected. The package is intended for offline, process-owned local state and
49
+ should not add networking, credentials, deployment, or service availability
50
+ requirements.
51
+
52
+ ## Consequences
53
+
54
+ ### Positive
55
+
56
+ - Consumers install and run without native database builds.
57
+ - State is inspectable and transferable as ordinary files.
58
+ - The hot path is simple and fast for bounded local tables.
59
+ - Recovery behavior is explicit and testable.
60
+ - The package can be vendored or published without LUMI-specific aliases.
61
+
62
+ ### Trade-offs
63
+
64
+ - Tables must fit in process memory.
65
+ - There is no SQL compatibility, migration engine, or relational join planner.
66
+ - The built-in mutex does not coordinate independent processes.
67
+ - Durability is explicit; an unflushed process crash can lose the latest buffer.
68
+ - Applications own backup policy, schema evolution, and record validation.
69
+
70
+ ## Compatibility impact
71
+
72
+ Changing the WAL frame shape, checkpoint structure, CAS encoding, export surface,
73
+ or lifecycle semantics is a compatibility change. Such changes require tests,
74
+ operations documentation, release notes, and a new or updated ADR.
@@ -0,0 +1,38 @@
1
+ # Architecture decision records
2
+
3
+ Architecture Decision Records (ADRs) capture decisions that affect the public
4
+ API, durable formats, portability boundary, or recovery behavior. They are
5
+ shorter than a whitepaper and more durable than a changelog entry.
6
+
7
+ ## Index
8
+
9
+ | ID | Decision | Status |
10
+ |---|---|---|
11
+ | [ADR-001](ADR-001-portable-inmemory-kernel.md) | Use a portable in-memory table kernel with explicit filesystem durability | Accepted |
12
+
13
+ ## When to write an ADR
14
+
15
+ Write or update an ADR when a change:
16
+
17
+ - adds/removes a public export;
18
+ - changes WAL, checkpoint, CAS, or recovery semantics;
19
+ - changes the runtime dependency or native-module policy;
20
+ - changes concurrency or cross-process assumptions;
21
+ - introduces a new persistence format or compatibility decision.
22
+
23
+ Do not use an ADR for a typo, a local refactor, or a test-only fixture change.
24
+
25
+ ## Lifecycle
26
+
27
+ 1. Create the next numbered file from [the template](TEMPLATE.md).
28
+ 2. State context, decision, alternatives, consequences, and compatibility impact.
29
+ 3. Link it from this index and the affected reference/operations documents.
30
+ 4. Mark it `Accepted`, `Superseded`, or `Rejected`; do not silently rewrite the
31
+ historical decision.
32
+
33
+ ## Status vocabulary
34
+
35
+ - **Proposed** — under review and not yet a supported contract.
36
+ - **Accepted** — current supported decision.
37
+ - **Superseded** — replaced by a later ADR; retain for history.
38
+ - **Rejected** — considered and declined; retain the reasoning.