@noorm/broccolidb 2.0.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.md +219 -0
- package/dist/TokenCompressionService.d.ts +59 -0
- package/dist/TokenCompressionService.d.ts.map +1 -0
- package/dist/TokenCompressionService.js +179 -0
- package/dist/TokenCompressionService.js.map +1 -0
- package/dist/broccolidb-aggregation.d.ts +14 -0
- package/dist/broccolidb-aggregation.d.ts.map +1 -0
- package/dist/broccolidb-aggregation.js +158 -0
- package/dist/broccolidb-aggregation.js.map +1 -0
- package/dist/broccolidb-cas.d.ts +56 -0
- package/dist/broccolidb-cas.d.ts.map +1 -0
- package/dist/broccolidb-cas.js +285 -0
- package/dist/broccolidb-cas.js.map +1 -0
- package/dist/broccolidb-kernel.d.ts +63 -0
- package/dist/broccolidb-kernel.d.ts.map +1 -0
- package/dist/broccolidb-kernel.js +287 -0
- package/dist/broccolidb-kernel.js.map +1 -0
- package/dist/broccolidb-mutex.d.ts +37 -0
- package/dist/broccolidb-mutex.d.ts.map +1 -0
- package/dist/broccolidb-mutex.js +121 -0
- package/dist/broccolidb-mutex.js.map +1 -0
- package/dist/broccolidb-natural-query.d.ts +13 -0
- package/dist/broccolidb-natural-query.d.ts.map +1 -0
- package/dist/broccolidb-natural-query.js +188 -0
- package/dist/broccolidb-natural-query.js.map +1 -0
- package/dist/broccolidb-table.d.ts +62 -0
- package/dist/broccolidb-table.d.ts.map +1 -0
- package/dist/broccolidb-table.js +893 -0
- package/dist/broccolidb-table.js.map +1 -0
- package/dist/broccolidb-wal.d.ts +49 -0
- package/dist/broccolidb-wal.d.ts.map +1 -0
- package/dist/broccolidb-wal.js +168 -0
- package/dist/broccolidb-wal.js.map +1 -0
- package/dist/broccolidb.contracts.d.ts +232 -0
- package/dist/broccolidb.contracts.d.ts.map +1 -0
- package/dist/broccolidb.contracts.js +7 -0
- package/dist/broccolidb.contracts.js.map +1 -0
- package/dist/index.d.ts +17 -0
- package/dist/index.d.ts.map +1 -0
- package/dist/index.js +17 -0
- package/dist/index.js.map +1 -0
- package/docs/API.md +208 -0
- package/docs/ARCHITECTURE.md +160 -0
- package/docs/BRIEF.md +56 -0
- package/docs/CONTRIBUTING.md +82 -0
- package/docs/GLOSSARY.md +23 -0
- package/docs/OPERATIONS.md +176 -0
- package/docs/PHILOSOPHY.md +91 -0
- package/docs/README.md +116 -0
- package/docs/RELEASE_NOTES.md +34 -0
- package/docs/TROUBLESHOOTING.md +145 -0
- package/docs/adr/ADR-001-portable-inmemory-kernel.md +74 -0
- package/docs/adr/README.md +38 -0
- package/docs/adr/TEMPLATE.md +35 -0
- package/package.json +43 -0
|
@@ -0,0 +1,176 @@
|
|
|
1
|
+
# Operations guide
|
|
2
|
+
|
|
3
|
+
BroccoliDB is embedded in the host process. There is no daemon to start, but
|
|
4
|
+
there is still an explicit lifecycle and a durable state directory to operate.
|
|
5
|
+
|
|
6
|
+
## Lifecycle
|
|
7
|
+
|
|
8
|
+
```ts
|
|
9
|
+
const db = new BroccoliDatabaseKernel({
|
|
10
|
+
workspaceRoot: "/var/lib/my-app/state",
|
|
11
|
+
walDebounceMs: 20,
|
|
12
|
+
})
|
|
13
|
+
|
|
14
|
+
await db.start()
|
|
15
|
+
try {
|
|
16
|
+
// register tables, read state, and perform mutations
|
|
17
|
+
await db.transaction(async () => {
|
|
18
|
+
// related writes share one process-local lock
|
|
19
|
+
})
|
|
20
|
+
await db.flush()
|
|
21
|
+
} finally {
|
|
22
|
+
await db.stop()
|
|
23
|
+
}
|
|
24
|
+
```
|
|
25
|
+
|
|
26
|
+
Rules:
|
|
27
|
+
|
|
28
|
+
1. Start once before reading or writing.
|
|
29
|
+
2. Keep one kernel owner per workspace root in a process.
|
|
30
|
+
3. Use `transaction()` for related async mutations.
|
|
31
|
+
4. Use `flush()` before reporting a durable handoff to another component.
|
|
32
|
+
5. Use `checkpoint()` before long imports, migrations performed by the host, or
|
|
33
|
+
risky workflows.
|
|
34
|
+
6. Always call `stop()` during normal shutdown.
|
|
35
|
+
|
|
36
|
+
`start()` and `stop()` are idempotent at the service level. A process crash can
|
|
37
|
+
still occur between an in-memory mutation and its asynchronous WAL append;
|
|
38
|
+
choose an explicit flush or transaction boundary when that window matters.
|
|
39
|
+
|
|
40
|
+
## Workspace selection
|
|
41
|
+
|
|
42
|
+
`workspaceRoot` defaults to `process.cwd()`. For applications with multiple
|
|
43
|
+
projects, use an explicit absolute or application-resolved path so state does
|
|
44
|
+
not accidentally follow the launch directory.
|
|
45
|
+
|
|
46
|
+
```ts
|
|
47
|
+
const db = new BroccoliDatabaseKernel({
|
|
48
|
+
workspaceRoot: path.join(appDataDir, "broccolidb"),
|
|
49
|
+
})
|
|
50
|
+
```
|
|
51
|
+
|
|
52
|
+
Do not point two independent processes at the same workspace unless the host
|
|
53
|
+
provides an external single-writer or lease protocol. The built-in mutex only
|
|
54
|
+
coordinates async work inside one Node.js process.
|
|
55
|
+
|
|
56
|
+
## On-disk layout
|
|
57
|
+
|
|
58
|
+
| Path | Purpose | Safe to edit manually? |
|
|
59
|
+
|---|---|---|
|
|
60
|
+
| `.broccolidb/wal.log` | Pending and historical mutation frames | No |
|
|
61
|
+
| `.broccolidb/wal.log.old` | Previous WAL after checkpoint rotation | No |
|
|
62
|
+
| `.broccolidb/checkpoint.db` | Latest base table snapshot | No |
|
|
63
|
+
| `.broccolidb/checkpoints/<id>.json` | Named checkpoint history | No |
|
|
64
|
+
| `.broccolidb/cas/blobs/<shard>/<hash>` | CAS payloads | No |
|
|
65
|
+
| `.broccolidb/cas/corrupt/` | Quarantined payloads and manifest | Preserve for investigation |
|
|
66
|
+
|
|
67
|
+
The directory is application state and should be included in the host's backup
|
|
68
|
+
policy. It is ignored by the package repository's `.gitignore` because it is
|
|
69
|
+
runtime data, not source.
|
|
70
|
+
|
|
71
|
+
## Backup and restore
|
|
72
|
+
|
|
73
|
+
### Consistent backup
|
|
74
|
+
|
|
75
|
+
1. Stop the owning application, or quiesce all writes.
|
|
76
|
+
2. Await `db.flush()` and preferably `db.checkpoint("backup-<label>")`.
|
|
77
|
+
3. Copy the complete `.broccolidb/` directory to the backup target.
|
|
78
|
+
4. Record the package version and Node.js version with the backup.
|
|
79
|
+
5. Restart the application.
|
|
80
|
+
|
|
81
|
+
Copying only `checkpoint.db` omits checkpoint history, WAL state, and CAS blobs.
|
|
82
|
+
Copying only the CAS directory omits table records and references.
|
|
83
|
+
|
|
84
|
+
### Restore
|
|
85
|
+
|
|
86
|
+
1. Stop the application.
|
|
87
|
+
2. Preserve the current state directory for forensic comparison.
|
|
88
|
+
3. Restore the complete backup directory into the configured workspace root.
|
|
89
|
+
4. Start the same or a compatible package version.
|
|
90
|
+
5. Run `health()` and verify representative table and blob reads.
|
|
91
|
+
|
|
92
|
+
Do not delete an invalid WAL before preserving it. A checksum failure is useful
|
|
93
|
+
evidence about an interrupted write, manual mutation, or storage problem.
|
|
94
|
+
|
|
95
|
+
## Checkpoints and rollback
|
|
96
|
+
|
|
97
|
+
Use checkpoints as application-level restore points:
|
|
98
|
+
|
|
99
|
+
```ts
|
|
100
|
+
const beforeUpgrade = await db.checkpoint("before-upgrade")
|
|
101
|
+
|
|
102
|
+
try {
|
|
103
|
+
await runHostUpgrade(db)
|
|
104
|
+
} catch (error) {
|
|
105
|
+
const restored = await db.rollback(beforeUpgrade.checkpointId)
|
|
106
|
+
if (!restored) throw new Error("Upgrade failed and rollback was unavailable", { cause: error })
|
|
107
|
+
}
|
|
108
|
+
```
|
|
109
|
+
|
|
110
|
+
`listCheckpoints()` reports records loaded or created in the current process.
|
|
111
|
+
After a restart, history files are available to `rollback(id)` when the ID is
|
|
112
|
+
known. Keep checkpoint IDs with the host's operation record if they are needed
|
|
113
|
+
for later recovery.
|
|
114
|
+
|
|
115
|
+
## Health and observability
|
|
116
|
+
|
|
117
|
+
```ts
|
|
118
|
+
const report = await db.health()
|
|
119
|
+
|
|
120
|
+
if (report.status !== "HEALTHY") {
|
|
121
|
+
console.warn(report.actionableRecommendations)
|
|
122
|
+
}
|
|
123
|
+
```
|
|
124
|
+
|
|
125
|
+
The report covers:
|
|
126
|
+
|
|
127
|
+
- state-directory writability and estimated CAS disk usage;
|
|
128
|
+
- CAS blob counts, quarantine counts, and reported compression savings;
|
|
129
|
+
- WAL frame totals, buffered frame count, and last sync time;
|
|
130
|
+
- table count, record count, and current index-parity status.
|
|
131
|
+
|
|
132
|
+
The report is a fast operational probe. It does not replace a backup restore
|
|
133
|
+
test, a full application invariant check, or cross-process coordination.
|
|
134
|
+
|
|
135
|
+
## WAL maintenance
|
|
136
|
+
|
|
137
|
+
The default WAL debounce is 20 ms. Increase it only when the host accepts a
|
|
138
|
+
larger durability window; decrease it when write visibility matters more than
|
|
139
|
+
batching. `checkpoint()` flushes before rotation and leaves a new checkpoint
|
|
140
|
+
marker in the WAL.
|
|
141
|
+
|
|
142
|
+
If WAL growth is persistent:
|
|
143
|
+
|
|
144
|
+
1. Confirm the process is calling `flush()` or `stop()`.
|
|
145
|
+
2. Inspect `health().pillars.walJournal`.
|
|
146
|
+
3. Create a checkpoint after quiescing writes.
|
|
147
|
+
4. Preserve `wal.log` and `wal.log.old` before manual intervention.
|
|
148
|
+
|
|
149
|
+
## CAS maintenance
|
|
150
|
+
|
|
151
|
+
Store a blob by retaining its returned hash in a record, conventionally as a
|
|
152
|
+
string such as `CAS:<hash>`:
|
|
153
|
+
|
|
154
|
+
```ts
|
|
155
|
+
const hash = await db.storeBlob(Buffer.from(payload))
|
|
156
|
+
table.put("document-1", { id: "document-1", blob: `CAS:${hash}` })
|
|
157
|
+
const bytes = await db.readBlob(hash)
|
|
158
|
+
```
|
|
159
|
+
|
|
160
|
+
`gc()` scans current table values for `CAS:` references and removes all other
|
|
161
|
+
blob files. Only call it when all durable references are represented in the
|
|
162
|
+
currently loaded tables. External manifests, pending imports, and backups are
|
|
163
|
+
not discovered automatically.
|
|
164
|
+
|
|
165
|
+
When CAS verification fails, the payload is moved under `cas/corrupt/` and an
|
|
166
|
+
entry is appended to `manifest.jsonl`. Preserve the quarantine directory before
|
|
167
|
+
attempting a restore.
|
|
168
|
+
|
|
169
|
+
## Security and data handling
|
|
170
|
+
|
|
171
|
+
- Choose a state directory with permissions appropriate for the records stored.
|
|
172
|
+
- Do not put secrets in natural-language queries, error messages, or checkpoint
|
|
173
|
+
labels; labels are persisted in checkpoint history.
|
|
174
|
+
- Treat WAL, checkpoint, CAS, and quarantine files as sensitive application data.
|
|
175
|
+
- Validate record content before persistence; BroccoliDB does not encrypt files.
|
|
176
|
+
- Use an external lock/lease when more than one process can write the same root.
|
|
@@ -0,0 +1,91 @@
|
|
|
1
|
+
# Design philosophy
|
|
2
|
+
|
|
3
|
+
BroccoliDB is intentionally smaller than a general-purpose database. Its design
|
|
4
|
+
optimizes for an embeddable application boundary where the owner can see the
|
|
5
|
+
tables, choose the workspace path, decide when to flush, and recover from
|
|
6
|
+
ordinary files.
|
|
7
|
+
|
|
8
|
+
## Principles
|
|
9
|
+
|
|
10
|
+
### 1. Memory is the hot path; durability is explicit
|
|
11
|
+
|
|
12
|
+
Table reads and indexes stay in memory. The WAL, checkpoint, and CAS layers are
|
|
13
|
+
explicit durability mechanisms rather than hidden database behavior. Callers
|
|
14
|
+
should know which boundary they have crossed:
|
|
15
|
+
|
|
16
|
+
| Boundary | Meaning |
|
|
17
|
+
|---|---|
|
|
18
|
+
| `put()`/`delete()` returned | The in-memory table changed; the WAL append was scheduled |
|
|
19
|
+
| `await db.flush()` | Buffered WAL frames were written |
|
|
20
|
+
| `await db.transaction(fn)` returned | The callback completed and the kernel flushed its WAL buffer |
|
|
21
|
+
| `await db.checkpoint()` returned | A base snapshot and checkpoint history were written and the WAL was rotated |
|
|
22
|
+
| `await db.stop()` returned | Kernel subsystems were flushed and stopped |
|
|
23
|
+
|
|
24
|
+
### 2. Files should be boring
|
|
25
|
+
|
|
26
|
+
The durable format uses JSON, JSONL, SHA-256, temporary files, and rename. This
|
|
27
|
+
makes state inspectable and transferable without a native reader. The trade-off
|
|
28
|
+
is that applications should not expect SQL joins, page-level indexes, or a
|
|
29
|
+
server-grade transaction log.
|
|
30
|
+
|
|
31
|
+
### 3. Integrity belongs at the boundary
|
|
32
|
+
|
|
33
|
+
WAL frames carry checksums and previous-frame metadata. CAS payloads are verified
|
|
34
|
+
against their requested SHA-256 address on read. Checkpoint snapshots include a
|
|
35
|
+
hash. Corrupt data should fail visibly or be quarantined; it should not silently
|
|
36
|
+
be treated as valid state.
|
|
37
|
+
|
|
38
|
+
### 4. Contracts are more stable than implementation details
|
|
39
|
+
|
|
40
|
+
Consumers program against `IDbTable`, `IBroccoliDatabaseKernel`, query types,
|
|
41
|
+
aggregate types, and change events. Internal index maps and file helpers may
|
|
42
|
+
change, but public signatures and documented lifecycle semantics require review.
|
|
43
|
+
|
|
44
|
+
### 5. Portability beats incidental compatibility
|
|
45
|
+
|
|
46
|
+
BroccoliDB does not preserve the removed SQLite package's SQL or driver API. The
|
|
47
|
+
supported path is the standalone `@noorm/broccolidb` package. Adapters should
|
|
48
|
+
translate application queries into the typed table/query contracts instead of
|
|
49
|
+
reintroducing a private compatibility layer.
|
|
50
|
+
|
|
51
|
+
### 6. Operational behavior must be legible
|
|
52
|
+
|
|
53
|
+
Health reports, checkpoint records, WAL metrics, CAS stats, and explicit error
|
|
54
|
+
types are preferable to opaque magic. The health report is a diagnostic probe,
|
|
55
|
+
not a substitute for an application-specific audit or backup test.
|
|
56
|
+
|
|
57
|
+
## Rejected alternatives
|
|
58
|
+
|
|
59
|
+
### Native SQLite binding as the package core
|
|
60
|
+
|
|
61
|
+
Rejected for this package because it adds native installation and ABI concerns.
|
|
62
|
+
Applications that need SQL should choose a dedicated SQL database; BroccoliDB is
|
|
63
|
+
the portable table substrate.
|
|
64
|
+
|
|
65
|
+
### A remote database service
|
|
66
|
+
|
|
67
|
+
Rejected because the package is designed for local, offline, process-owned state
|
|
68
|
+
and should not introduce networking, credentials, or service discovery.
|
|
69
|
+
|
|
70
|
+
### Implicit persistence on every mutation
|
|
71
|
+
|
|
72
|
+
Rejected because it couples latency and correctness decisions to an invisible
|
|
73
|
+
policy. Micro-batching plus explicit `flush()`, `transaction()`, and
|
|
74
|
+
`checkpoint()` calls make the boundary visible to the caller.
|
|
75
|
+
|
|
76
|
+
### Treating token estimates as billing truth
|
|
77
|
+
|
|
78
|
+
Rejected in `TokenCompressionService`. The four-characters-per-token estimate is
|
|
79
|
+
a fast budget signal; provider usage accounting remains the provider's concern.
|
|
80
|
+
|
|
81
|
+
## Boundaries
|
|
82
|
+
|
|
83
|
+
- The async mutex is process-local and re-entrant through async context.
|
|
84
|
+
- The kernel does not coordinate independent Node processes writing the same
|
|
85
|
+
workspace.
|
|
86
|
+
- WAL replay validates frame checksums and reconstructs table mutations; it is
|
|
87
|
+
not a general migration engine.
|
|
88
|
+
- CAS garbage collection is only safe when the referenced-hash set accurately
|
|
89
|
+
reflects application records.
|
|
90
|
+
- Health probes report the implementation's current checks; they are not a
|
|
91
|
+
full filesystem scrub or cross-process consistency protocol.
|
package/docs/README.md
ADDED
|
@@ -0,0 +1,116 @@
|
|
|
1
|
+
# BroccoliDB documentation
|
|
2
|
+
|
|
3
|
+
This is the documentation map for the standalone `@noorm/broccolidb` package.
|
|
4
|
+
The package docs follow a layered structure used by mature storage systems:
|
|
5
|
+
|
|
6
|
+
**Concepts → How it works → Reference → Operations → Decisions**
|
|
7
|
+
|
|
8
|
+
The implementation and tests are authoritative when documentation disagrees.
|
|
9
|
+
This map explains where a claim belongs and gives each reader a shortest useful
|
|
10
|
+
path through the package.
|
|
11
|
+
|
|
12
|
+
## Reading paths by role
|
|
13
|
+
|
|
14
|
+
| Role | Start here | Then read |
|
|
15
|
+
|---|---|---|
|
|
16
|
+
| **Application developer** | [Quick start](../README.md#quick-start) | [API reference](API.md) · [Operations](OPERATIONS.md) |
|
|
17
|
+
| **Architect / reviewer** | [Brief](BRIEF.md) | [Architecture](ARCHITECTURE.md) · [Philosophy](PHILOSOPHY.md) · [ADR-001](adr/ADR-001-portable-inmemory-kernel.md) |
|
|
18
|
+
| **Operator / support** | [Operations](OPERATIONS.md) | [Troubleshooting](TROUBLESHOOTING.md) · [Glossary](GLOSSARY.md) |
|
|
19
|
+
| **Contributor** | [Contributing](CONTRIBUTING.md) | [Architecture](ARCHITECTURE.md) · [API reference](API.md) · [ADR process](adr/README.md) |
|
|
20
|
+
| **Auditor / security reviewer** | [Boundaries](PHILOSOPHY.md#boundaries) | [Operations](OPERATIONS.md#integrity-and-recovery) · [Architecture](ARCHITECTURE.md#failure-model) |
|
|
21
|
+
| **Release owner** | [Release notes](RELEASE_NOTES.md) | [Contributing](CONTRIBUTING.md#release-checklist) · [ADR index](adr/README.md) |
|
|
22
|
+
|
|
23
|
+
## Document catalog
|
|
24
|
+
|
|
25
|
+
### Concepts — why
|
|
26
|
+
|
|
27
|
+
| Document | Purpose |
|
|
28
|
+
|---|---|
|
|
29
|
+
| [Brief](BRIEF.md) | One-page problem statement, solution, guarantees, and non-goals |
|
|
30
|
+
| [Philosophy](PHILOSOPHY.md) | Design principles, explicit trade-offs, and rejected alternatives |
|
|
31
|
+
| [Glossary](GLOSSARY.md) | Canonical terms for tables, WAL, CAS, checkpoints, and recovery |
|
|
32
|
+
|
|
33
|
+
### How it works — what happens
|
|
34
|
+
|
|
35
|
+
| Document | Purpose |
|
|
36
|
+
|---|---|
|
|
37
|
+
| [Architecture](ARCHITECTURE.md) | Runtime layers, mutation flow, startup, checkpointing, rollback, and failure model |
|
|
38
|
+
| [Operations](OPERATIONS.md) | Filesystem layout, lifecycle rules, backup/restore, integrity, and garbage collection |
|
|
39
|
+
|
|
40
|
+
### Reference — what to call
|
|
41
|
+
|
|
42
|
+
| Document | Purpose |
|
|
43
|
+
|---|---|
|
|
44
|
+
| [API reference](API.md) | Public exports, kernel/table methods, query operators, indexes, aggregation, and events |
|
|
45
|
+
| [Package README](../README.md) | Install, minimal example, portability statement, and command index |
|
|
46
|
+
| [Source entry point](../src/index.ts) | Export surface — the source of truth for package imports |
|
|
47
|
+
| [Contracts](../src/broccolidb.contracts.ts) | Type-level API and serialized record contracts |
|
|
48
|
+
|
|
49
|
+
### Operations — how to run and debug
|
|
50
|
+
|
|
51
|
+
| Document | Purpose |
|
|
52
|
+
|---|---|
|
|
53
|
+
| [Operations guide](OPERATIONS.md) | Start/stop, durability boundaries, backup, recovery, health, and CAS GC |
|
|
54
|
+
| [Troubleshooting](TROUBLESHOOTING.md) | Symptom → likely cause → action runbooks |
|
|
55
|
+
| [Release notes](RELEASE_NOTES.md) | Supported release history and compatibility policy |
|
|
56
|
+
|
|
57
|
+
### Decisions — why the shape is stable
|
|
58
|
+
|
|
59
|
+
| Document | Purpose |
|
|
60
|
+
|---|---|
|
|
61
|
+
| [ADR index](adr/README.md) | Decision inventory and writing rules |
|
|
62
|
+
| [ADR-001](adr/ADR-001-portable-inmemory-kernel.md) | Why BroccoliDB is a portable in-memory kernel with explicit file durability |
|
|
63
|
+
|
|
64
|
+
## Documentation conventions
|
|
65
|
+
|
|
66
|
+
1. **Describe the current package.** Do not document the removed in-repository
|
|
67
|
+
SQLite implementation or imply that it remains supported.
|
|
68
|
+
2. **Use contract language.** Distinguish what is guaranteed after an in-memory
|
|
69
|
+
mutation, after `flush()`, after `checkpoint()`, and after `stop()`.
|
|
70
|
+
3. **Name the source of truth.** Public exports live in `src/index.ts`; type
|
|
71
|
+
contracts live in `src/broccolidb.contracts.ts`; behavior is verified by
|
|
72
|
+
`test/`.
|
|
73
|
+
4. **Keep examples executable.** Examples should use the public package import,
|
|
74
|
+
explicit lifecycle calls, and a temporary or application-owned workspace.
|
|
75
|
+
5. **Document failure boundaries.** Say what is process-local, what is on disk,
|
|
76
|
+
what is recoverable, and what requires an external coordination system.
|
|
77
|
+
6. **Prefer stable vocabulary.** Say table, record, WAL frame, checkpoint, CAS
|
|
78
|
+
blob, replay, rollback, flush, and health report consistently.
|
|
79
|
+
7. **Update docs with contracts.** Export changes, file-format changes, and
|
|
80
|
+
durability changes require API/operations updates plus an ADR or release note.
|
|
81
|
+
|
|
82
|
+
## Source-of-truth matrix
|
|
83
|
+
|
|
84
|
+
| Question | Source of truth |
|
|
85
|
+
|---|---|
|
|
86
|
+
| What can consumers import? | `src/index.ts` and generated `dist/index.d.ts` |
|
|
87
|
+
| What does a method accept/return? | `src/broccolidb.contracts.ts` and class signatures |
|
|
88
|
+
| How does a write reach disk? | `src/broccolidb-kernel.ts` and `src/broccolidb-wal.ts` |
|
|
89
|
+
| How are blobs addressed and verified? | `src/broccolidb-cas.ts` |
|
|
90
|
+
| What behavior is protected? | `test/*.test.ts` |
|
|
91
|
+
| Is the package publishable? | `package.json`, `npm pack --dry-run`, and `npm run docs:check` |
|
|
92
|
+
|
|
93
|
+
## Status
|
|
94
|
+
|
|
95
|
+
| Metric | Current value |
|
|
96
|
+
|---|---|
|
|
97
|
+
| Package | `@noorm/broccolidb@2.0.1` |
|
|
98
|
+
| API status | Standalone supported package; modern ESM surface |
|
|
99
|
+
| Runtime dependencies | 0 |
|
|
100
|
+
| Node.js | `>=18` |
|
|
101
|
+
| Public entry point | `src/index.ts` / `dist/index.js` |
|
|
102
|
+
| Persistence | In-memory tables + optional filesystem WAL/checkpoints/CAS |
|
|
103
|
+
| License | MIT |
|
|
104
|
+
|
|
105
|
+
## Quick links
|
|
106
|
+
|
|
107
|
+
```bash
|
|
108
|
+
# Build and test the implementation
|
|
109
|
+
npm test
|
|
110
|
+
|
|
111
|
+
# Validate docs and relative Markdown links
|
|
112
|
+
npm run docs:check
|
|
113
|
+
|
|
114
|
+
# Inspect the publishable artifact without creating it
|
|
115
|
+
npm pack --dry-run
|
|
116
|
+
```
|
|
@@ -0,0 +1,34 @@
|
|
|
1
|
+
# Release notes
|
|
2
|
+
|
|
3
|
+
## 2.0.1 — first npm publication
|
|
4
|
+
|
|
5
|
+
This patch release publishes the standalone package under a new version because
|
|
6
|
+
the previously unused `2.0.0` version had already been unpublished on npm and
|
|
7
|
+
cannot be reused by the registry.
|
|
8
|
+
|
|
9
|
+
The package contents and supported API are unchanged from the standalone
|
|
10
|
+
release described below.
|
|
11
|
+
|
|
12
|
+
## 2.0.0 — standalone portable package
|
|
13
|
+
|
|
14
|
+
This release is the supported standalone BroccoliDB package.
|
|
15
|
+
|
|
16
|
+
### Included
|
|
17
|
+
|
|
18
|
+
- in-memory typed tables with equality, sorted, composite, and prefix indexes;
|
|
19
|
+
- operator filters, boolean query clauses, fluent queries, aggregation, CDC,
|
|
20
|
+
TTL expiration, and deterministic natural-language parsing;
|
|
21
|
+
- checksum-linked micro-batched WAL and restart replay;
|
|
22
|
+
- atomic JSON checkpoints, named history, and rollback;
|
|
23
|
+
- SHA-256 CAS storage with optional Brotli compression and corruption quarantine;
|
|
24
|
+
- re-entrant async mutex and kernel transaction boundary;
|
|
25
|
+
- dependency-free runtime package for Node.js `>=18`.
|
|
26
|
+
|
|
27
|
+
### Compatibility policy
|
|
28
|
+
|
|
29
|
+
The standalone package is the supported implementation. The removed SQLite
|
|
30
|
+
implementation is not a supported compatibility target, and this package does
|
|
31
|
+
not promise SQL, Kysely, or `better-sqlite3` API compatibility.
|
|
32
|
+
|
|
33
|
+
Changes to exported contracts or durable file formats require a release-note
|
|
34
|
+
entry and an architecture decision record.
|
|
@@ -0,0 +1,145 @@
|
|
|
1
|
+
# Troubleshooting
|
|
2
|
+
|
|
3
|
+
Use the symptom → likely cause → action flow below. Preserve `.broccolidb/`
|
|
4
|
+
before destructive recovery work.
|
|
5
|
+
|
|
6
|
+
## Startup and recovery
|
|
7
|
+
|
|
8
|
+
### `start()` throws `WalIntegrityError`
|
|
9
|
+
|
|
10
|
+
**Likely causes:** a truncated JSONL write, hand-edited WAL, filesystem
|
|
11
|
+
corruption, or a package/version mismatch that changed frame serialization.
|
|
12
|
+
|
|
13
|
+
**Action:** stop all writers, copy the entire `.broccolidb/` directory, inspect
|
|
14
|
+
the failing line and checksum message, and restore the last known-good backup if
|
|
15
|
+
the WAL cannot be repaired. Do not delete the WAL as a first response.
|
|
16
|
+
|
|
17
|
+
### Data is missing after a process crash
|
|
18
|
+
|
|
19
|
+
**Likely cause:** a mutation changed memory but its asynchronous WAL append had
|
|
20
|
+
not been flushed before termination.
|
|
21
|
+
|
|
22
|
+
**Action:** use `transaction()`, `flush()`, or `checkpoint()` at the host's
|
|
23
|
+
durability boundary. BroccoliDB cannot recover a frame that never reached the
|
|
24
|
+
WAL.
|
|
25
|
+
|
|
26
|
+
### Startup is slow
|
|
27
|
+
|
|
28
|
+
**Likely causes:** a large WAL tail, a large checkpoint, or too many records
|
|
29
|
+
being rebuilt into indexes.
|
|
30
|
+
|
|
31
|
+
**Action:** measure startup with the same workload, inspect WAL metrics, quiesce
|
|
32
|
+
writes, and create a checkpoint. Keep tables bounded or partition application
|
|
33
|
+
state when a single in-memory table becomes too large.
|
|
34
|
+
|
|
35
|
+
## Tables and queries
|
|
36
|
+
|
|
37
|
+
### A query is correct but slower than expected
|
|
38
|
+
|
|
39
|
+
**Likely cause:** no matching index exists or the predicate cannot use the
|
|
40
|
+
available index shape.
|
|
41
|
+
|
|
42
|
+
**Action:** call `select().where(...).explain()`, inspect `scanStrategy`, and add
|
|
43
|
+
an equality, sorted, composite, or prefix index that matches the hot predicate.
|
|
44
|
+
Do not assume an index changes semantics; it changes candidate selection.
|
|
45
|
+
|
|
46
|
+
### An index appears stale
|
|
47
|
+
|
|
48
|
+
**Likely causes:** records were restored with a custom snapshot, a host kept a
|
|
49
|
+
second table reference, or application code mutated an object after passing it
|
|
50
|
+
to the table.
|
|
51
|
+
|
|
52
|
+
**Action:** treat records as immutable application values, use table mutation
|
|
53
|
+
methods, and rebuild the table/indexes through a controlled restore if needed.
|
|
54
|
+
|
|
55
|
+
### TTL behavior is surprising
|
|
56
|
+
|
|
57
|
+
**Likely cause:** TTL timers are process-local and depend on the process staying
|
|
58
|
+
alive; an expired record is not a durable scheduler job.
|
|
59
|
+
|
|
60
|
+
**Action:** use TTL for local cache-like expiration. For business deadlines,
|
|
61
|
+
persist an explicit expiration field and run a host-owned reconciliation pass.
|
|
62
|
+
|
|
63
|
+
## Checkpoints and rollback
|
|
64
|
+
|
|
65
|
+
### `rollback(id)` returns `false`
|
|
66
|
+
|
|
67
|
+
**Likely causes:** the ID is unknown, history was not copied with the backup, or
|
|
68
|
+
the checkpoint file is unreadable.
|
|
69
|
+
|
|
70
|
+
**Action:** call `listCheckpoints()`, verify
|
|
71
|
+
`.broccolidb/checkpoints/<id>.json`, and restore the complete state directory if
|
|
72
|
+
the file is missing.
|
|
73
|
+
|
|
74
|
+
### Rollback did not remove a table created later
|
|
75
|
+
|
|
76
|
+
**Likely cause:** rollback rebuilds tables represented by the checkpoint; it is
|
|
77
|
+
not a schema registry that automatically deletes every table absent from a
|
|
78
|
+
historical snapshot.
|
|
79
|
+
|
|
80
|
+
**Action:** explicitly clear or recreate application tables as part of the
|
|
81
|
+
rollback procedure, then checkpoint the resulting state.
|
|
82
|
+
|
|
83
|
+
## CAS
|
|
84
|
+
|
|
85
|
+
### `readBlob(hash)` returns `null`
|
|
86
|
+
|
|
87
|
+
**Likely cause:** the hash is absent from the configured workspace or the host
|
|
88
|
+
opened a different `workspaceRoot`.
|
|
89
|
+
|
|
90
|
+
**Action:** log the resolved root, compare `cas.getBaseDir()`, and restore the
|
|
91
|
+
CAS directory from the matching backup.
|
|
92
|
+
|
|
93
|
+
### `StorageIntegrityError` is raised
|
|
94
|
+
|
|
95
|
+
**Likely cause:** a payload failed Brotli decompression or SHA-256 verification.
|
|
96
|
+
|
|
97
|
+
**Action:** preserve `cas/corrupt/manifest.jsonl`, inspect the quarantined blob,
|
|
98
|
+
and restore a known-good backup. Do not overwrite the quarantine entry before
|
|
99
|
+
capturing it.
|
|
100
|
+
|
|
101
|
+
### `gc()` deleted data needed by the application
|
|
102
|
+
|
|
103
|
+
**Likely cause:** the reference was not stored in a currently loaded table as a
|
|
104
|
+
`CAS:<hash>` string.
|
|
105
|
+
|
|
106
|
+
**Action:** restore from backup and make references explicit before rerunning GC.
|
|
107
|
+
`gc()` is conservative only with respect to the references it can see.
|
|
108
|
+
|
|
109
|
+
## Locking and concurrency
|
|
110
|
+
|
|
111
|
+
### `DeadlockTimeoutError`
|
|
112
|
+
|
|
113
|
+
**Likely causes:** a callback awaited a long-running operation while holding the
|
|
114
|
+
lock, nested operations are waiting on another resource, or two application
|
|
115
|
+
components use locks in inconsistent order.
|
|
116
|
+
|
|
117
|
+
**Action:** keep transaction callbacks short, avoid external network calls while
|
|
118
|
+
holding the kernel lock, and review lock ordering. The mutex is re-entrant for
|
|
119
|
+
the same async context, but it is not a general deadlock solver.
|
|
120
|
+
|
|
121
|
+
### Two processes see divergent state
|
|
122
|
+
|
|
123
|
+
**Likely cause:** both processes wrote the same `.broccolidb/` directory. The
|
|
124
|
+
package does not provide cross-process fencing.
|
|
125
|
+
|
|
126
|
+
**Action:** assign one writer, put an external lease around the workspace, or
|
|
127
|
+
choose a database designed for multi-process coordination.
|
|
128
|
+
|
|
129
|
+
## Packaging and imports
|
|
130
|
+
|
|
131
|
+
### `ERR_MODULE_NOT_FOUND` for internal imports
|
|
132
|
+
|
|
133
|
+
**Likely cause:** the package was copied without building `dist/`, or a consumer
|
|
134
|
+
is bypassing the package export and importing source files incorrectly.
|
|
135
|
+
|
|
136
|
+
**Action:** run `npm run build`, import from `@noorm/broccolidb`, and inspect the
|
|
137
|
+
`files` list with `npm pack --dry-run`.
|
|
138
|
+
|
|
139
|
+
### A native SQLite package appears in the dependency tree
|
|
140
|
+
|
|
141
|
+
**Likely cause:** the host application installed an unrelated database package
|
|
142
|
+
or a stale lockfile was copied into the package.
|
|
143
|
+
|
|
144
|
+
**Action:** run `npm ls --omit=dev --depth=0`, inspect the host's dependency
|
|
145
|
+
graph, and keep BroccoliDB's runtime `dependencies` object empty.
|
|
@@ -0,0 +1,74 @@
|
|
|
1
|
+
# ADR-001: Portable in-memory kernel with explicit filesystem durability
|
|
2
|
+
|
|
3
|
+
- **Status:** Accepted
|
|
4
|
+
- **Date:** 2026-09-20
|
|
5
|
+
- **Owners:** BroccoliDB maintainers
|
|
6
|
+
- **Scope:** package architecture, runtime dependencies, and persistence boundary
|
|
7
|
+
|
|
8
|
+
## Context
|
|
9
|
+
|
|
10
|
+
BroccoliDB is embedded in applications that need fast local tables, indexes, and
|
|
11
|
+
recoverable state. The previous SQLite-oriented implementation introduced a
|
|
12
|
+
native database dependency and a portability burden for consumers that only
|
|
13
|
+
needed table operations and local durability.
|
|
14
|
+
|
|
15
|
+
The standalone package must work across supported Node.js environments without a
|
|
16
|
+
native compiler, Electron ABI rebuild, database server, or SQL-specific adapter.
|
|
17
|
+
|
|
18
|
+
## Decision
|
|
19
|
+
|
|
20
|
+
Use an in-memory table kernel as the supported implementation and compose
|
|
21
|
+
durability from ordinary filesystem primitives:
|
|
22
|
+
|
|
23
|
+
- typed `Map`-backed tables and secondary indexes for the hot path;
|
|
24
|
+
- a micro-batched JSONL WAL with checksum metadata for mutation replay;
|
|
25
|
+
- atomic JSON checkpoints and named checkpoint history for restore points;
|
|
26
|
+
- SHA-256 content-addressable files with optional Brotli compression for blobs;
|
|
27
|
+
- a process-local re-entrant async mutex for kernel transactions;
|
|
28
|
+
- zero production dependencies and an ESM Node.js `>=18` package surface.
|
|
29
|
+
|
|
30
|
+
The public contract is the package export from `src/index.ts` and the interfaces
|
|
31
|
+
in `src/broccolidb.contracts.ts`. The removed SQLite implementation is not a
|
|
32
|
+
compatibility target.
|
|
33
|
+
|
|
34
|
+
## Alternatives considered
|
|
35
|
+
|
|
36
|
+
### Keep `better-sqlite3` in the package
|
|
37
|
+
|
|
38
|
+
Rejected. It creates native installation/ABI coupling and makes a small embedded
|
|
39
|
+
table substrate harder to transfer between environments.
|
|
40
|
+
|
|
41
|
+
### Use Kysely over a database driver
|
|
42
|
+
|
|
43
|
+
Rejected. A query builder does not remove the underlying driver and would force
|
|
44
|
+
the package to preserve SQL/dialect semantics it does not need.
|
|
45
|
+
|
|
46
|
+
### Use an external service
|
|
47
|
+
|
|
48
|
+
Rejected. The package is intended for offline, process-owned local state and
|
|
49
|
+
should not add networking, credentials, deployment, or service availability
|
|
50
|
+
requirements.
|
|
51
|
+
|
|
52
|
+
## Consequences
|
|
53
|
+
|
|
54
|
+
### Positive
|
|
55
|
+
|
|
56
|
+
- Consumers install and run without native database builds.
|
|
57
|
+
- State is inspectable and transferable as ordinary files.
|
|
58
|
+
- The hot path is simple and fast for bounded local tables.
|
|
59
|
+
- Recovery behavior is explicit and testable.
|
|
60
|
+
- The package can be vendored or published without LUMI-specific aliases.
|
|
61
|
+
|
|
62
|
+
### Trade-offs
|
|
63
|
+
|
|
64
|
+
- Tables must fit in process memory.
|
|
65
|
+
- There is no SQL compatibility, migration engine, or relational join planner.
|
|
66
|
+
- The built-in mutex does not coordinate independent processes.
|
|
67
|
+
- Durability is explicit; an unflushed process crash can lose the latest buffer.
|
|
68
|
+
- Applications own backup policy, schema evolution, and record validation.
|
|
69
|
+
|
|
70
|
+
## Compatibility impact
|
|
71
|
+
|
|
72
|
+
Changing the WAL frame shape, checkpoint structure, CAS encoding, export surface,
|
|
73
|
+
or lifecycle semantics is a compatibility change. Such changes require tests,
|
|
74
|
+
operations documentation, release notes, and a new or updated ADR.
|
|
@@ -0,0 +1,38 @@
|
|
|
1
|
+
# Architecture decision records
|
|
2
|
+
|
|
3
|
+
Architecture Decision Records (ADRs) capture decisions that affect the public
|
|
4
|
+
API, durable formats, portability boundary, or recovery behavior. They are
|
|
5
|
+
shorter than a whitepaper and more durable than a changelog entry.
|
|
6
|
+
|
|
7
|
+
## Index
|
|
8
|
+
|
|
9
|
+
| ID | Decision | Status |
|
|
10
|
+
|---|---|---|
|
|
11
|
+
| [ADR-001](ADR-001-portable-inmemory-kernel.md) | Use a portable in-memory table kernel with explicit filesystem durability | Accepted |
|
|
12
|
+
|
|
13
|
+
## When to write an ADR
|
|
14
|
+
|
|
15
|
+
Write or update an ADR when a change:
|
|
16
|
+
|
|
17
|
+
- adds/removes a public export;
|
|
18
|
+
- changes WAL, checkpoint, CAS, or recovery semantics;
|
|
19
|
+
- changes the runtime dependency or native-module policy;
|
|
20
|
+
- changes concurrency or cross-process assumptions;
|
|
21
|
+
- introduces a new persistence format or compatibility decision.
|
|
22
|
+
|
|
23
|
+
Do not use an ADR for a typo, a local refactor, or a test-only fixture change.
|
|
24
|
+
|
|
25
|
+
## Lifecycle
|
|
26
|
+
|
|
27
|
+
1. Create the next numbered file from [the template](TEMPLATE.md).
|
|
28
|
+
2. State context, decision, alternatives, consequences, and compatibility impact.
|
|
29
|
+
3. Link it from this index and the affected reference/operations documents.
|
|
30
|
+
4. Mark it `Accepted`, `Superseded`, or `Rejected`; do not silently rewrite the
|
|
31
|
+
historical decision.
|
|
32
|
+
|
|
33
|
+
## Status vocabulary
|
|
34
|
+
|
|
35
|
+
- **Proposed** — under review and not yet a supported contract.
|
|
36
|
+
- **Accepted** — current supported decision.
|
|
37
|
+
- **Superseded** — replaced by a later ADR; retain for history.
|
|
38
|
+
- **Rejected** — considered and declined; retain the reasoning.
|