@noorm/broccolidb 2.0.1

package/docs/API.md

@@ -0,0 +1,208 @@
+# API reference
+
+This reference documents the supported public entry point:
+
+```ts
+import { ... } from "@noorm/broccolidb"
+```
+
+The complete type-level contract is in [`src/broccolidb.contracts.ts`](../src/broccolidb.contracts.ts)
+and the export list is in [`src/index.ts`](../src/index.ts).
+
+## Kernel
+
+```ts
+interface DatabaseKernelOptions {
+  workspaceRoot?: string
+  walDebounceMs?: number
+}
+
+const db = new BroccoliDatabaseKernel(options)
+```
+
+| Method | Returns | Notes |
+|---|---|---|
+| `start()` | `Promise<void>` | Creates storage directories, loads checkpoint, and replays WAL. Idempotent. |
+| `stop()` | `Promise<void>` | Flushes WAL and stops WAL/CAS services. |
+| `flush()` | `Promise<void>` | Writes buffered WAL frames. |
+| `getTable<T>(name)` | `IDbTable<T>` | Returns or creates a typed in-memory table. |
+| `transaction(fn)` | `Promise<R>` | Concrete kernel method; runs an async callback under the re-entrant mutex and flushes afterward. |
+| `checkpoint(label?)` | `Promise<TimelineCheckpointRecord>` | Writes a base snapshot, history record, and WAL rotation. |
+| `rollback(id)` | `Promise<boolean>` | Restores a cached or on-disk checkpoint. |
+| `listCheckpoints()` | `readonly TimelineCheckpointRecord[]` | Lists checkpoint records known to the current process. |
+| `health()` | `Promise<DbHealthReport>` | Reports writeability, CAS metrics, WAL metrics, and table counts. |
+| `storeBlob(content)` | `Promise<string>` | Stores bytes/string in CAS and returns the SHA-256 address. |
+| `readBlob(hash)` | `Promise<Buffer \| null>` | Returns verified content or `null` when the blob is absent. |
+| `gc()` | `Promise<number>` | Removes CAS files not referenced by current table string values prefixed `CAS:`. |
+
+The package also exports `broccolidb`, a singleton constructed with the default
+workspace root. Prefer an explicitly constructed kernel when an application has
+more than one workspace, test isolation, or lifecycle owner.
+
+## Tables
+
+```ts
+type User = { id: string; name: string; score: number; active: boolean }
+const users = db.getTable<User>("users")
+```
+
+| Method | Description |
+|---|---|
+| `get(id)` | Read one record or return `undefined`. |
+| `getAll()` | Return a readonly snapshot of current records. |
+| `put(id, record, options?)` | Insert or replace a record; `ttlMs` schedules expiration. `idempotencyKey` is accepted by the contract but is not currently used for deduplication. |
+| `putMany(entries)` | Apply multiple puts. |
+| `compareAndSwap(id, predicate, updater, options?)` | Apply an update only when the predicate accepts the current record. |
+| `delete(id)` | Delete one record and return whether it existed. |
+| `deleteWhere(where)` | Delete matching records and return the count. |
+| `updateWhere(where, updater)` | Update matching records and return the count. |
+| `query(options?)` | Filter, order, paginate, and return readonly records. |
+| `aggregate(query)` | Group and calculate statistical metrics. |
+| `select()` | Start a fluent query builder. |
+| `subscribe(callback, filter?)` | Subscribe to change events; call `.unsubscribe()` to remove it. |
+| `count()` | Return current record count. |
+| `clear()` | Remove all records and emit a clear event. |
+| `createSnapshot()` / `restoreSnapshot()` | Capture or replace the table's in-memory map. |
+
+## Indexes
+
+```ts
+users.createIndex("active")
+users.createSortedIndex("score")
+users.createCompositeIndex(["active", "name"])
+users.createPrefixIndex("name")
+```
+
+Index creation is explicit and does not change query semantics. Available index
+types are `equality`, `sorted`, `composite`, and `prefix`. The fluent builder's
+`explain()` method returns the selected strategy and candidate counts.
+
+## Query options
+
+```ts
+const result = users.query({
+  where: {
+    active: true,
+    score: { $gte: 80, $lt: 100 },
+  },
+  sortBy: "score",
+  sortOrder: "desc",
+  limit: 20,
+  offset: 0,
+})
+```
+
+Supported field operators:
+
+| Operator | Meaning |
+|---|---|
+| `$eq`, `$ne` | Equality or inequality |
+| `$gt`, `$gte`, `$lt`, `$lte` | Ordered comparison |
+| `$in`, `$nin` | Membership or non-membership |
+| `$between` | Inclusive two-value range |
+| `$startsWith`, `$endsWith` | String prefix/suffix |
+| `$contains` | String containment |
+| `$regex` | Regular expression match |
+| `$exists` | Field presence check |
+
+Boolean clauses are available through `and`, `or`, and `not`. `sortBy` and
+`sortOrder` accept one field or parallel readonly arrays.
+
+## Fluent queries
+
+```ts
+const topUsers = users
+  .select()
+  .where("active").equals(true)
+  .and("score").greaterThanOrEqual(80)
+  .orderBy("score", "desc")
+  .limit(10)
+
+console.log(topUsers.explain())
+console.log(topUsers.execute())
+console.log(topUsers.first())
+console.log(topUsers.count())
+```
+
+Predicates include `equals`, `notEquals`, `greaterThan`,
+`greaterThanOrEqual`, `lessThan`, `lessThanOrEqual`, `in`, `notIn`, `between`,
+`startsWith`, `contains`, and `matches`.
+
+## Aggregation
+
+```ts
+const summary = users.aggregate({
+  groupBy: ["active"],
+  metrics: {
+    users: { metric: "count" },
+    averageScore: { metric: "avg", field: "score" },
+    scoreSpread: { metric: "stddev", field: "score" },
+  },
+  having: { users: { $gte: 2 } },
+})
+```
+
+Metrics are `sum`, `avg`, `min`, `max`, `count`, and `stddev`. Results include
+groups, grand totals, records evaluated, and elapsed microseconds.
+
+## Change events and TTL
+
+```ts
+const subscription = users.subscribe((event) => {
+  console.log(event.operation, event.recordId, event.before, event.after)
+})
+
+users.put("temporary", { id: "temporary", name: "Temp", score: 0, active: true }, { ttlMs: 5_000 })
+subscription.unsubscribe()
+```
+
+Change operations are `INSERT`, `UPDATE`, `DELETE`, `CLEAR`, and `EXPIRE`.
+Events may include `before`, `after`, and a field-level `diff`.
+
+## Natural-language parsing
+
+`BroccoliNaturalQueryParser` is a deterministic offline parser, not an LLM:
+
+```ts
+const parsed = BroccoliNaturalQueryParser.parse(
+  "from users where score >= 80 sorted by score desc limit 10",
+)
+
+const rows = db.getTable<User>(parsed.targetTable).query(parsed.queryOptions)
+```
+
+It recognizes table selection, limit/offset, sorting, comparisons, ranges,
+membership, prefixes, contains, and equality phrases. The returned `confidence`
+is a parsing coverage heuristic; applications should validate user-facing input.
+
+## WAL and CAS classes
+
+Advanced consumers can use the lower-level services directly:
+
+- `BroccoliWriteAheadLog` supports `start`, `appendFrame`, `flush`, `replay`,
+  `truncate`, and `getMetrics`.
+- `BroccoliCASStorageService` supports `start`, `store`, `read`, `exists`,
+  `pruneUnreferenced`, `getStats`, and `getBaseDir`.
+- `ReentrantAsyncMutex` supports `acquire`, `runLocked`, `isLocked`,
+  `getCurrentHolder`, and `getQueueLength`.
+
+Prefer the kernel for normal application use. Direct services are useful for
+specialized adapters, diagnostics, and tests.
+
+## Errors
+
+| Error | Raised when |
+|---|---|
+| `WalIntegrityError` | A WAL line is invalid JSON or its checksum does not match. |
+| `StorageIntegrityError` | A CAS blob cannot be decompressed or its content hash mismatches. |
+| `DatabaseLockError` | Base mutex coordination fails. |
+| `DeadlockTimeoutError` | A mutex waiter exceeds its configured timeout. |
+
+## Prompt compression
+
+`TokenCompressionService.getInstance().compactPrompt()` normalizes whitespace
+without flattening structured message blocks. Signed thinking blocks are left
+unchanged. Results include a SHA-256 prompt key, estimated token counts,
+compression ratio, requested-model echo, and an L1 cache flag.
+
+The estimate is `ceil(characters / 4)` and is not provider usage data.

package/docs/ARCHITECTURE.md

@@ -0,0 +1,160 @@
+# Architecture
+
+BroccoliDB is a four-layer embedded kernel with a process-local coordination
+boundary. The layers are deliberately composable: tables can be used directly,
+while the kernel adds WAL, checkpoints, CAS, and lifecycle management.
+
+## Layer map
+
+```mermaid
+flowchart TB
+    K[BroccoliDatabaseKernel]
+    T[Reactive in-memory tables\nCRUD · indexes · filters · CDC · TTL]
+    W[Write-ahead log\nJSONL · checksum frames · micro-batching]
+    C[Checkpoints\natomic base snapshot · timeline history]
+    S[CAS vault\nSHA-256 · Brotli · quarantine · GC]
+    M[ReentrantAsyncMutex\nprocess-local async coordination]
+
+    K --> T
+    K --> W
+    K --> C
+    K --> S
+    K --> M
+    T --> W
+    T --> C
+    T -. CAS:hash references .-> S
+```
+
+| Layer | Responsibility | Implementation |
+|---|---|---|
+| **L1 tables** | In-memory records, indexes, filtering, aggregation, fluent queries, TTL, CDC | `src/broccolidb-table.ts` |
+| **L2 WAL** | Append mutation frames, micro-batched flush, checksum validation, replay, rotation | `src/broccolidb-wal.ts` |
+| **L3 CAS** | Content-addressed blobs, optional Brotli compression, hash verification, quarantine, mark-sweep | `src/broccolidb-cas.ts` |
+| **L4 checkpointing** | Atomic base snapshot, checkpoint history, rollback metadata | `src/broccolidb-kernel.ts` |
+| **Coordination** | Re-entrant async lock for transactions and serialized WAL/checkpoint operations | `src/broccolidb-mutex.ts` |
+| **Contracts** | Public types and behavior vocabulary | `src/broccolidb.contracts.ts` |
+
+## Startup and recovery
+
+`start()` is idempotent. The kernel performs the following sequence:
+
+1. Create `.broccolidb/` and its checkpoint directory.
+2. Start the CAS service and WAL service.
+3. Load `.broccolidb/checkpoint.db` if it exists.
+4. Replay frames remaining in `.broccolidb/wal.log`.
+5. Mark the kernel started.
+
+Checkpoint data is loaded into tables before WAL replay. A missing base snapshot
+is treated as a fresh database. A malformed or checksum-invalid WAL frame is a
+`WalIntegrityError` and should be investigated rather than silently discarded.
+
+## Mutation flow
+
+```mermaid
+sequenceDiagram
+    participant App
+    participant Table
+    participant Kernel
+    participant WAL
+
+    App->>Kernel: getTable("users")
+    Kernel-->>App: IDbTable<User>
+    App->>Table: put(id, record)
+    Table->>Table: update record and indexes
+    Table-->>App: record
+    Table->>Kernel: WAL hook (async append)
+    Kernel->>WAL: append frame
+    WAL-->>WAL: debounce / batch
+    App->>Kernel: flush or transaction
+    Kernel->>WAL: write JSONL frames
+```
+
+The table mutation is synchronous from the caller's perspective. The table's
+WAL hook schedules an asynchronous frame append, so callers that need a durable
+boundary must await `flush()`, `transaction()`, `checkpoint()`, or `stop()`.
+
+## Checkpoint flow
+
+`checkpoint(label)` runs under the kernel mutex:
+
+1. Flush pending WAL frames.
+2. Serialize every currently registered table.
+3. Compute a SHA-256 snapshot hash.
+4. Write a temporary base file and rename it to `checkpoint.db`.
+5. Write a named history file under `checkpoints/<checkpointId>.json`.
+6. Cache the timeline record and in-memory snapshots.
+7. Rotate the WAL and append a checkpoint marker.
+
+The double-buffered base write protects the previous base snapshot if the
+process fails during the write. Checkpoint history is ordinary JSON and can be
+copied with the rest of `.broccolidb/`.
+
+## Rollback flow
+
+`rollback(checkpointId)` first checks the process-local snapshot cache. If the
+checkpoint was created in the current process, it restores those maps directly.
+Otherwise it loads the checkpoint history JSON and rebuilds the checkpoint's
+tables. It appends a rollback marker to the WAL. The method returns `false` when
+the requested history file cannot be read or parsed.
+
+Rollback is an application state operation, not a distributed transaction or a
+schema migration. Take a backup before destructive rollback workflows.
+
+## Table execution model
+
+Each table owns:
+
+- a `Map<string, T>` of records;
+- equality maps for exact lookups;
+- sorted entries for range/order queries;
+- composite maps keyed by field tuples;
+- prefix maps for string-prefix lookups;
+- subscriptions and TTL timers;
+- the query, aggregation, and fluent-builder helpers.
+
+The query planner reports the selected `scanStrategy`, matched index, candidate
+count, match count, and elapsed microseconds through `explain()`. A query can
+fall back to `FULL_TABLE_SCAN`; creating an index is an optimization, not a
+semantic requirement.
+
+## CAS model
+
+The CAS service hashes raw content before storage. Blobs are sharded by the
+first two hash characters and stored with a small format marker:
+
+- `BR_RAW\0` for uncompressed bytes;
+- `BR_BRZ\0` for Brotli-compressed bytes when the compressed representation
+  meets the savings threshold.
+
+Reads decompress when needed, recompute the raw SHA-256, and quarantine a blob
+when decompression or hash verification fails. `BroccoliDatabaseKernel.gc()`
+collects hashes referenced by string fields beginning with `CAS:` in current
+table records.
+
+## Failure model
+
+| Failure | Behavior | Caller action |
+|---|---|---|
+| Missing workspace directory | Created on `start()` | Normal startup |
+| Missing checkpoint | Treated as fresh state | Continue or restore a backup |
+| Invalid WAL JSON/checksum | `WalIntegrityError` during replay | Preserve the directory, inspect/restore, then retry |
+| Unreadable CAS blob | `null` for missing blob; integrity error for corruption | Inspect quarantine manifest and restore if needed |
+| Non-writable state directory | Health status is degraded/corrupted | Fix permissions or choose another root |
+| Mutex wait exceeds timeout | `DeadlockTimeoutError` | Find long-held/nested lock or reduce contention |
+| Process termination before flush | Recent buffered frames may be absent | Use explicit transaction/flush boundaries |
+
+## Implementation map
+
+| Concern | File |
+|---|---|
+| Export surface | `src/index.ts` |
+| Types and public contracts | `src/broccolidb.contracts.ts` |
+| Kernel lifecycle and recovery | `src/broccolidb-kernel.ts` |
+| Table/index/query behavior | `src/broccolidb-table.ts` |
+| Aggregation | `src/broccolidb-aggregation.ts` |
+| Natural-language parser | `src/broccolidb-natural-query.ts` |
+| WAL | `src/broccolidb-wal.ts` |
+| CAS | `src/broccolidb-cas.ts` |
+| Mutex | `src/broccolidb-mutex.ts` |
+| Prompt compression | `src/TokenCompressionService.ts` |
+| Contract tests | `test/` |

package/docs/BRIEF.md

@@ -0,0 +1,56 @@
+# BroccoliDB brief
+
+## Problem
+
+Applications often need a small, local state substrate with indexed records,
+durability, recovery, and predictable behavior. A native SQLite binding can be
+excellent, but it introduces platform-specific binaries, rebuilds, ABI coupling,
+and dependency installation failure modes that are disproportionate for an
+embedded table workload.
+
+## Solution
+
+BroccoliDB keeps the hot path in ordinary JavaScript/TypeScript memory and adds
+explicit filesystem layers only where durability needs them:
+
+1. **Reactive tables** hold typed records and maintain secondary indexes.
+2. **WAL** appends mutation frames and replays them after a restart.
+3. **Checkpoints** write a complete JSON base snapshot and rotate the WAL.
+4. **CAS** stores large or reusable byte payloads by SHA-256 content address.
+5. **Mutex and transactions** serialize related async mutations in one process.
+
+## Guarantees
+
+- The package has no production dependencies or native database driver.
+- Reads and index lookups use in-memory table state.
+- `flush()` writes buffered WAL frames; `stop()` flushes before returning.
+- `start()` loads a checkpoint and replays readable WAL frames.
+- Checkpoint files are written through a temporary file and rename.
+- CAS reads verify the content hash and quarantine corrupted payloads.
+- Query, aggregation, natural-query, and prompt-compression helpers are
+  deterministic and offline.
+
+These guarantees are deliberately narrower than a replicated database. See
+[Operations](OPERATIONS.md) for durability boundaries and process limitations.
+
+## Good fit
+
+- Extension or CLI-local state
+- Agent/session tables
+- Durable caches and indexes scoped to one workspace
+- Embedded tools that need rollback checkpoints
+- Portable test fixtures and hermetic local storage
+
+## Not a fit
+
+- Multiple writers across independent processes without external coordination
+- Cross-machine replication or consensus
+- SQL dialect compatibility or migrations from arbitrary relational schemas
+- Provider billing/token accounting
+- Large analytical datasets that do not fit comfortably in memory
+
+## Decision in one sentence
+
+Use a small, inspectable, dependency-free table kernel when portability and
+application-owned durability matter more than SQL compatibility or distributed
+database features.

package/docs/CONTRIBUTING.md

@@ -0,0 +1,82 @@
+# Contributing
+
+BroccoliDB is intentionally small and portable. Contributions should preserve
+that boundary and make behavior easier to inspect, test, and transfer.
+
+## Source map
+
+| Area | Location |
+|---|---|
+| Public exports | `src/index.ts` |
+| Public contracts | `src/broccolidb.contracts.ts` |
+| Kernel lifecycle | `src/broccolidb-kernel.ts` |
+| Tables and indexes | `src/broccolidb-table.ts` |
+| WAL | `src/broccolidb-wal.ts` |
+| CAS | `src/broccolidb-cas.ts` |
+| Queries and aggregation | `src/broccolidb-natural-query.ts`, `src/broccolidb-aggregation.ts` |
+| Locking | `src/broccolidb-mutex.ts` |
+| Prompt compression | `src/TokenCompressionService.ts` |
+| Tests | `test/` |
+| Documentation | `README.md`, `docs/`, `docs/adr/` |
+
+## Development workflow
+
+```bash
+npm install
+npm run build
+npm test
+npm run docs:check
+npm pack --dry-run
+```
+
+Run the smallest relevant test while iterating, then run `npm run check` before
+handoff. Tests should use temporary workspace roots and always clean them in a
+`finally` block.
+
+## Contract rules
+
+1. Keep the runtime dependency list empty unless a dependency is unavoidable,
+   justified in an ADR, and compatible with Node.js `>=18`.
+2. Preserve ESM-compatible relative imports and the public package export.
+3. Treat WAL, checkpoint, CAS, and serialized contract changes as compatibility
+   changes even when TypeScript still compiles.
+4. Add tests for startup/replay, checkpoint/rollback, integrity failures, and
+   query/index behavior when changing those areas.
+5. Keep `transaction()` callbacks deterministic and avoid network or long-lived
+   external work while holding the kernel mutex.
+6. Do not introduce an SQL or native-driver compatibility layer into the core.
+
+## Documentation rules
+
+Update the relevant layer whenever behavior changes:
+
+| Change | Required documentation |
+|---|---|
+| Export or signature | [API reference](API.md) and release notes |
+| Lifecycle, durability, or file format | [Architecture](ARCHITECTURE.md), [Operations](OPERATIONS.md), and an ADR |
+| Error or recovery behavior | [Troubleshooting](TROUBLESHOOTING.md) and operations guide |
+| Terminology | [Glossary](GLOSSARY.md) and affected docs |
+| Supported package/version policy | `README.md` and [release notes](RELEASE_NOTES.md) |
+
+Examples must import from `@noorm/broccolidb` and show lifecycle ownership.
+Avoid claims such as “transactional” or “durable” without naming the exact
+boundary (`flush`, `transaction`, `checkpoint`, or `stop`).
+
+## Pull request checklist
+
+- [ ] The public export surface is intentional.
+- [ ] Runtime dependencies remain portable and native-free.
+- [ ] TypeScript build passes.
+- [ ] Relevant tests pass, including restart/recovery tests where applicable.
+- [ ] `npm run docs:check` passes.
+- [ ] README/API/operations/ADR/release notes are updated as required.
+- [ ] No generated `dist/` or `.broccolidb/` runtime state was added accidentally.
+- [ ] `npm pack --dry-run` contains the intended docs and runtime files.
+
+## Release checklist
+
+1. Update `version` in `package.json` and the lockfile.
+2. Add a release-note entry describing API and persistence compatibility.
+3. Run `npm run check` and `npm pack --dry-run`.
+4. Inspect the tarball file list for `dist`, `README.md`, `docs`, and `LICENSE`.
+5. Verify a clean consumer can import the compiled package without dev tools.

package/docs/GLOSSARY.md

@@ -0,0 +1,23 @@
+# Glossary
+
+| Term | Meaning |
+|---|---|
+| **Table** | An in-memory collection of records keyed by string IDs and optionally backed by WAL frames. |
+| **Record** | A JSON-compatible application object stored under a table ID. |
+| **Index** | A secondary lookup structure maintained by a table for equality, sorted, composite, or prefix queries. |
+| **WAL** | Write-ahead log: append-only JSONL mutation frames used for durability and restart replay. |
+| **WAL frame** | One `INSERT`, `UPDATE`, `DELETE`, `CLEAR`, `CHECKPOINT`, `ROLLBACK`, or `BRANCH_MERGE` operation record with checksum metadata. |
+| **Flush** | Writing buffered WAL frames to `.broccolidb/wal.log`. |
+| **Checkpoint** | A complete JSON table snapshot plus a named history record that permits faster restart and rollback. |
+| **Rollback** | Restoring table state from a checkpoint held in memory or loaded from checkpoint history on disk. |
+| **CAS** | Content-addressable storage: blobs are named by the SHA-256 hash of their raw content. |
+| **Shard** | The two-character directory prefix used to spread CAS blob files across directories. |
+| **Quarantine** | Moving a corrupted CAS blob into `.broccolidb/cas/corrupt/` and recording an audit entry. |
+| **CDC** | Change-data capture: table subscriptions receive insert, update, delete, clear, and expiration events. |
+| **TTL** | Optional time-to-live on a write; expired records emit an `EXPIRE` change event. |
+| **Natural query** | An offline parser that turns constrained human-readable text into `DbQueryOptions`; it is not an LLM. |
+| **Fluent query** | A chainable `select().where(...).orderBy(...).execute()` builder over one table. |
+| **Transaction** | A kernel mutex scope that runs an async callback and flushes the WAL when the callback completes. |
+| **Health report** | A diagnostic snapshot covering directory writability, CAS metrics, WAL metrics, and table counts. |
+| **Workspace root** | The caller-selected filesystem root below which `.broccolidb/` is created. |
+| **Runtime dependency** | A package required by consumers at runtime. BroccoliDB has none; TypeScript, `tsx`, and Node types are development dependencies. |