@dikolab/kbdb 0.5.0 → 0.6.0

package/docs/goals/benchmarking.md

@@ -1,296 +0,0 @@
-# Scale Benchmarking -- Methodology and Performance Targets
-
-*Quantify search latency, rebuild time, and memory
-usage across realistic corpus sizes before trusting
-kbdb as a long-lived knowledge base.*
-
-## Goal
-
-kbdb is intended as a long-lived knowledge base for AI
-agents. Before trusting it at scale, search latency,
-rebuild time, and memory usage must be quantified across
-realistic corpus sizes.
-
-This document specifies the benchmark methodology, corpus
-generation, metrics, and regression targets. It addresses
-gap #4 from the
-[Opus 4.8 Second Brain Review](../issues/2026-06-23-opus-48-second-brain-review.md):
-
-> Scale of the lexical index at 50k-100k+ sections --
-> search latency and rebuild time are worth benchmarking
-> before trusting it as a long-lived brain.
-
-## Corpus Generation
-
-A synthetic corpus generator lives in `bench/` and
-creates realistic sections for repeatable benchmarks.
-
-### Generator Properties
-
-- **Term distribution** -- Word frequencies follow a
-  Zipfian distribution. A small vocabulary of high-
-  frequency terms dominates each section, with a long
-  tail of rare terms. This matches real-world prose.
-- **Title length** -- 3 to 8 words drawn from the
-  vocabulary.
-- **Body length** -- 100 to 500 words, configurable via
-  `avg body length` parameter.
-- **Vocabulary** -- Terms drawn from a 50,000-word
-  English dictionary subset. The same dictionary is used
-  for all corpus sizes so cross-size comparisons are
-  valid.
-- **Cross-references** -- Sections reference each other
-  via `$(link:)` markers at a configurable density
-  (default: 5% of sections link to one other section).
-- **Tags** -- Each section is assigned one to three tags
-  from a pool of 20 topic tags.
-- **Determinism** -- The generator accepts a seed
-  parameter. Fixed seed produces identical corpus across
-  runs. Results are reproducible across machines.
-
-### Generator Parameters
-
-| Parameter | Default | Description |
-|-----------|---------|-------------|
-| `--count` | 1000 | Number of sections to generate |
-| `--avg-body-length` | 250 | Average body word count |
-| `--ref-density` | 0.05 | Fraction of sections with a cross-reference |
-| `--seed` | 42 | RNG seed for reproducibility |
-| `--out` | `./bench/corpus/` | Output directory |
-
-## Corpus Sizes
-
-| Label | Sections | Estimated disk size |
-|-------|----------|---------------------|
-| Small | 1,000 | ~5 MB |
-| Medium | 10,000 | ~50 MB |
-| Large | 50,000 | ~250 MB |
-| XLarge | 100,000 | ~500 MB |
-
-All corpus sizes use the same generator with the same
-seed and vocabulary. The only difference is the
-`--count` parameter.
-
-## Operations to Benchmark
-
-| Operation | Description | Variants |
-|-----------|-------------|----------|
-| `search` | Single-query search | cold cache, warm cache |
-| `learn` | Add a single section | with/without embedding |
-| `batch-learn` | Add 100 sections | sequential |
-| `rebuild` | Full index rebuild | lexical only, hybrid |
-| `gc` | Garbage collection | after superseding 10% of sections |
-| `check` | Integrity check | full scan |
-
-### Variant Definitions
-
-**Cold cache** -- the worker daemon is started fresh
-immediately before the timed operation. The WASM
-linear memory page table and any in-process caches
-are empty.
-
-**Warm cache** -- the timed operation is preceded by
-10 identical operations that are not counted. The
-daemon's in-process caches are populated.
-
-**With embedding** -- an embedding provider is
-configured. The `learn` operation includes embedding
-computation time.
-
-**After superseding 10%** -- before timing `gc`, 10%
-of all sections are superseded (replaced) using the
-`replace` flag, leaving their originals as unreferenced
-candidates for collection.
-
-## Metrics
-
-For each operation at each corpus size, the harness
-collects:
-
-- **p50, p95, p99 latency** (ms) -- percentiles across
-  N runs (default N = 10).
-- **Throughput** (ops/sec) -- for batch operations
-  (`batch-learn`), calculated as
-  `(sections_per_batch * N) / total_elapsed_s`.
-- **WASM linear memory high watermark** -- the
-  `WebAssembly.Memory` buffer byte size sampled
-  immediately after the operation completes.
-- **Index file sizes** -- sizes in bytes of
-  `terms.idx`, `positions.idx`, `corpus.dat`, and
-  `vectors.idx` (if present) at each corpus size.
-- **Resident memory of the worker daemon** -- OS-
-  reported RSS of the `kbdb-worker` process, sampled
-  after warm-up.
-
-## Performance Targets
-
-The targets below define the maximum acceptable latency
-for each operation at each corpus size. Results
-exceeding a target mark that combination as a
-regression.
-
-| Operation | Corpus | Target |
-|-----------|--------|--------|
-| `search` (warm) | 1k | < 5 ms |
-| `search` (warm) | 10k | < 15 ms |
-| `search` (warm) | 50k | < 30 ms |
-| `search` (warm) | 100k | < 50 ms |
-| `search` (cold) | 100k | < 200 ms |
-| `learn` (single) | any | < 100 ms |
-| `rebuild` | 1k | < 2 s |
-| `rebuild` | 10k | < 10 s |
-| `rebuild` | 100k | < 30 s |
-| `gc` | 100k | < 10 s |
-| `check` | 100k | < 15 s |
-
-Targets are per-operation p99 latency unless noted.
-`learn (single)` and `rebuild` targets apply to p99
-of the N-run sample.
-
-## Benchmark Harness
-
-All benchmark tooling lives in `bench/`:
-
-```
-bench/
-   generate-corpus.ts    corpus generator script
-   run-benchmark.ts      harness: run operations,
-                          collect timing, emit JSON
-   report.ts             format JSON results as
-                          Markdown table
-   corpus/               generated corpus files
-                          (gitignored)
-   results/              timestamped JSON result files
-   baselines/            per-platform baseline files
-                          for regression comparison
-```
-
-### Harness Procedure
-
-1. Generate the corpus at target size using
-   `bench/generate-corpus.ts` with the configured
-   seed.
-2. Run `db init` to create a fresh `.kbdb` directory.
-3. Run `batch-learn` to populate the database with the
-   full generated corpus.
-4. For each operation variant, run the operation N
-   times (default N = 10), collecting wall-clock
-   elapsed time per run.
-5. Compute p50, p95, p99 from the N samples.
-6. Collect WASM memory and daemon RSS after each
-   operation.
-7. Write a JSON result file to `bench/results/` with
-   a timestamp suffix.
-8. Optionally compare against the baseline file in
-   `bench/baselines/` and emit a pass/fail summary.
-
-The harness uses the public CLI and library API
-exclusively. It does not import internal modules or
-call WASM exports directly.
-
-### Result File Format
-
-Each run produces one JSON file:
-
-```json
-{
-  "timestamp": "2026-06-23T10:00:00Z",
-  "platform": "linux-x86_64",
-  "runtime": "node-22",
-  "corpus_size": "medium",
-  "results": [
-    {
-      "operation": "search",
-      "variant": "warm",
-      "n": 10,
-      "p50_ms": 8,
-      "p95_ms": 11,
-      "p99_ms": 14,
-      "wasm_memory_bytes": 4194304,
-      "daemon_rss_bytes": 52428800
-    }
-  ],
-  "index_sizes": {
-    "terms_idx_bytes": 1048576,
-    "positions_idx_bytes": 524288,
-    "corpus_dat_bytes": 65536,
-    "vectors_idx_bytes": null
-  }
-}
-```
-
-`vectors_idx_bytes` is `null` when no embedding
-provider is configured.
-
-## Hybrid Search Benchmarks
-
-When an embedding provider is configured, the harness
-adds measurements for the vector path. These are
-collected as a separate benchmark suite and not mixed
-with the lexical-only results.
-
-| Additional measurement | Description |
-|------------------------|-------------|
-| Embedding time per section | Elapsed ms for embedding computation during `learn` |
-| Vector search latency | p50/p95/p99 of vector-only retrieval |
-| Lexical search latency | p50/p95/p99 of lexical-only retrieval for the same queries |
-| RRF fusion overhead | p50/p95/p99 of the fusion step alone (subtract lexical + vector from total) |
-| Vector index rebuild time | p50/p95/p99 of `rebuild` with embedding enabled |
-| `vectors.idx` size | File size in bytes at each corpus size |
-
-Hybrid benchmarks require a running embedding provider.
-The harness accepts `--embedding-provider` as a CLI
-flag. When the flag is absent, hybrid measurements are
-skipped and the harness prints a notice.
-
-## Regression Testing
-
-Benchmark results from the current release are stored
-in `bench/baselines/`. Baseline files are named by
-platform and runtime:
-
-```
-bench/baselines/
-   linux-x86_64-node-22.json
-   linux-x86_64-deno-2.json
-```
-
-The harness compares fresh results against the
-baseline and flags any p99 value that exceeds the
-baseline by more than 20%. A flagged regression prints
-to stderr and exits with a non-zero code.
-
-Baselines are updated manually when a performance
-improvement is intentional. The update command:
-
-```sh
-node bench/run-benchmark.ts --update-baseline
-```
-
-CI runs the full benchmark suite on every release
-branch push. Regression failures block the release.
-
-## Design Constraints
-
-- Benchmarks run on a single machine. No distributed
-  or multi-node testing is in scope.
-- Corpus generation is deterministic. A fixed seed
-  always produces the same sections in the same order.
-  Results are reproducible across machines with the
-  same platform and runtime.
-- The harness uses the public CLI and library API, not
-  internal modules. This ensures benchmarks measure the
-  end-to-end path that production callers experience.
-- WASM memory measurement uses the
-  `WebAssembly.Memory` buffer byte size sampled via
-  the daemon's status response, which exposes the
-  current WASM linear memory page count. This is an
-  approximation of the high watermark, not a
-  continuous trace.
-- Index file sizes are measured after the harness
-  operation completes by reading file sizes from the
-  `.kbdb/indexes/` directory. They are informational
-  and not subject to regression thresholds.
-- Baseline files are per-platform and per-runtime.
-  A Linux x86_64 Node.js baseline is not compared
-  against a Deno run or a different architecture.