@dikolab/kbdb 0.4.3 → 0.6.0

package/docs/goals/concurrency.md

@@ -1,259 +0,0 @@
-# Concurrent Write Safety
-
-*One writer at a time, unlimited concurrent readers --
-formalizing the locking model, multi-daemon prevention,
-crash recovery, and stress-test methodology.*
-
-## Goal
-
-The worker daemon serializes all write operations
-through a single-writer lock. Multiple concurrent
-readers proceed without any lock. This document
-formalizes the locking model, multi-daemon prevention,
-atomic write guarantees, crash recovery, and
-stress-test methodology.
-
-This goal addresses gap #3 from the issue
-[2026-06-23 Opus 4.8 Second Brain Review](../issues/2026-06-23-opus-48-second-brain-review.md):
-the single-writer lock in the worker daemon must be
-validated under concurrent load before concurrent
-multi-agent use can be trusted.
-
-## Single-Writer Lock
-
-The daemon acquires an exclusive file lock on
-`kbdb-{ctx}.lock` in the OS temporary directory during
-write operations. The lock file uses the same context
-ID as the PID file and IPC socket (see
-[Worker Daemon](worker-daemon.md) for context ID
-derivation).
-
-- **Read operations** -- search, recall, retrieve,
-  content, dbStatus, and listByType -- proceed without
-  acquiring the lock. Concurrent read requests from
-  multiple worker client connections are fully parallel.
-- **Write operations** -- addSection, updateSection,
-  removeSection, groupSections, removeGrouping,
-  initDirectory, gc, rebuildIndexes, and export --
-  acquire the exclusive lock before calling the
-  corresponding WASM export. If another write is
-  already in progress, the new write queues behind the
-  lock and waits.
-- The lock is held only for the duration of a single
-  write operation. It is released immediately after
-  the operation completes. It is never held across
-  multiple operations or across client connections.
-- The locking model is implemented in TypeScript inside
-  the worker daemon (`src/shared/worker-daemon/`), not
-  inside any WASM module.
-
-This model is equivalent to SQLite's default locking:
-readers never block, and writers take turns one at a
-time.
-
-### Write Operations Subject to the Lock
-
-| IPC Method               | WASM Export                |
-|--------------------------|----------------------------|
-| `addSection`             | `worker_add_section`       |
-| `updateSection`          | `worker_update_section`    |
-| `removeSection`          | `worker_remove_section`    |
-| `groupSections`          | `worker_group_sections`    |
-| `removeGrouping`         | `worker_remove_grouping`   |
-| `initDirectory`          | `worker_init_directory`    |
-| `gc`                     | `worker_garbage_collect`   |
-| `rebuildIndexes`         | `worker_rebuild_indexes`   |
-| `export`                 | `db_export`                |
-
-## Multi-Daemon Prevention
-
-Each database context gets exactly one daemon process
-at a time, enforced by a PID file keyed to the context
-ID.
-
-- The PID file is named `kbdb-{ctx}.pid` and lives in
-  the OS temporary directory. Its presence means a
-  daemon for that context is either running or died
-  without cleaning up.
-- Before spawning a new daemon, the worker client reads
-  the PID file (if present) and checks whether the
-  recorded process is alive.
-  - **PID file absent** -- no running daemon; spawn a
-    new one.
-  - **PID file present, process alive** -- connect to
-    the existing daemon; do not spawn.
-  - **PID file present, process dead** -- stale PID
-    file from a prior crash; remove the file and spawn
-    a new daemon.
-- The daemon writes its PID file only after all startup
-  preconditions pass and the WASM module is loaded
-  successfully. See [Worker Daemon](worker-daemon.md)
-  for the full startup sequence.
-- On graceful shutdown, the daemon removes its PID
-  file before exiting.
-
-This protocol ensures that the same database directory
-is never served by two concurrent daemon processes.
-
-## Atomic Write Guarantees
-
-All file writes in the database use the atomic
-write pattern: content is written to a temporary file
-first, then the temporary file is renamed to the final
-path. Rename is an atomic operation on all supported
-filesystems (POSIX and Windows NTFS).
-
-- **Section files** (`sections/{prefix}/{kbid}.sec`)
-  are written to `sections/{prefix}/{kbid}.sec.tmp`
-  then renamed.
-- **Document manifests** (`documents/{docid}.toml`)
-  are written to `documents/{docid}.toml.tmp` then
-  renamed.
-- **Index files** (`indexes/terms.idx`, etc.) are
-  rebuilt to temporary paths then renamed into place.
-- **Catalog** (`catalog.toml`) is written last,
-  after all data files are in place. A crash
-  before the catalog rename leaves the catalog
-  consistent with the previous complete state.
-
-If the daemon crashes mid-write, the temporary file
-is orphaned but the original file is intact. On next
-daemon startup, orphaned temporary files (matching
-`*.tmp`) in the database directory are cleaned up.
-
-The atomic write pattern is specified in
-[Database](database.md) under Storage Design.
-
-## Crash Recovery
-
-The table below describes every identified failure
-scenario, the resulting disk state, and the recovery
-action.
-
-| Scenario | State After Crash | Recovery |
-|---|---|---|
-| Crash during section write | `*.sec.tmp` orphaned, original `.sec` intact | Startup cleans `*.tmp` files |
-| Crash during index rebuild | Partial `*.idx.tmp`, originals intact | Startup cleans `*.tmp`; next `rebuildIndexes` overwrites |
-| Crash during GC | Some unreferenced sections deleted, others not | Re-run `gc`; idempotent |
-| Crash during catalog write | `catalog.toml.tmp` orphaned, original intact | Startup cleans `*.tmp` |
-| PID file not removed | Stale `kbdb-{ctx}.pid` on disk | Worker client detects dead process, removes PID, spawns new daemon |
-| Lock file not released | `kbdb-{ctx}.lock` orphaned | Daemon startup acquires the exclusive lock fresh; OS releases orphaned file locks when the process that held them exits |
-| WASM crash inside write | Partial in-memory state, temp file may exist | Daemon exits; startup cleans `*.tmp`; next operation starts clean |
-
-All recovery actions are automatic. No manual
-intervention is required for any scenario in this
-table.
-
-## Stress Test Specification
-
-The following test harness validates concurrent write
-safety. It must pass on both Deno and Node.js runtimes
-before a release can claim concurrent safety.
-
-### Setup
-
-1. Create a fresh, empty database directory.
-2. Prepare N unique documents (N in the range 10--50).
-   Each document must have distinct content to avoid
-   content-address collisions that would make section
-   count verification meaningless.
-
-### Execution
-
-3. Launch N concurrent `kbdb learn` invocations, all
-   targeting the same database directory. Each
-   invocation learns one of the N unique documents.
-4. Use a process pool or parallel shell to launch all
-   N invocations simultaneously -- not sequentially.
-5. Wait for all N invocations to complete. Record any
-   non-zero exit codes as failures.
-
-### Verification
-
-6. Run `kbdb check` against the database.
-   The response must have `ok: true`. Any errors in the
-   response are test failures.
-7. Query the section count. The count must equal the
-   expected number of sections derived from the N
-   documents (no duplicates from race conditions, no
-   losses from interleaved writes).
-8. Verify that no section has a checksum mismatch.
-   A mismatch indicates a torn write -- the atomic
-   rename pattern failed or was bypassed.
-
-### Pass Criteria
-
-| Check | Expected Result |
-|---|---|
-| All `kbdb learn` exit codes | `0` |
-| `kbdb check` response | `ok: true`, zero errors |
-| Section count | Exactly the expected count |
-| Checksum mismatches | Zero |
-
-### Runtime Matrix
-
-Run the full harness on both runtimes:
-
-| Runtime | Command prefix |
-|---|---|
-| Node.js >= 20 | `node dist/kbdb.mjs` |
-| Deno | `deno run --allow-all src/cli.ts` |
-
-Both must pass. A failure on one runtime but not the
-other indicates a runtime-specific locking bug.
-
-## Edge Cases
-
-### Two CLI Invocations Through the Same Daemon
-
-When two `kbdb` CLI processes connect to the same
-daemon (same database directory), their write requests
-share the single-writer lock automatically. No user
-action is required. The second write waits for the
-first to complete.
-
-### MCP Server Connections
-
-An MCP server connection routes all requests through
-a worker client to the daemon, the same as a CLI
-invocation. Multiple MCP client connections routed to
-the same daemon share the write lock:
-
-- Concurrent read requests from multiple MCP clients
-  are served in parallel.
-- Concurrent write requests from multiple MCP clients
-  are serialized through the lock one at a time.
-
-There is no per-connection or per-client lock; the
-lock is global to the daemon process.
-
-### Direct File Access
-
-Accessing or modifying `.kbdb` files directly -- via
-text editor, shell commands, or any process other than
-the daemon -- is not supported. Direct writes bypass
-the single-writer lock and the atomic write pattern,
-and may corrupt the database. All mutations must go
-through the daemon's IPC API.
-
-## Design Constraints
-
-- **Per-daemon locking, not per-file** -- The lock
-  covers all writes to a database context, not writes
-  to individual files. This means writes to unrelated
-  sections still serialize behind the same lock. The
-  simplicity of a single lock outweighs the throughput
-  cost at current scale. If per-section parallelism
-  becomes necessary, this constraint is the place to
-  revisit.
-- **Single-machine only** -- The locking model
-  relies on OS-level file locks and process identity
-  on a single machine. It provides no guarantees across
-  network filesystems or multiple machines. Cross-machine
-  scenarios require an external sync mechanism (see
-  [Sync](sync.md) when available).
-- **Lock implemented in TypeScript** -- The write lock
-  is managed by the TypeScript worker daemon layer, not
-  by any WASM module. WASM modules have no knowledge of
-  concurrent callers; they process one call at a time
-  through the host.