@ponraaj/dimple-sdk 0.0.0-alpha.5 → 0.0.0-alpha.7

package/README.md

@@ -1,35 +1,18 @@
 # @ponraaj/dimple-sdk
 
-A local-first memory system for AI agents. Store text memories (and whole
-codebases), retrieve them with hybrid semantic + lexical search, and let a
-**self-organizing topic tree + artifact graph + concepts** keep everything
-structured as it grows. Everything runs locally: ONNX embeddings on your
+A local-first memory system for AI agents: store text memories and whole
+codebases, retrieve them with hybrid semantic + lexical search, and let a
+self-organizing **topic tree + artifact graph + concepts** keep everything
+structured as it grows. Everything runs locally — ONNX embeddings on your
 CPU, SQLite files on disk.
 
-This is the programmatic SDK — the
-[`@ponraaj/dimple`](https://www.npmjs.com/package/@ponraaj/dimple) CLI is a
-thin facade over it.
+> **Documentation**: [dimple.ponraaj.me](https://dimple.ponraaj.me) —
+> quickstart, architecture, config reference, changelog ·
+> **Package**: [npmx.dev/package/@ponraaj/dimple](https://npmx.dev/package/@ponraaj/dimple) ·
+> **Status**: alpha (`0.0.0-alpha.n`) — the API is stabilizing.
 
-> **Status**: alpha (`0.0.0-alpha.n`). The API is stabilizing but may change.
-
-### The one-liner
-
-```ts
-import { makeDimple } from "@ponraaj/dimple-sdk"; // ← makeDimple, not dimple
-
-const dimple = await makeDimple({ config: "./dimple.jsonc" });
-const hits = await dimple.query("what did we decide about X", {
-  limit: 5,
-  contentType: "text/plain", // ← code-vs-message separation
-}); // query(text, options) — text is the FIRST positional argument
-await dimple.dispose();
-```
-
-Calling conventions: every method takes **plain arguments** (`query(text,
-opts)`, `write(input, opts)`, `index({ paths })`), every failure is a
-rejected promise with a closed `_tag` (`"Sdk.NotFound"`, `"Sdk.InputError"`,
-…), and `dispose()` is **required** before `process.exit(0)` — the durable
-worker keeps the event loop alive (`dispose()` stops it; it is idempotent).
+This is the programmatic SDK. The [`@ponraaj/dimple`](https://www.npmjs.com/package/@ponraaj/dimple)
+CLI is a thin facade over it.
 
 ## Install
 
@@ -37,16 +20,15 @@ worker keeps the event loop alive (`dispose()` stops it; it is idempotent).
 npm install @ponraaj/dimple-sdk
 ```
 
-Runtime dependencies are the only things installed (Effect, libSQL,
-Transformers.js, drizzle) — the workspace internals (`@dimple/*`) are
-bundled into the package, including fully **self-contained type
-declarations** (the publish build vendors every internal d.ts and rewrites
-the `@dimple/*` specifiers — plain consumers never need the workspace).
+Runtime deps only (Effect, libSQL, transformers) — workspace internals are
+bundled, types are self-contained, and the default install ships **zero
+postinstall scripts** (the local ONNX embedder is an optional peer,
+auto-installed on first use).
 
 ## Quickstart
 
 ```ts
-import { makeDimple } from "@ponraaj/dimple-sdk";
+import { makeDimple } from "@ponraaj/dimple-sdk"; // ← makeDimple, not dimple
 
 const dimple = await makeDimple({ config: "./dimple.jsonc" });
 
@@ -57,500 +39,59 @@ const mem = await dimple.write({
 });
 
 // hybrid semantic + FTS search
-const results = await dimple.query("how does the pipeline deploy", {
-  limit: 3,
-});
-
-// index real code — files + extracted functions/classes/methods become memories
-const indexed = await dimple.index({ paths: ["./src"] });
-
-await dimple.dispose(); // stop the worker, close DBs (idempotent)
-```
-
-## The problem & the approach
-
-**Problem**: naive memory stores scan every vector per query (cost grows
-with memory count), return flat results with no structure, and give the
-agent no notion of _why_ a memory is relevant, what supersedes it, or what
-contradicts it.
-
-**Approach** — three structures maintained automatically by a durable
-background worker:
-
-1. **Topic tree** — memories are routed by embedding similarity into tree
-   leaves. Queries descend the tree (beam search on centroids) and only
-   score the reached leaves: **search cost grows with depth, not with
-   memory count**. A durable worker splits overflowing leaves and merges
-   underflowing ones with deterministic algorithms (CF-triplet k-means) —
-   same embeddings ⇒ same tree, anywhere.
-2. **Hybrid retrieval** — vector (cosine) + FTS5 lexical legs fused with
-   Reciprocal Rank Fusion, optional recency boost, and per-result graph
-   context.
-3. **Artifact graph + concepts** — directed `memory_supports` /
-   `memory_supersedes` / `memory_contradicts` edges written by structural
-   rules, code indexing, the agent, and the optional **dream** job (LLM
-   classification of candidates + deterministic concept clustering).
-   Retrieval consumes the graph: superseded memories demote, conflicts
-   surface, supports ride along as provenance; concept filters give the
-   query sovereignty.
-
-All structural decisions are **deterministic**; everything is **durable**
-(two SQLite files, resume-safe jobs).
-
-## Configuration
-
-Pass a config file path or an inline object:
-
-```ts
-const dimple = await makeDimple({ config: "./dimple.jsonc" });
-// or — the inline object mirrors the JSON config exactly (quoted keys):
-const dimple = await makeDimple({
-  config: {
-    storeUrl: "file:./.dimple/dimple.db",
-    jobs: { dbUrl: "file:./.dimple/jobs.db" },
-    embedding: {
-      defaultModel: "minilm",
-      models: [
-        {
-          id: "minilm",
-          provider: "transformers",
-          model: "Xenova/all-MiniLM-L6-v2",
-          dimensions: 384,
-        },
-      ],
-    },
-  },
-});
-```
-
-### Config sections (`dimple.jsonc`)
+const hits = await dimple.query("how does the pipeline deploy", { limit: 3 });
 
-| Section                     | Fields                                                                                                                          | Notes                                      |
-| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------ |
-| `storeUrl`                  | string                                                                                                                          | memory store DB (`file:` or remote libSQL) |
-| `jobs`                      | `dbUrl`, `execution` (reserved), `concurrency`, `leaseTimeoutMs` (reserved), `maxAttempts` (1–100), `pollIntervalMs` (reserved) | separate DB for the durable queue          |
-| `embedding`                 | `defaultModel` + `models[]`                                                                                                     | **required** — see Models below            |
-| `topics`                    | tree shape (`minLeafItems`, `maxLeafItems`, …)                                                                                  | optional, defaults apply                   |
-| `graph`                     | graph config                                                                                                                    | optional                                   |
-| `llm`                       | `baseURL`, `apiKey`, `modelId`, `temperature`                                                                                   | enables `dream`/`repair`                   |
-| `logging`                   | `enabled`, `level`, `format` (`jsonl`\|`pretty`), `file`, `scopes`, `ringSize`                                                  | silent by default; `file` = durable JSONL  |
-| `defaultK` / `defaultLimit` | numbers                                                                                                                         | retrieval defaults                         |
-| `defaultTtlSecs`            | number                                                                                                                          | write TTL default                          |
+// index a codebase (tree-sitter: one unit per file + per definition)
+await dimple.index({ paths: ["./src"] });
 
-### Remote databases (Turso)
-
-`storeUrl` and `jobs.dbUrl` accept **remote libSQL URLs** (`libsql://`,
-`https://`, `ws://` — Turso or any libsql-compatible host). The same
-engine runs everything the local file does — FTS5, vector search, the
-topic tree, durable jobs — over the wire. Verified end-to-end against
-the real Turso service (`tests/store/turso-remote.test.ts`).
-
-Turso tokens are **per-database**, so each URL carries its own token.
-`dimple.jsonc` resolves `${ENV_VAR}` in values, so tokens stay out of
-config files:
-
-```jsonc
-{
-  "storeUrl": "libsql://<org>-<memdb>.turso.io?authToken=${TURSO_MEMORIES_TOKEN}",
-  "jobs": {
-    "dbUrl": "libsql://<org>-<jobsdb>.turso.io?authToken=${TURSO_JOBS_TOKEN}",
-  },
-  "embedding": {
-    /* hosted providers for serverless — local models are
-                    local-only (auto-install needs Node) */
-  },
-}
-```
-
-Notes:
-
-- The **jobs DB must also be remote** when the store is remote (the
-  durable queue + workflow state are cross-invocation)
-- `file:`-only behaviors (parent-dir creation, migration lock) skip
-  for remote URLs
-- A single-DB setup can share one URL + one token (`TURSO_TOKEN`)
-- For Cloudflare Workers the client swaps to the pure-JS entry
-  (`@libsql/client/web` — one bundler alias, no source changes)
-
-### Embedding models
-
-Local embeddings (`provider: "transformers"`) auto-install on first
-use — `@huggingface/transformers` is an **optional peer** so hosted-only
-consumers never pull its tree (the `sharp` + `onnxruntime-node`
-postinstall scripts that abort npm 11 installs). The first `embed()`
-installs it with the project's own package manager, scripts skipped
-(proven unnecessary for embedding), and retries in-process. Offline or
-restricted environments get a typed `Sdk.Embedding` error with the manual
-command. Set `autoInstall: false` on the model to disable:
-`embedding.models[].autoInstall: false`
-
-### Embedding models
-
-Two providers — **local** (Transformers.js ONNX, runs on your CPU, no
-keys) and **hosted** (any OpenAI-compatible endpoint, plus native
-provider families):
-
-```jsonc
-"embedding": {
-  "defaultModel": "bge",
-  "models": [
-    { "id": "minilm", "provider": "transformers",
-      "model": "Xenova/all-MiniLM-L6-v2", "dimensions": 384 },
-    { "id": "bge", "provider": "transformers",
-      "model": "Xenova/bge-small-en-v1.5", "dimensions": 384 },
-    { "id": "bge-base", "provider": "transformers",
-      "model": "Xenova/bge-base-en-v1.5", "dimensions": 768 },
-    { "id": "gemini", "provider": "openai",
-      "model": "gemini-embedding-2", "dimensions": 3072, "apiKey": "…",
-      "baseURL": "https://generativelanguage.googleapis.com/v1beta/openai/" },
-    { "id": "openai", "provider": "openai",
-      "model": "text-embedding-3-small", "dimensions": 1536, "apiKey": "…" }
-  ]
-}
+await dimple.dispose(); // required before process.exit — the durable worker
 ```
 
-**Hosted providers** — the config `provider` field selects the family:
-
-| `provider`         | Models                                                                                                                                                  | `baseURL` needed?                                           |
-| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------- |
-| `openai` (default) | any OpenAI-compatible endpoint — OpenAI, vLLM, **ollama** (`http://localhost:11434/v1`), Ollama Cloud, LM Studio, Together, Fireworks, Groq, Mistral, … | optional (defaults to OpenAI) — set it for custom endpoints |
-| `google`           | `gemini-embedding-001`, `gemini-embedding-2`, `gemini-embedding-2-preview` (native provider)                                                            | ❌ no                                                       |
-| `cohere`           | `embed-english-v3.0`, `embed-multilingual-v3.0`, … (native provider)                                                                                    | ❌ no                                                       |
-| `bedrock`          | `amazon.titan-embed-text-v2:0`, `amazon.nova-embed-text-v2:0`, `cohere.embed-*` (native provider; AWS creds from env)                                   | ❌ no                                                       |
-
-`dimensions` is sent as a provider option where the API supports it
-(openai-compatible/`outputDimensionality`/`outputDimension`/titan `dimensions`
-and nova `embeddingDimension`) and ALWAYS enforced on the output.
-Verified end-to-end against real `gemini-embedding-2` (3072-dim), local
-ollama `Qwen3-Embedding-0.6B` (1024-dim), and the openai-compatible
-contract.
-
-- `dimensions` must match the model's real output — a mismatch fails typed
-  on first use
-- Multiple models coexist: each gets its own vector tables
-  (`vec_<dims>_<model>`) and its own topic tree; queries route per model
-- **Per-model runtime options** (transformers provider, all optional):
-  `device` (`auto` | `gpu` | `cpu` | `wasm` | `webgpu` | `cuda` | `dml` |
-  `coreml` | `webnn` — `cpu`/`wasm` tested; GPU/CUDA needs the onnxruntime
-  CUDA libs, fails typed without them), `dtype` (`fp32` | `fp16` | `q8` |
-  `int8` | `uint8` | `q4` | `bnb4` | `q4f16` | `q2` | `q2f16` | `q1` |
-  `q1f16` — quantizes model WEIGHTS only, never the output vectors;
-  `fp32`/`q8`/`int8`/`q4`/`fp16` verified end-to-end, others depend on the
-  HF repo shipping that ONNX variant), `pooling` (`mean` | `cls` | `none`),
-  `normalize` (boolean), `cacheDir`
-- **DB-level vector quantization** via `vectorType` (`float32` |
-  `float16` | `float8` | `float1bit`): the per-memory search vectors are
-  stored as int8/bfloat16/binary blobs and searched with libSQL's native
-  `vector_distance_cos` — `float8` verified end-to-end at **395 B vs
-  1536 B (4× smaller)** for 384-dim. The topic tree's centroid math and
-  `memories.embedding` always stay float32.
-- **Switching models**: set `defaultModel` to a configured model (multi-
-  model configs coexist — each model gets its own tree + `vec_<dims>_<model>`
-  tables). After a restart the new default handles all writes/queries;
-  the old model's tree stays intact. Existing memories are backfilled via
-  the durable `memory.embed` job (re-embed + re-route into the new tree,
-  idempotent/resume-safe) — the store-level switch (`setActiveModel` +
-  `findStale`) drives it, reachable through the `{ core: … }` path. Until
-  backfilled, old memories remain in the old tree (queries descend the
-  ACTIVE model's tree), and the runtime is bound to one active model at a
-  time (per-model reads for another model fail typed with
-  `topics.ModelMismatch`).
-- Vectors are always stored as float32 (`F32_BLOB`, 4 B/dim: 384 = 1536 B,
-  768 = 3072 B) regardless of `dtype`
-
-### Power users: `{ core: … }`
-
-Instead of a config, pass full `CoreOptions` (store/jobs URLs, models,
-`defaultModelId`, `embedderFor` factory, topics/graph/logging/llm,
-`concurrency`, `maxAttempts`, `extraDefs`/`extraHandlers` for custom durable
-jobs — register `{ name: "my.job", handler }` and enqueue it via the
-runtime; `extraHandlers` entries override built-ins by name). This is the
-injection seam used by tests (synthetic embedders).
+Calling conventions: **plain arguments** (`query(text, opts)`,
+`write(input, opts)`, `index({ paths })`); every failure is a rejected
+promise with a closed `_tag` (`"Sdk.NotFound"`, `"Sdk.InputError"`, …);
+`dispose()` is idempotent.
 
 ## API surface
 
-### `write(input, options?)`
-
-```ts
-await dimple.write({
-  logicalKey: "deploy-1",        // required, non-empty
-  content: "…",                   // required
-  namespace?: "default",
-  contentType?: "text/plain",
-  ttlSecs?: 604800,
-  metadata?: "{\"json\":true}",   // JSON string
-  typedMetadata?: { … },
-}, { embed: true });              // embed + topic-route (default); false for externally-managed embeddings
-```
-
-Returns the persisted `MemoryUnit`: `{ id, logicalKey, content,
-contentType, state: "active" | "superseded" | …, createdAt, ttlSecs,
-namespace, metadata, embedding? }` (re-fetched after embedding). Duplicate
-active `logicalKey` → `Sdk.DuplicateKey` (carries `logicalKey`/`namespace`).
-
-**Memory expiry**: `ttlSecs` (or the config `defaultTtlSecs`) schedules
-expiry; a sweep job retires expired memories in the background.
+| Method                                                    | What it does                                              |
+| --------------------------------------------------------- | --------------------------------------------------------- |
+| `write(input, { embed })`                                 | Store a memory (idempotent per `logicalKey`)              |
+| `get(id)` / `forget(id)`                                  | Fetch / remove by id or logical key                       |
+| `query(text, options)`                                    | Hybrid search — profiles, budgets, content-type filter    |
+| `updateEmbedding(id, vec, modelKey?)`                     | Attach an external embedding                              |
+| `index({ paths })`                                        | Codebase ingestion (idempotent re-index)                  |
+| `graph.addEdge/confirm/reject/expire/candidates/coverage` | Memory relationships: supports / supersedes / contradicts |
+| `explain({ kind, id }, options)`                          | Bounded graph neighborhood                                |
+| `dream(modelKey?)` / `repair(modelKey?)`                  | Consolidation + maintenance jobs                          |
+| `health(modelKey?)`                                       | Topic-tree invariant health                               |
+| `reload(input?)`                                          | Blue-green config reload                                  |
+
+Errors are a closed hierarchy (`InputError`, `NotFound`, `DuplicateKey`,
+`JobFailed`, `LLMNotConfigured`, `StoreFailure`) — see the docs site.
 
-`query(text, { budget: { maxUnits, maxChars, maxMs } })` — LLM-facing
-context guardrail: caps the row count, truncates result content to the
-total char budget, and marks `results.truncated` when the soft time cap
-is exceeded.
-
-### `get(id)` / `forget(id)`
-
-Fetch one memory; soft-delete (cascades edges, rebalances the tree).
-
-### `query(text, options?)`
-
-```ts
-await dimple.query("how does the pipeline deploy", {
-  limit?: 10,
-  k?: 60,                          // RRF smoothing
-  namespace?: "default",
-  profile?: {
-    recencyBoost?: 0,              // 0 = off; hyperbolically decays with age
-    modeWeights?: { vector?: 0.5, fts?: 0.5 },
-    conceptFilter?: string[],      // concept ids (query sovereignty)
-  } | "hybrid" | "semantic" | "keyword",
-  topics?: { topicKeys?: string[] },
-});
-```
-
-Returns `AugmentedMemory[]` — each entry carries the memory, the fused
-score, its 1-based `rank`, and the graph context (`supersededBy`,
-`conflicts`, `supportedBy`). Zero-result queries resolve to `[]` (the
-absent-vector-table case is an empty dense leg, never an error).
-
-### `index(options)` — codebase ingestion
-
-```ts
-await dimple.index({
-  paths: ["./src", "./packages"],  // dirs (recursive) and/or files
-  keyPrefix?: "code-",
-  namespace?: "default",
-  embed?: true,
-  maxFileBytes?: 64 * 1024,
-  maxChunkBytes?: 8 * 1024,
-  extensions?: [".ts", ".tsx", ".js", ".jsx", …],
-});
-// → { files, indexed, updated, reconciled, skipped, commit, unsupported, failed }
-```
-
-- Tree-sitter parse (TypeScript/JavaScript/TSX/JSX) → the extractor
-  produces one **file unit** per file (with its import graph in metadata)
-  and one **chunk unit** per definition (function/class/method/interface/
-  type/enum/variable, with kind/name/line-range/docstring metadata)
-- Every unit embeds + topic-routes (unless `embed: false`)
-- AST-derived graph: relative imports resolve to indexed files and
-  class→method containment become **confirmed `memory_supports` edges**
-  (`source: "rule"`)
-- **Idempotent + upserting**: keys are cwd-anchored + deterministic
-  (chunk keys drop the line number, so edits ABOVE a definition update
-  it instead of duplicating it). Re-indexing skips unchanged units
-  (`skipped`), and changed units are **superseded + rewritten under the
-  same key** (`updated`) with a confirmed identity `memory_supersedes`
-  edge — stale copies never accumulate
-- **Edges re-form after `forget`**: phase-2 resolves endpoints by logical
-  key against the store (not just the current run), so a re-created unit
-  re-connects to its pre-existing dependencies
-- **Reconcile + rename detection**: units whose source vanished (deleted
-  file/definition) are demoted (`reconciled`, `source_removed` — gone
-  from search; domain-scoped to the walked roots, so `index lib` never
-  touches other files). A vanished unit whose embedding is a near-duplicate
-  of a new unit becomes a PENDING `memory_supersedes` (reason rename) —
-  `graph confirm` completes it. Code units are TTL-exempt
-  (`ttlSecs: null` — the index owns their lifecycle); `commit` stamps the
-  repo HEAD into every unit's metadata as `codeVersion`
-- **`code_calls` edges**: every call site inside a definition resolves to
-  the callee's chunk (same file, then imported files) → confirmed edges —
-  `explain(id, { edgeKinds: ["code_calls"] })` gives one-hop impact
-  analysis
-- Per-file failures are collected, never fatal
-
-### `updateEmbedding(id, embedding, modelKey?)` / `embed(text)`
-
-Attach an externally-computed embedding (completes the `{ embed: false }`
-workflow — the memory becomes semantically retrievable + topic-routed), or
-embed text with the SDK's own pipeline.
-
-### `dream(modelKey)` / `repair(modelKey)`
-
-Run the consolidation job (concept clustering + LLM edge classification;
-requires the `llm` config section) or the graph repair pass. Each call
-gets a fresh run id (always executes). Failures are typed:
-`Sdk.LLMNotConfigured`, `Sdk.JobFailed`.
-
-### `explain({ kind: "memory" | "topic" | "concept", id }, options?)`
-
-Graph neighborhood traversal — `{ nodes: [{ kind, id, depth, edgeKind }] }`.
-`options.edgeKinds` scopes the answer (memory-only navigation: pass the
-`memory_*` kinds; `code_calls` shows the call graph).
-
-### `health(modelKey)` / `graph.*`
-
-- `health` — topic-tree invariant sweep (`warnCount` + violation strings)
-- `graph.confirm(id)` / `graph.reject(id)` / `graph.expire(id)` — the
-  pending-edge lifecycle: promote to confirmed (a `memory_supersedes`
-  confirm demotes the old endpoint in the same transaction), reject, or
-  soft-delete. `Sdk.EdgeNotFound` for stale ids
-- `graph.addEdge({ fromMemoryId, toMemoryId, edgeKind: "memory_supports" |
-"memory_supersedes" | "memory_contradicts", source: "rule" | "llm" |
-"agent", weight?, confidence?, confirmed? })` — explicit edges; typed
-  validation (no self-loops, known kinds, weight > 0, confidence ∈ [0,1])
-- `graph.candidates()` — pending structural candidates + isolated count
-- `graph.coverage(modelKey)` — coverage %, pending, concept count
-
-### `reload(input?)` — apply a config change
-
-Pauses and re-boots with the re-resolved config:
-
-1. the config source is re-resolved **typed** — a `dimple.jsonc` path is
-   re-read from disk, an inline object re-validated (a bad config rejects
-   the reload with **zero impact** — nothing pauses);
-2. an **identical** config resolves as `{ reloaded: false }` — no pause,
-   no restart (safe to call from a file watcher);
-3. new calls wait, in-flight calls settle on the old stack;
-4. a fresh stack boots (new worker fleet, embedders, logger — jobs are
-   durable, so an interrupted handler's lease expires and re-claims after
-   boot);
-5. the swap: the old stack is disposed, the new one serves.
-
-Resolves with `{ config, changed, reloaded }` — `changed` flags which
-sections actually moved (`storeUrl` / `jobsUrl` / `rest`).
-
-Works for `{ config: … }`-built dimples (reload with no argument re-reads
-the original source). `{ core: … }`-built dimples have no config source
-and reject with `Sdk.Reload`. Boot failure leaves the previous system
-running (blue-green); a drain that cannot settle in-flight calls (30s)
-rejects with `Sdk.Reload` and nothing is torn down.
-
-**What happens to data when the database URL changes:** the old database
-file is preserved on disk but no longer served — nothing is migrated or
-merged, the new db is bootstrapped empty (a warning is logged for the
-switch; roll back with `reload(oldConfig)`). Pending jobs in a switched
-`jobs.dbUrl` stay durable in the old db, re-claimable by switching back
-or draining it externally.
-
-```ts
-await dimple.reload();                        // re-read dimple.jsonc
-await dimple.reload({ storeUrl: "file:…", … }); // or a fresh config
-```
-
-### `dispose()`
-
-Stops the worker, closes sinks + the store. Idempotent; post-dispose calls
-reject with a closed `Sdk.StoreFailure`.
-
-## Errors — the closed hierarchy
-
-Every promise rejects with typed errors from one family (never raw Effect
-internals). Export surface: `ConfigError`, `NotFoundError`,
-`DuplicateKeyError` (with `logicalKey`/`namespace`), `DuplicateEdgeError`,
-`LLMNotConfiguredError`, `JobFailedError`, `StoreFailure`, `InputError`,
-plus the `remapError` helper (maps unknown failures onto the hierarchy).
-Also importable from the `@ponraaj/dimple-sdk/errors` subpath.
-
-**Type exports** (all importable from the package root): `Dimple` (the
-facade interface), `MakeDimpleOptions`, `ConfigInput`, `QueryOptions`,
-`IndexOptions`, `IndexResult`, plus the error classes and the `DimpleError`
-union type. Errors also carry stable `_tag` values (`Sdk.DuplicateKey`,
-`Sdk.NotFound`, `Sdk.InputError`, `Sdk.JobFailed`, `Sdk.LLMNotConfigured`,
-`Sdk.StoreFailure`, …) — the recommended discriminator.
-
-## Background jobs (what runs while you work)
-
-| Job                                               | Trigger             | Work                                                                                                                       |
-| ------------------------------------------------- | ------------------- | -------------------------------------------------------------------------------------------------------------------------- |
-| `topic.split` / `topic.merge` / `topic.rebalance` | automatic per write | keep every leaf within its size budget (deterministic splits, donor borrow, nearest-sibling merge, uneven-split rebalance) |
-| `memory.embed`                                    | model backfill      | re-embed a memory with a target model + route into its tree                                                                |
-| `memory.dream`                                    | `dream()`           | concept clustering + LLM edge classification                                                                               |
-| `graph.repair`                                    | `repair()`          | graph invariant repair                                                                                                     |
-| `audit.write`                                     | query path          | async audit-log writes (never blocks queries) — one FUSED row per hybrid query with the per-leg breakdown                  |
-
-Jobs are durable (leased, retried up to `maxAttempts`, resume-safe across
-processes). Every `makeDimple` boots the worker; `dispose` stops it.
-
-Every `query` also writes a retrieval audit row (query, namespace, mode,
-per-leg counts/scores, sources) into `retrieval_log` — machine-readable
-query history, persisted asynchronously (never blocks the query).
-
-## Storage & logs
+## Configuration
 
-| What                           | Where                                                                                        |
-| ------------------------------ | -------------------------------------------------------------------------------------------- |
-| Memories, tree, graph, audit   | `storeUrl` (`file:…`)                                                                        |
-| Durable queue + workflow state | `jobs.dbUrl` (separate DB)                                                                   |
-| Logs                           | stdout + in-memory ring (4096) by default; durable JSONL **only** when `logging.file` is set |
-| Grammar WASMs                  | `~/.cache/dimple/tree-sitter/` (downloaded once, offline after)                              |
-| ONNX model                     | bun/npm cache (~90 MB, first use)                                                            |
+`makeDimple({ config: "./dimple.jsonc" })` — every section is optional
+except `embedding`. `dimple init` (CLI) writes the full annotated schema;
+`${ENV_VAR}` placeholders keep tokens out of files. The complete reference
+(store / jobs / embeddings / topics / graph / llm / logging / Turso) is on
+the docs site: [Configuration reference](https://dimple.ponraaj.me/reference/config).
 
-## Changelog
+## MCP server
 
-Every release creates a GitHub Release with the per-version changes (the
-tag-based publish workflow): repo → **Releases** page.
+`dimple mcp` (from the CLI package) exposes dimple to any MCP client over
+the 2026-07-28 protocol — API discovery plus sandboxed code execution with
+the whole API as a `dimple` namespace. Guide:
+[MCP server & code mode](https://dimple.ponraaj.me/guides/mcp).
 
 ## Agent integration patterns
 
-These are the shapes that work well when the consumer is another agent
-(a harness) rather than a human with a REPL.
-
-### Session-start injection
-
-The guaranteed-usage path: a warmup query returns a compact memory block
-the harness injects as context, and the pending-graph review list rides
-along:
-
-```ts
-const [context, pending] = await Promise.all([
-  dimple.query("ongoing work and decisions", { limit: 8, budget: { maxChars: 4000 } }),
-  dimple.graph.candidates(),
-]);
-// context  → inject the top-N content strings into the system prompt
-// pending  → offer each candidate as a confirm/reject decision
-```
-
-`graph.candidates()` pending `memory_supersedes` edges from the rename
-scanner (rule 0.9x near-duplicates) are the highest-value batch to review.
-
-### Supersedes recipe
+Session-start injection, the supersedes recipe, and namespaces as the
+code/message partition are covered on the docs site (Architecture section)
+and in `docs/MCP.md` in the repository.
 
-`addEdge({ … edgeKind: "memory_supersedes", confirmed: true })` performs
-the full state flip in one transaction (the old memory demotes
-immediately) — it is safe to use directly. For human-gated flows the
-pending → confirm pair keeps the old memory ranking until a reviewer
-agrees:
-
-```ts
-const edge = await dimple.graph.addEdge({
-  fromMemoryId: oldId,
-  toMemoryId: newId,
-  edgeKind: "memory_supersedes",
-  source: "agent",
-}); // pending — nothing changed yet
-await dimple.graph.confirm(edge.id); // ← the flip happens here
-```
-
-### Namespaces as the code/message partition
-
-Until a query-time content-type filter exists, namespaces give clean
-isolation: `index({ paths, namespace: "code" })` and
-`write({ …, namespace: "messages" })` partition the store; query with
-`namespace: "code"` for code and `namespace: "messages"` for
-conversations. (`query({ contentType })` now does the same within one
-namespace — the two levers compose.)
-
-### Remember
-
-- `dimple.reload(newConfig)` swaps the whole stack blue-green (validate →
-  pause → boot → drain → swap); a failed boot leaves the old stack
-  serving.
-- JSON output is compact by default: CLI `--json` omits embeddings (pass
-  `--embedding` to include them); the SDK always keeps them on
-  `memory.embedding`.
-
-## Notes
+## Changelog
 
-- Determinism: same data + same config ⇒ same tree, same clusters, same
-  audit hashes — reproducible across machines and runs
-- `migrationFolder` is accepted in `makeDimple` options for advanced users;
-  the default path uses migrations embedded at build time (nothing to ship)
+[Release history](https://dimple.ponraaj.me/changelog/sdk) · repository
+[`CHANGELOG.md`](./CHANGELOG.md)

package/dist/index.js

@@ -508,7 +508,10 @@ var consoleSink = (pretty) => {
   };
 };
 var fileSink = (path) => Effect5.tryPromise(async () => {
-  const { openSync, closeSync, fsyncSync, writeSync } = await import("node:fs");
+  const fs = await import("node:fs");
+  const { dirname } = await import("node:path");
+  fs.mkdirSync(dirname(path), { recursive: true });
+  const { openSync, closeSync, fsyncSync, writeSync } = fs;
   const fd = openSync(path, "a");
   let disposed = false;
   const flush = () => {
@@ -2971,17 +2974,13 @@ var workflowTagFor = (workflows, store, executionId) => Effect17.gen(function* (
     return;
   return workflows.get(record.workflowTag)?.workflow;
 });
-var ensureDbDir = (url) => {
-  if (url.startsWith("file:")) {
-    import("node:fs").then((fs) => import("node:path").then(({ dirname }) => {
-      fs.mkdirSync(dirname(url.slice("file:".length)), { recursive: true });
-    }));
-  }
-};
-var libsqlStack = (options) => {
-  ensureDbDir(options.url);
-  return Layer6.mergeAll(PersistenceNamespace2.layerSql, PersistedQueue2.layer.pipe(Layer6.provide(storeSqliteLayer))).pipe(Layer6.provide(exports_LibsqlClient.layer(options)));
-};
+var ensureDbDir = (url) => url.startsWith("file:") ? import("node:fs").then((fs) => import("node:path").then(({ dirname }) => {
+  fs.mkdirSync(dirname(url.slice("file:".length)), { recursive: true });
+})) : Promise.resolve();
+var libsqlStack = (options) => Layer6.unwrap(Effect17.tryPromise({
+  try: () => ensureDbDir(options.url),
+  catch: (cause) => new Error(String(cause))
+}).pipe(Effect17.catch(() => Effect17.void), Effect17.map(() => Layer6.mergeAll(PersistenceNamespace2.layerSql, PersistedQueue2.layer.pipe(Layer6.provide(storeSqliteLayer))).pipe(Layer6.provide(exports_LibsqlClient.layer(options))))));
 var layerDurableLibsql = (options) => layerDurable.pipe(Layer6.provide(libsqlStack(options)));
 
 // ../workflow/src/runtime.ts
@@ -4734,6 +4733,36 @@ var makeStore2 = (backend, modelConfig, topicsOptions, graphConfig, auditOptions
     if (!result)
       return yield* new NotFound({ id: key });
   });
+  const decrementTreeCf = (model, membership, x) => Effect26.tryPromise({
+    try: () => db.transaction(async (tx) => {
+      const leafRows = await tx.select({
+        topicKey: topics.topicKey,
+        memberCount: topics.memberCount,
+        sumSquares: topics.sumSquares
+      }).from(topics).where(and8(eq8(topics.topicKey, membership.topicKey), eq8(topics.modelKey, membership.modelKey)));
+      const leaf = leafRows[0];
+      if (!leaf)
+        return;
+      const next = cfRemove({
+        count: Number(leaf.memberCount),
+        ls: new Float32Array(0),
+        ss: Number(leaf.sumSquares)
+      }, x);
+      await tx.update(topics).set({
+        memberCount: Math.max(0, next.count),
+        sumSquares: Math.max(0, next.ss),
+        updatedAt: Date.now()
+      }).where(and8(eq8(topics.topicKey, membership.topicKey), eq8(topics.modelKey, membership.modelKey)));
+      const relRows = await tx.select({ parentKey: topicRelations.parentKey }).from(topicRelations).where(and8(eq8(topicRelations.childKey, membership.topicKey), eq8(topicRelations.modelKey, membership.modelKey), eq8(topicRelations.relationKind, "parent_child")));
+      let ancestorKey = relRows[0]?.parentKey;
+      while (ancestorKey) {
+        await tx.update(topics).set({ memberCount: sql7`max(0, ${topics.memberCount} - 1)`, updatedAt: Date.now() }).where(and8(eq8(topics.topicKey, ancestorKey), eq8(topics.modelKey, membership.modelKey)));
+        const up = await tx.select({ parentKey: topicRelations.parentKey }).from(topicRelations).where(and8(eq8(topicRelations.childKey, ancestorKey), eq8(topicRelations.modelKey, membership.modelKey), eq8(topicRelations.relationKind, "parent_child")));
+        ancestorKey = up[0]?.parentKey;
+      }
+    }),
+    catch: (cause) => new StoreError({ message: `Store: tree CF decrement: ${cause}` })
+  });
   const forget = Effect26.fn("Store.forget")(function* (id) {
     let membership;
     if (topicsEnabled) {
@@ -4747,37 +4776,7 @@ var makeStore2 = (backend, modelConfig, topicsOptions, graphConfig, auditOptions
         const vectorRows = yield* backend.fetchMemoryVectors([id], modelConfig);
         if (vectorRows.length > 0) {
           const x = new Float32Array(vectorRows[0].vector);
-          const now = Date.now();
-          yield* Effect26.tryPromise({
-            try: () => db.transaction(async (tx) => {
-              const leafRows = await tx.select({
-                topicKey: topics.topicKey,
-                memberCount: topics.memberCount,
-                sumSquares: topics.sumSquares
-              }).from(topics).where(and8(eq8(topics.topicKey, mem.topicKey), eq8(topics.modelKey, mem.modelKey)));
-              const leaf = leafRows[0];
-              if (!leaf)
-                return;
-              const next = cfRemove({
-                count: Number(leaf.memberCount),
-                ls: new Float32Array(0),
-                ss: Number(leaf.sumSquares)
-              }, x);
-              await tx.update(topics).set({
-                memberCount: Math.max(0, next.count),
-                sumSquares: Math.max(0, next.ss),
-                updatedAt: now
-              }).where(and8(eq8(topics.topicKey, mem.topicKey), eq8(topics.modelKey, mem.modelKey)));
-              const relRows = await tx.select({ parentKey: topicRelations.parentKey }).from(topicRelations).where(and8(eq8(topicRelations.childKey, mem.topicKey), eq8(topicRelations.modelKey, mem.modelKey), eq8(topicRelations.relationKind, "parent_child")));
-              let ancestorKey = relRows[0]?.parentKey;
-              while (ancestorKey) {
-                await tx.update(topics).set({ memberCount: sql7`max(0, ${topics.memberCount} - 1)`, updatedAt: now }).where(and8(eq8(topics.topicKey, ancestorKey), eq8(topics.modelKey, mem.modelKey)));
-                const up = await tx.select({ parentKey: topicRelations.parentKey }).from(topicRelations).where(and8(eq8(topicRelations.childKey, ancestorKey), eq8(topicRelations.modelKey, mem.modelKey), eq8(topicRelations.relationKind, "parent_child")));
-                ancestorKey = up[0]?.parentKey;
-              }
-            }),
-            catch: (cause) => new StoreError({ message: `Store.forget: tree CF: ${cause}` })
-          });
+          yield* decrementTreeCf(modelConfig, mem, x);
           yield* updateTopicVectorForget(backend, modelConfig, mem.topicKey, x);
         }
       }
@@ -5004,6 +5003,28 @@ var makeStore2 = (backend, modelConfig, topicsOptions, graphConfig, auditOptions
       return yield* new NotFound({ id });
     if (modelConfig && modelConfig.id === modelName) {
       yield* backend.ensureVectorTable(modelConfig);
+      if (topicsEnabled) {
+        const memberRows = yield* Effect26.tryPromise({
+          try: () => db.select({
+            topicKey: topicMemberships.topicKey,
+            modelKey: topicMemberships.modelKey
+          }).from(topicMemberships).where(eq8(topicMemberships.memoryId, id)),
+          catch: (cause) => new StoreError({ message: `Store.updateEmbedding: load membership: ${cause}` })
+        });
+        const mem = memberRows[0];
+        if (mem && mem.modelKey === modelName) {
+          const oldVec = yield* backend.fetchMemoryVectors([id], modelConfig);
+          if (oldVec.length > 0) {
+            const x = new Float32Array(oldVec[0].vector);
+            yield* decrementTreeCf(modelConfig, mem, x);
+            yield* updateTopicVectorForget(backend, modelConfig, mem.topicKey, x);
+          }
+          yield* Effect26.tryPromise({
+            try: () => db.delete(topicMemberships).where(and8(eq8(topicMemberships.memoryId, id), eq8(topicMemberships.modelKey, modelName))),
+            catch: (cause) => new StoreError({ message: `Store.updateEmbedding: delete membership: ${cause}` })
+          });
+        }
+      }
       yield* backend.insertVector(id, embedding, modelConfig);
       if (topicsEnabled) {
         const routed = yield* routeInsert(id, embedding, modelConfig, topicsConfig).pipe(Effect26.provideService(Service2, backend), Effect26.mapError((cause) => new StoreError({ message: `Store.updateEmbedding topic route: ${cause}` })));
@@ -6856,8 +6877,8 @@ var layer11 = (options) => Layer13.effect(Service10, Effect35.gen(function* () {
     const vWeight = profile.modeWeights?.vector ?? 1;
     const fWeight = profile.modeWeights?.fts ?? 1;
     const mode = vWeight === 0 && fWeight > 0 ? "keyword" : fWeight === 0 && vWeight > 0 ? "semantic" : "hybrid";
-    const denseRanks = vectorResults.length > 0 ? vectorResults.map((s) => [s.memory.id, s.score * vWeight]) : null;
-    const fts5Ranks = fts5Results.length > 0 ? fts5Results.map((s) => [s.memory.id, s.score * fWeight]) : null;
+    const denseRanks = vWeight > 0 && vectorResults.length > 0 ? vectorResults.map((s) => [s.memory.id, s.score * vWeight]) : null;
+    const fts5Ranks = fWeight > 0 && fts5Results.length > 0 ? fts5Results.map((s) => [s.memory.id, s.score * fWeight]) : null;
     if (!denseRanks && !fts5Ranks) {
       yield* store.logAudit({
         query: query2,

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@ponraaj/dimple-sdk",
-  "version": "0.0.0-alpha.5",
+  "version": "0.0.0-alpha.7",
   "private": false,
   "files": [
     "dist"