npm - @ponraaj/dimple-sdk - Versions diffs - 0.0.0-alpha.5 → 0.0.0-alpha.7 - Mend

@ponraaj/dimple-sdk 0.0.0-alpha.5 → 0.0.0-alpha.7

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (3) hide show

package/README.md CHANGED Viewed

@@ -1,35 +1,18 @@
 # @ponraaj/dimple-sdk
-A local-first memory system for AI agents. Store text memories (and whole
-codebases), retrieve them with hybrid semantic + lexical search, and let a
-**self-organizing topic tree + artifact graph + concepts** keep everything
-structured as it grows. Everything runs locally: ONNX embeddings on your
+A local-first memory system for AI agents: store text memories and whole
+codebases, retrieve them with hybrid semantic + lexical search, and let a
+self-organizing **topic tree + artifact graph + concepts** keep everything
+structured as it grows. Everything runs locally — ONNX embeddings on your
 CPU, SQLite files on disk.
-This is the programmatic SDK — the
-[`@ponraaj/dimple`](https://www.npmjs.com/package/@ponraaj/dimple) CLI is a
-thin facade over it.
+> **Documentation**: [dimple.ponraaj.me](https://dimple.ponraaj.me) —
+> quickstart, architecture, config reference, changelog ·
+> **Package**: [npmx.dev/package/@ponraaj/dimple](https://npmx.dev/package/@ponraaj/dimple) ·
+> **Status**: alpha (`0.0.0-alpha.n`) — the API is stabilizing.
-> **Status**: alpha (`0.0.0-alpha.n`). The API is stabilizing but may change.
-### The one-liner
-```ts
-import { makeDimple } from "@ponraaj/dimple-sdk"; // ← makeDimple, not dimple
-const dimple = await makeDimple({ config: "./dimple.jsonc" });
-const hits = await dimple.query("what did we decide about X", {
-  limit: 5,
-  contentType: "text/plain", // ← code-vs-message separation
-}); // query(text, options) — text is the FIRST positional argument
-await dimple.dispose();
-```
-Calling conventions: every method takes **plain arguments** (`query(text,
-opts)`, `write(input, opts)`, `index({ paths })`), every failure is a
-rejected promise with a closed `_tag` (`"Sdk.NotFound"`, `"Sdk.InputError"`,
-…), and `dispose()` is **required** before `process.exit(0)` — the durable
-worker keeps the event loop alive (`dispose()` stops it; it is idempotent).
+This is the programmatic SDK. The [`@ponraaj/dimple`](https://www.npmjs.com/package/@ponraaj/dimple)
+CLI is a thin facade over it.
 ## Install
@@ -37,16 +20,15 @@ worker keeps the event loop alive (`dispose()` stops it; it is idempotent).
 npm install @ponraaj/dimple-sdk
 ```
-Runtime dependencies are the only things installed (Effect, libSQL,
-Transformers.js, drizzle) — the workspace internals (`@dimple/*`) are
-bundled into the package, including fully **self-contained type
-declarations** (the publish build vendors every internal d.ts and rewrites
-the `@dimple/*` specifiers — plain consumers never need the workspace).
+Runtime deps only (Effect, libSQL, transformers) — workspace internals are
+bundled, types are self-contained, and the default install ships **zero
+postinstall scripts** (the local ONNX embedder is an optional peer,
+auto-installed on first use).
 ## Quickstart
 ```ts
-import { makeDimple } from "@ponraaj/dimple-sdk";
+import { makeDimple } from "@ponraaj/dimple-sdk"; // ← makeDimple, not dimple
 const dimple = await makeDimple({ config: "./dimple.jsonc" });
@@ -57,500 +39,59 @@ const mem = await dimple.write({
 });
 // hybrid semantic + FTS search
-const results = await dimple.query("how does the pipeline deploy", {
-  limit: 3,
-});
-// index real code — files + extracted functions/classes/methods become memories
-const indexed = await dimple.index({ paths: ["./src"] });
-await dimple.dispose(); // stop the worker, close DBs (idempotent)
-```
-## The problem & the approach
-**Problem**: naive memory stores scan every vector per query (cost grows
-with memory count), return flat results with no structure, and give the
-agent no notion of _why_ a memory is relevant, what supersedes it, or what
-contradicts it.
-**Approach** — three structures maintained automatically by a durable
-background worker:
-1. **Topic tree** — memories are routed by embedding similarity into tree
-   leaves. Queries descend the tree (beam search on centroids) and only
-   score the reached leaves: **search cost grows with depth, not with
-   memory count**. A durable worker splits overflowing leaves and merges
-   underflowing ones with deterministic algorithms (CF-triplet k-means) —
-   same embeddings ⇒ same tree, anywhere.
-2. **Hybrid retrieval** — vector (cosine) + FTS5 lexical legs fused with
-   Reciprocal Rank Fusion, optional recency boost, and per-result graph
-   context.
-3. **Artifact graph + concepts** — directed `memory_supports` /
-   `memory_supersedes` / `memory_contradicts` edges written by structural
-   rules, code indexing, the agent, and the optional **dream** job (LLM
-   classification of candidates + deterministic concept clustering).
-   Retrieval consumes the graph: superseded memories demote, conflicts
-   surface, supports ride along as provenance; concept filters give the
-   query sovereignty.
-All structural decisions are **deterministic**; everything is **durable**
-(two SQLite files, resume-safe jobs).
-## Configuration
-Pass a config file path or an inline object:
-```ts
-const dimple = await makeDimple({ config: "./dimple.jsonc" });
-// or — the inline object mirrors the JSON config exactly (quoted keys):
-const dimple = await makeDimple({
-  config: {
-    storeUrl: "file:./.dimple/dimple.db",
-    jobs: { dbUrl: "file:./.dimple/jobs.db" },
-    embedding: {
-      defaultModel: "minilm",
-      models: [
-        {
-          id: "minilm",
-          provider: "transformers",
-          model: "Xenova/all-MiniLM-L6-v2",
-          dimensions: 384,
-        },
-      ],
-    },
-  },
-});
-```
-### Config sections (`dimple.jsonc`)
+const hits = await dimple.query("how does the pipeline deploy", { limit: 3 });
-| Section                     | Fields                                                                                                                          | Notes                                      |
-| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------ |
-| `storeUrl`                  | string                                                                                                                          | memory store DB (`file:` or remote libSQL) |
-| `jobs`                      | `dbUrl`, `execution` (reserved), `concurrency`, `leaseTimeoutMs` (reserved), `maxAttempts` (1–100), `pollIntervalMs` (reserved) | separate DB for the durable queue          |
-| `embedding`                 | `defaultModel` + `models[]`                                                                                                     | **required** — see Models below            |
-| `topics`                    | tree shape (`minLeafItems`, `maxLeafItems`, …)                                                                                  | optional, defaults apply                   |
-| `graph`                     | graph config                                                                                                                    | optional                                   |
-| `llm`                       | `baseURL`, `apiKey`, `modelId`, `temperature`                                                                                   | enables `dream`/`repair`                   |
-| `logging`                   | `enabled`, `level`, `format` (`jsonl`\|`pretty`), `file`, `scopes`, `ringSize`                                                  | silent by default; `file` = durable JSONL  |
-| `defaultK` / `defaultLimit` | numbers                                                                                                                         | retrieval defaults                         |
-| `defaultTtlSecs`            | number                                                                                                                          | write TTL default                          |
+// index a codebase (tree-sitter: one unit per file + per definition)
+await dimple.index({ paths: ["./src"] });
-### Remote databases (Turso)
-`storeUrl` and `jobs.dbUrl` accept **remote libSQL URLs** (`libsql://`,
-`https://`, `ws://` — Turso or any libsql-compatible host). The same
-engine runs everything the local file does — FTS5, vector search, the
-topic tree, durable jobs — over the wire. Verified end-to-end against
-the real Turso service (`tests/store/turso-remote.test.ts`).
-Turso tokens are **per-database**, so each URL carries its own token.
-`dimple.jsonc` resolves `${ENV_VAR}` in values, so tokens stay out of
-config files:
-```jsonc
-{
-  "storeUrl": "libsql://<org>-<memdb>.turso.io?authToken=${TURSO_MEMORIES_TOKEN}",
-  "jobs": {
-    "dbUrl": "libsql://<org>-<jobsdb>.turso.io?authToken=${TURSO_JOBS_TOKEN}",
-  },
-  "embedding": {
-    /* hosted providers for serverless — local models are
-                    local-only (auto-install needs Node) */
-  },
-}
-```
-Notes:
-- The **jobs DB must also be remote** when the store is remote (the
-  durable queue + workflow state are cross-invocation)
-- `file:`-only behaviors (parent-dir creation, migration lock) skip
-  for remote URLs
-- A single-DB setup can share one URL + one token (`TURSO_TOKEN`)
-- For Cloudflare Workers the client swaps to the pure-JS entry
-  (`@libsql/client/web` — one bundler alias, no source changes)
-### Embedding models
-Local embeddings (`provider: "transformers"`) auto-install on first
-use — `@huggingface/transformers` is an **optional peer** so hosted-only
-consumers never pull its tree (the `sharp` + `onnxruntime-node`
-postinstall scripts that abort npm 11 installs). The first `embed()`
-installs it with the project's own package manager, scripts skipped
-(proven unnecessary for embedding), and retries in-process. Offline or
-restricted environments get a typed `Sdk.Embedding` error with the manual
-command. Set `autoInstall: false` on the model to disable:
-`embedding.models[].autoInstall: false`
-### Embedding models
-Two providers — **local** (Transformers.js ONNX, runs on your CPU, no
-keys) and **hosted** (any OpenAI-compatible endpoint, plus native
-provider families):
-```jsonc
-"embedding": {
-  "defaultModel": "bge",
-  "models": [
-    { "id": "minilm", "provider": "transformers",
-      "model": "Xenova/all-MiniLM-L6-v2", "dimensions": 384 },
-    { "id": "bge", "provider": "transformers",
-      "model": "Xenova/bge-small-en-v1.5", "dimensions": 384 },
-    { "id": "bge-base", "provider": "transformers",
-      "model": "Xenova/bge-base-en-v1.5", "dimensions": 768 },
-    { "id": "gemini", "provider": "openai",
-      "model": "gemini-embedding-2", "dimensions": 3072, "apiKey": "…",
-      "baseURL": "https://generativelanguage.googleapis.com/v1beta/openai/" },
-    { "id": "openai", "provider": "openai",
-      "model": "text-embedding-3-small", "dimensions": 1536, "apiKey": "…" }
-  ]
-}
+await dimple.dispose(); // required before process.exit — the durable worker
 ```
-**Hosted providers** — the config `provider` field selects the family:
-| `provider`         | Models                                                                                                                                                  | `baseURL` needed?                                           |
-| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------- |
-| `openai` (default) | any OpenAI-compatible endpoint — OpenAI, vLLM, **ollama** (`http://localhost:11434/v1`), Ollama Cloud, LM Studio, Together, Fireworks, Groq, Mistral, … | optional (defaults to OpenAI) — set it for custom endpoints |
-| `google`           | `gemini-embedding-001`, `gemini-embedding-2`, `gemini-embedding-2-preview` (native provider)                                                            | ❌ no                                                       |
-| `cohere`           | `embed-english-v3.0`, `embed-multilingual-v3.0`, … (native provider)                                                                                    | ❌ no                                                       |
-| `bedrock`          | `amazon.titan-embed-text-v2:0`, `amazon.nova-embed-text-v2:0`, `cohere.embed-*` (native provider; AWS creds from env)                                   | ❌ no                                                       |
-`dimensions` is sent as a provider option where the API supports it
-(openai-compatible/`outputDimensionality`/`outputDimension`/titan `dimensions`
-and nova `embeddingDimension`) and ALWAYS enforced on the output.
-Verified end-to-end against real `gemini-embedding-2` (3072-dim), local
-ollama `Qwen3-Embedding-0.6B` (1024-dim), and the openai-compatible
-contract.
-- `dimensions` must match the model's real output — a mismatch fails typed
-  on first use
-- Multiple models coexist: each gets its own vector tables
-  (`vec_<dims>_<model>`) and its own topic tree; queries route per model
-- **Per-model runtime options** (transformers provider, all optional):
-  `device` (`auto` | `gpu` | `cpu` | `wasm` | `webgpu` | `cuda` | `dml` |
-  `coreml` | `webnn` — `cpu`/`wasm` tested; GPU/CUDA needs the onnxruntime
-  CUDA libs, fails typed without them), `dtype` (`fp32` | `fp16` | `q8` |
-  `int8` | `uint8` | `q4` | `bnb4` | `q4f16` | `q2` | `q2f16` | `q1` |
-  `q1f16` — quantizes model WEIGHTS only, never the output vectors;
-  `fp32`/`q8`/`int8`/`q4`/`fp16` verified end-to-end, others depend on the
-  HF repo shipping that ONNX variant), `pooling` (`mean` | `cls` | `none`),
-  `normalize` (boolean), `cacheDir`
-- **DB-level vector quantization** via `vectorType` (`float32` |
-  `float16` | `float8` | `float1bit`): the per-memory search vectors are
-  stored as int8/bfloat16/binary blobs and searched with libSQL's native
-  `vector_distance_cos` — `float8` verified end-to-end at **395 B vs
-  1536 B (4× smaller)** for 384-dim. The topic tree's centroid math and
-  `memories.embedding` always stay float32.
-- **Switching models**: set `defaultModel` to a configured model (multi-
-  model configs coexist — each model gets its own tree + `vec_<dims>_<model>`
-  tables). After a restart the new default handles all writes/queries;
-  the old model's tree stays intact. Existing memories are backfilled via
-  the durable `memory.embed` job (re-embed + re-route into the new tree,
-  idempotent/resume-safe) — the store-level switch (`setActiveModel` +
-  `findStale`) drives it, reachable through the `{ core: … }` path. Until
-  backfilled, old memories remain in the old tree (queries descend the
-  ACTIVE model's tree), and the runtime is bound to one active model at a
-  time (per-model reads for another model fail typed with
-  `topics.ModelMismatch`).
-- Vectors are always stored as float32 (`F32_BLOB`, 4 B/dim: 384 = 1536 B,
-  768 = 3072 B) regardless of `dtype`
-### Power users: `{ core: … }`
-Instead of a config, pass full `CoreOptions` (store/jobs URLs, models,
-`defaultModelId`, `embedderFor` factory, topics/graph/logging/llm,
-`concurrency`, `maxAttempts`, `extraDefs`/`extraHandlers` for custom durable
-jobs — register `{ name: "my.job", handler }` and enqueue it via the
-runtime; `extraHandlers` entries override built-ins by name). This is the
-injection seam used by tests (synthetic embedders).
+Calling conventions: **plain arguments** (`query(text, opts)`,
+`write(input, opts)`, `index({ paths })`); every failure is a rejected
+promise with a closed `_tag` (`"Sdk.NotFound"`, `"Sdk.InputError"`, …);
+`dispose()` is idempotent.
 ## API surface
-### `write(input, options?)`
-```ts
-await dimple.write({
-  logicalKey: "deploy-1",        // required, non-empty
-  content: "…",                   // required
-  namespace?: "default",
-  contentType?: "text/plain",
-  ttlSecs?: 604800,
-  metadata?: "{\"json\":true}",   // JSON string
-  typedMetadata?: { … },
-}, { embed: true });              // embed + topic-route (default); false for externally-managed embeddings
-```
-Returns the persisted `MemoryUnit`: `{ id, logicalKey, content,
-contentType, state: "active" | "superseded" | …, createdAt, ttlSecs,
-namespace, metadata, embedding? }` (re-fetched after embedding). Duplicate
-active `logicalKey` → `Sdk.DuplicateKey` (carries `logicalKey`/`namespace`).
-**Memory expiry**: `ttlSecs` (or the config `defaultTtlSecs`) schedules
-expiry; a sweep job retires expired memories in the background.
+| Method                                                    | What it does                                              |
+| --------------------------------------------------------- | --------------------------------------------------------- |
+| `write(input, { embed })`                                 | Store a memory (idempotent per `logicalKey`)              |
+| `get(id)` / `forget(id)`                                  | Fetch / remove by id or logical key                       |
+| `query(text, options)`                                    | Hybrid search — profiles, budgets, content-type filter    |
+| `updateEmbedding(id, vec, modelKey?)`                     | Attach an external embedding                              |
+| `index({ paths })`                                        | Codebase ingestion (idempotent re-index)                  |
+| `graph.addEdge/confirm/reject/expire/candidates/coverage` | Memory relationships: supports / supersedes / contradicts |
+| `explain({ kind, id }, options)`                          | Bounded graph neighborhood                                |
+| `dream(modelKey?)` / `repair(modelKey?)`                  | Consolidation + maintenance jobs                          |
+| `health(modelKey?)`                                       | Topic-tree invariant health                               |
+| `reload(input?)`                                          | Blue-green config reload                                  |
+Errors are a closed hierarchy (`InputError`, `NotFound`, `DuplicateKey`,
+`JobFailed`, `LLMNotConfigured`, `StoreFailure`) — see the docs site.
-`query(text, { budget: { maxUnits, maxChars, maxMs } })` — LLM-facing
-context guardrail: caps the row count, truncates result content to the
-total char budget, and marks `results.truncated` when the soft time cap
-is exceeded.
-### `get(id)` / `forget(id)`
-Fetch one memory; soft-delete (cascades edges, rebalances the tree).
-### `query(text, options?)`
-```ts
-await dimple.query("how does the pipeline deploy", {
-  limit?: 10,
-  k?: 60,                          // RRF smoothing
-  namespace?: "default",
-  profile?: {
-    recencyBoost?: 0,              // 0 = off; hyperbolically decays with age
-    modeWeights?: { vector?: 0.5, fts?: 0.5 },
-    conceptFilter?: string[],      // concept ids (query sovereignty)
-  } | "hybrid" | "semantic" | "keyword",
-  topics?: { topicKeys?: string[] },
-});
-```
-Returns `AugmentedMemory[]` — each entry carries the memory, the fused
-score, its 1-based `rank`, and the graph context (`supersededBy`,
-`conflicts`, `supportedBy`). Zero-result queries resolve to `[]` (the
-absent-vector-table case is an empty dense leg, never an error).
-### `index(options)` — codebase ingestion
-```ts
-await dimple.index({
-  paths: ["./src", "./packages"],  // dirs (recursive) and/or files
-  keyPrefix?: "code-",
-  namespace?: "default",
-  embed?: true,
-  maxFileBytes?: 64 * 1024,
-  maxChunkBytes?: 8 * 1024,
-  extensions?: [".ts", ".tsx", ".js", ".jsx", …],
-});
-// → { files, indexed, updated, reconciled, skipped, commit, unsupported, failed }
-```
-- Tree-sitter parse (TypeScript/JavaScript/TSX/JSX) → the extractor
-  produces one **file unit** per file (with its import graph in metadata)
-  and one **chunk unit** per definition (function/class/method/interface/
-  type/enum/variable, with kind/name/line-range/docstring metadata)
-- Every unit embeds + topic-routes (unless `embed: false`)
-- AST-derived graph: relative imports resolve to indexed files and
-  class→method containment become **confirmed `memory_supports` edges**
-  (`source: "rule"`)
-- **Idempotent + upserting**: keys are cwd-anchored + deterministic
-  (chunk keys drop the line number, so edits ABOVE a definition update
-  it instead of duplicating it). Re-indexing skips unchanged units
-  (`skipped`), and changed units are **superseded + rewritten under the
-  same key** (`updated`) with a confirmed identity `memory_supersedes`
-  edge — stale copies never accumulate
-- **Edges re-form after `forget`**: phase-2 resolves endpoints by logical
-  key against the store (not just the current run), so a re-created unit
-  re-connects to its pre-existing dependencies
-- **Reconcile + rename detection**: units whose source vanished (deleted
-  file/definition) are demoted (`reconciled`, `source_removed` — gone
-  from search; domain-scoped to the walked roots, so `index lib` never
-  touches other files). A vanished unit whose embedding is a near-duplicate
-  of a new unit becomes a PENDING `memory_supersedes` (reason rename) —
-  `graph confirm` completes it. Code units are TTL-exempt
-  (`ttlSecs: null` — the index owns their lifecycle); `commit` stamps the
-  repo HEAD into every unit's metadata as `codeVersion`
-- **`code_calls` edges**: every call site inside a definition resolves to
-  the callee's chunk (same file, then imported files) → confirmed edges —
-  `explain(id, { edgeKinds: ["code_calls"] })` gives one-hop impact
-  analysis
-- Per-file failures are collected, never fatal
-### `updateEmbedding(id, embedding, modelKey?)` / `embed(text)`
-Attach an externally-computed embedding (completes the `{ embed: false }`
-workflow — the memory becomes semantically retrievable + topic-routed), or
-embed text with the SDK's own pipeline.
-### `dream(modelKey)` / `repair(modelKey)`
-Run the consolidation job (concept clustering + LLM edge classification;
-requires the `llm` config section) or the graph repair pass. Each call
-gets a fresh run id (always executes). Failures are typed:
-`Sdk.LLMNotConfigured`, `Sdk.JobFailed`.
-### `explain({ kind: "memory" | "topic" | "concept", id }, options?)`
-Graph neighborhood traversal — `{ nodes: [{ kind, id, depth, edgeKind }] }`.
-`options.edgeKinds` scopes the answer (memory-only navigation: pass the
-`memory_*` kinds; `code_calls` shows the call graph).
-### `health(modelKey)` / `graph.*`
-- `health` — topic-tree invariant sweep (`warnCount` + violation strings)
-- `graph.confirm(id)` / `graph.reject(id)` / `graph.expire(id)` — the
-  pending-edge lifecycle: promote to confirmed (a `memory_supersedes`
-  confirm demotes the old endpoint in the same transaction), reject, or
-  soft-delete. `Sdk.EdgeNotFound` for stale ids
-- `graph.addEdge({ fromMemoryId, toMemoryId, edgeKind: "memory_supports" |
-"memory_supersedes" | "memory_contradicts", source: "rule" | "llm" |
-"agent", weight?, confidence?, confirmed? })` — explicit edges; typed
-  validation (no self-loops, known kinds, weight > 0, confidence ∈ [0,1])
-- `graph.candidates()` — pending structural candidates + isolated count
-- `graph.coverage(modelKey)` — coverage %, pending, concept count
-### `reload(input?)` — apply a config change
-Pauses and re-boots with the re-resolved config:
-1. the config source is re-resolved **typed** — a `dimple.jsonc` path is
-   re-read from disk, an inline object re-validated (a bad config rejects
-   the reload with **zero impact** — nothing pauses);
-2. an **identical** config resolves as `{ reloaded: false }` — no pause,
-   no restart (safe to call from a file watcher);
-3. new calls wait, in-flight calls settle on the old stack;
-4. a fresh stack boots (new worker fleet, embedders, logger — jobs are
-   durable, so an interrupted handler's lease expires and re-claims after
-   boot);
-5. the swap: the old stack is disposed, the new one serves.
-Resolves with `{ config, changed, reloaded }` — `changed` flags which
-sections actually moved (`storeUrl` / `jobsUrl` / `rest`).
-Works for `{ config: … }`-built dimples (reload with no argument re-reads
-the original source). `{ core: … }`-built dimples have no config source
-and reject with `Sdk.Reload`. Boot failure leaves the previous system
-running (blue-green); a drain that cannot settle in-flight calls (30s)
-rejects with `Sdk.Reload` and nothing is torn down.
-**What happens to data when the database URL changes:** the old database
-file is preserved on disk but no longer served — nothing is migrated or
-merged, the new db is bootstrapped empty (a warning is logged for the
-switch; roll back with `reload(oldConfig)`). Pending jobs in a switched
-`jobs.dbUrl` stay durable in the old db, re-claimable by switching back
-or draining it externally.
-```ts
-await dimple.reload();                        // re-read dimple.jsonc
-await dimple.reload({ storeUrl: "file:…", … }); // or a fresh config
-```
-### `dispose()`
-Stops the worker, closes sinks + the store. Idempotent; post-dispose calls
-reject with a closed `Sdk.StoreFailure`.
-## Errors — the closed hierarchy
-Every promise rejects with typed errors from one family (never raw Effect
-internals). Export surface: `ConfigError`, `NotFoundError`,
-`DuplicateKeyError` (with `logicalKey`/`namespace`), `DuplicateEdgeError`,
-`LLMNotConfiguredError`, `JobFailedError`, `StoreFailure`, `InputError`,
-plus the `remapError` helper (maps unknown failures onto the hierarchy).
-Also importable from the `@ponraaj/dimple-sdk/errors` subpath.
-**Type exports** (all importable from the package root): `Dimple` (the
-facade interface), `MakeDimpleOptions`, `ConfigInput`, `QueryOptions`,
-`IndexOptions`, `IndexResult`, plus the error classes and the `DimpleError`
-union type. Errors also carry stable `_tag` values (`Sdk.DuplicateKey`,
-`Sdk.NotFound`, `Sdk.InputError`, `Sdk.JobFailed`, `Sdk.LLMNotConfigured`,
-`Sdk.StoreFailure`, …) — the recommended discriminator.
-## Background jobs (what runs while you work)
-| Job                                               | Trigger             | Work                                                                                                                       |
-| ------------------------------------------------- | ------------------- | -------------------------------------------------------------------------------------------------------------------------- |
-| `topic.split` / `topic.merge` / `topic.rebalance` | automatic per write | keep every leaf within its size budget (deterministic splits, donor borrow, nearest-sibling merge, uneven-split rebalance) |
-| `memory.embed`                                    | model backfill      | re-embed a memory with a target model + route into its tree                                                                |
-| `memory.dream`                                    | `dream()`           | concept clustering + LLM edge classification                                                                               |
-| `graph.repair`                                    | `repair()`          | graph invariant repair                                                                                                     |
-| `audit.write`                                     | query path          | async audit-log writes (never blocks queries) — one FUSED row per hybrid query with the per-leg breakdown                  |
-Jobs are durable (leased, retried up to `maxAttempts`, resume-safe across
-processes). Every `makeDimple` boots the worker; `dispose` stops it.
-Every `query` also writes a retrieval audit row (query, namespace, mode,
-per-leg counts/scores, sources) into `retrieval_log` — machine-readable
-query history, persisted asynchronously (never blocks the query).
-## Storage & logs
+## Configuration
-| What                           | Where                                                                                        |
-| ------------------------------ | -------------------------------------------------------------------------------------------- |
-| Memories, tree, graph, audit   | `storeUrl` (`file:…`)                                                                        |
-| Durable queue + workflow state | `jobs.dbUrl` (separate DB)                                                                   |
-| Logs                           | stdout + in-memory ring (4096) by default; durable JSONL **only** when `logging.file` is set |
-| Grammar WASMs                  | `~/.cache/dimple/tree-sitter/` (downloaded once, offline after)                              |
-| ONNX model                     | bun/npm cache (~90 MB, first use)                                                            |
+`makeDimple({ config: "./dimple.jsonc" })` — every section is optional
+except `embedding`. `dimple init` (CLI) writes the full annotated schema;
+`${ENV_VAR}` placeholders keep tokens out of files. The complete reference
+(store / jobs / embeddings / topics / graph / llm / logging / Turso) is on
+the docs site: [Configuration reference](https://dimple.ponraaj.me/reference/config).
-## Changelog
+## MCP server
-Every release creates a GitHub Release with the per-version changes (the
-tag-based publish workflow): repo → **Releases** page.
+`dimple mcp` (from the CLI package) exposes dimple to any MCP client over
+the 2026-07-28 protocol — API discovery plus sandboxed code execution with
+the whole API as a `dimple` namespace. Guide:
+[MCP server & code mode](https://dimple.ponraaj.me/guides/mcp).
 ## Agent integration patterns
-These are the shapes that work well when the consumer is another agent
-(a harness) rather than a human with a REPL.
-### Session-start injection
-The guaranteed-usage path: a warmup query returns a compact memory block
-the harness injects as context, and the pending-graph review list rides
-along:
-```ts
-const [context, pending] = await Promise.all([
-  dimple.query("ongoing work and decisions", { limit: 8, budget: { maxChars: 4000 } }),
-  dimple.graph.candidates(),
-]);
-// context  → inject the top-N content strings into the system prompt
-// pending  → offer each candidate as a confirm/reject decision
-```
-`graph.candidates()` pending `memory_supersedes` edges from the rename
-scanner (rule 0.9x near-duplicates) are the highest-value batch to review.
-### Supersedes recipe
+Session-start injection, the supersedes recipe, and namespaces as the
+code/message partition are covered on the docs site (Architecture section)
+and in `docs/MCP.md` in the repository.
-`addEdge({ … edgeKind: "memory_supersedes", confirmed: true })` performs
-the full state flip in one transaction (the old memory demotes
-immediately) — it is safe to use directly. For human-gated flows the
-pending → confirm pair keeps the old memory ranking until a reviewer
-agrees:
-```ts
-const edge = await dimple.graph.addEdge({
-  fromMemoryId: oldId,
-  toMemoryId: newId,
-  edgeKind: "memory_supersedes",
-  source: "agent",
-}); // pending — nothing changed yet
-await dimple.graph.confirm(edge.id); // ← the flip happens here
-```
-### Namespaces as the code/message partition
-Until a query-time content-type filter exists, namespaces give clean
-isolation: `index({ paths, namespace: "code" })` and
-`write({ …, namespace: "messages" })` partition the store; query with
-`namespace: "code"` for code and `namespace: "messages"` for
-conversations. (`query({ contentType })` now does the same within one
-namespace — the two levers compose.)
-### Remember
-- `dimple.reload(newConfig)` swaps the whole stack blue-green (validate →
-  pause → boot → drain → swap); a failed boot leaves the old stack
-  serving.
-- JSON output is compact by default: CLI `--json` omits embeddings (pass
-  `--embedding` to include them); the SDK always keeps them on
-  `memory.embedding`.
-## Notes
+## Changelog
-- Determinism: same data + same config ⇒ same tree, same clusters, same
-  audit hashes — reproducible across machines and runs
-- `migrationFolder` is accepted in `makeDimple` options for advanced users;
-  the default path uses migrations embedded at build time (nothing to ship)
+[Release history](https://dimple.ponraaj.me/changelog/sdk) · repository
+[`CHANGELOG.md`](./CHANGELOG.md)

package/dist/index.js CHANGED Viewed

@@ -508,7 +508,10 @@ var consoleSink = (pretty) => {
   };
 };
 var fileSink = (path) => Effect5.tryPromise(async () => {
-  const { openSync, closeSync, fsyncSync, writeSync } = await import("node:fs");
+  const fs = await import("node:fs");
+  const { dirname } = await import("node:path");
+  fs.mkdirSync(dirname(path), { recursive: true });
+  const { openSync, closeSync, fsyncSync, writeSync } = fs;
   const fd = openSync(path, "a");
   let disposed = false;
   const flush = () => {
@@ -2971,17 +2974,13 @@ var workflowTagFor = (workflows, store, executionId) => Effect17.gen(function* (
     return;
   return workflows.get(record.workflowTag)?.workflow;
 });
-var ensureDbDir = (url) => {
-  if (url.startsWith("file:")) {
-    import("node:fs").then((fs) => import("node:path").then(({ dirname }) => {
-      fs.mkdirSync(dirname(url.slice("file:".length)), { recursive: true });
-    }));
-  }
-};
-var libsqlStack = (options) => {
-  ensureDbDir(options.url);
-  return Layer6.mergeAll(PersistenceNamespace2.layerSql, PersistedQueue2.layer.pipe(Layer6.provide(storeSqliteLayer))).pipe(Layer6.provide(exports_LibsqlClient.layer(options)));
-};
+var ensureDbDir = (url) => url.startsWith("file:") ? import("node:fs").then((fs) => import("node:path").then(({ dirname }) => {
+  fs.mkdirSync(dirname(url.slice("file:".length)), { recursive: true });
+})) : Promise.resolve();
+var libsqlStack = (options) => Layer6.unwrap(Effect17.tryPromise({
+  try: () => ensureDbDir(options.url),
+  catch: (cause) => new Error(String(cause))
+}).pipe(Effect17.catch(() => Effect17.void), Effect17.map(() => Layer6.mergeAll(PersistenceNamespace2.layerSql, PersistedQueue2.layer.pipe(Layer6.provide(storeSqliteLayer))).pipe(Layer6.provide(exports_LibsqlClient.layer(options))))));
 var layerDurableLibsql = (options) => layerDurable.pipe(Layer6.provide(libsqlStack(options)));
 // ../workflow/src/runtime.ts
@@ -4734,6 +4733,36 @@ var makeStore2 = (backend, modelConfig, topicsOptions, graphConfig, auditOptions
     if (!result)
       return yield* new NotFound({ id: key });
   });
+  const decrementTreeCf = (model, membership, x) => Effect26.tryPromise({
+    try: () => db.transaction(async (tx) => {
+      const leafRows = await tx.select({
+        topicKey: topics.topicKey,
+        memberCount: topics.memberCount,
+        sumSquares: topics.sumSquares
+      }).from(topics).where(and8(eq8(topics.topicKey, membership.topicKey), eq8(topics.modelKey, membership.modelKey)));
+      const leaf = leafRows[0];
+      if (!leaf)
+        return;
+      const next = cfRemove({
+        count: Number(leaf.memberCount),
+        ls: new Float32Array(0),
+        ss: Number(leaf.sumSquares)
+      }, x);
+      await tx.update(topics).set({
+        memberCount: Math.max(0, next.count),
+        sumSquares: Math.max(0, next.ss),
+        updatedAt: Date.now()
+      }).where(and8(eq8(topics.topicKey, membership.topicKey), eq8(topics.modelKey, membership.modelKey)));
+      const relRows = await tx.select({ parentKey: topicRelations.parentKey }).from(topicRelations).where(and8(eq8(topicRelations.childKey, membership.topicKey), eq8(topicRelations.modelKey, membership.modelKey), eq8(topicRelations.relationKind, "parent_child")));
+      let ancestorKey = relRows[0]?.parentKey;
+      while (ancestorKey) {
+        await tx.update(topics).set({ memberCount: sql7`max(0, ${topics.memberCount} - 1)`, updatedAt: Date.now() }).where(and8(eq8(topics.topicKey, ancestorKey), eq8(topics.modelKey, membership.modelKey)));
+        const up = await tx.select({ parentKey: topicRelations.parentKey }).from(topicRelations).where(and8(eq8(topicRelations.childKey, ancestorKey), eq8(topicRelations.modelKey, membership.modelKey), eq8(topicRelations.relationKind, "parent_child")));
+        ancestorKey = up[0]?.parentKey;
+      }
+    }),
+    catch: (cause) => new StoreError({ message: `Store: tree CF decrement: ${cause}` })
+  });
   const forget = Effect26.fn("Store.forget")(function* (id) {
     let membership;
     if (topicsEnabled) {
@@ -4747,37 +4776,7 @@ var makeStore2 = (backend, modelConfig, topicsOptions, graphConfig, auditOptions
         const vectorRows = yield* backend.fetchMemoryVectors([id], modelConfig);
         if (vectorRows.length > 0) {
           const x = new Float32Array(vectorRows[0].vector);
-          const now = Date.now();
-          yield* Effect26.tryPromise({
-            try: () => db.transaction(async (tx) => {
-              const leafRows = await tx.select({
-                topicKey: topics.topicKey,
-                memberCount: topics.memberCount,
-                sumSquares: topics.sumSquares
-              }).from(topics).where(and8(eq8(topics.topicKey, mem.topicKey), eq8(topics.modelKey, mem.modelKey)));
-              const leaf = leafRows[0];
-              if (!leaf)
-                return;
-              const next = cfRemove({
-                count: Number(leaf.memberCount),
-                ls: new Float32Array(0),
-                ss: Number(leaf.sumSquares)
-              }, x);
-              await tx.update(topics).set({
-                memberCount: Math.max(0, next.count),
-                sumSquares: Math.max(0, next.ss),
-                updatedAt: now
-              }).where(and8(eq8(topics.topicKey, mem.topicKey), eq8(topics.modelKey, mem.modelKey)));
-              const relRows = await tx.select({ parentKey: topicRelations.parentKey }).from(topicRelations).where(and8(eq8(topicRelations.childKey, mem.topicKey), eq8(topicRelations.modelKey, mem.modelKey), eq8(topicRelations.relationKind, "parent_child")));
-              let ancestorKey = relRows[0]?.parentKey;
-              while (ancestorKey) {
-                await tx.update(topics).set({ memberCount: sql7`max(0, ${topics.memberCount} - 1)`, updatedAt: now }).where(and8(eq8(topics.topicKey, ancestorKey), eq8(topics.modelKey, mem.modelKey)));
-                const up = await tx.select({ parentKey: topicRelations.parentKey }).from(topicRelations).where(and8(eq8(topicRelations.childKey, ancestorKey), eq8(topicRelations.modelKey, mem.modelKey), eq8(topicRelations.relationKind, "parent_child")));
-                ancestorKey = up[0]?.parentKey;
-              }
-            }),
-            catch: (cause) => new StoreError({ message: `Store.forget: tree CF: ${cause}` })
-          });
+          yield* decrementTreeCf(modelConfig, mem, x);
           yield* updateTopicVectorForget(backend, modelConfig, mem.topicKey, x);
         }
       }
@@ -5004,6 +5003,28 @@ var makeStore2 = (backend, modelConfig, topicsOptions, graphConfig, auditOptions
       return yield* new NotFound({ id });
     if (modelConfig && modelConfig.id === modelName) {
       yield* backend.ensureVectorTable(modelConfig);
+      if (topicsEnabled) {
+        const memberRows = yield* Effect26.tryPromise({
+          try: () => db.select({
+            topicKey: topicMemberships.topicKey,
+            modelKey: topicMemberships.modelKey
+          }).from(topicMemberships).where(eq8(topicMemberships.memoryId, id)),
+          catch: (cause) => new StoreError({ message: `Store.updateEmbedding: load membership: ${cause}` })
+        });
+        const mem = memberRows[0];
+        if (mem && mem.modelKey === modelName) {
+          const oldVec = yield* backend.fetchMemoryVectors([id], modelConfig);
+          if (oldVec.length > 0) {
+            const x = new Float32Array(oldVec[0].vector);
+            yield* decrementTreeCf(modelConfig, mem, x);
+            yield* updateTopicVectorForget(backend, modelConfig, mem.topicKey, x);
+          }
+          yield* Effect26.tryPromise({
+            try: () => db.delete(topicMemberships).where(and8(eq8(topicMemberships.memoryId, id), eq8(topicMemberships.modelKey, modelName))),
+            catch: (cause) => new StoreError({ message: `Store.updateEmbedding: delete membership: ${cause}` })
+          });
+        }
+      }
       yield* backend.insertVector(id, embedding, modelConfig);
       if (topicsEnabled) {
         const routed = yield* routeInsert(id, embedding, modelConfig, topicsConfig).pipe(Effect26.provideService(Service2, backend), Effect26.mapError((cause) => new StoreError({ message: `Store.updateEmbedding topic route: ${cause}` })));
@@ -6856,8 +6877,8 @@ var layer11 = (options) => Layer13.effect(Service10, Effect35.gen(function* () {
     const vWeight = profile.modeWeights?.vector ?? 1;
     const fWeight = profile.modeWeights?.fts ?? 1;
     const mode = vWeight === 0 && fWeight > 0 ? "keyword" : fWeight === 0 && vWeight > 0 ? "semantic" : "hybrid";
-    const denseRanks = vectorResults.length > 0 ? vectorResults.map((s) => [s.memory.id, s.score * vWeight]) : null;
-    const fts5Ranks = fts5Results.length > 0 ? fts5Results.map((s) => [s.memory.id, s.score * fWeight]) : null;
+    const denseRanks = vWeight > 0 && vectorResults.length > 0 ? vectorResults.map((s) => [s.memory.id, s.score * vWeight]) : null;
+    const fts5Ranks = fWeight > 0 && fts5Results.length > 0 ? fts5Results.map((s) => [s.memory.id, s.score * fWeight]) : null;
     if (!denseRanks && !fts5Ranks) {
       yield* store.logAudit({
         query: query2,

package/package.json CHANGED Viewed

@@ -1,6 +1,6 @@
 {
   "name": "@ponraaj/dimple-sdk",
-  "version": "0.0.0-alpha.5",
+  "version": "0.0.0-alpha.7",
   "private": false,
   "files": [
     "dist"