@keb-org/consol 0.1.1 → 0.3.0

package/README.md

@@ -1,28 +1,64 @@
 # @keb-org/consol
 
-TypeScript SDK for the consol memory engine, for three uses:
+The consol memory engine as one package: the MCP server (`npx -y @keb-org/consol`) and a TypeScript SDK.
+The SDK has three uses:
 
-1. **Local**: a vault on disk (or S3) in your own process. The SDK starts `consol stdio` as a child.
+1. **Local**: a vault on disk (or S3), with the engine running in your own process as a native addon.
 2. **Hosted client**: the same `Vault` methods against a backend over HTTP.
 3. **Backend**: `Consol.handle` behind your own HTTP route, so clients of any language can use it.
 
 Node >= 22 or Bun, no dependencies; `import` and `require` share one implementation. The behaviour
 is the same in every mode.
 
-## The server binary and database
+## MCP server
 
-The package does not build the server. The SDK uses the first of these that exists:
+`npx -y @keb-org/consol --vault PATH --agent NAME` serves MCP on stdio for one vault that already
+exists (`consol create --vault PATH` makes it; the MCP never creates a vault). The engine runs in
+its process through the same addon as the SDK, so it needs the database too: every `CONSOL_*`
+variable of its process is an engine setting (`CONSOL_DB_URL` is required, `CONSOL_EMBED_*` turn on
+embeddings). Engine logs go to stderr at the level of `RUST_LOG` (default `warn`).
 
-1. the `binaryPath` option,
-2. the `CONSOL_BIN` environment variable,
-3. the platform package `@keb-org/consol-<platform>-<arch>`, resolved from the SDK's own location,
-4. `consol` (`consol.exe` on Windows) on `PATH`.
+| Flag | Variable | |
+|---|---|---|
+| `--vault` | `CONSOL_VAULT` | a directory or `s3://bucket/prefix/name`; with `--remote`, the vault's name on the backend |
+| `--remote` | `CONSOL_REMOTE_URL` | a backend URL (use 3) to forward to, instead of opening a vault; its headers come from `CONSOL_REMOTE_HEADERS` (one `Name: value` per line) |
+| `--agent` | `CONSOL_AGENT` | the default agent of every call; a call's `agent` argument overrides it |
+
+In an MCP client, or in Claude Code (the options go before the name):
+
+```bash
+claude mcp add --env CONSOL_DB_URL=postgres://postgres:pw@127.0.0.1:5432/postgres \
+  consol -- npx -y @keb-org/consol --vault /abs/path/vault --agent assistant
+```
+
+```json
+{ "mcpServers": { "consol": {
+  "command": "npx",
+  "args": ["-y", "@keb-org/consol", "--vault", "/abs/path/vault", "--agent", "assistant"],
+  "env": { "CONSOL_DB_URL": "postgres://postgres:pw@127.0.0.1:5432/postgres" }
+} } }
+```
+
+A hosted backend needs no database here:
+`npx -y @keb-org/consol --remote https://api.acme.dev/v1/consol --vault demo --agent assistant`, with the
+backend's credentials in `CONSOL_REMOTE_HEADERS`. The
+tool list, the instructions and each result are the engine's, the same ones `consol serve` mounts
+over streamable HTTP at `<path>/mcp`, which is unchanged.
+
+## The engine addon and database
+
+The engine is the native addon `consol.node`, loaded the first time you start a vault or a
+`Consol` (importing the SDK loads nothing). The SDK uses the first of these:
+
+1. the `addonPath` option,
+2. the `CONSOL_ADDON` environment variable,
+3. the platform package `@keb-org/consol-<platform>-<arch>`, resolved from the SDK's own location.
 
-A `binaryPath` or `CONSOL_BIN` that points nowhere fails at once with `binary_not_found` instead
-of falling through to the next place. Build the binary from the repository with
-`cargo build --release -p consol-cli`.
+An `addonPath` or `CONSOL_ADDON` that points nowhere, or at a file that does not load, fails at
+once with `addon_not_found` instead of falling through to the next place. The platform package
+also ships the `consol` CLI as `bin/consol`.
 
-The engine needs a database, PostgreSQL with pgvector or CockroachDB: pass `db` or set
+The engine still needs a database, PostgreSQL with pgvector or CockroachDB: pass `db` or set
 `CONSOL_DB_URL` in your process (the one `CONSOL_*` variable the SDK takes from its host).
 
 ## 1. Local
@@ -44,8 +80,8 @@ try {
 ```
 
 `createVault` fails with `vault_exists` when the directory already holds a vault, `openVault`
-(same options) with `vault_not_found` when it does not. Both start the server and check that its
-version matches the SDK's major.minor before they return.
+(same options) with `vault_not_found` when it does not. Both open the engine and check that the
+addon's version matches the SDK's major.minor before they return.
 
 ## 2. Hosted client
 
@@ -109,19 +145,25 @@ Each method takes one object with the tool's wire argument names, plus optional
 
 | Method | Arguments | Returns |
 |---|---|---|
-| `recall` | `{ query, include_history?, limit?, agent? }` | `RecallResult`: `text` is the progressive disclosure to give a model (cards by tier, intent block, hints); `candidates` are the same cards as data. Cards are leads: read a note before relying on it |
+| `recall` | `{ query, agent? }` | `RecallResult`: `text` is the progressive disclosure to give a model (cards by tier, the blocks the query's symbols ask for, hints); `candidates` are the same cards as data. Cards are leads: read a note before relying on it |
 | `read` | `{ id, continuation?, agent? }` | `ReadResult`; `text` is the exact file |
-| `remember` | `{ id, content?, expected_version?, summary?, entities?, facets?, scope?, team?, valid_from?, valid_to?, reflection?, agent? }` | `RememberResult` |
-| `remove` | `{ id, expected_version, reflection?, agent? }` | `RemoveResult`: the note is archived, its body stays readable |
-| `reflect` | one mode: `{}` discovery, `{ ids }`, `{ work_ids }` or `{ decisions }`; `limit?`, `agent?` | `ReflectResult`; delivered sources carry their `text` |
+| `remember` | `{ id, content?, meta?, expected_version?, token?, agent? }`; `meta` is `{ summary?, entities?, facets?, scope?, team?, valid_from?, valid_to? }` | `RememberResult` |
+| `remove` | `{ id, expected_version, token?, agent? }` | `RemoveResult`: the note is archived, its body stays readable |
+| `reflect` | `{ focus?, decisions?, agent? }`; call it with no arguments and follow `next` | `ReflectResult`: `next`, the delivered `items` (sources carry their `text`), every unresolved `writes` entry, the `blocked` tokens |
 | `callTool` | `(name, args)` with raw wire arguments | the raw `Reply`; tool errors come back as `reply.error` |
 | `close` | | final and idempotent |
 
 - `remember` without `expected_version` is create-only. To update, pass `source.content_version`
-  from your last `read`; a stale one fails with `version_conflict`. Omitted or `null` fields keep
-  their stored value, `""` or `[]` clears them.
-- `remember` and `remove` also take `{ resume: true, reflection, agent? }` to resume a pending
-  reflection-bound write. `reflection` is `{ work_id, basis_hash, action_key }`.
+  from your last `read`; a stale one fails with `version_conflict`. Omitted or `null` `content` keeps
+  the body. In `meta` a key left out keeps its stored value, `null`, `""` or `[]` clears it (not
+  `scope` or `team`); a key the server does not know is ignored and listed in the receipt's
+  `ignored`.
+- `token` on `remember` and `remove` is an item's work token from `reflect`: it binds that one write to the
+  work (the server derives the write's key). `{ token, agent? }` alone, with a write token from
+  `reflect().writes` or a write receipt, resumes that write.
+- `reflect` holds nothing between calls: a cold `reflect()` lists every unresolved write, the open work and
+  the blocked work. Decide an item with `{ token, verdict, reason, refs?, outputs? }`, `verdict` one of the
+  item's `verdicts`; the server sizes each batch (about 64 KiB of source Markdown, at most 16 items).
 - `byte_range` values are UTF-8 byte offsets: `sliceUtf8(text, start, end)` slices a JS string by them.
 - `vault.serverVersion` is the server version of the last reply. An agent exists from its first
   `remember`; reading as an agent that wrote nothing finds nothing.
@@ -147,7 +189,7 @@ Markdown inline.
 
 The tool list is not on the wire. `consol schema` prints its JSON Schema: `oneOf` holds one
 envelope per tool, with the `args` schema and the description the model reads. An MCP client can
-use `consol mcp` instead.
+use the MCP server of this package instead (above).
 
 ## Loading many notes
 
@@ -181,16 +223,16 @@ try {
 ```
 
 - `ConsolError.is(error)` also works across duplicate copies of the package, where `instanceof` fails.
-- `field` names the input at fault, relative to the call's arguments (`limit`,
-  `decisions[2].temporal`), `tool` the tool, and `result` keeps the raw `Reply` of a failed tool
+- `field` names the input at fault, relative to the call's arguments (`query`,
+  `decisions[2].refs[0]`), `tool` the tool, and `result` keeps the raw `Reply` of a failed tool
   call, including the durable state of a pending write (`error.result.result`).
 - Tool codes (contract section 4; accept unknown future ones): `invalid_argument`, `forbidden`,
   `not_found`, `already_exists`, `version_conflict`, `team_not_attached`, `secret_rejected`,
   `content_too_large`, `unsupported`, `work_not_found`, `basis_mismatch`, `already_decided`,
   `unresolved_writes`, `postcondition_failed`, `index_unavailable`, `storage_unavailable`, `vault_busy`, `internal`.
   Vault codes: `vault_exists`, `vault_not_found`.
-- SDK codes: `request_failed` (`status?`), `binary_not_found`, `incompatible_server` (server
-  major.minor differs from the SDK's), `server_exited`, `timeout`, `aborted`, `closed`.
+- SDK codes: `request_failed` (`status?`), `addon_not_found`, `incompatible_server` (addon or
+  server major.minor differs from the SDK's), `timeout`, `aborted`, `closed`.
 - `retryable: true` (`vault_busy`, `index_unavailable`, `storage_unavailable`, `request_failed` on a connect error or
   502/503/504) means the identical request may succeed later. The SDK never retries by itself.
 
@@ -198,22 +240,17 @@ try {
 
 - `close()` is final and idempotent: in-flight and later calls fail with `closed`, and concurrent
   calls share one completion. `await using vault = await openVault(...)` closes it for you.
-- An idle local vault does not keep Node or Bun alive: a script that forgets `close()` still
-  exits, and its server exits with it.
 - Every call has a deadline: 300 s by default, `timeoutMs` per vault or per call, `0` disables it.
   `signal` aborts a call. Both only stop waiting: a write may still complete, so read the note
   before you retry.
-- When a local server crashes, in-flight calls fail with `server_exited`. The next call starts the
-  server once more and opens the vault again (`vault_not_found` if it was deleted meanwhile).
-- Servers on one vault take turns on its lock, a lease in the database; a wait over 5 s fails with
-  `vault_busy`.
+- Engines on one vault, in this process or another, take turns on its lock, a lease in the
+  database; a wait over 5 s fails with `vault_busy`.
 
 ## Configuration and logging
 
-Options of `createVault`, `openVault` (local) and `Consol.start` become the server's environment,
-never its argv. Inherited `CONSOL_*` and `RUST_LOG` variables are dropped first (except
-`CONSOL_DB_URL` as the default for `db`), so a stray host variable cannot change the agent or the
-vault.
+Options of `createVault`, `openVault` (local) and `Consol.start` become the engine's settings (the
+`CONSOL_*` names). Your process's own `CONSOL_*` variables are not read (except `CONSOL_DB_URL` as
+the default for `db`), so a stray host variable cannot change the agent or the vault.
 
 | Option | Meaning |
 |---|---|
@@ -221,16 +258,16 @@ vault.
 | `vaultRoot` / `vault` | `Consol.start`: a directory or `s3://` prefix of many vaults, or one vault |
 | `agent` | default agent of every call; a call's `agent` argument overrides it |
 | `db` | database URL (`CONSOL_DB_URL`): PostgreSQL with pgvector, or CockroachDB |
-| `embedding` | `{ url, model, apiKey?, dims?, queryPrefix?, batchSize?, timeoutMs? }` (`CONSOL_EMBED_*`); without it recall is lexical-only |
+| `embedding` | `{ url, model, apiKey?, dims?, timeoutMs? }` (`CONSOL_EMBED_*`); without it recall is lexical-only |
 | `jev` | `{ url?, apiKey?, model?, auto? }` (`CONSOL_JEV_*`); experimental advice |
-| `env` | any other server variable, applied last |
-| `binaryPath` | server binary, see above |
+| `env` | any other engine setting (a `CONSOL_*` name), applied last |
+| `addonPath` | the `consol.node` to load, see above |
 | `timeoutMs` | default per-call deadline in ms |
-| `onLog` | `(line) => void`, receives each server stderr line |
+| `onLog` | `(line) => void`, receives each engine log line |
+| `logLevel` | which lines reach `onLog`, such as `info` or `consol_engine=debug` (default `warn`) |
 
-The server's stderr is captured and never written to your process's stderr. Pass `onLog` to see
-it, and `env: { RUST_LOG: 'debug' }` for more detail. A `server_exited` error carries the server's
-last lines in its message and in `stderrTail`.
+The addon has one log sink per process: every engine's `onLog` receives the same lines at the
+latest `logLevel` set, and the sink is released when the last of them closes.
 
 ## Types and stability
 
@@ -246,21 +283,22 @@ decisions, and every `jev`, `jev_triage` and `jev_advisory` field (typed as plai
 bun install
 bun run build       # dist/index.js (CJS), dist/index.mjs (ESM wrapper), dist/*.d.ts
 bun run typecheck   # tsc over src
-# the rest needs the real binary and a database:
-export CONSOL_BIN=/abs/consol CONSOL_DB_URL=postgres://...
-bun run test            # pure checks, hosted-client stub, real-binary suite on Node, strict tsc incl. conformance
-bun run test:consumers  # packed tarballs in Bun, Node ESM, Node CJS and platform-package projects
-node test/capture.mjs   # recapture test/fixtures from the real server (without CONSOL_BIN it only regenerates the types check)
+bun run test:offline    # pure checks, hosted-client stub, fake-addon transport checks, strict tsc incl. conformance
+# the rest needs the real addon and a database:
+export CONSOL_ADDON=/abs/consol.node CONSOL_DB_URL=postgres://...
+bun run test            # the offline checks plus the real-addon suite on Node
+CONSOL_BIN=/abs/consol bun run test:consumers  # packed tarballs in Bun, Node ESM, Node CJS and platform-package projects
+node test/capture.mjs   # recapture test/fixtures from the real engine (without CONSOL_ADDON it only regenerates the types check)
 ```
 
 `bun run test`, not `bun test` (that is Bun's own runner). Consumer logs go to
 `consumers/<project>.log`.
 
-## Publishing platform binaries
+## Publishing platform packages
 
 1. `bun run build`
-2. `bun scripts/make-platform-package.mjs <binary> <platform> <arch> <outDir>` per target
-   (win32-x64, darwin-arm64, darwin-x64, linux-x64, linux-arm64); the binary is stored as
-   `bin/consol[.exe]` whatever its file name
+2. `bun scripts/make-platform-package.mjs <binary> <addon> <platform> <arch> <outDir>` per target
+   (win32-x64, darwin-arm64, darwin-x64, linux-x64, linux-arm64); the CLI binary is stored as
+   `bin/consol[.exe]` and the addon as `consol.node`, whatever their file names
 3. Publish each platform package, then `bun scripts/inject-optional-deps.mjs <publish-copy-of-package.json>`
 4. `bun publish` the main package (`publishConfig.access` is `public`)

package/dist/consol.d.ts

@@ -1,4 +1,4 @@
-import { type EngineOptions } from "./stdio";
+import { type EngineOptions } from "./native";
 import type { CallOptions, Context } from "./transport";
 import type { Reply } from "./types";
 /** Exactly one of `vaultRoot` / `vault`, plus the engine settings (`db`, `embedding`, `jev`, ...). */
@@ -10,7 +10,7 @@ export interface ConsolOptions extends EngineOptions {
 }
 export declare function httpStatus(reply: Reply): number;
 /**
- * The engine in your own backend: one `consol stdio` child behind `handle`. A backend authorizes
+ * The engine in your own backend: the in-process engine behind `handle`. A backend authorizes
  * the request itself, then passes the result as `ctx`.
  *
  * SECURITY: `ctx` (the vaults the caller may use, the agent they are) must come from YOUR auth,

package/dist/errors.d.ts

@@ -1,13 +1,12 @@
 import type { Reply } from "./types";
 /** Branchable error codes: wire tool codes (contract section 4) plus the SDK codes of section 6. */
-export type ConsolErrorCode = "invalid_argument" | "not_found" | "already_exists" | "version_conflict" | "team_not_attached" | "secret_rejected" | "content_too_large" | "unsupported" | "work_not_found" | "basis_mismatch" | "already_decided" | "unresolved_writes" | "postcondition_failed" | "index_unavailable" | "storage_unavailable" | "vault_busy" | "forbidden" | "vault_exists" | "vault_not_found" | "internal" | "request_failed" | "binary_not_found" | "incompatible_server" | "server_exited" | "timeout" | "aborted" | "closed" | (string & {});
+export type ConsolErrorCode = "invalid_argument" | "not_found" | "already_exists" | "version_conflict" | "team_not_attached" | "secret_rejected" | "content_too_large" | "unsupported" | "work_not_found" | "basis_mismatch" | "already_decided" | "unresolved_writes" | "postcondition_failed" | "index_unavailable" | "storage_unavailable" | "vault_busy" | "forbidden" | "vault_exists" | "vault_not_found" | "internal" | "request_failed" | "addon_not_found" | "incompatible_server" | "timeout" | "aborted" | "closed" | (string & {});
 export interface ConsolErrorDetails {
     retryable?: boolean;
     field?: string;
     tool?: string;
     /** HTTP status of a `request_failed` / `forbidden` that came from the transport, not the engine. */
     status?: number;
-    stderrTail?: string;
     result?: Reply;
     cause?: unknown;
 }
@@ -18,7 +17,6 @@ export declare class ConsolError extends Error {
     readonly field?: string;
     readonly tool?: string;
     readonly status?: number;
-    readonly stderrTail?: string;
     /** Raw reply of a failed typed call: durable write state (`result.result`) survives the throw. */
     readonly result?: Reply;
     constructor(code: ConsolErrorCode, message: string, details?: ConsolErrorDetails);

package/dist/index.d.ts

@@ -1,6 +1,6 @@
 export { ConsolError, type ConsolErrorCode } from "./errors";
 export { Consol, httpStatus, type ConsolOptions } from "./consol";
 export type { CallOptions, Context } from "./transport";
-export type { EmbeddingOptions, EngineOptions, JevOptions } from "./stdio";
-export { Vault, createVault, openVault, sliceUtf8, type VaultOptions, type HostedOptions, type RecallArgs, type ReadArgs, type RememberArgs, type RemoveArgs, type ResumeArgs, type ReflectArgs, type ReflectionContext, type Decision, type InspectedSource, type InspectedSpan, type TemporalAssessment, type PairAssertion, } from "./vault";
-export type { Reply, ToolError, Unavailable, Status, ByteRange, Scope, SourceRef, RelationshipKind, Relationship, RecallRetrieval, SourceExcerpt, RecallCard, RecallResult, ReadResult, RememberResult, RemoveResult, ReflectResult, AffectedDoc, EntityAffectedGroup, WriteState, WriteFailureCode, WriteReceipt, WriteResolution, WriteOutcome, ReflectionWork, SourceWork, PairWork, HubWork, PromotionWork, DeliveredSource, CompleteDelivery, FailedDelivery, ReflectDelivery, DecisionReceipt, SourceDisposition, PairDisposition, } from "./types";
+export type { EmbeddingOptions, EngineOptions, JevOptions } from "./native";
+export { Vault, createVault, openVault, sliceUtf8, type VaultOptions, type HostedOptions, type RecallArgs, type ReadArgs, type RememberArgs, type RemoveArgs, type ResumeArgs, type ReflectArgs, type Decision, } from "./vault";
+export type { Reply, ToolError, Unavailable, Status, ByteRange, Scope, SourceRef, RelationshipKind, Relationship, RecallRetrieval, SourceExcerpt, RecallCard, RecallResult, ReadResult, RememberResult, RemoveResult, ReflectResult, AffectedDoc, EntityAffectedGroup, WriteState, WriteFailureCode, WriteReceipt, WriteResolution, WriteOutcome, UnresolvedWrite, ReflectItem, SourceItem, PairItem, HubItem, PromotionItem, DeliveredSource, DecisionReceipt, FocusError, ReflectNext, } from "./types";