melchizedek-agents 0.16.2 → 0.18.0

package/DOCUMENTATION.md

@@ -98,6 +98,8 @@ Field reference:
 | `memory_extraction_model` | root | The model that distils this syndicate's turns into memory records (long-term only); any model id. Default: the deployment's `MEMORY_EXTRACTION_MODEL`. |
 | `memory_retention_days` | root | Days a fact in this syndicate's namespace is kept; the server deletes older facts when it loads the syndicate and daily after. Requires `memory_namespace` (never applied to the shared default namespace). |
 | `dispatch` | root | Switches the syndicate from DELEGATE to PLAN-DISPATCH routing (§6). `default_route` (required) names the fail-static subagent; `route_key` / `reason_key` name the router's JSON properties (defaults `route` / `reason`). Honoured by every surface (the CLI, the server, the worker, evals). |
+| `workflow` | root | Switches the syndicate to a WORKFLOW (§6): a graph whose nodes are its agents plus declared `join`, `map`, `tool` and `ask_user` nodes, and whose `edges` say what runs after what and on which route. The orchestrator is a node like any other; nothing delegates. Cannot be combined with `dispatch`. Contract: `lib/workflow.ts`; worked example: `examples/pipeline.yaml`. |
+| `retries` | root | Self-correction, on by default ([ADR 0034](./wiki/decisions/0034-self-correction.md)): `model_errors` (default 2) retries a model reply ADK marks malformed (`MALFORMED_FUNCTION_CALL`) instead of failing the turn; `tool_errors` (default 3) answers a tool that threw with structured reflection guidance and caps its retries. `0` turns either off. Every retry is a model call under `max_steps`. |
 | `guards` | root | Optional list of post-answer guard NAMES (`lib/guards/index.ts`). Each runs after the answering turn and before the reply publishes, receiving the final text plus every tool-result text of that turn, and rewrites in place rather than re-asking the model; its notes land in the `[STATUS]` stream. Guards declared by a syndicate reached through `yaml_reference:` count too — the server resolves the union via `collectGuards()`. Resolved by name, never by module path, so adding one is a deliberate act in code; **the published registry ships one, `science`** (citation checks for `research.yaml`); `registerGuard()` adds your own from code, and an unregistered name is warned about and skipped. |
 | `name` / `model` / `instruction` | agent | The agent triple. Any Gemini id, `claude-*`, or `ollama/*` for open-weight local models (see §5). |
 | `description` | subagent | **The delegation API.** The orchestrator reads this when deciding to hand off — write it like a function signature ("Use this subagent to…, pass it…"). |
@@ -108,7 +110,15 @@ Field reference:
 | `a2a_agent_url` | subagent | A REMOTE agent over A2A (§6): the orchestrator delegates to it with one `request` argument; in plan-dispatch it can be a route. No `model`/`instruction` — the remote agent has its own. Credentials come from `A2A_AGENT_TOKENS`, never YAML. |
 | `max_steps` | root | Cap on model calls per turn, counted across every agent the turn reaches (orchestrator, subagents, nested syndicates). Exceeding it fails the turn with `STEP_LIMIT`. |
 | `includeContents` / `outputKey` / `globalInstruction` / `disallowTransferToParent` / `disallowTransferToPeers` | agent | Passed through to ADK's LlmAgent. `includeContents: none` makes an agent see only the current message. |
+| `fallback_model` | any agent | A model, ideally on another provider, that answers when this agent's model fails provider-side (5xx, 429, a connection reset, after its own retries) before producing any output, or while that provider's circuit is open: after `MODEL_BREAKER_THRESHOLD` consecutive provider failures (default 5; 0 disables) the provider is skipped for `MODEL_BREAKER_COOLDOWN_MS` (default 30 s). A 4xx and a canceled turn are never redirected, and a stream that already produced text is never replayed elsewhere ([ADR 0044](./wiki/decisions/0044-fallback-model-and-circuit-breaker.md)). |
+| `mcp_tools` | subagent | The MCP server's tools this agent may use; any other tool the server lists is not exposed. On a dispatch route `require_approval` may name them ([ADR 0041](./wiki/decisions/0041-tool-vendors-get-least-privilege.md)). |
 | `mcp_server_url` | subagent | Discover this subagent's tools from a remote MCP server at load time (§3). SSRF-guarded; `ALLOW_PRIVATE_MCP=true` permits localhost for development. |
+| `openapi` | any agent | HTTP APIs as tools, each from an OpenAPI 3 spec file (§3, OpenAPI tools): `spec`, and optionally `operations` (default: the GET operations only), `auth` (from environment variables), `base_url`, `prefix`. |
+| `code_execution` | any Gemini agent | `"gemini"`: the model writes Python and Gemini runs it in Google's server-side sandbox, returning the output to the model; nothing runs on this host. For arithmetic, data and checks a model gets wrong in its head. Gemini models only ([ADR 0033](./wiki/decisions/0033-context-task-code.md)). |
+| `context` | orchestrator | Compacts a long conversation: when the last request's prompt passed `compact_after_tokens`, earlier turns become one summary (written by `summary_model`, default the agent's own) and the last `keep_recent_events` stay verbatim. The full history stays stored; only what the model reads shrinks. The orchestrator of a delegate syndicate only: a dispatch route already reads a bounded projection, a workflow node sees only its input. |
+| `mode` | workflow node | `"task"`: the agent works with its tools until it calls `finish_task`, whose arguments (matching its `outputSchema`) become the node's output. Workflow nodes only. |
+| `examples` | any agent | Few-shot exchanges, `[{ input, output }]` (up to 20), added to every request's instruction by ADK's `ExampleTool`; the model never calls it. Keeps worked examples out of the prose of `instruction`. |
+| `skills` | any agent | Agent Skills (a directory of SKILL.md folders) the agent holds the way a coding harness does: every skill's name and description is appended to its instruction at compile time; `load_skill` reads one in full with the names of its files, `load_skill_resource` reads one file. `scripts: local` adds `run_skill_script`, which runs a skill's own scripts on this machine, each after a person approves (the `require_approval` pause); `tools:` names registry tools a skill's `allowed-tools` may unlock once loaded. Worked example: `examples/harness.yaml`; engine: `lib/tools/skillToolset.ts`. |
 
 Validation happens at load: missing names, legacy option blocks, and
 malformed agents fail with pointed errors before any model is called.
@@ -122,16 +132,18 @@ instance:
 
 | Name | Kind | Does |
 |---|---|---|
-| `web_search` | Provider-agnostic | Live web search via the agent model's NATIVE search: Gemini grounding, Anthropic `web_search` server tool, OpenAI Responses `web_search`, xAI Agent Tools `web_search`. On local `ollama/*` models the tool is omitted with a one-time warning (keyless stays keyless). On grok-* agents, optional server-side domain filters via `XAI_WEB_SEARCH_ALLOWED_DOMAINS` / `_EXCLUDED_DOMAINS` in `.env` (max 5, mutually exclusive; xAI accepts no date bounds — those are `x_search`-only). Prefer this in new YAMLs. |
+| `web_search` | Provider-agnostic | Live web search via the agent model's NATIVE search: Gemini grounding, Anthropic `web_search` server tool, OpenAI Responses `web_search`, xAI Agent Tools `web_search`. On local `ollama/*` models, and on `kimi-*` (Moonshot's model-side search retires 2026-10-20; its successor is a REST API, not a request field), the tool is omitted with a one-time warning (keyless stays keyless). On grok-* agents, optional server-side domain filters via `XAI_WEB_SEARCH_ALLOWED_DOMAINS` / `_EXCLUDED_DOMAINS` in `.env` (max 5, mutually exclusive; xAI accepts no date bounds — those are `x_search`-only). Prefer this in new YAMLs. |
 | `web_extract` | FunctionTool | Deterministic page reading: fetches 1–5 agent-chosen URLs and returns clean page text (no LLM summarization). Keyless — works on every provider including local `ollama/*`. Per-page char budget (default 15k, `WEB_EXTRACT_CHAR_LIMIT`); long pages return a head+tail window with an `offset` continuation call served from a 15-minute cache. SSRF-guarded (http(s) only, private/link-local hosts refused, redirects re-checked). Block pages (bot checks, paywall stubs, JS shells) are detected code-side and returned as labeled `Error:` blocks, never as content. Pair with `web_search`: search to find, extract to read past the headline. |
 | `x_search` | xAI-only | Live search over X (Twitter) posts via xAI Agent Tools. Self-gates to `grok-*` agents; a silent no-op on every other provider, so mixed-provider YAMLs stay safe. Optional server-side constraints in `.env`: `XAI_X_SEARCH_FROM_DATE`/`_TO_DATE` (inclusive `YYYY-MM-DD`) and `_ALLOWED_HANDLES`/`_EXCLUDED_HANDLES` (max 20, mutually exclusive — allowlist wins). |
 | `collections_search` | xAI-only | Semantic search over xAI **Collections** — hosted document stores (PDFs/text/CSVs) uploaded at console.x.ai — server-side RAG with `collections://…` citations. Which collections: `XAI_COLLECTION_IDS` in `.env` (optional `XAI_COLLECTIONS_MAX_RESULTS`). Declared with no ids → omitted with a warning; non-xAI providers → silent no-op. |
+| `url_context` | Gemini built-in | Gemini reads the pages at URLs in the conversation, server-side (Google fetches them, not this host). On any other provider it is a no-op the doctor reports as dropped; use `web_extract` there. |
 | `google_search` | ADK built-in | Live web search — Gemini agents only (legacy alias; use `web_search`). |
 | `preload_memory` | ADK built-in | Silently injects similarity-matched facts into every request (ambient recall). |
 | `load_memory` | ADK built-in | Explicit tool call to search the fact store (deliberate recall). |
 | `generate_image` | FunctionTool | Calls the Gemini image model directly, saves the result under `outputs/`, returns the path. A FunctionTool because binary `inlineData` cannot survive the AgentTool text boundary. |
 | `inspect_image` | FunctionTool | **Blind visual inventory** of a file under `outputs/`: subjects with exact counts, composition, light, palette, medium cues, artifacts — zero quality judgments. Its signature accepts *only* a file path, so an orchestrator cannot leak expectations into the observation (see `image_production.yaml`). |
 | `task_add` / `task_list` / `task_get` / `task_update` | FunctionTool | A to-do list and job queue. Default: a single-user JSON file (`MELCHIZEDEK_TASKS_FILE`, default `outputs/tasks.json`), so every caller of a shared endpoint shares one list. With `DATABASE_URL` (migration 0009) each caller has its own list, scoped by the caller's scope key, and any number of workers take jobs safely. |
+| `ask_user` | LongRunningFunctionTool | Asks the person one question (optionally with `options`) and ends the turn `input-required`; the next message on the conversation is the call's result. Orchestrator or plan-dispatch route only (§6, Questions). |
 | `task_queue` | FunctionTool | Queues a background job (a self-contained instruction). The tool only writes the queue; `npm run assistant:worker` (`melchizedek-worker`) claims each job, runs it through one agent compiled from YAML (default: the Assistant's Worker), and writes the result back for `task_get`. `--once` drains and exits, for cron. |
 
 **MCP tools** are the exception to the registry: a subagent with
@@ -147,12 +159,50 @@ loopback/private hosts unless `ALLOW_PRIVATE_MCP=true` (SSRF guard);
 treat any remote MCP server as an untrusted tool vendor whose results
 are data, never instructions.
 
+**OpenAPI tools** turn any HTTP API with an OpenAPI 3 spec into an
+agent's tools, with no tool code (`lib/tools/openapiTools.ts`, on ADK's
+`OpenAPIToolset`; [ADR 0032](./wiki/decisions/0032-openapi-tools.md)):
+
+```yaml
+orchestrator:
+  name: Forecaster
+  openapi:
+    - spec: "specs/open-meteo-forecast.json"     # relative to this YAML file
+      operations: [getForecast]                  # omitted: every GET operation
+      auth: { api_key: { env: "WEATHER_KEY", in: "header", name: "X-Api-Key" } }
+      # base_url: overrides servers[0].url · prefix: keeps two APIs apart
+```
+
+One tool per operation, named from its `operationId` in snake_case
+(`getForecast` → `get_forecast`; no `operationId`: path and method), with
+the parameters and request body as its arguments and the operation's
+`summary` as the description the model reads. Exposure is deliberate:
+without `operations`, only GET operations become tools, so anything that
+writes is exposed only by naming it, and a named operation can be listed
+under `require_approval` (as written under `operations`) so a person
+approves each call. `auth` names an environment variable, never a value
+(`bearer_env`, or `api_key` with `in: header | query` and `name`); an unset
+variable fails the compile, and so does one of the framework's own settings
+(the database URL, a provider key, an `A2A_` secret: anything `.env.example`
+documents), since the YAML chooses the host it goes to.
+`OPENAPI_CREDENTIAL_ENVS`, when set, is the exact list of variables an `auth`
+may name. A refused or unset variable fails the compile, and a static token is applied to the request,
+never stored in session state. Every server must be http(s) and pass the
+SSRF guard: its literal rules when the agent compiles, the full check with
+DNS before each call; `ALLOW_PRIVATE_OPENAPI=true` permits private hosts for
+local development. A response over 20,000 characters is cut and says so; a
+network failure comes back to the model as an error. Specs are files, never
+URLs: save the spec beside the YAML and review it like code (trim it to the
+operations the agent needs, and write each `summary` for the model). Worked
+example: `config/agents/examples/weather.yaml`, two keyless Open-Meteo specs
+in `examples/specs/`.
+
 > **Schema dialects, handled for you.** The factory emits Gemini-style
 > UPPERCASE schema types (`'OBJECT'`, `'STRING'`, …) because the ADK is
 > Gemini-native; every non-Gemini adapter normalizes them back to
 > standard lowercase JSON-Schema at request-build time
 > (`lib/models/schemaNormalize.ts`). MCP tools therefore work on any
-> provider's agents — Gemini, Claude, GPT, or Grok.
+> provider's agents — Gemini, Claude, GPT, Grok, or Kimi.
 
 ## 4. Sessions & long-term memory
 
@@ -179,7 +229,11 @@ npx melchizedek-db status    # schema version, hardening, session counts
 
 That runs the migrations in [`db/migrations/`](./db/migrations/) and then
 [`db/hardening.sql`](./db/hardening.sql). Both are idempotent, so re-running
-them is also the upgrade path from any earlier layout.
+them is also the upgrade path from any earlier layout. `apply` runs the whole
+install as one transaction under an advisory lock: if any statement fails,
+nothing is changed, and two applies started together run one after the other.
+Rolling back a migration that succeeded is a restore from backup (§6), since
+the migrations only move forward.
 
 The tables go in the `public` schema by default. To keep them out of reach of
 a REST layer (Supabase exposes `public`), install into a private schema and
@@ -260,6 +314,7 @@ accordingly — model optionality is a single YAML line per agent:
 | `claude-*` | Anthropic | `lib/models/claudeLlm.ts` | `ANTHROPIC_API_KEY` | ✅ server tool |
 | `gpt-*`, o-series | OpenAI | `lib/models/gptLlm.ts` (Responses API) | `OPENAI_API_KEY` | ✅ web_search tool |
 | `grok-*` | xAI | `lib/models/grokLlm.ts` (Responses API) | `XAI_API_KEY` | ✅ Agent Tools search |
+| `kimi-*` | Moonshot AI (Kimi) | `lib/models/kimiLlm.ts` (chat completions) | `MOONSHOT_API_KEY` | ⚠ omitted + warning |
 | `ollama/*` | Local Ollama | `lib/models/ollamaLlm.ts` | none | ⚠ omitted + warning |
 | *any cloud id whose direct key is absent* | the id's own provider, via a gateway | `lib/models/gatewayLlm.ts` (chat completions) | `MODEL_GATEWAY` + `MODEL_GATEWAY_API_KEY` | ⚠ omitted + reported |
 
@@ -271,6 +326,45 @@ structured outputs ride `outputSchema` → `text.format`, and two
 xAI-only tools — `x_search` and `collections_search` (§3) — turn on
 live X search and hosted-document RAG. All verified live on grok-4.5.
 
+**Moonshot AI (Kimi).** `kimi-*` ids route to `lib/models/kimiLlm.ts`, a
+subclass of the chat-completions base, against `https://api.moonshot.ai/v1`
+(`MOONSHOT_BASE_URL` for a proxy). Get a key at platform.moonshot.ai; the
+`.cn` console serves mainland China. The family (USD per 1M tokens, October
+2026): `kimi-k3`, the flagship — 2.8T-parameter open-weight MoE, 1M context,
+vision, always-on thinking, $3 in / $0.30 cache hit / $15 out; `kimi-k2.6`,
+the cheaper general tier (256K, vision, thinking switchable, $0.95 / $4);
+`kimi-k2.7-code` and `kimi-k2.7-code-highspeed` (256K, coding, thinking
+cannot be turned off; the highspeed variant streams at ~180 tokens/s for
+twice the price). There is no small or "flash" K3. On price, K3 is a mid-tier
+closed model, not a bargain: Claude Sonnet 4.6's rate, and Claude Sonnet 5.5
+($2 / $10) undercuts it per token while leading it on the coding boards;
+thinking is always on and billed as output, which is why the adapter pins
+effort below Moonshot's `max`. Choose K3 for the strongest open weights
+(self-hostable) and the flat-rate 1M context; choose `kimi-k2.6` for cost;
+or serve K3 through `MODEL_GATEWAY` (OpenRouter's endpoints run from about
+$0.88 to $3.45 per 1M input, the cheap ones quantized). What works: tool calling
+and delegation, strict `json_schema` structured output (K3 and K2.7 Code
+document it; K2.6 is unstable on `$ref`/`oneOf`), SSE streaming, images as
+base64 (Moonshot takes no public image URLs), MCP tools (they are function
+tools to the model), `reasoning_content` surfaced as THINKING. Reasoning:
+K3 takes a top-level `reasoning_effort` (`low` | `high` | `max`); the adapter
+pins `DEFAULT_KIMI_REASONING_EFFORT` (`high`, `lib/config.ts`) unless
+`generateContentConfig.reasoningEffort` says otherwise, and maps `none` to
+`low` because K3 cannot stop thinking. K2.x models take a `thinking:
+{ type }` switch instead: `reasoningEffort: "none"` or `thinkingBudget: 0`
+sends `disabled`, and any effort value is dropped from the body. What does
+not: native `web_search` (dropped with a warning; `web_extract` works, and
+Moonshot's standalone `POST /v1/tools/search`, billed per call, could back
+a client-side tool), and thinking carried across a tool loop (Moonshot asks
+for K3's `reasoning_content` back on the assistant message; this adapter
+keeps scratchpads out of history, so K3 re-reasons each step). K3's open
+weights mean the same id is served by OpenRouter, Together, Fireworks and
+Vercel AI Gateway; with no direct key, `MODEL_GATEWAY` maps it to
+`moonshotai/kimi-k3`. The row in the capability matrix is asserted against
+the request body the adapter sends; one plain turn was verified live on
+2026-10-03 (`npm run demo:models`, kimi-k3 at low effort, 17.9 s), tool
+loops and search were not.
+
 **Which keys do I need?** `npm run doctor` (the `melchizedek-doctor` bin)
 reads every syndicate YAML, resolves each agent's model under your `.env`,
 and prints one table — agent, model, provider, which declared server-side
@@ -340,7 +434,7 @@ line each. `config/agents/examples/claude.yaml` is the minimal Claude example;
 provider, and `npm run demo:models` proves the whole surface: one
 prompt to every available provider, printing input, thinking (qwen3
 `<think>` blocks, Claude extended thinking, GPT reasoning summaries,
-Grok reasoning), output, and a per-request token/latency trace; add
+Grok reasoning, Kimi reasoning_content), output, and a per-request token/latency trace; add
 `-- --search` to watch four native web searches plus the local
 omission. Providers without keys are skipped, never fatal.
 
@@ -415,7 +509,7 @@ options. `demo/a2a_demo.mjs` is a complete client.
 | `POST /a2a/jsonrpc`, `/a2a/rest` | bearer | the default syndicate |
 | `GET /<agentId>/.well-known/agent-card.json` | bearer | another syndicate's card; its URLs point at `/<agentId>/a2a/...` |
 | `POST /<agentId>/a2a/jsonrpc`, `/<agentId>/a2a/rest` | bearer | another syndicate (`A2A_SERVED_AGENTS` restricts which) |
-| `DELETE /memory` | bearer | erase everything stored for the calling scope: facts, sessions (with subagent rows) and ledger rows, with per-store counts; `?all=1` covers every memory namespace |
+| `DELETE /memory` | bearer | erase everything stored for the calling scope: facts, sessions (with subagent rows), ledger rows and A2A tasks, including conversations whose sessions have expired, with per-store counts; `?all=1` covers every memory namespace |
 
 "bearer" means the credential `A2A_AUTH` asks for (below); with no
 `A2A_SERVER_SECRET` and no authenticator the server binds `127.0.0.1` only.
@@ -461,9 +555,28 @@ or pass your own `resolveRequest`.
   The counts live in the `melchizedek_usage` table (`db/migrations/0004_usage.sql`)
   when Postgres or Supabase is configured, else in process memory. A store
   that cannot be read refuses the task.
-- **One record per task** (option `onTaskEnd`): agent, caller, a hash of the
-  scope, status, reason, duration, model calls and tokens.
+- **One record per task** (option `onTaskEnd`): task id, trace id, agent,
+  caller, a hash of the scope, status, reason, duration, model calls and
+  tokens. The trace id is the one the turn's spans and its ledger row carry,
+  and the audit trail's `task.end` row names both ids, so one id joins the
+  log, the ledger, the audit trail and your tracing backend. A request
+  carrying a W3C `traceparent` header is linked, not joined: the turn's root
+  span links to the caller's span and records `caller.trace_id`, and the
+  record carries `callerTraceId`. The turn keeps a trace id of its own, since
+  ledger attribution and erasure key on it and a caller must not choose it.
   `A2A_LOG_FORMAT=json` prints every server line as JSON, this record included.
+- **The audit trail** (`melchizedek_audit`, `db/migrations/0012_audit_log.sql`;
+  option `audit`, supplied by `postgresStorage`): one row per failed
+  authentication (`auth.failure`), task outcome (`task.end`) and erasure
+  (`memory.erase`), with the caller's name, the source address, the agent and
+  task ids, and a hash of the scope. Never a scope key and never conversation
+  content. A trigger refuses UPDATE and DELETE; rows leave only through
+  `SELECT melchizedek_prune_audit(<days>)`, which you schedule (pg_cron, or a
+  job) for the retention your evidence needs, since the source address is
+  personal data. A failed write is logged once and each event is printed to
+  stderr as JSON until writes recover. A database owner can still alter the
+  table; ship the rows to a store you do not administer when the evidence must
+  survive that.
 - **Rate limit**: with an authenticator it counts per caller (an operator's
   backend) or per scope (an end user); under the shared secret, per IP.
 - **The telemetry ledger is redacted before it is written**:
@@ -472,6 +585,30 @@ or pass your own `resolveRequest`.
   `setTelemetryRedactor(fn)`. Sessions and memory are not redacted; use
   `DELETE /memory` there.
 
+#### Where your data goes
+
+What a deployment sends where, how long it is kept, and the setting that
+changes it. "Your Postgres" is the database `DATABASE_URL` (or Supabase)
+names; nothing is kept by the framework anywhere else.
+
+| Data | Goes to | Kept | Change it with |
+|---|---|---|---|
+| The prompt: the user's message, the conversation, tool results | The model provider of each agent that runs (the model-id prefix) | The provider's policy | The `model:` lines in the YAML; `ollama/` ids stay on your machine |
+| Tool calls (`web_search`, `web_extract`, OpenAPI, MCP) | The tool's own host | That host's policy | The agent's `tools:`, `openapi:`, `mcp_server_url` / `mcp_tools` |
+| Sessions (conversation history) | Your Postgres, `adk_sessions` | 7 days after the last message | `ttlDays` (`postgresStorage`) |
+| Long-term memory facts | Your Postgres, `adk_memory_facts`; **the transcript is also sent to the extraction and embedding providers, Gemini by default whatever the agents run on** (the server warns at boot when they differ) | Until erased, or `memory_retention_days` | `MEMORY_EXTRACTION_MODEL`, `MEMORY_EMBEDDING_PROVIDER`, `memory_retention_days` |
+| The ledger: each turn's input, output and tool results (key-shaped secrets redacted) | Your Postgres, `adk_turns`, `adk_telemetry` | **Until you prune it**: schedule `melchizedek_prune_telemetry(<days>)` | `TELEMETRY_REDACT`, the prune's `turn_days` |
+| Sampled full model requests | Your Postgres, `adk_payloads` | 30 days | `TELEMETRY_PAYLOADS`, `TELEMETRY_PAYLOAD_TTL_DAYS` |
+| Traces, when `OTEL_EXPORTER_OTLP_ENDPOINT` is set | Your OTLP collector, conversation content included with key-shaped secrets redacted (`off` drops it) | Your collector's policy | `OTEL_EXPORT_CONTENT` (`redacted`, `off`, `raw`) |
+| The audit trail (no content) | Your Postgres, `melchizedek_audit` | Until `melchizedek_prune_audit(<days>)` | The prune schedule |
+| The task record (no content) | stdout | Your log platform's policy | `A2A_LOG_FORMAT` |
+
+`DELETE /memory` erases a user from every Postgres store above except the
+audit trail, which holds no content, only a hash of the scope. It does not
+reach a provider's own retention: read each provider's data terms, and the
+business associate or data processing agreement you need with it, before
+sending it regulated data.
+
 #### Who pays
 
 `A2A_KEY_MODE` (option `keyMode`), whatever the authenticator:
@@ -533,6 +670,8 @@ JSON); `file` parts are refused.
 
 #### Approval gates (`require_approval`)
 
+A skill script run (`skills.scripts: local`) pauses the same way, without being listed: every `run_skill_script` call waits for the person's answer.
+
 A tool can run only after a person approves the exact call
 ([ADR 0028](./wiki/decisions/0028-approval-gates.md)). The agent lists it:
 
@@ -564,6 +703,35 @@ or native-search sentinels. In code, `runSyndicateTurn` returns
 `status: 'input-required'` with `approval`, and the next turn's part
 `approvalResponsePart(approval.id, approved)` answers it.
 
+#### Questions (`ask_user`)
+
+An agent can ask the person something mid-turn and wait for the answer
+(`lib/runtime/questions.ts`, [ADR 0031](./wiki/decisions/0031-ask-user.md)).
+It lists the tool:
+
+```yaml
+orchestrator:
+  name: Desk
+  tools: [ask_user]   # ask_user(question, options?)
+```
+
+When the model calls `ask_user`, the turn ends `input-required`, final. The
+status message is the question ("Desk asks: Which account? (personal /
+work)") with a data part
+`{ type: 'input_request', interrupt_id, node, message, payload }`, where
+`payload.options` lists the choices when the agent gave some. The caller's
+next message on the same conversation is the answer: its text becomes the
+call's result, and the agent resumes its own tool loop where it asked. A
+workflow's `ask_user` node (§6, Workflows) publishes the same data part and
+is answered the same way, so a client handles both alike.
+
+`ask_user` is allowed where an approval gate is: the orchestrator and the
+subagents of a plan-dispatch syndicate (a dispatch turn that answers goes
+straight back to the route that asked, without the classifier). A delegated
+subagent or a workflow node listing it is a load error. In code,
+`runSyndicateTurn` returns `status: 'input-required'` with `input`
+(`node`, `message`, `payload`), and the next message's text answers it.
+
 #### Limits
 
 | Setting | Default |
@@ -572,9 +740,13 @@ or native-search sentinels. In code, `runSyndicateTurn` returns
 | `A2A_AUTH_FAILURE_MAX` failed logins per IP per 15 min, then blocked | 30 |
 | `A2A_TASK_TIMEOUT_MS` per task | 15 min |
 | `A2A_MAX_CONCURRENT_TASKS` | unlimited |
-| `max_steps` (YAML): model calls per turn, subagents included | none |
+| `A2A_MAX_CONCURRENT_PER_SCOPE`: tasks at once for one end user | 4 |
+| `A2A_MAX_CONCURRENT_PER_CALLER`: tasks at once for one caller | unlimited |
+| `max_steps` (YAML): model calls per turn, subagents included | 50 |
+| `A2A_TRUST_PROXY`: proxies in front (required with `PUBLIC_URL`) | none trusted |
 | `A2A_BODY_LIMIT` | 1 MB |
 | `A2A_SHUTDOWN_GRACE_MS`: SIGTERM waits for running tasks | 25 s |
+| `A2A_SHUTDOWN_DELAY_MS`: of that, `/readyz` fails while requests are still served | 5 s |
 
 The rate limit and the failed-login limit count in the process by default,
 so each replica keeps its own window. With `A2A_REDIS_URL` set (and the
@@ -604,7 +776,11 @@ Without `A2A_SERVER_SECRET` the server binds `127.0.0.1` only; binding
 another `HOST` requires the secret or `ALLOW_UNAUTHENTICATED=true`. With
 `PUBLIC_URL` set it refuses to start without the secret, with the
 `.env.example` placeholder as the secret, or against an unhardened
-database (unless `ALLOW_UNHARDENED_DB=true`). With durable storage it reads
+database (unless `ALLOW_UNHARDENED_DB=true`), and until the deployment
+states its posture: `A2A_AUTH` (prefer `callers` or `jwt`; `secret` lets any
+holder of the secret act as any user), `A2A_SERVED_AGENTS` (a list, or `*`
+for every agent) and `A2A_TRUST_PROXY` (the number of proxies in front, or
+`false`). The refusal names each one missing. With durable storage it reads
 `melchizedek_schema_version` and refuses to start on a database behind the
 migrations it ships, naming `melchizedek-db apply` (or
 `ALLOW_SCHEMA_MISMATCH=true` to start anyway); a database ahead of it only
@@ -630,9 +806,57 @@ local conversation keeps talking to the same remote conversation.
 `Dockerfile` builds the compiled server and runs it as a non-root user with
 a health check; `compose.yaml` adds optional Ollama and Phoenix (traces).
 `npx melchizedek-db print|apply|status` installs and checks the database.
-Set `PUBLIC_URL`, `A2A_SERVER_SECRET` and the provider keys from your
-secret manager; give the orchestrator's stop timeout at least
-`A2A_SHUTDOWN_GRACE_MS`.
+Set `PUBLIC_URL`, `A2A_SERVER_SECRET`, `A2A_AUTH`, `A2A_SERVED_AGENTS`,
+`A2A_TRUST_PROXY` and the provider keys from your secret manager; give the orchestrator's stop timeout at least
+`A2A_SHUTDOWN_GRACE_MS`. On SIGTERM the server first fails `/readyz` and
+keeps serving for `A2A_SHUTDOWN_DELAY_MS` so the load balancer deregisters
+it, then closes the listener and drains running tasks for the rest of the
+grace period; point the readiness probe at `/readyz`, which also fails while
+durable storage does not answer. `MELCHIZEDEK_DOTENV=off` makes every bin
+ignore `.env` files and run on its environment alone.
+
+#### Backups
+
+Everything durable is in the one Postgres you point the server at:
+conversations (`adk_sessions`, `adk_session_events`), long-term memory
+(`adk_memory_facts`), the telemetry ledger, the agent registry and its
+versions, A2A tasks, the task tools' lists and the budget counters. The
+engine takes no backups of it; back it up as you would any production
+database, with your provider's scheduled backups or point-in-time recovery
+(RDS and Cloud SQL automated backups; on Supabase, a paid plan, since the
+Free plan has none).
+
+What a backup protects is the history. The rest can be rebuilt: the schema
+with `npx melchizedek-db apply`, and registry agents by republishing their
+YAML files. Without a backup, lost conversations, memory and ledger rows
+stay lost.
+
+A logical backup works on any provider and is worth having beside its
+snapshots. Use a `pg_dump` at least as new as the server, and the
+schema the tables live in (`public`, or `MELCHIZEDEK_DB_SCHEMA`):
+
+```bash
+pg_dump --schema=public --no-owner --no-privileges -Fc -f melch.dump "$DATABASE_URL"
+# restore into a new database that has pgvector:
+psql "$TARGET_URL" -c 'CREATE EXTENSION IF NOT EXISTS vector'
+pg_restore --no-owner --no-privileges -d "$TARGET_URL" melch.dump
+npx melchizedek-db status    # with DATABASE_URL=$TARGET_URL: schema version and counts
+```
+
+Test the restore, not just the dump: restore into a scratch database and
+compare row counts per table with the source. Two things a backup changes:
+
+- **Erasure.** `DELETE /memory` removes a scope from the live database, not
+  from backups taken before it. Keep backup retention as short as your
+  recovery needs allow. After a restore, re-run any erasures made since the
+  backup was taken. The server logs each one with its scope (`Erasure for
+  scope …`); keep those lines for as long as you keep backups, since a
+  platform's log retention is usually shorter. The same holds for the
+  telemetry dead-letter spool (`outputs/telemetry-deadletter.ndjson`), which
+  holds ledger rows only while the database was unreachable: replay it
+  (`npm run telemetry:replay`) before honouring an erasure, then delete it.
+- **Secrets.** A dump holds every conversation in clear text. Encrypt it at
+  rest and give it the same access rules as the database.
 
 #### The agent registry
 
@@ -793,6 +1017,74 @@ Implemented in `scripts/a2a_server.ts`; like `yaml_reference`, it is
 A2A-only — the CLI runner (`scripts/syndicate_chat.ts`) still compiles
 every syndicate in DELEGATE mode.
 
+### Workflows (`workflow:`) — the third orchestration method
+
+A syndicate that declares a `workflow:` block is a **graph**. Its agents
+are the nodes; `edges` says what runs after what, and on which route;
+nothing delegates and nothing classifies. It is for the shape the other
+two methods cannot express: run these two at once, join what they
+produce, edit it, ask the person, then publish.
+
+```yaml
+workflow:
+  edges:
+    - [START, Planner, { article: [Writer, Checker], default: Answerer }]
+    - [[Writer, Checker], Both, Editor, Confirm, Publisher]
+  nodes:
+    Planner: { route_key: "kind" }                 # an agent: modifiers only
+    Both:    { join: true }                        # waits for every predecessor
+    Editor:  { retry: { max_attempts: 2 } }
+    Confirm: { ask_user: "Publish this draft? Reply yes, or say what to change." }
+```
+
+**Edges.** Each entry is a chain. `START` opens at least one chain. A
+name is an agent or a declared node; a list of names fans out (after one
+node) or fans in (before one node); a map `{ <route>: <node or nodes>,
+default: <node> }` ends a chain and routes on the output of the node
+before it — the `route_key` property of a JSON output (an agent with an
+`outputSchema`), else the trimmed text — with `default` catching what no
+key matched. Every node receives the previous node's output as its
+message; a join hands on `{ <predecessor>: <output>, … }`.
+
+**Nodes.** `workflow.nodes` declares the nodes that are not agents, each
+exactly one kind: `join: true`; `map: <Agent>` (runs the agent once per
+item of a list input, concurrently, `max_parallel` at a time, and outputs
+the list of results); `tool: <registry name>` (runs the tool with the
+node input as its arguments); `ask_user: "<question>"` (below). An entry
+named after an agent carries modifiers only: `route_key`, `retry`
+(`max_attempts`, `initial_delay`, `max_delay`, `backoff_factor`, in
+seconds) and `timeout` (seconds), which any node may carry.
+
+**The pause.** An `ask_user` node ends the turn `input-required`
+(`result.input`: the node, the question, and what it was asked about).
+The next message on the conversation is the answer: the node's output
+becomes `{ "reply": <the answer>, "input": <what it received> }` and the
+graph resumes where it waited. `melchizedek-chat` prints the question
+and takes the next line; the A2A server ends the task `input-required`
+with a data part `{ type: 'input_request', interrupt_id, node, message,
+payload, schema }`, and the client's next message on the same
+conversation resumes it. A later message after the graph has finished
+starts it again from `START`.
+
+**What a node sees.** Unless its YAML sets `includeContents`, a node
+agent sees only its input — not the conversation, not the other nodes —
+which is what makes a graph legible: each agent's input is one value a
+person can read in the trace. Every node runs in the shared session, so
+`memory_system: long-term` would ingest node inputs as user turns;
+`internal-only` is the sensible default for a workflow.
+
+**Failure.** A node's error is not the turn's: a node with `retry` tries
+again (the attempt is recorded in `answer.nodeErrors`), and a node that
+gives up fails the turn with `NODE_FAILED` naming it. `max_steps` and
+the deadline cap the whole graph as they cap any turn.
+
+**Not yet.** Approval gates (`require_approval`, `skills.scripts: local`)
+and remote `a2a_agent_url` subagents are refused inside a workflow by
+the schema, and a workflow cannot be another syndicate's `yaml_reference`
+(ADK cannot yet make a `Workflow` a subagent; nested, only its
+orchestrator runs). The record is [ADR 0030](./wiki/decisions/0030-workflow-graphs.md);
+the contract is `lib/workflow.ts`, on ADK's `Workflow`.
+
 ## 7. Extending the framework
 
 **Call ADK directly (no syndicate)**: the YAML layer is a convenience,
@@ -801,7 +1093,7 @@ the canonical minimal block — `LlmAgent` + `Runner` +
 `InMemorySessionService` straight from `@google/adk`, ~30 lines you can
 copy into any repo that has `@google/adk` installed. Add
 `registerAvailableProviders()` from `lib/models/registry.ts` and the
-same block runs `claude-*` / `gpt-*` / `grok-*` / `ollama/*` ids too.
+same block runs `claude-*` / `gpt-*` / `grok-*` / `kimi-*` / `ollama/*` ids too.
 
 **Add a syndicate**: create `config/agents/<name>.yaml` — copy the
 closest starter-pack file from `config/agents/examples/` or start from
@@ -852,8 +1144,12 @@ by a person; `skills/README.md` records the procedure.
   optional `adk_telemetry` sink and `adk_agent_registry` with its version
   history (an unprotected registry is worst of all: agent definitions writable with the anon key
   means anyone can rewrite the instructions your server boots). The A2A
-  server verifies hardening at boot and is fatal on public deployments
-  without it. Note `service_role` bypasses RLS by design — the hardening
+  server verifies hardening at boot on both storage paths (`DATABASE_URL`
+  and supabase-js) and is fatal on public deployments without it
+  (`ALLOW_UNHARDENED_DB=true` accepts the risk). Over `DATABASE_URL` the check
+  passes on its own when the database has no `anon` or `authenticated` role
+  (plain Postgres: no API serves the tables) or when the tables live in a
+  schema other than `public` (`MELCHIZEDEK_DB_SCHEMA`). Note `service_role` bypasses RLS by design — the hardening
   constrains the API surface, not the trusted server.
 - **A2A**: bearer auth, a failed-login limiter and a request rate limit
   are built in; without `A2A_SERVER_SECRET` the server binds loopback only.

package/QUICKSTART.md

@@ -14,7 +14,7 @@ fifteen. The full reference is [`DOCUMENTATION.md`](./DOCUMENTATION.md).
 - Optional: **Ollama** ([ollama.com](https://ollama.com)) for the
   open-weight local syndicates; provider keys only for the models you
   actually declare — **Anthropic** (`claude-*`), **OpenAI** (`gpt-*`),
-  **xAI** (`grok-*`) — or one `MODEL_GATEWAY` key as the fallback for
+  **xAI** (`grok-*`), **Moonshot** (`kimi-*`) — or one `MODEL_GATEWAY` key as the fallback for
   whatever direct key is missing (native search is lost on that path);
   and a free **Supabase project** (only for persistent sessions / memory).
   Not sure which you need? `npm run doctor` tells you, read-only.
@@ -243,7 +243,7 @@ console.log(result.text);
 |---|---|
 | `Gemini API Key is not configured` | `.env` missing or key not set — step 2. |
 | `[400] Tool call context circulation is not enabled` | The agent's `model:` is too old for agent transfer. Use `gemini-3.8-flash` or newer (all shipped configs already do). |
-| `Model not found` for `claude-*` / `gpt-*` / `grok-*` | The matching provider key (`ANTHROPIC_API_KEY` / `OPENAI_API_KEY` / `XAI_API_KEY`) is not set in `.env`, so the provider wasn't registered — `npm run doctor` names the variable. Or set `MODEL_GATEWAY` + `MODEL_GATEWAY_API_KEY` once for every cloud provider (native search is lost on that path). |
+| `Model not found` for `claude-*` / `gpt-*` / `grok-*` / `kimi-*` | The matching provider key (`ANTHROPIC_API_KEY` / `OPENAI_API_KEY` / `XAI_API_KEY` / `MOONSHOT_API_KEY`) is not set in `.env`, so the provider wasn't registered — `npm run doctor` names the variable. Or set `MODEL_GATEWAY` + `MODEL_GATEWAY_API_KEY` once for every cloud provider (native search is lost on that path). |
 | `GATEWAY_HTTP_ERROR … 404/400` | The gateway rejected the mapped model id. Fix the name once in `.env`: `MODEL_GATEWAY_MODEL_MAP=<your id>=<the gateway's id>`. `GATEWAY_KEY_MISSING` means `MODEL_GATEWAY` is set without `MODEL_GATEWAY_API_KEY`. |
 | `OLLAMA_UNREACHABLE` for `ollama/*` | Ollama isn't running — start the app or `ollama serve`; then check the model is pulled (`ollama list`). |
 | `Refusing to connect to private/loopback MCP host` | The SSRF guard is on (correctly). For the local demo set `ALLOW_PRIVATE_MCP=true` in `.env`. |