melchizedek-agents 0.16.2 → 0.18.0

package/.env.example

@@ -25,6 +25,13 @@ GOOGLE_GENAI_API_KEY=
 # turn it on for local development — e.g. the librarian demo (npm run mcp:demo).
 ALLOW_PRIVATE_MCP=false
 
+# ── Optional: OpenAPI tools (an agent's `openapi:` list) ──────────────────────
+# ALLOW_PRIVATE_OPENAPI: set to "true" to let an OpenAPI tool call a localhost /
+# private / link-local server. Off by default as an SSRF guard. Credentials an
+# `openapi:` entry names (auth.bearer_env, auth.api_key.env) are read from this
+# file or the environment, never written in YAML.
+# ALLOW_PRIVATE_OPENAPI=false
+
 # ── Optional: Anthropic / Claude ──────────────────────────────────────────────
 # Required only when using claude-* model identifiers in syndicate YAML configs.
 # Get your key at: https://console.anthropic.com
@@ -54,6 +61,16 @@ OPENAI_API_KEY=
 # Get your key at: https://console.x.ai
 XAI_API_KEY=
 
+# ── Optional: Moonshot AI / Kimi ─────────────────────────────────────────────
+# Required only when using kimi-* model identifiers in YAML (kimi-k3 is the
+# flagship; kimi-k2.6 the cheaper tier; kimi-k2.7-code[-highspeed] for code).
+# Get your key at: https://platform.moonshot.ai (the .cn console serves
+# mainland China). Without it, MODEL_GATEWAY serves kimi-* ids as
+# moonshotai/kimi-k3 and friends. web_search is never native on Kimi
+# (Moonshot's built-in search retires 2026-10-20); use web_extract.
+MOONSHOT_API_KEY=
+# MOONSHOT_BASE_URL=   # a proxy speaking the same chat-completions dialect
+
 # XAI_COLLECTION_IDS: comma-separated xAI Collection ids searched by the
 # `collections_search` tool (grok-* agents only). Create collections and
 # upload documents at console.x.ai. Declared with no ids set, the tool is
@@ -112,6 +129,8 @@ A2A_SERVER_SECRET=
 # Used when compiling the AgentCard that advertises this agent to A2A clients.
 # Defaults to http://localhost:4000 if not set.
 # Only needed for cloud deployments where the server has a real public URL.
+# With it set, the server also refuses to start until A2A_AUTH,
+# A2A_SERVED_AGENTS and A2A_TRUST_PROXY are set explicitly (ADR 0039).
 PUBLIC_URL=
 
 # ── A2A server: identity and billing (all optional; ADR 0017, 0025) ─────────
@@ -124,6 +143,7 @@ PUBLIC_URL=
 #   jwt     a JWT from your identity provider; the scope comes from its claims.
 #   header  an authenticating gateway in front sets the user header; requires
 #           A2A_SERVER_SECRET, which only the gateway holds.
+# Required with PUBLIC_URL; prefer callers or jwt there.
 # A2A_AUTH=secret
 # callers: `name:sha256[:scope]` entries separated by ';'. The config holds only
 # each token's SHA-256. Mint one: npx melchizedek-serve --new-caller <name>
@@ -154,8 +174,8 @@ PUBLIC_URL=
 # The server refuses a database behind the migrations it ships (run
 # `npx melchizedek-db apply`); this starts it anyway.
 # ALLOW_SCHEMA_MISMATCH=false
-# Only these agent ids are served at /<agentId>/... (comma list). Unset = any
-# syndicate the loader can find.
+# Only these agent ids are served at /<agentId>/... (comma list). Unset, or *,
+# = any syndicate the loader can find. Required with PUBLIC_URL (* allowed).
 # A2A_SERVED_AGENTS=support_triage,research_brief
 # Per-task wall-clock budget (ms; 0 = none). Default 15 minutes.
 # A2A_TASK_TIMEOUT_MS=900000
@@ -174,11 +194,23 @@ PUBLIC_URL=
 # A2A_RATE_LIMIT_MAX=60
 # A2A_RATE_LIMIT_WINDOW_MS=900000
 # A2A_AUTH_FAILURE_MAX=30
-# Express trust-proxy: proxy hop count, true/false, or a subnet list. Default 1.
+# Express trust-proxy: proxy hop count, true/false, or a subnet list. Default
+# false (no proxy is trusted). Required with PUBLIC_URL: 1 behind one load
+# balancer, false when clients connect directly.
 # A2A_TRUST_PROXY=1
+# Concurrent tasks for one end user (scope), and for one caller. 0 = unlimited.
+# A2A_MAX_CONCURRENT_PER_SCOPE=4
+# A2A_MAX_CONCURRENT_PER_CALLER=0
+
+# OpenAPI tools: the exact environment variables an `auth:` may name. Unset, any
+# variable but the framework's own settings (this file's) is allowed.
+# OPENAPI_CREDENTIAL_ENVS=WEATHER_API_KEY,CRM_TOKEN
 # A2A_BODY_LIMIT=1mb
 # How long SIGTERM waits for running tasks before canceling them.
 # A2A_SHUTDOWN_GRACE_MS=25000
+# Of that, how long /readyz fails while requests are still served, so a load
+# balancer deregisters this instance before the listener closes.
+# A2A_SHUTDOWN_DELAY_MS=5000
 # Daily budgets per UTC day (ADR 0026), per caller and per scope (one end
 # user), on tasks, model calls and tokens. A task over budget ends `rejected`
 # with the reason. Counted in Postgres/Supabase (db/migrations/0004) when
@@ -262,6 +294,11 @@ WIKI_AGENT_MODEL=
 # retried with jittered backoff, honouring Retry-After up to 60 s, before any
 # output is produced. Total attempts per model call; 1 turns retries off.
 # MODEL_RETRY_MAX_ATTEMPTS=3
+# For agents with a fallback_model: after this many consecutive provider-side
+# failures a provider's circuit opens and those agents answer from their
+# fallback for the cooldown. 0 disables the breaker.
+# MODEL_BREAKER_THRESHOLD=5
+# MODEL_BREAKER_COOLDOWN_MS=30000
 # Per-attempt timeout for xAI (Grok) calls, ms; floor 120000.
 # XAI_TIMEOUT_MS=600000
 

package/AGENT_SETUP.md

@@ -70,7 +70,8 @@ After the agent reports back:
    of them — set only what you'll declare in YAML:
    `ANTHROPIC_API_KEY` for `claude-*` (https://console.anthropic.com),
    `OPENAI_API_KEY` for `gpt-*` (https://platform.openai.com/api-keys),
-   `XAI_API_KEY` for `grok-*` (https://console.x.ai). Not sure which
+   `XAI_API_KEY` for `grok-*` (https://console.x.ai),
+   `MOONSHOT_API_KEY` for `kimi-*` (https://platform.moonshot.ai). Not sure which
    you need? `npm run doctor` reads every syndicate and prints what is
    ready, what is blocked, and which variable unlocks what — read-only,
    no key value shown. Verify the whole multi-provider surface live with
@@ -192,7 +193,8 @@ user id.
 
 Deploying the server itself (Heroku, Fly, a VPS) needs `PUBLIC_URL` set
 — at which point the server refuses to start without the bearer secret,
-by design. Run `db/hardening.sql` before real user data arrives, and
+and until `A2A_AUTH`, `A2A_SERVED_AGENTS` and `A2A_TRUST_PROXY` are set
+explicitly, by design. Run `db/hardening.sql` before real user data arrives, and
 read the deployment notes in `lib/memory/README.md` end to end. The
 patient in module 2.02's specimen would expect no less of you.
 

package/CHANGELOG.md

@@ -4,6 +4,252 @@ Consumers of the package read this file; it records changes to the
 **published API surface** (the exports map in `package.json`, the bins,
 the starter pack and the templates), not the repo's full history.
 
+## 0.18.0 — 2026-10-06
+
+### Breaking — read before upgrading
+
+- **A public server states its posture (ADR 0039).** With `PUBLIC_URL` set,
+  the server refuses to start until all three are set explicitly; the boot
+  message names each one missing:
+  - `A2A_AUTH`: `callers` or `jwt` (recommended), `header`, or `secret`
+    (one shared secret: any holder can act as any user via `X-User-Id`).
+  - `A2A_SERVED_AGENTS`: the agent ids to serve, or `*` for every agent.
+  - `A2A_TRUST_PROXY`: proxies in front (`1` behind one load balancer), or
+    `false` when clients connect directly.
+- **`A2A_TRUST_PROXY` defaults to `false`** (was 1): a server reached
+  directly no longer takes the client's `X-Forwarded-For` as its address.
+- **A turn stops at 50 model calls** when the YAML sets no `max_steps`
+  (`DEFAULT_MAX_STEPS`). A syndicate that needs more sets `max_steps`.
+- **One end user runs at most 4 tasks at once** (`A2A_MAX_CONCURRENT_PER_SCOPE`,
+  `maxConcurrentPerScope`; 0 = unlimited). `A2A_MAX_CONCURRENT_PER_CALLER`
+  adds a per-caller cap, off by default. The refusal names the cap.
+
+- **The `@google/adk` peer range is `~2.2.0`** (was `^2.2.0`): the tested
+  minor. A new ADK minor can bring a second `@google/genai` copy, whose
+  response shapes the framework pins; the range widens when a release is
+  tested against it. ADK 2.2.0 is the latest published, so no install
+  changes today.
+
+- **An OpenAPI `auth` may not name one of the framework's own settings**
+  (the database URL, a provider key, an `A2A_` secret: anything under the
+  framework's prefixes). The YAML chooses the host too, so this was a way
+  to send a secret anywhere. `OPENAPI_CREDENTIAL_ENVS` makes the rule an
+  exact allowlist. ADR 0041.
+- **MCP tool descriptions are cut at 1,000 characters and results at
+  20,000**, each marked where it was cut.
+
+### Changes
+
+- **An append-only audit trail** (`melchizedek_audit`, migration
+  `0012_audit_log`; ADR 0042). Every failed authentication, task outcome
+  and erasure is recorded with the caller, the source address, the agent
+  and task ids and a scope hash, and no content. A trigger refuses UPDATE
+  and DELETE; `melchizedek_prune_audit(days)` is the retention path.
+  `postgresStorage` supplies the sink (`audit`), and `createA2AApp` takes
+  an `audit` option for any other. Run `npx melchizedek-db apply`.
+- **The server says where memory sends transcripts.** Long-term memory's
+  extraction and embedding providers default to Gemini whatever the agents
+  run on; a syndicate whose agents use other providers now gets one boot
+  warning naming the providers memory reaches and the variables that move
+  it (`memoryDestinations`, `memoryCrossesProviders`). DOCUMENTATION gains
+  "Where your data goes": every destination, its retention, and the setting
+  that changes it.
+- **A request can be followed across systems.** Every task record
+  (`onTaskEnd`, and the JSON log line) carries `taskId` and `traceId`; the
+  trace id is the one the spans and the ledger row carry. A request with a
+  W3C `traceparent` header is linked from the turn's root span
+  (`caller.trace_id`, and `callerTraceId` on the record); the turn keeps
+  its own trace id. `validTraceparent` and `callerSpanContext` are exported
+  from `melchizedek-agents/observability/tracer`. ADR 0043.
+- **`fallback_model:`, an agent key, and a per-provider circuit breaker.**
+  When an agent's model fails provider-side (5xx, 429, a reset, after its
+  retries) before producing output, its `fallback_model` answers; after
+  `MODEL_BREAKER_THRESHOLD` consecutive failures (default 5) the provider
+  is skipped for `MODEL_BREAKER_COOLDOWN_MS` (30 s) by agents that have a
+  fallback. A 4xx, a canceled turn and a half-finished stream are never
+  redirected. ADR 0044.
+- **`mcp_tools:`, an agent key**: the MCP server's tools the agent may use;
+  any other tool the server lists is not exposed. On a dispatch route,
+  `require_approval` may name a listed MCP tool, so a write waits for a
+  person. `closeMcpConnections()` closes every open MCP connection.
+- **`melchizedek-db apply` is one transaction under an advisory lock.** The
+  whole install (migrations, hardening, telemetry) runs as one psql
+  `--single-transaction` script that takes `pg_advisory_xact_lock` first: a
+  failure anywhere rolls everything back, and two applies started together
+  run one after the other. The output still names each file.
+- `TaskLimiter` takes per-scope and per-caller limits and says which one
+  refused (`refusal()`); `createA2AApp` takes `maxConcurrentPerScope` and
+  `maxConcurrentPerCaller`; `DEFAULT_MAX_CONCURRENT_PER_SCOPE` is exported
+  from `melchizedek-agents/a2a` and `DEFAULT_MAX_STEPS` from
+  `melchizedek-agents/config`.
+
+## 0.17.1 — 2026-10-06
+
+- **Fix: a rolling deploy no longer refuses requests.** On SIGTERM the
+  server closed its listener at once, while the load balancer was still
+  sending it traffic. It now fails `/readyz` first and keeps serving for
+  `A2A_SHUTDOWN_DELAY_MS` (default 5 s), then closes and drains. The delay
+  counts inside `A2A_SHUTDOWN_GRACE_MS`, so a stop takes no longer than
+  before. `A2AApp.markUnready()` does the same for an embedding app.
+  ADR 0038.
+- **`/readyz` fails while durable storage does not answer** (503,
+  `reason: 'storage'`), so an instance that lost its database leaves the
+  rotation instead of failing turns. The cause is logged once per outage;
+  the answer is cached for 2 s and concurrent probes share one read, so the
+  unauthenticated route cannot be used to flood the database.
+- **Fix: the MCP client no longer follows redirects past the SSRF guard.**
+  The SSE transport used the default fetch, which follows redirects; it now
+  uses `mcpFetch`, under the rule 0.17.0 gave OpenAPI tools: same-origin hops
+  get the server's check, cross-origin hops the full guard and no
+  credential header.
+- **Fix: a failed MCP connect closes its transport.** The SSE stream's
+  reconnect timer kept running after `createMcpTools` gave up, for the life
+  of the process.
+- **`MELCHIZEDEK_DOTENV=off`** makes every bin ignore `.env` files and run
+  on its environment alone.
+
+## 0.17.0 — 2026-10-06
+
+### Read before upgrading
+
+- **Run `npx melchizedek-db apply`.** Migration `0011_erase_expired` ships
+  with this version, and the server refuses to start against a database
+  behind the shipped migrations.
+- **A public Supabase deployment on `DATABASE_URL` must be hardened.** The
+  boot-time RLS check now runs on that path; `db/hardening.sql` (which
+  `melchizedek-db apply` runs) satisfies it, `ALLOW_UNHARDENED_DB=true`
+  opts out.
+- **Self-correction is on by default.** Retries count against `max_steps`
+  and can raise spend; `retries: { model_errors: 0, tool_errors: 0 }`
+  restores the previous behaviour.
+
+### Changes
+
+- **Behaviour change: the hardening check runs on the `DATABASE_URL` path.**
+  The boot-time RLS check ran only on the deprecated supabase-js path, so a
+  deployment that followed the recommended setup was never checked, and
+  "fatal on public deployments without it" did not hold. `postgresStorage`
+  now supplies `rlsHardening()` and the server applies the same rules
+  (`lib/storage/rlsStatus.ts`): with `PUBLIC_URL` set, a Supabase database
+  whose `public` tables lack RLS stops the server (set
+  `ALLOW_UNHARDENED_DB=true` to accept the risk). A database with no `anon`
+  or `authenticated` role, or with the tables in a private schema, passes.
+  `createA2AApp`'s `storage` option accepts `rlsHardening` for custom storage.
+- **Fix: OpenAPI tools no longer follow redirects past the SSRF guard.**
+  ADK's REST tool calls `fetch`, which follows redirects, and the guard
+  checked only the configured server: an allowed API with an open redirect
+  could send a call to the cloud metadata service, and an `api_key` header
+  went with it. A call now follows redirects one hop at a time
+  (`lib/net/redirects.ts`): same-origin hops get the server's own check,
+  cross-origin hops the full guard (even with `ALLOW_PRIVATE_OPENAPI`) and
+  only content-negotiation headers. A refused hop returns an error to the
+  model. Nothing changes for a fetch made outside an OpenAPI tool call.
+- **Fix: `DELETE /memory` erases expired conversations.** A namespace erase
+  (the default) found conversations through their live session rows, so once
+  a session expired after seven idle days its ledger turns, spans, payloads
+  and A2A tasks survived the erase while the response reported success.
+  Migration `0011_erase_expired` replaces `melchizedek_erase_scope`: a
+  namespace erase now keeps a turn or task only when its conversation is
+  still live in another namespace. Run `npx melchizedek-db apply`; the server
+  refuses to start against a database behind the shipped migrations.
+- **Moonshot AI (Kimi) is a provider.** `kimi-*` model ids route to a new
+  direct adapter (`lib/models/kimiLlm.ts`, chat completions against
+  `api.moonshot.ai`), funded by `MOONSHOT_API_KEY` (`MOONSHOT_BASE_URL` for a
+  proxy). Tool calling, delegation, strict structured output, streaming,
+  images and `reasoning_content` work; native `web_search` is dropped with a
+  warning (Moonshot's model-side search retires 2026-10-20), and thinking is
+  not carried across a tool loop (K3 re-reasons each step; both stated in
+  the capability matrix). `kimi-k3` takes `reasoning_effort` (pinned `high`,
+  `DEFAULT_KIMI_REASONING_EFFORT`); K2.x ids take a thinking switch. With no
+  direct key the gateway serves the ids as `moonshotai/…`. The doctor,
+  `.env.example`, the schema comments and the docs know the provider;
+  `model_zoo.yaml` gains a sixth agent, `kimi` on `kimi-k3` at low effort,
+  and the Zookeeper names six providers. Cost note: `kimi-k3` is priced
+  like Claude Sonnet 4.6 ($3 / $15, thinking billed as output), so
+  `kimi-k2.6` is the budget tier and the gateway the cheap route to K3.
+  One plain turn verified live on 2026-10-03; tool loops were not.
+- **The doctor's providers line names each provider's own key.** It probed
+  every provider past the fourth with a `grok-` id, so Moonshot was reported
+  as missing `XAI_API_KEY`.
+- **Behaviour change: self-correction is on by default.** Every turn now runs
+  ADK's reflect-and-retry plugins. A model reply ADK marks malformed
+  (`MALFORMED_FUNCTION_CALL`) is retried up to twice with guidance instead of
+  failing the turn; a tool that throws is answered with structured reflection
+  guidance and retried at most three times. Every retry counts against
+  `max_steps`. Set `retries: { model_errors: 0, tool_errors: 0 }` at the root
+  of a syndicate to restore the previous behaviour. ADR 0034.
+- **`url_context` and `examples:`.** `url_context` lets a Gemini agent read
+  the pages at URLs in the conversation, server-side; on other providers it is
+  a no-op the doctor reports as dropped. An agent's `examples: [{ input,
+  output }]` adds few-shot exchanges to every request (ADK's `ExampleTool`).
+- **Three agent keys from ADK: `code_execution`, `context`, `mode`.**
+  `code_execution: gemini` lets a Gemini agent write and run Python in
+  Gemini's server-side sandbox (nothing runs on the host). `context:
+  { compact_after_tokens, keep_recent_events?, summary_model? }` on a delegate
+  orchestrator summarizes earlier turns once a prompt passes the threshold,
+  keeping the recent ones verbatim and the full history stored. `mode: task`
+  on a workflow node makes its output the arguments of its `finish_task`
+  call. The schema places each where it means something. ADR 0033.
+- **`openapi:`: any HTTP API as an agent's tools, from its spec.** An agent
+  lists OpenAPI 3 spec files (relative to the syndicate file); every operation
+  becomes a tool named from its `operationId`, its parameters the arguments,
+  its summary the description. Read-only by default (GET operations only,
+  unless `operations` names others); a named operation can be listed under
+  `require_approval`. Credentials come from environment variables
+  (`auth.bearer_env`, `auth.api_key`), never YAML, and are never stored in
+  session state. Every server passes the SSRF guard (literal rules at
+  compile, DNS before each call; `ALLOW_PRIVATE_OPENAPI=true` for local
+  development); results are capped at 20,000 characters. On ADK's
+  `OpenAPIToolset`. Worked example: `config/agents/examples/weather.yaml`
+  (`npm run syndicate:weather`), with two keyless Open-Meteo specs. ADR 0032.
+- **`ask_user`: an agent asks the person and waits.** A registry tool,
+  `ask_user(question, options?)`, on the orchestrator or a plan-dispatch
+  route. Called, it ends the turn `input-required` with `result.input`
+  (`node`, `message`, `payload.options`); the next plain-text message on the
+  conversation becomes the call's result and the agent resumes its own tool
+  loop (a dispatch turn goes straight back to the route that asked). Over A2A
+  the task carries an `input_request` data part — the same one a workflow's
+  `ask_user` node publishes. A delegated subagent or a workflow node listing
+  it is a load error. ADR 0031.
+- **Workflows: the `workflow:` block, the third orchestration method.** A
+  syndicate may be a graph: its agents are the nodes, `edges` says what runs
+  after what and on which route (a map after a node routes on the `route_key`
+  of its JSON output, else its text, with `default`), and `nodes` declares
+  what is not an agent — `join` (fan-in), `map` (one run per list item,
+  concurrently), `tool` (a registry tool as a node), `ask_user` (a pause: the
+  turn ends `input-required` with the question, the next message answers,
+  and the node outputs `{ reply, input }`). Any node may carry `retry` and
+  `timeout`. Every node receives the previous node's output as its message
+  and sees nothing else of the conversation unless its YAML says so. Runs on
+  ADK's `Workflow`; `runSyndicateTurn` gains `status: 'input-required'` with
+  `input` (the A2A server publishes an `input_request` data part;
+  `melchizedek-chat` prints the question and takes the next line);
+  `compileWorkflow` and `isWorkflowSyndicate` are exported. Not yet inside a
+  workflow: approval gates, skill scripts, remote subagents. Worked example:
+  `config/agents/examples/pipeline.yaml` (`npm run syndicate:pipeline`);
+  ADR 0030.
+- **Agent Skills in a syndicate: the `skills:` agent key and the Harness.**
+  An agent may declare `skills: { dir, scripts?, tools? }`. Every skill's
+  frontmatter (name, description) is appended to the agent's instruction at
+  compile time, so no turn is spent discovering skills; `load_skill` reads one
+  SKILL.md in full with the names of the files it ships, and
+  `load_skill_resource` reads one file (`references/`, `assets/`, `scripts/`,
+  the open standard's layout). A skill's `allowed-tools` frontmatter unlocks
+  the registry tools the YAML lists under `skills.tools`, once that skill is
+  loaded. With `scripts: local`, `run_skill_script` runs a skill's own scripts
+  on the host, each run only after a person approves it through the same
+  pause as `require_approval` (A2A `input-required`; `melchizedek-chat` now
+  asks `[y/N]`); the model's own code never executes. Built on ADK's
+  `SkillToolset` (`lib/tools/skillToolset.ts`). `config/agents/examples/harness.yaml`
+  is the specimen: a generic agent that works from whatever shelf of skills
+  `--bind skills_dir=…` points it at (default: this package's own suite).
+- **The suite's files move to the standard `assets/` directory.**
+  `melchizedek-author` ships `assets/minimal.yaml` (was `templates/minimal.yaml`)
+  and `melchizedek-scribe` ships `assets/brief.md` (was `templates/brief.md`),
+  so a harness, this one or any other, can read them as skill resources. A
+  skill installed by an earlier release keeps its old path until it is
+  reinstalled with `--force`.
+
 ## 0.16.2 — 2026-10-02
 
 - **`OTEL_EXPORT_CONTENT=off` also drops library content attributes.** In