melchizedek-agents 0.9.6 → 0.16.0

package/.env.example

@@ -0,0 +1,272 @@
+# ── Required for cloud models ─────────────────────────────────────────────────
+# Google AI Studio API key — powers all Gemini model inference.
+# Also required for the gemini-embedding-001 embedding model used by long-term memory.
+# Get yours free at: https://aistudio.google.com
+# NOT needed for all-local syndicates (tutor, council) — see the Ollama note below.
+GOOGLE_GENAI_API_KEY=
+
+# ── Optional: Ollama (open-weight local models) ───────────────────────────────
+# NO KEY NEEDED. Syndicates whose every agent uses an ollama/* model id (e.g.
+# ollama/qwen3:8b) run entirely on your machine via Ollama's OpenAI-compatible
+# API — install from https://ollama.com and `ollama pull qwen3:8b`.
+# Set this only to point at a non-default Ollama endpoint.
+# OLLAMA_BASE_URL=http://localhost:11434/v1
+# A local reasoning model that thinks without answering (OLLAMA_MAX_TOKENS /
+# OLLAMA_EMPTY_RESPONSE) is retried once with thinking off; false disables it.
+# OLLAMA_RETRY_WITHOUT_THINKING=true
+
+# Where the task tools and the background worker (npm run assistant:worker)
+# keep their task list and job queue. Default: outputs/tasks.json.
+# MELCHIZEDEK_TASKS_FILE=outputs/tasks.json
+
+# ── Optional: MCP (Model Context Protocol) ────────────────────────────────────
+# ALLOW_PRIVATE_MCP: set to "true" to permit mcp_server_url values pointing at
+# localhost / private / link-local hosts. Off by default as an SSRF guard;
+# turn it on for local development — e.g. the librarian demo (npm run mcp:demo).
+ALLOW_PRIVATE_MCP=false
+
+# ── Optional: Anthropic / Claude ──────────────────────────────────────────────
+# Required only when using claude-* model identifiers in syndicate YAML configs.
+# Get your key at: https://console.anthropic.com
+ANTHROPIC_API_KEY=
+
+# ── Optional: OpenAI / GPT ────────────────────────────────────────────────────
+# Required only when using gpt-* (or o-series) model identifiers in YAML.
+# Get your key at: https://platform.openai.com/api-keys
+OPENAI_API_KEY=
+
+# ── Optional: cloud platforms and proxies (ADR 0023) ─────────────────────────
+# The model id picks the provider; these pick how it is reached. Default:
+# the vendor's public API with the key above. Tested against mocks only.
+# GEMINI_PLATFORM=vertex          # + GOOGLE_CLOUD_PROJECT, GOOGLE_CLOUD_LOCATION (ADC)
+# ANTHROPIC_PLATFORM=bedrock      # + AWS_REGION (AWS chain; npm install @anthropic-ai/bedrock-sdk)
+# ANTHROPIC_PLATFORM=vertex       # + ANTHROPIC_VERTEX_PROJECT_ID, CLOUD_ML_REGION (npm install @anthropic-ai/vertex-sdk)
+# OPENAI_PLATFORM=azure           # + AZURE_OPENAI_ENDPOINT, and AZURE_OPENAI_API_KEY or Entra ID (@azure/identity)
+# AZURE_OPENAI_ENDPOINT=
+# AZURE_OPENAI_API_KEY=
+# ANTHROPIC_BASE_URL= / OPENAI_BASE_URL=   # an internal proxy in front of the vendor API
+# ANTHROPIC_MODEL_MAP={"claude-sonnet-4-6":"us.anthropic.claude-sonnet-4-6-v1:0"}
+# OPENAI_MODEL_MAP={"gpt-5-mini":"my-azure-deployment"}
+# GEMINI_MODEL_MAP={}
+
+# ── Optional: xAI / Grok ──────────────────────────────────────────────────────
+# Required only when using grok-* model identifiers in YAML.
+# Get your key at: https://console.x.ai
+XAI_API_KEY=
+
+# XAI_COLLECTION_IDS: comma-separated xAI Collection ids searched by the
+# `collections_search` tool (grok-* agents only). Create collections and
+# upload documents at console.x.ai. Declared with no ids set, the tool is
+# omitted with a warning. Optional cap: XAI_COLLECTIONS_MAX_RESULTS.
+# XAI_COLLECTION_IDS=collection_abc123,collection_def456
+# XAI_COLLECTIONS_MAX_RESULTS=10
+
+# Optional server-side constraints for the x_search tool (grok-* agents only).
+# Dates are YYYY-MM-DD (inclusive); handle lists are comma-separated (max 20,
+# leading @ tolerated) and mutually exclusive — if both are set the allowlist
+# wins. Malformed values degrade with a warning, never fatally.
+# XAI_X_SEARCH_FROM_DATE=2026-08-01
+# XAI_X_SEARCH_TO_DATE=2026-08-08
+# XAI_X_SEARCH_ALLOWED_HANDLES=Reuters,AP
+# XAI_X_SEARCH_EXCLUDED_HANDLES=spam_account
+
+# Optional server-side domain filters for the web_search tool on grok-* agents
+# (comma-separated, max 5, mutually exclusive — allowlist wins if both set;
+# OpenAI's web_search ignores these and always ships bare). NOTE: xAI's
+# web_search accepts NO date bounds — from_date/to_date are x_search-only.
+# XAI_WEB_SEARCH_ALLOWED_DOMAINS=reuters.com,apnews.com
+# XAI_WEB_SEARCH_EXCLUDED_DOMAINS=pinterest.com
+
+# ── Optional: Supabase (Session Persistence & Long-Term Memory) ───────────────
+# Required when memory_system is "session-only" or "long-term" in a syndicate YAML.
+# Without these, the framework falls back to in-memory sessions (no persistence).
+#
+# Setup: Create a free project at https://supabase.com, then go to:
+#   Project Settings → API → Project URL     → paste below as SUPABASE_URL
+#   Project Settings → API → service_role key → paste below as SUPABASE_SERVICE_ROLE_KEY
+#
+# Install the schema before sessions and memory will work:
+#   npx melchizedek-db print   (paste into the Supabase SQL Editor), or
+#   npx melchizedek-db apply   (with DATABASE_URL set; needs psql)
+# Clone: npm run db -- print | apply. It runs db/migrations/*.sql, then db/hardening.sql.
+SUPABASE_URL=
+SUPABASE_SERVICE_ROLE_KEY=
+
+# ── Optional: Telemetry Sink ──────────────────────────────────────────────────
+# Set to "true" (with the Supabase credentials above) to persist every model
+# request's tokens/latency (llm.request spans) and syndicate-run spans to the
+# adk_telemetry table. Provision it first: run db/telemetry.sql in the Supabase
+# SQL Editor, then re-run db/hardening.sql. See DOCUMENTATION.md §5.
+TELEMETRY_SUPABASE=false
+
+# ── Optional: A2A Server ──────────────────────────────────────────────────────
+# A2A_SERVER_SECRET: Bearer token used to authenticate incoming A2A requests.
+# If set, all requests must include: Authorization: Bearer <this_value>
+# If not set, the A2A server runs without authentication and binds 127.0.0.1 only.
+# Generate one with: openssl rand -hex 32
+# REQUIRED in production: if PUBLIC_URL is set, the server REFUSES TO START
+# without this secret, to avoid exposing an unauthenticated endpoint.
+A2A_SERVER_SECRET=
+
+# PUBLIC_URL: The publicly accessible base URL of the A2A server.
+# Used when compiling the AgentCard that advertises this agent to A2A clients.
+# Defaults to http://localhost:4000 if not set.
+# Only needed for cloud deployments where the server has a real public URL.
+PUBLIC_URL=
+
+# ── A2A server: identity and billing (all optional; ADR 0017, 0025) ─────────
+# A2A_AUTH decides who a caller is, and so whose data a request touches:
+#   secret  (default) every caller presents A2A_SERVER_SECRET; the calling
+#           backend names its end user in X-User-Id.
+#   callers one bearer token per calling backend. Each owns a scope, stable
+#           across model-key rotation. A2A_SERVER_SECRET, if still set, keeps
+#           working beside them (old scoping) while callers move over.
+#   jwt     a JWT from your identity provider; the scope comes from its claims.
+#   header  an authenticating gateway in front sets the user header; requires
+#           A2A_SERVER_SECRET, which only the gateway holds.
+# A2A_AUTH=secret
+# callers: `name:sha256[:scope]` entries separated by ';'. The config holds only
+# each token's SHA-256. Mint one: npx melchizedek-serve --new-caller <name>
+# A2A_CALLERS=
+# jwt: a JWKS URL (RS256/ES256) or an HS256 secret of 32+ characters.
+# A2A_JWT_JWKS_URL=https://your-idp.example/.well-known/jwks.json
+# A2A_JWT_SECRET=
+# A2A_JWT_ISSUER=https://your-idp.example/
+# A2A_JWT_AUDIENCE=melchizedek
+# A2A_JWT_SCOPE_CLAIM=sub
+# A2A_JWT_TENANT_CLAIM=
+# header: the header your gateway sets after authenticating the user.
+# A2A_TRUSTED_USER_HEADER=X-Authenticated-User
+# A2A_KEY_MODE: who pays for models. server (default): the server's own keys.
+# byok: each caller's X-API-Key funds its X-Provider. With A2A_AUTH=secret,
+# byok also scopes data by a hash of that key (the pre-0.16 behaviour).
+# A2A_KEY_MODE=server
+# Bare agent ids that load from the adk_agent_registry table (comma list).
+# Every other bare id is a file; registry:<id> always means the registry.
+# A2A_REGISTRY_AGENTS=
+
+# ── A2A server: binding, limits, lifecycle (all optional) ─────────────────────
+# Without A2A_SERVER_SECRET the server binds 127.0.0.1 only. To bind another
+# address without a secret you must also set ALLOW_UNAUTHENTICATED=true.
+# PORT=4000
+# HOST=0.0.0.0
+# ALLOW_UNAUTHENTICATED=false
+# The server refuses a database behind the migrations it ships (run
+# `npx melchizedek-db apply`); this starts it anyway.
+# ALLOW_SCHEMA_MISMATCH=false
+# Only these agent ids are served at /<agentId>/... (comma list). Unset = any
+# syndicate the loader can find.
+# A2A_SERVED_AGENTS=support_triage,research_brief
+# Per-task wall-clock budget (ms; 0 = none). Default 15 minutes.
+# A2A_TASK_TIMEOUT_MS=900000
+# Stream answers token by token on message/stream, as an `answer` artifact
+# (syndicates with guards never stream).
+# A2A_STREAM_TEXT=true
+# How long a second message on a busy conversation waits for the running
+# turn before it is rejected (ms).
+# A2A_TURN_LOCK_WAIT_MS=30000
+# Count the rate limits in Redis, one window across replicas (needs
+# `npm install redis`).
+# A2A_REDIS_URL=redis://localhost:6379
+# Concurrent tasks across all agents (0 = unlimited).
+# A2A_MAX_CONCURRENT_TASKS=0
+# Task submissions per window per client IP, and failed logins before an IP is blocked.
+# A2A_RATE_LIMIT_MAX=60
+# A2A_RATE_LIMIT_WINDOW_MS=900000
+# A2A_AUTH_FAILURE_MAX=30
+# Express trust-proxy: proxy hop count, true/false, or a subnet list. Default 1.
+# A2A_TRUST_PROXY=1
+# A2A_BODY_LIMIT=1mb
+# How long SIGTERM waits for running tasks before canceling them.
+# A2A_SHUTDOWN_GRACE_MS=25000
+# Daily budgets per UTC day (ADR 0026), per caller and per scope (one end
+# user), on tasks, model calls and tokens. A task over budget ends `rejected`
+# with the reason. Counted in Postgres/Supabase (db/migrations/0004) when
+# configured, else in process memory.
+# A2A_BUDGETS={"perCaller":{"tokens":5000000},"callers":{"reports":{"tasks":100}},"perScope":{"tasks":50}}
+# Bearer token for GET /metrics (Prometheus). Unset = no metrics route.
+# A2A_METRICS_TOKEN=
+# text (default) or json: one JSON object per line, with a `task` record per task.
+# A2A_LOG_FORMAT=text
+# Postgres for every durable store (sessions, memory, A2A tasks, erase), safe
+# across instances (ADR 0021); also what `melchizedek-db apply` connects to.
+# Without it: Supabase when its credentials are set, else process memory.
+# DATABASE_URL=
+# Connection counts per instance: the storage pool, and one connection per
+# running turn for the conversation lock. Behind Supabase's session-mode
+# pooler (port 5432) keep their sum under its client limit.
+# DATABASE_POOL_MAX=10
+# A2A_TURN_LOCK_POOL_MAX=20
+# The Postgres schema the tables live in (default public). A private schema
+# such as `melchizedek` keeps them off a Supabase REST layer; install into it
+# with `melchizedek-db apply --schema <name>` (or this variable) and run the
+# server with the same value. Needs DATABASE_URL.
+# MELCHIZEDEK_DB_SCHEMA=public
+
+# ── Optional: Knowledge Bundle (wiki) ─────────────────────────────────────────
+# WIKI_ROOT: filesystem path of the OKF knowledge bundle the wiki tools serve.
+# Defaults to <repo>/wiki (this repo ships its bundle there). Point it at any
+# other OKF bundle — the tooling is bundle-agnostic (npm run wiki:init).
+WIKI_ROOT=
+
+# MCP_WIKI_PORT: port for the wiki MCP server (npm run mcp:wiki). Default 8933.
+MCP_WIKI_PORT=
+
+# WIKI_AGENT_MODEL: model for wiki_query / wiki_garden. Any registry-routable
+# string (gemini-*, claude-*, ollama/*, ...). Default: gemini-3.8-flash.
+WIKI_AGENT_MODEL=
+
+# ── Optional: one gateway key for providers you have no key for ───────────────
+# A model whose direct key is ABSENT is served through the gateway's OpenAI-
+# compatible endpoint (native search tools are dropped on that path; the
+# doctor says which). A present direct key always wins.
+# MODEL_GATEWAY=vercel            # or openrouter
+# MODEL_GATEWAY_API_KEY=
+# MODEL_GATEWAY_BASE_URL=         # optional: a self-hosted proxy speaking the same dialect
+# MODEL_GATEWAY_MODEL_MAP=        # optional: "claude-sonnet-4-6=anthropic/claude-sonnet-4.6,..."
+
+# ── Optional: tracing ─────────────────────────────────────────────────────────
+# Send every span to any OTLP/HTTP backend (Phoenix, Langfuse, Jaeger, Tempo):
+# a local Phoenix is `docker run -p 6006:6006 arizephoenix/phoenix`.
+# OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:6006
+# OTEL_SERVICE_NAME=melchizedek
+# Print [OTEL_SPAN_JSON] lines to stdout. Off by default: the root span carries
+# the full user message and answer.
+# OTEL_CONSOLE_SPANS=false
+# Which model calls keep their full request/response in adk_payloads
+# (off | errors | sample | all), the sample fraction, and the expiry.
+# TELEMETRY_PAYLOADS=sample
+# TELEMETRY_PAYLOAD_SAMPLE=0.10
+# TELEMETRY_PAYLOAD_TTL_DAYS=30
+# TELEMETRY_REDACT: what is scrubbed from every ledger row before it is written
+# (lib/observability/redact.ts). secret (default) = key-shaped credentials;
+# add any of email, phone, card, ssn (comma list), or all, or off.
+# TELEMETRY_REDACT=secret
+
+# ── Optional: misc ────────────────────────────────────────────────────────────
+# Where syndicate YAMLs are found. Default: <cwd>/config/agents.
+# MELCHIZEDEK_AGENTS_DIR=
+# Bearer credentials for MCP servers that need one: a JSON object of
+# hostname → token, sent only to that exact host and only over https.
+# MCP_BEARER_TOKENS={"mcp.example.com":"..."}
+# Per-page character budget for web_extract (default 15000).
+# WEB_EXTRACT_CHAR_LIMIT=15000
+
+# ── Optional: provider retries ────────────────────────────────────────────────
+# Transient provider failures (408/409/425/429/5xx, connection resets) are
+# retried with jittered backoff, honouring Retry-After up to 60 s, before any
+# output is produced. Total attempts per model call; 1 turns retries off.
+# MODEL_RETRY_MAX_ATTEMPTS=3
+# Per-attempt timeout for xAI (Grok) calls, ms; floor 120000.
+# XAI_TIMEOUT_MS=600000
+
+# ── Optional: long-term memory providers ──────────────────────────────────────
+# Fact extraction runs on any model id (routed like an agent's model); the
+# embedder is gemini (default), openai, ollama or any OpenAI-compatible
+# endpoint. The dimension must match the vector column (768 in db/migrations).
+# MEMORY_EXTRACTION_MODEL=gemini-3.8-flash
+# MEMORY_EMBEDDING_PROVIDER=gemini
+# MEMORY_EMBEDDING_MODEL=
+# MEMORY_EMBEDDING_DIMENSIONS=768
+# MEMORY_EMBEDDING_BASE_URL=        # required for openai-compatible
+# MEMORY_EMBEDDING_API_KEY=         # for openai-compatible

package/AGENT_SETUP.md

@@ -5,7 +5,7 @@ the way people actually work now: your coding agent (Claude Code,
 Cursor, Codex — any of them) does the machine steps, you do the human
 steps, and the seams between the two are explicit.
 
-Repo: https://github.com/jhwadman/melchizedek-agents
+Repo: https://github.com/jhwadman/melch-ai
 Course: https://lyceumagents.com/curriculum/
 This file lives in both places and is identical in both.
 
@@ -19,8 +19,8 @@ Set up the melchizedek-agents framework on this machine.
 1. Check prerequisites: `node --version` must be 22 or newer. If it
    isn't, stop and tell me before doing anything else.
 2. Clone and install:
-     git clone https://github.com/jhwadman/melchizedek-agents.git
-     cd melchizedek-agents
+     git clone https://github.com/jhwadman/melch-ai.git
+     cd melch-ai
      npm install
 3. Create my env file: `cp .env.example .env`. Do NOT put any values in
    it. Never ask me to paste API keys into this chat, never read keys
@@ -49,10 +49,10 @@ conversation's history forever. The stop-conditions keep the agent from
 After the agent reports back:
 
 1. **Optional — zero keys, fully local**: install Ollama
-   (https://ollama.com) and run `ollama pull qwen3:8b`. The two
-   open-weight syndicates (`npm run syndicate:tutor`, `npm run
-   syndicate:council`) then run with an empty `.env` — every agent is a
-   local model, nothing leaves your machine. This is the open-weight course
+   (https://ollama.com) and run `ollama pull qwen3:8b`. The three
+   open-weight syndicates (`npm run syndicate:assistant`, `npm run
+   syndicate:tutor`, `npm run syndicate:council`) then run with an empty
+   `.env` — every agent is a local model, nothing leaves your machine. This is the open-weight course
    path; the cloud syndicates below still want a key.
 2. **Get a Gemini API key** (free): https://aistudio.google.com →
    create key → open `.env` in your editor and set
@@ -70,17 +70,28 @@ After the agent reports back:
    of them — set only what you'll declare in YAML:
    `ANTHROPIC_API_KEY` for `claude-*` (https://console.anthropic.com),
    `OPENAI_API_KEY` for `gpt-*` (https://platform.openai.com/api-keys),
-   `XAI_API_KEY` for `grok-*` (https://console.x.ai). Verify the whole
-   multi-provider surface at any time with `npm run demo:models` — it
-   sends one prompt to every provider you've configured (plus the local
-   model) and prints per-request token/latency traces; unconfigured
-   providers are skipped.
+   `XAI_API_KEY` for `grok-*` (https://console.x.ai). Not sure which
+   you need? `npm run doctor` reads every syndicate and prints what is
+   ready, what is blocked, and which variable unlocks what — read-only,
+   no key value shown. Verify the whole multi-provider surface live with
+   `npm run demo:models` — it sends one prompt to every provider you've
+   configured (plus the local model) and prints per-request token/latency
+   traces; unconfigured providers are skipped.
+5. **Optional — one key instead of several**: `MODEL_GATEWAY=vercel` (or
+   `openrouter`) plus `MODEL_GATEWAY_API_KEY` serves any cloud model id
+   whose direct key is absent through that gateway. It is a fallback: a
+   direct key set beside it always wins for its own provider, and adding
+   one later restores that provider's native search. Through the gateway,
+   Gemini grounding, Anthropic/OpenAI `web_search` and xAI `x_search` are
+   lost — the doctor says so per agent.
 
 ## 3. First tests — meet the agents
 
 Keyless, if you installed Ollama:
 
 ```bash
+npm run syndicate:assistant   # converse, summarize, keep a task list, queue background jobs
+npm run assistant:worker      # (second terminal) runs the jobs the Assistant queues
 npm run syndicate:tutor       # one open-weight agent: name a topic or paste material, be taught
 npm run syndicate:council     # advocate/skeptic council, all local
 ```
@@ -150,22 +161,26 @@ application calls like any other API:
    `A2A_SERVER_SECRET` in `.env`.
 2. `npm run start:a2a` — the endpoint speaks the open A2A protocol,
    publishes an agent card, and enforces bearer auth + rate limiting.
-3. Every request carries three headers: `Authorization: Bearer
-   <A2A_SERVER_SECRET>`, `X-API-Key: <a Gemini key>` (inference bills
-   the caller — your server key stays yours), and `X-User-Id: <your
-   app's opaque user id>` — the id your backend assigns after
-   authenticating its own user, never one the user chooses. That last
-   header is what keeps one user's memory out of another's session.
+3. Every request carries `Authorization: Bearer <A2A_SERVER_SECRET>`
+   and `X-User-Id: <your app's opaque user id>` — the id your backend
+   assigns after authenticating its own user, never one the user
+   chooses. That header is what keeps one user's sessions and memory out
+   of another's. The server's own provider keys pay for inference. (To
+   make callers fund their own, set `A2A_KEY_MODE=byok`; each request
+   then also carries `X-API-Key`.) To give each calling backend its own
+   token instead of the shared secret, set `A2A_AUTH=callers` and mint
+   one per caller with `npx melchizedek-serve --new-caller <name>`; for
+   your identity provider's JWTs, `A2A_AUTH=jwt` (DOCUMENTATION.md §6).
 4. `demo/a2a_demo.mjs` is a complete working client; `DOCUMENTATION.md`
-   §A2A has the protocol details, and `lib/memory/README.md` covers
-   per-user memory siloing and the right-to-erasure endpoint before you
+   §6 is the HTTP reference, and `DELETE /memory` erases everything
+   stored for a user — facts, transcripts and ledger rows — before you
    serve real people.
 
 When you're ready, hand your coding agent a prompt shaped like this:
 
 ```text
 My melchizedek-agents A2A server runs at <URL>. Read demo/a2a_demo.mjs
-and DOCUMENTATION.md §A2A in the melchizedek-agents repo, then wire my
+and DOCUMENTATION.md §A2A in the melch-ai repo, then wire my
 app's <feature> to it: send each authenticated user's message via
 message/send with the three required headers, stream or poll the reply,
 and render it in <where>. The bearer secret and Gemini key come from my
@@ -188,7 +203,8 @@ right place to learn. When a syndicate of yours deserves its own
 repository, the same engine installs as a typed npm dependency —
 `npm install melchizedek-agents` — with the starter pack inside the
 package to copy from, `loadSyndicate` reading your repo's own
-`config/agents/`, and the two CLIs (`npx melchizedek-chat`,
-`npx melchizedek-serve`) replacing the npm scripts. QUICKSTART §7 in the
+`config/agents/`, and its CLIs (`npx melchizedek-chat`,
+`npx melchizedek-serve`, `npx melchizedek-worker`) replacing the npm
+scripts. QUICKSTART §7 in the
 repo is the complete recipe; hand it to your coding agent when the time
 comes.