@rubytech/create-maxy-code 0.1.0

package/payload/platform/plugins/docs/references/internals.md

@@ -0,0 +1,512 @@
+# Platform Internals — Retrieval Architecture
+
+Technical architecture reference for the retrieval pipeline, knowledge delivery, and supporting infrastructure. This document covers how information moves from the Neo4j graph into an agent's context — the mechanics behind "{{productName}} searches this graph to retrieve relevant context."
+
+Use this reference when assessing capabilities, diagnosing retrieval behaviour, or answering questions about how the platform works internally. When a question asks "does {{productName}} have X?" — check here before asserting a gap.
+
+---
+
+## Retrieval Pipeline Overview
+
+Every knowledge query flows through a hybrid search pipeline that combines semantic similarity with keyword matching, applies layered access controls, expands results via graph traversal, and optionally re-ranks via LLM reasoning.
+
+```
+QUERY
+  │
+  ├── EMBED (EMBED_MODEL, default nomic-embed-text) ──► VECTOR SEARCH (per index, cosine)
+  │ │
+  │ ├──► MERGE ──► EXPAND ──► RESULTS
+  │ │
+  └── ESCAPE (Lucene special chars) ──────► BM25 FULL-TEXT ──┘
+                                            (entity_search index — universal coverage)
+
+Merge formula: combined = 0.7 × vector_score + 0.3 × normalised_bm25_score
+Deduplication: by nodeId — when a node appears in both paths, keep the max score from each method independently, then combine.
+Fallback: if the full-text index doesn't exist, vector-only results are returned (graceful degradation, no error).
+```
+
+### Hybrid Search Detail
+
+**Vector path:** The query is embedded via Ollama (model per `EMBED_MODEL` env var, default `nomic-embed-text`). The resulting vector is compared against Neo4j's HNSW cosine indexes — one per indexed label. Dimensions are configured at install time (default 768). The search runs against all discovered indexes (or a subset if the caller specifies label filters). Scores are in [0, 1] (cosine similarity).
+
+**BM25 path:** The raw query text is escaped for Lucene special characters and run against the `entity_search` full-text index (earlier platform fixes — universal coverage), which spans every operator-meaningful label written by the platform on the canonical text-property union (~28 properties: `name`, `firstName`, `lastName`, `givenName`, `familyName`, `title`, `summary`, `body`, `content`, `description`, `headline`, `email`, `subject`, `bodyPreview`, etc.). Pre-Task-748 the index was named `knowledge_fulltext` and covered only `KnowledgeDocument | Section | Chunk` — that gap silently hid Person/Organization/Task/Event/etc. from BM25 regardless of query. Raw BM25 scores are in [0, infinity) — they are normalised to [0, 1] via min-max scaling within the result set before merging. When all scores are equal (or a single result), all normalise to 1.0.
+
+**Merge:** Results from both paths are collected in a single map keyed by `nodeId`. A node appearing in both paths accumulates the max vector score and max BM25 score independently. The combined score is `0.7 * vectorScore + 0.3 * bm25Score`. Results are sorted descending by combined score, then sliced to the requested limit (default 10).
+
+### What the hybrid approach catches
+
+Vector search excels at semantic meaning — "how do I contact someone" finds nodes about communication even if the word "contact" doesn't appear. BM25 excels at exact terms — invoice numbers, product codes, proper nouns, technical identifiers. The hybrid combination ensures both modes contribute, with semantic similarity weighted higher (0.7) because most user queries are natural language.
+
+---
+
+## Embedding Infrastructure
+
+| Property | Value |
+|---|---|
+| Model | Default `nomic-embed-text` (via Ollama at `localhost:11434`), configurable at install time via `--embed-model` |
+| Dimensions | Default 768, configurable at install time (resolved from model lookup table or `--embed-dimensions`) |
+| Similarity function | Cosine |
+| Index algorithm | HNSW (approximate nearest-neighbor) |
+| Configurable via | `EMBED_MODEL` and `EMBED_DIMENSIONS` env vars (set by installer in `~/{configDir}/.env`), `OLLAMA_URL` |
+
+### Indexed node labels
+
+Every searchable node type has its own vector index. The `memory-search` tool discovers indexes at runtime via `SHOW INDEXES` and caches the label-to-index mapping. This means new index definitions in `schema.cypher` become searchable automatically without code changes.
+
+Indexed labels: `Question`, `DefinedTerm`, `Review`, `Service`, `Person`, `LocalBusiness`, `PriceSpecification`, `Task`, `CreativeWork`, `DigitalDocument`, `KnowledgeDocument`, `Section`, `Chunk`, `Conversation`, `Message`, `Event`, `Workflow`, `Preference`, `Email` (19 labels total).
+
+### Full-text index
+
+| Index name | Labels | Properties | Purpose |
+|---|---|---|---|
+| `entity_search` | All operator-meaningful labels (~40, see [`schema.cypher`](../../../neo4j/schema.cypher)) | Canonical text-property union (~28) | Universal BM25 keyword matching across the whole graph |
+
+### Embedding lifecycle
+
+Embeddings are computed when nodes are created or updated (via `memory-write`, `memory-ingest`, or any tool that persists to Neo4j). If Ollama is unavailable at write time, nodes are stored without embeddings. The `memory-reindex` tool backfills missing embeddings by iterating nodes where `embedding IS NULL`, calling Ollama's `/api/embed` endpoint, and storing the resulting vector. Batch embedding is supported for efficiency.
+
+---
+
+## Knowledge Document Hierarchy
+
+Large documents are decomposed into a three-level hierarchy for granular retrieval:
+
+```
+KnowledgeDocument
+  ├── summary (embedded) — document-level semantic anchor
+  ├── Section
+  │ ├── summary (embedded) — section-level semantic anchor
+  │ └── Chunk
+  │ ├── summary (embedded) — chunk-level semantic anchor
+  │ └── content (raw text, BM25-indexed) — full content for retrieval
+  └── attachmentId — links back to the source file
+```
+
+All three levels are independently vector-indexed and BM25-indexed. A query may match at the document level (broad topic), section level (sub-topic), or chunk level (specific passage). Graph expansion from a matched chunk retrieves its parent section and document for context.
+
+### Semantic chunking
+
+Documents are split by a semantic chunker that identifies topic boundaries rather than using fixed character counts. Each chunk gets a summary (used for embedding) and retains the raw content (used for BM25 and for returning to the agent).
+
+---
+
+## Guard Layers
+
+Every query path — vector search, BM25 search, keyword subscriptions, and graph expansion — applies a consistent set of access control filters. These are Cypher WHERE clauses, not middleware, so they cannot be bypassed by tool parameter manipulation.
+
+### Layer 1: Soft-delete filter
+
+```
+WHERE node.deletedAt IS NULL
+```
+
+Unconditional. No parameter controls it. Nodes with a `deletedAt` timestamp are excluded from all query paths. Soft-deleted `KnowledgeDocument` nodes cascade the timestamp to all child `Section` and `Chunk` nodes. Grace period before hard deletion: 7 days. Re-ingesting a soft-deleted document (same `attachmentId`) clears `deletedAt` and replaces the hierarchy.
+
+### Layer 2: Scope filter
+
+```
+WHERE (node.scope IS NULL OR node.scope IN $allowedScopes)
+```
+
+When `allowedScopes` is set (e.g., `["public", "shared"]` for public agents), only nodes with a matching `scope` property — or no scope at all (legacy transitional safety) — are returned. When `allowedScopes` is omitted (admin agent), no scope filtering is applied. Scope values: `public`, `shared`, `admin`.
+
+### Layer 3: Per-agent tag filter
+
+```
+WHERE node.agents IS NOT NULL AND $agentSlug IN node.agents
+```
+
+When `agentSlug` is set (public agent queries), only nodes explicitly tagged for that agent are returned. The `agents` property is a string array on each node — a node is visible to an agent only if the agent's slug appears in this array. No implicit "available to all" fallback. This is enforced at the MCP server level via the `AGENT_SLUG` environment variable — tool parameter overrides are rejected when the env var is set.
+
+**Defense in depth:** Both scope and agent filters must pass. An admin-scoped node tagged for a public agent is still invisible to that agent because the scope filter rejects it first.
+
+### Layer 4: Graph expansion enforcement
+
+Related nodes discovered during hop traversal are independently filtered:
+
+```
+WHERE (related.scope IS NULL OR related.scope IN $allowedScopes)
+AND (related.agents IS NULL OR $agentSlug IN related.agents)
+AND related.deletedAt IS NULL
+```
+
+This prevents cross-agent content leakage via graph traversal — a public agent cannot reach admin-scoped nodes by following relationships from a public node. Untagged related nodes (no `agents` property) pass through, allowing shared structural nodes (e.g., `PriceSpecification` linked to a `Service`) to be discoverable.
+
+### Layer 5: Account isolation
+
+```
+WHERE node.accountId = $accountId
+```
+
+Multi-tenancy boundary. Every query is scoped to the requesting account. The `ACCOUNT_ID` environment variable is set at MCP server startup — it is not a tool parameter and cannot be overridden by the agent.
+
+The read filter alone is not sufficient — it correctly *hides* alien-account nodes from every UI but does not prevent them existing. A writer that misresolves `accountId` (literal, undefined, or inferred-from-the-wrong-context) leaks nodes into the graph with no downstream symptom; the read filter then keeps them invisible indefinitely. The write-side doctrine is documented in `.docs/neo4j.md` "Account isolation invariant" — every writer that stamps `n.accountId` must verify the value against `${DATA_ROOT}/accounts/<id>/account.json` before write. The live floor is `writeNodeWithEdges` — every doctrine-primitive write is gated by an `accountId == process.env.ACCOUNT_ID` check (the spawning process validates `ACCOUNT_ID` at boot against the on-disk account set via the `account-enumeration` lib), with `[graph-write] reject reason=invalid-account-id …` as the rejection signal.
+
+**Two boot-time surfaces stamp + validate the env** (added 2026-05-07). The brand systemd unit emits `Environment=ACCOUNT_ID=<uuid>` (resolved by the installer from `INSTALL_DIR/data/accounts/<uuid>/account.json`); the Hono boot path then calls `validateAccountIdEnv` against the on-disk set and emits `[graph-health] account-id-env present=true id=<8> matches-on-disk=true` on success or `[graph-health] account-id-env FATAL reason=<missing|no-on-disk-account|mismatch>` + `process.exit(1)` on failure. No fallback — a misconfigured Pi cannot silently boot.
+
+---
+
+## Query Classification
+
+Before searching, a Haiku classifier decides whether a query needs knowledge retrieval at all. This prevents meta-queries ("hello", "thanks", "continue") from polluting the system prompt with irrelevant search results.
+
+| Property | Admin variant | Public variant |
+|---|---|---|
+| Model | `claude-haiku-4-5` | Same |
+| Timeout | 3 seconds | Same |
+| History window | Last 4 messages (2 user + 2 assistant) | Same |
+| Max tokens | 200 | 120 |
+| Query rewriting | Yes — resolves references from history into concrete search terms | Same |
+| Topic-change detection | Yes — detects shifts with confidence score | No (removed, earlier platform fixes) |
+| Fallback on failure | `search: true` (always search with raw message) | Same |
+
+### Classification output
+
+The classifier returns a JSON object:
+- `search` (boolean) — whether a knowledge search should run
+- `query` (string or null) — a search-optimised rewrite of the user's message, or null to use the raw message
+- `reason` (string) — short explanation of the decision
+
+When `search` is true and `query` is non-null, the rewritten query replaces the raw message for the memory-search call. This is important: the classifier resolves pronouns and references from conversation history into concrete terms, improving retrieval precision.
+
+### Knowledge retrieval gate
+
+The classifier is one input to a broader decision tree that determines whether `memory-search` runs:
+
+| Agent config | Classifier runs? | Search runs? |
+|---|---|---|
+| `liveMemory: true` | No | Always — the admin opted into live knowledge and the classifier cannot know what the graph contains. ~200ms cost is negligible vs. cost of a miss. |
+| `knowledgeBaked: true`, `liveMemory: false` | No | Never — KNOWLEDGE.md is the sole knowledge source. |
+| `knowledgeBaked: false` | Yes | Only if classifier says `search: true` (or classifier fails → fallback to always-search). |
+
+### Observability
+
+Admin: `[admin-query-classifier]` log line with `topicChange`, `topicChangeConfidence`, `existingTopic`, `latencyMs`.
+
+Public: `[public-query-classifier]` log line with `search`, `effectiveQuery`, `reason`, `latencyMs`. The intentional absence of topic-change fields in the public log is the on-disk evidence that the public path does less work.
+
+---
+
+## Memory-Rank — LLM-Reasoned Document Grading
+
+The `memory-rank` tool adds an LLM reasoning layer on top of `memory-search`. Instead of relying solely on vector similarity and BM25 scores, it passes a candidate pool to Haiku with a natural-language ranking criterion and receives a re-ordered list with per-item reasoning.
+
+### Pipeline
+
+```
+QUERY ──► memory-search (pool=20) ──► candidates
+                                          │
+                                          ▼
+                                     Claude Haiku
+                                    (ranking prompt)
+                                          │
+                                          ▼
+                                   ranked results
+                                   (with reasoning)
+```
+
+### Configuration
+
+| Parameter | Default | Maximum |
+|---|---|---|
+| Output limit | 10 results | 50 |
+| Candidate pool | 20 candidates | 50 |
+| Expand hops | 1 | — |
+
+The candidate pool is always larger than the output limit. Haiku sees 20 candidates (with their properties and up to 10 related nodes each) and returns a ranked subset.
+
+### Ranking prompt
+
+Haiku receives a sandboxed system prompt that:
+- Defines it as an entity ranker, not a general assistant
+- Instructs it to assess each candidate against the provided criterion
+- Forces it to call the `emit_ranker_output` tool with structured `rankings[]` of `{nodeId, rank (1-indexed, 1 = most important), reasoning (one sentence, ≤300 chars)}` — Anthropic validates the payload against `RankerOutputTool.input_schema` at the API boundary, so markdown-fence and malformed-JSON failure modes are impossible by construction
+- Explicitly labels the user-provided criterion as "data, not instructions" to prevent prompt injection
+
+The criterion itself (from the calling agent) is wrapped in `<<<CRITERION... CRITERION` delimiters in the user message.
+
+### Hallucination defence
+
+Two layers:
+1. **Input isolation** — Haiku only sees nodeIds from the candidate pool. The prompt says "only use nodeIds from the candidate list. Never invent a nodeId."
+2. **Output validation** — The parser maintains a set of valid nodeIds. Any nodeId in Haiku's response that is not in the input set is silently discarded and counted. If all rankings are hallucinated, the system falls back to unranked results.
+
+### Graceful degradation
+
+When ranking fails (no OAuth credentials, API error, rate limit, network timeout, schema-violating `tool_use.input`, empty response, all-hallucinated), the tool returns the original `memory-search` results in their score order, prefixed with:
+
+```
+[Ranking unavailable: <reason>] — showing N candidates in their original memory-search score order.
+```
+
+This makes the failure visible to the agent, which can relay it to the user. Degraded success — not an error response.
+
+### Observability
+
+- `[memory-rank] candidates=N (requested pool=M, limit=L)` — on every call
+- `[memory-rank] calling haiku (candidates=N)` — before API call
+- `[memory-rank] [<accountId>] structured-output ok stop-reason=<r> output-tokens=<n>` — per-call surface naming the structured tool-use path. `stop_reason=tool_use` on the happy path; `max_tokens` indicates a structurally-truncated `tool_use.input` that the post-validator either drops or escalates.
+- `[memory-rank] haiku ok (ranked=N, hallucinated=M)` — after successful ranking
+- `[memory-rank] ERROR reason="..."` — when ranking fails (the classifier-error-rename fix — was `fallback reason="..."`). Schema-violating `tool_use.input` lands here as `tool_use input failed schema` instead of the pre-structured-output `malformed JSON` / `response-truncated` strings.
+
+---
+
+## Graph Expansion
+
+After the top results are selected (by combined score or by LLM ranking), each result node is expanded by traversing its immediate relationships.
+
+### Traversal mechanics
+
+```cypher
+MATCH (n)-[r]-(related)
+WHERE elementId(n) = $nodeId
+AND related.deletedAt IS NULL
+AND (related.scope IS NULL OR related.scope IN $allowedScopes)
+AND (related.agents IS NULL OR $agentSlug IN related.agents)
+RETURN type(r), direction, labels(related), related
+LIMIT 20
+```
+
+- **Default hop depth:** 1 (immediate relationships only)
+- **Related nodes cap:** 20 per primary result
+- **Direction tracking:** Each relationship is labelled `outgoing` or `incoming`
+- **Scope enforcement:** All guard layers (soft-delete, scope, agent) apply to related nodes
+- **Configurable:** `expandHops: 0` produces compact output (properties only, no related nodes) — useful for listing/inventory queries
+
+### What expansion provides
+
+A `Service` node matched by vector search will have its `PriceSpecification`, `Review` nodes, and parent `LocalBusiness` attached as related nodes. A `Chunk` matched by BM25 will have its parent `Section` and `KnowledgeDocument`. This context enrichment means the agent receives not just the matched node but its immediate neighbourhood in the graph.
+
+---
+
+## Keyword Subscriptions — Reactive Per-Agent Knowledge
+
+Each public agent can subscribe to up to 5 keywords via `knowledgeKeywords` in its `config.json`. These subscriptions make the agent reactive to new graph content matching its topics — content added after the agent was created becomes discoverable without manual tag updates.
+
+### Dual search per keyword
+
+For each subscription keyword, two complementary searches run:
+
+1. **BM25 full-text search** — queries the universal `entity_search` index with the keyword as the search term. Catches content that mentions the keyword in its text across every operator-meaningful label.
+
+2. **Property-based search** — finds nodes whose `keywords` array property contains the subscription keyword (case-insensitive). Catches nodes explicitly tagged with that keyword topic. These matches are boosted to maximum BM25 score (1.0) since they are exact tag matches.
+
+Both searches run **without** the per-agent tag filter (`agentSlug`) — keyword subscriptions are scope-inclusive by design, meaning an agent's subscriptions can discover content not directly tagged for it. The scope filter (`allowedScopes`) still applies as defense in depth — admin-only content remains invisible to public agents regardless of keyword matches.
+
+### Union semantics
+
+Results from keyword subscription searches are merged into the same scored map as the primary vector+BM25 results. Deduplication by `nodeId` with `Math.max` on scores means a node found by both direct search and keyword subscription keeps the highest score from each method.
+
+### Lifecycle
+
+| Mode | How keywords are used |
+|---|---|
+| `liveMemory: true` | Keywords are passed as `KNOWLEDGE_KEYWORDS` env var to the memory MCP server at runtime. Every query includes keyword subscription searches alongside the primary search. |
+| `liveMemory: false` | Keywords are used by the `update-knowledge` admin skill when regenerating KNOWLEDGE.md. The regeneration queries the graph with both direct tags and keyword subscriptions. |
+
+---
+
+## Conversation Search
+
+Separate from the knowledge retrieval pipeline, `conversation-search` provides semantic search over past messages.
+
+- **Index:** `message_embedding` (768-dim cosine HNSW on `Message` nodes)
+- **Scope:** When `SESSION_ID` is set (public agent), results are limited to that conversation. Admin searches all conversations.
+- **Output:** Messages with role, content, timestamp, and relevance score.
+
+This tool is read-only and available to both public and admin agents.
+
+### When conversations are created
+
+`:Conversation` nodes on webchat (admin login, "New conversation" in the burger, a new public visitor) are created lazily. Opening the chat or logging in does not write anything to the graph — {{productName}} only records the conversation once the user sends a second message. This keeps `conversation-search` and the Conversations modal free of one-turn abandoned threads. WhatsApp and Telegram take the opposite posture: every inbound — DM or group, allowed or activation-off, agent-invoked or gated — MERGEs the `:Conversation` and writes a forensic `:Message:WhatsAppMessage` row before any access-control decision. The graph is the durable record of every message the device received, not just the ones the agent replied to. See `.docs/web-chat.md` "Deferred conversation persistence" and `.docs/whatsapp.md` "Session continuity" for the full contract.
+
+Each row in the Conversations modal exposes a `View logs` row-action that opens a popover with three links — **Stream**, **Errors**, **SSE** — each of which targets `/api/admin/logs?type={stream|error|sse}&conversationId={full-id}` in a new tab. The row's 8-char id chip is click-to-copy; hover reveals the full `conversationId` as a tooltip. See `.docs/web-chat.md` "In-chat retrieval" for the route contract and `console.debug` observability.
+
+### Static publish surface — `/sites/*`
+
+{{productName}} hosts a generic per-account static-tree publish surface at `https://public.<brand>/sites/<...>/<file>`. The route serves files from `<accountDir>/sites/<...>` with URL=disk mirroring — operator drops the tree on disk, no upload API. Extended MIME covers HTML/CSS/JS/woff2/fonts on top of images. Path traversal (`..`, encoded `..`, segments failing `SAFE_SEG_RE`) returns 403; symlinks escaping the sites root are rejected via a `realpathSync` re-check. `.html` responses carry `Content-Security-Policy: default-src 'self' https: data:; script-src 'none'` and `Cache-Control: no-cache`; assets are cached for an hour; every response carries `X-Content-Type-Options: nosniff`. Per-account isolation comes from `resolveAccount` — every brand's install sees only its own tree.
+
+**Directory canonicalisation.** A request whose disk target is a directory is `301`'d to the trailing-slash form (query string preserved) before any body is served — RFC 3986 §5.3 base resolution requires the trailing slash so relative refs in the served HTML resolve under the directory, not its parent. After the redirect the route serves `<dir>/index.html` if it exists on disk; otherwise `404`. There is **no** implicit-`index.html` invention for missing paths — the publisher owns canonical URLs. A brochure shipped without `index.html` is reached at `/sites/<slug>/<file>.html`, and the admin skill `publish-site` is the sanctioned surface that moves the extracted tree under `<accountDir>/sites/<slug>/` and emits the canonical path slug. Operator-side: drop a brochure at `<accountDir>/sites/properties/<id>/brochure/output/` and it serves at `<public-host>/sites/properties/<id>/brochure/output/brochure.html` (or `<public-host>/sites/properties/<id>/brochure/output/` if that directory contains an `index.html`). See `.docs/web-chat.md` `/sites/*` route entry for the wire contract and `[sites]` log lines (`serve|redirect-trailing-slash|not-found|path-traversal-rejected|symlink-escape-rejected|no-account`).
+
+**Deterministic public-hostname surface.** The `<public-host>` half of the URL the operator pastes is resolved by the `mcp__admin__public-hostname` MCP tool. It reads `<configDir>/cloudflared/config.yml` (ingress list) then falls back to `<configDir>/alias-domains.json` — the same two files `cloudflared` and `platform/ui/server/index.ts`'s `isPublicHost()` already trust to route. Returns `{hostname, isApex, source}` on hit (`source` is `"cloudflared-config.yml"` or `"alias-domains.json"`), or `{hostname:null, source:null, reason:"no-tunnel"}` on miss. Tiebreak: apex wins over subdomain (single-label, or `www.<apex>` stripped). `publish-site` step 6 calls it after the move and emits the full URL (`https://<hostname><path-slug>`) in the same turn. Graph queries are no longer involved — any earlier graph-backed resolver returned `(none)` on accounts bootstrapped without `cloudflare-task-tracker.ts` writes (laptop Real Agent, manual `cloudflared` setup), the `llm-framing-deterministic` recurrence class. The graph-mcp shim additionally runs a sequential envelope-warning probe on every read response — when Neo4j emits `gql_status` codes matching `^0[12]N5\d$` (e.g. `01N52` "property does not exist"), the shim stitches them into a prefix content block on the response so property-name misses surface to the agent inline instead of returning silent `[]`. Probe failure is best-effort: the upstream response forwards unchanged with `[mcp:graph] probe-error`.
+
+### Cross-tab session rotation
+
+When you click "New conversation" in the chat tab, {{productName}} mints a fresh admin session key on the server and clears the old one. Sibling admin tabs (`/graph`, `/data`) opened in the same browser keep working without re-login: the chat tab broadcasts the new key on a same-origin channel so each sibling tab updates its captured key instantly, and any in-flight admin request that 401s with the rotation-orphan code retries once after re-reading the latest key from per-tab storage. If neither path recovers (browser locked down, second 401 after retry, session expired), the tab shows a single banner — "Your admin session was renewed in another tab. Click to reload." — and one click sends you back through login. No silent 401s; no re-clicking through the same trash icon hoping it sticks. See `.docs/web-chat.md` "Cross-tab rotation contract" for the wire-level `code` taxonomy and observability surfaces.
+
+---
+
+## Context Assembly — How Retrieved Knowledge Reaches the Agent
+
+The final step in the retrieval pipeline is injecting retrieved content into the agent's system prompt. The path depends on agent configuration.
+
+### Public agent paths
+
+The server-side code in `claude-agent.ts` builds the system prompt before making the Anthropic API call. Three paths:
+
+**Path 1 — Baked only** (`knowledgeBaked: true`, `liveMemory: false`):
+- KNOWLEDGE.md content is embedded directly in the system prompt as `<knowledge>...</knowledge>`
+- No runtime search. No classifier. No memory MCP server call.
+- KNOWLEDGE.md quality is the ceiling of the agent's factual knowledge.
+
+**Path 2 — Baked + live** (`knowledgeBaked: true`, `liveMemory: true`):
+- KNOWLEDGE.md is embedded as `<knowledge>...</knowledge>` (static baseline)
+- Runtime search also runs (always, bypassing classifier)
+- Results appended as `<live-memory>...</live-memory>`
+- Agent sees both: curated knowledge file + fresh graph results
+
+**Path 3 — Memory-search only** (`knowledgeBaked: false`):
+- No KNOWLEDGE.md exists (e.g., before onboarding completes)
+- Classifier decides whether to search (or falls back to always-search)
+- Results injected as `<memory>...</memory>`
+
+### KNOWLEDGE.md staleness guard
+
+When both `KNOWLEDGE.md` and `KNOWLEDGE-SUMMARY.md` exist, the server compares modification times. If KNOWLEDGE.md is newer than the summary (summary is stale), the full KNOWLEDGE.md is used. Otherwise, the summary is preferred (smaller token footprint).
+
+### Admin agent path
+
+The admin agent runs via Claude Code CLI, which manages its own system prompt assembly. Knowledge reaches the admin agent through MCP tools — `memory-search` and `memory-rank` are available as tools the agent can call during a turn. The admin agent also receives session context via `loadSessionContext`, which injects:
+
+- Recent review digest (last public chat or review digest `CreativeWork`)
+- Open tasks (priority-ordered, capped)
+- Active review alerts (unsuppressed, last 24 hours, capped at 5)
+
+This is assembled as a `<previous-context>` block in the system prompt on each admin turn.
+
+### fetchMemoryContext — the MCP bridge
+
+For public agents, the server calls the memory MCP server via JSON-RPC over stdin/stdout:
+
+1. Spawn the memory MCP server as a subprocess with environment variables: `ACCOUNT_ID`, `ALLOWED_SCOPES=public,shared`, `AGENT_SLUG`, `KNOWLEDGE_KEYWORDS`, `SESSION_ID`
+2. Send `initialize` + `tools/call` (name: `memory-search`, arguments: `{query, account_id}`)
+3. Read the tool result text
+4. Timeout: 8 seconds. On any failure, returns null — the agent proceeds without memory context.
+
+This subprocess model means each public agent query gets an isolated, short-lived memory server instance with the correct scope constraints baked into its environment.
+
+---
+
+## Output Formatting and Budget
+
+The `memory-search` tool formats results as structured text with labels, properties, scores, and related nodes. An output character budget of 80,000 characters prevents results from exceeding Claude Code's tool result token limit (~100K chars). When results exceed the budget, related nodes are progressively dropped (compact mode) to fit within the limit.
+
+Each result is formatted as:
+```
+[Label1, Label2] (id: nodeId) (score: 0.XXX)
+  property1: value
+  property2: value
+  Related:
+    --[RELATIONSHIP]--> [RelatedLabel] {prop1: val, prop2: val}
+    <--[RELATIONSHIP]-- [RelatedLabel] {prop1: val, prop2: val}
+```
+
+Results are separated by `---` dividers. The `embedding` and `accountId` properties are stripped from output (internal fields, not useful to the agent).
+
+---
+
+## Index Discovery and Schema Evolution
+
+The memory MCP server does not hardcode index names. On first query, it runs `SHOW INDEXES YIELD name, labelsOrTypes, type WHERE type = 'VECTOR'` and builds a label-to-index-name map. This map is cached for the lifetime of the process.
+
+This means:
+- Adding a new vector index in `schema.cypher` makes a new label searchable without code changes
+- The `memory-reindex` tool can backfill embeddings for newly indexed labels
+- Index renames are transparent — the server discovers the current index names at startup
+
+The cache is cleared via `clearIndexCache` after schema changes (e.g., after `memory-reindex` detects new indexes).
+
+---
+
+## Inbound Message Gateway
+
+Every inbound message — regardless of channel (web admin, web public, WhatsApp DM, WhatsApp group) — passes through a centralised screening and classification step before reaching the agent. One Haiku call per message produces:
+
+- **Content screening** — CLEAN / SUSPICIOUS / DISCARD verdict plus a prompt injection flag. DISCARD verdicts on public channels return a polite refusal without invoking the agent. Admin messages receive advisory screening only — flagged in the log but never blocked or modified.
+- **Query rewriting** — retrieval-optimised reformulation of the message for memory-search (public channels only; admin text is unchanged).
+- **Intent classification** — question / instruction / complaint / greeting / follow-up.
+- **Language** — ISO 639-1 code.
+- **Complexity** — simple / complex.
+
+Short messages (under 5 words) skip the Haiku call but still get local pattern matching against the shared prompt injection vocabulary — this prevents short injection payloads from bypassing screening.
+
+On Haiku timeout, API error, or missing API key, the raw message passes through unmodified (graceful degradation). The gateway never blocks the user from reaching the agent due to its own failure.
+
+Gateway results are injected into the agent's system prompt as structured metadata, giving the agent context about the message before it begins processing.
+
+### Diagnostics
+
+Every gateway invocation logs to `server.log` with the `[inbound-gateway]` tag, including channel, verdict, intent, language, complexity, latency, and fallthrough status. Non-clean verdicts get an additional warning log.
+
+To check recent screening activity:
+```
+grep '[inbound-gateway]' server.log | tail -20
+```
+
+---
+
+## Tool Eagerness — eager-load vs deferred
+
+The Claude Code SDK marks every MCP tool as **deferred** by default. The model cannot invoke a deferred tool until it has first paid a `ToolSearch` round-trip to load the schema — one extra turn per unique schema. Built-in SDK tools (`Read`, `Write`, `Edit`, `Bash`, `Glob`, `Grep`, `Agent`, `WebSearch`, `WebFetch`) stay eager. There is no count threshold; the gate is per-tool.
+
+The SDK's per-tool override is `_meta["anthropic/alwaysLoad"]: true` on each MCP tool's `tools/list` entry. Two surfaces apply it:
+
+1. **In-process plugins.** Every admin-eager tool is registered via `eagerTool(server, name, description, inputSchema, handler)` from `platform/lib/mcp-eager/` instead of `server.tool(...)`. The helper calls `server.registerTool` with the `_meta` flag set.
+2. **Upstream graph proxy.** The upstream Python `mcp-neo4j-cypher` server has no `_meta` channel, so `platform/lib/graph-mcp/src/index.ts` intercepts every `tools/list` response on the wire and injects `_meta["anthropic/alwaysLoad"]: true` into each tool entry. The `[graph-mcp] tools/list eager-flagged count=<N>` stderr line confirms the injection fired.
+
+**Curation rule.** Every MCP tool the admin agent calls routinely should be eager. The migrated set is `ADMIN_CORE_TOOLS` in `platform/ui/app/lib/claude-agent/spawn-env.ts` — when a tool sits on that list it must be registered via `eagerTool` (or arrive through the graph-mcp interceptor). Admin-skill / specialist / public-agent tools that are NOT in `ADMIN_CORE_TOOLS` stay on `server.tool()` today; they pay the ToolSearch tax only when their caller invokes them. The allow-list is the source of truth for the eager set, not necessarily the complete set of tools admin can ever call — skills that route into Bash + plugin-toggle / agent-config / onboarding tools are an existing gap to widen the allow-list against, separately from the eager-load mechanism.
+
+**Observability.** Spawn-time emit: `[tool-surface] session=<convId> permission_allowed=N eager_intent=E eager_set_size=T`. Turn-end emit: `[admin-agent] turn-end ... toolsearch=N toolsearch_unique=U`. A non-zero `toolsearch` on a fresh turn for an eager-intended tool means a plugin reverted to `server.tool()` — fix at the plugin's MCP registration site, not the allow-list.
+
+## Tool Call Audit Trail
+
+Every tool invocation by the admin agent produces a durable `ToolCall` node in the knowledge graph, linked to the `Conversation` that triggered it. This covers all admin agent tool calls — the full history of what the agent did, when, and in what context.
+
+Each ToolCall record contains:
+
+| Field | Description |
+|-------|-------------|
+| toolName | The MCP tool that was invoked (e.g. `memory-search`, `workflow-execute`) |
+| pluginName | The plugin that owns the tool |
+| input | Truncated JSON of the tool's input arguments |
+| output | Truncated response text |
+| isError | Whether the tool call resulted in an error |
+| startedAt / completedAt | Timestamps for the invocation |
+| conversationId | Links back to the originating conversation |
+
+Records persist indefinitely and are queryable by the admin agent. Ask {{productName}} "what tools ran in the last session?" or "show me all tool calls from today" to review the audit trail.
+
+Workflow-dispatched tool calls are tracked separately via `StepResult` nodes (part of the workflow execution system) and are not duplicated as ToolCall nodes.
+
+### Diagnostics
+
+Tool call persistence logs to `server.log` with the `[persist]` tag:
+```
+grep '[persist] tool-call persisted' server.log | tail -10
+```
+
+Each log entry includes the tool name and a truncated conversation ID for correlation.
+
+## Process provenance — durable actions emit Tasks
+
+Every durable action — onboarding, cloudflare tunnel-login, brand publish, future deterministic flows — emits a `:Task {kind:"<flow>"}` node carrying the action's lifecycle and a `:PRODUCED` edge to every entity the action created. This makes the graph traversable from the originating Conversation to every entity created during it via `(c)<-[:RAISED_DURING]-(t:Task)-[:PRODUCED]->(e)` — answering "what did this turn produce" in one Cypher hop.
+
+The doctrine is enforced at the storage primitive: writes to `:Person`, `:UserProfile`, `:AdminUser`, `:Organization`, `:LocalBusiness`, `:CloudflareTunnel`, or `:CloudflareHostname` MUST include at least one inbound `:PRODUCED` edge whose source is one of `:Task`, `:Conversation`, or `:Message`. Subtype labels like `:AdminConversation`, `:UserMessage`, `:AssistantMessage`, `:AdminMessage` qualify because the gate checks the full `labels()` array. Bootstrap writes (PIN-setup, schema migrations, lazy first-session UserProfile creation) are exempt via `createdBy.agent === 'system'`.
+
+Two surfaces emit the lifecycle: agent-driven actions call `task-create`/`task-update`/`task-complete` over MCP (`task-create` accepts `kind`, the canonical `inputsProvided` call-shape record, `inputs` + `inputSchema` for the operator-meaningful form payload, and `raisedDuringConversationKey` to resolve the `RAISED_DURING` edge). Shell-driven actions wrap their script invocation in [platform/ui/app/lib/cloudflare-task-tracker.ts](../../../ui/app/lib/cloudflare-task-tracker.ts) (cloudflare is the first; installer / brand-publish / OAuth-login deferred). Both surfaces emit the same `[task] action-start|step|done` log lines so operators can grep one channel uniformly. Both also call the central `redactSecrets` primitive ([platform/lib/task-secrets/](../../../lib/task-secrets/)) to strip schema-tagged secret keys before persisting `inputs.<field>` props on the Task — see `.docs/neo4j.md § Audit Task input contract` for the contract that replaces per-kind allow-lists.
+
+Two surfaces feed the gate. (1) **Workflow path:** `memory-write` accepts an optional `producedByTaskId` parameter. When set, an inbound `:PRODUCED` edge from that Task is composed into the write's relationships before the gate runs — the typical agent-side pattern is to call `task-create` at the start of an autonomous flow, capture `taskId`, and pass it as `producedByTaskId` on every subsequent `memory-write` for a gated label. The gate verifies Task and write share the same `accountId`; mismatch is rejected loud. (2) **Direct-ask path:** the admin server resolves the active `:AdminConversation` elementId at MCP spawn time and stamps it as `CONVERSATION_NODE_ID` in the env. The `contact-create` and `memory-write` wrappers call `injectConversationProvenance` (exported from [`@maxy/graph-write`](../../../lib/graph-write/src/conversation-provenance.ts)) which verifies `accountId` parity and prepends the synthetic `:PRODUCED` edge — no agent-visible schema field changes. `memory-write` uses the env-stamp only as a fallback when `producedByTaskId` is unset; `contact-create` has no `producedByTaskId` parameter today and relies on the env-stamp alone.
+
+Operator audit cyphers:
+- "What entities did this conversation's actions produce?" — `MATCH (c:AdminConversation {conversationId:$id})<-[:RAISED_DURING]-(t:Task)-[:PRODUCED]->(e) RETURN labels(e), e.name, t.kind, t.status`
+- "What cloudflare resources did this onboarding produce?" — `MATCH (t:Task {kind:'cloudflare-tunnel-login', status:'completed'})-[:PRODUCED]->(r) RETURN t.taskId, r.tunnelId, r.hostnameValue ORDER BY t.completedAt DESC`
+
+See `.docs/neo4j.md § Process provenance doctrine` for the full enforcement contract, observability surface, and out-of-scope deferrals.
+
+## Context compaction
+
+When an admin turn crosses 75% of the model's context window, {{productName}} runs a silent compaction turn that asks the agent to call the `session-compact` MCP tool with a structured briefing (what you asked for, what was done, decisions made, work-in-progress, things you've shared about yourself). The briefing is written to Neo4j; the next admin turn injects it back into the system prompt, so continuity survives across the compaction boundary without re-sending the full transcript.
+
+The compaction runs against a transient one-shot pool entry separate from the long-lived admin Query. Operator-visible side effects:
+- Compaction logs land in `claude-agent-compaction-stream-YYYY-MM-DD.log` alongside the main stream log. Look for `[compaction-start]`, `[compaction-summary-captured]`, `[compaction-failed]`, `[compaction-timeout]`, `[compaction-crashed]`, or `[compaction-spawn-error]` to triage. Subprocess stderr is captured inline as `[subproc-stderr] <line>` — there is no longer a separate `claude-agent-compaction-stderr-…log` file.
+- The one-shot pool entry's lifecycle is greppable as `[client-cold-create] reason=compaction-one-shot …` paired with `[client-evict] reason=compaction-one-shot …`, distinguishable from the regular admin pool's lifecycle tags.