insika 0.3.0 → 0.8.0

data/docs/KNOWLEDGE.md

@@ -0,0 +1,290 @@
+---
+title: Knowledge
+parent: Improve
+nav_order: 4
+permalink: /knowledge/
+---
+
+# Knowledge — learning from conversations
+
+An agent is amnesiac in exactly one dimension. It has history (the session),
+memory (facts about one customer, written by the `remember` tool), skills
+(procedures a human curated), and tool traces — but nothing turns a finished
+conversation into knowledge the agent can use on someone else's turn. Every
+conversation teaches a store things — which product customers actually ask
+about, which objection keeps coming up, which CEP maps to which distribution
+center — and today all of it is write-once trace data nobody reads back.
+
+Knowledge is that loop. After a turn completes, the engine can extract durable
+**concepts** from it — facts, procedures, policies, objections — and persist
+them for the agent (never a customer, never a session) to build on.
+
+**What ships today: extraction, consolidation, retrieval, export, and a
+Studio page.** The engine writes what it learns, decides whether a repeat
+sighting confirms, merges with, or contradicts what it already knew,
+retrieves the concepts relevant to a turn's message back into the prompt,
+and an operator can see, edit and resolve all of it in the Studio. Only the
+optional FTS5 index remains, deferred with a measured trigger — see
+[What's not here yet](#whats-not-here-yet).
+
+## The concept format
+
+One concept is one record — a markdown document with a YAML frontmatter
+block, the same shape a `SKILL.md` uses:
+
+```markdown
+---
+name: cep-sudeste-cd-campinas
+description: CEPs 13xxx-13999 ship from the Campinas DC, 1-2 business days.
+type: fact                     # fact | entity | procedure | policy | objection
+provenance: observed           # policy (curated) | observed (learned from conversations)
+confidence: 0.6                # 0..1, evidence-weighted
+sources: ["sess_8f3c"]         # session ids, never message content
+occurrences: 1
+created_at: 2026-08-24T18:02:11Z
+updated_at: 2026-08-24T18:02:11Z
+---
+
+Orders to CEP range 13000-13999 are fulfilled by the Campinas DC. Quoted
+delivery is 1-2 business days. Related: [[frete-gratis-acima-199]].
+```
+
+The model only ever writes `name`, `description`, `type` and `body`. Every
+other field — `provenance`, `confidence`, `sources`, `occurrences`, the
+timestamps — is stamped by the engine; an extraction answer that tries to
+supply one of those is rejected outright (the same discipline Facts applies
+to a model-authored scope). `[[links]]` inside the body are plain text stored
+as-is; retrieval resolves them lazily by name at read time (below) — that one
+hop is the entire "graph," never a stored structure of its own.
+
+**`provenance` is not decoration.** Everything the extractor writes is
+`provenance: observed` — a claim learned from what people said in
+conversations, not official policy. A promise an agent made ("we'll get back
+to you in 48h") is observed practice, not a guarantee, and the field exists so
+nothing downstream states it to a customer as a commitment. `provenance:
+policy` is reserved for a concept a human authored or promoted by hand — the
+engine never sets it.
+
+## Enabling it — the `knowledge:` block
+
+Knowledge is pack data on the agent, exactly like `distill:` or `harvest:` —
+absent = the feature is off for that agent, byte-identical engine:
+
+```ruby
+agent = Insika.agent("store-support") do
+  instructions "…"
+  knowledge extract: true, retrieve: true,
+            types: %w[fact policy objection],  # what the extractor may emit
+            top_k: 5                            # concepts injected per turn (default 5)
+  # prompt: "<what counts as a concept for THIS store>" — the pack-authored
+  #   half; absent = the engine's generic prompt. `model:` (absent = the
+  #   platform utility_model) names the extractor AND the consolidator.
+  # index: "scan" (default, the only one built) | "fts5" (accepted, falls
+  #   back to "scan" — the optional SQLite index isn't built yet).
+end
+```
+
+`extract` and `retrieve` are independent switches: an agent can learn without
+recalling (write-only, inspected by hand), or recall without learning
+(curate every concept by hand in the Studio, `provenance: policy`).
+
+## The write path
+
+After a turn completes, off the turn's critical path (the customer already has
+the answer), the engine sends the turn's transcript slice to the platform
+`utility_model` and asks for a JSON array of candidate concepts. The answer is
+schema-validated, filtered against the agent's configured `types`, and any key
+the model should not be writing is dropped and counted, never trusted. The
+survivors are redacted for PII, stamped with provenance and a first-sighting
+confidence, and written to the store — one `:knowledge_learned` event per
+concept (name, type, agent — never content).
+
+**Durability, honestly.** Extraction is best-effort: a crash between a turn's
+terminal and the write loses that turn's concepts, not the conversation (which
+is durable) and not a previously learned concept. The recovery path is a
+re-scan, not a queue:
+
+```
+insika knowledge:backfill --agent store-support --since 2026-08-01T00:00:00Z
+```
+
+replays the agent's stored sessions through the same extractor a live turn
+uses.
+
+## Consolidation — what happens on a repeat sighting
+
+Writing a concept name that already exists is never a blind overwrite. The
+engine compares the new sighting against what is on record and picks one of
+three outcomes:
+
+- **Same claim, reworded or reconfirmed** — a cheap, deterministic check
+  (no model call): occurrences go up, the new session id joins `sources`,
+  confidence climbs (`min(0.95, 0.5 + 0.1 × distinct sources)` — more
+  independent sightings, more confidence, never certainty). The body itself
+  is untouched, so an operator's edit is never silently discarded by a
+  repeat sighting.
+- **Related claim** — the two bodies say compatible things that combine into
+  one coherent statement. A second model call (the only place this feature
+  spends a second call, and only when a name already exists) merges them;
+  the result bumps occurrences/sources/confidence the same way a same-claim
+  sighting does.
+- **Contradicting claim** — the two bodies say genuinely different things.
+  **Never merged, never silently overwritten.** The new claim is appended
+  under a `## Contradiction` heading, confidence drops to a flat `0.4`, and
+  a `:knowledge_conflict` event fires. A human resolves it in the Studio by
+  editing the concept directly — there is no separate "resolve" action,
+  because resolving IS editing the markdown to say what's actually true.
+
+When no model is configured for consolidation (or its answer is unusable),
+the engine defaults to the conservative outcome — contradicting. A concept
+this feature is unsure about becomes a human's problem, never a guess that
+looks confident and might be wrong.
+
+## The Studio page
+
+`/studio/knowledge` — single-agent-scoped like Harvest (`?agent=`), not
+shared like Skills, because a concept only ever belongs to one store. A
+drill-down list (name, type, confidence, occurrences, updated_at) with a
+conflict filter (`?status=conflict`) and the same CodeMirror markdown editor
+Skills uses, version history and restore, and delete. Editing the raw
+markdown is also how an operator promotes `provenance: observed` to `policy`
+— there is no separate "promote" button, the field is just another line in
+the file.
+
+## Retrieval — what reaches a turn's prompt
+
+With `knowledge.retrieve` on, every turn the engine searches the agent's
+concepts for the ones relevant to the customer's message (pure term overlap —
+no embeddings, no network call) and injects the top few as a level-1
+`<knowledge>` block, the same progressive-disclosure shape Skills uses:
+
+```
+<knowledge>
+  <concept name="cep-13-campinas" confidence="0.60" provenance="observed">CEPs 13xxx-13999 ship from the Campinas DC, 1-2 business days.</concept>
+</knowledge>
+
+If the customer's question needs more than the summary above, call
+`load_knowledge("name")` FIRST — before any other lookup for that topic.
+This is learned from past conversations, not official policy: never state
+a `provenance="observed"` concept to the customer as a guarantee.
+```
+
+Only `name`/`description`/`confidence`/`provenance` are shown — the same
+"summary now, body on demand" shape a skill's level-1 list uses. A `load_knowledge`
+tool (outside `tools_allow`, wired only when `retrieve` is on — same as
+`load_skill`) fetches one concept's complete body. Calling it fires
+`:knowledge_retrieved` — this, not the injection itself, is what the
+adoption metric tracks (see [the honest limits](#the-honest-limits)): a
+concept sitting unread in the prompt taught the agent nothing.
+
+**One hop through `[[links]]`.** A matched concept's body may reference
+`[[other-concept-name]]`; retrieval resolves those names against the store
+and injects them too (capped at `top_k` again), so a concept that names its
+neighbor arrives with it. No transitive walk — one level, deliberately, the
+same reasoning a skill's declared `companions:` uses.
+
+**Where it sits, and what gets cut first.** `<knowledge>` sits at priority 77
+— below curated skills (80), above a single conversation's memory (75): a
+human's playbook always outranks what the engine inferred, and what the
+engine inferred outranks one customer's chat facts. Never pinned; under
+budget pressure it goes before skills but survives longer than memory,
+briefing, history and request context — see [Context](CONTEXT.md).
+
+## External knowledge over MCP
+
+A native knowledge base is not the only shape this can take. Mounting a
+third-party knowledge or memory server as MCP tools on an agent is a
+supported pattern, complementary to the native loop above — a deployment can
+run both: this loop learns concepts from ITS OWN traffic automatically, while
+an MCP-mounted server can serve as a synthesis/consultant tool an agent calls
+explicitly.
+
+The one adoption lesson worth carrying over regardless of which shape is used:
+a model does not reliably call a recall/lookup tool just because it exists,
+even when it clearly should. A polite "use this when relevant" instruction
+measured close to zero calls under pressure; what worked was an explicit,
+ordered rule naming the competing tools directly — "call the recall tool
+FIRST, before any data lookup, on topics X" — not a sentiment about when it's
+a good idea. Tool *adoption* is a prompt-ordering problem, not a tool-quality
+one, and it is worth measuring (calls per conversation), not assuming.
+
+## The honest limits
+
+- **Best-effort extraction, re-scan recovery.** Same discipline as Facts: no
+  queue, no exactly-once claim. A concept is re-derivable from the session
+  transcript, so a missed extraction is recoverable, never lost.
+- **Consolidation trusts the same model that extracts.** The "related vs.
+  contradicting" call is a model judgment, not a proof — the conservative
+  default (contradicting, when unsure) bounds the failure mode to "a human
+  looks at it," never "two different claims silently became one wrong one."
+- **Precision is a forge audit.** The engine guarantees the gates (schema,
+  key-stripping, PII redaction, type allowlist); it cannot guarantee the
+  model's judgment about what is worth remembering. That is tuned per store,
+  the same way a Harvest or Facts prompt is.
+- **Retrieval quality is not adoption.** Injecting the right concept proves
+  nothing if the model never reads it — measure `:knowledge_retrieved`
+  (retrieval calls per conversation), not just whether the block appeared.
+  The explicit, ordered instruction in the block exists because a softer
+  "when to use" wording measured close to zero calls in practice.
+- **Term overlap, not understanding.** `Index::Scan` matches words, not
+  meaning — a concept phrased very differently from the customer's words
+  will not surface even if it answers the question. No embeddings by
+  design: revisit only with evidence that retrieval, not
+  extraction, is the bottleneck.
+
+## Index::Scan's performance, measured not assumed
+
+`Index::Scan` keeps a read cache per instance (the context provider holds
+one for the whole process, never rebuilt per turn): a concept's YAML
+frontmatter is parsed once and reused until that concept's own record
+`updated_at` changes, so a write invalidates itself for free. Reproduce
+these numbers with:
+
+```bash
+bundle exec ruby scripts/bench_knowledge_index.rb
+```
+
+| concepts/agent | p50 | p95 |
+|---|---|---|
+| 50 | 0.28 ms | 0.47 ms |
+| 200 | 1.17 ms | 1.56 ms |
+| 1000 | 6.4 ms | 8.0 ms |
+| 5000 | 35.6 ms | 41.3 ms |
+
+At the scale this feature targets for the first year — hundreds of concepts
+per agent — a warm-cache search costs a bit over a millisecond, close to
+the engine's own documented per-turn overhead (see [Benchmark](BENCHMARK.md)).
+Past roughly a thousand concepts it becomes a real, measurable cost again —
+that specific, numeric point is the trigger for building `Index::FTS5`, not
+a guess made in advance.
+
+## Export
+
+```
+insika knowledge:export --agent store-support --out ./export [--tenant loja-a]
+```
+
+Writes one `<name>.md` per concept — the storage format IS the export
+format, so this is a dump, not a converter (same discipline as Facts/
+Harvest's own append-only records): each file is the concept's markdown,
+byte for byte, directly consumable by okf-gem (`OKF::Bundle`) or graphify.
+Re-running it is safe — nothing here is lossy, so there is no `--force` to
+reason about.
+
+`--format graphml` writes one combined `knowledge.graphml` instead: a node
+per concept (`name`/`type`/`description`/`confidence`/`provenance` as node
+data) and an edge per `[[link]]` that resolves to another concept in the
+same export — a link to a concept outside the scope is dropped, never a
+dangling edge. Hand-built, hand-escaped XML (no new dependency), directly
+openable in Gephi, yEd, or graphify.
+
+## What's not here yet
+
+- **The optional FTS5 index** — `knowledge.index: "fts5"` is accepted but
+  falls back to `Index::Scan`. Deliberately not built yet: `Scan` was
+  measured (above), not assumed, and it comfortably meets this feature's
+  target scale. A deployment whose concept count is heading past ~1000 per
+  agent is the evidence that would justify building the SQLite
+  `MATCH`/`bm25()` adapter — not before.
+- **Decay** — recency is a ranking tiebreak today; a real confidence decay
+  curve is a later, evidence-driven addition, not a default.

data/docs/LOADTEST.md

@@ -1,7 +1,7 @@
 ---
 title: Load test
-parent: Operate & prove it
-nav_order: 3
+parent: Operate
+nav_order: 4
 permalink: /loadtest/
 ---
 
@@ -67,7 +67,7 @@ frame that carries it.
 
 ```bash
 INSIKA_URL=http://localhost:9292 \
-OPENCLAW_GATEWAY_TOKEN=xxx \
+INSIKA_GATEWAY_TOKEN=xxx \
 bundle exec ruby scripts/loadtest.rb \
   --agents demo,my-store --concurrency 16 --iterations 3 \
   --message "hi, how are you?"
@@ -80,7 +80,7 @@ Runs against a local server **or** a remote one (e.g. Railway) — just point
 
 | Flag | Default | Meaning |
 |------|---------|---------|
-| `--agents a,b,c` | `demo` | comma-separated agent ids (mapped to `model: openclaw:<agent>`) |
+| `--agents a,b,c` | `demo` | comma-separated agent ids (mapped to `model: insika:<agent>`) |
 | `--concurrency N` | `8` | concurrent turns per wave |
 | `--iterations N` | `1` | number of waves per agent |
 | `--message TEXT` | greeting | user message sent every turn |
@@ -95,14 +95,14 @@ Runs against a local server **or** a remote one (e.g. Railway) — just point
 | Env | Default | Meaning |
 |-----|---------|---------|
 | `INSIKA_URL` | `http://localhost:9292` | base URL of the engine |
-| `OPENCLAW_GATEWAY_TOKEN` | falls back to `ADMIN_TOKEN`, then `local-demo` | Bearer for `/v1/responses` |
+| `INSIKA_GATEWAY_TOKEN` | falls back to `ADMIN_TOKEN`, then `local-demo` | Bearer for `/v1/responses` |
 | `DEEPSEEK_API_KEY` | — | must be configured **on the server** for real turns (not read by the client) |
 
 Use `--dry-run` to sanity-check your flags/URL/token before firing real traffic
 (and to confirm the request body without needing a running server):
 
 ```bash
-INSIKA_URL=http://localhost:9292 OPENCLAW_GATEWAY_TOKEN=xxx \
+INSIKA_URL=http://localhost:9292 INSIKA_GATEWAY_TOKEN=xxx \
   bundle exec ruby scripts/loadtest.rb --agents demo --concurrency 16 --dry-run
 ```
 
@@ -133,7 +133,7 @@ DEEPSEEK_API_KEY=sk-... ./scripts/loadtest-local.sh [WORKERS] [CONCURRENCY]
 | Env | Default | Meaning |
 |-----|---------|---------|
 | `DEEPSEEK_API_KEY` | — (required) | real turns hit the provider; also auto-sourced from `.env.local` |
-| `OPENCLAW_GATEWAY_TOKEN` | falls back to `ADMIN_TOKEN`, then `local-demo` | Bearer for the sweep |
+| `INSIKA_GATEWAY_TOKEN` | falls back to `ADMIN_TOKEN`, then `local-demo` | Bearer for the sweep |
 | `PORT` | `9299` | bind port for the local Falcon |
 | `AGENT` | `demo` | agent id to load |
 
@@ -158,30 +158,18 @@ the engine, once at the gateway (its `/v1/responses` speaks the same protocol).
 Keep `--agents`, `--concurrency`, `--iterations` and `--message` identical, and use
 matching agents on both sides. Compare the printed TTFB/total/cache/error lines.
 
-### 4b. Reuse OpenClaw's `loadtest-gateway.mjs` unmodified
+### 4b. Comparing against an OpenClaw gateway
 
-`loadtest.rb` is the Ruby port of OpenClaw's `loadtest-gateway.mjs`. You do **not**
-need to change that script to point it at the engine — because the engine is a
-drop-in for the gateway, you only change **where it points**:
+The engine speaks its own wire names (`model: insika:<agent>`, header
+`X-Insika-Agent`), so OpenClaw's `loadtest-gateway.mjs` cannot be pointed at it
+unmodified. For a shadow comparison, run `loadtest.rb` (section 4a) against each
+side with identical `--agents`, `--concurrency`, `--iterations` and `--message`,
+and diff the reports. What you still need in hand:
 
-```bash
-# In the OpenClaw checkout, run its gateway loadtest against the HARNESS:
-OPENCLAW_GATEWAY_URL=http://localhost:9292 \
-OPENCLAW_GATEWAY_TOKEN=<same bearer the engine accepts> \
-node scripts/loadtest-gateway.mjs --agents demo --concurrency 16 --iterations 3
-```
-
-Then run the exact same command with `OPENCLAW_GATEWAY_URL` pointing at the real
-gateway, and diff the two reports. This is the shadow comparison the pilot needs.
-
-**What the operator must have in hand** (this repo does not vendor OpenClaw):
-
-- The OpenClaw checkout containing `scripts/loadtest-gateway.mjs` and Node installed.
-- A **bearer token accepted by both** sides. For the engine that is
-  `OPENCLAW_GATEWAY_TOKEN` (see DEPLOY.md); point the gateway run at its own token.
-- **The same agent id provisioned on both** sides (e.g. `demo`) so `model:
-  openclaw:<agent>` resolves on each. On the engine, provision via
-  `scripts/import_pack.rb`.
+- A bearer token accepted by each side — `INSIKA_GATEWAY_TOKEN` for the engine
+  (see DEPLOY.md), the gateway's own token for the gateway run.
+- **The same agent id provisioned on both** sides (e.g. `demo`). On the engine,
+  provision via `scripts/import_pack.rb`.
 - The **same provider** (or an equivalent-latency one) behind each, otherwise you
   are comparing providers, not engines.
 - Both endpoints reachable from where you run the client, warmed up (hit `/up` on

data/docs/MEDIA.md

@@ -0,0 +1,128 @@
+---
+title: Media
+parent: Integrate
+nav_order: 3
+permalink: /media/
+---
+
+# Media
+
+The engine transports media, it never means it. Photos, voice notes and documents
+travel through the message contract as additive content parts; what a picture
+*means* — a fitting room, a product mockup — stays a skill on top.
+
+## In — the message contract
+
+The engine transports media, it never means it. The message accepts additive
+**content parts** alongside the text — voice notes and photos travel, and any
+skill (a fitting room, an image QA) stays a consumer layer on top:
+
+```bash
+curl -X POST /v1/messages?stream=false -H "Authorization: Bearer $TOKEN" \
+  -d '{ "agent": "store-support", "session_id": "chat-7",
+        "message": "", "parts": [
+          { "type": "audio", "url": "https://cdn.example.com/voz.ogg" },
+          { "type": "image", "url": "https://cdn.example.com/sofa.jpg" },
+          { "type": "document", "url": "https://cdn.example.com/receita.pdf" }
+        ] }'
+```
+
+- **Audio** is transcribed (RubyLLM STT; model via `INSIKA_STT_MODEL`) and the
+  text enters the turn marked `source: "voice"` on the terminal event — the
+  consumer's signal the person spoke. A consumer that transcribes itself can
+  send the text with `"source": "voice"` directly. A domain vocabulary hint
+  (product names, brand terms) rides the transcription as `prompt:` —
+  per-agent `stt_prompt` (the DSL setter, or the Studio config form) beats the
+  deployment-wide `INSIKA_STT_PROMPT` env, which beats nothing. OPERATOR
+  config, never customer input.
+- **Images** attach to the model's ask (vision); the provider bills them and
+  the usage flows like any ask. The first image URL is also
+  `{{ctx.image_url}}` for data tools — photo analysis outside the prompt, the
+  tool's own egress applying when it fetches.
+- **Documents** (a prescription, a recipe, an invoice — most often a PDF)
+  attach the same way images do; the first document URL is
+  `{{ctx.document_url}}`. A model without document support fails the ask at
+  `:ruby_llm` with the provider's own error — the transport does not preflight
+  capability. Capped at 10 MB (`MAX_DOCUMENT_BYTES` — "a prescription, not an
+  archive"), separately from the 5 MB image cap.
+- Media URLs (audio, image AND document) are fetched by the engine through
+  the same egress guard (a private/metadata target is refused — SSRF) and a
+  size ceiling per kind (1 MB audio, 5 MB image, 10 MB document: the bytes
+  land in this process). A refused, oversized or unreadable part fails the turn
+  loudly at the `:media` stage, never a silent drop.
+- **Media alone is a turn.** A voice note with no caption is `parts` and an
+  empty `message` — the transcription becomes the message at the `:media`
+  stage. A media message never joins another turn (`collect`/`steer` move text
+  only, and the parts would be left behind), and a transcription that comes
+  back empty fails the turn instead of asking the model about nothing.
+- **Parts are contract at the edge** — a malformed part (unknown type, an
+  image/audio/document without `url`, a text without `text`) is a 422 before
+  dispatch on `/v1/messages` and `/v1/responses`. `document` is an ADDITIVE
+  part type (the compatibility rule in [the /v1 API](API.md) — no
+  `Insika-Version` bump needed).
+- `/v1/responses` accepts the OpenAI multimodal shape: `input` as an array of
+  text/image/audio/document parts.
+
+### Image editing
+
+`generate_image` (below) doesn't only generate — it can EDIT an existing
+image, using `RubyLLM.paint`'s `with:`/`mask:`. The tool exposes
+`source_image_urls` (an array — up to 4) and `mask_url`; when the model omits
+`source_image_urls` AND the turn carries an inbound photo, that photo is
+edited by default — no URL round-trip needed for "edit the photo the customer
+just sent". Explicit URLs always win over the default. A text-to-image call
+(no sources at all) is byte-identical to before this feature existed.
+
+What the edit MEANS — a virtual try-on, a product mockup on the customer's
+wall — is the calling skill's business; the tool only transports the bytes.
+Not every image model can edit (`dall-e-3` cannot); a call against a
+non-editing model fails at the provider, surfaced verbatim.
+
+## Out — generated media
+
+The turn can **produce** an image or a voice clip — but only when both sides of
+the gate agree, because nothing leaks by default. The agent declares it may
+generate media (`outputs` on the profile), and the **channel** declares it can
+receive it (`channel.capabilities` on the request):
+
+```bash
+curl -X POST /v1/responses -H "Authorization: Bearer $TOKEN" -d '{
+  "model": "insika:store-support", "user": "chat-7",
+  "input": "manda a foto do sofá da promoção",
+  "channel": { "capabilities": ["image_output", "audio_output"] }
+}'
+```
+
+```ruby
+agent = Insika.agent("store-support") do
+  instructions "…"
+  outputs image: { model: "gpt-image-1", size: "1024x1024" },   # the AGENT's half
+          tts:   { model: "tts-1", voice: "alloy" }
+end
+```
+
+- **Both gates** must pass for the model to even see the `generate_image` /
+  `tts` tools: the agent opted in (`outputs`) and the request declared the
+  matching capability (`image_output` / `audio_output` — an unknown value is a
+  422, never a silent ignore). The "abstraction admits only what leaks" rule.
+- **The media rides the envelope, never the answer text.** The terminal event
+  and the `/v1/responses` completed frame carry an additive `output_parts`
+  array — `{ type: "image", mime_type:, base64:, model: }` /
+  `{ type: "audio", mime_type:, base64:, model: }`. The model's prose stays
+  the `:content` answer; the channel consumes the bytes next to it.
+- **Generation is billed and counted.** Image tokens join the turn's usage
+  (like any ask). The speech API reports no token counts, so a TTS call adds
+  an honest `usage.media` counter and the part carries the `model` for
+  consumer-side pricing.
+- **Seams, not magic.** The generator is injectable per kind (specs stub it);
+  the defaults are lazy: images via RubyLLM (paint), speech via a thin POST to
+  the OpenAI-compatible `/audio/speech` endpoint using the same provider
+  config the chat uses — RubyLLM as of 1.16.0 has no speech API. A generated
+  part over 8 MB refuses loudly, never silently truncates.
+- **Not here:** what the generated image *means* — a fitting room, a product
+  mockup — is a skill on top. The engine transports bytes and cost.
+
+## See also
+
+- [The /v1 API](API.md) — the full message contract.
+- [Channels](CHANNELS.md) — how WhatsApp and the web widget carry these parts.

data/docs/OBSERVABILITY.md

@@ -1,6 +1,6 @@
 ---
 title: Observability
-parent: Operate & prove it
+parent: Operate
 nav_order: 1
 permalink: /observability/
 ---
@@ -25,7 +25,9 @@ authoring writes (`:golden_written`, `:agent_file_written`, …), queue bookkeep
 channel delivery (`:channel_delivered` — see [Channels](CHANNELS.md))
 travel the same stream and are **ignored** by the bridge: they open no span and
 touch no instrument, because they are not part of a turn's latency or cost. Any
-other subscriber still sees them.
+other subscriber still sees them. The one mid-turn event the bridge does consume
+is `:tool_loop_intervened` — it feeds the `insika.tool.loop_intervened` counter
+(no span: the intervention is a fact about the turn, not a timed operation).
 
 They are worth subscribing to even so, because each is the ONLY record of
 something that left no task of its own behind:
@@ -36,23 +38,28 @@ something that left no task of its own behind:
 | `:turn_steered` | `task_id`, `count`, `total` | a message arrived mid-run and was appended to the turn in flight |
 | `:turn_steer_released` | `task_id`, `released_as`, `count` | the run could not absorb it, so it became the turn `released_as` |
 | `:turn_interrupted` | `task_id`, `replaced_by` | the turn was abandoned mid-run, and which turn replaced it |
-| `:turn_stuck` | `task_id`, `agent`, `reason`, `message` | the agent declared it could not proceed (`signal_stuck`, WS5) — the deterministic signal a consumer escalates on |
+| `:turn_stuck` | `task_id`, `agent`, `reason`, `message` | the agent declared it could not proceed (`signal_stuck`) — the deterministic signal a consumer escalates on |
 | `:channel_delivered` | `channel`, `outbox_id`, `status`, `attempts`, `error` | the answer reached the platform (or did not) — the turn completing says nothing about that |
-| `:delivery_failed` | `channel`, `outbox_id`, `status`, `attempts`, `error` | a delivery exhausted its bounded retries — the alert face of the row above (WS6) |
-| `:budget_warning` | `agent`, `tenant`, `window`, `spent`, `cap` | a calendar budget crossed its threshold (`alert_at` or a soft cap) — once per window (WS2) |
-| `:breaker_open` | `agent`, `ref`, `tenant` | the reliability circuit breaker tripped for a `(tenant, provider/model)` — further turns fail fast until the cooldown (WS3/WS6) |
-| `:provider_failure` | `agent`, `ref`, `error`, `kind` | one attempt against `ref` failed and spent a retry — emitted with or without a circuit breaker (WS3) |
-| `:provider_fallback` | `agent`, `from`, `to`, `error`, `kind` | the turn ROTATED mid-flight to the next node of the fallback chain, and the error that caused it (WS3) |
-| `:ttft` | `task_id`, `session_id`, `ttft_ms` | the provider's time-to-first-token on the streaming envelope — only under `INSIKA_TURN_TIMING`, once per turn (WS6) |
-| `:distillation_completed` | `session_ref`, `agent`, `proposals`, `dropped{}`, `deduped`, `cost{}` | a session's traffic was distilled into N proposals (RFC-0034) — counts and ids only, never a fact value |
+| `:delivery_failed` | `channel`, `outbox_id`, `status`, `attempts`, `error` | a delivery exhausted its bounded retries — the alert face of the row above |
+| `:budget_warning` | `agent`, `tenant`, `window`, `spent`, `cap` | a calendar budget crossed its threshold (`alert_at` or a soft cap) — once per window |
+| `:breaker_open` | `agent`, `ref`, `tenant` | the reliability circuit breaker tripped for a `(tenant, provider/model)` — further turns fail fast until the cooldown |
+| `:provider_failure` | `agent`, `ref`, `error`, `kind` | one attempt against `ref` failed and spent a retry — emitted with or without a circuit breaker |
+| `:provider_fallback` | `agent`, `from`, `to`, `error`, `kind` | the turn ROTATED mid-flight to the next node of the fallback chain, and the error that caused it |
+| `:ttft` | `task_id`, `session_id`, `ttft_ms` | the provider's time-to-first-token on the streaming envelope — only under `INSIKA_TURN_TIMING`, once per turn |
+| `:distillation_completed` | `session_ref`, `agent`, `proposals`, `dropped{}`, `deduped`, `cost{}` | a session's traffic was distilled into N proposals — counts and ids only, never a fact value |
 | `:proposal_approved` | `proposal_id`, `status`, `operator` | a human approved a distilled fact on the Facts page — written to memory via CAS |
 | `:proposal_rejected` | `proposal_id`, `status`, `operator` | a human rejected it (the reason is on the proposal record, not in the event) |
 | `:proposal_dismissed` | `proposal_id`, `status`, `operator` | a human dismissed it — the tuple is latched, never proposed again |
 | `:proposal_stale` | `proposal_id`, `status`, `operator` | the CAS lost: the fact moved after distillation — re-presented with both values on the wiki, never silently overwritten |
-| `:harvest_mined` | `agent`, `run_id`, `candidates`, `rejected{}`, `cost{}` | a mining pass finished (RFC-0035) — counts and rule ids only, never a skill body |
+| `:harvest_mined` | `agent`, `run_id`, `candidates`, `rejected{}`, `cost{}` | a mining pass finished — counts and rule ids only, never a skill body |
 | `:harvest_gated` | `run_id`, `candidate_id`, `agent`, `eval_passed`, `conversion_passed`, `reason` | the double gate's verdict on one candidate — ids and verdicts only |
 | `:skill_promoted` | `agent`, `skill`, `candidate_id`, `snapshot_ref`, `promotion_ref`, `approver` | a human approved a mined skill — it is live for the store, with the snapshot for rollback |
 | `:skill_rolled_back` | `snapshot_ref`, `skill`, `agent`, `operator` | the snapshot was restored — the promotion row carries the `rolled_back_at` stamp |
+| `:knowledge_learned` | `name`, `type`, `agent` | a turn's transcript taught the agent a new concept, or a related sighting merged into an existing one — name and type only, never the body |
+| `:knowledge_conflict` | `name`, `agent` | a repeat sighting contradicted what's on record — never merged, appended under a heading, needs a human in the Studio |
+| `:knowledge_deleted` | `name`, `agent` | an operator removed a concept from the Studio |
+| `:knowledge_retrieved` | `name`, `agent` | the model called `load_knowledge` — the adoption signal (retrieval calls per conversation), not the `<knowledge>` injection itself |
+| `:knowledge_backfilled` | `agent`, `sessions`, `concepts`, `conflicts`, `dropped{}` | the recovery re-scan (`insika knowledge:backfill`) finished replaying an agent's stored sessions |
 
 `delivery_failed` and `breaker_open` are the two the operator config is pointed at
 (`alerts.webhook` on the profile): each only fires when something durable did
@@ -140,12 +147,32 @@ knows its outcome.
 | `insika.cost` | counter | `{USD}` | the turn's model is priced (see below) |
 | `insika.tool.calls` | counter | `{call}` | a tool call completes |
 | `insika.tool.duration` | histogram | `s` | a `tool_call`/`tool_result` pair completes |
+| `insika.cache.hit_rate` | histogram | `%` | a turn reported billed prompt tokens (see below) |
+| `insika.tool.loop_intervened` | counter | `{intervention}` | the loop detector delivered its one-shot warning |
+| `insika.context.compacted` | counter | `{compaction}` | an in-session compaction was persisted (RFC-0044) |
 
 `insika.tool.duration` is deliberately **not** recorded for data-tools: those are a
 single point-in-time event, so there is no measured duration to report. A tool left
 open by a mid-turn failure is not counted as a completed call either — its span is
 closed, but a failed call must not inflate the success histogram.
 
+`insika.cache.hit_rate` is the same arithmetic the Studio's per-agent series uses:
+cache **reads** over the whole **billed** prompt (fresh input + cache reads + cache
+writes), always in `[0,100]`. A turn with no billed prompt tokens records nothing —
+absence is not a 0% hit. It is recorded per turn on the terminal event, so it
+carries the turn labels (`insika.agent`, `insika.model`, …); the token-counter
+recipe below still works and answers the fleet-wide version of the question.
+
+`insika.tool.loop_intervened` counts deliveries of the loop detector's one
+warning per turn (see [Agents](AGENTS.md) `max_tool_repeat`), labelled with
+`insika.tool`. It counts interventions, not repeats: a turn contributes at most 1.
+
+`insika.context.compacted` counts persisted in-session compactions
+(`:context_compacted` — see [Context](CONTEXT.md)), labelled with
+`insika.agent` and `insika.model` (the summarizer's model, not the turn's). It
+fires post-turn, after the turn span already closed, so it deliberately rides
+its own labels rather than the open-turn set.
+
 ## Attribute reference
 
 The same names are used on spans and on metrics. **Metrics carry a deliberate
@@ -262,7 +289,14 @@ climbing while `input` stays flat.
 
 **Cache hit ratio**
 `insika.tokens` filtered to `insika.token.type="cached"` over the same counter
-filtered to `input`. This is the number that moves your bill.
+filtered to `input`. This is the number that moves your bill. For the per-turn
+distribution (does every turn hit, or do fleet averages hide cold agents?), chart
+`insika.cache.hit_rate` — p50 by `insika.agent`; a healthy agent sits near 100.
+
+**Loop interventions**
+`insika.tool.loop_intervened`, rate, grouped by `insika.agent` and `insika.tool`.
+Any sustained non-zero rate means one tool keeps being retried with identical
+arguments — fix the tool's contract or the prompt, not the detector.
 
 **Spend per tenant**
 `insika.cost`, rate (or `increase` over a billing window), grouped by