insika 0.2.0 → 0.7.0

data/docs/KNOWLEDGE.md

@@ -0,0 +1,290 @@
+---
+title: Knowledge
+parent: Improve
+nav_order: 4
+permalink: /knowledge/
+---
+
+# Knowledge — learning from conversations
+
+An agent is amnesiac in exactly one dimension. It has history (the session),
+memory (facts about one customer, written by the `remember` tool), skills
+(procedures a human curated), and tool traces — but nothing turns a finished
+conversation into knowledge the agent can use on someone else's turn. Every
+conversation teaches a store things — which product customers actually ask
+about, which objection keeps coming up, which CEP maps to which distribution
+center — and today all of it is write-once trace data nobody reads back.
+
+Knowledge is that loop. After a turn completes, the engine can extract durable
+**concepts** from it — facts, procedures, policies, objections — and persist
+them for the agent (never a customer, never a session) to build on.
+
+**What ships today: extraction, consolidation, retrieval, export, and a
+Studio page.** The engine writes what it learns, decides whether a repeat
+sighting confirms, merges with, or contradicts what it already knew,
+retrieves the concepts relevant to a turn's message back into the prompt,
+and an operator can see, edit and resolve all of it in the Studio. Only the
+optional FTS5 index remains, deferred with a measured trigger — see
+[What's not here yet](#whats-not-here-yet).
+
+## The concept format
+
+One concept is one record — a markdown document with a YAML frontmatter
+block, the same shape a `SKILL.md` uses:
+
+```markdown
+---
+name: cep-sudeste-cd-campinas
+description: CEPs 13xxx-13999 ship from the Campinas DC, 1-2 business days.
+type: fact                     # fact | entity | procedure | policy | objection
+provenance: observed           # policy (curated) | observed (learned from conversations)
+confidence: 0.6                # 0..1, evidence-weighted
+sources: ["sess_8f3c"]         # session ids, never message content
+occurrences: 1
+created_at: 2026-08-24T18:02:11Z
+updated_at: 2026-08-24T18:02:11Z
+---
+
+Orders to CEP range 13000-13999 are fulfilled by the Campinas DC. Quoted
+delivery is 1-2 business days. Related: [[frete-gratis-acima-199]].
+```
+
+The model only ever writes `name`, `description`, `type` and `body`. Every
+other field — `provenance`, `confidence`, `sources`, `occurrences`, the
+timestamps — is stamped by the engine; an extraction answer that tries to
+supply one of those is rejected outright (the same discipline Facts applies
+to a model-authored scope). `[[links]]` inside the body are plain text stored
+as-is; retrieval resolves them lazily by name at read time (below) — that one
+hop is the entire "graph," never a stored structure of its own.
+
+**`provenance` is not decoration.** Everything the extractor writes is
+`provenance: observed` — a claim learned from what people said in
+conversations, not official policy. A promise an agent made ("we'll get back
+to you in 48h") is observed practice, not a guarantee, and the field exists so
+nothing downstream states it to a customer as a commitment. `provenance:
+policy` is reserved for a concept a human authored or promoted by hand — the
+engine never sets it.
+
+## Enabling it — the `knowledge:` block
+
+Knowledge is pack data on the agent, exactly like `distill:` or `harvest:` —
+absent = the feature is off for that agent, byte-identical engine:
+
+```ruby
+agent = Insika.agent("store-support") do
+  instructions "…"
+  knowledge extract: true, retrieve: true,
+            types: %w[fact policy objection],  # what the extractor may emit
+            top_k: 5                            # concepts injected per turn (default 5)
+  # prompt: "<what counts as a concept for THIS store>" — the pack-authored
+  #   half; absent = the engine's generic prompt. `model:` (absent = the
+  #   platform utility_model) names the extractor AND the consolidator.
+  # index: "scan" (default, the only one built) | "fts5" (accepted, falls
+  #   back to "scan" — the optional SQLite index isn't built yet).
+end
+```
+
+`extract` and `retrieve` are independent switches: an agent can learn without
+recalling (write-only, inspected by hand), or recall without learning
+(curate every concept by hand in the Studio, `provenance: policy`).
+
+## The write path
+
+After a turn completes, off the turn's critical path (the customer already has
+the answer), the engine sends the turn's transcript slice to the platform
+`utility_model` and asks for a JSON array of candidate concepts. The answer is
+schema-validated, filtered against the agent's configured `types`, and any key
+the model should not be writing is dropped and counted, never trusted. The
+survivors are redacted for PII, stamped with provenance and a first-sighting
+confidence, and written to the store — one `:knowledge_learned` event per
+concept (name, type, agent — never content).
+
+**Durability, honestly.** Extraction is best-effort: a crash between a turn's
+terminal and the write loses that turn's concepts, not the conversation (which
+is durable) and not a previously learned concept. The recovery path is a
+re-scan, not a queue:
+
+```
+insika knowledge:backfill --agent store-support --since 2026-08-01T00:00:00Z
+```
+
+replays the agent's stored sessions through the same extractor a live turn
+uses.
+
+## Consolidation — what happens on a repeat sighting
+
+Writing a concept name that already exists is never a blind overwrite. The
+engine compares the new sighting against what is on record and picks one of
+three outcomes:
+
+- **Same claim, reworded or reconfirmed** — a cheap, deterministic check
+  (no model call): occurrences go up, the new session id joins `sources`,
+  confidence climbs (`min(0.95, 0.5 + 0.1 × distinct sources)` — more
+  independent sightings, more confidence, never certainty). The body itself
+  is untouched, so an operator's edit is never silently discarded by a
+  repeat sighting.
+- **Related claim** — the two bodies say compatible things that combine into
+  one coherent statement. A second model call (the only place this feature
+  spends a second call, and only when a name already exists) merges them;
+  the result bumps occurrences/sources/confidence the same way a same-claim
+  sighting does.
+- **Contradicting claim** — the two bodies say genuinely different things.
+  **Never merged, never silently overwritten.** The new claim is appended
+  under a `## Contradiction` heading, confidence drops to a flat `0.4`, and
+  a `:knowledge_conflict` event fires. A human resolves it in the Studio by
+  editing the concept directly — there is no separate "resolve" action,
+  because resolving IS editing the markdown to say what's actually true.
+
+When no model is configured for consolidation (or its answer is unusable),
+the engine defaults to the conservative outcome — contradicting. A concept
+this feature is unsure about becomes a human's problem, never a guess that
+looks confident and might be wrong.
+
+## The Studio page
+
+`/studio/knowledge` — single-agent-scoped like Harvest (`?agent=`), not
+shared like Skills, because a concept only ever belongs to one store. A
+drill-down list (name, type, confidence, occurrences, updated_at) with a
+conflict filter (`?status=conflict`) and the same CodeMirror markdown editor
+Skills uses, version history and restore, and delete. Editing the raw
+markdown is also how an operator promotes `provenance: observed` to `policy`
+— there is no separate "promote" button, the field is just another line in
+the file.
+
+## Retrieval — what reaches a turn's prompt
+
+With `knowledge.retrieve` on, every turn the engine searches the agent's
+concepts for the ones relevant to the customer's message (pure term overlap —
+no embeddings, no network call) and injects the top few as a level-1
+`<knowledge>` block, the same progressive-disclosure shape Skills uses:
+
+```
+<knowledge>
+  <concept name="cep-13-campinas" confidence="0.60" provenance="observed">CEPs 13xxx-13999 ship from the Campinas DC, 1-2 business days.</concept>
+</knowledge>
+
+If the customer's question needs more than the summary above, call
+`load_knowledge("name")` FIRST — before any other lookup for that topic.
+This is learned from past conversations, not official policy: never state
+a `provenance="observed"` concept to the customer as a guarantee.
+```
+
+Only `name`/`description`/`confidence`/`provenance` are shown — the same
+"summary now, body on demand" shape a skill's level-1 list uses. A `load_knowledge`
+tool (outside `tools_allow`, wired only when `retrieve` is on — same as
+`load_skill`) fetches one concept's complete body. Calling it fires
+`:knowledge_retrieved` — this, not the injection itself, is what the
+adoption metric tracks (see [the honest limits](#the-honest-limits)): a
+concept sitting unread in the prompt taught the agent nothing.
+
+**One hop through `[[links]]`.** A matched concept's body may reference
+`[[other-concept-name]]`; retrieval resolves those names against the store
+and injects them too (capped at `top_k` again), so a concept that names its
+neighbor arrives with it. No transitive walk — one level, deliberately, the
+same reasoning a skill's declared `companions:` uses.
+
+**Where it sits, and what gets cut first.** `<knowledge>` sits at priority 77
+— below curated skills (80), above a single conversation's memory (75): a
+human's playbook always outranks what the engine inferred, and what the
+engine inferred outranks one customer's chat facts. Never pinned; under
+budget pressure it goes before skills but survives longer than memory,
+briefing, history and request context — see [Context](CONTEXT.md).
+
+## External knowledge over MCP
+
+A native knowledge base is not the only shape this can take. Mounting a
+third-party knowledge or memory server as MCP tools on an agent is a
+supported pattern, complementary to the native loop above — a deployment can
+run both: this loop learns concepts from ITS OWN traffic automatically, while
+an MCP-mounted server can serve as a synthesis/consultant tool an agent calls
+explicitly.
+
+The one adoption lesson worth carrying over regardless of which shape is used:
+a model does not reliably call a recall/lookup tool just because it exists,
+even when it clearly should. A polite "use this when relevant" instruction
+measured close to zero calls under pressure; what worked was an explicit,
+ordered rule naming the competing tools directly — "call the recall tool
+FIRST, before any data lookup, on topics X" — not a sentiment about when it's
+a good idea. Tool *adoption* is a prompt-ordering problem, not a tool-quality
+one, and it is worth measuring (calls per conversation), not assuming.
+
+## The honest limits
+
+- **Best-effort extraction, re-scan recovery.** Same discipline as Facts: no
+  queue, no exactly-once claim. A concept is re-derivable from the session
+  transcript, so a missed extraction is recoverable, never lost.
+- **Consolidation trusts the same model that extracts.** The "related vs.
+  contradicting" call is a model judgment, not a proof — the conservative
+  default (contradicting, when unsure) bounds the failure mode to "a human
+  looks at it," never "two different claims silently became one wrong one."
+- **Precision is a forge audit.** The engine guarantees the gates (schema,
+  key-stripping, PII redaction, type allowlist); it cannot guarantee the
+  model's judgment about what is worth remembering. That is tuned per store,
+  the same way a Harvest or Facts prompt is.
+- **Retrieval quality is not adoption.** Injecting the right concept proves
+  nothing if the model never reads it — measure `:knowledge_retrieved`
+  (retrieval calls per conversation), not just whether the block appeared.
+  The explicit, ordered instruction in the block exists because a softer
+  "when to use" wording measured close to zero calls in practice.
+- **Term overlap, not understanding.** `Index::Scan` matches words, not
+  meaning — a concept phrased very differently from the customer's words
+  will not surface even if it answers the question. No embeddings by
+  design: revisit only with evidence that retrieval, not
+  extraction, is the bottleneck.
+
+## Index::Scan's performance, measured not assumed
+
+`Index::Scan` keeps a read cache per instance (the context provider holds
+one for the whole process, never rebuilt per turn): a concept's YAML
+frontmatter is parsed once and reused until that concept's own record
+`updated_at` changes, so a write invalidates itself for free. Reproduce
+these numbers with:
+
+```bash
+bundle exec ruby scripts/bench_knowledge_index.rb
+```
+
+| concepts/agent | p50 | p95 |
+|---|---|---|
+| 50 | 0.28 ms | 0.47 ms |
+| 200 | 1.17 ms | 1.56 ms |
+| 1000 | 6.4 ms | 8.0 ms |
+| 5000 | 35.6 ms | 41.3 ms |
+
+At the scale this feature targets for the first year — hundreds of concepts
+per agent — a warm-cache search costs a bit over a millisecond, close to
+the engine's own documented per-turn overhead (see [Benchmark](BENCHMARK.md)).
+Past roughly a thousand concepts it becomes a real, measurable cost again —
+that specific, numeric point is the trigger for building `Index::FTS5`, not
+a guess made in advance.
+
+## Export
+
+```
+insika knowledge:export --agent store-support --out ./export [--tenant loja-a]
+```
+
+Writes one `<name>.md` per concept — the storage format IS the export
+format, so this is a dump, not a converter (same discipline as Facts/
+Harvest's own append-only records): each file is the concept's markdown,
+byte for byte, directly consumable by okf-gem (`OKF::Bundle`) or graphify.
+Re-running it is safe — nothing here is lossy, so there is no `--force` to
+reason about.
+
+`--format graphml` writes one combined `knowledge.graphml` instead: a node
+per concept (`name`/`type`/`description`/`confidence`/`provenance` as node
+data) and an edge per `[[link]]` that resolves to another concept in the
+same export — a link to a concept outside the scope is dropped, never a
+dangling edge. Hand-built, hand-escaped XML (no new dependency), directly
+openable in Gephi, yEd, or graphify.
+
+## What's not here yet
+
+- **The optional FTS5 index** — `knowledge.index: "fts5"` is accepted but
+  falls back to `Index::Scan`. Deliberately not built yet: `Scan` was
+  measured (above), not assumed, and it comfortably meets this feature's
+  target scale. A deployment whose concept count is heading past ~1000 per
+  agent is the evidence that would justify building the SQLite
+  `MATCH`/`bm25()` adapter — not before.
+- **Decay** — recency is a ranking tiebreak today; a real confidence decay
+  curve is a later, evidence-driven addition, not a default.

data/docs/LOADTEST.md

@@ -1,7 +1,7 @@
 ---
 title: Load test
-parent: Operate & prove it
-nav_order: 3
+parent: Operate
+nav_order: 4
 permalink: /loadtest/
 ---
 
@@ -16,7 +16,7 @@ The whole point: the engine exposes `POST /v1/responses` as an **SSE drop-in** o
 the OpenClaw gateway. Same contract → the same load tools work against either side,
 so you can measure the engine you are about to ship against the gateway it replaces.
 
-There are three scripts, each answering a different question:
+There are four scripts, each answering a different question:
 
 | Script | Question it answers | Needs a provider? |
 |--------|---------------------|-------------------|
@@ -24,8 +24,13 @@ There are three scripts, each answering a different question:
 | `scripts/loadtest.rb` | End-to-end: TTFB/total/tokens/cache/error against `/v1/responses` | Yes |
 | `scripts/loadtest-local.sh` | Single-proc baseline vs N-worker multi-proc on one box | Yes |
 | `scripts/loadtest_session.rb` | A full multi-message session (CEP, searches, FAQ) under C concurrent sessions — direct to the engine (`--surface engine`, stream vs steer) or through the consumer's real ingress (`--surface web`, the consumer's widget API) | Yes |
+| `insika soak` | Does the deploy degrade over 72 h of steady load? | Yes |
 
-All three take `--help` / `-h`.
+The first four take `--help` / `-h`; the soak is a shipped command (`insika soak --help`) rather
+than a repo script. Bursts and uptime are different questions: a wave driver
+measures a burst, and the soak's arrival process measures *degradation over
+uptime* — the load-test table above deliberately stops where
+[Soak](SOAK.md) begins.
 
 ---
 
@@ -64,7 +69,7 @@ frame that carries it.
 INSIKA_URL=http://localhost:9292 \
 OPENCLAW_GATEWAY_TOKEN=xxx \
 bundle exec ruby scripts/loadtest.rb \
-  --agents bia,my-store --concurrency 16 --iterations 3 \
+  --agents demo,my-store --concurrency 16 --iterations 3 \
   --message "hi, how are you?"
 ```
 
@@ -75,7 +80,7 @@ Runs against a local server **or** a remote one (e.g. Railway) — just point
 
 | Flag | Default | Meaning |
 |------|---------|---------|
-| `--agents a,b,c` | `bia` | comma-separated agent ids (mapped to `model: openclaw:<agent>`) |
+| `--agents a,b,c` | `demo` | comma-separated agent ids (mapped to `model: openclaw:<agent>`) |
 | `--concurrency N` | `8` | concurrent turns per wave |
 | `--iterations N` | `1` | number of waves per agent |
 | `--message TEXT` | greeting | user message sent every turn |
@@ -98,7 +103,7 @@ Use `--dry-run` to sanity-check your flags/URL/token before firing real traffic
 
 ```bash
 INSIKA_URL=http://localhost:9292 OPENCLAW_GATEWAY_TOKEN=xxx \
-  bundle exec ruby scripts/loadtest.rb --agents bia --concurrency 16 --dry-run
+  bundle exec ruby scripts/loadtest.rb --agents demo --concurrency 16 --dry-run
 ```
 
 ### `--same-user` and the cache
@@ -130,7 +135,7 @@ DEEPSEEK_API_KEY=sk-... ./scripts/loadtest-local.sh [WORKERS] [CONCURRENCY]
 | `DEEPSEEK_API_KEY` | — (required) | real turns hit the provider; also auto-sourced from `.env.local` |
 | `OPENCLAW_GATEWAY_TOKEN` | falls back to `ADMIN_TOKEN`, then `local-demo` | Bearer for the sweep |
 | `PORT` | `9299` | bind port for the local Falcon |
-| `AGENT` | `bia` | agent id to load |
+| `AGENT` | `demo` | agent id to load |
 
 The final block prints the lock counts; the expected reading is `0` for both:
 
@@ -163,7 +168,7 @@ drop-in for the gateway, you only change **where it points**:
 # In the OpenClaw checkout, run its gateway loadtest against the HARNESS:
 OPENCLAW_GATEWAY_URL=http://localhost:9292 \
 OPENCLAW_GATEWAY_TOKEN=<same bearer the engine accepts> \
-node scripts/loadtest-gateway.mjs --agents bia --concurrency 16 --iterations 3
+node scripts/loadtest-gateway.mjs --agents demo --concurrency 16 --iterations 3
 ```
 
 Then run the exact same command with `OPENCLAW_GATEWAY_URL` pointing at the real
@@ -174,7 +179,7 @@ gateway, and diff the two reports. This is the shadow comparison the pilot needs
 - The OpenClaw checkout containing `scripts/loadtest-gateway.mjs` and Node installed.
 - A **bearer token accepted by both** sides. For the engine that is
   `OPENCLAW_GATEWAY_TOKEN` (see DEPLOY.md); point the gateway run at its own token.
-- **The same agent id provisioned on both** sides (e.g. `bia`) so `model:
+- **The same agent id provisioned on both** sides (e.g. `demo`) so `model:
   openclaw:<agent>` resolves on each. On the engine, provision via
   `scripts/import_pack.rb`.
 - The **same provider** (or an equivalent-latency one) behind each, otherwise you
@@ -224,6 +229,7 @@ Work top-down and **measure before assuming** — avoid premature topology optim
 | 4 | Insika vs gateway, identical knobs | §4 (either method) | Is the engine at parity with the engine it replaces before cut-over? |
 | 5 | Cold vs hot conversation (cache) | `loadtest.rb` with/without `--same-user 1` | Expected steady-state cost/latency once conversations warm up. |
 | 6 | Remote (Railway) vs local | `loadtest.rb` with `INSIKA_URL` remote | Network/deploy overhead of the real environment. |
+| 7 | Degradation over uptime (72 h) | `insika soak` (see [Soak](SOAK.md)) | The cut argument: does latency or memory grow with uptime? Run it last — it needs the topology settled first. |
 
 **Reaching for horizontal scale is only justified after 1–3 show the single box is
 the limit.** If it is, the paths are: sharding-by-tenant +

data/docs/MEDIA.md

@@ -0,0 +1,128 @@
+---
+title: Media
+parent: Integrate
+nav_order: 3
+permalink: /media/
+---
+
+# Media
+
+The engine transports media, it never means it. Photos, voice notes and documents
+travel through the message contract as additive content parts; what a picture
+*means* — a fitting room, a product mockup — stays a skill on top.
+
+## In — the message contract
+
+The engine transports media, it never means it. The message accepts additive
+**content parts** alongside the text — voice notes and photos travel, and any
+skill (a fitting room, an image QA) stays a consumer layer on top:
+
+```bash
+curl -X POST /v1/messages?stream=false -H "Authorization: Bearer $TOKEN" \
+  -d '{ "agent": "store-support", "session_id": "chat-7",
+        "message": "", "parts": [
+          { "type": "audio", "url": "https://cdn.example.com/voz.ogg" },
+          { "type": "image", "url": "https://cdn.example.com/sofa.jpg" },
+          { "type": "document", "url": "https://cdn.example.com/receita.pdf" }
+        ] }'
+```
+
+- **Audio** is transcribed (RubyLLM STT; model via `INSIKA_STT_MODEL`) and the
+  text enters the turn marked `source: "voice"` on the terminal event — the
+  consumer's signal the person spoke. A consumer that transcribes itself can
+  send the text with `"source": "voice"` directly. A domain vocabulary hint
+  (product names, brand terms) rides the transcription as `prompt:` —
+  per-agent `stt_prompt` (the DSL setter, or the Studio config form) beats the
+  deployment-wide `INSIKA_STT_PROMPT` env, which beats nothing. OPERATOR
+  config, never customer input.
+- **Images** attach to the model's ask (vision); the provider bills them and
+  the usage flows like any ask. The first image URL is also
+  `{{ctx.image_url}}` for data tools — photo analysis outside the prompt, the
+  tool's own egress applying when it fetches.
+- **Documents** (a prescription, a recipe, an invoice — most often a PDF)
+  attach the same way images do; the first document URL is
+  `{{ctx.document_url}}`. A model without document support fails the ask at
+  `:ruby_llm` with the provider's own error — the transport does not preflight
+  capability. Capped at 10 MB (`MAX_DOCUMENT_BYTES` — "a prescription, not an
+  archive"), separately from the 5 MB image cap.
+- Media URLs (audio, image AND document) are fetched by the engine through
+  the same egress guard (a private/metadata target is refused — SSRF) and a
+  size ceiling per kind (1 MB audio, 5 MB image, 10 MB document: the bytes
+  land in this process). A refused, oversized or unreadable part fails the turn
+  loudly at the `:media` stage, never a silent drop.
+- **Media alone is a turn.** A voice note with no caption is `parts` and an
+  empty `message` — the transcription becomes the message at the `:media`
+  stage. A media message never joins another turn (`collect`/`steer` move text
+  only, and the parts would be left behind), and a transcription that comes
+  back empty fails the turn instead of asking the model about nothing.
+- **Parts are contract at the edge** — a malformed part (unknown type, an
+  image/audio/document without `url`, a text without `text`) is a 422 before
+  dispatch on `/v1/messages` and `/v1/responses`. `document` is an ADDITIVE
+  part type (the compatibility rule in [the /v1 API](API.md) — no
+  `Insika-Version` bump needed).
+- `/v1/responses` accepts the OpenAI multimodal shape: `input` as an array of
+  text/image/audio/document parts.
+
+### Image editing
+
+`generate_image` (below) doesn't only generate — it can EDIT an existing
+image, using `RubyLLM.paint`'s `with:`/`mask:`. The tool exposes
+`source_image_urls` (an array — up to 4) and `mask_url`; when the model omits
+`source_image_urls` AND the turn carries an inbound photo, that photo is
+edited by default — no URL round-trip needed for "edit the photo the customer
+just sent". Explicit URLs always win over the default. A text-to-image call
+(no sources at all) is byte-identical to before this feature existed.
+
+What the edit MEANS — a virtual try-on, a product mockup on the customer's
+wall — is the calling skill's business; the tool only transports the bytes.
+Not every image model can edit (`dall-e-3` cannot); a call against a
+non-editing model fails at the provider, surfaced verbatim.
+
+## Out — generated media
+
+The turn can **produce** an image or a voice clip — but only when both sides of
+the gate agree, because nothing leaks by default. The agent declares it may
+generate media (`outputs` on the profile), and the **channel** declares it can
+receive it (`channel.capabilities` on the request):
+
+```bash
+curl -X POST /v1/responses -H "Authorization: Bearer $TOKEN" -d '{
+  "model": "openclaw:store-support", "user": "chat-7",
+  "input": "manda a foto do sofá da promoção",
+  "channel": { "capabilities": ["image_output", "audio_output"] }
+}'
+```
+
+```ruby
+agent = Insika.agent("store-support") do
+  instructions "…"
+  outputs image: { model: "gpt-image-1", size: "1024x1024" },   # the AGENT's half
+          tts:   { model: "tts-1", voice: "alloy" }
+end
+```
+
+- **Both gates** must pass for the model to even see the `generate_image` /
+  `tts` tools: the agent opted in (`outputs`) and the request declared the
+  matching capability (`image_output` / `audio_output` — an unknown value is a
+  422, never a silent ignore). The "abstraction admits only what leaks" rule.
+- **The media rides the envelope, never the answer text.** The terminal event
+  and the `/v1/responses` completed frame carry an additive `output_parts`
+  array — `{ type: "image", mime_type:, base64:, model: }` /
+  `{ type: "audio", mime_type:, base64:, model: }`. The model's prose stays
+  the `:content` answer; the channel consumes the bytes next to it.
+- **Generation is billed and counted.** Image tokens join the turn's usage
+  (like any ask). The speech API reports no token counts, so a TTS call adds
+  an honest `usage.media` counter and the part carries the `model` for
+  consumer-side pricing.
+- **Seams, not magic.** The generator is injectable per kind (specs stub it);
+  the defaults are lazy: images via RubyLLM (paint), speech via a thin POST to
+  the OpenAI-compatible `/audio/speech` endpoint using the same provider
+  config the chat uses — RubyLLM as of 1.16.0 has no speech API. A generated
+  part over 8 MB refuses loudly, never silently truncates.
+- **Not here:** what the generated image *means* — a fitting room, a product
+  mockup — is a skill on top. The engine transports bytes and cost.
+
+## See also
+
+- [The /v1 API](API.md) — the full message contract.
+- [Channels](CHANNELS.md) — how WhatsApp and the web widget carry these parts.

data/docs/OBSERVABILITY.md

@@ -1,6 +1,6 @@
 ---
 title: Observability
-parent: Operate & prove it
+parent: Operate
 nav_order: 1
 permalink: /observability/
 ---
@@ -36,12 +36,28 @@ something that left no task of its own behind:
 | `:turn_steered` | `task_id`, `count`, `total` | a message arrived mid-run and was appended to the turn in flight |
 | `:turn_steer_released` | `task_id`, `released_as`, `count` | the run could not absorb it, so it became the turn `released_as` |
 | `:turn_interrupted` | `task_id`, `replaced_by` | the turn was abandoned mid-run, and which turn replaced it |
-| `:turn_stuck` | `task_id`, `agent`, `reason`, `message` | the agent declared it could not proceed (`signal_stuck`, WS5) — the deterministic signal a consumer escalates on |
+| `:turn_stuck` | `task_id`, `agent`, `reason`, `message` | the agent declared it could not proceed (`signal_stuck`) — the deterministic signal a consumer escalates on |
 | `:channel_delivered` | `channel`, `outbox_id`, `status`, `attempts`, `error` | the answer reached the platform (or did not) — the turn completing says nothing about that |
-| `:delivery_failed` | `channel`, `outbox_id`, `status`, `attempts`, `error` | a delivery exhausted its bounded retries — the alert face of the row above (WS6) |
-| `:budget_warning` | `agent`, `tenant`, `window`, `spent`, `cap` | a calendar budget crossed its threshold (`alert_at` or a soft cap) — once per window (WS2) |
-| `:breaker_open` | `agent`, `ref`, `tenant` | the reliability circuit breaker tripped for a `(tenant, provider/model)` — further turns fail fast until the cooldown (WS3/WS6) |
-| `:ttft` | `task_id`, `session_id`, `ttft_ms` | the provider's time-to-first-token on the streaming envelope — only under `INSIKA_TURN_TIMING`, once per turn (WS6) |
+| `:delivery_failed` | `channel`, `outbox_id`, `status`, `attempts`, `error` | a delivery exhausted its bounded retries — the alert face of the row above |
+| `:budget_warning` | `agent`, `tenant`, `window`, `spent`, `cap` | a calendar budget crossed its threshold (`alert_at` or a soft cap) — once per window |
+| `:breaker_open` | `agent`, `ref`, `tenant` | the reliability circuit breaker tripped for a `(tenant, provider/model)` — further turns fail fast until the cooldown |
+| `:provider_failure` | `agent`, `ref`, `error`, `kind` | one attempt against `ref` failed and spent a retry — emitted with or without a circuit breaker |
+| `:provider_fallback` | `agent`, `from`, `to`, `error`, `kind` | the turn ROTATED mid-flight to the next node of the fallback chain, and the error that caused it |
+| `:ttft` | `task_id`, `session_id`, `ttft_ms` | the provider's time-to-first-token on the streaming envelope — only under `INSIKA_TURN_TIMING`, once per turn |
+| `:distillation_completed` | `session_ref`, `agent`, `proposals`, `dropped{}`, `deduped`, `cost{}` | a session's traffic was distilled into N proposals — counts and ids only, never a fact value |
+| `:proposal_approved` | `proposal_id`, `status`, `operator` | a human approved a distilled fact on the Facts page — written to memory via CAS |
+| `:proposal_rejected` | `proposal_id`, `status`, `operator` | a human rejected it (the reason is on the proposal record, not in the event) |
+| `:proposal_dismissed` | `proposal_id`, `status`, `operator` | a human dismissed it — the tuple is latched, never proposed again |
+| `:proposal_stale` | `proposal_id`, `status`, `operator` | the CAS lost: the fact moved after distillation — re-presented with both values on the wiki, never silently overwritten |
+| `:harvest_mined` | `agent`, `run_id`, `candidates`, `rejected{}`, `cost{}` | a mining pass finished — counts and rule ids only, never a skill body |
+| `:harvest_gated` | `run_id`, `candidate_id`, `agent`, `eval_passed`, `conversion_passed`, `reason` | the double gate's verdict on one candidate — ids and verdicts only |
+| `:skill_promoted` | `agent`, `skill`, `candidate_id`, `snapshot_ref`, `promotion_ref`, `approver` | a human approved a mined skill — it is live for the store, with the snapshot for rollback |
+| `:skill_rolled_back` | `snapshot_ref`, `skill`, `agent`, `operator` | the snapshot was restored — the promotion row carries the `rolled_back_at` stamp |
+| `:knowledge_learned` | `name`, `type`, `agent` | a turn's transcript taught the agent a new concept, or a related sighting merged into an existing one — name and type only, never the body |
+| `:knowledge_conflict` | `name`, `agent` | a repeat sighting contradicted what's on record — never merged, appended under a heading, needs a human in the Studio |
+| `:knowledge_deleted` | `name`, `agent` | an operator removed a concept from the Studio |
+| `:knowledge_retrieved` | `name`, `agent` | the model called `load_knowledge` — the adoption signal (retrieval calls per conversation), not the `<knowledge>` injection itself |
+| `:knowledge_backfilled` | `agent`, `sessions`, `concepts`, `conflicts`, `dropped{}` | the recovery re-scan (`insika knowledge:backfill`) finished replaying an agent's stored sessions |
 
 `delivery_failed` and `breaker_open` are the two the operator config is pointed at
 (`alerts.webhook` on the profile): each only fires when something durable did
@@ -366,6 +382,49 @@ consumer that fell far behind would have its subscription closed (telemetry stop
 the turn does not). Span and instrument operations are cheap, so there's ample
 headroom.
 
+## Process vitals — `GET /v1/vitals`
+
+OTel carries turn/tool telemetry; it says nothing about the **process**. For the
+questions a soak (or any operator) asks — *which process is this, how long has it
+been up, how much memory does it hold, and what is the Ruby heap doing?* — there
+is one read-only route:
+
+```bash
+curl -H "Authorization: Bearer $TOKEN" https://<target>/v1/vitals
+```
+
+```jsonc
+{
+  "boot_id": "20260820T09-…",     // one per container start; a change = a restart
+  "pid": 42,
+  "started_at": "2026-08-20T09:00:00Z",
+  "uptime_s": 259200,
+  "version": "0.2.0",
+  "ruby": "ruby 3.4.1 …",
+  "yjit": true,
+  "rss_bytes": 536870912,          // nil when unreadable — never a guess
+  "gc": { "heap_live_slots": …, "major_gc_count": …, "malloc_increase_bytes": … },
+  "threads": 8,
+  "in_flight": 1,                  // the executor's in-flight turns
+  "db_bytes": { "db": …, "wal": …, "shm": … },
+  "at": "2026-08-20T09:00:00Z"
+}
+```
+
+The two fields that make it a **restart detector**: `boot_id` (one per container
+start, exported by the entrypoint and shared by every worker) and `pid`. A
+`boot_id` change is a container restart; a `pid` change under the same `boot_id`
+is a worker respawn — the event a platform metrics API cannot see.
+
+- **Operator-only.** The route is not in the public allowlist (no bearer →
+  unauthorized) and not on the tenant surface, so only an operator reads
+  process internals. `/up` stays the public health probe and carries no
+  process data.
+- **Reads no store.** Pure OS/VM readings — safe to poll at any rate, and it
+  cannot contend with turns.
+- **The soak's sampler.** [Soak](SOAK.md) polls it hourly; a `nil` RSS reads as
+  missing coverage, never as zero.
+
 ---
 
 *Packaging note.* Today the bridge lives in the Insika repo as an opt-in core