insika 0.3.0 → 0.7.0

data/docs/HARVEST.md

@@ -1,7 +1,7 @@
 ---
 title: Harvest
-parent: Operate & prove it
-nav_order: 7
+parent: Improve
+nav_order: 6
 permalink: /harvest/
 ---
 
@@ -44,7 +44,7 @@ filters, each drop counted and logged:
    regex; phrases match case/accent-folded at word boundaries. Every rejected
    candidate is logged with the rule id.
 2. **The grounding filter** — every product reference in a proposal must be
-   in the union of the origin sessions' evidence ids (RFC-0029's ledger).
+   in the union of the origin sessions' evidence ids (the evidence ledger).
    A store without `grounding.matcher.sku` does not mine at all: product
    claims that cannot be verified are blocked by refusal, not by prompt.
 3. **Dedup** — an open `(agent, name)` tuple or a skill the store already has.
@@ -59,11 +59,10 @@ regression disqualifies** — the gate is a veto, never a score to argue with.
 Judges are mandatory in exactly the shapes the refinement gate already
 refuses: no recorded baseline, an all-red baseline, and a judged baseline
 replayed with no judge (a rubric'd case with no verdict would count as a
-pass — the P18 lesson, see [Evals](EVALS.md)).
+pass, see [Evals](EVALS.md)).
 
 The conversion gate is the second ruler: the store's funnel metric over the
-criterion's window, compared to the **frozen baseline** (the RFC-0032
-`freeze_funnel_baseline`). Outcome is evidence — this gate can only say "the
+criterion's window, compared to the **frozen baseline** (`freeze_funnel_baseline`). Outcome is evidence — this gate can only say "the
 store is measurably worse than the accepted state" or "there is nothing to
 compare against". It refuses on missing data, never passes: no frozen
 baseline, no criterion, no funnel store, a fold that has not converged — each

data/docs/KNOWLEDGE.md

@@ -0,0 +1,290 @@
+---
+title: Knowledge
+parent: Improve
+nav_order: 4
+permalink: /knowledge/
+---
+
+# Knowledge — learning from conversations
+
+An agent is amnesiac in exactly one dimension. It has history (the session),
+memory (facts about one customer, written by the `remember` tool), skills
+(procedures a human curated), and tool traces — but nothing turns a finished
+conversation into knowledge the agent can use on someone else's turn. Every
+conversation teaches a store things — which product customers actually ask
+about, which objection keeps coming up, which CEP maps to which distribution
+center — and today all of it is write-once trace data nobody reads back.
+
+Knowledge is that loop. After a turn completes, the engine can extract durable
+**concepts** from it — facts, procedures, policies, objections — and persist
+them for the agent (never a customer, never a session) to build on.
+
+**What ships today: extraction, consolidation, retrieval, export, and a
+Studio page.** The engine writes what it learns, decides whether a repeat
+sighting confirms, merges with, or contradicts what it already knew,
+retrieves the concepts relevant to a turn's message back into the prompt,
+and an operator can see, edit and resolve all of it in the Studio. Only the
+optional FTS5 index remains, deferred with a measured trigger — see
+[What's not here yet](#whats-not-here-yet).
+
+## The concept format
+
+One concept is one record — a markdown document with a YAML frontmatter
+block, the same shape a `SKILL.md` uses:
+
+```markdown
+---
+name: cep-sudeste-cd-campinas
+description: CEPs 13xxx-13999 ship from the Campinas DC, 1-2 business days.
+type: fact                     # fact | entity | procedure | policy | objection
+provenance: observed           # policy (curated) | observed (learned from conversations)
+confidence: 0.6                # 0..1, evidence-weighted
+sources: ["sess_8f3c"]         # session ids, never message content
+occurrences: 1
+created_at: 2026-08-24T18:02:11Z
+updated_at: 2026-08-24T18:02:11Z
+---
+
+Orders to CEP range 13000-13999 are fulfilled by the Campinas DC. Quoted
+delivery is 1-2 business days. Related: [[frete-gratis-acima-199]].
+```
+
+The model only ever writes `name`, `description`, `type` and `body`. Every
+other field — `provenance`, `confidence`, `sources`, `occurrences`, the
+timestamps — is stamped by the engine; an extraction answer that tries to
+supply one of those is rejected outright (the same discipline Facts applies
+to a model-authored scope). `[[links]]` inside the body are plain text stored
+as-is; retrieval resolves them lazily by name at read time (below) — that one
+hop is the entire "graph," never a stored structure of its own.
+
+**`provenance` is not decoration.** Everything the extractor writes is
+`provenance: observed` — a claim learned from what people said in
+conversations, not official policy. A promise an agent made ("we'll get back
+to you in 48h") is observed practice, not a guarantee, and the field exists so
+nothing downstream states it to a customer as a commitment. `provenance:
+policy` is reserved for a concept a human authored or promoted by hand — the
+engine never sets it.
+
+## Enabling it — the `knowledge:` block
+
+Knowledge is pack data on the agent, exactly like `distill:` or `harvest:` —
+absent = the feature is off for that agent, byte-identical engine:
+
+```ruby
+agent = Insika.agent("store-support") do
+  instructions "…"
+  knowledge extract: true, retrieve: true,
+            types: %w[fact policy objection],  # what the extractor may emit
+            top_k: 5                            # concepts injected per turn (default 5)
+  # prompt: "<what counts as a concept for THIS store>" — the pack-authored
+  #   half; absent = the engine's generic prompt. `model:` (absent = the
+  #   platform utility_model) names the extractor AND the consolidator.
+  # index: "scan" (default, the only one built) | "fts5" (accepted, falls
+  #   back to "scan" — the optional SQLite index isn't built yet).
+end
+```
+
+`extract` and `retrieve` are independent switches: an agent can learn without
+recalling (write-only, inspected by hand), or recall without learning
+(curate every concept by hand in the Studio, `provenance: policy`).
+
+## The write path
+
+After a turn completes, off the turn's critical path (the customer already has
+the answer), the engine sends the turn's transcript slice to the platform
+`utility_model` and asks for a JSON array of candidate concepts. The answer is
+schema-validated, filtered against the agent's configured `types`, and any key
+the model should not be writing is dropped and counted, never trusted. The
+survivors are redacted for PII, stamped with provenance and a first-sighting
+confidence, and written to the store — one `:knowledge_learned` event per
+concept (name, type, agent — never content).
+
+**Durability, honestly.** Extraction is best-effort: a crash between a turn's
+terminal and the write loses that turn's concepts, not the conversation (which
+is durable) and not a previously learned concept. The recovery path is a
+re-scan, not a queue:
+
+```
+insika knowledge:backfill --agent store-support --since 2026-08-01T00:00:00Z
+```
+
+replays the agent's stored sessions through the same extractor a live turn
+uses.
+
+## Consolidation — what happens on a repeat sighting
+
+Writing a concept name that already exists is never a blind overwrite. The
+engine compares the new sighting against what is on record and picks one of
+three outcomes:
+
+- **Same claim, reworded or reconfirmed** — a cheap, deterministic check
+  (no model call): occurrences go up, the new session id joins `sources`,
+  confidence climbs (`min(0.95, 0.5 + 0.1 × distinct sources)` — more
+  independent sightings, more confidence, never certainty). The body itself
+  is untouched, so an operator's edit is never silently discarded by a
+  repeat sighting.
+- **Related claim** — the two bodies say compatible things that combine into
+  one coherent statement. A second model call (the only place this feature
+  spends a second call, and only when a name already exists) merges them;
+  the result bumps occurrences/sources/confidence the same way a same-claim
+  sighting does.
+- **Contradicting claim** — the two bodies say genuinely different things.
+  **Never merged, never silently overwritten.** The new claim is appended
+  under a `## Contradiction` heading, confidence drops to a flat `0.4`, and
+  a `:knowledge_conflict` event fires. A human resolves it in the Studio by
+  editing the concept directly — there is no separate "resolve" action,
+  because resolving IS editing the markdown to say what's actually true.
+
+When no model is configured for consolidation (or its answer is unusable),
+the engine defaults to the conservative outcome — contradicting. A concept
+this feature is unsure about becomes a human's problem, never a guess that
+looks confident and might be wrong.
+
+## The Studio page
+
+`/studio/knowledge` — single-agent-scoped like Harvest (`?agent=`), not
+shared like Skills, because a concept only ever belongs to one store. A
+drill-down list (name, type, confidence, occurrences, updated_at) with a
+conflict filter (`?status=conflict`) and the same CodeMirror markdown editor
+Skills uses, version history and restore, and delete. Editing the raw
+markdown is also how an operator promotes `provenance: observed` to `policy`
+— there is no separate "promote" button, the field is just another line in
+the file.
+
+## Retrieval — what reaches a turn's prompt
+
+With `knowledge.retrieve` on, every turn the engine searches the agent's
+concepts for the ones relevant to the customer's message (pure term overlap —
+no embeddings, no network call) and injects the top few as a level-1
+`<knowledge>` block, the same progressive-disclosure shape Skills uses:
+
+```
+<knowledge>
+  <concept name="cep-13-campinas" confidence="0.60" provenance="observed">CEPs 13xxx-13999 ship from the Campinas DC, 1-2 business days.</concept>
+</knowledge>
+
+If the customer's question needs more than the summary above, call
+`load_knowledge("name")` FIRST — before any other lookup for that topic.
+This is learned from past conversations, not official policy: never state
+a `provenance="observed"` concept to the customer as a guarantee.
+```
+
+Only `name`/`description`/`confidence`/`provenance` are shown — the same
+"summary now, body on demand" shape a skill's level-1 list uses. A `load_knowledge`
+tool (outside `tools_allow`, wired only when `retrieve` is on — same as
+`load_skill`) fetches one concept's complete body. Calling it fires
+`:knowledge_retrieved` — this, not the injection itself, is what the
+adoption metric tracks (see [the honest limits](#the-honest-limits)): a
+concept sitting unread in the prompt taught the agent nothing.
+
+**One hop through `[[links]]`.** A matched concept's body may reference
+`[[other-concept-name]]`; retrieval resolves those names against the store
+and injects them too (capped at `top_k` again), so a concept that names its
+neighbor arrives with it. No transitive walk — one level, deliberately, the
+same reasoning a skill's declared `companions:` uses.
+
+**Where it sits, and what gets cut first.** `<knowledge>` sits at priority 77
+— below curated skills (80), above a single conversation's memory (75): a
+human's playbook always outranks what the engine inferred, and what the
+engine inferred outranks one customer's chat facts. Never pinned; under
+budget pressure it goes before skills but survives longer than memory,
+briefing, history and request context — see [Context](CONTEXT.md).
+
+## External knowledge over MCP
+
+A native knowledge base is not the only shape this can take. Mounting a
+third-party knowledge or memory server as MCP tools on an agent is a
+supported pattern, complementary to the native loop above — a deployment can
+run both: this loop learns concepts from ITS OWN traffic automatically, while
+an MCP-mounted server can serve as a synthesis/consultant tool an agent calls
+explicitly.
+
+The one adoption lesson worth carrying over regardless of which shape is used:
+a model does not reliably call a recall/lookup tool just because it exists,
+even when it clearly should. A polite "use this when relevant" instruction
+measured close to zero calls under pressure; what worked was an explicit,
+ordered rule naming the competing tools directly — "call the recall tool
+FIRST, before any data lookup, on topics X" — not a sentiment about when it's
+a good idea. Tool *adoption* is a prompt-ordering problem, not a tool-quality
+one, and it is worth measuring (calls per conversation), not assuming.
+
+## The honest limits
+
+- **Best-effort extraction, re-scan recovery.** Same discipline as Facts: no
+  queue, no exactly-once claim. A concept is re-derivable from the session
+  transcript, so a missed extraction is recoverable, never lost.
+- **Consolidation trusts the same model that extracts.** The "related vs.
+  contradicting" call is a model judgment, not a proof — the conservative
+  default (contradicting, when unsure) bounds the failure mode to "a human
+  looks at it," never "two different claims silently became one wrong one."
+- **Precision is a forge audit.** The engine guarantees the gates (schema,
+  key-stripping, PII redaction, type allowlist); it cannot guarantee the
+  model's judgment about what is worth remembering. That is tuned per store,
+  the same way a Harvest or Facts prompt is.
+- **Retrieval quality is not adoption.** Injecting the right concept proves
+  nothing if the model never reads it — measure `:knowledge_retrieved`
+  (retrieval calls per conversation), not just whether the block appeared.
+  The explicit, ordered instruction in the block exists because a softer
+  "when to use" wording measured close to zero calls in practice.
+- **Term overlap, not understanding.** `Index::Scan` matches words, not
+  meaning — a concept phrased very differently from the customer's words
+  will not surface even if it answers the question. No embeddings by
+  design: revisit only with evidence that retrieval, not
+  extraction, is the bottleneck.
+
+## Index::Scan's performance, measured not assumed
+
+`Index::Scan` keeps a read cache per instance (the context provider holds
+one for the whole process, never rebuilt per turn): a concept's YAML
+frontmatter is parsed once and reused until that concept's own record
+`updated_at` changes, so a write invalidates itself for free. Reproduce
+these numbers with:
+
+```bash
+bundle exec ruby scripts/bench_knowledge_index.rb
+```
+
+| concepts/agent | p50 | p95 |
+|---|---|---|
+| 50 | 0.28 ms | 0.47 ms |
+| 200 | 1.17 ms | 1.56 ms |
+| 1000 | 6.4 ms | 8.0 ms |
+| 5000 | 35.6 ms | 41.3 ms |
+
+At the scale this feature targets for the first year — hundreds of concepts
+per agent — a warm-cache search costs a bit over a millisecond, close to
+the engine's own documented per-turn overhead (see [Benchmark](BENCHMARK.md)).
+Past roughly a thousand concepts it becomes a real, measurable cost again —
+that specific, numeric point is the trigger for building `Index::FTS5`, not
+a guess made in advance.
+
+## Export
+
+```
+insika knowledge:export --agent store-support --out ./export [--tenant loja-a]
+```
+
+Writes one `<name>.md` per concept — the storage format IS the export
+format, so this is a dump, not a converter (same discipline as Facts/
+Harvest's own append-only records): each file is the concept's markdown,
+byte for byte, directly consumable by okf-gem (`OKF::Bundle`) or graphify.
+Re-running it is safe — nothing here is lossy, so there is no `--force` to
+reason about.
+
+`--format graphml` writes one combined `knowledge.graphml` instead: a node
+per concept (`name`/`type`/`description`/`confidence`/`provenance` as node
+data) and an edge per `[[link]]` that resolves to another concept in the
+same export — a link to a concept outside the scope is dropped, never a
+dangling edge. Hand-built, hand-escaped XML (no new dependency), directly
+openable in Gephi, yEd, or graphify.
+
+## What's not here yet
+
+- **The optional FTS5 index** — `knowledge.index: "fts5"` is accepted but
+  falls back to `Index::Scan`. Deliberately not built yet: `Scan` was
+  measured (above), not assumed, and it comfortably meets this feature's
+  target scale. A deployment whose concept count is heading past ~1000 per
+  agent is the evidence that would justify building the SQLite
+  `MATCH`/`bm25()` adapter — not before.
+- **Decay** — recency is a ranking tiebreak today; a real confidence decay
+  curve is a later, evidence-driven addition, not a default.

data/docs/LOADTEST.md

@@ -1,7 +1,7 @@
 ---
 title: Load test
-parent: Operate & prove it
-nav_order: 3
+parent: Operate
+nav_order: 4
 permalink: /loadtest/
 ---
 

data/docs/MEDIA.md

@@ -0,0 +1,128 @@
+---
+title: Media
+parent: Integrate
+nav_order: 3
+permalink: /media/
+---
+
+# Media
+
+The engine transports media, it never means it. Photos, voice notes and documents
+travel through the message contract as additive content parts; what a picture
+*means* — a fitting room, a product mockup — stays a skill on top.
+
+## In — the message contract
+
+The engine transports media, it never means it. The message accepts additive
+**content parts** alongside the text — voice notes and photos travel, and any
+skill (a fitting room, an image QA) stays a consumer layer on top:
+
+```bash
+curl -X POST /v1/messages?stream=false -H "Authorization: Bearer $TOKEN" \
+  -d '{ "agent": "store-support", "session_id": "chat-7",
+        "message": "", "parts": [
+          { "type": "audio", "url": "https://cdn.example.com/voz.ogg" },
+          { "type": "image", "url": "https://cdn.example.com/sofa.jpg" },
+          { "type": "document", "url": "https://cdn.example.com/receita.pdf" }
+        ] }'
+```
+
+- **Audio** is transcribed (RubyLLM STT; model via `INSIKA_STT_MODEL`) and the
+  text enters the turn marked `source: "voice"` on the terminal event — the
+  consumer's signal the person spoke. A consumer that transcribes itself can
+  send the text with `"source": "voice"` directly. A domain vocabulary hint
+  (product names, brand terms) rides the transcription as `prompt:` —
+  per-agent `stt_prompt` (the DSL setter, or the Studio config form) beats the
+  deployment-wide `INSIKA_STT_PROMPT` env, which beats nothing. OPERATOR
+  config, never customer input.
+- **Images** attach to the model's ask (vision); the provider bills them and
+  the usage flows like any ask. The first image URL is also
+  `{{ctx.image_url}}` for data tools — photo analysis outside the prompt, the
+  tool's own egress applying when it fetches.
+- **Documents** (a prescription, a recipe, an invoice — most often a PDF)
+  attach the same way images do; the first document URL is
+  `{{ctx.document_url}}`. A model without document support fails the ask at
+  `:ruby_llm` with the provider's own error — the transport does not preflight
+  capability. Capped at 10 MB (`MAX_DOCUMENT_BYTES` — "a prescription, not an
+  archive"), separately from the 5 MB image cap.
+- Media URLs (audio, image AND document) are fetched by the engine through
+  the same egress guard (a private/metadata target is refused — SSRF) and a
+  size ceiling per kind (1 MB audio, 5 MB image, 10 MB document: the bytes
+  land in this process). A refused, oversized or unreadable part fails the turn
+  loudly at the `:media` stage, never a silent drop.
+- **Media alone is a turn.** A voice note with no caption is `parts` and an
+  empty `message` — the transcription becomes the message at the `:media`
+  stage. A media message never joins another turn (`collect`/`steer` move text
+  only, and the parts would be left behind), and a transcription that comes
+  back empty fails the turn instead of asking the model about nothing.
+- **Parts are contract at the edge** — a malformed part (unknown type, an
+  image/audio/document without `url`, a text without `text`) is a 422 before
+  dispatch on `/v1/messages` and `/v1/responses`. `document` is an ADDITIVE
+  part type (the compatibility rule in [the /v1 API](API.md) — no
+  `Insika-Version` bump needed).
+- `/v1/responses` accepts the OpenAI multimodal shape: `input` as an array of
+  text/image/audio/document parts.
+
+### Image editing
+
+`generate_image` (below) doesn't only generate — it can EDIT an existing
+image, using `RubyLLM.paint`'s `with:`/`mask:`. The tool exposes
+`source_image_urls` (an array — up to 4) and `mask_url`; when the model omits
+`source_image_urls` AND the turn carries an inbound photo, that photo is
+edited by default — no URL round-trip needed for "edit the photo the customer
+just sent". Explicit URLs always win over the default. A text-to-image call
+(no sources at all) is byte-identical to before this feature existed.
+
+What the edit MEANS — a virtual try-on, a product mockup on the customer's
+wall — is the calling skill's business; the tool only transports the bytes.
+Not every image model can edit (`dall-e-3` cannot); a call against a
+non-editing model fails at the provider, surfaced verbatim.
+
+## Out — generated media
+
+The turn can **produce** an image or a voice clip — but only when both sides of
+the gate agree, because nothing leaks by default. The agent declares it may
+generate media (`outputs` on the profile), and the **channel** declares it can
+receive it (`channel.capabilities` on the request):
+
+```bash
+curl -X POST /v1/responses -H "Authorization: Bearer $TOKEN" -d '{
+  "model": "openclaw:store-support", "user": "chat-7",
+  "input": "manda a foto do sofá da promoção",
+  "channel": { "capabilities": ["image_output", "audio_output"] }
+}'
+```
+
+```ruby
+agent = Insika.agent("store-support") do
+  instructions "…"
+  outputs image: { model: "gpt-image-1", size: "1024x1024" },   # the AGENT's half
+          tts:   { model: "tts-1", voice: "alloy" }
+end
+```
+
+- **Both gates** must pass for the model to even see the `generate_image` /
+  `tts` tools: the agent opted in (`outputs`) and the request declared the
+  matching capability (`image_output` / `audio_output` — an unknown value is a
+  422, never a silent ignore). The "abstraction admits only what leaks" rule.
+- **The media rides the envelope, never the answer text.** The terminal event
+  and the `/v1/responses` completed frame carry an additive `output_parts`
+  array — `{ type: "image", mime_type:, base64:, model: }` /
+  `{ type: "audio", mime_type:, base64:, model: }`. The model's prose stays
+  the `:content` answer; the channel consumes the bytes next to it.
+- **Generation is billed and counted.** Image tokens join the turn's usage
+  (like any ask). The speech API reports no token counts, so a TTS call adds
+  an honest `usage.media` counter and the part carries the `model` for
+  consumer-side pricing.
+- **Seams, not magic.** The generator is injectable per kind (specs stub it);
+  the defaults are lazy: images via RubyLLM (paint), speech via a thin POST to
+  the OpenAI-compatible `/audio/speech` endpoint using the same provider
+  config the chat uses — RubyLLM as of 1.16.0 has no speech API. A generated
+  part over 8 MB refuses loudly, never silently truncates.
+- **Not here:** what the generated image *means* — a fitting room, a product
+  mockup — is a skill on top. The engine transports bytes and cost.
+
+## See also
+
+- [The /v1 API](API.md) — the full message contract.
+- [Channels](CHANNELS.md) — how WhatsApp and the web widget carry these parts.

data/docs/OBSERVABILITY.md

@@ -1,6 +1,6 @@
 ---
 title: Observability
-parent: Operate & prove it
+parent: Operate
 nav_order: 1
 permalink: /observability/
 ---
@@ -36,23 +36,28 @@ something that left no task of its own behind:
 | `:turn_steered` | `task_id`, `count`, `total` | a message arrived mid-run and was appended to the turn in flight |
 | `:turn_steer_released` | `task_id`, `released_as`, `count` | the run could not absorb it, so it became the turn `released_as` |
 | `:turn_interrupted` | `task_id`, `replaced_by` | the turn was abandoned mid-run, and which turn replaced it |
-| `:turn_stuck` | `task_id`, `agent`, `reason`, `message` | the agent declared it could not proceed (`signal_stuck`, WS5) — the deterministic signal a consumer escalates on |
+| `:turn_stuck` | `task_id`, `agent`, `reason`, `message` | the agent declared it could not proceed (`signal_stuck`) — the deterministic signal a consumer escalates on |
 | `:channel_delivered` | `channel`, `outbox_id`, `status`, `attempts`, `error` | the answer reached the platform (or did not) — the turn completing says nothing about that |
-| `:delivery_failed` | `channel`, `outbox_id`, `status`, `attempts`, `error` | a delivery exhausted its bounded retries — the alert face of the row above (WS6) |
-| `:budget_warning` | `agent`, `tenant`, `window`, `spent`, `cap` | a calendar budget crossed its threshold (`alert_at` or a soft cap) — once per window (WS2) |
-| `:breaker_open` | `agent`, `ref`, `tenant` | the reliability circuit breaker tripped for a `(tenant, provider/model)` — further turns fail fast until the cooldown (WS3/WS6) |
-| `:provider_failure` | `agent`, `ref`, `error`, `kind` | one attempt against `ref` failed and spent a retry — emitted with or without a circuit breaker (WS3) |
-| `:provider_fallback` | `agent`, `from`, `to`, `error`, `kind` | the turn ROTATED mid-flight to the next node of the fallback chain, and the error that caused it (WS3) |
-| `:ttft` | `task_id`, `session_id`, `ttft_ms` | the provider's time-to-first-token on the streaming envelope — only under `INSIKA_TURN_TIMING`, once per turn (WS6) |
-| `:distillation_completed` | `session_ref`, `agent`, `proposals`, `dropped{}`, `deduped`, `cost{}` | a session's traffic was distilled into N proposals (RFC-0034) — counts and ids only, never a fact value |
+| `:delivery_failed` | `channel`, `outbox_id`, `status`, `attempts`, `error` | a delivery exhausted its bounded retries — the alert face of the row above |
+| `:budget_warning` | `agent`, `tenant`, `window`, `spent`, `cap` | a calendar budget crossed its threshold (`alert_at` or a soft cap) — once per window |
+| `:breaker_open` | `agent`, `ref`, `tenant` | the reliability circuit breaker tripped for a `(tenant, provider/model)` — further turns fail fast until the cooldown |
+| `:provider_failure` | `agent`, `ref`, `error`, `kind` | one attempt against `ref` failed and spent a retry — emitted with or without a circuit breaker |
+| `:provider_fallback` | `agent`, `from`, `to`, `error`, `kind` | the turn ROTATED mid-flight to the next node of the fallback chain, and the error that caused it |
+| `:ttft` | `task_id`, `session_id`, `ttft_ms` | the provider's time-to-first-token on the streaming envelope — only under `INSIKA_TURN_TIMING`, once per turn |
+| `:distillation_completed` | `session_ref`, `agent`, `proposals`, `dropped{}`, `deduped`, `cost{}` | a session's traffic was distilled into N proposals — counts and ids only, never a fact value |
 | `:proposal_approved` | `proposal_id`, `status`, `operator` | a human approved a distilled fact on the Facts page — written to memory via CAS |
 | `:proposal_rejected` | `proposal_id`, `status`, `operator` | a human rejected it (the reason is on the proposal record, not in the event) |
 | `:proposal_dismissed` | `proposal_id`, `status`, `operator` | a human dismissed it — the tuple is latched, never proposed again |
 | `:proposal_stale` | `proposal_id`, `status`, `operator` | the CAS lost: the fact moved after distillation — re-presented with both values on the wiki, never silently overwritten |
-| `:harvest_mined` | `agent`, `run_id`, `candidates`, `rejected{}`, `cost{}` | a mining pass finished (RFC-0035) — counts and rule ids only, never a skill body |
+| `:harvest_mined` | `agent`, `run_id`, `candidates`, `rejected{}`, `cost{}` | a mining pass finished — counts and rule ids only, never a skill body |
 | `:harvest_gated` | `run_id`, `candidate_id`, `agent`, `eval_passed`, `conversion_passed`, `reason` | the double gate's verdict on one candidate — ids and verdicts only |
 | `:skill_promoted` | `agent`, `skill`, `candidate_id`, `snapshot_ref`, `promotion_ref`, `approver` | a human approved a mined skill — it is live for the store, with the snapshot for rollback |
 | `:skill_rolled_back` | `snapshot_ref`, `skill`, `agent`, `operator` | the snapshot was restored — the promotion row carries the `rolled_back_at` stamp |
+| `:knowledge_learned` | `name`, `type`, `agent` | a turn's transcript taught the agent a new concept, or a related sighting merged into an existing one — name and type only, never the body |
+| `:knowledge_conflict` | `name`, `agent` | a repeat sighting contradicted what's on record — never merged, appended under a heading, needs a human in the Studio |
+| `:knowledge_deleted` | `name`, `agent` | an operator removed a concept from the Studio |
+| `:knowledge_retrieved` | `name`, `agent` | the model called `load_knowledge` — the adoption signal (retrieval calls per conversation), not the `<knowledge>` injection itself |
+| `:knowledge_backfilled` | `agent`, `sessions`, `concepts`, `conflicts`, `dropped{}` | the recovery re-scan (`insika knowledge:backfill`) finished replaying an agent's stored sessions |
 
 `delivery_failed` and `breaker_open` are the two the operator config is pointed at
 (`alerts.webhook` on the profile): each only fires when something durable did

data/docs/OUTCOMES.md

@@ -0,0 +1,137 @@
+---
+title: Outcomes and follow-ups
+parent: Improve
+nav_order: 3
+permalink: /outcomes/
+---
+
+# Outcomes and follow-ups
+
+## Outcomes — business results over real traffic
+
+The engine measures what it is told to measure. The operator or the integration
+records a conversation's business outcome after the fact — `conversion`,
+`escalation`, `deflected`, anything, optionally with a monetary `value`:
+
+```bash
+curl -X POST /v1/outcomes -H "Authorization: Bearer $TOKEN" \
+  -d '{ "agent": "store-support", "session_id": "chat-7",
+        "outcome": "conversion", "value": 129.9 }'
+```
+
+The endpoint is **additive and outside the response contract** — the turn never
+knows or cares; the engine transports the outcome and never interprets it (what
+"conversion" means is yours). Records are tenant-stamped (a tenant principal
+writes and reads only its own), and `GET /v1/outcomes?agent=` serves the last
+outcome per agent plus the per-day series — the last-outcome pill on the Studio
+agent grid, and the per-day series on the agent detail.
+
+### The outcome funnel
+
+A store's funnel is pack data on the agent — the engine folds outcomes into
+the **declared** stages, and never hard-codes one itself (the stage vocabulary
+is the forge's):
+
+```ruby
+agent = Insika.agent("store-support") do
+  instructions "…"
+  funnel stages: %w[greeted qualified cart paid],
+         advance_on: { "abandoned_cart" => "cart", "pix_paid" => "paid" },
+         primary: "paid", attribution_window: "72h"
+end
+```
+
+The fold contract:
+
+- **Tick-driven, cumulative event counts on the declared order.** An outcome of
+  kind K means the session *reached* `advance_on[K]`; the fold increments
+  `stages[0..index]` for the reached stage. A per-stage-complete integration
+  and a terminal-event integration therefore produce identical counts — a
+  session that paid also emitted the earlier events. A duplicate event
+  double-counts (the integration's defect, not the engine's); **do not declare
+  a stage off the linear path** (a "handoff" stage would be inflated by every
+  later event). Counts are **event counts, not distinct sessions** — the
+  baseline is events-based.
+- **Idempotent**: a per-pair `{at, ids}` cursor inside one transaction; a crash
+  mid-fold never double counts, and a second pass folds only what is new.
+- **The attribution window is carried data, never computed** — `72h` is
+  validated, rendered, and copied into the baseline snapshot; causal
+  attribution stays human.
+- **The baseline freeze** (Studio > Funnel, or `:freeze_funnel_baseline` on the
+  bus) sums the folded cells over a span of **≥ 28 days** (shorter spans are
+  refused) into one current snapshot per `(tenant, agent)` — the number
+  the follow-up A/B and the harvest promotion gate compare against.
+- **Malformed declarations never crash the tick**: the fold skips them, the
+  doctor names the defect, the Studio shows nothing until it is fixed.
+- Vocabulary note: in the gem this is the **outcome funnel** — the stage names
+  are the forge's, and a bare install (no `funnel:` on any agent) shows no
+  funnel and no stage names at all.
+
+## Follow-ups — the seller who comes back
+
+The agent can book a follow-up with a customer at a future time — "te chamo
+amanhã se o PIX não cair" said in-conversation and meant. The engine fires the
+synthetic turn on its own tick, with consent and without spam. Everything is
+pack data on the profile:
+
+```ruby
+agent = Insika.agent("store-support") do
+  instructions "…"
+  followup arm: "schedule",
+           policy: { quiet_hours: { timezone: "America/Sao_Paulo",
+                                    start: "21:30", end: "09:00" },
+                     max_frequency: "2/24h",       # N outbound per window, per customer
+                     cancel_keywords: ["não quero mais contato"],
+                     silence_after_sends: 3 }      # N fires without a reply -> :unavailable
+end
+```
+
+The pieces:
+
+- **`schedule(at:, reason:)`** — a built-in tool the agent calls when the
+  customer agrees to be contacted again (a product, a cart, a pending payment).
+  The call itself IS the consent record — recorded without ever lifting
+  `:unavailable` or resetting the silence counter (ONLY a customer message
+  reopens, so a re-booking inside a follow-up turn cannot clear the silence
+  protection). `cancel_followup(id:)` is the sibling. A customer who opted out
+  can never be rescheduled.
+- **Contact state per customer** — `granted | revoked | unavailable` in a
+  durable cell per `(tenant, customer)`. Only `granted` may be messaged;
+  `revoked` is immediate and permanent until the customer speaks again;
+  `unavailable` means silence ≠ refusal — the engine stops firing after
+  `silence_after_sends` unanswered sends, and ANY customer message reopens.
+  The policy's `cancel_keywords` are matched on every inbound message: a
+  match revokes the contact and cancels its pending follow-ups in one
+  transaction.
+- **Firing is the tick's third duty** — the engine claims the due records
+  (one per claim window, at-most-once across workers), applies the policy in
+  force AT FIRE TIME (contact state, quiet hours, dedup per
+  `(customer, reason)`, frequency ceiling) and either enqueues the synthetic
+  turn or marks the record `blocked` with the failing rule — auditable, never
+  silent. Blocking happens at fire time, never at schedule time: the schedule
+  is a promise made in-conversation, and only the policy in force then may
+  revoke it.
+- **The synthetic turn** — a first-class inbound turn stamped
+  `origin: "scheduled"` (a refinement read can never mistake the engine's
+  kick for the customer repeating themselves), delivered through the full
+  pipeline on the channel the conversation came in on. It skips the edge's
+  ENTRY rate/token checks like a resume does — a follow-up she agreed to must
+  not receive the rate-limit reply; its usage still lands on the ledger.
+- **The Follow-ups page** (Studio) — per agent: the pending/fired/cancelled/
+  blocked records (blocked rows carry the reason), the read-only policy
+  summary and the A/B card: per arm, `sent` vs `conversions` (against the
+  frozen baseline) vs `opt-outs`. The only mutations — cancel a pending
+  record, force-revoke a contact — go through bus commands.
+- **LGPD** — the records and cells die with the customer (`forget_customer`),
+  the tenant (`delete_tenant_data`) and age out under the same
+  `retention_days` sweep as the rest of the footprint.
+
+Absent `followup:` = the feature is off for that agent — no tools wired, no
+records, byte-identical turns. The A/B against an existing cron is an
+operator experiment: the engine only keeps the records and the read card (the
+cron arm writes through the same store class with its own `arm` label).
+
+## See also
+
+- [Evals](EVALS.md) — the cases that grade an agent before traffic does.
+- [Refinement](REFINEMENT.md) — reading an agent's own traffic back as a report.