insika 0.2.0 → 0.3.0

data/docs/CONTEXT.md

@@ -20,19 +20,21 @@ Providers are chosen by a double gate (the provider opts in for the profile **an
 the agent's `context_providers` allowlist permits it), then assembled by priority
 into a deterministic prompt:
 
-| Provider | Block | Priority | Notes |
-|----------|-------|:--------:|-------|
-| **Identity** | system | **100 — pinned** | The agent's prompt files (global system files first). Never cut. |
-| **Skill trigger** | `<active_skill>` | 85 | Level-2 bodies: the agent's `skills_eager` set, plus the ones whose `triggers:` match the message — see [Skills](SKILLS.md). |
-| **Skills** | `<available_skills>` | 80 | Level-1 skill list, minus whatever is already eager — see [Skills](SKILLS.md). |
-| **Memory** | `<memory>` | 75 | Durable facts + recent notes, only if `memory` is on. Cuttable. |
-| **Tool search** | `<available_tools>` | 70 | Level-1 list of deferred tools — see [Tools](TOOLS.md). |
-| **Session** | history | 60–79 | The running transcript; priority scales with recency. |
-| **Request** | `<request_context>` | 40 | Turn variables + tenant. Most cuttable; sits last. |
-
-The ordering is deliberate: the **stable identity sits first**, the **volatile
-request context sits last**. That keeps the cacheable prefix byte-stable (see the
-prefix cache below).
+| Provider | Block | Priority | Layer | Notes |
+|----------|-------|:--------:|-------|-------|
+| **Identity** | system | **100 — pinned** | identity | The agent's prompt files (global system files first). Never cut. |
+| **Skills** | `<available_skills>` | 80 | identity | Level-1 skill list, minus whatever is already eager — see [Skills](SKILLS.md). |
+| **Tool search** | `<available_tools>` | 70 | identity | Level-1 list of deferred tools — see [Tools](TOOLS.md). |
+| **Skill trigger** | `<active_skill>` | 85 | volatile | Level-2 bodies: the agent's `skills_eager` set, plus the ones whose `triggers:` match the message — see [Skills](SKILLS.md). |
+| **Memory** | `<memory>` | 75 | volatile | Durable facts + recent notes, only if `memory` is on. Cuttable. |
+| **Briefing** | `<briefing>` | 65 | volatile | The session's working state (known fields, still-missing list, next step) — only if the pack declared `briefing_fields`. Cuttable. |
+| **Session** | history | 60–79 | volatile | The running transcript; priority scales with recency. |
+| **Request** | `<request_context>` | 40 | volatile | Turn variables + tenant. Most cuttable; sits last. |
+
+The ordering is deliberate: the render order is **identity layer first, volatile
+layer after** — nothing volatile can sit above the cache boundary, whatever its
+priority — and within each layer the priority sort above holds. That keeps the
+cacheable prefix byte-stable (see the prefix cache below).
 
 ## Budget and eviction — the actual "compaction"
 
@@ -78,19 +80,82 @@ budget first.
 With `memory` enabled, an agent gains a built-in `remember` tool for durable
 facts, and those facts (plus recent notes) are injected back into the prompt on
 later turns — **including turns in a different session**. Memory is scoped per
-agent. This is distinct from *session history*, which is the transcript of one
+agent, per `(tenant, customer)` when the message carries a `customer`, and per
+session otherwise — a session's own memory lives in a marked `memory:chat:<session id>`
+cell, never a bare one, so the Customers drill cannot read a conversation as a
+customer. This is distinct from *session history*, which is the transcript of one
 conversation; memory is the small set of facts that should outlive any single
 conversation. Facts and notes are editable from the Studio agent page. See
 [`examples/memory/`](https://github.com/guizaols/insika/tree/main/examples/memory/) for a runnable cross-session example.
 
+Facts carry **provenance metadata** (RFC-0031): every fact record stores `origin`
+(who wrote it — `"engine"`, `"operator"`, `"legacy"` or `"distilled"`),
+`created_at` / `updated_at` timestamps, and an optional `expires_at` (ISO8601) —
+**an expired fact is never injected**, even before the daily sweep prunes it. The
+Studio Customers drill reads and edits the same cell the next turn reads (injection
+unchanged), and every operator mutation lands in the content-free audit trail
+(digests, never values). The sweep honors the `memory_ttl_days` setting on its own
+knob — see [Security](SECURITY.md#memory-and-the-right-to-be-forgotten-lgpd-rfc-0031).
+
+An **approved distilled fact** (RFC-0034 — see [Facts](FACTS.md)) lands in the
+same cell this provider injects, stamped `distilled:<session_ref>` — approved on
+the Studio Facts page, never applied automatically.
+
+## Briefing — the session's working state
+
+The **briefing** is the per-conversation working state the agent keeps and asks
+for: which facts it already learned (size, budget, delivery day) and the agreed
+next step. It is **engine-owned data** — one `"briefing"` key on the session
+record, written only by the agent through tools — whose *fields* come from the
+pack:
+
+```jsonc
+// pack agent.config.json
+{ "id": "store-support", "briefing_fields": ["size", "budget", "delivery_day"] }
+```
+
+```ruby
+# or the DSL — [] = the feature is off (no block, no tools)
+briefing_fields "size", "budget", "delivery_day"
+```
+
+With fields declared, the turn's `:system` context gains a `<briefing>` block
+(priority 65 — below identity/skill/memory so it never breaks the cacheable
+prefix, above the turn's own `<request_context>`):
+
+```
+<briefing>
+known:
+  size: M
+still missing: budget, delivery_day
+next step: send the payment link tomorrow at 10
+</briefing>
+```
+
+The `still missing` list is the point: the *model* sees which declared fields are
+still unanswered, so it stops re-asking for something already given. Stored keys
+that the pack no longer declares are never rendered. The Studio session screen
+shows the persisted state (known fields + next step), read-only.
+
+The agent writes the briefing through two built-in tools, wired only when the
+pack declared fields:
+
+- `update_briefing(field:, value:)` — records a field. An undeclared `field`
+  returns an envelope error (`unknown field '…'; declared: …`) and nothing is
+  persisted; a blank `value` clears the field.
+- `set_next_step(text:)` — records the agreed next step; a blank `text` clears it.
+
+Both are deterministic in-process writes (never enveloped) and survive across
+turns and resumes — a resumed conversation re-opens with the briefing intact.
+
 ## The provider prefix cache
 
 Two distinct caching mechanisms — don't conflate them:
 
 - **Automatic server-side prefix cache.** Some providers prefix-cache a stable
   system prefix automatically, at no cost to configure. This works **only because**
-  the identity is at the top of the system block and the volatile
-  `<request_context>` is at the bottom, keeping the cacheable prefix byte-stable.
+  the engine renders the system in two layers (below) and the volatile half sits
+  **under** the identity boundary, keeping the cacheable prefix byte-stable.
   Anything that injects volatile content high in the system block breaks the cache.
 - **Manual cache breakpoints (opt-in).** With `prompt_caching` on **and** a
   provider that supports explicit cache control, the builder sets one cache
@@ -100,6 +165,52 @@ Two distinct caching mechanisms — don't conflate them:
 Cache accounting surfaces as `cached_tokens` (reads) and `cache_creation_tokens`
 (writes), visible in telemetry and the Studio tokens chip.
 
+### The two layers (RFC-0030)
+
+The system block is partitioned into two cache layers:
+
+- **Identity** — bytes that change only on deploy/config edit: the persona
+  prompt (`Prompt`), the level-1 skill list (`Skill`) and the deferred-tool
+  catalog (`ToolSearch`). This is the cacheable prefix.
+- **Volatile** — bytes that may change per turn: memory, session history,
+  triggered skill bodies, the `<request_context>`. Everything else.
+
+The layer is a **provider-class contract**, not profile data: `ContextProvider`
+declares `def layer = :volatile` (conservative — nothing gets pinned by
+accident) and the three identity builtins override to `:identity`. A pack does
+not set it — a pack reorganizes *which content goes into the Prompt provider vs
+the volatile providers*. The Builder stamps the layer on every fragment at
+production, and the render order is **identity first, volatile after** — a
+volatile block can never land above the cache boundary, whatever its priority.
+Within each partition the existing priority sort is untouched.
+
+The engine's own `doctor` check verifies the declaration: an engine-known
+volatile provider (Memory, Session, Request, SkillTrigger) that overrides to
+`:identity` is an **error** (guaranteed cache kill); any other custom
+`:identity` provider is a **warning** (purity unverifiable from outside — the
+output must be byte-stable across turns).
+
+### The observable cache: fingerprints and the invalidation reason
+
+Each turn, the Executor hashes the rendered prefix into a PII-free fingerprint
+chain — one SHA-256 per system category in render order, one for the tool
+schemas, one cumulative `prefix` — and compares it against the previous turn's
+entry. The **invalidation reason** is the first category whose bytes changed (or
+vanished); a turn whose prefix held reports nothing. History is deliberately
+excluded: a new user message is a divergence every turn, which would be noise,
+not a reason.
+
+The Studio surfaces it in two places: the **session Context card** shows the
+turn's cache-hit percentage and the `broke: <category>` line (plus the
+`identity` marker on the category rows), and the **agent detail** carries a
+cache tab with the per-agent hit series over time. The per-agent series lives
+in its own capped store, because a session does not stamp its author — the
+per-session trace cannot answer "cache-hit over time for *this* agent".
+
+With the prefix stable by construction, the existing `prompt_caching` breakpoint
+sits on bytes that stay put — the first (write) turn of a deployment pays the
+cache write once, every subsequent turn reads.
+
 ## The volume
 
 Agents, prompts, skills, and tools are **data in SQLite**, not files on a volume.

data/docs/DEPLOY.md

@@ -37,7 +37,9 @@ The image boots **N Falcon worker processes over one SQLite file**
 knob**: it decides which engine semantics hold cluster-wide and which are
 per-worker. The default is 1 because the per-worker semantics are the product
 — the queue modes (`collect`/`steer`) and FIFO ordering need one
-session actor; raise N only with sticky routing per session in front. The
+session actor; raise N only with sticky routing per session in front. It is
+also a [Soak](SOAK.md) precondition: with more than one worker the hourly RSS
+series is a random process per sample, and a soak refuses to start. The
 contract:
 
 > Everything here describes N workers of **one** deployment — one graph, replicated.
@@ -112,10 +114,11 @@ this section is the single source of truth for what changing it means.
 | `DEEPSEEK_MODEL` | `deepseek-v4-flash` | model |
 | `CONSUMER_INTERNAL_URL` | — | base URL for data-tools calling back a consumer's internal API (see below) |
 | `INSIKA_EGRESS_HOSTS` | — | outbound host allowlist (SSRF guard) |
-| `INSIKA_EGRESS_ALLOW_HTTP` / `_ALLOW_PRIVATE` | off | for `http`/loopback callbacks only (**never in cloud**) |
+| `INSIKA_EGRESS_ALLOW_HTTP` / `_ALLOW_PRIVATE` | off | for `http`/loopback callbacks only (**never in cloud**). Also opens inbound **media** fetches (audio/image URLs) to `http`/private targets — a local run serving media over `http://` needs it |
 | `INSIKA_RELAY_TOKEN` | — | **mounts the relay channel** at `POST /channels/relay/events`, and is the Bearer it requires. Empty = the route does not exist (`404`). See [Channels](CHANNELS.md) |
 | `INSIKA_RELAY_DELIVER_URL` | — | your callback; the engine POSTs each reply there. Goes through the egress guard |
 | `INSIKA_RELAY_DELIVER_TOKEN` | — | Bearer the engine sends **to** your callback (optional) |
+| `INSIKA_RELAY_DELIVERY` | `at_end` | how the relay flushes the outbox: `at_end` (one POST) or `progressive` (one POST per balloon — [RFC-0027](CHANNELS.md#delivery-policy)) |
 | `INSIKA_WIDGET_ORIGINS` | — | exact-match origins allowed to embed the [web widget](CHANNELS.md#the-web-widget), comma-separated. No wildcards. **Half the switch**: with `INSIKA_WIDGET_AGENTS` unset, nothing is mounted (`404`) |
 | `INSIKA_WIDGET_AGENTS` | — | agent ids a widget visitor may address, comma-separated. The other half of the switch. **A chat rate limit is also required** or the widget answers `503` |
 | `LITESTREAM_REPLICA_URL` | — | **enables Litestream** (backup/DR). Empty = disabled (default). See below |

data/docs/EVALS.md

@@ -232,6 +232,15 @@ That is what you run before merging a prompt, tool or model change. A case with
 baseline entry never blocks: it shows as failing in the report, but a brand-new case is
 not a regression.
 
+The same gate machinery guards the two automated loops — [Refinement](REFINEMENT.md)
+(cloned-agent replay of a proposed edit) and [Harvest](HARVEST.md) (cloned-agent
+replay of a mined skill). **Judges are mandatory for those gates** in exactly three
+shapes, the P18 lesson: a gate without a recorded baseline refuses, an all-red
+baseline refuses, and a baseline recorded WITH judge scores, replayed with no judge
+configured, refuses — a rubric'd case with no verdict reads as a pass, so the
+candidate would beat a measurement it never took. A store with no golden cases cannot
+gate, and cannot promote.
+
 ## Honest limits
 
 - **A judge is a model.** It has taste and it has bad days; that is why the
@@ -271,3 +280,13 @@ needs to score a candidate agent with the same judge, and a second copy of the j
 would be the worst possible outcome. It stays a **client** even so: it reaches a running
 deployment over HTTP through `POST /v1/responses` and never reads a store directly.
 `evals/run.rb` is a thin CLI over it.
+
+## Parity — the shadow criterion
+
+The shadow experiment's rule lives in a **deployment-side criterion file** —
+the path `INSIKA_PARITY_CRITERION` points at, prose a human reads and a `yaml`
+block the machine applies, in one file, so there is exactly one place to edit.
+The file's whole bytes are hashed; every shadow pair records that hash,
+and a window whose pairs disagree produces `:invalid`, never a verdict. Editing
+the criterion mid-experiment is *caught*, not averaged away. The fold itself is
+`lib/insika/parity/*` (see [Channels](CHANNELS.md#shadow-mode)).

data/docs/FACTS.md

@@ -0,0 +1,135 @@
+---
+title: Facts
+parent: Operate & prove it
+nav_order: 6
+permalink: /facts/
+---
+
+# Facts — distilled customer memory, human-gated
+
+Finished conversations teach a shop things — the customer's size, their budget,
+how they like to pay — and today that knowledge dies with the session. The
+engine can read it back out of the transcripts: **distillation** turns an idle,
+finished customer conversation into a list of proposed facts, a human approves
+or rejects them in the Studio, and an approved fact lands in the customer's
+memory cell — the same cell the `<memory>` block injects on every later turn.
+
+The loop has one hard rule: **nothing is ever applied automatically.** Zero
+facts reach the store without a click. Distillation proposes; the operator
+decides; the engine never applies its own proposal.
+
+## The loop
+
+1. A customer conversation (a session tagged with a `customer`, per
+   [Context](CONTEXT.md#memory)) goes idle — nobody has written for the
+   configured hours.
+2. The engine's distillation duty picks the session, sends its transcript to
+   the platform `utility_model` (off the turn path, on its own worker fiber),
+   and the model answers with a JSON list of durable facts: `name`, `value`,
+   an optional `confidence`, and the transcript message indexes that support
+   the fact (the **evidence**).
+3. The engine filters the answer against a safe subset (no invented scopes, no
+   oversized values, no out-of-range evidence), dedups it against the ledger,
+   and writes the survivors as **proposals**.
+4. The **Facts** page in the Studio shows the pending proposals with their
+   evidence excerpt. The operator **approves** (the fact is written to the
+   customer's memory, stamped with its origin), **rejects** (optionally with a
+   reason) or **dismisses** (it will never be proposed again).
+5. Approved facts join the customer's memory cell and are injected by the
+   Memory provider on the next turn of any session of that customer.
+
+## Enabling it — the `distill:` block
+
+Distillation is pack data on the agent, exactly like `refinement:` or
+`followup:` — absent = the feature is off for that agent, byte-identical engine:
+
+```ruby
+agent = Insika.agent("store-support") do
+  instructions "…"
+  distill enabled: true,
+          idle_hours: 6,        # how idle a session must be before it distills
+          min_messages: 3,      # a shorter session distills noise, not facts
+          max_proposals: 10     # cap per session pass
+  # prompt: "<what counts as a fact for THIS store>" — the pack-authored half;
+  #   absent = the engine's generic prompt. `model:` (absent = the platform
+  #   utility_model) can name the distiller explicitly.
+end
+```
+
+The same keys work in a pack's `agent.config.json`. `idle_hours` /
+`min_messages` / `max_proposals` are per-agent data; `prompt` is the store's
+half — what counts as a fact for a fashion store ("size, preference, budget")
+is not what counts for a logistics one ("address, delivery window, carrier").
+The engine never writes store vocabulary.
+
+## The human gate, precisely
+
+The **Facts** page (operate group, next to Follow-ups) shows:
+
+- **Pending** — oldest first, because evidence ages. Each card is the fact,
+  its confidence, its scope (`tenant:customer`), and the evidence excerpt read
+  from the transcript at request time (evidence is a link, never a copy). Three
+  buttons: **Approve & save to memory**, **Reject** (with an optional reason,
+  shown on the card), **Dismiss** (ghost — labelled "will not be proposed
+  again").
+- **Stale** — the CAS-lost re-present (below): the proposed value struck
+  through next to the operator's current value, resolved by dismissal.
+- **Recent** — every resolved proposal, most recent first, with operator and
+  note.
+
+**The latch** — a dismissed *or* rejected `(name, value)` tuple is never
+proposed again. The proposal rows ARE the ledger: a human saw that tuple and
+said no, and re-proposing it would train the operator to stop reading. An
+unanswered proposal is never piled on, either. A *different* value for the same
+name is a different tuple — "wears M" dismissed does not block "wears L".
+
+**The CAS guarantee** — approval never silently overwrites an operator edit.
+At distill time the engine records the target fact's existence and revision; at
+approve time it writes through the store's optimistic compare-and-swap. A fact
+the operator moved in between flips the proposal to `stale` with both values
+visible — the operator's edit always wins, never a silent overwrite.
+
+## Provenance
+
+An approved fact is written with `origin: "distilled:<session_ref>"` — the
+RFC-0031 provenance discipline: `"engine"` (the `remember` tool), `"operator"`
+(Studio edits), `"legacy"`, and `"distilled"` (+ the session that produced it).
+The Closed loop reads the same cell every later turn injects, and the
+distillation ledger suppresses re-proposing a fact that is already applied with
+a `distilled:` origin.
+
+## LGPD
+
+Distilled facts are personal data, and the engine treats them like it:
+
+- **Forget a customer** — `forget_customer` purges the customer's proposals
+  (every status) along with their memory cell and sessions.
+- **Delete a tenant** — `delete_tenant_data` purges the tenant's proposals.
+- **Retention** — proposals age out under the same `retention_days` sweep as
+  the rest of the footprint (pending included — a proposal is evidence of a
+  transcript, and when the transcript dies the pending fact is stale). The
+  session marker dies with its proposals: **an unreviewed proposal that ages
+  out is expired, not lost-locked** — the session is re-distillable, and a
+  duplicate survivor is filtered by the ledger, never applied.
+
+## The honest limits
+
+- **Best-effort extraction, re-scan recovery.** There is no distillation
+  queue. A crash mid-pass leaves the session unmarked; the next pass re-scans
+  it, and the ledger filters any duplicate proposal. Exactly-once is not
+  claimed — facts are re-derivable, and a duplicate is filtered, never applied.
+- **Precision is a forge audit.** The engine guarantees the *gates* (schema,
+  dedup, CAS, human approval); it cannot guarantee the *model's judgment*.
+  "Is this fact true and durable?" is audited on real traffic, per store — the
+  pack prompt is where that judgment is tuned.
+- **Scope comes from the session, never the model.** The proposal's landing
+  cell is assembled by the engine from the session's tenant and customer; the
+  schema rejects a model-authored scope outright (a cross-tenant escape), and
+  an untagged session is never distilled.
+- **Sessions only.** Distillation targets customer-tagged sessions; a
+  session without a customer has no landing zone and is skipped.
+- **One stamp, once.** A session receives `vars["agent"]` on the same write
+  that stamps its `customer` — the first tagged turn. The stamp is
+  idempotent: a session that already carries a `customer` (a pre-upgrade
+  conversation, or one that since moved to this agent) never gains `agent`
+  on its own, so it can never participate in distillation.

data/docs/HARVEST.md

@@ -0,0 +1,117 @@
+---
+title: Harvest
+parent: Operate & prove it
+nav_order: 7
+permalink: /harvest/
+---
+
+# Harvest — skills from real traffic, promoted only if eval AND conversion hold
+
+The harvest is the loop that makes a store smarter with its own traffic: it
+reads the store's **finished** conversations, asks a model to propose SKILLS
+for the agent's playbook, filters every proposal through two pre-registered
+lists (the negative list, the evidence ledger), scores it with a double gate,
+and — **only after a human approves** — lands it as a live skill. Nothing is
+ever applied automatically, and a skill that fails the gate is terminal: the
+same finding must re-surface with new evidence, there is no silent retry.
+
+Per-store data (`harvest:` on the agent), absent = the loop is off for that
+agent:
+
+```ruby
+harvest enabled: true,
+        negative_list: [ { rule: "no-competitor-prices", pattern: "concorrente" } ],
+        miner: { model: "deepseek-v4-flash",    # absent = the platform utility_model
+                 window: { last_sessions: 200 },
+                 max_proposals: 10,
+                 budget: { tokens: 100_000 } },
+        idle_hours: 24,      # how idle a session must be before it mines
+        min_messages: 3      # a shorter session mines noise
+```
+
+## The pass
+
+A mine reads ONLY durable, finished data — never an in-flight turn, never a
+live prompt prefix (the "fork" is structural: the mining writes nothing to the
+sessions it read, so a customer turn's cache is untouched by construction). The
+miner gets the transcript slices (masked), the session's evidence-ledger ids,
+and the agent's current skill names so it does not re-propose them. Then the
+filters, each drop counted and logged:
+
+1. **The negative list** — the versioned rules file (seed, imported
+   per store by `insika harvest:negative import --agent ID --file F`) and the
+   profile's hot-editable `harvest.negative_list`. A rule is a phrase or a
+   regex; phrases match case/accent-folded at word boundaries. Every rejected
+   candidate is logged with the rule id.
+2. **The grounding filter** — every product reference in a proposal must be
+   in the union of the origin sessions' evidence ids (RFC-0029's ledger).
+   A store without `grounding.matcher.sku` does not mine at all: product
+   claims that cannot be verified are blocked by refusal, not by prompt.
+3. **Dedup** — an open `(agent, name)` tuple or a skill the store already has.
+
+## The double gate
+
+A candidate that survives mining is scored by RUNNING it — the eval gate
+clones the agent, writes the candidate skill into the clone's agent-scoped
+store, enables it on the clone's allowlist, replays the golden set over the
+ordinary public surface and compares to the accepted baseline. **Any
+regression disqualifies** — the gate is a veto, never a score to argue with.
+Judges are mandatory in exactly the shapes the refinement gate already
+refuses: no recorded baseline, an all-red baseline, and a judged baseline
+replayed with no judge (a rubric'd case with no verdict would count as a
+pass — the P18 lesson, see [Evals](EVALS.md)).
+
+The conversion gate is the second ruler: the store's funnel metric over the
+criterion's window, compared to the **frozen baseline** (the RFC-0032
+`freeze_funnel_baseline`). Outcome is evidence — this gate can only say "the
+store is measurably worse than the accepted state" or "there is nothing to
+compare against". It refuses on missing data, never passes: no frozen
+baseline, no criterion, no funnel store, a fold that has not converged — each
+named on the Harvest page.
+
+Both passed → the candidate awaits **a human**. Promotion is snapshot-first
+(the pre-promotion content + allowlist), then the two existing write commands,
+then an append-only log row carrying `skill`, `origin`, `eval_ref`,
+`conversion_ref`, `approver`, `snapshot_ref` and the criterion's sha. The
+conversion ruler is re-read at the moment the skill lands — a store that
+dipped below its frozen baseline since gating parks the promotion with the
+current numbers, and a criterion file that changed since boot is a criterion
+nobody froze.
+
+## Rollback
+
+One click, deterministic: the snapshot is restored — `WriteSkill` with the
+pre-promotion bytes (or `DeleteSkill` when the skill did not exist), the
+allowlist restored to the snapshot's set, and the promotion row stamped
+`rolled_back_at`. The log stays the single ledger: a skill promoted, rolled
+back, re-promoted is three readable rows.
+
+## The honest limits
+
+- **A store with no golden cases cannot gate, and cannot promote.** The gate's
+  strength is entirely the golden set.
+- **Grounding first.** A store without a matcher does not mine — the product
+  loop is blocked until the evidence ledger is live (by refusal, not by
+  warning).
+- **A promoted skill is live text**, like any skill: the SkillStore's version
+  history plus the harvest snapshot plus the log make the rollback path
+  deterministic.
+- **The first-10 audit is human.** The negative list can carry a false
+  restriction (a phrase too broad) — that is exactly the Hermes failure, and
+  it is caught by the audit, not by the engine: the list grows and the loop
+  stops (`harvest.enabled: false`), both by data.
+- **The harvest costs provider money**: one miner call per window plus one
+  full golden replay per gated candidate. The `budget` cap, the recorded run
+  cost and the manual trigger bound it; the automated loop mines one session
+  per claim window.
+
+## The operator surface
+
+- `insika harvest --agent ID [--last-sessions N] [--since ISO] [--full]` — mine one window.
+- `insika harvest:negative import --agent ID --file F` — seed the profile's list from a rules file.
+- `insika harvest:criterion check --file F` — strict-load the frozen conversion criterion (the hook before any promotion).
+- The **Harvest page** in the Studio: the human's inbox (each candidate with
+  its evidence excerpt, the eval report, the conversion card, promote/reject),
+  the gated-but-blocked rows with the named ruler hole, the pending list, the
+  append-only promoted log with the rollback mirror, the negative list with
+  per-rule rejection counts, and the criterion block read-only.

data/docs/LOADTEST.md

@@ -16,7 +16,7 @@ The whole point: the engine exposes `POST /v1/responses` as an **SSE drop-in** o
 the OpenClaw gateway. Same contract → the same load tools work against either side,
 so you can measure the engine you are about to ship against the gateway it replaces.
 
-There are three scripts, each answering a different question:
+There are four scripts, each answering a different question:
 
 | Script | Question it answers | Needs a provider? |
 |--------|---------------------|-------------------|
@@ -24,8 +24,13 @@ There are three scripts, each answering a different question:
 | `scripts/loadtest.rb` | End-to-end: TTFB/total/tokens/cache/error against `/v1/responses` | Yes |
 | `scripts/loadtest-local.sh` | Single-proc baseline vs N-worker multi-proc on one box | Yes |
 | `scripts/loadtest_session.rb` | A full multi-message session (CEP, searches, FAQ) under C concurrent sessions — direct to the engine (`--surface engine`, stream vs steer) or through the consumer's real ingress (`--surface web`, the consumer's widget API) | Yes |
+| `insika soak` | Does the deploy degrade over 72 h of steady load? | Yes |
 
-All three take `--help` / `-h`.
+The first four take `--help` / `-h`; the soak is a shipped command (`insika soak --help`) rather
+than a repo script. Bursts and uptime are different questions: a wave driver
+measures a burst, and the soak's arrival process measures *degradation over
+uptime* — the load-test table above deliberately stops where
+[Soak](SOAK.md) begins.
 
 ---
 
@@ -64,7 +69,7 @@ frame that carries it.
 INSIKA_URL=http://localhost:9292 \
 OPENCLAW_GATEWAY_TOKEN=xxx \
 bundle exec ruby scripts/loadtest.rb \
-  --agents bia,my-store --concurrency 16 --iterations 3 \
+  --agents demo,my-store --concurrency 16 --iterations 3 \
   --message "hi, how are you?"
 ```
 
@@ -75,7 +80,7 @@ Runs against a local server **or** a remote one (e.g. Railway) — just point
 
 | Flag | Default | Meaning |
 |------|---------|---------|
-| `--agents a,b,c` | `bia` | comma-separated agent ids (mapped to `model: openclaw:<agent>`) |
+| `--agents a,b,c` | `demo` | comma-separated agent ids (mapped to `model: openclaw:<agent>`) |
 | `--concurrency N` | `8` | concurrent turns per wave |
 | `--iterations N` | `1` | number of waves per agent |
 | `--message TEXT` | greeting | user message sent every turn |
@@ -98,7 +103,7 @@ Use `--dry-run` to sanity-check your flags/URL/token before firing real traffic
 
 ```bash
 INSIKA_URL=http://localhost:9292 OPENCLAW_GATEWAY_TOKEN=xxx \
-  bundle exec ruby scripts/loadtest.rb --agents bia --concurrency 16 --dry-run
+  bundle exec ruby scripts/loadtest.rb --agents demo --concurrency 16 --dry-run
 ```
 
 ### `--same-user` and the cache
@@ -130,7 +135,7 @@ DEEPSEEK_API_KEY=sk-... ./scripts/loadtest-local.sh [WORKERS] [CONCURRENCY]
 | `DEEPSEEK_API_KEY` | — (required) | real turns hit the provider; also auto-sourced from `.env.local` |
 | `OPENCLAW_GATEWAY_TOKEN` | falls back to `ADMIN_TOKEN`, then `local-demo` | Bearer for the sweep |
 | `PORT` | `9299` | bind port for the local Falcon |
-| `AGENT` | `bia` | agent id to load |
+| `AGENT` | `demo` | agent id to load |
 
 The final block prints the lock counts; the expected reading is `0` for both:
 
@@ -163,7 +168,7 @@ drop-in for the gateway, you only change **where it points**:
 # In the OpenClaw checkout, run its gateway loadtest against the HARNESS:
 OPENCLAW_GATEWAY_URL=http://localhost:9292 \
 OPENCLAW_GATEWAY_TOKEN=<same bearer the engine accepts> \
-node scripts/loadtest-gateway.mjs --agents bia --concurrency 16 --iterations 3
+node scripts/loadtest-gateway.mjs --agents demo --concurrency 16 --iterations 3
 ```
 
 Then run the exact same command with `OPENCLAW_GATEWAY_URL` pointing at the real
@@ -174,7 +179,7 @@ gateway, and diff the two reports. This is the shadow comparison the pilot needs
 - The OpenClaw checkout containing `scripts/loadtest-gateway.mjs` and Node installed.
 - A **bearer token accepted by both** sides. For the engine that is
   `OPENCLAW_GATEWAY_TOKEN` (see DEPLOY.md); point the gateway run at its own token.
-- **The same agent id provisioned on both** sides (e.g. `bia`) so `model:
+- **The same agent id provisioned on both** sides (e.g. `demo`) so `model:
   openclaw:<agent>` resolves on each. On the engine, provision via
   `scripts/import_pack.rb`.
 - The **same provider** (or an equivalent-latency one) behind each, otherwise you
@@ -224,6 +229,7 @@ Work top-down and **measure before assuming** — avoid premature topology optim
 | 4 | Insika vs gateway, identical knobs | §4 (either method) | Is the engine at parity with the engine it replaces before cut-over? |
 | 5 | Cold vs hot conversation (cache) | `loadtest.rb` with/without `--same-user 1` | Expected steady-state cost/latency once conversations warm up. |
 | 6 | Remote (Railway) vs local | `loadtest.rb` with `INSIKA_URL` remote | Network/deploy overhead of the real environment. |
+| 7 | Degradation over uptime (72 h) | `insika soak` (see [Soak](SOAK.md)) | The cut argument: does latency or memory grow with uptime? Run it last — it needs the topology settled first. |
 
 **Reaching for horizontal scale is only justified after 1–3 show the single box is
 the limit.** If it is, the paths are: sharding-by-tenant +

data/docs/OBSERVABILITY.md

@@ -41,7 +41,18 @@ something that left no task of its own behind:
 | `:delivery_failed` | `channel`, `outbox_id`, `status`, `attempts`, `error` | a delivery exhausted its bounded retries — the alert face of the row above (WS6) |
 | `:budget_warning` | `agent`, `tenant`, `window`, `spent`, `cap` | a calendar budget crossed its threshold (`alert_at` or a soft cap) — once per window (WS2) |
 | `:breaker_open` | `agent`, `ref`, `tenant` | the reliability circuit breaker tripped for a `(tenant, provider/model)` — further turns fail fast until the cooldown (WS3/WS6) |
+| `:provider_failure` | `agent`, `ref`, `error`, `kind` | one attempt against `ref` failed and spent a retry — emitted with or without a circuit breaker (WS3) |
+| `:provider_fallback` | `agent`, `from`, `to`, `error`, `kind` | the turn ROTATED mid-flight to the next node of the fallback chain, and the error that caused it (WS3) |
 | `:ttft` | `task_id`, `session_id`, `ttft_ms` | the provider's time-to-first-token on the streaming envelope — only under `INSIKA_TURN_TIMING`, once per turn (WS6) |
+| `:distillation_completed` | `session_ref`, `agent`, `proposals`, `dropped{}`, `deduped`, `cost{}` | a session's traffic was distilled into N proposals (RFC-0034) — counts and ids only, never a fact value |
+| `:proposal_approved` | `proposal_id`, `status`, `operator` | a human approved a distilled fact on the Facts page — written to memory via CAS |
+| `:proposal_rejected` | `proposal_id`, `status`, `operator` | a human rejected it (the reason is on the proposal record, not in the event) |
+| `:proposal_dismissed` | `proposal_id`, `status`, `operator` | a human dismissed it — the tuple is latched, never proposed again |
+| `:proposal_stale` | `proposal_id`, `status`, `operator` | the CAS lost: the fact moved after distillation — re-presented with both values on the wiki, never silently overwritten |
+| `:harvest_mined` | `agent`, `run_id`, `candidates`, `rejected{}`, `cost{}` | a mining pass finished (RFC-0035) — counts and rule ids only, never a skill body |
+| `:harvest_gated` | `run_id`, `candidate_id`, `agent`, `eval_passed`, `conversion_passed`, `reason` | the double gate's verdict on one candidate — ids and verdicts only |
+| `:skill_promoted` | `agent`, `skill`, `candidate_id`, `snapshot_ref`, `promotion_ref`, `approver` | a human approved a mined skill — it is live for the store, with the snapshot for rollback |
+| `:skill_rolled_back` | `snapshot_ref`, `skill`, `agent`, `operator` | the snapshot was restored — the promotion row carries the `rolled_back_at` stamp |
 
 `delivery_failed` and `breaker_open` are the two the operator config is pointed at
 (`alerts.webhook` on the profile): each only fires when something durable did
@@ -366,6 +377,49 @@ consumer that fell far behind would have its subscription closed (telemetry stop
 the turn does not). Span and instrument operations are cheap, so there's ample
 headroom.
 
+## Process vitals — `GET /v1/vitals`
+
+OTel carries turn/tool telemetry; it says nothing about the **process**. For the
+questions a soak (or any operator) asks — *which process is this, how long has it
+been up, how much memory does it hold, and what is the Ruby heap doing?* — there
+is one read-only route:
+
+```bash
+curl -H "Authorization: Bearer $TOKEN" https://<target>/v1/vitals
+```
+
+```jsonc
+{
+  "boot_id": "20260820T09-…",     // one per container start; a change = a restart
+  "pid": 42,
+  "started_at": "2026-08-20T09:00:00Z",
+  "uptime_s": 259200,
+  "version": "0.2.0",
+  "ruby": "ruby 3.4.1 …",
+  "yjit": true,
+  "rss_bytes": 536870912,          // nil when unreadable — never a guess
+  "gc": { "heap_live_slots": …, "major_gc_count": …, "malloc_increase_bytes": … },
+  "threads": 8,
+  "in_flight": 1,                  // the executor's in-flight turns
+  "db_bytes": { "db": …, "wal": …, "shm": … },
+  "at": "2026-08-20T09:00:00Z"
+}
+```
+
+The two fields that make it a **restart detector**: `boot_id` (one per container
+start, exported by the entrypoint and shared by every worker) and `pid`. A
+`boot_id` change is a container restart; a `pid` change under the same `boot_id`
+is a worker respawn — the event a platform metrics API cannot see.
+
+- **Operator-only.** The route is not in the public allowlist (no bearer →
+  unauthorized) and not on the tenant surface, so only an operator reads
+  process internals. `/up` stays the public health probe and carries no
+  process data.
+- **Reads no store.** Pure OS/VM readings — safe to poll at any rate, and it
+  cannot contend with turns.
+- **The soak's sampler.** [Soak](SOAK.md) polls it hourly; a `nil` RSS reads as
+  missing coverage, never as zero.
+
 ---
 
 *Packaging note.* Today the bridge lives in the Insika repo as an opt-in core