insika 0.3.0 → 0.8.0

data/docs/DEPLOY.md

@@ -25,7 +25,7 @@ is durable SQLite (WAL) at `INSIKA_DB`; mount a volume and point it inside.
 docker build -t insika .
 docker run -p 9292:9292 -v insika-data:/data \
   -e DEEPSEEK_API_KEY=sk-... \
-  -e OPENCLAW_GATEWAY_TOKEN=change-me \
+  -e INSIKA_GATEWAY_TOKEN=change-me \
   insika
 curl localhost:9292/up      # {"status":"ok"}
 ```
@@ -57,7 +57,14 @@ contract:
    worker that holds the session's actor. The engine does **not** promise them
    across workers. A deploy that needs those semantics for a session must route
    that session's traffic to one worker (sticky routing) — or accept per-worker
-   best-effort.
+   best-effort. **On Railway there is no sticky-routing option at any layer** —
+   confirmed against Railway's own docs, which distribute traffic randomly and
+   explicitly do not support sticky sessions — so N>1 there is not "per-worker
+   best-effort," it is a guaranteed cross-session leak the first time two
+   requests for the same session land on different workers. See the Railway
+   section below; `insika doctor` errors on `WEB_CONCURRENCY>1` there — unless
+   [`insika-router`](ROUTER.md), the session-sticky proxy, is in front,
+   which is what makes N>1 safe on Railway too (N local workers behind it).
 3. **Recovery is part of boot, in every wiring.** Every worker boots through
    `Server::Boot`, which runs recovery **before the listen**. The per-record
    sweeps (undelivered outbox records, undelivered delegation results) run in
@@ -108,7 +115,7 @@ this section is the single source of truth for what changing it means.
 | `INSIKA_DRAIN_TIMEOUT` | `20` | seconds a stopping worker waits for in-flight turns before abandoning them to the next boot's recovery (process model, item 4). The entrypoint sizes Falcon's `--graceful-stop` from it; on Railway also set `RAILWAY_DEPLOYMENT_DRAINING_SECONDS` ≥ drain + 10 |
 | `INSIKA_TICK_INTERVAL` | `60` | seconds between tick passes — outbox drain + stale recovery sweep (process model, item 5). `0` disables |
 | `INSIKA_TICK_STALE_AFTER` | `900` | seconds a `:queued`/`:running` task must sit untouched before the tick sweeps it. Must exceed the largest `turn_timeout` of the deployment |
-| `OPENCLAW_GATEWAY_TOKEN` | falls back to `ADMIN_TOKEN` | Bearer for `/v1/responses` and `/v1/agents` (the API contract) |
+| `INSIKA_GATEWAY_TOKEN` | falls back to `ADMIN_TOKEN` | Bearer for `/v1/responses` and `/v1/agents` (the API contract) |
 | `ADMIN_TOKEN` | `local-demo` | login token for `/studio` (**change in production**) |
 | `DEEPSEEK_API_KEY` | — | provider key. **Without it the engine still boots** (`/up` green), but turns fail until it is configured (env or Studio → LLM providers) — cloud resilience |
 | `DEEPSEEK_MODEL` | `deepseek-v4-flash` | model |
@@ -118,7 +125,7 @@ this section is the single source of truth for what changing it means.
 | `INSIKA_RELAY_TOKEN` | — | **mounts the relay channel** at `POST /channels/relay/events`, and is the Bearer it requires. Empty = the route does not exist (`404`). See [Channels](CHANNELS.md) |
 | `INSIKA_RELAY_DELIVER_URL` | — | your callback; the engine POSTs each reply there. Goes through the egress guard |
 | `INSIKA_RELAY_DELIVER_TOKEN` | — | Bearer the engine sends **to** your callback (optional) |
-| `INSIKA_RELAY_DELIVERY` | `at_end` | how the relay flushes the outbox: `at_end` (one POST) or `progressive` (one POST per balloon — [RFC-0027](CHANNELS.md#delivery-policy)) |
+| `INSIKA_RELAY_DELIVERY` | `at_end` | how the relay flushes the outbox: `at_end` (one POST) or `progressive` (one POST per balloon — [delivery policy](CHANNELS.md#delivery-policy)) |
 | `INSIKA_WIDGET_ORIGINS` | — | exact-match origins allowed to embed the [web widget](CHANNELS.md#the-web-widget), comma-separated. No wildcards. **Half the switch**: with `INSIKA_WIDGET_AGENTS` unset, nothing is mounted (`404`) |
 | `INSIKA_WIDGET_AGENTS` | — | agent ids a widget visitor may address, comma-separated. The other half of the switch. **A chat rate limit is also required** or the widget answers `503` |
 | `LITESTREAM_REPLICA_URL` | — | **enables Litestream** (backup/DR). Empty = disabled (default). See below |
@@ -150,7 +157,7 @@ values (the API token falling back to `ADMIN_TOKEN` is a dev convenience only):
 - **`ADMIN_TOKEN`** — the `/studio` login (cookie auth). This is the **operator**
   surface (just you). Rotating it is **safe and independent**: change it, redeploy,
   log in with the new value. It does not affect any API consumer.
-- **`OPENCLAW_GATEWAY_TOKEN`** — the Bearer for `/v1/responses` and `/v1/agents`.
+- **`INSIKA_GATEWAY_TOKEN`** — the Bearer for `/v1/responses` and `/v1/agents`.
   This is the **contract with your API consumers**. Rotating it means **changing
   both sides together** (or the integration breaks): update the runtime var **and**
   each consumer's token in the same step.
@@ -180,15 +187,24 @@ insika doctor            # colored report; exits != 0 on any error
 insika doctor --json     # machine-readable (CI / monitoring)
 insika doctor --fix      # applies the safe autofixes and re-diagnoses
 insika env               # lists known keys + current values (secrets masked)
+insika tools:report      # tool audit over the stored traces: never-called
+                         # allowlisted tools, error rate > 30%, stale tools —
+                         # read-only, the operator removes ([--agent ID] [--days N] [--json])
 ```
 
 Checks: env (the schema above), settings schema version (a pending migration →
 `--fix` applies it), a missing platform `default_model` (`--fix` seeds it from
 `DEEPSEEK_MODEL`), durable vs ephemeral backend, LLM provider configured,
-`ADMIN_TOKEN` set, data-tool definitions still valid, **prompt files that hold
+`ADMIN_TOKEN` set, data-tool definitions still valid, **a stored agent whose
+declared tool allow/deny list does not name the `tool_allowlist` policy** (the
+engine repairs it on read, but as stored the list is inert — see
+[Agents](AGENTS.md#the-allowlist-convention)), **prompt files that hold
 text rather than a serialized object** (a file whose content is a stringified Hash
 serves a mangled prompt on every turn while looking perfectly healthy — present,
-non-empty, and the agent still answers), and **skill drift** — a shared skill whose
+non-empty, and the agent still answers), **a prompt file that outgrew a prompt**
+(WARN past ~6 000 estimated tokens or 600 lines — the LLM-generated pack shape
+that costs 20%+ extra tokens per turn for no better instruction-following), and
+**skill drift** — a shared skill whose
 body names one store, a prompt file routing to a skill the agent cannot load, a broken
 companion pair, a stale `eager:` key (see
 [Skills](SKILLS.md#drift-guards)). Settings-schema migrations are **explicit**
@@ -220,12 +236,65 @@ healthcheck, and a restart policy.
 2. **Volume**: mount it at `/data` (the default `INSIKA_DB` points there) —
    without a volume, SQLite is ephemeral and recovery resumes nothing after a
    redeploy.
-3. **Vars**: `DEEPSEEK_API_KEY`, `OPENCLAW_GATEWAY_TOKEN`, `CONSUMER_INTERNAL_URL`,
-   `INSIKA_EGRESS_HOSTS` (and `WEB_CONCURRENCY` to match your plan/CPU).
+3. **Vars**: `DEEPSEEK_API_KEY`, `INSIKA_GATEWAY_TOKEN`, `CONSUMER_INTERNAL_URL`,
+   `INSIKA_EGRESS_HOSTS`. **Leave `WEB_CONCURRENCY` at its default of 1** unless
+   you run [`insika-router`](ROUTER.md) in front. Railway's own docs say it
+   "does not support sticky sessions" and randomly distributes traffic across
+   replicas/workers — there is no way on this platform to satisfy the "sticky
+   routing per session in front" precondition item 2 of the process model
+   requires, at any layer (Railway replicas or Falcon's own `--count` workers
+   within one container). Raising `WEB_CONCURRENCY` without a router in front
+   is not a throughput knob, it is a guaranteed way to leak a reply across
+   sessions the first time two requests for the same session land on different
+   workers — `insika doctor` errors on this combination for exactly that
+   reason. `insika-router` is the fix: N local Falcon workers
+   behind one sticky proxy, still one Railway replica.
 4. The healthcheck hits `/up`.
 5. Point your consumer at the service's public URL, with a matching API token
    (see [RUNNING-LOCAL.md](RUNNING-LOCAL.md)).
 
+## Kubernetes
+
+Same contract as Railway, for the same reason, verified against
+ingress-nginx's own docs: **run one pod, `WEB_CONCURRENCY=1`.** Scale
+vertically (bigger pod) for now.
+
+Why a standard K8s setup does not get you out of this: the session id lives
+in the **JSON body** of `POST /v1/responses` (the `user` field), not in a
+header, cookie, or URL segment. That rules out every sticky-routing
+mechanism K8s gives you for free:
+
+- A vanilla `Service` (`ClusterIP`) load-balances across pod endpoints with
+  no notion of session at all — same failure mode as Railway's replicas,
+  one layer down.
+- `Service.spec.sessionAffinity: ClientIP` does not help either: the caller
+  is the consumer's own backend (achei-b2b, a server-to-server call), not
+  the end customer's browser, so many different customers' sessions arrive
+  from the same source IP and would collide on the same pod instead of
+  spreading — the opposite of what you want, and it still does not restore
+  a real per-session guarantee.
+- ingress-nginx's `nginx.ingress.kubernetes.io/upstream-hash-by` (the
+  standard sticky/consistent-hash annotation) hashes on nginx *variables* —
+  headers, cookies, `$request_uri`, client IP — not on a field parsed out of
+  a POST body. Reaching into the JSON body needs a custom Lua/OpenResty
+  snippet (or an Envoy filter) that parses the request and extracts `user`
+  before hashing. That is real, unbuilt engineering work, not a K8s
+  annotation to flip.
+
+If real horizontal throughput is ever needed, two paths get there, neither
+of them "just add replicas":
+1. **Build the body-aware sticky layer** above (Lua/Envoy consistent-hash on
+   the `user` field) in front of N pods, each still `WEB_CONCURRENCY=1`.
+2. **Move the session id into the URL**, the way the web widget transport
+   already does (`POST /api/widget/sessions/:token/messages` carries the
+   session token in the path) — that surface, unlike `/v1/responses`, *is*
+   sticky-routable today with a plain `upstream-hash-by: $request_uri`, no
+   custom scripting required.
+
+Neither is urgent while a single pod's throughput is enough — this section
+exists so scaling this deployment does not silently reintroduce the same
+cross-session leak the Railway incident above already found once.
+
 ## Backup / DR — Litestream (opt-in, configurable)
 
 A single volume is the **one point of total loss** between a pilot and production
@@ -300,6 +369,12 @@ per pod + **sticky-by-agent** routing (shard by tenant), or **LiteFS**, or an
 optional **Postgres** adapter. **Litestream** (above) for backup/DR from day one —
 orthogonal to topology.
 
+For the *session-routing* half specifically — as opposed to the storage
+topology above — see [`insika-router`](ROUTER.md): N pods behind a
+headless `Service`, one `insika-router` Deployment in front doing the
+consistent-hash routing ingress-nginx's own `upstream-hash-by` cannot (it
+hashes nginx variables, never a field parsed out of a POST body).
+
 ---
 
 ## Measuring performance / load
@@ -336,7 +411,7 @@ cache hits, P50/P95, error rate. Runs against local or a remote deployment. See
 [LOADTEST.md](LOADTEST.md).
 
 ```bash
-INSIKA_URL=http://localhost:9292 OPENCLAW_GATEWAY_TOKEN=xxx \
+INSIKA_URL=http://localhost:9292 INSIKA_GATEWAY_TOKEN=xxx \
   bundle exec ruby scripts/loadtest.rb --agents assistant --concurrency 16 --iterations 3
 ```
 
@@ -351,6 +426,8 @@ DEEPSEEK_API_KEY=sk-... ./scripts/loadtest-local.sh 4 24
 
 ## See also
 
+- [ROUTER.md](ROUTER.md) — `insika-router`, the session-sticky proxy for
+  scaling past one worker.
 - [RUNNING-LOCAL.md](RUNNING-LOCAL.md) — run the engine locally, single-process.
 - [Security](SECURITY.md) — tokens, egress, strict config.
 - [BENCHMARK.md](BENCHMARK.md) — the neutral, key-free engine benchmark.

data/docs/EMBEDDING.md

@@ -1,6 +1,6 @@
 ---
 title: Embedding
-parent: Ship it
+parent: Integrate
 nav_order: 4
 permalink: /embedding/
 ---

data/docs/EVALS.md

@@ -1,7 +1,7 @@
 ---
 title: Evals
-parent: Operate & prove it
-nav_order: 4
+parent: Improve
+nav_order: 1
 permalink: /evals/
 ---
 
@@ -157,6 +157,82 @@ Three rules make the number worth quoting:
 Cost: **two provider calls per judge per case**, which is why it is opt-in and never
 part of the gate.
 
+## Simulated users — conversations the agent never had
+
+A golden case fixes the user's turns in advance. Real customers branch — they answer
+the agent's question, or ignore it, or send an order number three messages later. A
+case with frozen turns can only ever test the path the author imagined.
+
+A **simulated** case generates the conversation instead: a persona model (the cheap
+platform `utility_model`) plays a customer with the persona as its **whole
+instruction**, and the target agent answers through the same transport a replay uses.
+They talk until the persona's `max_turns`, or until the persona emits a stop marker —
+`<<goal_met>>` when its goal is served, `<<gave_up>>` when it abandons. Both are
+recorded: "gave up at turn 3" is a finding.
+
+```yaml
+id: loja-objetivo-difuso
+agent: loja-chocolates
+persona:                          # the SIMULATED customer (alternative to `turns:`)
+  goal: "descobrir o que comprar de presente; não sabe nomes de produto"
+  style: "mensagens curtas, responde o que perguntam, desiste se tiver que explicar duas vezes"
+  opens_with: "oi, queria um presente"
+  knows:                          # the ONLY facts the persona may assert
+    orcamento: "até R$ 100"
+    ocasião: "aniversário"
+  max_turns: 8
+expect:
+  policy: investigate_first
+  rubric: |
+    Descobre o objetivo antes de recomendar (uma ou duas perguntas, não um formulário),
+    recomenda do catálogo real e fecha com um próximo passo claro. Reprova se despejar
+    catálogo antes de entender, ou se ficar perguntando sem nunca buscar.
+  min_score: 0.7
+```
+
+**The anti-invention rule is the soul of the feature.** The persona's prompt contains
+only the `knows` facts and the rule that it may assert exactly those and nothing else
+— asked about anything it does not have, it answers with ignorance ("não sei", "não
+tenho isso aqui"), like a real customer without that fact. A simulator that invents an
+order number produces a conversation the agent could never have had, and a case that
+tests nothing.
+
+```bash
+insika evals:simulate --persona persona.yml --target loja-chocolates --staging
+```
+
+- `--target <agent|url>` — an agent id over the deployment's `/v1/responses`, or a full
+  A2A URL (an agent that only speaks A2A rides the same Simulator through a thin A2A
+  transport; it polls once per second, bounded by `--timeout`, since a remote agent
+  takes seconds to finish a task).
+- `--staging` or `--eval-profile` — **one is required**. A simulated conversation
+  marks its transcript `simulated: true`; it does NOT disarm the agent's tools. So the
+  Simulator only runs against (a) a target the operator declares is a staging
+  deployment (`--staging`), or (b) an eval profile where the agent's side-effect tools
+  are swapped for dry-runs (`--eval-profile`).
+- `--eval-profile` is a **verified declaration, not a trust-me flag**. The CLI derives
+  the target's side-effect tools — the deployment's own registry over
+  `GET /v1/agents/:id` (`side_effect_tools`), or the local store when `INSIKA_DB`
+  points at it — and refuses unless `--eval-tools <a,b,c>` names every one of them.
+  The swap itself happens where the tools run (the deployment's eval profile; the
+  in-process overlay for a local graph, as in `examples/agent-tester/`); the
+  derivation is what keeps the client honest about what must be covered. An A2A target
+  has no reachable registry, so the explicit `--eval-tools` list is required there.
+- `--persona-model` — the model playing the customer (default: the platform
+  `utility_model`). The judge that scores the transcript comes from the same
+  panel configuration as a replay.
+- `--conv` — the conversation id. The default is unique per run
+  (`sim-<case id>-<random>`), because two runs on the same conv share the
+  deployment session: the second inherits the first's history — including
+  across the arms of an A/B comparison — and the grade stops measuring the
+  agent. Pass `--conv` only when continuing a session is the point.
+
+Every simulated run is **`simulated: true`** — the transcript carries the flag end to
+end, so a report never mixes a generated conversation with real traffic. A simulated
+case (`persona:`) is one of the two shapes of a case; the replay Runner **skips** it
+(the turns do not exist until a Simulator generates them) and only the simulate CLI
+drives it.
+
 Cases live in two places, and it is the same YAML in both:
 
 - **`evals/golden/**`** in the repo — the curated corpus, reviewable in a pull request,
@@ -178,6 +254,55 @@ A case whose stored YAML no longer validates is **listed as broken** on the Eval
 rather than skipped in silence — a test suite that quietly shrinks is worse than a red
 one.
 
+### `run_persona_eval` — a QA agent that tests other agents
+
+A tool (`lib/insika/tools/run_persona_eval.rb`), not a CLI: a QA agent picks one of its
+authored simulated cases and runs it **in-process**, against the case's declared
+target agent — the same Simulator + Judge machinery `evals:simulate` drives over HTTP,
+minus the network hop.
+
+```ruby
+tools %w[run_persona_eval save_artifact]
+```
+
+The model only sees `case_id`, enumerated with the ids the tool can actually run
+(never a free string it could invent) — every **simulated** case in the store, the same
+`persona:` shape as above.
+
+**Safety is derived here too, but there is no swap yet.** The tool computes the
+target's reachable side-effect tools (`Evals::EvalProfile`, the same derivation the CLI
+uses) and **refuses outright** — naming the tools — if that list is non-empty. Unlike
+the CLI, nothing here actually swaps a side-effect tool for a dry-run: `Evals::
+EvalProfile.registry` (the overlay) exists for exactly that, but nothing calls it yet.
+So `run_persona_eval` only runs against **read-only** target agents today; wiring the
+overlay into an in-process turn (so a target WITH a write tool can be tested safely) is
+follow-up work, not something this tool claims to do.
+
+**Budget**: the persona model + judge model calls are the cost of running the eval,
+charged to the **calling** agent's own turn — never the target's (the target's own
+turns bill normally, through the ordinary edge limiter, exactly as a real customer's
+would). A hard cap on the calling agent skips the run — visibly, in the tool's own
+result (`{skipped: true, reason: "budget", window: "daily"|"monthly"}`) — before a cent
+is spent; a soft cap just runs (the ledger's own alert already warns).
+
+Each run gets a **fresh session** (`eval-<case>-<random>`), never a reused one — a
+reused session lets the target agent "remember" earlier runs, which is not what a
+persona case is testing. See `examples/agent-tester/qa_scheduled.rb` for the full
+loop: `schedule` fires the QA agent, it calls `run_persona_eval`, then
+`save_artifact` publishes the quality report.
+
+**Tenant isolation.** A golden case carries a `tenant` (`platform` — the
+single-tenant default — unless the case declares one, the same convention
+`save_artifact` uses for its own binding tenant). `run_persona_eval` only ever
+lists or runs cases in the *calling* agent's own tenant: a case authored for
+another tenant is invisible, not merely refused — the model cannot even learn
+its id from the enum, and running it by a guessed id gets the same "unknown or
+invalid" error a nonexistent id would (the two must not be distinguishable).
+This is what makes "one QA agent per store" an actual boundary
+in a deployment where several stores' persona cases live in the same
+`GoldenStore` — without it, `qa-store-a` could enumerate and run
+`qa-store-b`'s persona and read its `knows` in the transcript.
+
 ## Running
 
 ```bash
@@ -235,7 +360,7 @@ not a regression.
 The same gate machinery guards the two automated loops — [Refinement](REFINEMENT.md)
 (cloned-agent replay of a proposed edit) and [Harvest](HARVEST.md) (cloned-agent
 replay of a mined skill). **Judges are mandatory for those gates** in exactly three
-shapes, the P18 lesson: a gate without a recorded baseline refuses, an all-red
+shapes: a gate without a recorded baseline refuses, an all-red
 baseline refuses, and a baseline recorded WITH judge scores, replayed with no judge
 configured, refuses — a rubric'd case with no verdict reads as a pass, so the
 candidate would beat a measurement it never took. A store with no golden cases cannot

data/docs/FACTS.md

@@ -1,7 +1,7 @@
 ---
 title: Facts
-parent: Operate & prove it
-nav_order: 6
+parent: Improve
+nav_order: 5
 permalink: /facts/
 ---
 
@@ -92,7 +92,7 @@ visible — the operator's edit always wins, never a silent overwrite.
 ## Provenance
 
 An approved fact is written with `origin: "distilled:<session_ref>"` — the
-RFC-0031 provenance discipline: `"engine"` (the `remember` tool), `"operator"`
+The provenance discipline: `"engine"` (the `remember` tool), `"operator"`
 (Studio edits), `"legacy"`, and `"distilled"` (+ the session that produced it).
 The Closed loop reads the same cell every later turn injects, and the
 distillation ledger suppresses re-proposing a fact that is already applied with

data/docs/HARVEST.md

@@ -1,7 +1,7 @@
 ---
 title: Harvest
-parent: Operate & prove it
-nav_order: 7
+parent: Improve
+nav_order: 6
 permalink: /harvest/
 ---
 
@@ -44,7 +44,7 @@ filters, each drop counted and logged:
    regex; phrases match case/accent-folded at word boundaries. Every rejected
    candidate is logged with the rule id.
 2. **The grounding filter** — every product reference in a proposal must be
-   in the union of the origin sessions' evidence ids (RFC-0029's ledger).
+   in the union of the origin sessions' evidence ids (the evidence ledger).
    A store without `grounding.matcher.sku` does not mine at all: product
    claims that cannot be verified are blocked by refusal, not by prompt.
 3. **Dedup** — an open `(agent, name)` tuple or a skill the store already has.
@@ -59,11 +59,10 @@ regression disqualifies** — the gate is a veto, never a score to argue with.
 Judges are mandatory in exactly the shapes the refinement gate already
 refuses: no recorded baseline, an all-red baseline, and a judged baseline
 replayed with no judge (a rubric'd case with no verdict would count as a
-pass — the P18 lesson, see [Evals](EVALS.md)).
+pass, see [Evals](EVALS.md)).
 
 The conversion gate is the second ruler: the store's funnel metric over the
-criterion's window, compared to the **frozen baseline** (the RFC-0032
-`freeze_funnel_baseline`). Outcome is evidence — this gate can only say "the
+criterion's window, compared to the **frozen baseline** (`freeze_funnel_baseline`). Outcome is evidence — this gate can only say "the
 store is measurably worse than the accepted state" or "there is nothing to
 compare against". It refuses on missing data, never passes: no frozen
 baseline, no criterion, no funnel store, a fold that has not converged — each