insika 0.0.1 → 0.2.0

data/docs/EVALS.md

@@ -0,0 +1,273 @@
+---
+title: Evals
+parent: Operate & prove it
+nav_order: 4
+permalink: /evals/
+---
+
+# Evals
+
+An agent has no unit test. The same prompt, the same tools and the same model can give
+a different answer twice, so "does it still work?" cannot be answered by asserting on a
+string. What you can do is keep a small set of **conversations you care about**, replay
+them against a running deployment, and check two different kinds of thing:
+
+- **deterministic** — did it call `search_voucher`? did the reply leak a CPF? did a
+  tool error? No model involved, no flakiness, no token cost.
+- **subjective** — did it actually resolve the doubt, without inventing a discount?
+  That one needs a reader, so a model reads it against a **rubric you write**.
+
+That is the whole idea. The rest of this page is the format, who grades, and how a run
+becomes a gate.
+
+## A case is data
+
+```yaml
+id: loja-chocolates-cupom
+agent: loja-chocolates
+turns:
+  - user: "tem algum cupom de desconto ativo?"
+expect:
+  tools_called:
+    - search_voucher        # a trailing "?" marks it optional
+  must_not:
+    - pii_leak
+    - tool_error
+  rubric: |
+    Consults the active coupon with the tool and says what it is — does NOT invent a
+    code or a percentage. If there is none, says so kindly.
+  min_score: 0.7
+```
+
+Turns replay **in order** under one conversation, so a case can build context ("what
+about the shipping?" after "I want the 70% bar"); the assertions run on the last turn.
+
+### `requires` — a case that cannot run here is skipped, not failed
+
+Deployments differ. Some stores have order tracking wired, some do not; some run
+promotions, some do not. A case asserting `search_orders` is **not a failure** for a
+store without it — it is a case that should never have run, and a red suite nobody
+trusts is worse than a small one.
+
+```yaml
+requires:
+  tools: [search_orders]        # must be available to the agent
+  capabilities: [promotions]    # facts the operator declared about the deployment
+expect:
+  tools_called:
+    - search_orders
+```
+
+The runner asks the deployment what that agent has (`GET /v1/agents/:id`, gated like
+every `/v1` read) **before** spending a turn, and reports a third outcome beside pass
+and fail: **skipped, with the reason**. The rule of thumb: a case that asserts a tool
+should require it.
+
+Capabilities are a flat list the operator writes on the agent — inferring "this store
+has promotions" from data is how a suite starts lying:
+
+```ruby
+declares "promotions", "human_handoff"
+```
+
+Three deliberate edges:
+
+- An agent with an **open** tool allowlist runs the case. "I could not rule it out" is
+  not a reason to stop testing something.
+- If the deployment cannot be read at all, cases with `requires` **run**, and the CLI
+  says so once. A suite must not shrink in silence.
+- The gate never blocks *on* a skip, and `--update-baseline` leaves skipped cases out
+  of the baseline entirely. But a case that used to pass here and is now skipped **is**
+  reported as a regression: it means the agent lost a tool.
+
+### `policy` — how much the agent should ask before acting
+
+One optional key, because this is the thing a rubric cannot carry alone. Whether the
+agent should establish the objective before searching, or act on the first plausible
+reading, is a decision **your store** makes — a universal rule would be wrong for half
+of them. Declare it and two things happen: a check that costs nothing runs, and the
+judge is told the rule instead of guessing it.
+
+```yaml
+expect:
+  policy: ask_once
+```
+
+| policy | the check | the judge is told |
+|---|---|---|
+| `ask_once` | no reply asks more than one question | two questions in one message is a failure |
+| `investigate_first` | turn 1 asks something and calls no tool | ask on a vague request, don't search immediately |
+| `act_fast` | turn 1 calls a tool | asking what a search would answer is a failure |
+
+Omit it and only the rubric decides. Unlike the other assertions, a policy is checked
+on **every** turn — "one question per reply" is a rule about each reply, and in the
+case that motivated this the violation was on the first one.
+
+Question counting is deliberately crude: a run of `?` counts once, and URLs are dropped
+so a tracking link's query string is not read as the agent asking something. It is a
+policy signal, not grammar — and crude was enough to catch an agent breaking a rule
+written in its own prompt, twice, with no model in the loop.
+
+### `reference` — compared against the system you want to replace
+
+`min_score: 0.7` says a reply cleared a bar you invented. It says nothing about
+whether the system already answering your customers would have done better. If you are
+replacing something, that is the only question that matters — and you have its
+transcripts.
+
+Give a case the incumbent's real conversation for the same opening, and a run with
+`--pairwise` asks one judge one question: **which one served the customer better?**
+
+```yaml
+reference:
+  source: "helpdesk chat 34403117"   # free text, so a reader can find the original
+  messages:
+    - role: user
+      text: "tem creatina?"
+    - role: assistant
+      text: "temos sim! qual seu objetivo?"
+    - role: assistant
+      text: "segue o link do produto"
+      origin: operator               # a HUMAN typed this one
+```
+
+```bash
+ruby evals/run.rb --agent loja-chocolates --pairwise
+```
+
+Three outcomes — `better`, `comparable`, `worse` — and two more the panel can produce
+and the report will not hide: `split` when the judges disagree, `unknown` when none of
+them answered readably. **It never changes pass/fail.** "Worse than the incumbent" is
+an answer about a replacement decision, not a regression in your suite, and it stays
+out of the gate.
+
+Three rules make the number worth quoting:
+
+- **The judge is not told which one is yours.** It sees "A" and "B". Told, it would
+  have an opinion about the new system instead of about the conversations.
+- **Every judge is asked twice, with the transcripts swapped.** Preferring whatever was
+  printed first is the classic failure of pairwise grading, so a verdict that flips is
+  reported as `comparable`, marked `order-dependent`.
+- **A person is not the incumbent's model.** `origin: operator` on any reference
+  message labels the whole pair `vs: human-assisted`, and the summary counts those
+  separately — comparing a model to a person and calling it a win is a lie in both
+  directions. The judge is not told; the reader is, which is where it changes a
+  decision.
+
+Cost: **two provider calls per judge per case**, which is why it is opt-in and never
+part of the gate.
+
+Cases live in two places, and it is the same YAML in both:
+
+- **`evals/golden/**`** in the repo — the curated corpus, reviewable in a pull request,
+  and the seed for a fresh deployment.
+- **the store** — what a deployment actually runs, editable in **Studio → Evals**
+  without a checkout. That matters because the rubric is the part of an eval a domain
+  owner can write, and asking them for a git branch means it never gets written.
+
+```bash
+insika evals:import              # corpus -> store (a fresh deploy starts here)
+insika evals:import --keep-existing   # don't overwrite what was authored in the Studio
+insika evals:export --dir /tmp/cases  # store -> YAML, at the paths it came from
+```
+
+Export refuses to overwrite an existing corpus unless you pass `--force`: `YAML.dump`
+drops the comments those files carry, and each one explains what its case is for.
+
+A case whose stored YAML no longer validates is **listed as broken** on the Evals page
+rather than skipped in silence — a test suite that quietly shrinks is worse than a red
+one.
+
+## Running
+
+```bash
+ADMIN_TOKEN=… ruby evals/run.rb                       # cases from the store, else the corpus
+ADMIN_TOKEN=… ruby evals/run.rb --source dir          # ignore the store (no database needed)
+ADMIN_TOKEN=… ruby evals/run.rb --agent loja-chocolates --mode both
+```
+
+It is **on-demand, not CI**: it costs tokens, needs a live provider key and needs the
+target agents provisioned. `--mode perf` reuses the same replay to report TTFB and
+total latency over real conversations, so one harness answers both questions.
+
+A run writes `evals/reports/<timestamp>.json` and prints a markdown summary.
+
+## Who grades: a panel, not a voice
+
+The judge reads (conversation, reply, rubric) and returns a score in `[0,1]` with one
+sentence of reason. An unparseable judge reply scores **0** — a broken grader must
+never look like a pass.
+
+Configure the graders in **Studio → Settings → Evals**, one `provider/model` per line:
+
+| Key | Meaning |
+|---|---|
+| `judges` | one entry per model. Empty = deterministic assertions only, and rubric'd cases read as `judge_pending` |
+| `aggregate` | `median` (default), `mean`, or `min` — how the panel's scores become the one number the report and the baseline read |
+| `min_agreement` | fraction of judges that must pass **on their own**. `0.5` = a majority, `1.0` = unanimous |
+| `quorum` | samples per judge, on top of the panel |
+| `tolerance` | max score drop before it counts as a regression |
+
+Several models is the point. Sampling **one** model three times measures that model's
+variance — at temperature 0 it mostly returns the same answer, including the same blind
+spot. Two *different* models disagreeing about a rubric is the signal worth having, and
+the report keeps each judge's score so a split panel is visible instead of hidden
+inside an average.
+
+`--judge-model` still overrides everything for a one-off run.
+
+## The gate
+
+A **baseline** (`evals/baseline.json`) is the accepted state of the corpus. A gated run
+blocks only on a **regression** — a case that used to pass and now fails, or a judge
+score that dropped past `tolerance` — so known failures don't wedge the gate while a
+real drop does:
+
+```bash
+ruby evals/run.rb --baseline evals/baseline.json   # exits non-zero on a regression
+ruby evals/run.rb --update-baseline                # accept the current state
+```
+
+That is what you run before merging a prompt, tool or model change. A case with no
+baseline entry never blocks: it shows as failing in the report, but a brand-new case is
+not a regression.
+
+## Honest limits
+
+- **A judge is a model.** It has taste and it has bad days; that is why the
+  deterministic layer carries the load and the rubric should be objective ("does not
+  invent a code" beats "is friendly").
+- **A green run is not a proof.** It says the cases you wrote still behave. Cases come
+  from real conversations — see [Refinement](REFINEMENT.md) for reading production
+  traffic back to find the ones worth adding.
+- **The corpus is small on purpose.** Twenty cases covering the hot flows beat two
+  hundred nobody curates.
+- **A pairwise verdict is a judgement, not a measurement.** It is one model's opinion
+  about two conversations. Read a batch of them by hand before quoting the number in a
+  decision — if `better` tracks length or politeness rather than whether the customer
+  got served, the comparison is measuring the wrong thing and should be dropped.
+- **A pair the deployment's DATA cannot satisfy is not a loss.** `requires` resolves
+  tools and capabilities; it says nothing about catalogue content. If the reference
+  conversation found a product that does not exist where you are replaying, no query
+  could have returned it, and the pair is unrunnable — exactly like a case needing a
+  tool the agent does not have. Exclude it by hand and say so; counting it is the same
+  lie `requires` exists to end.
+- **A tool that delivers out of band scores as silence.** When a backend sends the card,
+  the button or the support contact straight to the channel, the right behaviour is to
+  publish nothing ([`halt_when`](TOOLS.md#halt_when-when-the-answer-is-already-out)) —
+  and a comparison that only observes what *you* publish cannot tell that from an agent
+  that said nothing. Compare conversations whose tools deliver the same way, or read
+  those pairs by hand.
+- **Reuse a conversation and you are judging the wrong transcript.** A replay continues
+  whatever the conversation already holds, so a pair replayed into a conversation an
+  earlier run touched carries both runs' turns. It shows up as a judge that flips when
+  the transcripts are swapped — which is the swap-check earning its keep, but the run
+  is spent. Mint fresh conversation ids for every pairwise run.
+
+## Where it lives
+
+The harness is `lib/insika/evals/*` — inside the engine, because the refinement gate
+needs to score a candidate agent with the same judge, and a second copy of the judge
+would be the worst possible outcome. It stays a **client** even so: it reaches a running
+deployment over HTTP through `POST /v1/responses` and never reads a store directly.
+`evals/run.rb` is a thin CLI over it.

data/docs/LOADTEST.md

@@ -0,0 +1,232 @@
+---
+title: Load test
+parent: Operate & prove it
+nav_order: 3
+permalink: /loadtest/
+---
+
+# LOADTEST — Insika
+
+How to load-test the engine, how to compare it **apples-to-apples against another
+gateway**, and how to read the numbers — covering the SQLite-topology question
+(does one box hold up?) and loadtest parity. For deploy/env details see
+[DEPLOY.md](DEPLOY.md).
+
+The whole point: the engine exposes `POST /v1/responses` as an **SSE drop-in** of
+the OpenClaw gateway. Same contract → the same load tools work against either side,
+so you can measure the engine you are about to ship against the gateway it replaces.
+
+There are three scripts, each answering a different question:
+
+| Script | Question it answers | Needs a provider? |
+|--------|---------------------|-------------------|
+| `scripts/bench_store.rb` | Does SQLite (WAL) hold up N processes writing the same file? | No |
+| `scripts/loadtest.rb` | End-to-end: TTFB/total/tokens/cache/error against `/v1/responses` | Yes |
+| `scripts/loadtest-local.sh` | Single-proc baseline vs N-worker multi-proc on one box | Yes |
+| `scripts/loadtest_session.rb` | A full multi-message session (CEP, searches, FAQ) under C concurrent sessions — direct to the engine (`--surface engine`, stream vs steer) or through the consumer's real ingress (`--surface web`, the consumer's widget API) | Yes |
+
+All three take `--help` / `-h`.
+
+---
+
+## 1. `bench_store.rb` — SQLite write ceiling (no provider)
+
+Isolates "does SQLite survive multi-process?" from LLM-provider noise. N processes
+hammer writes against the **same file** using the engine's real production config
+(WAL + `busy_timeout` + `BEGIN IMMEDIATE` + in-process semaphore). It uses a fresh
+temp db per round, so it never touches your `INSIKA_DB`.
+
+```bash
+bundle exec ruby scripts/bench_store.rb [PROCS_CSV] [WRITES_PER_PROC]
+# defaults: 1,2,4,8 procs, 2000 writes each
+bundle exec ruby scripts/bench_store.rb 1,2,4,8 3000
+```
+
+Output columns: `procs | wall(s) | writes/s | p50(ms) | p95(ms) | max(ms) | locked`.
+
+**What to look for:** aggregate `writes/s` stays roughly flat as procs grow (the
+WAL "1 writer at a time" ceiling), and `locked` (i.e. `database is locked`) is **0**
+— the `busy_timeout` turns contention into tail latency (`max`), not errors. A real
+turn is provider-bound (seconds) and does only a handful of writes, so the workload
+sits ~100× below this ceiling. See DEPLOY.md for the measured numbers.
+
+---
+
+## 2. `loadtest.rb` — end-to-end against `/v1/responses` (with provider)
+
+Hits `POST /v1/responses` (SSE) directly — the production path
+(a consumer app, e.g. WhatsApp, → engine). Standard library only. Fires `agents × concurrency ×
+iterations` turns in waves of `concurrency`, and per turn records TTFB (time to
+first SSE byte), total time, and the `usage` block (tokens + cache hit) of the last
+frame that carries it.
+
+```bash
+INSIKA_URL=http://localhost:9292 \
+OPENCLAW_GATEWAY_TOKEN=xxx \
+bundle exec ruby scripts/loadtest.rb \
+  --agents bia,my-store --concurrency 16 --iterations 3 \
+  --message "hi, how are you?"
+```
+
+Runs against a local server **or** a remote one (e.g. Railway) — just point
+`INSIKA_URL` at it.
+
+### Flags
+
+| Flag | Default | Meaning |
+|------|---------|---------|
+| `--agents a,b,c` | `bia` | comma-separated agent ids (mapped to `model: openclaw:<agent>`) |
+| `--concurrency N` | `8` | concurrent turns per wave |
+| `--iterations N` | `1` | number of waves per agent |
+| `--message TEXT` | greeting | user message sent every turn |
+| `--timeout SECONDS` | `120` | per-request read timeout |
+| `--ports 9292,9293` | — | round-robin across local processes (see §4) |
+| `--same-user` | off | boolean toggle — reuse the same `user` per agent to measure the **hot-conversation cache** (legacy `--same-user 1` / `--same-user 0` still work) |
+| `--dry-run` | — | print the plan + one sample request (masked token) and exit; **sends no traffic** |
+| `--help` | — | show usage and exit |
+
+### Environment
+
+| Env | Default | Meaning |
+|-----|---------|---------|
+| `INSIKA_URL` | `http://localhost:9292` | base URL of the engine |
+| `OPENCLAW_GATEWAY_TOKEN` | falls back to `ADMIN_TOKEN`, then `local-demo` | Bearer for `/v1/responses` |
+| `DEEPSEEK_API_KEY` | — | must be configured **on the server** for real turns (not read by the client) |
+
+Use `--dry-run` to sanity-check your flags/URL/token before firing real traffic
+(and to confirm the request body without needing a running server):
+
+```bash
+INSIKA_URL=http://localhost:9292 OPENCLAW_GATEWAY_TOKEN=xxx \
+  bundle exec ruby scripts/loadtest.rb --agents bia --concurrency 16 --dry-run
+```
+
+### `--same-user` and the cache
+
+By default each turn uses a distinct `user` (`loadtest-<agent>-<idx>`), so every
+turn is a cold conversation. With `--same-user` all turns for an agent share one
+`user`, exercising the warm-conversation path — watch the **mean cache hit** rise
+and TTFB drop. Run both to bracket cold vs hot behaviour.
+
+---
+
+## 3. `loadtest-local.sh` — baseline vs multi-worker on one box
+
+Boots Falcon with `--count 1` (single-process baseline), runs the sweep, then boots
+`--count N` (multi-process) over the **same** SQLite file (WAL), runs the sweep
+again, and counts `database is locked` in each Falcon log. Since the provider is
+identical across both runs, if multi-proc recovers throughput the ceiling was
+CPU/event-loop (not the provider), and a `locked` count of **0** proves the WAL +
+`busy_timeout` config absorbs cross-process write contention.
+
+```bash
+DEEPSEEK_API_KEY=sk-... ./scripts/loadtest-local.sh [WORKERS] [CONCURRENCY]
+# defaults: 4 workers, 16 concurrency
+./scripts/loadtest-local.sh 4 24
+```
+
+| Env | Default | Meaning |
+|-----|---------|---------|
+| `DEEPSEEK_API_KEY` | — (required) | real turns hit the provider; also auto-sourced from `.env.local` |
+| `OPENCLAW_GATEWAY_TOKEN` | falls back to `ADMIN_TOKEN`, then `local-demo` | Bearer for the sweep |
+| `PORT` | `9299` | bind port for the local Falcon |
+| `AGENT` | `bia` | agent id to load |
+
+The final block prints the lock counts; the expected reading is `0` for both:
+
+```
+=== 'database is locked' (expected 0 — WAL + busy_timeout) ===
+  baseline (1):        0
+  multi (4):           0
+```
+
+---
+
+## 4. Apples-to-apples: Insika vs the OpenClaw gateway
+
+Two ways to compare, both valid because the SSE contract is identical.
+
+### 4a. Ruby native (`loadtest.rb`) against both
+
+Run the same `loadtest.rb` invocation twice — once with `INSIKA_URL` pointing at
+the engine, once at the gateway (its `/v1/responses` speaks the same protocol).
+Keep `--agents`, `--concurrency`, `--iterations` and `--message` identical, and use
+matching agents on both sides. Compare the printed TTFB/total/cache/error lines.
+
+### 4b. Reuse OpenClaw's `loadtest-gateway.mjs` unmodified
+
+`loadtest.rb` is the Ruby port of OpenClaw's `loadtest-gateway.mjs`. You do **not**
+need to change that script to point it at the engine — because the engine is a
+drop-in for the gateway, you only change **where it points**:
+
+```bash
+# In the OpenClaw checkout, run its gateway loadtest against the HARNESS:
+OPENCLAW_GATEWAY_URL=http://localhost:9292 \
+OPENCLAW_GATEWAY_TOKEN=<same bearer the engine accepts> \
+node scripts/loadtest-gateway.mjs --agents bia --concurrency 16 --iterations 3
+```
+
+Then run the exact same command with `OPENCLAW_GATEWAY_URL` pointing at the real
+gateway, and diff the two reports. This is the shadow comparison the pilot needs.
+
+**What the operator must have in hand** (this repo does not vendor OpenClaw):
+
+- The OpenClaw checkout containing `scripts/loadtest-gateway.mjs` and Node installed.
+- A **bearer token accepted by both** sides. For the engine that is
+  `OPENCLAW_GATEWAY_TOKEN` (see DEPLOY.md); point the gateway run at its own token.
+- **The same agent id provisioned on both** sides (e.g. `bia`) so `model:
+  openclaw:<agent>` resolves on each. On the engine, provision via
+  `scripts/import_pack.rb`.
+- The **same provider** (or an equivalent-latency one) behind each, otherwise you
+  are comparing providers, not engines.
+- Both endpoints reachable from where you run the client, warmed up (hit `/up` on
+  the engine first), and ideally driven from the same machine to remove network
+  skew.
+
+Keep every knob identical between the two runs — the only variable should be which
+engine is behind `/v1/responses`.
+
+---
+
+## 5. Reading the metrics
+
+- **TTFB** (time to first SSE byte) — how fast the user starts seeing a response.
+  Dominated by provider latency + the engine's per-turn setup (context build,
+  policy, first model call). This is the number that most shapes perceived latency.
+- **total** — full turn wall time including the whole tool-loop and streamed
+  output. `total − TTFB` is roughly the streaming/tool-loop tail.
+- **P50 vs P95** — P50 is the typical turn; **P95 is the tail you actually feel**
+  under load. A P50 that stays flat while P95 balloons as concurrency rises means
+  you are queueing (CPU/event-loop or writer contention) — that is the signal to
+  add workers (§3) or check `bench_store.rb`.
+- **mean tokens** — average `total_tokens`/`output_tokens` per turn; sanity-checks
+  that turns did real work and lets you compare cost between runs/engines.
+- **mean cache hit** — average cached prompt tokens; should rise sharply with
+  `--same-user 1`. A high hit rate is why warm conversations are cheaper and faster.
+- **error rate** (`turns ok: X/Y (errors: N)`) — non-2xx, timeouts, or connection
+  failures. Anything above ~0 under moderate load is a red flag; inspect server
+  logs. Note `loadtest.rb` errors are transport/HTTP-level; `database is locked`
+  specifically is counted by `loadtest-local.sh` from the Falcon logs, not here.
+- **throughput** (`turns/s`) — completed turns per wall second; the headline
+  capacity number for a given concurrency.
+
+---
+
+## 6. Checklist — what to measure before choosing a topology
+
+Work top-down and **measure before assuming** — avoid premature topology optimization.
+
+| # | Measure | Tool | Decision it informs |
+|---|---------|------|---------------------|
+| 1 | SQLite write ceiling & `locked` count under N procs | `bench_store.rb` | Is SQLite a bottleneck at all on one box? (Expected: no.) |
+| 2 | Single-proc baseline TTFB/total/P95/throughput | `loadtest.rb` (or `loadtest-local.sh` count 1) | The reference point for everything else. |
+| 3 | Multi-proc on one box: does throughput scale, `locked` = 0? | `loadtest-local.sh` | Do more Falcon workers help, and does the shared WAL hold? Sets `WEB_CONCURRENCY`. |
+| 4 | Insika vs gateway, identical knobs | §4 (either method) | Is the engine at parity with the engine it replaces before cut-over? |
+| 5 | Cold vs hot conversation (cache) | `loadtest.rb` with/without `--same-user 1` | Expected steady-state cost/latency once conversations warm up. |
+| 6 | Remote (Railway) vs local | `loadtest.rb` with `INSIKA_URL` remote | Network/deploy overhead of the real environment. |
+
+**Reaching for horizontal scale is only justified after 1–3 show the single box is
+the limit.** If it is, the paths are: sharding-by-tenant +
+sticky routing (recommended), LiteFS, or an optional Postgres adapter — plus
+Litestream for backup/DR regardless of topology. Do not skip straight to Postgres:
+the numbers usually show SQLite on one big box is not the bottleneck.