insika 0.3.0 → 0.7.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +180 -0
- data/README.md +45 -10
- data/bin/insika +684 -0
- data/bin/insika-router +87 -0
- data/docs/AGENTS.md +94 -403
- data/docs/API.md +5 -5
- data/docs/ARCHITECTURE.md +3 -2
- data/docs/ARTIFACTS.md +95 -0
- data/docs/BENCHMARK.md +2 -2
- data/docs/CHANNELS.md +14 -14
- data/docs/CONTEXT.md +9 -7
- data/docs/DEMO.md +80 -0
- data/docs/DEPLOY.md +71 -3
- data/docs/EMBEDDING.md +1 -1
- data/docs/EVALS.md +128 -3
- data/docs/FACTS.md +3 -3
- data/docs/HARVEST.md +5 -6
- data/docs/KNOWLEDGE.md +290 -0
- data/docs/LOADTEST.md +2 -2
- data/docs/MEDIA.md +128 -0
- data/docs/OBSERVABILITY.md +15 -10
- data/docs/OUTCOMES.md +137 -0
- data/docs/PLUGINS.md +51 -6
- data/docs/POLICY.md +216 -0
- data/docs/REFINEMENT.md +14 -9
- data/docs/RELEASING.md +4 -4
- data/docs/ROUTER.md +213 -0
- data/docs/RUNNING-LOCAL.md +3 -3
- data/docs/SCHEDULING.md +121 -0
- data/docs/SECURITY.md +22 -6
- data/docs/SKILLS.md +11 -2
- data/docs/SOAK.md +2 -2
- data/docs/TEMPLATES.md +134 -0
- data/docs/TOOLS.md +152 -27
- data/docs/WHY.md +1 -1
- data/docs/WORKFLOWS.md +2 -2
- data/docs/_includes/head_custom.html +5 -0
- data/docs/_includes/title.html +13 -0
- data/docs/_sass/color_schemes/insika.scss +32 -0
- data/docs/_sass/custom/custom.scss +199 -0
- data/docs/_sass/custom/setup.scss +26 -0
- data/docs/assets/img/favicon.svg +7 -0
- data/docs/assets/img/insika-mark.svg +7 -0
- data/docs/core-concepts.md +21 -0
- data/docs/domain.md +4 -4
- data/docs/improve.md +20 -0
- data/docs/index.md +8 -5
- data/docs/integrate.md +20 -0
- data/docs/operate.md +13 -6
- data/docs/prompts/ADD-TOOL.md +118 -0
- data/docs/prompts/DIAGNOSE-TURN.md +65 -0
- data/docs/prompts/GO-LIVE.md +138 -0
- data/docs/prompts/RUN-EXAMPLES.md +70 -0
- data/docs/reference.md +19 -0
- data/docs/ship.md +10 -2
- data/docs/start-here.md +18 -0
- data/lib/insika/agent_profile.rb +73 -16
- data/lib/insika/artifact_signing.rb +82 -0
- data/lib/insika/artifact_store.rb +160 -0
- data/lib/insika/channel_delivery.rb +1 -1
- data/lib/insika/chat_builder.rb +22 -2
- data/lib/insika/commands/agent_payload.rb +2 -2
- data/lib/insika/commands/backfill_knowledge.rb +145 -0
- data/lib/insika/commands/delete_artifact.rb +35 -0
- data/lib/insika/commands/delete_concept.rb +34 -0
- data/lib/insika/commands/delete_mcp.rb +6 -2
- data/lib/insika/commands/delete_tenant_data.rb +15 -3
- data/lib/insika/commands/gate_refinement.rb +1 -1
- data/lib/insika/commands/refresh_mcp_tools.rb +47 -0
- data/lib/insika/commands/restore_concept.rb +34 -0
- data/lib/insika/commands/seed_demo_data.rb +31 -0
- data/lib/insika/commands/upsert_mcp.rb +6 -3
- data/lib/insika/commands/write_concept.rb +57 -0
- data/lib/insika/context/priority.rb +2 -0
- data/lib/insika/context/providers/knowledge.rb +108 -0
- data/lib/insika/context/providers/prompt.rb +30 -24
- data/lib/insika/cron.rb +189 -0
- data/lib/insika/demo/agent_attrs.rb +43 -0
- data/lib/insika/demo/golden_cases.rb +81 -0
- data/lib/insika/demo/seeder.rb +336 -0
- data/lib/insika/doctor.rb +176 -8
- data/lib/insika/dsl/definition.rb +3 -2
- data/lib/insika/dsl/runtime.rb +60 -79
- data/lib/insika/dsl/server_boot.rb +23 -1
- data/lib/insika/dsl/system.rb +10 -2
- data/lib/insika/dsl.rb +103 -2
- data/lib/insika/env_schema.rb +16 -1
- data/lib/insika/evals/golden.rb +41 -4
- data/lib/insika/evals/judge.rb +47 -2
- data/lib/insika/evals/pairwise.rb +11 -0
- data/lib/insika/evals/persona.rb +98 -0
- data/lib/insika/evals/runner.rb +9 -0
- data/lib/insika/evals/simulator.rb +225 -0
- data/lib/insika/evals/transport.rb +83 -1
- data/lib/insika/event_stream.rb +10 -0
- data/lib/insika/executor.rb +231 -55
- data/lib/insika/followup_policy.rb +2 -25
- data/lib/insika/golden_store.rb +16 -1
- data/lib/insika/grounding/matcher.rb +1 -1
- data/lib/insika/knowledge.rb +680 -0
- data/lib/insika/knowledge_store.rb +140 -0
- data/lib/insika/mcp_client.rb +94 -0
- data/lib/insika/mcp_json.rb +74 -0
- data/lib/insika/mcp_live_tool.rb +43 -0
- data/lib/insika/mcp_store.rb +98 -26
- data/lib/insika/mcp_tool_ingestor.rb +30 -8
- data/lib/insika/mcp_tool_registry.rb +100 -0
- data/lib/insika/media.rb +115 -31
- data/lib/insika/message_origin.rb +1 -1
- data/lib/insika/middleware.rb +9 -0
- data/lib/insika/onboarding.rb +17 -1
- data/lib/insika/outcome_store.rb +1 -1
- data/lib/insika/overlay_tool_registry.rb +37 -17
- data/lib/insika/packaging.rb +2 -2
- data/lib/insika/profile_source.rb +8 -1
- data/lib/insika/prompt_catalog.rb +10 -0
- data/lib/insika/retention.rb +36 -1
- data/lib/insika/router/app.rb +157 -0
- data/lib/insika/router/backend_pool.rb +98 -0
- data/lib/insika/router/hash_ring.rb +55 -0
- data/lib/insika/router/proxy_body.rb +34 -0
- data/lib/insika/router/session_key.rb +54 -0
- data/lib/insika/router.rb +18 -0
- data/lib/insika/schedule.rb +177 -0
- data/lib/insika/schedule_engine.rb +314 -0
- data/lib/insika/schedule_store.rb +208 -0
- data/lib/insika/server/app.rb +105 -15
- data/lib/insika/server/rack_app.rb +5 -1
- data/lib/insika/server/responses.rb +1 -1
- data/lib/insika/skill_catalog.rb +12 -0
- data/lib/insika/steer_injector.rb +21 -10
- data/lib/insika/studio/app.rb +567 -45
- data/lib/insika/studio/assets/dist/application.css +1 -1
- data/lib/insika/studio/assets/dist/application.js +21 -21
- data/lib/insika/studio/forms.rb +46 -5
- data/lib/insika/studio/nav_icons.rb +14 -1
- data/lib/insika/studio/views/_agent_tab_cache.erb +25 -0
- data/lib/insika/studio/views/_agent_tab_config.erb +514 -0
- data/lib/insika/studio/views/_agent_tab_history.erb +24 -0
- data/lib/insika/studio/views/_agent_tab_loops.erb +54 -0
- data/lib/insika/studio/views/_agent_tab_memory.erb +51 -0
- data/lib/insika/studio/views/_agent_tab_outcomes.erb +31 -0
- data/lib/insika/studio/views/_agent_tab_prompts.erb +108 -0
- data/lib/insika/studio/views/_agent_tab_skills.erb +38 -0
- data/lib/insika/studio/views/_agents_master.erb +44 -0
- data/lib/insika/studio/views/_message.erb +49 -32
- data/lib/insika/studio/views/agent_detail.erb +61 -820
- data/lib/insika/studio/views/agents.erb +70 -57
- data/lib/insika/studio/views/artifact.erb +23 -0
- data/lib/insika/studio/views/artifacts.erb +59 -0
- data/lib/insika/studio/views/evals.erb +2 -2
- data/lib/insika/studio/views/facts.erb +1 -1
- data/lib/insika/studio/views/funnel.erb +1 -1
- data/lib/insika/studio/views/home.erb +106 -67
- data/lib/insika/studio/views/knowledge.erb +123 -0
- data/lib/insika/studio/views/layout.erb +14 -11
- data/lib/insika/studio/views/mcp.erb +174 -80
- data/lib/insika/studio/views/session.erb +231 -177
- data/lib/insika/studio/views/settings.erb +39 -1
- data/lib/insika/studio/views/skills.erb +1 -1
- data/lib/insika/studio/views/tools.erb +24 -9
- data/lib/insika/templates/browser-agent/README.md +36 -0
- data/lib/insika/templates/browser-agent/agent.rb +49 -0
- data/lib/insika/templates/daily-digest/README.md +38 -0
- data/lib/insika/templates/daily-digest/agent.rb +77 -0
- data/lib/insika/templates/repo-explorer/README.md +36 -0
- data/lib/insika/templates/repo-explorer/agent.rb +45 -0
- data/lib/insika/templates/research-analyst/README.md +26 -0
- data/lib/insika/templates/research-analyst/agent.rb +58 -0
- data/lib/insika/templates/review-panel/README.md +20 -0
- data/lib/insika/templates/review-panel/agent.rb +50 -0
- data/lib/insika/templates/travel-planner/README.md +35 -0
- data/lib/insika/templates/travel-planner/agent.rb +87 -0
- data/lib/insika/templates.rb +112 -0
- data/lib/insika/tick.rb +24 -12
- data/lib/insika/timezone.rb +45 -0
- data/lib/insika/tools/generate_image.rb +52 -7
- data/lib/insika/tools/load_knowledge.rb +74 -0
- data/lib/insika/tools/run_persona_eval.rb +328 -0
- data/lib/insika/tools/save_artifact.rb +95 -0
- data/lib/insika/turn_output.rb +1 -1
- data/lib/insika/turn_state.rb +15 -4
- data/lib/insika/version.rb +1 -1
- data/lib/insika/wiring/graph.rb +184 -12
- data/lib/insika/wiring/graph_chat.rb +102 -0
- data/lib/insika.rb +57 -0
- metadata +105 -5
- data/docs/build.md +0 -14
- data/docs/understand.md +0 -10
data/docs/ARTIFACTS.md
ADDED
|
@@ -0,0 +1,95 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: Artifacts
|
|
3
|
+
parent: Operate
|
|
4
|
+
nav_order: 3
|
|
5
|
+
permalink: /artifacts/
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
# Artifacts — a report the agent can hand you a URL to
|
|
9
|
+
|
|
10
|
+
A channel message is ephemeral, unformatted and capped. A scheduled report turn
|
|
11
|
+
(see [Schedules](SCHEDULING.md)) produces something no message can carry: a page
|
|
12
|
+
— tables, sections, inline charts. An **artifact** is that page as a thing that
|
|
13
|
+
exists afterwards: listable, linkable, and — because it carries customer PII —
|
|
14
|
+
deletable on the same terms as everything else the engine stores.
|
|
15
|
+
|
|
16
|
+
Deliberately small: a store, a tool, a route. Not a CMS.
|
|
17
|
+
|
|
18
|
+
## The `save_artifact` tool
|
|
19
|
+
|
|
20
|
+
A registry tool, allowlisted per agent like every tool — **the allowlist IS the
|
|
21
|
+
switch**:
|
|
22
|
+
|
|
23
|
+
```ruby
|
|
24
|
+
agent = Insika.agent("reporter") do
|
|
25
|
+
instructions "…"
|
|
26
|
+
tools_allow %w[save_artifact] # without this, the tool is not even offered
|
|
27
|
+
end
|
|
28
|
+
```
|
|
29
|
+
|
|
30
|
+
The agent hands in `title` + `content` (default mime `text/html`; also
|
|
31
|
+
`text/markdown` and `image/svg+xml`) and gets the URL back, which it can include
|
|
32
|
+
in a channel message ("today's report: <url>"). When a signing key is
|
|
33
|
+
configured (below), the result also carries a `signed_url` that expires.
|
|
34
|
+
|
|
35
|
+
```jsonc
|
|
36
|
+
{ "id": "…", "url": "https://…/studio/artifacts/<id>/content",
|
|
37
|
+
"signed_url": "https://…/studio/artifacts/s/<id>?exp=…&sig=…" } // only with a key
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
The tenant binding is **inherited, never chosen**: an artifact belongs to the
|
|
41
|
+
tenant of the agent that saved it — a binding of the tool instance, never a
|
|
42
|
+
parameter the model types. Store A's report can never appear in, or be linked
|
|
43
|
+
from, store B.
|
|
44
|
+
|
|
45
|
+
## Serving
|
|
46
|
+
|
|
47
|
+
- `GET /studio/artifacts` — the Studio's per-agent list (the listing IS the
|
|
48
|
+
history; no versioning — one report per run).
|
|
49
|
+
- `GET /studio/artifacts/:id` — the preview page, rendered **inside a sandboxed
|
|
50
|
+
iframe** (no scripts, no same-origin, no forms).
|
|
51
|
+
- `GET /studio/artifacts/:id/content` — the raw page (authenticated).
|
|
52
|
+
- `GET /studio/artifacts/s/:id?exp=…&sig=…` — the **signed link**: the only
|
|
53
|
+
artifact route that works without a Studio session. HMAC-SHA256 over
|
|
54
|
+
`(id, expiry)` with `INSIKA_ARTIFACT_SIGNING_KEY`, verified in constant time.
|
|
55
|
+
Expired or bad signatures **404 (never 403 — no oracle)**. Rotating the key
|
|
56
|
+
invalidates every outstanding link — the documented behavior, not a bug.
|
|
57
|
+
Without `INSIKA_ARTIFACT_SIGNING_KEY` there is no signed surface at all.
|
|
58
|
+
|
|
59
|
+
**Artifact content is untrusted.** It is LLM output. Both content routes send:
|
|
60
|
+
|
|
61
|
+
```http
|
|
62
|
+
Content-Security-Policy: default-src 'none'; style-src 'unsafe-inline'; img-src data:
|
|
63
|
+
X-Content-Type-Options: nosniff
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
No script, no external fetch, no forms. The model writes HTML with **inline
|
|
67
|
+
SVG** for charts — that is a skill instruction (palette, tables, pure-SVG bars),
|
|
68
|
+
not engine code. A "real" charting need is a plugin.
|
|
69
|
+
|
|
70
|
+
## Limits and retention
|
|
71
|
+
|
|
72
|
+
- **Size cap** — `INSIKA_ARTIFACT_MAX_BYTES` (default 1 MB): an artifact is a
|
|
73
|
+
page, not an attachment. The mime allowlist is `text/html`, `text/markdown`,
|
|
74
|
+
`image/svg+xml`; no binaries, no uploads.
|
|
75
|
+
- **Expiry** — the settings key `artifact_ttl_days` (Integer days; absent = OFF)
|
|
76
|
+
ages artifacts out on the retention sweep's own daily pass, **independent of
|
|
77
|
+
`retention_days`**: a deployment that keeps its conversations forever must
|
|
78
|
+
still expire the reports. This is the guarantee that PII inside a report
|
|
79
|
+
expires — the honest reach, because no reader can see inside the opaque HTML.
|
|
80
|
+
|
|
81
|
+
## Privacy
|
|
82
|
+
|
|
83
|
+
- `delete_tenant_data` deletes the tenant's artifacts (the tenant binding is
|
|
84
|
+
the isolation boundary).
|
|
85
|
+
- `forget_customer` **cannot** know which artifacts mention a customer (content
|
|
86
|
+
is opaque HTML), so per-customer redaction inside a report is not pretended to
|
|
87
|
+
exist; the `artifact_ttl_days` knob is the guarantee that a report's PII
|
|
88
|
+
expires.
|
|
89
|
+
|
|
90
|
+
## See also
|
|
91
|
+
|
|
92
|
+
- [Schedules](SCHEDULING.md) — the recurring turns whose output lands here.
|
|
93
|
+
- [Tools](TOOLS.md) — how a tool enters the per-agent allowlist.
|
|
94
|
+
- [`examples/scheduled-report/`](https://github.com/guizaols/insika/tree/main/examples/scheduled-report/)
|
|
95
|
+
— schedule + skill + data tool + artifact, tenant-bound, end to end.
|
data/docs/BENCHMARK.md
CHANGED
data/docs/CHANNELS.md
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
title: Channels
|
|
3
|
-
parent:
|
|
4
|
-
nav_order:
|
|
3
|
+
parent: Integrate
|
|
4
|
+
nav_order: 2
|
|
5
5
|
permalink: /channels/
|
|
6
6
|
---
|
|
7
7
|
|
|
@@ -176,14 +176,14 @@ again.
|
|
|
176
176
|
## Relay or the drop-in API?
|
|
177
177
|
|
|
178
178
|
If you already own your messaging platform, you can reach the engine two ways: the
|
|
179
|
-
drop-in [`POST /v1/responses`](
|
|
179
|
+
drop-in [`POST /v1/responses`](ARCHITECTURE.md) — you hold an SSE connection for the
|
|
180
180
|
whole turn and read the answer off it — or the relay, where the engine acks in
|
|
181
181
|
milliseconds and POSTs the answer to you when it exists.
|
|
182
182
|
|
|
183
183
|
The instinct is that streaming gets the customer their reply sooner, and that the
|
|
184
184
|
relay trades that away. **For the default relay it does not, and the reason is
|
|
185
185
|
structural:** the engine publishes `:content` as the ANSWER, whole, after the
|
|
186
|
-
turn's hooks ([what crosses the edge](
|
|
186
|
+
turn's hooks ([what crosses the edge](ARCHITECTURE.md#what-crosses-the-edge)).
|
|
187
187
|
During the turn the stream carries tool activity; the text arrives in one piece at
|
|
188
188
|
the end. Measured on a real store agent, the text frames span **0 ms** — there is
|
|
189
189
|
nothing to deliver progressively. That is the `:at_end` fact, true of `/v1/responses`
|
|
@@ -204,7 +204,7 @@ That last row is the one that cannot be had the other way. `/v1/responses` answe
|
|
|
204
204
|
the request it was given, so a message that arrives while a turn is running is a
|
|
205
205
|
second turn — the engine has no way to tell you "this joined the previous one". The
|
|
206
206
|
relay's `merged` / `steered` acks exist precisely to say that, which is why
|
|
207
|
-
[the inbound queue](
|
|
207
|
+
[the inbound queue](AGENTS.md#queue_mode--when-a-message-arrives-while-the-agent-is-busy)
|
|
208
208
|
is only reachable from here.
|
|
209
209
|
|
|
210
210
|
**Measured, so you can judge it rather than take our word:** same agent, same
|
|
@@ -282,7 +282,7 @@ answers, and **they are four different facts**:
|
|
|
282
282
|
answer two or three times.** That is the one contract mistake that is visible to
|
|
283
283
|
the end user, so it is worth a line of code: only deliver for a `202`.
|
|
284
284
|
|
|
285
|
-
`merged` and `steered` come from the [inbound queue](
|
|
285
|
+
`merged` and `steered` come from the [inbound queue](AGENTS.md#queue_mode--when-a-message-arrives-while-the-agent-is-busy)
|
|
286
286
|
(`limits[:queue_mode]`). They only ever occur if you turned that on for the agent;
|
|
287
287
|
with the default `followup` you will only see `202` and `duplicate`.
|
|
288
288
|
|
|
@@ -312,7 +312,7 @@ short backoff, and then the delivery is marked `failed` and stops.
|
|
|
312
312
|
`content` is the turn's **answer** — one message, whole. The model's narration on
|
|
313
313
|
the way to an answer ("vou verificar o cardápio…") does not come through here; it
|
|
314
314
|
stays internal unless the agent opts in. That contract is
|
|
315
|
-
[the edge contract](
|
|
315
|
+
[the edge contract](ARCHITECTURE.md#what-crosses-the-edge), and it is why you can
|
|
316
316
|
forward `content` straight to the customer.
|
|
317
317
|
|
|
318
318
|
### Delivery policy
|
|
@@ -379,7 +379,7 @@ your side: you already have the customer's conversation, and `GET /v1/tasks/:id`
|
|
|
379
379
|
tells you the turn's terminal state.
|
|
380
380
|
|
|
381
381
|
A turn that **failed** delivers nothing — an error string is not an answer. Watch
|
|
382
|
-
`GET /v1/tasks/:id` or the [event stream](
|
|
382
|
+
`GET /v1/tasks/:id` or the [event stream](OBSERVABILITY.md) for those.
|
|
383
383
|
|
|
384
384
|
## Setting up the relay
|
|
385
385
|
|
|
@@ -397,7 +397,7 @@ INSIKA_RELAY_DELIVERY=progressive # optional; "at_end" (the defaul
|
|
|
397
397
|
without a credential — a public inbound route with an LLM behind it is a money
|
|
398
398
|
faucet, so it fails closed by construction.
|
|
399
399
|
|
|
400
|
-
The delivery POST goes through the same [egress guard](
|
|
400
|
+
The delivery POST goes through the same [egress guard](SECURITY.md#egress-the-ssrf-boundary) as
|
|
401
401
|
data-tools: **https only**, and private/loopback destinations blocked. For local
|
|
402
402
|
development, where your consumer is on `localhost`:
|
|
403
403
|
|
|
@@ -415,7 +415,7 @@ A runnable consumer in ~40 lines lives in
|
|
|
415
415
|
|
|
416
416
|
## Shadow mode
|
|
417
417
|
|
|
418
|
-
Shadow mode
|
|
418
|
+
Shadow mode lets one channel run every turn **end to end and deliver
|
|
419
419
|
nothing** — the experiment that answers "can we replace the incumbent?" before
|
|
420
420
|
any customer is handed over. The incumbent keeps answering; the engine records
|
|
421
421
|
what it *would* have answered, and the two replies are judged pairwise against a
|
|
@@ -470,7 +470,7 @@ redirect its own conversation.
|
|
|
470
470
|
## Writing your own channel
|
|
471
471
|
|
|
472
472
|
A channel is a plain object — no base class. Register it from a plugin
|
|
473
|
-
(see [Plugins](
|
|
473
|
+
(see [Plugins](PLUGINS.md)) with `contracts: { channels: [<id>] }` in the manifest,
|
|
474
474
|
and it mounts under `/channels/<id>/`.
|
|
475
475
|
|
|
476
476
|
Two members are always there; the rest of the object decides which shape you get.
|
|
@@ -536,8 +536,8 @@ command carries `transport: "channel:<id>"`.
|
|
|
536
536
|
|
|
537
537
|
## See also
|
|
538
538
|
|
|
539
|
-
- [Security](
|
|
539
|
+
- [Security](SECURITY.md) — the tokens, the egress guard, and why the rate limit
|
|
540
540
|
matters for anything public.
|
|
541
|
-
- [Agents](
|
|
541
|
+
- [Agents](AGENTS.md) — `limits[:queue_mode]`, which is what produces `merged` and
|
|
542
542
|
`steered`.
|
|
543
|
-
- [Observability](
|
|
543
|
+
- [Observability](OBSERVABILITY.md) — the event stream and OpenTelemetry.
|
data/docs/CONTEXT.md
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
title: Context
|
|
3
|
-
parent:
|
|
4
|
-
nav_order:
|
|
3
|
+
parent: Core concepts
|
|
4
|
+
nav_order: 5
|
|
5
5
|
permalink: /context/
|
|
6
6
|
---
|
|
7
7
|
|
|
@@ -26,6 +26,7 @@ into a deterministic prompt:
|
|
|
26
26
|
| **Skills** | `<available_skills>` | 80 | identity | Level-1 skill list, minus whatever is already eager — see [Skills](SKILLS.md). |
|
|
27
27
|
| **Tool search** | `<available_tools>` | 70 | identity | Level-1 list of deferred tools — see [Tools](TOOLS.md). |
|
|
28
28
|
| **Skill trigger** | `<active_skill>` | 85 | volatile | Level-2 bodies: the agent's `skills_eager` set, plus the ones whose `triggers:` match the message — see [Skills](SKILLS.md). |
|
|
29
|
+
| **Knowledge** | `<knowledge>` | 77 | volatile | Level-1 top-K learned concepts for the turn's message (+ one-hop `[[links]]`), only if `knowledge.retrieve` is on. Cuttable — see [Knowledge](KNOWLEDGE.md). |
|
|
29
30
|
| **Memory** | `<memory>` | 75 | volatile | Durable facts + recent notes, only if `memory` is on. Cuttable. |
|
|
30
31
|
| **Briefing** | `<briefing>` | 65 | volatile | The session's working state (known fields, still-missing list, next step) — only if the pack declared `briefing_fields`. Cuttable. |
|
|
31
32
|
| **Session** | history | 60–79 | volatile | The running transcript; priority scales with recency. |
|
|
@@ -41,7 +42,8 @@ cacheable prefix byte-stable (see the prefix cache below).
|
|
|
41
42
|
- The cap is `profile.limits[:context_budget]`, **default 8000 tokens**.
|
|
42
43
|
- To fit the budget, the builder cuts **non-pinned** fragments
|
|
43
44
|
lowest-priority-first (ties broken by oldest history first). Under pressure you
|
|
44
|
-
lose
|
|
45
|
+
lose request context first, then old history, then briefing, then memory, then
|
|
46
|
+
learned knowledge, then the skill/tool-search level-1 lists — **the pinned
|
|
45
47
|
identity is never truncated**.
|
|
46
48
|
- A **pinned** fragment (the identity) that *alone* exceeds the budget raises an
|
|
47
49
|
error — the turn fails rather than shipping a truncated identity.
|
|
@@ -88,16 +90,16 @@ conversation; memory is the small set of facts that should outlive any single
|
|
|
88
90
|
conversation. Facts and notes are editable from the Studio agent page. See
|
|
89
91
|
[`examples/memory/`](https://github.com/guizaols/insika/tree/main/examples/memory/) for a runnable cross-session example.
|
|
90
92
|
|
|
91
|
-
Facts carry **provenance metadata
|
|
93
|
+
Facts carry **provenance metadata**: every fact record stores `origin`
|
|
92
94
|
(who wrote it — `"engine"`, `"operator"`, `"legacy"` or `"distilled"`),
|
|
93
95
|
`created_at` / `updated_at` timestamps, and an optional `expires_at` (ISO8601) —
|
|
94
96
|
**an expired fact is never injected**, even before the daily sweep prunes it. The
|
|
95
97
|
Studio Customers drill reads and edits the same cell the next turn reads (injection
|
|
96
98
|
unchanged), and every operator mutation lands in the content-free audit trail
|
|
97
99
|
(digests, never values). The sweep honors the `memory_ttl_days` setting on its own
|
|
98
|
-
knob — see [Security](SECURITY.md#memory-and-the-right-to-be-forgotten-lgpd
|
|
100
|
+
knob — see [Security](SECURITY.md#memory-and-the-right-to-be-forgotten-lgpd).
|
|
99
101
|
|
|
100
|
-
An **approved distilled fact** (
|
|
102
|
+
An **approved distilled fact** (see [Facts](FACTS.md)) lands in the
|
|
101
103
|
same cell this provider injects, stamped `distilled:<session_ref>` — approved on
|
|
102
104
|
the Studio Facts page, never applied automatically.
|
|
103
105
|
|
|
@@ -165,7 +167,7 @@ Two distinct caching mechanisms — don't conflate them:
|
|
|
165
167
|
Cache accounting surfaces as `cached_tokens` (reads) and `cache_creation_tokens`
|
|
166
168
|
(writes), visible in telemetry and the Studio tokens chip.
|
|
167
169
|
|
|
168
|
-
### The two layers
|
|
170
|
+
### The two layers
|
|
169
171
|
|
|
170
172
|
The system block is partitioned into two cache layers:
|
|
171
173
|
|
data/docs/DEMO.md
ADDED
|
@@ -0,0 +1,80 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: Demo data
|
|
3
|
+
parent: Start here
|
|
4
|
+
nav_order: 4
|
|
5
|
+
permalink: /demo/
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
# Demo data — see every loop working at once
|
|
9
|
+
|
|
10
|
+
Most of what makes Insika worth looking at only shows up after data has
|
|
11
|
+
accumulated: a [funnel](OUTCOMES.md#the-outcome-funnel) with a frozen
|
|
12
|
+
baseline needs weeks of folded outcomes, a [refinement](REFINEMENT.md)
|
|
13
|
+
proposal needs a run that actually found something, an
|
|
14
|
+
[approval](POLICY.md#layer-2-policies-and-approvals) needs a tool call
|
|
15
|
+
someone is waiting on. A fresh instance shows none of that — every one of
|
|
16
|
+
those pages renders its empty state, which makes it hard to tell "nothing
|
|
17
|
+
happened yet" from "this doesn't work."
|
|
18
|
+
|
|
19
|
+
`insika demo:seed` closes that gap in one shot: it provisions a single
|
|
20
|
+
fictional agent (`demo-store`, an e-commerce support bot) and writes enough
|
|
21
|
+
realistic-looking data to see every loop at once.
|
|
22
|
+
|
|
23
|
+
## What it creates
|
|
24
|
+
|
|
25
|
+
| Page | What you'll see |
|
|
26
|
+
|------|------------------|
|
|
27
|
+
| [Funnel](OUTCOMES.md#the-outcome-funnel) — `/studio/funnel?agent=demo-store` | 40 days of folded outcomes (`greeted → browsing → cart_started → checkout_started → purchased`) and a **frozen baseline** |
|
|
28
|
+
| [Follow-ups](OUTCOMES.md#follow-ups--the-seller-who-comes-back) — `/studio/followups?agent=demo-store` | one record in each state: `pending`, `fired` (one per arm, so the A/B card has something to compare), `cancelled`, `blocked` |
|
|
29
|
+
| [Refinement](REFINEMENT.md) — `/studio/refinement?agent=demo-store` | four runs across the lifecycle: `awaiting_approval`, `applied`, `rejected` (gate failed), `no_findings` |
|
|
30
|
+
| [Approvals](POLICY.md#layer-2-policies-and-approvals) — `/studio/approvals` | two pending tool calls waiting on a human, one already resolved |
|
|
31
|
+
| [Facts](FACTS.md) — `/studio/facts` | three distillation proposals (`pending`, `approved` — with the resulting memory fact, `rejected`) |
|
|
32
|
+
| [Evals](EVALS.md) — `/studio/evals?agent=demo-store` | six golden cases and a baseline run with a mix of passes and one failure |
|
|
33
|
+
|
|
34
|
+
Every record is written through the same store APIs a real turn would use
|
|
35
|
+
(`OutcomeStore#create` + the funnel fold, `FollowupStore#create` + its
|
|
36
|
+
transitions, and so on) — there is no bulk-insert shortcut, and no bundled
|
|
37
|
+
`.rb` script outside `lib/` (nothing here needs a checkout; it ships in the
|
|
38
|
+
gem).
|
|
39
|
+
|
|
40
|
+
## Running it
|
|
41
|
+
|
|
42
|
+
From the CLI, against whichever store the rest of your commands already use
|
|
43
|
+
([Running locally](RUNNING-LOCAL.md#variables-all-optional) — `INSIKA_DB`
|
|
44
|
+
unset means an ephemeral, in-memory store, which is a fine place to try this):
|
|
45
|
+
|
|
46
|
+
```bash
|
|
47
|
+
insika demo:seed
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
A second run is a safe no-op once `demo-store` exists; pass `--force` to seed
|
|
51
|
+
another batch on top (the funnel baseline recomputes cleanly, but follow-ups,
|
|
52
|
+
refinement runs, approvals, proposals and goldens accumulate rather than
|
|
53
|
+
reset — none of those stores expose a per-agent bulk-delete that a shared
|
|
54
|
+
"platform" tenant could call without risking another agent's data).
|
|
55
|
+
|
|
56
|
+
From the Studio, open **Settings → Demo data** and click **Seed demo data**.
|
|
57
|
+
It dispatches the exact same command the CLI runs — the Studio never writes a
|
|
58
|
+
store directly, here or anywhere else.
|
|
59
|
+
|
|
60
|
+
**This writes into whatever store the running instance already has open.**
|
|
61
|
+
There's no separate demo database and no isolation: point `INSIKA_DB` at a
|
|
62
|
+
scratch file (or leave it unset, for an ephemeral store) before seeding —
|
|
63
|
+
never at a deployment holding real tenant data.
|
|
64
|
+
|
|
65
|
+
## Then look around
|
|
66
|
+
|
|
67
|
+
Once seeded, the CLI prints the same six paths listed above. If Studio isn't
|
|
68
|
+
running yet, boot it the way you already do — see [Running
|
|
69
|
+
locally](RUNNING-LOCAL.md#boot) for a checkout, or [Embedding](EMBEDDING.md)
|
|
70
|
+
for `Insika.agent { … }.serve` — pointed at the same `INSIKA_DB`. Studio reads
|
|
71
|
+
every agent's profile from the same config store, so it will show
|
|
72
|
+
`demo-store` next to whatever agent you're actually building, no matter which
|
|
73
|
+
one the running process itself defines.
|
|
74
|
+
|
|
75
|
+
## See also
|
|
76
|
+
|
|
77
|
+
- [Refinement](REFINEMENT.md), [Facts](FACTS.md), [Evals](EVALS.md) — what
|
|
78
|
+
each seeded page actually means.
|
|
79
|
+
- [Running locally](RUNNING-LOCAL.md) — booting Studio against a durable
|
|
80
|
+
`INSIKA_DB`.
|
data/docs/DEPLOY.md
CHANGED
|
@@ -57,7 +57,14 @@ contract:
|
|
|
57
57
|
worker that holds the session's actor. The engine does **not** promise them
|
|
58
58
|
across workers. A deploy that needs those semantics for a session must route
|
|
59
59
|
that session's traffic to one worker (sticky routing) — or accept per-worker
|
|
60
|
-
best-effort.
|
|
60
|
+
best-effort. **On Railway there is no sticky-routing option at any layer** —
|
|
61
|
+
confirmed against Railway's own docs, which distribute traffic randomly and
|
|
62
|
+
explicitly do not support sticky sessions — so N>1 there is not "per-worker
|
|
63
|
+
best-effort," it is a guaranteed cross-session leak the first time two
|
|
64
|
+
requests for the same session land on different workers. See the Railway
|
|
65
|
+
section below; `insika doctor` errors on `WEB_CONCURRENCY>1` there — unless
|
|
66
|
+
[`insika-router`](ROUTER.md), the session-sticky proxy, is in front,
|
|
67
|
+
which is what makes N>1 safe on Railway too (N local workers behind it).
|
|
61
68
|
3. **Recovery is part of boot, in every wiring.** Every worker boots through
|
|
62
69
|
`Server::Boot`, which runs recovery **before the listen**. The per-record
|
|
63
70
|
sweeps (undelivered outbox records, undelivered delegation results) run in
|
|
@@ -118,7 +125,7 @@ this section is the single source of truth for what changing it means.
|
|
|
118
125
|
| `INSIKA_RELAY_TOKEN` | — | **mounts the relay channel** at `POST /channels/relay/events`, and is the Bearer it requires. Empty = the route does not exist (`404`). See [Channels](CHANNELS.md) |
|
|
119
126
|
| `INSIKA_RELAY_DELIVER_URL` | — | your callback; the engine POSTs each reply there. Goes through the egress guard |
|
|
120
127
|
| `INSIKA_RELAY_DELIVER_TOKEN` | — | Bearer the engine sends **to** your callback (optional) |
|
|
121
|
-
| `INSIKA_RELAY_DELIVERY` | `at_end` | how the relay flushes the outbox: `at_end` (one POST) or `progressive` (one POST per balloon — [
|
|
128
|
+
| `INSIKA_RELAY_DELIVERY` | `at_end` | how the relay flushes the outbox: `at_end` (one POST) or `progressive` (one POST per balloon — [delivery policy](CHANNELS.md#delivery-policy)) |
|
|
122
129
|
| `INSIKA_WIDGET_ORIGINS` | — | exact-match origins allowed to embed the [web widget](CHANNELS.md#the-web-widget), comma-separated. No wildcards. **Half the switch**: with `INSIKA_WIDGET_AGENTS` unset, nothing is mounted (`404`) |
|
|
123
130
|
| `INSIKA_WIDGET_AGENTS` | — | agent ids a widget visitor may address, comma-separated. The other half of the switch. **A chat rate limit is also required** or the widget answers `503` |
|
|
124
131
|
| `LITESTREAM_REPLICA_URL` | — | **enables Litestream** (backup/DR). Empty = disabled (default). See below |
|
|
@@ -221,11 +228,64 @@ healthcheck, and a restart policy.
|
|
|
221
228
|
without a volume, SQLite is ephemeral and recovery resumes nothing after a
|
|
222
229
|
redeploy.
|
|
223
230
|
3. **Vars**: `DEEPSEEK_API_KEY`, `OPENCLAW_GATEWAY_TOKEN`, `CONSUMER_INTERNAL_URL`,
|
|
224
|
-
`INSIKA_EGRESS_HOSTS
|
|
231
|
+
`INSIKA_EGRESS_HOSTS`. **Leave `WEB_CONCURRENCY` at its default of 1** unless
|
|
232
|
+
you run [`insika-router`](ROUTER.md) in front. Railway's own docs say it
|
|
233
|
+
"does not support sticky sessions" and randomly distributes traffic across
|
|
234
|
+
replicas/workers — there is no way on this platform to satisfy the "sticky
|
|
235
|
+
routing per session in front" precondition item 2 of the process model
|
|
236
|
+
requires, at any layer (Railway replicas or Falcon's own `--count` workers
|
|
237
|
+
within one container). Raising `WEB_CONCURRENCY` without a router in front
|
|
238
|
+
is not a throughput knob, it is a guaranteed way to leak a reply across
|
|
239
|
+
sessions the first time two requests for the same session land on different
|
|
240
|
+
workers — `insika doctor` errors on this combination for exactly that
|
|
241
|
+
reason. `insika-router` is the fix: N local Falcon workers
|
|
242
|
+
behind one sticky proxy, still one Railway replica.
|
|
225
243
|
4. The healthcheck hits `/up`.
|
|
226
244
|
5. Point your consumer at the service's public URL, with a matching API token
|
|
227
245
|
(see [RUNNING-LOCAL.md](RUNNING-LOCAL.md)).
|
|
228
246
|
|
|
247
|
+
## Kubernetes
|
|
248
|
+
|
|
249
|
+
Same contract as Railway, for the same reason, verified against
|
|
250
|
+
ingress-nginx's own docs: **run one pod, `WEB_CONCURRENCY=1`.** Scale
|
|
251
|
+
vertically (bigger pod) for now.
|
|
252
|
+
|
|
253
|
+
Why a standard K8s setup does not get you out of this: the session id lives
|
|
254
|
+
in the **JSON body** of `POST /v1/responses` (the `user` field), not in a
|
|
255
|
+
header, cookie, or URL segment. That rules out every sticky-routing
|
|
256
|
+
mechanism K8s gives you for free:
|
|
257
|
+
|
|
258
|
+
- A vanilla `Service` (`ClusterIP`) load-balances across pod endpoints with
|
|
259
|
+
no notion of session at all — same failure mode as Railway's replicas,
|
|
260
|
+
one layer down.
|
|
261
|
+
- `Service.spec.sessionAffinity: ClientIP` does not help either: the caller
|
|
262
|
+
is the consumer's own backend (achei-b2b, a server-to-server call), not
|
|
263
|
+
the end customer's browser, so many different customers' sessions arrive
|
|
264
|
+
from the same source IP and would collide on the same pod instead of
|
|
265
|
+
spreading — the opposite of what you want, and it still does not restore
|
|
266
|
+
a real per-session guarantee.
|
|
267
|
+
- ingress-nginx's `nginx.ingress.kubernetes.io/upstream-hash-by` (the
|
|
268
|
+
standard sticky/consistent-hash annotation) hashes on nginx *variables* —
|
|
269
|
+
headers, cookies, `$request_uri`, client IP — not on a field parsed out of
|
|
270
|
+
a POST body. Reaching into the JSON body needs a custom Lua/OpenResty
|
|
271
|
+
snippet (or an Envoy filter) that parses the request and extracts `user`
|
|
272
|
+
before hashing. That is real, unbuilt engineering work, not a K8s
|
|
273
|
+
annotation to flip.
|
|
274
|
+
|
|
275
|
+
If real horizontal throughput is ever needed, two paths get there, neither
|
|
276
|
+
of them "just add replicas":
|
|
277
|
+
1. **Build the body-aware sticky layer** above (Lua/Envoy consistent-hash on
|
|
278
|
+
the `user` field) in front of N pods, each still `WEB_CONCURRENCY=1`.
|
|
279
|
+
2. **Move the session id into the URL**, the way the web widget transport
|
|
280
|
+
already does (`POST /api/widget/sessions/:token/messages` carries the
|
|
281
|
+
session token in the path) — that surface, unlike `/v1/responses`, *is*
|
|
282
|
+
sticky-routable today with a plain `upstream-hash-by: $request_uri`, no
|
|
283
|
+
custom scripting required.
|
|
284
|
+
|
|
285
|
+
Neither is urgent while a single pod's throughput is enough — this section
|
|
286
|
+
exists so scaling this deployment does not silently reintroduce the same
|
|
287
|
+
cross-session leak the Railway incident above already found once.
|
|
288
|
+
|
|
229
289
|
## Backup / DR — Litestream (opt-in, configurable)
|
|
230
290
|
|
|
231
291
|
A single volume is the **one point of total loss** between a pilot and production
|
|
@@ -300,6 +360,12 @@ per pod + **sticky-by-agent** routing (shard by tenant), or **LiteFS**, or an
|
|
|
300
360
|
optional **Postgres** adapter. **Litestream** (above) for backup/DR from day one —
|
|
301
361
|
orthogonal to topology.
|
|
302
362
|
|
|
363
|
+
For the *session-routing* half specifically — as opposed to the storage
|
|
364
|
+
topology above — see [`insika-router`](ROUTER.md): N pods behind a
|
|
365
|
+
headless `Service`, one `insika-router` Deployment in front doing the
|
|
366
|
+
consistent-hash routing ingress-nginx's own `upstream-hash-by` cannot (it
|
|
367
|
+
hashes nginx variables, never a field parsed out of a POST body).
|
|
368
|
+
|
|
303
369
|
---
|
|
304
370
|
|
|
305
371
|
## Measuring performance / load
|
|
@@ -351,6 +417,8 @@ DEEPSEEK_API_KEY=sk-... ./scripts/loadtest-local.sh 4 24
|
|
|
351
417
|
|
|
352
418
|
## See also
|
|
353
419
|
|
|
420
|
+
- [ROUTER.md](ROUTER.md) — `insika-router`, the session-sticky proxy for
|
|
421
|
+
scaling past one worker.
|
|
354
422
|
- [RUNNING-LOCAL.md](RUNNING-LOCAL.md) — run the engine locally, single-process.
|
|
355
423
|
- [Security](SECURITY.md) — tokens, egress, strict config.
|
|
356
424
|
- [BENCHMARK.md](BENCHMARK.md) — the neutral, key-free engine benchmark.
|
data/docs/EMBEDDING.md
CHANGED
data/docs/EVALS.md
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
title: Evals
|
|
3
|
-
parent:
|
|
4
|
-
nav_order:
|
|
3
|
+
parent: Improve
|
|
4
|
+
nav_order: 1
|
|
5
5
|
permalink: /evals/
|
|
6
6
|
---
|
|
7
7
|
|
|
@@ -157,6 +157,82 @@ Three rules make the number worth quoting:
|
|
|
157
157
|
Cost: **two provider calls per judge per case**, which is why it is opt-in and never
|
|
158
158
|
part of the gate.
|
|
159
159
|
|
|
160
|
+
## Simulated users — conversations the agent never had
|
|
161
|
+
|
|
162
|
+
A golden case fixes the user's turns in advance. Real customers branch — they answer
|
|
163
|
+
the agent's question, or ignore it, or send an order number three messages later. A
|
|
164
|
+
case with frozen turns can only ever test the path the author imagined.
|
|
165
|
+
|
|
166
|
+
A **simulated** case generates the conversation instead: a persona model (the cheap
|
|
167
|
+
platform `utility_model`) plays a customer with the persona as its **whole
|
|
168
|
+
instruction**, and the target agent answers through the same transport a replay uses.
|
|
169
|
+
They talk until the persona's `max_turns`, or until the persona emits a stop marker —
|
|
170
|
+
`<<goal_met>>` when its goal is served, `<<gave_up>>` when it abandons. Both are
|
|
171
|
+
recorded: "gave up at turn 3" is a finding.
|
|
172
|
+
|
|
173
|
+
```yaml
|
|
174
|
+
id: loja-objetivo-difuso
|
|
175
|
+
agent: loja-chocolates
|
|
176
|
+
persona: # the SIMULATED customer (alternative to `turns:`)
|
|
177
|
+
goal: "descobrir o que comprar de presente; não sabe nomes de produto"
|
|
178
|
+
style: "mensagens curtas, responde o que perguntam, desiste se tiver que explicar duas vezes"
|
|
179
|
+
opens_with: "oi, queria um presente"
|
|
180
|
+
knows: # the ONLY facts the persona may assert
|
|
181
|
+
orcamento: "até R$ 100"
|
|
182
|
+
ocasião: "aniversário"
|
|
183
|
+
max_turns: 8
|
|
184
|
+
expect:
|
|
185
|
+
policy: investigate_first
|
|
186
|
+
rubric: |
|
|
187
|
+
Descobre o objetivo antes de recomendar (uma ou duas perguntas, não um formulário),
|
|
188
|
+
recomenda do catálogo real e fecha com um próximo passo claro. Reprova se despejar
|
|
189
|
+
catálogo antes de entender, ou se ficar perguntando sem nunca buscar.
|
|
190
|
+
min_score: 0.7
|
|
191
|
+
```
|
|
192
|
+
|
|
193
|
+
**The anti-invention rule is the soul of the feature.** The persona's prompt contains
|
|
194
|
+
only the `knows` facts and the rule that it may assert exactly those and nothing else
|
|
195
|
+
— asked about anything it does not have, it answers with ignorance ("não sei", "não
|
|
196
|
+
tenho isso aqui"), like a real customer without that fact. A simulator that invents an
|
|
197
|
+
order number produces a conversation the agent could never have had, and a case that
|
|
198
|
+
tests nothing.
|
|
199
|
+
|
|
200
|
+
```bash
|
|
201
|
+
insika evals:simulate --persona persona.yml --target loja-chocolates --staging
|
|
202
|
+
```
|
|
203
|
+
|
|
204
|
+
- `--target <agent|url>` — an agent id over the deployment's `/v1/responses`, or a full
|
|
205
|
+
A2A URL (an agent that only speaks A2A rides the same Simulator through a thin A2A
|
|
206
|
+
transport; it polls once per second, bounded by `--timeout`, since a remote agent
|
|
207
|
+
takes seconds to finish a task).
|
|
208
|
+
- `--staging` or `--eval-profile` — **one is required**. A simulated conversation
|
|
209
|
+
marks its transcript `simulated: true`; it does NOT disarm the agent's tools. So the
|
|
210
|
+
Simulator only runs against (a) a target the operator declares is a staging
|
|
211
|
+
deployment (`--staging`), or (b) an eval profile where the agent's side-effect tools
|
|
212
|
+
are swapped for dry-runs (`--eval-profile`).
|
|
213
|
+
- `--eval-profile` is a **verified declaration, not a trust-me flag**. The CLI derives
|
|
214
|
+
the target's side-effect tools — the deployment's own registry over
|
|
215
|
+
`GET /v1/agents/:id` (`side_effect_tools`), or the local store when `INSIKA_DB`
|
|
216
|
+
points at it — and refuses unless `--eval-tools <a,b,c>` names every one of them.
|
|
217
|
+
The swap itself happens where the tools run (the deployment's eval profile; the
|
|
218
|
+
in-process overlay for a local graph, as in `examples/agent-tester/`); the
|
|
219
|
+
derivation is what keeps the client honest about what must be covered. An A2A target
|
|
220
|
+
has no reachable registry, so the explicit `--eval-tools` list is required there.
|
|
221
|
+
- `--persona-model` — the model playing the customer (default: the platform
|
|
222
|
+
`utility_model`). The judge that scores the transcript comes from the same
|
|
223
|
+
panel configuration as a replay.
|
|
224
|
+
- `--conv` — the conversation id. The default is unique per run
|
|
225
|
+
(`sim-<case id>-<random>`), because two runs on the same conv share the
|
|
226
|
+
deployment session: the second inherits the first's history — including
|
|
227
|
+
across the arms of an A/B comparison — and the grade stops measuring the
|
|
228
|
+
agent. Pass `--conv` only when continuing a session is the point.
|
|
229
|
+
|
|
230
|
+
Every simulated run is **`simulated: true`** — the transcript carries the flag end to
|
|
231
|
+
end, so a report never mixes a generated conversation with real traffic. A simulated
|
|
232
|
+
case (`persona:`) is one of the two shapes of a case; the replay Runner **skips** it
|
|
233
|
+
(the turns do not exist until a Simulator generates them) and only the simulate CLI
|
|
234
|
+
drives it.
|
|
235
|
+
|
|
160
236
|
Cases live in two places, and it is the same YAML in both:
|
|
161
237
|
|
|
162
238
|
- **`evals/golden/**`** in the repo — the curated corpus, reviewable in a pull request,
|
|
@@ -178,6 +254,55 @@ A case whose stored YAML no longer validates is **listed as broken** on the Eval
|
|
|
178
254
|
rather than skipped in silence — a test suite that quietly shrinks is worse than a red
|
|
179
255
|
one.
|
|
180
256
|
|
|
257
|
+
### `run_persona_eval` — a QA agent that tests other agents
|
|
258
|
+
|
|
259
|
+
A tool (`lib/insika/tools/run_persona_eval.rb`), not a CLI: a QA agent picks one of its
|
|
260
|
+
authored simulated cases and runs it **in-process**, against the case's declared
|
|
261
|
+
target agent — the same Simulator + Judge machinery `evals:simulate` drives over HTTP,
|
|
262
|
+
minus the network hop.
|
|
263
|
+
|
|
264
|
+
```ruby
|
|
265
|
+
tools %w[run_persona_eval save_artifact]
|
|
266
|
+
```
|
|
267
|
+
|
|
268
|
+
The model only sees `case_id`, enumerated with the ids the tool can actually run
|
|
269
|
+
(never a free string it could invent) — every **simulated** case in the store, the same
|
|
270
|
+
`persona:` shape as above.
|
|
271
|
+
|
|
272
|
+
**Safety is derived here too, but there is no swap yet.** The tool computes the
|
|
273
|
+
target's reachable side-effect tools (`Evals::EvalProfile`, the same derivation the CLI
|
|
274
|
+
uses) and **refuses outright** — naming the tools — if that list is non-empty. Unlike
|
|
275
|
+
the CLI, nothing here actually swaps a side-effect tool for a dry-run: `Evals::
|
|
276
|
+
EvalProfile.registry` (the overlay) exists for exactly that, but nothing calls it yet.
|
|
277
|
+
So `run_persona_eval` only runs against **read-only** target agents today; wiring the
|
|
278
|
+
overlay into an in-process turn (so a target WITH a write tool can be tested safely) is
|
|
279
|
+
follow-up work, not something this tool claims to do.
|
|
280
|
+
|
|
281
|
+
**Budget**: the persona model + judge model calls are the cost of running the eval,
|
|
282
|
+
charged to the **calling** agent's own turn — never the target's (the target's own
|
|
283
|
+
turns bill normally, through the ordinary edge limiter, exactly as a real customer's
|
|
284
|
+
would). A hard cap on the calling agent skips the run — visibly, in the tool's own
|
|
285
|
+
result (`{skipped: true, reason: "budget", window: "daily"|"monthly"}`) — before a cent
|
|
286
|
+
is spent; a soft cap just runs (the ledger's own alert already warns).
|
|
287
|
+
|
|
288
|
+
Each run gets a **fresh session** (`eval-<case>-<random>`), never a reused one — a
|
|
289
|
+
reused session lets the target agent "remember" earlier runs, which is not what a
|
|
290
|
+
persona case is testing. See `examples/agent-tester/qa_scheduled.rb` for the full
|
|
291
|
+
loop: `schedule` fires the QA agent, it calls `run_persona_eval`, then
|
|
292
|
+
`save_artifact` publishes the quality report.
|
|
293
|
+
|
|
294
|
+
**Tenant isolation.** A golden case carries a `tenant` (`platform` — the
|
|
295
|
+
single-tenant default — unless the case declares one, the same convention
|
|
296
|
+
`save_artifact` uses for its own binding tenant). `run_persona_eval` only ever
|
|
297
|
+
lists or runs cases in the *calling* agent's own tenant: a case authored for
|
|
298
|
+
another tenant is invisible, not merely refused — the model cannot even learn
|
|
299
|
+
its id from the enum, and running it by a guessed id gets the same "unknown or
|
|
300
|
+
invalid" error a nonexistent id would (the two must not be distinguishable).
|
|
301
|
+
This is what makes "one QA agent per store" an actual boundary
|
|
302
|
+
in a deployment where several stores' persona cases live in the same
|
|
303
|
+
`GoldenStore` — without it, `qa-store-a` could enumerate and run
|
|
304
|
+
`qa-store-b`'s persona and read its `knows` in the transcript.
|
|
305
|
+
|
|
181
306
|
## Running
|
|
182
307
|
|
|
183
308
|
```bash
|
|
@@ -235,7 +360,7 @@ not a regression.
|
|
|
235
360
|
The same gate machinery guards the two automated loops — [Refinement](REFINEMENT.md)
|
|
236
361
|
(cloned-agent replay of a proposed edit) and [Harvest](HARVEST.md) (cloned-agent
|
|
237
362
|
replay of a mined skill). **Judges are mandatory for those gates** in exactly three
|
|
238
|
-
shapes
|
|
363
|
+
shapes: a gate without a recorded baseline refuses, an all-red
|
|
239
364
|
baseline refuses, and a baseline recorded WITH judge scores, replayed with no judge
|
|
240
365
|
configured, refuses — a rubric'd case with no verdict reads as a pass, so the
|
|
241
366
|
candidate would beat a measurement it never took. A store with no golden cases cannot
|
data/docs/FACTS.md
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
title: Facts
|
|
3
|
-
parent:
|
|
4
|
-
nav_order:
|
|
3
|
+
parent: Improve
|
|
4
|
+
nav_order: 5
|
|
5
5
|
permalink: /facts/
|
|
6
6
|
---
|
|
7
7
|
|
|
@@ -92,7 +92,7 @@ visible — the operator's edit always wins, never a silent overwrite.
|
|
|
92
92
|
## Provenance
|
|
93
93
|
|
|
94
94
|
An approved fact is written with `origin: "distilled:<session_ref>"` — the
|
|
95
|
-
|
|
95
|
+
The provenance discipline: `"engine"` (the `remember` tool), `"operator"`
|
|
96
96
|
(Studio edits), `"legacy"`, and `"distilled"` (+ the session that produced it).
|
|
97
97
|
The Closed loop reads the same cell every later turn injects, and the
|
|
98
98
|
distillation ledger suppresses re-proposing a fact that is already applied with
|