insika 0.9.0 → 1.0.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +237 -2
- data/README.md +5 -3
- data/bin/insika +3 -0
- data/bin/insika-router +0 -1
- data/docs/AGENTS.md +90 -7
- data/docs/ARCHITECTURE.md +12 -7
- data/docs/ARTIFACTS.md +33 -3
- data/docs/BENCHMARK.md +50 -7
- data/docs/CONTEXT.md +63 -0
- data/docs/CROSS-HARNESS-BENCH.md +157 -0
- data/docs/DEPLOY.md +57 -11
- data/docs/EMBEDDING.md +10 -0
- data/docs/EVALS.md +33 -4
- data/docs/FACTS.md +8 -6
- data/docs/HARVEST.md +8 -3
- data/docs/KNOWLEDGE.md +7 -0
- data/docs/MEDIA.md +6 -2
- data/docs/OBSERVABILITY.md +143 -22
- data/docs/PERFORMANCE.md +110 -0
- data/docs/PLUGINS.md +4 -4
- data/docs/POLICY.md +22 -13
- data/docs/RELEASING.md +5 -0
- data/docs/ROUTER.md +7 -0
- data/docs/RUBYLLM_2_MIGRATION.md +307 -0
- data/docs/SECURITY.md +7 -0
- data/docs/SHARED-CONVERSATIONS.md +40 -0
- data/docs/SKILLS.md +5 -3
- data/docs/STUDIO.md +122 -0
- data/docs/TOOLS.md +111 -9
- data/docs/operate.md +3 -0
- data/lib/insika/agent_defaults.rb +140 -0
- data/lib/insika/agent_file_store.rb +12 -0
- data/lib/insika/agent_profile.rb +78 -8
- data/lib/insika/artifact_signing.rb +11 -0
- data/lib/insika/chat_builder.rb +79 -25
- data/lib/insika/checkpoint.rb +6 -1
- data/lib/insika/checkpoint_store.rb +59 -7
- data/lib/insika/commands/agent_payload.rb +3 -3
- data/lib/insika/commands/delete_kit.rb +26 -0
- data/lib/insika/commands/gate_harvest.rb +3 -0
- data/lib/insika/commands/review_guardrail_flag.rb +38 -0
- data/lib/insika/commands/run_harvest.rb +17 -1
- data/lib/insika/commands/seed_session.rb +11 -1
- data/lib/insika/commands/send_message.rb +9 -3
- data/lib/insika/commands/update_settings.rb +23 -1
- data/lib/insika/commands/write_kit.rb +27 -0
- data/lib/insika/compaction.rb +33 -7
- data/lib/insika/config_store.rb +87 -7
- data/lib/insika/context/priority.rb +3 -0
- data/lib/insika/context/provider.rb +2 -2
- data/lib/insika/context/providers/knowledge.rb +25 -2
- data/lib/insika/context/providers/memory.rb +53 -4
- data/lib/insika/context/providers/pending_confirmation.rb +52 -0
- data/lib/insika/context/providers/prompt.rb +17 -4
- data/lib/insika/context/providers/session.rb +39 -3
- data/lib/insika/context/reranker.rb +75 -0
- data/lib/insika/distill.rb +22 -13
- data/lib/insika/doctor.rb +34 -4
- data/lib/insika/dsl/runtime.rb +36 -4
- data/lib/insika/dsl/server_boot.rb +4 -12
- data/lib/insika/dsl.rb +13 -5
- data/lib/insika/edge_limiter.rb +23 -15
- data/lib/insika/egress_guard.rb +11 -3
- data/lib/insika/env_schema.rb +33 -73
- data/lib/insika/evals/assertions.rb +171 -16
- data/lib/insika/evals/golden.rb +141 -4
- data/lib/insika/evals/judge.rb +19 -10
- data/lib/insika/evals/moderator_compare.rb +47 -0
- data/lib/insika/evals/report.rb +61 -9
- data/lib/insika/evals/runner.rb +44 -4
- data/lib/insika/evals/simulator.rb +10 -1
- data/lib/insika/evals/transport.rb +200 -0
- data/lib/insika/event_stream.rb +63 -7
- data/lib/insika/evidence.rb +29 -8
- data/lib/insika/executor.rb +668 -141
- data/lib/insika/harvest.rb +26 -14
- data/lib/insika/harvest_engine.rb +1 -0
- data/lib/insika/harvest_store.rb +12 -0
- data/lib/insika/http_client.rb +87 -4
- data/lib/insika/kits.rb +54 -0
- data/lib/insika/knowledge.rb +34 -20
- data/lib/insika/llm_configurator.rb +46 -4
- data/lib/insika/llm_provider_store.rb +1 -3
- data/lib/insika/llm_trace_store.rb +54 -0
- data/lib/insika/mcp_client.rb +27 -63
- data/lib/insika/mcp_live_tool.rb +55 -9
- data/lib/insika/mcp_store.rb +32 -1
- data/lib/insika/mcp_tool_registry.rb +69 -26
- data/lib/insika/media.rb +38 -25
- data/lib/insika/memory_store.rb +4 -3
- data/lib/insika/memory_trim.rb +60 -0
- data/lib/insika/middleware.rb +12 -3
- data/lib/insika/model_metrics_store.rb +285 -0
- data/lib/insika/model_registry.rb +110 -0
- data/lib/insika/model_resolver.rb +1 -1
- data/lib/insika/model_selection.rb +14 -21
- data/lib/insika/model_visible.rb +17 -6
- data/lib/insika/onboarding.rb +1 -0
- data/lib/insika/outcome_store.rb +1 -1
- data/lib/insika/overlay_tool_registry.rb +25 -2
- data/lib/insika/pack_importer.rb +54 -9
- data/lib/insika/packaging.rb +14 -5
- data/lib/insika/pending_action_store.rb +45 -18
- data/lib/insika/plugin/loader.rb +4 -2
- data/lib/insika/profile_source.rb +4 -0
- data/lib/insika/provider_error_classifier.rb +18 -3
- data/lib/insika/queue_policy.rb +7 -2
- data/lib/insika/refinement/proposer.rb +42 -17
- data/lib/insika/reliability.rb +12 -3
- data/lib/insika/router/app.rb +14 -5
- data/lib/insika/router/session_key.rb +35 -5
- data/lib/insika/safety/config.rb +20 -3
- data/lib/insika/safety/corpus.rb +10 -3
- data/lib/insika/safety/factory.rb +39 -1
- data/lib/insika/safety/input_guardrail.rb +21 -1
- data/lib/insika/safety/judge_moderator.rb +70 -0
- data/lib/insika/safety/output_validator.rb +29 -5
- data/lib/insika/safety/prompt_echo.rb +89 -0
- data/lib/insika/schema_guard.rb +53 -76
- data/lib/insika/server/app.rb +17 -8
- data/lib/insika/server/responses.rb +19 -1
- data/lib/insika/session_actor.rb +58 -28
- data/lib/insika/session_store.rb +159 -10
- data/lib/insika/settings_store.rb +47 -1
- data/lib/insika/shared_conversations.rb +305 -0
- data/lib/insika/skill_catalog.rb +28 -2
- data/lib/insika/skill_store.rb +6 -0
- data/lib/insika/soak/runner.rb +1 -1
- data/lib/insika/steer_injector.rb +7 -48
- data/lib/insika/store.rb +29 -1
- data/lib/insika/stores/json_value.rb +37 -0
- data/lib/insika/stores/memory.rb +22 -35
- data/lib/insika/stores/postgres.rb +295 -0
- data/lib/insika/stores/sqlite.rb +257 -43
- data/lib/insika/stores/turn_counter.rb +25 -0
- data/lib/insika/studio/app.rb +1453 -161
- data/lib/insika/studio/assets/dist/application.css +1 -1
- data/lib/insika/studio/assets/dist/application.js +25 -25
- data/lib/insika/studio/assets/dist/fonts/ibm-plex-mono-latin-400-normal.woff2 +0 -0
- data/lib/insika/studio/assets/dist/fonts/ibm-plex-mono-latin-500-normal.woff2 +0 -0
- data/lib/insika/studio/assets/dist/fonts/ibm-plex-mono-latin-600-normal.woff2 +0 -0
- data/lib/insika/studio/assets/dist/fonts/ibm-plex-sans-latin-wght-normal.woff2 +0 -0
- data/lib/insika/studio/forms.rb +171 -9
- data/lib/insika/studio/nav_icons.rb +3 -0
- data/lib/insika/studio/views/_agent_config_retrieval.erb +73 -0
- data/lib/insika/studio/views/_agent_picker.erb +15 -0
- data/lib/insika/studio/views/_agent_tab_config.erb +78 -242
- data/lib/insika/studio/views/_agent_tab_history.erb +56 -1
- data/lib/insika/studio/views/_agent_tab_performance.erb +34 -0
- data/lib/insika/studio/views/_agent_tab_prompts.erb +7 -9
- data/lib/insika/studio/views/_agent_tab_skills.erb +16 -7
- data/lib/insika/studio/views/_agent_tab_tools.erb +90 -0
- data/lib/insika/studio/views/_agents_master.erb +8 -12
- data/lib/insika/studio/views/_cfg_alerts.erb +4 -0
- data/lib/insika/studio/views/_cfg_budget.erb +20 -0
- data/lib/insika/studio/views/_cfg_distill.erb +26 -0
- data/lib/insika/studio/views/_cfg_followup.erb +30 -0
- data/lib/insika/studio/views/_cfg_grounding.erb +21 -0
- data/lib/insika/studio/views/_cfg_guardrails.erb +47 -0
- data/lib/insika/studio/views/_cfg_harvest.erb +35 -0
- data/lib/insika/studio/views/_cfg_reliability.erb +32 -0
- data/lib/insika/studio/views/_chats_master.erb +66 -0
- data/lib/insika/studio/views/_customers_master.erb +37 -0
- data/lib/insika/studio/views/_empty.erb +15 -0
- data/lib/insika/studio/views/_facts_master.erb +42 -0
- data/lib/insika/studio/views/_harvest_candidate_detail.erb +14 -0
- data/lib/insika/studio/views/_identity.erb +16 -0
- data/lib/insika/studio/views/_kpi.erb +17 -0
- data/lib/insika/studio/views/_message.erb +40 -2
- data/lib/insika/studio/views/_model_chart.erb +82 -0
- data/lib/insika/studio/views/_refinement_master.erb +38 -0
- data/lib/insika/studio/views/{tool_edit.erb → _tool_edit.erb} +32 -9
- data/lib/insika/studio/views/_tool_users.erb +17 -0
- data/lib/insika/studio/views/_why.erb +53 -0
- data/lib/insika/studio/views/agent_detail.erb +17 -10
- data/lib/insika/studio/views/agents.erb +28 -16
- data/lib/insika/studio/views/approvals.erb +16 -11
- data/lib/insika/studio/views/artifact.erb +20 -7
- data/lib/insika/studio/views/artifacts.erb +19 -14
- data/lib/insika/studio/views/chats.erb +22 -29
- data/lib/insika/studio/views/customer.erb +88 -43
- data/lib/insika/studio/views/customers.erb +16 -26
- data/lib/insika/studio/views/evals.erb +49 -31
- data/lib/insika/studio/views/fact_detail.erb +114 -0
- data/lib/insika/studio/views/facts.erb +15 -116
- data/lib/insika/studio/views/followups.erb +63 -52
- data/lib/insika/studio/views/funnel.erb +104 -43
- data/lib/insika/studio/views/harvest.erb +85 -59
- data/lib/insika/studio/views/home.erb +138 -23
- data/lib/insika/studio/views/inbox.erb +36 -0
- data/lib/insika/studio/views/kits.erb +176 -0
- data/lib/insika/studio/views/knowledge.erb +92 -80
- data/lib/insika/studio/views/layout.erb +42 -11
- data/lib/insika/studio/views/live.erb +45 -0
- data/lib/insika/studio/views/mcp.erb +43 -13
- data/lib/insika/studio/views/models.erb +187 -0
- data/lib/insika/studio/views/parity.erb +38 -32
- data/lib/insika/studio/views/playground.erb +13 -9
- data/lib/insika/studio/views/refinement.erb +15 -224
- data/lib/insika/studio/views/refinement_detail.erb +243 -0
- data/lib/insika/studio/views/session.erb +152 -117
- data/lib/insika/studio/views/settings.erb +86 -22
- data/lib/insika/studio/views/skills.erb +53 -19
- data/lib/insika/studio/views/system_files.erb +7 -7
- data/lib/insika/studio/views/task.erb +45 -15
- data/lib/insika/studio/views/tasks.erb +35 -18
- data/lib/insika/studio/views/tools.erb +62 -73
- data/lib/insika/task_actor.rb +4 -1
- data/lib/insika/task_store.rb +67 -1
- data/lib/insika/telemetry/pricing.rb +16 -14
- data/lib/insika/telemetry/recorder.rb +103 -9
- data/lib/insika/telemetry/ruby_llm_instrumenter.rb +110 -0
- data/lib/insika/telemetry.rb +12 -2
- data/lib/insika/templates/daily-digest/README.md +13 -0
- data/lib/insika/templates/daily-digest/agent.rb +22 -11
- data/lib/insika/testing/store_contract.rb +65 -0
- data/lib/insika/text_tool_call.rb +66 -0
- data/lib/insika/tick.rb +7 -1
- data/lib/insika/token_store.rb +5 -0
- data/lib/insika/tool_batch.rb +1 -1
- data/lib/insika/tool_catalog.rb +12 -2
- data/lib/insika/tool_definition.rb +10 -4
- data/lib/insika/tool_envelope.rb +165 -28
- data/lib/insika/tool_manifest.rb +1 -0
- data/lib/insika/tool_store.rb +3 -0
- data/lib/insika/tool_trace_store.rb +15 -8
- data/lib/insika/tool_usage_report.rb +2 -2
- data/lib/insika/tools/a2a_remote.rb +1 -1
- data/lib/insika/tools/agent_enum.rb +1 -1
- data/lib/insika/tools/confirm_pending.rb +112 -0
- data/lib/insika/tools/data_defined_tool.rb +23 -15
- data/lib/insika/tools/generate_image.rb +2 -2
- data/lib/insika/tools/load_knowledge.rb +7 -3
- data/lib/insika/tools/load_skill.rb +3 -3
- data/lib/insika/tools/present.rb +1 -8
- data/lib/insika/tools/remember.rb +7 -2
- data/lib/insika/tools/run_persona_eval.rb +3 -5
- data/lib/insika/tools/save_artifact.rb +16 -6
- data/lib/insika/tools/schedule_followup.rb +3 -3
- data/lib/insika/tools/stuck_signal.rb +3 -3
- data/lib/insika/tools/subagent.rb +5 -5
- data/lib/insika/tools/subagents.rb +3 -3
- data/lib/insika/tools/tool_search.rb +7 -5
- data/lib/insika/tools/tts.rb +2 -2
- data/lib/insika/tools/update_briefing.rb +4 -4
- data/lib/insika/turn_output.rb +2 -2
- data/lib/insika/turn_state.rb +18 -0
- data/lib/insika/turn_timing.rb +51 -14
- data/lib/insika/version.rb +1 -1
- data/lib/insika/wiring/graph.rb +81 -16
- data/lib/insika/workflow.rb +25 -64
- data/lib/insika.rb +19 -3
- metadata +69 -13
- data/lib/insika/commands/import_mcp_tools.rb +0 -48
- data/lib/insika/mcp_http_client.rb +0 -67
- data/lib/insika/mcp_tool_ingestor.rb +0 -165
data/docs/CONTEXT.md
CHANGED
|
@@ -87,6 +87,28 @@ Compaction does not replace the budget: eviction stays as the hard backstop
|
|
|
87
87
|
for a single oversized turn. Tune `context_budget` and keep the identity lean
|
|
88
88
|
regardless.
|
|
89
89
|
|
|
90
|
+
#### Native Responses compaction
|
|
91
|
+
|
|
92
|
+
Set `compaction.mode` to `"native"` alongside `enabled: true` to use RubyLLM's
|
|
93
|
+
manual compaction endpoint on supported OpenAI, Azure and xAI Responses chats.
|
|
94
|
+
The default is `"summary"`, described above. Native mode uses the turn's resolved
|
|
95
|
+
model and credential context; `compaction.model`, `prompt` and `utility_model`
|
|
96
|
+
do not apply. Unsupported providers remain uncompacted; there is no second
|
|
97
|
+
summary request or automatic switch to another provider.
|
|
98
|
+
|
|
99
|
+
After the turn commits, a fresh chat compacts the stored, filtered prefix.
|
|
100
|
+
The original messages remain stored. Opaque state is separate from the spoken
|
|
101
|
+
transcript and is replayed only with the same provider, model and protocol.
|
|
102
|
+
Fallbacks and protocol changes restore the original prefix. Checkpoints retain
|
|
103
|
+
both alternatives. A native marker is also skipped if another context provider
|
|
104
|
+
has already supplied history, because Responses would replace that history.
|
|
105
|
+
|
|
106
|
+
The budget conservatively counts the original prefix as well as the native state,
|
|
107
|
+
and can evict this entire unit. This version reduces provider input when that unit
|
|
108
|
+
fits; it does not increase local context-budget capacity. Each compaction rebuilds
|
|
109
|
+
the whole stored prefix rather than chaining opaque state. Native usage is reported
|
|
110
|
+
on `context_compacted`; the opaque payload is not included in that event.
|
|
111
|
+
|
|
90
112
|
One cheap half **is** wired, opt-in per agent: `tool_output_compression` (DSL
|
|
91
113
|
`tool_output_compression`, or `"tool_output_compression": true` in the pack).
|
|
92
114
|
When on, byte-identical repeated **tool results** in the replayed history
|
|
@@ -102,6 +124,15 @@ budget first.
|
|
|
102
124
|
> and the budget verdict (`used / cap`, evicted sources). Counts only, never
|
|
103
125
|
> fragment content.
|
|
104
126
|
|
|
127
|
+
### Repairing stored assistant context
|
|
128
|
+
|
|
129
|
+
An operator may attach a string `context_content` to a stored plain assistant
|
|
130
|
+
message. Session replay uses that text while preserving the original `content`
|
|
131
|
+
for display and audit. User messages, tool results, assistant tool calls, explicit
|
|
132
|
+
history and checkpoint replay ignore this field. Existing compaction summaries
|
|
133
|
+
are not rewritten. This is a targeted data repair, not an automatic tool-markup
|
|
134
|
+
parser; text is never converted into tool execution.
|
|
135
|
+
|
|
105
136
|
## Memory
|
|
106
137
|
|
|
107
138
|
With `memory` enabled, an agent gains a built-in `remember` tool for durable
|
|
@@ -115,6 +146,38 @@ conversation; memory is the small set of facts that should outlive any single
|
|
|
115
146
|
conversation. Facts and notes are editable from the Studio agent page. See
|
|
116
147
|
[`examples/memory/`](https://github.com/guizaols/insika/tree/main/examples/memory/) for a runnable cross-session example.
|
|
117
148
|
|
|
149
|
+
In Studio, open **Agents → select an agent → Configuration → Memory & knowledge**.
|
|
150
|
+
Memory, knowledge learning and knowledge retrieval have separate switches. Each
|
|
151
|
+
source has its own optional rerank provider, model, candidate limit and timeout;
|
|
152
|
+
reranking is off by default. Disabling rerank keeps memory and knowledge working
|
|
153
|
+
with their usual retrieval and clears only that source's rerank configuration.
|
|
154
|
+
The memory selection count applies only while memory reranking is on. The knowledge
|
|
155
|
+
selection count also applies without reranking. Provider credentials are configured
|
|
156
|
+
separately, never in this form. Changes apply to this agent's future turns.
|
|
157
|
+
|
|
158
|
+
To select a smaller memory block for each message, keep `"memory": true` and add
|
|
159
|
+
`"memory_retrieval": {"top_k": 3, "rerank": {"provider": "cohere", "model": "rerank-v3.5", "candidate_limit": 20, "timeout_seconds": 2}}`
|
|
160
|
+
to the agent profile or pack. The engine first reads only the current memory cell,
|
|
161
|
+
ranks active facts and considers the newest `candidate_limit` notes by word overlap and recency,
|
|
162
|
+
then sends at most `candidate_limit` redacted candidates to the configured reranker.
|
|
163
|
+
An empty message or reranker failure keeps the usual full facts and ten recent notes.
|
|
164
|
+
Older notes outside that read window remain available in the store and Studio, but
|
|
165
|
+
cannot be selected for the turn.
|
|
166
|
+
|
|
167
|
+
Knowledge can use the same opt-in reranker: set `knowledge.retrieve: true`,
|
|
168
|
+
`knowledge.top_k: 3`, and `knowledge.rerank` to the `provider`, `model`,
|
|
169
|
+
`candidate_limit`, and `timeout_seconds` object above. The Scan index first
|
|
170
|
+
finds matching concepts in the agent and tenant scope. The reranker can reorder
|
|
171
|
+
only those candidates; a paraphrase with no matching terms still finds nothing.
|
|
172
|
+
Knowledge sends the candidate name, description, and body; memory sends the
|
|
173
|
+
fact key/value or note text. The built-in PII and secret detector redacts
|
|
174
|
+
recognized patterns before transmission, but it cannot recognize every secret.
|
|
175
|
+
Use a provider and credentials appropriate for this external text transfer.
|
|
176
|
+
Reranker failure keeps lexical knowledge matches or the usual memory block.
|
|
177
|
+
The reranker adds a provider call and may add cost; missing provider usage is
|
|
178
|
+
reported as unknown, not zero. Keep reranking off until an agent's own live
|
|
179
|
+
evaluation shows better answers within its latency and cost budget.
|
|
180
|
+
|
|
118
181
|
Facts carry **provenance metadata**: every fact record stores `origin`
|
|
119
182
|
(who wrote it — `"engine"`, `"operator"`, `"legacy"` or `"distilled"`),
|
|
120
183
|
`created_at` / `updated_at` timestamps, and an optional `expires_at` (ISO8601) —
|
|
@@ -0,0 +1,157 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: Cross-harness bench
|
|
3
|
+
parent: Operate
|
|
4
|
+
nav_order: 7
|
|
5
|
+
permalink: /cross-harness-bench/
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
# Cross-harness bench — the same store, six harnesses
|
|
9
|
+
|
|
10
|
+
Six agent harnesses answer the same twelve customer conversations against the same
|
|
11
|
+
MCP store, on the same model at the same reasoning setting, graded on the store's own
|
|
12
|
+
state. It answers a question the engine benchmark deliberately does not touch: not
|
|
13
|
+
"what does the engine cost per turn" but "does a shop staffed by this agent behave".
|
|
14
|
+
|
|
15
|
+
This is a **different suite with a different contract** from
|
|
16
|
+
[Benchmark](BENCHMARK.md), which measures engine overhead with no provider, no key and
|
|
17
|
+
no competitor. This one needs a provider key by design — a commerce conversation
|
|
18
|
+
cannot be stubbed — and it names other harnesses. Read the publication rule at the
|
|
19
|
+
bottom before quoting anything from it.
|
|
20
|
+
|
|
21
|
+
## Running it
|
|
22
|
+
|
|
23
|
+
```bash
|
|
24
|
+
cd evals/bench
|
|
25
|
+
cp .env.example .env # OPENROUTER_API_KEY, BENCH_MODEL
|
|
26
|
+
docker compose build # six images, every version pinned
|
|
27
|
+
bash cut3.sh # ~4-5h, ~$6, resumable
|
|
28
|
+
```
|
|
29
|
+
|
|
30
|
+
`RUNBOOK.md` in that directory is the operator's copy: the traps already paid for, the
|
|
31
|
+
resumability contract, and what to do after the run. Every entrant is a pinned Docker
|
|
32
|
+
image driven through the same one-JSON-in, one-JSON-out adapter; the store is reset
|
|
33
|
+
between every cell; three rounds is the floor, because a single run is not a
|
|
34
|
+
measurement.
|
|
35
|
+
|
|
36
|
+
## Two scorecards, and why
|
|
37
|
+
|
|
38
|
+
**A is parity** — same tools, same model, same prompt, none of anyone's own product
|
|
39
|
+
layer, ours included. **B is out of the box** — each harness as it ships. The distance
|
|
40
|
+
between a harness's two rows is what its product adds over the bare harness.
|
|
41
|
+
|
|
42
|
+
That distance is the number this bench was built to produce. In cut 3 it was zero for
|
|
43
|
+
us: scorecard B did not show our commerce layer paying for itself, and it was published
|
|
44
|
+
as a null result. In cut 4, with the one irreversible write held for the customer's
|
|
45
|
+
word, it is one cell — and that cell is described below, because one cell is a
|
|
46
|
+
mechanism to read, not a number to quote.
|
|
47
|
+
|
|
48
|
+
## What cut 3 found, for us
|
|
49
|
+
|
|
50
|
+
Twelve tasks, three rounds, `deepseek/deepseek-v4-flash` at reasoning `medium`, run
|
|
51
|
+
2026-09-09.
|
|
52
|
+
|
|
53
|
+
| | Scorecard A | Scorecard B |
|
|
54
|
+
| --- | --- | --- |
|
|
55
|
+
| Passed | 34/36 (94%) | 34/36 (94%) |
|
|
56
|
+
| Tokens per success | 2170 | 2263 |
|
|
57
|
+
| Wall clock p50 | 10.0s | 11.8s |
|
|
58
|
+
|
|
59
|
+
Two findings hold regardless of where the other rows land:
|
|
60
|
+
|
|
61
|
+
**Cost is where the spread is.** Accuracy across the field sits inside eleven points;
|
|
62
|
+
tokens per success spans an order of magnitude. Ours is the cheapest row on both
|
|
63
|
+
cards, at comparable accuracy. When five harnesses answer equally well, what separates
|
|
64
|
+
them is what they spend answering.
|
|
65
|
+
|
|
66
|
+
**Take the rules out of the prompt and the field separates.** The shared `AGENTS.md`
|
|
67
|
+
carries the rules the three hardest tasks test, which means a guarded cut partly
|
|
68
|
+
measures the prompt. Re-run with those rules removed, our rows hold at 9/9 on both
|
|
69
|
+
scorecards while the field spreads from 44% to 100%. That is the engine holding a rule
|
|
70
|
+
the prompt no longer states — the closest thing to a claim in this bench, and it is
|
|
71
|
+
about the engine, not about scorecard B.
|
|
72
|
+
|
|
73
|
+
## What cut 4 found, for us
|
|
74
|
+
|
|
75
|
+
Same model, same setting, run 2026-09-11, with our scorecard-B entry declaring
|
|
76
|
+
`customer_confirm "create_order"` — the close is held until the customer's next
|
|
77
|
+
message confirms it — and task 07 given the third turn that makes that answerable.
|
|
78
|
+
|
|
79
|
+
| | Scorecard A | Scorecard B |
|
|
80
|
+
| --- | --- | --- |
|
|
81
|
+
| Passed | 35/36 (97%) | 36/36 (100%) |
|
|
82
|
+
| Tokens per success | 2349 | 2890 |
|
|
83
|
+
| Wall clock p50 | 11.6s | 8.7s |
|
|
84
|
+
|
|
85
|
+
The one A cell we lose is task 09: *"adiciona logo por favor"* over two units already
|
|
86
|
+
in the cart, and the parity entry closed the order. The B entry, same engine, wrote no
|
|
87
|
+
order in any round of that task and closed task 07 on the confirming turn every time.
|
|
88
|
+
That pair of rows is the distance between A and B made of exactly one thing, and it is
|
|
89
|
+
the thing [Agents](AGENTS.md#do-not-offer-the-irreversible-action) describes: an
|
|
90
|
+
action the reply may offer freely because the engine will not run it in the same turn.
|
|
91
|
+
Three rounds is the floor, not a proof; the mechanism is in `docs/TOOLS.md`, the cells
|
|
92
|
+
in `evals/bench/cuts/2026-09-11/`.
|
|
93
|
+
|
|
94
|
+
Claude Code did not run this cut — the provider's Anthropic-shaped endpoint returned
|
|
95
|
+
nothing visible to it, with the same image that scored 94% two days earlier. Its cells
|
|
96
|
+
are kept outside the scorecard; a row that cannot run is not a row that ran badly.
|
|
97
|
+
|
|
98
|
+
## Reasoning is a deployment variable, not an axis
|
|
99
|
+
|
|
100
|
+
Every entrant is pinned to `medium` because that is what production runs. Crossing
|
|
101
|
+
reasoning against the table is eight combinations and roughly 1500 cells, so it gets
|
|
102
|
+
its own arm instead: ours against itself, both scorecards, at `medium` and at `off`.
|
|
103
|
+
|
|
104
|
+
Better on one card, worse on the other, with the same three tasks failing at both
|
|
105
|
+
settings. A focused probe — the three tasks we lose, ten rounds each — came back 24/30
|
|
106
|
+
at medium and 27/30 at off, the three tasks contradicting each other. **Reasoning does
|
|
107
|
+
not decide these cases.** Choose it on cost and latency.
|
|
108
|
+
|
|
109
|
+
## Reading a cut
|
|
110
|
+
|
|
111
|
+
`scorecard.rb` says who scored what. `viewer.rb` says what was actually said, which is
|
|
112
|
+
the question every finding here has turned on:
|
|
113
|
+
|
|
114
|
+
```bash
|
|
115
|
+
evals/bench/viewer.rb --runs cuts/2026-09-09
|
|
116
|
+
open evals/bench/bench.html
|
|
117
|
+
```
|
|
118
|
+
|
|
119
|
+
One self-contained HTML file: a harness-by-task matrix, and behind each square the
|
|
120
|
+
verdict, the checks that failed with their detail, and the whole conversation — the
|
|
121
|
+
customer's line, the reply verbatim, the tool calls, the timing. Every cell is also a
|
|
122
|
+
JSON file under `evals/bench/cuts/<date>/`, so any table here can be recomputed or
|
|
123
|
+
contested.
|
|
124
|
+
|
|
125
|
+
## Publication rule
|
|
126
|
+
|
|
127
|
+
**Everything here is published with its evidence. The comparison table is not a
|
|
128
|
+
ranking we will quote until someone outside this repo has reviewed the configs.**
|
|
129
|
+
|
|
130
|
+
Every entrant's configuration in this repo was written by us, and cut 3 found two
|
|
131
|
+
failures in our own wiring before anyone else did:
|
|
132
|
+
|
|
133
|
+
- OpenClaw's config carried a thinking level under `reasoningDefault`, which is a
|
|
134
|
+
visibility setting taking only `on|off|stream`. OpenClaw rejected the whole config
|
|
135
|
+
and ran no turn; the adapter read an empty answer off the error object and scored it
|
|
136
|
+
as a harness that answered nothing. Seventy-two cells published a broken row as 33%.
|
|
137
|
+
Fixed, the same harness scores 97% on both cards.
|
|
138
|
+
- Fifteen Hermes cells resumed from an earlier session had run against a store image
|
|
139
|
+
that was never rebuilt after its code changed.
|
|
140
|
+
|
|
141
|
+
Both were caught by someone reading transcripts, not by the bench. So the tables ship
|
|
142
|
+
with every cell behind them — anyone can reproduce a row or show it is wrong — but
|
|
143
|
+
until each entrant's configuration has been reviewed by someone who does not maintain
|
|
144
|
+
this repo, **we do not cite another harness's row as a ranking** in comparative or
|
|
145
|
+
marketing material. Our own rows carry no such problem: we can read our own config.
|
|
146
|
+
|
|
147
|
+
Publishing the numbers and declining to market them is the point, not a hedge. A
|
|
148
|
+
ranking whose losing rows we wired ourselves is worth less than a bench anyone can
|
|
149
|
+
clone, rewire and re-run — and the two failures above are the argument for that, not
|
|
150
|
+
against it. Fix an entrant's config, re-run one harness, send back the cut.
|
|
151
|
+
|
|
152
|
+
## See also
|
|
153
|
+
|
|
154
|
+
- [Benchmark](BENCHMARK.md) — engine overhead: neutral, keyless, reproducible.
|
|
155
|
+
- [Evals](EVALS.md) — the grading corpus and the runner this bench drives.
|
|
156
|
+
- [Agents](AGENTS.md#do-not-offer-the-irreversible-action) — what the bench taught us
|
|
157
|
+
about writing a store agent's prompt.
|
data/docs/DEPLOY.md
CHANGED
|
@@ -117,10 +117,11 @@ this section is the single source of truth for what changing it means.
|
|
|
117
117
|
| `INSIKA_TICK_STALE_AFTER` | `900` | seconds a `:queued`/`:running` task must sit untouched before the tick sweeps it. Must exceed the largest `turn_timeout` of the deployment |
|
|
118
118
|
| `INSIKA_GATEWAY_TOKEN` | falls back to `ADMIN_TOKEN` | Bearer for `/v1/responses` and `/v1/agents` (the API contract) |
|
|
119
119
|
| `ADMIN_TOKEN` | `local-demo` | login token for `/studio` (**change in production**) |
|
|
120
|
+
| `INSIKA_ENV` | `RACK_ENV`, else `local` | environment name on the Studio banner; `production`/`prod` hides the banner |
|
|
120
121
|
| `DEEPSEEK_API_KEY` | — | provider key. **Without it the engine still boots** (`/up` green), but turns fail until it is configured (env or Studio → LLM providers) — cloud resilience |
|
|
121
122
|
| `DEEPSEEK_MODEL` | `deepseek-v4-flash` | model |
|
|
122
123
|
| `CONSUMER_INTERNAL_URL` | — | base URL for data-tools calling back a consumer's internal API (see below) |
|
|
123
|
-
| `INSIKA_EGRESS_HOSTS` | — | outbound host allowlist (SSRF guard) |
|
|
124
|
+
| `INSIKA_EGRESS_HOSTS` | — | outbound host allowlist for data tools (SSRF guard). An MCP server is pinned by it only when its host resolves to a private address |
|
|
124
125
|
| `INSIKA_EGRESS_ALLOW_HTTP` / `_ALLOW_PRIVATE` | off | for `http`/loopback callbacks only (**never in cloud**). Also opens inbound **media** fetches (audio/image URLs) to `http`/private targets — a local run serving media over `http://` needs it |
|
|
125
126
|
| `INSIKA_RELAY_TOKEN` | — | **mounts the relay channel** at `POST /channels/relay/events`, and is the Bearer it requires. Empty = the route does not exist (`404`). See [Channels](CHANNELS.md) |
|
|
126
127
|
| `INSIKA_RELAY_DELIVER_URL` | — | your callback; the engine POSTs each reply there. Goes through the egress guard |
|
|
@@ -128,16 +129,17 @@ this section is the single source of truth for what changing it means.
|
|
|
128
129
|
| `INSIKA_RELAY_DELIVERY` | `at_end` | how the relay flushes the outbox: `at_end` (one POST) or `progressive` (one POST per balloon — [delivery policy](CHANNELS.md#delivery-policy)) |
|
|
129
130
|
| `INSIKA_WIDGET_ORIGINS` | — | exact-match origins allowed to embed the [web widget](CHANNELS.md#the-web-widget), comma-separated. No wildcards. **Half the switch**: with `INSIKA_WIDGET_AGENTS` unset, nothing is mounted (`404`) |
|
|
130
131
|
| `INSIKA_WIDGET_AGENTS` | — | agent ids a widget visitor may address, comma-separated. The other half of the switch. **A chat rate limit is also required** or the widget answers `503` |
|
|
132
|
+
| `INSIKA_DATABASE_URL` | — | **Postgres store** (`postgres://…`, Postgres 13+). Set, it replaces `INSIKA_DB` and Litestream is skipped. Needs the `pg` gem. See [Postgres store](#postgres-store-optional) |
|
|
133
|
+
| `INSIKA_DATABASE_POOL` | `10` | Postgres connections per process |
|
|
131
134
|
| `LITESTREAM_REPLICA_URL` | — | **enables Litestream** (backup/DR). Empty = disabled (default). See below |
|
|
132
135
|
| `LITESTREAM_ENDPOINT` | — | S3-compatible endpoint (R2/MinIO). Empty = AWS S3 |
|
|
133
136
|
| `LITESTREAM_REGION` | — | bucket region (AWS: `us-east-1`; R2: `auto`) |
|
|
134
137
|
| `LITESTREAM_ACCESS_KEY_ID` / `LITESTREAM_SECRET_ACCESS_KEY` | — | bucket credentials (read natively by Litestream) |
|
|
135
138
|
|
|
136
|
-
> **Renamed from `HARNESS_*` → `INSIKA_*`.** Every engine variable
|
|
137
|
-
>
|
|
138
|
-
>
|
|
139
|
-
>
|
|
140
|
-
> will be dropped in a future release.
|
|
139
|
+
> **Renamed from `HARNESS_*` → `INSIKA_*`.** Every engine variable uses the `INSIKA_`
|
|
140
|
+
> prefix. The old `HARNESS_*` names were removed in 1.0 and are no longer read: a
|
|
141
|
+
> leftover one is reported as an error naming its replacement (at boot and in
|
|
142
|
+
> `insika doctor`). Rename it before upgrading.
|
|
141
143
|
|
|
142
144
|
> **The database filename changed too** — the image now defaults to
|
|
143
145
|
> `INSIKA_DB=/data/insika.db` (it was `/data/harness.db`). **Existing volumes are
|
|
@@ -347,6 +349,21 @@ LITESTREAM_SECRET_ACCESS_KEY=...
|
|
|
347
349
|
Credentials are read natively by Litestream (they are not in `deploy/litestream.yml`,
|
|
348
350
|
which only references URL/endpoint/region via `${VAR}`).
|
|
349
351
|
|
|
352
|
+
### One writer per replica path
|
|
353
|
+
|
|
354
|
+
- **Never point two deployments at the same `LITESTREAM_REPLICA_URL`** (two services,
|
|
355
|
+
or staging and production). Both replicate into the same lineage and a later restore
|
|
356
|
+
can bring back the wrong database.
|
|
357
|
+
- **A database already on the volume skips the restore** and is replicated as it is.
|
|
358
|
+
Give a new deployment an empty volume, and do not set the replica URL on a box whose
|
|
359
|
+
volume holds the wrong file — it will be copied over the backup.
|
|
360
|
+
- **Moving a database** to another deployment or path: stop replication on the old box
|
|
361
|
+
first (unset the variable and redeploy), and give the new one a fresh path. If the old
|
|
362
|
+
box's local state is stale (`LTX file is missing`), run `litestream reset <db>`.
|
|
363
|
+
- **Point-in-time recovery** into a separate file, to inspect before swapping it in:
|
|
364
|
+
`litestream ltx -level all <url>` lists the history, then
|
|
365
|
+
`litestream restore -txid <id> -o /data/recovered.db <url>`.
|
|
366
|
+
|
|
350
367
|
### Restore drill (the real "done" — an untested backup does not count)
|
|
351
368
|
|
|
352
369
|
The pilot→production gap only closes once a restore has been **exercised**. Two ways:
|
|
@@ -364,8 +381,8 @@ scripts/litestream-restore-drill.sh # needs docker, sqlite3, curl
|
|
|
364
381
|
|
|
365
382
|
```bash
|
|
366
383
|
# a. with the service live and replicating, generate some config/conversation and
|
|
367
|
-
# confirm the replica has
|
|
368
|
-
litestream
|
|
384
|
+
# confirm the replica has files (Litestream 0.5 lists them as LTX files):
|
|
385
|
+
litestream ltx -config deploy/litestream.yml "$INSIKA_DB"
|
|
369
386
|
|
|
370
387
|
# b. boot a NEW box (empty volume) with the same LITESTREAM_* vars → the entrypoint
|
|
371
388
|
# restores on boot. Verify manually in /studio that conversations and config came
|
|
@@ -373,12 +390,41 @@ litestream snapshots -config deploy/litestream.yml "$INSIKA_DB"
|
|
|
373
390
|
litestream restore -config deploy/litestream.yml -o /tmp/restored.db "$INSIKA_DB"
|
|
374
391
|
```
|
|
375
392
|
|
|
393
|
+
## Postgres store (optional)
|
|
394
|
+
|
|
395
|
+
SQLite is the default and the right choice for one box, including several worker
|
|
396
|
+
processes (`INSIKA_WORKERS` > 1): their writes share commits (group commit), so
|
|
397
|
+
they only queue behind the file's single write lock under sustained bursts of
|
|
398
|
+
writes. Postgres removes that lock — writers only meet on the keys they share — but
|
|
399
|
+
every store call becomes a network round trip, and a turn makes many store calls,
|
|
400
|
+
so on one box it is usually slower per turn, not faster.
|
|
401
|
+
|
|
402
|
+
Choose Postgres when the store must be shared: several nodes or replicas running
|
|
403
|
+
the engine, or other systems that query the data directly. Measure your own load
|
|
404
|
+
before switching for speed alone.
|
|
405
|
+
|
|
406
|
+
1. Add `gem "pg"` to your Gemfile (the reference image already has it and libpq).
|
|
407
|
+
2. Set `INSIKA_DATABASE_URL=postgres://user:pass@host:5432/db` (Postgres 13+).
|
|
408
|
+
It replaces `INSIKA_DB`; Litestream is skipped. `INSIKA_DATABASE_POOL` sets the
|
|
409
|
+
connections per process (default 10). A transaction keeps its connection while
|
|
410
|
+
its turn waits for the reactor, so under many concurrent turns a small pool is
|
|
411
|
+
where calls queue; raise it, keeping workers × pool under the server's
|
|
412
|
+
`max_connections` (100 by default).
|
|
413
|
+
3. The table (`insika_kv`) is created on first boot.
|
|
414
|
+
4. Moving an existing SQLite store into an **empty** database (the copy adds and
|
|
415
|
+
overwrites keys, it does not remove extra ones): stop the app, then
|
|
416
|
+
`INSIKA_DB=/data/insika.db INSIKA_DATABASE_URL=… bundle exec ruby scripts/store_copy.rb`,
|
|
417
|
+
then boot with `INSIKA_DATABASE_URL` set. Unsetting it goes back to the SQLite
|
|
418
|
+
file as it was at the copy.
|
|
419
|
+
|
|
420
|
+
Backups are Postgres's job (your provider's snapshots or point-in-time recovery).
|
|
421
|
+
|
|
376
422
|
## Kubernetes (evolution)
|
|
377
423
|
|
|
378
424
|
SQLite does not share one file across nodes. Paths forward: a StatefulSet + a PVC
|
|
379
|
-
per pod + **sticky-by-agent** routing (shard by tenant), or **LiteFS**, or
|
|
380
|
-
|
|
381
|
-
orthogonal to topology.
|
|
425
|
+
per pod + **sticky-by-agent** routing (shard by tenant), or **LiteFS**, or the
|
|
426
|
+
[Postgres store](#postgres-store-optional), which every pod can share.
|
|
427
|
+
**Litestream** (above) for backup/DR of SQLite — orthogonal to topology.
|
|
382
428
|
|
|
383
429
|
For the *session-routing* half specifically — as opposed to the storage
|
|
384
430
|
topology above — see [`insika-router`](ROUTER.md): N pods behind a
|
data/docs/EMBEDDING.md
CHANGED
|
@@ -98,6 +98,16 @@ It also holds for runtime reconfiguration. Editing a provider key in the Studio
|
|
|
98
98
|
(or dispatching `:upsert_llm_provider`) applies to **that graph only** — the
|
|
99
99
|
change is real, takes effect without a restart, and stops at the graph boundary.
|
|
100
100
|
|
|
101
|
+
Each turn's chat gets a copied RubyLLM configuration with Insika's diagnostic
|
|
102
|
+
instrumenter. It composes with a host instrumenter already on that configuration:
|
|
103
|
+
host notifications still run, and the graph and global configurations keep the
|
|
104
|
+
host instrumenter. Insika attributes only those chat operations to the task;
|
|
105
|
+
auxiliary model work on the graph's original context has no task correlation.
|
|
106
|
+
Each graph stores its task diagnostics in its own backend, so separate backends
|
|
107
|
+
keep those records isolated. Insika's diagnostic fields are allowlisted, but a
|
|
108
|
+
host instrumenter receives RubyLLM's original payload and must apply its own
|
|
109
|
+
data policy. See [Observability](OBSERVABILITY.md#native-model-diagnostics).
|
|
110
|
+
|
|
101
111
|
### 3. The process still owns signals and the reactor
|
|
102
112
|
|
|
103
113
|
Draining in-flight turns on SIGTERM ([the process model](DEPLOY.md)) is a process
|
data/docs/EVALS.md
CHANGED
|
@@ -109,7 +109,8 @@ check, and the report names the one that failed.
|
|
|
109
109
|
|
|
110
110
|
| Key | Checks |
|
|
111
111
|
|-----|--------|
|
|
112
|
-
| `tools_called: [name, name?]` | each required tool was called (`?` = optional, never fails) |
|
|
112
|
+
| `tools_called: [name, name?]` | each required tool was called on the LAST turn (`?` = optional, never fails) |
|
|
113
|
+
| `turns[i].tools_called: [names]` | the calls one specific turn must make — a two-turn case where turn one claimed an action and turn two repaired the store passes the key above and fails this one |
|
|
113
114
|
| `never_calls: [names]` | none of these was called — the negative every `tools_called` needs |
|
|
114
115
|
| `calls_one_of: [names]` | at least one of these was called |
|
|
115
116
|
| `first_tool: name` | the first call's name |
|
|
@@ -119,7 +120,8 @@ check, and the report names the one that failed.
|
|
|
119
120
|
| `blocked_gates: ["tool:gate"]` | each pair appears among the turn's blocked calls |
|
|
120
121
|
| `ui_components: [names]` | a presentation tool showed at least one card of each component (the `insika.ui` frames) |
|
|
121
122
|
| `no_ui: true` | the turn showed nothing — the negative of `ui_components` |
|
|
122
|
-
| `must_not: [detectors]` | the negative detectors (`pii_leak`, `tool_error`, …); a blocked call is not a tool error |
|
|
123
|
+
| `must_not: [detectors]` | the negative detectors (`pii_leak`, `tool_error`, `phantom_action`, …); a blocked call is not a tool error |
|
|
124
|
+
| `claims: { tool: pattern }` | what `phantom_action` reads — per mutation, the words a reply uses to say it did it (a regex, case-insensitive); the store's language, so it lives in the case |
|
|
123
125
|
|
|
124
126
|
**Every positive has a negative.** A case that only says `tools_called: [add_to_cart]`
|
|
125
127
|
passes an agent that also re-searched, or that echoed the SKU to the customer. Pin
|
|
@@ -128,6 +130,22 @@ in the same case. Snapshots avoid the model calls needed to reconstruct the setu
|
|
|
128
130
|
use the simulator when later customer messages need to branch on the replies.
|
|
129
131
|
|
|
130
132
|
`tools_called` confirms an attempted call, not a successful backend mutation.
|
|
133
|
+
|
|
134
|
+
`must_not: phantom_action` reads the other direction: a reply that SAYS it did
|
|
135
|
+
something ("adicionei", "removi", "fechei o pedido") on a turn where the matching
|
|
136
|
+
tool never ran, errored, or was blocked. It is cumulative over the conversation, so
|
|
137
|
+
"já adicionei" on turn two is true when turn one did it, and it only pins the words
|
|
138
|
+
against the calls — a cart that ended up right by accident is still a lie to the
|
|
139
|
+
customer. The words come from the case's `claims:` map, in the store's own
|
|
140
|
+
language, past tense on purpose: "quer adicionar?" is a question.
|
|
141
|
+
|
|
142
|
+
```yaml
|
|
143
|
+
expect:
|
|
144
|
+
must_not: [tool_error, phantom_action]
|
|
145
|
+
claims:
|
|
146
|
+
add_to_cart: '\b(?:adicion(?:ei|ad[oa]s?)|coloquei)\b'
|
|
147
|
+
create_order: '\b(?:fechei|finalizei|pedido (?:criado|fechado))\b'
|
|
148
|
+
```
|
|
131
149
|
Arguments are collected for inspection; there is no generic argument or cart-state
|
|
132
150
|
grader. `blocked_gates` needs completion statuses from the HTTP transport. The
|
|
133
151
|
in-process `GraphTransport` records tool names and UI events, but not completion
|
|
@@ -183,12 +201,13 @@ judge is told the rule instead of guessing it.
|
|
|
183
201
|
|
|
184
202
|
```yaml
|
|
185
203
|
expect:
|
|
186
|
-
policy: ask_once
|
|
204
|
+
policy: ask_once # the strict reading: one question, full stop
|
|
205
|
+
policy: { ask_once: { max: 2 } } # a store that greets its customers
|
|
187
206
|
```
|
|
188
207
|
|
|
189
208
|
| policy | the check | the judge is told |
|
|
190
209
|
|---|---|---|
|
|
191
|
-
| `ask_once` | no reply asks more than
|
|
210
|
+
| `ask_once` | no reply asks more than `max` questions (default 1) | at most `max` per reply, and a greeting formula is courtesy rather than a question |
|
|
192
211
|
| `investigate_first` | turn 1 asks something and calls no tool | ask on a vague request, don't search immediately |
|
|
193
212
|
| `act_fast` | turn 1 calls a tool | asking what a search would answer is a failure |
|
|
194
213
|
|
|
@@ -201,6 +220,16 @@ so a tracking link's query string is not read as the agent asking something. It
|
|
|
201
220
|
policy signal, not grammar — and crude was enough to catch an agent breaking a rule
|
|
202
221
|
written in its own prompt, twice, with no model in the loop.
|
|
203
222
|
|
|
223
|
+
**`max` is the store's, and it matters more than it looks.** The default of 1 comes from
|
|
224
|
+
a real store's rule, quoted down to the punctuation: *"Máximo 1 pergunta principal por
|
|
225
|
+
mensagem = 1 ponto de interrogação no fim"*. A store that greets its customers spends
|
|
226
|
+
one of those on courtesy — measured across six harnesses answering the same "oi", `max:
|
|
227
|
+
1` failed every reply containing "tudo bem?" and passed every reply without it, and
|
|
228
|
+
nothing else. That grades a greeting habit, not the rule the policy exists for, which is
|
|
229
|
+
the form: *"qual seu nome? qual o número do pedido? e o motivo?"*. Set `max` to what
|
|
230
|
+
your store actually tolerates; the judge is told the same number, so the two halves
|
|
231
|
+
never grade different rules.
|
|
232
|
+
|
|
204
233
|
### `reference` — compared against the system you want to replace
|
|
205
234
|
|
|
206
235
|
`min_score: 0.7` says a reply cleared a bar you invented. It says nothing about
|
data/docs/FACTS.md
CHANGED
|
@@ -70,12 +70,14 @@ The engine never writes store vocabulary.
|
|
|
70
70
|
|
|
71
71
|
The **Facts** page (operate group, next to Follow-ups) shows:
|
|
72
72
|
|
|
73
|
-
- **Pending** — oldest first, because evidence ages.
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
|
|
73
|
+
- **Pending** — oldest first, because evidence ages. Select a fact to review its
|
|
74
|
+
proposed value, model confidence and conversation evidence. Long excerpts scroll
|
|
75
|
+
within the evidence panel; **Open conversation** shows their source. Keys and
|
|
76
|
+
scope/session IDs live under **Technical details**. **Approve & save to memory**
|
|
77
|
+
saves the value; **Reject or dismiss** expands the alternatives, including an
|
|
78
|
+
optional rejection reason. Both prevent the same key and value from being proposed
|
|
79
|
+
again. Decisions refresh the queue and open the next pending fact in the selected
|
|
80
|
+
agent/scope.
|
|
79
81
|
- **Stale** — the CAS-lost re-present (below): the proposed value struck
|
|
80
82
|
through next to the operator's current value, resolved by dismissal.
|
|
81
83
|
- **Recent** — every resolved proposal, most recent first, with operator and
|
data/docs/HARVEST.md
CHANGED
|
@@ -35,8 +35,10 @@ A mine reads ONLY durable, finished data — never an in-flight turn, never a
|
|
|
35
35
|
live prompt prefix (the "fork" is structural: the mining writes nothing to the
|
|
36
36
|
sessions it read, so a customer turn's cache is untouched by construction). The
|
|
37
37
|
miner gets the transcript slices (masked), the session's evidence-ledger ids,
|
|
38
|
-
|
|
39
|
-
|
|
38
|
+
the agent's current skill names and the candidates still open (name and
|
|
39
|
+
description), so it does not re-propose an idea under another name.
|
|
40
|
+
Simulator and eval sessions (`sim-…`, `eval-…`) are never mined: they are
|
|
41
|
+
not customers. Then the filters, each drop counted and logged:
|
|
40
42
|
|
|
41
43
|
1. **The negative list** — the versioned rules file (seed, imported
|
|
42
44
|
per store by `insika harvest:negative import --agent ID --file F`) and the
|
|
@@ -111,6 +113,9 @@ back, re-promoted is three readable rows.
|
|
|
111
113
|
- `insika harvest:criterion check --file F` — strict-load the frozen conversion criterion (the hook before any promotion).
|
|
112
114
|
- The **Harvest page** in the Studio: the human's inbox (each candidate with
|
|
113
115
|
its evidence excerpt, the eval report, the conversion card, promote/reject),
|
|
114
|
-
the gated-but-blocked rows with the named ruler hole
|
|
116
|
+
the gated-but-blocked rows with the named ruler hole and a "Run checks
|
|
117
|
+
again" once the hole is fixed, the pending list (every row with the skill
|
|
118
|
+
text, the rationale and a link to its origin conversation), a notice when
|
|
119
|
+
the loop skips the agent (harvest off, or no grounding matcher), the
|
|
115
120
|
append-only promoted log with the rollback mirror, the negative list with
|
|
116
121
|
per-rule rejection counts, and the criterion block read-only.
|
data/docs/KNOWLEDGE.md
CHANGED
|
@@ -95,6 +95,13 @@ end
|
|
|
95
95
|
recalling (write-only, inspected by hand), or recall without learning
|
|
96
96
|
(curate every concept by hand in the Studio, `provenance: policy`).
|
|
97
97
|
|
|
98
|
+
The same settings are available in **Studio → Agents → select an agent →
|
|
99
|
+
Configuration → Memory & knowledge**: learn from conversations, use knowledge in
|
|
100
|
+
replies, concepts per turn, concept types, learning model and instructions.
|
|
101
|
+
Knowledge reranking has its own switch and settings, independent of memory
|
|
102
|
+
reranking, and defaults to off. Options authored in a pack but not exposed in the
|
|
103
|
+
form, such as `index`, are preserved when saving.
|
|
104
|
+
|
|
98
105
|
## The write path
|
|
99
106
|
|
|
100
107
|
After a turn completes, off the turn's critical path (the customer already has
|
data/docs/MEDIA.md
CHANGED
|
@@ -39,6 +39,8 @@ curl -X POST /v1/messages?stream=false -H "Authorization: Bearer $TOKEN" \
|
|
|
39
39
|
the usage flows like any ask. The first image URL is also
|
|
40
40
|
`{{ctx.image_url}}` for data tools — photo analysis outside the prompt, the
|
|
41
41
|
tool's own egress applying when it fetches.
|
|
42
|
+
DeepSeek images require RubyLLM's [image-attachment fix](https://github.com/crmne/ruby_llm/pull/972),
|
|
43
|
+
included in this deployment's pinned revision, and a vision-capable model.
|
|
42
44
|
- **Documents** (a prescription, a recipe, an invoice — most often a PDF)
|
|
43
45
|
attach the same way images do; the first document URL is
|
|
44
46
|
`{{ctx.document_url}}`. A model without document support fails the ask at
|
|
@@ -117,8 +119,10 @@ end
|
|
|
117
119
|
- **Seams, not magic.** The generator is injectable per kind (specs stub it);
|
|
118
120
|
the defaults are lazy: images via RubyLLM (paint), speech via a thin POST to
|
|
119
121
|
the OpenAI-compatible `/audio/speech` endpoint using the same provider
|
|
120
|
-
config the chat uses
|
|
121
|
-
|
|
122
|
+
config the chat uses. RubyLLM 2 has a speech API, but its inspected native
|
|
123
|
+
transport buffers non-audio/error responses without yielding chunks. Insika
|
|
124
|
+
retains its bounded HTTP transport until native limits cover those responses.
|
|
125
|
+
A generated part over 8 MB refuses loudly, never silently truncates.
|
|
122
126
|
- **Not here:** what the generated image *means* — a fitting room, a product
|
|
123
127
|
mockup — is a skill on top. The engine transports bytes and cost.
|
|
124
128
|
|