insika 0.8.0 → 1.0.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (276) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGELOG.md +312 -2
  3. data/README.md +10 -6
  4. data/bin/insika +4 -1
  5. data/bin/insika-router +0 -1
  6. data/docs/AGENTS.md +142 -18
  7. data/docs/API.md +73 -0
  8. data/docs/ARCHITECTURE.md +56 -50
  9. data/docs/ARTIFACTS.md +33 -3
  10. data/docs/BENCHMARK.md +50 -7
  11. data/docs/CHANNELS.md +19 -2
  12. data/docs/CONTEXT.md +96 -27
  13. data/docs/CROSS-HARNESS-BENCH.md +157 -0
  14. data/docs/DEPLOY.md +70 -13
  15. data/docs/EMBEDDING.md +10 -0
  16. data/docs/EVALS.md +129 -10
  17. data/docs/FACTS.md +12 -6
  18. data/docs/HARVEST.md +8 -3
  19. data/docs/KNOWLEDGE.md +14 -0
  20. data/docs/MEDIA.md +6 -2
  21. data/docs/OBSERVABILITY.md +164 -28
  22. data/docs/PERFORMANCE.md +110 -0
  23. data/docs/PLUGINS.md +4 -4
  24. data/docs/POLICY.md +26 -14
  25. data/docs/RELEASING.md +9 -0
  26. data/docs/ROUTER.md +7 -0
  27. data/docs/RUBYLLM_2_MIGRATION.md +307 -0
  28. data/docs/SECURITY.md +34 -1
  29. data/docs/SHARED-CONVERSATIONS.md +40 -0
  30. data/docs/SKILLS.md +5 -3
  31. data/docs/STUDIO.md +122 -0
  32. data/docs/TOOLS.md +238 -42
  33. data/docs/operate.md +3 -0
  34. data/docs/prompts/ADD-TOOL.md +12 -2
  35. data/docs/prompts/DIAGNOSE-TURN.md +3 -0
  36. data/docs/prompts/GO-LIVE.md +3 -1
  37. data/lib/insika/agent_defaults.rb +140 -0
  38. data/lib/insika/agent_file_store.rb +12 -0
  39. data/lib/insika/agent_profile.rb +99 -17
  40. data/lib/insika/artifact_signing.rb +11 -0
  41. data/lib/insika/channels/web/widget.js +33 -0
  42. data/lib/insika/channels/web.rb +5 -2
  43. data/lib/insika/chat_builder.rb +124 -28
  44. data/lib/insika/checkpoint.rb +6 -1
  45. data/lib/insika/checkpoint_store.rb +59 -7
  46. data/lib/insika/commands/agent_payload.rb +3 -3
  47. data/lib/insika/commands/delete_kit.rb +26 -0
  48. data/lib/insika/commands/gate_harvest.rb +3 -0
  49. data/lib/insika/commands/review_guardrail_flag.rb +38 -0
  50. data/lib/insika/commands/run_distillation.rb +5 -8
  51. data/lib/insika/commands/run_harvest.rb +17 -1
  52. data/lib/insika/commands/seed_session.rb +128 -0
  53. data/lib/insika/commands/send_message.rb +9 -3
  54. data/lib/insika/commands/update_settings.rb +23 -1
  55. data/lib/insika/commands/write_kit.rb +27 -0
  56. data/lib/insika/compaction.rb +33 -7
  57. data/lib/insika/config_store.rb +87 -7
  58. data/lib/insika/context/builder.rb +29 -9
  59. data/lib/insika/context/priority.rb +5 -0
  60. data/lib/insika/context/provider.rb +7 -2
  61. data/lib/insika/context/providers/briefing.rb +11 -8
  62. data/lib/insika/context/providers/fence_notice.rb +27 -0
  63. data/lib/insika/context/providers/knowledge.rb +32 -6
  64. data/lib/insika/context/providers/memory.rb +61 -8
  65. data/lib/insika/context/providers/pending_confirmation.rb +52 -0
  66. data/lib/insika/context/providers/prompt.rb +17 -4
  67. data/lib/insika/context/providers/session.rb +46 -6
  68. data/lib/insika/context/reranker.rb +75 -0
  69. data/lib/insika/distill.rb +22 -13
  70. data/lib/insika/doctor.rb +142 -4
  71. data/lib/insika/dsl/runtime.rb +37 -4
  72. data/lib/insika/dsl/server_boot.rb +4 -12
  73. data/lib/insika/dsl.rb +19 -5
  74. data/lib/insika/edge_limiter.rb +27 -16
  75. data/lib/insika/egress_guard.rb +11 -3
  76. data/lib/insika/env_schema.rb +33 -73
  77. data/lib/insika/errors.rb +1 -0
  78. data/lib/insika/evals/assertions.rb +259 -18
  79. data/lib/insika/evals/golden.rb +230 -4
  80. data/lib/insika/evals/judge.rb +19 -10
  81. data/lib/insika/evals/moderator_compare.rb +47 -0
  82. data/lib/insika/evals/report.rb +61 -9
  83. data/lib/insika/evals/runner.rb +64 -4
  84. data/lib/insika/evals/simulator.rb +21 -3
  85. data/lib/insika/evals/transport.rb +317 -15
  86. data/lib/insika/event_stream.rb +63 -7
  87. data/lib/insika/evidence.rb +105 -17
  88. data/lib/insika/executor.rb +698 -164
  89. data/lib/insika/fence.rb +96 -0
  90. data/lib/insika/golden_store.rb +3 -0
  91. data/lib/insika/harvest.rb +26 -14
  92. data/lib/insika/harvest_engine.rb +1 -0
  93. data/lib/insika/harvest_store.rb +12 -0
  94. data/lib/insika/http_client.rb +87 -4
  95. data/lib/insika/kits.rb +54 -0
  96. data/lib/insika/knowledge.rb +34 -20
  97. data/lib/insika/llm_configurator.rb +46 -4
  98. data/lib/insika/llm_provider_store.rb +1 -3
  99. data/lib/insika/llm_trace_store.rb +54 -0
  100. data/lib/insika/mcp_client.rb +27 -63
  101. data/lib/insika/mcp_live_tool.rb +55 -9
  102. data/lib/insika/mcp_store.rb +37 -3
  103. data/lib/insika/mcp_tool_registry.rb +76 -26
  104. data/lib/insika/media.rb +38 -25
  105. data/lib/insika/memory_store.rb +16 -3
  106. data/lib/insika/memory_trim.rb +60 -0
  107. data/lib/insika/middleware.rb +12 -3
  108. data/lib/insika/model_metrics_store.rb +285 -0
  109. data/lib/insika/model_registry.rb +110 -0
  110. data/lib/insika/model_resolver.rb +1 -1
  111. data/lib/insika/model_selection.rb +14 -21
  112. data/lib/insika/model_visible.rb +17 -6
  113. data/lib/insika/onboarding.rb +1 -0
  114. data/lib/insika/outcome_store.rb +1 -1
  115. data/lib/insika/overlay_tool_registry.rb +30 -2
  116. data/lib/insika/pack_importer.rb +54 -9
  117. data/lib/insika/packaging.rb +14 -5
  118. data/lib/insika/pending_action_store.rb +45 -18
  119. data/lib/insika/plugin/loader.rb +4 -2
  120. data/lib/insika/prefix_fingerprint.rb +32 -27
  121. data/lib/insika/profile_source.rb +5 -0
  122. data/lib/insika/provider_error_classifier.rb +18 -3
  123. data/lib/insika/queue_policy.rb +7 -2
  124. data/lib/insika/refinement/proposer.rb +42 -17
  125. data/lib/insika/reliability.rb +12 -3
  126. data/lib/insika/router/app.rb +14 -5
  127. data/lib/insika/router/session_key.rb +35 -5
  128. data/lib/insika/safety/config.rb +20 -3
  129. data/lib/insika/safety/corpus.rb +10 -3
  130. data/lib/insika/safety/factory.rb +39 -1
  131. data/lib/insika/safety/input_guardrail.rb +21 -1
  132. data/lib/insika/safety/judge_moderator.rb +70 -0
  133. data/lib/insika/safety/output_validator.rb +29 -5
  134. data/lib/insika/safety/prompt_echo.rb +89 -0
  135. data/lib/insika/schema_guard.rb +53 -76
  136. data/lib/insika/server/app.rb +58 -7
  137. data/lib/insika/server/rack_app.rb +2 -0
  138. data/lib/insika/server/responses.rb +50 -5
  139. data/lib/insika/session_actor.rb +58 -28
  140. data/lib/insika/session_store.rb +163 -11
  141. data/lib/insika/settings_store.rb +57 -2
  142. data/lib/insika/shared_conversations.rb +305 -0
  143. data/lib/insika/skill_catalog.rb +28 -2
  144. data/lib/insika/skill_store.rb +6 -0
  145. data/lib/insika/soak/runner.rb +1 -1
  146. data/lib/insika/spoken_transcript.rb +31 -0
  147. data/lib/insika/steer_injector.rb +7 -48
  148. data/lib/insika/store.rb +29 -1
  149. data/lib/insika/stores/json_value.rb +37 -0
  150. data/lib/insika/stores/memory.rb +22 -35
  151. data/lib/insika/stores/postgres.rb +295 -0
  152. data/lib/insika/stores/sqlite.rb +257 -43
  153. data/lib/insika/stores/turn_counter.rb +25 -0
  154. data/lib/insika/studio/app.rb +1459 -163
  155. data/lib/insika/studio/assets/dist/application.css +1 -1
  156. data/lib/insika/studio/assets/dist/application.js +25 -25
  157. data/lib/insika/studio/assets/dist/fonts/ibm-plex-mono-latin-400-normal.woff2 +0 -0
  158. data/lib/insika/studio/assets/dist/fonts/ibm-plex-mono-latin-500-normal.woff2 +0 -0
  159. data/lib/insika/studio/assets/dist/fonts/ibm-plex-mono-latin-600-normal.woff2 +0 -0
  160. data/lib/insika/studio/assets/dist/fonts/ibm-plex-sans-latin-wght-normal.woff2 +0 -0
  161. data/lib/insika/studio/forms.rb +189 -12
  162. data/lib/insika/studio/nav_icons.rb +3 -0
  163. data/lib/insika/studio/views/_agent_config_retrieval.erb +73 -0
  164. data/lib/insika/studio/views/_agent_picker.erb +15 -0
  165. data/lib/insika/studio/views/_agent_tab_config.erb +83 -243
  166. data/lib/insika/studio/views/_agent_tab_history.erb +56 -1
  167. data/lib/insika/studio/views/_agent_tab_performance.erb +34 -0
  168. data/lib/insika/studio/views/_agent_tab_prompts.erb +7 -9
  169. data/lib/insika/studio/views/_agent_tab_skills.erb +16 -7
  170. data/lib/insika/studio/views/_agent_tab_tools.erb +90 -0
  171. data/lib/insika/studio/views/_agents_master.erb +8 -12
  172. data/lib/insika/studio/views/_cfg_alerts.erb +4 -0
  173. data/lib/insika/studio/views/_cfg_budget.erb +20 -0
  174. data/lib/insika/studio/views/_cfg_distill.erb +26 -0
  175. data/lib/insika/studio/views/_cfg_followup.erb +30 -0
  176. data/lib/insika/studio/views/_cfg_grounding.erb +21 -0
  177. data/lib/insika/studio/views/_cfg_guardrails.erb +47 -0
  178. data/lib/insika/studio/views/_cfg_harvest.erb +35 -0
  179. data/lib/insika/studio/views/_cfg_reliability.erb +32 -0
  180. data/lib/insika/studio/views/_chats_master.erb +66 -0
  181. data/lib/insika/studio/views/_customers_master.erb +37 -0
  182. data/lib/insika/studio/views/_empty.erb +15 -0
  183. data/lib/insika/studio/views/_facts_master.erb +42 -0
  184. data/lib/insika/studio/views/_harvest_candidate_detail.erb +14 -0
  185. data/lib/insika/studio/views/_identity.erb +16 -0
  186. data/lib/insika/studio/views/_kpi.erb +17 -0
  187. data/lib/insika/studio/views/_message.erb +40 -2
  188. data/lib/insika/studio/views/_model_chart.erb +82 -0
  189. data/lib/insika/studio/views/_refinement_master.erb +38 -0
  190. data/lib/insika/studio/views/{tool_edit.erb → _tool_edit.erb} +38 -11
  191. data/lib/insika/studio/views/_tool_users.erb +17 -0
  192. data/lib/insika/studio/views/_why.erb +53 -0
  193. data/lib/insika/studio/views/agent_detail.erb +17 -10
  194. data/lib/insika/studio/views/agents.erb +28 -16
  195. data/lib/insika/studio/views/approvals.erb +16 -11
  196. data/lib/insika/studio/views/artifact.erb +20 -7
  197. data/lib/insika/studio/views/artifacts.erb +19 -14
  198. data/lib/insika/studio/views/chats.erb +22 -29
  199. data/lib/insika/studio/views/customer.erb +88 -43
  200. data/lib/insika/studio/views/customers.erb +16 -26
  201. data/lib/insika/studio/views/evals.erb +49 -31
  202. data/lib/insika/studio/views/fact_detail.erb +114 -0
  203. data/lib/insika/studio/views/facts.erb +15 -116
  204. data/lib/insika/studio/views/followups.erb +63 -52
  205. data/lib/insika/studio/views/funnel.erb +104 -43
  206. data/lib/insika/studio/views/harvest.erb +85 -59
  207. data/lib/insika/studio/views/home.erb +138 -23
  208. data/lib/insika/studio/views/inbox.erb +36 -0
  209. data/lib/insika/studio/views/kits.erb +176 -0
  210. data/lib/insika/studio/views/knowledge.erb +92 -80
  211. data/lib/insika/studio/views/layout.erb +42 -11
  212. data/lib/insika/studio/views/live.erb +45 -0
  213. data/lib/insika/studio/views/mcp.erb +43 -13
  214. data/lib/insika/studio/views/models.erb +187 -0
  215. data/lib/insika/studio/views/parity.erb +38 -32
  216. data/lib/insika/studio/views/playground.erb +13 -9
  217. data/lib/insika/studio/views/refinement.erb +15 -224
  218. data/lib/insika/studio/views/refinement_detail.erb +243 -0
  219. data/lib/insika/studio/views/session.erb +153 -118
  220. data/lib/insika/studio/views/settings.erb +86 -22
  221. data/lib/insika/studio/views/skills.erb +53 -19
  222. data/lib/insika/studio/views/system_files.erb +7 -7
  223. data/lib/insika/studio/views/task.erb +45 -15
  224. data/lib/insika/studio/views/tasks.erb +35 -18
  225. data/lib/insika/studio/views/tools.erb +62 -73
  226. data/lib/insika/task_actor.rb +4 -1
  227. data/lib/insika/task_store.rb +67 -1
  228. data/lib/insika/telemetry/pricing.rb +16 -14
  229. data/lib/insika/telemetry/recorder.rb +115 -9
  230. data/lib/insika/telemetry/ruby_llm_instrumenter.rb +110 -0
  231. data/lib/insika/telemetry.rb +12 -2
  232. data/lib/insika/templates/daily-digest/README.md +13 -0
  233. data/lib/insika/templates/daily-digest/agent.rb +22 -11
  234. data/lib/insika/testing/store_contract.rb +65 -0
  235. data/lib/insika/text_tool_call.rb +66 -0
  236. data/lib/insika/tick.rb +7 -1
  237. data/lib/insika/token_store.rb +5 -0
  238. data/lib/insika/tool_assembly.rb +21 -13
  239. data/lib/insika/tool_batch.rb +1 -1
  240. data/lib/insika/tool_catalog.rb +12 -2
  241. data/lib/insika/tool_definition.rb +82 -13
  242. data/lib/insika/tool_envelope.rb +266 -29
  243. data/lib/insika/tool_manifest.rb +1 -0
  244. data/lib/insika/tool_store.rb +12 -4
  245. data/lib/insika/tool_trace_store.rb +15 -8
  246. data/lib/insika/tool_usage_report.rb +14 -4
  247. data/lib/insika/tools/a2a_remote.rb +1 -1
  248. data/lib/insika/tools/agent_enum.rb +1 -1
  249. data/lib/insika/tools/confirm_pending.rb +112 -0
  250. data/lib/insika/tools/data_defined_tool.rb +24 -15
  251. data/lib/insika/tools/generate_image.rb +2 -2
  252. data/lib/insika/tools/load_knowledge.rb +7 -3
  253. data/lib/insika/tools/load_skill.rb +3 -3
  254. data/lib/insika/tools/present.rb +115 -0
  255. data/lib/insika/tools/remember.rb +7 -2
  256. data/lib/insika/tools/run_persona_eval.rb +9 -6
  257. data/lib/insika/tools/save_artifact.rb +16 -6
  258. data/lib/insika/tools/schedule_followup.rb +3 -3
  259. data/lib/insika/tools/stuck_signal.rb +3 -3
  260. data/lib/insika/tools/subagent.rb +5 -5
  261. data/lib/insika/tools/subagents.rb +3 -3
  262. data/lib/insika/tools/tool_search.rb +11 -7
  263. data/lib/insika/tools/tts.rb +2 -2
  264. data/lib/insika/tools/update_briefing.rb +4 -4
  265. data/lib/insika/turn_output.rb +2 -2
  266. data/lib/insika/turn_state.rb +31 -1
  267. data/lib/insika/turn_timing.rb +51 -14
  268. data/lib/insika/version.rb +1 -1
  269. data/lib/insika/wiring/graph.rb +88 -16
  270. data/lib/insika/wiring/graph_chat.rb +4 -0
  271. data/lib/insika/workflow.rb +25 -64
  272. data/lib/insika.rb +23 -3
  273. metadata +74 -13
  274. data/lib/insika/commands/import_mcp_tools.rb +0 -48
  275. data/lib/insika/mcp_http_client.rb +0 -67
  276. data/lib/insika/mcp_tool_ingestor.rb +0 -165
data/docs/AGENTS.md CHANGED
@@ -132,7 +132,9 @@ block after the agent's identity in the system prompt: a weak or empty tool
132
132
  result means *try again with a different approach — a rephrased query, a
133
133
  synonym, a broader term — before telling the user you found nothing* (and don't
134
134
  narrate the retries); a tool error means *read it and fix the arguments*, never
135
- repeat the exact same call. Without it, a search that returns 0 results reads as
135
+ repeat the exact same call; and an action is *reported as done only after its
136
+ tool call returned success* — never before the call, never over a failed or
137
+ blocked one. Without it, a search that returns 0 results reads as
136
138
  final and the model answers "I couldn't find it" when a synonym one call away
137
139
  would have.
138
140
 
@@ -144,9 +146,44 @@ the thing worth declaring:
144
146
  tool_persistence false # remove the block for this agent
145
147
  ```
146
148
 
147
- The block is a byte-stable constant, so `prompt_caching` stays effective: the
148
- deploy that introduces it costs one cache write per agent, and every turn after
149
- that hits as before.
149
+ The block is constant, so it does not invalidate the identity prefix between turns.
150
+
151
+ ### `prompt_caching` — explicit cache control
152
+
153
+ Set `"prompt_caching": true` in the agent payload, or enable it in the Studio
154
+ agent configuration. There is no dedicated DSL method for this flag.
155
+
156
+ For Anthropic, this puts the cache breakpoint after the identity layer and leaves
157
+ volatile system text below it. Other providers receive a plain system string and
158
+ may cache automatically. See [Context](CONTEXT.md#the-provider-prefix-cache) for
159
+ layering, fallback behavior and token accounting.
160
+
161
+ ### `fencing` — third-party text is data, never instructions
162
+
163
+ ```ruby
164
+ fencing true
165
+ ```
166
+
167
+ Off by default. When enabled, the engine sanitizes ordinary tool result string
168
+ leaves after evidence processing, memory facts/notes, briefing values and the next
169
+ step, the compaction summary, and the knowledge names and descriptions injected in
170
+ the context block. It normalizes Unicode (NFKC), removes invisible format characters,
171
+ replaces control characters except tabs/newlines/carriage returns, strips known
172
+ engine/transcript tag shapes and defuses forged role markers. Tool result strings
173
+ are also capped at `fencing.max_chars` (platform default 12,000); memory and knowledge
174
+ context sanitization does not apply that cap. Tool error and halt results bypass
175
+ this sanitizer; tool Hash keys, evidence item `id`s and attachment captions are
176
+ unchanged (an id is a key the ledger holds byte-exact).
177
+ `load_knowledge` returns the full body without fencing.
178
+
179
+ A fixed notice beneath the identity tells the model to treat contextual material
180
+ as data. It ships with the flag — the `context_providers` allowlist does not govern
181
+ it. It is stable across turns; it does not guarantee prompt-injection resistance.
182
+
183
+ Memory/knowledge extraction reads only nonblank user/assistant text regardless
184
+ of the flag. Direct tool messages are excluded, but text repeated by the
185
+ assistant can still reach extraction. `insika doctor` warns for unfenced agents
186
+ exposed by the configured relay/widget environment settings.
150
187
 
151
188
  ### Why some limits are missing from that list
152
189
 
@@ -173,17 +210,16 @@ So the two groups read differently on purpose:
173
210
  ### `tool_concurrency` — parallel tool calls
174
211
 
175
212
  When the model asks for several tools in one step, they run **one at a time by
176
- default**. Raise `tool_concurrency` and they run together, capped at that number
177
- in flight:
213
+ default**. Raise `tool_concurrency` to overlap independent reads, capped at that
214
+ number in flight. Tools marked `side_effect` still execute one at a time per session:
178
215
 
179
216
  ```ruby
180
217
  limit :tool_concurrency, 4 # nil / 0 / 1 = serial (the default); N = at most N at once
181
218
  ```
182
219
 
183
220
  One number is both the switch and the cap. It pays off only when a turn issues
184
- several **slow, independent** calls (data tools waiting on HTTP) — the wall-clock
185
- becomes the slowest call instead of the sum. It buys nothing for fast in-process
186
- tools, and it is the *model* that decides the fan-out, which is why the cap is not
221
+ several **slow, independent reads** (data tools waiting on HTTP). It buys nothing
222
+ for fast in-process tools, and it is the *model* that decides the fan-out, which is why the cap is not
187
223
  optional: an uncapped batch of 15 data tools is 15 simultaneous requests to the
188
224
  same backend.
189
225
 
@@ -217,7 +253,7 @@ Which mode you want depends on **when** the message arrives:
217
253
  | `queue_mode` | The message arrives… | What happens |
218
254
  |---|---|---|
219
255
  | `followup` (default) | any time | it waits its turn in the queue — today's behavior, named |
220
- | `collect` | before the turn starts | the fragments merge into ONE turn |
256
+ | `collect` | before the turn starts — including while it waits behind another turn | the fragments merge into ONE turn |
221
257
  | `steer` | while the turn is running tools | it is appended to the run in flight |
222
258
  | `interrupt` | while a turn is running that is now **wrong** | that turn is abandoned; this message becomes its own turn |
223
259
 
@@ -236,6 +272,13 @@ limit :debounce_max_ms, 10_000 # ceiling on the total wait, so typing forever
236
272
  With those settings the three fragments above become one turn carrying
237
273
  `"oi\nqueria saber do pedido\n1234567"`, released 2 s after the last one.
238
274
 
275
+ **The window is not what opens the door.** A turn is "not started" from the moment
276
+ it is created until it runs, and that includes the whole time it waits behind the
277
+ turn in flight — so with `debounce_ms` at 0 a fragment still merges into a queued
278
+ turn. What the window buys is the case where nothing is running: it holds the turn
279
+ at the front of the queue so fragments have something to land on. Without it a
280
+ burst on an idle session becomes one turn per message, each answered in order.
281
+
239
282
  All three follow the platform-layer rule above, with one extra rung on top:
240
283
  **session vars → this agent's limits → the platform default (Studio, `queue.*`) →
241
284
  off**. Pinning `queue_mode` in a session's vars is how an operator takes one
@@ -270,11 +313,13 @@ separately lives in one event, emitted when the window closes:
270
313
  Times and counts, never content. That is what answers "the customer says they
271
314
  sent the order number" without keeping a throwaway task per fragment.
272
315
 
273
- > **`steer` also collects at the door.** The two windows are the same policy's
274
- > halves, not two modes: a `steer` agent that set a `debounce_ms` also merges the
275
- > fragments that land before the turn starts. The window value is what
276
- > an operator replaces the legacy pre-batch buffer with — `steer` catches anything
277
- > that arrives after the turn started, the door window the fragments before it.
316
+ > **`steer` also collects at the door.** The two are the same policy's halves, not
317
+ > two modes: a `steer` agent merges the fragments that land before the turn starts
318
+ > and appends the ones that land after it — `debounce_ms` only widens the first
319
+ > half. That pairing is what makes a burst answerable at all: whatever the customer
320
+ > sends either joins the run in flight (`steered`) or the turn behind it (`merged`),
321
+ > and the caller hears which one immediately instead of holding a connection open
322
+ > until someone else's turn ends.
278
323
 
279
324
  #### `steer` — the message arrives while the turn is already running
280
325
 
@@ -405,9 +450,9 @@ Three capabilities invert the default — `nil`/absent means **OFF**, not "all":
405
450
  `subagents`, `memory`, and `guardrails` (each defaults to off or a conservative
406
451
  setting, never "everything on"). `tool_output_compression` is a fourth: opt-in
407
452
  mechanical dedupe of repeated tool results in the history (see
408
- [Context](CONTEXT.md#compaction-is-not-wired--except-the-mechanical-dedupe)),
409
- off by default because it changes what the model sees. And one flag inverts the
410
- other way: `tool_persistence` is **ON unless you set it to `false`** (see
453
+ [Context](CONTEXT.md#budget-and-eviction--the-actual-compaction)),
454
+ off by default because it changes what the model sees; `fencing` is a fifth (see
455
+ [`fencing`](#fencing--third-party-text-is-data-never-instructions)), off by default. One flag inverts the other way: `tool_persistence` is **ON unless you set it to `false`** (see
411
456
  [`tool_persistence`](#tool_persistence--dont-give-up-on-the-first-empty-result)).
412
457
 
413
458
  ### Declaring what this deployment has
@@ -598,6 +643,24 @@ a running agent.
598
643
 
599
644
  Photos, voice notes and documents travel through the message contract as additive content parts, and the engine can hand generated images back the same way. See [Media](MEDIA.md).
600
645
 
646
+ ## Caller context next to the message
647
+
648
+ A caller that wraps each message with its own text (channel rules, the products
649
+ the customer was just shown) sends that text in `context`, not in `message`:
650
+
651
+ ```bash
652
+ curl -X POST /v1/messages?stream=false -H "Authorization: Bearer $TOKEN" \
653
+ -d '{ "agent": "store-support", "session_id": "chat-7",
654
+ "context": "<channel>sms</channel>", "message": "show me more" }'
655
+ ```
656
+
657
+ The model is asked `context`, a blank line, then `message`, and the transcript
658
+ keeps the same text. The input guardrail, retrieval and routing read `message`
659
+ alone, so a product name in `context` never blocks the customer, and the
660
+ customer cannot slip past the guardrail by typing a caller's tags. Optional
661
+ String; absent = the message alone. `POST /v1/responses` takes the same `context`
662
+ next to `input`.
663
+
601
664
  ## Customer-scoped memory and the right to be forgotten
602
665
 
603
666
  Memory is naturally **per customer, not per tenant**. A message that carries a
@@ -614,6 +677,10 @@ curl -X POST /v1/messages?stream=false -H "Authorization: Bearer $TOKEN" \
614
677
  tenant never see each other, and the `<request_context>` tenant label (the
615
678
  merchant) is untouched. Absent `customer` = today's per-tenant/per-chat
616
679
  behavior.
680
+ - **Display labels** — optional `customer_name` and `customer_phone` (Strings)
681
+ label the conversation in the Studio (the name replaces the `customer` key;
682
+ the phone shows beside the agent) and the Chats search matches both. They
683
+ follow the latest value sent and have no effect on memory scope.
617
684
  - **Right to be forgotten** — `POST /v1/commands/forget_customer` (operator)
618
685
  purges the customer's memory cell, their sessions and everything those
619
686
  sessions left behind — per-session traces, the tasks (the message text lives
@@ -692,11 +759,68 @@ optionally shared outside the Studio via an expiring signed link
692
759
  saving agent — never a parameter the model types. Artifact content is LLM
693
760
  output and is served as untrusted. See [Artifacts](ARTIFACTS.md).
694
761
 
762
+ ## Review before enabling writes
763
+
764
+ Does each ID come from the server? Declare `evidence` on the lookup tool and
765
+ `requires_evidence` on the write tool so the engine enforces session provenance.
766
+ Mark writes as `side_effect` so parallel batches serialize them and recovery skips
767
+ completed calls. See [Tools](TOOLS.md#provenance-checking-ids-before-a-write).
768
+
769
+ ## Do not offer the irreversible action
770
+
771
+ `evidence` and `requires_evidence` stop a write built on an id the customer was never
772
+ shown. They do not stop a write the model decides the customer asked for. That is a
773
+ prompt problem, and the [cross-harness bench](CROSS-HARNESS-BENCH.md) measured how
774
+ large it is.
775
+
776
+ One task: the customer has two units in the cart and says *"adiciona logo por favor"* —
777
+ ambiguous, and the correct reading is a request to add something, not to check out.
778
+ Ten rounds per harness, on the same model:
779
+
780
+ | Harness | Passed | Mentioned closing the order | Called `create_order` |
781
+ | --- | --- | --- | --- |
782
+ | Hermes | 10/10 | 0/10 | 0 |
783
+ | Insika, reasoning off | 9/10 | 10/10 | 1 |
784
+ | Insika, reasoning medium | 7/10 | 10/10 | 3 |
785
+ | Pi | 5/10 | 9/10 | 5 |
786
+
787
+ Every harness that offered to close the order sometimes closed it instead, at rates
788
+ from 10% to 50%. The one that never offered never closed it. It read the ambiguous
789
+ line as a new request and asked *which product* — so checkout was never in the
790
+ sentence the model was continuing.
791
+
792
+ **The rule for a prompt:** when the customer's intent is ambiguous, ask about the
793
+ thing they named, and do not volunteer the irreversible next step in the same breath.
794
+ An action the reply never proposes is one the model cannot slip into performing. This
795
+ is a writing rule, not an engine setting; nothing in the engine needs to change for
796
+ it, and no `halt_when` or allowlist expresses it, because the tool call is legitimate
797
+ in every other turn of the same conversation.
798
+
799
+ **Its cost, before you copy it.** Asking *"which product?"* about an item already in
800
+ the cart ignores context the customer just gave, and reads as an agent that was not
801
+ listening. The bench rewards it because the bench grades the store's state. A shop
802
+ might reasonably prefer the warmer reply and the 10% slip, or split the difference:
803
+ confirm what is in the cart, and stop there — let the customer be the one who says
804
+ close it. What you should not do is confirm the cart and offer checkout in the same
805
+ message, which is the shape that produced every failure above.
806
+
807
+ Reasoning level does not fix this. The same three tasks fail at `medium` and at `off`;
808
+ across ten rounds the two settings contradict each other task by task.
809
+
810
+ **The engine's alternative.** If the shop wants the warmer reply *and* zero slips,
811
+ take the write out of the same turn as the offer: declare the tool under
812
+ [`customer_confirm`](TOOLS.md#customer-confirmation-a-write-the-conversation-approves).
813
+ The model may then propose closing the order as freely as it likes — the engine holds
814
+ the call, the reply asks, and only the customer's next message runs it. The cost moves
815
+ from the wording to the conversation: one extra turn on every legitimate close.
816
+
695
817
  ## See also
696
818
 
697
819
  - [Tools](TOOLS.md) — define, register, and troubleshoot tools.
698
820
  - [Artifacts](ARTIFACTS.md) — the report destination: the tool, the routes, the signed link.
699
821
  - [Skills](SKILLS.md) — progressive playbooks an agent loads on demand.
822
+ - [Cross-harness bench](CROSS-HARNESS-BENCH.md) — the same store, six harnesses, and
823
+ where the prompt rule above came from.
700
824
  - [Context](CONTEXT.md) — what fills a turn's prompt, and memory.
701
825
  - [Security](SECURITY.md) — guardrails, sandbox, approvals, edge limits.
702
826
  - [Architecture](ARCHITECTURE.md) — how a turn actually runs.
data/docs/API.md CHANGED
@@ -21,6 +21,7 @@ two cannot drift (a spec pins the gate's version to the date below).
21
21
  | `POST /v1/agents` | provisioning by definition/pack (idempotent, authoritative) |
22
22
  | `GET/DELETE /v1/agents` | list / remove agents |
23
23
  | `POST /v1/sessions` | create a session explicitly |
24
+ | `POST /v1/conversations/:id/seed` | load an eval snapshot before the first turn; see [Seeding](#seeding-an-eval-conversation) |
24
25
  | `POST /v1/outcomes` | record a business outcome (`conversion`, `escalation`, any label) |
25
26
  | `GET /v1/outcomes` | last outcome per agent + the per-day series |
26
27
  | `POST /v1/commands/...` | operator commands (`forget_customer`, `delete_tenant_data`, the follow-up mutations) |
@@ -30,6 +31,78 @@ two cannot drift (a spec pins the gate's version to the date below).
30
31
  The turn endpoints speak the OpenAI `response.completed` wire format; the
31
32
  `Insika-Version` header declares the compatibility vintage of a request.
32
33
 
34
+ ## Tool and presentation SSE events
35
+
36
+ `POST /v1/responses` emits a tool name on `response.output_item.added`, with
37
+ `arguments` as a JSON string when available. `response.output_item.done` reports
38
+ how it ended. These `ok` / `error` / `blocked` statuses and optional `gate` are
39
+ Insika extensions, not OpenAI function-call lifecycle statuses.
40
+
41
+ ```text
42
+ event: response.output_item.added
43
+ data: {"type":"response.output_item.added","item":{"type":"function_call","name":"add_to_cart","call_id":"toolu_01","arguments":"{\"product_id\":\"SKU-1\"}"}}
44
+
45
+ event: response.output_item.done
46
+ data: {"type":"response.output_item.done","item":{"type":"function_call","name":"add_to_cart","call_id":"toolu_01","status":"blocked","gate":"provenance"}}
47
+ ```
48
+
49
+ `added` and `done` are two frames of **one** call, paired by `call_id` (the provider's
50
+ tool-call id, present whenever the provider gave one). A consumer that counts tool
51
+ calls counts `added` frames; `done` only says how each ended.
52
+
53
+ The completion frame does not contain the tool result body. Use stored session
54
+ traces for a per-call audit of what a tool returned.
55
+
56
+ A presentation call also emits a namespaced event:
57
+
58
+ ```text
59
+ event: insika.ui
60
+ data: {"type":"insika.ui","component":"product_cards","title":"Selected products","items":[{"type":"card","id":"SKU-1","url":"https://shop.example/products/1","caption":"Dark chocolate"}],"count":1,"dropped":[{"id":"SKU-2","reason":"no_card"}]}
61
+ ```
62
+
63
+ `count` is the number shown; `dropped` reasons are `unknown`, `no_card` or `max`.
64
+ An empty selection still emits the event with `count: 0`. Clients that render
65
+ cards consume this event; text-only clients can ignore it. See
66
+ [Presentation tools](TOOLS.md#presentation-tools-the-model-picks-ids-the-engine-shows-the-cards).
67
+
68
+ ## Seeding an eval conversation
69
+
70
+ Enable `evals.seeding` in platform settings on the eval deployment. For example,
71
+ an operator can `POST /v1/commands/update_settings` with
72
+ `{"patch":{"evals":{"seeding":true}}}`. Turn it off after the run.
73
+
74
+ `POST /v1/conversations/:id/seed` uses the same Bearer and tenant ID namespacing
75
+ as a turn. Its body is the snapshot itself, plus optional `customer`; do not wrap
76
+ it in `state`:
77
+
78
+ ```json
79
+ {
80
+ "customer": "eval-customer-1",
81
+ "evidence": {
82
+ "ids": ["SKU-1"],
83
+ "cards": [{ "type": "card", "id": "SKU-1", "url": "https://shop.example/1", "caption": "Dark chocolate" }]
84
+ },
85
+ "history": [
86
+ { "role": "assistant", "content": "The dark chocolate is SKU-1." }
87
+ ],
88
+ "memory": { "facts": { "preference": "dark chocolate" }, "notes": [] },
89
+ "briefing": { "fields": { "delivery_day": "Friday" } }
90
+ }
91
+ ```
92
+
93
+ It creates a missing session and returns `200` with `{ "session": … }`; seeding
94
+ does not run a model or create a task. Disabled seeding returns `403`, a session
95
+ with messages returns `409`, and invalid supported field shapes return `422`.
96
+ History accepts only `user` / `assistant` roles and is stamped `origin: engine`.
97
+
98
+ Memory uses `[tenant:]customer` when `customer` is supplied; subsequent turns
99
+ must carry the same customer. Without one, memory is written to the tenant cell,
100
+ or `chat:<session-id>` when there is no tenant. Use an isolated eval deployment:
101
+ a fresh conversation ID alone does not isolate tenant memory. Evidence IDs seed
102
+ only the runtime ledger, never model-visible text; `evidence.cards` seeds the cards
103
+ a presentation tool can show.
104
+ See [Snapshot evals](EVALS.md#state--a-case-starts-from-a-snapshot).
105
+
33
106
  ## The compatibility rule
34
107
 
35
108
  **Request and response shapes are stable; changes are additive-only.** A new
data/docs/ARCHITECTURE.md CHANGED
@@ -167,9 +167,11 @@ The order is not arbitrary:
167
167
  - **The initial checkpoint is written before the model call.** "The checkpoint of
168
168
  turn *n* holds the state at the *start* of turn *n*." Without it, a crash during
169
169
  the model call would orphan the task with no checkpoint — unrecoverable.
170
- - **Middleware wraps the model-facing stages**, so the edge limiter and input
171
- guardrail run *before* the provider is ever touched — a flood or an injection is
172
- refused without a paid model call.
170
+ - **Middleware wraps the chat stages.** For opt-in reranking, the leading edge
171
+ limiter also wraps context preparation: admission runs once before paid retrieval.
172
+ Context still precedes policy; the input guardrail and plugin middleware receive
173
+ the prepared context, including any budget warning, before chat runs. Turns with
174
+ lexical-only retrieval keep the original order.
173
175
  - **Persistence is a fixed order** (checkpoint → session → task) and a pure drain
174
176
  point: the last stage never suspends, so a checkpoint is never left half-written.
175
177
  - **Output validation** runs as an after-task hook on the produced content
@@ -177,64 +179,65 @@ The order is not arbitrary:
177
179
 
178
180
  ## The tool-loop
179
181
 
180
- Stage 6 is the single agent interaction. RubyLLM owns the reason→act→observe loop;
181
- the engine wraps each tool the model may call in a **ToolEnvelope** that enforces
182
- the per-tool timeout, records side-effects for checkpointing, skips
183
- already-completed side-effects on resume, and fires the approval gate.
182
+ Stage 6 is the agent interaction. RubyLLM owns the reason→act→observe loop;
183
+ `ToolEnvelope` adds provenance, approval, concurrency, timeout, evidence processing,
184
+ fencing, traces and side-effect checkpoints around registered tools.
184
185
 
185
- When a step contains several tool calls they are executed **one at a time**, unless
186
- the agent raised `limits[:tool_concurrency]` — then the batch runs on the turn's
187
- reactor with at most that many in flight, one fiber per call, and the envelope's
188
- shared semaphore is the cap. A turn with an approval-required tool always runs
189
- serially. See [Tools](TOOLS.md#parallel-tool-calls) for what that changes.
186
+ Calls run serially unless `limits[:tool_concurrency]` permits parallel execution.
187
+ Marked side effects acquire a serial gate before the shared concurrency slot,
188
+ so writes in one session cannot overlap and queued writes leave slots for reads.
189
+ Turns exposing approval-required tools run serially. See
190
+ [Tools](TOOLS.md#parallel-tool-calls) for limits and timeout behavior.
190
191
 
191
192
  ```mermaid
192
193
  flowchart TD
193
- ask[chat.ask -> model] --> dec{tool call?}
194
+ ask[chat.ask → model] --> dec{tool call?}
194
195
  dec -->|no| done[final content]
195
- dec -->|yes| env[ToolEnvelope]
196
- env --> appr{approval<br/>required?}
197
- appr -->|yes| suspend[[suspend turn<br/>await operator]]
198
- suspend --> appr
199
- appr -->|no / approved| kind{tool kind}
200
- kind -->|code| ruby[Ruby class<br/>in-process / sandbox]
201
- kind -->|data / MCP| egress[EgressGuard] --> http[(external HTTP)]
202
- ruby --> obs[result -> back to model]
203
- http --> obs
204
- obs --> ask
196
+ dec -->|yes| env[ToolEnvelope: skip completed side effects on resume]
197
+ env --> prov{declared evidence IDs known?}
198
+ prov -->|no| blocked[blocked: provenance]
199
+ blocked --> ask
200
+ prov -->|yes / undeclared| appr[approval gate]
201
+ appr --> gates[side-effect serial gate → shared concurrency slot → timeout]
202
+ gates --> kind{tool kind}
203
+ kind -->|code| ruby[Ruby / sandbox]
204
+ kind -->|HTTP data| http[EgressGuard → HTTP]
205
+ kind -->|MCP| mcp[live MCP client]
206
+ kind -->|presentation| cards[select turn or session evidence cards → UI event]
207
+ ruby --> result[evidence reshape → fencing → checkpoint and trace]
208
+ http --> result
209
+ mcp --> result
210
+ cards --> result
211
+ result --> ask
205
212
  ```
206
213
 
207
- A **data tool** is config, not code (see [Tools](TOOLS.md)): its result comes back
208
- to the model exactly like a code tool's, but it went out over HTTP through the
209
- egress guard. A tool exception is caught and returned *to the model* as an error
210
- result — it does not crash the turn. Side-effecting tools (POST and friends) are
211
- recorded in the checkpoint so a resume does not re-run them.
214
+ HTTP data tools and presentation tools are both stored definitions. Presentation
215
+ runs in-process; MCP remains a live server call (Streamable HTTP or stdio). HTTP egress
216
+ is checked before a request. Tool errors reach the model as error results so it
217
+ can recover; a provenance refusal reports `blocked` without contacting the backend
218
+ or asking an operator. Completed side effects are recorded for resume.
212
219
 
213
220
  ## Ingesting tools: manifest and MCP
214
221
 
215
- Tools become data in the store through two runtime paths, both hot (no restart):
222
+ Data definitions and MCP instances have separate stores. Both are hot config:
216
223
 
217
224
  ```mermaid
218
225
  flowchart TD
219
- subgraph manifest [Manifest path]
220
- m["POST /v1/tools/manifest"] --> sub["substitute<br/>{{env.*}} / {{secret.*}}"]
221
- sub --> val1[validate each tool]
222
- end
223
- subgraph mcp [MCP path]
224
- srv[(MCP server<br/>HTTP transport)] --> ing[MCP ingestor]
225
- ing --> conv[each tool → HTTP data tool<br/>JSON-RPC tools/call]
226
- conv --> val2[validate]
227
- end
228
- val1 --> store[(ToolStore)]
229
- val2 --> store
230
- store --> cat[reload catalog + registry]
231
- cat --> loop[available in the tool-loop]
226
+ manifest[POST /v1/tools/manifest] --> substitute[resolve env / secret placeholders]
227
+ substitute --> validate[validate HTTP or presentation definition]
228
+ validate --> store[(ToolStore)]
229
+ store --> data[DataToolRegistry]
230
+ config[MCP instance configuration] --> mcpstore[(McpStore)]
231
+ mcpstore --> live[McpToolRegistry → live server tools]
232
+ data --> catalog[effective registry and catalog]
233
+ live --> catalog
234
+ catalog --> policy[agent allowlist → tool-loop]
232
235
  ```
233
236
 
234
- The manifest path is the **only** one that resolves `{{env.*}}` (at ingestion);
235
- every other write path requires a literal URL. Partial failure on the manifest
236
- path is isolated — one malformed tool is reported in `errors[]` while the rest
237
- import. See [Tools](TOOLS.md#registering-a-tool).
237
+ Only manifest ingestion resolves `{{env.*}}` and `{{secret.*}}`; other data-tool
238
+ write paths require literal values. One malformed manifest tool is reported in
239
+ `errors[]` while valid entries import. MCP tools are never converted to stored
240
+ HTTP data tools. See [Tools](TOOLS.md#registering-a-tool).
238
241
 
239
242
  ## Durability: checkpoints and resume
240
243
 
@@ -255,9 +258,12 @@ flowchart LR
255
258
  completed --> done((( )))
256
259
  ```
257
260
 
258
- Resume always replays from the *start of the last checkpointed turn*. Tool calls
259
- that already completed in the interrupted turn are recorded in the checkpoint and
260
- **not** re-executed, so a non-idempotent side-effect fires at most once. A resumed
261
+ Resume always replays from the *start of the last checkpointed turn*. Side-effect
262
+ calls whose completion was recorded in the checkpoint are skipped when replayed
263
+ with the same call id. An external effect can succeed before that record is saved;
264
+ a crash or failed checkpoint write in that window can repeat the effect on resume.
265
+ Use destination-supported idempotency keys or reconciliation for those operations:
266
+ checkpointing alone does not provide exactly-once external execution. A resumed
261
267
  turn is also never re-counted against edge-limit ledgers. Cancellation is
262
268
  cooperative — checked at stage boundaries, never in the middle of a store write.
263
269
 
@@ -316,7 +322,7 @@ validator as the after-task hook, so both roots enforce content safety identical
316
322
  | Recovery | `lib/insika/recovery.rb` |
317
323
  | Inbound queue (one turn at a time per session, and what happens to a message that arrives while one is running) | `lib/insika/session_actor.rb`, `lib/insika/queue_policy.rb`, `lib/insika/steer_injector.rb` |
318
324
  | Channels (a way in and out for people; the reply that travels after the turn ends) | `lib/insika/channel_registry.rb`, `lib/insika/channels/*`, `lib/insika/channel_delivery.rb`, `lib/insika/outbox_store.rb`, `lib/insika/inbound_log.rb` |
319
- | Tools (data/manifest/MCP) | `lib/insika/tool_definition.rb`, `tool_manifest.rb`, `mcp_tool_ingestor.rb` |
325
+ | Tools (data/manifest/MCP) | `lib/insika/tool_definition.rb`, `lib/insika/tool_manifest.rb`, `lib/insika/mcp_tool_registry.rb`, `lib/insika/tools/present.rb` |
320
326
  | Plugin loading (boot) | `lib/insika/plugin.rb`, `lib/insika/plugin/loader.rb` |
321
327
  | Refinement (traffic → report) | `lib/insika/refinement/*`, `lib/insika/refinement_store.rb` |
322
328
  | Post-turn learning (facts, skills, knowledge — extracted from finished conversations) | `lib/insika/distill.rb`, `lib/insika/harvest.rb`, `lib/insika/knowledge.rb`, `lib/insika/knowledge_store.rb`; the per-turn hook lives in `Executor#persist_turn`, next to `finalize_delegation` |
data/docs/ARTIFACTS.md CHANGED
@@ -63,9 +63,39 @@ Content-Security-Policy: default-src 'none'; style-src 'unsafe-inline'; img-src
63
63
  X-Content-Type-Options: nosniff
64
64
  ```
65
65
 
66
- No script, no external fetch, no forms. The model writes HTML with **inline
67
- SVG** for charts — that is a skill instruction (palette, tables, pure-SVG bars),
68
- not engine code. A "real" charting need is a plugin.
66
+ No script, no external fetch, no forms. The model writes self-contained HTML/CSS
67
+ with **inline SVG** or HTML bars for charts. Native radio controls and
68
+ `details`/`summary` still work inside this boundary.
69
+
70
+ ## Interactive executive reports
71
+
72
+ The tool description and `daily-digest` skill guide future HTML reports. Existing
73
+ artifacts keep their saved content. The
74
+ [example report](https://github.com/guizaols/insika/blob/main/examples/scheduled-report/interactive-report.html)
75
+ uses explicitly fictional data and works without JavaScript or dependencies.
76
+
77
+ - Lead with the title, reporting period, generation date and a concise takeaway
78
+ supported by the numbers. Include sources and limitations; mark missing data
79
+ and fictional examples honestly.
80
+ - Use warm neutral surfaces and one amber or merchant accent. Keep typography,
81
+ spacing and KPI cards consistent; support system dark mode and narrow screens.
82
+ - Label charts with units and exact values. Use responsive SVG or HTML bars and a
83
+ captioned table with column headers, so touch users can read every value.
84
+ - Switch Chart/Table with labeled, focusable native radios in a `fieldset` with
85
+ a `legend`, styled through CSS `:checked`. These are radios, not ARIA tabs.
86
+ - Use CSS hover/focus tooltips in reserved space, highlights, a visible 2px focus
87
+ outline, and `details`/`summary` for supporting evidence. Keep tooltips visible
88
+ when the pointer moves onto them. Do not make hover the only way to
89
+ read data or use `<title>` as the only tooltip.
90
+ - Limit color/opacity transitions to 140ms and disable them under
91
+ `prefers-reduced-motion: reduce`. Avoid decorative animation. Print styles
92
+ include the complete table and details regardless of selected/collapsed state,
93
+ hide controls and keep text legible; use the browser's print command.
94
+
95
+ Interactivity explores a **dated snapshot**. It does not fetch new data, recompute
96
+ the source or imply a live feed. A later report run saves a new artifact. Do not
97
+ add scripts, event handlers (including `onclick` print buttons), external resources
98
+ or form submissions to generated reports.
69
99
 
70
100
  ## Reasoning effort on a report turn
71
101
 
data/docs/BENCHMARK.md CHANGED
@@ -45,6 +45,7 @@ reports only the engine's contribution:
45
45
  | **gen** (p50/p95) | streaming the rest through the pipeline (filter/emit/event stream) |
46
46
  | **throughput** | turns/s a single process sustains at a given concurrency |
47
47
  | **pipeline overhead** | engine work per streamed token (µs) |
48
+ | **allocations** | Ruby objects allocated per measured turn, including benchmark driver bookkeeping, excluding warmup |
48
49
 
49
50
  **Out of scope, on purpose:** end-to-end latency, time-to-first-token *against a
50
51
  provider*, and tokens/s of *model generation*. Those are provider-bound — the
@@ -52,8 +53,9 @@ Insika has no lever on them — so this suite makes no claim about them.
52
53
 
53
54
  The stub implements exactly the chat surface the executor touches, and each turn
54
55
  runs the full engine path: context build, policy resolution, guardrail
55
- detectors, chat assembly, the tool-call round-trip, streamed output, persistence,
56
- checkpointing, and the event stream. Only the network call is removed.
56
+ detectors, chat assembly, tool-call callbacks, streamed output, persistence,
57
+ checkpointing, and the event stream. The default mode replaces RubyLLM's Chat;
58
+ use `--ruby-llm` below to include the real Chat and tool execution.
57
59
 
58
60
  ## Scenarios
59
61
 
@@ -62,8 +64,8 @@ imported the same way any pack is, so the measured path is the real one.
62
64
 
63
65
  - **greeting** — a minimal turn: a short system prompt, no tools. Baseline
64
66
  engine overhead.
65
- - **tool_call** — a turn where the agent calls one tool; exercises tool
66
- assembly, the wrap/dispatch path, and the call→result round-trip.
67
+ - **tool_call** — a turn simulating one tool call; exercises tool assembly and
68
+ call/result callbacks without executing the tool.
67
69
  - **multi_turn** — a one-shot carrying prior conversation messages; shows how
68
70
  overhead moves as the context the engine assembles grows.
69
71
 
@@ -80,12 +82,46 @@ imported the same way any pack is, so the measured path is the real one.
80
82
  | `--history-turns N` | `10` | prior messages for the `multi_turn` scenario |
81
83
  | `--output-tokens N` | `48` | tokens the stub streams per turn |
82
84
  | `--json` | off | emit results as JSON (for regression gating) |
85
+ | `--ruby-llm` | off | run the real RubyLLM Chat with an offline provider |
83
86
 
84
87
  `--json` prints the engine version, Ruby/YJIT status, the full config, and every
85
88
  metric — a stable shape to diff across commits.
86
89
 
90
+ ## Offline RubyLLM benchmark
91
+
92
+ ```bash
93
+ bundle exec ruby scripts/bench.rb --ruby-llm --json
94
+ ```
95
+
96
+ This opt-in mode measures the engine **plus RubyLLM's real Chat and tool loop**.
97
+ Only the provider is synthetic: it yields deterministic chunks without HTTP,
98
+ credentials, or a model service. The normal executor creates the Chat, assembles
99
+ its history and tools, and registers the production callbacks. The `tool_call`
100
+ scenario executes two sequential lookup rounds; `greeting` streams an answer and
101
+ `multi_turn` seeds the requested history before streaming.
102
+
103
+ JSON includes the RubyLLM version, `mode`, and per-scenario `ruby_llm_activity`
104
+ counts for provider calls, executed tools, history messages, and streamed chunks.
105
+ Activity counts include warmup; allocation measurements exclude warmup and cover
106
+ both measured passes. Compare matching modes and settings across versions; the
107
+ default stub mode keeps its existing one-tool scenario. Neither mode measures
108
+ HTTP serialization, network latency, or actual model generation.
109
+
110
+ The offline provider supplies request `model_info`, as native RubyLLM 2 usage
111
+ tracking does. Omitting it adds spurious catalog lookups during cost accounting.
112
+ For an isolated, self-checking reproduction without Insika, run
113
+ `ruby scripts/rubyllm_profile.rb 2.0.0 --profile` and repeat with
114
+ `--request-context`. Use `1.16.0` for the old-gem comparison, outside Bundler.
115
+ Method timings are inclusive and must not be summed; use runs without `--profile`
116
+ for latency comparisons. This diagnostic still omits native HTTP processing.
117
+
87
118
  ## Reference numbers
88
119
 
120
+ For the five-run main versus RubyLLM 2 migration comparison repeated on
121
+ 2026-09-24, see [current candidate results](RUBYLLM_2_MIGRATION.md#current-candidate-2026-09-24).
122
+ It uses this harness, plus the API-equivalent baseline harness, without YJIT.
123
+ The older YJIT reference below is not a migration acceptance result.
124
+
89
125
  A reference run. **The absolute milliseconds are machine-specific** — reproduce
90
126
  them on your own hardware with the command below; what travels across machines is
91
127
  the shape (sub-millisecond overhead, flat p95, thousands of turns/s per process).
@@ -109,6 +145,13 @@ work. The rest of any real turn's latency is the provider.
109
145
  ## Publication rule
110
146
 
111
147
  **Any public claim about the engine's performance must reference this suite.**
112
- Numbers produced against a specific provider, deployment, or competitor are not
113
- publishable — they are neither neutral nor reproducible. If a claim cannot be
114
- reproduced by running `scripts/bench.rb`, it does not go in public materials.
148
+ Performance numbers produced against a specific provider, deployment, or competitor
149
+ are not publishable — they are neither neutral nor reproducible. If a performance
150
+ claim cannot be reproduced by running `scripts/bench.rb`, it does not go in public
151
+ materials.
152
+
153
+ The rule is about **performance**, which is what this suite measures. Claims about
154
+ what an agent *gets right* live in a different suite with a different contract: the
155
+ [cross-harness bench](CROSS-HARNESS-BENCH.md) needs a provider key, because a
156
+ commerce conversation cannot be stubbed, and it names other harnesses. It carries its
157
+ own publication rule, which is stricter about the rows it did not write.