insika 0.8.0 → 1.0.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +312 -2
- data/README.md +10 -6
- data/bin/insika +4 -1
- data/bin/insika-router +0 -1
- data/docs/AGENTS.md +142 -18
- data/docs/API.md +73 -0
- data/docs/ARCHITECTURE.md +56 -50
- data/docs/ARTIFACTS.md +33 -3
- data/docs/BENCHMARK.md +50 -7
- data/docs/CHANNELS.md +19 -2
- data/docs/CONTEXT.md +96 -27
- data/docs/CROSS-HARNESS-BENCH.md +157 -0
- data/docs/DEPLOY.md +70 -13
- data/docs/EMBEDDING.md +10 -0
- data/docs/EVALS.md +129 -10
- data/docs/FACTS.md +12 -6
- data/docs/HARVEST.md +8 -3
- data/docs/KNOWLEDGE.md +14 -0
- data/docs/MEDIA.md +6 -2
- data/docs/OBSERVABILITY.md +164 -28
- data/docs/PERFORMANCE.md +110 -0
- data/docs/PLUGINS.md +4 -4
- data/docs/POLICY.md +26 -14
- data/docs/RELEASING.md +9 -0
- data/docs/ROUTER.md +7 -0
- data/docs/RUBYLLM_2_MIGRATION.md +307 -0
- data/docs/SECURITY.md +34 -1
- data/docs/SHARED-CONVERSATIONS.md +40 -0
- data/docs/SKILLS.md +5 -3
- data/docs/STUDIO.md +122 -0
- data/docs/TOOLS.md +238 -42
- data/docs/operate.md +3 -0
- data/docs/prompts/ADD-TOOL.md +12 -2
- data/docs/prompts/DIAGNOSE-TURN.md +3 -0
- data/docs/prompts/GO-LIVE.md +3 -1
- data/lib/insika/agent_defaults.rb +140 -0
- data/lib/insika/agent_file_store.rb +12 -0
- data/lib/insika/agent_profile.rb +99 -17
- data/lib/insika/artifact_signing.rb +11 -0
- data/lib/insika/channels/web/widget.js +33 -0
- data/lib/insika/channels/web.rb +5 -2
- data/lib/insika/chat_builder.rb +124 -28
- data/lib/insika/checkpoint.rb +6 -1
- data/lib/insika/checkpoint_store.rb +59 -7
- data/lib/insika/commands/agent_payload.rb +3 -3
- data/lib/insika/commands/delete_kit.rb +26 -0
- data/lib/insika/commands/gate_harvest.rb +3 -0
- data/lib/insika/commands/review_guardrail_flag.rb +38 -0
- data/lib/insika/commands/run_distillation.rb +5 -8
- data/lib/insika/commands/run_harvest.rb +17 -1
- data/lib/insika/commands/seed_session.rb +128 -0
- data/lib/insika/commands/send_message.rb +9 -3
- data/lib/insika/commands/update_settings.rb +23 -1
- data/lib/insika/commands/write_kit.rb +27 -0
- data/lib/insika/compaction.rb +33 -7
- data/lib/insika/config_store.rb +87 -7
- data/lib/insika/context/builder.rb +29 -9
- data/lib/insika/context/priority.rb +5 -0
- data/lib/insika/context/provider.rb +7 -2
- data/lib/insika/context/providers/briefing.rb +11 -8
- data/lib/insika/context/providers/fence_notice.rb +27 -0
- data/lib/insika/context/providers/knowledge.rb +32 -6
- data/lib/insika/context/providers/memory.rb +61 -8
- data/lib/insika/context/providers/pending_confirmation.rb +52 -0
- data/lib/insika/context/providers/prompt.rb +17 -4
- data/lib/insika/context/providers/session.rb +46 -6
- data/lib/insika/context/reranker.rb +75 -0
- data/lib/insika/distill.rb +22 -13
- data/lib/insika/doctor.rb +142 -4
- data/lib/insika/dsl/runtime.rb +37 -4
- data/lib/insika/dsl/server_boot.rb +4 -12
- data/lib/insika/dsl.rb +19 -5
- data/lib/insika/edge_limiter.rb +27 -16
- data/lib/insika/egress_guard.rb +11 -3
- data/lib/insika/env_schema.rb +33 -73
- data/lib/insika/errors.rb +1 -0
- data/lib/insika/evals/assertions.rb +259 -18
- data/lib/insika/evals/golden.rb +230 -4
- data/lib/insika/evals/judge.rb +19 -10
- data/lib/insika/evals/moderator_compare.rb +47 -0
- data/lib/insika/evals/report.rb +61 -9
- data/lib/insika/evals/runner.rb +64 -4
- data/lib/insika/evals/simulator.rb +21 -3
- data/lib/insika/evals/transport.rb +317 -15
- data/lib/insika/event_stream.rb +63 -7
- data/lib/insika/evidence.rb +105 -17
- data/lib/insika/executor.rb +698 -164
- data/lib/insika/fence.rb +96 -0
- data/lib/insika/golden_store.rb +3 -0
- data/lib/insika/harvest.rb +26 -14
- data/lib/insika/harvest_engine.rb +1 -0
- data/lib/insika/harvest_store.rb +12 -0
- data/lib/insika/http_client.rb +87 -4
- data/lib/insika/kits.rb +54 -0
- data/lib/insika/knowledge.rb +34 -20
- data/lib/insika/llm_configurator.rb +46 -4
- data/lib/insika/llm_provider_store.rb +1 -3
- data/lib/insika/llm_trace_store.rb +54 -0
- data/lib/insika/mcp_client.rb +27 -63
- data/lib/insika/mcp_live_tool.rb +55 -9
- data/lib/insika/mcp_store.rb +37 -3
- data/lib/insika/mcp_tool_registry.rb +76 -26
- data/lib/insika/media.rb +38 -25
- data/lib/insika/memory_store.rb +16 -3
- data/lib/insika/memory_trim.rb +60 -0
- data/lib/insika/middleware.rb +12 -3
- data/lib/insika/model_metrics_store.rb +285 -0
- data/lib/insika/model_registry.rb +110 -0
- data/lib/insika/model_resolver.rb +1 -1
- data/lib/insika/model_selection.rb +14 -21
- data/lib/insika/model_visible.rb +17 -6
- data/lib/insika/onboarding.rb +1 -0
- data/lib/insika/outcome_store.rb +1 -1
- data/lib/insika/overlay_tool_registry.rb +30 -2
- data/lib/insika/pack_importer.rb +54 -9
- data/lib/insika/packaging.rb +14 -5
- data/lib/insika/pending_action_store.rb +45 -18
- data/lib/insika/plugin/loader.rb +4 -2
- data/lib/insika/prefix_fingerprint.rb +32 -27
- data/lib/insika/profile_source.rb +5 -0
- data/lib/insika/provider_error_classifier.rb +18 -3
- data/lib/insika/queue_policy.rb +7 -2
- data/lib/insika/refinement/proposer.rb +42 -17
- data/lib/insika/reliability.rb +12 -3
- data/lib/insika/router/app.rb +14 -5
- data/lib/insika/router/session_key.rb +35 -5
- data/lib/insika/safety/config.rb +20 -3
- data/lib/insika/safety/corpus.rb +10 -3
- data/lib/insika/safety/factory.rb +39 -1
- data/lib/insika/safety/input_guardrail.rb +21 -1
- data/lib/insika/safety/judge_moderator.rb +70 -0
- data/lib/insika/safety/output_validator.rb +29 -5
- data/lib/insika/safety/prompt_echo.rb +89 -0
- data/lib/insika/schema_guard.rb +53 -76
- data/lib/insika/server/app.rb +58 -7
- data/lib/insika/server/rack_app.rb +2 -0
- data/lib/insika/server/responses.rb +50 -5
- data/lib/insika/session_actor.rb +58 -28
- data/lib/insika/session_store.rb +163 -11
- data/lib/insika/settings_store.rb +57 -2
- data/lib/insika/shared_conversations.rb +305 -0
- data/lib/insika/skill_catalog.rb +28 -2
- data/lib/insika/skill_store.rb +6 -0
- data/lib/insika/soak/runner.rb +1 -1
- data/lib/insika/spoken_transcript.rb +31 -0
- data/lib/insika/steer_injector.rb +7 -48
- data/lib/insika/store.rb +29 -1
- data/lib/insika/stores/json_value.rb +37 -0
- data/lib/insika/stores/memory.rb +22 -35
- data/lib/insika/stores/postgres.rb +295 -0
- data/lib/insika/stores/sqlite.rb +257 -43
- data/lib/insika/stores/turn_counter.rb +25 -0
- data/lib/insika/studio/app.rb +1459 -163
- data/lib/insika/studio/assets/dist/application.css +1 -1
- data/lib/insika/studio/assets/dist/application.js +25 -25
- data/lib/insika/studio/assets/dist/fonts/ibm-plex-mono-latin-400-normal.woff2 +0 -0
- data/lib/insika/studio/assets/dist/fonts/ibm-plex-mono-latin-500-normal.woff2 +0 -0
- data/lib/insika/studio/assets/dist/fonts/ibm-plex-mono-latin-600-normal.woff2 +0 -0
- data/lib/insika/studio/assets/dist/fonts/ibm-plex-sans-latin-wght-normal.woff2 +0 -0
- data/lib/insika/studio/forms.rb +189 -12
- data/lib/insika/studio/nav_icons.rb +3 -0
- data/lib/insika/studio/views/_agent_config_retrieval.erb +73 -0
- data/lib/insika/studio/views/_agent_picker.erb +15 -0
- data/lib/insika/studio/views/_agent_tab_config.erb +83 -243
- data/lib/insika/studio/views/_agent_tab_history.erb +56 -1
- data/lib/insika/studio/views/_agent_tab_performance.erb +34 -0
- data/lib/insika/studio/views/_agent_tab_prompts.erb +7 -9
- data/lib/insika/studio/views/_agent_tab_skills.erb +16 -7
- data/lib/insika/studio/views/_agent_tab_tools.erb +90 -0
- data/lib/insika/studio/views/_agents_master.erb +8 -12
- data/lib/insika/studio/views/_cfg_alerts.erb +4 -0
- data/lib/insika/studio/views/_cfg_budget.erb +20 -0
- data/lib/insika/studio/views/_cfg_distill.erb +26 -0
- data/lib/insika/studio/views/_cfg_followup.erb +30 -0
- data/lib/insika/studio/views/_cfg_grounding.erb +21 -0
- data/lib/insika/studio/views/_cfg_guardrails.erb +47 -0
- data/lib/insika/studio/views/_cfg_harvest.erb +35 -0
- data/lib/insika/studio/views/_cfg_reliability.erb +32 -0
- data/lib/insika/studio/views/_chats_master.erb +66 -0
- data/lib/insika/studio/views/_customers_master.erb +37 -0
- data/lib/insika/studio/views/_empty.erb +15 -0
- data/lib/insika/studio/views/_facts_master.erb +42 -0
- data/lib/insika/studio/views/_harvest_candidate_detail.erb +14 -0
- data/lib/insika/studio/views/_identity.erb +16 -0
- data/lib/insika/studio/views/_kpi.erb +17 -0
- data/lib/insika/studio/views/_message.erb +40 -2
- data/lib/insika/studio/views/_model_chart.erb +82 -0
- data/lib/insika/studio/views/_refinement_master.erb +38 -0
- data/lib/insika/studio/views/{tool_edit.erb → _tool_edit.erb} +38 -11
- data/lib/insika/studio/views/_tool_users.erb +17 -0
- data/lib/insika/studio/views/_why.erb +53 -0
- data/lib/insika/studio/views/agent_detail.erb +17 -10
- data/lib/insika/studio/views/agents.erb +28 -16
- data/lib/insika/studio/views/approvals.erb +16 -11
- data/lib/insika/studio/views/artifact.erb +20 -7
- data/lib/insika/studio/views/artifacts.erb +19 -14
- data/lib/insika/studio/views/chats.erb +22 -29
- data/lib/insika/studio/views/customer.erb +88 -43
- data/lib/insika/studio/views/customers.erb +16 -26
- data/lib/insika/studio/views/evals.erb +49 -31
- data/lib/insika/studio/views/fact_detail.erb +114 -0
- data/lib/insika/studio/views/facts.erb +15 -116
- data/lib/insika/studio/views/followups.erb +63 -52
- data/lib/insika/studio/views/funnel.erb +104 -43
- data/lib/insika/studio/views/harvest.erb +85 -59
- data/lib/insika/studio/views/home.erb +138 -23
- data/lib/insika/studio/views/inbox.erb +36 -0
- data/lib/insika/studio/views/kits.erb +176 -0
- data/lib/insika/studio/views/knowledge.erb +92 -80
- data/lib/insika/studio/views/layout.erb +42 -11
- data/lib/insika/studio/views/live.erb +45 -0
- data/lib/insika/studio/views/mcp.erb +43 -13
- data/lib/insika/studio/views/models.erb +187 -0
- data/lib/insika/studio/views/parity.erb +38 -32
- data/lib/insika/studio/views/playground.erb +13 -9
- data/lib/insika/studio/views/refinement.erb +15 -224
- data/lib/insika/studio/views/refinement_detail.erb +243 -0
- data/lib/insika/studio/views/session.erb +153 -118
- data/lib/insika/studio/views/settings.erb +86 -22
- data/lib/insika/studio/views/skills.erb +53 -19
- data/lib/insika/studio/views/system_files.erb +7 -7
- data/lib/insika/studio/views/task.erb +45 -15
- data/lib/insika/studio/views/tasks.erb +35 -18
- data/lib/insika/studio/views/tools.erb +62 -73
- data/lib/insika/task_actor.rb +4 -1
- data/lib/insika/task_store.rb +67 -1
- data/lib/insika/telemetry/pricing.rb +16 -14
- data/lib/insika/telemetry/recorder.rb +115 -9
- data/lib/insika/telemetry/ruby_llm_instrumenter.rb +110 -0
- data/lib/insika/telemetry.rb +12 -2
- data/lib/insika/templates/daily-digest/README.md +13 -0
- data/lib/insika/templates/daily-digest/agent.rb +22 -11
- data/lib/insika/testing/store_contract.rb +65 -0
- data/lib/insika/text_tool_call.rb +66 -0
- data/lib/insika/tick.rb +7 -1
- data/lib/insika/token_store.rb +5 -0
- data/lib/insika/tool_assembly.rb +21 -13
- data/lib/insika/tool_batch.rb +1 -1
- data/lib/insika/tool_catalog.rb +12 -2
- data/lib/insika/tool_definition.rb +82 -13
- data/lib/insika/tool_envelope.rb +266 -29
- data/lib/insika/tool_manifest.rb +1 -0
- data/lib/insika/tool_store.rb +12 -4
- data/lib/insika/tool_trace_store.rb +15 -8
- data/lib/insika/tool_usage_report.rb +14 -4
- data/lib/insika/tools/a2a_remote.rb +1 -1
- data/lib/insika/tools/agent_enum.rb +1 -1
- data/lib/insika/tools/confirm_pending.rb +112 -0
- data/lib/insika/tools/data_defined_tool.rb +24 -15
- data/lib/insika/tools/generate_image.rb +2 -2
- data/lib/insika/tools/load_knowledge.rb +7 -3
- data/lib/insika/tools/load_skill.rb +3 -3
- data/lib/insika/tools/present.rb +115 -0
- data/lib/insika/tools/remember.rb +7 -2
- data/lib/insika/tools/run_persona_eval.rb +9 -6
- data/lib/insika/tools/save_artifact.rb +16 -6
- data/lib/insika/tools/schedule_followup.rb +3 -3
- data/lib/insika/tools/stuck_signal.rb +3 -3
- data/lib/insika/tools/subagent.rb +5 -5
- data/lib/insika/tools/subagents.rb +3 -3
- data/lib/insika/tools/tool_search.rb +11 -7
- data/lib/insika/tools/tts.rb +2 -2
- data/lib/insika/tools/update_briefing.rb +4 -4
- data/lib/insika/turn_output.rb +2 -2
- data/lib/insika/turn_state.rb +31 -1
- data/lib/insika/turn_timing.rb +51 -14
- data/lib/insika/version.rb +1 -1
- data/lib/insika/wiring/graph.rb +88 -16
- data/lib/insika/wiring/graph_chat.rb +4 -0
- data/lib/insika/workflow.rb +25 -64
- data/lib/insika.rb +23 -3
- metadata +74 -13
- data/lib/insika/commands/import_mcp_tools.rb +0 -48
- data/lib/insika/mcp_http_client.rb +0 -67
- data/lib/insika/mcp_tool_ingestor.rb +0 -165
data/docs/AGENTS.md
CHANGED
|
@@ -132,7 +132,9 @@ block after the agent's identity in the system prompt: a weak or empty tool
|
|
|
132
132
|
result means *try again with a different approach — a rephrased query, a
|
|
133
133
|
synonym, a broader term — before telling the user you found nothing* (and don't
|
|
134
134
|
narrate the retries); a tool error means *read it and fix the arguments*, never
|
|
135
|
-
repeat the exact same call
|
|
135
|
+
repeat the exact same call; and an action is *reported as done only after its
|
|
136
|
+
tool call returned success* — never before the call, never over a failed or
|
|
137
|
+
blocked one. Without it, a search that returns 0 results reads as
|
|
136
138
|
final and the model answers "I couldn't find it" when a synonym one call away
|
|
137
139
|
would have.
|
|
138
140
|
|
|
@@ -144,9 +146,44 @@ the thing worth declaring:
|
|
|
144
146
|
tool_persistence false # remove the block for this agent
|
|
145
147
|
```
|
|
146
148
|
|
|
147
|
-
The block is
|
|
148
|
-
|
|
149
|
-
|
|
149
|
+
The block is constant, so it does not invalidate the identity prefix between turns.
|
|
150
|
+
|
|
151
|
+
### `prompt_caching` — explicit cache control
|
|
152
|
+
|
|
153
|
+
Set `"prompt_caching": true` in the agent payload, or enable it in the Studio
|
|
154
|
+
agent configuration. There is no dedicated DSL method for this flag.
|
|
155
|
+
|
|
156
|
+
For Anthropic, this puts the cache breakpoint after the identity layer and leaves
|
|
157
|
+
volatile system text below it. Other providers receive a plain system string and
|
|
158
|
+
may cache automatically. See [Context](CONTEXT.md#the-provider-prefix-cache) for
|
|
159
|
+
layering, fallback behavior and token accounting.
|
|
160
|
+
|
|
161
|
+
### `fencing` — third-party text is data, never instructions
|
|
162
|
+
|
|
163
|
+
```ruby
|
|
164
|
+
fencing true
|
|
165
|
+
```
|
|
166
|
+
|
|
167
|
+
Off by default. When enabled, the engine sanitizes ordinary tool result string
|
|
168
|
+
leaves after evidence processing, memory facts/notes, briefing values and the next
|
|
169
|
+
step, the compaction summary, and the knowledge names and descriptions injected in
|
|
170
|
+
the context block. It normalizes Unicode (NFKC), removes invisible format characters,
|
|
171
|
+
replaces control characters except tabs/newlines/carriage returns, strips known
|
|
172
|
+
engine/transcript tag shapes and defuses forged role markers. Tool result strings
|
|
173
|
+
are also capped at `fencing.max_chars` (platform default 12,000); memory and knowledge
|
|
174
|
+
context sanitization does not apply that cap. Tool error and halt results bypass
|
|
175
|
+
this sanitizer; tool Hash keys, evidence item `id`s and attachment captions are
|
|
176
|
+
unchanged (an id is a key the ledger holds byte-exact).
|
|
177
|
+
`load_knowledge` returns the full body without fencing.
|
|
178
|
+
|
|
179
|
+
A fixed notice beneath the identity tells the model to treat contextual material
|
|
180
|
+
as data. It ships with the flag — the `context_providers` allowlist does not govern
|
|
181
|
+
it. It is stable across turns; it does not guarantee prompt-injection resistance.
|
|
182
|
+
|
|
183
|
+
Memory/knowledge extraction reads only nonblank user/assistant text regardless
|
|
184
|
+
of the flag. Direct tool messages are excluded, but text repeated by the
|
|
185
|
+
assistant can still reach extraction. `insika doctor` warns for unfenced agents
|
|
186
|
+
exposed by the configured relay/widget environment settings.
|
|
150
187
|
|
|
151
188
|
### Why some limits are missing from that list
|
|
152
189
|
|
|
@@ -173,17 +210,16 @@ So the two groups read differently on purpose:
|
|
|
173
210
|
### `tool_concurrency` — parallel tool calls
|
|
174
211
|
|
|
175
212
|
When the model asks for several tools in one step, they run **one at a time by
|
|
176
|
-
default**. Raise `tool_concurrency`
|
|
177
|
-
in flight:
|
|
213
|
+
default**. Raise `tool_concurrency` to overlap independent reads, capped at that
|
|
214
|
+
number in flight. Tools marked `side_effect` still execute one at a time per session:
|
|
178
215
|
|
|
179
216
|
```ruby
|
|
180
217
|
limit :tool_concurrency, 4 # nil / 0 / 1 = serial (the default); N = at most N at once
|
|
181
218
|
```
|
|
182
219
|
|
|
183
220
|
One number is both the switch and the cap. It pays off only when a turn issues
|
|
184
|
-
several **slow, independent**
|
|
185
|
-
|
|
186
|
-
tools, and it is the *model* that decides the fan-out, which is why the cap is not
|
|
221
|
+
several **slow, independent reads** (data tools waiting on HTTP). It buys nothing
|
|
222
|
+
for fast in-process tools, and it is the *model* that decides the fan-out, which is why the cap is not
|
|
187
223
|
optional: an uncapped batch of 15 data tools is 15 simultaneous requests to the
|
|
188
224
|
same backend.
|
|
189
225
|
|
|
@@ -217,7 +253,7 @@ Which mode you want depends on **when** the message arrives:
|
|
|
217
253
|
| `queue_mode` | The message arrives… | What happens |
|
|
218
254
|
|---|---|---|
|
|
219
255
|
| `followup` (default) | any time | it waits its turn in the queue — today's behavior, named |
|
|
220
|
-
| `collect` | before the turn starts | the fragments merge into ONE turn |
|
|
256
|
+
| `collect` | before the turn starts — including while it waits behind another turn | the fragments merge into ONE turn |
|
|
221
257
|
| `steer` | while the turn is running tools | it is appended to the run in flight |
|
|
222
258
|
| `interrupt` | while a turn is running that is now **wrong** | that turn is abandoned; this message becomes its own turn |
|
|
223
259
|
|
|
@@ -236,6 +272,13 @@ limit :debounce_max_ms, 10_000 # ceiling on the total wait, so typing forever
|
|
|
236
272
|
With those settings the three fragments above become one turn carrying
|
|
237
273
|
`"oi\nqueria saber do pedido\n1234567"`, released 2 s after the last one.
|
|
238
274
|
|
|
275
|
+
**The window is not what opens the door.** A turn is "not started" from the moment
|
|
276
|
+
it is created until it runs, and that includes the whole time it waits behind the
|
|
277
|
+
turn in flight — so with `debounce_ms` at 0 a fragment still merges into a queued
|
|
278
|
+
turn. What the window buys is the case where nothing is running: it holds the turn
|
|
279
|
+
at the front of the queue so fragments have something to land on. Without it a
|
|
280
|
+
burst on an idle session becomes one turn per message, each answered in order.
|
|
281
|
+
|
|
239
282
|
All three follow the platform-layer rule above, with one extra rung on top:
|
|
240
283
|
**session vars → this agent's limits → the platform default (Studio, `queue.*`) →
|
|
241
284
|
off**. Pinning `queue_mode` in a session's vars is how an operator takes one
|
|
@@ -270,11 +313,13 @@ separately lives in one event, emitted when the window closes:
|
|
|
270
313
|
Times and counts, never content. That is what answers "the customer says they
|
|
271
314
|
sent the order number" without keeping a throwaway task per fragment.
|
|
272
315
|
|
|
273
|
-
> **`steer` also collects at the door.** The two
|
|
274
|
-
>
|
|
275
|
-
>
|
|
276
|
-
>
|
|
277
|
-
>
|
|
316
|
+
> **`steer` also collects at the door.** The two are the same policy's halves, not
|
|
317
|
+
> two modes: a `steer` agent merges the fragments that land before the turn starts
|
|
318
|
+
> and appends the ones that land after it — `debounce_ms` only widens the first
|
|
319
|
+
> half. That pairing is what makes a burst answerable at all: whatever the customer
|
|
320
|
+
> sends either joins the run in flight (`steered`) or the turn behind it (`merged`),
|
|
321
|
+
> and the caller hears which one immediately instead of holding a connection open
|
|
322
|
+
> until someone else's turn ends.
|
|
278
323
|
|
|
279
324
|
#### `steer` — the message arrives while the turn is already running
|
|
280
325
|
|
|
@@ -405,9 +450,9 @@ Three capabilities invert the default — `nil`/absent means **OFF**, not "all":
|
|
|
405
450
|
`subagents`, `memory`, and `guardrails` (each defaults to off or a conservative
|
|
406
451
|
setting, never "everything on"). `tool_output_compression` is a fourth: opt-in
|
|
407
452
|
mechanical dedupe of repeated tool results in the history (see
|
|
408
|
-
[Context](CONTEXT.md#
|
|
409
|
-
off by default because it changes what the model sees
|
|
410
|
-
other way: `tool_persistence` is **ON unless you set it to `false`** (see
|
|
453
|
+
[Context](CONTEXT.md#budget-and-eviction--the-actual-compaction)),
|
|
454
|
+
off by default because it changes what the model sees; `fencing` is a fifth (see
|
|
455
|
+
[`fencing`](#fencing--third-party-text-is-data-never-instructions)), off by default. One flag inverts the other way: `tool_persistence` is **ON unless you set it to `false`** (see
|
|
411
456
|
[`tool_persistence`](#tool_persistence--dont-give-up-on-the-first-empty-result)).
|
|
412
457
|
|
|
413
458
|
### Declaring what this deployment has
|
|
@@ -598,6 +643,24 @@ a running agent.
|
|
|
598
643
|
|
|
599
644
|
Photos, voice notes and documents travel through the message contract as additive content parts, and the engine can hand generated images back the same way. See [Media](MEDIA.md).
|
|
600
645
|
|
|
646
|
+
## Caller context next to the message
|
|
647
|
+
|
|
648
|
+
A caller that wraps each message with its own text (channel rules, the products
|
|
649
|
+
the customer was just shown) sends that text in `context`, not in `message`:
|
|
650
|
+
|
|
651
|
+
```bash
|
|
652
|
+
curl -X POST /v1/messages?stream=false -H "Authorization: Bearer $TOKEN" \
|
|
653
|
+
-d '{ "agent": "store-support", "session_id": "chat-7",
|
|
654
|
+
"context": "<channel>sms</channel>", "message": "show me more" }'
|
|
655
|
+
```
|
|
656
|
+
|
|
657
|
+
The model is asked `context`, a blank line, then `message`, and the transcript
|
|
658
|
+
keeps the same text. The input guardrail, retrieval and routing read `message`
|
|
659
|
+
alone, so a product name in `context` never blocks the customer, and the
|
|
660
|
+
customer cannot slip past the guardrail by typing a caller's tags. Optional
|
|
661
|
+
String; absent = the message alone. `POST /v1/responses` takes the same `context`
|
|
662
|
+
next to `input`.
|
|
663
|
+
|
|
601
664
|
## Customer-scoped memory and the right to be forgotten
|
|
602
665
|
|
|
603
666
|
Memory is naturally **per customer, not per tenant**. A message that carries a
|
|
@@ -614,6 +677,10 @@ curl -X POST /v1/messages?stream=false -H "Authorization: Bearer $TOKEN" \
|
|
|
614
677
|
tenant never see each other, and the `<request_context>` tenant label (the
|
|
615
678
|
merchant) is untouched. Absent `customer` = today's per-tenant/per-chat
|
|
616
679
|
behavior.
|
|
680
|
+
- **Display labels** — optional `customer_name` and `customer_phone` (Strings)
|
|
681
|
+
label the conversation in the Studio (the name replaces the `customer` key;
|
|
682
|
+
the phone shows beside the agent) and the Chats search matches both. They
|
|
683
|
+
follow the latest value sent and have no effect on memory scope.
|
|
617
684
|
- **Right to be forgotten** — `POST /v1/commands/forget_customer` (operator)
|
|
618
685
|
purges the customer's memory cell, their sessions and everything those
|
|
619
686
|
sessions left behind — per-session traces, the tasks (the message text lives
|
|
@@ -692,11 +759,68 @@ optionally shared outside the Studio via an expiring signed link
|
|
|
692
759
|
saving agent — never a parameter the model types. Artifact content is LLM
|
|
693
760
|
output and is served as untrusted. See [Artifacts](ARTIFACTS.md).
|
|
694
761
|
|
|
762
|
+
## Review before enabling writes
|
|
763
|
+
|
|
764
|
+
Does each ID come from the server? Declare `evidence` on the lookup tool and
|
|
765
|
+
`requires_evidence` on the write tool so the engine enforces session provenance.
|
|
766
|
+
Mark writes as `side_effect` so parallel batches serialize them and recovery skips
|
|
767
|
+
completed calls. See [Tools](TOOLS.md#provenance-checking-ids-before-a-write).
|
|
768
|
+
|
|
769
|
+
## Do not offer the irreversible action
|
|
770
|
+
|
|
771
|
+
`evidence` and `requires_evidence` stop a write built on an id the customer was never
|
|
772
|
+
shown. They do not stop a write the model decides the customer asked for. That is a
|
|
773
|
+
prompt problem, and the [cross-harness bench](CROSS-HARNESS-BENCH.md) measured how
|
|
774
|
+
large it is.
|
|
775
|
+
|
|
776
|
+
One task: the customer has two units in the cart and says *"adiciona logo por favor"* —
|
|
777
|
+
ambiguous, and the correct reading is a request to add something, not to check out.
|
|
778
|
+
Ten rounds per harness, on the same model:
|
|
779
|
+
|
|
780
|
+
| Harness | Passed | Mentioned closing the order | Called `create_order` |
|
|
781
|
+
| --- | --- | --- | --- |
|
|
782
|
+
| Hermes | 10/10 | 0/10 | 0 |
|
|
783
|
+
| Insika, reasoning off | 9/10 | 10/10 | 1 |
|
|
784
|
+
| Insika, reasoning medium | 7/10 | 10/10 | 3 |
|
|
785
|
+
| Pi | 5/10 | 9/10 | 5 |
|
|
786
|
+
|
|
787
|
+
Every harness that offered to close the order sometimes closed it instead, at rates
|
|
788
|
+
from 10% to 50%. The one that never offered never closed it. It read the ambiguous
|
|
789
|
+
line as a new request and asked *which product* — so checkout was never in the
|
|
790
|
+
sentence the model was continuing.
|
|
791
|
+
|
|
792
|
+
**The rule for a prompt:** when the customer's intent is ambiguous, ask about the
|
|
793
|
+
thing they named, and do not volunteer the irreversible next step in the same breath.
|
|
794
|
+
An action the reply never proposes is one the model cannot slip into performing. This
|
|
795
|
+
is a writing rule, not an engine setting; nothing in the engine needs to change for
|
|
796
|
+
it, and no `halt_when` or allowlist expresses it, because the tool call is legitimate
|
|
797
|
+
in every other turn of the same conversation.
|
|
798
|
+
|
|
799
|
+
**Its cost, before you copy it.** Asking *"which product?"* about an item already in
|
|
800
|
+
the cart ignores context the customer just gave, and reads as an agent that was not
|
|
801
|
+
listening. The bench rewards it because the bench grades the store's state. A shop
|
|
802
|
+
might reasonably prefer the warmer reply and the 10% slip, or split the difference:
|
|
803
|
+
confirm what is in the cart, and stop there — let the customer be the one who says
|
|
804
|
+
close it. What you should not do is confirm the cart and offer checkout in the same
|
|
805
|
+
message, which is the shape that produced every failure above.
|
|
806
|
+
|
|
807
|
+
Reasoning level does not fix this. The same three tasks fail at `medium` and at `off`;
|
|
808
|
+
across ten rounds the two settings contradict each other task by task.
|
|
809
|
+
|
|
810
|
+
**The engine's alternative.** If the shop wants the warmer reply *and* zero slips,
|
|
811
|
+
take the write out of the same turn as the offer: declare the tool under
|
|
812
|
+
[`customer_confirm`](TOOLS.md#customer-confirmation-a-write-the-conversation-approves).
|
|
813
|
+
The model may then propose closing the order as freely as it likes — the engine holds
|
|
814
|
+
the call, the reply asks, and only the customer's next message runs it. The cost moves
|
|
815
|
+
from the wording to the conversation: one extra turn on every legitimate close.
|
|
816
|
+
|
|
695
817
|
## See also
|
|
696
818
|
|
|
697
819
|
- [Tools](TOOLS.md) — define, register, and troubleshoot tools.
|
|
698
820
|
- [Artifacts](ARTIFACTS.md) — the report destination: the tool, the routes, the signed link.
|
|
699
821
|
- [Skills](SKILLS.md) — progressive playbooks an agent loads on demand.
|
|
822
|
+
- [Cross-harness bench](CROSS-HARNESS-BENCH.md) — the same store, six harnesses, and
|
|
823
|
+
where the prompt rule above came from.
|
|
700
824
|
- [Context](CONTEXT.md) — what fills a turn's prompt, and memory.
|
|
701
825
|
- [Security](SECURITY.md) — guardrails, sandbox, approvals, edge limits.
|
|
702
826
|
- [Architecture](ARCHITECTURE.md) — how a turn actually runs.
|
data/docs/API.md
CHANGED
|
@@ -21,6 +21,7 @@ two cannot drift (a spec pins the gate's version to the date below).
|
|
|
21
21
|
| `POST /v1/agents` | provisioning by definition/pack (idempotent, authoritative) |
|
|
22
22
|
| `GET/DELETE /v1/agents` | list / remove agents |
|
|
23
23
|
| `POST /v1/sessions` | create a session explicitly |
|
|
24
|
+
| `POST /v1/conversations/:id/seed` | load an eval snapshot before the first turn; see [Seeding](#seeding-an-eval-conversation) |
|
|
24
25
|
| `POST /v1/outcomes` | record a business outcome (`conversion`, `escalation`, any label) |
|
|
25
26
|
| `GET /v1/outcomes` | last outcome per agent + the per-day series |
|
|
26
27
|
| `POST /v1/commands/...` | operator commands (`forget_customer`, `delete_tenant_data`, the follow-up mutations) |
|
|
@@ -30,6 +31,78 @@ two cannot drift (a spec pins the gate's version to the date below).
|
|
|
30
31
|
The turn endpoints speak the OpenAI `response.completed` wire format; the
|
|
31
32
|
`Insika-Version` header declares the compatibility vintage of a request.
|
|
32
33
|
|
|
34
|
+
## Tool and presentation SSE events
|
|
35
|
+
|
|
36
|
+
`POST /v1/responses` emits a tool name on `response.output_item.added`, with
|
|
37
|
+
`arguments` as a JSON string when available. `response.output_item.done` reports
|
|
38
|
+
how it ended. These `ok` / `error` / `blocked` statuses and optional `gate` are
|
|
39
|
+
Insika extensions, not OpenAI function-call lifecycle statuses.
|
|
40
|
+
|
|
41
|
+
```text
|
|
42
|
+
event: response.output_item.added
|
|
43
|
+
data: {"type":"response.output_item.added","item":{"type":"function_call","name":"add_to_cart","call_id":"toolu_01","arguments":"{\"product_id\":\"SKU-1\"}"}}
|
|
44
|
+
|
|
45
|
+
event: response.output_item.done
|
|
46
|
+
data: {"type":"response.output_item.done","item":{"type":"function_call","name":"add_to_cart","call_id":"toolu_01","status":"blocked","gate":"provenance"}}
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
`added` and `done` are two frames of **one** call, paired by `call_id` (the provider's
|
|
50
|
+
tool-call id, present whenever the provider gave one). A consumer that counts tool
|
|
51
|
+
calls counts `added` frames; `done` only says how each ended.
|
|
52
|
+
|
|
53
|
+
The completion frame does not contain the tool result body. Use stored session
|
|
54
|
+
traces for a per-call audit of what a tool returned.
|
|
55
|
+
|
|
56
|
+
A presentation call also emits a namespaced event:
|
|
57
|
+
|
|
58
|
+
```text
|
|
59
|
+
event: insika.ui
|
|
60
|
+
data: {"type":"insika.ui","component":"product_cards","title":"Selected products","items":[{"type":"card","id":"SKU-1","url":"https://shop.example/products/1","caption":"Dark chocolate"}],"count":1,"dropped":[{"id":"SKU-2","reason":"no_card"}]}
|
|
61
|
+
```
|
|
62
|
+
|
|
63
|
+
`count` is the number shown; `dropped` reasons are `unknown`, `no_card` or `max`.
|
|
64
|
+
An empty selection still emits the event with `count: 0`. Clients that render
|
|
65
|
+
cards consume this event; text-only clients can ignore it. See
|
|
66
|
+
[Presentation tools](TOOLS.md#presentation-tools-the-model-picks-ids-the-engine-shows-the-cards).
|
|
67
|
+
|
|
68
|
+
## Seeding an eval conversation
|
|
69
|
+
|
|
70
|
+
Enable `evals.seeding` in platform settings on the eval deployment. For example,
|
|
71
|
+
an operator can `POST /v1/commands/update_settings` with
|
|
72
|
+
`{"patch":{"evals":{"seeding":true}}}`. Turn it off after the run.
|
|
73
|
+
|
|
74
|
+
`POST /v1/conversations/:id/seed` uses the same Bearer and tenant ID namespacing
|
|
75
|
+
as a turn. Its body is the snapshot itself, plus optional `customer`; do not wrap
|
|
76
|
+
it in `state`:
|
|
77
|
+
|
|
78
|
+
```json
|
|
79
|
+
{
|
|
80
|
+
"customer": "eval-customer-1",
|
|
81
|
+
"evidence": {
|
|
82
|
+
"ids": ["SKU-1"],
|
|
83
|
+
"cards": [{ "type": "card", "id": "SKU-1", "url": "https://shop.example/1", "caption": "Dark chocolate" }]
|
|
84
|
+
},
|
|
85
|
+
"history": [
|
|
86
|
+
{ "role": "assistant", "content": "The dark chocolate is SKU-1." }
|
|
87
|
+
],
|
|
88
|
+
"memory": { "facts": { "preference": "dark chocolate" }, "notes": [] },
|
|
89
|
+
"briefing": { "fields": { "delivery_day": "Friday" } }
|
|
90
|
+
}
|
|
91
|
+
```
|
|
92
|
+
|
|
93
|
+
It creates a missing session and returns `200` with `{ "session": … }`; seeding
|
|
94
|
+
does not run a model or create a task. Disabled seeding returns `403`, a session
|
|
95
|
+
with messages returns `409`, and invalid supported field shapes return `422`.
|
|
96
|
+
History accepts only `user` / `assistant` roles and is stamped `origin: engine`.
|
|
97
|
+
|
|
98
|
+
Memory uses `[tenant:]customer` when `customer` is supplied; subsequent turns
|
|
99
|
+
must carry the same customer. Without one, memory is written to the tenant cell,
|
|
100
|
+
or `chat:<session-id>` when there is no tenant. Use an isolated eval deployment:
|
|
101
|
+
a fresh conversation ID alone does not isolate tenant memory. Evidence IDs seed
|
|
102
|
+
only the runtime ledger, never model-visible text; `evidence.cards` seeds the cards
|
|
103
|
+
a presentation tool can show.
|
|
104
|
+
See [Snapshot evals](EVALS.md#state--a-case-starts-from-a-snapshot).
|
|
105
|
+
|
|
33
106
|
## The compatibility rule
|
|
34
107
|
|
|
35
108
|
**Request and response shapes are stable; changes are additive-only.** A new
|
data/docs/ARCHITECTURE.md
CHANGED
|
@@ -167,9 +167,11 @@ The order is not arbitrary:
|
|
|
167
167
|
- **The initial checkpoint is written before the model call.** "The checkpoint of
|
|
168
168
|
turn *n* holds the state at the *start* of turn *n*." Without it, a crash during
|
|
169
169
|
the model call would orphan the task with no checkpoint — unrecoverable.
|
|
170
|
-
- **Middleware wraps the
|
|
171
|
-
|
|
172
|
-
|
|
170
|
+
- **Middleware wraps the chat stages.** For opt-in reranking, the leading edge
|
|
171
|
+
limiter also wraps context preparation: admission runs once before paid retrieval.
|
|
172
|
+
Context still precedes policy; the input guardrail and plugin middleware receive
|
|
173
|
+
the prepared context, including any budget warning, before chat runs. Turns with
|
|
174
|
+
lexical-only retrieval keep the original order.
|
|
173
175
|
- **Persistence is a fixed order** (checkpoint → session → task) and a pure drain
|
|
174
176
|
point: the last stage never suspends, so a checkpoint is never left half-written.
|
|
175
177
|
- **Output validation** runs as an after-task hook on the produced content
|
|
@@ -177,64 +179,65 @@ The order is not arbitrary:
|
|
|
177
179
|
|
|
178
180
|
## The tool-loop
|
|
179
181
|
|
|
180
|
-
Stage 6 is the
|
|
181
|
-
|
|
182
|
-
|
|
183
|
-
already-completed side-effects on resume, and fires the approval gate.
|
|
182
|
+
Stage 6 is the agent interaction. RubyLLM owns the reason→act→observe loop;
|
|
183
|
+
`ToolEnvelope` adds provenance, approval, concurrency, timeout, evidence processing,
|
|
184
|
+
fencing, traces and side-effect checkpoints around registered tools.
|
|
184
185
|
|
|
185
|
-
|
|
186
|
-
|
|
187
|
-
|
|
188
|
-
|
|
189
|
-
|
|
186
|
+
Calls run serially unless `limits[:tool_concurrency]` permits parallel execution.
|
|
187
|
+
Marked side effects acquire a serial gate before the shared concurrency slot,
|
|
188
|
+
so writes in one session cannot overlap and queued writes leave slots for reads.
|
|
189
|
+
Turns exposing approval-required tools run serially. See
|
|
190
|
+
[Tools](TOOLS.md#parallel-tool-calls) for limits and timeout behavior.
|
|
190
191
|
|
|
191
192
|
```mermaid
|
|
192
193
|
flowchart TD
|
|
193
|
-
ask[chat.ask
|
|
194
|
+
ask[chat.ask → model] --> dec{tool call?}
|
|
194
195
|
dec -->|no| done[final content]
|
|
195
|
-
dec -->|yes| env[ToolEnvelope]
|
|
196
|
-
env -->
|
|
197
|
-
|
|
198
|
-
|
|
199
|
-
|
|
200
|
-
|
|
201
|
-
|
|
202
|
-
|
|
203
|
-
http
|
|
204
|
-
|
|
196
|
+
dec -->|yes| env[ToolEnvelope: skip completed side effects on resume]
|
|
197
|
+
env --> prov{declared evidence IDs known?}
|
|
198
|
+
prov -->|no| blocked[blocked: provenance]
|
|
199
|
+
blocked --> ask
|
|
200
|
+
prov -->|yes / undeclared| appr[approval gate]
|
|
201
|
+
appr --> gates[side-effect serial gate → shared concurrency slot → timeout]
|
|
202
|
+
gates --> kind{tool kind}
|
|
203
|
+
kind -->|code| ruby[Ruby / sandbox]
|
|
204
|
+
kind -->|HTTP data| http[EgressGuard → HTTP]
|
|
205
|
+
kind -->|MCP| mcp[live MCP client]
|
|
206
|
+
kind -->|presentation| cards[select turn or session evidence cards → UI event]
|
|
207
|
+
ruby --> result[evidence reshape → fencing → checkpoint and trace]
|
|
208
|
+
http --> result
|
|
209
|
+
mcp --> result
|
|
210
|
+
cards --> result
|
|
211
|
+
result --> ask
|
|
205
212
|
```
|
|
206
213
|
|
|
207
|
-
|
|
208
|
-
|
|
209
|
-
|
|
210
|
-
|
|
211
|
-
|
|
214
|
+
HTTP data tools and presentation tools are both stored definitions. Presentation
|
|
215
|
+
runs in-process; MCP remains a live server call (Streamable HTTP or stdio). HTTP egress
|
|
216
|
+
is checked before a request. Tool errors reach the model as error results so it
|
|
217
|
+
can recover; a provenance refusal reports `blocked` without contacting the backend
|
|
218
|
+
or asking an operator. Completed side effects are recorded for resume.
|
|
212
219
|
|
|
213
220
|
## Ingesting tools: manifest and MCP
|
|
214
221
|
|
|
215
|
-
|
|
222
|
+
Data definitions and MCP instances have separate stores. Both are hot config:
|
|
216
223
|
|
|
217
224
|
```mermaid
|
|
218
225
|
flowchart TD
|
|
219
|
-
|
|
220
|
-
|
|
221
|
-
|
|
222
|
-
|
|
223
|
-
|
|
224
|
-
|
|
225
|
-
|
|
226
|
-
|
|
227
|
-
|
|
228
|
-
val1 --> store[(ToolStore)]
|
|
229
|
-
val2 --> store
|
|
230
|
-
store --> cat[reload catalog + registry]
|
|
231
|
-
cat --> loop[available in the tool-loop]
|
|
226
|
+
manifest[POST /v1/tools/manifest] --> substitute[resolve env / secret placeholders]
|
|
227
|
+
substitute --> validate[validate HTTP or presentation definition]
|
|
228
|
+
validate --> store[(ToolStore)]
|
|
229
|
+
store --> data[DataToolRegistry]
|
|
230
|
+
config[MCP instance configuration] --> mcpstore[(McpStore)]
|
|
231
|
+
mcpstore --> live[McpToolRegistry → live server tools]
|
|
232
|
+
data --> catalog[effective registry and catalog]
|
|
233
|
+
live --> catalog
|
|
234
|
+
catalog --> policy[agent allowlist → tool-loop]
|
|
232
235
|
```
|
|
233
236
|
|
|
234
|
-
|
|
235
|
-
|
|
236
|
-
|
|
237
|
-
|
|
237
|
+
Only manifest ingestion resolves `{{env.*}}` and `{{secret.*}}`; other data-tool
|
|
238
|
+
write paths require literal values. One malformed manifest tool is reported in
|
|
239
|
+
`errors[]` while valid entries import. MCP tools are never converted to stored
|
|
240
|
+
HTTP data tools. See [Tools](TOOLS.md#registering-a-tool).
|
|
238
241
|
|
|
239
242
|
## Durability: checkpoints and resume
|
|
240
243
|
|
|
@@ -255,9 +258,12 @@ flowchart LR
|
|
|
255
258
|
completed --> done((( )))
|
|
256
259
|
```
|
|
257
260
|
|
|
258
|
-
Resume always replays from the *start of the last checkpointed turn*.
|
|
259
|
-
|
|
260
|
-
|
|
261
|
+
Resume always replays from the *start of the last checkpointed turn*. Side-effect
|
|
262
|
+
calls whose completion was recorded in the checkpoint are skipped when replayed
|
|
263
|
+
with the same call id. An external effect can succeed before that record is saved;
|
|
264
|
+
a crash or failed checkpoint write in that window can repeat the effect on resume.
|
|
265
|
+
Use destination-supported idempotency keys or reconciliation for those operations:
|
|
266
|
+
checkpointing alone does not provide exactly-once external execution. A resumed
|
|
261
267
|
turn is also never re-counted against edge-limit ledgers. Cancellation is
|
|
262
268
|
cooperative — checked at stage boundaries, never in the middle of a store write.
|
|
263
269
|
|
|
@@ -316,7 +322,7 @@ validator as the after-task hook, so both roots enforce content safety identical
|
|
|
316
322
|
| Recovery | `lib/insika/recovery.rb` |
|
|
317
323
|
| Inbound queue (one turn at a time per session, and what happens to a message that arrives while one is running) | `lib/insika/session_actor.rb`, `lib/insika/queue_policy.rb`, `lib/insika/steer_injector.rb` |
|
|
318
324
|
| Channels (a way in and out for people; the reply that travels after the turn ends) | `lib/insika/channel_registry.rb`, `lib/insika/channels/*`, `lib/insika/channel_delivery.rb`, `lib/insika/outbox_store.rb`, `lib/insika/inbound_log.rb` |
|
|
319
|
-
| Tools (data/manifest/MCP) | `lib/insika/tool_definition.rb`, `tool_manifest.rb`, `
|
|
325
|
+
| Tools (data/manifest/MCP) | `lib/insika/tool_definition.rb`, `lib/insika/tool_manifest.rb`, `lib/insika/mcp_tool_registry.rb`, `lib/insika/tools/present.rb` |
|
|
320
326
|
| Plugin loading (boot) | `lib/insika/plugin.rb`, `lib/insika/plugin/loader.rb` |
|
|
321
327
|
| Refinement (traffic → report) | `lib/insika/refinement/*`, `lib/insika/refinement_store.rb` |
|
|
322
328
|
| Post-turn learning (facts, skills, knowledge — extracted from finished conversations) | `lib/insika/distill.rb`, `lib/insika/harvest.rb`, `lib/insika/knowledge.rb`, `lib/insika/knowledge_store.rb`; the per-turn hook lives in `Executor#persist_turn`, next to `finalize_delegation` |
|
data/docs/ARTIFACTS.md
CHANGED
|
@@ -63,9 +63,39 @@ Content-Security-Policy: default-src 'none'; style-src 'unsafe-inline'; img-src
|
|
|
63
63
|
X-Content-Type-Options: nosniff
|
|
64
64
|
```
|
|
65
65
|
|
|
66
|
-
No script, no external fetch, no forms. The model writes HTML
|
|
67
|
-
SVG**
|
|
68
|
-
|
|
66
|
+
No script, no external fetch, no forms. The model writes self-contained HTML/CSS
|
|
67
|
+
with **inline SVG** or HTML bars for charts. Native radio controls and
|
|
68
|
+
`details`/`summary` still work inside this boundary.
|
|
69
|
+
|
|
70
|
+
## Interactive executive reports
|
|
71
|
+
|
|
72
|
+
The tool description and `daily-digest` skill guide future HTML reports. Existing
|
|
73
|
+
artifacts keep their saved content. The
|
|
74
|
+
[example report](https://github.com/guizaols/insika/blob/main/examples/scheduled-report/interactive-report.html)
|
|
75
|
+
uses explicitly fictional data and works without JavaScript or dependencies.
|
|
76
|
+
|
|
77
|
+
- Lead with the title, reporting period, generation date and a concise takeaway
|
|
78
|
+
supported by the numbers. Include sources and limitations; mark missing data
|
|
79
|
+
and fictional examples honestly.
|
|
80
|
+
- Use warm neutral surfaces and one amber or merchant accent. Keep typography,
|
|
81
|
+
spacing and KPI cards consistent; support system dark mode and narrow screens.
|
|
82
|
+
- Label charts with units and exact values. Use responsive SVG or HTML bars and a
|
|
83
|
+
captioned table with column headers, so touch users can read every value.
|
|
84
|
+
- Switch Chart/Table with labeled, focusable native radios in a `fieldset` with
|
|
85
|
+
a `legend`, styled through CSS `:checked`. These are radios, not ARIA tabs.
|
|
86
|
+
- Use CSS hover/focus tooltips in reserved space, highlights, a visible 2px focus
|
|
87
|
+
outline, and `details`/`summary` for supporting evidence. Keep tooltips visible
|
|
88
|
+
when the pointer moves onto them. Do not make hover the only way to
|
|
89
|
+
read data or use `<title>` as the only tooltip.
|
|
90
|
+
- Limit color/opacity transitions to 140ms and disable them under
|
|
91
|
+
`prefers-reduced-motion: reduce`. Avoid decorative animation. Print styles
|
|
92
|
+
include the complete table and details regardless of selected/collapsed state,
|
|
93
|
+
hide controls and keep text legible; use the browser's print command.
|
|
94
|
+
|
|
95
|
+
Interactivity explores a **dated snapshot**. It does not fetch new data, recompute
|
|
96
|
+
the source or imply a live feed. A later report run saves a new artifact. Do not
|
|
97
|
+
add scripts, event handlers (including `onclick` print buttons), external resources
|
|
98
|
+
or form submissions to generated reports.
|
|
69
99
|
|
|
70
100
|
## Reasoning effort on a report turn
|
|
71
101
|
|
data/docs/BENCHMARK.md
CHANGED
|
@@ -45,6 +45,7 @@ reports only the engine's contribution:
|
|
|
45
45
|
| **gen** (p50/p95) | streaming the rest through the pipeline (filter/emit/event stream) |
|
|
46
46
|
| **throughput** | turns/s a single process sustains at a given concurrency |
|
|
47
47
|
| **pipeline overhead** | engine work per streamed token (µs) |
|
|
48
|
+
| **allocations** | Ruby objects allocated per measured turn, including benchmark driver bookkeeping, excluding warmup |
|
|
48
49
|
|
|
49
50
|
**Out of scope, on purpose:** end-to-end latency, time-to-first-token *against a
|
|
50
51
|
provider*, and tokens/s of *model generation*. Those are provider-bound — the
|
|
@@ -52,8 +53,9 @@ Insika has no lever on them — so this suite makes no claim about them.
|
|
|
52
53
|
|
|
53
54
|
The stub implements exactly the chat surface the executor touches, and each turn
|
|
54
55
|
runs the full engine path: context build, policy resolution, guardrail
|
|
55
|
-
detectors, chat assembly,
|
|
56
|
-
checkpointing, and the event stream.
|
|
56
|
+
detectors, chat assembly, tool-call callbacks, streamed output, persistence,
|
|
57
|
+
checkpointing, and the event stream. The default mode replaces RubyLLM's Chat;
|
|
58
|
+
use `--ruby-llm` below to include the real Chat and tool execution.
|
|
57
59
|
|
|
58
60
|
## Scenarios
|
|
59
61
|
|
|
@@ -62,8 +64,8 @@ imported the same way any pack is, so the measured path is the real one.
|
|
|
62
64
|
|
|
63
65
|
- **greeting** — a minimal turn: a short system prompt, no tools. Baseline
|
|
64
66
|
engine overhead.
|
|
65
|
-
- **tool_call** — a turn
|
|
66
|
-
|
|
67
|
+
- **tool_call** — a turn simulating one tool call; exercises tool assembly and
|
|
68
|
+
call/result callbacks without executing the tool.
|
|
67
69
|
- **multi_turn** — a one-shot carrying prior conversation messages; shows how
|
|
68
70
|
overhead moves as the context the engine assembles grows.
|
|
69
71
|
|
|
@@ -80,12 +82,46 @@ imported the same way any pack is, so the measured path is the real one.
|
|
|
80
82
|
| `--history-turns N` | `10` | prior messages for the `multi_turn` scenario |
|
|
81
83
|
| `--output-tokens N` | `48` | tokens the stub streams per turn |
|
|
82
84
|
| `--json` | off | emit results as JSON (for regression gating) |
|
|
85
|
+
| `--ruby-llm` | off | run the real RubyLLM Chat with an offline provider |
|
|
83
86
|
|
|
84
87
|
`--json` prints the engine version, Ruby/YJIT status, the full config, and every
|
|
85
88
|
metric — a stable shape to diff across commits.
|
|
86
89
|
|
|
90
|
+
## Offline RubyLLM benchmark
|
|
91
|
+
|
|
92
|
+
```bash
|
|
93
|
+
bundle exec ruby scripts/bench.rb --ruby-llm --json
|
|
94
|
+
```
|
|
95
|
+
|
|
96
|
+
This opt-in mode measures the engine **plus RubyLLM's real Chat and tool loop**.
|
|
97
|
+
Only the provider is synthetic: it yields deterministic chunks without HTTP,
|
|
98
|
+
credentials, or a model service. The normal executor creates the Chat, assembles
|
|
99
|
+
its history and tools, and registers the production callbacks. The `tool_call`
|
|
100
|
+
scenario executes two sequential lookup rounds; `greeting` streams an answer and
|
|
101
|
+
`multi_turn` seeds the requested history before streaming.
|
|
102
|
+
|
|
103
|
+
JSON includes the RubyLLM version, `mode`, and per-scenario `ruby_llm_activity`
|
|
104
|
+
counts for provider calls, executed tools, history messages, and streamed chunks.
|
|
105
|
+
Activity counts include warmup; allocation measurements exclude warmup and cover
|
|
106
|
+
both measured passes. Compare matching modes and settings across versions; the
|
|
107
|
+
default stub mode keeps its existing one-tool scenario. Neither mode measures
|
|
108
|
+
HTTP serialization, network latency, or actual model generation.
|
|
109
|
+
|
|
110
|
+
The offline provider supplies request `model_info`, as native RubyLLM 2 usage
|
|
111
|
+
tracking does. Omitting it adds spurious catalog lookups during cost accounting.
|
|
112
|
+
For an isolated, self-checking reproduction without Insika, run
|
|
113
|
+
`ruby scripts/rubyllm_profile.rb 2.0.0 --profile` and repeat with
|
|
114
|
+
`--request-context`. Use `1.16.0` for the old-gem comparison, outside Bundler.
|
|
115
|
+
Method timings are inclusive and must not be summed; use runs without `--profile`
|
|
116
|
+
for latency comparisons. This diagnostic still omits native HTTP processing.
|
|
117
|
+
|
|
87
118
|
## Reference numbers
|
|
88
119
|
|
|
120
|
+
For the five-run main versus RubyLLM 2 migration comparison repeated on
|
|
121
|
+
2026-09-24, see [current candidate results](RUBYLLM_2_MIGRATION.md#current-candidate-2026-09-24).
|
|
122
|
+
It uses this harness, plus the API-equivalent baseline harness, without YJIT.
|
|
123
|
+
The older YJIT reference below is not a migration acceptance result.
|
|
124
|
+
|
|
89
125
|
A reference run. **The absolute milliseconds are machine-specific** — reproduce
|
|
90
126
|
them on your own hardware with the command below; what travels across machines is
|
|
91
127
|
the shape (sub-millisecond overhead, flat p95, thousands of turns/s per process).
|
|
@@ -109,6 +145,13 @@ work. The rest of any real turn's latency is the provider.
|
|
|
109
145
|
## Publication rule
|
|
110
146
|
|
|
111
147
|
**Any public claim about the engine's performance must reference this suite.**
|
|
112
|
-
|
|
113
|
-
publishable — they are neither neutral nor reproducible. If a
|
|
114
|
-
reproduced by running `scripts/bench.rb`, it does not go in public
|
|
148
|
+
Performance numbers produced against a specific provider, deployment, or competitor
|
|
149
|
+
are not publishable — they are neither neutral nor reproducible. If a performance
|
|
150
|
+
claim cannot be reproduced by running `scripts/bench.rb`, it does not go in public
|
|
151
|
+
materials.
|
|
152
|
+
|
|
153
|
+
The rule is about **performance**, which is what this suite measures. Claims about
|
|
154
|
+
what an agent *gets right* live in a different suite with a different contract: the
|
|
155
|
+
[cross-harness bench](CROSS-HARNESS-BENCH.md) needs a provider key, because a
|
|
156
|
+
commerce conversation cannot be stubbed, and it names other harnesses. It carries its
|
|
157
|
+
own publication rule, which is stricter about the rows it did not write.
|