insika 0.9.0 → 1.0.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (257) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGELOG.md +237 -2
  3. data/README.md +5 -3
  4. data/bin/insika +3 -0
  5. data/bin/insika-router +0 -1
  6. data/docs/AGENTS.md +90 -7
  7. data/docs/ARCHITECTURE.md +12 -7
  8. data/docs/ARTIFACTS.md +33 -3
  9. data/docs/BENCHMARK.md +50 -7
  10. data/docs/CONTEXT.md +63 -0
  11. data/docs/CROSS-HARNESS-BENCH.md +157 -0
  12. data/docs/DEPLOY.md +57 -11
  13. data/docs/EMBEDDING.md +10 -0
  14. data/docs/EVALS.md +33 -4
  15. data/docs/FACTS.md +8 -6
  16. data/docs/HARVEST.md +8 -3
  17. data/docs/KNOWLEDGE.md +7 -0
  18. data/docs/MEDIA.md +6 -2
  19. data/docs/OBSERVABILITY.md +143 -22
  20. data/docs/PERFORMANCE.md +110 -0
  21. data/docs/PLUGINS.md +4 -4
  22. data/docs/POLICY.md +22 -13
  23. data/docs/RELEASING.md +5 -0
  24. data/docs/ROUTER.md +7 -0
  25. data/docs/RUBYLLM_2_MIGRATION.md +307 -0
  26. data/docs/SECURITY.md +7 -0
  27. data/docs/SHARED-CONVERSATIONS.md +40 -0
  28. data/docs/SKILLS.md +5 -3
  29. data/docs/STUDIO.md +122 -0
  30. data/docs/TOOLS.md +111 -9
  31. data/docs/operate.md +3 -0
  32. data/lib/insika/agent_defaults.rb +140 -0
  33. data/lib/insika/agent_file_store.rb +12 -0
  34. data/lib/insika/agent_profile.rb +78 -8
  35. data/lib/insika/artifact_signing.rb +11 -0
  36. data/lib/insika/chat_builder.rb +79 -25
  37. data/lib/insika/checkpoint.rb +6 -1
  38. data/lib/insika/checkpoint_store.rb +59 -7
  39. data/lib/insika/commands/agent_payload.rb +3 -3
  40. data/lib/insika/commands/delete_kit.rb +26 -0
  41. data/lib/insika/commands/gate_harvest.rb +3 -0
  42. data/lib/insika/commands/review_guardrail_flag.rb +38 -0
  43. data/lib/insika/commands/run_harvest.rb +17 -1
  44. data/lib/insika/commands/seed_session.rb +11 -1
  45. data/lib/insika/commands/send_message.rb +9 -3
  46. data/lib/insika/commands/update_settings.rb +23 -1
  47. data/lib/insika/commands/write_kit.rb +27 -0
  48. data/lib/insika/compaction.rb +33 -7
  49. data/lib/insika/config_store.rb +87 -7
  50. data/lib/insika/context/priority.rb +3 -0
  51. data/lib/insika/context/provider.rb +2 -2
  52. data/lib/insika/context/providers/knowledge.rb +25 -2
  53. data/lib/insika/context/providers/memory.rb +53 -4
  54. data/lib/insika/context/providers/pending_confirmation.rb +52 -0
  55. data/lib/insika/context/providers/prompt.rb +17 -4
  56. data/lib/insika/context/providers/session.rb +39 -3
  57. data/lib/insika/context/reranker.rb +75 -0
  58. data/lib/insika/distill.rb +22 -13
  59. data/lib/insika/doctor.rb +34 -4
  60. data/lib/insika/dsl/runtime.rb +36 -4
  61. data/lib/insika/dsl/server_boot.rb +4 -12
  62. data/lib/insika/dsl.rb +13 -5
  63. data/lib/insika/edge_limiter.rb +23 -15
  64. data/lib/insika/egress_guard.rb +11 -3
  65. data/lib/insika/env_schema.rb +33 -73
  66. data/lib/insika/evals/assertions.rb +171 -16
  67. data/lib/insika/evals/golden.rb +141 -4
  68. data/lib/insika/evals/judge.rb +19 -10
  69. data/lib/insika/evals/moderator_compare.rb +47 -0
  70. data/lib/insika/evals/report.rb +61 -9
  71. data/lib/insika/evals/runner.rb +44 -4
  72. data/lib/insika/evals/simulator.rb +10 -1
  73. data/lib/insika/evals/transport.rb +200 -0
  74. data/lib/insika/event_stream.rb +63 -7
  75. data/lib/insika/evidence.rb +29 -8
  76. data/lib/insika/executor.rb +668 -141
  77. data/lib/insika/harvest.rb +26 -14
  78. data/lib/insika/harvest_engine.rb +1 -0
  79. data/lib/insika/harvest_store.rb +12 -0
  80. data/lib/insika/http_client.rb +87 -4
  81. data/lib/insika/kits.rb +54 -0
  82. data/lib/insika/knowledge.rb +34 -20
  83. data/lib/insika/llm_configurator.rb +46 -4
  84. data/lib/insika/llm_provider_store.rb +1 -3
  85. data/lib/insika/llm_trace_store.rb +54 -0
  86. data/lib/insika/mcp_client.rb +27 -63
  87. data/lib/insika/mcp_live_tool.rb +55 -9
  88. data/lib/insika/mcp_store.rb +32 -1
  89. data/lib/insika/mcp_tool_registry.rb +69 -26
  90. data/lib/insika/media.rb +38 -25
  91. data/lib/insika/memory_store.rb +4 -3
  92. data/lib/insika/memory_trim.rb +60 -0
  93. data/lib/insika/middleware.rb +12 -3
  94. data/lib/insika/model_metrics_store.rb +285 -0
  95. data/lib/insika/model_registry.rb +110 -0
  96. data/lib/insika/model_resolver.rb +1 -1
  97. data/lib/insika/model_selection.rb +14 -21
  98. data/lib/insika/model_visible.rb +17 -6
  99. data/lib/insika/onboarding.rb +1 -0
  100. data/lib/insika/outcome_store.rb +1 -1
  101. data/lib/insika/overlay_tool_registry.rb +25 -2
  102. data/lib/insika/pack_importer.rb +54 -9
  103. data/lib/insika/packaging.rb +14 -5
  104. data/lib/insika/pending_action_store.rb +45 -18
  105. data/lib/insika/plugin/loader.rb +4 -2
  106. data/lib/insika/profile_source.rb +4 -0
  107. data/lib/insika/provider_error_classifier.rb +18 -3
  108. data/lib/insika/queue_policy.rb +7 -2
  109. data/lib/insika/refinement/proposer.rb +42 -17
  110. data/lib/insika/reliability.rb +12 -3
  111. data/lib/insika/router/app.rb +14 -5
  112. data/lib/insika/router/session_key.rb +35 -5
  113. data/lib/insika/safety/config.rb +20 -3
  114. data/lib/insika/safety/corpus.rb +10 -3
  115. data/lib/insika/safety/factory.rb +39 -1
  116. data/lib/insika/safety/input_guardrail.rb +21 -1
  117. data/lib/insika/safety/judge_moderator.rb +70 -0
  118. data/lib/insika/safety/output_validator.rb +29 -5
  119. data/lib/insika/safety/prompt_echo.rb +89 -0
  120. data/lib/insika/schema_guard.rb +53 -76
  121. data/lib/insika/server/app.rb +17 -8
  122. data/lib/insika/server/responses.rb +19 -1
  123. data/lib/insika/session_actor.rb +58 -28
  124. data/lib/insika/session_store.rb +159 -10
  125. data/lib/insika/settings_store.rb +47 -1
  126. data/lib/insika/shared_conversations.rb +305 -0
  127. data/lib/insika/skill_catalog.rb +28 -2
  128. data/lib/insika/skill_store.rb +6 -0
  129. data/lib/insika/soak/runner.rb +1 -1
  130. data/lib/insika/steer_injector.rb +7 -48
  131. data/lib/insika/store.rb +29 -1
  132. data/lib/insika/stores/json_value.rb +37 -0
  133. data/lib/insika/stores/memory.rb +22 -35
  134. data/lib/insika/stores/postgres.rb +295 -0
  135. data/lib/insika/stores/sqlite.rb +257 -43
  136. data/lib/insika/stores/turn_counter.rb +25 -0
  137. data/lib/insika/studio/app.rb +1453 -161
  138. data/lib/insika/studio/assets/dist/application.css +1 -1
  139. data/lib/insika/studio/assets/dist/application.js +25 -25
  140. data/lib/insika/studio/assets/dist/fonts/ibm-plex-mono-latin-400-normal.woff2 +0 -0
  141. data/lib/insika/studio/assets/dist/fonts/ibm-plex-mono-latin-500-normal.woff2 +0 -0
  142. data/lib/insika/studio/assets/dist/fonts/ibm-plex-mono-latin-600-normal.woff2 +0 -0
  143. data/lib/insika/studio/assets/dist/fonts/ibm-plex-sans-latin-wght-normal.woff2 +0 -0
  144. data/lib/insika/studio/forms.rb +171 -9
  145. data/lib/insika/studio/nav_icons.rb +3 -0
  146. data/lib/insika/studio/views/_agent_config_retrieval.erb +73 -0
  147. data/lib/insika/studio/views/_agent_picker.erb +15 -0
  148. data/lib/insika/studio/views/_agent_tab_config.erb +78 -242
  149. data/lib/insika/studio/views/_agent_tab_history.erb +56 -1
  150. data/lib/insika/studio/views/_agent_tab_performance.erb +34 -0
  151. data/lib/insika/studio/views/_agent_tab_prompts.erb +7 -9
  152. data/lib/insika/studio/views/_agent_tab_skills.erb +16 -7
  153. data/lib/insika/studio/views/_agent_tab_tools.erb +90 -0
  154. data/lib/insika/studio/views/_agents_master.erb +8 -12
  155. data/lib/insika/studio/views/_cfg_alerts.erb +4 -0
  156. data/lib/insika/studio/views/_cfg_budget.erb +20 -0
  157. data/lib/insika/studio/views/_cfg_distill.erb +26 -0
  158. data/lib/insika/studio/views/_cfg_followup.erb +30 -0
  159. data/lib/insika/studio/views/_cfg_grounding.erb +21 -0
  160. data/lib/insika/studio/views/_cfg_guardrails.erb +47 -0
  161. data/lib/insika/studio/views/_cfg_harvest.erb +35 -0
  162. data/lib/insika/studio/views/_cfg_reliability.erb +32 -0
  163. data/lib/insika/studio/views/_chats_master.erb +66 -0
  164. data/lib/insika/studio/views/_customers_master.erb +37 -0
  165. data/lib/insika/studio/views/_empty.erb +15 -0
  166. data/lib/insika/studio/views/_facts_master.erb +42 -0
  167. data/lib/insika/studio/views/_harvest_candidate_detail.erb +14 -0
  168. data/lib/insika/studio/views/_identity.erb +16 -0
  169. data/lib/insika/studio/views/_kpi.erb +17 -0
  170. data/lib/insika/studio/views/_message.erb +40 -2
  171. data/lib/insika/studio/views/_model_chart.erb +82 -0
  172. data/lib/insika/studio/views/_refinement_master.erb +38 -0
  173. data/lib/insika/studio/views/{tool_edit.erb → _tool_edit.erb} +32 -9
  174. data/lib/insika/studio/views/_tool_users.erb +17 -0
  175. data/lib/insika/studio/views/_why.erb +53 -0
  176. data/lib/insika/studio/views/agent_detail.erb +17 -10
  177. data/lib/insika/studio/views/agents.erb +28 -16
  178. data/lib/insika/studio/views/approvals.erb +16 -11
  179. data/lib/insika/studio/views/artifact.erb +20 -7
  180. data/lib/insika/studio/views/artifacts.erb +19 -14
  181. data/lib/insika/studio/views/chats.erb +22 -29
  182. data/lib/insika/studio/views/customer.erb +88 -43
  183. data/lib/insika/studio/views/customers.erb +16 -26
  184. data/lib/insika/studio/views/evals.erb +49 -31
  185. data/lib/insika/studio/views/fact_detail.erb +114 -0
  186. data/lib/insika/studio/views/facts.erb +15 -116
  187. data/lib/insika/studio/views/followups.erb +63 -52
  188. data/lib/insika/studio/views/funnel.erb +104 -43
  189. data/lib/insika/studio/views/harvest.erb +85 -59
  190. data/lib/insika/studio/views/home.erb +138 -23
  191. data/lib/insika/studio/views/inbox.erb +36 -0
  192. data/lib/insika/studio/views/kits.erb +176 -0
  193. data/lib/insika/studio/views/knowledge.erb +92 -80
  194. data/lib/insika/studio/views/layout.erb +42 -11
  195. data/lib/insika/studio/views/live.erb +45 -0
  196. data/lib/insika/studio/views/mcp.erb +43 -13
  197. data/lib/insika/studio/views/models.erb +187 -0
  198. data/lib/insika/studio/views/parity.erb +38 -32
  199. data/lib/insika/studio/views/playground.erb +13 -9
  200. data/lib/insika/studio/views/refinement.erb +15 -224
  201. data/lib/insika/studio/views/refinement_detail.erb +243 -0
  202. data/lib/insika/studio/views/session.erb +152 -117
  203. data/lib/insika/studio/views/settings.erb +86 -22
  204. data/lib/insika/studio/views/skills.erb +53 -19
  205. data/lib/insika/studio/views/system_files.erb +7 -7
  206. data/lib/insika/studio/views/task.erb +45 -15
  207. data/lib/insika/studio/views/tasks.erb +35 -18
  208. data/lib/insika/studio/views/tools.erb +62 -73
  209. data/lib/insika/task_actor.rb +4 -1
  210. data/lib/insika/task_store.rb +67 -1
  211. data/lib/insika/telemetry/pricing.rb +16 -14
  212. data/lib/insika/telemetry/recorder.rb +103 -9
  213. data/lib/insika/telemetry/ruby_llm_instrumenter.rb +110 -0
  214. data/lib/insika/telemetry.rb +12 -2
  215. data/lib/insika/templates/daily-digest/README.md +13 -0
  216. data/lib/insika/templates/daily-digest/agent.rb +22 -11
  217. data/lib/insika/testing/store_contract.rb +65 -0
  218. data/lib/insika/text_tool_call.rb +66 -0
  219. data/lib/insika/tick.rb +7 -1
  220. data/lib/insika/token_store.rb +5 -0
  221. data/lib/insika/tool_batch.rb +1 -1
  222. data/lib/insika/tool_catalog.rb +12 -2
  223. data/lib/insika/tool_definition.rb +10 -4
  224. data/lib/insika/tool_envelope.rb +165 -28
  225. data/lib/insika/tool_manifest.rb +1 -0
  226. data/lib/insika/tool_store.rb +3 -0
  227. data/lib/insika/tool_trace_store.rb +15 -8
  228. data/lib/insika/tool_usage_report.rb +2 -2
  229. data/lib/insika/tools/a2a_remote.rb +1 -1
  230. data/lib/insika/tools/agent_enum.rb +1 -1
  231. data/lib/insika/tools/confirm_pending.rb +112 -0
  232. data/lib/insika/tools/data_defined_tool.rb +23 -15
  233. data/lib/insika/tools/generate_image.rb +2 -2
  234. data/lib/insika/tools/load_knowledge.rb +7 -3
  235. data/lib/insika/tools/load_skill.rb +3 -3
  236. data/lib/insika/tools/present.rb +1 -8
  237. data/lib/insika/tools/remember.rb +7 -2
  238. data/lib/insika/tools/run_persona_eval.rb +3 -5
  239. data/lib/insika/tools/save_artifact.rb +16 -6
  240. data/lib/insika/tools/schedule_followup.rb +3 -3
  241. data/lib/insika/tools/stuck_signal.rb +3 -3
  242. data/lib/insika/tools/subagent.rb +5 -5
  243. data/lib/insika/tools/subagents.rb +3 -3
  244. data/lib/insika/tools/tool_search.rb +7 -5
  245. data/lib/insika/tools/tts.rb +2 -2
  246. data/lib/insika/tools/update_briefing.rb +4 -4
  247. data/lib/insika/turn_output.rb +2 -2
  248. data/lib/insika/turn_state.rb +18 -0
  249. data/lib/insika/turn_timing.rb +51 -14
  250. data/lib/insika/version.rb +1 -1
  251. data/lib/insika/wiring/graph.rb +81 -16
  252. data/lib/insika/workflow.rb +25 -64
  253. data/lib/insika.rb +19 -3
  254. metadata +69 -13
  255. data/lib/insika/commands/import_mcp_tools.rb +0 -48
  256. data/lib/insika/mcp_http_client.rb +0 -67
  257. data/lib/insika/mcp_tool_ingestor.rb +0 -165
@@ -0,0 +1,307 @@
1
+ ---
2
+ title: RubyLLM 2 Migration
3
+ parent: Ship it
4
+ nav_order: 6
5
+ permalink: /rubyllm-2-migration/
6
+ ---
7
+
8
+ # RubyLLM 2 Migration
9
+
10
+ Status reviewed on 2026-09-24: implemented on `codex/rubyllm-2-migration`,
11
+ and **approved for merge and testing on the existing Railway staging service**.
12
+ Keep Insika's durable runtime and policy;
13
+ delegate provider protocols, tool schemas, and cache formatting to RubyLLM.
14
+ No Rails, package extraction or database migration. JSONSchemer is a direct
15
+ dependency for workflow and tool argument validation.
16
+
17
+ The follow-up ecosystem audit and execution status (kept with the internal plans)
18
+ records replacements and the concrete gates that still block others.
19
+
20
+ ## Dependency
21
+
22
+ The Gemfile and lockfile pin RubyLLM commit
23
+ `099c381b44fec6f1d72b2b63b9a9eea7635755f1`, the merge of
24
+ [native MCP PR 959](https://github.com/crmne/ruby_llm/pull/959).
25
+ The external `ruby_llm-mcp` dependency and its Origin-header patch are removed.
26
+ The pinned source still reports version 2.0.0, but published 2.0.0 does not contain
27
+ this client. Before publishing Insika, require a released RubyLLM version with
28
+ native MCP in the gemspec and remove the Git pin. A gemspec cannot carry it.
29
+
30
+ Existing MCP records and cached tool descriptors remain readable. Discovery and
31
+ execution use native Streamable HTTP or stdio; Insika retains policy, evidence,
32
+ credential masking, the stdio opt-in, and the egress allowlist. Refresh preserves
33
+ the server's strict boolean read-only hint. Legacy SSE records fail with a request
34
+ to configure a Streamable HTTP endpoint; they are not silently rewritten. Native
35
+ MCP also requires HTTPS outside loopback, even with the HTTP egress opt-in.
36
+ Sampling, roots and server logging are not supported by the native client. Insika
37
+ does not yet expose native MCP input-request resumption or its OAuth UI; unanswered
38
+ input requests surface as tool errors through the existing wrapper.
39
+
40
+ Native MCP validation on 2026-09-23: **5,614 examples, zero failures**, seed 46560.
41
+ Local protocol fixtures cover real stdio and HTTP sockets, modern discovery,
42
+ legacy handshake/session headers, event-stream responses, authorization headers,
43
+ tool errors, structured content, attachments, and read-only metadata. No external
44
+ production MCP server was invoked for these fixture checks. The current-candidate
45
+ benchmark below was repeated after this dependency switch; it does not call MCP.
46
+
47
+ The official RubyLLM coding skill (now from 2.1.0) is copied into `.agents/skills/rubyllm`.
48
+ Refresh it from the installed gem when upgrading RubyLLM, not from upstream main.
49
+
50
+ ## Preserved Behavior
51
+
52
+ - Native `step` drives the existing runtime; a terminal tool finishes its batch
53
+ and prevents another model request. Siblings, result IDs, steering, and output remain covered.
54
+ - Tools use `parameter` and `parameters_schema`; data-defined JSON schemas stay unchanged.
55
+ - Native cache markers preserve the stable/volatile Anthropic instruction boundary.
56
+ - Native tokens and model readers retain Insika's usage fields. Auxiliary billing
57
+ includes cache reads and writes without counting thinking twice.
58
+ - RubyLLM now handles operator approval gating for native chats. Insika retains
59
+ authorization, persisted decisions, customer confirmation, security checks and UI.
60
+ Decisions bind to task, turn, call ID, real tool name and arguments. Checkpoints
61
+ preserve pending calls, completed results, usage, evidence and filtered halt text.
62
+ Recorded side effects are skipped on replay; terminal results survive interruption.
63
+ Direct workflow/customer-confirmation paths retain the existing blocking gate.
64
+ Legacy tool-name decisions do not authorize newly generated native calls.
65
+ A crash between an external effect and its local marker still requires downstream
66
+ idempotency; this is not an exactly-once guarantee.
67
+ - Opt-in native
68
+ compaction now uses separate model-bound state and filtered stored history, with
69
+ original-prefix recovery on fallback. See [native compaction](CONTEXT.md#native-responses-compaction)
70
+ for its conservative budget and supported providers.
71
+
72
+ ## Verification
73
+
74
+ Latest completed checks (2026-09-24, before merge): **5,616 RSpec
75
+ examples, zero failures**, seed 19184; **53 Studio tests passed**. Assets were
76
+ rebuilt during the preceding Studio implementation and remain unchanged.
77
+ Final diff review reran 200 focused runtime/MCP/contract examples with no failures
78
+ and found no actionable introduced regression in the reviewed scope.
79
+ Browser checks covered tool grouping by origin/server, filtering, collapsible
80
+ sections and section selection without changing other sections or denied tools.
81
+ Changes still require **Save tools**; filtering does not change permissions.
82
+
83
+ Local development validation also exercised a real provider reply and MCP tool
84
+ discovery. The user confirmed WhatsApp replies and execution through the Metabase
85
+ MCP. This is smoke-test evidence, not a comparative quality eval or a new benchmark.
86
+ An HTTP data tool and an MCP instance may use different credentials: successful
87
+ MCP discovery does not validate a separately configured HTTP tool.
88
+
89
+ ### Earlier Verification
90
+
91
+ Native transport retries are enabled for reliable native chats, with isolated
92
+ per-chat configuration and no outer Insika retries. The approved behavior is
93
+ per-request retries, stopping once streaming starts. Insika still owns fallback,
94
+ model policy, tenant circuits and the overall per-model deadline. Circuits count
95
+ an exhausted native sequence once, not each internal HTTP failure. The native
96
+ request timeout is also configured from the profile. Completed tool results
97
+ survive fallback, and failed-stream redactor tails are discarded.
98
+ Native `with_fallbacks` adoption remains deferred: it restores the original model
99
+ before the next generation and requires adaptation for model-bound compaction.
100
+
101
+ After native compaction, operator approvals and native transport retries: **5,603 examples, zero failures** (seed 32098).
102
+ The execution record (kept with the internal plans) has the new contracts, remaining replacement gates and offline benchmark.
103
+
104
+ Full RSpec after the benchmark correction: **5,519 examples, zero failures**
105
+ (seed 55367). Studio: **50 tests passed** during the preceding migration validation.
106
+ The real-gem contract has 17 behavioral cases, with no missing-gem skip.
107
+ Synthetic SQLite records passed 1.16 write -> 2.0 read/write -> 1.16 read/write,
108
+ including history, pending/resolved approvals, and recorded side-effect markers.
109
+ An isolated installed gem passed reply, SQLite embed, Rack mount, actual HTTP/SSE,
110
+ Studio login, packaged assets/docs, and lazy-load guards using an offline provider.
111
+ That earlier install used the former external MCP Git dependency. On 2026-09-24,
112
+ the current native-MCP candidate was built and installed in a fresh GEM_HOME
113
+ outside the checkout and passed all those checks again. The synthetic SQLite
114
+ 1.16 -> 2.0 -> 1.16 record rehearsal also passed again. A Linux ARM64 Docker
115
+ build with the deployment lockfile passed and loaded native MCP while preserving
116
+ the core lazy-load guard. The isolated gem install explicitly pins RubyLLM's Git
117
+ revision and resolves other allowed dependencies; it is not a public gem-install
118
+ proof or a substitute for verifying Railway's x86_64 deployment.
119
+ Browser checks passed for login, agent configuration, skill/provider creation,
120
+ Playground replies, completed tasks, and session transcripts. Approval/resume
121
+ flows were covered by automated tests, not separately exercised in the browser.
122
+
123
+ ## Performance
124
+
125
+ ### Current Candidate (2026-09-24)
126
+
127
+ Five alternating runs per version in each of three modes (stub, real Chat with
128
+ a synthetic model, and real Chat with catalog model `gpt-4o`), plus five SQLite
129
+ runs per version. The catalog experiment was subsequently corrected and rerun
130
+ five times per version, as described below. Ruby 4.0.6,
131
+ arm64-darwin25, YJIT off; 300 latency turns, 30 warmups, concurrency 16 and
132
+ 20 waves per scenario. Native MCP and the latest Studio changes are present.
133
+ The same existing benchmark harnesses load each checkout's actual runtime.
134
+
135
+ All 40 benchmark processes succeeded: 58,500 engine turns including warmups,
136
+ matching provider/tool/history/chunk counts, and 300,000 SQLite writes with zero
137
+ reported lock errors. Automated suites ran after the measurement. Development
138
+ servers remained running, so workstation noise is not eliminated.
139
+
140
+ Profiling exposed a flaw in the earlier catalog experiment: its flag assertion
141
+ wrapped `RubyLLM.chat`, but `main` constructs chats through a context. Its
142
+ `assume_model_exists: false` override therefore did not reach the actual Chat.
143
+ Those earlier catalog numbers do not establish equivalent registry behavior.
144
+ The corrected local harness forces the option at `Chat#initialize` and asserts
145
+ that the resulting model is the exact registry object. The separate profiler
146
+ confirms one registry lookup per measured turn in both versions.
147
+
148
+ All ten corrected benchmark processes succeeded, with matching activity counts
149
+ across 19,500 additional turns including warmups. Median corrected measurements:
150
+
151
+ | Scenario | Main p50 / p95 ms | Migration p50 / p95 ms | Turns/s, before -> after |
152
+ | --- | --- | --- | --- |
153
+ | Greeting | 1.37 / 2.01 | 1.48 / 2.48 | 654.1 -> 603.8 |
154
+ | Two tool rounds | 1.46 / 2.10 | 1.80 / 2.42 | 603.5 -> 515.7 |
155
+ | History | 1.51 / 2.08 | 1.62 / 2.25 | 573.9 -> 558.2 |
156
+
157
+ Catalog p50 increases are **7.3-23.3%**, or **0.11-0.34 ms per turn**.
158
+ Peak RSS increased **13.9%** (87,769,088 -> 99,991,552 bytes), and tool
159
+ allocations increased 8.4%. Tool p50 ranges did not overlap (main 1.35-1.58 ms,
160
+ migration 1.65-1.85 ms); the other scenarios' ranges overlap. These are new runs,
161
+ so differences from the invalid comparison are not attributable solely to the
162
+ harness correction. Synthetic-model p50 increased 18.3-28.2% and peak
163
+ RSS 29.4%. Stub p50 changes were -5.3%, +4.5% and +9.1%, with peak RSS +2.3%.
164
+ SQLite median writes/s stayed within 2.3% of baseline in either direction.
165
+
166
+ The proposed 10% gate still fails numerically. This isolates an increase along the real-Chat
167
+ path, not its cause; it does not measure native HTTP, MCP calls, UI speed or
168
+ provider latency. No safety or accounting was disabled. Earlier measurements
169
+ below are retained as historical comparisons, not current acceptance evidence.
170
+
171
+ **Decision (2026-09-24):** The project owner accepted the measured performance
172
+ tradeoff for this migration and approved continuing with RubyLLM 2's native
173
+ runtime. The additional 0.34 ms in the two-tool benchmark does not block ongoing
174
+ development validation. This is an explicit exception, not a passing benchmark
175
+ or acceptance of future regressions. There are no real operational workflows
176
+ beyond the tests already executed. Continue validation with the existing
177
+ automated tests, controlled integration checks and offline benchmarks. Measure
178
+ live latency, errors and memory when actual usage exists; that is not a current
179
+ development prerequisite. Investigate
180
+ upstream improvements with isolated evidence; do not add custom caches or a
181
+ replacement tool loop without demonstrated waste. The performance decision alone
182
+ did not authorize deployment. The project owner subsequently authorized merge
183
+ to `main` and testing through its automatic Railway staging deployment. Gem
184
+ publication remains outside that authorization.
185
+
186
+ ### Targeted Profiling (2026-09-24)
187
+
188
+ Three alternating instrumented runs per version, each with 1,001 measured
189
+ two-tool turns after 30 warmups, completed without errors. Both versions made
190
+ one registry lookup, three provider calls and two tool calls per measured turn.
191
+ The probe wraps selected methods using the monotonic clock and `GC.stat`; it is
192
+ local diagnostic tooling, not a runtime dependency.
193
+
194
+ Median inclusive time per turn, rounded, across those instrumented runs:
195
+
196
+ | Scope | Main ms | Migration ms |
197
+ | --- | --- | --- |
198
+ | `Executor#ask_on` (generation and tool cycle) | 0.402 | 0.659 |
199
+ | `Executor#create_chat` | 0.261 | 0.279 |
200
+ | `Executor#record_model_visible` | 0.011 | 0.042 |
201
+ | `Executor#persist_turn` | 0.326 | 0.322 |
202
+ | `ContextBuilder#call` | 0.251 | 0.267 |
203
+
204
+ The largest observed increase is inside the generation/tool cycle. Native
205
+ `instrumentation_payload`, `record_completion_event`, and
206
+ `record_out_of_band_usage` account for approximately 0.030, 0.022 and 0.038 ms
207
+ per turn respectively. The latter is the offline provider's fallback accounting
208
+ path, not the accounting path used by native HTTP providers. Message creation
209
+ also grows from approximately 0.063 to 0.079 ms for the same 55 messages/chunks
210
+ per turn. These nested times overlap other scopes and must not be added to them.
211
+
212
+ Registry lookup is slower (approximately 0.073 -> 0.111 ms), but native 2.0
213
+ creates one connection per turn instead of 1.16's two; the overall Chat creation
214
+ difference is much smaller. The trace increase includes corrected behavior:
215
+ `main` treats the tool Hash as pairs and misses schemas, whereas the migration
216
+ captures actual tool schemas and system instructions. Removing those records
217
+ would reintroduce a correctness problem, not provide an equivalent optimization.
218
+
219
+ Instrumented timings include probe overhead and cannot apportion the exact
220
+ uninstrumented regression. Instrumented allocation counts are not clean
221
+ allocation measurements. RSS growth remains unattributed; MCP and HTTP were not
222
+ exercised. The next investigation should isolate RubyLLM's per-round bookkeeping
223
+ without Insika before proposing an upstream optimization. Keep accounting,
224
+ approval checks and trace correctness intact; no custom tool loop or cache was
225
+ added during profiling.
226
+
227
+ ### Earlier Catalog-Model Experiment (Invalid Flag Assertion)
228
+
229
+ These results used the ineffective `RubyLLM.chat` assertion described above.
230
+ They are historical only, not an equivalent-registry comparison. Five
231
+ alternating runs per version named `gpt-4o`. The offline provider is
232
+ registered under `openai` only inside the experiment; it does not make HTTP calls.
233
+ Its intended assertions were bypassed by context-created chats. No application
234
+ model-selection behavior was changed.
235
+ Settings match the earlier comparison: Ruby 4.0.6, YJIT off, 300 latency turns,
236
+ 30 warmups, concurrency 16, 20 waves, and identical tool/history/chunk counts.
237
+ Every scenario in all ten runs completed without errors.
238
+
239
+ | Scenario | 1.16 p50 / p95 ms | 2.0 p50 / p95 ms | Turns/s, before -> after |
240
+ | --- | --- | --- | --- |
241
+ | Greeting | 1.23 / 1.65 | 1.32 / 2.06 | 549.2 -> 629.2 |
242
+ | Two tool rounds | 1.36 / 1.75 | 1.62 / 2.09 | 620.3 -> 539.8 |
243
+ | History | 1.28 / 1.78 | 1.44 / 1.95 | 590.6 -> 561.9 |
244
+
245
+ Median p50 changes are +7.3%, +19.1%, and +12.5%. Median peak RSS is
246
+ 89,194,496 -> 105,971,712 bytes (+18.8%); tool allocations rise 4.8%.
247
+ These numbers cannot determine whether registry lookup avoidance explains the
248
+ difference. They are offline engine-plus-Chat results, not
249
+ live-provider latency or proof of a RubyLLM defect. Investigate the remaining
250
+ tool-loop and memory differences before making an upstream performance claim.
251
+
252
+ ### Earlier Unknown-Model Experiment
253
+
254
+ The following results are retained for comparison, not generalized to catalog
255
+ models. The fake model used `assume_model_exists: true`, which bypassed registry
256
+ resolution in 1.16 but not 2.0.
257
+
258
+ Five interleaved runs per version, Ruby 4.0.6, YJIT off, same host and settings.
259
+ Real Chat with a synthetic provider, no network; provider/tool/chunk/history counts match.
260
+
261
+ | Scenario | 1.16 p50 / p95 ms | 2.0 p50 / p95 ms | Turns/s, before -> after |
262
+ | --- | --- | --- | --- |
263
+ | Greeting | 1.38 / 1.83 | 1.60 / 2.23 | 476.3 -> 571.2 |
264
+ | Two tool rounds | 1.35 / 1.95 | 1.79 / 2.44 | 619.8 -> 491.5 |
265
+ | History | 1.34 / 1.87 | 1.78 / 2.37 | 597.3 -> 498.2 |
266
+
267
+ These replace the initial measurements: the offline provider omitted request
268
+ `model_info`, unlike native `Accounting::Usage::Tracker#succeed`. Supplying it
269
+ removes three spurious pricing catalog lookups per two-tool turn. The benchmark
270
+ spec now rejects those lookups. No production runtime workaround was added.
271
+
272
+ Corrected real-Chat p50 increased 15.9-32.8%; peak RSS increased 24.0%
273
+ (87,310,336 -> 108,232,704 bytes); tool allocations increased 7.4%.
274
+ RSS also increased 11.3% in stub mode, so the entire memory difference cannot
275
+ be attributed to RubyLLM catalog loading. SQLite had zero lock errors, but
276
+ throughput varied from +1.6% to -17.2%; the earlier no-regression claim is withdrawn.
277
+ These are host-specific samples, not a controlled attribution of every difference.
278
+
279
+ The standalone `scripts/rubyllm_profile.rb` reproduces the fixture defect and
280
+ the remaining catalog lookup without Insika. With `--request-context --profile`,
281
+ 1,000 two-tool turns make 1,000 `Models#find` calls in 2.0 versus none in 1.16.
282
+ RubyLLM 2 resolves catalog metadata even with `assume_model_exists`; 1.16 skipped
283
+ it for this non-local provider. In one instrumented run that lookup consumed
284
+ 112 ms total; fallback usage accounting consumed 39 ms. Inclusive timings must
285
+ not be summed. All three requests, both tools, history and aggregate tokens are
286
+ checked. The fake provider bypasses native transport and uses fallback usage
287
+ accounting, so this is not an end-to-end native-provider performance claim.
288
+
289
+ For upstream discussion, run the script outside Bundler with `1.16.0 --profile`
290
+ and `2.0.0 --request-context --profile`. Omitting `--request-context` deliberately
291
+ recreates the old fixture defect, not native provider behavior. The useful upstream
292
+ question is whether repeated registry resolution can be cheaper while preserving
293
+ model metadata and refresh semantics. No upstream patch or issue was published;
294
+ remaining latency and memory attribution is open.
295
+ The proposed 10% performance gate **failed**. No accounting or safety was disabled.
296
+ Reproduce the candidate with the commands in [Benchmark](BENCHMARK.md).
297
+
298
+ ## Release Gate
299
+
300
+ Public gem release still requires a published RubyLLM native-MCP dependency.
301
+ The measured performance tradeoff is accepted; current installed-artifact and
302
+ synthetic record-compatibility checks passed. Merge to `main` and testing of its
303
+ automatic Railway staging deployment are authorized. Confirm that deployment's
304
+ status and HTTP behavior before reporting success. No production rollout,
305
+ database schema migration or gem publication is part of this validation.
306
+ Record compatibility does not guarantee exactly-once external effects; keep
307
+ destination idempotency or reconciliation for an execution-before-checkpoint failure.
data/docs/SECURITY.md CHANGED
@@ -246,6 +246,13 @@ resume path is durable: an approval that arrives after a restart still resumes t
246
246
  turn from its checkpoint (see
247
247
  [Architecture](ARCHITECTURE.md#durability-checkpoints-and-resume)).
248
248
 
249
+ There is a second approver: the customer. `customer_confirm: [tool names]` holds a
250
+ call the same way, but the turn *ends* with a question instead of suspending, and the
251
+ next message of the conversation decides it through `confirm_pending` /
252
+ `cancel_pending`. A hold nobody resolves expires when that next turn ends. Use it for
253
+ the write the person on the other end must agree to; use approval for the one a staff
254
+ member must. A tool cannot be on both lists. See [Tools](TOOLS.md#customer-confirmation-a-write-the-conversation-approves).
255
+
249
256
  ## Egress: the SSRF boundary
250
257
 
251
258
  Every outbound HTTP call from a data tool passes through the **EgressGuard**, a
@@ -0,0 +1,40 @@
1
+ ---
2
+ layout: default
3
+ title: Shared conversations
4
+ ---
5
+
6
+ # Shared conversations
7
+
8
+ `shared_conversations: true` opts an agent into required central history. The
9
+ DSL accepts `shared_conversations true`; stored profiles preserve the flag.
10
+ Default is false. Configure `INSIKA_CONVERSATIONS_URL`, a scoped
11
+ `INSIKA_CONVERSATIONS_TOKEN`, and a durable native backend (`INSIKA_DB` or
12
+ `INSIKA_DATABASE_URL`). The client provisions the central conversation first.
13
+
14
+ The tenant-authenticated `/v1/responses` request adds `user_text` (original speech)
15
+ and `shared_conversation` with UUID `tenant_id`, `user_id`, `agent_id`,
16
+ `conversation_id`, `turn_id`, and `message_id`. Keep the
17
+ same turn/message IDs across retries. Existing `input` still contains the
18
+ presenter/context text used for this request. The authenticated tenant must match.
19
+ Omit `generation` to acquire a settled conversation automatically when admitting
20
+ the next turn. The engine reads and writes the acquired generation internally.
21
+ An explicit integer `generation` retains the legacy ownership check.
22
+
23
+ Upload incoming media to the central attachment endpoint first. Optional
24
+ `shared_conversation.content` carries canonical text/media parts with attachment
25
+ IDs; its text must match `user_text`. Generated image/audio bytes upload before
26
+ completion. Native checkpoints store attachment bytes and hydrate them at the
27
+ RubyLLM boundary. Full tool results retain secret-key masking without prompt caps.
28
+
29
+ Admission precedes model/tool work. Delivery follows central completion. A lost
30
+ completion acknowledgment reuses the native durable result; it never reruns tools.
31
+ An uncertain execution stays blocked for operator reconciliation. Only the same
32
+ native waiting approval can resume from its checkpoint under the current generation.
33
+ Unresolved customer confirmations block switching. Central deletion/outage fails
34
+ before cached history is used. Local native purge still requires an explicit cleanup.
35
+
36
+ Run the isolated HTTP proof (synthetic model/tools, temporary databases):
37
+
38
+ ```sh
39
+ SHARED_CONVERSATIONS_SOURCE=/path/to/shared-memory-repo bundle exec rspec spec/insika/shared_conversations_http_spec.rb
40
+ ```
data/docs/SKILLS.md CHANGED
@@ -249,9 +249,11 @@ in the allowlist → the model never sees it in `<available_skills>`.
249
249
  Two ways to satisfy both:
250
250
 
251
251
  - **Via a definition/pack import.** The import writes each skill directory into
252
- the store and sets the agent's `skills` allowlist **authoritatively** from the
253
- skills present — so a re-import that drops a skill also removes it. Keep the
254
- definition complete.
252
+ that agent's scope, without overwriting another agent's or a shared skill, and
253
+ sets the agent's `skills` allowlist **authoritatively** from the skills present.
254
+ Dropping a skill removes it from the allowlist, not storage. Keep the definition
255
+ complete. Packs previously imported into shared scope must be reimported from
256
+ their original sources; the overwritten shared body cannot recover each agent's version.
255
257
  - **Directly (Studio / API / DSL).** Write the skill (upserts the row and reloads
256
258
  the catalog atomically — live immediately), then attach it to the agent(s) by
257
259
  adding its name to the `skills` allowlist.
data/docs/STUDIO.md ADDED
@@ -0,0 +1,122 @@
1
+ ---
2
+ title: Studio
3
+ parent: Operate
4
+ nav_order: 0
5
+ permalink: /studio/
6
+ ---
7
+
8
+ # Studio — the operator UI
9
+
10
+ The Studio is the web UI at `/studio`. Sign in with `ADMIN_TOKEN` (see
11
+ [Deploy](DEPLOY.md#environment-variables)). This page covers the screens for
12
+ watching conversations, reviewing what waits on a person, and seeing what a
13
+ change did. Model metrics are on [Observability](OBSERVABILITY.md#studio-models).
14
+
15
+ Outside production, a banner above every page names the environment
16
+ (`INSIKA_ENV`, else `RACK_ENV`, else `local`): amber for staging, blue for
17
+ anything else. Production (`production` or `prod`) shows no banner.
18
+
19
+ ## Navigation
20
+
21
+ The sidebar groups pages as Operate, Review, Build, Insights and Platform. Each
22
+ Review page shows a badge with how many items wait on it. `⌘K` (or Search in the
23
+ sidebar) opens a palette that jumps to any page or agent.
24
+
25
+ ## Overview
26
+
27
+ `/studio/home`. The Overview shows the headline numbers for the agent filter and
28
+ a **Needs attention** row. The row lists the review queues, customers waiting for
29
+ a reply and tools failing in the last 24 hours, each linking to its page. Only
30
+ non-zero items show.
31
+
32
+ ## Live
33
+
34
+ `/studio/live` shows one card per recent conversation: customer, agent, the last
35
+ message, a presence dot (activity in the last 5 minutes) and the guardrail flag.
36
+ Cards move with the event stream. While a turn runs, a card shows the tools it
37
+ calls and "agent is writing…". A turn paused for an approval stops writing until
38
+ it resumes. A ticker lists the latest events, without the streaming deltas.
39
+
40
+ `?agent=<id>` shows only that agent's conversations.
41
+
42
+ ## Chats
43
+
44
+ `/studio/chats` lists conversations with three tabs: **All**, **Active now**
45
+ (activity in the last 5 minutes) and **Guardrail flagged**, each with its count.
46
+ The session page shows the customer (key, name, phone) and links to their other
47
+ conversations.
48
+
49
+ ### Why this reply
50
+
51
+ Each agent reply has a collapsed **why this reply** panel for its own turn:
52
+
53
+ - context tokens by category, and the skills injected with the reason;
54
+ - the tool calls with status and timing;
55
+ - the guardrail flags on the customer's message.
56
+
57
+ From the panel, **Replay in Playground** opens the conversation in the
58
+ Playground, and **Save as eval
59
+ case** opens the Evals editor prefilled from the reply (the customer's turns,
60
+ the transcript as the reference, the tools the turn called). A reply that
61
+ streams in live offers the panel when its turn finishes.
62
+
63
+ ### Tool calls
64
+
65
+ A turn with two or more tool calls draws a waterfall of when each one ran.
66
+ Tool calls are grouped by turn within each task, since the turn number restarts
67
+ at 1 for every task.
68
+
69
+ ### Markup tags
70
+
71
+ Tags an agent or a host writes in the text, like `<carrossel/>`,
72
+ `<vitrine formato="lista">` or `</b>`, show as highlighted tokens: in replies,
73
+ customer messages, and the Live and Chats previews. They are escaped and never
74
+ rendered. The channel decides what a tag does; the Studio only shows it is
75
+ there. A tag inside inline code stays code.
76
+
77
+ ## Guardrail false positives
78
+
79
+ On a guardrail flag in the session page, **Mark false positive** records the
80
+ verdict on the flag itself, inside the session, so retention and tenant purges
81
+ cover it. The flag can later be marked resolved, or the verdict undone.
82
+ Reviewing a flag does not count as conversation activity: the session keeps its
83
+ place in the recent lists. The command is `:review_guardrail_flag`.
84
+
85
+ ## Inbox
86
+
87
+ `/studio/inbox` is one list, oldest first, of everything that waits on a person:
88
+ approvals, fact proposals, mined skills, refinement runs and reported guardrail
89
+ false positives. Tabs filter by type, with counts. Each row opens the page where
90
+ the item is decided.
91
+
92
+ ## Agent History
93
+
94
+ The **History** tab of an agent shows outcomes per day over the last 30 days:
95
+ the funnel's primary outcome when the agent declares a funnel, else all
96
+ outcomes. A dashed line marks each change to a prompt file or to one of this
97
+ agent's own skills (shared skills change for every agent, so they are not
98
+ marked).
99
+
100
+ Under the chart, one card per change shows the file or skill, when, a short line
101
+ diff, and the outcomes in the 7 days after against the 7 days before. A window
102
+ that is not complete yet, or that starts before the outcome data, reads "not
103
+ enough data". A prompt card has the Restore form; a skill card links to the
104
+ skill.
105
+
106
+ ## Funnel: where conversations stop
107
+
108
+ The Funnel page marks the biggest loss between two consecutive declared stages.
109
+ Each loss opens (`?drop=<stage>`) the conversations that reached the stage but
110
+ not the next one in the period, grouped by the last thing that happened:
111
+
112
+ | Reason | Meaning |
113
+ |--------|---------|
114
+ | Last tool call failed | the conversation's last tool call returned an error |
115
+ | A guardrail flagged a message | a guardrail flagged one of its messages |
116
+ | Customer waiting on a reply | the customer wrote last |
117
+ | Customer went quiet after the agent's reply | the agent wrote last |
118
+ | No message to read | the conversation has no message to classify |
119
+ | Conversation no longer stored | the conversation was deleted |
120
+
121
+ Only the newest 200 stuck conversations are classified; the page says how many
122
+ were left out.