insika 0.0.1 → 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (260) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGELOG.md +295 -0
  3. data/LICENSE +21 -0
  4. data/README.md +136 -2
  5. data/bin/insika +351 -0
  6. data/docs/AGENTS.md +494 -0
  7. data/docs/ARCHITECTURE.md +333 -0
  8. data/docs/BENCHMARK.md +114 -0
  9. data/docs/CHANNELS.md +453 -0
  10. data/docs/CONTEXT.md +100 -0
  11. data/docs/DEPLOY.md +334 -0
  12. data/docs/EMBEDDING.md +194 -0
  13. data/docs/EVALS.md +273 -0
  14. data/docs/LOADTEST.md +231 -0
  15. data/docs/OBSERVABILITY.md +365 -0
  16. data/docs/PLUGINS.md +211 -0
  17. data/docs/REFINEMENT.md +477 -0
  18. data/docs/RELEASING.md +70 -0
  19. data/docs/RUNNING-LOCAL.md +153 -0
  20. data/docs/SANDBOX.md +114 -0
  21. data/docs/SECURITY.md +362 -0
  22. data/docs/SKILLS.md +98 -0
  23. data/docs/TOOLS.md +302 -0
  24. data/docs/WHY.md +137 -0
  25. data/docs/WORKFLOWS.md +225 -0
  26. data/docs/build.md +14 -0
  27. data/docs/index.md +68 -0
  28. data/docs/onboarding/start.md +126 -0
  29. data/docs/operate.md +12 -0
  30. data/docs/ship.md +10 -0
  31. data/docs/understand.md +10 -0
  32. data/lib/insika/agent_file_store.rb +125 -0
  33. data/lib/insika/agent_profile.rb +188 -0
  34. data/lib/insika/allowlist.rb +28 -0
  35. data/lib/insika/baseline_store.rb +74 -0
  36. data/lib/insika/capability/resolved_tool.rb +34 -0
  37. data/lib/insika/capability_registry.rb +112 -0
  38. data/lib/insika/channel_delivery.rb +150 -0
  39. data/lib/insika/channel_registry.rb +30 -0
  40. data/lib/insika/channels/relay.rb +178 -0
  41. data/lib/insika/channels/web/widget.js +283 -0
  42. data/lib/insika/channels/web.rb +211 -0
  43. data/lib/insika/chat_builder.rb +254 -0
  44. data/lib/insika/checkpoint.rb +13 -0
  45. data/lib/insika/checkpoint_store.rb +153 -0
  46. data/lib/insika/coercion.rb +50 -0
  47. data/lib/insika/command.rb +32 -0
  48. data/lib/insika/command_bus.rb +39 -0
  49. data/lib/insika/commands/agent_payload.rb +41 -0
  50. data/lib/insika/commands/approve_action.rb +46 -0
  51. data/lib/insika/commands/cancel_task.rb +33 -0
  52. data/lib/insika/commands/create_agent.rb +54 -0
  53. data/lib/insika/commands/create_session.rb +67 -0
  54. data/lib/insika/commands/delete_agent.rb +33 -0
  55. data/lib/insika/commands/delete_agent_file.rb +50 -0
  56. data/lib/insika/commands/delete_data_tool.rb +33 -0
  57. data/lib/insika/commands/delete_llm_provider.rb +36 -0
  58. data/lib/insika/commands/delete_mcp.rb +30 -0
  59. data/lib/insika/commands/delete_system_file.rb +29 -0
  60. data/lib/insika/commands/gate_refinement.rb +245 -0
  61. data/lib/insika/commands/import_mcp_tools.rb +48 -0
  62. data/lib/insika/commands/import_tools.rb +81 -0
  63. data/lib/insika/commands/memory_add_note.rb +32 -0
  64. data/lib/insika/commands/memory_forget_fact.rb +32 -0
  65. data/lib/insika/commands/memory_put_fact.rb +35 -0
  66. data/lib/insika/commands/pause_task.rb +29 -0
  67. data/lib/insika/commands/resolve_refinement.rb +126 -0
  68. data/lib/insika/commands/restore_agent_file.rb +36 -0
  69. data/lib/insika/commands/restore_data_tool.rb +34 -0
  70. data/lib/insika/commands/restore_system_file.rb +31 -0
  71. data/lib/insika/commands/resume_task.rb +85 -0
  72. data/lib/insika/commands/run_refinement.rb +133 -0
  73. data/lib/insika/commands/send_message.rb +150 -0
  74. data/lib/insika/commands/set_agent_tools.rb +39 -0
  75. data/lib/insika/commands/set_skill_agents.rb +71 -0
  76. data/lib/insika/commands/trigger_workflow.rb +80 -0
  77. data/lib/insika/commands/update_agent.rb +49 -0
  78. data/lib/insika/commands/update_settings.rb +33 -0
  79. data/lib/insika/commands/upsert_llm_provider.rb +34 -0
  80. data/lib/insika/commands/upsert_mcp.rb +32 -0
  81. data/lib/insika/commands/write_agent_file.rb +57 -0
  82. data/lib/insika/commands/write_data_tool.rb +43 -0
  83. data/lib/insika/commands/write_golden.rb +58 -0
  84. data/lib/insika/commands/write_skill.rb +50 -0
  85. data/lib/insika/commands/write_system_file.rb +31 -0
  86. data/lib/insika/config_store.rb +85 -0
  87. data/lib/insika/context/builder.rb +166 -0
  88. data/lib/insika/context/catalog_provider.rb +23 -0
  89. data/lib/insika/context/fragment.rb +19 -0
  90. data/lib/insika/context/priority.rb +29 -0
  91. data/lib/insika/context/provider.rb +19 -0
  92. data/lib/insika/context/providers/memory.rb +60 -0
  93. data/lib/insika/context/providers/prompt.rb +105 -0
  94. data/lib/insika/context/providers/request.rb +32 -0
  95. data/lib/insika/context/providers/session.rb +108 -0
  96. data/lib/insika/context/providers/skill.rb +20 -0
  97. data/lib/insika/context/providers/tool_search.rb +20 -0
  98. data/lib/insika/delegation_store.rb +153 -0
  99. data/lib/insika/doctor.rb +294 -0
  100. data/lib/insika/dsl/definition.rb +55 -0
  101. data/lib/insika/dsl/runtime.rb +379 -0
  102. data/lib/insika/dsl/server_boot.rb +97 -0
  103. data/lib/insika/dsl/system.rb +93 -0
  104. data/lib/insika/dsl/workflow_adapter.rb +59 -0
  105. data/lib/insika/dsl.rb +307 -0
  106. data/lib/insika/edge_limiter.rb +130 -0
  107. data/lib/insika/egress_guard.rb +75 -0
  108. data/lib/insika/env_schema.rb +246 -0
  109. data/lib/insika/errors.rb +145 -0
  110. data/lib/insika/evals/assertions.rb +247 -0
  111. data/lib/insika/evals/baseline.rb +69 -0
  112. data/lib/insika/evals/golden.rb +172 -0
  113. data/lib/insika/evals/judge.rb +225 -0
  114. data/lib/insika/evals/pairwise.rb +178 -0
  115. data/lib/insika/evals/report.rb +115 -0
  116. data/lib/insika/evals/runner.rb +141 -0
  117. data/lib/insika/evals/transport.rb +178 -0
  118. data/lib/insika/event.rb +18 -0
  119. data/lib/insika/event_stream.rb +114 -0
  120. data/lib/insika/executor.rb +1680 -0
  121. data/lib/insika/frontmatter.rb +42 -0
  122. data/lib/insika/golden_store.rb +145 -0
  123. data/lib/insika/hooks.rb +48 -0
  124. data/lib/insika/http_client.rb +63 -0
  125. data/lib/insika/inbound_log.rb +84 -0
  126. data/lib/insika/llm_configurator.rb +99 -0
  127. data/lib/insika/llm_provider_store.rb +83 -0
  128. data/lib/insika/mcp_http_client.rb +67 -0
  129. data/lib/insika/mcp_store.rb +115 -0
  130. data/lib/insika/mcp_tool_ingestor.rb +143 -0
  131. data/lib/insika/memory_store.rb +93 -0
  132. data/lib/insika/message_origin.rb +76 -0
  133. data/lib/insika/middleware.rb +36 -0
  134. data/lib/insika/model_policy.rb +52 -0
  135. data/lib/insika/model_resolver.rb +176 -0
  136. data/lib/insika/model_selection.rb +114 -0
  137. data/lib/insika/onboarding.rb +208 -0
  138. data/lib/insika/outbox_store.rb +166 -0
  139. data/lib/insika/overlay_tool_registry.rb +103 -0
  140. data/lib/insika/pack.rb +102 -0
  141. data/lib/insika/pack_importer.rb +121 -0
  142. data/lib/insika/pending_action_store.rb +120 -0
  143. data/lib/insika/plugin/loader.rb +356 -0
  144. data/lib/insika/plugin.rb +35 -0
  145. data/lib/insika/policy/engine.rb +83 -0
  146. data/lib/insika/policy/policy.rb +120 -0
  147. data/lib/insika/policy_registry.rb +23 -0
  148. data/lib/insika/profile_source.rb +137 -0
  149. data/lib/insika/prompt_catalog.rb +61 -0
  150. data/lib/insika/queue_policy.rb +167 -0
  151. data/lib/insika/recovery.rb +127 -0
  152. data/lib/insika/refinement/candidate.rb +159 -0
  153. data/lib/insika/refinement/evidence_collector.rb +371 -0
  154. data/lib/insika/refinement/gate.rb +234 -0
  155. data/lib/insika/refinement/panel.rb +222 -0
  156. data/lib/insika/refinement/proposer.rb +262 -0
  157. data/lib/insika/refinement_store.rb +295 -0
  158. data/lib/insika/registry.rb +59 -0
  159. data/lib/insika/safety/config.rb +109 -0
  160. data/lib/insika/safety/detectors.rb +176 -0
  161. data/lib/insika/safety/factory.rb +102 -0
  162. data/lib/insika/safety/input_guardrail.rb +87 -0
  163. data/lib/insika/safety/moderator.rb +86 -0
  164. data/lib/insika/safety/output_filter.rb +79 -0
  165. data/lib/insika/safety/output_validator.rb +101 -0
  166. data/lib/insika/safety/safe_responses.rb +47 -0
  167. data/lib/insika/sandbox/boundary.rb +93 -0
  168. data/lib/insika/sandbox/docker.rb +74 -0
  169. data/lib/insika/sandbox/local.rb +33 -0
  170. data/lib/insika/sandbox/runner.rb +80 -0
  171. data/lib/insika/sandbox.rb +85 -0
  172. data/lib/insika/schema_guard.rb +147 -0
  173. data/lib/insika/secret_masking.rb +34 -0
  174. data/lib/insika/server/a2a/agent_card.rb +27 -0
  175. data/lib/insika/server/a2a/app.rb +112 -0
  176. data/lib/insika/server/a2a/client.rb +101 -0
  177. data/lib/insika/server/a2a/errors.rb +32 -0
  178. data/lib/insika/server/a2a/http.rb +42 -0
  179. data/lib/insika/server/a2a/message.rb +27 -0
  180. data/lib/insika/server/a2a/protocol.rb +45 -0
  181. data/lib/insika/server/a2a/remotes.rb +25 -0
  182. data/lib/insika/server/a2a/task_projection.rb +40 -0
  183. data/lib/insika/server/admin_auth.rb +29 -0
  184. data/lib/insika/server/app.rb +850 -0
  185. data/lib/insika/server/boot.rb +119 -0
  186. data/lib/insika/server/rack_app.rb +110 -0
  187. data/lib/insika/server/responses.rb +155 -0
  188. data/lib/insika/server/sse_body.rb +96 -0
  189. data/lib/insika/session_actor.rb +162 -0
  190. data/lib/insika/session_store.rb +143 -0
  191. data/lib/insika/settings_store.rb +154 -0
  192. data/lib/insika/shutdown.rb +125 -0
  193. data/lib/insika/skill_catalog.rb +113 -0
  194. data/lib/insika/skill_store.rb +79 -0
  195. data/lib/insika/steer_injector.rb +110 -0
  196. data/lib/insika/store.rb +52 -0
  197. data/lib/insika/stores/memory.rb +123 -0
  198. data/lib/insika/stores/sqlite.rb +183 -0
  199. data/lib/insika/studio/app.rb +1571 -0
  200. data/lib/insika/studio/assets/dist/application.css +1 -0
  201. data/lib/insika/studio/assets/dist/application.js +69 -0
  202. data/lib/insika/studio/forms.rb +340 -0
  203. data/lib/insika/studio/nav_icons.rb +31 -0
  204. data/lib/insika/studio/views/_message.erb +44 -0
  205. data/lib/insika/studio/views/agent_detail.erb +285 -0
  206. data/lib/insika/studio/views/agents.erb +63 -0
  207. data/lib/insika/studio/views/approvals.erb +41 -0
  208. data/lib/insika/studio/views/chats.erb +34 -0
  209. data/lib/insika/studio/views/evals.erb +83 -0
  210. data/lib/insika/studio/views/home.erb +72 -0
  211. data/lib/insika/studio/views/layout.erb +94 -0
  212. data/lib/insika/studio/views/login.erb +17 -0
  213. data/lib/insika/studio/views/mcp.erb +91 -0
  214. data/lib/insika/studio/views/not_found.erb +5 -0
  215. data/lib/insika/studio/views/playground.erb +47 -0
  216. data/lib/insika/studio/views/refinement.erb +234 -0
  217. data/lib/insika/studio/views/session.erb +62 -0
  218. data/lib/insika/studio/views/settings.erb +173 -0
  219. data/lib/insika/studio/views/skills.erb +86 -0
  220. data/lib/insika/studio/views/system_files.erb +65 -0
  221. data/lib/insika/studio/views/task.erb +105 -0
  222. data/lib/insika/studio/views/tasks.erb +33 -0
  223. data/lib/insika/studio/views/tool_edit.erb +107 -0
  224. data/lib/insika/studio/views/tools.erb +89 -0
  225. data/lib/insika/subagent_graph.rb +96 -0
  226. data/lib/insika/system_file_store.rb +96 -0
  227. data/lib/insika/task_actor.rb +128 -0
  228. data/lib/insika/task_store.rb +250 -0
  229. data/lib/insika/telemetry/pricing.rb +104 -0
  230. data/lib/insika/telemetry/recorder.rb +228 -0
  231. data/lib/insika/telemetry.rb +127 -0
  232. data/lib/insika/testing/store_contract.rb +270 -0
  233. data/lib/insika/token_estimator.rb +16 -0
  234. data/lib/insika/tool_assembly.rb +140 -0
  235. data/lib/insika/tool_catalog.rb +89 -0
  236. data/lib/insika/tool_definition.rb +518 -0
  237. data/lib/insika/tool_envelope.rb +140 -0
  238. data/lib/insika/tool_manifest.rb +218 -0
  239. data/lib/insika/tool_registry.rb +21 -0
  240. data/lib/insika/tool_store.rb +135 -0
  241. data/lib/insika/tool_trace_store.rb +92 -0
  242. data/lib/insika/tools/a2a_remote.rb +48 -0
  243. data/lib/insika/tools/agent_enum.rb +68 -0
  244. data/lib/insika/tools/concurrency.rb +54 -0
  245. data/lib/insika/tools/data_defined_tool.rb +220 -0
  246. data/lib/insika/tools/load_skill.rb +41 -0
  247. data/lib/insika/tools/remember.rb +53 -0
  248. data/lib/insika/tools/subagent.rb +75 -0
  249. data/lib/insika/tools/subagents.rb +77 -0
  250. data/lib/insika/tools/tool_search.rb +94 -0
  251. data/lib/insika/turn_output.rb +139 -0
  252. data/lib/insika/turn_state.rb +158 -0
  253. data/lib/insika/turn_timing.rb +56 -0
  254. data/lib/insika/usage_ledger.rb +47 -0
  255. data/lib/insika/version.rb +3 -1
  256. data/lib/insika/wiring/graph.rb +198 -0
  257. data/lib/insika/workflow.rb +185 -0
  258. data/lib/insika/workflow_registry.rb +33 -0
  259. data/lib/insika.rb +203 -4
  260. metadata +395 -8
data/docs/SKILLS.md ADDED
@@ -0,0 +1,98 @@
1
+ ---
2
+ title: Skills
3
+ parent: Build an agent
4
+ nav_order: 3
5
+ permalink: /skills/
6
+ ---
7
+
8
+ # Skills
9
+
10
+ A **skill** is a named playbook an agent loads **on demand**. It is a directory
11
+ with a `SKILL.md` file: YAML frontmatter (`name` + `description`) followed by a
12
+ Markdown body. The agent always sees the *name and description* of each skill it
13
+ is allowed; it pulls the full *body* into context only when a turn actually calls
14
+ for it. That is **progressive loading** — an agent can "know" twenty skills exist
15
+ while paying for the text of only the ones it opens.
16
+
17
+ See [`examples/skills/`](https://github.com/guizaols/insika/tree/main/examples/skills/) for a runnable one.
18
+
19
+ ## Format
20
+
21
+ ```markdown
22
+ ---
23
+ name: refunds # must equal the directory name
24
+ description: When and how to process a refund # the Level-1 trigger text
25
+ ---
26
+
27
+ <the full playbook body — loaded only on demand>
28
+ ```
29
+
30
+ Frontmatter is parsed tolerantly. A skill's canonical name is its directory name,
31
+ which the `name:` field must match.
32
+
33
+ ## Progressive loading: two levels
34
+
35
+ - **Level 1 — metadata only.** A context provider injects an `<available_skills>`
36
+ list (the name + one-line description of every allowed skill) into the system
37
+ prompt, telling the model to load a skill before acting on it. Cheap, and always
38
+ present for allowed skills.
39
+ - **Level 2 — the full body.** A built-in `load_skill` tool returns the skill body
40
+ on demand. It enforces the agent's skill allowlist and is wired **automatically**
41
+ whenever the agent has any allowed skills — you do not add it to `tools_allow`.
42
+
43
+ The Level-1 list is budgeted like any other context fragment
44
+ (see [Context](CONTEXT.md)); the Level-2 body only costs tokens on the turns that
45
+ open it.
46
+
47
+ ## Where skills live: the store, over a disk seed
48
+
49
+ - Skills live as **rows in SQLite** — one row per skill, holding the entire
50
+ `SKILL.md`, versioned (recent revisions are retained).
51
+ - On-disk `SKILL.md` files in configured roots are loaded as a **seed**, then the
52
+ store is **overlaid on top — the store wins**. A reload swaps the index
53
+ atomically, so edits take effect **without a restart**.
54
+
55
+ > ⚠️ **Committing a `.md` file to the repo does not make a skill show up on a
56
+ > running deployment.** The on-disk file is only a seed for a *fresh* box; a live
57
+ > box serves the store, and a deploy does not rewrite the database. Editing is a
58
+ > runtime operation (Studio / API / DSL), not a commit. See
59
+ > [Context](CONTEXT.md#the-volume).
60
+
61
+ ## Making a new skill "show up"
62
+
63
+ For an agent to actually use a skill, **both** conditions must hold:
64
+
65
+ 1. The skill exists as a row in the store (a `SKILL.md` written into it).
66
+ 2. The skill is in that agent's `skills` allowlist
67
+ (`nil` = all, `[]` = none, `[names]` = those — see
68
+ [Agents](AGENTS.md#the-allowlist-convention)).
69
+
70
+ Miss either and the skill is invisible: not in the store → nothing to load; not
71
+ in the allowlist → the model never sees it in `<available_skills>`.
72
+
73
+ Two ways to satisfy both:
74
+
75
+ - **Via a definition/pack import.** The import writes each skill directory into
76
+ the store and sets the agent's `skills` allowlist **authoritatively** from the
77
+ skills present — so a re-import that drops a skill also removes it. Keep the
78
+ definition complete.
79
+ - **Directly (Studio / API / DSL).** Write the skill (upserts the row and reloads
80
+ the catalog atomically — live immediately), then attach it to the agent(s) by
81
+ adding its name to the `skills` allowlist.
82
+
83
+ ## Verify it showed up
84
+
85
+ - In the Studio agent's Skills section, the skill is listed and allowed.
86
+ - In a turn, the skill appears in the `<available_skills>` list and the model can
87
+ `load_skill` it (the body loads on demand).
88
+ - If the model never mentions it → check the allowlist (condition 2). If
89
+ `load_skill` errors → the store row is missing or misnamed (condition 1); the
90
+ `name:` frontmatter must equal the directory name.
91
+
92
+ ## See also
93
+
94
+ - [Context](CONTEXT.md) — how the skills list is budgeted into a turn.
95
+ - [Agents](AGENTS.md) — the skills allowlist.
96
+ - [Tools](TOOLS.md) — `load_skill` and deferred-tool progressive disclosure.
97
+ - [Plugins](PLUGINS.md) — shipping skills inside a plugin, and the two extension tiers.
98
+ - [`examples/skills/`](https://github.com/guizaols/insika/tree/main/examples/skills/) — progressive loading, runnable.
data/docs/TOOLS.md ADDED
@@ -0,0 +1,302 @@
1
+ ---
2
+ title: Tools
3
+ parent: Build an agent
4
+ nav_order: 2
5
+ permalink: /tools/
6
+ ---
7
+
8
+ # Tools
9
+
10
+ A **tool** is a function the model can call inside a turn. Insika has three
11
+ kinds, and the distinction that matters is **who can change one at runtime**:
12
+
13
+ | | **Code tool** | **Data tool** | **MCP tool** |
14
+ |---|---|---|---|
15
+ | What | a Ruby class (`< RubyLLM::Tool`) | an HTTP call described by config, no Ruby | an MCP server's tool, ingested |
16
+ | Lives | in the deployment image | as a row in SQLite | as data-tool rows in SQLite |
17
+ | Editable at runtime | no (shipped in the image) | **yes** (DSL / API / manifest / Studio) | **yes** (re-ingest) |
18
+ | Reach for it when | logic must run in-process (file edit, shell, subagent) | calling an external HTTP API | adopting a whole MCP toolset at once |
19
+
20
+ **MCP tools are not a separate runtime type.** An MCP ingestor discovers an MCP
21
+ server's tools and turns each into an HTTP **data tool** that posts a JSON-RPC
22
+ `tools/call`, tagged with a `group` naming the source instance. (Only
23
+ HTTP-transport MCP servers are ingestible; stdio is rejected.)
24
+
25
+ Code tools **win name collisions** — you cannot register a data tool whose name
26
+ shadows a code tool.
27
+
28
+ ## Data tools: a tool is a row
29
+
30
+ A data tool is defined entirely by config — this is the operator-facing kind, and
31
+ the one you create and change without a rebuild. See
32
+ [`examples/data-tool/`](https://github.com/guizaols/insika/tree/main/examples/data-tool/) for a runnable one.
33
+
34
+ ```jsonc
35
+ {
36
+ "name": "search_products", // /\A[a-z][a-z0-9_]*\z/
37
+ "description": "Search the catalog", // required — this is what the model reads
38
+ "parameters": { /* JSON Schema, safe subset */ },
39
+ "request": {
40
+ "method": "POST", // GET | HEAD | POST | PUT | PATCH | DELETE
41
+ "url": "https://api.example.com/search",
42
+ "headers": { "X-Session": "{{ctx.chat_id}}",
43
+ "Authorization": "Bearer {{secret.api_token}}" },
44
+ "query": {}, "body": "…"
45
+ },
46
+ "response": { "extract": "json_path", "path": "$.results" },
47
+ "secret_headers": ["Authorization"],
48
+ "side_effect": true, "timeout": 30, "group": "catalog", "tags": []
49
+ }
50
+ ```
51
+
52
+ ### Parameters: the schema is the contract
53
+
54
+ `parameters` is **JSON Schema**, and it reaches the provider verbatim — it is the only
55
+ thing telling the model what shape to send. The engine never fills a gap in it.
56
+
57
+ For simple params there is a flat sugar (what the Studio's textarea and a hand-written
58
+ manifest accept), one line per param:
59
+
60
+ ```
61
+ cep | string | required | The ZIP code to look up
62
+ tags | array:string | optional | Labels to filter by
63
+ quantity | integer | required | How many
64
+ ```
65
+
66
+ Types are `string`, `number`, `integer`, `boolean`, and `array:<scalar>` for a list.
67
+ There is **no bare `array`**: a list without an item type is an incomplete declaration,
68
+ and it is rejected instead of being guessed at. A list of **objects** — the common
69
+ `[{query, filters}]` shape — cannot be written in the flat form at all; write the JSON
70
+ Schema, which is what the Studio field reads when the text starts with `{`:
71
+
72
+ ```jsonc
73
+ { "type": "object",
74
+ "properties": {
75
+ "query_filter_pairs": {
76
+ "type": "array",
77
+ "items": { "type": "object",
78
+ "properties": { "query": { "type": "string" },
79
+ "filters": { "type": "object", "properties": {} } },
80
+ "required": ["query"] } } },
81
+ "required": ["query_filter_pairs"] }
82
+ ```
83
+
84
+ **Arguments are checked against the schema at call time.** A call the schema does not
85
+ allow never becomes a request: it returns an `{ error: … }` naming the path
86
+ (`query_filter_pairs[0]: expected an object, got a string`), which the model reads and
87
+ retries against. Structure is strict; a scalar may arrive in its lossless string form
88
+ (`"2"`, `"true"`) and is never coerced — what the model sent is what the request carries.
89
+
90
+ **Placeholders** are resolved at turn time:
91
+
92
+ - `{{param}}` — a declared top-level parameter, filled from the model's call.
93
+ - `{{ctx.*}}` — turn context set **server-side, never by the model**: a closed set
94
+ of `chat_id`, `store_id`, `agent_id`, `tenant`. This is how a tool knows *which*
95
+ session/agent it is acting for without trusting the model.
96
+ - `{{secret.*}}` — allowed **only** inside a header named in `secret_headers`.
97
+ A secret placeholder anywhere else is rejected (it would leak unmasked). The
98
+ real secret value is injected at provision time and never lives on disk.
99
+
100
+ **Validation** happens on ingestion. Common rejections:
101
+
102
+ - `url` must be `http`/`https` — anything else is a 422.
103
+ - `parameters` is a **safe subset** of JSON Schema
104
+ (`object/array/string/number/integer/boolean`); `oneOf`/`anyOf`/`allOf`/`$ref`/
105
+ `if`/`then`/`else` are forbidden (not every provider supports them).
106
+ - `side_effect` defaults from the method (GET/HEAD → false, else true) and drives
107
+ checkpoint/replay semantics (a completed side-effecting tool is not re-run on
108
+ resume — see [Architecture](ARCHITECTURE.md#durability-checkpoints-and-resume)).
109
+
110
+ ### `halt_when`: when the answer is already out
111
+
112
+ Some tools do the work **and** deliver the news. A backend that subscribes a customer
113
+ and sends its own confirmation over the channel has already said everything there is to
114
+ say: if the model then writes "all set, you're subscribed!", the person gets the message
115
+ twice. The usual patch is to ask the model to stay quiet in the tool's instructions —
116
+ which works until the turn it doesn't, and the failure lands in front of a customer.
117
+
118
+ `halt_when` moves the decision from the prompt to the engine. It reads the tool's own
119
+ **response**, and when it matches, the turn ends right there — no further provider call:
120
+
121
+ ```jsonc
122
+ { "name": "subscribe_to_learning_path",
123
+ "request": { "method": "POST", "url": "https://app.example/subscribe" },
124
+ "halt_when": { "json_path": "tool_result.status", "equals": ["SUBSCRIBED"] } }
125
+ ```
126
+
127
+ By **result**, not by tool. The same call that goes silent on `SUBSCRIBED` must let the
128
+ model explain a `SUBSCRIPTION_FAILED` ("you are already enrolled") — one tool, two
129
+ endings, decided by what the backend actually returned.
130
+
131
+ - `json_path` is a dotted path into the parsed response body, and `equals` a list of
132
+ values compared **as strings** (a status is a label; JSON types vary by backend).
133
+ - It reads the **body**, independently of `response.extract` — which shapes what the
134
+ *model* sees, not what the engine decides on.
135
+ - It only fires on a **2xx**. An error response that happens to carry the value is a
136
+ failure, and a failure must reach the model.
137
+ - A non-JSON body or a missing path simply does not match: a turn never ends on a guess.
138
+
139
+ A halted turn keeps whatever the model had already streamed *before* the call (usually a
140
+ "let me get that for you") and adds nothing after it.
141
+
142
+ #### `say`: what the customer gets when the model wrote nothing first
143
+
144
+ The model does not always introduce the call. Then the lead-in is empty, and the turn
145
+ used to publish **nothing** — measured on a real store, two escalation turns in a row
146
+ delivered silence to the customer. `say` is the answer for that turn, and only that
147
+ turn: when there **is** a lead-in it still wins, because two messages for one
148
+ escalation is what `halt_when` exists to prevent.
149
+
150
+ It cannot be inferred. `json_path` + `equals` cannot supply it either — the matched
151
+ value is by definition one of the `equals` tokens, so publishing it would ship
152
+ `SUBSCRIBED` to a person as often as it ships a sentence. So you name it, in one of two
153
+ shapes:
154
+
155
+ ```jsonc
156
+ // the sentence the backend itself returned
157
+ "halt_when": { "json_path": "tool_result.status", "equals": ["SUBSCRIBED"],
158
+ "say": { "json_path": "tool_result.message" } }
159
+
160
+ // a literal the CHANNEL knows how to resolve
161
+ "halt_when": { "json_path": "tool_result", "equals": ["…"],
162
+ "say": { "text": "CALL_SUPPORT" } }
163
+ ```
164
+
165
+ The literal form replaces the usual workaround: instructing the model to emit a control
166
+ token and parsing it downstream. The token now comes from the **tool's contract**,
167
+ deterministically, instead of depending on the model complying with a sentence in a
168
+ prompt.
169
+
170
+ - Exactly one of `text` or `json_path` — two answers to "what does the customer get" is
171
+ a configuration nobody can read, so both (or neither) is refused at load.
172
+ - A `json_path` that does not resolve to a **string** publishes nothing: a hash or a
173
+ number reaching a customer as the answer is never what someone meant.
174
+ - Omit `say` and the behaviour is unchanged — a halt with no lead-in completes empty,
175
+ which is what a channel consumer drops.
176
+
177
+ `say` is declared on the **tool**, because what a backend answers is a property of that
178
+ backend, not of whoever calls it. Every agent sharing the tool gets the same value.
179
+
180
+ > The Studio's tool editor does not render this field (nor `group`/`tags`), but a save
181
+ > there **preserves** it — the form carries the stored values through instead of
182
+ > replacing the record with only what it shows.
183
+
184
+ ## Registering a tool
185
+
186
+ A tool appears in the Studio panel and enters an agent's tool-loop when it is
187
+ **registered** in the catalog **and** allowed by the agent's policy allowlist.
188
+ Four ways to write a data tool into the store — all **hot** (registry and catalog
189
+ reload, no restart):
190
+
191
+ 1. **DSL** — `data_tool(name:, …)` in a `Insika.agent { … }` block.
192
+ 2. **Studio** — the Tools panel editor.
193
+ 3. **Manifest** — `POST /v1/tools/manifest`. Partial failure is isolated: one
194
+ malformed tool becomes an `errors[]` entry; only a structural manifest error
195
+ fails the whole request. The response reports `{ version, created, updated, errors }`.
196
+ 4. **MCP ingestion** — import a server; each of its tools becomes a data tool.
197
+
198
+ ### The one gotcha: env templating is manifest-only
199
+
200
+ `{{env.*}}` (and `{{secret.*}}`) are substituted **at ingestion, on the manifest
201
+ path**. Other write paths do **not** resolve `{{env.*}}` — a literal
202
+ `{{env.API_URL}}` there fails the `http`/`https` URL check and 422s. Rule:
203
+ **manifest tools may template the URL with `{{env.*}}`; tools written any other
204
+ way must ship a literal URL.** `{{ctx.*}}` and `{{param}}` work everywhere (they
205
+ resolve at turn time, not ingestion).
206
+
207
+ ## Making it appear — and enter the tool-loop
208
+
209
+ 1. **Panel visibility** = registered in the catalog. Data tools are marked
210
+ editable; code tools are allow/deny only.
211
+ 2. **Per-agent exposure** is set from the same panel, or by the agent's allowlist.
212
+ 3. **Entering the tool-loop** is decided by the **policy allowlist**, not by tool
213
+ type: deny wins, otherwise the agent sees `tools_allow ∪ tools_allow_groups`
214
+ (or all, when both are absent). See [Agents](AGENTS.md#the-allowlist-convention).
215
+ 4. **Deferred tools** (`tools_deferred`) are *not* offered directly — they appear
216
+ as a short "available tools" list and the model must call `tool_search` to
217
+ enable one. This is progressive disclosure for large toolsets — see
218
+ [Context](CONTEXT.md).
219
+
220
+ ## Parallel tool calls
221
+
222
+ A model can ask for several tools in one step. By default the engine runs them one
223
+ at a time. Set `limits[:tool_concurrency]` above 1 (see
224
+ [Agents](AGENTS.md#tool_concurrency--parallel-tool-calls)) and the calls in that
225
+ batch run concurrently, **at most N in flight**, on the turn's own reactor — so
226
+ the wall-clock of a batch of slow data tools approaches the slowest call rather
227
+ than their sum. The cap covers every enveloped tool of the turn, including the
228
+ ones `tool_search` promotes mid-turn.
229
+
230
+ It applies only to what the *model* fans out. Two primitives already parallelize
231
+ deterministically and are unaffected: `spawn_subagents` (capped at 8 children) and
232
+ `Insika::Tools::Concurrency.gather` (fan-out inside one tool). System tools —
233
+ `tool_search`, `load_skill`, `remember`, `spawn_subagent` — are not enveloped and
234
+ so are not gated by the cap; they are trivial or capped on their own.
235
+
236
+ Turning it on changes three things, all of them worth knowing before you do:
237
+
238
+ - **`max_tool_calls` becomes approximate.** The limit is checked per call, but a
239
+ call that trips it does not stop its siblings — the whole batch finishes and the
240
+ turn then fails. With a cap of 4, up to 3 extra tools may have executed. The turn
241
+ still fails at the right boundary; the count is just no longer exact.
242
+ - **The transcript records results in completion order.** Providers key results by
243
+ `tool_call_id`, so the wire stays valid and persistence is faithful to what was
244
+ sent — but a replayed transcript no longer reads in call order.
245
+ - **`turn_timeout` can overrun by up to `tool_timeout`.** A turn deadline does not
246
+ cancel a tool call already in flight in a sibling fiber; it waits for it. Each
247
+ call is still bounded by its own `tool_timeout`, which is what bounds the
248
+ overrun. Serial execution is unaffected (there, the deadline lands directly in
249
+ the fiber running the tool).
250
+
251
+ Approvals and concurrency are mutually exclusive per turn — the approval gate wins
252
+ and the turn goes serial. That is a deadlock avoided, not a preference.
253
+
254
+ ## Egress: the SSRF guard (and its silent failure)
255
+
256
+ Data tools make outbound HTTP, so every call passes through the **EgressGuard**, a
257
+ Server-Side Request Forgery defense. The default posture is **strict: public
258
+ `https` only.** Three env vars widen it:
259
+
260
+ | Env | Effect |
261
+ |-----|--------|
262
+ | `INSIKA_EGRESS_HOSTS` | allowlist of hosts (CSV). The safe way to permit a specific backend. |
263
+ | `INSIKA_EGRESS_ALLOW_HTTP=1` | permit plain `http` — **loopback dev only** |
264
+ | `INSIKA_EGRESS_ALLOW_PRIVATE=1` | permit private/loopback IPs — **dev only** |
265
+
266
+ > ⚠️ **Egress failures are silent.** When a tool targets a blocked host (e.g. a
267
+ > plain-`http` localhost backend without the opt-ins), the guard turns the block
268
+ > into a `{ error: … }` returned **to the model** — the request never leaves the
269
+ > process, yet the stream still emits a tool call, so the model narrates a
270
+ > plausible failure and the conversation *looks* like it worked. You will not see
271
+ > an exception.
272
+ >
273
+ > **Always verify by the trace, never by the reply:** open the Studio session
274
+ > viewer — a healthy call shows the request, args, and the backend's `200`; a
275
+ > missing or errored call is almost always egress (host not in the allowlist, or
276
+ > `http`/private without the opt-in).
277
+
278
+ Egress is **orthogonal** to registration and allowlisting: a tool can be
279
+ registered, allowed, offered to the model, and still blocked at call time.
280
+
281
+ ## Troubleshooting: "the tool is missing"
282
+
283
+ Work down this checklist:
284
+
285
+ 1. **Registered?** Is it in the catalog (Studio Tools panel)? If not, the write
286
+ or import failed — check the manifest `errors[]`, and run `insika doctor`: a stored
287
+ definition that no longer validates is dropped from the catalog, and the
288
+ `data-tools` check is the only place that says so.
289
+ 2. **Allowed for this agent?** In `tools_allow` (or an allowed group), and not in
290
+ `tools_deny`?
291
+ 3. **Egress?** If it *appears and is called* but "fails", open the trace — a
292
+ blocked call is ~99% egress.
293
+ 4. **URL literal?** For non-manifest tools, an unresolved `{{env.*}}` would have
294
+ 422'd at import — re-check the definition.
295
+
296
+ ## See also
297
+
298
+ - [Agents](AGENTS.md) — allowlists, groups, and per-agent tool exposure.
299
+ - [Plugins](PLUGINS.md) — where a code tool comes from, and how to package one.
300
+ - [Security](SECURITY.md) — egress, sandbox, and approval gating together.
301
+ - [Architecture](ARCHITECTURE.md) — the tool-loop and side-effect checkpointing.
302
+ - [`examples/data-tool/`](https://github.com/guizaols/insika/tree/main/examples/data-tool/) — a runnable data tool + the egress note.
data/docs/WHY.md ADDED
@@ -0,0 +1,137 @@
1
+ ---
2
+ title: Why Insika
3
+ parent: Understand the idea
4
+ nav_order: 1
5
+ permalink: /why/
6
+ ---
7
+
8
+ # Why Insika
9
+
10
+ **Ruby is ready for production AI.** The narrative that you must reach for Python
11
+ to ship serious LLM applications is out of date. [RubyLLM](https://rubyllm.com)
12
+ gave the language a first-class, provider-agnostic LLM client — chat, tools,
13
+ streaming, embeddings, moderation — a genuinely valuable foundation that closed
14
+ the "Ruby can't do AI" gap. Insika is the next layer up: it takes those
15
+ primitives and makes an agent *dependable in production*.
16
+
17
+ Because **an agent without a harness is just a chat loop.** The model does the
18
+ reasoning, but everything that makes an agent survive contact with real traffic —
19
+ durable state, tools, guardrails, evals, operations — lives in the engine around
20
+ it. Those harnesses exist, mature, for other ecosystems. Ruby teams have mostly
21
+ been told to glue libraries together. They shouldn't have to.
22
+
23
+ Insika is an **agent runtime for Ruby**: the turn pipeline, the operational
24
+ surface, and the safety layer, in one deployable piece — behind an
25
+ OpenAI-Responses-compatible API. It stands on RubyLLM for the provider layer and
26
+ adds everything between a chat call and a production agent.
27
+
28
+ And it does it *ergonomically*. A complete, running agent is a few lines of Ruby —
29
+ the same speed-to-first-agent story Python teams tell, without leaving Ruby:
30
+
31
+ ```ruby
32
+ require "insika"
33
+
34
+ agent = Insika.agent("assistant") do
35
+ model "deepseek-chat"
36
+ provider :deepseek
37
+ instructions "You are a concise, friendly assistant."
38
+ end
39
+
40
+ puts agent.reply("hi, what can you do?") # one turn, in-process
41
+ agent.serve # ...or a full server: /studio + /v1/responses
42
+ ```
43
+
44
+ Every capability below is reachable the same way — see [`examples/`](https://github.com/guizaols/insika/tree/main/examples/),
45
+ one small runnable project per capability.
46
+
47
+ ## Four ways to run an agent
48
+
49
+ There are, roughly, four ways to put an LLM agent into production. They're not
50
+ wrong — they're different amounts of "build it yourself":
51
+
52
+ - **Raw SDK loop.** Call the provider SDK directly and hand-roll the
53
+ reason→act→observe loop. Minimal to start; everything operational is on you.
54
+ - **Assemble a framework.** Compose library pieces — a tool abstraction here, a
55
+ memory store there, your own persistence and moderation. Flexible, but you own
56
+ the integration and the gaps between the parts.
57
+ - **Hosted agent gateway.** A managed service runs the agent behind an API. You
58
+ get operations for free, but your agents, prompts, and conversation data live in
59
+ someone else's system.
60
+ - **An agent runtime (this project).** The loop, durability, tools, safety, and an
61
+ operations UI as one thing you deploy in your own infrastructure.
62
+
63
+ How they compare on what production actually demands:
64
+
65
+ | | Raw SDK loop | Assemble a framework | Hosted gateway | **Insika (runtime)** |
66
+ |---|:---:|:---:|:---:|:---:|
67
+ | Get a first agent talking | ✅ | ⚠️ | ✅ | ✅ |
68
+ | Add/change a tool without a redeploy | ❌ | ❌ | ⚠️ | ✅ |
69
+ | Durable, resumable turns (crash mid-conversation) | ❌ | ⚠️ | ✅ | ✅ |
70
+ | Content-safety guardrails built in | ❌ | ⚠️ | ⚠️ | ✅ |
71
+ | Evals / regression gating as a primitive | ❌ | ⚠️ | ⚠️ | ✅ |
72
+ | Operations UI included | ❌ | ❌ | ✅ | ✅ |
73
+ | Runs in your infrastructure (you own the data) | ✅ | ✅ | ❌ | ✅ |
74
+ | One container, no orchestrator to start | ✅ | ⚠️ | — | ✅ |
75
+
76
+ ✅ built in · ⚠️ possible, but you build/integrate it · ❌ not addressed · — n/a
77
+
78
+ ## What we do differently
79
+
80
+ - **Tools are data, not code.** Define a tool with a JSON Schema manifest and a
81
+ declarative binding, and the running agent picks it up — no rebuild, no redeploy.
82
+ Import whole toolsets from an MCP server or a manifest at runtime. In most setups
83
+ a new tool is a code change; here it's a row.
84
+ - **Durability is the default.** Every turn checkpoints; sessions and tasks recover
85
+ across restarts; continuous replication is one env var away, with a documented
86
+ restore drill. If the box dies mid-conversation, the conversation doesn't.
87
+ - **Safety is built in, not bolted on.** Input guardrails (prompt-injection and
88
+ abuse handling that fails gracefully *without* burning a model turn), content
89
+ moderation, PII/secret redaction in the output stream, and a post-turn validator
90
+ — configured, not hand-rolled, and opt-in per agent.
91
+ - **Evals are a primitive.** Golden conversations, LLM-as-judge, and baseline
92
+ gating run against the *same* API your users hit — so a prompt or model change
93
+ that regresses behavior fails loudly before it ships.
94
+ - **An operations UI is included.** Studio lets an operator manage agents, tools,
95
+ approvals, and traces — pause a task, approve a sensitive action, inspect a
96
+ tool-call trace — without writing code. Headless when you want it, operable when
97
+ you need it.
98
+ - **One container, one file.** SQLite in WAL mode with streaming replication. No
99
+ queue cluster, no orchestrator, no managed database required to start — and it
100
+ runs a real production workload today.
101
+ - **Fibers, not a thread per request.** An LLM turn is almost all *waiting* on the
102
+ provider. Insika runs on the [Async](https://github.com/socketry/async) fiber
103
+ scheduler and serves under [Falcon](https://github.com/socketry/falcon), so one
104
+ process handles thousands of concurrent turns on a handful of connections instead
105
+ of pinning a heavyweight thread per call. This is exactly the model
106
+ [RubyLLM's async guide](https://rubyllm.com/async/) recommends — *"Falcon is a
107
+ Ruby application server built on fibers; with Falcon, async just works"* — and it
108
+ composes for free, because RubyLLM's HTTP cooperates with Ruby's fiber scheduler.
109
+ It's a big part of why one box goes so far.
110
+ - **The engine gets out of the way.** On top of that concurrency, all the machinery
111
+ above adds **well under a millisecond of overhead per turn**; a single process
112
+ sustains thousands of turns per second of pure engine work. A neutral,
113
+ provider-free benchmark in the repo reproduces it with one command — see
114
+ [BENCHMARK.md](BENCHMARK.md). The rest of a turn's latency is the model, not us.
115
+ *(That number is engine overhead only — end-to-end latency is provider-bound and
116
+ not claimed here.)*
117
+
118
+ ## When a lighter tool is the right call
119
+
120
+ Insika is for when an agent has to run **unattended, durably, and safely, in
121
+ production**. If that's not you yet, reach for less:
122
+
123
+ - If you only need LLM calls with tools in Ruby and nothing operational around
124
+ them, **[RubyLLM](https://rubyllm.com) alone is excellent** — and it's exactly
125
+ what Insika builds on. Reach for Insika when "nothing operational around them"
126
+ stops being true: when the agent has to run unattended, recover from crashes,
127
+ enforce safety, and be operable by someone who isn't you.
128
+ - If your team is committed to another language ecosystem, use a runtime native to
129
+ it — the ideas here travel, the code doesn't.
130
+ - If you want a minimal terminal coding assistant rather than a product platform,
131
+ a single-purpose CLI agent will be simpler.
132
+
133
+ ## See also
134
+
135
+ - [README](https://github.com/guizaols/insika#readme) — quickstart and the drop-in `/v1/responses` API.
136
+ - [BENCHMARK.md](BENCHMARK.md) — the neutral, reproducible engine benchmark.
137
+ - [OBSERVABILITY.md](OBSERVABILITY.md) — OpenTelemetry tracing (opt-in).