insika 0.2.0 → 0.7.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (300) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGELOG.md +325 -17
  3. data/README.md +52 -11
  4. data/bin/insika +895 -8
  5. data/bin/insika-router +87 -0
  6. data/docs/AGENTS.md +229 -163
  7. data/docs/API.md +56 -0
  8. data/docs/ARCHITECTURE.md +3 -2
  9. data/docs/ARTIFACTS.md +95 -0
  10. data/docs/BENCHMARK.md +2 -2
  11. data/docs/CHANNELS.md +111 -21
  12. data/docs/CONTEXT.md +132 -19
  13. data/docs/DEMO.md +80 -0
  14. data/docs/DEPLOY.md +75 -4
  15. data/docs/EMBEDDING.md +1 -1
  16. data/docs/EVALS.md +146 -2
  17. data/docs/FACTS.md +135 -0
  18. data/docs/HARVEST.md +116 -0
  19. data/docs/KNOWLEDGE.md +290 -0
  20. data/docs/LOADTEST.md +16 -10
  21. data/docs/MEDIA.md +128 -0
  22. data/docs/OBSERVABILITY.md +65 -6
  23. data/docs/OUTCOMES.md +137 -0
  24. data/docs/PLUGINS.md +51 -6
  25. data/docs/POLICY.md +216 -0
  26. data/docs/REFINEMENT.md +17 -12
  27. data/docs/RELEASING.md +27 -0
  28. data/docs/ROUTER.md +213 -0
  29. data/docs/RUNNING-LOCAL.md +6 -6
  30. data/docs/SCHEDULING.md +121 -0
  31. data/docs/SECURITY.md +79 -2
  32. data/docs/SKILLS.md +13 -4
  33. data/docs/SOAK.md +127 -0
  34. data/docs/TEMPLATES.md +134 -0
  35. data/docs/TOOLS.md +220 -27
  36. data/docs/WHY.md +1 -1
  37. data/docs/WORKFLOWS.md +2 -2
  38. data/docs/_includes/head_custom.html +5 -0
  39. data/docs/_includes/title.html +13 -0
  40. data/docs/_sass/color_schemes/insika.scss +32 -0
  41. data/docs/_sass/custom/custom.scss +199 -0
  42. data/docs/_sass/custom/setup.scss +26 -0
  43. data/docs/assets/img/favicon.svg +7 -0
  44. data/docs/assets/img/insika-mark.svg +7 -0
  45. data/docs/core-concepts.md +21 -0
  46. data/docs/domain.md +115 -0
  47. data/docs/improve.md +20 -0
  48. data/docs/index.md +9 -6
  49. data/docs/integrate.md +20 -0
  50. data/docs/operate.md +13 -6
  51. data/docs/prompts/ADD-TOOL.md +118 -0
  52. data/docs/prompts/DIAGNOSE-TURN.md +65 -0
  53. data/docs/prompts/GO-LIVE.md +138 -0
  54. data/docs/prompts/RUN-EXAMPLES.md +70 -0
  55. data/docs/reference.md +19 -0
  56. data/docs/ship.md +10 -2
  57. data/docs/start-here.md +18 -0
  58. data/lib/insika/agent_profile.rb +202 -10
  59. data/lib/insika/artifact_signing.rb +82 -0
  60. data/lib/insika/artifact_store.rb +160 -0
  61. data/lib/insika/balloon_splitter.rb +102 -0
  62. data/lib/insika/budget_ledger.rb +34 -3
  63. data/lib/insika/cache_series_store.rb +49 -0
  64. data/lib/insika/channel_delivery.rb +119 -14
  65. data/lib/insika/channels/relay.rb +77 -3
  66. data/lib/insika/channels/web.rb +2 -2
  67. data/lib/insika/chat_builder.rb +105 -2
  68. data/lib/insika/checkpoint_store.rb +16 -0
  69. data/lib/insika/commands/agent_payload.rb +3 -3
  70. data/lib/insika/commands/backfill_knowledge.rb +145 -0
  71. data/lib/insika/commands/cancel_followup.rb +49 -0
  72. data/lib/insika/commands/delete_artifact.rb +35 -0
  73. data/lib/insika/commands/delete_concept.rb +34 -0
  74. data/lib/insika/commands/delete_mcp.rb +6 -2
  75. data/lib/insika/commands/delete_tenant_data.rb +107 -0
  76. data/lib/insika/commands/export_customer_memory.rb +48 -0
  77. data/lib/insika/commands/forget_customer.rb +117 -0
  78. data/lib/insika/commands/freeze_funnel_baseline.rb +113 -0
  79. data/lib/insika/commands/gate_harvest.rb +138 -0
  80. data/lib/insika/commands/gate_refinement.rb +1 -1
  81. data/lib/insika/commands/judge_shadow_pairs.rb +124 -0
  82. data/lib/insika/commands/memory_forget_fact.rb +20 -4
  83. data/lib/insika/commands/memory_put_fact.rb +23 -4
  84. data/lib/insika/commands/promote_harvest.rb +130 -0
  85. data/lib/insika/commands/record_outcome.rb +46 -0
  86. data/lib/insika/commands/record_shadow_reply.rb +68 -0
  87. data/lib/insika/commands/refresh_mcp_tools.rb +47 -0
  88. data/lib/insika/commands/reject_harvest.rb +38 -0
  89. data/lib/insika/commands/resolve_proposal.rb +108 -0
  90. data/lib/insika/commands/restore_concept.rb +34 -0
  91. data/lib/insika/commands/revoke_contact.rb +49 -0
  92. data/lib/insika/commands/rollback_harvest.rb +86 -0
  93. data/lib/insika/commands/run_distillation.rb +186 -0
  94. data/lib/insika/commands/run_harvest.rb +393 -0
  95. data/lib/insika/commands/seed_demo_data.rb +31 -0
  96. data/lib/insika/commands/send_message.rb +103 -6
  97. data/lib/insika/commands/session_purge.rb +67 -0
  98. data/lib/insika/commands/upsert_mcp.rb +6 -3
  99. data/lib/insika/commands/write_concept.rb +57 -0
  100. data/lib/insika/contact_store.rb +183 -0
  101. data/lib/insika/context/builder.rb +21 -3
  102. data/lib/insika/context/fragment.rb +7 -3
  103. data/lib/insika/context/priority.rb +5 -0
  104. data/lib/insika/context/provider.rb +17 -3
  105. data/lib/insika/context/providers/briefing.rb +96 -0
  106. data/lib/insika/context/providers/knowledge.rb +108 -0
  107. data/lib/insika/context/providers/memory.rb +16 -7
  108. data/lib/insika/context/providers/prompt.rb +57 -23
  109. data/lib/insika/context/providers/skill.rb +2 -0
  110. data/lib/insika/context/providers/tool_search.rb +2 -0
  111. data/lib/insika/context_trace_store.rb +38 -2
  112. data/lib/insika/cron.rb +189 -0
  113. data/lib/insika/demo/agent_attrs.rb +43 -0
  114. data/lib/insika/demo/golden_cases.rb +81 -0
  115. data/lib/insika/demo/seeder.rb +336 -0
  116. data/lib/insika/distill.rb +224 -0
  117. data/lib/insika/distill_engine.rb +169 -0
  118. data/lib/insika/doctor.rb +889 -11
  119. data/lib/insika/dsl/definition.rb +3 -2
  120. data/lib/insika/dsl/runtime.rb +69 -82
  121. data/lib/insika/dsl/server_boot.rb +92 -1
  122. data/lib/insika/dsl/system.rb +10 -2
  123. data/lib/insika/dsl.rb +186 -5
  124. data/lib/insika/edge_limiter.rb +28 -7
  125. data/lib/insika/env_schema.rb +25 -3
  126. data/lib/insika/errors.rb +11 -0
  127. data/lib/insika/evals/assertions.rb +3 -2
  128. data/lib/insika/evals/golden.rb +41 -4
  129. data/lib/insika/evals/judge.rb +47 -2
  130. data/lib/insika/evals/pairwise.rb +29 -6
  131. data/lib/insika/evals/persona.rb +98 -0
  132. data/lib/insika/evals/runner.rb +9 -0
  133. data/lib/insika/evals/simulator.rb +225 -0
  134. data/lib/insika/evals/transport.rb +83 -1
  135. data/lib/insika/event_stream.rb +10 -0
  136. data/lib/insika/evidence.rb +183 -0
  137. data/lib/insika/executor.rb +869 -76
  138. data/lib/insika/followup_engine.rb +207 -0
  139. data/lib/insika/followup_policy.rb +198 -0
  140. data/lib/insika/followup_store.rb +306 -0
  141. data/lib/insika/funnel_declaration.rb +106 -0
  142. data/lib/insika/funnel_fold.rb +179 -0
  143. data/lib/insika/funnel_store.rb +163 -0
  144. data/lib/insika/golden_store.rb +17 -2
  145. data/lib/insika/grounding/matcher.rb +69 -0
  146. data/lib/insika/grounding.rb +44 -0
  147. data/lib/insika/harvest/conversion_gate.rb +159 -0
  148. data/lib/insika/harvest/criterion.rb +98 -0
  149. data/lib/insika/harvest/gate.rb +194 -0
  150. data/lib/insika/harvest/negative_list.rb +199 -0
  151. data/lib/insika/harvest.rb +241 -0
  152. data/lib/insika/harvest_engine.rb +193 -0
  153. data/lib/insika/harvest_store.rb +548 -0
  154. data/lib/insika/knowledge.rb +680 -0
  155. data/lib/insika/knowledge_store.rb +140 -0
  156. data/lib/insika/mcp_client.rb +94 -0
  157. data/lib/insika/mcp_json.rb +74 -0
  158. data/lib/insika/mcp_live_tool.rb +43 -0
  159. data/lib/insika/mcp_store.rb +98 -26
  160. data/lib/insika/mcp_tool_ingestor.rb +30 -8
  161. data/lib/insika/mcp_tool_registry.rb +100 -0
  162. data/lib/insika/media.rb +382 -0
  163. data/lib/insika/memory_audit_store.rb +85 -0
  164. data/lib/insika/memory_store.rb +264 -23
  165. data/lib/insika/message_origin.rb +7 -2
  166. data/lib/insika/middleware.rb +9 -0
  167. data/lib/insika/model_visible.rb +87 -0
  168. data/lib/insika/model_visible_trace_store.rb +66 -0
  169. data/lib/insika/onboarding.rb +22 -1
  170. data/lib/insika/outbox_store.rb +42 -4
  171. data/lib/insika/outcome_store.rb +147 -0
  172. data/lib/insika/overlay_tool_registry.rb +37 -17
  173. data/lib/insika/packaging.rb +163 -0
  174. data/lib/insika/parity/criterion.rb +79 -0
  175. data/lib/insika/parity/verdict.rb +318 -0
  176. data/lib/insika/prefix_fingerprint.rb +58 -0
  177. data/lib/insika/profile_source.rb +29 -1
  178. data/lib/insika/prompt_catalog.rb +10 -0
  179. data/lib/insika/proposal_store.rb +271 -0
  180. data/lib/insika/queue_policy.rb +4 -1
  181. data/lib/insika/refinement/proposer.rb +1 -1
  182. data/lib/insika/reliability.rb +32 -6
  183. data/lib/insika/retention.rb +316 -0
  184. data/lib/insika/router/app.rb +157 -0
  185. data/lib/insika/router/backend_pool.rb +98 -0
  186. data/lib/insika/router/hash_ring.rb +55 -0
  187. data/lib/insika/router/proxy_body.rb +34 -0
  188. data/lib/insika/router/session_key.rb +54 -0
  189. data/lib/insika/router.rb +18 -0
  190. data/lib/insika/routing.rb +101 -0
  191. data/lib/insika/safety/config.rb +44 -4
  192. data/lib/insika/safety/corpus.rb +255 -0
  193. data/lib/insika/safety/detectors.rb +32 -113
  194. data/lib/insika/safety/factory.rb +15 -2
  195. data/lib/insika/safety/grounding_enforcer.rb +59 -0
  196. data/lib/insika/safety/grounding_validator.rb +49 -0
  197. data/lib/insika/safety/input_guardrail.rb +1 -1
  198. data/lib/insika/safety/moderator.rb +1 -1
  199. data/lib/insika/safety/output_filter.rb +9 -5
  200. data/lib/insika/safety/output_validator.rb +11 -5
  201. data/lib/insika/schedule.rb +177 -0
  202. data/lib/insika/schedule_engine.rb +314 -0
  203. data/lib/insika/schedule_store.rb +208 -0
  204. data/lib/insika/schema_guard.rb +35 -0
  205. data/lib/insika/server/app.rb +247 -17
  206. data/lib/insika/server/rack_app.rb +21 -1
  207. data/lib/insika/server/responses.rb +40 -1
  208. data/lib/insika/session_actor.rb +8 -4
  209. data/lib/insika/session_store.rb +65 -2
  210. data/lib/insika/settings_store.rb +10 -0
  211. data/lib/insika/shadow_pair_store.rb +258 -0
  212. data/lib/insika/skill_catalog.rb +16 -0
  213. data/lib/insika/soak/envelope.rb +140 -0
  214. data/lib/insika/soak/report.rb +392 -0
  215. data/lib/insika/soak/runner.rb +554 -0
  216. data/lib/insika/steer_injector.rb +21 -10
  217. data/lib/insika/store.rb +10 -1
  218. data/lib/insika/stores/memory.rb +6 -0
  219. data/lib/insika/stores/sqlite.rb +8 -0
  220. data/lib/insika/studio/app.rb +1437 -54
  221. data/lib/insika/studio/assets/dist/application.css +1 -1
  222. data/lib/insika/studio/assets/dist/application.js +23 -23
  223. data/lib/insika/studio/assets/dist/favicon.svg +6 -0
  224. data/lib/insika/studio/forms.rb +308 -10
  225. data/lib/insika/studio/nav_icons.rb +19 -1
  226. data/lib/insika/studio/views/_agent_tab_cache.erb +25 -0
  227. data/lib/insika/studio/views/_agent_tab_config.erb +514 -0
  228. data/lib/insika/studio/views/_agent_tab_history.erb +24 -0
  229. data/lib/insika/studio/views/_agent_tab_loops.erb +54 -0
  230. data/lib/insika/studio/views/_agent_tab_memory.erb +51 -0
  231. data/lib/insika/studio/views/_agent_tab_outcomes.erb +31 -0
  232. data/lib/insika/studio/views/_agent_tab_prompts.erb +108 -0
  233. data/lib/insika/studio/views/_agent_tab_skills.erb +38 -0
  234. data/lib/insika/studio/views/_agents_master.erb +44 -0
  235. data/lib/insika/studio/views/_message.erb +49 -32
  236. data/lib/insika/studio/views/agent_detail.erb +62 -278
  237. data/lib/insika/studio/views/agents.erb +71 -54
  238. data/lib/insika/studio/views/approvals.erb +4 -1
  239. data/lib/insika/studio/views/artifact.erb +23 -0
  240. data/lib/insika/studio/views/artifacts.erb +59 -0
  241. data/lib/insika/studio/views/chats.erb +4 -1
  242. data/lib/insika/studio/views/customer.erb +94 -0
  243. data/lib/insika/studio/views/customers.erb +32 -0
  244. data/lib/insika/studio/views/evals.erb +6 -3
  245. data/lib/insika/studio/views/facts.erb +133 -0
  246. data/lib/insika/studio/views/followups.erb +125 -0
  247. data/lib/insika/studio/views/funnel.erb +106 -0
  248. data/lib/insika/studio/views/harvest.erb +234 -0
  249. data/lib/insika/studio/views/home.erb +106 -66
  250. data/lib/insika/studio/views/knowledge.erb +123 -0
  251. data/lib/insika/studio/views/layout.erb +15 -11
  252. data/lib/insika/studio/views/mcp.erb +174 -80
  253. data/lib/insika/studio/views/parity.erb +147 -0
  254. data/lib/insika/studio/views/playground.erb +7 -1
  255. data/lib/insika/studio/views/session.erb +233 -124
  256. data/lib/insika/studio/views/settings.erb +41 -1
  257. data/lib/insika/studio/views/skills.erb +1 -2
  258. data/lib/insika/studio/views/system_files.erb +1 -1
  259. data/lib/insika/studio/views/task.erb +13 -0
  260. data/lib/insika/studio/views/tasks.erb +4 -1
  261. data/lib/insika/studio/views/tools.erb +24 -10
  262. data/lib/insika/task_store.rb +21 -1
  263. data/lib/insika/templates/browser-agent/README.md +36 -0
  264. data/lib/insika/templates/browser-agent/agent.rb +49 -0
  265. data/lib/insika/templates/daily-digest/README.md +38 -0
  266. data/lib/insika/templates/daily-digest/agent.rb +77 -0
  267. data/lib/insika/templates/repo-explorer/README.md +36 -0
  268. data/lib/insika/templates/repo-explorer/agent.rb +45 -0
  269. data/lib/insika/templates/research-analyst/README.md +26 -0
  270. data/lib/insika/templates/research-analyst/agent.rb +58 -0
  271. data/lib/insika/templates/review-panel/README.md +20 -0
  272. data/lib/insika/templates/review-panel/agent.rb +50 -0
  273. data/lib/insika/templates/travel-planner/README.md +35 -0
  274. data/lib/insika/templates/travel-planner/agent.rb +87 -0
  275. data/lib/insika/templates.rb +112 -0
  276. data/lib/insika/testing/store_contract.rb +27 -6
  277. data/lib/insika/tick.rb +47 -11
  278. data/lib/insika/timezone.rb +45 -0
  279. data/lib/insika/tool_definition.rb +17 -7
  280. data/lib/insika/tool_envelope.rb +69 -0
  281. data/lib/insika/tool_manifest.rb +5 -1
  282. data/lib/insika/tools/data_defined_tool.rb +10 -0
  283. data/lib/insika/tools/generate_image.rb +89 -0
  284. data/lib/insika/tools/load_knowledge.rb +74 -0
  285. data/lib/insika/tools/run_persona_eval.rb +328 -0
  286. data/lib/insika/tools/save_artifact.rb +95 -0
  287. data/lib/insika/tools/schedule_followup.rb +164 -0
  288. data/lib/insika/tools/tts.rb +47 -0
  289. data/lib/insika/tools/update_briefing.rb +126 -0
  290. data/lib/insika/turn_output.rb +1 -1
  291. data/lib/insika/turn_state.rb +49 -1
  292. data/lib/insika/turn_timing.rb +22 -2
  293. data/lib/insika/version.rb +1 -1
  294. data/lib/insika/vitals.rb +84 -0
  295. data/lib/insika/wiring/graph.rb +472 -13
  296. data/lib/insika/wiring/graph_chat.rb +102 -0
  297. data/lib/insika.rb +149 -3
  298. metadata +181 -5
  299. data/docs/build.md +0 -14
  300. data/docs/understand.md +0 -10
data/docs/OUTCOMES.md ADDED
@@ -0,0 +1,137 @@
1
+ ---
2
+ title: Outcomes and follow-ups
3
+ parent: Improve
4
+ nav_order: 3
5
+ permalink: /outcomes/
6
+ ---
7
+
8
+ # Outcomes and follow-ups
9
+
10
+ ## Outcomes — business results over real traffic
11
+
12
+ The engine measures what it is told to measure. The operator or the integration
13
+ records a conversation's business outcome after the fact — `conversion`,
14
+ `escalation`, `deflected`, anything, optionally with a monetary `value`:
15
+
16
+ ```bash
17
+ curl -X POST /v1/outcomes -H "Authorization: Bearer $TOKEN" \
18
+ -d '{ "agent": "store-support", "session_id": "chat-7",
19
+ "outcome": "conversion", "value": 129.9 }'
20
+ ```
21
+
22
+ The endpoint is **additive and outside the response contract** — the turn never
23
+ knows or cares; the engine transports the outcome and never interprets it (what
24
+ "conversion" means is yours). Records are tenant-stamped (a tenant principal
25
+ writes and reads only its own), and `GET /v1/outcomes?agent=` serves the last
26
+ outcome per agent plus the per-day series — the last-outcome pill on the Studio
27
+ agent grid, and the per-day series on the agent detail.
28
+
29
+ ### The outcome funnel
30
+
31
+ A store's funnel is pack data on the agent — the engine folds outcomes into
32
+ the **declared** stages, and never hard-codes one itself (the stage vocabulary
33
+ is the forge's):
34
+
35
+ ```ruby
36
+ agent = Insika.agent("store-support") do
37
+ instructions "…"
38
+ funnel stages: %w[greeted qualified cart paid],
39
+ advance_on: { "abandoned_cart" => "cart", "pix_paid" => "paid" },
40
+ primary: "paid", attribution_window: "72h"
41
+ end
42
+ ```
43
+
44
+ The fold contract:
45
+
46
+ - **Tick-driven, cumulative event counts on the declared order.** An outcome of
47
+ kind K means the session *reached* `advance_on[K]`; the fold increments
48
+ `stages[0..index]` for the reached stage. A per-stage-complete integration
49
+ and a terminal-event integration therefore produce identical counts — a
50
+ session that paid also emitted the earlier events. A duplicate event
51
+ double-counts (the integration's defect, not the engine's); **do not declare
52
+ a stage off the linear path** (a "handoff" stage would be inflated by every
53
+ later event). Counts are **event counts, not distinct sessions** — the
54
+ baseline is events-based.
55
+ - **Idempotent**: a per-pair `{at, ids}` cursor inside one transaction; a crash
56
+ mid-fold never double counts, and a second pass folds only what is new.
57
+ - **The attribution window is carried data, never computed** — `72h` is
58
+ validated, rendered, and copied into the baseline snapshot; causal
59
+ attribution stays human.
60
+ - **The baseline freeze** (Studio > Funnel, or `:freeze_funnel_baseline` on the
61
+ bus) sums the folded cells over a span of **≥ 28 days** (shorter spans are
62
+ refused) into one current snapshot per `(tenant, agent)` — the number
63
+ the follow-up A/B and the harvest promotion gate compare against.
64
+ - **Malformed declarations never crash the tick**: the fold skips them, the
65
+ doctor names the defect, the Studio shows nothing until it is fixed.
66
+ - Vocabulary note: in the gem this is the **outcome funnel** — the stage names
67
+ are the forge's, and a bare install (no `funnel:` on any agent) shows no
68
+ funnel and no stage names at all.
69
+
70
+ ## Follow-ups — the seller who comes back
71
+
72
+ The agent can book a follow-up with a customer at a future time — "te chamo
73
+ amanhã se o PIX não cair" said in-conversation and meant. The engine fires the
74
+ synthetic turn on its own tick, with consent and without spam. Everything is
75
+ pack data on the profile:
76
+
77
+ ```ruby
78
+ agent = Insika.agent("store-support") do
79
+ instructions "…"
80
+ followup arm: "schedule",
81
+ policy: { quiet_hours: { timezone: "America/Sao_Paulo",
82
+ start: "21:30", end: "09:00" },
83
+ max_frequency: "2/24h", # N outbound per window, per customer
84
+ cancel_keywords: ["não quero mais contato"],
85
+ silence_after_sends: 3 } # N fires without a reply -> :unavailable
86
+ end
87
+ ```
88
+
89
+ The pieces:
90
+
91
+ - **`schedule(at:, reason:)`** — a built-in tool the agent calls when the
92
+ customer agrees to be contacted again (a product, a cart, a pending payment).
93
+ The call itself IS the consent record — recorded without ever lifting
94
+ `:unavailable` or resetting the silence counter (ONLY a customer message
95
+ reopens, so a re-booking inside a follow-up turn cannot clear the silence
96
+ protection). `cancel_followup(id:)` is the sibling. A customer who opted out
97
+ can never be rescheduled.
98
+ - **Contact state per customer** — `granted | revoked | unavailable` in a
99
+ durable cell per `(tenant, customer)`. Only `granted` may be messaged;
100
+ `revoked` is immediate and permanent until the customer speaks again;
101
+ `unavailable` means silence ≠ refusal — the engine stops firing after
102
+ `silence_after_sends` unanswered sends, and ANY customer message reopens.
103
+ The policy's `cancel_keywords` are matched on every inbound message: a
104
+ match revokes the contact and cancels its pending follow-ups in one
105
+ transaction.
106
+ - **Firing is the tick's third duty** — the engine claims the due records
107
+ (one per claim window, at-most-once across workers), applies the policy in
108
+ force AT FIRE TIME (contact state, quiet hours, dedup per
109
+ `(customer, reason)`, frequency ceiling) and either enqueues the synthetic
110
+ turn or marks the record `blocked` with the failing rule — auditable, never
111
+ silent. Blocking happens at fire time, never at schedule time: the schedule
112
+ is a promise made in-conversation, and only the policy in force then may
113
+ revoke it.
114
+ - **The synthetic turn** — a first-class inbound turn stamped
115
+ `origin: "scheduled"` (a refinement read can never mistake the engine's
116
+ kick for the customer repeating themselves), delivered through the full
117
+ pipeline on the channel the conversation came in on. It skips the edge's
118
+ ENTRY rate/token checks like a resume does — a follow-up she agreed to must
119
+ not receive the rate-limit reply; its usage still lands on the ledger.
120
+ - **The Follow-ups page** (Studio) — per agent: the pending/fired/cancelled/
121
+ blocked records (blocked rows carry the reason), the read-only policy
122
+ summary and the A/B card: per arm, `sent` vs `conversions` (against the
123
+ frozen baseline) vs `opt-outs`. The only mutations — cancel a pending
124
+ record, force-revoke a contact — go through bus commands.
125
+ - **LGPD** — the records and cells die with the customer (`forget_customer`),
126
+ the tenant (`delete_tenant_data`) and age out under the same
127
+ `retention_days` sweep as the rest of the footprint.
128
+
129
+ Absent `followup:` = the feature is off for that agent — no tools wired, no
130
+ records, byte-identical turns. The A/B against an existing cron is an
131
+ operator experiment: the engine only keeps the records and the read card (the
132
+ cron arm writes through the same store class with its own `arm` label).
133
+
134
+ ## See also
135
+
136
+ - [Evals](EVALS.md) — the cases that grade an agent before traffic does.
137
+ - [Refinement](REFINEMENT.md) — reading an agent's own traffic back as a report.
data/docs/PLUGINS.md CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  title: Plugins
3
- parent: Build an agent
4
- nav_order: 7
3
+ parent: Integrate
4
+ nav_order: 5
5
5
  permalink: /plugins/
6
6
  ---
7
7
 
@@ -131,16 +131,34 @@ keyword is itself an error rather than being silently ignored.
131
131
 
132
132
  ### Discovery and enabling
133
133
 
134
+ The loader runs **at boot, in every composition root** — the server wirings and
135
+ a DSL-run agent alike — single-threaded, before the first request is accepted.
136
+ A plugin therefore either exists for the whole life of the process or not at
137
+ all; there is no half-loaded state a turn can observe.
138
+
134
139
  Plugins come from three kinds of root, and they differ in **who has to say yes**:
135
140
 
136
141
  | Root | How it is found | Enabled by default |
137
142
  |---|---|---|
138
143
  | **Gem** | the gem calls `Insika::Plugin.announce(__dir__)` when its `lib/` loads | **yes** — installing it is the consent |
139
- | **Workspace** | a directory in the deployment's plugin roots | no — must be listed in `enabled:` |
140
- | **Bundled** | `plugins/` in this repo | no — must be listed in `enabled:` |
144
+ | **Workspace** | the directory named by `INSIKA_PLUGIN_DIR` | no — list the id in `INSIKA_PLUGINS` |
145
+ | **Bundled** | `plugins/` in this repo | no — list the id in `INSIKA_PLUGINS` |
146
+
147
+ ```sh
148
+ INSIKA_PLUGIN_DIR=/srv/insika/plugins # workspace root to scan
149
+ INSIKA_PLUGINS=weather,acme # ids to enable from workspace/bundled roots
150
+ INSIKA_PLUGINS_DISABLED=insika-code # ids that never load, no matter what
151
+ ```
141
152
 
142
- `disabled:` is an absolute veto: an id listed there never loads, even if it is
143
- also in `enabled:` (deny wins, the same rule as every allowlist in the engine).
153
+ `INSIKA_PLUGINS_DISABLED` is an absolute veto: an id listed there never loads,
154
+ even if it is enabled or shipped by an installed gem (deny wins, the same rule
155
+ as every allowlist in the engine).
156
+
157
+ When two roots ship the same `id`, precedence is workspace → gems → bundled:
158
+ the first root wins, so an operator's local copy always overrides an installed
159
+ one. Skills and prompts a plugin ships join the catalogs at the **lowest**
160
+ precedence for the same reason — a workspace or Studio-authored skill beats a
161
+ plugin's same-named one.
144
162
 
145
163
  A gem announces itself explicitly — Insika never scans the load path or your
146
164
  installed gems:
@@ -173,6 +191,33 @@ config_schema:
173
191
  The manifest is committed; the secret is not. This mirrors how data tools handle
174
192
  `{{secret.*}}`.
175
193
 
194
+ ## Why it is built this way
195
+
196
+ Five decisions carry the whole design; knowing them explains every behavior
197
+ above.
198
+
199
+ 1. **Data before code.** Most agent runtimes make code the unit of extension
200
+ and a restart the price of every integration. Insika inverts that: tier 1
201
+ (data tools, MCP imports, skills) covers most integrations hot, with no
202
+ deploy, and tier 2 exists only for what genuinely needs to run in-process.
203
+ A smaller code-plugin surface is a feature — less to audit, less to break.
204
+ 2. **Manifest before code.** Discovery never executes anything: the manifest is
205
+ read, validated, and gated first, and only then is the entry `require`d.
206
+ You can inventory, enable, and veto plugins without running them.
207
+ 3. **Contracts are the public API.** Anything addressable by name — tools,
208
+ workflows, capabilities, channels — must be declared in `contracts`, and an
209
+ undeclared registration is ignored with a warning. A plugin cannot quietly
210
+ widen its surface between versions, and a channel cannot mount a route
211
+ nobody asked for.
212
+ 4. **Announce, never scan.** A gem opts in with one explicit
213
+ `Insika::Plugin.announce` call; Insika never walks the load path or your
214
+ installed gems looking for candidates. What loads is exactly what was
215
+ announced or configured — auditable from the boot log alone.
216
+ 5. **One bad plugin never takes the deployment down.** Registration is staged
217
+ and committed atomically; a raise inside `register(api)` rolls back
218
+ everything that plugin staged and boot continues. Enablement is explicit,
219
+ and deny wins.
220
+
176
221
  ## Publishing a plugin
177
222
 
178
223
  - **Name it `insika-plugin-<thing>`.** The convention *is* the registry for now:
data/docs/POLICY.md ADDED
@@ -0,0 +1,216 @@
1
+ ---
2
+ title: Limits and policy
3
+ parent: Core concepts
4
+ nav_order: 2
5
+ permalink: /policy/
6
+ ---
7
+
8
+ # Limits and policy
9
+
10
+ What an agent may do is layered. Each layer is independent, opt-in where it
11
+ matters, and editable hot — the same profile keys covered in [Agents](AGENTS.md),
12
+ grouped here because they answer one question: what is this agent allowed to do,
13
+ and what stops it when it tries to do more.
14
+
15
+ What an agent may do is layered. Each layer is independent, opt-in where it
16
+ matters, and editable hot.
17
+
18
+ ## Layer 1: Tools (what it can call)
19
+
20
+ `tools_allow` / `tools_deny` / `tools_allow_groups` decide which tools enter the
21
+ turn's tool-loop, enforced by the tool-allowlist policy. See [Tools](TOOLS.md)
22
+ for how tools are defined and registered, and [`examples/data-tool/`](https://github.com/guizaols/insika/tree/main/examples/data-tool/).
23
+
24
+ ## Layer 2: Policies and approvals
25
+
26
+ Policies are named entries evaluated before the turn runs. Builtins cover
27
+ tool-, skill-, and workflow-allowlisting, plus **`ApprovalRequired`** — which
28
+ does not allow or deny but *tags* a tool as needing human approval. Set
29
+ `approvals_required: [tool names]`; the gate then fires when the model tries to
30
+ call that tool, suspending the turn until an operator approves it in the Studio.
31
+ See [Security](SECURITY.md#human-approval).
32
+
33
+ ## Layer 3: Guardrails (content safety)
34
+
35
+ `guardrails` configures input/output content safety per agent — **opt-in**, so an
36
+ agent that says nothing gets a conservative default (deterministic detectors on,
37
+ LLM moderator off). See [Security](SECURITY.md#guardrails) and
38
+ [`examples/guardrails/`](https://github.com/guizaols/insika/tree/main/examples/guardrails/).
39
+
40
+ ## Layer 4: Edge limits (flood and spend control)
41
+
42
+ Two independent, opt-in ceilings, enforced *before* the model is ever called —
43
+ opt-in everywhere except on a public channel, where `chat_rate_limit` is
44
+ [required](CHANNELS.md#a-rate-limit-is-required-not-suggested) and the
45
+ [web widget](CHANNELS.md#the-web-widget) refuses to serve without one:
46
+
47
+ - **`chat_rate_limit`** — turn attempts per session per `chat_rate_window`.
48
+ - **`agent_token_ceiling`** — total tokens per agent per `agent_token_window`.
49
+
50
+ On breach the turn halts gracefully with a configurable `limit_response` and
51
+ **zero LLM calls**. Windows are set at the platform level; the ceilings can be set
52
+ per agent (blank inherits the platform value, `0` explicitly disables it).
53
+
54
+ > ⚠️ The token window default is **86400 (daily)**. To express "500k tokens per
55
+ > **hour**", set `agent_token_window = 3600` explicitly. A per-agent key that is
56
+ > *present but nil* reads as OFF for that agent — leave the key **absent** to
57
+ > inherit. See [Security](SECURITY.md#edge-limits).
58
+
59
+ ### Calendar budgets — the daily/monthly cost wall
60
+
61
+ A third, opt-in ceiling for the *billing* shape the windows above cannot express:
62
+ a spend cap over a CALENDAR day or month, per `(tenant, agent)` when
63
+ multi-tenant. Data on the profile (DSL `budget` or the pack's `budget` key):
64
+
65
+ ```ruby
66
+ budget daily: 100_000, monthly: 2_000_000, soft: false # or soft: true
67
+ ```
68
+
69
+ - Tokens count the **billed** spend — `input + output + cached + cache_creation`
70
+ (the cached prefix is the bulk of the bill, not an afterthought).
71
+ - **Hard** (the default, `soft` absent/false): a turn that arrives with the
72
+ window's spend already at/over the cap **fails** with the typed
73
+ `Insika::BudgetExceeded` — the envelope reads `budget_exceeded` +
74
+ `retry_after` (seconds until the window rolls). It is NOT a customer reply; it
75
+ is an operator signal.
76
+ - **Soft** (`soft: true`): the same turn RUNS — crossing the cap emits one
77
+ `budget_warning` event per window and injects a note into the context (the
78
+ model sees it, the transcript does not).
79
+ - Either way, crossing `alert_at` (default `0.8` of the cap) fires the same
80
+ warning **before** the wall, once per window.
81
+
82
+ > ⚠️ Unlike the ceilings above, the cap that counts is per **calendar** window —
83
+ > a `daily` budget rolls at UTC midnight, a `monthly` one on the 1st, whatever
84
+ > the sun. `agent_token_window` is a fixed seconds window and cannot express
85
+ > "the day resets at midnight".
86
+
87
+ ### Reliability — retries, fallback, circuit breaker
88
+
89
+ The provider interaction is a single attempt by default (RubyLLM's own 2
90
+ transport retries aside). For a store that cannot have a dead model take the
91
+ chat down, the reliability policy is DATA on the profile:
92
+
93
+ ```ruby
94
+ reliability retries: 2, backoff: "exponential",
95
+ fallback: ["openai/gpt-4o-mini"],
96
+ circuit_breaker: { after: 10, within: 60, cooldown: 300 }
97
+ ```
98
+
99
+ - **Retries** — transient failures (`:retryable` / `:rate_limited_*` per the
100
+ error classification) retry with exponential backoff, up to `retries`.
101
+ A `:fatal` (auth, billing, bad request) is NEVER retried or rotated. Each
102
+ attempt runs on a fresh chat — the customer-visible answer comes only from
103
+ the attempt that returns.
104
+ - **Fallback** — after a node's retries, the turn ROTATES to the next model in
105
+ the chain: the profile's `fallback` refs first, then the platform
106
+ `fallback_models`. The turn's usage is attributed to the model that actually
107
+ spoke (`model_source: "fallback"`).
108
+ - **Circuit breaker** — per `(tenant, provider/model)`: `after` failures within
109
+ `within` seconds open the circuit; while open, the turn fail-fasts with the
110
+ typed `circuit_open` + `retry_after` (remaining cooldown) and the provider is
111
+ never touched. After `cooldown` a half-open trial closes the circuit on
112
+ success or reopens it on failure.
113
+ - **`timeout`** — per-attempt ceiling (default 30s), counted as a retryable
114
+ failure.
115
+
116
+ Absent `reliability` = the plain single attempt, byte-for-byte today's
117
+ behavior.
118
+
119
+ ### Intent routing — classify before you answer
120
+
121
+ For a store that must tell "shopping" from "order" from "human" up front,
122
+ routing is data on the profile:
123
+
124
+ ```ruby
125
+ routes "shopping" => "the customer wants to browse products",
126
+ "order" => { "description" => "asks about an existing order",
127
+ "delegate" => "order-agent" },
128
+ "human" => { "description" => "the customer asks for a person",
129
+ "stuck" => true, "message" => "A person will help you." },
130
+ "default" => "shopping",
131
+ "model" => "deepseek-v4-flash" # the cheap classifier (absent = the agent's own)
132
+ ```
133
+
134
+ - **Classification** — when `routes:` is present, the message is classified into
135
+ one route with the configured model BEFORE the agent chat is assembled, from
136
+ a prompt auto-generated out of the descriptions (no per-route prompt file).
137
+ The route rides the turn: `state.route`, the `:route_classified` event, and
138
+ the terminal event additively.
139
+ - **Deterministic default** — the model's answer must be a route name; prose,
140
+ an unknown name, or an empty answer falls back to `default`, never invents.
141
+ A classifier call that FAILS leaves the turn unrouted (routing is additive —
142
+ it must not break the turn).
143
+ - **Cost** — the classification is an extra provider call, counted in the
144
+ turn's usage (the trace, the token ceiling and the budget all see it).
145
+ - **Actions** — a route value may be a description string, or a Hash:
146
+ `delegate: "<agent-id>"` hands the turn to that existing agent and its
147
+ answer becomes the parent's; `stuck: true` ends the turn with the [stuck
148
+ outcome](AGENTS.md#the-stuck-signal--i-cannot-proceed) and the route's `message`
149
+ (or description) as the lead-in — the consumer interprets it. A route with
150
+ neither is just a label. A delegation counts against the same delegation
151
+ depth cap as a subagent (`INSIKA_SUBAGENT_DEPTH_CAP`, default 5), so a pair
152
+ of agents routing to each other stops instead of looping.
153
+
154
+ Absent `routes` = no classification, no extra call, byte-identical turn.
155
+
156
+ ### Operator alerts — the webhook
157
+
158
+ Three operational events — `budget_warning`, `breaker_open`, `delivery_failed` —
159
+ can be answered per agent with a webhook:
160
+
161
+ ```ruby
162
+ alerts webhook: "https://ops.example.com/insika-alerts"
163
+ ```
164
+
165
+ When present, each such event is POSTed to the URL as JSON (the event's
166
+ type/data/meta, plus the agent). Delivery rides the same outbox + claim +
167
+ bounded-retry pipeline as channel answers — at-most-once, crashed deliveries
168
+ recovered at boot. The engine transports the event and does not interpret it: a
169
+ Slack/CRM adapter is the consumer's. Absent `alerts` = nothing is sent.
170
+
171
+ Separately, with `INSIKA_TURN_TIMING`, the provider's **live TTFB** is carried in
172
+ the streaming envelope: the first content chunk emits an `insika.ttft` frame
173
+ (`ttft_ms`) on `/v1/responses`, alongside the per-turn `timing` breakdown on the
174
+ final `response.completed`. Additive and opt-in — absent by default.
175
+
176
+ ## Layer 5: Reasoning (thinking)
177
+
178
+ Controls the model's thinking budget, resolved by precedence
179
+ **Chat > Agent > Model > Global** (first non-blank wins):
180
+
181
+ | Scope | Where |
182
+ |-------|-------|
183
+ | Chat | session var `__llm__.thinking` |
184
+ | Agent | `profile.params["thinking"]` |
185
+ | Model | platform `model_params[<ref>].thinking` |
186
+ | Global | platform `thinking` |
187
+
188
+ Values: `off | on | low | medium | high`. `off`/`on` toggle thinking; the effort
189
+ levels map to the provider's thinking-effort parameter. This is a control
190
+ primitive, not a latency lever — turning reasoning off does not necessarily speed
191
+ up a turn, because most of a turn's latency is the provider itself, not thinking.
192
+
193
+ Whether the reasoning ever reaches the **customer** is a separate switch, off by
194
+ default:
195
+
196
+ ```ruby
197
+ edge_stream thinking: true, intermediate: false
198
+ ```
199
+
200
+ `thinking` is the provider's reasoning; `intermediate` is the model narrating its
201
+ own tool loop ("let me look that up"). Both are always on the event stream for the
202
+ Studio and the trace — this decides only whether `/v1/responses` translates them,
203
+ and each opted-in channel gets its own frame type, never the answer's. See
204
+ [Architecture](ARCHITECTURE.md#what-crosses-the-edge).
205
+
206
+ > ⚠️ Turn it on knowing your consumer. One that concatenates every text delta into
207
+ > a single message — a WhatsApp adapter — will only be affected once it learns to
208
+ > read the new frames, and when it does, the deliberation is what the customer
209
+ > reads. That is the operator's call, which is why it is neither a default nor a
210
+ > global.
211
+
212
+ ## See also
213
+
214
+ - [Agents](AGENTS.md) — the profile these keys live on.
215
+ - [Tools](TOOLS.md) — how a tool gets defined, registered and allowed.
216
+ - [Security](SECURITY.md) — the deployment-side counterpart: sandbox, egress, secrets.
data/docs/REFINEMENT.md CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  title: Refinement
3
- parent: Operate & prove it
4
- nav_order: 5
3
+ parent: Improve
4
+ nav_order: 2
5
5
  permalink: /refinement/
6
6
  ---
7
7
 
@@ -32,11 +32,11 @@ it reaches anyone. Every part of that is below, including what it cannot catch.
32
32
  From the CLI, against the same database the engine uses:
33
33
 
34
34
  ```bash
35
- INSIKA_DB=insika.db bin/insika refine --agent bia
35
+ INSIKA_DB=insika.db bin/insika refine --agent demo
36
36
  ```
37
37
 
38
38
  ```text
39
- bia — completed (last 200 session(s))
39
+ demo — completed (last 200 session(s))
40
40
  tool_error ×24 shipping_quote failed: cep is required
41
41
  sessions: 9f2c1a04…, 4b7e5590…, c1d0aa31…
42
42
  repetition ×7 customer repeated themselves
@@ -51,11 +51,14 @@ Or from **Studio → Refinement**: pick the agent, press Run, and the same repor
51
51
  renders with each session id linking to its transcript. Every finding is a claim
52
52
  you can go and check.
53
53
 
54
- There is no scheduler in the engine, by design. A run is one command
55
- (`run_refinement`), and the CLI and the button are the two ways to fire it. If you
56
- want it on a timer, point your own cron at the authenticated route the button uses
57
- that keeps the engine free of a background ticker and of the single-node
58
- assumption one would bring.
54
+ A run is one command (`run_refinement`), and the CLI and the button are the
55
+ two ways to fire it. There is still no built-in timer *for the report itself*,
56
+ but there is one for the *agent*: fine-grained recurring turns live in
57
+ [Schedules](SCHEDULING.md), and a refinement run scheduled like any other turn
58
+ is just a message the agent's schedule sends it. If you prefer to stay outside
59
+ the engine, pointing your own cron at the authenticated route the button uses
60
+ works just the same — both paths are supported, the engine's trigger is the
61
+ built-in one.
59
62
 
60
63
  ```
61
64
  insika refine ─┐
@@ -173,7 +176,7 @@ an agent with no configuration at all can be run. The optional block on the agen
173
176
  sets the defaults:
174
177
 
175
178
  ```ruby
176
- Insika.agent "bia" do
179
+ Insika.agent "demo" do
177
180
  model "deepseek-v4-flash"
178
181
  refine window: { last_sessions: 200 }, max_findings: 20
179
182
  end
@@ -466,8 +469,10 @@ prompt edits have real leverage, and it is also where they do damage.
466
469
 
467
470
  ## What this is not
468
471
 
469
- It has no scheduler: a run happens because a person or a cron asked for one, never
470
- because a timer inside the engine went off. A proposal is written when you ask for
472
+ A run happens because a person or a cron asked for one — the refinement
473
+ pipeline has no timer of its own, and if you want one, a schedule
474
+ ([Schedules](SCHEDULING.md)) makes a run a turn any agent can be scheduled to
475
+ send. A proposal is written when you ask for
471
476
  one, and — unless you turned on [`auto_apply`](#applying-without-a-human), which is
472
477
  off until you do and bounded when you do — applied when you approve it. And it cannot
473
478
  touch your guardrails, tools, policies, model pins or limits, and not because a prompt
data/docs/RELEASING.md CHANGED
@@ -59,6 +59,33 @@ GEM_HOME="$T/gemhome" GEM_PATH="$T/gemhome" ruby -e '
59
59
  puts "clean"'
60
60
  ```
61
61
 
62
+ ### The 1.0 release gate — the clean-install proof
63
+
64
+ For the 1.0 release the proof above is scripted and its **installed-bytes**
65
+ half is asserted by the domain-boundary suite on the artifact, not the
66
+ repo. Run the runbook, with a key (the smoke turn is one `reply` through the
67
+ installed gem):
68
+
69
+ ```bash
70
+ DEEPSEEK_API_KEY=sk-... scripts/install_proof/install_proof.sh # prints PASS
71
+ ```
72
+
73
+ The script builds, installs into a FRESH `GEM_HOME`, asserts `gem contents
74
+ insika` carries no `deploy/ packs/ examples/ plugins/ evals/ scripts/ spec/`
75
+ path and no demo-persona-name string, and answers one turn from an app dir that follows
76
+ only the public docs. Archive the PASS output with the release notes — it is
77
+ the 1.0 exit criterion "install proof by the docs alone".
78
+
79
+ The same gate writes the freeze date: a breaking `/v1` change needs a new
80
+ `Insika-Version` entry (server/app.rb), a compatibility branch
81
+ and a rewritten `**Frozen as of:**` line in `docs/API.md` — the version-gate
82
+ spec pins the two together, and the 1.0 release writes the date at release
83
+ time.
84
+
85
+ Catalog submission checklist: verify the best-of-Agent-Harnesses
86
+ catalog size at submission time (161 vs 154 — the counts diverge across the
87
+ catalog's own pages) and cite the conformance suite as the `durable` evidence.
88
+
62
89
  ## Publish
63
90
 
64
91
  ```bash