insika 0.3.0 → 0.8.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (204) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGELOG.md +296 -0
  3. data/README.md +48 -12
  4. data/bin/insika +725 -0
  5. data/bin/insika-router +87 -0
  6. data/docs/AGENTS.md +116 -406
  7. data/docs/API.md +5 -5
  8. data/docs/ARCHITECTURE.md +3 -2
  9. data/docs/ARTIFACTS.md +137 -0
  10. data/docs/BENCHMARK.md +2 -2
  11. data/docs/CHANNELS.md +14 -14
  12. data/docs/CONTEXT.md +63 -19
  13. data/docs/DEMO.md +80 -0
  14. data/docs/DEPLOY.md +87 -10
  15. data/docs/EMBEDDING.md +1 -1
  16. data/docs/EVALS.md +128 -3
  17. data/docs/FACTS.md +3 -3
  18. data/docs/HARVEST.md +5 -6
  19. data/docs/KNOWLEDGE.md +290 -0
  20. data/docs/LOADTEST.md +17 -29
  21. data/docs/MEDIA.md +128 -0
  22. data/docs/OBSERVABILITY.md +46 -12
  23. data/docs/OUTCOMES.md +137 -0
  24. data/docs/PLUGINS.md +51 -6
  25. data/docs/POLICY.md +222 -0
  26. data/docs/REFINEMENT.md +14 -9
  27. data/docs/RELEASING.md +4 -4
  28. data/docs/ROUTER.md +213 -0
  29. data/docs/RUNNING-LOCAL.md +5 -5
  30. data/docs/SCHEDULING.md +121 -0
  31. data/docs/SECURITY.md +23 -7
  32. data/docs/SKILLS.md +11 -2
  33. data/docs/SOAK.md +3 -3
  34. data/docs/TEMPLATES.md +134 -0
  35. data/docs/TOOLS.md +176 -27
  36. data/docs/WHY.md +1 -1
  37. data/docs/WORKFLOWS.md +2 -2
  38. data/docs/_includes/head_custom.html +5 -0
  39. data/docs/_includes/title.html +13 -0
  40. data/docs/_sass/color_schemes/insika.scss +32 -0
  41. data/docs/_sass/custom/custom.scss +199 -0
  42. data/docs/_sass/custom/setup.scss +26 -0
  43. data/docs/assets/img/favicon.svg +7 -0
  44. data/docs/assets/img/insika-mark.svg +7 -0
  45. data/docs/core-concepts.md +21 -0
  46. data/docs/domain.md +4 -4
  47. data/docs/improve.md +20 -0
  48. data/docs/index.md +8 -5
  49. data/docs/integrate.md +20 -0
  50. data/docs/operate.md +13 -6
  51. data/docs/prompts/ADD-TOOL.md +118 -0
  52. data/docs/prompts/DIAGNOSE-TURN.md +65 -0
  53. data/docs/prompts/GO-LIVE.md +138 -0
  54. data/docs/prompts/RUN-EXAMPLES.md +70 -0
  55. data/docs/reference.md +19 -0
  56. data/docs/ship.md +10 -2
  57. data/docs/start-here.md +18 -0
  58. data/lib/insika/agent_profile.rb +99 -17
  59. data/lib/insika/artifact_signing.rb +82 -0
  60. data/lib/insika/artifact_store.rb +160 -0
  61. data/lib/insika/channel_delivery.rb +1 -1
  62. data/lib/insika/chat_builder.rb +50 -19
  63. data/lib/insika/commands/agent_payload.rb +2 -2
  64. data/lib/insika/commands/backfill_knowledge.rb +145 -0
  65. data/lib/insika/commands/delete_artifact.rb +35 -0
  66. data/lib/insika/commands/delete_concept.rb +34 -0
  67. data/lib/insika/commands/delete_mcp.rb +6 -2
  68. data/lib/insika/commands/delete_tenant_data.rb +15 -3
  69. data/lib/insika/commands/gate_refinement.rb +1 -1
  70. data/lib/insika/commands/refresh_mcp_tools.rb +47 -0
  71. data/lib/insika/commands/restore_concept.rb +34 -0
  72. data/lib/insika/commands/seed_demo_data.rb +31 -0
  73. data/lib/insika/commands/upsert_mcp.rb +6 -3
  74. data/lib/insika/commands/write_concept.rb +57 -0
  75. data/lib/insika/compaction.rb +196 -0
  76. data/lib/insika/context/builder.rb +6 -2
  77. data/lib/insika/context/fragment.rb +4 -1
  78. data/lib/insika/context/priority.rb +8 -0
  79. data/lib/insika/context/providers/briefing.rb +53 -24
  80. data/lib/insika/context/providers/knowledge.rb +108 -0
  81. data/lib/insika/context/providers/prompt.rb +30 -24
  82. data/lib/insika/context/providers/session.rb +46 -10
  83. data/lib/insika/context_trace_store.rb +11 -1
  84. data/lib/insika/cron.rb +189 -0
  85. data/lib/insika/demo/agent_attrs.rb +43 -0
  86. data/lib/insika/demo/golden_cases.rb +81 -0
  87. data/lib/insika/demo/seeder.rb +336 -0
  88. data/lib/insika/doctor.rb +280 -17
  89. data/lib/insika/dsl/definition.rb +3 -2
  90. data/lib/insika/dsl/runtime.rb +64 -79
  91. data/lib/insika/dsl/server_boot.rb +23 -1
  92. data/lib/insika/dsl/system.rb +10 -2
  93. data/lib/insika/dsl.rb +103 -2
  94. data/lib/insika/env_schema.rb +21 -7
  95. data/lib/insika/evals/golden.rb +41 -4
  96. data/lib/insika/evals/judge.rb +47 -2
  97. data/lib/insika/evals/pairwise.rb +11 -0
  98. data/lib/insika/evals/persona.rb +98 -0
  99. data/lib/insika/evals/runner.rb +9 -0
  100. data/lib/insika/evals/simulator.rb +225 -0
  101. data/lib/insika/evals/transport.rb +84 -2
  102. data/lib/insika/event_stream.rb +10 -0
  103. data/lib/insika/executor.rb +295 -55
  104. data/lib/insika/followup_policy.rb +2 -25
  105. data/lib/insika/golden_store.rb +16 -1
  106. data/lib/insika/grounding/matcher.rb +1 -1
  107. data/lib/insika/knowledge.rb +680 -0
  108. data/lib/insika/knowledge_store.rb +140 -0
  109. data/lib/insika/loop_detector.rb +5 -34
  110. data/lib/insika/mcp_client.rb +94 -0
  111. data/lib/insika/mcp_json.rb +74 -0
  112. data/lib/insika/mcp_live_tool.rb +43 -0
  113. data/lib/insika/mcp_store.rb +98 -26
  114. data/lib/insika/mcp_tool_ingestor.rb +30 -8
  115. data/lib/insika/mcp_tool_registry.rb +100 -0
  116. data/lib/insika/media.rb +115 -31
  117. data/lib/insika/message_origin.rb +1 -1
  118. data/lib/insika/middleware.rb +9 -0
  119. data/lib/insika/onboarding.rb +17 -1
  120. data/lib/insika/outcome_store.rb +1 -1
  121. data/lib/insika/overlay_tool_registry.rb +37 -17
  122. data/lib/insika/packaging.rb +2 -2
  123. data/lib/insika/profile_source.rb +15 -1
  124. data/lib/insika/prompt_catalog.rb +10 -0
  125. data/lib/insika/retention.rb +36 -1
  126. data/lib/insika/router/app.rb +157 -0
  127. data/lib/insika/router/backend_pool.rb +98 -0
  128. data/lib/insika/router/hash_ring.rb +55 -0
  129. data/lib/insika/router/proxy_body.rb +34 -0
  130. data/lib/insika/router/session_key.rb +54 -0
  131. data/lib/insika/router.rb +18 -0
  132. data/lib/insika/schedule.rb +177 -0
  133. data/lib/insika/schedule_engine.rb +314 -0
  134. data/lib/insika/schedule_store.rb +208 -0
  135. data/lib/insika/server/app.rb +105 -15
  136. data/lib/insika/server/rack_app.rb +5 -1
  137. data/lib/insika/server/responses.rb +5 -5
  138. data/lib/insika/session_store.rb +34 -4
  139. data/lib/insika/settings_store.rb +8 -1
  140. data/lib/insika/skill_catalog.rb +12 -0
  141. data/lib/insika/soak/runner.rb +4 -4
  142. data/lib/insika/steer_injector.rb +21 -10
  143. data/lib/insika/studio/app.rb +591 -47
  144. data/lib/insika/studio/assets/dist/application.css +1 -1
  145. data/lib/insika/studio/assets/dist/application.js +21 -21
  146. data/lib/insika/studio/forms.rb +57 -5
  147. data/lib/insika/studio/nav_icons.rb +14 -1
  148. data/lib/insika/studio/views/_agent_tab_cache.erb +25 -0
  149. data/lib/insika/studio/views/_agent_tab_config.erb +514 -0
  150. data/lib/insika/studio/views/_agent_tab_history.erb +24 -0
  151. data/lib/insika/studio/views/_agent_tab_loops.erb +54 -0
  152. data/lib/insika/studio/views/_agent_tab_memory.erb +51 -0
  153. data/lib/insika/studio/views/_agent_tab_outcomes.erb +31 -0
  154. data/lib/insika/studio/views/_agent_tab_prompts.erb +108 -0
  155. data/lib/insika/studio/views/_agent_tab_skills.erb +38 -0
  156. data/lib/insika/studio/views/_agents_master.erb +44 -0
  157. data/lib/insika/studio/views/_message.erb +49 -32
  158. data/lib/insika/studio/views/agent_detail.erb +61 -820
  159. data/lib/insika/studio/views/agents.erb +70 -57
  160. data/lib/insika/studio/views/artifact.erb +23 -0
  161. data/lib/insika/studio/views/artifacts.erb +59 -0
  162. data/lib/insika/studio/views/evals.erb +2 -2
  163. data/lib/insika/studio/views/facts.erb +1 -1
  164. data/lib/insika/studio/views/funnel.erb +1 -1
  165. data/lib/insika/studio/views/home.erb +106 -67
  166. data/lib/insika/studio/views/knowledge.erb +123 -0
  167. data/lib/insika/studio/views/layout.erb +14 -11
  168. data/lib/insika/studio/views/mcp.erb +174 -80
  169. data/lib/insika/studio/views/session.erb +231 -177
  170. data/lib/insika/studio/views/settings.erb +50 -1
  171. data/lib/insika/studio/views/skills.erb +1 -1
  172. data/lib/insika/studio/views/tools.erb +24 -9
  173. data/lib/insika/telemetry/recorder.rb +49 -1
  174. data/lib/insika/templates/browser-agent/README.md +36 -0
  175. data/lib/insika/templates/browser-agent/agent.rb +49 -0
  176. data/lib/insika/templates/daily-digest/README.md +47 -0
  177. data/lib/insika/templates/daily-digest/agent.rb +77 -0
  178. data/lib/insika/templates/repo-explorer/README.md +36 -0
  179. data/lib/insika/templates/repo-explorer/agent.rb +45 -0
  180. data/lib/insika/templates/research-analyst/README.md +26 -0
  181. data/lib/insika/templates/research-analyst/agent.rb +68 -0
  182. data/lib/insika/templates/review-panel/README.md +20 -0
  183. data/lib/insika/templates/review-panel/agent.rb +50 -0
  184. data/lib/insika/templates/travel-planner/README.md +35 -0
  185. data/lib/insika/templates/travel-planner/agent.rb +87 -0
  186. data/lib/insika/templates.rb +112 -0
  187. data/lib/insika/tick.rb +24 -12
  188. data/lib/insika/timezone.rb +45 -0
  189. data/lib/insika/tool_batch.rb +67 -0
  190. data/lib/insika/tool_usage_report.rb +162 -0
  191. data/lib/insika/tools/generate_image.rb +52 -7
  192. data/lib/insika/tools/load_knowledge.rb +74 -0
  193. data/lib/insika/tools/run_persona_eval.rb +328 -0
  194. data/lib/insika/tools/save_artifact.rb +95 -0
  195. data/lib/insika/turn_budget.rb +91 -0
  196. data/lib/insika/turn_output.rb +1 -1
  197. data/lib/insika/turn_state.rb +15 -4
  198. data/lib/insika/version.rb +1 -1
  199. data/lib/insika/wiring/graph.rb +184 -12
  200. data/lib/insika/wiring/graph_chat.rb +102 -0
  201. data/lib/insika.rb +64 -0
  202. metadata +109 -5
  203. data/docs/build.md +0 -14
  204. data/docs/understand.md +0 -10
data/docs/ARCHITECTURE.md CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  title: Architecture
3
- parent: Understand the idea
4
- nav_order: 2
3
+ parent: Start here
4
+ nav_order: 3
5
5
  permalink: /architecture/
6
6
  ---
7
7
 
@@ -319,6 +319,7 @@ validator as the after-task hook, so both roots enforce content safety identical
319
319
  | Tools (data/manifest/MCP) | `lib/insika/tool_definition.rb`, `tool_manifest.rb`, `mcp_tool_ingestor.rb` |
320
320
  | Plugin loading (boot) | `lib/insika/plugin.rb`, `lib/insika/plugin/loader.rb` |
321
321
  | Refinement (traffic → report) | `lib/insika/refinement/*`, `lib/insika/refinement_store.rb` |
322
+ | Post-turn learning (facts, skills, knowledge — extracted from finished conversations) | `lib/insika/distill.rb`, `lib/insika/harvest.rb`, `lib/insika/knowledge.rb`, `lib/insika/knowledge_store.rb`; the per-turn hook lives in `Executor#persist_turn`, next to `finalize_delegation` |
322
323
  | Evals (cases, judges, gate) | `lib/insika/evals/*`, `lib/insika/golden_store.rb`; `evals/run.rb` is the CLI |
323
324
  | HTTP/SSE surface | `lib/insika/server/*` |
324
325
 
data/docs/ARTIFACTS.md ADDED
@@ -0,0 +1,137 @@
1
+ ---
2
+ title: Artifacts
3
+ parent: Operate
4
+ nav_order: 3
5
+ permalink: /artifacts/
6
+ ---
7
+
8
+ # Artifacts — a report the agent can hand you a URL to
9
+
10
+ A channel message is ephemeral, unformatted and capped. A scheduled report turn
11
+ (see [Schedules](SCHEDULING.md)) produces something no message can carry: a page
12
+ — tables, sections, inline charts. An **artifact** is that page as a thing that
13
+ exists afterwards: listable, linkable, and — because it carries customer PII —
14
+ deletable on the same terms as everything else the engine stores.
15
+
16
+ Deliberately small: a store, a tool, a route. Not a CMS.
17
+
18
+ ## The `save_artifact` tool
19
+
20
+ A registry tool, allowlisted per agent like every tool — **the allowlist IS the
21
+ switch**:
22
+
23
+ ```ruby
24
+ agent = Insika.agent("reporter") do
25
+ instructions "…"
26
+ tools_allow %w[save_artifact] # without this, the tool is not even offered
27
+ end
28
+ ```
29
+
30
+ The agent hands in `title` + `content` (default mime `text/html`; also
31
+ `text/markdown` and `image/svg+xml`) and gets the URL back, which it can include
32
+ in a channel message ("today's report: <url>"). When a signing key is
33
+ configured (below), the result also carries a `signed_url` that expires.
34
+
35
+ ```jsonc
36
+ { "id": "…", "url": "https://…/studio/artifacts/<id>/content",
37
+ "signed_url": "https://…/studio/artifacts/s/<id>?exp=…&sig=…" } // only with a key
38
+ ```
39
+
40
+ The tenant binding is **inherited, never chosen**: an artifact belongs to the
41
+ tenant of the agent that saved it — a binding of the tool instance, never a
42
+ parameter the model types. Store A's report can never appear in, or be linked
43
+ from, store B.
44
+
45
+ ## Serving
46
+
47
+ - `GET /studio/artifacts` — the Studio's per-agent list (the listing IS the
48
+ history; no versioning — one report per run).
49
+ - `GET /studio/artifacts/:id` — the preview page, rendered **inside a sandboxed
50
+ iframe** (no scripts, no same-origin, no forms).
51
+ - `GET /studio/artifacts/:id/content` — the raw page (authenticated).
52
+ - `GET /studio/artifacts/s/:id?exp=…&sig=…` — the **signed link**: the only
53
+ artifact route that works without a Studio session. HMAC-SHA256 over
54
+ `(id, expiry)` with `INSIKA_ARTIFACT_SIGNING_KEY`, verified in constant time.
55
+ Expired or bad signatures **404 (never 403 — no oracle)**. Rotating the key
56
+ invalidates every outstanding link — the documented behavior, not a bug.
57
+ Without `INSIKA_ARTIFACT_SIGNING_KEY` there is no signed surface at all.
58
+
59
+ **Artifact content is untrusted.** It is LLM output. Both content routes send:
60
+
61
+ ```http
62
+ Content-Security-Policy: default-src 'none'; style-src 'unsafe-inline'; img-src data:
63
+ X-Content-Type-Options: nosniff
64
+ ```
65
+
66
+ No script, no external fetch, no forms. The model writes HTML with **inline
67
+ SVG** for charts — that is a skill instruction (palette, tables, pure-SVG bars),
68
+ not engine code. A "real" charting need is a plugin.
69
+
70
+ ## Reasoning effort on a report turn
71
+
72
+ A report turn is not one shape of work — it plans, then it mines, then it
73
+ writes. `thinking` is one value per agent for the whole turn, so an agent set to
74
+ `high` pays deliberation on every one of the 30–50 tool calls a real report
75
+ makes, and that is where the 300 s turn timeout gets spent.
76
+
77
+ Split the phases across agents instead, which the engine already supports today:
78
+
79
+ ```ruby
80
+ Insika.system do
81
+ # The miners: one narrow question each, no judgement to make.
82
+ agent("sales_miner") do
83
+ model "deepseek-v4-flash"
84
+ params thinking: "low"
85
+ tools %w[query_sales]
86
+ instructions "Answer ONE question about sales from the store data. Numbers, no prose."
87
+ end
88
+
89
+ # The orchestrator: it plans the report and writes it. This is the turn
90
+ # that deserves the deliberation.
91
+ agent("reporter") do
92
+ model "deepseek-v4-flash"
93
+ params thinking: "high"
94
+ tools %w[save_artifact]
95
+ subagents "sales_miner"
96
+ instructions "Plan the report, call spawn_subagents ONCE for every number you need, then write the page and save_artifact it."
97
+ end
98
+ end
99
+ ```
100
+
101
+ Two things make this work: a child inherits the *environment* (model, thinking)
102
+ only as a **default**, so its own `params thinking:` wins; and `spawn_subagents`
103
+ runs the children in parallel, so wall-clock is the slowest miner rather than
104
+ the sum. Each child also mines in its own isolated context, which is what keeps
105
+ the orchestrator's context from filling with raw rows.
106
+
107
+ Measure it before reaching for anything cleverer: the numbers that matter are
108
+ the turn's wall-clock, the timeout rate, and the judge score on the same report.
109
+ Per-phase effort *inside* a single turn is a real idea, but it is only worth
110
+ building once this recipe is shown not to be enough.
111
+
112
+ ## Limits and retention
113
+
114
+ - **Size cap** — `INSIKA_ARTIFACT_MAX_BYTES` (default 1 MB): an artifact is a
115
+ page, not an attachment. The mime allowlist is `text/html`, `text/markdown`,
116
+ `image/svg+xml`; no binaries, no uploads.
117
+ - **Expiry** — the settings key `artifact_ttl_days` (Integer days; absent = OFF)
118
+ ages artifacts out on the retention sweep's own daily pass, **independent of
119
+ `retention_days`**: a deployment that keeps its conversations forever must
120
+ still expire the reports. This is the guarantee that PII inside a report
121
+ expires — the honest reach, because no reader can see inside the opaque HTML.
122
+
123
+ ## Privacy
124
+
125
+ - `delete_tenant_data` deletes the tenant's artifacts (the tenant binding is
126
+ the isolation boundary).
127
+ - `forget_customer` **cannot** know which artifacts mention a customer (content
128
+ is opaque HTML), so per-customer redaction inside a report is not pretended to
129
+ exist; the `artifact_ttl_days` knob is the guarantee that a report's PII
130
+ expires.
131
+
132
+ ## See also
133
+
134
+ - [Schedules](SCHEDULING.md) — the recurring turns whose output lands here.
135
+ - [Tools](TOOLS.md) — how a tool enters the per-agent allowlist.
136
+ - [`examples/scheduled-report/`](https://github.com/guizaols/insika/tree/main/examples/scheduled-report/)
137
+ — schedule + skill + data tool + artifact, tenant-bound, end to end.
data/docs/BENCHMARK.md CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  title: Benchmark
3
- parent: Operate & prove it
4
- nav_order: 2
3
+ parent: Operate
4
+ nav_order: 6
5
5
  permalink: /benchmark/
6
6
  ---
7
7
 
data/docs/CHANNELS.md CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  title: Channels
3
- parent: Build an agent
4
- nav_order: 6
3
+ parent: Integrate
4
+ nav_order: 2
5
5
  permalink: /channels/
6
6
  ---
7
7
 
@@ -176,14 +176,14 @@ again.
176
176
  ## Relay or the drop-in API?
177
177
 
178
178
  If you already own your messaging platform, you can reach the engine two ways: the
179
- drop-in [`POST /v1/responses`](/architecture/) — you hold an SSE connection for the
179
+ drop-in [`POST /v1/responses`](ARCHITECTURE.md) — you hold an SSE connection for the
180
180
  whole turn and read the answer off it — or the relay, where the engine acks in
181
181
  milliseconds and POSTs the answer to you when it exists.
182
182
 
183
183
  The instinct is that streaming gets the customer their reply sooner, and that the
184
184
  relay trades that away. **For the default relay it does not, and the reason is
185
185
  structural:** the engine publishes `:content` as the ANSWER, whole, after the
186
- turn's hooks ([what crosses the edge](/architecture/#what-crosses-the-edge)).
186
+ turn's hooks ([what crosses the edge](ARCHITECTURE.md#what-crosses-the-edge)).
187
187
  During the turn the stream carries tool activity; the text arrives in one piece at
188
188
  the end. Measured on a real store agent, the text frames span **0 ms** — there is
189
189
  nothing to deliver progressively. That is the `:at_end` fact, true of `/v1/responses`
@@ -204,7 +204,7 @@ That last row is the one that cannot be had the other way. `/v1/responses` answe
204
204
  the request it was given, so a message that arrives while a turn is running is a
205
205
  second turn — the engine has no way to tell you "this joined the previous one". The
206
206
  relay's `merged` / `steered` acks exist precisely to say that, which is why
207
- [the inbound queue](/agents/#queue_mode--when-a-message-arrives-while-the-agent-is-busy)
207
+ [the inbound queue](AGENTS.md#queue_mode--when-a-message-arrives-while-the-agent-is-busy)
208
208
  is only reachable from here.
209
209
 
210
210
  **Measured, so you can judge it rather than take our word:** same agent, same
@@ -282,7 +282,7 @@ answers, and **they are four different facts**:
282
282
  answer two or three times.** That is the one contract mistake that is visible to
283
283
  the end user, so it is worth a line of code: only deliver for a `202`.
284
284
 
285
- `merged` and `steered` come from the [inbound queue](/agents/#queue_mode--when-a-message-arrives-while-the-agent-is-busy)
285
+ `merged` and `steered` come from the [inbound queue](AGENTS.md#queue_mode--when-a-message-arrives-while-the-agent-is-busy)
286
286
  (`limits[:queue_mode]`). They only ever occur if you turned that on for the agent;
287
287
  with the default `followup` you will only see `202` and `duplicate`.
288
288
 
@@ -312,7 +312,7 @@ short backoff, and then the delivery is marked `failed` and stops.
312
312
  `content` is the turn's **answer** — one message, whole. The model's narration on
313
313
  the way to an answer ("vou verificar o cardápio…") does not come through here; it
314
314
  stays internal unless the agent opts in. That contract is
315
- [the edge contract](/architecture/#what-crosses-the-edge), and it is why you can
315
+ [the edge contract](ARCHITECTURE.md#what-crosses-the-edge), and it is why you can
316
316
  forward `content` straight to the customer.
317
317
 
318
318
  ### Delivery policy
@@ -379,7 +379,7 @@ your side: you already have the customer's conversation, and `GET /v1/tasks/:id`
379
379
  tells you the turn's terminal state.
380
380
 
381
381
  A turn that **failed** delivers nothing — an error string is not an answer. Watch
382
- `GET /v1/tasks/:id` or the [event stream](/observability/) for those.
382
+ `GET /v1/tasks/:id` or the [event stream](OBSERVABILITY.md) for those.
383
383
 
384
384
  ## Setting up the relay
385
385
 
@@ -397,7 +397,7 @@ INSIKA_RELAY_DELIVERY=progressive # optional; "at_end" (the defaul
397
397
  without a credential — a public inbound route with an LLM behind it is a money
398
398
  faucet, so it fails closed by construction.
399
399
 
400
- The delivery POST goes through the same [egress guard](/security/#egress-the-ssrf-boundary) as
400
+ The delivery POST goes through the same [egress guard](SECURITY.md#egress-the-ssrf-boundary) as
401
401
  data-tools: **https only**, and private/loopback destinations blocked. For local
402
402
  development, where your consumer is on `localhost`:
403
403
 
@@ -415,7 +415,7 @@ A runnable consumer in ~40 lines lives in
415
415
 
416
416
  ## Shadow mode
417
417
 
418
- Shadow mode (RFC-0025) lets one channel run every turn **end to end and deliver
418
+ Shadow mode lets one channel run every turn **end to end and deliver
419
419
  nothing** — the experiment that answers "can we replace the incumbent?" before
420
420
  any customer is handed over. The incumbent keeps answering; the engine records
421
421
  what it *would* have answered, and the two replies are judged pairwise against a
@@ -470,7 +470,7 @@ redirect its own conversation.
470
470
  ## Writing your own channel
471
471
 
472
472
  A channel is a plain object — no base class. Register it from a plugin
473
- (see [Plugins](/plugins/)) with `contracts: { channels: [<id>] }` in the manifest,
473
+ (see [Plugins](PLUGINS.md)) with `contracts: { channels: [<id>] }` in the manifest,
474
474
  and it mounts under `/channels/<id>/`.
475
475
 
476
476
  Two members are always there; the rest of the object decides which shape you get.
@@ -536,8 +536,8 @@ command carries `transport: "channel:<id>"`.
536
536
 
537
537
  ## See also
538
538
 
539
- - [Security](/security/) — the tokens, the egress guard, and why the rate limit
539
+ - [Security](SECURITY.md) — the tokens, the egress guard, and why the rate limit
540
540
  matters for anything public.
541
- - [Agents](/agents/) — `limits[:queue_mode]`, which is what produces `merged` and
541
+ - [Agents](AGENTS.md) — `limits[:queue_mode]`, which is what produces `merged` and
542
542
  `steered`.
543
- - [Observability](/observability/) — the event stream and OpenTelemetry.
543
+ - [Observability](OBSERVABILITY.md) — the event stream and OpenTelemetry.
data/docs/CONTEXT.md CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  title: Context
3
- parent: Build an agent
4
- nav_order: 4
3
+ parent: Core concepts
4
+ nav_order: 5
5
5
  permalink: /context/
6
6
  ---
7
7
 
@@ -26,10 +26,12 @@ into a deterministic prompt:
26
26
  | **Skills** | `<available_skills>` | 80 | identity | Level-1 skill list, minus whatever is already eager — see [Skills](SKILLS.md). |
27
27
  | **Tool search** | `<available_tools>` | 70 | identity | Level-1 list of deferred tools — see [Tools](TOOLS.md). |
28
28
  | **Skill trigger** | `<active_skill>` | 85 | volatile | Level-2 bodies: the agent's `skills_eager` set, plus the ones whose `triggers:` match the message — see [Skills](SKILLS.md). |
29
+ | **Knowledge** | `<knowledge>` | 77 | volatile | Level-1 top-K learned concepts for the turn's message (+ one-hop `[[links]]`), only if `knowledge.retrieve` is on. Cuttable — see [Knowledge](KNOWLEDGE.md). |
29
30
  | **Memory** | `<memory>` | 75 | volatile | Durable facts + recent notes, only if `memory` is on. Cuttable. |
30
- | **Briefing** | `<briefing>` | 65 | volatile | The session's working state (known fields, still-missing list, next step) — only if the pack declared `briefing_fields`. Cuttable. |
31
+ | **Briefing** | `<briefing>` | 65 | volatile | The session's working state — the *known* fields only. Only if the pack declared `briefing_fields`. Cuttable. |
31
32
  | **Session** | history | 60–79 | volatile | The running transcript; priority scales with recency. |
32
33
  | **Request** | `<request_context>` | 40 | volatile | Turn variables + tenant. Most cuttable; sits last. |
34
+ | **Briefing (tail)** | `<recitation>` | 95 | volatile | The still-missing list + next step, rendered **after the whole history** as a `user` message — the last thing the model reads before the current message. |
33
35
 
34
36
  The ordering is deliberate: the render order is **identity layer first, volatile
35
37
  layer after** — nothing volatile can sit above the cache boundary, whatever its
@@ -41,7 +43,8 @@ cacheable prefix byte-stable (see the prefix cache below).
41
43
  - The cap is `profile.limits[:context_budget]`, **default 8000 tokens**.
42
44
  - To fit the budget, the builder cuts **non-pinned** fragments
43
45
  lowest-priority-first (ties broken by oldest history first). Under pressure you
44
- lose old history, then the memory block, then request context — **the pinned
46
+ lose request context first, then old history, then briefing, then memory, then
47
+ learned knowledge, then the skill/tool-search level-1 lists — **the pinned
45
48
  identity is never truncated**.
46
49
  - A **pinned** fragment (the identity) that *alone* exceeds the budget raises an
47
50
  error — the turn fails rather than shipping a truncated identity.
@@ -51,14 +54,37 @@ cacheable prefix byte-stable (see the prefix cache below).
51
54
  > returns empty turns, raise `context_budget` (e.g. to `60000`) before looking
52
55
  > anywhere else. See [Agents](AGENTS.md#default-limits).
53
56
 
54
- ### Compaction is not wired — except the mechanical dedupe
55
-
56
- There is a settings stub for LLM-summarization compaction (`enabled: false`,
57
- `keep_last`, a reserved utility-model slot), but **nothing consumes it today**
58
- — and the Studio no longer shows a form for it, so the setting cannot be
59
- switched on by accident. Size is managed purely by hard budget eviction.
60
- Do not rely on compaction to shrink a bloated agent: tune `context_budget` and
61
- keep the identity lean.
57
+ ### In-session compaction (opt-in, platform Settings)
58
+
59
+ When enabled (Studio → Settings → General, or the `compaction` settings hash),
60
+ the engine summarizes old turns *inside* the session instead of losing them to
61
+ eviction:
62
+
63
+ - **Trigger:** after a turn commits, if the session's *uncompacted* message
64
+ count exceeds `compact_after` (default 40), everything but the last
65
+ `keep_last` messages (default 20) is summarized by a cheap model
66
+ (`compaction.model`, falling back to the platform `utility_model`; neither
67
+ set = the feature is inert and `insika doctor` warns). Runs off the critical
68
+ path — the customer already has the answer.
69
+ - **Read path:** the Session provider replaces the compacted prefix with ONE
70
+ history fragment — a `user` message wrapped in `<conversation_summary>` tags,
71
+ priority 59 (one step below the oldest verbatim message), source
72
+ `compaction` (its own category in the context trace). The tail stays
73
+ verbatim and the boundary is **stable** between compactions, so the prompt
74
+ cache holds after it.
75
+ - **What survives:** the default prompt orders the summary to preserve customer
76
+ facts (sizes, CEP, order numbers), the assistant's commitments, the still-open
77
+ questions and the decisions already made; on re-compaction the previous
78
+ summary is folded in, so a fact from turn 3 survives every later batch.
79
+ A platform `compaction.prompt` replaces the default wholesale.
80
+ - **Scope:** store-sourced history only — a checkpoint resume replays its own
81
+ tape and an explicit `history` is the caller's contract; neither is
82
+ rewritten. Observability: the `:context_compacted` event, the
83
+ `insika.context.compacted` counter and `{upto, runs}` in the context trace.
84
+
85
+ Compaction does not replace the budget: eviction stays as the hard backstop
86
+ for a single oversized turn. Tune `context_budget` and keep the identity lean
87
+ regardless.
62
88
 
63
89
  One cheap half **is** wired, opt-in per agent: `tool_output_compression` (DSL
64
90
  `tool_output_compression`, or `"tool_output_compression": true` in the pack).
@@ -88,16 +114,16 @@ conversation; memory is the small set of facts that should outlive any single
88
114
  conversation. Facts and notes are editable from the Studio agent page. See
89
115
  [`examples/memory/`](https://github.com/guizaols/insika/tree/main/examples/memory/) for a runnable cross-session example.
90
116
 
91
- Facts carry **provenance metadata** (RFC-0031): every fact record stores `origin`
117
+ Facts carry **provenance metadata**: every fact record stores `origin`
92
118
  (who wrote it — `"engine"`, `"operator"`, `"legacy"` or `"distilled"`),
93
119
  `created_at` / `updated_at` timestamps, and an optional `expires_at` (ISO8601) —
94
120
  **an expired fact is never injected**, even before the daily sweep prunes it. The
95
121
  Studio Customers drill reads and edits the same cell the next turn reads (injection
96
122
  unchanged), and every operator mutation lands in the content-free audit trail
97
123
  (digests, never values). The sweep honors the `memory_ttl_days` setting on its own
98
- knob — see [Security](SECURITY.md#memory-and-the-right-to-be-forgotten-lgpd-rfc-0031).
124
+ knob — see [Security](SECURITY.md#memory-and-the-right-to-be-forgotten-lgpd).
99
125
 
100
- An **approved distilled fact** (RFC-0034 — see [Facts](FACTS.md)) lands in the
126
+ An **approved distilled fact** (see [Facts](FACTS.md)) lands in the
101
127
  same cell this provider injects, stamped `distilled:<session_ref>` — approved on
102
128
  the Studio Facts page, never applied automatically.
103
129
 
@@ -119,19 +145,37 @@ pack:
119
145
  briefing_fields "size", "budget", "delivery_day"
120
146
  ```
121
147
 
122
- With fields declared, the turn's `:system` context gains a `<briefing>` block
123
- (priority 65 — below identity/skill/memory so it never breaks the cacheable
148
+ With fields declared, the briefing renders in **two places**, and the split is
149
+ deliberate.
150
+
151
+ The durable half — what is already known — sits in the `:system` context
152
+ (priority 65: below identity/skill/memory so it never breaks the cacheable
124
153
  prefix, above the turn's own `<request_context>`):
125
154
 
126
155
  ```
127
156
  <briefing>
128
157
  known:
129
158
  size: M
159
+ </briefing>
160
+ ```
161
+
162
+ The half that is a *goal* — what is still missing and the agreed next step — is
163
+ **recited at the tail**, after the whole history, as the last thing the model
164
+ reads before the current user message:
165
+
166
+ ```
167
+ <recitation>
130
168
  still missing: budget, delivery_day
131
169
  next step: send the payment link tomorrow at 10
132
- </briefing>
170
+ </recitation>
133
171
  ```
134
172
 
173
+ Attention is strongest at the end of the context: a goal stated only at the top
174
+ is the first thing a 30-call turn forgets. So the recitation was **moved** there,
175
+ not copied — the head never repeats it, and the turn pays for it once. It rides
176
+ as a `user` message, like every other engine append inside a turn, so the system
177
+ prefix stays byte-stable and the cache breakpoint at its end keeps hitting.
178
+
135
179
  The `still missing` list is the point: the *model* sees which declared fields are
136
180
  still unanswered, so it stops re-asking for something already given. Stored keys
137
181
  that the pack no longer declares are never rendered. The Studio session screen
@@ -165,7 +209,7 @@ Two distinct caching mechanisms — don't conflate them:
165
209
  Cache accounting surfaces as `cached_tokens` (reads) and `cache_creation_tokens`
166
210
  (writes), visible in telemetry and the Studio tokens chip.
167
211
 
168
- ### The two layers (RFC-0030)
212
+ ### The two layers
169
213
 
170
214
  The system block is partitioned into two cache layers:
171
215
 
data/docs/DEMO.md ADDED
@@ -0,0 +1,80 @@
1
+ ---
2
+ title: Demo data
3
+ parent: Start here
4
+ nav_order: 4
5
+ permalink: /demo/
6
+ ---
7
+
8
+ # Demo data — see every loop working at once
9
+
10
+ Most of what makes Insika worth looking at only shows up after data has
11
+ accumulated: a [funnel](OUTCOMES.md#the-outcome-funnel) with a frozen
12
+ baseline needs weeks of folded outcomes, a [refinement](REFINEMENT.md)
13
+ proposal needs a run that actually found something, an
14
+ [approval](POLICY.md#layer-2-policies-and-approvals) needs a tool call
15
+ someone is waiting on. A fresh instance shows none of that — every one of
16
+ those pages renders its empty state, which makes it hard to tell "nothing
17
+ happened yet" from "this doesn't work."
18
+
19
+ `insika demo:seed` closes that gap in one shot: it provisions a single
20
+ fictional agent (`demo-store`, an e-commerce support bot) and writes enough
21
+ realistic-looking data to see every loop at once.
22
+
23
+ ## What it creates
24
+
25
+ | Page | What you'll see |
26
+ |------|------------------|
27
+ | [Funnel](OUTCOMES.md#the-outcome-funnel) — `/studio/funnel?agent=demo-store` | 40 days of folded outcomes (`greeted → browsing → cart_started → checkout_started → purchased`) and a **frozen baseline** |
28
+ | [Follow-ups](OUTCOMES.md#follow-ups--the-seller-who-comes-back) — `/studio/followups?agent=demo-store` | one record in each state: `pending`, `fired` (one per arm, so the A/B card has something to compare), `cancelled`, `blocked` |
29
+ | [Refinement](REFINEMENT.md) — `/studio/refinement?agent=demo-store` | four runs across the lifecycle: `awaiting_approval`, `applied`, `rejected` (gate failed), `no_findings` |
30
+ | [Approvals](POLICY.md#layer-2-policies-and-approvals) — `/studio/approvals` | two pending tool calls waiting on a human, one already resolved |
31
+ | [Facts](FACTS.md) — `/studio/facts` | three distillation proposals (`pending`, `approved` — with the resulting memory fact, `rejected`) |
32
+ | [Evals](EVALS.md) — `/studio/evals?agent=demo-store` | six golden cases and a baseline run with a mix of passes and one failure |
33
+
34
+ Every record is written through the same store APIs a real turn would use
35
+ (`OutcomeStore#create` + the funnel fold, `FollowupStore#create` + its
36
+ transitions, and so on) — there is no bulk-insert shortcut, and no bundled
37
+ `.rb` script outside `lib/` (nothing here needs a checkout; it ships in the
38
+ gem).
39
+
40
+ ## Running it
41
+
42
+ From the CLI, against whichever store the rest of your commands already use
43
+ ([Running locally](RUNNING-LOCAL.md#variables-all-optional) — `INSIKA_DB`
44
+ unset means an ephemeral, in-memory store, which is a fine place to try this):
45
+
46
+ ```bash
47
+ insika demo:seed
48
+ ```
49
+
50
+ A second run is a safe no-op once `demo-store` exists; pass `--force` to seed
51
+ another batch on top (the funnel baseline recomputes cleanly, but follow-ups,
52
+ refinement runs, approvals, proposals and goldens accumulate rather than
53
+ reset — none of those stores expose a per-agent bulk-delete that a shared
54
+ "platform" tenant could call without risking another agent's data).
55
+
56
+ From the Studio, open **Settings → Demo data** and click **Seed demo data**.
57
+ It dispatches the exact same command the CLI runs — the Studio never writes a
58
+ store directly, here or anywhere else.
59
+
60
+ **This writes into whatever store the running instance already has open.**
61
+ There's no separate demo database and no isolation: point `INSIKA_DB` at a
62
+ scratch file (or leave it unset, for an ephemeral store) before seeding —
63
+ never at a deployment holding real tenant data.
64
+
65
+ ## Then look around
66
+
67
+ Once seeded, the CLI prints the same six paths listed above. If Studio isn't
68
+ running yet, boot it the way you already do — see [Running
69
+ locally](RUNNING-LOCAL.md#boot) for a checkout, or [Embedding](EMBEDDING.md)
70
+ for `Insika.agent { … }.serve` — pointed at the same `INSIKA_DB`. Studio reads
71
+ every agent's profile from the same config store, so it will show
72
+ `demo-store` next to whatever agent you're actually building, no matter which
73
+ one the running process itself defines.
74
+
75
+ ## See also
76
+
77
+ - [Refinement](REFINEMENT.md), [Facts](FACTS.md), [Evals](EVALS.md) — what
78
+ each seeded page actually means.
79
+ - [Running locally](RUNNING-LOCAL.md) — booting Studio against a durable
80
+ `INSIKA_DB`.