insika 0.2.0 → 0.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (185) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGELOG.md +145 -17
  3. data/README.md +7 -1
  4. data/bin/insika +211 -8
  5. data/docs/AGENTS.md +377 -2
  6. data/docs/API.md +56 -0
  7. data/docs/CHANNELS.md +99 -9
  8. data/docs/CONTEXT.md +127 -16
  9. data/docs/DEPLOY.md +5 -2
  10. data/docs/EVALS.md +19 -0
  11. data/docs/FACTS.md +135 -0
  12. data/docs/HARVEST.md +117 -0
  13. data/docs/LOADTEST.md +14 -8
  14. data/docs/OBSERVABILITY.md +54 -0
  15. data/docs/REFINEMENT.md +3 -3
  16. data/docs/RELEASING.md +27 -0
  17. data/docs/RUNNING-LOCAL.md +3 -3
  18. data/docs/SECURITY.md +61 -0
  19. data/docs/SKILLS.md +2 -2
  20. data/docs/SOAK.md +127 -0
  21. data/docs/TOOLS.md +70 -2
  22. data/docs/domain.md +115 -0
  23. data/docs/index.md +1 -1
  24. data/lib/insika/agent_profile.rb +145 -10
  25. data/lib/insika/balloon_splitter.rb +102 -0
  26. data/lib/insika/budget_ledger.rb +34 -3
  27. data/lib/insika/cache_series_store.rb +49 -0
  28. data/lib/insika/channel_delivery.rb +118 -13
  29. data/lib/insika/channels/relay.rb +77 -3
  30. data/lib/insika/channels/web.rb +2 -2
  31. data/lib/insika/chat_builder.rb +84 -1
  32. data/lib/insika/checkpoint_store.rb +16 -0
  33. data/lib/insika/commands/agent_payload.rb +3 -3
  34. data/lib/insika/commands/cancel_followup.rb +49 -0
  35. data/lib/insika/commands/delete_tenant_data.rb +95 -0
  36. data/lib/insika/commands/export_customer_memory.rb +48 -0
  37. data/lib/insika/commands/forget_customer.rb +117 -0
  38. data/lib/insika/commands/freeze_funnel_baseline.rb +113 -0
  39. data/lib/insika/commands/gate_harvest.rb +138 -0
  40. data/lib/insika/commands/gate_refinement.rb +1 -1
  41. data/lib/insika/commands/judge_shadow_pairs.rb +124 -0
  42. data/lib/insika/commands/memory_forget_fact.rb +20 -4
  43. data/lib/insika/commands/memory_put_fact.rb +23 -4
  44. data/lib/insika/commands/promote_harvest.rb +130 -0
  45. data/lib/insika/commands/record_outcome.rb +46 -0
  46. data/lib/insika/commands/record_shadow_reply.rb +68 -0
  47. data/lib/insika/commands/reject_harvest.rb +38 -0
  48. data/lib/insika/commands/resolve_proposal.rb +108 -0
  49. data/lib/insika/commands/revoke_contact.rb +49 -0
  50. data/lib/insika/commands/rollback_harvest.rb +86 -0
  51. data/lib/insika/commands/run_distillation.rb +186 -0
  52. data/lib/insika/commands/run_harvest.rb +393 -0
  53. data/lib/insika/commands/send_message.rb +103 -6
  54. data/lib/insika/commands/session_purge.rb +67 -0
  55. data/lib/insika/contact_store.rb +183 -0
  56. data/lib/insika/context/builder.rb +21 -3
  57. data/lib/insika/context/fragment.rb +7 -3
  58. data/lib/insika/context/priority.rb +3 -0
  59. data/lib/insika/context/provider.rb +17 -3
  60. data/lib/insika/context/providers/briefing.rb +96 -0
  61. data/lib/insika/context/providers/memory.rb +16 -7
  62. data/lib/insika/context/providers/prompt.rb +30 -2
  63. data/lib/insika/context/providers/skill.rb +2 -0
  64. data/lib/insika/context/providers/tool_search.rb +2 -0
  65. data/lib/insika/context_trace_store.rb +38 -2
  66. data/lib/insika/distill.rb +224 -0
  67. data/lib/insika/distill_engine.rb +169 -0
  68. data/lib/insika/doctor.rb +716 -6
  69. data/lib/insika/dsl/runtime.rb +9 -3
  70. data/lib/insika/dsl/server_boot.rb +70 -1
  71. data/lib/insika/dsl.rb +83 -3
  72. data/lib/insika/edge_limiter.rb +28 -7
  73. data/lib/insika/env_schema.rb +9 -2
  74. data/lib/insika/errors.rb +11 -0
  75. data/lib/insika/evals/assertions.rb +3 -2
  76. data/lib/insika/evals/pairwise.rb +18 -6
  77. data/lib/insika/evidence.rb +183 -0
  78. data/lib/insika/executor.rb +687 -70
  79. data/lib/insika/followup_engine.rb +207 -0
  80. data/lib/insika/followup_policy.rb +221 -0
  81. data/lib/insika/followup_store.rb +306 -0
  82. data/lib/insika/funnel_declaration.rb +106 -0
  83. data/lib/insika/funnel_fold.rb +179 -0
  84. data/lib/insika/funnel_store.rb +163 -0
  85. data/lib/insika/golden_store.rb +1 -1
  86. data/lib/insika/grounding/matcher.rb +69 -0
  87. data/lib/insika/grounding.rb +44 -0
  88. data/lib/insika/harvest/conversion_gate.rb +159 -0
  89. data/lib/insika/harvest/criterion.rb +98 -0
  90. data/lib/insika/harvest/gate.rb +194 -0
  91. data/lib/insika/harvest/negative_list.rb +199 -0
  92. data/lib/insika/harvest.rb +241 -0
  93. data/lib/insika/harvest_engine.rb +193 -0
  94. data/lib/insika/harvest_store.rb +548 -0
  95. data/lib/insika/media.rb +298 -0
  96. data/lib/insika/memory_audit_store.rb +85 -0
  97. data/lib/insika/memory_store.rb +264 -23
  98. data/lib/insika/message_origin.rb +6 -1
  99. data/lib/insika/model_visible.rb +87 -0
  100. data/lib/insika/model_visible_trace_store.rb +66 -0
  101. data/lib/insika/onboarding.rb +6 -1
  102. data/lib/insika/outbox_store.rb +42 -4
  103. data/lib/insika/outcome_store.rb +147 -0
  104. data/lib/insika/packaging.rb +163 -0
  105. data/lib/insika/parity/criterion.rb +79 -0
  106. data/lib/insika/parity/verdict.rb +318 -0
  107. data/lib/insika/prefix_fingerprint.rb +58 -0
  108. data/lib/insika/profile_source.rb +22 -1
  109. data/lib/insika/proposal_store.rb +271 -0
  110. data/lib/insika/queue_policy.rb +4 -1
  111. data/lib/insika/refinement/proposer.rb +1 -1
  112. data/lib/insika/reliability.rb +32 -6
  113. data/lib/insika/retention.rb +281 -0
  114. data/lib/insika/routing.rb +101 -0
  115. data/lib/insika/safety/config.rb +44 -4
  116. data/lib/insika/safety/corpus.rb +255 -0
  117. data/lib/insika/safety/detectors.rb +32 -113
  118. data/lib/insika/safety/factory.rb +15 -2
  119. data/lib/insika/safety/grounding_enforcer.rb +59 -0
  120. data/lib/insika/safety/grounding_validator.rb +49 -0
  121. data/lib/insika/safety/input_guardrail.rb +1 -1
  122. data/lib/insika/safety/moderator.rb +1 -1
  123. data/lib/insika/safety/output_filter.rb +9 -5
  124. data/lib/insika/safety/output_validator.rb +11 -5
  125. data/lib/insika/schema_guard.rb +35 -0
  126. data/lib/insika/server/app.rb +144 -4
  127. data/lib/insika/server/rack_app.rb +16 -0
  128. data/lib/insika/server/responses.rb +40 -1
  129. data/lib/insika/session_actor.rb +8 -4
  130. data/lib/insika/session_store.rb +65 -2
  131. data/lib/insika/settings_store.rb +10 -0
  132. data/lib/insika/shadow_pair_store.rb +258 -0
  133. data/lib/insika/skill_catalog.rb +4 -0
  134. data/lib/insika/soak/envelope.rb +140 -0
  135. data/lib/insika/soak/report.rb +392 -0
  136. data/lib/insika/soak/runner.rb +554 -0
  137. data/lib/insika/store.rb +10 -1
  138. data/lib/insika/stores/memory.rb +6 -0
  139. data/lib/insika/stores/sqlite.rb +8 -0
  140. data/lib/insika/studio/app.rb +875 -14
  141. data/lib/insika/studio/assets/dist/application.css +1 -1
  142. data/lib/insika/studio/assets/dist/application.js +22 -22
  143. data/lib/insika/studio/assets/dist/favicon.svg +6 -0
  144. data/lib/insika/studio/forms.rb +262 -5
  145. data/lib/insika/studio/nav_icons.rb +6 -1
  146. data/lib/insika/studio/views/agent_detail.erb +627 -84
  147. data/lib/insika/studio/views/agents.erb +10 -6
  148. data/lib/insika/studio/views/approvals.erb +4 -1
  149. data/lib/insika/studio/views/chats.erb +4 -1
  150. data/lib/insika/studio/views/customer.erb +94 -0
  151. data/lib/insika/studio/views/customers.erb +32 -0
  152. data/lib/insika/studio/views/evals.erb +4 -1
  153. data/lib/insika/studio/views/facts.erb +133 -0
  154. data/lib/insika/studio/views/followups.erb +125 -0
  155. data/lib/insika/studio/views/funnel.erb +106 -0
  156. data/lib/insika/studio/views/harvest.erb +234 -0
  157. data/lib/insika/studio/views/home.erb +2 -1
  158. data/lib/insika/studio/views/layout.erb +1 -0
  159. data/lib/insika/studio/views/parity.erb +147 -0
  160. data/lib/insika/studio/views/playground.erb +7 -1
  161. data/lib/insika/studio/views/session.erb +56 -1
  162. data/lib/insika/studio/views/settings.erb +2 -0
  163. data/lib/insika/studio/views/skills.erb +0 -1
  164. data/lib/insika/studio/views/system_files.erb +1 -1
  165. data/lib/insika/studio/views/task.erb +13 -0
  166. data/lib/insika/studio/views/tasks.erb +4 -1
  167. data/lib/insika/studio/views/tools.erb +0 -1
  168. data/lib/insika/task_store.rb +21 -1
  169. data/lib/insika/testing/store_contract.rb +27 -6
  170. data/lib/insika/tick.rb +25 -1
  171. data/lib/insika/tool_definition.rb +17 -7
  172. data/lib/insika/tool_envelope.rb +69 -0
  173. data/lib/insika/tool_manifest.rb +5 -1
  174. data/lib/insika/tools/data_defined_tool.rb +10 -0
  175. data/lib/insika/tools/generate_image.rb +44 -0
  176. data/lib/insika/tools/schedule_followup.rb +164 -0
  177. data/lib/insika/tools/tts.rb +47 -0
  178. data/lib/insika/tools/update_briefing.rb +126 -0
  179. data/lib/insika/turn_state.rb +38 -1
  180. data/lib/insika/turn_timing.rb +22 -2
  181. data/lib/insika/version.rb +1 -1
  182. data/lib/insika/vitals.rb +84 -0
  183. data/lib/insika/wiring/graph.rb +300 -13
  184. data/lib/insika.rb +92 -3
  185. metadata +77 -1
data/docs/AGENTS.md CHANGED
@@ -41,7 +41,7 @@ ergonomics, not in what they produce.
41
41
  re-import that drops a tool or skill also removes it. `DELETE /v1/agents/:id`
42
42
  removes an agent.
43
43
  - **Studio** — create and edit an agent by hand in the control UI (Config /
44
- Prompts / Skills / Memory / History tabs), backed by the same commands.
44
+ Prompts / Skills / Memory / Outcomes / Cache / History / **loops** tabs), backed by the same commands.
45
45
 
46
46
  Creating an agent validates its id (required, must be unique) and its subagent
47
47
  graph (cycle/depth — see [subagents](#delegation-subagents)) **before**
@@ -97,6 +97,31 @@ the next tool-batch boundary: "you already ran this, answer with what you
97
97
  have"). A repeat after the warning aborts the turn like `max_tool_calls` does.
98
98
  Set it below 2 to switch it off.
99
99
 
100
+ ### `tool_persistence` — don't give up on the first empty result
101
+
102
+ The loop guard's mirror image. `max_tool_repeat` stops the model from repeating
103
+ the *same* call; `tool_persistence` stops it from giving up after *one* weak
104
+ call. When ON (the default), the engine appends a short **"Tool discipline"**
105
+ block after the agent's identity in the system prompt: a weak or empty tool
106
+ result means *try again with a different approach — a rephrased query, a
107
+ synonym, a broader term — before telling the user you found nothing* (and don't
108
+ narrate the retries); a tool error means *read it and fix the arguments*, never
109
+ repeat the exact same call. Without it, a search that returns 0 results reads as
110
+ final and the model answers "I couldn't find it" when a synonym one call away
111
+ would have.
112
+
113
+ This is the **one default-ON profile flag** — every field above is opt-in, this
114
+ one is opt-out, because the behavior is the proven default and the exception is
115
+ the thing worth declaring:
116
+
117
+ ```ruby
118
+ tool_persistence false # remove the block for this agent
119
+ ```
120
+
121
+ The block is a byte-stable constant, so `prompt_caching` stays effective: the
122
+ deploy that introduces it costs one cache write per agent, and every turn after
123
+ that hits as before.
124
+
100
125
  ### Why some limits are missing from that list
101
126
 
102
127
  `chat_rate_limit`, `agent_token_ceiling`, `queue_mode`, `debounce_ms`,
@@ -219,6 +244,12 @@ separately lives in one event, emitted when the window closes:
219
244
  Times and counts, never content. That is what answers "the customer says they
220
245
  sent the order number" without keeping a throwaway task per fragment.
221
246
 
247
+ > **`steer` also collects at the door.** The two windows are the same policy's
248
+ > halves, not two modes: a `steer` agent that set a `debounce_ms` also merges the
249
+ > fragments that land before the turn starts (RFC-0027). The window value is what
250
+ > an operator replaces the legacy pre-batch buffer with — `steer` catches anything
251
+ > that arrives after the turn started, the door window the fragments before it.
252
+
222
253
  #### `steer` — the message arrives while the turn is already running
223
254
 
224
255
  `collect` only ever touches a turn that has **not started**. Once the agent is
@@ -339,7 +370,9 @@ Three capabilities invert the default — `nil`/absent means **OFF**, not "all":
339
370
  setting, never "everything on"). `tool_output_compression` is a fourth: opt-in
340
371
  mechanical dedupe of repeated tool results in the history (see
341
372
  [Context](CONTEXT.md#compaction-is-not-wired--except-the-mechanical-dedupe)),
342
- off by default because it changes what the model sees.
373
+ off by default because it changes what the model sees. And one flag inverts the
374
+ other way: `tool_persistence` is **ON unless you set it to `false`** (see
375
+ [`tool_persistence`](#tool_persistence--dont-give-up-on-the-first-empty-result)).
343
376
 
344
377
  ### Declaring what this deployment has
345
378
 
@@ -455,6 +488,43 @@ reliability retries: 2, backoff: "exponential",
455
488
  Absent `reliability` = the plain single attempt, byte-for-byte today's
456
489
  behavior.
457
490
 
491
+ #### Intent routing — classify before you answer (WS4)
492
+
493
+ For a store that must tell "shopping" from "order" from "human" up front,
494
+ routing is data on the profile:
495
+
496
+ ```ruby
497
+ routes "shopping" => "the customer wants to browse products",
498
+ "order" => { "description" => "asks about an existing order",
499
+ "delegate" => "order-agent" },
500
+ "human" => { "description" => "the customer asks for a person",
501
+ "stuck" => true, "message" => "A person will help you." },
502
+ "default" => "shopping",
503
+ "model" => "deepseek-v4-flash" # the cheap classifier (absent = the agent's own)
504
+ ```
505
+
506
+ - **Classification** — when `routes:` is present, the message is classified into
507
+ one route with the configured model BEFORE the agent chat is assembled, from
508
+ a prompt auto-generated out of the descriptions (no per-route prompt file).
509
+ The route rides the turn: `state.route`, the `:route_classified` event, and
510
+ the terminal event additively.
511
+ - **Deterministic default** — the model's answer must be a route name; prose,
512
+ an unknown name, or an empty answer falls back to `default`, never invents.
513
+ A classifier call that FAILS leaves the turn unrouted (routing is additive —
514
+ it must not break the turn).
515
+ - **Cost** — the classification is an extra provider call, counted in the
516
+ turn's usage (the trace, the token ceiling and the budget all see it).
517
+ - **Actions** — a route value may be a description string, or a Hash:
518
+ `delegate: "<agent-id>"` hands the turn to that existing agent and its
519
+ answer becomes the parent's; `stuck: true` ends the turn with the [stuck
520
+ outcome](#the-stuck-signal--i-cannot-proceed-ws5) and the route's `message`
521
+ (or description) as the lead-in — the consumer interprets it. A route with
522
+ neither is just a label. A delegation counts against the same delegation
523
+ depth cap as a subagent (`INSIKA_SUBAGENT_DEPTH_CAP`, default 5), so a pair
524
+ of agents routing to each other stops instead of looping.
525
+
526
+ Absent `routes` = no classification, no extra call, byte-identical turn.
527
+
458
528
  #### Operator alerts — the webhook (WS6)
459
529
 
460
530
  Three operational events — `budget_warning`, `breaker_open`, `delivery_failed` —
@@ -562,6 +632,56 @@ allowlist, one or more `proposers`, a token `budget`, and a gate that replays th
562
632
  golden set before anything reaches a human. All of it is in
563
633
  [Refinement](REFINEMENT.md); none of it is on until you name it.
564
634
 
635
+ ## Distillation of customer facts
636
+
637
+ `distill` configures how finished, idle customer conversations are read back as
638
+ proposed facts — the human-gated loop documented in [Facts](FACTS.md). Pack
639
+ data, `refinement:`'s shape, and absent = off for that agent:
640
+
641
+ ```ruby
642
+ distill enabled: true,
643
+ idle_hours: 6, # how idle a session must be before it distills
644
+ min_messages: 3, # a shorter session distills noise
645
+ max_proposals: 10 # cap per session pass
646
+ # prompt: "<what counts as a fact for THIS store>" (the forge's half)
647
+ # model: "<ref — absent = the platform utility_model>"
648
+ ```
649
+
650
+ Nothing is ever applied automatically: the engine writes **proposals**, the
651
+ operator approves/rejects/dismisses them on the Studio **Facts** page (the
652
+ latch: a dismissed or rejected tuple is never proposed again), and an approval
653
+ writes the fact to the customer's memory cell stamped
654
+ `distilled:<session_ref>` through an optimistic CAS — an approval never
655
+ silently overwrites an operator edit. Sessions are the only candidates, and
656
+ the distiller rides the platform `utility_model`, never a new model slot.
657
+
658
+ ## Harvest of skills from real traffic
659
+
660
+ `harvest` configures how finished, idle conversations are read back as
661
+ proposed **SKILLS** for the agent's playbook — the human-gated loop documented
662
+ in [Harvest](HARVEST.md). Pack data, `distill:`'s shape, and absent = off for
663
+ that agent:
664
+
665
+ ```ruby
666
+ harvest enabled: true,
667
+ negative_list: [ { rule: "no-competitor-prices", pattern: "concorrente" } ],
668
+ miner: { model: "deepseek-v4-flash", # absent = the platform utility_model
669
+ window: { last_sessions: 200 } },
670
+ idle_hours: 24,
671
+ min_messages: 3
672
+ # prompt: "<what a harvestable skill is for THIS store>" (the forge's half)
673
+ ```
674
+
675
+ The loop reads only finished traffic (the fork is structural — the mining
676
+ writes nothing to the sessions it read), filters every proposal through the
677
+ negative list and the evidence ledger (product claims must reference IDs the
678
+ origin sessions actually saw — an agent without `grounding.matcher.sku` does
679
+ not mine at all), scores survivors with a double gate (the eval replay against
680
+ the clone's golden set, judges mandatory in the three P18 shapes; the
681
+ conversion "not worse" check against the RFC-0032 frozen baseline), and lands
682
+ a skill **only after a human approves** — snapshot-first, append-only
683
+ promotion log, deterministic rollback. Nothing is ever applied automatically.
684
+
565
685
  ## Delegation (subagents)
566
686
 
567
687
  An agent can delegate to **subagents**: named child agents it may invoke as a
@@ -608,6 +728,261 @@ next turn with no restart. See [Deploy](DEPLOY.md) for the durable-volume setup
608
728
  [Context](CONTEXT.md#the-volume) for why editing a committed file does *not* change
609
729
  a running agent.
610
730
 
731
+ ## Media in the message contract (WS9)
732
+
733
+ The engine transports media, it never means it. The message accepts additive
734
+ **content parts** alongside the text — voice notes and photos travel, and any
735
+ skill (a fitting room, an image QA) stays a consumer layer on top:
736
+
737
+ ```bash
738
+ curl -X POST /v1/messages?stream=false -H "Authorization: Bearer $TOKEN" \
739
+ -d '{ "agent": "store-support", "session_id": "chat-7",
740
+ "message": "", "parts": [
741
+ { "type": "audio", "url": "https://cdn.example.com/voz.ogg" },
742
+ { "type": "image", "url": "https://cdn.example.com/sofa.jpg" }
743
+ ] }'
744
+ ```
745
+
746
+ - **Audio** is transcribed (RubyLLM STT; model via `INSIKA_STT_MODEL`) and the
747
+ text enters the turn marked `source: "voice"` on the terminal event — the
748
+ consumer's signal the person spoke. A consumer that transcribes itself can
749
+ send the text with `"source": "voice"` directly.
750
+ - **Images** attach to the model's ask (vision); the provider bills them and
751
+ the usage flows like any ask. The first image URL is also
752
+ `{{ctx.image_url}}` for data tools — photo analysis outside the prompt, the
753
+ tool's own egress applying when it fetches. Media URLs (audio AND image) are fetched by
754
+ the engine through the same egress guard (a private/metadata target is
755
+ refused — SSRF) and the same size ceiling (1 MB audio, 5 MB image: the bytes
756
+ land in this process). A refused, oversized or unreadable part fails the turn
757
+ loudly at the `:media` stage, never a silent drop.
758
+ - **Media alone is a turn.** A voice note with no caption is `parts` and an
759
+ empty `message` — the transcription becomes the message at the `:media`
760
+ stage. A media message never joins another turn (`collect`/`steer` move text
761
+ only, and the parts would be left behind), and a transcription that comes
762
+ back empty fails the turn instead of asking the model about nothing.
763
+ - **Parts are contract at the edge** — a malformed part (unknown type, an
764
+ image/audio without `url`, a text without `text`) is a 422 before dispatch
765
+ on `/v1/messages` and `/v1/responses`.
766
+ - `/v1/responses` accepts the OpenAI multimodal shape: `input` as an array of
767
+ text/image/audio parts.
768
+
769
+ ## Generated media as outputs (WS9, saída)
770
+
771
+ The turn can **produce** an image or a voice clip — but only when both sides of
772
+ the gate agree, because nothing leaks by default. The agent declares it may
773
+ generate media (`outputs` on the profile), and the **channel** declares it can
774
+ receive it (`channel.capabilities` on the request):
775
+
776
+ ```bash
777
+ curl -X POST /v1/responses -H "Authorization: Bearer $TOKEN" -d '{
778
+ "model": "openclaw:store-support", "user": "chat-7",
779
+ "input": "manda a foto do sofá da promoção",
780
+ "channel": { "capabilities": ["image_output", "audio_output"] }
781
+ }'
782
+ ```
783
+
784
+ ```ruby
785
+ agent = Insika.agent("store-support") do
786
+ instructions "…"
787
+ outputs image: { model: "gpt-image-1", size: "1024x1024" }, # the AGENT's half
788
+ tts: { model: "tts-1", voice: "alloy" }
789
+ end
790
+ ```
791
+
792
+ - **Both gates** must pass for the model to even see the `generate_image` /
793
+ `tts` tools: the agent opted in (`outputs`) and the request declared the
794
+ matching capability (`image_output` / `audio_output` — an unknown value is a
795
+ 422, never a silent ignore). The "abstraction admits only what leaks" rule.
796
+ - **The media rides the envelope, never the answer text.** The terminal event
797
+ and the `/v1/responses` completed frame carry an additive `output_parts`
798
+ array — `{ type: "image", mime_type:, base64:, model: }` /
799
+ `{ type: "audio", mime_type:, base64:, model: }`. The model's prose stays
800
+ the `:content` answer; the channel consumes the bytes next to it.
801
+ - **Generation is billed and counted.** Image tokens join the turn's usage
802
+ (like any ask). The speech API reports no token counts, so a TTS call adds
803
+ an honest `usage.media` counter and the part carries the `model` for
804
+ consumer-side pricing.
805
+ - **Seams, not magic.** The generator is injectable per kind (specs stub it);
806
+ the defaults are lazy: images via RubyLLM (paint), speech via a thin POST to
807
+ the OpenAI-compatible `/audio/speech` endpoint using the same provider
808
+ config the chat uses — RubyLLM as of 1.16.0 has no speech API. A generated
809
+ part over 8 MB refuses loudly, never silently truncates.
810
+ - **Not here:** what the generated image *means* — a fitting room, a product
811
+ mockup — is a skill on top. The engine transports bytes and cost.
812
+
813
+ ## Customer-scoped memory and the right to be forgotten (WS8)
814
+
815
+ Memory is naturally **per customer, not per tenant**. A message that carries a
816
+ `customer` key moves the engine's memory scope to that person:
817
+
818
+ ```bash
819
+ curl -X POST /v1/messages?stream=false -H "Authorization: Bearer $TOKEN" \
820
+ -d '{ "agent": "store-support", "session_id": "chat-7",
821
+ "customer": "c-123", "message": "cadê meu pedido" }'
822
+ ```
823
+
824
+ - **Scope** — with `customer` present, the `remember` tool and the `<memory>`
825
+ block read/write the `[tenant:]customer` cell: two customers under the same
826
+ tenant never see each other, and the `<request_context>` tenant label (the
827
+ merchant) is untouched. Absent `customer` = today's per-tenant/per-chat
828
+ behavior.
829
+ - **Right to be forgotten** — `POST /v1/commands/forget_customer` (operator)
830
+ purges the customer's memory cell, their sessions and everything those
831
+ sessions left behind — per-session traces, the tasks (the message text lives
832
+ in the persisted command), their checkpoints (the transcript) and the outbox
833
+ deliveries (the answer as it was handed to the channel) — and nothing else's:
834
+ `{ "customer": "c-123", "tenant": "acme" }`. **Name the tenant**: the
835
+ operator credential carries none, and without one the purge means the whole
836
+ deployment (the untagged memory cell, plus that customer's sessions in every
837
+ tenant) — right for a single-tenant deployment, never what a multi-tenant
838
+ operator means. Facts also support an optimistic CAS write
839
+ (`replace_if_revision`) for an integration that must not clobber a concurrent
840
+ edit.
841
+ - **Tenant deletion** — `POST /v1/commands/delete_tenant_data` (operator)
842
+ purges EVERYTHING the engine holds about one tenant: its sessions and their
843
+ whole footprint (traces, tasks, checkpoints, outbox deliveries), every memory
844
+ cell under the tenant (its own + the customer cells — enumerated from the
845
+ store, so even a cell whose session was already deleted goes) and its outcome
846
+ records: `{ "tenant": "acme" }`. Its API tokens are **revoked first** (before
847
+ the sweep): an offboarded tenant whose credentials still resolved kept
848
+ authenticating and could open a new session over the erasure. The tenant
849
+ string is the isolation boundary; a neighbour is untouched.
850
+ - **Retention** — the age-based counterpart, as data: the settings key
851
+ `retention_days` (Integer days; absent/0 = OFF, the engine never sweeps by
852
+ default). The tick's daily sweep (at most once per 24 h, behind the same
853
+ single-key claim the stale-task sweep uses) purges sessions (+traces),
854
+ terminal tasks (+checkpoints), delivered/failed outbox records, memory
855
+ facts/notes and outcomes older than the window. A non-terminal task is never
856
+ touched — the Recovery sweep owns those lives — and neither is a delivery
857
+ still owed to somebody. One thing the same daily pass sweeps **regardless of
858
+ `retention_days`**: the budget counter cells whose window already rolled over
859
+ (and their once-per-window alert markers). Those are engine bookkeeping, not
860
+ customer content, and nothing else ever collected them.
861
+
862
+ ## Outcomes — business results over real traffic (WS7)
863
+
864
+ The engine measures what it is told to measure. The operator or the integration
865
+ records a conversation's business outcome after the fact — `conversion`,
866
+ `escalation`, `deflected`, anything, optionally with a monetary `value`:
867
+
868
+ ```bash
869
+ curl -X POST /v1/outcomes -H "Authorization: Bearer $TOKEN" \
870
+ -d '{ "agent": "store-support", "session_id": "chat-7",
871
+ "outcome": "conversion", "value": 129.9 }'
872
+ ```
873
+
874
+ The endpoint is **additive and outside the response contract** — the turn never
875
+ knows or cares; the engine transports the outcome and never interprets it (what
876
+ "conversion" means is yours). Records are tenant-stamped (a tenant principal
877
+ writes and reads only its own), and `GET /v1/outcomes?agent=` serves the last
878
+ outcome per agent plus the per-day series — the last-outcome pill on the Studio
879
+ agent grid, and the per-day series on the agent detail.
880
+
881
+ ### The outcome funnel (RFC-0032)
882
+
883
+ A store's funnel is pack data on the agent — the engine folds WS7 outcomes into
884
+ the **declared** stages, and never hard-codes one itself (the stage vocabulary
885
+ is the forge's):
886
+
887
+ ```ruby
888
+ agent = Insika.agent("store-support") do
889
+ instructions "…"
890
+ funnel stages: %w[greeted qualified cart paid],
891
+ advance_on: { "abandoned_cart" => "cart", "pix_paid" => "paid" },
892
+ primary: "paid", attribution_window: "72h"
893
+ end
894
+ ```
895
+
896
+ The fold contract:
897
+
898
+ - **Tick-driven, cumulative event counts on the declared order.** An outcome of
899
+ kind K means the session *reached* `advance_on[K]`; the fold increments
900
+ `stages[0..index]` for the reached stage. A per-stage-complete integration
901
+ and a terminal-event integration therefore produce identical counts — a
902
+ session that paid also emitted the earlier events. A duplicate event
903
+ double-counts (the integration's defect, not the engine's); **do not declare
904
+ a stage off the linear path** (a "handoff" stage would be inflated by every
905
+ later event). Counts are **event counts, not distinct sessions** — the
906
+ baseline is events-based.
907
+ - **Idempotent**: a per-pair `{at, ids}` cursor inside one transaction; a crash
908
+ mid-fold never double counts, and a second pass folds only what is new.
909
+ - **The attribution window is carried data, never computed** — `72h` is
910
+ validated, rendered, and copied into the baseline snapshot; causal
911
+ attribution stays human.
912
+ - **The baseline freeze** (Studio > Funnel, or `:freeze_funnel_baseline` on the
913
+ bus) sums the folded cells over a span of **≥ 28 days** (shorter spans are
914
+ refused) into one current snapshot per `(tenant, agent)` — the number
915
+ RFC-0033 (follow-up A/B) and RFC-0035 (promotion gate) compare against.
916
+ - **Malformed declarations never crash the tick**: the fold skips them, the
917
+ doctor names the defect, the Studio shows nothing until it is fixed.
918
+ - Vocabulary note: in the gem this is the **outcome funnel** — the stage names
919
+ are the forge's, and a bare install (no `funnel:` on any agent) shows no
920
+ funnel and no stage names at all.
921
+
922
+ ## Follow-ups — the seller who comes back (RFC-0033)
923
+
924
+ The agent can book a follow-up with a customer at a future time — "te chamo
925
+ amanhã se o PIX não cair" said in-conversation and meant. The engine fires the
926
+ synthetic turn on its own tick, with consent and without spam. Everything is
927
+ pack data on the profile:
928
+
929
+ ```ruby
930
+ agent = Insika.agent("store-support") do
931
+ instructions "…"
932
+ followup arm: "schedule",
933
+ policy: { quiet_hours: { timezone: "America/Sao_Paulo",
934
+ start: "21:30", end: "09:00" },
935
+ max_frequency: "2/24h", # N outbound per window, per customer
936
+ cancel_keywords: ["não quero mais contato"],
937
+ silence_after_sends: 3 } # N fires without a reply -> :unavailable
938
+ end
939
+ ```
940
+
941
+ The pieces:
942
+
943
+ - **`schedule(at:, reason:)`** — a built-in tool the agent calls when the
944
+ customer agrees to be contacted again (a product, a cart, a pending payment).
945
+ The call itself IS the consent record — recorded without ever lifting
946
+ `:unavailable` or resetting the silence counter (ONLY a customer message
947
+ reopens, so a re-booking inside a follow-up turn cannot clear the silence
948
+ protection). `cancel_followup(id:)` is the sibling. A customer who opted out
949
+ can never be rescheduled.
950
+ - **Contact state per customer** — `granted | revoked | unavailable` in a
951
+ durable cell per `(tenant, customer)`. Only `granted` may be messaged;
952
+ `revoked` is immediate and permanent until the customer speaks again;
953
+ `unavailable` means silence ≠ refusal — the engine stops firing after
954
+ `silence_after_sends` unanswered sends, and ANY customer message reopens.
955
+ The policy's `cancel_keywords` are matched on every inbound message: a
956
+ match revokes the contact and cancels its pending follow-ups in one
957
+ transaction.
958
+ - **Firing is the tick's third duty** — the engine claims the due records
959
+ (one per claim window, at-most-once across workers), applies the policy in
960
+ force AT FIRE TIME (contact state, quiet hours, dedup per
961
+ `(customer, reason)`, frequency ceiling) and either enqueues the synthetic
962
+ turn or marks the record `blocked` with the failing rule — auditable, never
963
+ silent. Blocking happens at fire time, never at schedule time: the schedule
964
+ is a promise made in-conversation, and only the policy in force then may
965
+ revoke it.
966
+ - **The synthetic turn** — a first-class inbound turn stamped
967
+ `origin: "scheduled"` (a refinement read can never mistake the engine's
968
+ kick for the customer repeating themselves), delivered through the full
969
+ pipeline on the channel the conversation came in on. It skips the edge's
970
+ ENTRY rate/token checks like a resume does — a follow-up she agreed to must
971
+ not receive the rate-limit reply; its usage still lands on the ledger.
972
+ - **The Follow-ups page** (Studio) — per agent: the pending/fired/cancelled/
973
+ blocked records (blocked rows carry the reason), the read-only policy
974
+ summary and the A/B card: per arm, `sent` vs `conversions` (against the
975
+ RFC-0032 baseline) vs `opt-outs`. The only mutations — cancel a pending
976
+ record, force-revoke a contact — go through bus commands.
977
+ - **LGPD** — the records and cells die with the customer (`forget_customer`),
978
+ the tenant (`delete_tenant_data`) and age out under the same
979
+ `retention_days` sweep as the rest of the footprint.
980
+
981
+ Absent `followup:` = the feature is off for that agent — no tools wired, no
982
+ records, byte-identical turns. The A/B against an existing cron is an
983
+ operator experiment: the engine only keeps the records and the read card (the
984
+ cron arm writes through the same store class with its own `arm` label).
985
+
611
986
  ## See also
612
987
 
613
988
  - [Tools](TOOLS.md) — define, register, and troubleshoot tools.
data/docs/API.md ADDED
@@ -0,0 +1,56 @@
1
+ ---
2
+ title: The /v1 API
3
+ parent: Build an agent
4
+ nav_order: 9
5
+ permalink: /api/
6
+ ---
7
+
8
+ # The /v1 API — the frozen compatibility contract
9
+
10
+ The HTTP surface is the drop-in OpenAI-Responses-compatible contract: a client
11
+ that speaks `/v1` today speaks it tomorrow. This page is the WRITTEN promise
12
+ (RFC-0036 C6) — the mechanical half is the version gate in the server, and the
13
+ two cannot drift (a spec pins the gate's version to the date below).
14
+
15
+ ## The surface
16
+
17
+ | Route | What it is |
18
+ |-------|------------|
19
+ | `POST /v1/responses` | the OpenAI-Responses-compatible turn ingress (streaming SSE) |
20
+ | `POST /v1/messages` | the channel-style message ingress (one turn per message, ack-fast) |
21
+ | `POST /v1/agents` | provisioning by definition/pack (idempotent, authoritative) |
22
+ | `GET/DELETE /v1/agents` | list / remove agents |
23
+ | `POST /v1/sessions` | create a session explicitly |
24
+ | `POST /v1/outcomes` | record a business outcome (`conversion`, `escalation`, any label) |
25
+ | `GET /v1/outcomes` | last outcome per agent + the per-day series |
26
+ | `POST /v1/commands/...` | operator commands (`forget_customer`, `delete_tenant_data`, the follow-up mutations) |
27
+ | `GET /v1/vitals` | process health/uptime vitals |
28
+ | `GET /v1/...` | the onboarding surface (`start.md`, `models.json`, `/docs/<name>.md`) |
29
+
30
+ The turn endpoints speak the OpenAI `response.completed` wire format; the
31
+ `Insika-Version` header declares the compatibility vintage of a request.
32
+
33
+ ## The compatibility rule
34
+
35
+ **Request and response shapes are stable; changes are additive-only.** A new
36
+ field on a request or response is a release. Removing, renaming or re-typing a
37
+ field is a BREAKING change and requires, together, all three:
38
+
39
+ 1. a new `Insika-Version` entry in the server's known-versions gate (an
40
+ unknown value is refused with 400 before auth — fail-closed);
41
+ 2. a route branch that serves both vintages (a compatibility branch, per
42
+ RFC-0016 A5), so an old client keeps working while the new shape rolls;
43
+ 3. a new `**Frozen as of:**` line below.
44
+
45
+ Additive changes need none of the three.
46
+
47
+ **Frozen as of: 2026-08-08** — the date that is the 1.0 promise. The gate's
48
+ first known version, pinned by `spec/insika/server/version_gate_spec.rb`; the
49
+ release writes the 1.0 freeze date here at release time.
50
+
51
+ ## The 400-on-unknown rule
52
+
53
+ A request carrying an `Insika-Version` the gate does not know is refused with
54
+ `400 Bad Request` BEFORE authentication — a client that speaks a future or
55
+ misspelled vintage learns immediately, and an unauthenticated probe cannot
56
+ distinguish the gate from auth (fail-closed, constant-time).
data/docs/CHANNELS.md CHANGED
@@ -181,16 +181,19 @@ whole turn and read the answer off it — or the relay, where the engine acks in
181
181
  milliseconds and POSTs the answer to you when it exists.
182
182
 
183
183
  The instinct is that streaming gets the customer their reply sooner, and that the
184
- relay trades that away. **It does not, and the reason is structural:** the engine
185
- publishes `:content` as the ANSWER, whole, after the turn's hooks
186
- ([what crosses the edge](/architecture/#what-crosses-the-edge)). During the turn the
187
- stream carries tool activity; the text arrives in one piece at the end. Measured on
188
- a real store agent, the text frames span **0 ms** — there is nothing to deliver
189
- progressively, on either path.
184
+ relay trades that away. **For the default relay it does not, and the reason is
185
+ structural:** the engine publishes `:content` as the ANSWER, whole, after the
186
+ turn's hooks ([what crosses the edge](/architecture/#what-crosses-the-edge)).
187
+ During the turn the stream carries tool activity; the text arrives in one piece at
188
+ the end. Measured on a real store agent, the text frames span **0 ms** — there is
189
+ nothing to deliver progressively. That is the `:at_end` fact, true of `/v1/responses`
190
+ and of a relay that never opted into [progressive delivery](#delivery-policy).
191
+ The relay can opt out of it per channel — the opt-in is below, and it is the one
192
+ thing that changes the 0 ms span.
190
193
 
191
194
  | | drop-in `/v1/responses` | relay |
192
195
  |---|---|---|
193
- | What the customer receives | one message, at the end | one message, at the end |
196
+ | What the customer receives | one message, at the end | one message at the end (`:at_end`), or one WhatsApp balloon per paragraph with the first one as soon as the answer exists (`delivery: :progressive`) |
194
197
  | Your app's request | held open for the whole turn (seconds) | acked in **milliseconds** |
195
198
  | A turn that outlives your HTTP timeout | your problem | already handled — the answer arrives later |
196
199
  | Retry on a failed handover | yours to build | the engine's outbox, bounded, at-most-once |
@@ -285,12 +288,14 @@ with the default `followup` you will only see `202` and `duplicate`.
285
288
 
286
289
  ### Outbound
287
290
 
288
- One POST per reply, to the URL you configured:
291
+ One POST per **balloon**, to the URL you configured. For the default `:at_end`
292
+ delivery a turn emits exactly one; a progressive turn may emit several — `task_id`
293
+ is the correlation, `index` is the order.
289
294
 
290
295
  ```jsonc
291
296
  POST <INSIKA_RELAY_DELIVER_URL>
292
297
  Authorization: Bearer <INSIKA_RELAY_DELIVER_TOKEN> // omitted if unset
293
- X-Insika-Delivery: 0f2c… // stable idempotency key
298
+ X-Insika-Delivery: 0f2c… // stable idempotency key, PER balloon
294
299
  Content-Type: application/json
295
300
 
296
301
  {
@@ -310,6 +315,46 @@ stays internal unless the agent opts in. That contract is
310
315
  [the edge contract](/architecture/#what-crosses-the-edge), and it is why you can
311
316
  forward `content` straight to the customer.
312
317
 
318
+ ### Delivery policy
319
+
320
+ A relay declares **how the outbox flushes** — a property of the surface
321
+ (WhatsApp balloons), not of the model:
322
+
323
+ - **`:at_end`** (the default, also `INSIKA_RELAY_DELIVERY=at_end` or unset): one
324
+ POST at the end with the whole answer. Byte-identical to the contract above.
325
+ - **`:progressive`** (`INSIKA_RELAY_DELIVERY=progressive`): the answer is split
326
+ into balloons at paragraph boundaries (a soft 600-char cap splits a single long
327
+ paragraph on sentences; fenced code blocks are atomic), and each balloon is its
328
+ own POST, in order, starting as soon as the answer exists. The consumer sends
329
+ each POST as its own platform message.
330
+
331
+ A progressive POST carries two additive fields **only when the turn split into
332
+ more than one balloon** — a one-balloon progressive turn is indistinguishable
333
+ from `:at_end` on the wire:
334
+
335
+ ```jsonc
336
+ {
337
+ "external_id": "5511999998888",
338
+ "session_id": "relay:5511999998888",
339
+ "task_id": "…",
340
+ "content": "Seu pedido saiu para entrega hoje",
341
+ "index": 0, // this balloon's position, 0-based
342
+ "final": false // true on the LAST balloon of this task_id
343
+ }
344
+ ```
345
+
346
+ When `index`/`final` are present, this POST is **one balloon of several for the
347
+ same `task_id`** — forward `content` as its own platform message and honor
348
+ `X-Insika-Delivery` per balloon. If you only forward `content` and ignore the new
349
+ fields, progressive turns still read as N messages in arrival order (which is
350
+ index order — the engine dispatches the chain sequentially); single-balloon turns
351
+ behave exactly as before. A consumer that ignores unknown keys keeps working.
352
+
353
+ The engine measures the win in-process: every channel turn records
354
+ `first_balloon_ms` (inbound receipt → first outbox flush) on the task record and
355
+ the terminal event, so the Studio task page shows whether the 2 s target is being
356
+ hit without toggling any flag.
357
+
313
358
  ### Deduplication
314
359
 
315
360
  Send `event_id` and a retried webhook costs you nothing: the engine recognizes the
@@ -344,6 +389,7 @@ Three environment variables on the engine:
344
389
  INSIKA_RELAY_TOKEN=<a long random secret> # the switch AND the credential
345
390
  INSIKA_RELAY_DELIVER_URL=https://you.example/insika/deliver
346
391
  INSIKA_RELAY_DELIVER_TOKEN=<another secret> # optional; what we send to you
392
+ INSIKA_RELAY_DELIVERY=progressive # optional; "at_end" (the default) = one POST
347
393
  ```
348
394
 
349
395
  `INSIKA_RELAY_TOKEN` is the switch: without it the channel is not mounted and
@@ -367,6 +413,50 @@ nothing at all).
367
413
  A runnable consumer in ~40 lines lives in
368
414
  [`examples/relay-channel/`](https://github.com/guizaols/insika/tree/main/examples/relay-channel).
369
415
 
416
+ ## Shadow mode
417
+
418
+ Shadow mode (RFC-0025) lets one channel run every turn **end to end and deliver
419
+ nothing** — the experiment that answers "can we replace the incumbent?" before
420
+ any customer is handed over. The incumbent keeps answering; the engine records
421
+ what it *would* have answered, and the two replies are judged pairwise against a
422
+ **frozen criterion** — the file `INSIKA_PARITY_CRITERION` points at.
423
+
424
+ ```bash
425
+ INSIKA_RELAY_SHADOW=1 # the switch
426
+ # INSIKA_PARITY_CRITERION — required in shadow mode
427
+ ```
428
+
429
+ Three things change when it is on:
430
+
431
+ - The turn still runs; the reply is recorded as a **pair** and never reaches the
432
+ customer. Zero outbox records, ever — and `Relay#deliver` refuses loudly if one
433
+ somehow exists.
434
+ - The inbound ack becomes `200 {"task_id": …, "shadow": true}` instead of `202`,
435
+ so a consumer wired to "202 means a reply is coming" cannot be misled.
436
+ - `event_id` becomes **required** — it is the correlation key both halves of the
437
+ pair are built from.
438
+
439
+ The incumbent's reply enters the same pair through one of two shapes: alongside
440
+ the mirror call itself (`"incumbent_reply": "…"` on `POST /channels/relay/events`),
441
+ or as a follow-up when the consumer answers first:
442
+
443
+ ```jsonc
444
+ POST /channels/relay/shadow-reply
445
+ Authorization: Bearer <INSIKA_RELAY_TOKEN>
446
+ { "external_id": "5511999998888", "event_id": "wamid.HBg…",
447
+ "reply": "Claro! Me passa o número do pedido?", "at": "2026-…Z" }
448
+ → 202 { "pair_id": "9f2c…", "status": "open" }
449
+ ```
450
+
451
+ Both shapes land in one command; a retried reply is ignored (first write wins —
452
+ the customer received one reply, and a retry must not rewrite evidence).
453
+
454
+ **No criterion, no shadow.** Boot refuses when shadow is on and
455
+ the criterion file is missing or unparseable — a number nobody pre-registered
456
+ does not count. The Studio's Parity page folds the running verdict on demand
457
+ from the pair store; `insika doctor` reports the shadow configuration before
458
+ boot does.
459
+
370
460
  ## Sessions
371
461
 
372
462
  The engine mints the session id: `relay:<your external_id>`. Namespacing is not