insika 0.0.1 → 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (260) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGELOG.md +295 -0
  3. data/LICENSE +21 -0
  4. data/README.md +136 -2
  5. data/bin/insika +351 -0
  6. data/docs/AGENTS.md +494 -0
  7. data/docs/ARCHITECTURE.md +333 -0
  8. data/docs/BENCHMARK.md +114 -0
  9. data/docs/CHANNELS.md +453 -0
  10. data/docs/CONTEXT.md +100 -0
  11. data/docs/DEPLOY.md +334 -0
  12. data/docs/EMBEDDING.md +194 -0
  13. data/docs/EVALS.md +273 -0
  14. data/docs/LOADTEST.md +231 -0
  15. data/docs/OBSERVABILITY.md +365 -0
  16. data/docs/PLUGINS.md +211 -0
  17. data/docs/REFINEMENT.md +477 -0
  18. data/docs/RELEASING.md +70 -0
  19. data/docs/RUNNING-LOCAL.md +153 -0
  20. data/docs/SANDBOX.md +114 -0
  21. data/docs/SECURITY.md +362 -0
  22. data/docs/SKILLS.md +98 -0
  23. data/docs/TOOLS.md +302 -0
  24. data/docs/WHY.md +137 -0
  25. data/docs/WORKFLOWS.md +225 -0
  26. data/docs/build.md +14 -0
  27. data/docs/index.md +68 -0
  28. data/docs/onboarding/start.md +126 -0
  29. data/docs/operate.md +12 -0
  30. data/docs/ship.md +10 -0
  31. data/docs/understand.md +10 -0
  32. data/lib/insika/agent_file_store.rb +125 -0
  33. data/lib/insika/agent_profile.rb +188 -0
  34. data/lib/insika/allowlist.rb +28 -0
  35. data/lib/insika/baseline_store.rb +74 -0
  36. data/lib/insika/capability/resolved_tool.rb +34 -0
  37. data/lib/insika/capability_registry.rb +112 -0
  38. data/lib/insika/channel_delivery.rb +150 -0
  39. data/lib/insika/channel_registry.rb +30 -0
  40. data/lib/insika/channels/relay.rb +178 -0
  41. data/lib/insika/channels/web/widget.js +283 -0
  42. data/lib/insika/channels/web.rb +211 -0
  43. data/lib/insika/chat_builder.rb +254 -0
  44. data/lib/insika/checkpoint.rb +13 -0
  45. data/lib/insika/checkpoint_store.rb +153 -0
  46. data/lib/insika/coercion.rb +50 -0
  47. data/lib/insika/command.rb +32 -0
  48. data/lib/insika/command_bus.rb +39 -0
  49. data/lib/insika/commands/agent_payload.rb +41 -0
  50. data/lib/insika/commands/approve_action.rb +46 -0
  51. data/lib/insika/commands/cancel_task.rb +33 -0
  52. data/lib/insika/commands/create_agent.rb +54 -0
  53. data/lib/insika/commands/create_session.rb +67 -0
  54. data/lib/insika/commands/delete_agent.rb +33 -0
  55. data/lib/insika/commands/delete_agent_file.rb +50 -0
  56. data/lib/insika/commands/delete_data_tool.rb +33 -0
  57. data/lib/insika/commands/delete_llm_provider.rb +36 -0
  58. data/lib/insika/commands/delete_mcp.rb +30 -0
  59. data/lib/insika/commands/delete_system_file.rb +29 -0
  60. data/lib/insika/commands/gate_refinement.rb +245 -0
  61. data/lib/insika/commands/import_mcp_tools.rb +48 -0
  62. data/lib/insika/commands/import_tools.rb +81 -0
  63. data/lib/insika/commands/memory_add_note.rb +32 -0
  64. data/lib/insika/commands/memory_forget_fact.rb +32 -0
  65. data/lib/insika/commands/memory_put_fact.rb +35 -0
  66. data/lib/insika/commands/pause_task.rb +29 -0
  67. data/lib/insika/commands/resolve_refinement.rb +126 -0
  68. data/lib/insika/commands/restore_agent_file.rb +36 -0
  69. data/lib/insika/commands/restore_data_tool.rb +34 -0
  70. data/lib/insika/commands/restore_system_file.rb +31 -0
  71. data/lib/insika/commands/resume_task.rb +85 -0
  72. data/lib/insika/commands/run_refinement.rb +133 -0
  73. data/lib/insika/commands/send_message.rb +150 -0
  74. data/lib/insika/commands/set_agent_tools.rb +39 -0
  75. data/lib/insika/commands/set_skill_agents.rb +71 -0
  76. data/lib/insika/commands/trigger_workflow.rb +80 -0
  77. data/lib/insika/commands/update_agent.rb +49 -0
  78. data/lib/insika/commands/update_settings.rb +33 -0
  79. data/lib/insika/commands/upsert_llm_provider.rb +34 -0
  80. data/lib/insika/commands/upsert_mcp.rb +32 -0
  81. data/lib/insika/commands/write_agent_file.rb +57 -0
  82. data/lib/insika/commands/write_data_tool.rb +43 -0
  83. data/lib/insika/commands/write_golden.rb +58 -0
  84. data/lib/insika/commands/write_skill.rb +50 -0
  85. data/lib/insika/commands/write_system_file.rb +31 -0
  86. data/lib/insika/config_store.rb +85 -0
  87. data/lib/insika/context/builder.rb +166 -0
  88. data/lib/insika/context/catalog_provider.rb +23 -0
  89. data/lib/insika/context/fragment.rb +19 -0
  90. data/lib/insika/context/priority.rb +29 -0
  91. data/lib/insika/context/provider.rb +19 -0
  92. data/lib/insika/context/providers/memory.rb +60 -0
  93. data/lib/insika/context/providers/prompt.rb +105 -0
  94. data/lib/insika/context/providers/request.rb +32 -0
  95. data/lib/insika/context/providers/session.rb +108 -0
  96. data/lib/insika/context/providers/skill.rb +20 -0
  97. data/lib/insika/context/providers/tool_search.rb +20 -0
  98. data/lib/insika/delegation_store.rb +153 -0
  99. data/lib/insika/doctor.rb +294 -0
  100. data/lib/insika/dsl/definition.rb +55 -0
  101. data/lib/insika/dsl/runtime.rb +379 -0
  102. data/lib/insika/dsl/server_boot.rb +97 -0
  103. data/lib/insika/dsl/system.rb +93 -0
  104. data/lib/insika/dsl/workflow_adapter.rb +59 -0
  105. data/lib/insika/dsl.rb +307 -0
  106. data/lib/insika/edge_limiter.rb +130 -0
  107. data/lib/insika/egress_guard.rb +75 -0
  108. data/lib/insika/env_schema.rb +246 -0
  109. data/lib/insika/errors.rb +145 -0
  110. data/lib/insika/evals/assertions.rb +247 -0
  111. data/lib/insika/evals/baseline.rb +69 -0
  112. data/lib/insika/evals/golden.rb +172 -0
  113. data/lib/insika/evals/judge.rb +225 -0
  114. data/lib/insika/evals/pairwise.rb +178 -0
  115. data/lib/insika/evals/report.rb +115 -0
  116. data/lib/insika/evals/runner.rb +141 -0
  117. data/lib/insika/evals/transport.rb +178 -0
  118. data/lib/insika/event.rb +18 -0
  119. data/lib/insika/event_stream.rb +114 -0
  120. data/lib/insika/executor.rb +1680 -0
  121. data/lib/insika/frontmatter.rb +42 -0
  122. data/lib/insika/golden_store.rb +145 -0
  123. data/lib/insika/hooks.rb +48 -0
  124. data/lib/insika/http_client.rb +63 -0
  125. data/lib/insika/inbound_log.rb +84 -0
  126. data/lib/insika/llm_configurator.rb +99 -0
  127. data/lib/insika/llm_provider_store.rb +83 -0
  128. data/lib/insika/mcp_http_client.rb +67 -0
  129. data/lib/insika/mcp_store.rb +115 -0
  130. data/lib/insika/mcp_tool_ingestor.rb +143 -0
  131. data/lib/insika/memory_store.rb +93 -0
  132. data/lib/insika/message_origin.rb +76 -0
  133. data/lib/insika/middleware.rb +36 -0
  134. data/lib/insika/model_policy.rb +52 -0
  135. data/lib/insika/model_resolver.rb +176 -0
  136. data/lib/insika/model_selection.rb +114 -0
  137. data/lib/insika/onboarding.rb +208 -0
  138. data/lib/insika/outbox_store.rb +166 -0
  139. data/lib/insika/overlay_tool_registry.rb +103 -0
  140. data/lib/insika/pack.rb +102 -0
  141. data/lib/insika/pack_importer.rb +121 -0
  142. data/lib/insika/pending_action_store.rb +120 -0
  143. data/lib/insika/plugin/loader.rb +356 -0
  144. data/lib/insika/plugin.rb +35 -0
  145. data/lib/insika/policy/engine.rb +83 -0
  146. data/lib/insika/policy/policy.rb +120 -0
  147. data/lib/insika/policy_registry.rb +23 -0
  148. data/lib/insika/profile_source.rb +137 -0
  149. data/lib/insika/prompt_catalog.rb +61 -0
  150. data/lib/insika/queue_policy.rb +167 -0
  151. data/lib/insika/recovery.rb +127 -0
  152. data/lib/insika/refinement/candidate.rb +159 -0
  153. data/lib/insika/refinement/evidence_collector.rb +371 -0
  154. data/lib/insika/refinement/gate.rb +234 -0
  155. data/lib/insika/refinement/panel.rb +222 -0
  156. data/lib/insika/refinement/proposer.rb +262 -0
  157. data/lib/insika/refinement_store.rb +295 -0
  158. data/lib/insika/registry.rb +59 -0
  159. data/lib/insika/safety/config.rb +109 -0
  160. data/lib/insika/safety/detectors.rb +176 -0
  161. data/lib/insika/safety/factory.rb +102 -0
  162. data/lib/insika/safety/input_guardrail.rb +87 -0
  163. data/lib/insika/safety/moderator.rb +86 -0
  164. data/lib/insika/safety/output_filter.rb +79 -0
  165. data/lib/insika/safety/output_validator.rb +101 -0
  166. data/lib/insika/safety/safe_responses.rb +47 -0
  167. data/lib/insika/sandbox/boundary.rb +93 -0
  168. data/lib/insika/sandbox/docker.rb +74 -0
  169. data/lib/insika/sandbox/local.rb +33 -0
  170. data/lib/insika/sandbox/runner.rb +80 -0
  171. data/lib/insika/sandbox.rb +85 -0
  172. data/lib/insika/schema_guard.rb +147 -0
  173. data/lib/insika/secret_masking.rb +34 -0
  174. data/lib/insika/server/a2a/agent_card.rb +27 -0
  175. data/lib/insika/server/a2a/app.rb +112 -0
  176. data/lib/insika/server/a2a/client.rb +101 -0
  177. data/lib/insika/server/a2a/errors.rb +32 -0
  178. data/lib/insika/server/a2a/http.rb +42 -0
  179. data/lib/insika/server/a2a/message.rb +27 -0
  180. data/lib/insika/server/a2a/protocol.rb +45 -0
  181. data/lib/insika/server/a2a/remotes.rb +25 -0
  182. data/lib/insika/server/a2a/task_projection.rb +40 -0
  183. data/lib/insika/server/admin_auth.rb +29 -0
  184. data/lib/insika/server/app.rb +850 -0
  185. data/lib/insika/server/boot.rb +119 -0
  186. data/lib/insika/server/rack_app.rb +110 -0
  187. data/lib/insika/server/responses.rb +155 -0
  188. data/lib/insika/server/sse_body.rb +96 -0
  189. data/lib/insika/session_actor.rb +162 -0
  190. data/lib/insika/session_store.rb +143 -0
  191. data/lib/insika/settings_store.rb +154 -0
  192. data/lib/insika/shutdown.rb +125 -0
  193. data/lib/insika/skill_catalog.rb +113 -0
  194. data/lib/insika/skill_store.rb +79 -0
  195. data/lib/insika/steer_injector.rb +110 -0
  196. data/lib/insika/store.rb +52 -0
  197. data/lib/insika/stores/memory.rb +123 -0
  198. data/lib/insika/stores/sqlite.rb +183 -0
  199. data/lib/insika/studio/app.rb +1571 -0
  200. data/lib/insika/studio/assets/dist/application.css +1 -0
  201. data/lib/insika/studio/assets/dist/application.js +69 -0
  202. data/lib/insika/studio/forms.rb +340 -0
  203. data/lib/insika/studio/nav_icons.rb +31 -0
  204. data/lib/insika/studio/views/_message.erb +44 -0
  205. data/lib/insika/studio/views/agent_detail.erb +285 -0
  206. data/lib/insika/studio/views/agents.erb +63 -0
  207. data/lib/insika/studio/views/approvals.erb +41 -0
  208. data/lib/insika/studio/views/chats.erb +34 -0
  209. data/lib/insika/studio/views/evals.erb +83 -0
  210. data/lib/insika/studio/views/home.erb +72 -0
  211. data/lib/insika/studio/views/layout.erb +94 -0
  212. data/lib/insika/studio/views/login.erb +17 -0
  213. data/lib/insika/studio/views/mcp.erb +91 -0
  214. data/lib/insika/studio/views/not_found.erb +5 -0
  215. data/lib/insika/studio/views/playground.erb +47 -0
  216. data/lib/insika/studio/views/refinement.erb +234 -0
  217. data/lib/insika/studio/views/session.erb +62 -0
  218. data/lib/insika/studio/views/settings.erb +173 -0
  219. data/lib/insika/studio/views/skills.erb +86 -0
  220. data/lib/insika/studio/views/system_files.erb +65 -0
  221. data/lib/insika/studio/views/task.erb +105 -0
  222. data/lib/insika/studio/views/tasks.erb +33 -0
  223. data/lib/insika/studio/views/tool_edit.erb +107 -0
  224. data/lib/insika/studio/views/tools.erb +89 -0
  225. data/lib/insika/subagent_graph.rb +96 -0
  226. data/lib/insika/system_file_store.rb +96 -0
  227. data/lib/insika/task_actor.rb +128 -0
  228. data/lib/insika/task_store.rb +250 -0
  229. data/lib/insika/telemetry/pricing.rb +104 -0
  230. data/lib/insika/telemetry/recorder.rb +228 -0
  231. data/lib/insika/telemetry.rb +127 -0
  232. data/lib/insika/testing/store_contract.rb +270 -0
  233. data/lib/insika/token_estimator.rb +16 -0
  234. data/lib/insika/tool_assembly.rb +140 -0
  235. data/lib/insika/tool_catalog.rb +89 -0
  236. data/lib/insika/tool_definition.rb +518 -0
  237. data/lib/insika/tool_envelope.rb +140 -0
  238. data/lib/insika/tool_manifest.rb +218 -0
  239. data/lib/insika/tool_registry.rb +21 -0
  240. data/lib/insika/tool_store.rb +135 -0
  241. data/lib/insika/tool_trace_store.rb +92 -0
  242. data/lib/insika/tools/a2a_remote.rb +48 -0
  243. data/lib/insika/tools/agent_enum.rb +68 -0
  244. data/lib/insika/tools/concurrency.rb +54 -0
  245. data/lib/insika/tools/data_defined_tool.rb +220 -0
  246. data/lib/insika/tools/load_skill.rb +41 -0
  247. data/lib/insika/tools/remember.rb +53 -0
  248. data/lib/insika/tools/subagent.rb +75 -0
  249. data/lib/insika/tools/subagents.rb +77 -0
  250. data/lib/insika/tools/tool_search.rb +94 -0
  251. data/lib/insika/turn_output.rb +139 -0
  252. data/lib/insika/turn_state.rb +158 -0
  253. data/lib/insika/turn_timing.rb +56 -0
  254. data/lib/insika/usage_ledger.rb +47 -0
  255. data/lib/insika/version.rb +3 -1
  256. data/lib/insika/wiring/graph.rb +198 -0
  257. data/lib/insika/workflow.rb +185 -0
  258. data/lib/insika/workflow_registry.rb +33 -0
  259. data/lib/insika.rb +203 -4
  260. metadata +395 -8
data/docs/DEPLOY.md ADDED
@@ -0,0 +1,334 @@
1
+ ---
2
+ title: Deploy
3
+ parent: Ship it
4
+ nav_order: 3
5
+ permalink: /deploy/
6
+ ---
7
+
8
+ # Deploy
9
+
10
+ How to run the engine in a container (Railway today, Kubernetes later) and how to
11
+ measure performance and load.
12
+
13
+ > **This page deploys the repo, not the gem.** The reference deployment is a
14
+ > checkout of this same repository — `config/` is its composition root, and the
15
+ > `Gemfile` consumes the engine through the gemspec. An adopter deploys
16
+ > `gem install insika` (or a `Gemfile` line) instead.
17
+
18
+ ## Image (Docker)
19
+
20
+ The `Dockerfile` (multi-stage, YJIT on) serves `config.ru` under Falcon. The Studio
21
+ ships with its `dist/` built and vendored — **no Node in the build**. The backend
22
+ is durable SQLite (WAL) at `INSIKA_DB`; mount a volume and point it inside.
23
+
24
+ ```bash
25
+ docker build -t insika .
26
+ docker run -p 9292:9292 -v insika-data:/data \
27
+ -e DEEPSEEK_API_KEY=sk-... \
28
+ -e OPENCLAW_GATEWAY_TOKEN=change-me \
29
+ insika
30
+ curl localhost:9292/up # {"status":"ok"}
31
+ ```
32
+
33
+ ## The process model
34
+
35
+ The image boots **N Falcon worker processes over one SQLite file**
36
+ (`WEB_CONCURRENCY`, default 2). That number is a **contract input, not a tuning
37
+ knob**: it decides which engine semantics hold cluster-wide and which are
38
+ per-worker. The contract:
39
+
40
+ > Everything here describes N workers of **one** deployment — one graph, replicated.
41
+ > N *graphs* inside one process is a different contract, and it is
42
+ > [Embedding](EMBEDDING.md): there each graph owns its own store and credentials,
43
+ > and the host — not the engine — installs the drain.
44
+
45
+ 1. **N workers share one SQLite store.** Everything durable — sessions, tasks,
46
+ checkpoints, outbox, delegations — is cross-process state. Any status
47
+ transition that hands work to "whoever gets there first" goes through a
48
+ transactional claim (`Store#transaction`); a bare read-check-write on a
49
+ shared status field is a bug by definition.
50
+ 2. **A session's live semantics are per-worker.** Per-session FIFO ordering,
51
+ `steer`, `interrupt`, `pause`/`cancel` and the SSE watch operate on the
52
+ worker that holds the session's actor. The engine does **not** promise them
53
+ across workers. A deploy that needs those semantics for a session must route
54
+ that session's traffic to one worker (sticky routing) — or accept per-worker
55
+ best-effort.
56
+ 3. **Recovery is part of boot, in every wiring.** Every worker boots through
57
+ `Server::Boot`, which runs recovery **before the listen**. The per-record
58
+ sweeps (undelivered outbox records, undelivered delegation results) run in
59
+ every worker — each record carries its own transactional claim (item 1), so
60
+ at-most-once holds however many workers sweep. The **task sweep** runs
61
+ **once per boot generation**: the sweep's "orphaned `:running`" test cannot
62
+ see a fiber living in a *sibling* process, so the first worker to claim
63
+ `INSIKA_BOOT_ID` (one id per container start, exported by
64
+ `deploy/entrypoint.sh`) sweeps and the rest skip. A worker respawned
65
+ mid-generation skips too — sweeping then would steal its siblings' live
66
+ turns; its own orphans wait for the next generation (the next deploy).
67
+ Without `INSIKA_BOOT_ID` (single-process runs) every boot sweeps.
68
+ 4. **Shutdown is a drain, not a kill.** On SIGTERM (or SIGINT) a worker stops
69
+ accepting new turns — a turn that arrives mid-drain stays `:queued` and the
70
+ next boot's recovery replays it — and waits up to `INSIKA_DRAIN_TIMEOUT`
71
+ (default **20s**) for the in-flight ones. A second signal skips the wait.
72
+ Whatever the deadline abandons dies `:running`, and item 3 picks it up at the
73
+ next boot. The layers above must grant the time: `deploy/entrypoint.sh`
74
+ passes Falcon `--graceful-stop` = drain + 5 (Falcon's own default is 1s),
75
+ and the platform's SIGTERM→SIGKILL buffer must be ≥ drain + 10. **On Railway
76
+ that buffer defaults to 0** — SIGKILL right after SIGTERM, which cancels the
77
+ whole drain — so set `RAILWAY_DEPLOYMENT_DRAINING_SECONDS=30` on the
78
+ service.
79
+
80
+ `deploy/entrypoint.sh` sets `WEB_CONCURRENCY` next to a pointer to this section;
81
+ this section is the single source of truth for what changing it means.
82
+
83
+ ## Environment variables
84
+
85
+ | Env | Default | Effect |
86
+ |-----|---------|--------|
87
+ | `INSIKA_DB` | `/data/insika.db` (in the image) | durable SQLite path (**mount a volume!**) |
88
+ | `PORT` | `9292` | HTTP bind port |
89
+ | `WEB_CONCURRENCY` | `2` | number of Falcon worker processes — a contract input, see [The process model](#the-process-model) |
90
+ | `INSIKA_BOOT_ID` | set by `deploy/entrypoint.sh` | boot generation id; the recovery **task sweep** runs once per id (process model, item 3). Unset = every boot sweeps (single-process default) |
91
+ | `INSIKA_DRAIN_TIMEOUT` | `20` | seconds a stopping worker waits for in-flight turns before abandoning them to the next boot's recovery (process model, item 4). The entrypoint sizes Falcon's `--graceful-stop` from it; on Railway also set `RAILWAY_DEPLOYMENT_DRAINING_SECONDS` ≥ drain + 10 |
92
+ | `OPENCLAW_GATEWAY_TOKEN` | falls back to `ADMIN_TOKEN` | Bearer for `/v1/responses` and `/v1/agents` (the API contract) |
93
+ | `ADMIN_TOKEN` | `local-demo` | login token for `/studio` (**change in production**) |
94
+ | `DEEPSEEK_API_KEY` | — | provider key. **Without it the engine still boots** (`/up` green), but turns fail until it is configured (env or Studio → LLM providers) — cloud resilience |
95
+ | `DEEPSEEK_MODEL` | `deepseek-chat` | model |
96
+ | `ACHEI_INTERNAL_URL` | — | base URL for data-tools calling back a consumer's internal API (see below) |
97
+ | `INSIKA_EGRESS_HOSTS` | — | outbound host allowlist (SSRF guard) |
98
+ | `INSIKA_EGRESS_ALLOW_HTTP` / `_ALLOW_PRIVATE` | off | for `http`/loopback callbacks only (**never in cloud**) |
99
+ | `INSIKA_RELAY_TOKEN` | — | **mounts the relay channel** at `POST /channels/relay/events`, and is the Bearer it requires. Empty = the route does not exist (`404`). See [Channels](CHANNELS.md) |
100
+ | `INSIKA_RELAY_DELIVER_URL` | — | your callback; the engine POSTs each reply there. Goes through the egress guard |
101
+ | `INSIKA_RELAY_DELIVER_TOKEN` | — | Bearer the engine sends **to** your callback (optional) |
102
+ | `INSIKA_WIDGET_ORIGINS` | — | exact-match origins allowed to embed the [web widget](CHANNELS.md#the-web-widget), comma-separated. No wildcards. **Half the switch**: with `INSIKA_WIDGET_AGENTS` unset, nothing is mounted (`404`) |
103
+ | `INSIKA_WIDGET_AGENTS` | — | agent ids a widget visitor may address, comma-separated. The other half of the switch. **A chat rate limit is also required** or the widget answers `503` |
104
+ | `LITESTREAM_REPLICA_URL` | — | **enables Litestream** (backup/DR). Empty = disabled (default). See below |
105
+ | `LITESTREAM_ENDPOINT` | — | S3-compatible endpoint (R2/MinIO). Empty = AWS S3 |
106
+ | `LITESTREAM_REGION` | — | bucket region (AWS: `us-east-1`; R2: `auto`) |
107
+ | `LITESTREAM_ACCESS_KEY_ID` / `LITESTREAM_SECRET_ACCESS_KEY` | — | bucket credentials (read natively by Litestream) |
108
+
109
+ > **Renamed from `HARNESS_*` → `INSIKA_*`.** Every engine variable now uses the
110
+ > `INSIKA_` prefix. The old `HARNESS_*` names are still honored as deprecated aliases
111
+ > — set either one and the engine reads it, logging a one-line deprecation notice at
112
+ > boot (`insika doctor` reports it too). Migrate at your convenience; the legacy names
113
+ > will be dropped in a future release.
114
+
115
+ > **The database filename changed too** — the image now defaults to
116
+ > `INSIKA_DB=/data/insika.db` (it was `/data/harness.db`). **Existing volumes are
117
+ > adopted automatically:** the container entrypoint renames the old file — with its
118
+ > `-wal`/`-shm` siblings, before anything opens it — when the configured path does
119
+ > not exist yet. Nothing to run by hand, no data lost, and a no-op from the second
120
+ > boot on. To keep the old filename instead, point `INSIKA_DB` at it: the variable
121
+ > is the knob, the image only picks a default. (The adoption lives in
122
+ > `deploy/entrypoint.sh`, not in the engine — it is deploy baggage, not a runtime
123
+ > behavior.)
124
+
125
+ ### Tokens & rotation (keep the two separate!)
126
+
127
+ There are **two** secrets with distinct purposes — in production use **different**
128
+ values (the API token falling back to `ADMIN_TOKEN` is a dev convenience only):
129
+
130
+ - **`ADMIN_TOKEN`** — the `/studio` login (cookie auth). This is the **operator**
131
+ surface (just you). Rotating it is **safe and independent**: change it, redeploy,
132
+ log in with the new value. It does not affect any API consumer.
133
+ - **`OPENCLAW_GATEWAY_TOKEN`** — the Bearer for `/v1/responses` and `/v1/agents`.
134
+ This is the **contract with your API consumers**. Rotating it means **changing
135
+ both sides together** (or the integration breaks): update the runtime var **and**
136
+ each consumer's token in the same step.
137
+
138
+ Generate a strong token: `ruby -rsecurerandom -e 'puts SecureRandom.hex(24)'`.
139
+
140
+ ### Strict config and `insika doctor`
141
+
142
+ Config discipline that **rejects unknown keys — no silent schema tolerance**. Two
143
+ parts:
144
+
145
+ **1. Boot gate.** On boot, the engine validates the environment against a schema of
146
+ known keys (`Insika::EnvSchema`): a wrong type (`INSIKA_PORT=abc`) or an
147
+ **unknown key in the `INSIKA_` namespace** (a typo like `INSIKA_EGRES_ALLOW_HTTP`
148
+ the runtime would silently ignore). By **default it only warns** and boots anyway
149
+ (*last-known-good* — a rotated key or a typo never takes the whole service down).
150
+ To **refuse boot** on any finding, set `INSIKA_CONFIG_STRICT=1`. Unknown-key
151
+ detection is scoped to the `INSIKA_` prefix; the shared `OPENCLAW_`, `LITESTREAM_`,
152
+ and `OTEL_` namespaces are never flagged.
153
+
154
+ **2. `bin/insika doctor` — on-demand diagnostics.** Reads the **same** durable
155
+ backend the server uses (`INSIKA_DB`) without booting the whole app (no provider,
156
+ no seed) — safe to run against a production volume:
157
+
158
+ ```bash
159
+ insika doctor # colored report; exits != 0 on any error
160
+ insika doctor --json # machine-readable (CI / monitoring)
161
+ insika doctor --fix # applies the safe autofixes and re-diagnoses
162
+ insika env # lists known keys + current values (secrets masked)
163
+ ```
164
+
165
+ Checks: env (the schema above), settings schema version (a pending migration →
166
+ `--fix` applies it), a missing platform `default_model` (`--fix` seeds it from
167
+ `DEEPSEEK_MODEL`), durable vs ephemeral backend, LLM provider configured,
168
+ `ADMIN_TOKEN` set, data-tool definitions still valid, and **prompt files that hold
169
+ text rather than a serialized object** (a file whose content is a stringified Hash
170
+ serves a mangled prompt on every turn while looking perfectly healthy — present,
171
+ non-empty, and the agent still answers). Settings-schema migrations are **explicit**
172
+ — no Studio save silently reinterprets old-shape data.
173
+
174
+ ### Data-tool callbacks to a backend — via a tunnel
175
+
176
+ Data-tools call back a consumer's internal HTTP API. With the engine **in the
177
+ cloud** and your backend **on your machine** (`:3000`), expose it over a public
178
+ `https` tunnel and point the engine at it:
179
+
180
+ ```bash
181
+ # in the tool/manifest: base_url = {{env.ACHEI_INTERNAL_URL}}
182
+ ACHEI_INTERNAL_URL=https://your-tunnel.example.dev
183
+ INSIKA_EGRESS_HOSTS=your-tunnel.example.dev
184
+ ```
185
+
186
+ Because the tunnel is **public `https`**, the strict egress guard (the default)
187
+ **already allows it** — you do **not** need `ALLOW_HTTP`/`ALLOW_PRIVATE` (those are
188
+ only for a fully-local loop). Restricting `INSIKA_EGRESS_HOSTS` to the tunnel host
189
+ is the secure posture. See [Security](SECURITY.md#egress-the-ssrf-boundary).
190
+
191
+ ## Railway
192
+
193
+ `railway.json` already configures the Dockerfile builder, `startCommand`, the `/up`
194
+ healthcheck, and a restart policy.
195
+
196
+ 1. Create the project/service from this repo (builder = Dockerfile).
197
+ 2. **Volume**: mount it at `/data` (the default `INSIKA_DB` points there) —
198
+ without a volume, SQLite is ephemeral and recovery resumes nothing after a
199
+ redeploy.
200
+ 3. **Vars**: `DEEPSEEK_API_KEY`, `OPENCLAW_GATEWAY_TOKEN`, `ACHEI_INTERNAL_URL`,
201
+ `INSIKA_EGRESS_HOSTS` (and `WEB_CONCURRENCY` to match your plan/CPU).
202
+ 4. The healthcheck hits `/up`.
203
+ 5. Point your consumer at the service's public URL, with a matching API token
204
+ (see [RUNNING-LOCAL.md](RUNNING-LOCAL.md)).
205
+
206
+ ## Backup / DR — Litestream (opt-in, configurable)
207
+
208
+ A single volume is the **one point of total loss** between a pilot and production
209
+ (disk corruption/loss = goodbye conversations + config). Litestream does
210
+ **continuous replication** of `insika.db` (its WAL) to an S3/R2 bucket, without
211
+ changing databases and **without a line of Ruby**.
212
+
213
+ It is **off by default** and turns on by env — a single-box ephemeral deploy pays
214
+ nothing; a durable deploy enables it by pointing at a bucket. The trigger is one
215
+ variable, `LITESTREAM_REPLICA_URL`:
216
+
217
+ - **empty (default):** the entrypoint `exec`s Falcon directly. The Litestream binary
218
+ is never invoked — behavior identical to not having it.
219
+ - **set:** on a fresh box the entrypoint **restores** `insika.db` from the replica
220
+ *before* the app opens it (`litestream restore -if-replica-exists`; a no-op if the
221
+ bucket is still empty), then **supervises** the app (`litestream replicate -exec`),
222
+ replicating the WAL continuously and doing a final sync on shutdown (Railway's
223
+ SIGTERM).
224
+
225
+ ### Enable in production (Railway)
226
+
227
+ Add the vars (keep the volume at `/data`):
228
+
229
+ ```bash
230
+ # AWS S3
231
+ LITESTREAM_REPLICA_URL=s3://my-bucket/insika
232
+ LITESTREAM_REGION=us-east-1
233
+ LITESTREAM_ACCESS_KEY_ID=AKIA...
234
+ LITESTREAM_SECRET_ACCESS_KEY=...
235
+
236
+ # Cloudflare R2 (S3-compatible): same, + endpoint and region=auto
237
+ LITESTREAM_REPLICA_URL=s3://my-bucket/insika
238
+ LITESTREAM_ENDPOINT=https://<accountid>.r2.cloudflarestorage.com
239
+ LITESTREAM_REGION=auto
240
+ LITESTREAM_ACCESS_KEY_ID=...
241
+ LITESTREAM_SECRET_ACCESS_KEY=...
242
+ ```
243
+
244
+ Credentials are read natively by Litestream (they are not in `deploy/litestream.yml`,
245
+ which only references URL/endpoint/region via `${VAR}`).
246
+
247
+ ### Restore drill (the real "done" — an untested backup does not count)
248
+
249
+ The pilot→production gap only closes once a restore has been **exercised**. Two ways:
250
+
251
+ **1. Local, automated (proves the mechanism, zero credentials):** uses the real
252
+ image + a `file://` replica; boots → replicates → deletes the volume → boots a new
253
+ box → restores → confirms the marker row survived and `/up` is green.
254
+
255
+ ```bash
256
+ scripts/litestream-restore-drill.sh # needs docker, sqlite3, curl
257
+ # → [drill] PASS — marker … restored from replica; /up green on the new box.
258
+ ```
259
+
260
+ **2. Production (the drill that counts for go-live):** against the real bucket.
261
+
262
+ ```bash
263
+ # a. with the service live and replicating, generate some config/conversation and
264
+ # confirm the replica has generations:
265
+ litestream snapshots -config deploy/litestream.yml "$INSIKA_DB"
266
+
267
+ # b. boot a NEW box (empty volume) with the same LITESTREAM_* vars → the entrypoint
268
+ # restores on boot. Verify manually in /studio that conversations and config came
269
+ # back. Manual restore alternative:
270
+ litestream restore -config deploy/litestream.yml -o /tmp/restored.db "$INSIKA_DB"
271
+ ```
272
+
273
+ ## Kubernetes (evolution)
274
+
275
+ SQLite does not share one file across nodes. Paths forward: a StatefulSet + a PVC
276
+ per pod + **sticky-by-agent** routing (shard by tenant), or **LiteFS**, or an
277
+ optional **Postgres** adapter. **Litestream** (above) for backup/DR from day one —
278
+ orthogonal to topology.
279
+
280
+ ---
281
+
282
+ ## Measuring performance / load
283
+
284
+ ### 1. SQLite write ceiling (no provider) — `bench_store.rb`
285
+
286
+ Isolates "can SQLite take multi-process writes?" from LLM noise: N processes
287
+ hammering writes on the **same** file (WAL + busy_timeout — the real config).
288
+
289
+ ```bash
290
+ bundle exec ruby scripts/bench_store.rb 1,2,4,8 3000
291
+ ```
292
+
293
+ **Measured (mid-2026, laptop, ~481B payload):**
294
+
295
+ | procs | writes/s (aggregate) | p50 | p95 | max | locked |
296
+ |------:|--------------------:|----:|----:|----:|-------:|
297
+ | 1 | ~29.6k | 0.02ms | 0.04ms | 2.4ms | **0** |
298
+ | 2 | ~29.9k | 0.02ms | 0.04ms | 59ms | **0** |
299
+ | 4 | ~23.7k | 0.02ms | 0.05ms | 336ms | **0** |
300
+ | 8 | ~28.4k | 0.03ms | 0.05ms | 539ms | **0** |
301
+
302
+ **Reading:** aggregate throughput stays ~25–30k writes/s regardless of process
303
+ count (the WAL's one-writer-at-a-time ceiling), with **zero "database is locked"**
304
+ (the `busy_timeout` absorbs contention into tail latency, not errors), and a
305
+ microscopic p95. A real turn is **provider-bound (seconds)** and does a handful of
306
+ writes → the workload sits ~100× under the ceiling. **Empirically, SQLite is not
307
+ the bottleneck on a single box.**
308
+
309
+ ### 2. End-to-end load (with provider) — `loadtest.rb`
310
+
311
+ Hits `POST /v1/responses` (SSE), the production path. Measures TTFB, total, tokens,
312
+ cache hits, P50/P95, error rate. Runs against local or a remote deployment. See
313
+ [LOADTEST.md](LOADTEST.md).
314
+
315
+ ```bash
316
+ INSIKA_URL=http://localhost:9292 OPENCLAW_GATEWAY_TOKEN=xxx \
317
+ bundle exec ruby scripts/loadtest.rb --agents assistant --concurrency 16 --iterations 3
318
+ ```
319
+
320
+ ### 3. Baseline vs multi-worker on one box — `loadtest-local.sh`
321
+
322
+ Boots Falcon with 1 worker, then N, over the **same** SQLite, and counts "database
323
+ is locked" in the logs.
324
+
325
+ ```bash
326
+ DEEPSEEK_API_KEY=sk-... ./scripts/loadtest-local.sh 4 24
327
+ ```
328
+
329
+ ## See also
330
+
331
+ - [RUNNING-LOCAL.md](RUNNING-LOCAL.md) — run the engine locally, single-process.
332
+ - [Security](SECURITY.md) — tokens, egress, strict config.
333
+ - [BENCHMARK.md](BENCHMARK.md) — the neutral, key-free engine benchmark.
334
+ - [OBSERVABILITY.md](OBSERVABILITY.md) — OpenTelemetry traces + metrics (opt-in).
data/docs/EMBEDDING.md ADDED
@@ -0,0 +1,194 @@
1
+ ---
2
+ title: Embedding
3
+ parent: Ship it
4
+ nav_order: 4
5
+ permalink: /embedding/
6
+ ---
7
+
8
+ # Embedding
9
+
10
+ Insika does not have to be a service you stand up next to your app. It can be an
11
+ object inside the app you already have: a graph you build in an initializer and a
12
+ Rack app you mount in your router. This page is the contract for doing that —
13
+ what an embedded graph owns, what it still shares with the process, and what that
14
+ makes your responsibility.
15
+
16
+ ---
17
+
18
+ ## The short version
19
+
20
+ ```ruby
21
+ # config/initializers/insika.rb
22
+ INSIKA = Insika.embed(backend: Insika::Stores::SQLite.new(path: Rails.root.join("storage/insika.sqlite3").to_s)) do
23
+ agent "support" do
24
+ model "deepseek-chat"
25
+ provider :deepseek
26
+ api_key ENV.fetch("DEEPSEEK_API_KEY")
27
+ instructions "You answer questions about orders. Be brief."
28
+ end
29
+ end
30
+
31
+ # turns are born as children of a long-lived supervisor, not of the request
32
+ INSIKA.runtime.graph.executor.supervised = true
33
+ ```
34
+
35
+ ```ruby
36
+ # config/routes.rb — the transport is lazy: `require "insika"` never loads it
37
+ require "insika/server/rack_app"
38
+
39
+ mount Insika::Server.rack_app(INSIKA, token: ENV.fetch("INSIKA_TOKEN")), at: "/ai"
40
+ ```
41
+
42
+ ```ruby
43
+ # anywhere in your app — no HTTP involved
44
+ INSIKA.reply("support", "where is order 8123?", session: current_user.id)
45
+ ```
46
+
47
+ `Insika.embed` takes the same block as [`Insika.system`](AGENTS.md) and adds one
48
+ obligation: **you name the store**. That single argument is what turns "one
49
+ program" into "one object" — see [why](#why-the-store-is-an-argument) below.
50
+
51
+ `Insika::Server.rack_app` returns a plain Rack app. It routes on `PATH_INFO`, so
52
+ Rails' `mount` (and `Rack::URLMap`, and anything else that moves the prefix into
53
+ `SCRIPT_NAME`) leaves every route intact: mounted at `/ai`, the drop-in API is at
54
+ `/ai/v1/responses`.
55
+
56
+ ---
57
+
58
+ ## Why the store is an argument
59
+
60
+ Until RFC-0017 the engine was not one process — it was one *program*. Two graphs
61
+ built in the same Ruby process shared things they never declared:
62
+
63
+ | What | What actually happened |
64
+ |---|---|
65
+ | LLM credentials | one process-wide slot **per provider**: two graphs on `deepseek` with different keys, and the second one won **for both**. No error, no warning — agent A simply called the provider with tenant B's key. |
66
+ | The store | `INSIKA_DB` unset → each graph got its own in-memory store and nothing persisted; `INSIKA_DB` set → every graph opened the **same file** and read the others' agents, sessions and tasks. |
67
+
68
+ Neither failure raised. That is the whole reason the embed contract is written
69
+ down rather than left to good sense.
70
+
71
+ ---
72
+
73
+ ## The contract
74
+
75
+ ### 1. A graph owns its store
76
+
77
+ Every durable thing — sessions, tasks, checkpoints, the outbox, delegations,
78
+ memory — belongs to the backend the graph was built with. Two graphs given the
79
+ **same** backend share all of it, by your choice. Two graphs given **different**
80
+ backends share nothing.
81
+
82
+ ```ruby
83
+ support = Insika.embed(backend: Insika::Stores::SQLite.new(path: "storage/support.sqlite3")) { … }
84
+ ops = Insika.embed(backend: Insika::Stores::SQLite.new(path: "storage/ops.sqlite3")) { … }
85
+ # a session created in `support` does not exist in `ops`
86
+ ```
87
+
88
+ ### 2. A graph owns its LLM credentials
89
+
90
+ Provider keys and bases resolve through the graph's own
91
+ [`RubyLLM.context`](https://rubyllm.com) — an isolated dup of the configuration.
92
+ An embedded graph never mutates the global `RubyLLM.config`, and never reads
93
+ another graph's credentials. That holds for the turn itself *and* for everything
94
+ else in the graph that asks a model: the content-safety moderator and the output
95
+ validator ask on the same context the turn does.
96
+
97
+ It also holds for runtime reconfiguration. Editing a provider key in the Studio
98
+ (or dispatching `:upsert_llm_provider`) applies to **that graph only** — the
99
+ change is real, takes effect without a restart, and stops at the graph boundary.
100
+
101
+ ### 3. The process still owns signals and the reactor
102
+
103
+ Draining in-flight turns on SIGTERM ([the process model](DEPLOY.md)) is a process
104
+ concern, not a graph concern: `Signal.trap` keeps only the last handler, so a
105
+ shutdown installed once per graph would drain the last graph and let the others
106
+ die mid-turn. An embedded graph therefore installs **nothing** behind your back.
107
+ You install it once, naming every graph:
108
+
109
+ ```ruby
110
+ Insika::Shutdown.install(executors: [SUPPORT, OPS].map { |g| g.runtime.graph.executor })
111
+ ```
112
+
113
+ One signal closes every graph's intake first, then waits — up to
114
+ `INSIKA_DRAIN_TIMEOUT` (default 20s) — for the in-flight turns of all of them.
115
+ Whatever outlives the deadline stays `:running` and is replayed by the next
116
+ boot's recovery sweep.
117
+
118
+ The reactor is yours too, and this one has teeth:
119
+
120
+ > **The mounted app needs an async server.** `rack_app` is a value — nothing in it
121
+ > starts a server or a reactor. A turn is a fiber, so the routes that start one
122
+ > (`POST /v1/responses`, `/v1/messages`, `/v1/commands`, `POST /channels/:id/events`)
123
+ > must be called from inside a running reactor. Under **Falcon** you already are.
124
+ > Under Puma or Unicorn they answer `500 {"error":{"message":"No async task
125
+ > available!"}}` — the read-only routes (`GET /v1/agents/:id`, `/v1/tasks/:id`, `/up`)
126
+ > are fine either way. Serve an embedded Insika from Falcon, or reach the graph
127
+ > in-process with `INSIKA.reply(...)`, which builds its own reactor.
128
+
129
+ And once you are inside one, set `supervised = true` (as in the quickstart): it
130
+ makes a turn a child of a long-lived supervisor instead of a child of the
131
+ request, so the turn survives a client disconnect. Without it the runtime cancels
132
+ the turn when the connection drops.
133
+
134
+ ### 4. The Studio is not part of the embeddable surface
135
+
136
+ `rack_app` returns the `/v1` transport and nothing else. `Studio::App.configure`
137
+ freezes its collaborators on the **class**, which makes it one operator UI per
138
+ process: a second graph configuring the Studio replaces the first's wiring, for
139
+ both. So a host that wants the UI does a plain `require "insika/studio/app"`,
140
+ calls `Studio::App.configure` with exactly one graph's collaborators, and mounts
141
+ the class — accepting that the UI shows that graph and no other.
142
+
143
+ This is a stated limitation, not an oversight. Making the Studio instantiable is
144
+ its own change, and it needs its own reason.
145
+
146
+ ### 5. ENV is a default, never a requirement
147
+
148
+ Anything an embedded graph reads from the environment has an injectable
149
+ equivalent, and the injected value wins. `INSIKA_DB` still works — it is the
150
+ default for the DSL quickstart and for `config.ru` — but an embedded graph that
151
+ was given a backend never looks at it.
152
+
153
+ ---
154
+
155
+ ## What you are responsible for
156
+
157
+ | Concern | Who |
158
+ |---|---|
159
+ | Which store each graph gets | You, via `backend:` |
160
+ | Which credentials each graph gets | You, via the agent's `api_key`/`provider` |
161
+ | Who may call the mounted app | You. `token:` is one shared Bearer for the whole mount — see below |
162
+ | Signals and the drain | You, once per process (`Shutdown.install(executors:)`) |
163
+ | The reactor / `supervised` | You, matching your server |
164
+ | Recovery of orphaned turns at boot | You, if you want it — the sweep is `Insika::Recovery`, wired by `Insika::Server::Boot` for the standalone deployment, not by `embed` |
165
+
166
+ ### Embedding is not multi-tenancy
167
+
168
+ Two graphs stop corrupting each other. That is all this contract says. **Who is
169
+ allowed to talk to which graph** is authorization, and it is not here: `token:` is
170
+ a single Bearer gating the whole mounted app, exactly as it does for the
171
+ standalone server — the written decision is [one deployment, one token;
172
+ multi-tenancy belongs to the host](SECURITY.md#the-bearer-gate). If your app has
173
+ users, put the mounted app behind your own authentication and pass `session:`
174
+ yourself — do not hand the mount point to the browser.
175
+
176
+ ---
177
+
178
+ ## What this is not
179
+
180
+ - **Not a Rails engine.** `insika-rails` would be a separate gem; this is the Rack
181
+ app such a gem would mount. The core has no Rails knowledge and takes no Rails
182
+ dependency.
183
+ - **Not a second assembly path.** `Insika.embed` is a front door over the same
184
+ pipeline `Insika.agent`/`Insika.system` use — the profile it produces is
185
+ identical, and there is a spec that holds it to that.
186
+
187
+ ---
188
+
189
+ ## See also
190
+
191
+ - [Architecture](ARCHITECTURE.md) — how a turn actually runs.
192
+ - [Deploy](DEPLOY.md) — the standalone process model (N workers of *one*
193
+ deployment), which is a different question from N graphs in one process.
194
+ - [Agents](AGENTS.md) — the DSL block `embed` takes.