insika 0.2.0 → 0.7.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 9e7913378bd8be19a7d9422524e62c6786c2abc8d26b5194ef8b1db8fef82be5
-  data.tar.gz: f1a279b7284d9756d7ad3dcef56329314062ef77c0fbb3d24df98e10c9bcb4ce
+  metadata.gz: 5c2b2fbe064af45aa132f93de00954b24dc504e052db2e059b4f6daa151e248b
+  data.tar.gz: 50cabcc10d82eabdfd08de562bd61feddf1a9de68ee6427343c80c5cf6ad9d8c
 SHA512:
-  metadata.gz: dcbf5403a982de5ee9dc8ed0c5fde2b21b5b63667cde0b47d2b9985b99fe7a9708d53fb82d43ce8464f46c38989d5b2b261156a84c8245e16199017515d24191
-  data.tar.gz: 349d532c16f5baed05fcd79318f3f68056e40f2a931e5e33ff12c339791edaa6fcdfe811d419cf3aff490118bd46698486470f234d9e820f0e568bb875e8b2cc
+  metadata.gz: 3004e30a234349bcb91db185fcc31f9b52cbdeb7660c613b1fc229adf6bbbe717d9abc8ebcdc5bf9ecc125986694c0d2f3a82e1d48eb83105ff74db0467e4d91
+  data.tar.gz: ca4bec6a8b8cd03ee1a5c2df627e7da778d3dd157f30f848c70f8077ac467570284bf934e3a4c8040fb223503855d9cdcad111b9139ed093c702b6a8850ea1ff

data/CHANGELOG.md

@@ -8,44 +8,352 @@ it is released. Entries land with the pull request that makes the change.
 
 ## [Unreleased]
 
-Nothing yet.
+## [0.7.0] - 2026-08-29
+
+The pre-1.0 consolidation: MCP servers as first-class tools, a template gallery,
+the knowledge layer (extract → consolidate → retrieve → export), a session-sticky
+router for N>1 capacity, and the Studio brought to visual parity. The `/v1`
+contract is unchanged. The version jumps from 0.3 to 0.7 to say what the code
+already says — the surface is the 1.0 surface; what remains is operational proof,
+not features.
+
+### Fixed
+
+- **An agent's identity never falls back to another agent's.** An agent with
+  neither `prompt_files` nor `base_prompt` silently inherited the deployment-wide
+  default identity, so one agent could answer as another's persona (seen live,
+  deterministic, previously misdiagnosed as a concurrency bug). It now raises
+  `Insika::ContextError` and aborts the turn instead.
+- **A text-only turn absorbs a burst of messages instead of leaking a follow-up.**
+  Steering only drained the mailbox at a tool-batch boundary; a turn with no tool
+  calls never had one, so a customer who sent four messages in a row got one
+  answer that only saw the first. The executor now runs one extra round in the
+  same turn when the mailbox is not empty.
+- **The turn supervisor starts at boot, not on the first served turn** — a
+  deployment whose only agents are scheduled never fired tick/alert/distill
+  workers until unrelated traffic arrived.
+- **Every Linux/Docker build was failing** on two independent packaging gaps:
+  `Gemfile.lock` only listed `arm64-darwin` (#185), and the Dockerfile's builder
+  stage did not copy `lib/insika/packaging.rb`, which the gemspec requires (#184;
+  a spec now parses the Dockerfile against the gemspec's `require_relative`s).
+- **MCP**: `refresh` reused a permanently broken memoized client, so an edited
+  server config needed a full process restart; and ruby_llm-mcp sent a
+  spec-invalid `Origin` header (URL with a path) that Origin-checking servers
+  (GitHub, Grafana, Metabase) rejected on every request — now stripped.
+- **Media**: STT and image-generation calls always raised past the RubyLLM
+  boundary (raw bytes where a path was expected, `assume_model_exists` with no
+  provider). Found live-validating against real audio/PDF fixtures; a contract
+  spec now guards the boundary.
+- **`insika harvest`/`knowledge:backfill` never wired the CLI's provider
+  keys** — both build their own miner/extractor `ask` outside a full app
+  boot (no `LlmConfigurator`), so `RubyLLM.config` was empty and a real run
+  raised `RubyLLM::ConfigurationError` on the first live call, no matter how
+  the agent was configured. Found running `knowledge:backfill` against real
+  staging traffic for the first time. `configure_eval_llm!` (added for
+  `evals:simulate`, PR #190) is now `configure_cli_llm!` and both commands
+  call it before building their factory.
+
+### Added
+
+- **MCP servers are first-class tools** (#178–#182). `Insika::McpClient` builds a
+  real stdio/http/sse client (ruby_llm-mcp, lazy-loaded) from an `McpStore`
+  record; `McpToolRegistry` executes live through it, merged into the overlay
+  registry as a third source (base > data-tools > mcp) so allowlists, traces,
+  budgets and checkpoints work unchanged. Config surfaces: JSON import/export in
+  the `mcpServers` format, `insika mcp list|add|remove|import|test|refresh`, the
+  `mcp` DSL declaration (which auto-grants `mcp:<name>` on the agent's allowlist),
+  `/v1/mcp`, and a transport-aware `/studio/mcp` page with per-instance status and
+  "Test connection". stdio is arbitrary command execution by config and stays
+  behind `INSIKA_MCP_STDIO=1`.
+- **Template gallery** (#183). Six example agents ship inside the gem
+  (`lib/insika/templates/<name>/agent.rb` + README) — one DSL file is both doors:
+  `insika new` copies it, the Studio gallery instantiates it. Starter
+  (travel-planner, keyless public APIs), Advanced (research-analyst fan-out),
+  Always-on (daily-digest: schedule + artifact), Teams (review-panel) and more.
+- **Session-sticky router** — `insika-router`, a standalone Async/Falcon proxy that
+  runs N engine backends (`WEB_CONCURRENCY=1` each) behind a consistent-hash ring
+  keyed on the session id, so per-session FIFO/collect/steer guarantees hold at
+  N>1 capacity with no change to the executor. Static (Railway) and DNS
+  (Kubernetes headless Service) backend discovery; an unreachable backend answers
+  the retry envelope. Opt-in.
+- **Studio visual parity** — view transitions and motion (CSS only, reduced-motion
+  aware), the transcript as product (mono label rows, tool calls as pill chips,
+  three-zone session viewer), a live home over `/studio/events`, master-detail
+  lists.
+- **`queue_mode` and `steer_max_messages` editable per agent** in the Studio
+  Config form — operations can switch steering on/off without touching the store.
+  Blank inherits the platform default; `steer_max_messages: 0` never steers.
+- **Paste-prompt journeys for coding agents** — four skill-shaped prompts under
+  `docs/prompts/`, served at `GET /docs/<name>.md` next to `start.md`:
+  run-examples, add-tool, diagnose-turn and go-live.
+- **`run_persona_eval` swaps side-effect tools instead of refusing** — a target
+  agent with a reachable POST tool gets a throwaway executor whose registry
+  resolves every side-effect name to a recorder, sharing every other collaborator
+  of the real graph.
+- **`examples/analytics`** — MCP wired to a real, key-authenticated server
+  (OpenSEO over http, bearer via `OPENSEO_API_KEY`).
+- **`insika knowledge:export`** — writes one `<name>.md` per learned concept
+  under a directory (`KnowledgeStore#export_dir`). The storage format IS the
+  export format, so it's a dump, not a converter, directly consumable by
+  okf-gem/graphify. Idempotent — nothing here is lossy, so there's no
+  `--force` to reason about. `--format graphml` writes one combined
+  `knowledge.graphml` instead — a node per concept, an edge per `[[link]]`
+  resolved against the same export set (a dangling link is dropped, never a
+  broken edge) — hand-built, hand-escaped XML, no new dependency.
+
+- **Knowledge retrieval** — with `knowledge: {retrieve: true}`, every turn
+  the engine now searches the agent's learned concepts for the ones
+  relevant to the customer's message (`Insika::Knowledge::Index::Scan` —
+  pure term overlap × confidence × recency, no embeddings, no network call)
+  and injects the top few as a level-1 `<knowledge>` block (name/
+  description/confidence/provenance, new `Context::Priority::KNOWLEDGE =
+  77`, between skills and memory), expanding one hop through a concept's
+  `[[links]]`. A `load_knowledge` tool — outside `tools_allow`, wired
+  exactly like `load_skill` — loads a concept's full body on demand;
+  calling it fires `:knowledge_retrieved`, the adoption metric that
+  actually matters (a concept sitting unread in the prompt taught nothing).
+  `knowledge.index: "fts5"` is accepted but falls back to the built-in scan
+  index — deliberately not built yet: `scripts/bench_knowledge_index.rb`
+  measured `Index::Scan` at hundreds of milliseconds per search before a
+  read-cache fix (re-parsing a concept's frontmatter on every search was
+  ~90% of the cost, not the store I/O); after the fix it's sub-1.5ms at the
+  concept counts this feature targets, and building the SQLite adapter is
+  deferred to a real, documented latency trigger (~1000 concepts/agent)
+  instead of being built speculatively.
+
+- **Knowledge consolidation + the Studio page** — a repeat concept name no
+  longer blindly overwrites. The engine now decides same claim (bumps
+  occurrences/sources/confidence — `min(0.95, 0.5 + 0.1 × distinct_sources)`,
+  no model call), related claim (one extra model call merges the two
+  bodies — `Insika::Knowledge::Consolidator`/`ConsolidatorFactory`), or
+  contradicting claim (never merged — appended under a `## Contradiction`
+  heading, confidence dropped to `0.4`, `:knowledge_conflict` emitted). No
+  consolidator configured, or an unusable answer, defaults to contradicting —
+  the conservative choice. `/studio/knowledge` (single-agent-scoped like
+  Harvest) lists every concept with a conflict filter, the same CodeMirror
+  editor Skills uses (also how an operator hand-promotes `provenance:
+  observed` to `policy`), version history/restore, and delete — all through
+  three new bus commands (`write_concept`/`delete_concept`/`restore_concept`,
+  wired through every composition root).
+
+- **Knowledge, layer 1 (extraction)** — the engine can now learn durable
+  **concepts** (facts, procedures, policies, objections) from finished
+  conversations, opt-in per agent via `knowledge extract: true`. After a turn
+  completes, off the critical path, the platform `utility_model` proposes
+  candidate concepts; the engine schema-validates the answer, drops any
+  model-authored `provenance`/`confidence`/`sources` (stamped by the engine
+  instead — every extracted concept is `provenance: observed`, never
+  `policy`), redacts the body for PII, and persists it as a markdown+
+  frontmatter record in the new `KnowledgeStore` (`Insika::KnowledgeStore`,
+  scoped per agent/tenant like `MemoryStore`, versioned like `SkillStore`).
+  Emits `:knowledge_learned` (name/type/agent only, never content). The
+  recovery path is `insika knowledge:backfill --agent ID [--since DATE]`,
+  replaying stored sessions through the same extractor. No retrieval into a
+  turn's prompt yet — see `docs/KNOWLEDGE.md` for what's shipped and what's
+  still planned.
+
+- **Plugin loading is on in every root** — `Server::Boot`'s `load_plugins`
+  step was a no-op in both composition roots, so the tested
+  `Insika::Plugin::Loader` never ran outside the `insika-code`
+  example. It now runs at boot in the minimal wiring, the demo deployment
+  and DSL-run agents, via a shared `Wiring::Graph.load_plugins`. Discovery
+  roots: announced gems (default-enabled), `INSIKA_PLUGIN_DIR` (workspace)
+  and the repo's `plugins/` — the latter two gated by `INSIKA_PLUGINS`,
+  with `INSIKA_PLUGINS_DISABLED` as the absolute veto. Plugin skills and
+  prompts join the catalogs at the lowest precedence (a workspace or
+  authored skill still wins). The dead `OPENCLAW_PLUGIN_DIR` env spec
+  (nothing ever read it) is replaced by the three new keys.
+
+- **Media parity** — three transport gaps closed, all additive:
+  `generate_image` can now EDIT as well as generate — `source_image_urls`
+  (or, absent those, the turn's own inbound photo by default) ride
+  `RubyLLM.paint(with:)`, an optional `mask_url` rides `paint(mask:)`; a
+  fourth inbound part type, `document` (`{ "type": "document", "url": … }`,
+  capped at 10 MB), attaches to the ask like an image and deposits
+  `{{ctx.document_url}}` for data tools; and audio transcription now carries
+  a vocabulary `prompt:` (per-agent `stt_prompt`, falling back to the
+  deployment-wide `INSIKA_STT_PROMPT`) so domain terms (product names, brand
+  terms) transcribe correctly instead of phonetically. Text-to-image and
+  plain audio transcription stay byte-identical when neither feature is used.
+- **Demo data** — `insika demo:seed` (and a matching "Seed demo data" button
+  under Studio Settings) provisions a bundled `demo-store` agent and writes
+  enough realistic data to see every loop working at once: a funnel with a
+  frozen baseline, follow-ups in all four states, refinement runs across the
+  lifecycle, pending and resolved approvals, distillation proposals and a
+  memory fact, and a golden set with a mixed-result baseline. One code path
+  (`Insika::Demo::Seeder`) behind both front doors, same discipline as every
+  other Studio button. Fixed two pre-existing gaps this surfaced: `config.ru`
+  never wired `proposal_store`/`budget_ledger` into the Studio (the Facts page
+  and the funnel's spend pill were unreachable in production), and
+  `scripts/serve_real.rb` never wired the outcome/funnel/follow-up/proposal
+  stores at all.
+
+## [0.3.0] - 2026-08-19
+
+The proof-and-consultant wave: shadow parity against a frozen criterion, the
+72h soak, perceived latency (progressive delivery), the session briefing,
+evidence grounding, the layered identity cache, customer memory in the Studio,
+the conversion ledger, `schedule()` with consent, human-gated facts, the gated
+skill harvest — and a domain-free core: the e-commerce defaults are removable,
+and what ships in the gem is asserted by the suite.
+
+### Added
+
+- **Shadow mode** — the relay channel can run every turn end to end and deliver
+  nothing: the incumbent keeps answering, the engine records what it *would*
+  have answered, and the two replies are judged pairwise against a **frozen,
+  pre-registered criterion** (the file `INSIKA_PARITY_CRITERION` points at; its
+  SHA-256 stamps every pair, and editing it mid-window turns the verdict
+  `invalid`, never stale). A panel of judges scores each pair twice with the
+  sides swapped, so a preference that flips with presentation order is recorded
+  as comparable, not as a preference. The Studio's Parity page folds the
+  running verdict on demand. No criterion, no shadow: boot refuses.
+- **The 72h soak** — `insika soak` sustains a declared arrival rate for days and
+  asks whether the *process itself* degrades with uptime. A deployment-side
+  envelope freezes the load shape and the gated ceilings before the first hour
+  (its hash stamps every snapshot); `--preflight` refuses to start on a missing
+  precondition; the runner appends hourly vitals as they happen, so a crash at
+  hour 60 leaves 60 usable hours; `--verify` recomputes the whole verdict
+  offline from the raw records. A fail means *find the leak* — never cut, and
+  never loosen the envelope.
+- **Progressive delivery** — the relay can flush the outbox one balloon per
+  paragraph (`INSIKA_RELAY_DELIVERY=progressive`), with the first balloon
+  posted as soon as the answer exists; the default `:at_end` is byte-identical
+  to before. Every channel turn records `first_balloon_ms` (inbound receipt →
+  first outbox flush) so the target is measured, not assumed.
+- **Session briefing** — the pack can declare `briefing_fields`; the engine
+  keeps the session's working state as data (known fields, still-missing list,
+  next step), a tool updates it, and the context provider injects it into the
+  turn — the consultant stops re-asking what it already knows.
+- **Grounding** — an `evidence:` contract on the tool envelope: a model claim
+  must cite ids that came out of a tool call, with `:enforce` / `:flag` / `:off`
+  modes and a per-turn ledger. A claim without evidence is refused or flagged —
+  never silently passed through.
+- **Layered identity + observable cache** — context providers are split into
+  identity/volatile layers behind a byte-stable cache prefix; the per-agent
+  cache-hit series is recorded (`CacheSeriesStore`) and shown on the Studio
+  agent detail. Invalidations and hits are measurable, not guessed.
+- **Customer memory in the Studio** — a Customers drill reads and edits the
+  per-customer memory cell, with an append-only operator-mutation audit trail
+  (content-free digests), provenance metadata, expiry, the LGPD export
+  (`/v1/commands/export_customer_memory`) and the existing right to be
+  forgotten.
+- **The outcome funnel** — outcomes fold into a store-declared stage funnel
+  (`funnel:` on the pack) on the tick: idempotent cumulative counts, an
+  attribution window carried as data, a frozen baseline (`:freeze_funnel_baseline`)
+  and a Studio page per agent. The stage vocabulary is the deployment's; a bare
+  install shows no funnel at all.
+- **Follow-ups** — the agent can book a future contact: `schedule()` and
+  `cancel_followup()` tools, a durable per-customer contact state
+  (`granted | revoked | unavailable`, where silence is not a refusal), a policy
+  engine over the tick (quiet hours, frequency ceilings, dedup, opt-out
+  keywords) and a Studio page per agent. A customer who opted out can never be
+  rescheduled; blocking happens at fire time, never at schedule time.
+- **Distilled facts** — finished conversations are read back as proposed
+  customer facts; a human approves/rejects/dismisses each on the Studio Facts
+  page, with a latched dedup ledger (a dismissed tuple is never proposed again),
+  optimistic CAS writes and LGPD-friendly retention. Nothing is ever applied
+  automatically.
+- **Gated skill harvest** — finished traffic can be mined for skill proposals
+  behind a versioned negative list and an evidence ledger (product claims must
+  reference ids the origin sessions actually saw); promotion requires the eval
+  replay **and** the store's conversion ruler not to regress, with an
+  append-only log and snapshot rollback. Nothing is ever applied automatically.
+- **A domain-free core** — the gem payload is a spec-asserted boundary
+  (`lib/` + `docs/` + the four root files; `deploy/`, `packs/`, `examples/`,
+  `evals/`, `scripts/` and `spec/` never ship, even when tracked). The pt-BR
+  guardrail corpus is language-tagged removable data, and `insika doctor --domain`
+  inventories what a deployment declares. A bare install names no store.
+- **Model-visible conformance** — every byte that reaches the provider is
+  reconstructable from checkpoints + traces, proven byte-for-byte by the
+  conformance suite (`spec/insika/conformance/`): what the chat held == the
+  checkpoint transcript == the model trace, across plain, tool-calling, steered,
+  subagent, scheduled and resuming turns.
+- **Intent routing** — `AgentProfile#routes` classifies the turn's
+  message into one configured route with a cheap model before the ask, from a
+  prompt auto-generated out of the route descriptions. The route rides the
+  turn (`state.route`, the `:route_classified` event, the terminal event), its
+  provider cost is counted in the usage, and a route can `delegate` to an
+  existing agent (its answer becomes the parent's) or end the turn with the
+  stuck outcome (`stuck: true`). Deterministic `default` fallback;
+  classifier failure leaves the turn unrouted (additive).
+- **Outcomes** — `POST /v1/outcomes` records a conversation's business
+  outcome (`conversion`/`escalation`/`deflected`/…, optional monetary value)
+  from the operator or the integration — additive, outside the response
+  contract, tenant-stamped. `GET /v1/outcomes` serves the last outcome
+  per agent + per-day series; the Studio's agent grid shows the last-outcome
+  pill and the agent detail shows the per-day series.
+- **Customer-scoped memory + right to be forgotten** — a message
+  carrying a `customer` key moves the memory scope to the `[tenant:]customer`
+  cell: two customers under one tenant never read each other (phase 1), and
+  the `<request_context>` merchant label stays untouched. Facts gained an
+  optimistic CAS write (`replace_if_revision`, microsecond revisions).
+  `POST /v1/commands/forget_customer` (phase 2, LGPD) purges one customer's
+  memory cell, their sessions and per-session traces — nothing else's.
+- **Tenant deletion + retention** —
+  `POST /v1/commands/delete_tenant_data` purges everything the engine holds
+  about one tenant (sessions, traces, every memory cell under the tenant and
+  its outcome records); the `retention_days` settings key turns on the tick's
+  daily age-based sweep (sessions + traces, terminal tasks + checkpoints,
+  memory cells and outcomes — OFF by default). The KV store contract gained
+  an additive `scopes(prefix)` enumeration.
+- **Media in the message contract** — messages accept
+  additive content parts (`text`/`image`/`audio` with a URL). Audio is
+  transcribed via RubyLLM STT (model/language via `INSIKA_STT_MODEL`/
+  `INSIKA_STT_LANGUAGE`) and the text enters the turn marked
+  `source: "voice"` on the terminal event; images attach to the model's ask
+  (provider-billed, usage flows) and the first image URL is
+  `{{ctx.image_url}}` for data/HTTP tools; media URLs pass the egress guard. The
+  OpenAI multimodal `input` array shape works on `/v1/responses`.
+- **Generated media as outputs** — the turn can produce an
+  image or a voice clip when BOTH gates agree: the agent opts in
+  (`AgentProfile#outputs` — per-kind model/voice/size config) and the request
+  declares the channel can receive it (`channel.capabilities`, one of
+  `image_output`/`audio_output`; unknown values are a 422 — the abstraction
+  admits only what leaks). The `generate_image`/`tts` system tools are wired
+  only then, the media rides the envelope additively (`output_parts` on the
+  terminal event and the `/v1/responses` completed frame — base64 parts, never
+  the answer text), image tokens merge into the turn's usage and every call
+  adds an honest `usage.media` counter (the speech API reports no tokens; the
+  part carries the model for consumer-side pricing). Generators are injectable
+  seams; the defaults are lazy (RubyLLM paint; a thin POST to the
+  OpenAI-compatible `/audio/speech` using the same provider config).
 
 ## [0.2.0] - 2026-08-13
 
 The workstreams between the first release and the one the gem actually became:
-multi-tenancy at the edge (WS1), calendar budgets (WS2), provider reliability
-(WS3), the stuck signal (WS5), operator alerts + live TTFB (WS6), and the
-failure-classification core (B9) — plus the two fix rounds that made them
-safe to ship.
+multi-tenancy at the edge, calendar budgets, provider reliability, the stuck
+signal, operator alerts + live TTFB, and the failure-classification core — plus
+the two fix rounds that made them safe to ship.
 
 ### Added
 
-- **Multi-tenant at the edge (WS1)** — `INSIKA_TENANCY=multi_tenant` resolves the
+- **Multi-tenant at the edge** — `INSIKA_TENANCY=multi_tenant` resolves the
   Bearer to a principal before the routes: per-tenant + operator tokens stored
   only as SHA-256 hashes, a tenant's sessions/tasks/streams living under its own
   `<tenant>:` namespace (fail-closed: another tenant's reads as `404`), and every
   authoring/config surface refused to a tenant.
-- **Calendar budgets (WS2)** — `AgentProfile#budget` caps the billed spend
+- **Calendar budgets** — `AgentProfile#budget` caps the billed spend
   (input + output + cached + cache-creation) per calendar day/month and
   (tenant, agent): HARD (default) fails the turn with the typed
   `Insika::BudgetExceeded` + `retry_after`; `soft: true` runs the turn and warns
   once per window — with the `alert_at` (`0.8`) crossing and the real cap
   crossing as separate events.
-- **Reliability (WS3)** — retries with backoff, mid-turn rotation to the
+- **Reliability** — retries with backoff, mid-turn rotation to the
   fallback chain, a per-`(tenant, provider/model)` circuit breaker with
   half-open trials (a failed trial reopens), and a per-attempt `timeout`
   (default 30s) counted as retryable. A `:fatal` provider error is never
   retried.
-- **Stuck signal (WS5)** — an agent declared stuck ends its turn with
+- **Stuck signal** — an agent declared stuck ends its turn with
   `outcome: "stuck"` on the envelope and a dedicated `:turn_stuck` event — the
   deterministic point a consumer escalates on.
-- **Operator alerts + live TTFB (WS6)** — `budget_warning`, `breaker_open` and
+- **Operator alerts + live TTFB** — `budget_warning`, `breaker_open` and
   `delivery_failed` POSTed to a per-agent `alerts.webhook` over the at-most-once
   outbox pipeline (boot-recoverable); under `INSIKA_TURN_TIMING` the first
   content chunk emits a live `:ttft` on the streaming envelope.
-- **Failure classification (B9)** — provider/transport failures classified by
+- **Failure classification** — provider/transport failures classified by
   action (`:fatal` / `:retryable` / `:rate_limited_*`) and wrapped with the
-  provider's `retry_after`; mechanical tool-output dedupe (C3) back-references a
+  provider's `retry_after`; mechanical tool-output dedupe back-references a
   byte-identical repeat only when the reference is genuinely shorter.
 - **The periodic tick** — durability no longer waits for a reboot. Serving
   workers run a tick every `INSIKA_TICK_INTERVAL` (default 60s, `0` disables)
@@ -58,7 +366,7 @@ safe to ship.
 
 ### Fixed
 
-- **WS2/WS3/WS6 criticals** — the budget alert marker no longer returns inside
+- **Budget/reliability/alerts criticals** — the budget alert marker no longer returns inside
   the store transaction (a leaked `BEGIN IMMEDIATE` locked SQLite on the 2nd
   over-threshold turn); the monthly reset is December-safe and UTC-aligned; an
   unset reliability timeout is 30s, not 1s, and a timeout retries/rotates
@@ -66,10 +374,10 @@ safe to ship.
   webhook deliveries pass the egress guard (SSRF); `:ttft` is emitted once per
   turn; webhook channels pre-register so the boot sweep recovers pending alerts;
   the alert dispatcher subscribes typed and re-subscribes on overflow.
-- **WS1** — `#revoke` rides the store transaction; a `tenant_id` containing
+- **Tenancy** — `#revoke` rides the store transaction; a `tenant_id` containing
   `:` is refused (the session-namespace delimiter); `POST /v1/sessions` mints a
   tenant's session under its own prefix.
-- **WS2/WS3 softs** — a failed turn's consumed tokens count against the budget;
+- **Budget and reliability softs** — a failed turn's consumed tokens count against the budget;
   `:breaker_open` alerts only on the closed→open transition; the fallback chain
   dedupes `"model"` vs `"provider/model"` spellings.
 
@@ -143,7 +451,7 @@ The first release: `gem install insika`.
 - **The gate refuses to grade a judged baseline without a judge** — a rubric'd case
   with no verdict counts as a pass, so replaying without a judge against a baseline
   recorded with one does not measure less, it measures backwards. Found by running the
-  panel against the real pilot: the gate reported *6/6, no regression* against a
+  panel against real traffic: the gate reported *6/6, no regression* against a
   baseline the same corpus had just scored *2/6*, and both candidates cleared. With the
   judge configured, the same two candidates were correctly rejected on judge-score
   drops. Third member of the same family as the missing and all-red baseline refusals.
@@ -151,7 +459,7 @@ The first release: `gem install insika`.
   input + output and excludes the cached prefix (a 27 KB pack reports `88` total against
   `26624` cached), so a ceiling built on it alone let a run send hundreds of times what
   it said. Cost now bills `total + cached` and records the cached share, which on a real
-  panel run was 95% of the spend. A run that cannot be gated is also refused **before**
+  run was 95% of the spend. A run that cannot be gated is also refused **before**
   the proposal is paid for, not after.
 - **The eval baseline is a per-agent record, not only a file** — `evals/baseline.json`
   works from a checkout; the refinement gate runs inside a deployment that has none.

data/README.md

@@ -22,6 +22,10 @@ deployment.
 - **Safe by default** — content-safety guardrails, an egress guard, confined execution, approvals, edge limits.
 - **Observable** — an event stream, per-session tool-call traces, optional OpenTelemetry.
 - **~0.4 ms of engine per turn** — p50 overhead on a neutral, key-free benchmark you can rerun yourself ([methodology](docs/BENCHMARK.md)).
+- **Domain-free core** — the gem carries no store vocabulary, no persona and no fixed
+  conversation language: the built-in pt-BR guardrail corpus is removable data, a
+  deployment declares what it is, and the payload boundary is a spec the suite asserts
+  on ([the domain-free map](docs/domain.md)).
 
 ## Your first agent
 
@@ -37,7 +41,7 @@ require "insika"
 assistant = Insika.agent("assistant") do
   model "deepseek-v4-flash"
   provider :deepseek
-  instructions "You are Bia, a concise and friendly assistant. Answer briefly."
+  instructions "You are a concise and friendly assistant. Answer briefly."
 end
 
 puts assistant.reply("hi, what can you do?")   # one turn, in-process
@@ -76,39 +80,76 @@ served — no secrets) and **`GET /docs`** + **`GET /docs/<name>.md`** (these do
 markdown). Public and on by default when you `serve`; opt-in in production
 (`INSIKA_ONBOARDING=1`).
 
+Three more paste-prompts cover the steps after that, served and structured the same way
+(one file each under `docs/prompts/`):
+
+```
+Read http://localhost:9292/docs/add-tool.md then add a currency tool to my agent
+Read http://localhost:9292/docs/diagnose-turn.md — this turn failed, investigate
+Read http://localhost:9292/docs/run-examples.md then run every example for me
+Read http://localhost:9292/docs/go-live.md then take this agent to production
+```
+
 ## Docs by goal
 
-**Understand the idea**
+The groups below are the same six the [site](https://guizaols.github.io/insika/)
+navigates by.
+
+**Start here**
 
 - [Why Insika](docs/WHY.md) — a runtime vs a DIY loop, an assembled framework, or a hosted gateway.
+- [Running locally](docs/RUNNING-LOCAL.md) — the local demo, the control UI, wiring tools to your own backend.
 - [Architecture](docs/ARCHITECTURE.md) — the turn pipeline, the tool-loop, checkpoint recovery, composition roots, diagrams.
+- [Demo data](docs/DEMO.md) — `insika demo:seed` populates one agent with a funnel, follow-ups, refinement runs, approvals, facts and evals at once, so a fresh instance shows every loop working before you build your own.
 
-**Build an agent**
+**Core concepts**
 
-- [Agents](docs/AGENTS.md) — the AgentProfile and its five access layers; create and edit at runtime.
+- [Agents](docs/AGENTS.md) — the AgentProfile and every key on it; create and edit at runtime.
+- [Limits and policy](docs/POLICY.md) — the five layers: allowed tools, approvals, guardrails, edge limits, reasoning.
 - [Tools](docs/TOOLS.md) — code vs data vs MCP tools, manifests, egress troubleshooting.
 - [Skills](docs/SKILLS.md) — the SKILL.md format and progressive loading.
 - [Context](docs/CONTEXT.md) — what fills a turn's prompt; budget, eviction, memory.
 - [Workflows](docs/WORKFLOWS.md) — deterministic orchestration of several agents: the five patterns, and when to let the model choose instead.
+- [examples/](examples/) — one small runnable project per capability.
+
+**Integrate**
+
+- [The /v1 API](docs/API.md) — the frozen compatibility contract: the surface, the additive-only rule, the version gate.
 - [Channels](docs/CHANNELS.md) — how people reach the agent: a widget on your site in one `<script>` tag, or keep your own WhatsApp/Slack stack (relay).
+- [Media](docs/MEDIA.md) — photos, voice notes and documents through the message contract; generated images back out.
+- [Embedding](docs/EMBEDDING.md) — mount Insika into the Ruby app you already have: `Insika.embed(backend:)` and a Rack app for your router.
 - [Plugins](docs/PLUGINS.md) — the two extension tiers: config-only, or a gem the engine loads.
-- [Running locally](docs/RUNNING-LOCAL.md) — the local demo, the control UI, wiring tools to your own backend.
-- [examples/](examples/) — one small runnable project per capability.
+- [Templates](docs/TEMPLATES.md) — example agents shipped in the gem: `insika new <name>` or a Studio click gets you a running, editable app in one command.
 
 **Ship it**
 
 - [Security](docs/SECURITY.md) — guardrails, egress, approvals, edge limits, secrets.
 - [Sandbox](docs/SANDBOX.md) — the confined-execution primitive.
-- [Deploy](docs/DEPLOY.md) — Falcon, a durable SQLite volume, tokens.
-- [Embedding](docs/EMBEDDING.md) — mount Insika into the Ruby app you already have: `Insika.embed(backend:)` and a Rack app for your router.
+- [Deploy](docs/DEPLOY.md) — Falcon, a durable SQLite volume, tokens, the process model.
+- [Router](docs/ROUTER.md) — the session-sticky proxy that keeps per-session guarantees past one worker.
+- [Releasing](docs/RELEASING.md) — how the gem is cut, and the install proof that runs before it is published.
 
-**Operate & prove it**
+**Operate**
 
-- [Observability](docs/OBSERVABILITY.md) — OpenTelemetry (opt-in): turns as traces and metrics, the attribute convention, dashboard recipes.
-- [Benchmark](docs/BENCHMARK.md) — the neutral, reproducible, provider-free engine benchmark.
+- [Observability](docs/OBSERVABILITY.md) — the event stream, plus opt-in OpenTelemetry: turns as traces and metrics, the attribute convention, dashboard recipes.
+- [Schedules](docs/SCHEDULING.md) — recurring turns the engine fires on its own tick, with no cron on another box.
+- [Artifacts](docs/ARTIFACTS.md) — a report the agent can hand you a URL to.
 - [Load test](docs/LOADTEST.md) — load-testing and data topology.
+- [Soak](docs/SOAK.md) — the long run that finds what a short one cannot.
+- [Benchmark](docs/BENCHMARK.md) — the neutral, reproducible, provider-free engine benchmark.
+
+**Improve**
+
 - [Evals](docs/EVALS.md) — the cases that grade an agent: rubrics, the judge panel, and the pre-merge gate.
 - [Refinement](docs/REFINEMENT.md) — read an agent's own traffic back as a ranked report of what broke.
+- [Outcomes and follow-ups](docs/OUTCOMES.md) — what the traffic was worth in business terms, and the tool that comes back on a promise.
+- [Knowledge](docs/KNOWLEDGE.md) — extract durable concepts from finished conversations into a per-agent knowledge base; provenance-stamped, PII-redacted, best-effort with a re-scan recovery path (retrieval and consolidation are not shipped yet).
+- [Facts](docs/FACTS.md) — distill finished customer conversations into proposed facts; a human approves them into memory (nothing is ever applied automatically).
+- [Harvest](docs/HARVEST.md) — mine real traffic for SKILL proposals; the negative list, the evidence-ledger grounding filter and the double gate hold every proposal to a human approval and the append-only log (nothing is ever applied automatically).
+
+**Reference**
+
+- [The domain-free core](docs/domain.md) — what the gem ships, what a deployment declares, and how to clear it.
 
 All of the above is also browsable, searchable and cross-linked at
 **[guizaols.github.io/insika](https://guizaols.github.io/insika/)** — the same files,