insika 0.3.0 → 0.7.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 9ac198fc4ba91d6f3d3b55ebdf782b326bee455d45be766befaff3fc761009ff
-  data.tar.gz: e315b3f30cbe8365bbb3c6db6477209935ac71f5d0cadca26b6a5bc3458fa1cb
+  metadata.gz: 5c2b2fbe064af45aa132f93de00954b24dc504e052db2e059b4f6daa151e248b
+  data.tar.gz: 50cabcc10d82eabdfd08de562bd61feddf1a9de68ee6427343c80c5cf6ad9d8c
 SHA512:
-  metadata.gz: 985b9924a6996aeb9a180ae6c800191029aaaad48ea55c18ffb35e8df159b14fbbc801bf7fc7cc33dc2291c6f8e80ffba078578f715ee062410c0504579819d3
-  data.tar.gz: a3a7dd5ff765f9fd652b116f7a8b03c05d3fb29cc582e1a14508e5e0ee5afb088b2d755a194da9e378b262871f84a884adbbc18a8e5e65f8af36b49706b76047
+  metadata.gz: 3004e30a234349bcb91db185fcc31f9b52cbdeb7660c613b1fc229adf6bbbe717d9abc8ebcdc5bf9ecc125986694c0d2f3a82e1d48eb83105ff74db0467e4d91
+  data.tar.gz: ca4bec6a8b8cd03ee1a5c2df627e7da778d3dd157f30f848c70f8077ac467570284bf934e3a4c8040fb223503855d9cdcad111b9139ed093c702b6a8850ea1ff

data/CHANGELOG.md

@@ -8,6 +8,186 @@ it is released. Entries land with the pull request that makes the change.
 
 ## [Unreleased]
 
+## [0.7.0] - 2026-08-29
+
+The pre-1.0 consolidation: MCP servers as first-class tools, a template gallery,
+the knowledge layer (extract → consolidate → retrieve → export), a session-sticky
+router for N>1 capacity, and the Studio brought to visual parity. The `/v1`
+contract is unchanged. The version jumps from 0.3 to 0.7 to say what the code
+already says — the surface is the 1.0 surface; what remains is operational proof,
+not features.
+
+### Fixed
+
+- **An agent's identity never falls back to another agent's.** An agent with
+  neither `prompt_files` nor `base_prompt` silently inherited the deployment-wide
+  default identity, so one agent could answer as another's persona (seen live,
+  deterministic, previously misdiagnosed as a concurrency bug). It now raises
+  `Insika::ContextError` and aborts the turn instead.
+- **A text-only turn absorbs a burst of messages instead of leaking a follow-up.**
+  Steering only drained the mailbox at a tool-batch boundary; a turn with no tool
+  calls never had one, so a customer who sent four messages in a row got one
+  answer that only saw the first. The executor now runs one extra round in the
+  same turn when the mailbox is not empty.
+- **The turn supervisor starts at boot, not on the first served turn** — a
+  deployment whose only agents are scheduled never fired tick/alert/distill
+  workers until unrelated traffic arrived.
+- **Every Linux/Docker build was failing** on two independent packaging gaps:
+  `Gemfile.lock` only listed `arm64-darwin` (#185), and the Dockerfile's builder
+  stage did not copy `lib/insika/packaging.rb`, which the gemspec requires (#184;
+  a spec now parses the Dockerfile against the gemspec's `require_relative`s).
+- **MCP**: `refresh` reused a permanently broken memoized client, so an edited
+  server config needed a full process restart; and ruby_llm-mcp sent a
+  spec-invalid `Origin` header (URL with a path) that Origin-checking servers
+  (GitHub, Grafana, Metabase) rejected on every request — now stripped.
+- **Media**: STT and image-generation calls always raised past the RubyLLM
+  boundary (raw bytes where a path was expected, `assume_model_exists` with no
+  provider). Found live-validating against real audio/PDF fixtures; a contract
+  spec now guards the boundary.
+- **`insika harvest`/`knowledge:backfill` never wired the CLI's provider
+  keys** — both build their own miner/extractor `ask` outside a full app
+  boot (no `LlmConfigurator`), so `RubyLLM.config` was empty and a real run
+  raised `RubyLLM::ConfigurationError` on the first live call, no matter how
+  the agent was configured. Found running `knowledge:backfill` against real
+  staging traffic for the first time. `configure_eval_llm!` (added for
+  `evals:simulate`, PR #190) is now `configure_cli_llm!` and both commands
+  call it before building their factory.
+
+### Added
+
+- **MCP servers are first-class tools** (#178–#182). `Insika::McpClient` builds a
+  real stdio/http/sse client (ruby_llm-mcp, lazy-loaded) from an `McpStore`
+  record; `McpToolRegistry` executes live through it, merged into the overlay
+  registry as a third source (base > data-tools > mcp) so allowlists, traces,
+  budgets and checkpoints work unchanged. Config surfaces: JSON import/export in
+  the `mcpServers` format, `insika mcp list|add|remove|import|test|refresh`, the
+  `mcp` DSL declaration (which auto-grants `mcp:<name>` on the agent's allowlist),
+  `/v1/mcp`, and a transport-aware `/studio/mcp` page with per-instance status and
+  "Test connection". stdio is arbitrary command execution by config and stays
+  behind `INSIKA_MCP_STDIO=1`.
+- **Template gallery** (#183). Six example agents ship inside the gem
+  (`lib/insika/templates/<name>/agent.rb` + README) — one DSL file is both doors:
+  `insika new` copies it, the Studio gallery instantiates it. Starter
+  (travel-planner, keyless public APIs), Advanced (research-analyst fan-out),
+  Always-on (daily-digest: schedule + artifact), Teams (review-panel) and more.
+- **Session-sticky router** — `insika-router`, a standalone Async/Falcon proxy that
+  runs N engine backends (`WEB_CONCURRENCY=1` each) behind a consistent-hash ring
+  keyed on the session id, so per-session FIFO/collect/steer guarantees hold at
+  N>1 capacity with no change to the executor. Static (Railway) and DNS
+  (Kubernetes headless Service) backend discovery; an unreachable backend answers
+  the retry envelope. Opt-in.
+- **Studio visual parity** — view transitions and motion (CSS only, reduced-motion
+  aware), the transcript as product (mono label rows, tool calls as pill chips,
+  three-zone session viewer), a live home over `/studio/events`, master-detail
+  lists.
+- **`queue_mode` and `steer_max_messages` editable per agent** in the Studio
+  Config form — operations can switch steering on/off without touching the store.
+  Blank inherits the platform default; `steer_max_messages: 0` never steers.
+- **Paste-prompt journeys for coding agents** — four skill-shaped prompts under
+  `docs/prompts/`, served at `GET /docs/<name>.md` next to `start.md`:
+  run-examples, add-tool, diagnose-turn and go-live.
+- **`run_persona_eval` swaps side-effect tools instead of refusing** — a target
+  agent with a reachable POST tool gets a throwaway executor whose registry
+  resolves every side-effect name to a recorder, sharing every other collaborator
+  of the real graph.
+- **`examples/analytics`** — MCP wired to a real, key-authenticated server
+  (OpenSEO over http, bearer via `OPENSEO_API_KEY`).
+- **`insika knowledge:export`** — writes one `<name>.md` per learned concept
+  under a directory (`KnowledgeStore#export_dir`). The storage format IS the
+  export format, so it's a dump, not a converter, directly consumable by
+  okf-gem/graphify. Idempotent — nothing here is lossy, so there's no
+  `--force` to reason about. `--format graphml` writes one combined
+  `knowledge.graphml` instead — a node per concept, an edge per `[[link]]`
+  resolved against the same export set (a dangling link is dropped, never a
+  broken edge) — hand-built, hand-escaped XML, no new dependency.
+
+- **Knowledge retrieval** — with `knowledge: {retrieve: true}`, every turn
+  the engine now searches the agent's learned concepts for the ones
+  relevant to the customer's message (`Insika::Knowledge::Index::Scan` —
+  pure term overlap × confidence × recency, no embeddings, no network call)
+  and injects the top few as a level-1 `<knowledge>` block (name/
+  description/confidence/provenance, new `Context::Priority::KNOWLEDGE =
+  77`, between skills and memory), expanding one hop through a concept's
+  `[[links]]`. A `load_knowledge` tool — outside `tools_allow`, wired
+  exactly like `load_skill` — loads a concept's full body on demand;
+  calling it fires `:knowledge_retrieved`, the adoption metric that
+  actually matters (a concept sitting unread in the prompt taught nothing).
+  `knowledge.index: "fts5"` is accepted but falls back to the built-in scan
+  index — deliberately not built yet: `scripts/bench_knowledge_index.rb`
+  measured `Index::Scan` at hundreds of milliseconds per search before a
+  read-cache fix (re-parsing a concept's frontmatter on every search was
+  ~90% of the cost, not the store I/O); after the fix it's sub-1.5ms at the
+  concept counts this feature targets, and building the SQLite adapter is
+  deferred to a real, documented latency trigger (~1000 concepts/agent)
+  instead of being built speculatively.
+
+- **Knowledge consolidation + the Studio page** — a repeat concept name no
+  longer blindly overwrites. The engine now decides same claim (bumps
+  occurrences/sources/confidence — `min(0.95, 0.5 + 0.1 × distinct_sources)`,
+  no model call), related claim (one extra model call merges the two
+  bodies — `Insika::Knowledge::Consolidator`/`ConsolidatorFactory`), or
+  contradicting claim (never merged — appended under a `## Contradiction`
+  heading, confidence dropped to `0.4`, `:knowledge_conflict` emitted). No
+  consolidator configured, or an unusable answer, defaults to contradicting —
+  the conservative choice. `/studio/knowledge` (single-agent-scoped like
+  Harvest) lists every concept with a conflict filter, the same CodeMirror
+  editor Skills uses (also how an operator hand-promotes `provenance:
+  observed` to `policy`), version history/restore, and delete — all through
+  three new bus commands (`write_concept`/`delete_concept`/`restore_concept`,
+  wired through every composition root).
+
+- **Knowledge, layer 1 (extraction)** — the engine can now learn durable
+  **concepts** (facts, procedures, policies, objections) from finished
+  conversations, opt-in per agent via `knowledge extract: true`. After a turn
+  completes, off the critical path, the platform `utility_model` proposes
+  candidate concepts; the engine schema-validates the answer, drops any
+  model-authored `provenance`/`confidence`/`sources` (stamped by the engine
+  instead — every extracted concept is `provenance: observed`, never
+  `policy`), redacts the body for PII, and persists it as a markdown+
+  frontmatter record in the new `KnowledgeStore` (`Insika::KnowledgeStore`,
+  scoped per agent/tenant like `MemoryStore`, versioned like `SkillStore`).
+  Emits `:knowledge_learned` (name/type/agent only, never content). The
+  recovery path is `insika knowledge:backfill --agent ID [--since DATE]`,
+  replaying stored sessions through the same extractor. No retrieval into a
+  turn's prompt yet — see `docs/KNOWLEDGE.md` for what's shipped and what's
+  still planned.
+
+- **Plugin loading is on in every root** — `Server::Boot`'s `load_plugins`
+  step was a no-op in both composition roots, so the tested
+  `Insika::Plugin::Loader` never ran outside the `insika-code`
+  example. It now runs at boot in the minimal wiring, the demo deployment
+  and DSL-run agents, via a shared `Wiring::Graph.load_plugins`. Discovery
+  roots: announced gems (default-enabled), `INSIKA_PLUGIN_DIR` (workspace)
+  and the repo's `plugins/` — the latter two gated by `INSIKA_PLUGINS`,
+  with `INSIKA_PLUGINS_DISABLED` as the absolute veto. Plugin skills and
+  prompts join the catalogs at the lowest precedence (a workspace or
+  authored skill still wins). The dead `OPENCLAW_PLUGIN_DIR` env spec
+  (nothing ever read it) is replaced by the three new keys.
+
+- **Media parity** — three transport gaps closed, all additive:
+  `generate_image` can now EDIT as well as generate — `source_image_urls`
+  (or, absent those, the turn's own inbound photo by default) ride
+  `RubyLLM.paint(with:)`, an optional `mask_url` rides `paint(mask:)`; a
+  fourth inbound part type, `document` (`{ "type": "document", "url": … }`,
+  capped at 10 MB), attaches to the ask like an image and deposits
+  `{{ctx.document_url}}` for data tools; and audio transcription now carries
+  a vocabulary `prompt:` (per-agent `stt_prompt`, falling back to the
+  deployment-wide `INSIKA_STT_PROMPT`) so domain terms (product names, brand
+  terms) transcribe correctly instead of phonetically. Text-to-image and
+  plain audio transcription stay byte-identical when neither feature is used.
+- **Demo data** — `insika demo:seed` (and a matching "Seed demo data" button
+  under Studio Settings) provisions a bundled `demo-store` agent and writes
+  enough realistic data to see every loop working at once: a funnel with a
+  frozen baseline, follow-ups in all four states, refinement runs across the
+  lifecycle, pending and resolved approvals, distillation proposals and a
+  memory fact, and a golden set with a mixed-result baseline. One code path
+  (`Insika::Demo::Seeder`) behind both front doors, same discipline as every
+  other Studio button. Fixed two pre-existing gaps this surfaced: `config.ru`
+  never wired `proposal_store`/`budget_ledger` into the Studio (the Facts page
+  and the funnel's spend pill were unreachable in production), and
+  `scripts/serve_real.rb` never wired the outcome/funnel/follow-up/proposal
+  stores at all.
+
 ## [0.3.0] - 2026-08-19
 
 The proof-and-consultant wave: shadow parity against a frozen criterion, the

data/README.md

@@ -80,42 +80,77 @@ served — no secrets) and **`GET /docs`** + **`GET /docs/<name>.md`** (these do
 markdown). Public and on by default when you `serve`; opt-in in production
 (`INSIKA_ONBOARDING=1`).
 
+Three more paste-prompts cover the steps after that, served and structured the same way
+(one file each under `docs/prompts/`):
+
+```
+Read http://localhost:9292/docs/add-tool.md then add a currency tool to my agent
+Read http://localhost:9292/docs/diagnose-turn.md — this turn failed, investigate
+Read http://localhost:9292/docs/run-examples.md then run every example for me
+Read http://localhost:9292/docs/go-live.md then take this agent to production
+```
+
 ## Docs by goal
 
-**Understand the idea**
+The groups below are the same six the [site](https://guizaols.github.io/insika/)
+navigates by.
+
+**Start here**
 
 - [Why Insika](docs/WHY.md) — a runtime vs a DIY loop, an assembled framework, or a hosted gateway.
+- [Running locally](docs/RUNNING-LOCAL.md) — the local demo, the control UI, wiring tools to your own backend.
 - [Architecture](docs/ARCHITECTURE.md) — the turn pipeline, the tool-loop, checkpoint recovery, composition roots, diagrams.
+- [Demo data](docs/DEMO.md) — `insika demo:seed` populates one agent with a funnel, follow-ups, refinement runs, approvals, facts and evals at once, so a fresh instance shows every loop working before you build your own.
 
-**Build an agent**
+**Core concepts**
 
-- [Agents](docs/AGENTS.md) — the AgentProfile and its five access layers; create and edit at runtime.
+- [Agents](docs/AGENTS.md) — the AgentProfile and every key on it; create and edit at runtime.
+- [Limits and policy](docs/POLICY.md) — the five layers: allowed tools, approvals, guardrails, edge limits, reasoning.
 - [Tools](docs/TOOLS.md) — code vs data vs MCP tools, manifests, egress troubleshooting.
 - [Skills](docs/SKILLS.md) — the SKILL.md format and progressive loading.
 - [Context](docs/CONTEXT.md) — what fills a turn's prompt; budget, eviction, memory.
 - [Workflows](docs/WORKFLOWS.md) — deterministic orchestration of several agents: the five patterns, and when to let the model choose instead.
+- [examples/](examples/) — one small runnable project per capability.
+
+**Integrate**
+
+- [The /v1 API](docs/API.md) — the frozen compatibility contract: the surface, the additive-only rule, the version gate.
 - [Channels](docs/CHANNELS.md) — how people reach the agent: a widget on your site in one `<script>` tag, or keep your own WhatsApp/Slack stack (relay).
+- [Media](docs/MEDIA.md) — photos, voice notes and documents through the message contract; generated images back out.
+- [Embedding](docs/EMBEDDING.md) — mount Insika into the Ruby app you already have: `Insika.embed(backend:)` and a Rack app for your router.
 - [Plugins](docs/PLUGINS.md) — the two extension tiers: config-only, or a gem the engine loads.
-- [Running locally](docs/RUNNING-LOCAL.md) — the local demo, the control UI, wiring tools to your own backend.
-- [examples/](examples/) — one small runnable project per capability.
+- [Templates](docs/TEMPLATES.md) — example agents shipped in the gem: `insika new <name>` or a Studio click gets you a running, editable app in one command.
 
 **Ship it**
 
 - [Security](docs/SECURITY.md) — guardrails, egress, approvals, edge limits, secrets.
 - [Sandbox](docs/SANDBOX.md) — the confined-execution primitive.
-- [Deploy](docs/DEPLOY.md) — Falcon, a durable SQLite volume, tokens.
-- [Embedding](docs/EMBEDDING.md) — mount Insika into the Ruby app you already have: `Insika.embed(backend:)` and a Rack app for your router.
+- [Deploy](docs/DEPLOY.md) — Falcon, a durable SQLite volume, tokens, the process model.
+- [Router](docs/ROUTER.md) — the session-sticky proxy that keeps per-session guarantees past one worker.
+- [Releasing](docs/RELEASING.md) — how the gem is cut, and the install proof that runs before it is published.
 
-**Operate & prove it**
+**Operate**
 
-- [Observability](docs/OBSERVABILITY.md) — OpenTelemetry (opt-in): turns as traces and metrics, the attribute convention, dashboard recipes.
-- [Benchmark](docs/BENCHMARK.md) — the neutral, reproducible, provider-free engine benchmark.
+- [Observability](docs/OBSERVABILITY.md) — the event stream, plus opt-in OpenTelemetry: turns as traces and metrics, the attribute convention, dashboard recipes.
+- [Schedules](docs/SCHEDULING.md) — recurring turns the engine fires on its own tick, with no cron on another box.
+- [Artifacts](docs/ARTIFACTS.md) — a report the agent can hand you a URL to.
 - [Load test](docs/LOADTEST.md) — load-testing and data topology.
+- [Soak](docs/SOAK.md) — the long run that finds what a short one cannot.
+- [Benchmark](docs/BENCHMARK.md) — the neutral, reproducible, provider-free engine benchmark.
+
+**Improve**
+
 - [Evals](docs/EVALS.md) — the cases that grade an agent: rubrics, the judge panel, and the pre-merge gate.
 - [Refinement](docs/REFINEMENT.md) — read an agent's own traffic back as a ranked report of what broke.
+- [Outcomes and follow-ups](docs/OUTCOMES.md) — what the traffic was worth in business terms, and the tool that comes back on a promise.
+- [Knowledge](docs/KNOWLEDGE.md) — extract durable concepts from finished conversations into a per-agent knowledge base; provenance-stamped, PII-redacted, best-effort with a re-scan recovery path (retrieval and consolidation are not shipped yet).
 - [Facts](docs/FACTS.md) — distill finished customer conversations into proposed facts; a human approves them into memory (nothing is ever applied automatically).
 - [Harvest](docs/HARVEST.md) — mine real traffic for SKILL proposals; the negative list, the evidence-ledger grounding filter and the double gate hold every proposal to a human approval and the append-only log (nothing is ever applied automatically).
 
+**Reference**
+
+- [The domain-free core](docs/domain.md) — what the gem ships, what a deployment declares, and how to clear it.
+
 All of the above is also browsable, searchable and cross-linked at
 **[guizaols.github.io/insika](https://guizaols.github.io/insika/)** — the same files,
 rendered. Reading this repo as an agent? [llms.txt](llms.txt) indexes the docs;