insika 0.3.0 → 0.8.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 9ac198fc4ba91d6f3d3b55ebdf782b326bee455d45be766befaff3fc761009ff
-  data.tar.gz: e315b3f30cbe8365bbb3c6db6477209935ac71f5d0cadca26b6a5bc3458fa1cb
+  metadata.gz: 345e57c18a277e8ecd49b2481300c1474c60b2e0c3c384a086740e0de5ec3a2e
+  data.tar.gz: 800794431e279fec34f135e4e203a0488b04066b667bc691806ea5f575d81530
 SHA512:
-  metadata.gz: 985b9924a6996aeb9a180ae6c800191029aaaad48ea55c18ffb35e8df159b14fbbc801bf7fc7cc33dc2291c6f8e80ffba078578f715ee062410c0504579819d3
-  data.tar.gz: a3a7dd5ff765f9fd652b116f7a8b03c05d3fb29cc582e1a14508e5e0ee5afb088b2d755a194da9e378b262871f84a884adbbc18a8e5e65f8af36b49706b76047
+  metadata.gz: 199cc16b9ea651ec2da1d2e6f7ddea5020194882da00cf96981e8d5d44086233862873d14640d11cd1d445f9d202992ed46691210f9fb1699b1a3d56a99b3145
+  data.tar.gz: 44d24bc53036da3803ee6eba6203d1addc8ccf0bd27c1aa15d1e9acfe763bca807ede55c2b7136ee2c245e5cd725541c719baa018c99e09638e964803bf688a9

data/CHANGELOG.md

@@ -8,6 +8,302 @@ it is released. Entries land with the pull request that makes the change.
 
 ## [Unreleased]
 
+## [0.8.0] - 2026-08-31
+
+The post-consolidation hardening wave: the last two OpenClaw-era names are gone
+from the wire (**breaking** — `insika:<agent>` / `X-Insika-Agent` and
+`INSIKA_GATEWAY_TOKEN`, no aliases), and the engine learned to manage its own
+long conversations — in-session compaction (RFC-0044), a tool budget that warns
+before it kills, goal recitation at the tail of the context, and a tool-usage
+audit report. Update `/v1` consumers and the gateway-token variable together
+with this upgrade.
+
+### Added
+
+- **In-session compaction (RFC-0044).** When a session's uncompacted
+  transcript grows past `compact_after` messages (default 40), everything but
+  the last `keep_last` (default 20) is summarized by a cheap model
+  (`compaction.model` → platform `utility_model`) into one
+  `<conversation_summary>` history fragment; the tail stays verbatim and the
+  boundary is stable between compactions, so the prompt cache holds after it.
+  Runs post-turn, off the critical path, best-effort; persisted on the session
+  record with a monotonic boundary. Opt-in via the long-reserved Settings
+  `compaction` hash (the Studio general form is back), `insika doctor` warns
+  when it is enabled with no model, and every compaction is observable:
+  `:context_compacted` event, `insika.context.compacted` counter, `{upto,
+  runs}` + its own category in the context trace. See
+  [Context](docs/CONTEXT.md).
+
+- **`insika tools:report` — the tool audit.** The per-session trace never
+  aggregated, so "which tools does this agent carry and never use?" had no
+  answer. The new CLI report reads the stored traces per agent (tasks →
+  sessions → `tool_traces`) and flags allowlisted-but-never-called tools,
+  tools over 30% errors in the window, and tools not called in N days
+  (default 14). Read-only — the operator removes. `--agent`, `--days`,
+  `--json`. See [Tools](docs/TOOLS.md).
+
+- **Two series promoted to OTEL metrics.** `insika.cache.hit_rate` (histogram,
+  `%` — cache reads over the billed prompt, per turn, same arithmetic as the
+  Studio's per-agent series) and `insika.tool.loop_intervened` (counter — the
+  loop detector's one-shot warning, labelled by tool). Both computed in the
+  telemetry bridge from events that already existed: zero cost when OTEL is
+  off, no engine change. See [Observability](docs/OBSERVABILITY.md).
+
+- **The doctor warns on a prompt file that outgrew a prompt.** The
+  `prompt-files` sweep now WARNs (never errors) past ~6 000 estimated tokens
+  or 600 lines, with the actual counts — the LLM-generated merchant-pack
+  shape that costs 20%+ extra tokens per turn. Corrupted-wrapper detection
+  is unchanged.
+
+- **The tool budget is announced before it kills.** `max_tool_calls` was
+  enforced but silent: the model met the ceiling only when the turn died with
+  `stage: :tool_limit`, having delivered nothing. At **10 / 5 / 2** calls
+  remaining the engine now appends a short, escalating notice at the next
+  tool-batch boundary and emits `:tool_budget_warned`. It is a `user` message,
+  never a system one, so the cacheable system prefix stays byte-stable.
+  Verified live (DeepSeek): a turn asking for 30 lookups on a budget of 20 used
+  to die at `tool_limit`; it now stops at 18 and answers with what it has. The
+  turn's counter, the notices and the abort are one object (`TurnBudget`), and
+  the batch-boundary arithmetic it shares with the loop detector moved into
+  `ToolBatch`.
+
+### Changed
+
+- **The briefing recites the goal at the TAIL of the context.** `<briefing>`
+  used to carry known facts, the still-missing list and the next step in one
+  `:system` block at the top. The goal half was **moved** (not copied) into a
+  new `<recitation>` block rendered after the whole history, as the last thing
+  the model reads before the current message — attention is strongest at the
+  end, and a goal stated only at the top is the first thing a 30-call turn
+  forgets. New `:tail` fragment placement; the head keeps the durable facts.
+
+- **The `/v1/responses` wire names are insika-native**: the agent arrives as
+  `model: "insika:<agent>"` and the fallback header is `X-Insika-Agent`. Hard
+  rename, no alias — a consumer still sending `openclaw:<agent>` gets
+  "model/agent missing". Update every `/v1` consumer together with this upgrade.
+- **`INSIKA_GATEWAY_TOKEN` replaces `OPENCLAW_GATEWAY_TOKEN`** as the Bearer for
+  `/v1` + `/a2a`. Hard rename, no alias: the old name was a leftover of the
+  OpenClaw migration and is now a foreign var the engine ignores. Update the
+  variable on every deployment before upgrading. The dead `OPENCLAW_AGENTS_DIR`
+  env spec (nothing read it) is gone.
+
+### Fixed
+
+- **The Studio home charts went blank for three hours every day.** The activity
+  buckets key on calendar parts (date, hour) of `updated_at`, which the engine
+  writes in UTC, but `render_home` read `Time.now` in the host's local zone. On a
+  UTC-3 host, from 21:00 local onward "today" was already tomorrow in UTC: the
+  14-day chart matched no session and the 24h sparkline's floor was set three
+  hours in the future, dropping the newest buckets. Stamps are now read through
+  one `utc_time` helper and `now` is UTC. Instant comparisons (`active_now`)
+  never had the bug — only the calendar arithmetic did.
+- **The Conversations trend showed "−1" for the day's first conversation.** The
+  14-day series is oldest-first and ends at today, so its last pair reads
+  `[yesterday, today]`; it was destructured the other way round, inverting the
+  sign on every conversations delta. The Messages card was always correct.
+- **Declaring a tool allow/deny list now opts the profile into the policy.** The
+  lists are applied by exactly one policy — the builtin `tool_allowlist` — and
+  the policy engine runs only the policies a profile *names*. So an agent with
+  `tools_allow: ["a", "b"]` and no `policies` sent **every registered tool** to
+  the model, silently: 508 tools / 52 405 tokens of schemas per request against
+  an allowlist of 2, observed on a real run. `tools_deny` was ignored the same
+  way. `AgentProfile.build` now appends `tool_allowlist` whenever `tools_allow`,
+  `tools_deny` or `tools_allow_groups` is declared — presence, not emptiness, so
+  `tools_allow: []` still means "no tools". A stored record left in the old
+  state is repaired on read but stays wrong on disk, so `doctor` gained a
+  `tool-allowlist` **error** naming each agent to re-save.
+- **A DSL agent could write its briefing and never read it back.** The
+  `Insika.agent`/`Insika.system` runtime wired every context provider except
+  `Briefing`, so `update_briefing` / `set_next_step` persisted state that never
+  reached a prompt. Now wired, before `Session`, like `config/wiring.rb`.
+- **Docker image shipped without `docs/`, so `GET /start.md` was a 500 with
+  `INSIKA_ONBOARDING=1`.** `.dockerignore` excluded the whole tree; the onboarding
+  surface reads `docs/onboarding/start.md`, `docs/prompts/` and the public
+  `docs/*.md` from disk at request time. Only the Jekyll build output and the
+  internal notes are excluded now, and a spec asserts `.dockerignore` never drops
+  a file `Insika::Onboarding` serves. Found on the first Railway deploy with
+  onboarding on; the gem was never affected (its payload keeps `docs/`).
+
+## [0.7.0] - 2026-08-29
+
+The pre-1.0 consolidation: MCP servers as first-class tools, a template gallery,
+the knowledge layer (extract → consolidate → retrieve → export), a session-sticky
+router for N>1 capacity, and the Studio brought to visual parity. The `/v1`
+contract is unchanged. The version jumps from 0.3 to 0.7 to say what the code
+already says — the surface is the 1.0 surface; what remains is operational proof,
+not features.
+
+### Fixed
+
+- **An agent's identity never falls back to another agent's.** An agent with
+  neither `prompt_files` nor `base_prompt` silently inherited the deployment-wide
+  default identity, so one agent could answer as another's persona (seen live,
+  deterministic, previously misdiagnosed as a concurrency bug). It now raises
+  `Insika::ContextError` and aborts the turn instead.
+- **A text-only turn absorbs a burst of messages instead of leaking a follow-up.**
+  Steering only drained the mailbox at a tool-batch boundary; a turn with no tool
+  calls never had one, so a customer who sent four messages in a row got one
+  answer that only saw the first. The executor now runs one extra round in the
+  same turn when the mailbox is not empty.
+- **The turn supervisor starts at boot, not on the first served turn** — a
+  deployment whose only agents are scheduled never fired tick/alert/distill
+  workers until unrelated traffic arrived.
+- **Every Linux/Docker build was failing** on two independent packaging gaps:
+  `Gemfile.lock` only listed `arm64-darwin` (#185), and the Dockerfile's builder
+  stage did not copy `lib/insika/packaging.rb`, which the gemspec requires (#184;
+  a spec now parses the Dockerfile against the gemspec's `require_relative`s).
+- **MCP**: `refresh` reused a permanently broken memoized client, so an edited
+  server config needed a full process restart; and ruby_llm-mcp sent a
+  spec-invalid `Origin` header (URL with a path) that Origin-checking servers
+  (GitHub, Grafana, Metabase) rejected on every request — now stripped.
+- **Media**: STT and image-generation calls always raised past the RubyLLM
+  boundary (raw bytes where a path was expected, `assume_model_exists` with no
+  provider). Found live-validating against real audio/PDF fixtures; a contract
+  spec now guards the boundary.
+- **`insika harvest`/`knowledge:backfill` never wired the CLI's provider
+  keys** — both build their own miner/extractor `ask` outside a full app
+  boot (no `LlmConfigurator`), so `RubyLLM.config` was empty and a real run
+  raised `RubyLLM::ConfigurationError` on the first live call, no matter how
+  the agent was configured. Found running `knowledge:backfill` against real
+  staging traffic for the first time. `configure_eval_llm!` (added for
+  `evals:simulate`, PR #190) is now `configure_cli_llm!` and both commands
+  call it before building their factory.
+
+### Added
+
+- **MCP servers are first-class tools** (#178–#182). `Insika::McpClient` builds a
+  real stdio/http/sse client (ruby_llm-mcp, lazy-loaded) from an `McpStore`
+  record; `McpToolRegistry` executes live through it, merged into the overlay
+  registry as a third source (base > data-tools > mcp) so allowlists, traces,
+  budgets and checkpoints work unchanged. Config surfaces: JSON import/export in
+  the `mcpServers` format, `insika mcp list|add|remove|import|test|refresh`, the
+  `mcp` DSL declaration (which auto-grants `mcp:<name>` on the agent's allowlist),
+  `/v1/mcp`, and a transport-aware `/studio/mcp` page with per-instance status and
+  "Test connection". stdio is arbitrary command execution by config and stays
+  behind `INSIKA_MCP_STDIO=1`.
+- **Template gallery** (#183). Six example agents ship inside the gem
+  (`lib/insika/templates/<name>/agent.rb` + README) — one DSL file is both doors:
+  `insika new` copies it, the Studio gallery instantiates it. Starter
+  (travel-planner, keyless public APIs), Advanced (research-analyst fan-out),
+  Always-on (daily-digest: schedule + artifact), Teams (review-panel) and more.
+- **Session-sticky router** — `insika-router`, a standalone Async/Falcon proxy that
+  runs N engine backends (`WEB_CONCURRENCY=1` each) behind a consistent-hash ring
+  keyed on the session id, so per-session FIFO/collect/steer guarantees hold at
+  N>1 capacity with no change to the executor. Static (Railway) and DNS
+  (Kubernetes headless Service) backend discovery; an unreachable backend answers
+  the retry envelope. Opt-in.
+- **Studio visual parity** — view transitions and motion (CSS only, reduced-motion
+  aware), the transcript as product (mono label rows, tool calls as pill chips,
+  three-zone session viewer), a live home over `/studio/events`, master-detail
+  lists.
+- **`queue_mode` and `steer_max_messages` editable per agent** in the Studio
+  Config form — operations can switch steering on/off without touching the store.
+  Blank inherits the platform default; `steer_max_messages: 0` never steers.
+- **Paste-prompt journeys for coding agents** — four skill-shaped prompts under
+  `docs/prompts/`, served at `GET /docs/<name>.md` next to `start.md`:
+  run-examples, add-tool, diagnose-turn and go-live.
+- **`run_persona_eval` swaps side-effect tools instead of refusing** — a target
+  agent with a reachable POST tool gets a throwaway executor whose registry
+  resolves every side-effect name to a recorder, sharing every other collaborator
+  of the real graph.
+- **`examples/analytics`** — MCP wired to a real, key-authenticated server
+  (OpenSEO over http, bearer via `OPENSEO_API_KEY`).
+- **`insika knowledge:export`** — writes one `<name>.md` per learned concept
+  under a directory (`KnowledgeStore#export_dir`). The storage format IS the
+  export format, so it's a dump, not a converter, directly consumable by
+  okf-gem/graphify. Idempotent — nothing here is lossy, so there's no
+  `--force` to reason about. `--format graphml` writes one combined
+  `knowledge.graphml` instead — a node per concept, an edge per `[[link]]`
+  resolved against the same export set (a dangling link is dropped, never a
+  broken edge) — hand-built, hand-escaped XML, no new dependency.
+
+- **Knowledge retrieval** — with `knowledge: {retrieve: true}`, every turn
+  the engine now searches the agent's learned concepts for the ones
+  relevant to the customer's message (`Insika::Knowledge::Index::Scan` —
+  pure term overlap × confidence × recency, no embeddings, no network call)
+  and injects the top few as a level-1 `<knowledge>` block (name/
+  description/confidence/provenance, new `Context::Priority::KNOWLEDGE =
+  77`, between skills and memory), expanding one hop through a concept's
+  `[[links]]`. A `load_knowledge` tool — outside `tools_allow`, wired
+  exactly like `load_skill` — loads a concept's full body on demand;
+  calling it fires `:knowledge_retrieved`, the adoption metric that
+  actually matters (a concept sitting unread in the prompt taught nothing).
+  `knowledge.index: "fts5"` is accepted but falls back to the built-in scan
+  index — deliberately not built yet: `scripts/bench_knowledge_index.rb`
+  measured `Index::Scan` at hundreds of milliseconds per search before a
+  read-cache fix (re-parsing a concept's frontmatter on every search was
+  ~90% of the cost, not the store I/O); after the fix it's sub-1.5ms at the
+  concept counts this feature targets, and building the SQLite adapter is
+  deferred to a real, documented latency trigger (~1000 concepts/agent)
+  instead of being built speculatively.
+
+- **Knowledge consolidation + the Studio page** — a repeat concept name no
+  longer blindly overwrites. The engine now decides same claim (bumps
+  occurrences/sources/confidence — `min(0.95, 0.5 + 0.1 × distinct_sources)`,
+  no model call), related claim (one extra model call merges the two
+  bodies — `Insika::Knowledge::Consolidator`/`ConsolidatorFactory`), or
+  contradicting claim (never merged — appended under a `## Contradiction`
+  heading, confidence dropped to `0.4`, `:knowledge_conflict` emitted). No
+  consolidator configured, or an unusable answer, defaults to contradicting —
+  the conservative choice. `/studio/knowledge` (single-agent-scoped like
+  Harvest) lists every concept with a conflict filter, the same CodeMirror
+  editor Skills uses (also how an operator hand-promotes `provenance:
+  observed` to `policy`), version history/restore, and delete — all through
+  three new bus commands (`write_concept`/`delete_concept`/`restore_concept`,
+  wired through every composition root).
+
+- **Knowledge, layer 1 (extraction)** — the engine can now learn durable
+  **concepts** (facts, procedures, policies, objections) from finished
+  conversations, opt-in per agent via `knowledge extract: true`. After a turn
+  completes, off the critical path, the platform `utility_model` proposes
+  candidate concepts; the engine schema-validates the answer, drops any
+  model-authored `provenance`/`confidence`/`sources` (stamped by the engine
+  instead — every extracted concept is `provenance: observed`, never
+  `policy`), redacts the body for PII, and persists it as a markdown+
+  frontmatter record in the new `KnowledgeStore` (`Insika::KnowledgeStore`,
+  scoped per agent/tenant like `MemoryStore`, versioned like `SkillStore`).
+  Emits `:knowledge_learned` (name/type/agent only, never content). The
+  recovery path is `insika knowledge:backfill --agent ID [--since DATE]`,
+  replaying stored sessions through the same extractor. No retrieval into a
+  turn's prompt yet — see `docs/KNOWLEDGE.md` for what's shipped and what's
+  still planned.
+
+- **Plugin loading is on in every root** — `Server::Boot`'s `load_plugins`
+  step was a no-op in both composition roots, so the tested
+  `Insika::Plugin::Loader` never ran outside the `insika-code`
+  example. It now runs at boot in the minimal wiring, the demo deployment
+  and DSL-run agents, via a shared `Wiring::Graph.load_plugins`. Discovery
+  roots: announced gems (default-enabled), `INSIKA_PLUGIN_DIR` (workspace)
+  and the repo's `plugins/` — the latter two gated by `INSIKA_PLUGINS`,
+  with `INSIKA_PLUGINS_DISABLED` as the absolute veto. Plugin skills and
+  prompts join the catalogs at the lowest precedence (a workspace or
+  authored skill still wins). The dead `OPENCLAW_PLUGIN_DIR` env spec
+  (nothing ever read it) is replaced by the three new keys.
+
+- **Media parity** — three transport gaps closed, all additive:
+  `generate_image` can now EDIT as well as generate — `source_image_urls`
+  (or, absent those, the turn's own inbound photo by default) ride
+  `RubyLLM.paint(with:)`, an optional `mask_url` rides `paint(mask:)`; a
+  fourth inbound part type, `document` (`{ "type": "document", "url": … }`,
+  capped at 10 MB), attaches to the ask like an image and deposits
+  `{{ctx.document_url}}` for data tools; and audio transcription now carries
+  a vocabulary `prompt:` (per-agent `stt_prompt`, falling back to the
+  deployment-wide `INSIKA_STT_PROMPT`) so domain terms (product names, brand
+  terms) transcribe correctly instead of phonetically. Text-to-image and
+  plain audio transcription stay byte-identical when neither feature is used.
+- **Demo data** — `insika demo:seed` (and a matching "Seed demo data" button
+  under Studio Settings) provisions a bundled `demo-store` agent and writes
+  enough realistic data to see every loop working at once: a funnel with a
+  frozen baseline, follow-ups in all four states, refinement runs across the
+  lifecycle, pending and resolved approvals, distillation proposals and a
+  memory fact, and a golden set with a mixed-result baseline. One code path
+  (`Insika::Demo::Seeder`) behind both front doors, same discipline as every
+  other Studio button. Fixed two pre-existing gaps this surfaced: `config.ru`
+  never wired `proposal_store`/`budget_ledger` into the Studio (the Facts page
+  and the funnel's spend pill were unreachable in production), and
+  `scripts/serve_real.rb` never wired the outcome/funnel/follow-up/proposal
+  stores at all.
+
 ## [0.3.0] - 2026-08-19
 
 The proof-and-consultant wave: shadow parity against a frozen criterion, the

data/README.md

@@ -80,42 +80,77 @@ served — no secrets) and **`GET /docs`** + **`GET /docs/<name>.md`** (these do
 markdown). Public and on by default when you `serve`; opt-in in production
 (`INSIKA_ONBOARDING=1`).
 
+Three more paste-prompts cover the steps after that, served and structured the same way
+(one file each under `docs/prompts/`):
+
+```
+Read http://localhost:9292/docs/add-tool.md then add a currency tool to my agent
+Read http://localhost:9292/docs/diagnose-turn.md — this turn failed, investigate
+Read http://localhost:9292/docs/run-examples.md then run every example for me
+Read http://localhost:9292/docs/go-live.md then take this agent to production
+```
+
 ## Docs by goal
 
-**Understand the idea**
+The groups below are the same six the [site](https://guizaols.github.io/insika/)
+navigates by.
+
+**Start here**
 
 - [Why Insika](docs/WHY.md) — a runtime vs a DIY loop, an assembled framework, or a hosted gateway.
+- [Running locally](docs/RUNNING-LOCAL.md) — the local demo, the control UI, wiring tools to your own backend.
 - [Architecture](docs/ARCHITECTURE.md) — the turn pipeline, the tool-loop, checkpoint recovery, composition roots, diagrams.
+- [Demo data](docs/DEMO.md) — `insika demo:seed` populates one agent with a funnel, follow-ups, refinement runs, approvals, facts and evals at once, so a fresh instance shows every loop working before you build your own.
 
-**Build an agent**
+**Core concepts**
 
-- [Agents](docs/AGENTS.md) — the AgentProfile and its five access layers; create and edit at runtime.
+- [Agents](docs/AGENTS.md) — the AgentProfile and every key on it; create and edit at runtime.
+- [Limits and policy](docs/POLICY.md) — the five layers: allowed tools, approvals, guardrails, edge limits, reasoning.
 - [Tools](docs/TOOLS.md) — code vs data vs MCP tools, manifests, egress troubleshooting.
 - [Skills](docs/SKILLS.md) — the SKILL.md format and progressive loading.
 - [Context](docs/CONTEXT.md) — what fills a turn's prompt; budget, eviction, memory.
 - [Workflows](docs/WORKFLOWS.md) — deterministic orchestration of several agents: the five patterns, and when to let the model choose instead.
+- [examples/](examples/) — one small runnable project per capability.
+
+**Integrate**
+
+- [The /v1 API](docs/API.md) — the frozen compatibility contract: the surface, the additive-only rule, the version gate.
 - [Channels](docs/CHANNELS.md) — how people reach the agent: a widget on your site in one `<script>` tag, or keep your own WhatsApp/Slack stack (relay).
+- [Media](docs/MEDIA.md) — photos, voice notes and documents through the message contract; generated images back out.
+- [Embedding](docs/EMBEDDING.md) — mount Insika into the Ruby app you already have: `Insika.embed(backend:)` and a Rack app for your router.
 - [Plugins](docs/PLUGINS.md) — the two extension tiers: config-only, or a gem the engine loads.
-- [Running locally](docs/RUNNING-LOCAL.md) — the local demo, the control UI, wiring tools to your own backend.
-- [examples/](examples/) — one small runnable project per capability.
+- [Templates](docs/TEMPLATES.md) — example agents shipped in the gem: `insika new <name>` or a Studio click gets you a running, editable app in one command.
 
 **Ship it**
 
 - [Security](docs/SECURITY.md) — guardrails, egress, approvals, edge limits, secrets.
 - [Sandbox](docs/SANDBOX.md) — the confined-execution primitive.
-- [Deploy](docs/DEPLOY.md) — Falcon, a durable SQLite volume, tokens.
-- [Embedding](docs/EMBEDDING.md) — mount Insika into the Ruby app you already have: `Insika.embed(backend:)` and a Rack app for your router.
+- [Deploy](docs/DEPLOY.md) — Falcon, a durable SQLite volume, tokens, the process model.
+- [Router](docs/ROUTER.md) — the session-sticky proxy that keeps per-session guarantees past one worker.
+- [Releasing](docs/RELEASING.md) — how the gem is cut, and the install proof that runs before it is published.
 
-**Operate & prove it**
+**Operate**
 
-- [Observability](docs/OBSERVABILITY.md) — OpenTelemetry (opt-in): turns as traces and metrics, the attribute convention, dashboard recipes.
-- [Benchmark](docs/BENCHMARK.md) — the neutral, reproducible, provider-free engine benchmark.
+- [Observability](docs/OBSERVABILITY.md) — the event stream, plus opt-in OpenTelemetry: turns as traces and metrics, the attribute convention, dashboard recipes.
+- [Schedules](docs/SCHEDULING.md) — recurring turns the engine fires on its own tick, with no cron on another box.
+- [Artifacts](docs/ARTIFACTS.md) — a report the agent can hand you a URL to.
 - [Load test](docs/LOADTEST.md) — load-testing and data topology.
+- [Soak](docs/SOAK.md) — the long run that finds what a short one cannot.
+- [Benchmark](docs/BENCHMARK.md) — the neutral, reproducible, provider-free engine benchmark.
+
+**Improve**
+
 - [Evals](docs/EVALS.md) — the cases that grade an agent: rubrics, the judge panel, and the pre-merge gate.
 - [Refinement](docs/REFINEMENT.md) — read an agent's own traffic back as a ranked report of what broke.
+- [Outcomes and follow-ups](docs/OUTCOMES.md) — what the traffic was worth in business terms, and the tool that comes back on a promise.
+- [Knowledge](docs/KNOWLEDGE.md) — extract durable concepts from finished conversations into a per-agent knowledge base; provenance-stamped, PII-redacted, best-effort with a re-scan recovery path.
 - [Facts](docs/FACTS.md) — distill finished customer conversations into proposed facts; a human approves them into memory (nothing is ever applied automatically).
 - [Harvest](docs/HARVEST.md) — mine real traffic for SKILL proposals; the negative list, the evidence-ledger grounding filter and the double gate hold every proposal to a human approval and the append-only log (nothing is ever applied automatically).
 
+**Reference**
+
+- [The domain-free core](docs/domain.md) — what the gem ships, what a deployment declares, and how to clear it.
+
 All of the above is also browsable, searchable and cross-linked at
 **[guizaols.github.io/insika](https://guizaols.github.io/insika/)** — the same files,
 rendered. Reading this repo as an agent? [llms.txt](llms.txt) indexes the docs;
@@ -141,5 +176,6 @@ and [CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md). Vulnerabilities go through
 
 ## Status
 
-Pre-release — APIs may still change, nothing is tagged yet
-([CHANGELOG.md](CHANGELOG.md)). Licensed **MIT** ([LICENSE](LICENSE)).
+Pre-1.0 — published on [RubyGems](https://rubygems.org/gems/insika); APIs may
+still change between minor versions ([CHANGELOG.md](CHANGELOG.md)). Licensed
+**MIT** ([LICENSE](LICENSE)).