insika 0.8.0 → 1.0.1

data/docs/PERFORMANCE.md

@@ -0,0 +1,110 @@
+---
+title: Performance
+parent: Operate
+nav_order: 8
+permalink: /performance/
+---
+
+# Performance
+
+What the engine does to stay cheap per turn, and what was tried and left out. A turn
+mostly waits on the model provider, so the engine's job is to add little CPU, hold
+little memory and never block the other turns in the same process.
+
+To measure your own load, see [Deploy › Measuring performance](DEPLOY.md#measuring-performance--load),
+[Load test](LOADTEST.md) and the [Benchmark](BENCHMARK.md).
+
+## Numbers you can reproduce
+
+Each one comes from a script in this repository, with no API key and no real data.
+
+| What | Number | Run |
+|---|---|---|
+| Engine work per turn, model stubbed | p50 ≈ 0.4 ms, ~1,700–1,900 turns/s per process | `bundle exec ruby scripts/bench.rb` ([Benchmark](BENCHMARK.md)) |
+| SQLite writes, 1 to 8 processes on one file | ~25–30k writes/s in total, zero "database is locked" | `bundle exec ruby scripts/bench_store.rb 1,2,4,8 3000` ([Deploy](DEPLOY.md#1-sqlite-write-ceiling-no-provider--bench_storerb)) |
+| One key lookup, `execute` vs a cached prepared statement | ~2.7 µs vs ~0.9 µs (Ruby 4.0, laptop) | the snippet below |
+
+A turn makes a few hundred store calls at most. Saving ~2 µs on each is well under a
+millisecond per turn, which is why statement caching is in the "left out" table.
+
+```ruby
+require "sqlite3"; require "benchmark"
+db = SQLite3::Database.new(":memory:")
+db.execute("CREATE TABLE kv (scope TEXT, key TEXT, value TEXT, PRIMARY KEY (scope, key))")
+1000.times { |i| db.execute("INSERT INTO kv VALUES (?, ?, ?)", ["s", "k#{i}", "x" * 500]) }
+sql = "SELECT value FROM kv WHERE scope = ? AND key = ?"
+st = db.prepare(sql)
+Benchmark.bm(9) do |b|
+  b.report("execute")  { 200_000.times { |i| db.get_first_value(sql, ["s", "k#{i % 1000}"]) } }
+  b.report("prepared") { 200_000.times { |i| st.reset!; st.bind_params("s", "k#{i % 1000}"); st.step } }
+end
+```
+
+## What the engine does
+
+**No Rails, no ORM.** The server is Falcon with Rack and Roda. The store is a key-value
+table driven by the `sqlite3` gem directly. Nothing loads a framework or builds model
+objects on the hot path.
+
+**One fiber per turn.** Each turn runs as an `Async` task. While it waits on the provider,
+a tool or the store, the same process runs other turns. One worker holds many concurrent
+conversations without threads. See [Architecture](ARCHITECTURE.md).
+
+**Session actors end when idle.** Turns of the same session run in order through a small
+actor. Its loop ends as soon as its queue drains and it leaves the executor's table, so an
+idle session holds no fiber.
+
+**SQLite tuned for many writers.**
+
+- WAL mode with `synchronous = NORMAL`.
+- `busy_handler_timeout=` instead of `busy_timeout=`. The C-level handler sleeps while
+  holding the GVL, so a worker waiting on another process's write lock would freeze every
+  fiber it runs. The Ruby-level handler lets the scheduler keep the other turns going.
+- **Group commit.** Writes that arrive while one is waiting for the lock join the same
+  transaction and share one `COMMIT`. Each caller still returns only after its own write is
+  committed, so durability and per-fiber order do not change. Fewer commits means less
+  fighting over the file's single write lock when several workers share it.
+- Prefix and recent-items reads are one range query, not one query per key.
+- `INSIKA_SLOW_STORE_MS` logs every slow store call, and every slow group-commit batch with
+  its time split into wait, statements and commit. A long wait with a short commit means
+  another process holds the lock. A long commit means the batch itself is heavy.
+
+**Memory goes back to the OS.**
+
+- `MALLOC_ARENA_MAX=2` caps glibc's malloc arenas. Under a multi-worker load, the peak
+  stays lower with the same latency.
+- `MALLOC_MMAP_THRESHOLD_=131072` pins the mmap threshold. Large buffers stay mmapped and
+  return to the OS when freed instead of fragmenting the arenas.
+- `Insika::MemoryTrim` runs on a quiet worker: no turn in flight and enough allocated since
+  the last pass. It runs a full GC and then `malloc_trim(0)`. Without it, a worker stays at
+  its peak after a burst, because a GC alone gives nothing back to the OS. It needs the
+  `fiddle` gem and does nothing where `malloc_trim` does not exist (macOS, musl).
+- Long-lived fibers return `nil` from their blocks, so a finished task's result is not kept
+  alive by a stale reference.
+
+**YJIT on.** The reference image sets `RUBY_YJIT_ENABLE=1`. It lowers CPU per turn
+(serialization, streaming, context building) for a small memory cost per worker.
+
+**Workers are a contract input, not a tuning knob.** More worker processes means more
+memory and more writers on one SQLite file. Start with one. Add workers, and the
+[Router](ROUTER.md) in front of them, only when one process's CPU is the limit. See
+[Deploy › The process model](DEPLOY.md#the-process-model).
+
+## Tried and left out
+
+Each of these was measured on a production-like load and did not pay off. Measure again
+before you reopen one.
+
+| Idea | Why it is out |
+|---|---|
+| jemalloc or mimalloc instead of glibc | `MALLOC_ARENA_MAX` plus `MemoryTrim` already got most of the gain. Both left a higher memory floor after repeated loads, and `malloc_trim` does nothing under them. |
+| Preforking workers from a parent with the code loaded (copy-on-write) | Saves memory only right after boot. After a load, each worker's floor is its own heap, not duplicated code. |
+| `GC.compact` during the trim | Fewer heap pages, same RSS. |
+| Async HTTP adapter for the provider | Slower end to end than plain `Net::HTTP`, and it breaks Streamable-HTTP MCP outside an Async reactor. |
+| Write-behind for traces, compressing large stored values | No measurable gain over group commit. Old compressed rows are still read. |
+| Postgres store for speed on one box | Every store call becomes a network round trip, and a turn makes many store calls. Use it to share the store across nodes, not for speed. See [Deploy › Postgres store](DEPLOY.md#postgres-store-optional). |
+| Caching prepared SQLite statements | Saves microseconds per call. Too small next to a turn that waits on the provider. |
+
+Once group commit is in place, writer contention on one SQLite file under sustained heavy
+load is a matter of architecture: fewer writer processes, a faster volume, or a shared
+store. More application-level tuning does not move it.

data/docs/PLUGINS.md

@@ -73,7 +73,7 @@ require "ruby_llm"
 module WeatherPlugin
   class GetWeather < RubyLLM::Tool
     description "Looks up the current weather for a city"
-    param :city, desc: "City name"
+    parameter :city, description: "City name"
 
     def execute(city:) = { city: city, temp_c: 24, condition: "sunny" }
   end
@@ -169,9 +169,9 @@ require "insika/plugin"
 Insika::Plugin.announce(File.expand_path("../..", __dir__))
 ```
 
-Discovery details worth knowing: the manifest may be named `insika.plugin.yml`
-(preferred) or `plugin.yml` (deprecated — it warns); one manifest per directory,
-with `insika.plugin.yml` winning if both exist; and if two roots ship the same
+Discovery details worth knowing: the manifest is named `insika.plugin.yml` (a leftover
+`plugin.yml` from before 1.0 is not loaded and warns to rename it); one manifest per
+directory; and if two roots ship the same
 `id`, the **first root wins** and the second is skipped.
 
 ### Secrets

data/docs/POLICY.md

@@ -34,7 +34,15 @@ Builtins cover tool-, skill-, and workflow-allowlisting, plus
 does not allow or deny but *tags* a tool as needing human approval. Set
 `approvals_required: [tool names]`; the gate then fires when the model tries to
 call that tool, suspending the turn until an operator approves it in the Studio.
-See [Security](SECURITY.md#human-approval).
+A data tool's `requires_evidence` check runs before this approval gate; an unknown
+ID is blocked without asking the operator. See
+[Tools](TOOLS.md#provenance-checking-ids-before-a-write) and
+[Security](SECURITY.md#human-approval).
+
+Checkpoints preserve tools enabled by `tool_search`; resuming restores only those
+still allowed by the current policy. If a tool halts the turn, unanswered sibling
+calls receive a non-execution result instead of waiting for approval or executing.
+Their outstanding operator approvals are rejected with `resolved_by: engine:halted`.
 
 ## Layer 3: Guardrails (content safety)
 
@@ -92,8 +100,8 @@ budget daily: 100_000, monthly: 2_000_000, soft: false   # or soft: true
 
 ### Reliability — retries, fallback, circuit breaker
 
-The provider interaction is a single attempt by default (RubyLLM's own 2
-transport retries aside). For a store that cannot have a dead model take the
+Without a reliability policy, transport retries use the provider context's
+configuration. For a store that cannot have a dead model take the
 chat down, the reliability policy is DATA on the profile:
 
 ```ruby
@@ -102,11 +110,12 @@ reliability retries: 2, backoff: "exponential",
             circuit_breaker: { after: 10, within: 60, cooldown: 300 }
 ```
 
-- **Retries** — transient failures (`:retryable` / `:rate_limited_*` per the
-  error classification) retry with exponential backoff, up to `retries`.
-  A `:fatal` (auth, billing, bad request) is NEVER retried or rotated. Each
-  attempt runs on a fresh chat — the customer-visible answer comes only from
-  the attempt that returns.
+- **Retries** — RubyLLM owns transport retries for native chats, up to `retries`
+  per request, using its transient-error classification and backoff. Once stream
+  data has arrived, the same request is not retried. `backoff: "exponential"`
+  doubles the native interval; `"constant"` keeps it fixed. Interval and jitter
+  come from the provider context. Insika does not add another retry loop.
+  Fatal authentication/billing/request errors do not trigger model rotation.
 - **Fallback** — after a node's retries, the turn ROTATES to the next model in
   the chain: the profile's `fallback` refs first, then the platform
   `fallback_models`. The turn's usage is attributed to the model that actually
@@ -115,12 +124,15 @@ reliability retries: 2, backoff: "exponential",
   `within` seconds open the circuit; while open, the turn fail-fasts with the
   typed `circuit_open` + `retry_after` (remaining cooldown) and the provider is
   never touched. After `cooldown` a half-open trial closes the circuit on
-  success or reopens it on failure.
-- **`timeout`** — per-attempt ceiling (default 30s), counted as a retryable
-  failure.
-
-Absent `reliability` = the plain single attempt, byte-for-byte today's
-behavior.
+  success or reopens it on failure. A native retry sequence that ultimately
+  fails counts once, not once per internal HTTP attempt. `provider_failure`
+  reports that exhausted sequence; native usage records retain individual attempts.
+- **`timeout`** — the native request timeout and the overall ceiling for one
+  model's execution (default 30s), including its retries. Exhaustion can rotate
+  to the next allowed model but never adds another same-model retry.
+
+Absent `reliability` leaves the provider context unchanged and does not enable
+Insika's model rotation or circuit breaker.
 
 ### Intent routing — classify before you answer
 

data/docs/RELEASING.md

@@ -14,11 +14,20 @@ is invisible to it. Do not publish on rspec alone.
 
 ## Before anything
 
+RubyLLM comes from RubyGems (`~> 2.1` in the gemspec); the development Gemfile must
+not Git-pin it, or the published gem resolves a different RubyLLM than the suite ran.
+The measured performance tradeoff was explicitly accepted for the RubyLLM 2 migration,
+not for future regressions; see [migration results](RUBYLLM_2_MIGRATION.md).
+
 1. The suite is green: `bundle exec rspec`.
 2. `lib/insika/version.rb` carries the version being published.
 3. Every new `lib/` file is **tracked in git**. The gemspec's `files` come from
    `git ls-files`: an untracked file builds without a warning and the installed
    gem fails at `require` — this is exactly the failure this proof exists to catch.
+4. If changing the `fencing` default (currently off), first run the deployment's
+   golden cases with it enabled, retain the comparison report, and document the
+   behavior change in `CHANGELOG.md`. A default change needs evaluation evidence;
+   a scheduled version number alone is not the release gate.
 
 ## Cut the gem
 

data/docs/ROUTER.md

@@ -106,6 +106,13 @@ INSIKA_ROUTER_BACKENDS=http://127.0.0.1:9292,http://127.0.0.1:9293,http://127.0.
   bundle exec insika-router
 ```
 
+The image does this for you: set `INSIKA_WORKERS=N` (and keep
+`WEB_CONCURRENCY=1`) and `deploy/entrypoint.sh` runs `deploy/workers.sh`, which
+starts the N workers on ports from `INSIKA_WORKER_BASE_PORT` (default 9300),
+waits until each accepts connections, then binds the router on `$PORT` with a
+backend timeout of `INSIKA_ROUTER_BACKEND_TIMEOUT` (default 1300s there, so a
+long turn is never cut). Unset or `1` boots exactly as before.
+
 `insika doctor` treats `WEB_CONCURRENCY>1` as `ok` (not `error`/`warn`) once
 it sees `INSIKA_ROUTER_BACKENDS` or `INSIKA_ROUTER_BACKENDS_DNS` set — it
 cannot verify a router process is actually running at those addresses, only

data/docs/RUBYLLM_2_MIGRATION.md

@@ -0,0 +1,307 @@
+---
+title: RubyLLM 2 Migration
+parent: Ship it
+nav_order: 6
+permalink: /rubyllm-2-migration/
+---
+
+# RubyLLM 2 Migration
+
+Status reviewed on 2026-09-24: implemented on `codex/rubyllm-2-migration`,
+and **approved for merge and testing on the existing Railway staging service**.
+Keep Insika's durable runtime and policy;
+delegate provider protocols, tool schemas, and cache formatting to RubyLLM.
+No Rails, package extraction or database migration. JSONSchemer is a direct
+dependency for workflow and tool argument validation.
+
+The follow-up ecosystem audit and execution status (kept with the internal plans)
+records replacements and the concrete gates that still block others.
+
+## Dependency
+
+The Gemfile and lockfile pin RubyLLM commit
+`099c381b44fec6f1d72b2b63b9a9eea7635755f1`, the merge of
+[native MCP PR 959](https://github.com/crmne/ruby_llm/pull/959).
+The external `ruby_llm-mcp` dependency and its Origin-header patch are removed.
+The pinned source still reports version 2.0.0, but published 2.0.0 does not contain
+this client. Before publishing Insika, require a released RubyLLM version with
+native MCP in the gemspec and remove the Git pin. A gemspec cannot carry it.
+
+Existing MCP records and cached tool descriptors remain readable. Discovery and
+execution use native Streamable HTTP or stdio; Insika retains policy, evidence,
+credential masking, the stdio opt-in, and the egress allowlist. Refresh preserves
+the server's strict boolean read-only hint. Legacy SSE records fail with a request
+to configure a Streamable HTTP endpoint; they are not silently rewritten. Native
+MCP also requires HTTPS outside loopback, even with the HTTP egress opt-in.
+Sampling, roots and server logging are not supported by the native client. Insika
+does not yet expose native MCP input-request resumption or its OAuth UI; unanswered
+input requests surface as tool errors through the existing wrapper.
+
+Native MCP validation on 2026-09-23: **5,614 examples, zero failures**, seed 46560.
+Local protocol fixtures cover real stdio and HTTP sockets, modern discovery,
+legacy handshake/session headers, event-stream responses, authorization headers,
+tool errors, structured content, attachments, and read-only metadata. No external
+production MCP server was invoked for these fixture checks. The current-candidate
+benchmark below was repeated after this dependency switch; it does not call MCP.
+
+The official RubyLLM coding skill (now from 2.1.0) is copied into `.agents/skills/rubyllm`.
+Refresh it from the installed gem when upgrading RubyLLM, not from upstream main.
+
+## Preserved Behavior
+
+- Native `step` drives the existing runtime; a terminal tool finishes its batch
+  and prevents another model request. Siblings, result IDs, steering, and output remain covered.
+- Tools use `parameter` and `parameters_schema`; data-defined JSON schemas stay unchanged.
+- Native cache markers preserve the stable/volatile Anthropic instruction boundary.
+- Native tokens and model readers retain Insika's usage fields. Auxiliary billing
+  includes cache reads and writes without counting thinking twice.
+- RubyLLM now handles operator approval gating for native chats. Insika retains
+  authorization, persisted decisions, customer confirmation, security checks and UI.
+  Decisions bind to task, turn, call ID, real tool name and arguments. Checkpoints
+  preserve pending calls, completed results, usage, evidence and filtered halt text.
+  Recorded side effects are skipped on replay; terminal results survive interruption.
+  Direct workflow/customer-confirmation paths retain the existing blocking gate.
+  Legacy tool-name decisions do not authorize newly generated native calls.
+  A crash between an external effect and its local marker still requires downstream
+  idempotency; this is not an exactly-once guarantee.
+- Opt-in native
+  compaction now uses separate model-bound state and filtered stored history, with
+  original-prefix recovery on fallback. See [native compaction](CONTEXT.md#native-responses-compaction)
+  for its conservative budget and supported providers.
+
+## Verification
+
+Latest completed checks (2026-09-24, before merge): **5,616 RSpec
+examples, zero failures**, seed 19184; **53 Studio tests passed**. Assets were
+rebuilt during the preceding Studio implementation and remain unchanged.
+Final diff review reran 200 focused runtime/MCP/contract examples with no failures
+and found no actionable introduced regression in the reviewed scope.
+Browser checks covered tool grouping by origin/server, filtering, collapsible
+sections and section selection without changing other sections or denied tools.
+Changes still require **Save tools**; filtering does not change permissions.
+
+Local development validation also exercised a real provider reply and MCP tool
+discovery. The user confirmed WhatsApp replies and execution through the Metabase
+MCP. This is smoke-test evidence, not a comparative quality eval or a new benchmark.
+An HTTP data tool and an MCP instance may use different credentials: successful
+MCP discovery does not validate a separately configured HTTP tool.
+
+### Earlier Verification
+
+Native transport retries are enabled for reliable native chats, with isolated
+per-chat configuration and no outer Insika retries. The approved behavior is
+per-request retries, stopping once streaming starts. Insika still owns fallback,
+model policy, tenant circuits and the overall per-model deadline. Circuits count
+an exhausted native sequence once, not each internal HTTP failure. The native
+request timeout is also configured from the profile. Completed tool results
+survive fallback, and failed-stream redactor tails are discarded.
+Native `with_fallbacks` adoption remains deferred: it restores the original model
+before the next generation and requires adaptation for model-bound compaction.
+
+After native compaction, operator approvals and native transport retries: **5,603 examples, zero failures** (seed 32098).
+The execution record (kept with the internal plans) has the new contracts, remaining replacement gates and offline benchmark.
+
+Full RSpec after the benchmark correction: **5,519 examples, zero failures**
+(seed 55367). Studio: **50 tests passed** during the preceding migration validation.
+The real-gem contract has 17 behavioral cases, with no missing-gem skip.
+Synthetic SQLite records passed 1.16 write -> 2.0 read/write -> 1.16 read/write,
+including history, pending/resolved approvals, and recorded side-effect markers.
+An isolated installed gem passed reply, SQLite embed, Rack mount, actual HTTP/SSE,
+Studio login, packaged assets/docs, and lazy-load guards using an offline provider.
+That earlier install used the former external MCP Git dependency. On 2026-09-24,
+the current native-MCP candidate was built and installed in a fresh GEM_HOME
+outside the checkout and passed all those checks again. The synthetic SQLite
+1.16 -> 2.0 -> 1.16 record rehearsal also passed again. A Linux ARM64 Docker
+build with the deployment lockfile passed and loaded native MCP while preserving
+the core lazy-load guard. The isolated gem install explicitly pins RubyLLM's Git
+revision and resolves other allowed dependencies; it is not a public gem-install
+proof or a substitute for verifying Railway's x86_64 deployment.
+Browser checks passed for login, agent configuration, skill/provider creation,
+Playground replies, completed tasks, and session transcripts. Approval/resume
+flows were covered by automated tests, not separately exercised in the browser.
+
+## Performance
+
+### Current Candidate (2026-09-24)
+
+Five alternating runs per version in each of three modes (stub, real Chat with
+a synthetic model, and real Chat with catalog model `gpt-4o`), plus five SQLite
+runs per version. The catalog experiment was subsequently corrected and rerun
+five times per version, as described below. Ruby 4.0.6,
+arm64-darwin25, YJIT off; 300 latency turns, 30 warmups, concurrency 16 and
+20 waves per scenario. Native MCP and the latest Studio changes are present.
+The same existing benchmark harnesses load each checkout's actual runtime.
+
+All 40 benchmark processes succeeded: 58,500 engine turns including warmups,
+matching provider/tool/history/chunk counts, and 300,000 SQLite writes with zero
+reported lock errors. Automated suites ran after the measurement. Development
+servers remained running, so workstation noise is not eliminated.
+
+Profiling exposed a flaw in the earlier catalog experiment: its flag assertion
+wrapped `RubyLLM.chat`, but `main` constructs chats through a context. Its
+`assume_model_exists: false` override therefore did not reach the actual Chat.
+Those earlier catalog numbers do not establish equivalent registry behavior.
+The corrected local harness forces the option at `Chat#initialize` and asserts
+that the resulting model is the exact registry object. The separate profiler
+confirms one registry lookup per measured turn in both versions.
+
+All ten corrected benchmark processes succeeded, with matching activity counts
+across 19,500 additional turns including warmups. Median corrected measurements:
+
+| Scenario | Main p50 / p95 ms | Migration p50 / p95 ms | Turns/s, before -> after |
+| --- | --- | --- | --- |
+| Greeting | 1.37 / 2.01 | 1.48 / 2.48 | 654.1 -> 603.8 |
+| Two tool rounds | 1.46 / 2.10 | 1.80 / 2.42 | 603.5 -> 515.7 |
+| History | 1.51 / 2.08 | 1.62 / 2.25 | 573.9 -> 558.2 |
+
+Catalog p50 increases are **7.3-23.3%**, or **0.11-0.34 ms per turn**.
+Peak RSS increased **13.9%** (87,769,088 -> 99,991,552 bytes), and tool
+allocations increased 8.4%. Tool p50 ranges did not overlap (main 1.35-1.58 ms,
+migration 1.65-1.85 ms); the other scenarios' ranges overlap. These are new runs,
+so differences from the invalid comparison are not attributable solely to the
+harness correction. Synthetic-model p50 increased 18.3-28.2% and peak
+RSS 29.4%. Stub p50 changes were -5.3%, +4.5% and +9.1%, with peak RSS +2.3%.
+SQLite median writes/s stayed within 2.3% of baseline in either direction.
+
+The proposed 10% gate still fails numerically. This isolates an increase along the real-Chat
+path, not its cause; it does not measure native HTTP, MCP calls, UI speed or
+provider latency. No safety or accounting was disabled. Earlier measurements
+below are retained as historical comparisons, not current acceptance evidence.
+
+**Decision (2026-09-24):** The project owner accepted the measured performance
+tradeoff for this migration and approved continuing with RubyLLM 2's native
+runtime. The additional 0.34 ms in the two-tool benchmark does not block ongoing
+development validation. This is an explicit exception, not a passing benchmark
+or acceptance of future regressions. There are no real operational workflows
+beyond the tests already executed. Continue validation with the existing
+automated tests, controlled integration checks and offline benchmarks. Measure
+live latency, errors and memory when actual usage exists; that is not a current
+development prerequisite. Investigate
+upstream improvements with isolated evidence; do not add custom caches or a
+replacement tool loop without demonstrated waste. The performance decision alone
+did not authorize deployment. The project owner subsequently authorized merge
+to `main` and testing through its automatic Railway staging deployment. Gem
+publication remains outside that authorization.
+
+### Targeted Profiling (2026-09-24)
+
+Three alternating instrumented runs per version, each with 1,001 measured
+two-tool turns after 30 warmups, completed without errors. Both versions made
+one registry lookup, three provider calls and two tool calls per measured turn.
+The probe wraps selected methods using the monotonic clock and `GC.stat`; it is
+local diagnostic tooling, not a runtime dependency.
+
+Median inclusive time per turn, rounded, across those instrumented runs:
+
+| Scope | Main ms | Migration ms |
+| --- | --- | --- |
+| `Executor#ask_on` (generation and tool cycle) | 0.402 | 0.659 |
+| `Executor#create_chat` | 0.261 | 0.279 |
+| `Executor#record_model_visible` | 0.011 | 0.042 |
+| `Executor#persist_turn` | 0.326 | 0.322 |
+| `ContextBuilder#call` | 0.251 | 0.267 |
+
+The largest observed increase is inside the generation/tool cycle. Native
+`instrumentation_payload`, `record_completion_event`, and
+`record_out_of_band_usage` account for approximately 0.030, 0.022 and 0.038 ms
+per turn respectively. The latter is the offline provider's fallback accounting
+path, not the accounting path used by native HTTP providers. Message creation
+also grows from approximately 0.063 to 0.079 ms for the same 55 messages/chunks
+per turn. These nested times overlap other scopes and must not be added to them.
+
+Registry lookup is slower (approximately 0.073 -> 0.111 ms), but native 2.0
+creates one connection per turn instead of 1.16's two; the overall Chat creation
+difference is much smaller. The trace increase includes corrected behavior:
+`main` treats the tool Hash as pairs and misses schemas, whereas the migration
+captures actual tool schemas and system instructions. Removing those records
+would reintroduce a correctness problem, not provide an equivalent optimization.
+
+Instrumented timings include probe overhead and cannot apportion the exact
+uninstrumented regression. Instrumented allocation counts are not clean
+allocation measurements. RSS growth remains unattributed; MCP and HTTP were not
+exercised. The next investigation should isolate RubyLLM's per-round bookkeeping
+without Insika before proposing an upstream optimization. Keep accounting,
+approval checks and trace correctness intact; no custom tool loop or cache was
+added during profiling.
+
+### Earlier Catalog-Model Experiment (Invalid Flag Assertion)
+
+These results used the ineffective `RubyLLM.chat` assertion described above.
+They are historical only, not an equivalent-registry comparison. Five
+alternating runs per version named `gpt-4o`. The offline provider is
+registered under `openai` only inside the experiment; it does not make HTTP calls.
+Its intended assertions were bypassed by context-created chats. No application
+model-selection behavior was changed.
+Settings match the earlier comparison: Ruby 4.0.6, YJIT off, 300 latency turns,
+30 warmups, concurrency 16, 20 waves, and identical tool/history/chunk counts.
+Every scenario in all ten runs completed without errors.
+
+| Scenario | 1.16 p50 / p95 ms | 2.0 p50 / p95 ms | Turns/s, before -> after |
+| --- | --- | --- | --- |
+| Greeting | 1.23 / 1.65 | 1.32 / 2.06 | 549.2 -> 629.2 |
+| Two tool rounds | 1.36 / 1.75 | 1.62 / 2.09 | 620.3 -> 539.8 |
+| History | 1.28 / 1.78 | 1.44 / 1.95 | 590.6 -> 561.9 |
+
+Median p50 changes are +7.3%, +19.1%, and +12.5%. Median peak RSS is
+89,194,496 -> 105,971,712 bytes (+18.8%); tool allocations rise 4.8%.
+These numbers cannot determine whether registry lookup avoidance explains the
+difference. They are offline engine-plus-Chat results, not
+live-provider latency or proof of a RubyLLM defect. Investigate the remaining
+tool-loop and memory differences before making an upstream performance claim.
+
+### Earlier Unknown-Model Experiment
+
+The following results are retained for comparison, not generalized to catalog
+models. The fake model used `assume_model_exists: true`, which bypassed registry
+resolution in 1.16 but not 2.0.
+
+Five interleaved runs per version, Ruby 4.0.6, YJIT off, same host and settings.
+Real Chat with a synthetic provider, no network; provider/tool/chunk/history counts match.
+
+| Scenario | 1.16 p50 / p95 ms | 2.0 p50 / p95 ms | Turns/s, before -> after |
+| --- | --- | --- | --- |
+| Greeting | 1.38 / 1.83 | 1.60 / 2.23 | 476.3 -> 571.2 |
+| Two tool rounds | 1.35 / 1.95 | 1.79 / 2.44 | 619.8 -> 491.5 |
+| History | 1.34 / 1.87 | 1.78 / 2.37 | 597.3 -> 498.2 |
+
+These replace the initial measurements: the offline provider omitted request
+`model_info`, unlike native `Accounting::Usage::Tracker#succeed`. Supplying it
+removes three spurious pricing catalog lookups per two-tool turn. The benchmark
+spec now rejects those lookups. No production runtime workaround was added.
+
+Corrected real-Chat p50 increased 15.9-32.8%; peak RSS increased 24.0%
+(87,310,336 -> 108,232,704 bytes); tool allocations increased 7.4%.
+RSS also increased 11.3% in stub mode, so the entire memory difference cannot
+be attributed to RubyLLM catalog loading. SQLite had zero lock errors, but
+throughput varied from +1.6% to -17.2%; the earlier no-regression claim is withdrawn.
+These are host-specific samples, not a controlled attribution of every difference.
+
+The standalone `scripts/rubyllm_profile.rb` reproduces the fixture defect and
+the remaining catalog lookup without Insika. With `--request-context --profile`,
+1,000 two-tool turns make 1,000 `Models#find` calls in 2.0 versus none in 1.16.
+RubyLLM 2 resolves catalog metadata even with `assume_model_exists`; 1.16 skipped
+it for this non-local provider. In one instrumented run that lookup consumed
+112 ms total; fallback usage accounting consumed 39 ms. Inclusive timings must
+not be summed. All three requests, both tools, history and aggregate tokens are
+checked. The fake provider bypasses native transport and uses fallback usage
+accounting, so this is not an end-to-end native-provider performance claim.
+
+For upstream discussion, run the script outside Bundler with `1.16.0 --profile`
+and `2.0.0 --request-context --profile`. Omitting `--request-context` deliberately
+recreates the old fixture defect, not native provider behavior. The useful upstream
+question is whether repeated registry resolution can be cheaper while preserving
+model metadata and refresh semantics. No upstream patch or issue was published;
+remaining latency and memory attribution is open.
+The proposed 10% performance gate **failed**. No accounting or safety was disabled.
+Reproduce the candidate with the commands in [Benchmark](BENCHMARK.md).
+
+## Release Gate
+
+Public gem release still requires a published RubyLLM native-MCP dependency.
+The measured performance tradeoff is accepted; current installed-artifact and
+synthetic record-compatibility checks passed. Merge to `main` and testing of its
+automatic Railway staging deployment are authorized. Confirm that deployment's
+status and HTTP behavior before reporting success. No production rollout,
+database schema migration or gem publication is part of this validation.
+Record compatibility does not guarantee exactly-once external effects; keep
+destination idempotency or reconciliation for an execution-before-checkpoint failure.

data/docs/SECURITY.md

@@ -205,6 +205,32 @@ category: the agent's category reply → the agent's default → the builtin
 category → the builtin default. All of it is editable in the Studio Configuration
 form. See [Agents §Layer 3](POLICY.md#layer-3-guardrails-content-safety).
 
+## Third-party text is data (fencing)
+
+Input guardrails cover the incoming message. The opt-in `fencing` flag also
+sanitizes ordinary tool result strings, memory text and the injected knowledge
+names/descriptions, and places a
+fixed notice under the identity. It reduces known markup and Unicode tricks;
+it does not make arbitrary third-party instructions safe.
+
+The user/assistant-only extraction filter applies even when fencing is off.
+Attachment captions and `load_knowledge` bodies are not fenced. The extraction
+filter excludes direct tool messages,
+but assistant paraphrases can still become extraction input. See
+[Agents](AGENTS.md#fencing--third-party-text-is-data-never-instructions) for the
+exact scope, exceptions, default and size cap.
+
+## Write provenance and serial execution
+
+A data tool's `requires_evidence` gate checks IDs against the session ledger
+before approval or backend execution. A known ID proves a prior lookup; it does
+not authorize access or establish current stock, price or quantity limits.
+Those checks remain the backend's responsibility.
+
+Tools marked `side_effect` execute one at a time within a session, including in
+parallel batches. This is not a cross-session or distributed backend lock.
+See [Tools](TOOLS.md#provenance-checking-ids-before-a-write).
+
 ## Human approval
 
 Some tool calls should not happen unattended. Mark them with
@@ -220,6 +246,13 @@ resume path is durable: an approval that arrives after a restart still resumes t
 turn from its checkpoint (see
 [Architecture](ARCHITECTURE.md#durability-checkpoints-and-resume)).
 
+There is a second approver: the customer. `customer_confirm: [tool names]` holds a
+call the same way, but the turn *ends* with a question instead of suspending, and the
+next message of the conversation decides it through `confirm_pending` /
+`cancel_pending`. A hold nobody resolves expires when that next turn ends. Use it for
+the write the person on the other end must agree to; use approval for the one a staff
+member must. A tool cannot be on both lists. See [Tools](TOOLS.md#customer-confirmation-a-write-the-conversation-approves).
+
 ## Egress: the SSRF boundary
 
 Every outbound HTTP call from a data tool passes through the **EgressGuard**, a
@@ -242,7 +275,7 @@ defense-in-depth: without it, `ALLOW_PRIVATE` opens *any* private destination.
 > the request never leaves the process, and the conversation *looks* fine. Verify
 > tool health by the Studio session **trace** (a healthy call shows the backend's
 > `200`), never by the model's reply. Full detail in
-> [Tools §Egress](TOOLS.md#egress-the-ssrf-guard-and-its-silent-failure).
+> [Tools §Egress](TOOLS.md#egress-the-ssrf-guard).
 
 ## Sandbox: confined execution
 

data/docs/SHARED-CONVERSATIONS.md

@@ -0,0 +1,40 @@
+---
+layout: default
+title: Shared conversations
+---
+
+# Shared conversations
+
+`shared_conversations: true` opts an agent into required central history. The
+DSL accepts `shared_conversations true`; stored profiles preserve the flag.
+Default is false. Configure `INSIKA_CONVERSATIONS_URL`, a scoped
+`INSIKA_CONVERSATIONS_TOKEN`, and a durable native backend (`INSIKA_DB` or
+`INSIKA_DATABASE_URL`). The client provisions the central conversation first.
+
+The tenant-authenticated `/v1/responses` request adds `user_text` (original speech)
+and `shared_conversation` with UUID `tenant_id`, `user_id`, `agent_id`,
+`conversation_id`, `turn_id`, and `message_id`. Keep the
+same turn/message IDs across retries. Existing `input` still contains the
+presenter/context text used for this request. The authenticated tenant must match.
+Omit `generation` to acquire a settled conversation automatically when admitting
+the next turn. The engine reads and writes the acquired generation internally.
+An explicit integer `generation` retains the legacy ownership check.
+
+Upload incoming media to the central attachment endpoint first. Optional
+`shared_conversation.content` carries canonical text/media parts with attachment
+IDs; its text must match `user_text`. Generated image/audio bytes upload before
+completion. Native checkpoints store attachment bytes and hydrate them at the
+RubyLLM boundary. Full tool results retain secret-key masking without prompt caps.
+
+Admission precedes model/tool work. Delivery follows central completion. A lost
+completion acknowledgment reuses the native durable result; it never reruns tools.
+An uncertain execution stays blocked for operator reconciliation. Only the same
+native waiting approval can resume from its checkpoint under the current generation.
+Unresolved customer confirmations block switching. Central deletion/outage fails
+before cached history is used. Local native purge still requires an explicit cleanup.
+
+Run the isolated HTTP proof (synthetic model/tools, temporary databases):
+
+```sh
+SHARED_CONVERSATIONS_SOURCE=/path/to/shared-memory-repo bundle exec rspec spec/insika/shared_conversations_http_spec.rb
+```