legion-llm 0.14.10 → 0.14.15

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 259d94128ef1590e757e4929a78e0f260db10ccc447c92033164db40b236198b
-  data.tar.gz: 9b51f6f2bed159bbdbbd19d30af9c1b54e1905bbc284996865ab5da20b581450
+  metadata.gz: 6720d66411b7d2125aab243354ca9f0bf19292b10895695c7e762fe6f23180d0
+  data.tar.gz: 87910fc8387d28d65759e207f79bf6d1a2b39e178a4fefe5c8b6bd5f17187f6b
 SHA512:
-  metadata.gz: 03c431fb67fdb8a8f319bacde1afad35f98141412b7a721c2ff59b94d8f43ae6f2c0d89369e4d662ce310b4c1e0b4354087c90cf796dfed5a4585e79e4133a7a
-  data.tar.gz: e8480e859c80a2c98562fe6619de8f1a16ddd6ba7079d3006092c24f08f1ab9a6e31443080fb7e48d7647c1696fe8917e6b2a21f2ea48780ae079d4750732f34
+  metadata.gz: 83e7680e3083b13e523f66cdba62ff28680f85bc4e63d0e59ef15b4d7cb260c28c02d1325fa76566f5bd8a6606dc55d552bce9dbf097d45d197a02a2a486ac9d
+  data.tar.gz: 9cbc000d34b4afdc8e24dc677ea9a0a22a397868bf1e765d96e2bf7ea173016d293441dbdb6401edd2de30380dd0892cfd73e65550797aca1f4fdaa81e08db0d

data/CHANGELOG.md

@@ -1,5 +1,54 @@
 # Legion LLM Changelog
 
+## [0.14.15] - 2026-07-09
+
+### Fixed
+
+- **OpenAI Responses translator now produces a canonical request equivalent to the Anthropic translator (SSOT / one-oracle).** The same logical request must parse to an identical `Canonical::Request` regardless of client dialect; three divergences were breaking that (caught by the canonical-equivalence checks):
+  - **`upstream_body` no longer leaks into `Canonical::Request.metadata`.** It was written at parse, stripped again before dispatch, and never read (the native `call_responses` path is fed the raw body directly), so it was dead weight that made the two dialects' canonical requests differ. Removed.
+  - **System prompt goes to the canonical `system:` field, not a `role: system` message.** Responses `instructions` (and inline `developer`/`system` input items) now populate `system:` like the Anthropic translator, instead of being injected as a system message.
+  - **Text-then-tool-call is one assistant turn.** A Responses assistant text item immediately followed by `function_call`(s) now merges into a single assistant message (`content` + `tool_calls`), matching the Anthropic single-message shape, instead of splitting into a text message plus an empty-content tool_calls message.
+
+## [0.14.14] - 2026-07-09
+
+### Fixed
+
+- **Curator now strips client-harness noise from the mid-conversation system role (GH#168).** The context curator curated `tool` and `assistant` roles but blindly passed every `system`-role message through — so Claude Code's harness injections (task-tool nags repeated 18×, linter file-dumps) accumulated unbounded, consuming up to ~25% of a 60–80K dispatch window and confusing non-Claude models (e.g. ornith/gemma) into hallucinating or dropping tool calls mid-turn. A new `strip_harness_noise` stage (first in both the heuristic and structural curation pipelines) drops a system message only when it (1) matches a known harness pattern or (2) is a verbatim duplicate of an earlier system message. Fail-safe: unrecognized, non-duplicate system messages and all non-system roles pass through untouched.
+
+### Added
+
+- `llm.context_curation.harness_noise_strip` (default `true`) and `llm.context_curation.harness_noise_patterns` (default: Claude Code task-nag and linter file-dump markers). The pattern list is a plain substring array designed to grow as new harness injections are identified.
+
+### Internal
+
+- Guarded four `String#include?` substring checks (`structured_output`, `classification`, `inventory`, `curator`) with `# rubocop:disable Style/ArrayIntersect`. The cop's autocorrect rewrites `array.any? { |x| str.include?(x) }` to `array.intersect?(str)`, which raises `TypeError` on a String argument — a latent trap for `rubocop -A`.
+
+## [0.14.13] - 2026-07-09
+
+### Fixed
+
+- **Leaked chat-template tool-call tokens are synthesized into real tool calls (native_tool_loop pattern 4).** Some models (observed live with gemma served via vLLM; ledger rows 337049/337111/337145/337174) emit tool-call intent as a raw literal token in the response content instead of the structured `tool_calls` field: `<|tool_call>call:NAME{key:<|"|>value<|"|>,...}<tool_call|>` (where `<|"|>` is the string delimiter). Without synthesis the token reached the client verbatim — a "tool call" printed as text with nothing executed. `maybe_synthesize_tool_call_from_content` now recognizes this as a fourth format (after forced-choice JSON, Qwen tag markup, and Qwen single-tag): it parses the token via a scan-based parser (values terminated only by `<|"|>`, so embedded regular quotes and newlines in e.g. browser `code:` args survive — a gsub-to-quotes + JSON.parse approach breaks on those), validates the tool name against `native_dispatch_tools`, supports multiple tokens in one blob, and lets the executor run the tool server-side. Covered by unit specs (captured-verbatim ledger strings) and an in-process matrix scenario asserting all three client formats never surface the raw markers.
+
+## [0.14.12] - 2026-07-09
+
+### Fixed
+
+- **BYOK model discovery returns 200 for passthrough models.** Clients like GitHub Copilot validate model availability via `GET /v1/models/:id` before sending inference requests. Previously returned 404 for models not in Inventory (e.g. `copilot-utility-small`), even though the daemon auto-routes them successfully. Now returns a synthetic model object for models listed in `llm.routing.model_passthrough_ids`.
+
+### Added
+
+- `llm.routing.model_passthrough_ids` setting (default: `["copilot-utility-small"]`). Models in this list get dedicated `/v1/models/<id>` routes that return a synthetic model object backed by the best available lane's limits. Configurable for additional BYOK client model names.
+
+## [0.14.11] - 2026-07-03
+
+### Fixed
+
+- **Routing token estimate now measures the payload dispatch actually sends (one oracle).** `estimate_request_tokens` walks canonical structs/content blocks via `ContextAccounting` instead of a naive `m[:content].to_s` shadow estimator (which read structured content — the shape Claude Code sends — as near-zero), and now counts the *injected* system prompt (baseline + RAG + skills + prior history) and the *full dispatch tool catalog* (special + client + registry), not just `@request.tools`. Fixes small-context local lanes passing the router's context-window filter then overflowing with `ContextOverflow` at dispatch.
+- **Routing estimates the lane-independent-reduced message set.** Extracted a pure `reduce_messages_for_dispatch` (empty-assistant prune + leading-thinking strip + oversized-tool-result trim) so routing measures exactly what `native_dispatch_messages` puts on the wire, minus the lane-dependent `enforce_context_window` compaction (correctly excluded — the lane isn't chosen yet).
+- **History double-send eliminated.** For a client-managed conversation (e.g. Claude Code) the client resends the full history in `@request.messages` every turn; `step_context_load` also injected the server-stored copy as "Prior conversation history" system text — sending the same turns to the provider twice (real tokens, real cost, real context pressure). `reject_client_resent_history` now drops stored turns the client already re-sent, matched by `(role, content-text)` fingerprint, while still injecting the prior turns a server-managed client omitted.
+- **Router/dispatch context-window agreement.** New `llm.routing.context_headroom` (default `0.90`) applies the same headroom the dispatch budget guard uses, so a lane the router picks must still clear `RouteAttempts#enforce_final_context_budget!`.
+- `token_budget` step now shares the same `ContextAccounting` estimator as the router filter and dispatch guard.
+
 ## [0.14.10] - 2026-06-26
 
 ### Fixed

data/README.md

@@ -93,6 +93,22 @@ Both produced correct, specific technical answers and passed a multi-question re
 
 Curation runs **after** each turn (async), never inside the tool loop, so it never adds latency to an in-flight request. With a 60K-token target the validated payloads stay comfortably in the 40–56K range.
 
+### The knowledge loop
+
+`drop_and_archive` is the strategy that makes curation more than a bigger trash can: instead of discarding overflow context, it sends it to the Apollo knowledge store, where it stays retrievable on future turns. Client-side compaction is lossy truncation — once it's gone, it's gone. This is archival: a fact from turn 3 can resurface at turn 300.
+
+Apollo is two stores. The local per-node store ([legion-apollo](https://github.com/LegionIO/legion-apollo)) starts knowledge at confidence 0.5, corroborates at +0.15, decays at 0.005 per pass, and archives below 0.1 ([confidence.rb](https://github.com/LegionIO/legion-apollo/blob/main/lib/legion/apollo/helpers/confidence.rb)). The shared cross-node store ([lex-apollo](https://github.com/LegionIO/lex-apollo)) corroborates at +0.3 — only when a ≥0.9-cosine match arrives from a *different* provider — and begins time decay after 168 hours. The pipeline steps here use the local store (`Legion::Apollo::Local`).
+
+Lifecycle status in the local store moves through `pending` → `confirmed` / `disputed` → `deprecated` / `archived`. The shared store is backed by PostgreSQL+pgvector (`lib/legion/extensions/apollo.rb`).
+
+The loop closes back up in the Inference pipeline, across three steps:
+
+- `knowledge_capture` (`lib/legion/llm/inference/steps/knowledge_capture.rb`) evaluates the response after each turn and writes new knowledge back to Apollo.
+- `rag_context` (`lib/legion/llm/inference/steps/rag_context.rb`) queries Apollo — via the shared-store runner when available, falling back to `Legion::Apollo.retrieve(scope: :all)`, which merges local and shared results — and injects relevant knowledge into the request before it goes out.
+- `gaia_advisory` (`lib/legion/llm/inference/steps/gaia_advisory.rb`) queries `Legion::Apollo::Local` — the local store only — for advisory context: tone, partner history, routing hints. GAIA advises; it doesn't inject prompts. The pipeline is what pulls context in.
+
+The receipts for this loop aren't a separate claim: RAG-injected tokens are a recorded column in the lex-llm-ledger aggregates already published in [docs/curation-production-metrics.md](docs/curation-production-metrics.md).
+
 ### And everything else you'd expect from a production gateway
 
 Dynamic weighted routing across five tiers, automatic escalation and mid-stream provider failover, per-instance circuit breakers, capability-aware model selection, prompt compression, structured output, embeddings, fleet dispatch over AMQP, unified metering/audit on every request exit, and config-driven model-policy compliance (`model_whitelist`/`model_blacklist`, fail-closed). All of it is documented below.

data/docs/curation-production-metrics.md

@@ -0,0 +1,80 @@
+# Context Curation: Production Metrics
+
+Aggregate metrics from a production LegionIO deployment, exported from
+[lex-llm-ledger](https://github.com/LegionIO/lex-llm-ledger) — the audit ledger that
+records every LLM request the framework makes. All requests over the measurement
+window, bucketed by conversation length. Nothing filtered, nothing excluded.
+
+**The honest headline:** curation saves approximately nothing on short conversations
+and ~98% on long agent sessions — and that asymmetry is the point, because that's
+where the cost lives. In this window, 97.8% of all would-be tokens came from the 50+
+turn bucket, while 99% of conversations by count were single-turn and gained nothing
+(-0.1%, the overhead is noise). Curation doesn't make cheap traffic cheaper; it
+prevents the runaway sessions that dominate spend.
+
+## Baseline definition (read before quoting any percentage)
+
+"Naive" means: no client-side context management at all — the full accumulated
+conversation resent on every turn. That is the default behavior of a plain agent loop,
+but it is a worst-case counterfactual, not a claim about what your current tooling
+does. A client that already trims context will see smaller deltas. All percentages
+below use this single denominator: `actually_sent / naive_would_have_sent`.
+
+## Savings vs. naive resend, by conversation length
+
+| Turns in conversation | 1 | 2–3 | 4–5 | 6–9 | 10–19 | 20–49 | 50+ |
+|---|---|---|---|---|---|---|---|
+| Total requests | 126,202 | 1,941 | 299 | 343 | 559 | 826 | 10,956 |
+| Conversations | 126,202 | 947 | 69 | 50 | 43 | 29 | 37 |
+| Avg turns/conversation | 1 | 2 | 4.3 | 6.9 | 13 | 28.5 | 296.1 |
+| Naive would have sent (tokens) | 163.4M | 2.59M | 4.00M | 6.65M | 28.2M | 74.4M | 12.35B |
+| Actually sent (tokens) | 163.6M | 2.35M | 3.47M | 5.08M | 12.9M | 20.3M | 285.6M |
+| Avg naive per turn | 1,295 | 1,336 | 13,382 | 19,376 | 50,360 | 90,019 | 1,127,381 |
+| Avg actual per turn | 1,296 | 1,208 | 11,607 | 14,812 | 22,998 | 24,521 | 26,064 |
+| **Reduction vs. naive** | **-0.1%** | **9.6%** | **13.3%** | **23.6%** | **54.3%** | **72.8%** | **97.7%** |
+
+The mechanism is visible in the per-turn averages: in the 50+ bucket (dominated by
+long-running agent sessions averaging 296 turns), naive resend grows unbounded toward
+1.13M tokens per turn while curated context stays flat around 26K. Bounded per-turn
+context is the claim this table proves; percentage reductions are a consequence of it.
+
+Token-weighted across all buckets the reduction is ~96%, but quote that number with
+its context: it is almost entirely the 50+ bucket's 97.7%. Reduction scales
+monotonically with conversation depth, and the 50+ figure is not a small-sample
+artifact — it covers 10,956 requests across 37 conversations.
+
+## How the savings happen
+
+The Curator runs asynchronously after each turn (no added request latency) and applies
+deterministic strategies — each one a named method in
+[curator.rb](../lib/legion/llm/context/curator.rb):
+
+- `strip_thinking` — drops stale reasoning blocks from prior turns
+- `distill_tool_result` — summarizes large tool outputs
+- `fold_resolved_exchanges` — collapses completed tool-call/result pairs
+- `evict_superseded` — drops an old file-read when a newer read replaces it
+- `dedup_similar` — Jaccard-similarity dedup of near-identical content
+- `drop_and_archive` — overflow goes to the knowledge store, not the trash
+
+In the 50+ bucket, loaded history of 300.2M tokens was curated down to 106.6M per
+request on average, with a further 75.8M archived to the knowledge store rather than
+resent. The ledger records curation and archival as separate stages that partially
+overlap on the same content, so their per-stage "saved" counters are not additive;
+the end-to-end truth is the naive-vs-actual row above, which is stage-independent.
+
+## Caveats and known gaps, stated plainly
+
+- **Single deployment, single author's workload.** Your mix of conversation lengths
+  determines your savings; the -0.1% single-turn column is as real as the 97.7% one.
+- **The baseline is modeled, not A/B tested.** See the baseline definition above.
+- **Provider prompt caching was off in this window** (cached and cache-creation token
+  columns were zero). These savings come from payload reduction only, so they stack
+  with provider-side caching rather than replacing it.
+- **Tool definitions are not yet curated.** They are static, resent every turn, and
+  material at scale — a known optimization gap, not a solved problem.
+
+## Reproduce
+
+Every number above is a `SELECT` over lex-llm-ledger tables — the same ledger any
+LegionIO deployment gets by installing the extension. The ledger exists for
+finance/audit; these metrics are a side effect of never throwing usage data away.

data/legion-llm.gemspec

@@ -7,8 +7,12 @@ Gem::Specification.new do |spec|
   spec.version       = Legion::LLM::VERSION
   spec.authors       = ['Esity']
   spec.email         = ['matthewdiverson@gmail.com']
-  spec.summary       = 'LLM routing and provider orchestration for the LegionIO framework'
-  spec.description   = 'Routes LLM chat, embeddings, tool use, fleet dispatch, auditing, and provider metadata for LegionIO extensions'
+  spec.summary       = 'LLM gateway: tiered routing, mid-stream failover, and context curation in one proxy'
+  spec.description   = 'A proxy between AI clients and model backends. Requests translate once to a canonical form, ' \
+                       'route to the cheapest capable tier (local, direct, fleet, cloud, frontier), and escalate on ' \
+                       'failure with circuit breakers and mid-stream failover. A post-turn Curator bounds long agent ' \
+                       'sessions (97.7% context reduction vs naive resend at 50+ turns, production-measured). Provider ' \
+                       'adapters are separate lex-llm-* gems.'
   spec.homepage      = 'https://github.com/LegionIO/legion-llm'
   spec.license       = 'Apache-2.0'
   spec.require_paths = ['lib']

data/lib/legion/llm/api/client_translators/openai_responses.rb

@@ -39,15 +39,15 @@ module Legion
             log.debug('[llm][client_translator][openai_responses] action=parse_request')
             body = symbolize(body)
 
-            messages = build_messages(body[:input], body[:instructions])
+            system, messages = build_system_and_messages(body[:input], body[:instructions])
             tools = build_tools(body[:tools])
             params = build_params(body)
             thinking = build_thinking(body[:reasoning])
             tool_choice = build_tool_choice(body[:tool_choice])
-            upstream_body = ensure_reasoning_summary(body)
 
             Canonical::Request.build(
               id:              env['HTTP_X_CLIENT_REQUEST_ID'] || "resp_#{SecureRandom.hex(16)}",
+              system:          system,
               messages:        messages,
               tools:           tools,
               tool_choice:     tool_choice,
@@ -60,8 +60,7 @@ module Legion
                 client_model:     body[:model],
                 tier:             env['HTTP_X_LEGION_TIER'],
                 routing_explicit: legion_routing_explicit_from_env(env),
-                external_refs:    external_refs(body, env),
-                upstream_body:    upstream_body # preserved for native call_responses path until executor is canonical
+                external_refs:    external_refs(body, env)
               }.compact
             )
           end
@@ -107,7 +106,7 @@ module Legion
               thinking:        thinking_to_inference(canonical_request.thinking),
               cache:           { strategy: :default, cacheable: true },
               extra:           extra,
-              metadata:        canonical_request.metadata.except(:upstream_body)
+              metadata:        canonical_request.metadata
             }
 
             apply_canonical_params_to_inference(request_kwargs, canonical_request.params)
@@ -533,18 +532,30 @@ module Legion
             body
           end
 
-          def build_messages(input, instructions)
-            messages = case input
-                       when Array
-                         normalize_input_array(input)
-                       when String
-                         [{ role: 'user', content: input }]
-                       else
-                         []
-                       end
-            messages = [{ role: 'system', content: instructions.to_s }] + messages if instructions
-
-            messages.map do |m|
+          # SSOT: system content belongs in the canonical `system:` field (as the
+          # Anthropic translator does), NOT as a role:system message. Extract it
+          # from top-level `instructions` and from any inline system/developer
+          # input items, and return [system_string_or_nil, non_system_messages].
+          def build_system_and_messages(input, instructions)
+            raw = case input
+                  when Array  then normalize_input_array(input)
+                  when String then [{ role: 'user', content: input }]
+                  else             []
+                  end
+
+            system_parts = []
+            system_parts << instructions.to_s if instructions
+            messages = []
+            raw.each do |m|
+              role = (m[:role] || m['role']).to_s
+              if role == 'system'
+                system_parts << (m[:content] || m['content']).to_s
+              else
+                messages << m
+              end
+            end
+
+            normalized = messages.map do |m|
               {
                 role:         m[:role] || m['role'],
                 content:      m[:content] || m['content'],
@@ -552,6 +563,9 @@ module Legion
                 tool_call_id: m[:tool_call_id] || m['tool_call_id']
               }.compact
             end
+
+            system = system_parts.reject(&:empty?).join("\n\n")
+            [system.empty? ? nil : system, normalized]
           end
 
           def normalize_input_array(input)
@@ -596,15 +610,22 @@ module Legion
           def flush_pending_tool_calls(messages, pending)
             return if pending.empty?
 
-            messages << {
-              role:       'assistant',
-              content:    '',
-              tool_calls: pending.map do |tc|
-                args = tc[:arguments]
-                args = safe_parse_json(args) if args.is_a?(String)
-                { id: tc[:id], name: tc[:name].to_s, arguments: args || {} }
-              end
-            }
+            tool_calls = pending.map do |tc|
+              args = tc[:arguments]
+              args = safe_parse_json(args) if args.is_a?(String)
+              { id: tc[:id], name: tc[:name].to_s, arguments: args || {} }
+            end
+
+            # SSOT: an assistant text item immediately followed by function_call(s)
+            # is ONE assistant turn (text + tool_calls), matching the Anthropic
+            # translator's single-message shape. Merge into the trailing assistant
+            # text message rather than emitting a separate empty-content message.
+            last = messages.last
+            if last && last[:role] == 'assistant' && !last.key?(:tool_calls)
+              last[:tool_calls] = tool_calls
+            else
+              messages << { role: 'assistant', content: '', tool_calls: tool_calls }
+            end
             pending.clear
           end
 

data/lib/legion/llm/api/namespaces/openai/models.rb

@@ -44,6 +44,26 @@ module Legion
                 openai_error(e.message, type: 'server_error', status_code: 500)
               end
 
+              Models.passthrough_model_ids.each do |passthrough_id|
+                app.get "/v1/models/#{passthrough_id}" do
+                  require_llm!
+                  log.debug("[llm][api][namespaces][openai][models] action=passthrough_model id=#{passthrough_id}")
+                  found = Models.synthetic_model_for_auto_route(passthrough_id)
+
+                  unless found
+                    return openai_error("Model '#{passthrough_id}' not found",
+                                        type: 'invalid_request_error', code: 'model_not_found', status_code: 404)
+                  end
+
+                  content_type :json
+                  Legion::JSON.dump(found)
+                rescue StandardError => e
+                  handle_exception(e, level: :error, handled: true,
+                                      operation: 'llm.api.namespaces.openai.models.passthrough')
+                  openai_error(e.message, type: 'server_error', status_code: 500)
+                end
+              end
+
               app.get '/v1/models/:id' do
                 require_llm!
                 model_id = params[:id]
@@ -99,6 +119,23 @@ module Legion
               []
             end
 
+            def self.passthrough_model_ids
+              Legion::Settings[:llm][:routing][:model_passthrough_ids]
+            end
+
+            def self.synthetic_model_for_auto_route(model_id)
+              return nil unless Legion::LLM::Inventory.lanes.any?
+
+              best_lane = Legion::LLM::Inventory.lanes.max_by { |l| l[:lane_weight].to_i }
+              return nil unless best_lane
+
+              Legion::LLM::API::Translators::OpenAIResponse.format_model_object(
+                model_id,
+                owned_by: 'legionio',
+                limits:   best_lane[:limits]
+              )
+            end
+
             def self.to_anthropic_model_list(openai_list)
               openai_list.map { |m| openai_to_anthropic_model(m) }
             end

data/lib/legion/llm/call/structured_output.rb

@@ -126,7 +126,11 @@ module Legion
           end
 
           def supports_response_format?(model)
+            # rubocop:disable Style/ArrayIntersect -- substring check (each model
+            # fragment `include?`d in the model name), NOT array intersection.
+            # `intersect?` raises TypeError on the String arg.
             SCHEMA_CAPABLE_MODELS.any? { |m| model.to_s.include?(m) }
+            # rubocop:enable Style/ArrayIntersect
           end
 
           def retry_enabled?

data/lib/legion/llm/context/curator.rb

@@ -194,6 +194,43 @@ module Legion
           result
         end
 
+        # Strip client-harness noise from the mid-conversation system role
+        # (GH#168). Claude Code and similar harnesses inject unbounded
+        # system-role messages — task nags (identical text, repeated), linter
+        # file dumps — that the rest of the curator never touched, letting them
+        # consume ~25% of a dispatch window.
+        #
+        # System-role only, and fail-safe: a system message is dropped ONLY when
+        # it (a) matches a known harness pattern, or (b) is a verbatim duplicate
+        # of an earlier system message. Unrecognized, non-duplicate system
+        # messages and every non-system message pass through untouched.
+        def strip_harness_noise(messages)
+          return messages unless setting(:harness_noise_strip, true)
+
+          patterns = Array(setting(:harness_noise_patterns, []))
+          seen_system = {}
+          result = messages.reject do |msg|
+            next false unless (msg[:role] || msg['role']).to_s == 'system'
+
+            content = (msg[:content] || msg['content']).to_s
+            next true if patterns.any? { |p| content.include?(p.to_s) }
+
+            if seen_system.key?(content)
+              true
+            else
+              seen_system[content] = true
+              false
+            end
+          end
+
+          stripped = messages.size - result.size
+          if stripped.positive?
+            log.info "[llm][curator] action=strip_harness_noise conversation_id=#{@conversation_id} " \
+                     "stripped=#{stripped} messages_before=#{messages.size} messages_after=#{result.size}"
+          end
+          result
+        end
+
         # Heuristic: deduplicate near-identical messages using Jaccard similarity.
         def dedup_similar(messages, threshold: nil)
           return messages unless setting(:dedup_enabled, true)
@@ -352,6 +389,7 @@ module Legion
             result = messages.map { |msg| strip_thinking(msg) }
           end
 
+          result = strip_harness_noise(result)
           result = fold_resolved_exchanges(result)
           result = evict_superseded(result)
           dedup_similar(result)
@@ -368,7 +406,8 @@ module Legion
         end
 
         def apply_structural_curation_pipeline(messages)
-          result = fold_resolved_exchanges(messages)
+          result = strip_harness_noise(messages)
+          result = fold_resolved_exchanges(result)
           result = evict_superseded(result)
           dedup_similar(result)
         rescue StandardError => e
@@ -575,8 +614,12 @@ module Legion
           clarification_signals = ['clarif', 'what do you mean', 'i see', 'understood', 'got it', 'correct', 'exactly', 'yes', 'right', 'agree']
           conclusion_signals    = ['in summary', 'to summarize', 'in conclusion', 'therefore', 'so to answer', 'the answer is']
 
+          # rubocop:disable Style/ArrayIntersect -- these are substring checks
+          # (signal `include?` against a String), NOT array intersection. The
+          # cop's `intersect?` suggestion raises TypeError on a String arg.
           has_clarification = contents.any? { |c| clarification_signals.any? { |s| c.include?(s) } }
           has_conclusion    = contents.last.length < 500 || conclusion_signals.any? { |s| contents.last.include?(s) }
+          # rubocop:enable Style/ArrayIntersect
 
           has_clarification && has_conclusion
         end

data/lib/legion/llm/inference/executor/context_window.rb

@@ -108,11 +108,23 @@ module Legion
             end
           end
 
-          def strip_thinking_from_history(messages)
-            before_tokens = ContextAccounting.estimate_message_tokens(messages)
+          # Lane-INDEPENDENT reduction applied before dispatch: empty-assistant
+          # prune + leading-thinking strip + oversized-tool-result trim. PURE — no
+          # @context_accounting writes, no logging — so the routing estimate can
+          # call it to measure exactly what native_dispatch_messages will send
+          # (minus the lane-dependent enforce_context_window compaction, which is
+          # correctly excluded from routing since it depends on the chosen lane).
+          def reduce_messages_for_dispatch(messages)
+            msgs = Array(messages).reject { |m| empty_assistant_message?(m) }
+            msgs = strip_thinking_pure(msgs)
+            trim_oversized_tool_results_pure(msgs)
+          end
+
+          # Pure leading-thinking strip. Shared by strip_thinking_from_history
+          # (which adds accounting) and reduce_messages_for_dispatch.
+          def strip_thinking_pure(messages)
             preserve_after = last_user_message_index(messages)
-            stripped_count = 0
-            result = messages.each_with_index.map do |msg, idx|
+            messages.each_with_index.map do |msg, idx|
               next msg if idx >= preserve_after
               next msg unless (msg[:role] || msg['role']).to_s == 'assistant'
 
@@ -122,9 +134,14 @@ module Legion
               cleaned = strip_leading_thinking_block(content)
               next msg if cleaned == content
 
-              stripped_count += 1
               msg.merge(content: cleaned)
             end
+          end
+
+          def strip_thinking_from_history(messages)
+            before_tokens = ContextAccounting.estimate_message_tokens(messages)
+            result = strip_thinking_pure(messages)
+            stripped_count = messages.zip(result).count { |before, after| before != after }
 
             after_tokens = ContextAccounting.estimate_message_tokens(result)
             saved = [before_tokens - after_tokens, 0].max
@@ -158,28 +175,32 @@ module Legion
             text
           end
 
-          def trim_oversized_tool_results(messages)
+          # Pure oversized-tool-result trim. Shared by trim_oversized_tool_results
+          # (which adds logging) and reduce_messages_for_dispatch.
+          def trim_oversized_tool_results_pure(messages)
             max_chars = Legion::Settings[:llm][:tool_result_max_dispatch_chars].to_i
             return messages unless max_chars.positive?
 
             preserve_after = last_user_message_index(messages)
-            trimmed_count = 0
-            result = messages.each_with_index.map do |msg, idx|
+            messages.each_with_index.map do |msg, idx|
               next msg if idx >= preserve_after
               next msg unless tool_result_message?(msg)
 
               content = msg[:content] || msg['content']
               next msg unless content.is_a?(String) && content.length > max_chars
 
-              trimmed_count += 1
               msg.merge(content: "#{content[0, max_chars]}\n\n[TRUNCATED: showing first #{max_chars} of #{content.length} chars. " \
                                  'If you need more content, make multiple smaller targeted requests ' \
                                  '(e.g. read specific line ranges, grep for specific patterns, or request smaller sections).]')
             end
+          end
 
+          def trim_oversized_tool_results(messages)
+            result = trim_oversized_tool_results_pure(messages)
+            trimmed_count = messages.zip(result).count { |before, after| before != after }
             if trimmed_count.positive?
               log.info "[llm][executor] action=trim_tool_results request_id=#{@request.id} trimmed=#{trimmed_count} " \
-                       "max_chars=#{max_chars} preserved_after=#{preserve_after}"
+                       "max_chars=#{Legion::Settings[:llm][:tool_result_max_dispatch_chars].to_i}"
             end
             result
           end

data/lib/legion/llm/inference/executor/routing.rb

@@ -198,18 +198,52 @@ module Legion
             Legion::Settings.dig(:llm, :routing, :allow_body_routing_hints) == true
           end
 
+          # One oracle: routing must estimate the SAME payload the dispatch guard
+          # (RouteAttempts#final_dispatch_token_estimate) measures, or a too-small
+          # lane passes the router's context-window filter then overflows at
+          # dispatch. Use ContextAccounting (walks canonical structs / content
+          # blocks) for messages, estimate the INJECTED system (baseline + RAG +
+          # skills + prior-history text), and count client tools. Messages are the
+          # lane-independent-reduced set — what actually goes on the wire.
           def estimate_request_tokens
-            all_messages = []
-            all_messages.concat(@enrichments['context:conversation_history'] || [])
-            all_messages.concat(@request.messages || [])
-
-            estimated = all_messages.empty? ? 0 : estimate_message_tokens(all_messages)
-            estimated += ((@request.system || '').length / 4.0).ceil
-            estimated += estimate_tool_token_budget if @request.tools&.any?
-            estimated += (Legion::JSON.dump(@request.thinking).length / 3.5).ceil if @request.thinking.is_a?(Hash) && @request.thinking.any?
+            # Mirror the dispatch shape EXACTLY: native_dispatch_messages sends
+            # @request.messages (lane-independent-reduced); the conversation
+            # history lives in the injected SYSTEM (via EnrichmentInjector), not
+            # in the messages array — counting it in both would double-count.
+            reduced = reduce_messages_for_dispatch(Array(@request.messages))
+
+            estimated  = ContextAccounting.estimate_message_tokens(reduced)
+            estimated += ContextAccounting.estimate_text_tokens(routing_estimate_injected_system)
+            estimated += ContextAccounting.estimate_json_tokens(routing_estimate_tool_defs)
+            estimated += ContextAccounting.estimate_json_tokens(@request.tool_choice) if @request.tool_choice
+            estimated += ContextAccounting.estimate_json_tokens(@request.thinking) if @request.thinking.is_a?(Hash) && @request.thinking.any?
             estimated
           end
 
+          # The system prompt as it will be dispatched — baseline + GAIA + prior
+          # history + RAG + skills injected. EnrichmentInjector.inject is pure, so
+          # calling it here and again at dispatch has no side effects.
+          def routing_estimate_injected_system
+            EnrichmentInjector.inject(system: @request.system, enrichments: @enrichments || {})
+          end
+
+          # The tool catalog dispatch will send (special + client + registry), NOT
+          # just @request.tools — the dispatch guard counts native_dispatch_tools,
+          # so routing must too or a heavy-tool request under-counts and picks a
+          # too-small lane. Built read-only here (never memoized) because
+          # native_tool_definitions caches and applies the local-provider cap
+          # against @resolved_provider, which is not set at routing time; an
+          # uncapped over-estimate is the safe direction for a "will it fit" filter.
+          def routing_estimate_tool_defs
+            defs = Tools::Special.pinned_definitions.map(&:to_h)
+            Array(@request.tools).each { |t| defs << (t.respond_to?(:to_h) ? t.to_h : t) }
+            Legion::Settings::Extensions.filter_tools(deferred: false).each { |e| defs << e } if registry_tool_injection_requested?
+            defs
+          rescue StandardError => e
+            handle_exception(e, level: :warn, handled: true, operation: 'llm.pipeline.routing_estimate_tool_defs')
+            Array(@request.tools).map { |t| t.respond_to?(:to_h) ? t.to_h : t }
+          end
+
           def chain_required_capabilities
             caps = []
             caps << :streaming if @request.stream == true

data/lib/legion/llm/inference/executor.rb

@@ -350,6 +350,15 @@ module Legion
             @context_accounting[:component_status][:archive] = :observed
           end
 
+          # Kill the history double-send: a client-managed client (e.g. Claude
+          # Code) resends the full conversation in @request.messages every turn,
+          # and those go on the wire as structured messages. Injecting the same
+          # turns AGAIN as "Prior conversation history" system text (via
+          # EnrichmentInjector) would send them twice — real tokens, real cost,
+          # real context pressure. Only inject the prior turns the client did NOT
+          # resend: empty for a fully client-managed turn, the whole stored prefix
+          # for a server-managed client.
+          history = reject_client_resent_history(history)
           @enrichments['context:conversation_history'] = history
           @timeline.record(
             category: :internal, key: 'context:loaded',
@@ -358,6 +367,21 @@ module Legion
           )
         end
 
+        # Drop stored-history messages the client already re-sent in
+        # @request.messages, matched by (role, content-text) fingerprint. Prevents
+        # the same turns reaching the provider twice (structured messages +
+        # injected system text). See step_context_load.
+        def reject_client_resent_history(history)
+          resent = Array(@request.messages).to_set { |m| history_dedup_key(m) }
+          Array(history).reject { |m| resent.include?(history_dedup_key(m)) }
+        end
+
+        def history_dedup_key(msg)
+          role = (msg[:role] || msg['role']).to_s
+          content = msg.is_a?(Hash) ? (msg[:content] || msg['content']) : msg
+          [role, ContextAccounting.content_text(content).to_s.strip]
+        end
+
         def maybe_compact_history(conv_id, history)
           # Guard against recursive compaction: if this thread is already compacting,
           # skip to prevent infinite loops when compressor calls back into chat_direct

data/lib/legion/llm/inference/native_tool_loop.rb

@@ -22,6 +22,18 @@ module Legion
           \s*<tool_use_value>(?<value>[^<]*)</tool_use_value>
         }mx
 
+        # Leaked chat-template tool-call token some models (e.g. gemma via vLLM)
+        # emit as literal text instead of the structured tool_calls field. One
+        # block per tool call; captured live from the ledger (2026-07):
+        #   <|tool_call>call:NAME{key:<|"|>value<|"|>,key2:<|"|>value2<|"|>}<tool_call|>
+        # LEAKED_TOKEN_RE captures NAME + the raw {..} body; LEAKED_TOKEN_ARG_RE
+        # walks the body as key:<|"|>value<|"|> pairs. The value capture is
+        # non-greedy and terminated only by the <|"|> delimiter, so embedded
+        # regular quotes and newlines (common in browser `code:` args) survive —
+        # a gsub-to-quotes + parse-as-object approach would break on those.
+        LEAKED_TOKEN_RE     = /<\|tool_call>call:(?<name>[^{]+?)(?<body>\{.*?\})<tool_call\|>/m
+        LEAKED_TOKEN_ARG_RE = /(?<key>\w+):<\|"\|>(?<value>.*?)<\|"\|>/m
+
         private
 
         def execute_native_tool_loop
@@ -337,7 +349,7 @@ module Legion
         # When the provider's response carries tool-call intent in plain text
         # rather than the structured tool_calls field, synthesize a structured
         # tool call so downstream translators emit the correct client-native
-        # shape. Three formats are recognized (in order):
+        # shape. Four formats are recognized (in order):
         #   1. forced-choice JSON args (vLLM with tool_choice forcing a tool):
         #      result text starts with `{"` and parses to a Hash.
         #   2. Qwen tag markup (no forced choice required):
@@ -346,6 +358,9 @@ module Legion
         #   3. Qwen single-tag form (live capture from qwen3.6-27b 2026-06):
         #      <TOOL>arg-string</TOOL> — only synthesizable when the named
         #      tool's schema declares a single required string param.
+        #   4. Leaked chat-template token (gemma via vLLM, live capture 2026-07):
+        #      <|tool_call>call:NAME{key:<|"|>value<|"|>,...}<tool_call|>
+        #      The named tool must be in native_dispatch_tools.
         #
         # NONDETERMINISM NOTE: qwen3.6-27b also frequently emits plain
         # narrative text (e.g. "Running `ls -la`...") with NO recoverable
@@ -364,7 +379,33 @@ module Legion
           markup = synthesize_qwen_markup_tool_call(text, round)
           return markup if markup.any?
 
-          synthesize_qwen_single_tag_tool_call(text, round)
+          single_tag = synthesize_qwen_single_tag_tool_call(text, round)
+          return single_tag if single_tag.any?
+
+          synthesize_leaked_token_tool_call(text, round)
+        end
+
+        # Pattern 4. Parse leaked chat-template tokens
+        # (<|tool_call>call:NAME{key:<|"|>value<|"|>,...}<tool_call|>) into
+        # structured tool calls. Supports multiple tokens in one blob. Only
+        # synthesizes calls whose NAME is a known native tool; unknown names are
+        # skipped (the response surfaces as text). Arguments are always strings —
+        # that is all the token encodes.
+        def synthesize_leaked_token_tool_call(text, round)
+          synthesized = text.scan(LEAKED_TOKEN_RE).filter_map do |name, body|
+            name = name.to_s.strip
+            next if name.empty? || lookup_native_tool_definition(name).nil?
+
+            arguments = body.to_s.scan(LEAKED_TOKEN_ARG_RE).each_with_object({}) do |(key, value), acc|
+              acc[key.to_s.strip] = value.to_s
+            end
+            { id: "call_#{SecureRandom.hex(10)}", name: name, arguments: arguments }
+          end
+          return [] if synthesized.empty?
+
+          log.info "[llm][native_tool_loop] action=synthesized_tool_call source=leaked_token round=#{round} " \
+                   "count=#{synthesized.size} tools=#{synthesized.map { |s| s[:name] }.join(',')}"
+          synthesized
         end
 
         def synthesize_forced_choice_tool_call(text, round)

data/lib/legion/llm/inference/steps/classification.rb

@@ -110,7 +110,11 @@ module Legion
               patterns << name if text.match?(regex)
             end
 
+            # rubocop:disable Style/ArrayIntersect -- substring check (each keyword
+            # `include?`d in the text), NOT array intersection. `intersect?` raises
+            # TypeError on the String arg.
             phi_found = PHI_KEYWORDS.any? { |kw| text.downcase.include?(kw) }
+            # rubocop:enable Style/ArrayIntersect
             patterns << :phi_keyword if phi_found
             if text.match?(EMAIL_PATTERN) && (standalone_email_pii? || phi_found || patterns.any?)
               patterns.delete(:email)

data/lib/legion/llm/inference/steps/token_budget.rb

@@ -54,22 +54,16 @@ module Legion
                   "session token budget exceeded: #{total} >= #{limit}"
           end
 
+          # One oracle: same estimator the router filter and dispatch guard use.
+          # ContextAccounting walks canonical structs / content blocks; the
+          # injected system carries baseline + history + RAG + skills.
           def estimate_input_tokens
-            content_chars = @request.messages.sum { |m| message_content_chars(m[:content]) }
             injected_system = EnrichmentInjector.inject(
               system:      @request.system,
               enrichments: @enrichments || {}
             )
-            system_chars = injected_system.to_s.length
-            (content_chars + system_chars) / 4
-          end
-
-          def message_content_chars(content)
-            return content.to_s.length unless content.is_a?(Array)
-
-            content.sum do |part|
-              part.is_a?(Hash) ? (part[:text] || part['text']).to_s.length : part.to_s.length
-            end
+            ContextAccounting.estimate_message_tokens(@request.messages) +
+              ContextAccounting.estimate_text_tokens(injected_system)
           end
         end
       end

data/lib/legion/llm/inventory.rb

@@ -206,6 +206,9 @@ module Legion
 
           # Whitelist takes precedence over blacklist (M2).  Substring match —
           # a model is denied if NO whitelist pattern is contained in its name.
+          # rubocop:disable Style/ArrayIntersect -- substring checks (patterns
+          # `include?`d in the model name), NOT array intersection; `intersect?`
+          # raises TypeError on the String arg.
           if sets[:whitelist]
             return true if sets[:whitelist].none? { |p| model.include?(p) }
 
@@ -213,6 +216,7 @@ module Legion
           end
 
           sets[:blacklist]&.any? { |p| model.include?(p) } || false
+          # rubocop:enable Style/ArrayIntersect
         end
 
         # Specificity cascade — same precedence as lex-llm Provider#model_whitelist:

data/lib/legion/llm/router.rb

@@ -209,8 +209,15 @@ module Legion
 
           return false if thinking == :require && !available.include?(:thinking)
 
+          # Apply the same headroom the dispatch budget guard uses: a lane is only
+          # eligible when estimated_context fits within context_window * headroom.
+          # Keeps routing and RouteAttempts#enforce_final_context_budget! in
+          # agreement so the router never picks a lane the pre-dispatch guard rejects.
           context_window = lane.dig(:limits, :context_window)
-          return false if estimated_context && context_window && context_window.to_i < estimated_context
+          if estimated_context && context_window
+            usable = (context_window.to_i * context_headroom).to_i
+            return false if usable < estimated_context
+          end
           return false if privacy == :strict && %i[cloud frontier].include?(lane[:tier])
 
           true
@@ -229,6 +236,18 @@ module Legion
           model
         end
 
+        # Fraction of a lane's context_window the router treats as usable when
+        # applying the estimated_context filter. Mirrors the dispatch-time budget
+        # guard so routing and dispatch agree on what fits. Clamped to (0, 1].
+        def context_headroom
+          value = Legion::Settings.dig(:llm, :routing, :context_headroom)
+          value = value.to_f
+          value.positive? && value <= 1.0 ? value : 1.0
+        rescue StandardError => e
+          handle_exception(e, level: :warn, handled: true, operation: 'router.context_headroom')
+          1.0
+        end
+
         def default_rng
           @default_rng ||= Random.new
         end

data/lib/legion/llm/settings.rb

@@ -220,9 +220,18 @@ module Legion
           # Used by Inventory.write_lane to compute lane_weight = tier_w * provider_w * instance_w * model_w * health_mult.
           tier_weights:               { direct: 105, local: 110, fleet: 110, cloud: 120, frontier: 150 },
           max_attempts:               3,
+          # Fraction of a lane's context_window the router treats as usable when
+          # applying the estimated_context hard filter. A lane is excluded when
+          # estimated_context >= context_window * context_headroom. Mirrors the
+          # dispatch-time headroom (context_curation.context_window_threshold and
+          # RouteAttempts#enforce_final_context_budget!'s 0.90) so routing and
+          # dispatch agree on what "fits" — a lane the router picks must still
+          # clear the pre-dispatch budget guard. 1.0 disables headroom.
+          context_headroom:           0.90,
           # Body-level routing hints are gated by this flag. Auto-routing aliases
           # like legionio/auto are still accepted as "you pick" intent.
           allow_body_routing_hints:   false,
+          model_passthrough_ids:      %w[copilot-utility-small],
           auto_routing_model_aliases: %w[legionio auto],
           default_intent:             { privacy: 'normal', effort: 'moderate', operation: 'chat', cost: 'normal' },
           # Last-resort fallback model when both `default_model` and the
@@ -409,6 +418,23 @@ module Legion
           superseded_eviction:      true,
           dedup_enabled:            true,
           dedup_threshold:          0.85,
+          # Strip client-harness noise that accumulates unbounded in the
+          # mid-conversation system role (Claude Code task nags, linter file
+          # dumps, etc.). See GH#168. Two mechanisms, both system-role only:
+          #   1. exact-dedup: a verbatim-duplicate system message is dropped
+          #      (the first occurrence is kept).
+          #   2. pattern strip: a system message whose content matches any
+          #      substring in harness_noise_patterns is dropped entirely.
+          # Fail-safe: an unrecognized, non-duplicate system message is never
+          # touched. The pattern list is designed to grow over time as new
+          # harness injections are identified.
+          harness_noise_strip:      true,
+          harness_noise_patterns:   [
+            # Claude Code "task tools haven't been used recently" nag.
+            "task tools haven't been used recently",
+            # Claude Code linter/formatter file-dump note.
+            'was modified, either by the user or by a linter'
+          ],
           target_context_tokens:    60_000,
           context_window_threshold: 0.90,
           archive_dropped_turns:    true,

data/lib/legion/llm/version.rb

@@ -2,6 +2,6 @@
 
 module Legion
   module LLM
-    VERSION = '0.14.10'
+    VERSION = '0.14.15'
   end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: legion-llm
 version: !ruby/object:Gem::Version
-  version: 0.14.10
+  version: 0.14.15
 platform: ruby
 authors:
 - Esity
@@ -191,8 +191,12 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '2.0'
-description: Routes LLM chat, embeddings, tool use, fleet dispatch, auditing, and
-  provider metadata for LegionIO extensions
+description: A proxy between AI clients and model backends. Requests translate once
+  to a canonical form, route to the cheapest capable tier (local, direct, fleet, cloud,
+  frontier), and escalate on failure with circuit breakers and mid-stream failover.
+  A post-turn Curator bounds long agent sessions (97.7% context reduction vs naive
+  resend at 50+ turns, production-measured). Provider adapters are separate lex-llm-*
+  gems.
 email:
 - matthewdiverson@gmail.com
 executables: []
@@ -220,6 +224,7 @@ files:
 - bin/h200-setup-prereboot.sh
 - bin/h200-setup-remaining-prereboot.sh
 - bin/h200-setup-resume-safe.sh
+- docs/curation-production-metrics.md
 - docs/work/planning/p1-results.md
 - docs/work/planning/p2-results.md
 - docs/work/planning/p3-results.md
@@ -470,5 +475,6 @@ required_rubygems_version: !ruby/object:Gem::Requirement
 requirements: []
 rubygems_version: 3.6.9
 specification_version: 4
-summary: LLM routing and provider orchestration for the LegionIO framework
+summary: 'LLM gateway: tiered routing, mid-stream failover, and context curation in
+  one proxy'
 test_files: []