lynkr 9.9.0 → 9.9.1

package/README.md

@@ -246,9 +246,9 @@ POLICY_MAX_STEPS=50
 POLICY_MAX_TOOL_CALLS=100
 ```
 
-Lynkr analyzes each request and routes it to the appropriate tier. Simple questions use fast models. Complex refactoring uses powerful models.
+Lynkr analyzes each request and routes it to the appropriate tier. Simple questions use fast models. Complex refactoring uses powerful models. The scorer combines anchor-embedding classification (WS7, payload-invariant) with an LLM difficulty classifier (Phase 6, added 2026-07-19) that catches topic-vs-difficulty confounding — `list the exports from this file` correctly routes to MEDIUM instead of getting boosted to REASONING by "technical vocabulary" alone.
 
-**Result:** 70-90% of requests use cheaper/faster models. Only hard problems hit expensive models.
+**Result:** 70-90% of requests use cheaper/faster models. Only hard problems hit expensive models. **15× reduction in expensive-tier over-routing** vs the anchor-only baseline on the 381-prompt eval set (0.6% vs ~15%).
 
 Tier configuration is strictly authoritative — bandit exploration is constrained to the models you've listed in `TIER_*`, and multi-turn conversations score with a recency-weighted sliding window so context isn't lost on short follow-ups. Conversations get a content-fingerprint session id (clients like Claude Code send none), the decision pins for the session, and a guarded escape ladder (risk keywords, force phrases, score drift, context overflow) re-escalates the moment a task outgrows its model. Full pipeline: [`docs/routing-intelligence.md`](docs/routing-intelligence.md) · intent scorer: [`docs/intent-window-routing.md`](docs/intent-window-routing.md) · verify any change: [`docs/benchmarking.md`](docs/benchmarking.md).
 
@@ -413,7 +413,7 @@ npm start
 
 Head-to-head against **LiteLLM** on the **same backends** (Ollama `minimax-m2.5`, Moonshot, Azure OpenAI), 9 scenarios across 4 feature categories. Apples-to-apples comparison is Lynkr vs LiteLLM **billed tokens on the same scenario**. Run with `node benchmark-tier-routing.js`.
 
-> _Run: June 5, 2026 · Lynkr v9.3.2 · LiteLLM v1.87.1 · macOS, Apple Silicon._
+> _Runs: token benchmarks — June 5, 2026 (Lynkr v9.3.2 · LiteLLM v1.87.1). Tier routing head-to-head — **re-run July 19, 2026** (Lynkr working tree w/ Phase-6 classifier + config B · LiteLLM v1.94.0.dev1 Auto Router v2). macOS, Apple Silicon._
 
 ### Token reduction (vs LiteLLM, same model & prompt)
 
@@ -433,28 +433,56 @@ Lynkr strips irrelevant tool schemas (smart tool selection) and binary-compresse
 
 Near-identical prompts return cached responses in 171ms. Zero model tokens billed on a cache hit.
 
-### Tier routing — vs LiteLLM Auto Router v2 (July 15, 2026)
+### Tier routing — vs LiteLLM Auto Router v2 (re-run July 19, 2026)
 
-LiteLLM v1.94 shipped a native complexity router (`auto_router/complexity_router`) with the same four tier names Lynkr uses. Head-to-head on the **same backends** with identical prompts, both proxies judged against the same acceptable-tier sets (11 routing scenarios, `MODE=routing node benchmark-tier-routing.js`, LiteLLM config in `litellm-autorouter-v2.yaml`):
+LiteLLM v1.94 shipped a native complexity router (`auto_router/complexity_router`) with the same four tier names Lynkr uses. Head-to-head on the **same backends** with identical prompts, both proxies live, both judged against the same acceptable-tier sets (11 routing scenarios, `MODE=routing node benchmark-tier-routing.js`, LiteLLM config in `litellm-autorouter-v2.yaml`):
 
-| Router | Routing-correct | Routing overhead |
+| Router | Routing-correct | Notes |
 |---|---|---|
-| **Lynkr** | **11/11** | local embedding, ~0 marginal cost/latency after cache |
-| LiteLLM v2 — heuristic (default) | 4/11 | <1ms, free — but every miss under-routed hard work (banking security analysis, autonomous agentic loop) to the free local model |
-| LiteLLM v2 — LLM classifier | 6–8/11 (non-deterministic across runs) | a paid GPT-5.2 call + ~2–3s on **every** request; fails outright with local classifier models (structured-output errors → silent heuristic fallback) |
+| **Lynkr (Phase-6 classifier + config B)** | **11/11 ✅** | anchor embedding + local LLM classifier reconcile + FORCE_REASONING patterns + risk-remap; ~500ms warm classifier call, cached |
+| LiteLLM v2 — heuristic (default) | **4/11** | every miss *under-routed* — banking security analysis, whole-pipeline refactor, prod auth-file fix, autonomous agentic loop all sent to a 7B local model |
+| LiteLLM v2 — LLM classifier | 6–8/11 (non-deterministic) | paid GPT-5.2 call + ~2–3s on **every** request; fails outright with local classifier models |
 
-Lynkr scores cleaned user-authored text against embedding anchors plus 13 weighted heuristic dimensions, detects agentic workflows, and strips harness-injected noise before scoring. LiteLLM's router reads the raw last message and has no verify-then-escalate cascade — its fallbacks trigger only on HTTP errors, never on a bad answer.
+**The 7 requests LiteLLM v2 misroutes to a 7B ollama model (should be COMPLEX or REASONING):**
 
-_Fairness notes: the 11 scenarios derive from Lynkr's own regression suite, so Lynkr has home-field advantage — the transferable finding is the direction of LiteLLM's failures (systematic under-routing on defaults; per-request cost and instability with the LLM classifier), not the exact scores. Lynkr's top tiers used Azure gpt-5.2-chat / Z.ai GLM-5.2; LiteLLM was given the identical tier targets._
+| What you typed | Lynkr sends to | LiteLLM sends to |
+|---|---|---|
+| "Analyse security trade-offs of JWT vs httpOnly cookies for a banking app" | Claude Opus (top) | **ollama 7B** ❌ |
+| "Refactor the entire ingestion pipeline and give me the plan" | z.ai GLM-5.2 (mid) | **ollama 7B** ❌ |
+| "Fix the null-check bug in `src/auth/middleware.ts`" | Claude Opus (risk-remap) | **ollama 7B** ❌ |
+| "Figure out why this test is flaky. You have full autonomy — iterate until 10 runs pass" | Claude Opus (autonomous) | **ollama 7B** ❌ |
+| "please summarize the exports of this file quickly" | ollama minimax (correct — MEDIUM) | ollama minimax — but for the wrong reason (LiteLLM under-routed to SIMPLE) |
+
+Lynkr layers the following, in decreasing priority, to catch every miss LiteLLM leaks:
+
+- FORCE_REASONING regex (`ultrathink`, `prove`, `security audit`, `from first principles`) → deterministic top-tier
+- Risk classifier (auth/middleware/credentials paths) → REASONING under config B
+- Agentic detector (`AUTONOMOUS` workflows) → REASONING minimum
+- Anchor-embedding classifier over 4 difficulty classes + FRONTIER_MIN_SIM floor
+- Local LLM classifier (qwen2.5:3b) reconciles borderline anchor calls
+- Envelope invariance — scores cleaned user text only, ignoring tool schemas / history / system-reminders
+
+LiteLLM's router reads the raw last message with no envelope-stripping, no verify-then-escalate cascade, no risk classifier, no agentic detection — its fallbacks trigger only on HTTP errors, never on a bad answer.
+
+_Fairness notes: the 11 scenarios derive from Lynkr's own regression suite, so Lynkr has home-field advantage — the transferable finding is the **direction** of LiteLLM's failures (systematic under-routing to cheap on hard prompts), not the exact scores. Lynkr's top tiers used Claude Opus 4.8 / z.ai GLM-5.2; LiteLLM was given the identical tier targets._
 
 ### Cost projection (100,000 requests/month, same backend)
 
+**Direct cost (raw meter):**
+
 | | Monthly cost | vs LiteLLM |
 |---|---|---|
-| LiteLLM | ~$818 | baseline |
-| **Lynkr** | **~$409** | **~50% cheaper** |
+| LiteLLM (TOON tool-heavy scenario) | ~$818 | baseline |
+| **Lynkr (TOON tool-heavy)** | **~$409** | **~50% cheaper** via token optimization |
+
+**Effective cost per correct-tier answer** (July 19 head-to-head, all 11 routing scenarios):
+
+| | Direct $/mo | Effective $/mo if you re-issue misroutes at correct tier |
+|---|---|---|
+| **Lynkr** | ~$5.50 (spends where it should, doesn't where it shouldn't) | ~$5.50 |
+| LiteLLM v2 heuristic | $0 (under-routes everything to ollama) | ~$89 (7 misroutes need re-issue on Moonshot/COMPLEX) |
 
-_Based on a tool-heavy agentic session (TOON scenario). On equal footing — same provider, same model — Lynkr is cheaper due to token optimization._
+The story isn't "Lynkr always spends less." It's **Lynkr spends where it should and doesn't where it shouldn't**. LiteLLM's zero-dollar cost on complex prompts is bought with wrong-tier answers to hard work — a 7B model answering "prove this rate limiter is fair under concurrent refill" is silent quality collapse, not a savings.
 
 → [Full benchmark report with methodology](BENCHMARK_REPORT.md)
 

package/bin/cli.js

@@ -14,21 +14,23 @@ const SUBCOMMANDS = {
 };
 
 const sub = process.argv[2];
-if (sub && Object.prototype.hasOwnProperty.call(SUBCOMMANDS, sub)) {
+// `lynkr start` is an alias for `lynkr` (start the proxy). Drop the token
+// and fall through so users can use whichever spelling they prefer.
+if (sub === 'start') {
+  process.argv.splice(2, 1);
+} else if (sub && Object.prototype.hasOwnProperty.call(SUBCOMMANDS, sub)) {
   process.argv.splice(2, 1); // drop the subcommand token so the script's own arg parser is happy
   // Subcommand scripts check this to decide whether to invoke their main()
   // when they're require()'d (vs being loaded by a test for unit-checking).
   process.env._LYNKR_SUBCMD = sub;
   require(SUBCOMMANDS[sub]);
   return;
-}
-
-// Unknown positional commands must error loudly, not silently boot the
-// server (`lynkr codex` used to start a bare gateway and never say why).
-if (sub && !sub.startsWith('-')) {
+} else if (sub && !sub.startsWith('-')) {
+  // Unknown positional commands must error loudly, not silently boot the
+  // server (`lynkr codex` used to start a bare gateway and never say why).
   console.error(`Error: unknown command '${sub}'.`);
   console.error(`Did you mean: lynkr wrap ${sub}`);
-  console.error(`Known commands: ${Object.keys(SUBCOMMANDS).join(', ')} (or no command to start the server)`);
+  console.error(`Known commands: ${Object.keys(SUBCOMMANDS).join(', ')}, start (or no command to start the server)`);
   process.exit(1);
 }
 
@@ -45,6 +47,8 @@ ${pkg.description}
 
 Usage:
   lynkr [options]                  Start the proxy server (default)
+  lynkr start [options]            Alias for the above
+  lynkr init [options]             Interactive setup wizard (writes .env, pulls classifier model)
   lynkr wrap <target> [options]    Wrap CLI tools through Lynkr proxy
   lynkr usage [options]            Show AI spend report and tier-routing savings
   lynkr stats [options]            Shareable savings-receipt card (also: lynkr usage --card)

package/bin/lynkr-init.js

@@ -550,6 +550,36 @@ async function runInteractive(opts) {
     close();
     console.log('');
     writeEnvFile(buildEnvContent(env, isWrap, tierConfig), opts);
+
+    // ── Classifier bootstrap: detect ollama, pull model, warm-up ──
+    // Runs AFTER .env is written so the classifier can read tier config
+    // (currently uses hardcoded qwen2.5:3b, but leaves room for the future
+    // fine-tuned model to plug in via SIMPLE tier or an env override).
+    if (!opts.dryRun) {
+      console.log('');
+      console.log('Setting up the difficulty classifier…');
+      try {
+        const { ensureClassifierReady } = require('../src/routing/classifier-setup');
+        const rl = readline.createInterface({ input: process.stdin, output: process.stdout });
+        const prompt = (q) => new Promise((res) => rl.question(q, (a) => res(a)));
+        const result = await ensureClassifierReady({
+          mode: 'interactive',
+          log: (m) => console.log(m),
+          warn: (m) => console.warn(m),
+          prompt,
+        });
+        rl.close();
+        if (result.ready) {
+          console.log('✓ Classifier ready.');
+        } else if (result.reason === 'ollama_missing') {
+          console.log('(Install Ollama and re-run `lynkr init` to enable the classifier — Lynkr will still start without it, using anchor-only scoring.)');
+        } else {
+          console.log(`(Classifier disabled: ${result.reason}. Lynkr will fall back to anchor-only scoring.)`);
+        }
+      } catch (err) {
+        console.warn(`⚠ Classifier setup skipped: ${err.message}`);
+      }
+    }
   } catch (err) {
     close();
     throw err;

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "lynkr",
-  "version": "9.9.0",
+  "version": "9.9.1",
   "files": [
     "index.js",
     "install.sh",
@@ -36,7 +36,7 @@
     "dev": "nodemon index.js",
     "lint": "eslint src index.js",
     "test": "npm run test:unit && npm run test:performance",
-    "test:unit": "DATABRICKS_API_KEY=test-key DATABRICKS_API_BASE=http://test.com LOG_FILE_ENABLED=false node --test test/routing.test.js test/hybrid-routing-integration.test.js test/web-tools.test.js test/passthrough-mode.test.js test/openrouter-error-resilience.test.js test/format-conversion.test.js test/azure-openai-config.test.js test/azure-openai-format-conversion.test.js test/azure-openai-routing.test.js test/azure-openai-streaming.test.js test/azure-openai-error-resilience.test.js test/azure-openai-integration.test.js test/openai-integration.test.js test/toon-compression.test.js test/gcf-compression.test.js test/llamacpp-integration.test.js test/resilience.test.js test/telemetry-routing.test.js test/memory/store.test.js test/memory/surprise.test.js test/memory/extractor.test.js test/memory/search.test.js test/memory/retriever.test.js test/distill.test.js test/large-payload.test.js test/code-mode.test.js test/prompt-cache-injection.test.js test/risk-analyzer.test.js test/interaction-block.test.js test/preflight.test.js test/token-reduction.test.js test/session-affinity.test.js test/model-registry-cost.test.js test/task-decomposition.test.js test/output-format-guard.test.js test/tier-fallback.test.js test/wrap.test.js test/init.test.js test/agent-learning.test.js test/tool-call-response-metadata.test.js test/degradation.test.js test/routing-telemetry-columns.test.js test/sticky-routing.test.js test/knn-ambiguous-escalate.test.js test/deescalator.test.js test/client-profiles.test.js test/strip-internal-fields.test.js test/complexity-tool-subtraction.test.js test/bandit.test.js test/routing-propensity.test.js test/reward-pipeline.test.js test/knn-cold-start.test.js test/calibration.test.js test/feedback-loop.test.js test/session-fingerprint.test.js test/side-request-guards.test.js test/verifier.test.js test/intent-score.test.js test/usage-stats.test.js test/loop-guard.test.js",
+    "test:unit": "DATABRICKS_API_KEY=test-key DATABRICKS_API_BASE=http://test.com LOG_FILE_ENABLED=false node --test test/routing.test.js test/hybrid-routing-integration.test.js test/web-tools.test.js test/passthrough-mode.test.js test/openrouter-error-resilience.test.js test/format-conversion.test.js test/azure-openai-config.test.js test/azure-openai-format-conversion.test.js test/azure-openai-routing.test.js test/azure-openai-streaming.test.js test/azure-openai-error-resilience.test.js test/azure-openai-integration.test.js test/openai-integration.test.js test/toon-compression.test.js test/gcf-compression.test.js test/llamacpp-integration.test.js test/resilience.test.js test/telemetry-routing.test.js test/memory/store.test.js test/memory/surprise.test.js test/memory/extractor.test.js test/memory/search.test.js test/memory/retriever.test.js test/distill.test.js test/large-payload.test.js test/code-mode.test.js test/prompt-cache-injection.test.js test/risk-analyzer.test.js test/interaction-block.test.js test/preflight.test.js test/token-reduction.test.js test/session-affinity.test.js test/model-registry-cost.test.js test/task-decomposition.test.js test/output-format-guard.test.js test/tier-fallback.test.js test/wrap.test.js test/init.test.js test/agent-learning.test.js test/tool-call-response-metadata.test.js test/degradation.test.js test/routing-telemetry-columns.test.js test/sticky-routing.test.js test/knn-ambiguous-escalate.test.js test/deescalator.test.js test/client-profiles.test.js test/strip-internal-fields.test.js test/complexity-tool-subtraction.test.js test/bandit.test.js test/routing-propensity.test.js test/reward-pipeline.test.js test/knn-cold-start.test.js test/calibration.test.js test/feedback-loop.test.js test/session-fingerprint.test.js test/side-request-guards.test.js test/verifier.test.js test/intent-score.test.js test/difficulty-classifier.test.js test/classifier-setup.test.js test/usage-stats.test.js test/loop-guard.test.js",
     "test:memory": "DATABRICKS_API_KEY=test-key DATABRICKS_API_BASE=http://test.com node --test test/memory/store.test.js test/memory/surprise.test.js test/memory/extractor.test.js test/memory/search.test.js test/memory/retriever.test.js",
     "test:new-features": "DATABRICKS_API_KEY=test-key DATABRICKS_API_BASE=http://test.com node --test test/passthrough-mode.test.js test/openrouter-error-resilience.test.js test/format-conversion.test.js",
     "test:performance": "DATABRICKS_API_KEY=test-key DATABRICKS_API_BASE=http://test.com node test/hybrid-routing-performance.test.js && DATABRICKS_API_KEY=test-key DATABRICKS_API_BASE=http://test.com node test/performance-tests.js",
@@ -124,4 +124,4 @@
     "nodemon": "^3.1.0",
     "pino-pretty": "^10.2.0"
   }
-}
+}

package/scripts/build-eval-set.js

@@ -0,0 +1,256 @@
+#!/usr/bin/env node
+/**
+ * Build the difficulty-classifier evaluation set.
+ *
+ * Sources (all zero-cost, no LLM judge required):
+ *  1. RouterArena unused rows — native empirical difficulty labels
+ *     (easy/medium/hard from 42-model pass rate). No LLM involved.
+ *  2. routellm/gpt4_dataset unused rows — native mixtral_score 1–5
+ *     (GPT-4-judged when the dataset was built). No LLM involved.
+ *  3. Hand-crafted coding-agent-flavored prompts — representative of
+ *     Lynkr's real traffic (real logs proved too thin, ~7 unique prompts).
+ *
+ * Leak avoidance: skip any text already in data/difficulty-anchors.provenance.json
+ * (the mined anchor set) so eval and anchors don't overlap.
+ *
+ * Label → tier mapping:
+ *   RouterArena easy   → MEDIUM  (any competent model handles)
+ *   RouterArena medium → COMPLEX (needs strong general model)
+ *   RouterArena hard   → REASONING (frontier model territory)
+ *   gpt4_dataset score 5      → MEDIUM   (Mixtral got it right — trivially easy)
+ *   gpt4_dataset score 3–4    → COMPLEX  (Mixtral partial — real difficulty)
+ *   gpt4_dataset score 1–2    → REASONING (Mixtral failed — top-tier work)
+ *
+ * Output: data/difficulty-eval.jsonl (gitignored)
+ * Usage: node scripts/build-eval-set.js
+ */
+
+const fs = require("fs");
+const path = require("path");
+
+const DS_SERVER = "https://datasets-server.huggingface.co/rows";
+const OUT_FILE = path.join(__dirname, "../data/difficulty-eval.jsonl");
+const PROVENANCE = path.join(__dirname, "../data/difficulty-anchors.provenance.json");
+
+const TARGETS = { routerarena: 200, gpt4_dataset: 200 };
+
+function sleep(ms) { return new Promise(r => setTimeout(r, ms)); }
+
+async function fetchRows(dataset, config, split, offset, length) {
+  const url = `${DS_SERVER}?dataset=${encodeURIComponent(dataset)}&config=${config}&split=${split}&offset=${offset}&length=${length}`;
+  for (let a = 0; a < 5; a++) {
+    try {
+      const res = await fetch(url);
+      if (res.status === 429) { await sleep(3000 * (a + 1)); continue; }
+      if (!res.ok) throw new Error(`HTTP ${res.status}`);
+      return (await res.json()).rows?.map(r => r.row) || [];
+    } catch (err) {
+      if (a === 4) return [];
+      await sleep(1500 * (a + 1));
+    }
+  }
+  return [];
+}
+
+function clean(text) {
+  if (typeof text !== "string") return null;
+  const t = text.replace(/\s+/g, " ").trim();
+  if (t.length < 15 || t.length > 400) return null;
+  if (/\(\s*\)/.test(t) || /_{3,}/.test(t)) return null;
+  if (/\b[A-D]\)\s/.test(t) || /which of the following/i.test(t)) return null;
+  if (/\b(premise|hypothesis)\b/i.test(t) && /\b(entail|inference|contradict)\b/i.test(t)) return null;
+  if (/NAME_\d|\{\{.*\}\}/.test(t)) return null;
+  const nonAscii = (t.match(/[^\x20-\x7E]/g) || []).length;
+  if (nonAscii > t.length * 0.1) return null;
+  return t;
+}
+
+function isQuizbowlStyle(t) {
+  return /^(this |one |a \d{4} (book|work|paper)|fictional |in one (work|section))/i.test(t) ||
+    /\([A-Za-z]+-[A-Z][A-Za-z-]+\)/.test(t);
+}
+
+// Hand-crafted coding-agent-flavored prompts by tier. Reflects real Lynkr
+// traffic shape (imperatives, technical vocabulary, mix of simple/mechanical/
+// systemic/rigorous). Labels are mine — the honest reviewer for this session.
+const HAND_EVAL = [
+  // SIMPLE — casual/acks
+  { text: "hi", tier: "SIMPLE" },
+  { text: "hello", tier: "SIMPLE" },
+  { text: "ok thanks", tier: "SIMPLE" },
+  { text: "yes continue", tier: "SIMPLE" },
+  { text: "no skip that", tier: "SIMPLE" },
+  { text: "got it", tier: "SIMPLE" },
+  { text: "sure go ahead", tier: "SIMPLE" },
+  { text: "what time is it", tier: "SIMPLE" },
+  { text: "cool", tier: "SIMPLE" },
+  { text: "makes sense", tier: "SIMPLE" },
+  { text: "thanks a lot", tier: "SIMPLE" },
+  { text: "hey there", tier: "SIMPLE" },
+  { text: "yep proceed", tier: "SIMPLE" },
+  { text: "great work", tier: "SIMPLE" },
+  { text: "no worries", tier: "SIMPLE" },
+  // MEDIUM — one specific mechanical task
+  { text: "list the exports from src/router.js", tier: "MEDIUM" },
+  { text: "run the unit tests and summarize failures", tier: "MEDIUM" },
+  { text: "show me the git diff for this file", tier: "MEDIUM" },
+  { text: "what does this function do", tier: "MEDIUM" },
+  { text: "explain this regex pattern to me", tier: "MEDIUM" },
+  { text: "add error handling to this try block", tier: "MEDIUM" },
+  { text: "fix the linter warnings in this file", tier: "MEDIUM" },
+  { text: "write a unit test for the login helper", tier: "MEDIUM" },
+  { text: "convert this callback to async/await", tier: "MEDIUM" },
+  { text: "add a null check before this dereference", tier: "MEDIUM" },
+  { text: "rename this variable to something more descriptive", tier: "MEDIUM" },
+  { text: "extract this into a helper function", tier: "MEDIUM" },
+  { text: "add a docstring to this method", tier: "MEDIUM" },
+  { text: "check if this file has any TODO comments", tier: "MEDIUM" },
+  { text: "search the codebase for uses of deprecated API", tier: "MEDIUM" },
+  { text: "install the axios package as a dev dependency", tier: "MEDIUM" },
+  { text: "revert the last commit but keep the working tree", tier: "MEDIUM" },
+  { text: "delete unused imports from this file", tier: "MEDIUM" },
+  { text: "format this JSON blob nicely", tier: "MEDIUM" },
+  { text: "which files were changed in the last commit", tier: "MEDIUM" },
+  { text: "how do I use the fetch API with timeouts", tier: "MEDIUM" },
+  { text: "what's the difference between let and const in JavaScript", tier: "MEDIUM" },
+  { text: "add a rate limit to this endpoint", tier: "MEDIUM" },
+  { text: "grep for TODO in the routing folder", tier: "MEDIUM" },
+  { text: "make this loop use Promise.all instead of sequential awaits", tier: "MEDIUM" },
+  // COMPLEX — systemic / design / multi-file
+  { text: "do an architecture review of the orchestrator", tier: "COMPLEX" },
+  { text: "review this retry helper for bugs and race conditions", tier: "COMPLEX" },
+  { text: "refactor the entire ingestion pipeline and give me a plan", tier: "COMPLEX" },
+  { text: "design a horizontally scalable architecture for the router with failure-mode analysis", tier: "COMPLEX" },
+  { text: "code review the PR #84 routing hardening changes", tier: "COMPLEX" },
+  { text: "analyze every module in src/ for circular dependencies", tier: "COMPLEX" },
+  { text: "debug this complex race condition in the connection pool", tier: "COMPLEX" },
+  { text: "plan a zero-downtime migration to the new schema", tier: "COMPLEX" },
+  { text: "implement a distributed rate limiter with Redis backing", tier: "COMPLEX" },
+  { text: "design the caching strategy for this API gateway including invalidation rules", tier: "COMPLEX" },
+  { text: "trace through the request lifecycle end to end and identify bottlenecks", tier: "COMPLEX" },
+  { text: "restructure the module boundaries to reduce coupling between core and plugins", tier: "COMPLEX" },
+  { text: "propose a testing strategy for the new streaming pipeline covering edge cases", tier: "COMPLEX" },
+  { text: "identify all the places in the codebase that depend on the legacy auth flow and plan the deprecation", tier: "COMPLEX" },
+  { text: "walk me through how the tier routing decision cascade works and highlight the weak points", tier: "COMPLEX" },
+  { text: "compare our current queueing implementation against BullMQ and recommend a migration path", tier: "COMPLEX" },
+  { text: "extract the shared session logic into a separate package and update all consumers", tier: "COMPLEX" },
+  { text: "design the observability layer for the multi-tenant deployment with per-tenant metrics", tier: "COMPLEX" },
+  { text: "diagnose why the latency P99 spiked yesterday afternoon across the fleet", tier: "COMPLEX" },
+  { text: "plan the sharding strategy for the telemetry database as we scale to 10x traffic", tier: "COMPLEX" },
+  { text: "audit the codebase for uses of unsafe eval and propose safe replacements", tier: "COMPLEX" },
+  { text: "propose a graceful degradation plan for when the primary embedding model is unavailable", tier: "COMPLEX" },
+  { text: "review the error handling across the client SDK and standardize on a single pattern", tier: "COMPLEX" },
+  { text: "outline the migration from the monolith to a services split with backwards compatibility", tier: "COMPLEX" },
+  { text: "reason about whether this cache invalidation scheme is correct under concurrent writes", tier: "COMPLEX" },
+  // REASONING — proof / audit / formal / deep
+  { text: "prove the correctness of this lock-free queue implementation and identify any ABA hazards", tier: "REASONING" },
+  { text: "security audit the authentication middleware for token reuse vulnerabilities", tier: "REASONING" },
+  { text: "derive the optimal cache eviction policy for this access pattern and prove its competitive ratio", tier: "REASONING" },
+  { text: "formally verify that this state machine can never deadlock or produce a counterexample trace", tier: "REASONING" },
+  { text: "from first principles design a Byzantine-fault-tolerant consensus variant tolerating f faults with 2f+1 replicas", tier: "REASONING" },
+  { text: "think deeply about why this floating-point summation loses precision at scale and derive the compensated algorithm", tier: "REASONING" },
+  { text: "prove this rate limiter is fair under concurrent refill or construct the starvation schedule that breaks it", tier: "REASONING" },
+  { text: "reason through the exact memory ordering constraints this concurrent hashmap needs on ARM and justify each barrier", tier: "REASONING" },
+  { text: "given these heap dumps reason to the retention path causing the leak and rule out the two most plausible alternatives", tier: "REASONING" },
+  { text: "ultrathink: analyze whether this online schema migration can preserve every invariant the application relies on", tier: "REASONING" },
+  { text: "verify no path in this state machine violates the safety property that read never observes a partial write", tier: "REASONING" },
+  { text: "derive the amortized complexity bound for this splay tree under this adversarial access pattern and prove it tight", tier: "REASONING" },
+  { text: "prove that this distributed algorithm makes progress under fair scheduling regardless of message reordering", tier: "REASONING" },
+  { text: "penetration test the session token handling and enumerate every replay attack vector", tier: "REASONING" },
+  { text: "formal proof that this compensation transaction preserves ACID under partial-failure scenarios", tier: "REASONING" },
+  { text: "reason from first principles about whether exactly-once semantics are achievable on top of this at-least-once queue", tier: "REASONING" },
+  { text: "vulnerability scan the recently-added crypto path for downgrade attacks and misuse of primitives", tier: "REASONING" },
+  { text: "prove that this garbage collector will always terminate on any input given the described root set", tier: "REASONING" },
+  { text: "verify the linearizability of this concurrent skip list using the linearization-point argument", tier: "REASONING" },
+  { text: "think hard about which of these three algorithms actually preserves invariant I under crash recovery and which only appears to", tier: "REASONING" },
+];
+
+async function main() {
+  const seenTexts = new Set();
+  try {
+    const prov = JSON.parse(fs.readFileSync(PROVENANCE, "utf8"));
+    for (const text of Object.keys(prov)) seenTexts.add(text);
+    console.log(`Loaded ${seenTexts.size} anchor texts from provenance (excluded from eval)`);
+  } catch { /* first run may have no provenance */ }
+
+  const eval_ = [];
+  const seen = new Set([...seenTexts].map(t => t.slice(0, 80)));
+
+  const add = (text, tier, source, label) => {
+    const key = text.slice(0, 80);
+    if (seen.has(key)) return false;
+    seen.add(key);
+    eval_.push({ text, tier, source, label });
+    return true;
+  };
+
+  // 1. Hand-crafted prompts (canary + real-traffic representative)
+  for (const item of HAND_EVAL) {
+    add(item.text, item.tier, "hand", "author-labeled");
+  }
+  console.log(`Hand-crafted: ${eval_.length} added`);
+
+  // 2. RouterArena — technical domains only, empirical difficulty labels
+  console.log("Fetching RouterArena rows...");
+  const TECH_DOMAIN_RE = /computer science|technology|engineering|mathematic|science/i;
+  const DATASET_EXCLUDE_RE = /superglue|cloze|entailment/i;
+  let raAdded = 0;
+  outer: for (let offset = 4000; offset < 8400; offset += 100) {
+    const rows = await fetchRows("RouteWorks/RouterArena", "default", "full", offset, 100);
+    for (const row of rows) {
+      if (raAdded >= TARGETS.routerarena) break outer;
+      if (row.Context && String(row.Context).trim()) continue;
+      if (Array.isArray(row.Options) && row.Options.length > 0) continue;
+      if (DATASET_EXCLUDE_RE.test(String(row["Dataset name"] || ""))) continue;
+      const text = clean(row.Question);
+      if (!text || isQuizbowlStyle(text)) continue;
+      const technical = TECH_DOMAIN_RE.test(String(row.Domain || ""));
+      const diff = String(row.Difficulty || "").toLowerCase();
+      let tier = null;
+      if (diff === "easy") tier = "MEDIUM";
+      else if (diff === "medium" && technical) tier = "COMPLEX";
+      else if (diff === "hard" && technical) tier = "REASONING";
+      if (!tier) continue;
+      if (add(text, tier, "RouterArena", `${diff}/${row["Dataset name"]}`)) raAdded++;
+    }
+    await sleep(200);
+  }
+  console.log(`RouterArena: +${raAdded}`);
+
+  // 3. gpt4_dataset — skip pages we mined for anchors (offset 0, 1817, 3634...)
+  console.log("Fetching gpt4_dataset rows...");
+  let gpAdded = 0;
+  outer2: for (let offset = 60000; offset < 109000; offset += 500) {
+    const rows = await fetchRows("routellm/gpt4_dataset", "default", "train", offset, 100);
+    for (const row of rows) {
+      if (gpAdded >= TARGETS.gpt4_dataset) break outer2;
+      const text = clean(row.prompt);
+      if (!text) continue;
+      const score = Number(row.mixtral_score);
+      let tier = null;
+      if (score === 5) tier = "MEDIUM";
+      else if (score === 3 || score === 4) tier = "COMPLEX";
+      else if (score === 1 || score === 2) tier = text.length < 60 ? null : "REASONING";
+      if (!tier) continue;
+      if (add(text, tier, "gpt4_dataset", `mixtral_score=${score}`)) gpAdded++;
+    }
+    await sleep(200);
+  }
+  console.log(`gpt4_dataset: +${gpAdded}`);
+
+  // Shuffle deterministically (session-repro via fixed permutation)
+  eval_.sort((a, b) => a.text.length - b.text.length);
+  const shuffled = [];
+  for (let i = 0; i < eval_.length; i++) {
+    const j = (i * 37 + 11) % eval_.length;
+    shuffled.push(eval_[j]);
+  }
+
+  fs.writeFileSync(OUT_FILE, shuffled.map(r => JSON.stringify(r)).join("\n") + "\n");
+  console.log(`\nWrote ${OUT_FILE}: ${shuffled.length} rows`);
+  const perTier = {};
+  for (const r of shuffled) perTier[r.tier] = (perTier[r.tier] || 0) + 1;
+  console.log("Per-tier:", JSON.stringify(perTier));
+}
+
+main().catch(err => { console.error(err); process.exit(1); });