@tangle-network/agent-knowledge 1.9.0 → 1.11.0

package/docs/eval/investment-material-facts.md

@@ -0,0 +1,252 @@
+# Held-out investment-research eval set — material facts + provenance
+
+This is the answer key and the provenance ledger for `tests/eval/investment-thesis-set.ts`.
+
+**What the set measures.** Give a research loop a company + ticker + an as-of
+**cutoff** date and ask it to write an investment thesis. Then grade that thesis
+against the held-out **material facts** below — facts the loop never saw. A high
+score means the thesis surfaced the buried, material, non-obvious drivers a
+thorough analyst would flag and a single ticker search would miss; it is **not**
+teaching-to-the-test, because the answer key is firewalled from every loop and
+the grader is a `$0`, model-free substring check (`gradeFactAgainstText`).
+
+**Three hard rules, enforced by how the data was gathered:**
+
+1. **Specific + checkable.** Every fact carries keyword groups (a number, a name,
+   a phrase) so the deterministic grader can score "did the thesis surface it".
+2. **Derived from real fetched evidence.** Every fact cites the primary SEC EDGAR
+   10-K it was read from and the literal value in that document. Nothing is
+   invented; an item that could not be independently sourced was **dropped**, not
+   guessed (see the drop log).
+3. **Knowable at the cutoff.** Every value was disclosed in, or computable from, a
+   filing available on or before the cutoff. The eventual collapse is **not** a
+   checklist item — it is recorded as `knownOutcome`, for the reader only, and is
+   never graded.
+
+All five primary documents were fetched live from `https://www.sec.gov/Archives/`
+during curation (a `curl` with a descriptive `User-Agent`, per SEC fair-access
+rules). Every dollar figure below was read directly out of the de-tagged filing
+text. Provenance is verifiable: each `sourceUrl` contains the company's SEC CIK,
+and `tests/eval/investment-thesis-set.test.ts` asserts that invariant.
+
+---
+
+## Companies + cutoffs
+
+| Ticker | Company | CIK | Cutoff (as-of) | Sector | Primary source (10-K) |
+|---|---|---|---|---|---|
+| SIVB | SVB Financial Group | 719739 | 2023-02-24 | Banking | FY2022 10-K, filed 2023-02-24 |
+| BBBY | Bed Bath & Beyond Inc. | 886158 | 2022-04-21 | Specialty retail | FY2021 10-K, filed 2022-04-21 |
+| CVNA | Carvana Co. | 1690820 | 2023-02-23 | Auto e-commerce | FY2022 10-K, filed 2023-02-23 |
+| PTON | Peloton Interactive, Inc. | 1639825 | 2022-09-07 | Consumer fitness hardware | FY2022 10-K, filed 2022-09-07 |
+| SI | Silvergate Capital Corporation | 1312109 | 2022-02-28 | Banking (digital-asset) | FY2021 10-K, filed 2022-02-28 |
+
+Each cutoff is set to the filing date of the primary 10-K, so the entire document
+was public on the as-of date. All five cutoffs are **>= 18 months** before this
+set was curated (June 2026); `investment-thesis-set.test.ts` asserts this.
+
+---
+
+## SIVB — SVB Financial Group (cutoff 2023-02-24)
+
+Source: [FY2022 10-K (`sivb-20221231.htm`)](https://www.sec.gov/Archives/edgar/data/719739/000071973923000021/sivb-20221231.htm)
+**Known outcome (not graded):** FDIC receivership March 10, 2023; holding-company Chapter 11 March 17, 2023.
+
+| ID | Lens | Material fact | Value read from the filing |
+|---|---|---|---|
+| SIVB/f1 | off-balance-sheet | HTM securities carried at amortized cost were far above fair value | "Held-to-maturity securities, at amortized cost ... **91,321** (fair value of $ **76,169**)" → ~**$15.15B** unrealized loss, footnote-only |
+| SIVB/f2 | off-balance-sheet | That HTM loss ~= the entire equity base | "Total SVBFG stockholders' equity **16,004**" → ~$15.15B is ~95% of $16.0B equity |
+| SIVB/f3 | concentration | Run-prone uninsured deposit base | "estimated uninsured deposits in U.S. offices that exceed the FDIC insurance limit were **$151.5 billion**" |
+| SIVB/f4 | margin-trend | Cheap deposits fleeing → funding cost set to rise | "Noninterest-bearing demand deposits to total deposits decreased by **20 percentage points to 47 percent**" |
+| SIVB/f5 | concentration | Single-client-type (innovation-economy) deposit + credit base | 10-K frames the franchise around "the innovation economy" (technology, life-science, venture) |
+| SIVB/f6 | off-balance-sheet | AFS loss in AOCI — the visible, smaller tip | "Available-for-sale securities, at fair value (cost of $ **28,602**) **26,069**" → ~$2.5B AFS loss in AOCI |
+
+The decisive, non-obvious fact is SIVB/f1+f2: an interest-rate loss roughly equal
+to all of equity, sitting in the footnotes because HTM accounting keeps it out of
+both earnings and book equity. A ticker search shows a profitable bank; the
+filing shows a mark-to-market hole the size of its capital.
+
+## BBBY — Bed Bath & Beyond Inc. (cutoff 2022-04-21)
+
+Source: [FY2021 10-K (`bbby-20220226.htm`)](https://www.sec.gov/Archives/edgar/data/886158/000088615822000047/bbby-20220226.htm)
+**Known outcome (not graded):** Chapter 11 on April 23, 2023; equity wiped out.
+
+| ID | Lens | Material fact | Value read from the filing |
+|---|---|---|---|
+| BBBY/f1 | capital-return | Buybacks drained a loss-making balance sheet | "we have repurchased approximately **$11.685 billion**"; FY2021 alone "**$574.9 million** ... two years ahead of schedule" |
+| BBBY/f2 | liquidity | Operating cash flow nearly vanished | "Net cash provided by operating activities **17,854** 268,108 590,941" ($ thousands) |
+| BBBY/f3 | liquidity | Equity collapsed ~86% in one year | "Total shareholders' equity **174,145** 1,276,936" ($ thousands) |
+| BBBY/f4 | liquidity | A net loss the same year it kept buying back stock | "Net loss $ (**559,623**)" ($ thousands) |
+| BBBY/f5 | margin-trend | Inventory building into a demand decline | "Merchandise inventories **1,725,410** 1,671,909" ($ thousands) |
+
+The non-obvious fact is BBBY/f1+f2+f4 together: in FY2021 the company **lost $560M,
+generated only $18M of operating cash, and still spent $575M buying back stock** —
+returning more cash than it had. The buyback, not the income statement alone, is
+why a $1.3B equity base became $174M.
+
+## CVNA — Carvana Co. (cutoff 2023-02-23)
+
+Source: [FY2022 10-K (`cvna-20221231.htm`)](https://www.sec.gov/Archives/edgar/data/1690820/000169082023000052/cvna-20221231.htm)
+**Known outcome (not graded):** Stock fell ~98% from its 2021 peak; a 2023 debt exchange cut and extended obligations, narrowly avoiding bankruptcy.
+
+| ID | Lens | Material fact | Value read from the filing |
+|---|---|---|---|
+| CVNA/f1 | leverage | A debt load far above the equity base | "Total debt **8,391** 5,447" ($ millions) |
+| CVNA/f2 | leverage | Interest expense nearly tripled in a year | "Interest expense **486** 176" ($ millions) |
+| CVNA/f3 | leverage | A ~$2.2B debt-funded acquisition as demand turned | "physical auction business of ADESA US Auction, LLC for approximately **$2.2 billion** in cash" (closed 2022-05-09) |
+| CVNA/f4 | governance | Recurring related-party leases with the founder's family | Related-Party note: DriveTime, controlled by "Ernest Garcia II, Ernest Garcia III, and entities controlled by one or both of them" |
+| CVNA/f5 | liquidity | A wide loss showing unit economics had not turned | "Net loss $ (**2,894**)" ($ millions) |
+
+The non-obvious facts are CVNA/f2 (interest expense up 2.8x — the debt was now
+expensive, not just large) and CVNA/f4 (the controlling Garcia family on both
+sides of material leases via DriveTime), neither of which a ticker quote shows.
+
+## PTON — Peloton Interactive, Inc. (cutoff 2022-09-07)
+
+Source: [FY2022 10-K (`pton-20220630.htm`)](https://www.sec.gov/Archives/edgar/data/1639825/000163982522000117/pton-20220630.htm)
+**Known outcome (not graded):** Stock fell ~95% from its 2021 peak; founder-CEO departed; mass layoffs and a multi-year turnaround.
+
+| ID | Lens | Material fact | Value read from the filing |
+|---|---|---|---|
+| PTON/f1 | margin-trend | Hardware gross margin turned **negative** | Connected Fitness "Gross Margin decreased to (**11**)" percent — losing money per unit sold |
+| PTON/f2 | liquidity | Inventory glut as pandemic demand normalized | "Inventories, net **1,104.5** 937" ($ millions) |
+| PTON/f3 | governance | Dual-class super-voting control | "Class B common stock has **20 votes per share** and our Class A common stock has one vote per share" |
+| PTON/f4 | liquidity | An order-of-magnitude wider loss | "Net loss $ (**2,827**)" ($ millions) |
+| PTON/f5 | regulatory | An open CPSC product-safety recall | "recall on **Tread+** ... in collaboration with the **Consumer Product Safety Commission ('CPSC')**" |
+| PTON/f6 | leverage | Locked-in purchase commitments into falling demand | "purchase commitments related to the manufacture of Peloton products were estimated to be approximately **$334**" million |
+
+The non-obvious fact is PTON/f1: revenue was still large, but the **hardware was
+sold below cost** (−11% gross margin) — the unit economics, not just the growth
+rate, had broken. PTON/f6 compounds it: the company was contractually obliged to
+buy more inventory it could not sell.
+
+## SI — Silvergate Capital Corporation (cutoff 2022-02-28)
+
+Source: [FY2021 10-K (`si-20211231.htm`)](https://www.sec.gov/Archives/edgar/data/1312109/000131210922000051/si-20211231.htm)
+**Known outcome (not graded):** After the FTX collapse triggered a deposit run, Silvergate announced a voluntary wind-down and liquidation of Silvergate Bank in March 2023.
+
+| ID | Lens | Material fact | Value read from the filing |
+|---|---|---|---|
+| SI/f1 | concentration | Essentially all funding was on-demand money | "noninterest bearing deposits as a percentage of total deposits was **99.5%** as of December 31, 2021" |
+| SI/f2 | concentration | Deposits dominated by crypto exchanges | "Deposits from digital currency exchanges represent approximately **58%**" |
+| SI/f3 | concentration | The whole deposit base tied to one volatile industry | Strategy + risk factors center on "digital currency customers" and "the concentration of our deposits" |
+| SI/f4 | liquidity | Hot-money deposit growth that could reverse fast | "Total deposits $ **14,290,628**" vs prior year "$ 10,411,278" ($ thousands) |
+| SI/f5 | concentration | The moat AND the funding are the same crypto-only bet | "Silvergate Exchange Network ('SEN'), our proprietary ... payment network for participants in the digital currency industry" |
+
+The non-obvious fact is SI/f1+f2 together: **99.5% noninterest-bearing deposits,
+~58% from crypto exchanges** — a funding base with no contractual term and a
+single correlated counterparty type. A ticker quote shows a fast-growing,
+low-cost-funding bank; the filing shows a bank that could be emptied in days if
+crypto sentiment turned.
+
+---
+
+## Curation-bias disclosure (read this before trusting a score on this set)
+
+This set is **not** a representative sample of public companies, and a score on it
+should not be read as a general "investment-research quality" number. The biases:
+
+1. **Survivorship / outcome selection.** All five companies are ones whose buried
+   risks later **materialized** (four failed or near-failed; one was forced into a
+   multi-year turnaround). They were chosen partly because that is where the
+   material-vs-surface distinction is sharpest, and partly because the figures are
+   easy to verify after the fact. A set built only on known blow-ups will reward a
+   loop that pattern-matches "find the downside" and will not test whether a loop
+   can surface a buried **positive** driver or correctly conclude a company is
+   sound. A production set must add survivors and upside cases.
+
+2. **Lens skew toward downside risk.** The 27 facts are not evenly spread across
+   the eight analyst lenses. Measured by `lensDistribution()`:
+
+   | Lens | Facts |
+   |---|---|
+   | liquidity | 7 |
+   | concentration | 6 |
+   | leverage | 4 |
+   | off-balance-sheet | 3 |
+   | margin-trend | 3 |
+   | governance | 2 |
+   | capital-return | 1 |
+   | regulatory | 1 |
+
+   Liquidity / concentration / leverage dominate (17 of 27). Governance,
+   capital-return, and regulatory are thin. The set therefore tests "can you find
+   the cash/funding/debt hole" much more than "can you find the governance or
+   regulatory landmine."
+
+3. **Sector skew toward financials + distressed consumer.** Two of five are banks
+   (SIVB, SI). The two banks are deliberately given **different** buried-risk
+   lenses — SIVB is an interest-rate / duration / off-balance-sheet story, SI is a
+   single-industry deposit-concentration story — so they are not redundant, but
+   the set still over-indexes on balance-sheet fragility and under-tests, e.g.,
+   technology platform risk, supply-chain concentration, or accounting-policy
+   aggressiveness in a healthy grower.
+
+4. **Grader leniency vs strictness is a knob, not a truth.** The grader counts a
+   fact "surfaced" on a case-insensitive substring of any synonym in a group. This
+   can **over-credit** a thesis that name-drops a number without understanding it
+   (e.g. mentions "$151.5 billion" in passing), and can **under-credit** a thesis
+   that explains the risk in numbers the grader did not anticipate. The synonym
+   groups were written to be faithful, but they are a human artifact; treat the
+   absolute score as ordinal (arm A vs arm B), not cardinal.
+
+5. **Single-source provenance.** Every fact is sourced to the company's own 10-K.
+   That makes provenance clean and checkable, but it means the set tests "did you
+   read the filing", not "did you triangulate the filing against an independent
+   source" (analyst reports, court filings, short-seller research). A fact that
+   required a second independent source to establish was dropped (below) rather
+   than sourced to the filing alone.
+
+## Drop log — items considered and NOT included (honesty over coverage)
+
+These were candidate facts I could not independently ground to a primary source
+available at the cutoff, so I **dropped them rather than guess**:
+
+- **First Republic Bank (FRC)** — dropped as a company entirely. First Republic
+  was a state-chartered bank that filed its annual reports with the **FDIC**, not
+  on SEC EDGAR, so its 10-K is not at a `sec.gov/Archives` URL and I could not give
+  it the same clean, CIK-verifiable provenance as the other five. Its widely cited
+  figures (~$15B HTM-style loss, ~$119.5B uninsured deposits) are real but are best
+  sourced from the FDIC OIG Material Loss Review, a **post**-cutoff document — using
+  it would violate rule 3. Replaced with Silvergate, whose 10-K is on EDGAR.
+
+- **SIVB exact total unrealized-loss footnote line** — I report the HTM gap as the
+  arithmetic difference of two figures printed on the balance sheet
+  ($91,321M − $76,169M), which is exact and on-cutoff. I did **not** include a
+  separately-quoted "$15.1B net unrealized loss" sentence because I did not locate
+  that exact phrasing in the de-tagged text; quoting a number I could not point to
+  verbatim would break rule 2. The computed value is conservative and checkable.
+
+- **CVNA negative gross-profit-per-unit** — a frequently-cited Carvana red flag,
+  but the per-unit figure I could find cleanly was a derived/analyst number, not a
+  single line item I could quote verbatim from the 10-K at the cutoff. Dropped in
+  favor of the directly-quoted total debt, interest expense, ADESA price, related
+  party, and net loss.
+
+- **PTON / SI specific debt-covenant or going-concern language** — I searched for
+  explicit "substantial doubt / going concern" wording in both filings and did
+  **not** find it at these cutoffs (it came later). I did not invent it. The facts
+  included are the ones actually present in the cutoff-date document.
+
+## How a loop is graded against this set
+
+```ts
+import {
+  investmentThesisSet,
+  gradeCompanyAgainstText,
+  totalMaterialFacts,
+} from '../../tests/eval/investment-thesis-set'
+
+// For each company, the loop writes a thesis BLIND (it sees only company +
+// ticker + cutoff, never the facts above). Then:
+for (const company of investmentThesisSet) {
+  const thesisText = /* the loop's full thesis for this company */ ''
+  const { surfaced, total } = gradeCompanyAgainstText(company, thesisText)
+  // surfaced / total = held-out material facts this thesis caught.
+}
+// totalMaterialFacts() = 27 is the denominator across the whole set.
+```
+
+The grader is deterministic and model-free, so the same thesis always scores the
+same and the answer key never reaches a model the loop could observe — the same
+firewall the deep-question exam (`tests/loops/held-out-exam.ts`) uses.

package/docs/eval/rag-eval-roadmap.md

@@ -0,0 +1,119 @@
+# RAG Eval Completion Roadmap
+
+Verdict: `runRetrievalImprovementLoop()` is the right first loop, but it is only the retrieval layer.
+SOTA RAG evaluation requires retrieval quality, context quality, generated-answer quality, abstention behavior, robustness, and operating budgets.
+
+## Research Basis
+
+| Source | What matters for us |
+| --- | --- |
+| [Ragas](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/) / [paper](https://arxiv.org/abs/2309.15217) | Standard RAG eval splits retrieval and generation into context precision, context recall, faithfulness, and answer relevance. |
+| [ARES](https://arxiv.org/abs/2311.09476) | Strong RAG eval scores context relevance, answer faithfulness, and answer relevance, and uses small human-labeled sets to calibrate automated judges. |
+| [TruLens RAG Triad](https://www.trulens.org/getting_started/core_concepts/rag_triad/) | The minimal end-to-end triad is context relevance, groundedness, and answer relevance. |
+| [RAGChecker](https://papers.nips.cc/paper_files/paper/2024/hash/27245589131d17368cccdfa990cbf16e-Abstract-Datasets_and_Benchmarks_Track.html) | Fine-grained diagnosis should separate retrieval misses, noisy context, and unsupported generated claims. |
+| [BEIR](https://arxiv.org/abs/2104.08663) / TREC-style retrieval | Retrieval still needs classical rank metrics: Recall@k, Precision@k, MRR, MAP, and nDCG. |
+| [CRAG](https://arxiv.org/abs/2406.04744) | Real RAG evals must include long-tail, dynamic, multi-hop, and unanswerable questions, not only easy static facts. |
+| [DeepEval RAG metrics](https://deepeval.com/docs/metrics-faithfulness) | Production tools converge on faithfulness, answer relevance, and context relevance as generator/retriever checks. |
+
+## Current Repo Status
+
+Done:
+
+- `runRetrievalImprovementLoop()` auto-searches retrieval configs through `agent-eval`.
+- `runRagKnowledgeImprovementLoop()` exposes the whole RAG lifecycle as typed phases:
+  retrieval tuning, gap diagnosis, knowledge acquisition, knowledge update, answer-quality eval, and promotion.
+- Retrieval scenarios can label pages, page paths, sources, source anchors, and source spans.
+- The retrieval judge reports recall, MRR, nDCG, precision@k, cost, and held-out promotion.
+- The loop is tested with a real `agent-eval` run where `{ k: 2 }` beats `{ k: 1 }`.
+- The lifecycle loop is tested both with pluggable phase hooks and with a real local KB update through `runKnowledgeResearchLoop()`.
+- `ragAnswerQualityJudge()` and `createRagAnswerQualityHook()` score context precision/recall/relevance/sufficiency, faithfulness, answer relevance/correctness, citation support, abstention, and unsupported-answer rate.
+- `normalizeExternalRagScores()` and the row exporters make Ragas, DeepEval, TruLens, RAGChecker, and custom evaluator outputs pluggable instead of hard dependencies.
+- `scoreKnowledgeBaseIndex()` validates generic wiki/KB health: citation coverage, source-backed pages, stale sources, duplicate source hashes, and lint/validation errors.
+- `calibrateRagAnswerJudge()` enforces the strong-vs-weak metric check before trusting a RAG answer metric.
+
+Not done:
+
+- Slice-level reporting for freshness, distractors, multi-hop, and long-tail cases.
+- Packaged runtime adapters for browser/coding/research agents.
+  The lifecycle API accepts those agents as hooks today; it does not hardcode provider-specific workers.
+
+## Completion Criteria
+
+### Phase 1: Retrieval Quality
+
+Build a retrieval eval pack with at least 100 labeled scenarios.
+Use source-span labels wherever possible.
+
+Required slices:
+
+- 25 known-answer questions.
+- 25 paraphrase questions.
+- 20 distractor questions.
+- 10 freshness/version questions.
+- 10 multi-source questions.
+- 10 unanswerable or forbidden-source questions.
+
+Ship criteria:
+
+- Holdout source-span Recall@5 is at least 0.90.
+- Holdout nDCG@5 is at least 0.80.
+- Train-to-holdout recall gap is at most 0.08.
+- Stale or forbidden source hit rate is at most 0.02.
+- p95 retrieval latency and cost do not regress by more than 10 percent versus baseline.
+
+### Phase 2: Answer Quality
+
+Add a generated-answer eval artifact that includes query, retrieved context, answer text, citations, cost, latency, and trace ids.
+Score it with deterministic checks first and LLM judges only for semantic quality.
+
+Ship criteria:
+
+- Faithfulness or groundedness is at least 0.95 on holdout.
+- Answer relevance is at least 0.90 on holdout.
+- Answer correctness is at least 0.85 on human-labeled holdout.
+- Citation support is at least 0.95 for claims that cite sources.
+- Unsupported-answer rate on unanswerable questions is at most 0.05.
+
+### Phase 3: Diagnosis
+
+Add RAGChecker-style failure attribution.
+Every failed case must classify as one primary cause.
+
+Required failure classes:
+
+- Retrieval miss.
+- Retrieval noisy context.
+- Stale retrieval.
+- Missing multi-hop evidence.
+- Generator ignored evidence.
+- Generator hallucinated unsupported claim.
+- Citation mismatch.
+- Correct abstention.
+- Incorrect abstention.
+
+Ship criteria:
+
+- Every failed eval has one primary failure class.
+- At least 95 percent of generated claims can be mapped to supporting context, contradicted context, or no context.
+- Reports show metrics by slice and by failure class, not only the aggregate score.
+
+### Phase 4: Production Loop
+
+Run the same eval pack on every retrieval or prompt change.
+Keep train/dev/holdout isolated.
+Never tune on holdout.
+
+Ship criteria:
+
+- `runRetrievalImprovementLoop()` gates retrieval config changes.
+- `runRagKnowledgeImprovementLoop()` is the default front door when retrieval changes, source acquisition, KB updates, answer checks, and promotion must run together.
+- Answer-quality eval gates prompt and synthesis changes.
+- Reports persist run id, commit, config hash, dataset hash, metric versions, cost, latency, and traces.
+- A promoted candidate must improve the target metric without violating faithfulness, abstention, cost, or latency limits.
+
+## Next Implementation Steps
+
+1. Add slice-level aggregation helpers for freshness, distractors, multi-hop, long-tail, and unanswerable cases.
+2. Add forbidden/stale source targets to retrieval scenarios.
+3. Add packaged adapters that turn `agent-runtime` browser, research, and coding agents into `runRagKnowledgeImprovementLoop()` hooks.
+4. Add a CLI command that runs the lifecycle loop and writes a reproducible report under `.agent-knowledge/eval/`.

package/docs/results/adaptive.md

@@ -0,0 +1,120 @@
+# Adaptive topology: spend the LLM verifier only when it pays
+
+The cost/quality A/B (`docs/results/cost-quality.md`) found the LLM relevance
+verifier's cleanliness win is dominated by **de-duplication** — which a
+deterministic content-hash / canonical-URL check captures at ~none of the LLM
+premium — and that an LLM check only earns its dollar on the off-scope tail. The
+production move it named was: do the cheap deterministic work first, reserve the
+LLM for the ambiguous tail. `createAdaptiveResearchDriver`
+(`src/adaptive-driver.ts`) is that driver, and this is its measurement.
+
+Per candidate source the adaptive driver runs three stages, cheapest first,
+stopping at the first that decides:
+
+1. **Dedup ($0).** Reject a source whose canonical URL (scheme/`www`/trailing
+   slash/tracking params stripped) or normalized-text content hash matches one
+   already accepted this round or in the KB.
+2. **Heuristic triage ($0).** Classify a unique survivor with host/title/length
+   signals only: an authoritative host (arxiv, `*.edu`, `*.gov`, official docs,
+   github, …) with a substantial body is **kept**; an obvious spam/listicle
+   title or a too-thin body is **dropped**; everything else is **ambiguous**.
+3. **LLM escalation ($).** Only ambiguous survivors reach the shipped LLM
+   relevance verifier (`createVerifyingResearchDriver`) — one call each.
+
+## Live frontier, n=5 topics (glm-5.2)
+
+Real web-research worker fetches each topic once; the same fetched proposals
+(plus one planted tracking-decorated mirror of the first source, so the dedup
+stage has a real duplicate to catch) are gated through all three drivers. Cost
+is the per-arm `RouterClient.usage()` diff (#36). Total spend for the run:
+**$0.033**.
+
+| topic | fetched | single admit | full-LLM admit / calls / $ | adaptive admit / LLM calls / $ |
+|---|---|---|---|---|
+| self-speculative decoding | 3 | 3 | 1 / 3 / $0.0027 | 2 / **0** / **$0.0000** |
+| rotary position embeddings | 3 | 3 | 1 / 3 / $0.0031 | 2 / **0** / **$0.0000** |
+| grouped-query attention | 7 | 7 | 3 / 7 / $0.0072 | 6 / 3 / $0.0030 |
+| KV-cache quantization | 5 | 5 | 3 / 5 / $0.0052 | 4 / **0** / **$0.0000** |
+| LoRA fine-tuning | 7 | 7 | 4 / 7 / $0.0079 | 6 / 3 / $0.0037 |
+| **total** | **25** | **25** | **12 / 25 / $0.0261** | **20 / 6 / $0.0068** |
+
+**Cost.** Adaptive cuts LLM verifier calls **76%** (25 → 6) and dollars **74%**
+($0.0261 → $0.0068). On 3 of the 5 topics it spent **zero** LLM calls — every
+unique survivor was on an authoritative host, so the $0 stages decided
+everything.
+
+## The honest reading: adaptive is a frontier POINT, not a free lunch
+
+Admitted-source counts (lower = cleaner KB): **single 25, adaptive 20,
+full-LLM 12**. Adaptive sits **between** the two:
+
+- It removes **5 of the 13 sources** the full-LLM judge rejects that the
+  single-agent loop keeps (every one of them a real duplicate caught by the $0
+  dedup stage — exactly the de-dup-dominated win the cost/quality result
+  predicted).
+- It does **NOT** match full-LLM cleanliness. The remaining 8 sources full-LLM
+  rejects, adaptive keeps. The cause is structural and visible in the trace: on
+  this topic set every non-duplicate survivor landed on an authoritative host
+  (arxiv / github / official docs), so the heuristic **kept** it without ever
+  asking the LLM — and the LLM, when full-LLM did ask it, judged several of
+  those same authoritative pages not-quite-on-topic and dropped them. The host
+  prior is coarser than the relevance judge.
+
+So the frontier tradeoff is concrete: **adaptive recovers the deterministic
+de-dup half of full-LLM's cleanliness for ~26% of full-LLM's dollars, and gives
+up the relevance-judgment half.** Whether that is the right point depends on the
+cost of a kept-but-marginal source. If a slightly-off-topic authoritative page
+is cheap to carry, adaptive dominates. If every admitted source must clear a
+relevance bar, the host heuristic is too permissive and you want the full LLM —
+or a tightened heuristic.
+
+## Where the heuristic is weak — stated plainly
+
+The escalation count is the diagnostic. On 3 of 5 topics it was **zero**: the
+heuristic never deferred to the LLM, so on those topics adaptive is a
+**pure host/title/length rule**, and its cleanliness is exactly that rule's
+cleanliness — no smarter than "trust arxiv/github, drop spam." That is fine when
+the worker's sources are dominated by authoritative hosts (as here), but it
+means the LLM's relevance judgment is contributing nothing on those topics, by
+construction. The two topics where adaptive *did* escalate (grouped-query
+attention, LoRA) are where some survivors were on unknown hosts — and there the
+3 LLM calls per topic are the off-scope tail the verifier is actually for.
+
+The heuristic would mis-route in two directions a richer worker would expose,
+neither seen on this authoritative-host-heavy set:
+
+- **False keep:** an authoritative-host page that is off-topic or low-value
+  (an arxiv paper on an unrelated subject) is kept without the LLM ever seeing
+  it. The host prior cannot catch this; only the relevance judge can.
+- **False drop:** a genuinely good source on an unknown blog/host with a
+  spam-shaped title, or under the 400-char body floor, is dropped before the LLM
+  could rescue it.
+
+## What this changes
+
+The deployable recommendation from the cost/quality result was "deterministic
+dedup first, reserve the LLM for the tail." This driver ships that and the
+measurement confirms the **cost** half cleanly (76% fewer calls, 74% cheaper)
+and qualifies the **quality** half honestly: adaptive captures the de-dup
+cleanliness (the dominant, free win) but not the LLM's relevance cleanliness,
+because the host heuristic resolves authoritative survivors without asking. For
+a worker whose sources are mostly authoritative, adaptive is the right frontier
+point. For one whose junk is on-topic-looking pages on unknown hosts, the
+ambiguous tail grows and adaptive converges toward full-LLM cost — which is the
+correct behavior: it pays for the LLM exactly when the cheap signals can't
+decide.
+
+## Run it
+
+```
+# offline (controlled wiring + escalates-only-ambiguous proof, no creds)
+pnpm exec vitest run tests/loops/adaptive-ab.test.ts
+
+# live three-topology frontier (needs TANGLE_API_KEY with glm-5.2 credits)
+AGENT_KNOWLEDGE_LIVE=1 \
+ADAPTIVE_LIVE_GOALS="self-speculative decoding|rotary position embeddings|grouped-query attention|KV-cache quantization|LoRA fine-tuning" \
+pnpm exec vitest run tests/loops/adaptive-ab.test.ts -t "three-topology"
+```
+
+A cheap one-call glm-5.2 smoke gates the multi-topic burn (fails fast if the key
+or the reasoning-token floor is broken) before any dollars are spent.

package/docs/results/claim-grounding.md

@@ -0,0 +1,97 @@
+# Claim-grounding: the band where the verifier earns its dollar
+
+`docs/results/cost-quality.md` found the relevance verifier's cleanliness win is
+**dominated by de-duplication** — a deterministic content-hash captures most of it
+at ~none of the LLM premium. So the open question was: is there an error band where
+a verifier earns its cost — something a hash AND a relevance judge both miss?
+
+**Yes: misattributed citations.** A source that is on-topic, unique, and real, but
+whose cited CLAIM does not appear in the page (the LLM wrote a plausible sentence
+and hung a real URL off it). De-dup passes it (it's unique). A relevance judge
+passes it (the page is on-topic). Only checking the claim against the fetched text
+catches it — and that check is **deterministic text presence, $0 inference**.
+
+## The mode
+
+Each proposed source now carries the specific claim it is cited for
+(`withCitedClaim` → `metadata.citedClaim`). The verifier
+(`createClaimGroundingVerifier`) runs `groundClaimInText(claim, pageText)` over the
+`htmlToText` output of the page the worker actually fetched — verbatim, normalized
+(punctuation/whitespace-insensitive), or a ≥70% content-word overlap close
+paraphrase. A claim that isn't present is rejected as **misattributed**. The oracle
+is text presence, not a model call, so it composes with the LLM relevance verifier
+(reject off-topic AND misattributed) or runs alone at zero inference cost.
+
+## Live A/B (glm-5.2, real web fetch, planted misattribution band)
+
+Real worker (glm-5.2 query-gen → live `/v1/search` → `politeFetch` → `htmlToText`)
+fetches the sources once per topic; we then plant ONE misattribution per topic (real
+fetched page + a deliberately-wrong claim) and run three verifier arms over the SAME
+proposals. Cost diffed per arm with the #36 `RouterClient.usage()` instrumentation.
+
+| n=5 topics | misattributions caught | marginal $ | $/topic | per-$ caught |
+|---|---|---|---|---|
+| no-verifier | 0 / 5 | $0.0000 | — | — |
+| relevance (LLM judge) | **4 / 5** | $0.0157 | ~$0.0031 | 254 |
+| claim-grounding (text) | **5 / 5** | **$0.0000** | $0 | ∞ |
+
+Per-topic (caught relevance / grounding): self-speculative decoding 1/1, rotary
+position embeddings 1/1, grouped-query attention 1/1, **KV-cache quantization 0/1**,
+LoRA 1/1. (An earlier 3-topic run missed self-speculative decoding instead — the
+miss moves around; it is not a fixed topic.)
+
+**Reading.** Claim-grounding catches every misattribution at **$0**; the relevance
+judge catches most but **misses one in five at ~$0.003/topic**. The miss is the
+point: the relevance verifier only ever sees the page text, never the cited claim,
+so it is *structurally blind* to misattribution. It catches one only by accident —
+when the fabricated claim happens to also make the page read off-topic
+(e.g. a "12-billion-parameter draft transformer" claim on a rotary-embeddings page).
+When the fabrication stays on-topic (the KV-cache case), the judge waves it through.
+
+So on THIS band the verifier-per-dollar comparison inverts the cost/quality result:
+there, the LLM verifier bought a dedup-shaped gain a free hash already captures —
+expensive for what a cheap rule does. Here the cheap, deterministic check
+**dominates** the expensive judge: it catches strictly more (5/5 vs 4/5) at strictly
+less ($0 vs $0.0157). The verifier earns its dollar on misattribution; it does not on
+de-duplication.
+
+## Why this is a real correctable band (not dedup, not relevance)
+
+- **Not de-duplication.** Every planted source has a unique URL and unique text; a
+  content-hash / canonical-URL dedup keeps all of them.
+- **Not generic relevance.** Every planted source is on-topic; the relevance judge
+  (and the offline relevance stand-in) accept them. The error is in the *claim*, not
+  the *topic*.
+- **Executable ground truth.** The check is presence/close-paraphrase of the claim
+  in the fetched text — deployable in production with no oracle and no model call.
+
+The offline arm proves the floor with a controlled 4-source pool (2 grounded, 2
+misattributed): claim-grounding admits **0/2** misattributions and keeps **2/2**
+grounded sources, while relevance and no-verifier both admit **2/2**.
+
+## Threats to validity
+
+- **n=5 topics, 1 misattribution each.** The direction (grounding ≥ relevance caught,
+  at ≤ cost) is asserted in the test on every run; the magnitude is small-n. The
+  relevance miss-rate (1/5 here, 1/3 earlier) is an existence proof of the blind
+  spot, not a calibrated rate.
+- **Planted misattributions, not naturally-occurring ones.** Like the cost/quality
+  offline floor, the misattribution is injected so the band is measurable. It models
+  the real LLM citation-fabrication failure but does not measure its base rate in the
+  wild — that needs a corpus of model-written citations checked by hand.
+- **The grounding oracle is conservative.** A real paraphrase whose inflected words
+  differ from the page ("drafts" vs "draft") can score below 0.7 and be rejected —
+  a false-positive misattribution flag. `minOverlap` tunes this; the worker should
+  cite the page's own key terms (the `createClaimDecorator` extractor is told to).
+
+## Run it
+
+```bash
+# offline floor (no creds)
+pnpm exec vitest run tests/loops/claim-grounding-ab.test.ts -t "offline"
+
+# live A/B (creds-gated). A cheap glm-5.2 smoke runs BEFORE the multi-topic burn.
+AGENT_KNOWLEDGE_LIVE=1 TANGLE_API_KEY=… \
+  CLAIM_GROUNDING_LIVE_GOALS='self-speculative decoding|rotary position embeddings|grouped-query attention|KV-cache quantization|LoRA' \
+  pnpm exec vitest run tests/loops/claim-grounding-ab.test.ts -t "three verifier arms"
+```

package/docs/results/cost-quality.md

@@ -0,0 +1,27 @@
+# Cost/quality: the two-agent loop's inference premium
+
+Live 9-topic A/B (glm-5.2, budget B ≤ 4 passes/arm), measured per arm with the
+router-client instrumentation. The original A/B reported only admitted-sources at
+"equal passes" — which charged the two-agent verify step as one pass while it is
+actually N `verifySource` LLM calls. Pricing the calls shows what that hid.
+
+| per topic (mean) | two-agent | single-agent | ratio |
+|---|---|---|---|
+| LLM chat calls | 5.4 | 1.0 | ~5.4× |
+| tokens (in+out) | ~4,900 | ~530 | ~9× |
+| cost (USD) | ~$0.0072 | ~$0.0013 | ~5.5× |
+| latency (wall) | ~37 s | ~11 s | ~3.4× |
+| cleanliness Δ (single − two admitted) | — | — | +1.56, 95% CI [0.33, 2.67] |
+
+Per-topic Δ (single − two admitted) this run: self-speculative decoding +4,
+grouped-query attention 0, rotary position embeddings +1, KV-cache quantization
+**−1**, LoRA +4, ring attention +2, constitutional AI +3, transformer +2, gradient
+descent **−1**. Coverage 1.00 every topic, both arms.
+
+**Reading.** The verifier buys ~1.5–2.7 fewer junk sources for roughly **5× the
+dollars, 9× the tokens, and 3× the latency** — and on two topics it admitted *more*
+than the single agent (the cleanliness signal is real but noisier than the +2.3 /
++2.7 of earlier runs). The cleanliness gain is dominated by de-duplication, so the
+honest production move is a deterministic content-hash / canonical-URL dedup, which
+captures most of the cleanliness at ~none of this premium; reserve an LLM check for
+the off-scope tail. This is the cost half the "equal passes" framing left out.