@tangle-network/agent-knowledge 1.9.0 → 1.11.0

package/docs/results/investment-thesis.md

@@ -0,0 +1,306 @@
+# From paper-retrieval to investigation — a research eval where one search is not enough, a metric that discriminates depth, and a 3-arm topology A/B
+
+*Tangle Network · `agent-knowledge`*
+
+## Verdict (BLUF)
+
+We moved the research eval off ML-paper retrieval — where a **single** web search
+already returns the answer, so the only thing the metric could measure was
+*collection* — onto **investment research**, where the material facts are buried in
+the footnotes of an SEC filing and genuinely require *investigation* to surface. On
+this harder domain we (1) **calibrated** a `$0`, model-free metric and proved it
+discriminates research depth — a shallow ticker summary scores **1/27 (4%)**, a
+deep filings-grounded thesis scores **27/27 (100%)**, a **+96-point** gap — and then
+(2) ran **three research topologies head-to-head** on the same five held-out
+companies at matched compute, grading each one's knowledge base against a firewalled
+checklist of **27 buried material facts** a ticker search misses.
+
+**The honest A/B verdict: the research-DRIVING loop surfaced the most buried facts
+(16/27, 59%), +5 over blind collection (11/27, 41%) — but at n=5 companies that lift
+is NOT statistically clean: the 95% confidence interval crosses zero (P(Δ≤0)=0.08).**
+The verify/dedup loop did **not** beat collection at all (10/27, 37% — a wash,
+slightly worse). So no topology *significantly* beats collection here either —
+**driving is the only arm that even points the right way, and it points there
+suggestively, not significantly.** What the reframe *did* buy is a domain and a meter
+where the question is finally well-posed: the metric is no longer measuring whether a
+single search ran, but whether investigation reached the buried fact.
+
+| arm | what the coordinator does | material facts | cost (5 cos.) | chats | searches | tokens |
+|---|---|---|---|---|---|---|
+| **A · collection** | nothing — accepts every source (1 agent collects) | 11/27 (41%) | $0.082 | 10 | 20 | 64,248 |
+| **B · verify/dedup** | LLM gates each source for relevance, rejects off-topic | 10/27 (37%) | $0.125 | 56 | 52 | 84,678 |
+| **C · driving (deepen)** | extracts claims, demands corroboration, asks deep follow-ups | **16/27 (59%)** | $0.157 | 33 | 28 | 124,387 |
+
+Cost is real provenance, not an estimate: every `$`/call/token is diffed from
+`RouterClient.usage()` per company (the router's own `usage` field, priced at
+glm-5.2 rates), so "driving cost 1.9× collection" is measured, not modelled.
+
+## 1. The domain reframe — why we left ML-paper retrieval
+
+The companion paper's depth eval (`docs/two-agent-research-ab.md` §9) measures a
+research loop on **20 deep questions across 5 ML topics**. That apparatus is sound —
+its grader scores 0/20 on a one-line topic definition and 20/20 on a mechanism-rich
+paragraph, so it *can* tell depth from surface. But the topology A/B on top of it
+came back a clean null (driving 16/20 @ budget 4, 13/20 @ budget 6 — the winner
+*flips with the compute budget*, and the within-arm swing is as large as the
+between-arm gap). The autopsy named the cause: **on an ML topic a single good search
+already collects the answer.** Every arm finished in one effective round because the
+generic "one source closes the gap" readiness was met by the *first* fetch — so the
+driving driver, whose entire mechanism is steering a *second* round, never got to
+act. When one search suffices, there is no investigation for a smarter coordinator to
+do, and the metric can only reward collection. That is the failure-mode-in-spirit the
+ML exam couldn't escape: not that the grader was loose, but that the *domain* was too
+easy for topology to matter.
+
+So we changed the domain to one where a single search **provably cannot** suffice.
+**Investment research**: give a loop a company + ticker + an as-of cutoff date and ask
+for a thesis; grade it on the buried, material, non-obvious drivers a thorough analyst
+flags and a ticker search misses. The decisive facts here live in 10-K footnotes — an
+HTM securities mark roughly equal to a bank's entire equity (SIVB), a deposit base
+that is 97% uninsured, a negative per-unit gross margin, a related-party lease. A web
+search for the company name returns "profitable regional bank" or "high-growth auto
+e-commerce"; the filing shows the mark-to-market hole the size of capital. **The
+answer is not collectable in one fetch — it has to be investigated for.** That is the
+property the ML domain lacked, and it is what makes a topology A/B finally meaningful:
+if a smarter coordinator can ever beat blind collection, a domain where the answer is
+buried is where it has the room to.
+
+The held-out set is **5 companies, 27 material facts**, every fact read live out of
+the primary SEC EDGAR 10-K during curation, every dollar figure quoted from the
+de-tagged filing text, every value knowable *at the cutoff* (the eventual collapse is
+recorded for the reader only and is **never graded**). The companies skew distressed /
+downside-risk — a documented curation bias, not a hidden one (full provenance, the
+drop log, and the per-fact keyword groups: `docs/eval/investment-material-facts.md`):
+
+| ticker | company | as-of cutoff | CIK | facts |
+|---|---|---|---|---|
+| SIVB | SVB Financial Group | 2023-02-24 | 719739 | 6 |
+| BBBY | Bed Bath & Beyond | 2022-04-21 | 886158 | 5 |
+| CVNA | Carvana | 2023-02-23 | 1690820 | 5 |
+| PTON | Peloton | 2022-09-07 | 1639825 | 6 |
+| SI | Silvergate | 2022-02-28 | 1312109 | 5 |
+
+## 2. Calibration — does the metric discriminate depth? (the gate the ML exam passed only weakly)
+
+A topology A/B is meaningless unless the metric grading it can tell a *deep* thesis
+from a *shallow* one. If it can't, it is measuring word-collection — the exact failure
+the reframe set out to escape — and any A/B on top of it is noise. So **before** the
+A/B, we ran a calibration gate (`$0`, offline, the binding result):
+
+For each of the 5 companies we hand-wrote two theses and scored each with the metric's
+pure core (`materialFactsSurfacedInText`):
+
+- **shallow** — a one-paragraph ticker summary: what the company does, a vibe on the
+  stock, generic macro/competition risk. The kind a single name-search returns. Names
+  none of the buried, filing-level facts.
+- **deep** — a filings-grounded analyst memo naming the buried drivers (the duration
+  loss, the buyback drain, the negative unit margin, the deposit concentration, the
+  related party) in independent prose, with the real numbers.
+
+The grader is the same `$0`, model-free, case-insensitive substring check the held-out
+checklist ships (`gradeFactAgainstText`); the checklist is firewalled — read only by
+the metric, never shown to a loop.
+
+| ticker | shallow | deep | gap |
+|---|---|---|---|
+| SIVB | 1/6 (17%) | 6/6 (100%) | +83pp |
+| BBBY | 0/5 (0%) | 5/5 (100%) | +100pp |
+| CVNA | 0/5 (0%) | 5/5 (100%) | +100pp |
+| PTON | 0/6 (0%) | 6/6 (100%) | +100pp |
+| SI | 0/5 (0%) | 5/5 (100%) | +100pp |
+| **total** | **1/27 (4%)** | **27/27 (100%)** | **+96pp** |
+
+**The metric is VALID:** it cleanly separates a shallow ticker-summary from a deep,
+filings-grounded thesis — a +96-point aggregate gap, every company clearing the bars
+(shallow `< 30%`, deep `> 70%`) with wide margin. Two guards make the gap real and not
+a teaching-to-the-test artifact:
+
+- **Anti-circularity.** The gate asserts **no deep thesis verbatim-embeds any
+  checklist `evidence` string** — the deep theses state the same publicly-documented
+  facts in independent analyst prose. A high deep score is the meter catching real,
+  independently-phrased depth, not an answer-key echo.
+- **The one honest shallow hit.** The single shallow surface (SIVB 1/6) is SIVB/f5 —
+  the innovation-economy / venture-client concentration — which fires on "technology"
+  / "venture" in the SVB summary. That genuinely *is* the least-buried of SVB's six
+  facts (a ticker search does return "SVB banks tech and venture startups"). We kept it
+  as a 4% leak rather than tighten the group, because it honestly reflects one
+  near-surface fact while the five truly buried ones (the ~$15B HTM loss, the
+  equity-sized mark, the $151.5B uninsured base, the 20-point deposit-mix shift, the
+  AFS/AOCI loss) stay unsurfaced. The firewall holds (17% << 30%).
+
+Calibration also did its job as more than a rubber stamp: the first pass had the
+**Silvergate shallow thesis scoring 3/5 (60%)**, blowing the bar, because three fact
+groups accepted bare crypto-bank vocabulary (`crypto`, `grew rapidly`, `proprietary` /
+`payment network`) that any one-line summary trips. We tightened those three to require
+the *buried* signal — the deposit-**concentration** framing, the specific **$14.3B**
+figure, the **SEN** (Silvergate Exchange Network) name — and dropped the generic vocab.
+The deep thesis still hits all three; the shallow one no longer does. The metric found
+a way it could be fooled and we closed it before any A/B depended on it.
+
+**This is the gate the ML exam passed only in spirit.** The ML grader discriminated
+0/20 vs 20/20 too — but on a domain where the deep answer was *collectable* in one
+search, so the discrimination measured grammar, not investigation. Here the +96-point
+gap is over facts that are *buried by construction*, so a high score is reachable only
+by reaching into the filing. Same shape of result, materially harder domain.
+
+## 3. The 3-arm A/B — what each topology surfaced, head-to-head
+
+For each company the loop is told **only** `{company, ticker, cik, cutoff}` plus a
+generic set of analyst-lens readiness specs (balance-sheet risk, concentration,
+leverage, margins, liquidity, governance, regulatory — the *lenses* and where they
+live, the latest SEC 10-K — **not the answers**). It researches the company *as of* the
+cutoff over web + SEC EDGAR (both public), writes a thesis, and we grade the resulting
+KB with `materialFactsSurfaced` — the firewalled checklist the loop never sees.
+
+**Compute is matched by construction.** All three arms run the *same* web worker, the
+*same* 3-round budget, the *same* worker config (`resultsPerQuery: 3, queriesPerGap: 1,
+maxSourcesPerRound: 6`). The **only** thing that varies is the driver — the coordinator
+between the worker and the knowledge base:
+
+- **A · collection** (`createCollectionResearchDriver`) — an inert rubber stamp:
+  accepts every source, gates nothing, steers only with the loop's built-in open-gap
+  list. The driver adds **zero** router calls. This is the blind-collection floor.
+- **B · verify/dedup** (`createVerifyingResearchDriver`) — an LLM relevance gate: one
+  chat call per candidate source to accept-or-reject for on-topic relevance and
+  near-duplication. The worker ADDS; the driver GATES.
+- **C · driving** (`createResearchDrivingDriver`) — extracts each source's claims,
+  tracks how many *independent* sources corroborate each, and synthesizes deep
+  follow-up sub-questions (comparative / mechanism / gap / contradiction) it folds into
+  the worker's next prompt to push depth and demand corroboration.
+
+So any quality difference is attributable to topology, and any cost difference is the
+price each topology pays in extra inference.
+
+### Per-company matrix
+
+| ticker | A · collection | B · verify | C · driving |
+|---|---|---|---|
+| SIVB | 2/6 · $0.017 | 0/6 · $0.019 | **5/6 · $0.017** |
+| BBBY | 4/5 · $0.014 | 0/5 · $0.026 | **5/5 · $0.027** |
+| CVNA | **3/5 · $0.016** | 2/5 · $0.028 | 2/5 · $0.025 |
+| PTON | 2/6 · $0.019 | **4/6 · $0.026** | 2/6 · $0.054 |
+| SI | 0/5 · $0.016 | **4/5 · $0.027** | 2/5 · $0.033 |
+| **total** | **11/27 (41%)** | **10/27 (37%)** | **16/27 (59%)** |
+| **cost** | **$0.082** | **$0.125** | **$0.157** |
+
+No arm dominates company-by-company. Driving owns the two banks; verify owns PTON and
+SI; collection owns CVNA. That spread is the whole story at n=5: the topology that wins
+depends heavily on which pages the web returned for that company that minute.
+
+### Significance (paired bootstrap, unit = company, 10k resamples)
+
+| comparison | total facts | per-company Δ | mean Δ/co. | 95% CI | P(Δ≤0) |
+|---|---|---|---|---|---|
+| driving − collection | 16 vs 11 (+19pp) | `[+3,+1,−1,0,+2]` | +1.0 | **[−0.20, +2.20]** | 0.08 |
+| verify − collection | 10 vs 11 (−4pp) | `[−2,−4,−1,+2,+4]` | −0.2 | [−2.60, +2.40] | 0.60 |
+| driving − verify | 16 vs 10 (+22pp) | `[+5,+5,0,−2,−2]` | +1.2 | [−1.60, +4.00] | 0.23 |
+
+Every interval crosses zero. **Driving vs collection is the closest to clean
+(P=0.08)** but does not pass the project's significance bar. Verify vs collection is a
+coin flip. This is the project's well-documented small-n mirage: exciting deltas born
+at n=5 do not survive a paired bootstrap.
+
+## 4. Autopsy — the two things worth understanding
+
+### 4.1 Why driving wins where it wins
+
+Driving's mechanism is multi-round: extract claims from round 1, then steer the worker
+in rounds 2–3 to corroborate the weak ones and chase the deep questions. It helps most
+where the **first** fetch lands real filing data the driver can build on — the two
+banks, where SEC bank-call-report / 10-K data is dense and reachable. SIVB jumps from 2
+buried facts (collection) to 5 (driving): the duration loss, the deposit concentration,
+the AFS/AOCI mark all surface once the driver demands the balance-sheet detail a second
+time. Where the first fetch is thin (PTON, SI), the driver has little to deepen and the
+extra rounds mostly burn searches (PTON driving: 12 searches, 2 facts, the most
+expensive cell at $0.054). This is the reframe paying off mechanically: it is exactly
+the *second-round investigation* the ML domain never reached, and it is the only thing
+that moved the number.
+
+### 4.2 Why verify scored ZERO on SIVB and BBBY (a real effect, not a bug)
+
+This was the surprising result, so we probed it directly (a 2-round live replay of the
+BBBY verify arm with round-level accept/reject logging). The verifier rejected **every**
+source both rounds, accepting nothing, writing no KB pages:
+
+```
+ROUND 1: accepted=0 rejected=3 writtenPages=0
+  REJECT stockanalysis.com/stocks/bbby/financials  :: Third-party aggregator, not the SEC EDGAR 10-K primary source…
+  REJECT stocktitan.net/financials/BBBY            :: Third-party financial data aggregator, not the authoritative SEC 10-K…
+  REJECT last10k.com/sec-filings/bbby              :: Third-party aggregator showing a 2025/2026 10-K, well after the 2022-04-21 research date…
+ROUND 2: accepted=0 rejected=2 writtenPages=0
+```
+
+The verifier was **correct on the merits** — those are aggregators, not the primary
+filing, and last10k showed a post-cutoff filing. But the worker never surfaced the
+EDGAR primary for BBBY, so a strict primary-only gate left the KB **empty**. Collection
+accepts the same aggregator pages and scores 4/5 on BBBY; driving accepts them and
+scores 5/5. **The gate's strictness is a liability when the worker's sourcing is
+imperfect:** it throws away the only evidence the loop had. This is a genuine topology
+trade-off worth naming, not a harness break — the verify arm surfaced facts fine on
+CVNA (2), PTON (4), SI (4), so the harness works.
+
+### 4.3 The empty-thesis caveat (honest)
+
+Some runs show `thesis=0ch` (empty synthesis) yet still score facts. That is expected:
+`materialFactsSurfaced` grades the **whole KB** (the worker's fetched `knowledge/*.md`
+pages), not only the final thesis page. glm-5.2 occasionally spends its entire output
+budget on hidden reasoning and returns empty visible content on the synthesis call — a
+known reasoning-model behavior we floor at 1200 tokens but can't fully eliminate. The
+score still reflects what the loop fetched, so an empty thesis does not invalidate a
+run; it just means the synthesis prose was lost while the curated evidence was not.
+
+## 5. What this does and does not establish
+
+- **Does**: the metric *discriminates depth on a domain where one search is not
+  enough* (calibration: 1/27 shallow vs 27/27 deep, +96pp) — so the topology A/B on top
+  of it is finally well-posed, unlike the ML retrieval exam where one search already
+  collected the answer. At matched compute on the 5-company held-out set, the
+  research-driving topology surfaced the most buried material facts (16/27 vs 11/27 for
+  blind collection), a +5-fact / +18pp lift, for ~1.9× the cost. Every dollar is real,
+  per-call provenance from `RouterClient.usage()`.
+- **Does NOT**: prove driving *significantly* beats collection. At n=5 the
+  paired-bootstrap CI for the driving lift crosses zero (P(Δ≤0)=0.08). The verdict is
+  "promising, under-powered," consistent with the project's prior topology nulls (the
+  ML exam, depth-vs-breadth, native-skills), not "driving wins." No topology cleared the
+  bar. Nor does the 27-fact checklist generalize beyond its documented curation bias
+  (downside-risk, distressed-name skew).
+- **The next rung** (to turn the P=0.08 lean into a verdict): expand the held-out set
+  well past 5 companies (the checklist is the constraint, not the harness) and re-run at
+  n≥24 so a +1-fact/company effect can clear a paired bootstrap. The driving arm is the
+  one worth funding that test on; verify is not.
+
+## 6. Reproduce
+
+```bash
+# The calibration gate that must pass FIRST ($0, offline) — the metric is valid.
+pnpm test investment-calibration
+
+# The offline task wiring ($0) — proves page → index → grade end-to-end.
+pnpm test investment-thesis-task
+
+# The full live 3-arm A/B (costs ~$0.36 total at 5 companies × 3 arms × 3 rounds).
+# Needs a router key that can reach glm-5.2.
+export TANGLE_API_KEY=<router key>
+AGENT_KNOWLEDGE_LIVE=1 IT_LIVE_ROUNDS=3 \
+  npx vitest run tests/eval/investment-thesis-ab.test.ts --reporter=basic
+
+# A single arm / single company (cheap smoke before the full burn):
+AGENT_KNOWLEDGE_LIVE=1 IT_LIVE_TICKERS=CVNA IT_LIVE_ARMS=collection \
+  npx vitest run tests/eval/investment-thesis-ab.test.ts --reporter=basic
+```
+
+`IT_LIVE_ARMS` (`|`-separated subset of `collection|verify|driving`) and
+`IT_LIVE_TICKERS` scope the run; `IT_LIVE_ROUNDS` sets the per-arm round budget
+(default 3 — driving needs > 1). The smoke (one cheap glm-5.2 call) runs once before any
+arm, so a bad key fails fast, before the burn.
+
+---
+
+*Run provenance. Calibration: `$0`, offline, reproducible — numbers from
+`tests/eval/investment-calibration.test.ts` (shallow `< 30%`, deep `> 70%`,
+gap `> 40pp` per company and aggregate; all pass). A/B: 5 companies × 3 arms × 3 rounds
+= 15 live company-runs, glm-5.2 over the Tangle router, ~37.5 min wall, $0.36 total;
+grader `materialFactsSurfaced` (firewalled, `$0`, model-free substring check); numbers
+transcribed verbatim from the test's `[IT 3-ARM TOTALS]` console output and recorded in
+commit `338bc54`; statistics from a paired bootstrap over the per-company fact deltas.
+Held-out set + per-fact provenance: `docs/eval/investment-material-facts.md`.*

package/docs/results/research-driving.md

@@ -0,0 +1,265 @@
+# Does driving research DEEPER beat just collecting it? A held-out deep-question A/B — and an honest null
+
+*Tangle Network · `agent-knowledge`*
+
+## Verdict (BLUF)
+
+We built a research driver whose job is to push a web-research loop **deeper** —
+extract each source's claims, demand a second independent source for every claim,
+generate comparative / mechanism / contradiction sub-questions, and steer the
+worker to chase them. The hypothesis: a KB built this way answers **more held-out
+deep questions** than one built by plain collection or by a relevance/dedup
+verifier.
+
+On a **firewalled exam of 20 deep questions across 5 ML topics**, graded with a
+$0 deterministic check the loop never sees, at equal compute, **the driving loop
+did NOT reliably beat plain collection — and cost ~12–16× more.** The verdict
+flips with the compute budget, which is the tell that it is **noise, not signal**:
+
+| arm | answered @ B=4 | answered @ B=6 | cost (5 topics) | tokens |
+|---|---|---|---|---|
+| single-agent (collect) | 13/20 | **15/20** | $0.005–0.007 | ~2.4–3.0k |
+| verify/dedup | **15/20** | **15/20** | $0.031–0.027 | ~21k |
+| **DRIVING (deepen)** | **16/20** | 13/20 | **$0.089–0.084** | ~69–71k |
+
+Driving "wins" at B=4 (16 > 15 > 13) and "loses" at B=6 (13 < 15), while the
+single-agent arm itself swings 15→13 and driving swings 16→13 on the *same* exam.
+At n=5 topics a ±1–3 question difference is within the run-to-run web variance —
+the arms are not separated by topology, they are separated by **which pages the
+web returned that minute**. The one thing that is stable and large is the **cost**:
+driving spends an order of magnitude more for no reliable quality gain.
+
+**Why** (the autopsy that explains it, §4): every arm finished in **one effective
+round** (`passes=2` on every topic, every budget). The generic readiness gate —
+"one source closes the gap" — is satisfied by the first round's fetch, so the
+loop *stops before the driving driver ever steers a second round*. Driving's
+entire mechanism is multi-round (extract → demand corroboration → re-search). So
+the headline A/B under-tests it. **We then gave it its fairest test** — a
+controlled probe (§5) that *forces* 3 rounds so the driver actually steers — and
+it **still does not win: 8 vs 8 against a blind worker, at ~9× the cost.** The
+negative result is robust, not a gate artifact.
+
+This is a real negative result, and we report it as one.
+
+## 1. Why a new metric
+
+The prior A/B in this repo
+([two-agent-research-ab.md](../two-agent-research-ab.md)) measured **cleanliness**
+— how *few* sources a verifier admits at equal coverage. That is the right metric
+for a verifier whose job is to filter. It is the **wrong** metric for the driving
+driver, whose thesis is the opposite: it is not trying to admit fewer sources, it
+is trying to build a KB that *answers more*. Measuring it on admitted-count would
+score its whole point as a regression (it admits *more*, because it accepts every
+source with an extractable claim and then chases more).
+
+So we need a metric for research **quality**, not hygiene. We use:
+
+> **QUALITY = how many held-out deep questions the resulting KB can answer.**
+
+## 2. The held-out exam (the firewall)
+
+[`tests/loops/held-out-exam.ts`](../../tests/loops/held-out-exam.ts). 5 ML topics,
+4 questions each = **20 deep questions**. Each is a *depth* question by
+construction — comparative, mechanism-level, or contradiction-aware — chosen so a
+single web search for the topic name does **not** surface the answer:
+
+- *speculative decoding* — the rejection-sampling acceptance rule and why output
+  is lossless; how self-speculative/Medusa (no draft model) trades off vs the
+  two-model scheme; what bounds the speedup; why one verify pass is ~free.
+- *LoRA* — the `W = W₀ + BA` update and which matrices train; QLoRA's 4-bit
+  tradeoff; why LoRA adds zero inference latency (merge); the very-low-rank claim.
+- *grouped-query attention* — the MQA↔GQA↔MHA spectrum of KV heads; the real
+  bottleneck (KV cache / memory bandwidth, not FLOPs); uptraining from a
+  checkpoint; the MQA quality cost GQA recovers.
+- *RLHF/PPO* — the KL-to-reference penalty and what it prevents; the pairwise
+  preference reward model + Bradley-Terry loss; DPO's reward-model-free insight;
+  PPO's clipped surrogate.
+- *mixture-of-experts* — top-k routing and sparse activation; load imbalance + the
+  auxiliary balancing loss; params-decoupled-from-FLOPs; the memory cost at
+  inference.
+
+**The firewall.** The questions and their expected answers are **never shown to
+any loop.** A loop is told only the topic name and the *same generic readiness
+specs* every arm gets ("what X is and how it works" / "results, mechanisms,
+trade-offs"). It researches blind. **After** it finishes, we grade the KB it built
+against the held-out questions with a **$0 deterministic substring grader** (no
+LLM — so the exam cannot leak into a model the loop observes), where each question
+carries the specific load-bearing answer tokens as keyword groups (a number, a
+name, a mechanism phrase), with synonym groups so a faithful page in its own words
+still grades as answered.
+
+**The exam discriminates depth, not surface facts** (calibration, run offline):
+
+| graded against | held-out answered |
+|---|---|
+| a one-line topic-definition snippet (what a single search returns) | **0 / 20** |
+| a deep, mechanism-rich paragraph | **20 / 20** |
+
+So a high score is only reachable by depth — the firewall is real, and the gap
+between arms (if any) would be real depth, not grader slack.
+
+## 3. The three arms, at equal compute
+
+[`tests/loops/research-driving-ab.test.ts`](../../tests/loops/research-driving-ab.test.ts).
+All three arms run the **same** real web worker
+([`createWebResearchWorker`](../../src/web-research-worker.ts) — glm-5.2 query-gen
+→ live `/v1/search` → `politeFetch` → `htmlToText`). They differ **only** in the
+driver:
+
+- **(A) single-agent collection** — the worker alone, no driver. It collects.
+- **(B) verify/dedup** — [`createVerifyingResearchDriver`](../../src/web-research-worker.ts):
+  a second glm-5.2 pass filters each source for relevance / near-duplicates.
+- **(C) DRIVING** — [`createResearchDrivingDriver`](../../src/research-driving-driver.ts):
+  the driver extracts each source's claims (glm-5.2), tracks independent-source
+  support + contradictions, and folds comparative / mechanism / gap /
+  contradiction sub-questions into the worker's next prompt.
+
+**Equal compute** is counted in agent passes (same unit as the prior A/B): a
+single-agent iteration is 1 worker pass; a two-agent round is 1 worker + 1 driver
+pass = 2. Each arm gets the same pass ceiling B; the single-agent arm gets more
+iterations to spend the budget the two-agent arms burn on their driver. Cost is
+read per-arm from `RouterClient.usage()` — measured dollars/tokens/calls, not
+estimates.
+
+## 4. Result — driving does not reliably win, and the verdict flips with budget
+
+Per-topic held-out questions answered (out of 4 each), at both budgets:
+
+| topic | single B4 / B6 | verify B4 / B6 | driving B4 / B6 |
+|---|---|---|---|
+| speculative decoding | 1 / 2 | 2 / 2 | 2 / 1 |
+| LoRA | 2 / 3 | 2 / 3 | 3 / 3 |
+| grouped-query attention | 4 / 4 | 4 / 4 | 4 / 4 |
+| RLHF / PPO | 2 / 2 | 3 / 3 | 3 / 2 |
+| mixture-of-experts | 4 / 4 | 4 / 3 | 4 / 3 |
+| **total /20** | **13 / 15** | **15 / 15** | **16 / 13** |
+
+The driving arm answers **16** at B=4 and **13** at B=6 — a 3-question swing on
+the *same* exam, the same arm, just a different compute ceiling and a different
+minute of web results. Single swings 13→15; verify is flat 15→15. **The within-arm
+swing is as large as the between-arm gap**, which is the signature of a null:
+whatever separates a "win" from a "loss" here is web variance, not the driver.
+
+The **cost**, by contrast, is stable and large. Driving spends **~$0.084–0.089**
+across 5 topics vs **~$0.005–0.007** for single-agent — **12–16× the dollars** and
+**~24× the tokens** (~70k vs ~2.5k) — because it runs a claim-extraction LLM call
+on every fetched source. For that 12–16× it buys no reliable held-out-question
+gain over plain collection.
+
+### Why every arm stopped after one round
+
+The decisive diagnostic is `passes=2` on **every** topic at **every** budget: the
+two-agent loop ran exactly **one** worker round before the readiness gate reported
+done, even with B=6 budget for three rounds. The generic specs require one source
+to close a gap, and the worker's first-round fetch closes them — so the loop stops
+before round 2. The driving driver's mechanism is *multi-round*: round 1 extracts
+claims and flags the weakly-supported ones; round 2+ is where it steers the worker
+to corroborate and go deeper. **It never got a round 2.** So in this setting
+driving's only active effect was a one-round claim-extraction tax with no chance to
+use what it extracted — which is exactly what the numbers show: same answers as
+collection, much higher cost.
+
+That makes the equal-compute/generic-gate A/B a test of the wrong thing for the
+driving thesis. The fair test has to *force* multiple rounds.
+
+## 5. Controlled multi-round probe — the driving thesis's fairest test
+
+To isolate "does depth-steering help **when it actually runs**?", we raise the
+readiness bar so the gate stays unmet and the loop runs the full round budget,
+forcing the driving driver to steer each round. Same real worker, same number of
+rounds; the only difference is whether the driver **steers** (driving) or the
+worker **re-searches the same gaps blind** (a no-op driver that accepts every
+source and never steers). If driving's steering has any value, this is where it
+shows.
+
+**Result — 3 rounds, 3 topics, driving steers vs blind re-search:**
+
+| topic | driving (steered) answered / cost | blind (no steer) answered / cost |
+|---|---|---|
+| speculative decoding | 2/4 — $0.032 | **4/4 — $0.004** |
+| LoRA | 3/4 — $0.028 | 3/4 — $0.003 |
+| RLHF / PPO | **3/4** — $0.033 | 1/4 — $0.003 |
+| **total /12** | **8/12 — $0.093** | **8/12 — $0.010** |
+
+**Steering does not help: 8 vs 8, at ~9× the cost.** Given its fairest test — the
+full multi-round regime its mechanism was designed for — the driving driver ties a
+blind worker that just re-searches the same gaps three times. It is **better on
+RLHF** (3 vs 1, the one topic where chasing corroboration found a page blind
+re-search missed) and **worse on speculative decoding** (2 vs 4 — steering pulled
+the worker *off* the pages that answered the exam and toward corroborating a
+narrower claim set). Those cancel. So the depth-steering does change *what* gets
+fetched, but not *how many* held-out questions get answered — and it pays ~9× the
+dollars for the privilege. The headline null (§4) is therefore **not** merely a
+gate artifact: even forced to run, driving does not beat blind collection on
+research quality at this n.
+
+## 6. Threats to validity
+
+- **Small n, high web variance.** n = 5 topics; one live run per arm per budget.
+  The §4 budget-flip is itself the evidence that the per-run magnitudes are
+  variance-bound. We did not run a paired bootstrap because the within-arm swing
+  already exceeds the between-arm gap — the honest read is "no separation," and a
+  significance test on a known-null at n=5 would dress it up, not clarify it.
+- **The gate, not the driver, ended the headline loop (§4) — but the probe
+  controls for it.** The headline A/B's generic one-source readiness gate closed
+  every loop after one round, so on its own it would only show "driving adds a
+  one-round extraction tax." The §5 probe removes that confound by forcing 3
+  rounds, and driving still ties blind (8 vs 8) — so the null survives the fix,
+  it is not an artifact of the permissive gate.
+- **The worker is shared and shallow.** All arms use the same ~500-line direct
+  pipeline (query-gen → search → fetch), not an `AgentProfile` on a harness. A
+  richer worker that follows citations or reads PDFs might give depth-steering more
+  to work with.
+- **glm-5.2-specific.** A stronger extractor/judge would change both the cost and
+  the per-round depth. The grader is conservative (substring/synonym presence); a
+  faithful paraphrase using none of the listed synonyms would read as unanswered.
+- **Depth-components is a proxy.** "Distinct expected-answer groups present" tracks
+  the binary answered-count closely here; it is a finer-grained view, not an
+  independent oracle.
+
+## 7. What this says, plainly
+
+Adding a "drive it deeper" agent did **not** make the research measurably better at
+answering hard, held-out questions — at equal compute (§4) *and* forced to run its
+full multi-round mechanism (§5), on this worker, at n=5 — and it cost 9–16× more.
+The steering changes *what* gets fetched (it helped on RLHF, hurt on speculative
+decoding) but not *how many* held-out questions get answered. The most durable
+thing the experiment produced is the **measurement apparatus**: a firewalled
+deep-question exam with a $0 deterministic grader that *can* tell depth from
+surface (0/20 vs 20/20), reusable for any future research-quality claim in this
+repo. The driving thesis — that pursuing depth + corroboration beats plain
+collection — is, on the evidence here, **not supported**; the cheaper paths
+(collect, or dedup) match it. Where it might still earn its cost: a worker rich
+enough that "go corroborate this claim" reaches a page blind re-search can't
+(the RLHF case), measured at an n large enough to separate that from variance.
+
+## 8. Reproduce
+
+```bash
+git clone https://github.com/tangle-network/agent-knowledge
+cd agent-knowledge && pnpm install
+
+# offline: the exam wiring + the $0 grader (no credentials)
+pnpm exec vitest run tests/loops/research-driving-ab.test.ts
+
+# the live 3-arm A/B — real web search + glm-5.2, per-arm cost reported
+export TANGLE_API_KEY=<router key with glm-5.2 credits>
+AGENT_KNOWLEDGE_LIVE=1 RQ_LIVE_BUDGET=4 \
+  pnpm exec vitest run tests/loops/research-driving-ab.test.ts -t "3-arm A/B"
+# re-run at RQ_LIVE_BUDGET=6 to see the verdict flip (the §4 variance point)
+
+# the controlled multi-round probe — forces N rounds so driving actually steers
+AGENT_KNOWLEDGE_LIVE=1 RQ_PROBE=1 RQ_PROBE_ROUNDS=3 TANGLE_API_KEY=<…> \
+  pnpm exec vitest run tests/loops/research-driving-ab.test.ts -t "multi-round probe"
+# (~$0.10 for the 5-topic A/B at one budget; ~$0.10 for the 3-topic probe)
+```
+
+`RQ_LIVE_TOPICS` takes a `|`-separated subset of the exam topic names to run a
+cheaper slice. The exam is held out by construction — no flag shows it to a loop.
+
+**Source:** the exam + grader —
+[`tests/loops/held-out-exam.ts`](../../tests/loops/held-out-exam.ts);
+the 3-arm A/B + multi-round probe —
+[`tests/loops/research-driving-ab.test.ts`](../../tests/loops/research-driving-ab.test.ts);
+the driving driver under test —
+[`src/research-driving-driver.ts`](../../src/research-driving-driver.ts).