@tangle-network/agent-knowledge 4.0.0 → 4.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (35) hide show
  1. package/AGENTS.md +32 -18
  2. package/CHANGELOG.md +32 -0
  3. package/README.md +31 -19
  4. package/dist/benchmarks/index.d.ts +51 -3
  5. package/dist/benchmarks/index.js +1 -1
  6. package/dist/{chunk-EUAXSBMH.js → chunk-AKYJG2MR.js} +224 -224
  7. package/dist/chunk-AKYJG2MR.js.map +1 -0
  8. package/dist/{chunk-DW6APRTX.js → chunk-BBCIB4UL.js} +2486 -2457
  9. package/dist/chunk-BBCIB4UL.js.map +1 -0
  10. package/dist/{chunk-UWOTQNBI.js → chunk-CPMLJYA3.js} +1886 -1842
  11. package/dist/chunk-CPMLJYA3.js.map +1 -0
  12. package/dist/{chunk-WCYW2GDA.js → chunk-MYFM6LKH.js} +2 -2
  13. package/dist/chunk-MYFM6LKH.js.map +1 -0
  14. package/dist/cli.js +1 -1
  15. package/dist/index.d.ts +48 -52
  16. package/dist/index.js +2906 -2874
  17. package/dist/index.js.map +1 -1
  18. package/dist/memory/index.d.ts +129 -6
  19. package/dist/memory/index.js +2 -2
  20. package/dist/sources/index.d.ts +17 -33
  21. package/dist/sources/index.js +1 -1
  22. package/dist/{index-Bqg1mBPt.d.ts → types-DP38encz.d.ts} +127 -284
  23. package/docs/eval/investment-material-facts.md +28 -29
  24. package/docs/eval/rag-eval-roadmap.md +6 -5
  25. package/docs/results/adaptive.md +13 -13
  26. package/docs/results/claim-grounding.md +11 -11
  27. package/docs/results/cost-quality.md +4 -4
  28. package/docs/results/investment-thesis.md +56 -56
  29. package/docs/results/research-driving.md +54 -54
  30. package/docs/two-agent-research-ab.md +94 -94
  31. package/package.json +4 -7
  32. package/dist/chunk-DW6APRTX.js.map +0 -1
  33. package/dist/chunk-EUAXSBMH.js.map +0 -1
  34. package/dist/chunk-UWOTQNBI.js.map +0 -1
  35. package/dist/chunk-WCYW2GDA.js.map +0 -1
@@ -1,25 +1,25 @@
1
- # From paper-retrieval to investigation — a research eval where one search is not enough, a metric that discriminates depth, and a 3-arm topology A/B
1
+ # Investment-research depth eval and three-arm comparison
2
2
 
3
3
  *Tangle Network · `agent-knowledge`*
4
4
 
5
- ## Verdict (BLUF)
5
+ ## Verdict
6
6
 
7
- We moved the research eval off ML-paper retrieval where a **single** web search
7
+ We moved the research eval off ML-paper retrieval, where a **single** web search
8
8
  already returns the answer, so the only thing the metric could measure was
9
- *collection* onto **investment research**, where the material facts are buried in
9
+ *collection*, onto **investment research**, where the material facts are buried in
10
10
  the footnotes of an SEC filing and genuinely require *investigation* to surface. On
11
11
  this harder domain we (1) **calibrated** a `$0`, model-free metric and proved it
12
- discriminates research depth a shallow ticker summary scores **1/27 (4%)**, a
13
- deep filings-grounded thesis scores **27/27 (100%)**, a **+96-point** gap and then
12
+ discriminates research depth, a shallow ticker summary scores **1/27 (4%)**, a
13
+ deep filings-grounded thesis scores **27/27 (100%)**, a **+96-point** gap, and then
14
14
  (2) ran **three research topologies head-to-head** on the same five held-out
15
15
  companies at matched compute, grading each one's knowledge base against a firewalled
16
16
  checklist of **27 buried material facts** a ticker search misses.
17
17
 
18
- **The honest A/B verdict: the research-DRIVING loop surfaced the most buried facts
19
- (16/27, 59%), +5 over blind collection (11/27, 41%) but at n=5 companies that lift
18
+ **A/B result: the research-driving loop surfaced the most buried facts
19
+ (16/27, 59%), +5 over blind collection (11/27, 41%), but at n=5 companies that lift
20
20
  is NOT statistically clean: the 95% confidence interval crosses zero (P(Δ≤0)=0.08).**
21
- The verify/dedup loop did **not** beat collection at all (10/27, 37% a wash,
22
- slightly worse). So no topology *significantly* beats collection here either —
21
+ The verify/dedup loop did **not** beat collection at all (10/27, 37%, a wash,
22
+ slightly worse). No approach *significantly* beats collection here.
23
23
  **driving is the only arm that even points the right way, and it points there
24
24
  suggestively, not significantly.** What the reframe *did* buy is a domain and a meter
25
25
  where the question is finally well-posed: the metric is no longer measuring whether a
@@ -27,7 +27,7 @@ single search ran, but whether investigation reached the buried fact.
27
27
 
28
28
  | arm | what the coordinator does | material facts | cost (5 cos.) | chats | searches | tokens |
29
29
  |---|---|---|---|---|---|---|
30
- | **A · collection** | nothing accepts every source (1 agent collects) | 11/27 (41%) | $0.082 | 10 | 20 | 64,248 |
30
+ | **A · collection** | nothing; accepts every source (1 agent collects) | 11/27 (41%) | $0.082 | 10 | 20 | 64,248 |
31
31
  | **B · verify/dedup** | LLM gates each source for relevance, rejects off-topic | 10/27 (37%) | $0.125 | 56 | 52 | 84,678 |
32
32
  | **C · driving (deepen)** | extracts claims, demands corroboration, asks deep follow-ups | **16/27 (59%)** | $0.157 | 33 | 28 | 124,387 |
33
33
 
@@ -35,17 +35,17 @@ Cost is real provenance, not an estimate: every `$`/call/token is diffed from
35
35
  `RouterClient.usage()` per company (the router's own `usage` field, priced at
36
36
  glm-5.2 rates), so "driving cost 1.9× collection" is measured, not modelled.
37
37
 
38
- ## 1. The domain reframe why we left ML-paper retrieval
38
+ ## 1. The domain reframe: why we left ML-paper retrieval
39
39
 
40
40
  The companion paper's depth eval (`docs/two-agent-research-ab.md` §9) measures a
41
- research loop on **20 deep questions across 5 ML topics**. That apparatus is sound
41
+ research loop on **20 deep questions across 5 ML topics**. That method is sound:
42
42
  its grader scores 0/20 on a one-line topic definition and 20/20 on a mechanism-rich
43
43
  paragraph, so it *can* tell depth from surface. But the topology A/B on top of it
44
- came back a clean null (driving 16/20 @ budget 4, 13/20 @ budget 6 the winner
44
+ came back a clean null (driving 16/20 @ budget 4, 13/20 @ budget 6, the winner
45
45
  *flips with the compute budget*, and the within-arm swing is as large as the
46
46
  between-arm gap). The autopsy named the cause: **on an ML topic a single good search
47
47
  already collects the answer.** Every arm finished in one effective round because the
48
- generic "one source closes the gap" readiness was met by the *first* fetch so the
48
+ generic "one source closes the gap" readiness was met by the *first* fetch, so the
49
49
  driving driver, whose entire mechanism is steering a *second* round, never got to
50
50
  act. When one search suffices, there is no investigation for a smarter coordinator to
51
51
  do, and the metric can only reward collection. That is the failure-mode-in-spirit the
@@ -55,12 +55,12 @@ easy for topology to matter.
55
55
  So we changed the domain to one where a single search **provably cannot** suffice.
56
56
  **Investment research**: give a loop a company + ticker + an as-of cutoff date and ask
57
57
  for a thesis; grade it on the buried, material, non-obvious drivers a thorough analyst
58
- flags and a ticker search misses. The decisive facts here live in 10-K footnotes an
58
+ flags and a ticker search misses. The decisive facts here live in 10-K footnotes, an
59
59
  HTM securities mark roughly equal to a bank's entire equity (SIVB), a deposit base
60
60
  that is 97% uninsured, a negative per-unit gross margin, a related-party lease. A web
61
61
  search for the company name returns "profitable regional bank" or "high-growth auto
62
62
  e-commerce"; the filing shows the mark-to-market hole the size of capital. **The
63
- answer is not collectable in one fetch it has to be investigated for.** That is the
63
+ answer is not collectable in one fetch, it has to be investigated for.** That is the
64
64
  property the ML domain lacked, and it is what makes a topology A/B finally meaningful:
65
65
  if a smarter coordinator can ever beat blind collection, a domain where the answer is
66
66
  buried is where it has the room to.
@@ -69,7 +69,7 @@ The held-out set is **5 companies, 27 material facts**, every fact read live out
69
69
  the primary SEC EDGAR 10-K during curation, every dollar figure quoted from the
70
70
  de-tagged filing text, every value knowable *at the cutoff* (the eventual collapse is
71
71
  recorded for the reader only and is **never graded**). The companies skew distressed /
72
- downside-risk a documented curation bias, not a hidden one (full provenance, the
72
+ downside-risk, a documented curation bias, not a hidden one (full provenance, the
73
73
  drop log, and the per-fact keyword groups: `docs/eval/investment-material-facts.md`):
74
74
 
75
75
  | ticker | company | as-of cutoff | CIK | facts |
@@ -80,25 +80,25 @@ drop log, and the per-fact keyword groups: `docs/eval/investment-material-facts.
80
80
  | PTON | Peloton | 2022-09-07 | 1639825 | 6 |
81
81
  | SI | Silvergate | 2022-02-28 | 1312109 | 5 |
82
82
 
83
- ## 2. Calibration does the metric discriminate depth? (the gate the ML exam passed only weakly)
83
+ ## 2. Calibration: does the metric discriminate depth? (the gate the ML exam passed only weakly)
84
84
 
85
85
  A topology A/B is meaningless unless the metric grading it can tell a *deep* thesis
86
- from a *shallow* one. If it can't, it is measuring word-collection the exact failure
87
- the reframe set out to escape and any A/B on top of it is noise. So **before** the
86
+ from a *shallow* one. If it can't, it is measuring word-collection, the exact failure
87
+ the reframe set out to escape, and any A/B on top of it is noise. So **before** the
88
88
  A/B, we ran a calibration gate (`$0`, offline, the binding result):
89
89
 
90
90
  For each of the 5 companies we hand-wrote two theses and scored each with the metric's
91
91
  pure core (`materialFactsSurfacedInText`):
92
92
 
93
- - **shallow** a one-paragraph ticker summary: what the company does, a vibe on the
93
+ - **shallow**: a one-paragraph ticker summary: what the company does, a vibe on the
94
94
  stock, generic macro/competition risk. The kind a single name-search returns. Names
95
95
  none of the buried, filing-level facts.
96
- - **deep** a filings-grounded analyst memo naming the buried drivers (the duration
96
+ - **deep**: a filings-grounded analyst memo naming the buried drivers (the duration
97
97
  loss, the buyback drain, the negative unit margin, the deposit concentration, the
98
98
  related party) in independent prose, with the real numbers.
99
99
 
100
100
  The grader is the same `$0`, model-free, case-insensitive substring check the held-out
101
- checklist ships (`gradeFactAgainstText`); the checklist is firewalled read only by
101
+ checklist ships (`gradeFactAgainstText`); the checklist is firewalled, read only by
102
102
  the metric, never shown to a loop.
103
103
 
104
104
  | ticker | shallow | deep | gap |
@@ -110,20 +110,20 @@ the metric, never shown to a loop.
110
110
  | SI | 0/5 (0%) | 5/5 (100%) | +100pp |
111
111
  | **total** | **1/27 (4%)** | **27/27 (100%)** | **+96pp** |
112
112
 
113
- **The metric is VALID:** it cleanly separates a shallow ticker-summary from a deep,
114
- filings-grounded thesis a +96-point aggregate gap, every company clearing the bars
113
+ **Calibration result:** the metric separates a shallow ticker-summary from a deep,
114
+ filings-grounded thesis, a +96-point aggregate gap, every company clearing the bars
115
115
  (shallow `< 30%`, deep `> 70%`) with wide margin. Two guards make the gap real and not
116
116
  a teaching-to-the-test artifact:
117
117
 
118
118
  - **Anti-circularity.** The gate asserts **no deep thesis verbatim-embeds any
119
- checklist `evidence` string** the deep theses state the same publicly-documented
119
+ checklist `evidence` string**, the deep theses state the same publicly-documented
120
120
  facts in independent analyst prose. A high deep score is the meter catching real,
121
121
  independently-phrased depth, not an answer-key echo.
122
- - **The one honest shallow hit.** The single shallow surface (SIVB 1/6) is SIVB/f5
123
- the innovation-economy / venture-client concentration which fires on "technology"
122
+ - **The one shallow hit.** The single shallow surface (SIVB 1/6) is SIVB/f5:
123
+ the innovation-economy / venture-client concentration, which fires on "technology"
124
124
  / "venture" in the SVB summary. That genuinely *is* the least-buried of SVB's six
125
125
  facts (a ticker search does return "SVB banks tech and venture startups"). We kept it
126
- as a 4% leak rather than tighten the group, because it honestly reflects one
126
+ as a 4% leak rather than tighten the group, because it reflects one
127
127
  near-surface fact while the five truly buried ones (the ~$15B HTM loss, the
128
128
  equity-sized mark, the $151.5B uninsured base, the 20-point deposit-mix shift, the
129
129
  AFS/AOCI loss) stay unsurfaced. The firewall holds (17% << 30%).
@@ -132,38 +132,38 @@ Calibration also did its job as more than a rubber stamp: the first pass had the
132
132
  **Silvergate shallow thesis scoring 3/5 (60%)**, blowing the bar, because three fact
133
133
  groups accepted bare crypto-bank vocabulary (`crypto`, `grew rapidly`, `proprietary` /
134
134
  `payment network`) that any one-line summary trips. We tightened those three to require
135
- the *buried* signal the deposit-**concentration** framing, the specific **$14.3B**
136
- figure, the **SEN** (Silvergate Exchange Network) name and dropped the generic vocab.
135
+ the *buried* signal, the deposit-**concentration** framing, the specific **$14.3B**
136
+ figure, the **SEN** (Silvergate Exchange Network) name, and dropped the generic vocab.
137
137
  The deep thesis still hits all three; the shallow one no longer does. The metric found
138
138
  a way it could be fooled and we closed it before any A/B depended on it.
139
139
 
140
140
  **This is the gate the ML exam passed only in spirit.** The ML grader discriminated
141
- 0/20 vs 20/20 too but on a domain where the deep answer was *collectable* in one
141
+ 0/20 vs 20/20 too, but on a domain where the deep answer was *collectable* in one
142
142
  search, so the discrimination measured grammar, not investigation. Here the +96-point
143
143
  gap is over facts that are *buried by construction*, so a high score is reachable only
144
144
  by reaching into the filing. Same shape of result, materially harder domain.
145
145
 
146
- ## 3. The 3-arm A/B what each topology surfaced, head-to-head
146
+ ## 3. The 3-arm A/B: what each topology surfaced, head-to-head
147
147
 
148
148
  For each company the loop is told **only** `{company, ticker, cik, cutoff}` plus a
149
149
  generic set of analyst-lens readiness specs (balance-sheet risk, concentration,
150
- leverage, margins, liquidity, governance, regulatory the *lenses* and where they
151
- live, the latest SEC 10-K **not the answers**). It researches the company *as of* the
150
+ leverage, margins, liquidity, governance, regulatory, the *lenses* and where they
151
+ live, the latest SEC 10-K, **not the answers**). It researches the company *as of* the
152
152
  cutoff over web + SEC EDGAR (both public), writes a thesis, and we grade the resulting
153
- KB with `materialFactsSurfaced` the firewalled checklist the loop never sees.
153
+ KB with `materialFactsSurfaced`, the firewalled checklist the loop never sees.
154
154
 
155
155
  **Compute is matched by construction.** All three arms run the *same* web worker, the
156
156
  *same* 3-round budget, the *same* worker config (`resultsPerQuery: 3, queriesPerGap: 1,
157
- maxSourcesPerRound: 6`). The **only** thing that varies is the driver the coordinator
157
+ maxSourcesPerRound: 6`). The **only** thing that varies is the driver, the coordinator
158
158
  between the worker and the knowledge base:
159
159
 
160
- - **A · collection** (`createCollectionResearchDriver`) an inert rubber stamp:
160
+ - **A · collection** (`createCollectionResearchDriver`), an inert rubber stamp:
161
161
  accepts every source, gates nothing, steers only with the loop's built-in open-gap
162
162
  list. The driver adds **zero** router calls. This is the blind-collection floor.
163
- - **B · verify/dedup** (`createVerifyingResearchDriver`) an LLM relevance gate: one
163
+ - **B · verify/dedup** (`createVerifyingResearchDriver`), an LLM relevance gate: one
164
164
  chat call per candidate source to accept-or-reject for on-topic relevance and
165
165
  near-duplication. The worker ADDS; the driver GATES.
166
- - **C · driving** (`createResearchDrivingDriver`) extracts each source's claims,
166
+ - **C · driving** (`createResearchDrivingDriver`), extracts each source's claims,
167
167
  tracks how many *independent* sources corroborate each, and synthesizes deep
168
168
  follow-up sub-questions (comparative / mechanism / gap / contradiction) it folds into
169
169
  the worker's next prompt to push depth and demand corroboration.
@@ -200,13 +200,13 @@ Every interval crosses zero. **Driving vs collection is the closest to clean
200
200
  coin flip. This is the project's well-documented small-n mirage: exciting deltas born
201
201
  at n=5 do not survive a paired bootstrap.
202
202
 
203
- ## 4. Autopsy the two things worth understanding
203
+ ## 4. Autopsy: the two things worth understanding
204
204
 
205
205
  ### 4.1 Why driving wins where it wins
206
206
 
207
207
  Driving's mechanism is multi-round: extract claims from round 1, then steer the worker
208
208
  in rounds 2–3 to corroborate the weak ones and chase the deep questions. It helps most
209
- where the **first** fetch lands real filing data the driver can build on the two
209
+ where the **first** fetch lands real filing data the driver can build on, the two
210
210
  banks, where SEC bank-call-report / 10-K data is dense and reachable. SIVB jumps from 2
211
211
  buried facts (collection) to 5 (driving): the duration loss, the deposit concentration,
212
212
  the AFS/AOCI mark all surface once the driver demands the balance-sheet detail a second
@@ -230,21 +230,21 @@ ROUND 1: accepted=0 rejected=3 writtenPages=0
230
230
  ROUND 2: accepted=0 rejected=2 writtenPages=0
231
231
  ```
232
232
 
233
- The verifier was **correct on the merits** those are aggregators, not the primary
233
+ The verifier was **correct on the merits**, those are aggregators, not the primary
234
234
  filing, and last10k showed a post-cutoff filing. But the worker never surfaced the
235
235
  EDGAR primary for BBBY, so a strict primary-only gate left the KB **empty**. Collection
236
236
  accepts the same aggregator pages and scores 4/5 on BBBY; driving accepts them and
237
237
  scores 5/5. **The gate's strictness is a liability when the worker's sourcing is
238
- imperfect:** it throws away the only evidence the loop had. This is a genuine topology
239
- trade-off worth naming, not a harness break the verify arm surfaced facts fine on
240
- CVNA (2), PTON (4), SI (4), so the harness works.
238
+ imperfect:** it throws away the only evidence the loop had. This is a real coordination
239
+ trade-off. The verify arm still surfaced facts on CVNA (2), PTON (4), and SI (4),
240
+ which rules out a general execution failure.
241
241
 
242
- ### 4.3 The empty-thesis caveat (honest)
242
+ ### 4.3 The empty-thesis caveat
243
243
 
244
244
  Some runs show `thesis=0ch` (empty synthesis) yet still score facts. That is expected:
245
- `materialFactsSurfaced` grades the **whole KB** (the worker's fetched `knowledge/*.md`
246
- pages), not only the final thesis page. glm-5.2 occasionally spends its entire output
247
- budget on hidden reasoning and returns empty visible content on the synthesis call a
245
+ `materialFactsSurfaced` grades the **whole KB**, including the worker's fetched
246
+ `knowledge/*.md` pages and the final thesis page. glm-5.2 occasionally spends its entire output
247
+ budget on hidden reasoning and returns empty visible content on the synthesis call, a
248
248
  known reasoning-model behavior we floor at 1200 tokens but can't fully eliminate. The
249
249
  score still reflects what the loop fetched, so an empty thesis does not invalidate a
250
250
  run; it just means the synthesis prose was lost while the curated evidence was not.
@@ -252,7 +252,7 @@ run; it just means the synthesis prose was lost while the curated evidence was n
252
252
  ## 5. What this does and does not establish
253
253
 
254
254
  - **Does**: the metric *discriminates depth on a domain where one search is not
255
- enough* (calibration: 1/27 shallow vs 27/27 deep, +96pp) so the topology A/B on top
255
+ enough* (calibration: 1/27 shallow vs 27/27 deep, +96pp), so the topology A/B on top
256
256
  of it is finally well-posed, unlike the ML retrieval exam where one search already
257
257
  collected the answer. At matched compute on the 5-company held-out set, the
258
258
  research-driving topology surfaced the most buried material facts (16/27 vs 11/27 for
@@ -272,10 +272,10 @@ run; it just means the synthesis prose was lost while the curated evidence was n
272
272
  ## 6. Reproduce
273
273
 
274
274
  ```bash
275
- # The calibration gate that must pass FIRST ($0, offline) the metric is valid.
275
+ # The calibration gate that must pass FIRST ($0, offline): the metric is valid.
276
276
  pnpm test investment-calibration
277
277
 
278
- # The offline task wiring ($0) proves page → index → grade end-to-end.
278
+ # The offline task wiring ($0): proves page → index → grade end-to-end.
279
279
  pnpm test investment-thesis-task
280
280
 
281
281
  # The full live 3-arm A/B (costs ~$0.36 total at 5 companies × 3 arms × 3 rounds).
@@ -291,12 +291,12 @@ AGENT_KNOWLEDGE_LIVE=1 IT_LIVE_TICKERS=CVNA IT_LIVE_ARMS=collection \
291
291
 
292
292
  `IT_LIVE_ARMS` (`|`-separated subset of `collection|verify|driving`) and
293
293
  `IT_LIVE_TICKERS` scope the run; `IT_LIVE_ROUNDS` sets the per-arm round budget
294
- (default 3 driving needs > 1). The smoke (one cheap glm-5.2 call) runs once before any
294
+ (default 3, driving needs > 1). The smoke (one cheap glm-5.2 call) runs once before any
295
295
  arm, so a bad key fails fast, before the burn.
296
296
 
297
297
  ---
298
298
 
299
- *Run provenance. Calibration: `$0`, offline, reproducible numbers from
299
+ *Run provenance. Calibration: `$0`, offline, reproducible, numbers from
300
300
  `tests/eval/investment-calibration.test.ts` (shallow `< 30%`, deep `> 70%`,
301
301
  gap `> 40pp` per company and aggregate; all pass). A/B: 5 companies × 3 arms × 3 rounds
302
302
  = 15 live company-runs, glm-5.2 over the Tangle router, ~37.5 min wall, $0.36 total;
@@ -1,10 +1,10 @@
1
- # Does driving research DEEPER beat just collecting it? A held-out deep-question A/B — and an honest null
1
+ # Research-driving versus collection: a held-out deep-question result
2
2
 
3
3
  *Tangle Network · `agent-knowledge`*
4
4
 
5
- ## Verdict (BLUF)
5
+ ## Verdict
6
6
 
7
- We built a research driver whose job is to push a web-research loop **deeper** —
7
+ We built a research driver whose job is to push a web-research loop **deeper**:
8
8
  extract each source's claims, demand a second independent source for every claim,
9
9
  generate comparative / mechanism / contradiction sub-questions, and steer the
10
10
  worker to chase them. The hypothesis: a KB built this way answers **more held-out
@@ -13,7 +13,7 @@ verifier.
13
13
 
14
14
  On a **firewalled exam of 20 deep questions across 5 ML topics**, graded with a
15
15
  $0 deterministic check the loop never sees, at equal compute, **the driving loop
16
- did NOT reliably beat plain collection and cost ~12–16× more.** The verdict
16
+ did NOT reliably beat plain collection, and cost ~12–16× more.** The verdict
17
17
  flips with the compute budget, which is the tell that it is **noise, not signal**:
18
18
 
19
19
  | arm | answered @ B=4 | answered @ B=6 | cost (5 topics) | tokens |
@@ -24,20 +24,20 @@ flips with the compute budget, which is the tell that it is **noise, not signal*
24
24
 
25
25
  Driving "wins" at B=4 (16 > 15 > 13) and "loses" at B=6 (13 < 15), while the
26
26
  single-agent arm itself swings 15→13 and driving swings 16→13 on the *same* exam.
27
- At n=5 topics a ±1–3 question difference is within the run-to-run web variance
27
+ At n=5 topics a ±1–3 question difference is within the run-to-run web variance;
28
28
  the arms are not separated by topology, they are separated by **which pages the
29
29
  web returned that minute**. The one thing that is stable and large is the **cost**:
30
30
  driving spends an order of magnitude more for no reliable quality gain.
31
31
 
32
32
  **Why** (the autopsy that explains it, §4): every arm finished in **one effective
33
- round** (`passes=2` on every topic, every budget). The generic readiness gate
34
- "one source closes the gap" is satisfied by the first round's fetch, so the
33
+ round** (`passes=2` on every topic, every budget). The generic readiness gate,
34
+ "one source closes the gap", is satisfied by the first round's fetch, so the
35
35
  loop *stops before the driving driver ever steers a second round*. Driving's
36
36
  entire mechanism is multi-round (extract → demand corroboration → re-search). So
37
- the headline A/B under-tests it. **We then gave it its fairest test** a
38
- controlled probe (§5) that *forces* 3 rounds so the driver actually steers and
37
+ the headline A/B under-tests it. **We then ran a controlled test**, a
38
+ controlled probe (§5) that *forces* 3 rounds so the driver actually steers, and
39
39
  it **still does not win: 8 vs 8 against a blind worker, at ~9× the cost.** The
40
- negative result is robust, not a gate artifact.
40
+ same result remains after controlling the stopping condition.
41
41
 
42
42
  This is a real negative result, and we report it as one.
43
43
 
@@ -45,7 +45,7 @@ This is a real negative result, and we report it as one.
45
45
 
46
46
  The prior A/B in this repo
47
47
  ([two-agent-research-ab.md](../two-agent-research-ab.md)) measured **cleanliness**
48
- how *few* sources a verifier admits at equal coverage. That is the right metric
48
+ - how *few* sources a verifier admits at equal coverage. That is the right metric
49
49
  for a verifier whose job is to filter. It is the **wrong** metric for the driving
50
50
  driver, whose thesis is the opposite: it is not trying to admit fewer sources, it
51
51
  is trying to build a KB that *answers more*. Measuring it on admitted-count would
@@ -60,21 +60,21 @@ So we need a metric for research **quality**, not hygiene. We use:
60
60
 
61
61
  [`tests/loops/held-out-exam.ts`](../../tests/loops/held-out-exam.ts). 5 ML topics,
62
62
  4 questions each = **20 deep questions**. Each is a *depth* question by
63
- construction comparative, mechanism-level, or contradiction-aware chosen so a
63
+ construction, comparative, mechanism-level, or contradiction-aware, chosen so a
64
64
  single web search for the topic name does **not** surface the answer:
65
65
 
66
- - *speculative decoding* the rejection-sampling acceptance rule and why output
66
+ - *speculative decoding*, the rejection-sampling acceptance rule and why output
67
67
  is lossless; how self-speculative/Medusa (no draft model) trades off vs the
68
68
  two-model scheme; what bounds the speedup; why one verify pass is ~free.
69
- - *LoRA* the `W = W₀ + BA` update and which matrices train; QLoRA's 4-bit
69
+ - *LoRA*, the `W = W₀ + BA` update and which matrices train; QLoRA's 4-bit
70
70
  tradeoff; why LoRA adds zero inference latency (merge); the very-low-rank claim.
71
- - *grouped-query attention* the MQA↔GQA↔MHA spectrum of KV heads; the real
71
+ - *grouped-query attention*, the MQA↔GQA↔MHA spectrum of KV heads; the real
72
72
  bottleneck (KV cache / memory bandwidth, not FLOPs); uptraining from a
73
73
  checkpoint; the MQA quality cost GQA recovers.
74
- - *RLHF/PPO* the KL-to-reference penalty and what it prevents; the pairwise
74
+ - *RLHF/PPO*, the KL-to-reference penalty and what it prevents; the pairwise
75
75
  preference reward model + Bradley-Terry loss; DPO's reward-model-free insight;
76
76
  PPO's clipped surrogate.
77
- - *mixture-of-experts* top-k routing and sparse activation; load imbalance + the
77
+ - *mixture-of-experts*, top-k routing and sparse activation; load imbalance + the
78
78
  auxiliary balancing loss; params-decoupled-from-FLOPs; the memory cost at
79
79
  inference.
80
80
 
@@ -83,7 +83,7 @@ any loop.** A loop is told only the topic name and the *same generic readiness
83
83
  specs* every arm gets ("what X is and how it works" / "results, mechanisms,
84
84
  trade-offs"). It researches blind. **After** it finishes, we grade the KB it built
85
85
  against the held-out questions with a **$0 deterministic substring grader** (no
86
- LLM so the exam cannot leak into a model the loop observes), where each question
86
+ LLM, so the exam cannot leak into a model the loop observes), where each question
87
87
  carries the specific load-bearing answer tokens as keyword groups (a number, a
88
88
  name, a mechanism phrase), with synonym groups so a faithful page in its own words
89
89
  still grades as answered.
@@ -95,21 +95,21 @@ still grades as answered.
95
95
  | a one-line topic-definition snippet (what a single search returns) | **0 / 20** |
96
96
  | a deep, mechanism-rich paragraph | **20 / 20** |
97
97
 
98
- So a high score is only reachable by depth the firewall is real, and the gap
98
+ So a high score is only reachable by depth, the firewall is real, and the gap
99
99
  between arms (if any) would be real depth, not grader slack.
100
100
 
101
101
  ## 3. The three arms, at equal compute
102
102
 
103
103
  [`tests/loops/research-driving-ab.test.ts`](../../tests/loops/research-driving-ab.test.ts).
104
104
  All three arms run the **same** real web worker
105
- ([`createWebResearchWorker`](../../src/web-research-worker.ts) glm-5.2 query-gen
105
+ ([`createWebResearchWorker`](../../src/web-research-worker.ts), glm-5.2 query-gen
106
106
  → live `/v1/search` → `politeFetch` → `htmlToText`). They differ **only** in the
107
107
  driver:
108
108
 
109
- - **(A) single-agent collection** the worker alone, no driver. It collects.
110
- - **(B) verify/dedup** [`createVerifyingResearchDriver`](../../src/web-research-worker.ts):
109
+ - **(A) single-agent collection**: the worker alone, no driver. It collects.
110
+ - **(B) verify/dedup**: [`createVerifyingResearchDriver`](../../src/web-research-worker.ts):
111
111
  a second glm-5.2 pass filters each source for relevance / near-duplicates.
112
- - **(C) DRIVING** [`createResearchDrivingDriver`](../../src/research-driving-driver.ts):
112
+ - **(C) DRIVING**: [`createResearchDrivingDriver`](../../src/research-driving-driver.ts):
113
113
  the driver extracts each source's claims (glm-5.2), tracks independent-source
114
114
  support + contradictions, and folds comparative / mechanism / gap /
115
115
  contradiction sub-questions into the worker's next prompt.
@@ -118,10 +118,10 @@ driver:
118
118
  single-agent iteration is 1 worker pass; a two-agent round is 1 worker + 1 driver
119
119
  pass = 2. Each arm gets the same pass ceiling B; the single-agent arm gets more
120
120
  iterations to spend the budget the two-agent arms burn on their driver. Cost is
121
- read per-arm from `RouterClient.usage()` measured dollars/tokens/calls, not
121
+ read per-arm from `RouterClient.usage()`, measured dollars/tokens/calls, not
122
122
  estimates.
123
123
 
124
- ## 4. Result driving does not reliably win, and the verdict flips with budget
124
+ ## 4. Result: driving does not reliably win, and the verdict flips with budget
125
125
 
126
126
  Per-topic held-out questions answered (out of 4 each), at both budgets:
127
127
 
@@ -134,15 +134,15 @@ Per-topic held-out questions answered (out of 4 each), at both budgets:
134
134
  | mixture-of-experts | 4 / 4 | 4 / 3 | 4 / 3 |
135
135
  | **total /20** | **13 / 15** | **15 / 15** | **16 / 13** |
136
136
 
137
- The driving arm answers **16** at B=4 and **13** at B=6 a 3-question swing on
137
+ The driving arm answers **16** at B=4 and **13** at B=6, a 3-question swing on
138
138
  the *same* exam, the same arm, just a different compute ceiling and a different
139
139
  minute of web results. Single swings 13→15; verify is flat 15→15. **The within-arm
140
140
  swing is as large as the between-arm gap**, which is the signature of a null:
141
141
  whatever separates a "win" from a "loss" here is web variance, not the driver.
142
142
 
143
143
  The **cost**, by contrast, is stable and large. Driving spends **~$0.084–0.089**
144
- across 5 topics vs **~$0.005–0.007** for single-agent **12–16× the dollars** and
145
- **~24× the tokens** (~70k vs ~2.5k) because it runs a claim-extraction LLM call
144
+ across 5 topics vs **~$0.005–0.007** for single-agent, **12–16× the dollars** and
145
+ **~24× the tokens** (~70k vs ~2.5k), because it runs a claim-extraction LLM call
146
146
  on every fetched source. For that 12–16× it buys no reliable held-out-question
147
147
  gain over plain collection.
148
148
 
@@ -151,18 +151,18 @@ gain over plain collection.
151
151
  The decisive diagnostic is `passes=2` on **every** topic at **every** budget: the
152
152
  two-agent loop ran exactly **one** worker round before the readiness gate reported
153
153
  done, even with B=6 budget for three rounds. The generic specs require one source
154
- to close a gap, and the worker's first-round fetch closes them so the loop stops
154
+ to close a gap, and the worker's first-round fetch closes them, so the loop stops
155
155
  before round 2. The driving driver's mechanism is *multi-round*: round 1 extracts
156
156
  claims and flags the weakly-supported ones; round 2+ is where it steers the worker
157
157
  to corroborate and go deeper. **It never got a round 2.** So in this setting
158
158
  driving's only active effect was a one-round claim-extraction tax with no chance to
159
- use what it extracted which is exactly what the numbers show: same answers as
159
+ use what it extracted, which is exactly what the numbers show: same answers as
160
160
  collection, much higher cost.
161
161
 
162
162
  That makes the equal-compute/generic-gate A/B a test of the wrong thing for the
163
163
  driving thesis. The fair test has to *force* multiple rounds.
164
164
 
165
- ## 5. Controlled multi-round probe — the driving thesis's fairest test
165
+ ## 5. Controlled multi-round probe
166
166
 
167
167
  To isolate "does depth-steering help **when it actually runs**?", we raise the
168
168
  readiness bar so the gate stays unmet and the loop runs the full round budget,
@@ -172,23 +172,23 @@ worker **re-searches the same gaps blind** (a no-op driver that accepts every
172
172
  source and never steers). If driving's steering has any value, this is where it
173
173
  shows.
174
174
 
175
- **Result 3 rounds, 3 topics, driving steers vs blind re-search:**
175
+ **Result, 3 rounds, 3 topics, driving steers vs blind re-search:**
176
176
 
177
177
  | topic | driving (steered) answered / cost | blind (no steer) answered / cost |
178
178
  |---|---|---|
179
- | speculative decoding | 2/4 $0.032 | **4/4 $0.004** |
180
- | LoRA | 3/4 $0.028 | 3/4 $0.003 |
181
- | RLHF / PPO | **3/4** $0.033 | 1/4 $0.003 |
182
- | **total /12** | **8/12 $0.093** | **8/12 $0.010** |
179
+ | speculative decoding | 2/4; $0.032 | **4/4; $0.004** |
180
+ | LoRA | 3/4; $0.028 | 3/4; $0.003 |
181
+ | RLHF / PPO | **3/4**; $0.033 | 1/4; $0.003 |
182
+ | **total /12** | **8/12; $0.093** | **8/12; $0.010** |
183
183
 
184
- **Steering does not help: 8 vs 8, at ~9× the cost.** Given its fairest test the
185
- full multi-round regime its mechanism was designed for the driving driver ties a
184
+ **Steering does not help: 8 vs 8, at ~9× the cost.** In the controlled test, the
185
+ full multi-round regime its mechanism was designed for, the driving driver ties a
186
186
  blind worker that just re-searches the same gaps three times. It is **better on
187
187
  RLHF** (3 vs 1, the one topic where chasing corroboration found a page blind
188
- re-search missed) and **worse on speculative decoding** (2 vs 4 steering pulled
188
+ re-search missed) and **worse on speculative decoding** (2 vs 4, steering pulled
189
189
  the worker *off* the pages that answered the exam and toward corroborating a
190
190
  narrower claim set). Those cancel. So the depth-steering does change *what* gets
191
- fetched, but not *how many* held-out questions get answered and it pays ~9× the
191
+ fetched, but not *how many* held-out questions get answered, and it pays ~9× the
192
192
  dollars for the privilege. The headline null (§4) is therefore **not** merely a
193
193
  gate artifact: even forced to run, driving does not beat blind collection on
194
194
  research quality at this n.
@@ -198,13 +198,13 @@ research quality at this n.
198
198
  - **Small n, high web variance.** n = 5 topics; one live run per arm per budget.
199
199
  The §4 budget-flip is itself the evidence that the per-run magnitudes are
200
200
  variance-bound. We did not run a paired bootstrap because the within-arm swing
201
- already exceeds the between-arm gap the honest read is "no separation," and a
201
+ already exceeds the between-arm gap, the supported conclusion is "no separation," and a
202
202
  significance test on a known-null at n=5 would dress it up, not clarify it.
203
- - **The gate, not the driver, ended the headline loop (§4) but the probe
203
+ - **The gate, not the driver, ended the headline loop (§4), but the probe
204
204
  controls for it.** The headline A/B's generic one-source readiness gate closed
205
205
  every loop after one round, so on its own it would only show "driving adds a
206
206
  one-round extraction tax." The §5 probe removes that confound by forcing 3
207
- rounds, and driving still ties blind (8 vs 8) so the null survives the fix,
207
+ rounds, and driving still ties blind (8 vs 8), so the null survives the fix,
208
208
  it is not an artifact of the permissive gate.
209
209
  - **The worker is shared and shallow.** All arms use the same ~500-line direct
210
210
  pipeline (query-gen → search → fetch), not an `AgentProfile` on a harness. A
@@ -220,15 +220,15 @@ research quality at this n.
220
220
  ## 7. What this says, plainly
221
221
 
222
222
  Adding a "drive it deeper" agent did **not** make the research measurably better at
223
- answering hard, held-out questions at equal compute (§4) *and* forced to run its
224
- full multi-round mechanism (§5), on this worker, at n=5 and it cost 9–16× more.
223
+ answering hard, held-out questions, at equal compute (§4) *and* forced to run its
224
+ full multi-round mechanism (§5), on this worker, at n=5, and it cost 9–16× more.
225
225
  The steering changes *what* gets fetched (it helped on RLHF, hurt on speculative
226
226
  decoding) but not *how many* held-out questions get answered. The most durable
227
- thing the experiment produced is the **measurement apparatus**: a firewalled
227
+ thing the experiment produced is the **measurement method**: a firewalled
228
228
  deep-question exam with a $0 deterministic grader that *can* tell depth from
229
229
  surface (0/20 vs 20/20), reusable for any future research-quality claim in this
230
- repo. The driving thesis that pursuing depth + corroboration beats plain
231
- collection is, on the evidence here, **not supported**; the cheaper paths
230
+ repo. The driving thesis, that pursuing depth + corroboration beats plain
231
+ collection, is, on the evidence here, **not supported**; the cheaper paths
232
232
  (collect, or dedup) match it. Where it might still earn its cost: a worker rich
233
233
  enough that "go corroborate this claim" reaches a page blind re-search can't
234
234
  (the RLHF case), measured at an n large enough to separate that from variance.
@@ -242,24 +242,24 @@ cd agent-knowledge && pnpm install
242
242
  # offline: the exam wiring + the $0 grader (no credentials)
243
243
  pnpm exec vitest run tests/loops/research-driving-ab.test.ts
244
244
 
245
- # the live 3-arm A/B real web search + glm-5.2, per-arm cost reported
245
+ # the live 3-arm A/B: real web search + glm-5.2, per-arm cost reported
246
246
  export TANGLE_API_KEY=<router key with glm-5.2 credits>
247
247
  AGENT_KNOWLEDGE_LIVE=1 RQ_LIVE_BUDGET=4 \
248
248
  pnpm exec vitest run tests/loops/research-driving-ab.test.ts -t "3-arm A/B"
249
249
  # re-run at RQ_LIVE_BUDGET=6 to see the verdict flip (the §4 variance point)
250
250
 
251
- # the controlled multi-round probe forces N rounds so driving actually steers
251
+ # the controlled multi-round probe: forces N rounds so driving actually steers
252
252
  AGENT_KNOWLEDGE_LIVE=1 RQ_PROBE=1 RQ_PROBE_ROUNDS=3 TANGLE_API_KEY=<…> \
253
253
  pnpm exec vitest run tests/loops/research-driving-ab.test.ts -t "multi-round probe"
254
254
  # (~$0.10 for the 5-topic A/B at one budget; ~$0.10 for the 3-topic probe)
255
255
  ```
256
256
 
257
257
  `RQ_LIVE_TOPICS` takes a `|`-separated subset of the exam topic names to run a
258
- cheaper slice. The exam is held out by construction no flag shows it to a loop.
258
+ cheaper slice. The exam is held out by construction, no flag shows it to a loop.
259
259
 
260
- **Source:** the exam + grader
260
+ **Source:** the exam and grader:
261
261
  [`tests/loops/held-out-exam.ts`](../../tests/loops/held-out-exam.ts);
262
- the 3-arm A/B + multi-round probe
262
+ the 3-arm A/B and multi-round probe:
263
263
  [`tests/loops/research-driving-ab.test.ts`](../../tests/loops/research-driving-ab.test.ts);
264
- the driving driver under test
264
+ the driving driver under test:
265
265
  [`src/research-driving-driver.ts`](../../src/research-driving-driver.ts).