@tangle-network/agent-knowledge 4.0.0 → 4.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENTS.md +32 -18
- package/CHANGELOG.md +32 -0
- package/README.md +31 -19
- package/dist/benchmarks/index.d.ts +51 -3
- package/dist/benchmarks/index.js +1 -1
- package/dist/{chunk-EUAXSBMH.js → chunk-AKYJG2MR.js} +224 -224
- package/dist/chunk-AKYJG2MR.js.map +1 -0
- package/dist/{chunk-DW6APRTX.js → chunk-BBCIB4UL.js} +2486 -2457
- package/dist/chunk-BBCIB4UL.js.map +1 -0
- package/dist/{chunk-UWOTQNBI.js → chunk-CPMLJYA3.js} +1886 -1842
- package/dist/chunk-CPMLJYA3.js.map +1 -0
- package/dist/{chunk-WCYW2GDA.js → chunk-MYFM6LKH.js} +2 -2
- package/dist/chunk-MYFM6LKH.js.map +1 -0
- package/dist/cli.js +1 -1
- package/dist/index.d.ts +48 -52
- package/dist/index.js +2906 -2874
- package/dist/index.js.map +1 -1
- package/dist/memory/index.d.ts +129 -6
- package/dist/memory/index.js +2 -2
- package/dist/sources/index.d.ts +17 -33
- package/dist/sources/index.js +1 -1
- package/dist/{index-Bqg1mBPt.d.ts → types-DP38encz.d.ts} +127 -284
- package/docs/eval/investment-material-facts.md +28 -29
- package/docs/eval/rag-eval-roadmap.md +6 -5
- package/docs/results/adaptive.md +13 -13
- package/docs/results/claim-grounding.md +11 -11
- package/docs/results/cost-quality.md +4 -4
- package/docs/results/investment-thesis.md +56 -56
- package/docs/results/research-driving.md +54 -54
- package/docs/two-agent-research-ab.md +94 -94
- package/package.json +4 -7
- package/dist/chunk-DW6APRTX.js.map +0 -1
- package/dist/chunk-EUAXSBMH.js.map +0 -1
- package/dist/chunk-UWOTQNBI.js.map +0 -1
- package/dist/chunk-WCYW2GDA.js.map +0 -1
|
@@ -1,25 +1,25 @@
|
|
|
1
|
-
#
|
|
1
|
+
# Investment-research depth eval and three-arm comparison
|
|
2
2
|
|
|
3
3
|
*Tangle Network · `agent-knowledge`*
|
|
4
4
|
|
|
5
|
-
## Verdict
|
|
5
|
+
## Verdict
|
|
6
6
|
|
|
7
|
-
We moved the research eval off ML-paper retrieval
|
|
7
|
+
We moved the research eval off ML-paper retrieval, where a **single** web search
|
|
8
8
|
already returns the answer, so the only thing the metric could measure was
|
|
9
|
-
*collection
|
|
9
|
+
*collection*, onto **investment research**, where the material facts are buried in
|
|
10
10
|
the footnotes of an SEC filing and genuinely require *investigation* to surface. On
|
|
11
11
|
this harder domain we (1) **calibrated** a `$0`, model-free metric and proved it
|
|
12
|
-
discriminates research depth
|
|
13
|
-
deep filings-grounded thesis scores **27/27 (100%)**, a **+96-point** gap
|
|
12
|
+
discriminates research depth, a shallow ticker summary scores **1/27 (4%)**, a
|
|
13
|
+
deep filings-grounded thesis scores **27/27 (100%)**, a **+96-point** gap, and then
|
|
14
14
|
(2) ran **three research topologies head-to-head** on the same five held-out
|
|
15
15
|
companies at matched compute, grading each one's knowledge base against a firewalled
|
|
16
16
|
checklist of **27 buried material facts** a ticker search misses.
|
|
17
17
|
|
|
18
|
-
**
|
|
19
|
-
(16/27, 59%), +5 over blind collection (11/27, 41%)
|
|
18
|
+
**A/B result: the research-driving loop surfaced the most buried facts
|
|
19
|
+
(16/27, 59%), +5 over blind collection (11/27, 41%), but at n=5 companies that lift
|
|
20
20
|
is NOT statistically clean: the 95% confidence interval crosses zero (P(Δ≤0)=0.08).**
|
|
21
|
-
The verify/dedup loop did **not** beat collection at all (10/27, 37
|
|
22
|
-
slightly worse).
|
|
21
|
+
The verify/dedup loop did **not** beat collection at all (10/27, 37%, a wash,
|
|
22
|
+
slightly worse). No approach *significantly* beats collection here.
|
|
23
23
|
**driving is the only arm that even points the right way, and it points there
|
|
24
24
|
suggestively, not significantly.** What the reframe *did* buy is a domain and a meter
|
|
25
25
|
where the question is finally well-posed: the metric is no longer measuring whether a
|
|
@@ -27,7 +27,7 @@ single search ran, but whether investigation reached the buried fact.
|
|
|
27
27
|
|
|
28
28
|
| arm | what the coordinator does | material facts | cost (5 cos.) | chats | searches | tokens |
|
|
29
29
|
|---|---|---|---|---|---|---|
|
|
30
|
-
| **A · collection** | nothing
|
|
30
|
+
| **A · collection** | nothing; accepts every source (1 agent collects) | 11/27 (41%) | $0.082 | 10 | 20 | 64,248 |
|
|
31
31
|
| **B · verify/dedup** | LLM gates each source for relevance, rejects off-topic | 10/27 (37%) | $0.125 | 56 | 52 | 84,678 |
|
|
32
32
|
| **C · driving (deepen)** | extracts claims, demands corroboration, asks deep follow-ups | **16/27 (59%)** | $0.157 | 33 | 28 | 124,387 |
|
|
33
33
|
|
|
@@ -35,17 +35,17 @@ Cost is real provenance, not an estimate: every `$`/call/token is diffed from
|
|
|
35
35
|
`RouterClient.usage()` per company (the router's own `usage` field, priced at
|
|
36
36
|
glm-5.2 rates), so "driving cost 1.9× collection" is measured, not modelled.
|
|
37
37
|
|
|
38
|
-
## 1. The domain reframe
|
|
38
|
+
## 1. The domain reframe: why we left ML-paper retrieval
|
|
39
39
|
|
|
40
40
|
The companion paper's depth eval (`docs/two-agent-research-ab.md` §9) measures a
|
|
41
|
-
research loop on **20 deep questions across 5 ML topics**. That
|
|
41
|
+
research loop on **20 deep questions across 5 ML topics**. That method is sound:
|
|
42
42
|
its grader scores 0/20 on a one-line topic definition and 20/20 on a mechanism-rich
|
|
43
43
|
paragraph, so it *can* tell depth from surface. But the topology A/B on top of it
|
|
44
|
-
came back a clean null (driving 16/20 @ budget 4, 13/20 @ budget 6
|
|
44
|
+
came back a clean null (driving 16/20 @ budget 4, 13/20 @ budget 6, the winner
|
|
45
45
|
*flips with the compute budget*, and the within-arm swing is as large as the
|
|
46
46
|
between-arm gap). The autopsy named the cause: **on an ML topic a single good search
|
|
47
47
|
already collects the answer.** Every arm finished in one effective round because the
|
|
48
|
-
generic "one source closes the gap" readiness was met by the *first* fetch
|
|
48
|
+
generic "one source closes the gap" readiness was met by the *first* fetch, so the
|
|
49
49
|
driving driver, whose entire mechanism is steering a *second* round, never got to
|
|
50
50
|
act. When one search suffices, there is no investigation for a smarter coordinator to
|
|
51
51
|
do, and the metric can only reward collection. That is the failure-mode-in-spirit the
|
|
@@ -55,12 +55,12 @@ easy for topology to matter.
|
|
|
55
55
|
So we changed the domain to one where a single search **provably cannot** suffice.
|
|
56
56
|
**Investment research**: give a loop a company + ticker + an as-of cutoff date and ask
|
|
57
57
|
for a thesis; grade it on the buried, material, non-obvious drivers a thorough analyst
|
|
58
|
-
flags and a ticker search misses. The decisive facts here live in 10-K footnotes
|
|
58
|
+
flags and a ticker search misses. The decisive facts here live in 10-K footnotes, an
|
|
59
59
|
HTM securities mark roughly equal to a bank's entire equity (SIVB), a deposit base
|
|
60
60
|
that is 97% uninsured, a negative per-unit gross margin, a related-party lease. A web
|
|
61
61
|
search for the company name returns "profitable regional bank" or "high-growth auto
|
|
62
62
|
e-commerce"; the filing shows the mark-to-market hole the size of capital. **The
|
|
63
|
-
answer is not collectable in one fetch
|
|
63
|
+
answer is not collectable in one fetch, it has to be investigated for.** That is the
|
|
64
64
|
property the ML domain lacked, and it is what makes a topology A/B finally meaningful:
|
|
65
65
|
if a smarter coordinator can ever beat blind collection, a domain where the answer is
|
|
66
66
|
buried is where it has the room to.
|
|
@@ -69,7 +69,7 @@ The held-out set is **5 companies, 27 material facts**, every fact read live out
|
|
|
69
69
|
the primary SEC EDGAR 10-K during curation, every dollar figure quoted from the
|
|
70
70
|
de-tagged filing text, every value knowable *at the cutoff* (the eventual collapse is
|
|
71
71
|
recorded for the reader only and is **never graded**). The companies skew distressed /
|
|
72
|
-
downside-risk
|
|
72
|
+
downside-risk, a documented curation bias, not a hidden one (full provenance, the
|
|
73
73
|
drop log, and the per-fact keyword groups: `docs/eval/investment-material-facts.md`):
|
|
74
74
|
|
|
75
75
|
| ticker | company | as-of cutoff | CIK | facts |
|
|
@@ -80,25 +80,25 @@ drop log, and the per-fact keyword groups: `docs/eval/investment-material-facts.
|
|
|
80
80
|
| PTON | Peloton | 2022-09-07 | 1639825 | 6 |
|
|
81
81
|
| SI | Silvergate | 2022-02-28 | 1312109 | 5 |
|
|
82
82
|
|
|
83
|
-
## 2. Calibration
|
|
83
|
+
## 2. Calibration: does the metric discriminate depth? (the gate the ML exam passed only weakly)
|
|
84
84
|
|
|
85
85
|
A topology A/B is meaningless unless the metric grading it can tell a *deep* thesis
|
|
86
|
-
from a *shallow* one. If it can't, it is measuring word-collection
|
|
87
|
-
the reframe set out to escape
|
|
86
|
+
from a *shallow* one. If it can't, it is measuring word-collection, the exact failure
|
|
87
|
+
the reframe set out to escape, and any A/B on top of it is noise. So **before** the
|
|
88
88
|
A/B, we ran a calibration gate (`$0`, offline, the binding result):
|
|
89
89
|
|
|
90
90
|
For each of the 5 companies we hand-wrote two theses and scored each with the metric's
|
|
91
91
|
pure core (`materialFactsSurfacedInText`):
|
|
92
92
|
|
|
93
|
-
- **shallow
|
|
93
|
+
- **shallow**: a one-paragraph ticker summary: what the company does, a vibe on the
|
|
94
94
|
stock, generic macro/competition risk. The kind a single name-search returns. Names
|
|
95
95
|
none of the buried, filing-level facts.
|
|
96
|
-
- **deep
|
|
96
|
+
- **deep**: a filings-grounded analyst memo naming the buried drivers (the duration
|
|
97
97
|
loss, the buyback drain, the negative unit margin, the deposit concentration, the
|
|
98
98
|
related party) in independent prose, with the real numbers.
|
|
99
99
|
|
|
100
100
|
The grader is the same `$0`, model-free, case-insensitive substring check the held-out
|
|
101
|
-
checklist ships (`gradeFactAgainstText`); the checklist is firewalled
|
|
101
|
+
checklist ships (`gradeFactAgainstText`); the checklist is firewalled, read only by
|
|
102
102
|
the metric, never shown to a loop.
|
|
103
103
|
|
|
104
104
|
| ticker | shallow | deep | gap |
|
|
@@ -110,20 +110,20 @@ the metric, never shown to a loop.
|
|
|
110
110
|
| SI | 0/5 (0%) | 5/5 (100%) | +100pp |
|
|
111
111
|
| **total** | **1/27 (4%)** | **27/27 (100%)** | **+96pp** |
|
|
112
112
|
|
|
113
|
-
**
|
|
114
|
-
filings-grounded thesis
|
|
113
|
+
**Calibration result:** the metric separates a shallow ticker-summary from a deep,
|
|
114
|
+
filings-grounded thesis, a +96-point aggregate gap, every company clearing the bars
|
|
115
115
|
(shallow `< 30%`, deep `> 70%`) with wide margin. Two guards make the gap real and not
|
|
116
116
|
a teaching-to-the-test artifact:
|
|
117
117
|
|
|
118
118
|
- **Anti-circularity.** The gate asserts **no deep thesis verbatim-embeds any
|
|
119
|
-
checklist `evidence` string
|
|
119
|
+
checklist `evidence` string**, the deep theses state the same publicly-documented
|
|
120
120
|
facts in independent analyst prose. A high deep score is the meter catching real,
|
|
121
121
|
independently-phrased depth, not an answer-key echo.
|
|
122
|
-
- **The one
|
|
123
|
-
the innovation-economy / venture-client concentration
|
|
122
|
+
- **The one shallow hit.** The single shallow surface (SIVB 1/6) is SIVB/f5:
|
|
123
|
+
the innovation-economy / venture-client concentration, which fires on "technology"
|
|
124
124
|
/ "venture" in the SVB summary. That genuinely *is* the least-buried of SVB's six
|
|
125
125
|
facts (a ticker search does return "SVB banks tech and venture startups"). We kept it
|
|
126
|
-
as a 4% leak rather than tighten the group, because it
|
|
126
|
+
as a 4% leak rather than tighten the group, because it reflects one
|
|
127
127
|
near-surface fact while the five truly buried ones (the ~$15B HTM loss, the
|
|
128
128
|
equity-sized mark, the $151.5B uninsured base, the 20-point deposit-mix shift, the
|
|
129
129
|
AFS/AOCI loss) stay unsurfaced. The firewall holds (17% << 30%).
|
|
@@ -132,38 +132,38 @@ Calibration also did its job as more than a rubber stamp: the first pass had the
|
|
|
132
132
|
**Silvergate shallow thesis scoring 3/5 (60%)**, blowing the bar, because three fact
|
|
133
133
|
groups accepted bare crypto-bank vocabulary (`crypto`, `grew rapidly`, `proprietary` /
|
|
134
134
|
`payment network`) that any one-line summary trips. We tightened those three to require
|
|
135
|
-
the *buried* signal
|
|
136
|
-
figure, the **SEN** (Silvergate Exchange Network) name
|
|
135
|
+
the *buried* signal, the deposit-**concentration** framing, the specific **$14.3B**
|
|
136
|
+
figure, the **SEN** (Silvergate Exchange Network) name, and dropped the generic vocab.
|
|
137
137
|
The deep thesis still hits all three; the shallow one no longer does. The metric found
|
|
138
138
|
a way it could be fooled and we closed it before any A/B depended on it.
|
|
139
139
|
|
|
140
140
|
**This is the gate the ML exam passed only in spirit.** The ML grader discriminated
|
|
141
|
-
0/20 vs 20/20 too
|
|
141
|
+
0/20 vs 20/20 too, but on a domain where the deep answer was *collectable* in one
|
|
142
142
|
search, so the discrimination measured grammar, not investigation. Here the +96-point
|
|
143
143
|
gap is over facts that are *buried by construction*, so a high score is reachable only
|
|
144
144
|
by reaching into the filing. Same shape of result, materially harder domain.
|
|
145
145
|
|
|
146
|
-
## 3. The 3-arm A/B
|
|
146
|
+
## 3. The 3-arm A/B: what each topology surfaced, head-to-head
|
|
147
147
|
|
|
148
148
|
For each company the loop is told **only** `{company, ticker, cik, cutoff}` plus a
|
|
149
149
|
generic set of analyst-lens readiness specs (balance-sheet risk, concentration,
|
|
150
|
-
leverage, margins, liquidity, governance, regulatory
|
|
151
|
-
live, the latest SEC 10-K
|
|
150
|
+
leverage, margins, liquidity, governance, regulatory, the *lenses* and where they
|
|
151
|
+
live, the latest SEC 10-K, **not the answers**). It researches the company *as of* the
|
|
152
152
|
cutoff over web + SEC EDGAR (both public), writes a thesis, and we grade the resulting
|
|
153
|
-
KB with `materialFactsSurfaced
|
|
153
|
+
KB with `materialFactsSurfaced`, the firewalled checklist the loop never sees.
|
|
154
154
|
|
|
155
155
|
**Compute is matched by construction.** All three arms run the *same* web worker, the
|
|
156
156
|
*same* 3-round budget, the *same* worker config (`resultsPerQuery: 3, queriesPerGap: 1,
|
|
157
|
-
maxSourcesPerRound: 6`). The **only** thing that varies is the driver
|
|
157
|
+
maxSourcesPerRound: 6`). The **only** thing that varies is the driver, the coordinator
|
|
158
158
|
between the worker and the knowledge base:
|
|
159
159
|
|
|
160
|
-
- **A · collection** (`createCollectionResearchDriver`)
|
|
160
|
+
- **A · collection** (`createCollectionResearchDriver`), an inert rubber stamp:
|
|
161
161
|
accepts every source, gates nothing, steers only with the loop's built-in open-gap
|
|
162
162
|
list. The driver adds **zero** router calls. This is the blind-collection floor.
|
|
163
|
-
- **B · verify/dedup** (`createVerifyingResearchDriver`)
|
|
163
|
+
- **B · verify/dedup** (`createVerifyingResearchDriver`), an LLM relevance gate: one
|
|
164
164
|
chat call per candidate source to accept-or-reject for on-topic relevance and
|
|
165
165
|
near-duplication. The worker ADDS; the driver GATES.
|
|
166
|
-
- **C · driving** (`createResearchDrivingDriver`)
|
|
166
|
+
- **C · driving** (`createResearchDrivingDriver`), extracts each source's claims,
|
|
167
167
|
tracks how many *independent* sources corroborate each, and synthesizes deep
|
|
168
168
|
follow-up sub-questions (comparative / mechanism / gap / contradiction) it folds into
|
|
169
169
|
the worker's next prompt to push depth and demand corroboration.
|
|
@@ -200,13 +200,13 @@ Every interval crosses zero. **Driving vs collection is the closest to clean
|
|
|
200
200
|
coin flip. This is the project's well-documented small-n mirage: exciting deltas born
|
|
201
201
|
at n=5 do not survive a paired bootstrap.
|
|
202
202
|
|
|
203
|
-
## 4. Autopsy
|
|
203
|
+
## 4. Autopsy: the two things worth understanding
|
|
204
204
|
|
|
205
205
|
### 4.1 Why driving wins where it wins
|
|
206
206
|
|
|
207
207
|
Driving's mechanism is multi-round: extract claims from round 1, then steer the worker
|
|
208
208
|
in rounds 2–3 to corroborate the weak ones and chase the deep questions. It helps most
|
|
209
|
-
where the **first** fetch lands real filing data the driver can build on
|
|
209
|
+
where the **first** fetch lands real filing data the driver can build on, the two
|
|
210
210
|
banks, where SEC bank-call-report / 10-K data is dense and reachable. SIVB jumps from 2
|
|
211
211
|
buried facts (collection) to 5 (driving): the duration loss, the deposit concentration,
|
|
212
212
|
the AFS/AOCI mark all surface once the driver demands the balance-sheet detail a second
|
|
@@ -230,21 +230,21 @@ ROUND 1: accepted=0 rejected=3 writtenPages=0
|
|
|
230
230
|
ROUND 2: accepted=0 rejected=2 writtenPages=0
|
|
231
231
|
```
|
|
232
232
|
|
|
233
|
-
The verifier was **correct on the merits
|
|
233
|
+
The verifier was **correct on the merits**, those are aggregators, not the primary
|
|
234
234
|
filing, and last10k showed a post-cutoff filing. But the worker never surfaced the
|
|
235
235
|
EDGAR primary for BBBY, so a strict primary-only gate left the KB **empty**. Collection
|
|
236
236
|
accepts the same aggregator pages and scores 4/5 on BBBY; driving accepts them and
|
|
237
237
|
scores 5/5. **The gate's strictness is a liability when the worker's sourcing is
|
|
238
|
-
imperfect:** it throws away the only evidence the loop had. This is a
|
|
239
|
-
trade-off
|
|
240
|
-
|
|
238
|
+
imperfect:** it throws away the only evidence the loop had. This is a real coordination
|
|
239
|
+
trade-off. The verify arm still surfaced facts on CVNA (2), PTON (4), and SI (4),
|
|
240
|
+
which rules out a general execution failure.
|
|
241
241
|
|
|
242
|
-
### 4.3 The empty-thesis caveat
|
|
242
|
+
### 4.3 The empty-thesis caveat
|
|
243
243
|
|
|
244
244
|
Some runs show `thesis=0ch` (empty synthesis) yet still score facts. That is expected:
|
|
245
|
-
`materialFactsSurfaced` grades the **whole KB
|
|
246
|
-
pages
|
|
247
|
-
budget on hidden reasoning and returns empty visible content on the synthesis call
|
|
245
|
+
`materialFactsSurfaced` grades the **whole KB**, including the worker's fetched
|
|
246
|
+
`knowledge/*.md` pages and the final thesis page. glm-5.2 occasionally spends its entire output
|
|
247
|
+
budget on hidden reasoning and returns empty visible content on the synthesis call, a
|
|
248
248
|
known reasoning-model behavior we floor at 1200 tokens but can't fully eliminate. The
|
|
249
249
|
score still reflects what the loop fetched, so an empty thesis does not invalidate a
|
|
250
250
|
run; it just means the synthesis prose was lost while the curated evidence was not.
|
|
@@ -252,7 +252,7 @@ run; it just means the synthesis prose was lost while the curated evidence was n
|
|
|
252
252
|
## 5. What this does and does not establish
|
|
253
253
|
|
|
254
254
|
- **Does**: the metric *discriminates depth on a domain where one search is not
|
|
255
|
-
enough* (calibration: 1/27 shallow vs 27/27 deep, +96pp)
|
|
255
|
+
enough* (calibration: 1/27 shallow vs 27/27 deep, +96pp), so the topology A/B on top
|
|
256
256
|
of it is finally well-posed, unlike the ML retrieval exam where one search already
|
|
257
257
|
collected the answer. At matched compute on the 5-company held-out set, the
|
|
258
258
|
research-driving topology surfaced the most buried material facts (16/27 vs 11/27 for
|
|
@@ -272,10 +272,10 @@ run; it just means the synthesis prose was lost while the curated evidence was n
|
|
|
272
272
|
## 6. Reproduce
|
|
273
273
|
|
|
274
274
|
```bash
|
|
275
|
-
# The calibration gate that must pass FIRST ($0, offline)
|
|
275
|
+
# The calibration gate that must pass FIRST ($0, offline): the metric is valid.
|
|
276
276
|
pnpm test investment-calibration
|
|
277
277
|
|
|
278
|
-
# The offline task wiring ($0)
|
|
278
|
+
# The offline task wiring ($0): proves page → index → grade end-to-end.
|
|
279
279
|
pnpm test investment-thesis-task
|
|
280
280
|
|
|
281
281
|
# The full live 3-arm A/B (costs ~$0.36 total at 5 companies × 3 arms × 3 rounds).
|
|
@@ -291,12 +291,12 @@ AGENT_KNOWLEDGE_LIVE=1 IT_LIVE_TICKERS=CVNA IT_LIVE_ARMS=collection \
|
|
|
291
291
|
|
|
292
292
|
`IT_LIVE_ARMS` (`|`-separated subset of `collection|verify|driving`) and
|
|
293
293
|
`IT_LIVE_TICKERS` scope the run; `IT_LIVE_ROUNDS` sets the per-arm round budget
|
|
294
|
-
(default 3
|
|
294
|
+
(default 3, driving needs > 1). The smoke (one cheap glm-5.2 call) runs once before any
|
|
295
295
|
arm, so a bad key fails fast, before the burn.
|
|
296
296
|
|
|
297
297
|
---
|
|
298
298
|
|
|
299
|
-
*Run provenance. Calibration: `$0`, offline, reproducible
|
|
299
|
+
*Run provenance. Calibration: `$0`, offline, reproducible, numbers from
|
|
300
300
|
`tests/eval/investment-calibration.test.ts` (shallow `< 30%`, deep `> 70%`,
|
|
301
301
|
gap `> 40pp` per company and aggregate; all pass). A/B: 5 companies × 3 arms × 3 rounds
|
|
302
302
|
= 15 live company-runs, glm-5.2 over the Tangle router, ~37.5 min wall, $0.36 total;
|
|
@@ -1,10 +1,10 @@
|
|
|
1
|
-
#
|
|
1
|
+
# Research-driving versus collection: a held-out deep-question result
|
|
2
2
|
|
|
3
3
|
*Tangle Network · `agent-knowledge`*
|
|
4
4
|
|
|
5
|
-
## Verdict
|
|
5
|
+
## Verdict
|
|
6
6
|
|
|
7
|
-
We built a research driver whose job is to push a web-research loop **deeper
|
|
7
|
+
We built a research driver whose job is to push a web-research loop **deeper**:
|
|
8
8
|
extract each source's claims, demand a second independent source for every claim,
|
|
9
9
|
generate comparative / mechanism / contradiction sub-questions, and steer the
|
|
10
10
|
worker to chase them. The hypothesis: a KB built this way answers **more held-out
|
|
@@ -13,7 +13,7 @@ verifier.
|
|
|
13
13
|
|
|
14
14
|
On a **firewalled exam of 20 deep questions across 5 ML topics**, graded with a
|
|
15
15
|
$0 deterministic check the loop never sees, at equal compute, **the driving loop
|
|
16
|
-
did NOT reliably beat plain collection
|
|
16
|
+
did NOT reliably beat plain collection, and cost ~12–16× more.** The verdict
|
|
17
17
|
flips with the compute budget, which is the tell that it is **noise, not signal**:
|
|
18
18
|
|
|
19
19
|
| arm | answered @ B=4 | answered @ B=6 | cost (5 topics) | tokens |
|
|
@@ -24,20 +24,20 @@ flips with the compute budget, which is the tell that it is **noise, not signal*
|
|
|
24
24
|
|
|
25
25
|
Driving "wins" at B=4 (16 > 15 > 13) and "loses" at B=6 (13 < 15), while the
|
|
26
26
|
single-agent arm itself swings 15→13 and driving swings 16→13 on the *same* exam.
|
|
27
|
-
At n=5 topics a ±1–3 question difference is within the run-to-run web variance
|
|
27
|
+
At n=5 topics a ±1–3 question difference is within the run-to-run web variance;
|
|
28
28
|
the arms are not separated by topology, they are separated by **which pages the
|
|
29
29
|
web returned that minute**. The one thing that is stable and large is the **cost**:
|
|
30
30
|
driving spends an order of magnitude more for no reliable quality gain.
|
|
31
31
|
|
|
32
32
|
**Why** (the autopsy that explains it, §4): every arm finished in **one effective
|
|
33
|
-
round** (`passes=2` on every topic, every budget). The generic readiness gate
|
|
34
|
-
"one source closes the gap"
|
|
33
|
+
round** (`passes=2` on every topic, every budget). The generic readiness gate,
|
|
34
|
+
"one source closes the gap", is satisfied by the first round's fetch, so the
|
|
35
35
|
loop *stops before the driving driver ever steers a second round*. Driving's
|
|
36
36
|
entire mechanism is multi-round (extract → demand corroboration → re-search). So
|
|
37
|
-
the headline A/B under-tests it. **We then
|
|
38
|
-
controlled probe (§5) that *forces* 3 rounds so the driver actually steers
|
|
37
|
+
the headline A/B under-tests it. **We then ran a controlled test**, a
|
|
38
|
+
controlled probe (§5) that *forces* 3 rounds so the driver actually steers, and
|
|
39
39
|
it **still does not win: 8 vs 8 against a blind worker, at ~9× the cost.** The
|
|
40
|
-
|
|
40
|
+
same result remains after controlling the stopping condition.
|
|
41
41
|
|
|
42
42
|
This is a real negative result, and we report it as one.
|
|
43
43
|
|
|
@@ -45,7 +45,7 @@ This is a real negative result, and we report it as one.
|
|
|
45
45
|
|
|
46
46
|
The prior A/B in this repo
|
|
47
47
|
([two-agent-research-ab.md](../two-agent-research-ab.md)) measured **cleanliness**
|
|
48
|
-
|
|
48
|
+
- how *few* sources a verifier admits at equal coverage. That is the right metric
|
|
49
49
|
for a verifier whose job is to filter. It is the **wrong** metric for the driving
|
|
50
50
|
driver, whose thesis is the opposite: it is not trying to admit fewer sources, it
|
|
51
51
|
is trying to build a KB that *answers more*. Measuring it on admitted-count would
|
|
@@ -60,21 +60,21 @@ So we need a metric for research **quality**, not hygiene. We use:
|
|
|
60
60
|
|
|
61
61
|
[`tests/loops/held-out-exam.ts`](../../tests/loops/held-out-exam.ts). 5 ML topics,
|
|
62
62
|
4 questions each = **20 deep questions**. Each is a *depth* question by
|
|
63
|
-
construction
|
|
63
|
+
construction, comparative, mechanism-level, or contradiction-aware, chosen so a
|
|
64
64
|
single web search for the topic name does **not** surface the answer:
|
|
65
65
|
|
|
66
|
-
- *speculative decoding
|
|
66
|
+
- *speculative decoding*, the rejection-sampling acceptance rule and why output
|
|
67
67
|
is lossless; how self-speculative/Medusa (no draft model) trades off vs the
|
|
68
68
|
two-model scheme; what bounds the speedup; why one verify pass is ~free.
|
|
69
|
-
- *LoRA
|
|
69
|
+
- *LoRA*, the `W = W₀ + BA` update and which matrices train; QLoRA's 4-bit
|
|
70
70
|
tradeoff; why LoRA adds zero inference latency (merge); the very-low-rank claim.
|
|
71
|
-
- *grouped-query attention
|
|
71
|
+
- *grouped-query attention*, the MQA↔GQA↔MHA spectrum of KV heads; the real
|
|
72
72
|
bottleneck (KV cache / memory bandwidth, not FLOPs); uptraining from a
|
|
73
73
|
checkpoint; the MQA quality cost GQA recovers.
|
|
74
|
-
- *RLHF/PPO
|
|
74
|
+
- *RLHF/PPO*, the KL-to-reference penalty and what it prevents; the pairwise
|
|
75
75
|
preference reward model + Bradley-Terry loss; DPO's reward-model-free insight;
|
|
76
76
|
PPO's clipped surrogate.
|
|
77
|
-
- *mixture-of-experts
|
|
77
|
+
- *mixture-of-experts*, top-k routing and sparse activation; load imbalance + the
|
|
78
78
|
auxiliary balancing loss; params-decoupled-from-FLOPs; the memory cost at
|
|
79
79
|
inference.
|
|
80
80
|
|
|
@@ -83,7 +83,7 @@ any loop.** A loop is told only the topic name and the *same generic readiness
|
|
|
83
83
|
specs* every arm gets ("what X is and how it works" / "results, mechanisms,
|
|
84
84
|
trade-offs"). It researches blind. **After** it finishes, we grade the KB it built
|
|
85
85
|
against the held-out questions with a **$0 deterministic substring grader** (no
|
|
86
|
-
LLM
|
|
86
|
+
LLM, so the exam cannot leak into a model the loop observes), where each question
|
|
87
87
|
carries the specific load-bearing answer tokens as keyword groups (a number, a
|
|
88
88
|
name, a mechanism phrase), with synonym groups so a faithful page in its own words
|
|
89
89
|
still grades as answered.
|
|
@@ -95,21 +95,21 @@ still grades as answered.
|
|
|
95
95
|
| a one-line topic-definition snippet (what a single search returns) | **0 / 20** |
|
|
96
96
|
| a deep, mechanism-rich paragraph | **20 / 20** |
|
|
97
97
|
|
|
98
|
-
So a high score is only reachable by depth
|
|
98
|
+
So a high score is only reachable by depth, the firewall is real, and the gap
|
|
99
99
|
between arms (if any) would be real depth, not grader slack.
|
|
100
100
|
|
|
101
101
|
## 3. The three arms, at equal compute
|
|
102
102
|
|
|
103
103
|
[`tests/loops/research-driving-ab.test.ts`](../../tests/loops/research-driving-ab.test.ts).
|
|
104
104
|
All three arms run the **same** real web worker
|
|
105
|
-
([`createWebResearchWorker`](../../src/web-research-worker.ts)
|
|
105
|
+
([`createWebResearchWorker`](../../src/web-research-worker.ts), glm-5.2 query-gen
|
|
106
106
|
→ live `/v1/search` → `politeFetch` → `htmlToText`). They differ **only** in the
|
|
107
107
|
driver:
|
|
108
108
|
|
|
109
|
-
- **(A) single-agent collection
|
|
110
|
-
- **(B) verify/dedup
|
|
109
|
+
- **(A) single-agent collection**: the worker alone, no driver. It collects.
|
|
110
|
+
- **(B) verify/dedup**: [`createVerifyingResearchDriver`](../../src/web-research-worker.ts):
|
|
111
111
|
a second glm-5.2 pass filters each source for relevance / near-duplicates.
|
|
112
|
-
- **(C) DRIVING
|
|
112
|
+
- **(C) DRIVING**: [`createResearchDrivingDriver`](../../src/research-driving-driver.ts):
|
|
113
113
|
the driver extracts each source's claims (glm-5.2), tracks independent-source
|
|
114
114
|
support + contradictions, and folds comparative / mechanism / gap /
|
|
115
115
|
contradiction sub-questions into the worker's next prompt.
|
|
@@ -118,10 +118,10 @@ driver:
|
|
|
118
118
|
single-agent iteration is 1 worker pass; a two-agent round is 1 worker + 1 driver
|
|
119
119
|
pass = 2. Each arm gets the same pass ceiling B; the single-agent arm gets more
|
|
120
120
|
iterations to spend the budget the two-agent arms burn on their driver. Cost is
|
|
121
|
-
read per-arm from `RouterClient.usage()
|
|
121
|
+
read per-arm from `RouterClient.usage()`, measured dollars/tokens/calls, not
|
|
122
122
|
estimates.
|
|
123
123
|
|
|
124
|
-
## 4. Result
|
|
124
|
+
## 4. Result: driving does not reliably win, and the verdict flips with budget
|
|
125
125
|
|
|
126
126
|
Per-topic held-out questions answered (out of 4 each), at both budgets:
|
|
127
127
|
|
|
@@ -134,15 +134,15 @@ Per-topic held-out questions answered (out of 4 each), at both budgets:
|
|
|
134
134
|
| mixture-of-experts | 4 / 4 | 4 / 3 | 4 / 3 |
|
|
135
135
|
| **total /20** | **13 / 15** | **15 / 15** | **16 / 13** |
|
|
136
136
|
|
|
137
|
-
The driving arm answers **16** at B=4 and **13** at B=6
|
|
137
|
+
The driving arm answers **16** at B=4 and **13** at B=6, a 3-question swing on
|
|
138
138
|
the *same* exam, the same arm, just a different compute ceiling and a different
|
|
139
139
|
minute of web results. Single swings 13→15; verify is flat 15→15. **The within-arm
|
|
140
140
|
swing is as large as the between-arm gap**, which is the signature of a null:
|
|
141
141
|
whatever separates a "win" from a "loss" here is web variance, not the driver.
|
|
142
142
|
|
|
143
143
|
The **cost**, by contrast, is stable and large. Driving spends **~$0.084–0.089**
|
|
144
|
-
across 5 topics vs **~$0.005–0.007** for single-agent
|
|
145
|
-
**~24× the tokens** (~70k vs ~2.5k)
|
|
144
|
+
across 5 topics vs **~$0.005–0.007** for single-agent, **12–16× the dollars** and
|
|
145
|
+
**~24× the tokens** (~70k vs ~2.5k), because it runs a claim-extraction LLM call
|
|
146
146
|
on every fetched source. For that 12–16× it buys no reliable held-out-question
|
|
147
147
|
gain over plain collection.
|
|
148
148
|
|
|
@@ -151,18 +151,18 @@ gain over plain collection.
|
|
|
151
151
|
The decisive diagnostic is `passes=2` on **every** topic at **every** budget: the
|
|
152
152
|
two-agent loop ran exactly **one** worker round before the readiness gate reported
|
|
153
153
|
done, even with B=6 budget for three rounds. The generic specs require one source
|
|
154
|
-
to close a gap, and the worker's first-round fetch closes them
|
|
154
|
+
to close a gap, and the worker's first-round fetch closes them, so the loop stops
|
|
155
155
|
before round 2. The driving driver's mechanism is *multi-round*: round 1 extracts
|
|
156
156
|
claims and flags the weakly-supported ones; round 2+ is where it steers the worker
|
|
157
157
|
to corroborate and go deeper. **It never got a round 2.** So in this setting
|
|
158
158
|
driving's only active effect was a one-round claim-extraction tax with no chance to
|
|
159
|
-
use what it extracted
|
|
159
|
+
use what it extracted, which is exactly what the numbers show: same answers as
|
|
160
160
|
collection, much higher cost.
|
|
161
161
|
|
|
162
162
|
That makes the equal-compute/generic-gate A/B a test of the wrong thing for the
|
|
163
163
|
driving thesis. The fair test has to *force* multiple rounds.
|
|
164
164
|
|
|
165
|
-
## 5. Controlled multi-round probe
|
|
165
|
+
## 5. Controlled multi-round probe
|
|
166
166
|
|
|
167
167
|
To isolate "does depth-steering help **when it actually runs**?", we raise the
|
|
168
168
|
readiness bar so the gate stays unmet and the loop runs the full round budget,
|
|
@@ -172,23 +172,23 @@ worker **re-searches the same gaps blind** (a no-op driver that accepts every
|
|
|
172
172
|
source and never steers). If driving's steering has any value, this is where it
|
|
173
173
|
shows.
|
|
174
174
|
|
|
175
|
-
**Result
|
|
175
|
+
**Result, 3 rounds, 3 topics, driving steers vs blind re-search:**
|
|
176
176
|
|
|
177
177
|
| topic | driving (steered) answered / cost | blind (no steer) answered / cost |
|
|
178
178
|
|---|---|---|
|
|
179
|
-
| speculative decoding | 2/4
|
|
180
|
-
| LoRA | 3/4
|
|
181
|
-
| RLHF / PPO | **3/4
|
|
182
|
-
| **total /12** | **8/12
|
|
179
|
+
| speculative decoding | 2/4; $0.032 | **4/4; $0.004** |
|
|
180
|
+
| LoRA | 3/4; $0.028 | 3/4; $0.003 |
|
|
181
|
+
| RLHF / PPO | **3/4**; $0.033 | 1/4; $0.003 |
|
|
182
|
+
| **total /12** | **8/12; $0.093** | **8/12; $0.010** |
|
|
183
183
|
|
|
184
|
-
**Steering does not help: 8 vs 8, at ~9× the cost.**
|
|
185
|
-
full multi-round regime its mechanism was designed for
|
|
184
|
+
**Steering does not help: 8 vs 8, at ~9× the cost.** In the controlled test, the
|
|
185
|
+
full multi-round regime its mechanism was designed for, the driving driver ties a
|
|
186
186
|
blind worker that just re-searches the same gaps three times. It is **better on
|
|
187
187
|
RLHF** (3 vs 1, the one topic where chasing corroboration found a page blind
|
|
188
|
-
re-search missed) and **worse on speculative decoding** (2 vs 4
|
|
188
|
+
re-search missed) and **worse on speculative decoding** (2 vs 4, steering pulled
|
|
189
189
|
the worker *off* the pages that answered the exam and toward corroborating a
|
|
190
190
|
narrower claim set). Those cancel. So the depth-steering does change *what* gets
|
|
191
|
-
fetched, but not *how many* held-out questions get answered
|
|
191
|
+
fetched, but not *how many* held-out questions get answered, and it pays ~9× the
|
|
192
192
|
dollars for the privilege. The headline null (§4) is therefore **not** merely a
|
|
193
193
|
gate artifact: even forced to run, driving does not beat blind collection on
|
|
194
194
|
research quality at this n.
|
|
@@ -198,13 +198,13 @@ research quality at this n.
|
|
|
198
198
|
- **Small n, high web variance.** n = 5 topics; one live run per arm per budget.
|
|
199
199
|
The §4 budget-flip is itself the evidence that the per-run magnitudes are
|
|
200
200
|
variance-bound. We did not run a paired bootstrap because the within-arm swing
|
|
201
|
-
already exceeds the between-arm gap
|
|
201
|
+
already exceeds the between-arm gap, the supported conclusion is "no separation," and a
|
|
202
202
|
significance test on a known-null at n=5 would dress it up, not clarify it.
|
|
203
|
-
- **The gate, not the driver, ended the headline loop (§4)
|
|
203
|
+
- **The gate, not the driver, ended the headline loop (§4), but the probe
|
|
204
204
|
controls for it.** The headline A/B's generic one-source readiness gate closed
|
|
205
205
|
every loop after one round, so on its own it would only show "driving adds a
|
|
206
206
|
one-round extraction tax." The §5 probe removes that confound by forcing 3
|
|
207
|
-
rounds, and driving still ties blind (8 vs 8)
|
|
207
|
+
rounds, and driving still ties blind (8 vs 8), so the null survives the fix,
|
|
208
208
|
it is not an artifact of the permissive gate.
|
|
209
209
|
- **The worker is shared and shallow.** All arms use the same ~500-line direct
|
|
210
210
|
pipeline (query-gen → search → fetch), not an `AgentProfile` on a harness. A
|
|
@@ -220,15 +220,15 @@ research quality at this n.
|
|
|
220
220
|
## 7. What this says, plainly
|
|
221
221
|
|
|
222
222
|
Adding a "drive it deeper" agent did **not** make the research measurably better at
|
|
223
|
-
answering hard, held-out questions
|
|
224
|
-
full multi-round mechanism (§5), on this worker, at n=5
|
|
223
|
+
answering hard, held-out questions, at equal compute (§4) *and* forced to run its
|
|
224
|
+
full multi-round mechanism (§5), on this worker, at n=5, and it cost 9–16× more.
|
|
225
225
|
The steering changes *what* gets fetched (it helped on RLHF, hurt on speculative
|
|
226
226
|
decoding) but not *how many* held-out questions get answered. The most durable
|
|
227
|
-
thing the experiment produced is the **measurement
|
|
227
|
+
thing the experiment produced is the **measurement method**: a firewalled
|
|
228
228
|
deep-question exam with a $0 deterministic grader that *can* tell depth from
|
|
229
229
|
surface (0/20 vs 20/20), reusable for any future research-quality claim in this
|
|
230
|
-
repo. The driving thesis
|
|
231
|
-
collection
|
|
230
|
+
repo. The driving thesis, that pursuing depth + corroboration beats plain
|
|
231
|
+
collection, is, on the evidence here, **not supported**; the cheaper paths
|
|
232
232
|
(collect, or dedup) match it. Where it might still earn its cost: a worker rich
|
|
233
233
|
enough that "go corroborate this claim" reaches a page blind re-search can't
|
|
234
234
|
(the RLHF case), measured at an n large enough to separate that from variance.
|
|
@@ -242,24 +242,24 @@ cd agent-knowledge && pnpm install
|
|
|
242
242
|
# offline: the exam wiring + the $0 grader (no credentials)
|
|
243
243
|
pnpm exec vitest run tests/loops/research-driving-ab.test.ts
|
|
244
244
|
|
|
245
|
-
# the live 3-arm A/B
|
|
245
|
+
# the live 3-arm A/B: real web search + glm-5.2, per-arm cost reported
|
|
246
246
|
export TANGLE_API_KEY=<router key with glm-5.2 credits>
|
|
247
247
|
AGENT_KNOWLEDGE_LIVE=1 RQ_LIVE_BUDGET=4 \
|
|
248
248
|
pnpm exec vitest run tests/loops/research-driving-ab.test.ts -t "3-arm A/B"
|
|
249
249
|
# re-run at RQ_LIVE_BUDGET=6 to see the verdict flip (the §4 variance point)
|
|
250
250
|
|
|
251
|
-
# the controlled multi-round probe
|
|
251
|
+
# the controlled multi-round probe: forces N rounds so driving actually steers
|
|
252
252
|
AGENT_KNOWLEDGE_LIVE=1 RQ_PROBE=1 RQ_PROBE_ROUNDS=3 TANGLE_API_KEY=<…> \
|
|
253
253
|
pnpm exec vitest run tests/loops/research-driving-ab.test.ts -t "multi-round probe"
|
|
254
254
|
# (~$0.10 for the 5-topic A/B at one budget; ~$0.10 for the 3-topic probe)
|
|
255
255
|
```
|
|
256
256
|
|
|
257
257
|
`RQ_LIVE_TOPICS` takes a `|`-separated subset of the exam topic names to run a
|
|
258
|
-
cheaper slice. The exam is held out by construction
|
|
258
|
+
cheaper slice. The exam is held out by construction, no flag shows it to a loop.
|
|
259
259
|
|
|
260
|
-
**Source:** the exam
|
|
260
|
+
**Source:** the exam and grader:
|
|
261
261
|
[`tests/loops/held-out-exam.ts`](../../tests/loops/held-out-exam.ts);
|
|
262
|
-
the 3-arm A/B
|
|
262
|
+
the 3-arm A/B and multi-round probe:
|
|
263
263
|
[`tests/loops/research-driving-ab.test.ts`](../../tests/loops/research-driving-ab.test.ts);
|
|
264
|
-
the driving driver under test
|
|
264
|
+
the driving driver under test:
|
|
265
265
|
[`src/research-driving-driver.ts`](../../src/research-driving-driver.ts).
|