@tangle-network/agent-knowledge 1.8.0 → 1.10.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +6 -0
- package/dist/index.d.ts +739 -321
- package/dist/index.js +653 -0
- package/dist/index.js.map +1 -1
- package/dist/memory/index.d.ts +1 -1
- package/dist/{types-BEDGlXB-.d.ts → types-C17sAROL.d.ts} +1 -1
- package/dist/viz/index.d.ts +1 -1
- package/docs/results/adaptive.md +120 -0
- package/docs/results/claim-grounding.md +97 -0
- package/docs/results/cost-quality.md +27 -0
- package/docs/supervisor-profiles.md +295 -0
- package/docs/two-agent-research-ab.md +428 -0
- package/package.json +1 -1
|
@@ -0,0 +1,428 @@
|
|
|
1
|
+
# A verifier agent mostly deduplicates: a controlled A/B on two-agent web research, and what its cost buys
|
|
2
|
+
|
|
3
|
+
*Tangle Network · `agent-knowledge`*
|
|
4
|
+
|
|
5
|
+
## Abstract
|
|
6
|
+
|
|
7
|
+
We test whether adding a second "verifier" agent to a web-research loop produces a
|
|
8
|
+
cleaner knowledge base than a single agent doing the same work, with compute held
|
|
9
|
+
fixed. A *worker* agent searches the web and proposes sources for the knowledge
|
|
10
|
+
base's open gaps; a *driver* agent vets each proposed source before it commits,
|
|
11
|
+
fills gaps the worker missed, and decides when the base is complete. Over 9
|
|
12
|
+
machine-learning topics at equal compute, the two-agent loop admitted **2.3–2.7
|
|
13
|
+
fewer sources per topic at identical coverage** — 95% bootstrap intervals
|
|
14
|
+
[1.78, 2.89] and [2.22, 3.00] across two independent runs, both above zero. The
|
|
15
|
+
effect is real and reproduces. But the mechanism is not the one we set out to
|
|
16
|
+
test: reading the rejection logs, most of the gain is **de-duplication** — the
|
|
17
|
+
same paper fetched from arXiv, OpenReview, and the NeurIPS proceedings — not the
|
|
18
|
+
relevance filtering we expected. Pricing the verifier's calls (we added per-arm
|
|
19
|
+
router-usage instrumentation) shows the cleanliness costs roughly **5× the
|
|
20
|
+
dollars, 9× the tokens, and 3× the latency** of the single agent — and that the
|
|
21
|
+
original "equal passes" framing hid this, because it charged the verify step as
|
|
22
|
+
one pass while it is actually one LLM call per proposed source. Since the win is
|
|
23
|
+
de-dup-dominated, a deterministic content hash recovers most of the cleanliness at
|
|
24
|
+
~none of the premium. We then asked the sharper question — is there an error band
|
|
25
|
+
where an LLM verifier *does* earn its dollar? — and found two, on opposite sides of
|
|
26
|
+
the ledger. **Misattributed citations** (an on-topic, unique, real source whose
|
|
27
|
+
cited claim never appears in the page) are caught by a $0 deterministic
|
|
28
|
+
text-presence check that the LLM relevance judge misses 1 in 5 times, because the
|
|
29
|
+
judge structurally never sees the claim. And we built the deployable shape the
|
|
30
|
+
cost result implied — an **adaptive driver** that runs free dedup, then free
|
|
31
|
+
heuristic triage, and escalates to the LLM only on the ambiguous tail: it cuts
|
|
32
|
+
LLM verifier calls 76% and dollars 74%, recovering the de-dup half of the
|
|
33
|
+
verifier's cleanliness while honestly giving up the relevance-judgment half on a
|
|
34
|
+
source pool dominated by authoritative hosts. The verifier earns its dollar on
|
|
35
|
+
misattribution, not on de-duplication; the right production loop spends it only
|
|
36
|
+
where the cheap signals can't decide.
|
|
37
|
+
|
|
38
|
+
## 1. Setup
|
|
39
|
+
|
|
40
|
+
A research agent building a knowledge base accumulates sources. A single agent both
|
|
41
|
+
finds sources and, implicitly, decides which to keep — it grades its own work. The
|
|
42
|
+
hypothesis is the usual one for verification: separating the producer (find
|
|
43
|
+
sources) from the checker (keep the good ones) should yield a cleaner result, for
|
|
44
|
+
the same reason a second pair of eyes catches typos the author misses.
|
|
45
|
+
|
|
46
|
+
The trap in any "more agents help" claim is compute. Two agents that simply do more
|
|
47
|
+
work will of course produce more — that is a bigger budget, not a finding. So the
|
|
48
|
+
comparison must hold total compute fixed and ask whether the *topology* — splitting
|
|
49
|
+
find from check — beats spending the same compute on a single agent that just finds
|
|
50
|
+
more. And once topology shows an effect, the second question is what it costs: a
|
|
51
|
+
cleaner base bought at 5× the inference is a different product decision than one
|
|
52
|
+
bought for free.
|
|
53
|
+
|
|
54
|
+
## 2. Method
|
|
55
|
+
|
|
56
|
+
### 2.1 The loop
|
|
57
|
+
|
|
58
|
+
The loop has two roles (`src/two-agent-research-loop.ts`, `runTwoAgentResearchLoop`):
|
|
59
|
+
|
|
60
|
+
- **Worker** — primary research. Each round it reads the open gaps and proposes
|
|
61
|
+
sources to close them (`ResearchWorker: (ctx: { gaps, steer }) => proposals`).
|
|
62
|
+
- **Driver** — does three things (`ResearchDriver`): `verifySource` vets each
|
|
63
|
+
proposed source before it commits (dedup against the base, then reject sources
|
|
64
|
+
that aren't relevant); `research` runs the driver's own gap-fill pass over gaps
|
|
65
|
+
the worker missed; `foldGaps` turns the still-open gaps into a `steer` string for
|
|
66
|
+
the worker's next round.
|
|
67
|
+
|
|
68
|
+
A round is therefore: worker proposes → driver verifies each proposal (rejections
|
|
69
|
+
never reach the base) → driver gap-fills → the readiness gate checks the base →
|
|
70
|
+
remaining gaps are folded into the next worker prompt. The loop stops when the gate
|
|
71
|
+
reports no blocking gaps left.
|
|
72
|
+
|
|
73
|
+
Note what the driver→worker hand-off is and isn't: the driver *steers* the worker
|
|
74
|
+
by handing it the remaining readiness gaps (`foldGaps`), which is a deterministic
|
|
75
|
+
formatting of unmet requirements — not an LLM authoring a fresh instruction. The
|
|
76
|
+
driver's LLM work is in `verifySource` (one call per proposed source) and its own
|
|
77
|
+
`research` pass. This matters for §4.2: the verify step is N calls, not one, and
|
|
78
|
+
the cost framing turns on that.
|
|
79
|
+
|
|
80
|
+
The readiness gate is `scoreKnowledgeReadiness` (from `agent-eval`). It scores
|
|
81
|
+
*pages* (curated `knowledge/*.md`), not raw sources, and only `importance:
|
|
82
|
+
'blocking'` requirements gate. Coverage below is the fraction of a topic's blocking
|
|
83
|
+
requirements met.
|
|
84
|
+
|
|
85
|
+
### 2.2 What the agents are — an honest note
|
|
86
|
+
|
|
87
|
+
The agents in the live run are **not** `AgentProfile`s on a coding harness. The
|
|
88
|
+
worker is a hand-wired pipeline (`src/web-research-worker.ts`,
|
|
89
|
+
`createWebResearchWorker`): glm-5.2 turns the gaps into search queries → a real web
|
|
90
|
+
search over the Tangle router (`POST /v1/search`) → each hit is fetched with the
|
|
91
|
+
repo's `politeFetch` and reduced to text with `htmlToText` → citing pages are
|
|
92
|
+
proposed. It talks to the router directly through `createTangleRouterClient` — no
|
|
93
|
+
claude-code / opencode / sandbox harness, and no dynamic harness selection. The
|
|
94
|
+
driver (`createVerifyingResearchDriver`) is one glm-5.2 chat call per source.
|
|
95
|
+
|
|
96
|
+
The repo *does* ship a real `AgentProfile` for research (`researcherProfile`), and
|
|
97
|
+
the **offline** control arm uses it with a stub harness — but the live arm bypasses
|
|
98
|
+
it for the direct pipeline. This is a deliberate shortcut (no harness to stand up,
|
|
99
|
+
~$0.20 to run) and also the loop's main simplification debt; see §7.
|
|
100
|
+
|
|
101
|
+
### 2.3 Equal compute
|
|
102
|
+
|
|
103
|
+
Compute is counted in agent passes. A two-agent round = 1 worker pass + 1
|
|
104
|
+
verify pass = 2 passes; a single-agent iteration = 1 pass. Both arms gate on the
|
|
105
|
+
*same* readiness criterion and stop as soon as it is met, so neither is starved.
|
|
106
|
+
We budget-match by passes, not rounds — the single-agent loop gets more rounds to
|
|
107
|
+
spend the compute the two-agent loop spends on verification. The harness asserts,
|
|
108
|
+
per topic, that the two-agent loop spent no more passes than the single-agent loop
|
|
109
|
+
and that both stayed under the ceiling; if that ever fails the comparison has
|
|
110
|
+
drifted to unequal compute and the result is void.
|
|
111
|
+
|
|
112
|
+
The pass-accounting has a known soft spot, which §4.2 exposes: a "verify pass" is
|
|
113
|
+
not one LLM call, it is one `verifySource` call per *proposed* source that round.
|
|
114
|
+
Charging it as a single pass keeps the topology comparison fair on agent passes but
|
|
115
|
+
understates the verifier's dollar cost. We added explicit per-arm cost
|
|
116
|
+
instrumentation to measure that directly.
|
|
117
|
+
|
|
118
|
+
### 2.4 Cost instrumentation
|
|
119
|
+
|
|
120
|
+
The router client now records usage per call (`RouterClient.usage()`,
|
|
121
|
+
`src/web-research-worker.ts`): cumulative chat-completion count, prompt/completion
|
|
122
|
+
tokens, glm-5.2 priced cost, and wall latency. Each A/B arm reads the accumulator
|
|
123
|
+
before and after its run and diffs, so every reported dollar and token figure is a
|
|
124
|
+
measured per-arm delta, not an estimate.
|
|
125
|
+
|
|
126
|
+
### 2.5 Topics and readiness
|
|
127
|
+
|
|
128
|
+
9 topics, each with two blocking requirements (the defining mechanism, and reported
|
|
129
|
+
results / trade-offs). Seven are "narrow-scope-inside-a-broad-space" (e.g.
|
|
130
|
+
*self-speculative decoding* inside *speculative decoding*), where we expected the
|
|
131
|
+
broad space to leak in; two are clean controls (*the transformer architecture*,
|
|
132
|
+
*gradient descent*).
|
|
133
|
+
|
|
134
|
+
## 3. Result 1 — cleanliness: the verifier admits fewer sources at equal coverage
|
|
135
|
+
|
|
136
|
+
The cleanliness signal is the **admitted-source count**: on live data there is no
|
|
137
|
+
oracle, so "fewer sources admitted at equal coverage" is the measurable proxy for
|
|
138
|
+
"cleaner." Δ = single-agent admitted − two-agent admitted, per topic.
|
|
139
|
+
|
|
140
|
+
| Topic | band | Run 1 Δ | Run 2 Δ |
|
|
141
|
+
|---|---|---|---|
|
|
142
|
+
| self-speculative decoding | narrow | 3 | 3 |
|
|
143
|
+
| grouped-query attention | narrow | 3 | 3 |
|
|
144
|
+
| rotary position embeddings | narrow | 2 | 3 |
|
|
145
|
+
| KV-cache quantization | narrow | 1 | 1 |
|
|
146
|
+
| LoRA | narrow | 1 | 3 |
|
|
147
|
+
| ring attention | narrow | 2 | 3 |
|
|
148
|
+
| constitutional AI | narrow | 3 | 3 |
|
|
149
|
+
| the transformer architecture | clean | 3 | 3 |
|
|
150
|
+
| gradient descent | clean | 3 | 2 |
|
|
151
|
+
| **mean Δ** | | **2.33** | **2.67** |
|
|
152
|
+
| **95% CI** (paired bootstrap) | | **[1.78, 2.89]** | **[2.22, 3.00]** |
|
|
153
|
+
|
|
154
|
+
Coverage was **1.00 on every topic, both arms, both runs** — the verifier never
|
|
155
|
+
cost completeness. Both bootstrap intervals (`pairedBootstrap`, from `agent-eval`)
|
|
156
|
+
are above zero. The effect reproduces; its exact magnitude varies run-to-run with
|
|
157
|
+
what the web returns (one topic swung Δ = 0→1→3 across separate runs during
|
|
158
|
+
development).
|
|
159
|
+
|
|
160
|
+
A third, cost-instrumented run (the one priced in §4.2) was noisier: mean Δ **+1.56,
|
|
161
|
+
95% CI [0.33, 2.67]**, with two topics where the two-agent loop admitted *more* than
|
|
162
|
+
the single agent (KV-cache quantization −1, gradient descent −1). The interval still
|
|
163
|
+
clears zero, but it is the lower-bound run — a reminder that the magnitude is
|
|
164
|
+
web-variance-bound, while the sign is stable.
|
|
165
|
+
|
|
166
|
+
## 4. Result 2 — what the verifier does, and what it costs
|
|
167
|
+
|
|
168
|
+
### 4.1 Mostly de-duplication
|
|
169
|
+
|
|
170
|
+
We classified each rejection by the verifier's own stated reason:
|
|
171
|
+
|
|
172
|
+
| rejection reason | narrow (7) | clean (2) |
|
|
173
|
+
|---|---|---|
|
|
174
|
+
| near-duplicate (same paper, different host) | 6 | 4 |
|
|
175
|
+
| off-scope (broad space leaked in) | 3 | 2 |
|
|
176
|
+
| junk page (aggregator / marketing / explainer) | 3 | 0 |
|
|
177
|
+
|
|
178
|
+
The dominant mechanism is **de-duplication** — canonical papers mirrored across
|
|
179
|
+
arXiv, OpenReview, and the NeurIPS proceedings — and it fires regardless of band.
|
|
180
|
+
The off-scope rejection we set out to measure is real (on *self-speculative
|
|
181
|
+
decoding* the verifier correctly dropped three general *speculative decoding* papers
|
|
182
|
+
that use a *separate* draft model) but it is the minority, and it does not
|
|
183
|
+
concentrate on the narrow topics as hypothesized: narrow mean Δ = 2.14 vs clean
|
|
184
|
+
3.00. The strong form of our hypothesis — narrow-in-broad pays more — is **refuted
|
|
185
|
+
in magnitude, confirmed only in mechanism**.
|
|
186
|
+
|
|
187
|
+
The practical reading: most of the win is "you fetched the same PDF three times,"
|
|
188
|
+
which a content hash catches for free. The LLM's distinctive contribution is the
|
|
189
|
+
page that *looks* on-topic but isn't — the self-speculative-vs-separate-draft
|
|
190
|
+
distinction a string match would miss.
|
|
191
|
+
|
|
192
|
+
### 4.2 The inference premium (and what "equal passes" hid)
|
|
193
|
+
|
|
194
|
+
`docs/results/cost-quality.md`. The original A/B reported only admitted-sources at
|
|
195
|
+
"equal passes," which charged the verify step as one pass while it is actually N
|
|
196
|
+
`verifySource` LLM calls. Pricing the calls per arm (B ≤ 4 passes/arm, glm-5.2):
|
|
197
|
+
|
|
198
|
+
| per topic (mean) | two-agent | single-agent | ratio |
|
|
199
|
+
|---|---|---|---|
|
|
200
|
+
| LLM chat calls | 5.4 | 1.0 | ~5.4× |
|
|
201
|
+
| tokens (in+out) | ~4,900 | ~530 | ~9× |
|
|
202
|
+
| cost (USD) | ~$0.0072 | ~$0.0013 | ~5.5× |
|
|
203
|
+
| latency (wall) | ~37 s | ~11 s | ~3.4× |
|
|
204
|
+
| cleanliness Δ (single − two admitted) | — | — | +1.56, 95% CI [0.33, 2.67] |
|
|
205
|
+
|
|
206
|
+
The verifier buys ~1.5–2.7 fewer junk sources for roughly **5× the dollars, 9× the
|
|
207
|
+
tokens, and 3× the latency**. Since the cleanliness gain is de-dup-dominated
|
|
208
|
+
(§4.1), the honest production move is a deterministic content-hash / canonical-URL
|
|
209
|
+
dedup, which captures most of the cleanliness at ~none of this premium, reserving
|
|
210
|
+
an LLM check only for the off-scope tail. This is the cost half the "equal passes"
|
|
211
|
+
framing left out — and the rest of the paper is what we built once we saw it.
|
|
212
|
+
|
|
213
|
+
## 5. Result 3 — the two bands where an LLM verifier does, and doesn't, earn its dollar
|
|
214
|
+
|
|
215
|
+
If de-dup is free and dominates the win, when is the LLM verifier worth its 5×? We
|
|
216
|
+
found two bands, and they cut in opposite directions.
|
|
217
|
+
|
|
218
|
+
### 5.1 Misattributed citations — the cheap check beats the expensive judge
|
|
219
|
+
|
|
220
|
+
`docs/results/claim-grounding.md`. A source can be on-topic, unique, and real, yet
|
|
221
|
+
the cited *claim* never appears in the page — the LLM wrote a plausible sentence and
|
|
222
|
+
hung a real URL off it. De-dup passes it (unique). A relevance judge passes it (the
|
|
223
|
+
page is on-topic). Only checking the claim against the fetched text catches it — and
|
|
224
|
+
that check is **deterministic text presence, $0 inference**.
|
|
225
|
+
|
|
226
|
+
Each proposed source now carries the claim it is cited for (`withCitedClaim` →
|
|
227
|
+
`metadata.citedClaim`). The claim-grounding verifier (`createClaimGroundingVerifier`,
|
|
228
|
+
`src/claim-grounding.ts`) runs `groundClaimInText(claim, pageText)` over the
|
|
229
|
+
`htmlToText` output of the page the worker actually fetched — verbatim, normalized
|
|
230
|
+
(punctuation/whitespace-insensitive), or a ≥70% content-word overlap close
|
|
231
|
+
paraphrase. A claim that isn't present is rejected as misattributed. The oracle is
|
|
232
|
+
text presence, not a model call, so it composes with the LLM relevance verifier or
|
|
233
|
+
runs alone at zero cost.
|
|
234
|
+
|
|
235
|
+
Live A/B (glm-5.2, real web fetch, one planted misattribution per topic — a real
|
|
236
|
+
fetched page plus a deliberately-wrong claim — over three verifier arms on the same
|
|
237
|
+
proposals):
|
|
238
|
+
|
|
239
|
+
| n=5 topics | misattributions caught | marginal $ | per-$ caught |
|
|
240
|
+
|---|---|---|---|
|
|
241
|
+
| no-verifier | 0 / 5 | $0.0000 | — |
|
|
242
|
+
| relevance (LLM judge) | 4 / 5 | $0.0157 | 254 |
|
|
243
|
+
| claim-grounding (text) | **5 / 5** | **$0.0000** | ∞ |
|
|
244
|
+
|
|
245
|
+
The relevance judge catches one only by accident — when the fabricated claim also
|
|
246
|
+
makes the page read off-topic (a "12-billion-parameter draft transformer" claim on a
|
|
247
|
+
rotary-embeddings page). When the fabrication stays on-topic (the KV-cache case), the
|
|
248
|
+
judge waves it through, because the relevance verifier only ever sees the page text,
|
|
249
|
+
never the cited claim — it is **structurally blind** to misattribution. On this band
|
|
250
|
+
the verifier-per-dollar comparison inverts §4.2: the cheap, deterministic check
|
|
251
|
+
catches strictly more (5/5 vs 4/5) at strictly less ($0 vs $0.0157). The offline
|
|
252
|
+
floor confirms the wiring: on a controlled 4-source pool (2 grounded, 2
|
|
253
|
+
misattributed), claim-grounding admits **0/2** misattributions and keeps **2/2**
|
|
254
|
+
grounded, while relevance and no-verifier both admit **2/2**.
|
|
255
|
+
|
|
256
|
+
### 5.2 Adaptive topology — pay the LLM only on the ambiguous tail
|
|
257
|
+
|
|
258
|
+
`docs/results/adaptive.md`. The deployable shape §4.2 implied: do the free
|
|
259
|
+
deterministic work first, reserve the LLM for what the cheap signals can't decide.
|
|
260
|
+
`createAdaptiveResearchDriver` (`src/adaptive-driver.ts`) is that driver. Per
|
|
261
|
+
candidate source it runs three stages, cheapest first, stopping at the first that
|
|
262
|
+
decides:
|
|
263
|
+
|
|
264
|
+
1. **Dedup ($0).** Reject a source whose canonical URL (scheme / `www` / trailing
|
|
265
|
+
slash / tracking params stripped) or normalized-text content hash matches one
|
|
266
|
+
already accepted this round or in the KB.
|
|
267
|
+
2. **Heuristic triage ($0).** Classify a unique survivor with host/title/length
|
|
268
|
+
signals only: an authoritative host (arxiv, `*.edu`, `*.gov`, official docs,
|
|
269
|
+
github, …) with a substantial body is **kept**; an obvious spam/listicle title
|
|
270
|
+
or a too-thin body is **dropped**; everything else is **ambiguous**.
|
|
271
|
+
3. **LLM escalation ($).** Only ambiguous survivors reach the shipped LLM relevance
|
|
272
|
+
verifier — one call each.
|
|
273
|
+
|
|
274
|
+
Live frontier, n=5 topics, glm-5.2, same fetched proposals gated through all three
|
|
275
|
+
drivers (plus one planted tracking-decorated mirror of the first source, so the
|
|
276
|
+
dedup stage has a real duplicate to catch). Total spend $0.033:
|
|
277
|
+
|
|
278
|
+
| topic | fetched | single admit | full-LLM admit / calls / $ | adaptive admit / LLM calls / $ |
|
|
279
|
+
|---|---|---|---|---|
|
|
280
|
+
| self-speculative decoding | 3 | 3 | 1 / 3 / $0.0027 | 2 / **0** / **$0.0000** |
|
|
281
|
+
| rotary position embeddings | 3 | 3 | 1 / 3 / $0.0031 | 2 / **0** / **$0.0000** |
|
|
282
|
+
| grouped-query attention | 7 | 7 | 3 / 7 / $0.0072 | 6 / 3 / $0.0030 |
|
|
283
|
+
| KV-cache quantization | 5 | 5 | 3 / 5 / $0.0052 | 4 / **0** / **$0.0000** |
|
|
284
|
+
| LoRA fine-tuning | 7 | 7 | 4 / 7 / $0.0079 | 6 / 3 / $0.0037 |
|
|
285
|
+
| **total** | **25** | **25** | **12 / 25 / $0.0261** | **20 / 6 / $0.0068** |
|
|
286
|
+
|
|
287
|
+
Adaptive cuts LLM verifier calls **76%** (25 → 6) and dollars **74%** ($0.0261 →
|
|
288
|
+
$0.0068). On 3 of the 5 topics it spent **zero** LLM calls — every unique survivor
|
|
289
|
+
was on an authoritative host, so the $0 stages decided everything.
|
|
290
|
+
|
|
291
|
+
It is a frontier point, not a free lunch. Admitted counts (lower = cleaner): single
|
|
292
|
+
25, **adaptive 20**, full-LLM 12. Adaptive removes the 5 real duplicates the $0 dedup
|
|
293
|
+
catches — exactly the de-dup-dominated win — but keeps the 8 sources full-LLM rejects
|
|
294
|
+
on relevance, because on this authoritative-host-heavy set the heuristic resolved
|
|
295
|
+
every non-duplicate survivor without ever asking the LLM, and the host prior is
|
|
296
|
+
coarser than the relevance judge. So adaptive **recovers the deterministic de-dup
|
|
297
|
+
half of full-LLM's cleanliness for ~26% of its dollars, and gives up the
|
|
298
|
+
relevance-judgment half**. The escalation count is the diagnostic: on the 3 topics
|
|
299
|
+
where it was zero, adaptive *is* a pure host/title/length rule and the LLM
|
|
300
|
+
contributes nothing by construction; on the 2 topics with unknown-host survivors
|
|
301
|
+
(grouped-query attention, LoRA) it escalated 3 calls each — the off-scope tail the
|
|
302
|
+
verifier is actually for.
|
|
303
|
+
|
|
304
|
+
## 6. Discussion
|
|
305
|
+
|
|
306
|
+
The three results compose into one rule. The LLM verifier's headline cleanliness win
|
|
307
|
+
is real (§3) but **de-dup-dominated** (§4.1) and **expensive** (§4.2, ~5×/9×/3×), so
|
|
308
|
+
spending an LLM call on every source is the wrong default — a free content hash buys
|
|
309
|
+
most of it. The verifier earns its 5× exactly where the cheap signals are blind: on
|
|
310
|
+
**misattribution** (§5.1), where a $0 text-presence check beats the LLM judge
|
|
311
|
+
outright because the judge never sees the claim; and on the **off-scope tail** (§5.2),
|
|
312
|
+
where a page looks on-topic, is unique, and isn't fabricated, so only a relevance
|
|
313
|
+
judgment can settle it. The deployable loop therefore stratifies by cost: free dedup,
|
|
314
|
+
free claim-grounding, free heuristic triage, then an LLM call only on what survives —
|
|
315
|
+
which is what the adaptive driver ships.
|
|
316
|
+
|
|
317
|
+
Two cross-cutting lessons. First, **the accounting unit decides the verdict**:
|
|
318
|
+
charging the verify step as one pass made the topology look near-free; pricing it per
|
|
319
|
+
LLM call (§4.2) is what surfaced the 5× and motivated everything after it. Second,
|
|
320
|
+
**the same verifier inverts in value across bands** — on de-dup the LLM is expensive
|
|
321
|
+
for what a hash does; on misattribution a deterministic check is free for what the LLM
|
|
322
|
+
can't do; on the off-scope tail the LLM is the only thing that works. "Add a verifier"
|
|
323
|
+
is not a setting; it is a cost-stratified decision per error type.
|
|
324
|
+
|
|
325
|
+
## 7. Limitations
|
|
326
|
+
|
|
327
|
+
- **The verifier is also the judge.** Admitted-count is a proxy; we have no
|
|
328
|
+
independent oracle for whether a dropped source was genuinely redundant. The
|
|
329
|
+
verifier's stated reasons hold up on inspection, but this is the load-bearing
|
|
330
|
+
caveat for §3–§4.
|
|
331
|
+
- **Deltas are conservative.** The single-agent loop stops on the same readiness
|
|
332
|
+
gate, capping its admits; with more iterations it would admit even more junk, so
|
|
333
|
+
the true gap is at least this large.
|
|
334
|
+
- **Small n.** n = 2 clean controls is too thin to compare bands; the misattribution
|
|
335
|
+
and adaptive frontiers are n = 5 each. The directions are asserted in the tests on
|
|
336
|
+
every run; the magnitudes are small-n and web-variance-bound (the §3 third run swung
|
|
337
|
+
to +1.56 from +2.3/+2.7).
|
|
338
|
+
- **Planted error bands.** The misattributions (§5.1) and the adaptive duplicate
|
|
339
|
+
(§5.2) are injected so the band is measurable. They model the real LLM
|
|
340
|
+
citation-fabrication and mirror-host failures but do not measure their base rate in
|
|
341
|
+
the wild — that needs a hand-checked corpus of model-written citations.
|
|
342
|
+
- **Adaptive's quality is host-prior-bound.** On an authoritative-host-heavy source
|
|
343
|
+
pool the heuristic resolves everything and the LLM's relevance judgment contributes
|
|
344
|
+
nothing; a richer worker (good sources on unknown hosts, junk on on-topic-looking
|
|
345
|
+
pages) would grow the ambiguous tail and converge adaptive toward full-LLM cost.
|
|
346
|
+
- **glm-5.2-specific.** A weaker or stronger judge would shift rejection rates and the
|
|
347
|
+
relevance miss-rate. The grounding oracle is also conservative: a real paraphrase
|
|
348
|
+
whose inflected words differ ("drafts" vs "draft") can fall below the 0.7 overlap and
|
|
349
|
+
be flagged misattributed; `minOverlap` tunes this.
|
|
350
|
+
- **High web variance.** One live run per topic per result; numbers move with what
|
|
351
|
+
search returns.
|
|
352
|
+
|
|
353
|
+
## 8. A simpler loop — built, not deferred
|
|
354
|
+
|
|
355
|
+
The original write-up named two simplifications as future work. Both are now built and
|
|
356
|
+
measured; this is what changed.
|
|
357
|
+
|
|
358
|
+
1. **Deterministic dedup before the LLM, LLM only on the tail — shipped.** The
|
|
359
|
+
adaptive driver (`src/adaptive-driver.ts`, §5.2) does exactly this: free
|
|
360
|
+
canonical-URL / content-hash dedup, free host/title/length triage, LLM relevance
|
|
361
|
+
only on the ambiguous survivors. Measured: **76% fewer LLM calls, 74% cheaper**,
|
|
362
|
+
recovering the de-dup half of the verifier's cleanliness. The remaining gap to
|
|
363
|
+
full-LLM is the relevance-judgment half, kept honest in §5.2 — adaptive is a
|
|
364
|
+
frontier point you choose by how much a kept-but-marginal source costs you, not a
|
|
365
|
+
strict improvement.
|
|
366
|
+
2. **A free check the LLM judge can't replicate — shipped.** Claim-grounding
|
|
367
|
+
(`src/claim-grounding.ts`, §5.1) adds the one verification an LLM relevance judge is
|
|
368
|
+
structurally blind to: does the cited claim actually appear in the page? It catches
|
|
369
|
+
5/5 planted misattributions at **$0**, vs the judge's 4/5 at ~$0.003/topic.
|
|
370
|
+
|
|
371
|
+
What is still **not** built remains the worker: the live worker is a ~500-line
|
|
372
|
+
hand-wired pipeline (query-gen, search, fetch, propose) against the router directly,
|
|
373
|
+
where the repo's own pattern is to *author* an `AgentProfile` (`researcherProfile`)
|
|
374
|
+
and run it on a harness with a web-search tool — reusable and harness-agnostic. The
|
|
375
|
+
direct pipeline is cheaper to run today (no harness, no creds beyond the router) but it
|
|
376
|
+
is the loop's main remaining piece of duplication, and the obvious next step if this
|
|
377
|
+
loop graduates from experiment to production.
|
|
378
|
+
|
|
379
|
+
## 9. Reproduce
|
|
380
|
+
|
|
381
|
+
The loop, the worker, the verifier, the claim-grounding mode, the adaptive driver, the
|
|
382
|
+
cost instrumentation, and every A/B are all in this repository. Each live test gates a
|
|
383
|
+
cheap one-call glm-5.2 smoke before any multi-topic burn.
|
|
384
|
+
|
|
385
|
+
```bash
|
|
386
|
+
git clone https://github.com/tangle-network/agent-knowledge
|
|
387
|
+
cd agent-knowledge && pnpm install
|
|
388
|
+
|
|
389
|
+
# offline A/B — deterministic, no credentials (a controlled lower bound that
|
|
390
|
+
# exercises the same harness against a planted source pool)
|
|
391
|
+
pnpm exec vitest run tests/loops/research-loop-equal-compute.test.ts
|
|
392
|
+
|
|
393
|
+
# offline claim-grounding + adaptive floors (no credentials)
|
|
394
|
+
pnpm exec vitest run tests/loops/claim-grounding-ab.test.ts -t "offline"
|
|
395
|
+
pnpm exec vitest run tests/loops/adaptive-ab.test.ts
|
|
396
|
+
|
|
397
|
+
# the live cleanliness sweep — real web search + a real glm-5.2 verifier, with
|
|
398
|
+
# per-arm cost reported (~$0.20 for 9 topics)
|
|
399
|
+
export TANGLE_API_KEY=<router key with glm-5.2 credits>
|
|
400
|
+
AGENT_KNOWLEDGE_LIVE=1 \
|
|
401
|
+
AGENT_KNOWLEDGE_LIVE_GOALS="self-speculative decoding|grouped-query attention|rotary position embeddings|KV-cache quantization|LoRA|ring attention|constitutional AI|the transformer architecture|gradient descent" \
|
|
402
|
+
pnpm exec vitest run tests/loops/research-loop-equal-compute.test.ts
|
|
403
|
+
|
|
404
|
+
# live misattribution band — three verifier arms over the same proposals
|
|
405
|
+
AGENT_KNOWLEDGE_LIVE=1 TANGLE_API_KEY=<…> \
|
|
406
|
+
CLAIM_GROUNDING_LIVE_GOALS='self-speculative decoding|rotary position embeddings|grouped-query attention|KV-cache quantization|LoRA' \
|
|
407
|
+
pnpm exec vitest run tests/loops/claim-grounding-ab.test.ts -t "three verifier arms"
|
|
408
|
+
|
|
409
|
+
# live adaptive frontier — single / full-LLM / adaptive on the same fetched proposals
|
|
410
|
+
AGENT_KNOWLEDGE_LIVE=1 TANGLE_API_KEY=<…> \
|
|
411
|
+
ADAPTIVE_LIVE_GOALS="self-speculative decoding|rotary position embeddings|grouped-query attention|KV-cache quantization|LoRA fine-tuning" \
|
|
412
|
+
pnpm exec vitest run tests/loops/adaptive-ab.test.ts -t "three-topology"
|
|
413
|
+
```
|
|
414
|
+
|
|
415
|
+
`AGENT_KNOWLEDGE_LIVE_GOALS` (and the per-result `*_LIVE_GOALS`) take a `|`-separated
|
|
416
|
+
topic list; the live arms run the loops on each at equal compute and report the paired
|
|
417
|
+
bootstrap and per-arm cost.
|
|
418
|
+
|
|
419
|
+
**Source:** the loop — [`src/two-agent-research-loop.ts`](../src/two-agent-research-loop.ts);
|
|
420
|
+
the live worker + verifier + cost instrumentation — [`src/web-research-worker.ts`](../src/web-research-worker.ts);
|
|
421
|
+
the misattribution check — [`src/claim-grounding.ts`](../src/claim-grounding.ts);
|
|
422
|
+
the adaptive driver — [`src/adaptive-driver.ts`](../src/adaptive-driver.ts);
|
|
423
|
+
the A/B harnesses — [`tests/loops/`](../tests/loops/).
|
|
424
|
+
Per-result detail: [`docs/results/cost-quality.md`](results/cost-quality.md),
|
|
425
|
+
[`docs/results/claim-grounding.md`](results/claim-grounding.md),
|
|
426
|
+
[`docs/results/adaptive.md`](results/adaptive.md).
|
|
427
|
+
</content>
|
|
428
|
+
</invoke>
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@tangle-network/agent-knowledge",
|
|
3
|
-
"version": "1.
|
|
3
|
+
"version": "1.10.0",
|
|
4
4
|
"description": "Source-grounded, eval-gated knowledge growth primitives for agents.",
|
|
5
5
|
"homepage": "https://github.com/tangle-network/agent-knowledge#readme",
|
|
6
6
|
"repository": {
|