@tangle-network/agent-knowledge 4.0.0 → 4.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENTS.md +32 -18
- package/CHANGELOG.md +32 -0
- package/README.md +31 -19
- package/dist/benchmarks/index.d.ts +51 -3
- package/dist/benchmarks/index.js +1 -1
- package/dist/{chunk-EUAXSBMH.js → chunk-AKYJG2MR.js} +224 -224
- package/dist/chunk-AKYJG2MR.js.map +1 -0
- package/dist/{chunk-DW6APRTX.js → chunk-BBCIB4UL.js} +2486 -2457
- package/dist/chunk-BBCIB4UL.js.map +1 -0
- package/dist/{chunk-UWOTQNBI.js → chunk-CPMLJYA3.js} +1886 -1842
- package/dist/chunk-CPMLJYA3.js.map +1 -0
- package/dist/{chunk-WCYW2GDA.js → chunk-MYFM6LKH.js} +2 -2
- package/dist/chunk-MYFM6LKH.js.map +1 -0
- package/dist/cli.js +1 -1
- package/dist/index.d.ts +48 -52
- package/dist/index.js +2906 -2874
- package/dist/index.js.map +1 -1
- package/dist/memory/index.d.ts +129 -6
- package/dist/memory/index.js +2 -2
- package/dist/sources/index.d.ts +17 -33
- package/dist/sources/index.js +1 -1
- package/dist/{index-Bqg1mBPt.d.ts → types-DP38encz.d.ts} +127 -284
- package/docs/eval/investment-material-facts.md +28 -29
- package/docs/eval/rag-eval-roadmap.md +6 -5
- package/docs/results/adaptive.md +13 -13
- package/docs/results/claim-grounding.md +11 -11
- package/docs/results/cost-quality.md +4 -4
- package/docs/results/investment-thesis.md +56 -56
- package/docs/results/research-driving.md +54 -54
- package/docs/two-agent-research-ab.md +94 -94
- package/package.json +4 -7
- package/dist/chunk-DW6APRTX.js.map +0 -1
- package/dist/chunk-EUAXSBMH.js.map +0 -1
- package/dist/chunk-UWOTQNBI.js.map +0 -1
- package/dist/chunk-WCYW2GDA.js.map +0 -1
|
@@ -1,10 +1,10 @@
|
|
|
1
|
-
# Held-out investment-research eval set
|
|
1
|
+
# Held-out investment-research eval set: material facts + provenance
|
|
2
2
|
|
|
3
3
|
This is the answer key and the provenance ledger for `tests/eval/investment-thesis-set.ts`.
|
|
4
4
|
|
|
5
5
|
**What the set measures.** Give a research loop a company + ticker + an as-of
|
|
6
6
|
**cutoff** date and ask it to write an investment thesis. Then grade that thesis
|
|
7
|
-
against the held-out **material facts** below
|
|
7
|
+
against the held-out **material facts** below, facts the loop never saw. A high
|
|
8
8
|
score means the thesis surfaced the buried, material, non-obvious drivers a
|
|
9
9
|
thorough analyst would flag and a single ticker search would miss; it is **not**
|
|
10
10
|
teaching-to-the-test, because the answer key is firewalled from every loop and
|
|
@@ -20,14 +20,13 @@ the grader is a `$0`, model-free substring check (`gradeFactAgainstText`).
|
|
|
20
20
|
guessed (see the drop log).
|
|
21
21
|
3. **Knowable at the cutoff.** Every value was disclosed in, or computable from, a
|
|
22
22
|
filing available on or before the cutoff. The eventual collapse is **not** a
|
|
23
|
-
checklist item
|
|
24
|
-
never graded.
|
|
23
|
+
checklist item. It is recorded as `knownOutcome` for the reader and is never graded.
|
|
25
24
|
|
|
26
25
|
All five primary documents were fetched live from `https://www.sec.gov/Archives/`
|
|
27
26
|
during curation (a `curl` with a descriptive `User-Agent`, per SEC fair-access
|
|
28
27
|
rules). Every dollar figure below was read directly out of the de-tagged filing
|
|
29
|
-
text.
|
|
30
|
-
|
|
28
|
+
text. Each `sourceUrl` contains the company's SEC CIK, and
|
|
29
|
+
`tests/eval/investment-thesis-set.test.ts` checks that invariant.
|
|
31
30
|
|
|
32
31
|
---
|
|
33
32
|
|
|
@@ -47,7 +46,7 @@ set was curated (June 2026); `investment-thesis-set.test.ts` asserts this.
|
|
|
47
46
|
|
|
48
47
|
---
|
|
49
48
|
|
|
50
|
-
## SIVB
|
|
49
|
+
## SIVB: SVB Financial Group (cutoff 2023-02-24)
|
|
51
50
|
|
|
52
51
|
Source: [FY2022 10-K (`sivb-20221231.htm`)](https://www.sec.gov/Archives/edgar/data/719739/000071973923000021/sivb-20221231.htm)
|
|
53
52
|
**Known outcome (not graded):** FDIC receivership March 10, 2023; holding-company Chapter 11 March 17, 2023.
|
|
@@ -59,14 +58,14 @@ Source: [FY2022 10-K (`sivb-20221231.htm`)](https://www.sec.gov/Archives/edgar/d
|
|
|
59
58
|
| SIVB/f3 | concentration | Run-prone uninsured deposit base | "estimated uninsured deposits in U.S. offices that exceed the FDIC insurance limit were **$151.5 billion**" |
|
|
60
59
|
| SIVB/f4 | margin-trend | Cheap deposits fleeing → funding cost set to rise | "Noninterest-bearing demand deposits to total deposits decreased by **20 percentage points to 47 percent**" |
|
|
61
60
|
| SIVB/f5 | concentration | Single-client-type (innovation-economy) deposit + credit base | 10-K frames the franchise around "the innovation economy" (technology, life-science, venture) |
|
|
62
|
-
| SIVB/f6 | off-balance-sheet | AFS loss in AOCI
|
|
61
|
+
| SIVB/f6 | off-balance-sheet | AFS loss in AOCI; the visible, smaller tip | "Available-for-sale securities, at fair value (cost of $ **28,602**) **26,069**" → ~$2.5B AFS loss in AOCI |
|
|
63
62
|
|
|
64
63
|
The decisive, non-obvious fact is SIVB/f1+f2: an interest-rate loss roughly equal
|
|
65
64
|
to all of equity, sitting in the footnotes because HTM accounting keeps it out of
|
|
66
65
|
both earnings and book equity. A ticker search shows a profitable bank; the
|
|
67
66
|
filing shows a mark-to-market hole the size of its capital.
|
|
68
67
|
|
|
69
|
-
## BBBY
|
|
68
|
+
## BBBY: Bed Bath & Beyond Inc. (cutoff 2022-04-21)
|
|
70
69
|
|
|
71
70
|
Source: [FY2021 10-K (`bbby-20220226.htm`)](https://www.sec.gov/Archives/edgar/data/886158/000088615822000047/bbby-20220226.htm)
|
|
72
71
|
**Known outcome (not graded):** Chapter 11 on April 23, 2023; equity wiped out.
|
|
@@ -80,11 +79,11 @@ Source: [FY2021 10-K (`bbby-20220226.htm`)](https://www.sec.gov/Archives/edgar/d
|
|
|
80
79
|
| BBBY/f5 | margin-trend | Inventory building into a demand decline | "Merchandise inventories **1,725,410** 1,671,909" ($ thousands) |
|
|
81
80
|
|
|
82
81
|
The non-obvious fact is BBBY/f1+f2+f4 together: in FY2021 the company **lost $560M,
|
|
83
|
-
generated only $18M of operating cash, and still spent $575M buying back stock
|
|
82
|
+
generated only $18M of operating cash, and still spent $575M buying back stock**,
|
|
84
83
|
returning more cash than it had. The buyback, not the income statement alone, is
|
|
85
84
|
why a $1.3B equity base became $174M.
|
|
86
85
|
|
|
87
|
-
## CVNA
|
|
86
|
+
## CVNA: Carvana Co. (cutoff 2023-02-23)
|
|
88
87
|
|
|
89
88
|
Source: [FY2022 10-K (`cvna-20221231.htm`)](https://www.sec.gov/Archives/edgar/data/1690820/000169082023000052/cvna-20221231.htm)
|
|
90
89
|
**Known outcome (not graded):** Stock fell ~98% from its 2021 peak; a 2023 debt exchange cut and extended obligations, narrowly avoiding bankruptcy.
|
|
@@ -97,18 +96,18 @@ Source: [FY2022 10-K (`cvna-20221231.htm`)](https://www.sec.gov/Archives/edgar/d
|
|
|
97
96
|
| CVNA/f4 | governance | Recurring related-party leases with the founder's family | Related-Party note: DriveTime, controlled by "Ernest Garcia II, Ernest Garcia III, and entities controlled by one or both of them" |
|
|
98
97
|
| CVNA/f5 | liquidity | A wide loss showing unit economics had not turned | "Net loss $ (**2,894**)" ($ millions) |
|
|
99
98
|
|
|
100
|
-
The non-obvious facts are CVNA/f2 (interest expense up 2.8x
|
|
101
|
-
|
|
99
|
+
The non-obvious facts are CVNA/f2 (interest expense up 2.8x, making the debt expensive
|
|
100
|
+
as well as large) and CVNA/f4 (the controlling Garcia family on both
|
|
102
101
|
sides of material leases via DriveTime), neither of which a ticker quote shows.
|
|
103
102
|
|
|
104
|
-
## PTON
|
|
103
|
+
## PTON: Peloton Interactive, Inc. (cutoff 2022-09-07)
|
|
105
104
|
|
|
106
105
|
Source: [FY2022 10-K (`pton-20220630.htm`)](https://www.sec.gov/Archives/edgar/data/1639825/000163982522000117/pton-20220630.htm)
|
|
107
106
|
**Known outcome (not graded):** Stock fell ~95% from its 2021 peak; founder-CEO departed; mass layoffs and a multi-year turnaround.
|
|
108
107
|
|
|
109
108
|
| ID | Lens | Material fact | Value read from the filing |
|
|
110
109
|
|---|---|---|---|
|
|
111
|
-
| PTON/f1 | margin-trend | Hardware gross margin turned **negative** | Connected Fitness "Gross Margin decreased to (**11**)" percent
|
|
110
|
+
| PTON/f1 | margin-trend | Hardware gross margin turned **negative** | Connected Fitness "Gross Margin decreased to (**11**)" percent; losing money per unit sold |
|
|
112
111
|
| PTON/f2 | liquidity | Inventory glut as pandemic demand normalized | "Inventories, net **1,104.5** 937" ($ millions) |
|
|
113
112
|
| PTON/f3 | governance | Dual-class super-voting control | "Class B common stock has **20 votes per share** and our Class A common stock has one vote per share" |
|
|
114
113
|
| PTON/f4 | liquidity | An order-of-magnitude wider loss | "Net loss $ (**2,827**)" ($ millions) |
|
|
@@ -116,11 +115,11 @@ Source: [FY2022 10-K (`pton-20220630.htm`)](https://www.sec.gov/Archives/edgar/d
|
|
|
116
115
|
| PTON/f6 | leverage | Locked-in purchase commitments into falling demand | "purchase commitments related to the manufacture of Peloton products were estimated to be approximately **$334**" million |
|
|
117
116
|
|
|
118
117
|
The non-obvious fact is PTON/f1: revenue was still large, but the **hardware was
|
|
119
|
-
sold below cost** (−11% gross margin)
|
|
120
|
-
|
|
118
|
+
sold below cost** (−11% gross margin). The unit economics had broken despite the
|
|
119
|
+
top-line growth. PTON/f6 compounds it: the company was contractually obliged to
|
|
121
120
|
buy more inventory it could not sell.
|
|
122
121
|
|
|
123
|
-
## SI
|
|
122
|
+
## SI: Silvergate Capital Corporation (cutoff 2022-02-28)
|
|
124
123
|
|
|
125
124
|
Source: [FY2021 10-K (`si-20211231.htm`)](https://www.sec.gov/Archives/edgar/data/1312109/000131210922000051/si-20211231.htm)
|
|
126
125
|
**Known outcome (not graded):** After the FTX collapse triggered a deposit run, Silvergate announced a voluntary wind-down and liquidation of Silvergate Bank in March 2023.
|
|
@@ -134,7 +133,7 @@ Source: [FY2021 10-K (`si-20211231.htm`)](https://www.sec.gov/Archives/edgar/dat
|
|
|
134
133
|
| SI/f5 | concentration | The moat AND the funding are the same crypto-only bet | "Silvergate Exchange Network ('SEN'), our proprietary ... payment network for participants in the digital currency industry" |
|
|
135
134
|
|
|
136
135
|
The non-obvious fact is SI/f1+f2 together: **99.5% noninterest-bearing deposits,
|
|
137
|
-
~58% from crypto exchanges
|
|
136
|
+
~58% from crypto exchanges**, a funding base with no contractual term and a
|
|
138
137
|
single correlated counterparty type. A ticker quote shows a fast-growing,
|
|
139
138
|
low-cost-funding bank; the filing shows a bank that could be emptied in days if
|
|
140
139
|
crypto sentiment turned.
|
|
@@ -176,8 +175,8 @@ should not be read as a general "investment-research quality" number. The biases
|
|
|
176
175
|
|
|
177
176
|
3. **Sector skew toward financials + distressed consumer.** Two of five are banks
|
|
178
177
|
(SIVB, SI). The two banks are deliberately given **different** buried-risk
|
|
179
|
-
lenses
|
|
180
|
-
single-industry deposit-concentration story
|
|
178
|
+
lenses, SIVB is an interest-rate / duration / off-balance-sheet story, SI is a
|
|
179
|
+
single-industry deposit-concentration story, so they are not redundant, but
|
|
181
180
|
the set still over-indexes on balance-sheet fragility and under-tests, e.g.,
|
|
182
181
|
technology platform risk, supply-chain concentration, or accounting-policy
|
|
183
182
|
aggressiveness in a healthy grower.
|
|
@@ -197,33 +196,33 @@ should not be read as a general "investment-research quality" number. The biases
|
|
|
197
196
|
required a second independent source to establish was dropped (below) rather
|
|
198
197
|
than sourced to the filing alone.
|
|
199
198
|
|
|
200
|
-
## Drop log
|
|
199
|
+
## Drop log: excluded items
|
|
201
200
|
|
|
202
201
|
These were candidate facts I could not independently ground to a primary source
|
|
203
202
|
available at the cutoff, so I **dropped them rather than guess**:
|
|
204
203
|
|
|
205
|
-
- **First Republic Bank (FRC)
|
|
204
|
+
- **First Republic Bank (FRC)**: dropped as a company entirely. First Republic
|
|
206
205
|
was a state-chartered bank that filed its annual reports with the **FDIC**, not
|
|
207
206
|
on SEC EDGAR, so its 10-K is not at a `sec.gov/Archives` URL and I could not give
|
|
208
|
-
it the same
|
|
207
|
+
it the same CIK-matched provenance as the other five. Its widely cited
|
|
209
208
|
figures (~$15B HTM-style loss, ~$119.5B uninsured deposits) are real but are best
|
|
210
|
-
sourced from the FDIC OIG Material Loss Review, a **post**-cutoff document
|
|
209
|
+
sourced from the FDIC OIG Material Loss Review, a **post**-cutoff document, using
|
|
211
210
|
it would violate rule 3. Replaced with Silvergate, whose 10-K is on EDGAR.
|
|
212
211
|
|
|
213
|
-
- **SIVB exact total unrealized-loss footnote line
|
|
212
|
+
- **SIVB exact total unrealized-loss footnote line**: I report the HTM gap as the
|
|
214
213
|
arithmetic difference of two figures printed on the balance sheet
|
|
215
214
|
($91,321M − $76,169M), which is exact and on-cutoff. I did **not** include a
|
|
216
215
|
separately-quoted "$15.1B net unrealized loss" sentence because I did not locate
|
|
217
216
|
that exact phrasing in the de-tagged text; quoting a number I could not point to
|
|
218
217
|
verbatim would break rule 2. The computed value is conservative and checkable.
|
|
219
218
|
|
|
220
|
-
- **CVNA negative gross-profit-per-unit
|
|
219
|
+
- **CVNA negative gross-profit-per-unit**: a frequently-cited Carvana red flag,
|
|
221
220
|
but the per-unit figure I could find cleanly was a derived/analyst number, not a
|
|
222
221
|
single line item I could quote verbatim from the 10-K at the cutoff. Dropped in
|
|
223
222
|
favor of the directly-quoted total debt, interest expense, ADESA price, related
|
|
224
223
|
party, and net loss.
|
|
225
224
|
|
|
226
|
-
- **PTON / SI specific debt-covenant or going-concern language
|
|
225
|
+
- **PTON / SI specific debt-covenant or going-concern language**: I searched for
|
|
227
226
|
explicit "substantial doubt / going concern" wording in both filings and did
|
|
228
227
|
**not** find it at these cutoffs (it came later). I did not invent it. The facts
|
|
229
228
|
included are the ones actually present in the cutoff-date document.
|
|
@@ -248,5 +247,5 @@ for (const company of investmentThesisSet) {
|
|
|
248
247
|
```
|
|
249
248
|
|
|
250
249
|
The grader is deterministic and model-free, so the same thesis always scores the
|
|
251
|
-
same and the answer key never reaches a model the loop could observe
|
|
250
|
+
same and the answer key never reaches a model the loop could observe, the same
|
|
252
251
|
firewall the deep-question exam (`tests/loops/held-out-exam.ts`) uses.
|
|
@@ -12,7 +12,7 @@ SOTA RAG evaluation requires retrieval quality, context quality, generated-answe
|
|
|
12
12
|
| [TruLens RAG Triad](https://www.trulens.org/getting_started/core_concepts/rag_triad/) | The minimal end-to-end triad is context relevance, groundedness, and answer relevance. |
|
|
13
13
|
| [RAGChecker](https://papers.nips.cc/paper_files/paper/2024/hash/27245589131d17368cccdfa990cbf16e-Abstract-Datasets_and_Benchmarks_Track.html) | Fine-grained diagnosis should separate retrieval misses, noisy context, and unsupported generated claims. |
|
|
14
14
|
| [BEIR](https://arxiv.org/abs/2104.08663) / TREC-style retrieval | Retrieval still needs classical rank metrics: Recall@k, Precision@k, MRR, MAP, and nDCG. |
|
|
15
|
-
| [CRAG](https://arxiv.org/abs/2406.04744) | Real RAG evals must include long-tail, dynamic, multi-hop, and unanswerable questions
|
|
15
|
+
| [CRAG](https://arxiv.org/abs/2406.04744) | Real RAG evals must include long-tail, dynamic, multi-hop, and unanswerable questions alongside easy static facts. |
|
|
16
16
|
| [DeepEval RAG metrics](https://deepeval.com/docs/metrics-faithfulness) | Production tools converge on faithfulness, answer relevance, and context relevance as generator/retriever checks. |
|
|
17
17
|
|
|
18
18
|
## Current Repo Status
|
|
@@ -30,12 +30,13 @@ Done:
|
|
|
30
30
|
- `normalizeExternalRagScores()` and the row exporters make Ragas, DeepEval, TruLens, RAGChecker, and custom evaluator outputs pluggable instead of hard dependencies.
|
|
31
31
|
- `scoreKnowledgeBaseIndex()` validates generic wiki/KB health: citation coverage, source-backed pages, stale sources, duplicate source hashes, and lint/validation errors.
|
|
32
32
|
- `calibrateRagAnswerJudge()` enforces the strong-vs-weak metric check before trusting a RAG answer metric.
|
|
33
|
+
- `runKnowledgeImprovementJob()` in `@tangle-network/agent-runtime` connects runtime backends and worker factories to candidate KB creation, readiness checks, frozen comparisons, spend measurement, and explicit activation.
|
|
34
|
+
- `agent-knowledge` remains runtime-free; applications with their own agent runner pass an `updateKnowledge` callback directly.
|
|
33
35
|
|
|
34
36
|
Not done:
|
|
35
37
|
|
|
36
38
|
- Slice-level reporting for freshness, distractors, multi-hop, and long-tail cases.
|
|
37
|
-
-
|
|
38
|
-
The lifecycle API accepts those agents as hooks today; it does not hardcode provider-specific workers.
|
|
39
|
+
- A maintained, public benchmark pack with at least 100 labeled scenarios and published baseline results.
|
|
39
40
|
|
|
40
41
|
## Completion Criteria
|
|
41
42
|
|
|
@@ -95,7 +96,7 @@ Ship criteria:
|
|
|
95
96
|
|
|
96
97
|
- Every failed eval has one primary failure class.
|
|
97
98
|
- At least 95 percent of generated claims can be mapped to supporting context, contradicted context, or no context.
|
|
98
|
-
- Reports show metrics by slice and
|
|
99
|
+
- Reports show metrics by slice and failure class as well as the aggregate score.
|
|
99
100
|
|
|
100
101
|
### Phase 4: Production Loop
|
|
101
102
|
|
|
@@ -115,5 +116,5 @@ Ship criteria:
|
|
|
115
116
|
|
|
116
117
|
1. Add slice-level aggregation helpers for freshness, distractors, multi-hop, long-tail, and unanswerable cases.
|
|
117
118
|
2. Add forbidden/stale source targets to retrieval scenarios.
|
|
118
|
-
3. Add
|
|
119
|
+
3. Add a maintained benchmark pack with at least 100 labeled scenarios and a reproducible baseline report.
|
|
119
120
|
4. Add a CLI command that runs the lifecycle loop and writes a reproducible report under `.agent-knowledge/eval/`.
|
package/docs/results/adaptive.md
CHANGED
|
@@ -1,9 +1,9 @@
|
|
|
1
1
|
# Adaptive topology: spend the LLM verifier only when it pays
|
|
2
2
|
|
|
3
3
|
The cost/quality A/B (`docs/results/cost-quality.md`) found the LLM relevance
|
|
4
|
-
verifier's cleanliness win is dominated by **de-duplication
|
|
4
|
+
verifier's cleanliness win is dominated by **de-duplication**, which a
|
|
5
5
|
deterministic content-hash / canonical-URL check captures at ~none of the LLM
|
|
6
|
-
premium
|
|
6
|
+
premium, and that an LLM check only earns its dollar on the off-scope tail. The
|
|
7
7
|
production move it named was: do the cheap deterministic work first, reserve the
|
|
8
8
|
LLM for the ambiguous tail. `createAdaptiveResearchDriver`
|
|
9
9
|
(`src/adaptive-driver.ts`) is that driver, and this is its measurement.
|
|
@@ -19,7 +19,7 @@ stopping at the first that decides:
|
|
|
19
19
|
github, …) with a substantial body is **kept**; an obvious spam/listicle
|
|
20
20
|
title or a too-thin body is **dropped**; everything else is **ambiguous**.
|
|
21
21
|
3. **LLM escalation ($).** Only ambiguous survivors reach the shipped LLM
|
|
22
|
-
relevance verifier (`createVerifyingResearchDriver`)
|
|
22
|
+
relevance verifier (`createVerifyingResearchDriver`), one call each.
|
|
23
23
|
|
|
24
24
|
## Live frontier, n=5 topics (glm-5.2)
|
|
25
25
|
|
|
@@ -39,24 +39,24 @@ is the per-arm `RouterClient.usage()` diff (#36). Total spend for the run:
|
|
|
39
39
|
| **total** | **25** | **25** | **12 / 25 / $0.0261** | **20 / 6 / $0.0068** |
|
|
40
40
|
|
|
41
41
|
**Cost.** Adaptive cuts LLM verifier calls **76%** (25 → 6) and dollars **74%**
|
|
42
|
-
($0.0261 → $0.0068). On 3 of the 5 topics it spent **zero** LLM calls
|
|
42
|
+
($0.0261 → $0.0068). On 3 of the 5 topics it spent **zero** LLM calls, every
|
|
43
43
|
unique survivor was on an authoritative host, so the $0 stages decided
|
|
44
44
|
everything.
|
|
45
45
|
|
|
46
|
-
##
|
|
46
|
+
## Interpretation: adaptive occupies one cost-quality point
|
|
47
47
|
|
|
48
48
|
Admitted-source counts (lower = cleaner KB): **single 25, adaptive 20,
|
|
49
49
|
full-LLM 12**. Adaptive sits **between** the two:
|
|
50
50
|
|
|
51
51
|
- It removes **5 of the 13 sources** the full-LLM judge rejects that the
|
|
52
52
|
single-agent loop keeps (every one of them a real duplicate caught by the $0
|
|
53
|
-
dedup stage
|
|
53
|
+
dedup stage, exactly the de-dup-dominated win the cost/quality result
|
|
54
54
|
predicted).
|
|
55
55
|
- It does **NOT** match full-LLM cleanliness. The remaining 8 sources full-LLM
|
|
56
56
|
rejects, adaptive keeps. The cause is structural and visible in the trace: on
|
|
57
57
|
this topic set every non-duplicate survivor landed on an authoritative host
|
|
58
58
|
(arxiv / github / official docs), so the heuristic **kept** it without ever
|
|
59
|
-
asking the LLM
|
|
59
|
+
asking the LLM, and the LLM, when full-LLM did ask it, judged several of
|
|
60
60
|
those same authoritative pages not-quite-on-topic and dropped them. The host
|
|
61
61
|
prior is coarser than the relevance judge.
|
|
62
62
|
|
|
@@ -65,19 +65,19 @@ de-dup half of full-LLM's cleanliness for ~26% of full-LLM's dollars, and gives
|
|
|
65
65
|
up the relevance-judgment half.** Whether that is the right point depends on the
|
|
66
66
|
cost of a kept-but-marginal source. If a slightly-off-topic authoritative page
|
|
67
67
|
is cheap to carry, adaptive dominates. If every admitted source must clear a
|
|
68
|
-
relevance bar, the host heuristic is too permissive and you want the full LLM
|
|
68
|
+
relevance bar, the host heuristic is too permissive and you want the full LLM,
|
|
69
69
|
or a tightened heuristic.
|
|
70
70
|
|
|
71
|
-
## Where the heuristic is weak
|
|
71
|
+
## Where the heuristic is weak: stated plainly
|
|
72
72
|
|
|
73
73
|
The escalation count is the diagnostic. On 3 of 5 topics it was **zero**: the
|
|
74
74
|
heuristic never deferred to the LLM, so on those topics adaptive is a
|
|
75
75
|
**pure host/title/length rule**, and its cleanliness is exactly that rule's
|
|
76
|
-
cleanliness
|
|
76
|
+
cleanliness, no smarter than "trust arxiv/github, drop spam." That is fine when
|
|
77
77
|
the worker's sources are dominated by authoritative hosts (as here), but it
|
|
78
78
|
means the LLM's relevance judgment is contributing nothing on those topics, by
|
|
79
79
|
construction. The two topics where adaptive *did* escalate (grouped-query
|
|
80
|
-
attention, LoRA) are where some survivors were on unknown hosts
|
|
80
|
+
attention, LoRA) are where some survivors were on unknown hosts, and there the
|
|
81
81
|
3 LLM calls per topic are the off-scope tail the verifier is actually for.
|
|
82
82
|
|
|
83
83
|
The heuristic would mis-route in two directions a richer worker would expose,
|
|
@@ -95,12 +95,12 @@ neither seen on this authoritative-host-heavy set:
|
|
|
95
95
|
The deployable recommendation from the cost/quality result was "deterministic
|
|
96
96
|
dedup first, reserve the LLM for the tail." This driver ships that and the
|
|
97
97
|
measurement confirms the **cost** half cleanly (76% fewer calls, 74% cheaper)
|
|
98
|
-
and qualifies the **quality** half
|
|
98
|
+
and qualifies the **quality** half accurately: adaptive captures the de-dup
|
|
99
99
|
cleanliness (the dominant, free win) but not the LLM's relevance cleanliness,
|
|
100
100
|
because the host heuristic resolves authoritative survivors without asking. For
|
|
101
101
|
a worker whose sources are mostly authoritative, adaptive is the right frontier
|
|
102
102
|
point. For one whose junk is on-topic-looking pages on unknown hosts, the
|
|
103
|
-
ambiguous tail grows and adaptive converges toward full-LLM cost
|
|
103
|
+
ambiguous tail grows and adaptive converges toward full-LLM cost, which is the
|
|
104
104
|
correct behavior: it pays for the LLM exactly when the cheap signals can't
|
|
105
105
|
decide.
|
|
106
106
|
|
|
@@ -1,22 +1,22 @@
|
|
|
1
1
|
# Claim-grounding: the band where the verifier earns its dollar
|
|
2
2
|
|
|
3
3
|
`docs/results/cost-quality.md` found the relevance verifier's cleanliness win is
|
|
4
|
-
**dominated by de-duplication
|
|
4
|
+
**dominated by de-duplication**, a deterministic content-hash captures most of it
|
|
5
5
|
at ~none of the LLM premium. So the open question was: is there an error band where
|
|
6
|
-
a verifier earns its cost
|
|
6
|
+
a verifier earns its cost, something a hash AND a relevance judge both miss?
|
|
7
7
|
|
|
8
8
|
**Yes: misattributed citations.** A source that is on-topic, unique, and real, but
|
|
9
9
|
whose cited CLAIM does not appear in the page (the LLM wrote a plausible sentence
|
|
10
10
|
and hung a real URL off it). De-dup passes it (it's unique). A relevance judge
|
|
11
11
|
passes it (the page is on-topic). Only checking the claim against the fetched text
|
|
12
|
-
catches it
|
|
12
|
+
catches it, and that check is **deterministic text presence, $0 inference**.
|
|
13
13
|
|
|
14
14
|
## The mode
|
|
15
15
|
|
|
16
16
|
Each proposed source now carries the specific claim it is cited for
|
|
17
17
|
(`withCitedClaim` → `metadata.citedClaim`). The verifier
|
|
18
18
|
(`createClaimGroundingVerifier`) runs `groundClaimInText(claim, pageText)` over the
|
|
19
|
-
`htmlToText` output of the page the worker actually fetched
|
|
19
|
+
`htmlToText` output of the page the worker actually fetched, verbatim, normalized
|
|
20
20
|
(punctuation/whitespace-insensitive), or a ≥70% content-word overlap close
|
|
21
21
|
paraphrase. A claim that isn't present is rejected as **misattributed**. The oracle
|
|
22
22
|
is text presence, not a model call, so it composes with the LLM relevance verifier
|
|
@@ -31,25 +31,25 @@ proposals. Cost diffed per arm with the #36 `RouterClient.usage()` instrumentati
|
|
|
31
31
|
|
|
32
32
|
| n=5 topics | misattributions caught | marginal $ | $/topic | per-$ caught |
|
|
33
33
|
|---|---|---|---|---|
|
|
34
|
-
| no-verifier | 0 / 5 | $0.0000 |
|
|
34
|
+
| no-verifier | 0 / 5 | $0.0000 | n/a | n/a |
|
|
35
35
|
| relevance (LLM judge) | **4 / 5** | $0.0157 | ~$0.0031 | 254 |
|
|
36
36
|
| claim-grounding (text) | **5 / 5** | **$0.0000** | $0 | ∞ |
|
|
37
37
|
|
|
38
38
|
Per-topic (caught relevance / grounding): self-speculative decoding 1/1, rotary
|
|
39
39
|
position embeddings 1/1, grouped-query attention 1/1, **KV-cache quantization 0/1**,
|
|
40
|
-
LoRA 1/1. (An earlier 3-topic run missed self-speculative decoding instead
|
|
40
|
+
LoRA 1/1. (An earlier 3-topic run missed self-speculative decoding instead, the
|
|
41
41
|
miss moves around; it is not a fixed topic.)
|
|
42
42
|
|
|
43
43
|
**Reading.** Claim-grounding catches every misattribution at **$0**; the relevance
|
|
44
44
|
judge catches most but **misses one in five at ~$0.003/topic**. The miss is the
|
|
45
45
|
point: the relevance verifier only ever sees the page text, never the cited claim,
|
|
46
|
-
so it is *structurally blind* to misattribution. It catches one only by accident
|
|
46
|
+
so it is *structurally blind* to misattribution. It catches one only by accident:
|
|
47
47
|
when the fabricated claim happens to also make the page read off-topic
|
|
48
48
|
(e.g. a "12-billion-parameter draft transformer" claim on a rotary-embeddings page).
|
|
49
49
|
When the fabrication stays on-topic (the KV-cache case), the judge waves it through.
|
|
50
50
|
|
|
51
51
|
So on THIS band the verifier-per-dollar comparison inverts the cost/quality result:
|
|
52
|
-
there, the LLM verifier bought a dedup-shaped gain a free hash already captures
|
|
52
|
+
there, the LLM verifier bought a dedup-shaped gain a free hash already captures;
|
|
53
53
|
expensive for what a cheap rule does. Here the cheap, deterministic check
|
|
54
54
|
**dominates** the expensive judge: it catches strictly more (5/5 vs 4/5) at strictly
|
|
55
55
|
less ($0 vs $0.0157). The verifier earns its dollar on misattribution; it does not on
|
|
@@ -63,7 +63,7 @@ de-duplication.
|
|
|
63
63
|
(and the offline relevance stand-in) accept them. The error is in the *claim*, not
|
|
64
64
|
the *topic*.
|
|
65
65
|
- **Executable ground truth.** The check is presence/close-paraphrase of the claim
|
|
66
|
-
in the fetched text
|
|
66
|
+
in the fetched text, deployable in production with no oracle and no model call.
|
|
67
67
|
|
|
68
68
|
The offline arm proves the floor with a controlled 4-source pool (2 grounded, 2
|
|
69
69
|
misattributed): claim-grounding admits **0/2** misattributions and keeps **2/2**
|
|
@@ -78,9 +78,9 @@ grounded sources, while relevance and no-verifier both admit **2/2**.
|
|
|
78
78
|
- **Planted misattributions, not naturally-occurring ones.** Like the cost/quality
|
|
79
79
|
offline floor, the misattribution is injected so the band is measurable. It models
|
|
80
80
|
the real LLM citation-fabrication failure but does not measure its base rate in the
|
|
81
|
-
wild
|
|
81
|
+
wild, that needs a corpus of model-written citations checked by hand.
|
|
82
82
|
- **The grounding oracle is conservative.** A real paraphrase whose inflected words
|
|
83
|
-
differ from the page ("drafts" vs "draft") can score below 0.7 and be rejected
|
|
83
|
+
differ from the page ("drafts" vs "draft") can score below 0.7 and be rejected,
|
|
84
84
|
a false-positive misattribution flag. `minOverlap` tunes this; the worker should
|
|
85
85
|
cite the page's own key terms (the `createClaimDecorator` extractor is told to).
|
|
86
86
|
|
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
Live 9-topic A/B (glm-5.2, budget B ≤ 4 passes/arm), measured per arm with the
|
|
4
4
|
router-client instrumentation. The original A/B reported only admitted-sources at
|
|
5
|
-
"equal passes"
|
|
5
|
+
"equal passes", which charged the two-agent verify step as one pass while it is
|
|
6
6
|
actually N `verifySource` LLM calls. Pricing the calls shows what that hid.
|
|
7
7
|
|
|
8
8
|
| per topic (mean) | two-agent | single-agent | ratio |
|
|
@@ -11,7 +11,7 @@ actually N `verifySource` LLM calls. Pricing the calls shows what that hid.
|
|
|
11
11
|
| tokens (in+out) | ~4,900 | ~530 | ~9× |
|
|
12
12
|
| cost (USD) | ~$0.0072 | ~$0.0013 | ~5.5× |
|
|
13
13
|
| latency (wall) | ~37 s | ~11 s | ~3.4× |
|
|
14
|
-
| cleanliness Δ (single − two admitted) |
|
|
14
|
+
| cleanliness Δ (single − two admitted) | n/a | n/a | +1.56, 95% CI [0.33, 2.67] |
|
|
15
15
|
|
|
16
16
|
Per-topic Δ (single − two admitted) this run: self-speculative decoding +4,
|
|
17
17
|
grouped-query attention 0, rotary position embeddings +1, KV-cache quantization
|
|
@@ -19,9 +19,9 @@ grouped-query attention 0, rotary position embeddings +1, KV-cache quantization
|
|
|
19
19
|
descent **−1**. Coverage 1.00 every topic, both arms.
|
|
20
20
|
|
|
21
21
|
**Reading.** The verifier buys ~1.5–2.7 fewer junk sources for roughly **5× the
|
|
22
|
-
dollars, 9× the tokens, and 3× the latency
|
|
22
|
+
dollars, 9× the tokens, and 3× the latency**, and on two topics it admitted *more*
|
|
23
23
|
than the single agent (the cleanliness signal is real but noisier than the +2.3 /
|
|
24
24
|
+2.7 of earlier runs). The cleanliness gain is dominated by de-duplication, so the
|
|
25
|
-
|
|
25
|
+
production move is a deterministic content-hash / canonical-URL dedup, which
|
|
26
26
|
captures most of the cleanliness at ~none of this premium; reserve an LLM check for
|
|
27
27
|
the off-scope tail. This is the cost half the "equal passes" framing left out.
|