@tangle-network/agent-knowledge 4.0.0 → 4.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (35) hide show
  1. package/AGENTS.md +32 -18
  2. package/CHANGELOG.md +32 -0
  3. package/README.md +31 -19
  4. package/dist/benchmarks/index.d.ts +51 -3
  5. package/dist/benchmarks/index.js +1 -1
  6. package/dist/{chunk-EUAXSBMH.js → chunk-AKYJG2MR.js} +224 -224
  7. package/dist/chunk-AKYJG2MR.js.map +1 -0
  8. package/dist/{chunk-DW6APRTX.js → chunk-BBCIB4UL.js} +2486 -2457
  9. package/dist/chunk-BBCIB4UL.js.map +1 -0
  10. package/dist/{chunk-UWOTQNBI.js → chunk-CPMLJYA3.js} +1886 -1842
  11. package/dist/chunk-CPMLJYA3.js.map +1 -0
  12. package/dist/{chunk-WCYW2GDA.js → chunk-MYFM6LKH.js} +2 -2
  13. package/dist/chunk-MYFM6LKH.js.map +1 -0
  14. package/dist/cli.js +1 -1
  15. package/dist/index.d.ts +48 -52
  16. package/dist/index.js +2906 -2874
  17. package/dist/index.js.map +1 -1
  18. package/dist/memory/index.d.ts +129 -6
  19. package/dist/memory/index.js +2 -2
  20. package/dist/sources/index.d.ts +17 -33
  21. package/dist/sources/index.js +1 -1
  22. package/dist/{index-Bqg1mBPt.d.ts → types-DP38encz.d.ts} +127 -284
  23. package/docs/eval/investment-material-facts.md +28 -29
  24. package/docs/eval/rag-eval-roadmap.md +6 -5
  25. package/docs/results/adaptive.md +13 -13
  26. package/docs/results/claim-grounding.md +11 -11
  27. package/docs/results/cost-quality.md +4 -4
  28. package/docs/results/investment-thesis.md +56 -56
  29. package/docs/results/research-driving.md +54 -54
  30. package/docs/two-agent-research-ab.md +94 -94
  31. package/package.json +4 -7
  32. package/dist/chunk-DW6APRTX.js.map +0 -1
  33. package/dist/chunk-EUAXSBMH.js.map +0 -1
  34. package/dist/chunk-UWOTQNBI.js.map +0 -1
  35. package/dist/chunk-WCYW2GDA.js.map +0 -1
@@ -1,10 +1,10 @@
1
- # Held-out investment-research eval set material facts + provenance
1
+ # Held-out investment-research eval set: material facts + provenance
2
2
 
3
3
  This is the answer key and the provenance ledger for `tests/eval/investment-thesis-set.ts`.
4
4
 
5
5
  **What the set measures.** Give a research loop a company + ticker + an as-of
6
6
  **cutoff** date and ask it to write an investment thesis. Then grade that thesis
7
- against the held-out **material facts** below facts the loop never saw. A high
7
+ against the held-out **material facts** below, facts the loop never saw. A high
8
8
  score means the thesis surfaced the buried, material, non-obvious drivers a
9
9
  thorough analyst would flag and a single ticker search would miss; it is **not**
10
10
  teaching-to-the-test, because the answer key is firewalled from every loop and
@@ -20,14 +20,13 @@ the grader is a `$0`, model-free substring check (`gradeFactAgainstText`).
20
20
  guessed (see the drop log).
21
21
  3. **Knowable at the cutoff.** Every value was disclosed in, or computable from, a
22
22
  filing available on or before the cutoff. The eventual collapse is **not** a
23
- checklist item it is recorded as `knownOutcome`, for the reader only, and is
24
- never graded.
23
+ checklist item. It is recorded as `knownOutcome` for the reader and is never graded.
25
24
 
26
25
  All five primary documents were fetched live from `https://www.sec.gov/Archives/`
27
26
  during curation (a `curl` with a descriptive `User-Agent`, per SEC fair-access
28
27
  rules). Every dollar figure below was read directly out of the de-tagged filing
29
- text. Provenance is verifiable: each `sourceUrl` contains the company's SEC CIK,
30
- and `tests/eval/investment-thesis-set.test.ts` asserts that invariant.
28
+ text. Each `sourceUrl` contains the company's SEC CIK, and
29
+ `tests/eval/investment-thesis-set.test.ts` checks that invariant.
31
30
 
32
31
  ---
33
32
 
@@ -47,7 +46,7 @@ set was curated (June 2026); `investment-thesis-set.test.ts` asserts this.
47
46
 
48
47
  ---
49
48
 
50
- ## SIVB SVB Financial Group (cutoff 2023-02-24)
49
+ ## SIVB: SVB Financial Group (cutoff 2023-02-24)
51
50
 
52
51
  Source: [FY2022 10-K (`sivb-20221231.htm`)](https://www.sec.gov/Archives/edgar/data/719739/000071973923000021/sivb-20221231.htm)
53
52
  **Known outcome (not graded):** FDIC receivership March 10, 2023; holding-company Chapter 11 March 17, 2023.
@@ -59,14 +58,14 @@ Source: [FY2022 10-K (`sivb-20221231.htm`)](https://www.sec.gov/Archives/edgar/d
59
58
  | SIVB/f3 | concentration | Run-prone uninsured deposit base | "estimated uninsured deposits in U.S. offices that exceed the FDIC insurance limit were **$151.5 billion**" |
60
59
  | SIVB/f4 | margin-trend | Cheap deposits fleeing → funding cost set to rise | "Noninterest-bearing demand deposits to total deposits decreased by **20 percentage points to 47 percent**" |
61
60
  | SIVB/f5 | concentration | Single-client-type (innovation-economy) deposit + credit base | 10-K frames the franchise around "the innovation economy" (technology, life-science, venture) |
62
- | SIVB/f6 | off-balance-sheet | AFS loss in AOCI the visible, smaller tip | "Available-for-sale securities, at fair value (cost of $ **28,602**) **26,069**" → ~$2.5B AFS loss in AOCI |
61
+ | SIVB/f6 | off-balance-sheet | AFS loss in AOCI; the visible, smaller tip | "Available-for-sale securities, at fair value (cost of $ **28,602**) **26,069**" → ~$2.5B AFS loss in AOCI |
63
62
 
64
63
  The decisive, non-obvious fact is SIVB/f1+f2: an interest-rate loss roughly equal
65
64
  to all of equity, sitting in the footnotes because HTM accounting keeps it out of
66
65
  both earnings and book equity. A ticker search shows a profitable bank; the
67
66
  filing shows a mark-to-market hole the size of its capital.
68
67
 
69
- ## BBBY Bed Bath & Beyond Inc. (cutoff 2022-04-21)
68
+ ## BBBY: Bed Bath & Beyond Inc. (cutoff 2022-04-21)
70
69
 
71
70
  Source: [FY2021 10-K (`bbby-20220226.htm`)](https://www.sec.gov/Archives/edgar/data/886158/000088615822000047/bbby-20220226.htm)
72
71
  **Known outcome (not graded):** Chapter 11 on April 23, 2023; equity wiped out.
@@ -80,11 +79,11 @@ Source: [FY2021 10-K (`bbby-20220226.htm`)](https://www.sec.gov/Archives/edgar/d
80
79
  | BBBY/f5 | margin-trend | Inventory building into a demand decline | "Merchandise inventories **1,725,410** 1,671,909" ($ thousands) |
81
80
 
82
81
  The non-obvious fact is BBBY/f1+f2+f4 together: in FY2021 the company **lost $560M,
83
- generated only $18M of operating cash, and still spent $575M buying back stock** —
82
+ generated only $18M of operating cash, and still spent $575M buying back stock**,
84
83
  returning more cash than it had. The buyback, not the income statement alone, is
85
84
  why a $1.3B equity base became $174M.
86
85
 
87
- ## CVNA Carvana Co. (cutoff 2023-02-23)
86
+ ## CVNA: Carvana Co. (cutoff 2023-02-23)
88
87
 
89
88
  Source: [FY2022 10-K (`cvna-20221231.htm`)](https://www.sec.gov/Archives/edgar/data/1690820/000169082023000052/cvna-20221231.htm)
90
89
  **Known outcome (not graded):** Stock fell ~98% from its 2021 peak; a 2023 debt exchange cut and extended obligations, narrowly avoiding bankruptcy.
@@ -97,18 +96,18 @@ Source: [FY2022 10-K (`cvna-20221231.htm`)](https://www.sec.gov/Archives/edgar/d
97
96
  | CVNA/f4 | governance | Recurring related-party leases with the founder's family | Related-Party note: DriveTime, controlled by "Ernest Garcia II, Ernest Garcia III, and entities controlled by one or both of them" |
98
97
  | CVNA/f5 | liquidity | A wide loss showing unit economics had not turned | "Net loss $ (**2,894**)" ($ millions) |
99
98
 
100
- The non-obvious facts are CVNA/f2 (interest expense up 2.8x the debt was now
101
- expensive, not just large) and CVNA/f4 (the controlling Garcia family on both
99
+ The non-obvious facts are CVNA/f2 (interest expense up 2.8x, making the debt expensive
100
+ as well as large) and CVNA/f4 (the controlling Garcia family on both
102
101
  sides of material leases via DriveTime), neither of which a ticker quote shows.
103
102
 
104
- ## PTON Peloton Interactive, Inc. (cutoff 2022-09-07)
103
+ ## PTON: Peloton Interactive, Inc. (cutoff 2022-09-07)
105
104
 
106
105
  Source: [FY2022 10-K (`pton-20220630.htm`)](https://www.sec.gov/Archives/edgar/data/1639825/000163982522000117/pton-20220630.htm)
107
106
  **Known outcome (not graded):** Stock fell ~95% from its 2021 peak; founder-CEO departed; mass layoffs and a multi-year turnaround.
108
107
 
109
108
  | ID | Lens | Material fact | Value read from the filing |
110
109
  |---|---|---|---|
111
- | PTON/f1 | margin-trend | Hardware gross margin turned **negative** | Connected Fitness "Gross Margin decreased to (**11**)" percent losing money per unit sold |
110
+ | PTON/f1 | margin-trend | Hardware gross margin turned **negative** | Connected Fitness "Gross Margin decreased to (**11**)" percent; losing money per unit sold |
112
111
  | PTON/f2 | liquidity | Inventory glut as pandemic demand normalized | "Inventories, net **1,104.5** 937" ($ millions) |
113
112
  | PTON/f3 | governance | Dual-class super-voting control | "Class B common stock has **20 votes per share** and our Class A common stock has one vote per share" |
114
113
  | PTON/f4 | liquidity | An order-of-magnitude wider loss | "Net loss $ (**2,827**)" ($ millions) |
@@ -116,11 +115,11 @@ Source: [FY2022 10-K (`pton-20220630.htm`)](https://www.sec.gov/Archives/edgar/d
116
115
  | PTON/f6 | leverage | Locked-in purchase commitments into falling demand | "purchase commitments related to the manufacture of Peloton products were estimated to be approximately **$334**" million |
117
116
 
118
117
  The non-obvious fact is PTON/f1: revenue was still large, but the **hardware was
119
- sold below cost** (−11% gross margin) the unit economics, not just the growth
120
- rate, had broken. PTON/f6 compounds it: the company was contractually obliged to
118
+ sold below cost** (−11% gross margin). The unit economics had broken despite the
119
+ top-line growth. PTON/f6 compounds it: the company was contractually obliged to
121
120
  buy more inventory it could not sell.
122
121
 
123
- ## SI Silvergate Capital Corporation (cutoff 2022-02-28)
122
+ ## SI: Silvergate Capital Corporation (cutoff 2022-02-28)
124
123
 
125
124
  Source: [FY2021 10-K (`si-20211231.htm`)](https://www.sec.gov/Archives/edgar/data/1312109/000131210922000051/si-20211231.htm)
126
125
  **Known outcome (not graded):** After the FTX collapse triggered a deposit run, Silvergate announced a voluntary wind-down and liquidation of Silvergate Bank in March 2023.
@@ -134,7 +133,7 @@ Source: [FY2021 10-K (`si-20211231.htm`)](https://www.sec.gov/Archives/edgar/dat
134
133
  | SI/f5 | concentration | The moat AND the funding are the same crypto-only bet | "Silvergate Exchange Network ('SEN'), our proprietary ... payment network for participants in the digital currency industry" |
135
134
 
136
135
  The non-obvious fact is SI/f1+f2 together: **99.5% noninterest-bearing deposits,
137
- ~58% from crypto exchanges** a funding base with no contractual term and a
136
+ ~58% from crypto exchanges**, a funding base with no contractual term and a
138
137
  single correlated counterparty type. A ticker quote shows a fast-growing,
139
138
  low-cost-funding bank; the filing shows a bank that could be emptied in days if
140
139
  crypto sentiment turned.
@@ -176,8 +175,8 @@ should not be read as a general "investment-research quality" number. The biases
176
175
 
177
176
  3. **Sector skew toward financials + distressed consumer.** Two of five are banks
178
177
  (SIVB, SI). The two banks are deliberately given **different** buried-risk
179
- lenses SIVB is an interest-rate / duration / off-balance-sheet story, SI is a
180
- single-industry deposit-concentration story so they are not redundant, but
178
+ lenses, SIVB is an interest-rate / duration / off-balance-sheet story, SI is a
179
+ single-industry deposit-concentration story, so they are not redundant, but
181
180
  the set still over-indexes on balance-sheet fragility and under-tests, e.g.,
182
181
  technology platform risk, supply-chain concentration, or accounting-policy
183
182
  aggressiveness in a healthy grower.
@@ -197,33 +196,33 @@ should not be read as a general "investment-research quality" number. The biases
197
196
  required a second independent source to establish was dropped (below) rather
198
197
  than sourced to the filing alone.
199
198
 
200
- ## Drop log items considered and NOT included (honesty over coverage)
199
+ ## Drop log: excluded items
201
200
 
202
201
  These were candidate facts I could not independently ground to a primary source
203
202
  available at the cutoff, so I **dropped them rather than guess**:
204
203
 
205
- - **First Republic Bank (FRC)** dropped as a company entirely. First Republic
204
+ - **First Republic Bank (FRC)**: dropped as a company entirely. First Republic
206
205
  was a state-chartered bank that filed its annual reports with the **FDIC**, not
207
206
  on SEC EDGAR, so its 10-K is not at a `sec.gov/Archives` URL and I could not give
208
- it the same clean, CIK-verifiable provenance as the other five. Its widely cited
207
+ it the same CIK-matched provenance as the other five. Its widely cited
209
208
  figures (~$15B HTM-style loss, ~$119.5B uninsured deposits) are real but are best
210
- sourced from the FDIC OIG Material Loss Review, a **post**-cutoff document using
209
+ sourced from the FDIC OIG Material Loss Review, a **post**-cutoff document, using
211
210
  it would violate rule 3. Replaced with Silvergate, whose 10-K is on EDGAR.
212
211
 
213
- - **SIVB exact total unrealized-loss footnote line** I report the HTM gap as the
212
+ - **SIVB exact total unrealized-loss footnote line**: I report the HTM gap as the
214
213
  arithmetic difference of two figures printed on the balance sheet
215
214
  ($91,321M − $76,169M), which is exact and on-cutoff. I did **not** include a
216
215
  separately-quoted "$15.1B net unrealized loss" sentence because I did not locate
217
216
  that exact phrasing in the de-tagged text; quoting a number I could not point to
218
217
  verbatim would break rule 2. The computed value is conservative and checkable.
219
218
 
220
- - **CVNA negative gross-profit-per-unit** a frequently-cited Carvana red flag,
219
+ - **CVNA negative gross-profit-per-unit**: a frequently-cited Carvana red flag,
221
220
  but the per-unit figure I could find cleanly was a derived/analyst number, not a
222
221
  single line item I could quote verbatim from the 10-K at the cutoff. Dropped in
223
222
  favor of the directly-quoted total debt, interest expense, ADESA price, related
224
223
  party, and net loss.
225
224
 
226
- - **PTON / SI specific debt-covenant or going-concern language** I searched for
225
+ - **PTON / SI specific debt-covenant or going-concern language**: I searched for
227
226
  explicit "substantial doubt / going concern" wording in both filings and did
228
227
  **not** find it at these cutoffs (it came later). I did not invent it. The facts
229
228
  included are the ones actually present in the cutoff-date document.
@@ -248,5 +247,5 @@ for (const company of investmentThesisSet) {
248
247
  ```
249
248
 
250
249
  The grader is deterministic and model-free, so the same thesis always scores the
251
- same and the answer key never reaches a model the loop could observe the same
250
+ same and the answer key never reaches a model the loop could observe, the same
252
251
  firewall the deep-question exam (`tests/loops/held-out-exam.ts`) uses.
@@ -12,7 +12,7 @@ SOTA RAG evaluation requires retrieval quality, context quality, generated-answe
12
12
  | [TruLens RAG Triad](https://www.trulens.org/getting_started/core_concepts/rag_triad/) | The minimal end-to-end triad is context relevance, groundedness, and answer relevance. |
13
13
  | [RAGChecker](https://papers.nips.cc/paper_files/paper/2024/hash/27245589131d17368cccdfa990cbf16e-Abstract-Datasets_and_Benchmarks_Track.html) | Fine-grained diagnosis should separate retrieval misses, noisy context, and unsupported generated claims. |
14
14
  | [BEIR](https://arxiv.org/abs/2104.08663) / TREC-style retrieval | Retrieval still needs classical rank metrics: Recall@k, Precision@k, MRR, MAP, and nDCG. |
15
- | [CRAG](https://arxiv.org/abs/2406.04744) | Real RAG evals must include long-tail, dynamic, multi-hop, and unanswerable questions, not only easy static facts. |
15
+ | [CRAG](https://arxiv.org/abs/2406.04744) | Real RAG evals must include long-tail, dynamic, multi-hop, and unanswerable questions alongside easy static facts. |
16
16
  | [DeepEval RAG metrics](https://deepeval.com/docs/metrics-faithfulness) | Production tools converge on faithfulness, answer relevance, and context relevance as generator/retriever checks. |
17
17
 
18
18
  ## Current Repo Status
@@ -30,12 +30,13 @@ Done:
30
30
  - `normalizeExternalRagScores()` and the row exporters make Ragas, DeepEval, TruLens, RAGChecker, and custom evaluator outputs pluggable instead of hard dependencies.
31
31
  - `scoreKnowledgeBaseIndex()` validates generic wiki/KB health: citation coverage, source-backed pages, stale sources, duplicate source hashes, and lint/validation errors.
32
32
  - `calibrateRagAnswerJudge()` enforces the strong-vs-weak metric check before trusting a RAG answer metric.
33
+ - `runKnowledgeImprovementJob()` in `@tangle-network/agent-runtime` connects runtime backends and worker factories to candidate KB creation, readiness checks, frozen comparisons, spend measurement, and explicit activation.
34
+ - `agent-knowledge` remains runtime-free; applications with their own agent runner pass an `updateKnowledge` callback directly.
33
35
 
34
36
  Not done:
35
37
 
36
38
  - Slice-level reporting for freshness, distractors, multi-hop, and long-tail cases.
37
- - Packaged runtime adapters for browser/coding/research agents.
38
- The lifecycle API accepts those agents as hooks today; it does not hardcode provider-specific workers.
39
+ - A maintained, public benchmark pack with at least 100 labeled scenarios and published baseline results.
39
40
 
40
41
  ## Completion Criteria
41
42
 
@@ -95,7 +96,7 @@ Ship criteria:
95
96
 
96
97
  - Every failed eval has one primary failure class.
97
98
  - At least 95 percent of generated claims can be mapped to supporting context, contradicted context, or no context.
98
- - Reports show metrics by slice and by failure class, not only the aggregate score.
99
+ - Reports show metrics by slice and failure class as well as the aggregate score.
99
100
 
100
101
  ### Phase 4: Production Loop
101
102
 
@@ -115,5 +116,5 @@ Ship criteria:
115
116
 
116
117
  1. Add slice-level aggregation helpers for freshness, distractors, multi-hop, long-tail, and unanswerable cases.
117
118
  2. Add forbidden/stale source targets to retrieval scenarios.
118
- 3. Add packaged adapters that turn `agent-runtime` browser, research, and coding agents into `runRagKnowledgeImprovementLoop()` hooks.
119
+ 3. Add a maintained benchmark pack with at least 100 labeled scenarios and a reproducible baseline report.
119
120
  4. Add a CLI command that runs the lifecycle loop and writes a reproducible report under `.agent-knowledge/eval/`.
@@ -1,9 +1,9 @@
1
1
  # Adaptive topology: spend the LLM verifier only when it pays
2
2
 
3
3
  The cost/quality A/B (`docs/results/cost-quality.md`) found the LLM relevance
4
- verifier's cleanliness win is dominated by **de-duplication** which a
4
+ verifier's cleanliness win is dominated by **de-duplication**, which a
5
5
  deterministic content-hash / canonical-URL check captures at ~none of the LLM
6
- premium and that an LLM check only earns its dollar on the off-scope tail. The
6
+ premium, and that an LLM check only earns its dollar on the off-scope tail. The
7
7
  production move it named was: do the cheap deterministic work first, reserve the
8
8
  LLM for the ambiguous tail. `createAdaptiveResearchDriver`
9
9
  (`src/adaptive-driver.ts`) is that driver, and this is its measurement.
@@ -19,7 +19,7 @@ stopping at the first that decides:
19
19
  github, …) with a substantial body is **kept**; an obvious spam/listicle
20
20
  title or a too-thin body is **dropped**; everything else is **ambiguous**.
21
21
  3. **LLM escalation ($).** Only ambiguous survivors reach the shipped LLM
22
- relevance verifier (`createVerifyingResearchDriver`) one call each.
22
+ relevance verifier (`createVerifyingResearchDriver`), one call each.
23
23
 
24
24
  ## Live frontier, n=5 topics (glm-5.2)
25
25
 
@@ -39,24 +39,24 @@ is the per-arm `RouterClient.usage()` diff (#36). Total spend for the run:
39
39
  | **total** | **25** | **25** | **12 / 25 / $0.0261** | **20 / 6 / $0.0068** |
40
40
 
41
41
  **Cost.** Adaptive cuts LLM verifier calls **76%** (25 → 6) and dollars **74%**
42
- ($0.0261 → $0.0068). On 3 of the 5 topics it spent **zero** LLM calls every
42
+ ($0.0261 → $0.0068). On 3 of the 5 topics it spent **zero** LLM calls, every
43
43
  unique survivor was on an authoritative host, so the $0 stages decided
44
44
  everything.
45
45
 
46
- ## The honest reading: adaptive is a frontier POINT, not a free lunch
46
+ ## Interpretation: adaptive occupies one cost-quality point
47
47
 
48
48
  Admitted-source counts (lower = cleaner KB): **single 25, adaptive 20,
49
49
  full-LLM 12**. Adaptive sits **between** the two:
50
50
 
51
51
  - It removes **5 of the 13 sources** the full-LLM judge rejects that the
52
52
  single-agent loop keeps (every one of them a real duplicate caught by the $0
53
- dedup stage exactly the de-dup-dominated win the cost/quality result
53
+ dedup stage, exactly the de-dup-dominated win the cost/quality result
54
54
  predicted).
55
55
  - It does **NOT** match full-LLM cleanliness. The remaining 8 sources full-LLM
56
56
  rejects, adaptive keeps. The cause is structural and visible in the trace: on
57
57
  this topic set every non-duplicate survivor landed on an authoritative host
58
58
  (arxiv / github / official docs), so the heuristic **kept** it without ever
59
- asking the LLM and the LLM, when full-LLM did ask it, judged several of
59
+ asking the LLM, and the LLM, when full-LLM did ask it, judged several of
60
60
  those same authoritative pages not-quite-on-topic and dropped them. The host
61
61
  prior is coarser than the relevance judge.
62
62
 
@@ -65,19 +65,19 @@ de-dup half of full-LLM's cleanliness for ~26% of full-LLM's dollars, and gives
65
65
  up the relevance-judgment half.** Whether that is the right point depends on the
66
66
  cost of a kept-but-marginal source. If a slightly-off-topic authoritative page
67
67
  is cheap to carry, adaptive dominates. If every admitted source must clear a
68
- relevance bar, the host heuristic is too permissive and you want the full LLM
68
+ relevance bar, the host heuristic is too permissive and you want the full LLM,
69
69
  or a tightened heuristic.
70
70
 
71
- ## Where the heuristic is weak stated plainly
71
+ ## Where the heuristic is weak: stated plainly
72
72
 
73
73
  The escalation count is the diagnostic. On 3 of 5 topics it was **zero**: the
74
74
  heuristic never deferred to the LLM, so on those topics adaptive is a
75
75
  **pure host/title/length rule**, and its cleanliness is exactly that rule's
76
- cleanliness no smarter than "trust arxiv/github, drop spam." That is fine when
76
+ cleanliness, no smarter than "trust arxiv/github, drop spam." That is fine when
77
77
  the worker's sources are dominated by authoritative hosts (as here), but it
78
78
  means the LLM's relevance judgment is contributing nothing on those topics, by
79
79
  construction. The two topics where adaptive *did* escalate (grouped-query
80
- attention, LoRA) are where some survivors were on unknown hosts and there the
80
+ attention, LoRA) are where some survivors were on unknown hosts, and there the
81
81
  3 LLM calls per topic are the off-scope tail the verifier is actually for.
82
82
 
83
83
  The heuristic would mis-route in two directions a richer worker would expose,
@@ -95,12 +95,12 @@ neither seen on this authoritative-host-heavy set:
95
95
  The deployable recommendation from the cost/quality result was "deterministic
96
96
  dedup first, reserve the LLM for the tail." This driver ships that and the
97
97
  measurement confirms the **cost** half cleanly (76% fewer calls, 74% cheaper)
98
- and qualifies the **quality** half honestly: adaptive captures the de-dup
98
+ and qualifies the **quality** half accurately: adaptive captures the de-dup
99
99
  cleanliness (the dominant, free win) but not the LLM's relevance cleanliness,
100
100
  because the host heuristic resolves authoritative survivors without asking. For
101
101
  a worker whose sources are mostly authoritative, adaptive is the right frontier
102
102
  point. For one whose junk is on-topic-looking pages on unknown hosts, the
103
- ambiguous tail grows and adaptive converges toward full-LLM cost which is the
103
+ ambiguous tail grows and adaptive converges toward full-LLM cost, which is the
104
104
  correct behavior: it pays for the LLM exactly when the cheap signals can't
105
105
  decide.
106
106
 
@@ -1,22 +1,22 @@
1
1
  # Claim-grounding: the band where the verifier earns its dollar
2
2
 
3
3
  `docs/results/cost-quality.md` found the relevance verifier's cleanliness win is
4
- **dominated by de-duplication** a deterministic content-hash captures most of it
4
+ **dominated by de-duplication**, a deterministic content-hash captures most of it
5
5
  at ~none of the LLM premium. So the open question was: is there an error band where
6
- a verifier earns its cost something a hash AND a relevance judge both miss?
6
+ a verifier earns its cost, something a hash AND a relevance judge both miss?
7
7
 
8
8
  **Yes: misattributed citations.** A source that is on-topic, unique, and real, but
9
9
  whose cited CLAIM does not appear in the page (the LLM wrote a plausible sentence
10
10
  and hung a real URL off it). De-dup passes it (it's unique). A relevance judge
11
11
  passes it (the page is on-topic). Only checking the claim against the fetched text
12
- catches it and that check is **deterministic text presence, $0 inference**.
12
+ catches it, and that check is **deterministic text presence, $0 inference**.
13
13
 
14
14
  ## The mode
15
15
 
16
16
  Each proposed source now carries the specific claim it is cited for
17
17
  (`withCitedClaim` → `metadata.citedClaim`). The verifier
18
18
  (`createClaimGroundingVerifier`) runs `groundClaimInText(claim, pageText)` over the
19
- `htmlToText` output of the page the worker actually fetched verbatim, normalized
19
+ `htmlToText` output of the page the worker actually fetched, verbatim, normalized
20
20
  (punctuation/whitespace-insensitive), or a ≥70% content-word overlap close
21
21
  paraphrase. A claim that isn't present is rejected as **misattributed**. The oracle
22
22
  is text presence, not a model call, so it composes with the LLM relevance verifier
@@ -31,25 +31,25 @@ proposals. Cost diffed per arm with the #36 `RouterClient.usage()` instrumentati
31
31
 
32
32
  | n=5 topics | misattributions caught | marginal $ | $/topic | per-$ caught |
33
33
  |---|---|---|---|---|
34
- | no-verifier | 0 / 5 | $0.0000 | | |
34
+ | no-verifier | 0 / 5 | $0.0000 | n/a | n/a |
35
35
  | relevance (LLM judge) | **4 / 5** | $0.0157 | ~$0.0031 | 254 |
36
36
  | claim-grounding (text) | **5 / 5** | **$0.0000** | $0 | ∞ |
37
37
 
38
38
  Per-topic (caught relevance / grounding): self-speculative decoding 1/1, rotary
39
39
  position embeddings 1/1, grouped-query attention 1/1, **KV-cache quantization 0/1**,
40
- LoRA 1/1. (An earlier 3-topic run missed self-speculative decoding instead the
40
+ LoRA 1/1. (An earlier 3-topic run missed self-speculative decoding instead, the
41
41
  miss moves around; it is not a fixed topic.)
42
42
 
43
43
  **Reading.** Claim-grounding catches every misattribution at **$0**; the relevance
44
44
  judge catches most but **misses one in five at ~$0.003/topic**. The miss is the
45
45
  point: the relevance verifier only ever sees the page text, never the cited claim,
46
- so it is *structurally blind* to misattribution. It catches one only by accident
46
+ so it is *structurally blind* to misattribution. It catches one only by accident:
47
47
  when the fabricated claim happens to also make the page read off-topic
48
48
  (e.g. a "12-billion-parameter draft transformer" claim on a rotary-embeddings page).
49
49
  When the fabrication stays on-topic (the KV-cache case), the judge waves it through.
50
50
 
51
51
  So on THIS band the verifier-per-dollar comparison inverts the cost/quality result:
52
- there, the LLM verifier bought a dedup-shaped gain a free hash already captures
52
+ there, the LLM verifier bought a dedup-shaped gain a free hash already captures;
53
53
  expensive for what a cheap rule does. Here the cheap, deterministic check
54
54
  **dominates** the expensive judge: it catches strictly more (5/5 vs 4/5) at strictly
55
55
  less ($0 vs $0.0157). The verifier earns its dollar on misattribution; it does not on
@@ -63,7 +63,7 @@ de-duplication.
63
63
  (and the offline relevance stand-in) accept them. The error is in the *claim*, not
64
64
  the *topic*.
65
65
  - **Executable ground truth.** The check is presence/close-paraphrase of the claim
66
- in the fetched text deployable in production with no oracle and no model call.
66
+ in the fetched text, deployable in production with no oracle and no model call.
67
67
 
68
68
  The offline arm proves the floor with a controlled 4-source pool (2 grounded, 2
69
69
  misattributed): claim-grounding admits **0/2** misattributions and keeps **2/2**
@@ -78,9 +78,9 @@ grounded sources, while relevance and no-verifier both admit **2/2**.
78
78
  - **Planted misattributions, not naturally-occurring ones.** Like the cost/quality
79
79
  offline floor, the misattribution is injected so the band is measurable. It models
80
80
  the real LLM citation-fabrication failure but does not measure its base rate in the
81
- wild that needs a corpus of model-written citations checked by hand.
81
+ wild, that needs a corpus of model-written citations checked by hand.
82
82
  - **The grounding oracle is conservative.** A real paraphrase whose inflected words
83
- differ from the page ("drafts" vs "draft") can score below 0.7 and be rejected
83
+ differ from the page ("drafts" vs "draft") can score below 0.7 and be rejected,
84
84
  a false-positive misattribution flag. `minOverlap` tunes this; the worker should
85
85
  cite the page's own key terms (the `createClaimDecorator` extractor is told to).
86
86
 
@@ -2,7 +2,7 @@
2
2
 
3
3
  Live 9-topic A/B (glm-5.2, budget B ≤ 4 passes/arm), measured per arm with the
4
4
  router-client instrumentation. The original A/B reported only admitted-sources at
5
- "equal passes" which charged the two-agent verify step as one pass while it is
5
+ "equal passes", which charged the two-agent verify step as one pass while it is
6
6
  actually N `verifySource` LLM calls. Pricing the calls shows what that hid.
7
7
 
8
8
  | per topic (mean) | two-agent | single-agent | ratio |
@@ -11,7 +11,7 @@ actually N `verifySource` LLM calls. Pricing the calls shows what that hid.
11
11
  | tokens (in+out) | ~4,900 | ~530 | ~9× |
12
12
  | cost (USD) | ~$0.0072 | ~$0.0013 | ~5.5× |
13
13
  | latency (wall) | ~37 s | ~11 s | ~3.4× |
14
- | cleanliness Δ (single − two admitted) | | | +1.56, 95% CI [0.33, 2.67] |
14
+ | cleanliness Δ (single − two admitted) | n/a | n/a | +1.56, 95% CI [0.33, 2.67] |
15
15
 
16
16
  Per-topic Δ (single − two admitted) this run: self-speculative decoding +4,
17
17
  grouped-query attention 0, rotary position embeddings +1, KV-cache quantization
@@ -19,9 +19,9 @@ grouped-query attention 0, rotary position embeddings +1, KV-cache quantization
19
19
  descent **−1**. Coverage 1.00 every topic, both arms.
20
20
 
21
21
  **Reading.** The verifier buys ~1.5–2.7 fewer junk sources for roughly **5× the
22
- dollars, 9× the tokens, and 3× the latency** and on two topics it admitted *more*
22
+ dollars, 9× the tokens, and 3× the latency**, and on two topics it admitted *more*
23
23
  than the single agent (the cleanliness signal is real but noisier than the +2.3 /
24
24
  +2.7 of earlier runs). The cleanliness gain is dominated by de-duplication, so the
25
- honest production move is a deterministic content-hash / canonical-URL dedup, which
25
+ production move is a deterministic content-hash / canonical-URL dedup, which
26
26
  captures most of the cleanliness at ~none of this premium; reserve an LLM check for
27
27
  the off-scope tail. This is the cost half the "equal passes" framing left out.