@tangle-network/agent-knowledge 4.0.1 → 5.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (37) hide show
  1. package/AGENTS.md +35 -19
  2. package/CHANGELOG.md +48 -0
  3. package/README.md +220 -943
  4. package/dist/benchmarks/index.d.ts +3 -2
  5. package/dist/benchmarks/index.js +1 -1
  6. package/dist/{chunk-EUAXSBMH.js → chunk-AKYJG2MR.js} +224 -224
  7. package/dist/chunk-AKYJG2MR.js.map +1 -0
  8. package/dist/{chunk-BBCIB4UL.js → chunk-EYIA5PLQ.js} +27 -195
  9. package/dist/chunk-EYIA5PLQ.js.map +1 -0
  10. package/dist/{chunk-3CAAGJCE.js → chunk-LMR53POQ.js} +1227 -574
  11. package/dist/chunk-LMR53POQ.js.map +1 -0
  12. package/dist/{chunk-WCYW2GDA.js → chunk-MYFM6LKH.js} +2 -2
  13. package/dist/chunk-MYFM6LKH.js.map +1 -0
  14. package/dist/cli.js +1 -1
  15. package/dist/index-Cf7txrYP.d.ts +790 -0
  16. package/dist/index.d.ts +252 -73
  17. package/dist/index.js +2813 -2229
  18. package/dist/index.js.map +1 -1
  19. package/dist/memory/index.d.ts +8 -709
  20. package/dist/memory/index.js +2 -2
  21. package/dist/sources/index.d.ts +17 -33
  22. package/dist/sources/index.js +1 -1
  23. package/dist/{types-ZzY_x0r7.d.ts → types-BY-xLVw-.d.ts} +28 -67
  24. package/docs/eval/investment-material-facts.md +28 -29
  25. package/docs/eval/rag-eval-roadmap.md +21 -17
  26. package/docs/results/adaptive.md +13 -13
  27. package/docs/results/claim-grounding.md +11 -11
  28. package/docs/results/cost-quality.md +4 -4
  29. package/docs/results/investment-thesis.md +56 -56
  30. package/docs/results/research-driving.md +54 -54
  31. package/docs/two-agent-research-ab.md +94 -94
  32. package/package.json +11 -11
  33. package/skills/build-with-agent-knowledge/SKILL.md +97 -0
  34. package/dist/chunk-3CAAGJCE.js.map +0 -1
  35. package/dist/chunk-BBCIB4UL.js.map +0 -1
  36. package/dist/chunk-EUAXSBMH.js.map +0 -1
  37. package/dist/chunk-WCYW2GDA.js.map +0 -1
@@ -1,9 +1,9 @@
1
1
  # Adaptive topology: spend the LLM verifier only when it pays
2
2
 
3
3
  The cost/quality A/B (`docs/results/cost-quality.md`) found the LLM relevance
4
- verifier's cleanliness win is dominated by **de-duplication** which a
4
+ verifier's cleanliness win is dominated by **de-duplication**, which a
5
5
  deterministic content-hash / canonical-URL check captures at ~none of the LLM
6
- premium and that an LLM check only earns its dollar on the off-scope tail. The
6
+ premium, and that an LLM check only earns its dollar on the off-scope tail. The
7
7
  production move it named was: do the cheap deterministic work first, reserve the
8
8
  LLM for the ambiguous tail. `createAdaptiveResearchDriver`
9
9
  (`src/adaptive-driver.ts`) is that driver, and this is its measurement.
@@ -19,7 +19,7 @@ stopping at the first that decides:
19
19
  github, …) with a substantial body is **kept**; an obvious spam/listicle
20
20
  title or a too-thin body is **dropped**; everything else is **ambiguous**.
21
21
  3. **LLM escalation ($).** Only ambiguous survivors reach the shipped LLM
22
- relevance verifier (`createVerifyingResearchDriver`) one call each.
22
+ relevance verifier (`createVerifyingResearchDriver`), one call each.
23
23
 
24
24
  ## Live frontier, n=5 topics (glm-5.2)
25
25
 
@@ -39,24 +39,24 @@ is the per-arm `RouterClient.usage()` diff (#36). Total spend for the run:
39
39
  | **total** | **25** | **25** | **12 / 25 / $0.0261** | **20 / 6 / $0.0068** |
40
40
 
41
41
  **Cost.** Adaptive cuts LLM verifier calls **76%** (25 → 6) and dollars **74%**
42
- ($0.0261 → $0.0068). On 3 of the 5 topics it spent **zero** LLM calls every
42
+ ($0.0261 → $0.0068). On 3 of the 5 topics it spent **zero** LLM calls, every
43
43
  unique survivor was on an authoritative host, so the $0 stages decided
44
44
  everything.
45
45
 
46
- ## The honest reading: adaptive is a frontier POINT, not a free lunch
46
+ ## Interpretation: adaptive occupies one cost-quality point
47
47
 
48
48
  Admitted-source counts (lower = cleaner KB): **single 25, adaptive 20,
49
49
  full-LLM 12**. Adaptive sits **between** the two:
50
50
 
51
51
  - It removes **5 of the 13 sources** the full-LLM judge rejects that the
52
52
  single-agent loop keeps (every one of them a real duplicate caught by the $0
53
- dedup stage exactly the de-dup-dominated win the cost/quality result
53
+ dedup stage, exactly the de-dup-dominated win the cost/quality result
54
54
  predicted).
55
55
  - It does **NOT** match full-LLM cleanliness. The remaining 8 sources full-LLM
56
56
  rejects, adaptive keeps. The cause is structural and visible in the trace: on
57
57
  this topic set every non-duplicate survivor landed on an authoritative host
58
58
  (arxiv / github / official docs), so the heuristic **kept** it without ever
59
- asking the LLM and the LLM, when full-LLM did ask it, judged several of
59
+ asking the LLM, and the LLM, when full-LLM did ask it, judged several of
60
60
  those same authoritative pages not-quite-on-topic and dropped them. The host
61
61
  prior is coarser than the relevance judge.
62
62
 
@@ -65,19 +65,19 @@ de-dup half of full-LLM's cleanliness for ~26% of full-LLM's dollars, and gives
65
65
  up the relevance-judgment half.** Whether that is the right point depends on the
66
66
  cost of a kept-but-marginal source. If a slightly-off-topic authoritative page
67
67
  is cheap to carry, adaptive dominates. If every admitted source must clear a
68
- relevance bar, the host heuristic is too permissive and you want the full LLM
68
+ relevance bar, the host heuristic is too permissive and you want the full LLM,
69
69
  or a tightened heuristic.
70
70
 
71
- ## Where the heuristic is weak stated plainly
71
+ ## Where the heuristic is weak: stated plainly
72
72
 
73
73
  The escalation count is the diagnostic. On 3 of 5 topics it was **zero**: the
74
74
  heuristic never deferred to the LLM, so on those topics adaptive is a
75
75
  **pure host/title/length rule**, and its cleanliness is exactly that rule's
76
- cleanliness no smarter than "trust arxiv/github, drop spam." That is fine when
76
+ cleanliness, no smarter than "trust arxiv/github, drop spam." That is fine when
77
77
  the worker's sources are dominated by authoritative hosts (as here), but it
78
78
  means the LLM's relevance judgment is contributing nothing on those topics, by
79
79
  construction. The two topics where adaptive *did* escalate (grouped-query
80
- attention, LoRA) are where some survivors were on unknown hosts and there the
80
+ attention, LoRA) are where some survivors were on unknown hosts, and there the
81
81
  3 LLM calls per topic are the off-scope tail the verifier is actually for.
82
82
 
83
83
  The heuristic would mis-route in two directions a richer worker would expose,
@@ -95,12 +95,12 @@ neither seen on this authoritative-host-heavy set:
95
95
  The deployable recommendation from the cost/quality result was "deterministic
96
96
  dedup first, reserve the LLM for the tail." This driver ships that and the
97
97
  measurement confirms the **cost** half cleanly (76% fewer calls, 74% cheaper)
98
- and qualifies the **quality** half honestly: adaptive captures the de-dup
98
+ and qualifies the **quality** half accurately: adaptive captures the de-dup
99
99
  cleanliness (the dominant, free win) but not the LLM's relevance cleanliness,
100
100
  because the host heuristic resolves authoritative survivors without asking. For
101
101
  a worker whose sources are mostly authoritative, adaptive is the right frontier
102
102
  point. For one whose junk is on-topic-looking pages on unknown hosts, the
103
- ambiguous tail grows and adaptive converges toward full-LLM cost which is the
103
+ ambiguous tail grows and adaptive converges toward full-LLM cost, which is the
104
104
  correct behavior: it pays for the LLM exactly when the cheap signals can't
105
105
  decide.
106
106
 
@@ -1,22 +1,22 @@
1
1
  # Claim-grounding: the band where the verifier earns its dollar
2
2
 
3
3
  `docs/results/cost-quality.md` found the relevance verifier's cleanliness win is
4
- **dominated by de-duplication** a deterministic content-hash captures most of it
4
+ **dominated by de-duplication**, a deterministic content-hash captures most of it
5
5
  at ~none of the LLM premium. So the open question was: is there an error band where
6
- a verifier earns its cost something a hash AND a relevance judge both miss?
6
+ a verifier earns its cost, something a hash AND a relevance judge both miss?
7
7
 
8
8
  **Yes: misattributed citations.** A source that is on-topic, unique, and real, but
9
9
  whose cited CLAIM does not appear in the page (the LLM wrote a plausible sentence
10
10
  and hung a real URL off it). De-dup passes it (it's unique). A relevance judge
11
11
  passes it (the page is on-topic). Only checking the claim against the fetched text
12
- catches it and that check is **deterministic text presence, $0 inference**.
12
+ catches it, and that check is **deterministic text presence, $0 inference**.
13
13
 
14
14
  ## The mode
15
15
 
16
16
  Each proposed source now carries the specific claim it is cited for
17
17
  (`withCitedClaim` → `metadata.citedClaim`). The verifier
18
18
  (`createClaimGroundingVerifier`) runs `groundClaimInText(claim, pageText)` over the
19
- `htmlToText` output of the page the worker actually fetched verbatim, normalized
19
+ `htmlToText` output of the page the worker actually fetched, verbatim, normalized
20
20
  (punctuation/whitespace-insensitive), or a ≥70% content-word overlap close
21
21
  paraphrase. A claim that isn't present is rejected as **misattributed**. The oracle
22
22
  is text presence, not a model call, so it composes with the LLM relevance verifier
@@ -31,25 +31,25 @@ proposals. Cost diffed per arm with the #36 `RouterClient.usage()` instrumentati
31
31
 
32
32
  | n=5 topics | misattributions caught | marginal $ | $/topic | per-$ caught |
33
33
  |---|---|---|---|---|
34
- | no-verifier | 0 / 5 | $0.0000 | | |
34
+ | no-verifier | 0 / 5 | $0.0000 | n/a | n/a |
35
35
  | relevance (LLM judge) | **4 / 5** | $0.0157 | ~$0.0031 | 254 |
36
36
  | claim-grounding (text) | **5 / 5** | **$0.0000** | $0 | ∞ |
37
37
 
38
38
  Per-topic (caught relevance / grounding): self-speculative decoding 1/1, rotary
39
39
  position embeddings 1/1, grouped-query attention 1/1, **KV-cache quantization 0/1**,
40
- LoRA 1/1. (An earlier 3-topic run missed self-speculative decoding instead the
40
+ LoRA 1/1. (An earlier 3-topic run missed self-speculative decoding instead, the
41
41
  miss moves around; it is not a fixed topic.)
42
42
 
43
43
  **Reading.** Claim-grounding catches every misattribution at **$0**; the relevance
44
44
  judge catches most but **misses one in five at ~$0.003/topic**. The miss is the
45
45
  point: the relevance verifier only ever sees the page text, never the cited claim,
46
- so it is *structurally blind* to misattribution. It catches one only by accident
46
+ so it is *structurally blind* to misattribution. It catches one only by accident:
47
47
  when the fabricated claim happens to also make the page read off-topic
48
48
  (e.g. a "12-billion-parameter draft transformer" claim on a rotary-embeddings page).
49
49
  When the fabrication stays on-topic (the KV-cache case), the judge waves it through.
50
50
 
51
51
  So on THIS band the verifier-per-dollar comparison inverts the cost/quality result:
52
- there, the LLM verifier bought a dedup-shaped gain a free hash already captures
52
+ there, the LLM verifier bought a dedup-shaped gain a free hash already captures;
53
53
  expensive for what a cheap rule does. Here the cheap, deterministic check
54
54
  **dominates** the expensive judge: it catches strictly more (5/5 vs 4/5) at strictly
55
55
  less ($0 vs $0.0157). The verifier earns its dollar on misattribution; it does not on
@@ -63,7 +63,7 @@ de-duplication.
63
63
  (and the offline relevance stand-in) accept them. The error is in the *claim*, not
64
64
  the *topic*.
65
65
  - **Executable ground truth.** The check is presence/close-paraphrase of the claim
66
- in the fetched text deployable in production with no oracle and no model call.
66
+ in the fetched text, deployable in production with no oracle and no model call.
67
67
 
68
68
  The offline arm proves the floor with a controlled 4-source pool (2 grounded, 2
69
69
  misattributed): claim-grounding admits **0/2** misattributions and keeps **2/2**
@@ -78,9 +78,9 @@ grounded sources, while relevance and no-verifier both admit **2/2**.
78
78
  - **Planted misattributions, not naturally-occurring ones.** Like the cost/quality
79
79
  offline floor, the misattribution is injected so the band is measurable. It models
80
80
  the real LLM citation-fabrication failure but does not measure its base rate in the
81
- wild that needs a corpus of model-written citations checked by hand.
81
+ wild, that needs a corpus of model-written citations checked by hand.
82
82
  - **The grounding oracle is conservative.** A real paraphrase whose inflected words
83
- differ from the page ("drafts" vs "draft") can score below 0.7 and be rejected
83
+ differ from the page ("drafts" vs "draft") can score below 0.7 and be rejected,
84
84
  a false-positive misattribution flag. `minOverlap` tunes this; the worker should
85
85
  cite the page's own key terms (the `createClaimDecorator` extractor is told to).
86
86
 
@@ -2,7 +2,7 @@
2
2
 
3
3
  Live 9-topic A/B (glm-5.2, budget B ≤ 4 passes/arm), measured per arm with the
4
4
  router-client instrumentation. The original A/B reported only admitted-sources at
5
- "equal passes" which charged the two-agent verify step as one pass while it is
5
+ "equal passes", which charged the two-agent verify step as one pass while it is
6
6
  actually N `verifySource` LLM calls. Pricing the calls shows what that hid.
7
7
 
8
8
  | per topic (mean) | two-agent | single-agent | ratio |
@@ -11,7 +11,7 @@ actually N `verifySource` LLM calls. Pricing the calls shows what that hid.
11
11
  | tokens (in+out) | ~4,900 | ~530 | ~9× |
12
12
  | cost (USD) | ~$0.0072 | ~$0.0013 | ~5.5× |
13
13
  | latency (wall) | ~37 s | ~11 s | ~3.4× |
14
- | cleanliness Δ (single − two admitted) | | | +1.56, 95% CI [0.33, 2.67] |
14
+ | cleanliness Δ (single − two admitted) | n/a | n/a | +1.56, 95% CI [0.33, 2.67] |
15
15
 
16
16
  Per-topic Δ (single − two admitted) this run: self-speculative decoding +4,
17
17
  grouped-query attention 0, rotary position embeddings +1, KV-cache quantization
@@ -19,9 +19,9 @@ grouped-query attention 0, rotary position embeddings +1, KV-cache quantization
19
19
  descent **−1**. Coverage 1.00 every topic, both arms.
20
20
 
21
21
  **Reading.** The verifier buys ~1.5–2.7 fewer junk sources for roughly **5× the
22
- dollars, 9× the tokens, and 3× the latency** and on two topics it admitted *more*
22
+ dollars, 9× the tokens, and 3× the latency**, and on two topics it admitted *more*
23
23
  than the single agent (the cleanliness signal is real but noisier than the +2.3 /
24
24
  +2.7 of earlier runs). The cleanliness gain is dominated by de-duplication, so the
25
- honest production move is a deterministic content-hash / canonical-URL dedup, which
25
+ production move is a deterministic content-hash / canonical-URL dedup, which
26
26
  captures most of the cleanliness at ~none of this premium; reserve an LLM check for
27
27
  the off-scope tail. This is the cost half the "equal passes" framing left out.
@@ -1,25 +1,25 @@
1
- # From paper-retrieval to investigation — a research eval where one search is not enough, a metric that discriminates depth, and a 3-arm topology A/B
1
+ # Investment-research depth eval and three-arm comparison
2
2
 
3
3
  *Tangle Network · `agent-knowledge`*
4
4
 
5
- ## Verdict (BLUF)
5
+ ## Verdict
6
6
 
7
- We moved the research eval off ML-paper retrieval where a **single** web search
7
+ We moved the research eval off ML-paper retrieval, where a **single** web search
8
8
  already returns the answer, so the only thing the metric could measure was
9
- *collection* onto **investment research**, where the material facts are buried in
9
+ *collection*, onto **investment research**, where the material facts are buried in
10
10
  the footnotes of an SEC filing and genuinely require *investigation* to surface. On
11
11
  this harder domain we (1) **calibrated** a `$0`, model-free metric and proved it
12
- discriminates research depth a shallow ticker summary scores **1/27 (4%)**, a
13
- deep filings-grounded thesis scores **27/27 (100%)**, a **+96-point** gap and then
12
+ discriminates research depth, a shallow ticker summary scores **1/27 (4%)**, a
13
+ deep filings-grounded thesis scores **27/27 (100%)**, a **+96-point** gap, and then
14
14
  (2) ran **three research topologies head-to-head** on the same five held-out
15
15
  companies at matched compute, grading each one's knowledge base against a firewalled
16
16
  checklist of **27 buried material facts** a ticker search misses.
17
17
 
18
- **The honest A/B verdict: the research-DRIVING loop surfaced the most buried facts
19
- (16/27, 59%), +5 over blind collection (11/27, 41%) but at n=5 companies that lift
18
+ **A/B result: the research-driving loop surfaced the most buried facts
19
+ (16/27, 59%), +5 over blind collection (11/27, 41%), but at n=5 companies that lift
20
20
  is NOT statistically clean: the 95% confidence interval crosses zero (P(Δ≤0)=0.08).**
21
- The verify/dedup loop did **not** beat collection at all (10/27, 37% a wash,
22
- slightly worse). So no topology *significantly* beats collection here either —
21
+ The verify/dedup loop did **not** beat collection at all (10/27, 37%, a wash,
22
+ slightly worse). No approach *significantly* beats collection here.
23
23
  **driving is the only arm that even points the right way, and it points there
24
24
  suggestively, not significantly.** What the reframe *did* buy is a domain and a meter
25
25
  where the question is finally well-posed: the metric is no longer measuring whether a
@@ -27,7 +27,7 @@ single search ran, but whether investigation reached the buried fact.
27
27
 
28
28
  | arm | what the coordinator does | material facts | cost (5 cos.) | chats | searches | tokens |
29
29
  |---|---|---|---|---|---|---|
30
- | **A · collection** | nothing accepts every source (1 agent collects) | 11/27 (41%) | $0.082 | 10 | 20 | 64,248 |
30
+ | **A · collection** | nothing; accepts every source (1 agent collects) | 11/27 (41%) | $0.082 | 10 | 20 | 64,248 |
31
31
  | **B · verify/dedup** | LLM gates each source for relevance, rejects off-topic | 10/27 (37%) | $0.125 | 56 | 52 | 84,678 |
32
32
  | **C · driving (deepen)** | extracts claims, demands corroboration, asks deep follow-ups | **16/27 (59%)** | $0.157 | 33 | 28 | 124,387 |
33
33
 
@@ -35,17 +35,17 @@ Cost is real provenance, not an estimate: every `$`/call/token is diffed from
35
35
  `RouterClient.usage()` per company (the router's own `usage` field, priced at
36
36
  glm-5.2 rates), so "driving cost 1.9× collection" is measured, not modelled.
37
37
 
38
- ## 1. The domain reframe why we left ML-paper retrieval
38
+ ## 1. The domain reframe: why we left ML-paper retrieval
39
39
 
40
40
  The companion paper's depth eval (`docs/two-agent-research-ab.md` §9) measures a
41
- research loop on **20 deep questions across 5 ML topics**. That apparatus is sound
41
+ research loop on **20 deep questions across 5 ML topics**. That method is sound:
42
42
  its grader scores 0/20 on a one-line topic definition and 20/20 on a mechanism-rich
43
43
  paragraph, so it *can* tell depth from surface. But the topology A/B on top of it
44
- came back a clean null (driving 16/20 @ budget 4, 13/20 @ budget 6 the winner
44
+ came back a clean null (driving 16/20 @ budget 4, 13/20 @ budget 6, the winner
45
45
  *flips with the compute budget*, and the within-arm swing is as large as the
46
46
  between-arm gap). The autopsy named the cause: **on an ML topic a single good search
47
47
  already collects the answer.** Every arm finished in one effective round because the
48
- generic "one source closes the gap" readiness was met by the *first* fetch so the
48
+ generic "one source closes the gap" readiness was met by the *first* fetch, so the
49
49
  driving driver, whose entire mechanism is steering a *second* round, never got to
50
50
  act. When one search suffices, there is no investigation for a smarter coordinator to
51
51
  do, and the metric can only reward collection. That is the failure-mode-in-spirit the
@@ -55,12 +55,12 @@ easy for topology to matter.
55
55
  So we changed the domain to one where a single search **provably cannot** suffice.
56
56
  **Investment research**: give a loop a company + ticker + an as-of cutoff date and ask
57
57
  for a thesis; grade it on the buried, material, non-obvious drivers a thorough analyst
58
- flags and a ticker search misses. The decisive facts here live in 10-K footnotes an
58
+ flags and a ticker search misses. The decisive facts here live in 10-K footnotes, an
59
59
  HTM securities mark roughly equal to a bank's entire equity (SIVB), a deposit base
60
60
  that is 97% uninsured, a negative per-unit gross margin, a related-party lease. A web
61
61
  search for the company name returns "profitable regional bank" or "high-growth auto
62
62
  e-commerce"; the filing shows the mark-to-market hole the size of capital. **The
63
- answer is not collectable in one fetch it has to be investigated for.** That is the
63
+ answer is not collectable in one fetch, it has to be investigated for.** That is the
64
64
  property the ML domain lacked, and it is what makes a topology A/B finally meaningful:
65
65
  if a smarter coordinator can ever beat blind collection, a domain where the answer is
66
66
  buried is where it has the room to.
@@ -69,7 +69,7 @@ The held-out set is **5 companies, 27 material facts**, every fact read live out
69
69
  the primary SEC EDGAR 10-K during curation, every dollar figure quoted from the
70
70
  de-tagged filing text, every value knowable *at the cutoff* (the eventual collapse is
71
71
  recorded for the reader only and is **never graded**). The companies skew distressed /
72
- downside-risk a documented curation bias, not a hidden one (full provenance, the
72
+ downside-risk, a documented curation bias, not a hidden one (full provenance, the
73
73
  drop log, and the per-fact keyword groups: `docs/eval/investment-material-facts.md`):
74
74
 
75
75
  | ticker | company | as-of cutoff | CIK | facts |
@@ -80,25 +80,25 @@ drop log, and the per-fact keyword groups: `docs/eval/investment-material-facts.
80
80
  | PTON | Peloton | 2022-09-07 | 1639825 | 6 |
81
81
  | SI | Silvergate | 2022-02-28 | 1312109 | 5 |
82
82
 
83
- ## 2. Calibration does the metric discriminate depth? (the gate the ML exam passed only weakly)
83
+ ## 2. Calibration: does the metric discriminate depth? (the gate the ML exam passed only weakly)
84
84
 
85
85
  A topology A/B is meaningless unless the metric grading it can tell a *deep* thesis
86
- from a *shallow* one. If it can't, it is measuring word-collection the exact failure
87
- the reframe set out to escape and any A/B on top of it is noise. So **before** the
86
+ from a *shallow* one. If it can't, it is measuring word-collection, the exact failure
87
+ the reframe set out to escape, and any A/B on top of it is noise. So **before** the
88
88
  A/B, we ran a calibration gate (`$0`, offline, the binding result):
89
89
 
90
90
  For each of the 5 companies we hand-wrote two theses and scored each with the metric's
91
91
  pure core (`materialFactsSurfacedInText`):
92
92
 
93
- - **shallow** a one-paragraph ticker summary: what the company does, a vibe on the
93
+ - **shallow**: a one-paragraph ticker summary: what the company does, a vibe on the
94
94
  stock, generic macro/competition risk. The kind a single name-search returns. Names
95
95
  none of the buried, filing-level facts.
96
- - **deep** a filings-grounded analyst memo naming the buried drivers (the duration
96
+ - **deep**: a filings-grounded analyst memo naming the buried drivers (the duration
97
97
  loss, the buyback drain, the negative unit margin, the deposit concentration, the
98
98
  related party) in independent prose, with the real numbers.
99
99
 
100
100
  The grader is the same `$0`, model-free, case-insensitive substring check the held-out
101
- checklist ships (`gradeFactAgainstText`); the checklist is firewalled read only by
101
+ checklist ships (`gradeFactAgainstText`); the checklist is firewalled, read only by
102
102
  the metric, never shown to a loop.
103
103
 
104
104
  | ticker | shallow | deep | gap |
@@ -110,20 +110,20 @@ the metric, never shown to a loop.
110
110
  | SI | 0/5 (0%) | 5/5 (100%) | +100pp |
111
111
  | **total** | **1/27 (4%)** | **27/27 (100%)** | **+96pp** |
112
112
 
113
- **The metric is VALID:** it cleanly separates a shallow ticker-summary from a deep,
114
- filings-grounded thesis a +96-point aggregate gap, every company clearing the bars
113
+ **Calibration result:** the metric separates a shallow ticker-summary from a deep,
114
+ filings-grounded thesis, a +96-point aggregate gap, every company clearing the bars
115
115
  (shallow `< 30%`, deep `> 70%`) with wide margin. Two guards make the gap real and not
116
116
  a teaching-to-the-test artifact:
117
117
 
118
118
  - **Anti-circularity.** The gate asserts **no deep thesis verbatim-embeds any
119
- checklist `evidence` string** the deep theses state the same publicly-documented
119
+ checklist `evidence` string**, the deep theses state the same publicly-documented
120
120
  facts in independent analyst prose. A high deep score is the meter catching real,
121
121
  independently-phrased depth, not an answer-key echo.
122
- - **The one honest shallow hit.** The single shallow surface (SIVB 1/6) is SIVB/f5
123
- the innovation-economy / venture-client concentration which fires on "technology"
122
+ - **The one shallow hit.** The single shallow surface (SIVB 1/6) is SIVB/f5:
123
+ the innovation-economy / venture-client concentration, which fires on "technology"
124
124
  / "venture" in the SVB summary. That genuinely *is* the least-buried of SVB's six
125
125
  facts (a ticker search does return "SVB banks tech and venture startups"). We kept it
126
- as a 4% leak rather than tighten the group, because it honestly reflects one
126
+ as a 4% leak rather than tighten the group, because it reflects one
127
127
  near-surface fact while the five truly buried ones (the ~$15B HTM loss, the
128
128
  equity-sized mark, the $151.5B uninsured base, the 20-point deposit-mix shift, the
129
129
  AFS/AOCI loss) stay unsurfaced. The firewall holds (17% << 30%).
@@ -132,38 +132,38 @@ Calibration also did its job as more than a rubber stamp: the first pass had the
132
132
  **Silvergate shallow thesis scoring 3/5 (60%)**, blowing the bar, because three fact
133
133
  groups accepted bare crypto-bank vocabulary (`crypto`, `grew rapidly`, `proprietary` /
134
134
  `payment network`) that any one-line summary trips. We tightened those three to require
135
- the *buried* signal the deposit-**concentration** framing, the specific **$14.3B**
136
- figure, the **SEN** (Silvergate Exchange Network) name and dropped the generic vocab.
135
+ the *buried* signal, the deposit-**concentration** framing, the specific **$14.3B**
136
+ figure, the **SEN** (Silvergate Exchange Network) name, and dropped the generic vocab.
137
137
  The deep thesis still hits all three; the shallow one no longer does. The metric found
138
138
  a way it could be fooled and we closed it before any A/B depended on it.
139
139
 
140
140
  **This is the gate the ML exam passed only in spirit.** The ML grader discriminated
141
- 0/20 vs 20/20 too but on a domain where the deep answer was *collectable* in one
141
+ 0/20 vs 20/20 too, but on a domain where the deep answer was *collectable* in one
142
142
  search, so the discrimination measured grammar, not investigation. Here the +96-point
143
143
  gap is over facts that are *buried by construction*, so a high score is reachable only
144
144
  by reaching into the filing. Same shape of result, materially harder domain.
145
145
 
146
- ## 3. The 3-arm A/B what each topology surfaced, head-to-head
146
+ ## 3. The 3-arm A/B: what each topology surfaced, head-to-head
147
147
 
148
148
  For each company the loop is told **only** `{company, ticker, cik, cutoff}` plus a
149
149
  generic set of analyst-lens readiness specs (balance-sheet risk, concentration,
150
- leverage, margins, liquidity, governance, regulatory the *lenses* and where they
151
- live, the latest SEC 10-K **not the answers**). It researches the company *as of* the
150
+ leverage, margins, liquidity, governance, regulatory, the *lenses* and where they
151
+ live, the latest SEC 10-K, **not the answers**). It researches the company *as of* the
152
152
  cutoff over web + SEC EDGAR (both public), writes a thesis, and we grade the resulting
153
- KB with `materialFactsSurfaced` the firewalled checklist the loop never sees.
153
+ KB with `materialFactsSurfaced`, the firewalled checklist the loop never sees.
154
154
 
155
155
  **Compute is matched by construction.** All three arms run the *same* web worker, the
156
156
  *same* 3-round budget, the *same* worker config (`resultsPerQuery: 3, queriesPerGap: 1,
157
- maxSourcesPerRound: 6`). The **only** thing that varies is the driver the coordinator
157
+ maxSourcesPerRound: 6`). The **only** thing that varies is the driver, the coordinator
158
158
  between the worker and the knowledge base:
159
159
 
160
- - **A · collection** (`createCollectionResearchDriver`) an inert rubber stamp:
160
+ - **A · collection** (`createCollectionResearchDriver`), an inert rubber stamp:
161
161
  accepts every source, gates nothing, steers only with the loop's built-in open-gap
162
162
  list. The driver adds **zero** router calls. This is the blind-collection floor.
163
- - **B · verify/dedup** (`createVerifyingResearchDriver`) an LLM relevance gate: one
163
+ - **B · verify/dedup** (`createVerifyingResearchDriver`), an LLM relevance gate: one
164
164
  chat call per candidate source to accept-or-reject for on-topic relevance and
165
165
  near-duplication. The worker ADDS; the driver GATES.
166
- - **C · driving** (`createResearchDrivingDriver`) extracts each source's claims,
166
+ - **C · driving** (`createResearchDrivingDriver`), extracts each source's claims,
167
167
  tracks how many *independent* sources corroborate each, and synthesizes deep
168
168
  follow-up sub-questions (comparative / mechanism / gap / contradiction) it folds into
169
169
  the worker's next prompt to push depth and demand corroboration.
@@ -200,13 +200,13 @@ Every interval crosses zero. **Driving vs collection is the closest to clean
200
200
  coin flip. This is the project's well-documented small-n mirage: exciting deltas born
201
201
  at n=5 do not survive a paired bootstrap.
202
202
 
203
- ## 4. Autopsy the two things worth understanding
203
+ ## 4. Autopsy: the two things worth understanding
204
204
 
205
205
  ### 4.1 Why driving wins where it wins
206
206
 
207
207
  Driving's mechanism is multi-round: extract claims from round 1, then steer the worker
208
208
  in rounds 2–3 to corroborate the weak ones and chase the deep questions. It helps most
209
- where the **first** fetch lands real filing data the driver can build on the two
209
+ where the **first** fetch lands real filing data the driver can build on, the two
210
210
  banks, where SEC bank-call-report / 10-K data is dense and reachable. SIVB jumps from 2
211
211
  buried facts (collection) to 5 (driving): the duration loss, the deposit concentration,
212
212
  the AFS/AOCI mark all surface once the driver demands the balance-sheet detail a second
@@ -230,21 +230,21 @@ ROUND 1: accepted=0 rejected=3 writtenPages=0
230
230
  ROUND 2: accepted=0 rejected=2 writtenPages=0
231
231
  ```
232
232
 
233
- The verifier was **correct on the merits** those are aggregators, not the primary
233
+ The verifier was **correct on the merits**, those are aggregators, not the primary
234
234
  filing, and last10k showed a post-cutoff filing. But the worker never surfaced the
235
235
  EDGAR primary for BBBY, so a strict primary-only gate left the KB **empty**. Collection
236
236
  accepts the same aggregator pages and scores 4/5 on BBBY; driving accepts them and
237
237
  scores 5/5. **The gate's strictness is a liability when the worker's sourcing is
238
- imperfect:** it throws away the only evidence the loop had. This is a genuine topology
239
- trade-off worth naming, not a harness break the verify arm surfaced facts fine on
240
- CVNA (2), PTON (4), SI (4), so the harness works.
238
+ imperfect:** it throws away the only evidence the loop had. This is a real coordination
239
+ trade-off. The verify arm still surfaced facts on CVNA (2), PTON (4), and SI (4),
240
+ which rules out a general execution failure.
241
241
 
242
- ### 4.3 The empty-thesis caveat (honest)
242
+ ### 4.3 The empty-thesis caveat
243
243
 
244
244
  Some runs show `thesis=0ch` (empty synthesis) yet still score facts. That is expected:
245
- `materialFactsSurfaced` grades the **whole KB** (the worker's fetched `knowledge/*.md`
246
- pages), not only the final thesis page. glm-5.2 occasionally spends its entire output
247
- budget on hidden reasoning and returns empty visible content on the synthesis call a
245
+ `materialFactsSurfaced` grades the **whole KB**, including the worker's fetched
246
+ `knowledge/*.md` pages and the final thesis page. glm-5.2 occasionally spends its entire output
247
+ budget on hidden reasoning and returns empty visible content on the synthesis call, a
248
248
  known reasoning-model behavior we floor at 1200 tokens but can't fully eliminate. The
249
249
  score still reflects what the loop fetched, so an empty thesis does not invalidate a
250
250
  run; it just means the synthesis prose was lost while the curated evidence was not.
@@ -252,7 +252,7 @@ run; it just means the synthesis prose was lost while the curated evidence was n
252
252
  ## 5. What this does and does not establish
253
253
 
254
254
  - **Does**: the metric *discriminates depth on a domain where one search is not
255
- enough* (calibration: 1/27 shallow vs 27/27 deep, +96pp) so the topology A/B on top
255
+ enough* (calibration: 1/27 shallow vs 27/27 deep, +96pp), so the topology A/B on top
256
256
  of it is finally well-posed, unlike the ML retrieval exam where one search already
257
257
  collected the answer. At matched compute on the 5-company held-out set, the
258
258
  research-driving topology surfaced the most buried material facts (16/27 vs 11/27 for
@@ -272,10 +272,10 @@ run; it just means the synthesis prose was lost while the curated evidence was n
272
272
  ## 6. Reproduce
273
273
 
274
274
  ```bash
275
- # The calibration gate that must pass FIRST ($0, offline) the metric is valid.
275
+ # The calibration gate that must pass FIRST ($0, offline): the metric is valid.
276
276
  pnpm test investment-calibration
277
277
 
278
- # The offline task wiring ($0) proves page → index → grade end-to-end.
278
+ # The offline task wiring ($0): proves page → index → grade end-to-end.
279
279
  pnpm test investment-thesis-task
280
280
 
281
281
  # The full live 3-arm A/B (costs ~$0.36 total at 5 companies × 3 arms × 3 rounds).
@@ -291,12 +291,12 @@ AGENT_KNOWLEDGE_LIVE=1 IT_LIVE_TICKERS=CVNA IT_LIVE_ARMS=collection \
291
291
 
292
292
  `IT_LIVE_ARMS` (`|`-separated subset of `collection|verify|driving`) and
293
293
  `IT_LIVE_TICKERS` scope the run; `IT_LIVE_ROUNDS` sets the per-arm round budget
294
- (default 3 driving needs > 1). The smoke (one cheap glm-5.2 call) runs once before any
294
+ (default 3, driving needs > 1). The smoke (one cheap glm-5.2 call) runs once before any
295
295
  arm, so a bad key fails fast, before the burn.
296
296
 
297
297
  ---
298
298
 
299
- *Run provenance. Calibration: `$0`, offline, reproducible numbers from
299
+ *Run provenance. Calibration: `$0`, offline, reproducible, numbers from
300
300
  `tests/eval/investment-calibration.test.ts` (shallow `< 30%`, deep `> 70%`,
301
301
  gap `> 40pp` per company and aggregate; all pass). A/B: 5 companies × 3 arms × 3 rounds
302
302
  = 15 live company-runs, glm-5.2 over the Tangle router, ~37.5 min wall, $0.36 total;