@tangle-network/agent-knowledge 4.0.1 → 5.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENTS.md +35 -19
- package/CHANGELOG.md +48 -0
- package/README.md +220 -943
- package/dist/benchmarks/index.d.ts +3 -2
- package/dist/benchmarks/index.js +1 -1
- package/dist/{chunk-EUAXSBMH.js → chunk-AKYJG2MR.js} +224 -224
- package/dist/chunk-AKYJG2MR.js.map +1 -0
- package/dist/{chunk-BBCIB4UL.js → chunk-EYIA5PLQ.js} +27 -195
- package/dist/chunk-EYIA5PLQ.js.map +1 -0
- package/dist/{chunk-3CAAGJCE.js → chunk-LMR53POQ.js} +1227 -574
- package/dist/chunk-LMR53POQ.js.map +1 -0
- package/dist/{chunk-WCYW2GDA.js → chunk-MYFM6LKH.js} +2 -2
- package/dist/chunk-MYFM6LKH.js.map +1 -0
- package/dist/cli.js +1 -1
- package/dist/index-Cf7txrYP.d.ts +790 -0
- package/dist/index.d.ts +252 -73
- package/dist/index.js +2813 -2229
- package/dist/index.js.map +1 -1
- package/dist/memory/index.d.ts +8 -709
- package/dist/memory/index.js +2 -2
- package/dist/sources/index.d.ts +17 -33
- package/dist/sources/index.js +1 -1
- package/dist/{types-ZzY_x0r7.d.ts → types-BY-xLVw-.d.ts} +28 -67
- package/docs/eval/investment-material-facts.md +28 -29
- package/docs/eval/rag-eval-roadmap.md +21 -17
- package/docs/results/adaptive.md +13 -13
- package/docs/results/claim-grounding.md +11 -11
- package/docs/results/cost-quality.md +4 -4
- package/docs/results/investment-thesis.md +56 -56
- package/docs/results/research-driving.md +54 -54
- package/docs/two-agent-research-ab.md +94 -94
- package/package.json +11 -11
- package/skills/build-with-agent-knowledge/SKILL.md +97 -0
- package/dist/chunk-3CAAGJCE.js.map +0 -1
- package/dist/chunk-BBCIB4UL.js.map +0 -1
- package/dist/chunk-EUAXSBMH.js.map +0 -1
- package/dist/chunk-WCYW2GDA.js.map +0 -1
package/docs/results/adaptive.md
CHANGED
|
@@ -1,9 +1,9 @@
|
|
|
1
1
|
# Adaptive topology: spend the LLM verifier only when it pays
|
|
2
2
|
|
|
3
3
|
The cost/quality A/B (`docs/results/cost-quality.md`) found the LLM relevance
|
|
4
|
-
verifier's cleanliness win is dominated by **de-duplication
|
|
4
|
+
verifier's cleanliness win is dominated by **de-duplication**, which a
|
|
5
5
|
deterministic content-hash / canonical-URL check captures at ~none of the LLM
|
|
6
|
-
premium
|
|
6
|
+
premium, and that an LLM check only earns its dollar on the off-scope tail. The
|
|
7
7
|
production move it named was: do the cheap deterministic work first, reserve the
|
|
8
8
|
LLM for the ambiguous tail. `createAdaptiveResearchDriver`
|
|
9
9
|
(`src/adaptive-driver.ts`) is that driver, and this is its measurement.
|
|
@@ -19,7 +19,7 @@ stopping at the first that decides:
|
|
|
19
19
|
github, …) with a substantial body is **kept**; an obvious spam/listicle
|
|
20
20
|
title or a too-thin body is **dropped**; everything else is **ambiguous**.
|
|
21
21
|
3. **LLM escalation ($).** Only ambiguous survivors reach the shipped LLM
|
|
22
|
-
relevance verifier (`createVerifyingResearchDriver`)
|
|
22
|
+
relevance verifier (`createVerifyingResearchDriver`), one call each.
|
|
23
23
|
|
|
24
24
|
## Live frontier, n=5 topics (glm-5.2)
|
|
25
25
|
|
|
@@ -39,24 +39,24 @@ is the per-arm `RouterClient.usage()` diff (#36). Total spend for the run:
|
|
|
39
39
|
| **total** | **25** | **25** | **12 / 25 / $0.0261** | **20 / 6 / $0.0068** |
|
|
40
40
|
|
|
41
41
|
**Cost.** Adaptive cuts LLM verifier calls **76%** (25 → 6) and dollars **74%**
|
|
42
|
-
($0.0261 → $0.0068). On 3 of the 5 topics it spent **zero** LLM calls
|
|
42
|
+
($0.0261 → $0.0068). On 3 of the 5 topics it spent **zero** LLM calls, every
|
|
43
43
|
unique survivor was on an authoritative host, so the $0 stages decided
|
|
44
44
|
everything.
|
|
45
45
|
|
|
46
|
-
##
|
|
46
|
+
## Interpretation: adaptive occupies one cost-quality point
|
|
47
47
|
|
|
48
48
|
Admitted-source counts (lower = cleaner KB): **single 25, adaptive 20,
|
|
49
49
|
full-LLM 12**. Adaptive sits **between** the two:
|
|
50
50
|
|
|
51
51
|
- It removes **5 of the 13 sources** the full-LLM judge rejects that the
|
|
52
52
|
single-agent loop keeps (every one of them a real duplicate caught by the $0
|
|
53
|
-
dedup stage
|
|
53
|
+
dedup stage, exactly the de-dup-dominated win the cost/quality result
|
|
54
54
|
predicted).
|
|
55
55
|
- It does **NOT** match full-LLM cleanliness. The remaining 8 sources full-LLM
|
|
56
56
|
rejects, adaptive keeps. The cause is structural and visible in the trace: on
|
|
57
57
|
this topic set every non-duplicate survivor landed on an authoritative host
|
|
58
58
|
(arxiv / github / official docs), so the heuristic **kept** it without ever
|
|
59
|
-
asking the LLM
|
|
59
|
+
asking the LLM, and the LLM, when full-LLM did ask it, judged several of
|
|
60
60
|
those same authoritative pages not-quite-on-topic and dropped them. The host
|
|
61
61
|
prior is coarser than the relevance judge.
|
|
62
62
|
|
|
@@ -65,19 +65,19 @@ de-dup half of full-LLM's cleanliness for ~26% of full-LLM's dollars, and gives
|
|
|
65
65
|
up the relevance-judgment half.** Whether that is the right point depends on the
|
|
66
66
|
cost of a kept-but-marginal source. If a slightly-off-topic authoritative page
|
|
67
67
|
is cheap to carry, adaptive dominates. If every admitted source must clear a
|
|
68
|
-
relevance bar, the host heuristic is too permissive and you want the full LLM
|
|
68
|
+
relevance bar, the host heuristic is too permissive and you want the full LLM,
|
|
69
69
|
or a tightened heuristic.
|
|
70
70
|
|
|
71
|
-
## Where the heuristic is weak
|
|
71
|
+
## Where the heuristic is weak: stated plainly
|
|
72
72
|
|
|
73
73
|
The escalation count is the diagnostic. On 3 of 5 topics it was **zero**: the
|
|
74
74
|
heuristic never deferred to the LLM, so on those topics adaptive is a
|
|
75
75
|
**pure host/title/length rule**, and its cleanliness is exactly that rule's
|
|
76
|
-
cleanliness
|
|
76
|
+
cleanliness, no smarter than "trust arxiv/github, drop spam." That is fine when
|
|
77
77
|
the worker's sources are dominated by authoritative hosts (as here), but it
|
|
78
78
|
means the LLM's relevance judgment is contributing nothing on those topics, by
|
|
79
79
|
construction. The two topics where adaptive *did* escalate (grouped-query
|
|
80
|
-
attention, LoRA) are where some survivors were on unknown hosts
|
|
80
|
+
attention, LoRA) are where some survivors were on unknown hosts, and there the
|
|
81
81
|
3 LLM calls per topic are the off-scope tail the verifier is actually for.
|
|
82
82
|
|
|
83
83
|
The heuristic would mis-route in two directions a richer worker would expose,
|
|
@@ -95,12 +95,12 @@ neither seen on this authoritative-host-heavy set:
|
|
|
95
95
|
The deployable recommendation from the cost/quality result was "deterministic
|
|
96
96
|
dedup first, reserve the LLM for the tail." This driver ships that and the
|
|
97
97
|
measurement confirms the **cost** half cleanly (76% fewer calls, 74% cheaper)
|
|
98
|
-
and qualifies the **quality** half
|
|
98
|
+
and qualifies the **quality** half accurately: adaptive captures the de-dup
|
|
99
99
|
cleanliness (the dominant, free win) but not the LLM's relevance cleanliness,
|
|
100
100
|
because the host heuristic resolves authoritative survivors without asking. For
|
|
101
101
|
a worker whose sources are mostly authoritative, adaptive is the right frontier
|
|
102
102
|
point. For one whose junk is on-topic-looking pages on unknown hosts, the
|
|
103
|
-
ambiguous tail grows and adaptive converges toward full-LLM cost
|
|
103
|
+
ambiguous tail grows and adaptive converges toward full-LLM cost, which is the
|
|
104
104
|
correct behavior: it pays for the LLM exactly when the cheap signals can't
|
|
105
105
|
decide.
|
|
106
106
|
|
|
@@ -1,22 +1,22 @@
|
|
|
1
1
|
# Claim-grounding: the band where the verifier earns its dollar
|
|
2
2
|
|
|
3
3
|
`docs/results/cost-quality.md` found the relevance verifier's cleanliness win is
|
|
4
|
-
**dominated by de-duplication
|
|
4
|
+
**dominated by de-duplication**, a deterministic content-hash captures most of it
|
|
5
5
|
at ~none of the LLM premium. So the open question was: is there an error band where
|
|
6
|
-
a verifier earns its cost
|
|
6
|
+
a verifier earns its cost, something a hash AND a relevance judge both miss?
|
|
7
7
|
|
|
8
8
|
**Yes: misattributed citations.** A source that is on-topic, unique, and real, but
|
|
9
9
|
whose cited CLAIM does not appear in the page (the LLM wrote a plausible sentence
|
|
10
10
|
and hung a real URL off it). De-dup passes it (it's unique). A relevance judge
|
|
11
11
|
passes it (the page is on-topic). Only checking the claim against the fetched text
|
|
12
|
-
catches it
|
|
12
|
+
catches it, and that check is **deterministic text presence, $0 inference**.
|
|
13
13
|
|
|
14
14
|
## The mode
|
|
15
15
|
|
|
16
16
|
Each proposed source now carries the specific claim it is cited for
|
|
17
17
|
(`withCitedClaim` → `metadata.citedClaim`). The verifier
|
|
18
18
|
(`createClaimGroundingVerifier`) runs `groundClaimInText(claim, pageText)` over the
|
|
19
|
-
`htmlToText` output of the page the worker actually fetched
|
|
19
|
+
`htmlToText` output of the page the worker actually fetched, verbatim, normalized
|
|
20
20
|
(punctuation/whitespace-insensitive), or a ≥70% content-word overlap close
|
|
21
21
|
paraphrase. A claim that isn't present is rejected as **misattributed**. The oracle
|
|
22
22
|
is text presence, not a model call, so it composes with the LLM relevance verifier
|
|
@@ -31,25 +31,25 @@ proposals. Cost diffed per arm with the #36 `RouterClient.usage()` instrumentati
|
|
|
31
31
|
|
|
32
32
|
| n=5 topics | misattributions caught | marginal $ | $/topic | per-$ caught |
|
|
33
33
|
|---|---|---|---|---|
|
|
34
|
-
| no-verifier | 0 / 5 | $0.0000 |
|
|
34
|
+
| no-verifier | 0 / 5 | $0.0000 | n/a | n/a |
|
|
35
35
|
| relevance (LLM judge) | **4 / 5** | $0.0157 | ~$0.0031 | 254 |
|
|
36
36
|
| claim-grounding (text) | **5 / 5** | **$0.0000** | $0 | ∞ |
|
|
37
37
|
|
|
38
38
|
Per-topic (caught relevance / grounding): self-speculative decoding 1/1, rotary
|
|
39
39
|
position embeddings 1/1, grouped-query attention 1/1, **KV-cache quantization 0/1**,
|
|
40
|
-
LoRA 1/1. (An earlier 3-topic run missed self-speculative decoding instead
|
|
40
|
+
LoRA 1/1. (An earlier 3-topic run missed self-speculative decoding instead, the
|
|
41
41
|
miss moves around; it is not a fixed topic.)
|
|
42
42
|
|
|
43
43
|
**Reading.** Claim-grounding catches every misattribution at **$0**; the relevance
|
|
44
44
|
judge catches most but **misses one in five at ~$0.003/topic**. The miss is the
|
|
45
45
|
point: the relevance verifier only ever sees the page text, never the cited claim,
|
|
46
|
-
so it is *structurally blind* to misattribution. It catches one only by accident
|
|
46
|
+
so it is *structurally blind* to misattribution. It catches one only by accident:
|
|
47
47
|
when the fabricated claim happens to also make the page read off-topic
|
|
48
48
|
(e.g. a "12-billion-parameter draft transformer" claim on a rotary-embeddings page).
|
|
49
49
|
When the fabrication stays on-topic (the KV-cache case), the judge waves it through.
|
|
50
50
|
|
|
51
51
|
So on THIS band the verifier-per-dollar comparison inverts the cost/quality result:
|
|
52
|
-
there, the LLM verifier bought a dedup-shaped gain a free hash already captures
|
|
52
|
+
there, the LLM verifier bought a dedup-shaped gain a free hash already captures;
|
|
53
53
|
expensive for what a cheap rule does. Here the cheap, deterministic check
|
|
54
54
|
**dominates** the expensive judge: it catches strictly more (5/5 vs 4/5) at strictly
|
|
55
55
|
less ($0 vs $0.0157). The verifier earns its dollar on misattribution; it does not on
|
|
@@ -63,7 +63,7 @@ de-duplication.
|
|
|
63
63
|
(and the offline relevance stand-in) accept them. The error is in the *claim*, not
|
|
64
64
|
the *topic*.
|
|
65
65
|
- **Executable ground truth.** The check is presence/close-paraphrase of the claim
|
|
66
|
-
in the fetched text
|
|
66
|
+
in the fetched text, deployable in production with no oracle and no model call.
|
|
67
67
|
|
|
68
68
|
The offline arm proves the floor with a controlled 4-source pool (2 grounded, 2
|
|
69
69
|
misattributed): claim-grounding admits **0/2** misattributions and keeps **2/2**
|
|
@@ -78,9 +78,9 @@ grounded sources, while relevance and no-verifier both admit **2/2**.
|
|
|
78
78
|
- **Planted misattributions, not naturally-occurring ones.** Like the cost/quality
|
|
79
79
|
offline floor, the misattribution is injected so the band is measurable. It models
|
|
80
80
|
the real LLM citation-fabrication failure but does not measure its base rate in the
|
|
81
|
-
wild
|
|
81
|
+
wild, that needs a corpus of model-written citations checked by hand.
|
|
82
82
|
- **The grounding oracle is conservative.** A real paraphrase whose inflected words
|
|
83
|
-
differ from the page ("drafts" vs "draft") can score below 0.7 and be rejected
|
|
83
|
+
differ from the page ("drafts" vs "draft") can score below 0.7 and be rejected,
|
|
84
84
|
a false-positive misattribution flag. `minOverlap` tunes this; the worker should
|
|
85
85
|
cite the page's own key terms (the `createClaimDecorator` extractor is told to).
|
|
86
86
|
|
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
Live 9-topic A/B (glm-5.2, budget B ≤ 4 passes/arm), measured per arm with the
|
|
4
4
|
router-client instrumentation. The original A/B reported only admitted-sources at
|
|
5
|
-
"equal passes"
|
|
5
|
+
"equal passes", which charged the two-agent verify step as one pass while it is
|
|
6
6
|
actually N `verifySource` LLM calls. Pricing the calls shows what that hid.
|
|
7
7
|
|
|
8
8
|
| per topic (mean) | two-agent | single-agent | ratio |
|
|
@@ -11,7 +11,7 @@ actually N `verifySource` LLM calls. Pricing the calls shows what that hid.
|
|
|
11
11
|
| tokens (in+out) | ~4,900 | ~530 | ~9× |
|
|
12
12
|
| cost (USD) | ~$0.0072 | ~$0.0013 | ~5.5× |
|
|
13
13
|
| latency (wall) | ~37 s | ~11 s | ~3.4× |
|
|
14
|
-
| cleanliness Δ (single − two admitted) |
|
|
14
|
+
| cleanliness Δ (single − two admitted) | n/a | n/a | +1.56, 95% CI [0.33, 2.67] |
|
|
15
15
|
|
|
16
16
|
Per-topic Δ (single − two admitted) this run: self-speculative decoding +4,
|
|
17
17
|
grouped-query attention 0, rotary position embeddings +1, KV-cache quantization
|
|
@@ -19,9 +19,9 @@ grouped-query attention 0, rotary position embeddings +1, KV-cache quantization
|
|
|
19
19
|
descent **−1**. Coverage 1.00 every topic, both arms.
|
|
20
20
|
|
|
21
21
|
**Reading.** The verifier buys ~1.5–2.7 fewer junk sources for roughly **5× the
|
|
22
|
-
dollars, 9× the tokens, and 3× the latency
|
|
22
|
+
dollars, 9× the tokens, and 3× the latency**, and on two topics it admitted *more*
|
|
23
23
|
than the single agent (the cleanliness signal is real but noisier than the +2.3 /
|
|
24
24
|
+2.7 of earlier runs). The cleanliness gain is dominated by de-duplication, so the
|
|
25
|
-
|
|
25
|
+
production move is a deterministic content-hash / canonical-URL dedup, which
|
|
26
26
|
captures most of the cleanliness at ~none of this premium; reserve an LLM check for
|
|
27
27
|
the off-scope tail. This is the cost half the "equal passes" framing left out.
|
|
@@ -1,25 +1,25 @@
|
|
|
1
|
-
#
|
|
1
|
+
# Investment-research depth eval and three-arm comparison
|
|
2
2
|
|
|
3
3
|
*Tangle Network · `agent-knowledge`*
|
|
4
4
|
|
|
5
|
-
## Verdict
|
|
5
|
+
## Verdict
|
|
6
6
|
|
|
7
|
-
We moved the research eval off ML-paper retrieval
|
|
7
|
+
We moved the research eval off ML-paper retrieval, where a **single** web search
|
|
8
8
|
already returns the answer, so the only thing the metric could measure was
|
|
9
|
-
*collection
|
|
9
|
+
*collection*, onto **investment research**, where the material facts are buried in
|
|
10
10
|
the footnotes of an SEC filing and genuinely require *investigation* to surface. On
|
|
11
11
|
this harder domain we (1) **calibrated** a `$0`, model-free metric and proved it
|
|
12
|
-
discriminates research depth
|
|
13
|
-
deep filings-grounded thesis scores **27/27 (100%)**, a **+96-point** gap
|
|
12
|
+
discriminates research depth, a shallow ticker summary scores **1/27 (4%)**, a
|
|
13
|
+
deep filings-grounded thesis scores **27/27 (100%)**, a **+96-point** gap, and then
|
|
14
14
|
(2) ran **three research topologies head-to-head** on the same five held-out
|
|
15
15
|
companies at matched compute, grading each one's knowledge base against a firewalled
|
|
16
16
|
checklist of **27 buried material facts** a ticker search misses.
|
|
17
17
|
|
|
18
|
-
**
|
|
19
|
-
(16/27, 59%), +5 over blind collection (11/27, 41%)
|
|
18
|
+
**A/B result: the research-driving loop surfaced the most buried facts
|
|
19
|
+
(16/27, 59%), +5 over blind collection (11/27, 41%), but at n=5 companies that lift
|
|
20
20
|
is NOT statistically clean: the 95% confidence interval crosses zero (P(Δ≤0)=0.08).**
|
|
21
|
-
The verify/dedup loop did **not** beat collection at all (10/27, 37
|
|
22
|
-
slightly worse).
|
|
21
|
+
The verify/dedup loop did **not** beat collection at all (10/27, 37%, a wash,
|
|
22
|
+
slightly worse). No approach *significantly* beats collection here.
|
|
23
23
|
**driving is the only arm that even points the right way, and it points there
|
|
24
24
|
suggestively, not significantly.** What the reframe *did* buy is a domain and a meter
|
|
25
25
|
where the question is finally well-posed: the metric is no longer measuring whether a
|
|
@@ -27,7 +27,7 @@ single search ran, but whether investigation reached the buried fact.
|
|
|
27
27
|
|
|
28
28
|
| arm | what the coordinator does | material facts | cost (5 cos.) | chats | searches | tokens |
|
|
29
29
|
|---|---|---|---|---|---|---|
|
|
30
|
-
| **A · collection** | nothing
|
|
30
|
+
| **A · collection** | nothing; accepts every source (1 agent collects) | 11/27 (41%) | $0.082 | 10 | 20 | 64,248 |
|
|
31
31
|
| **B · verify/dedup** | LLM gates each source for relevance, rejects off-topic | 10/27 (37%) | $0.125 | 56 | 52 | 84,678 |
|
|
32
32
|
| **C · driving (deepen)** | extracts claims, demands corroboration, asks deep follow-ups | **16/27 (59%)** | $0.157 | 33 | 28 | 124,387 |
|
|
33
33
|
|
|
@@ -35,17 +35,17 @@ Cost is real provenance, not an estimate: every `$`/call/token is diffed from
|
|
|
35
35
|
`RouterClient.usage()` per company (the router's own `usage` field, priced at
|
|
36
36
|
glm-5.2 rates), so "driving cost 1.9× collection" is measured, not modelled.
|
|
37
37
|
|
|
38
|
-
## 1. The domain reframe
|
|
38
|
+
## 1. The domain reframe: why we left ML-paper retrieval
|
|
39
39
|
|
|
40
40
|
The companion paper's depth eval (`docs/two-agent-research-ab.md` §9) measures a
|
|
41
|
-
research loop on **20 deep questions across 5 ML topics**. That
|
|
41
|
+
research loop on **20 deep questions across 5 ML topics**. That method is sound:
|
|
42
42
|
its grader scores 0/20 on a one-line topic definition and 20/20 on a mechanism-rich
|
|
43
43
|
paragraph, so it *can* tell depth from surface. But the topology A/B on top of it
|
|
44
|
-
came back a clean null (driving 16/20 @ budget 4, 13/20 @ budget 6
|
|
44
|
+
came back a clean null (driving 16/20 @ budget 4, 13/20 @ budget 6, the winner
|
|
45
45
|
*flips with the compute budget*, and the within-arm swing is as large as the
|
|
46
46
|
between-arm gap). The autopsy named the cause: **on an ML topic a single good search
|
|
47
47
|
already collects the answer.** Every arm finished in one effective round because the
|
|
48
|
-
generic "one source closes the gap" readiness was met by the *first* fetch
|
|
48
|
+
generic "one source closes the gap" readiness was met by the *first* fetch, so the
|
|
49
49
|
driving driver, whose entire mechanism is steering a *second* round, never got to
|
|
50
50
|
act. When one search suffices, there is no investigation for a smarter coordinator to
|
|
51
51
|
do, and the metric can only reward collection. That is the failure-mode-in-spirit the
|
|
@@ -55,12 +55,12 @@ easy for topology to matter.
|
|
|
55
55
|
So we changed the domain to one where a single search **provably cannot** suffice.
|
|
56
56
|
**Investment research**: give a loop a company + ticker + an as-of cutoff date and ask
|
|
57
57
|
for a thesis; grade it on the buried, material, non-obvious drivers a thorough analyst
|
|
58
|
-
flags and a ticker search misses. The decisive facts here live in 10-K footnotes
|
|
58
|
+
flags and a ticker search misses. The decisive facts here live in 10-K footnotes, an
|
|
59
59
|
HTM securities mark roughly equal to a bank's entire equity (SIVB), a deposit base
|
|
60
60
|
that is 97% uninsured, a negative per-unit gross margin, a related-party lease. A web
|
|
61
61
|
search for the company name returns "profitable regional bank" or "high-growth auto
|
|
62
62
|
e-commerce"; the filing shows the mark-to-market hole the size of capital. **The
|
|
63
|
-
answer is not collectable in one fetch
|
|
63
|
+
answer is not collectable in one fetch, it has to be investigated for.** That is the
|
|
64
64
|
property the ML domain lacked, and it is what makes a topology A/B finally meaningful:
|
|
65
65
|
if a smarter coordinator can ever beat blind collection, a domain where the answer is
|
|
66
66
|
buried is where it has the room to.
|
|
@@ -69,7 +69,7 @@ The held-out set is **5 companies, 27 material facts**, every fact read live out
|
|
|
69
69
|
the primary SEC EDGAR 10-K during curation, every dollar figure quoted from the
|
|
70
70
|
de-tagged filing text, every value knowable *at the cutoff* (the eventual collapse is
|
|
71
71
|
recorded for the reader only and is **never graded**). The companies skew distressed /
|
|
72
|
-
downside-risk
|
|
72
|
+
downside-risk, a documented curation bias, not a hidden one (full provenance, the
|
|
73
73
|
drop log, and the per-fact keyword groups: `docs/eval/investment-material-facts.md`):
|
|
74
74
|
|
|
75
75
|
| ticker | company | as-of cutoff | CIK | facts |
|
|
@@ -80,25 +80,25 @@ drop log, and the per-fact keyword groups: `docs/eval/investment-material-facts.
|
|
|
80
80
|
| PTON | Peloton | 2022-09-07 | 1639825 | 6 |
|
|
81
81
|
| SI | Silvergate | 2022-02-28 | 1312109 | 5 |
|
|
82
82
|
|
|
83
|
-
## 2. Calibration
|
|
83
|
+
## 2. Calibration: does the metric discriminate depth? (the gate the ML exam passed only weakly)
|
|
84
84
|
|
|
85
85
|
A topology A/B is meaningless unless the metric grading it can tell a *deep* thesis
|
|
86
|
-
from a *shallow* one. If it can't, it is measuring word-collection
|
|
87
|
-
the reframe set out to escape
|
|
86
|
+
from a *shallow* one. If it can't, it is measuring word-collection, the exact failure
|
|
87
|
+
the reframe set out to escape, and any A/B on top of it is noise. So **before** the
|
|
88
88
|
A/B, we ran a calibration gate (`$0`, offline, the binding result):
|
|
89
89
|
|
|
90
90
|
For each of the 5 companies we hand-wrote two theses and scored each with the metric's
|
|
91
91
|
pure core (`materialFactsSurfacedInText`):
|
|
92
92
|
|
|
93
|
-
- **shallow
|
|
93
|
+
- **shallow**: a one-paragraph ticker summary: what the company does, a vibe on the
|
|
94
94
|
stock, generic macro/competition risk. The kind a single name-search returns. Names
|
|
95
95
|
none of the buried, filing-level facts.
|
|
96
|
-
- **deep
|
|
96
|
+
- **deep**: a filings-grounded analyst memo naming the buried drivers (the duration
|
|
97
97
|
loss, the buyback drain, the negative unit margin, the deposit concentration, the
|
|
98
98
|
related party) in independent prose, with the real numbers.
|
|
99
99
|
|
|
100
100
|
The grader is the same `$0`, model-free, case-insensitive substring check the held-out
|
|
101
|
-
checklist ships (`gradeFactAgainstText`); the checklist is firewalled
|
|
101
|
+
checklist ships (`gradeFactAgainstText`); the checklist is firewalled, read only by
|
|
102
102
|
the metric, never shown to a loop.
|
|
103
103
|
|
|
104
104
|
| ticker | shallow | deep | gap |
|
|
@@ -110,20 +110,20 @@ the metric, never shown to a loop.
|
|
|
110
110
|
| SI | 0/5 (0%) | 5/5 (100%) | +100pp |
|
|
111
111
|
| **total** | **1/27 (4%)** | **27/27 (100%)** | **+96pp** |
|
|
112
112
|
|
|
113
|
-
**
|
|
114
|
-
filings-grounded thesis
|
|
113
|
+
**Calibration result:** the metric separates a shallow ticker-summary from a deep,
|
|
114
|
+
filings-grounded thesis, a +96-point aggregate gap, every company clearing the bars
|
|
115
115
|
(shallow `< 30%`, deep `> 70%`) with wide margin. Two guards make the gap real and not
|
|
116
116
|
a teaching-to-the-test artifact:
|
|
117
117
|
|
|
118
118
|
- **Anti-circularity.** The gate asserts **no deep thesis verbatim-embeds any
|
|
119
|
-
checklist `evidence` string
|
|
119
|
+
checklist `evidence` string**, the deep theses state the same publicly-documented
|
|
120
120
|
facts in independent analyst prose. A high deep score is the meter catching real,
|
|
121
121
|
independently-phrased depth, not an answer-key echo.
|
|
122
|
-
- **The one
|
|
123
|
-
the innovation-economy / venture-client concentration
|
|
122
|
+
- **The one shallow hit.** The single shallow surface (SIVB 1/6) is SIVB/f5:
|
|
123
|
+
the innovation-economy / venture-client concentration, which fires on "technology"
|
|
124
124
|
/ "venture" in the SVB summary. That genuinely *is* the least-buried of SVB's six
|
|
125
125
|
facts (a ticker search does return "SVB banks tech and venture startups"). We kept it
|
|
126
|
-
as a 4% leak rather than tighten the group, because it
|
|
126
|
+
as a 4% leak rather than tighten the group, because it reflects one
|
|
127
127
|
near-surface fact while the five truly buried ones (the ~$15B HTM loss, the
|
|
128
128
|
equity-sized mark, the $151.5B uninsured base, the 20-point deposit-mix shift, the
|
|
129
129
|
AFS/AOCI loss) stay unsurfaced. The firewall holds (17% << 30%).
|
|
@@ -132,38 +132,38 @@ Calibration also did its job as more than a rubber stamp: the first pass had the
|
|
|
132
132
|
**Silvergate shallow thesis scoring 3/5 (60%)**, blowing the bar, because three fact
|
|
133
133
|
groups accepted bare crypto-bank vocabulary (`crypto`, `grew rapidly`, `proprietary` /
|
|
134
134
|
`payment network`) that any one-line summary trips. We tightened those three to require
|
|
135
|
-
the *buried* signal
|
|
136
|
-
figure, the **SEN** (Silvergate Exchange Network) name
|
|
135
|
+
the *buried* signal, the deposit-**concentration** framing, the specific **$14.3B**
|
|
136
|
+
figure, the **SEN** (Silvergate Exchange Network) name, and dropped the generic vocab.
|
|
137
137
|
The deep thesis still hits all three; the shallow one no longer does. The metric found
|
|
138
138
|
a way it could be fooled and we closed it before any A/B depended on it.
|
|
139
139
|
|
|
140
140
|
**This is the gate the ML exam passed only in spirit.** The ML grader discriminated
|
|
141
|
-
0/20 vs 20/20 too
|
|
141
|
+
0/20 vs 20/20 too, but on a domain where the deep answer was *collectable* in one
|
|
142
142
|
search, so the discrimination measured grammar, not investigation. Here the +96-point
|
|
143
143
|
gap is over facts that are *buried by construction*, so a high score is reachable only
|
|
144
144
|
by reaching into the filing. Same shape of result, materially harder domain.
|
|
145
145
|
|
|
146
|
-
## 3. The 3-arm A/B
|
|
146
|
+
## 3. The 3-arm A/B: what each topology surfaced, head-to-head
|
|
147
147
|
|
|
148
148
|
For each company the loop is told **only** `{company, ticker, cik, cutoff}` plus a
|
|
149
149
|
generic set of analyst-lens readiness specs (balance-sheet risk, concentration,
|
|
150
|
-
leverage, margins, liquidity, governance, regulatory
|
|
151
|
-
live, the latest SEC 10-K
|
|
150
|
+
leverage, margins, liquidity, governance, regulatory, the *lenses* and where they
|
|
151
|
+
live, the latest SEC 10-K, **not the answers**). It researches the company *as of* the
|
|
152
152
|
cutoff over web + SEC EDGAR (both public), writes a thesis, and we grade the resulting
|
|
153
|
-
KB with `materialFactsSurfaced
|
|
153
|
+
KB with `materialFactsSurfaced`, the firewalled checklist the loop never sees.
|
|
154
154
|
|
|
155
155
|
**Compute is matched by construction.** All three arms run the *same* web worker, the
|
|
156
156
|
*same* 3-round budget, the *same* worker config (`resultsPerQuery: 3, queriesPerGap: 1,
|
|
157
|
-
maxSourcesPerRound: 6`). The **only** thing that varies is the driver
|
|
157
|
+
maxSourcesPerRound: 6`). The **only** thing that varies is the driver, the coordinator
|
|
158
158
|
between the worker and the knowledge base:
|
|
159
159
|
|
|
160
|
-
- **A · collection** (`createCollectionResearchDriver`)
|
|
160
|
+
- **A · collection** (`createCollectionResearchDriver`), an inert rubber stamp:
|
|
161
161
|
accepts every source, gates nothing, steers only with the loop's built-in open-gap
|
|
162
162
|
list. The driver adds **zero** router calls. This is the blind-collection floor.
|
|
163
|
-
- **B · verify/dedup** (`createVerifyingResearchDriver`)
|
|
163
|
+
- **B · verify/dedup** (`createVerifyingResearchDriver`), an LLM relevance gate: one
|
|
164
164
|
chat call per candidate source to accept-or-reject for on-topic relevance and
|
|
165
165
|
near-duplication. The worker ADDS; the driver GATES.
|
|
166
|
-
- **C · driving** (`createResearchDrivingDriver`)
|
|
166
|
+
- **C · driving** (`createResearchDrivingDriver`), extracts each source's claims,
|
|
167
167
|
tracks how many *independent* sources corroborate each, and synthesizes deep
|
|
168
168
|
follow-up sub-questions (comparative / mechanism / gap / contradiction) it folds into
|
|
169
169
|
the worker's next prompt to push depth and demand corroboration.
|
|
@@ -200,13 +200,13 @@ Every interval crosses zero. **Driving vs collection is the closest to clean
|
|
|
200
200
|
coin flip. This is the project's well-documented small-n mirage: exciting deltas born
|
|
201
201
|
at n=5 do not survive a paired bootstrap.
|
|
202
202
|
|
|
203
|
-
## 4. Autopsy
|
|
203
|
+
## 4. Autopsy: the two things worth understanding
|
|
204
204
|
|
|
205
205
|
### 4.1 Why driving wins where it wins
|
|
206
206
|
|
|
207
207
|
Driving's mechanism is multi-round: extract claims from round 1, then steer the worker
|
|
208
208
|
in rounds 2–3 to corroborate the weak ones and chase the deep questions. It helps most
|
|
209
|
-
where the **first** fetch lands real filing data the driver can build on
|
|
209
|
+
where the **first** fetch lands real filing data the driver can build on, the two
|
|
210
210
|
banks, where SEC bank-call-report / 10-K data is dense and reachable. SIVB jumps from 2
|
|
211
211
|
buried facts (collection) to 5 (driving): the duration loss, the deposit concentration,
|
|
212
212
|
the AFS/AOCI mark all surface once the driver demands the balance-sheet detail a second
|
|
@@ -230,21 +230,21 @@ ROUND 1: accepted=0 rejected=3 writtenPages=0
|
|
|
230
230
|
ROUND 2: accepted=0 rejected=2 writtenPages=0
|
|
231
231
|
```
|
|
232
232
|
|
|
233
|
-
The verifier was **correct on the merits
|
|
233
|
+
The verifier was **correct on the merits**, those are aggregators, not the primary
|
|
234
234
|
filing, and last10k showed a post-cutoff filing. But the worker never surfaced the
|
|
235
235
|
EDGAR primary for BBBY, so a strict primary-only gate left the KB **empty**. Collection
|
|
236
236
|
accepts the same aggregator pages and scores 4/5 on BBBY; driving accepts them and
|
|
237
237
|
scores 5/5. **The gate's strictness is a liability when the worker's sourcing is
|
|
238
|
-
imperfect:** it throws away the only evidence the loop had. This is a
|
|
239
|
-
trade-off
|
|
240
|
-
|
|
238
|
+
imperfect:** it throws away the only evidence the loop had. This is a real coordination
|
|
239
|
+
trade-off. The verify arm still surfaced facts on CVNA (2), PTON (4), and SI (4),
|
|
240
|
+
which rules out a general execution failure.
|
|
241
241
|
|
|
242
|
-
### 4.3 The empty-thesis caveat
|
|
242
|
+
### 4.3 The empty-thesis caveat
|
|
243
243
|
|
|
244
244
|
Some runs show `thesis=0ch` (empty synthesis) yet still score facts. That is expected:
|
|
245
|
-
`materialFactsSurfaced` grades the **whole KB
|
|
246
|
-
pages
|
|
247
|
-
budget on hidden reasoning and returns empty visible content on the synthesis call
|
|
245
|
+
`materialFactsSurfaced` grades the **whole KB**, including the worker's fetched
|
|
246
|
+
`knowledge/*.md` pages and the final thesis page. glm-5.2 occasionally spends its entire output
|
|
247
|
+
budget on hidden reasoning and returns empty visible content on the synthesis call, a
|
|
248
248
|
known reasoning-model behavior we floor at 1200 tokens but can't fully eliminate. The
|
|
249
249
|
score still reflects what the loop fetched, so an empty thesis does not invalidate a
|
|
250
250
|
run; it just means the synthesis prose was lost while the curated evidence was not.
|
|
@@ -252,7 +252,7 @@ run; it just means the synthesis prose was lost while the curated evidence was n
|
|
|
252
252
|
## 5. What this does and does not establish
|
|
253
253
|
|
|
254
254
|
- **Does**: the metric *discriminates depth on a domain where one search is not
|
|
255
|
-
enough* (calibration: 1/27 shallow vs 27/27 deep, +96pp)
|
|
255
|
+
enough* (calibration: 1/27 shallow vs 27/27 deep, +96pp), so the topology A/B on top
|
|
256
256
|
of it is finally well-posed, unlike the ML retrieval exam where one search already
|
|
257
257
|
collected the answer. At matched compute on the 5-company held-out set, the
|
|
258
258
|
research-driving topology surfaced the most buried material facts (16/27 vs 11/27 for
|
|
@@ -272,10 +272,10 @@ run; it just means the synthesis prose was lost while the curated evidence was n
|
|
|
272
272
|
## 6. Reproduce
|
|
273
273
|
|
|
274
274
|
```bash
|
|
275
|
-
# The calibration gate that must pass FIRST ($0, offline)
|
|
275
|
+
# The calibration gate that must pass FIRST ($0, offline): the metric is valid.
|
|
276
276
|
pnpm test investment-calibration
|
|
277
277
|
|
|
278
|
-
# The offline task wiring ($0)
|
|
278
|
+
# The offline task wiring ($0): proves page → index → grade end-to-end.
|
|
279
279
|
pnpm test investment-thesis-task
|
|
280
280
|
|
|
281
281
|
# The full live 3-arm A/B (costs ~$0.36 total at 5 companies × 3 arms × 3 rounds).
|
|
@@ -291,12 +291,12 @@ AGENT_KNOWLEDGE_LIVE=1 IT_LIVE_TICKERS=CVNA IT_LIVE_ARMS=collection \
|
|
|
291
291
|
|
|
292
292
|
`IT_LIVE_ARMS` (`|`-separated subset of `collection|verify|driving`) and
|
|
293
293
|
`IT_LIVE_TICKERS` scope the run; `IT_LIVE_ROUNDS` sets the per-arm round budget
|
|
294
|
-
(default 3
|
|
294
|
+
(default 3, driving needs > 1). The smoke (one cheap glm-5.2 call) runs once before any
|
|
295
295
|
arm, so a bad key fails fast, before the burn.
|
|
296
296
|
|
|
297
297
|
---
|
|
298
298
|
|
|
299
|
-
*Run provenance. Calibration: `$0`, offline, reproducible
|
|
299
|
+
*Run provenance. Calibration: `$0`, offline, reproducible, numbers from
|
|
300
300
|
`tests/eval/investment-calibration.test.ts` (shallow `< 30%`, deep `> 70%`,
|
|
301
301
|
gap `> 40pp` per company and aggregate; all pass). A/B: 5 companies × 3 arms × 3 rounds
|
|
302
302
|
= 15 live company-runs, glm-5.2 over the Tangle router, ~37.5 min wall, $0.36 total;
|