@tangle-network/agent-knowledge 1.8.0 → 1.10.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +6 -0
- package/dist/index.d.ts +739 -321
- package/dist/index.js +653 -0
- package/dist/index.js.map +1 -1
- package/dist/memory/index.d.ts +1 -1
- package/dist/{types-BEDGlXB-.d.ts → types-C17sAROL.d.ts} +1 -1
- package/dist/viz/index.d.ts +1 -1
- package/docs/results/adaptive.md +120 -0
- package/docs/results/claim-grounding.md +97 -0
- package/docs/results/cost-quality.md +27 -0
- package/docs/supervisor-profiles.md +295 -0
- package/docs/two-agent-research-ab.md +428 -0
- package/package.json +1 -1
package/dist/memory/index.d.ts
CHANGED
|
@@ -170,4 +170,4 @@ interface KnowledgeRelease {
|
|
|
170
170
|
metadata?: Record<string, unknown>;
|
|
171
171
|
}
|
|
172
172
|
|
|
173
|
-
export type { ClaimRef as C, KnowledgeGraphEdge as K, SourceRecord as S, KnowledgeGraphNode as a, KnowledgeGraph as b, KnowledgeIndex as c, KnowledgeSearchResult as d,
|
|
173
|
+
export type { ClaimRef as C, KnowledgeGraphEdge as K, SourceRecord as S, KnowledgeGraphNode as a, KnowledgeGraph as b, KnowledgeIndex as c, KnowledgeSearchResult as d, SourceRegistry as e, KnowledgeEvent as f, KnowledgeEventType as g, KnowledgePage as h, KnowledgeLintFinding as i, KnowledgeWriteBlock as j, KnowledgeClaim as k, KnowledgeRelease as l, KnowledgeWriteParseResult as m, KnowledgeBaseCandidate as n, KnowledgeId as o, KnowledgePolicy as p, KnowledgeRelation as q, KnowledgeUnit as r, SourceAnchor as s };
|
package/dist/viz/index.d.ts
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
import { K as KnowledgeGraphEdge, a as KnowledgeGraphNode, b as KnowledgeGraph } from '../types-
|
|
1
|
+
import { K as KnowledgeGraphEdge, a as KnowledgeGraphNode, b as KnowledgeGraph } from '../types-C17sAROL.js';
|
|
2
2
|
|
|
3
3
|
interface KnowledgeVizNode extends KnowledgeGraphNode {
|
|
4
4
|
degree: number;
|
|
@@ -0,0 +1,120 @@
|
|
|
1
|
+
# Adaptive topology: spend the LLM verifier only when it pays
|
|
2
|
+
|
|
3
|
+
The cost/quality A/B (`docs/results/cost-quality.md`) found the LLM relevance
|
|
4
|
+
verifier's cleanliness win is dominated by **de-duplication** — which a
|
|
5
|
+
deterministic content-hash / canonical-URL check captures at ~none of the LLM
|
|
6
|
+
premium — and that an LLM check only earns its dollar on the off-scope tail. The
|
|
7
|
+
production move it named was: do the cheap deterministic work first, reserve the
|
|
8
|
+
LLM for the ambiguous tail. `createAdaptiveResearchDriver`
|
|
9
|
+
(`src/adaptive-driver.ts`) is that driver, and this is its measurement.
|
|
10
|
+
|
|
11
|
+
Per candidate source the adaptive driver runs three stages, cheapest first,
|
|
12
|
+
stopping at the first that decides:
|
|
13
|
+
|
|
14
|
+
1. **Dedup ($0).** Reject a source whose canonical URL (scheme/`www`/trailing
|
|
15
|
+
slash/tracking params stripped) or normalized-text content hash matches one
|
|
16
|
+
already accepted this round or in the KB.
|
|
17
|
+
2. **Heuristic triage ($0).** Classify a unique survivor with host/title/length
|
|
18
|
+
signals only: an authoritative host (arxiv, `*.edu`, `*.gov`, official docs,
|
|
19
|
+
github, …) with a substantial body is **kept**; an obvious spam/listicle
|
|
20
|
+
title or a too-thin body is **dropped**; everything else is **ambiguous**.
|
|
21
|
+
3. **LLM escalation ($).** Only ambiguous survivors reach the shipped LLM
|
|
22
|
+
relevance verifier (`createVerifyingResearchDriver`) — one call each.
|
|
23
|
+
|
|
24
|
+
## Live frontier, n=5 topics (glm-5.2)
|
|
25
|
+
|
|
26
|
+
Real web-research worker fetches each topic once; the same fetched proposals
|
|
27
|
+
(plus one planted tracking-decorated mirror of the first source, so the dedup
|
|
28
|
+
stage has a real duplicate to catch) are gated through all three drivers. Cost
|
|
29
|
+
is the per-arm `RouterClient.usage()` diff (#36). Total spend for the run:
|
|
30
|
+
**$0.033**.
|
|
31
|
+
|
|
32
|
+
| topic | fetched | single admit | full-LLM admit / calls / $ | adaptive admit / LLM calls / $ |
|
|
33
|
+
|---|---|---|---|---|
|
|
34
|
+
| self-speculative decoding | 3 | 3 | 1 / 3 / $0.0027 | 2 / **0** / **$0.0000** |
|
|
35
|
+
| rotary position embeddings | 3 | 3 | 1 / 3 / $0.0031 | 2 / **0** / **$0.0000** |
|
|
36
|
+
| grouped-query attention | 7 | 7 | 3 / 7 / $0.0072 | 6 / 3 / $0.0030 |
|
|
37
|
+
| KV-cache quantization | 5 | 5 | 3 / 5 / $0.0052 | 4 / **0** / **$0.0000** |
|
|
38
|
+
| LoRA fine-tuning | 7 | 7 | 4 / 7 / $0.0079 | 6 / 3 / $0.0037 |
|
|
39
|
+
| **total** | **25** | **25** | **12 / 25 / $0.0261** | **20 / 6 / $0.0068** |
|
|
40
|
+
|
|
41
|
+
**Cost.** Adaptive cuts LLM verifier calls **76%** (25 → 6) and dollars **74%**
|
|
42
|
+
($0.0261 → $0.0068). On 3 of the 5 topics it spent **zero** LLM calls — every
|
|
43
|
+
unique survivor was on an authoritative host, so the $0 stages decided
|
|
44
|
+
everything.
|
|
45
|
+
|
|
46
|
+
## The honest reading: adaptive is a frontier POINT, not a free lunch
|
|
47
|
+
|
|
48
|
+
Admitted-source counts (lower = cleaner KB): **single 25, adaptive 20,
|
|
49
|
+
full-LLM 12**. Adaptive sits **between** the two:
|
|
50
|
+
|
|
51
|
+
- It removes **5 of the 13 sources** the full-LLM judge rejects that the
|
|
52
|
+
single-agent loop keeps (every one of them a real duplicate caught by the $0
|
|
53
|
+
dedup stage — exactly the de-dup-dominated win the cost/quality result
|
|
54
|
+
predicted).
|
|
55
|
+
- It does **NOT** match full-LLM cleanliness. The remaining 8 sources full-LLM
|
|
56
|
+
rejects, adaptive keeps. The cause is structural and visible in the trace: on
|
|
57
|
+
this topic set every non-duplicate survivor landed on an authoritative host
|
|
58
|
+
(arxiv / github / official docs), so the heuristic **kept** it without ever
|
|
59
|
+
asking the LLM — and the LLM, when full-LLM did ask it, judged several of
|
|
60
|
+
those same authoritative pages not-quite-on-topic and dropped them. The host
|
|
61
|
+
prior is coarser than the relevance judge.
|
|
62
|
+
|
|
63
|
+
So the frontier tradeoff is concrete: **adaptive recovers the deterministic
|
|
64
|
+
de-dup half of full-LLM's cleanliness for ~26% of full-LLM's dollars, and gives
|
|
65
|
+
up the relevance-judgment half.** Whether that is the right point depends on the
|
|
66
|
+
cost of a kept-but-marginal source. If a slightly-off-topic authoritative page
|
|
67
|
+
is cheap to carry, adaptive dominates. If every admitted source must clear a
|
|
68
|
+
relevance bar, the host heuristic is too permissive and you want the full LLM —
|
|
69
|
+
or a tightened heuristic.
|
|
70
|
+
|
|
71
|
+
## Where the heuristic is weak — stated plainly
|
|
72
|
+
|
|
73
|
+
The escalation count is the diagnostic. On 3 of 5 topics it was **zero**: the
|
|
74
|
+
heuristic never deferred to the LLM, so on those topics adaptive is a
|
|
75
|
+
**pure host/title/length rule**, and its cleanliness is exactly that rule's
|
|
76
|
+
cleanliness — no smarter than "trust arxiv/github, drop spam." That is fine when
|
|
77
|
+
the worker's sources are dominated by authoritative hosts (as here), but it
|
|
78
|
+
means the LLM's relevance judgment is contributing nothing on those topics, by
|
|
79
|
+
construction. The two topics where adaptive *did* escalate (grouped-query
|
|
80
|
+
attention, LoRA) are where some survivors were on unknown hosts — and there the
|
|
81
|
+
3 LLM calls per topic are the off-scope tail the verifier is actually for.
|
|
82
|
+
|
|
83
|
+
The heuristic would mis-route in two directions a richer worker would expose,
|
|
84
|
+
neither seen on this authoritative-host-heavy set:
|
|
85
|
+
|
|
86
|
+
- **False keep:** an authoritative-host page that is off-topic or low-value
|
|
87
|
+
(an arxiv paper on an unrelated subject) is kept without the LLM ever seeing
|
|
88
|
+
it. The host prior cannot catch this; only the relevance judge can.
|
|
89
|
+
- **False drop:** a genuinely good source on an unknown blog/host with a
|
|
90
|
+
spam-shaped title, or under the 400-char body floor, is dropped before the LLM
|
|
91
|
+
could rescue it.
|
|
92
|
+
|
|
93
|
+
## What this changes
|
|
94
|
+
|
|
95
|
+
The deployable recommendation from the cost/quality result was "deterministic
|
|
96
|
+
dedup first, reserve the LLM for the tail." This driver ships that and the
|
|
97
|
+
measurement confirms the **cost** half cleanly (76% fewer calls, 74% cheaper)
|
|
98
|
+
and qualifies the **quality** half honestly: adaptive captures the de-dup
|
|
99
|
+
cleanliness (the dominant, free win) but not the LLM's relevance cleanliness,
|
|
100
|
+
because the host heuristic resolves authoritative survivors without asking. For
|
|
101
|
+
a worker whose sources are mostly authoritative, adaptive is the right frontier
|
|
102
|
+
point. For one whose junk is on-topic-looking pages on unknown hosts, the
|
|
103
|
+
ambiguous tail grows and adaptive converges toward full-LLM cost — which is the
|
|
104
|
+
correct behavior: it pays for the LLM exactly when the cheap signals can't
|
|
105
|
+
decide.
|
|
106
|
+
|
|
107
|
+
## Run it
|
|
108
|
+
|
|
109
|
+
```
|
|
110
|
+
# offline (controlled wiring + escalates-only-ambiguous proof, no creds)
|
|
111
|
+
pnpm exec vitest run tests/loops/adaptive-ab.test.ts
|
|
112
|
+
|
|
113
|
+
# live three-topology frontier (needs TANGLE_API_KEY with glm-5.2 credits)
|
|
114
|
+
AGENT_KNOWLEDGE_LIVE=1 \
|
|
115
|
+
ADAPTIVE_LIVE_GOALS="self-speculative decoding|rotary position embeddings|grouped-query attention|KV-cache quantization|LoRA fine-tuning" \
|
|
116
|
+
pnpm exec vitest run tests/loops/adaptive-ab.test.ts -t "three-topology"
|
|
117
|
+
```
|
|
118
|
+
|
|
119
|
+
A cheap one-call glm-5.2 smoke gates the multi-topic burn (fails fast if the key
|
|
120
|
+
or the reasoning-token floor is broken) before any dollars are spent.
|
|
@@ -0,0 +1,97 @@
|
|
|
1
|
+
# Claim-grounding: the band where the verifier earns its dollar
|
|
2
|
+
|
|
3
|
+
`docs/results/cost-quality.md` found the relevance verifier's cleanliness win is
|
|
4
|
+
**dominated by de-duplication** — a deterministic content-hash captures most of it
|
|
5
|
+
at ~none of the LLM premium. So the open question was: is there an error band where
|
|
6
|
+
a verifier earns its cost — something a hash AND a relevance judge both miss?
|
|
7
|
+
|
|
8
|
+
**Yes: misattributed citations.** A source that is on-topic, unique, and real, but
|
|
9
|
+
whose cited CLAIM does not appear in the page (the LLM wrote a plausible sentence
|
|
10
|
+
and hung a real URL off it). De-dup passes it (it's unique). A relevance judge
|
|
11
|
+
passes it (the page is on-topic). Only checking the claim against the fetched text
|
|
12
|
+
catches it — and that check is **deterministic text presence, $0 inference**.
|
|
13
|
+
|
|
14
|
+
## The mode
|
|
15
|
+
|
|
16
|
+
Each proposed source now carries the specific claim it is cited for
|
|
17
|
+
(`withCitedClaim` → `metadata.citedClaim`). The verifier
|
|
18
|
+
(`createClaimGroundingVerifier`) runs `groundClaimInText(claim, pageText)` over the
|
|
19
|
+
`htmlToText` output of the page the worker actually fetched — verbatim, normalized
|
|
20
|
+
(punctuation/whitespace-insensitive), or a ≥70% content-word overlap close
|
|
21
|
+
paraphrase. A claim that isn't present is rejected as **misattributed**. The oracle
|
|
22
|
+
is text presence, not a model call, so it composes with the LLM relevance verifier
|
|
23
|
+
(reject off-topic AND misattributed) or runs alone at zero inference cost.
|
|
24
|
+
|
|
25
|
+
## Live A/B (glm-5.2, real web fetch, planted misattribution band)
|
|
26
|
+
|
|
27
|
+
Real worker (glm-5.2 query-gen → live `/v1/search` → `politeFetch` → `htmlToText`)
|
|
28
|
+
fetches the sources once per topic; we then plant ONE misattribution per topic (real
|
|
29
|
+
fetched page + a deliberately-wrong claim) and run three verifier arms over the SAME
|
|
30
|
+
proposals. Cost diffed per arm with the #36 `RouterClient.usage()` instrumentation.
|
|
31
|
+
|
|
32
|
+
| n=5 topics | misattributions caught | marginal $ | $/topic | per-$ caught |
|
|
33
|
+
|---|---|---|---|---|
|
|
34
|
+
| no-verifier | 0 / 5 | $0.0000 | — | — |
|
|
35
|
+
| relevance (LLM judge) | **4 / 5** | $0.0157 | ~$0.0031 | 254 |
|
|
36
|
+
| claim-grounding (text) | **5 / 5** | **$0.0000** | $0 | ∞ |
|
|
37
|
+
|
|
38
|
+
Per-topic (caught relevance / grounding): self-speculative decoding 1/1, rotary
|
|
39
|
+
position embeddings 1/1, grouped-query attention 1/1, **KV-cache quantization 0/1**,
|
|
40
|
+
LoRA 1/1. (An earlier 3-topic run missed self-speculative decoding instead — the
|
|
41
|
+
miss moves around; it is not a fixed topic.)
|
|
42
|
+
|
|
43
|
+
**Reading.** Claim-grounding catches every misattribution at **$0**; the relevance
|
|
44
|
+
judge catches most but **misses one in five at ~$0.003/topic**. The miss is the
|
|
45
|
+
point: the relevance verifier only ever sees the page text, never the cited claim,
|
|
46
|
+
so it is *structurally blind* to misattribution. It catches one only by accident —
|
|
47
|
+
when the fabricated claim happens to also make the page read off-topic
|
|
48
|
+
(e.g. a "12-billion-parameter draft transformer" claim on a rotary-embeddings page).
|
|
49
|
+
When the fabrication stays on-topic (the KV-cache case), the judge waves it through.
|
|
50
|
+
|
|
51
|
+
So on THIS band the verifier-per-dollar comparison inverts the cost/quality result:
|
|
52
|
+
there, the LLM verifier bought a dedup-shaped gain a free hash already captures —
|
|
53
|
+
expensive for what a cheap rule does. Here the cheap, deterministic check
|
|
54
|
+
**dominates** the expensive judge: it catches strictly more (5/5 vs 4/5) at strictly
|
|
55
|
+
less ($0 vs $0.0157). The verifier earns its dollar on misattribution; it does not on
|
|
56
|
+
de-duplication.
|
|
57
|
+
|
|
58
|
+
## Why this is a real correctable band (not dedup, not relevance)
|
|
59
|
+
|
|
60
|
+
- **Not de-duplication.** Every planted source has a unique URL and unique text; a
|
|
61
|
+
content-hash / canonical-URL dedup keeps all of them.
|
|
62
|
+
- **Not generic relevance.** Every planted source is on-topic; the relevance judge
|
|
63
|
+
(and the offline relevance stand-in) accept them. The error is in the *claim*, not
|
|
64
|
+
the *topic*.
|
|
65
|
+
- **Executable ground truth.** The check is presence/close-paraphrase of the claim
|
|
66
|
+
in the fetched text — deployable in production with no oracle and no model call.
|
|
67
|
+
|
|
68
|
+
The offline arm proves the floor with a controlled 4-source pool (2 grounded, 2
|
|
69
|
+
misattributed): claim-grounding admits **0/2** misattributions and keeps **2/2**
|
|
70
|
+
grounded sources, while relevance and no-verifier both admit **2/2**.
|
|
71
|
+
|
|
72
|
+
## Threats to validity
|
|
73
|
+
|
|
74
|
+
- **n=5 topics, 1 misattribution each.** The direction (grounding ≥ relevance caught,
|
|
75
|
+
at ≤ cost) is asserted in the test on every run; the magnitude is small-n. The
|
|
76
|
+
relevance miss-rate (1/5 here, 1/3 earlier) is an existence proof of the blind
|
|
77
|
+
spot, not a calibrated rate.
|
|
78
|
+
- **Planted misattributions, not naturally-occurring ones.** Like the cost/quality
|
|
79
|
+
offline floor, the misattribution is injected so the band is measurable. It models
|
|
80
|
+
the real LLM citation-fabrication failure but does not measure its base rate in the
|
|
81
|
+
wild — that needs a corpus of model-written citations checked by hand.
|
|
82
|
+
- **The grounding oracle is conservative.** A real paraphrase whose inflected words
|
|
83
|
+
differ from the page ("drafts" vs "draft") can score below 0.7 and be rejected —
|
|
84
|
+
a false-positive misattribution flag. `minOverlap` tunes this; the worker should
|
|
85
|
+
cite the page's own key terms (the `createClaimDecorator` extractor is told to).
|
|
86
|
+
|
|
87
|
+
## Run it
|
|
88
|
+
|
|
89
|
+
```bash
|
|
90
|
+
# offline floor (no creds)
|
|
91
|
+
pnpm exec vitest run tests/loops/claim-grounding-ab.test.ts -t "offline"
|
|
92
|
+
|
|
93
|
+
# live A/B (creds-gated). A cheap glm-5.2 smoke runs BEFORE the multi-topic burn.
|
|
94
|
+
AGENT_KNOWLEDGE_LIVE=1 TANGLE_API_KEY=… \
|
|
95
|
+
CLAIM_GROUNDING_LIVE_GOALS='self-speculative decoding|rotary position embeddings|grouped-query attention|KV-cache quantization|LoRA' \
|
|
96
|
+
pnpm exec vitest run tests/loops/claim-grounding-ab.test.ts -t "three verifier arms"
|
|
97
|
+
```
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
# Cost/quality: the two-agent loop's inference premium
|
|
2
|
+
|
|
3
|
+
Live 9-topic A/B (glm-5.2, budget B ≤ 4 passes/arm), measured per arm with the
|
|
4
|
+
router-client instrumentation. The original A/B reported only admitted-sources at
|
|
5
|
+
"equal passes" — which charged the two-agent verify step as one pass while it is
|
|
6
|
+
actually N `verifySource` LLM calls. Pricing the calls shows what that hid.
|
|
7
|
+
|
|
8
|
+
| per topic (mean) | two-agent | single-agent | ratio |
|
|
9
|
+
|---|---|---|---|
|
|
10
|
+
| LLM chat calls | 5.4 | 1.0 | ~5.4× |
|
|
11
|
+
| tokens (in+out) | ~4,900 | ~530 | ~9× |
|
|
12
|
+
| cost (USD) | ~$0.0072 | ~$0.0013 | ~5.5× |
|
|
13
|
+
| latency (wall) | ~37 s | ~11 s | ~3.4× |
|
|
14
|
+
| cleanliness Δ (single − two admitted) | — | — | +1.56, 95% CI [0.33, 2.67] |
|
|
15
|
+
|
|
16
|
+
Per-topic Δ (single − two admitted) this run: self-speculative decoding +4,
|
|
17
|
+
grouped-query attention 0, rotary position embeddings +1, KV-cache quantization
|
|
18
|
+
**−1**, LoRA +4, ring attention +2, constitutional AI +3, transformer +2, gradient
|
|
19
|
+
descent **−1**. Coverage 1.00 every topic, both arms.
|
|
20
|
+
|
|
21
|
+
**Reading.** The verifier buys ~1.5–2.7 fewer junk sources for roughly **5× the
|
|
22
|
+
dollars, 9× the tokens, and 3× the latency** — and on two topics it admitted *more*
|
|
23
|
+
than the single agent (the cleanliness signal is real but noisier than the +2.3 /
|
|
24
|
+
+2.7 of earlier runs). The cleanliness gain is dominated by de-duplication, so the
|
|
25
|
+
honest production move is a deterministic content-hash / canonical-URL dedup, which
|
|
26
|
+
captures most of the cleanliness at ~none of this premium; reserve an LLM check for
|
|
27
|
+
the off-scope tail. This is the cost half the "equal passes" framing left out.
|
|
@@ -0,0 +1,295 @@
|
|
|
1
|
+
# Supervisor prompt + agent-profile ideas
|
|
2
|
+
|
|
3
|
+
Four `AgentProfile` sketches for the research/knowledge domain — a **researcher worker**, a
|
|
4
|
+
**verifier driver**, a **research supervisor**, and a thin **dedup-first verifier** — plus a port
|
|
5
|
+
plan for the existing `~/code/supervisor-lab`.
|
|
6
|
+
|
|
7
|
+
This is a *design doc*, not shipped code. Each sketch names the real primitive it composes (in
|
|
8
|
+
`agent-knowledge` or `supervisor-lab`) so building it is "assemble", not "invent". Where a primitive
|
|
9
|
+
already exists, the sketch says so and points at it — we extend, we do not fork.
|
|
10
|
+
|
|
11
|
+
## What we borrowed from emilkowalski/skills
|
|
12
|
+
|
|
13
|
+
[`emilkowalski/skills`](https://github.com/emilkowalski/skills) is a small pack of Claude skills for
|
|
14
|
+
design engineering (`emil-design-eng`, `review-animations`). It is not about agents, but its *skill
|
|
15
|
+
shape* is worth copying, and one structural choice maps directly onto our verifier:
|
|
16
|
+
|
|
17
|
+
- **Tight frontmatter, two fields.** `name` + a one-paragraph `description` that says *when* to
|
|
18
|
+
invoke. Nothing else. Our profiles' `description` should read the same way: a trigger, not a
|
|
19
|
+
feature list.
|
|
20
|
+
- **Terse non-negotiable rules over prose.** `review-animations` is "The Ten Non-Negotiable
|
|
21
|
+
Standards" — numbered, imperative, absolute ("Animate `transform` and `opacity` only").
|
|
22
|
+
A worker system prompt is more legible as a short numbered contract than as paragraphs.
|
|
23
|
+
- **"Default to flagging; approval is earned."** The review skill is *adversarial by posture* — it
|
|
24
|
+
exists to reject, and its output is a findings table + a verdict. This is exactly the right posture
|
|
25
|
+
for a verifier, and it informs the dedup point below.
|
|
26
|
+
- **`disable-model-invocation: true` on the reviewer.** The review skill is not auto-invoked by the
|
|
27
|
+
model; it is routed in deliberately. Our verifier is the same: a driver *calls* it at a gate, the
|
|
28
|
+
worker never invokes it on a whim.
|
|
29
|
+
- **Cheap because narrow.** Each skill does one thing. The review skill carries no design *generation*
|
|
30
|
+
ability — it only judges. That separation (generate vs. judge) is the whole reason the judge can be
|
|
31
|
+
thin.
|
|
32
|
+
|
|
33
|
+
### The dedup insight (why the verifier is a thin profile, not a heavy one)
|
|
34
|
+
|
|
35
|
+
`review-animations` is mostly a deduplicated rule set: the craft knowledge lives in `emil-design-eng`,
|
|
36
|
+
and the reviewer is a thin lens that *applies the same rules* to a diff. It does not re-derive taste;
|
|
37
|
+
it checks against a list.
|
|
38
|
+
|
|
39
|
+
Our verifier is the same. In this repo the "rules" are already deterministic code:
|
|
40
|
+
|
|
41
|
+
- `createResearcherValidator` (`src/profiles/researcher.ts:235`) — citation-density floor + namespace
|
|
42
|
+
check, **no LLM**.
|
|
43
|
+
- the `lint` / `validate --strict` CLI path — citation, link, and schema checks over markdown, **no
|
|
44
|
+
LLM** (`docs/architecture.md`, "The CLI ... does not call an LLM").
|
|
45
|
+
- `assessAuthoredProfile` / `profileRichnessFinding` in supervisor-lab's bench — a deterministic
|
|
46
|
+
"is this profile a stub" gate.
|
|
47
|
+
|
|
48
|
+
So the verifier profile should be **thin and cheap**: it mostly *calls the deterministic checker and
|
|
49
|
+
reports the verdict*, escalating to an LLM judge **only** for the residual a deterministic check can't
|
|
50
|
+
cover (does the cited source actually support the claim — a semantic check). A heavy verifier that
|
|
51
|
+
re-reasons every claim from scratch is wasted spend: the dedup already happened in the validator.
|
|
52
|
+
**Spend the model budget on the worker (generation is hard); keep the verifier a cheap lens (judging
|
|
53
|
+
against a list is easy).**
|
|
54
|
+
|
|
55
|
+
---
|
|
56
|
+
|
|
57
|
+
## Sketch 1 — `grounded-researcher` (leaf worker)
|
|
58
|
+
|
|
59
|
+
**Status: already exists, twice.** `agent-knowledge` ships `researcherProfile()`
|
|
60
|
+
(`src/profiles/researcher.ts:130`) — `tools: { web_search, fs, shell }`, propose-don't-apply, with a
|
|
61
|
+
matching `createResearcherValidator`. `supervisor-lab` ships
|
|
62
|
+
`profiles/research/grounded-researcher.ts` (web-search MCP, search→cite→synthesize). This sketch is
|
|
63
|
+
the **canonical shape both converge on** — build new only if you need a variant; otherwise
|
|
64
|
+
`mergeAgentProfiles` over the existing one.
|
|
65
|
+
|
|
66
|
+
**When to use:** one self-contained research question, spawned one-per-sub-topic by a driver. The
|
|
67
|
+
leaf — it does not spawn.
|
|
68
|
+
|
|
69
|
+
**Tools / resources:** `web_search` (the `tcloud mcp` stdio server, or the `web_search: true` tool),
|
|
70
|
+
`fs` for writing the cited dossier, `shell` for `agent-knowledge index/lint`.
|
|
71
|
+
|
|
72
|
+
**System prompt (the contract):**
|
|
73
|
+
|
|
74
|
+
```
|
|
75
|
+
You are a grounded research WORKER. You answer ONE question, and every load-bearing
|
|
76
|
+
claim is traceable to a real source you actually retrieved.
|
|
77
|
+
|
|
78
|
+
THE LOOP — search → cite → synthesize:
|
|
79
|
+
1. SEARCH issue multiple focused web_search queries; never stop at the first hit.
|
|
80
|
+
2. CITE attach every load-bearing claim to its source (title + url/identifier).
|
|
81
|
+
A claim you cannot cite is a claim you do NOT make. Never invent a source.
|
|
82
|
+
3. SYNTH lead with the answer; note where sources agree, conflict, or leave a gap.
|
|
83
|
+
|
|
84
|
+
NON-NEGOTIABLE:
|
|
85
|
+
- Honest "the evidence is thin / conflicting" beats a confident hallucination.
|
|
86
|
+
- Citation density floor is enforced downstream — under-citing is a hard fail, not a warning.
|
|
87
|
+
- You are the LEAF. You do not spawn sub-workers.
|
|
88
|
+
|
|
89
|
+
Return your cited synthesis as the settled output (a structured object if a schema was passed).
|
|
90
|
+
```
|
|
91
|
+
|
|
92
|
+
**Why this shape:** the validator (`createResearcherValidator`) checks citation density and namespace
|
|
93
|
+
*deterministically*, so the prompt's job is only to make the worker *produce* citable output — the
|
|
94
|
+
checking is not the worker's job. That split is what keeps the verifier cheap (Sketch 4).
|
|
95
|
+
|
|
96
|
+
---
|
|
97
|
+
|
|
98
|
+
## Sketch 2 — `verifier-driver` (the gate, NOT the judge)
|
|
99
|
+
|
|
100
|
+
**Status: composes existing primitives.** This is `agent-runtime`'s `verify({ implement, verifier })`
|
|
101
|
+
combinator (`src/runtime/personify/combinators.ts:333`) wired to a research worker + a thin checker.
|
|
102
|
+
**Do not write a new verify-loop** — the 2-node implement→gate already exists.
|
|
103
|
+
|
|
104
|
+
**When to use:** when "did the worker actually deliver" must be proven before the result counts —
|
|
105
|
+
i.e. always, on a graded run. A driver that runs ONE worker behind ONE gate.
|
|
106
|
+
|
|
107
|
+
**Tools:** the coordination verbs (`spawn_agent`, `await_event`, `steer_agent`) over a `Scope`, plus
|
|
108
|
+
the ability to call the deterministic checker (`agent-knowledge lint` / `createResearcherValidator`).
|
|
109
|
+
|
|
110
|
+
**System prompt (the contract):**
|
|
111
|
+
|
|
112
|
+
```
|
|
113
|
+
You are a VERIFIER-DRIVER. You spawn ONE research worker, then you GATE its output —
|
|
114
|
+
you never accept the worker's own say-so.
|
|
115
|
+
|
|
116
|
+
THE GATE — generate → check → decide:
|
|
117
|
+
1. SPAWN author a grounded-researcher worker for the question and spawn it.
|
|
118
|
+
2. AWAIT pull its settled output (await_event). Read the REAL output, not a summary.
|
|
119
|
+
3. CHECK run the DETERMINISTIC checker first (citation density, namespace, lint,
|
|
120
|
+
schema). This is cheap and catches most failures. Only if it PASSES do you
|
|
121
|
+
escalate to the one semantic question a checker can't answer:
|
|
122
|
+
"does each cited source actually support the claim it's attached to?"
|
|
123
|
+
4. DECIDE PASS → settle. FAIL → steer_agent with the SPECIFIC gap named
|
|
124
|
+
("claim X cites source Y but Y does not say X"), or respawn a sharper worker.
|
|
125
|
+
|
|
126
|
+
NON-NEGOTIABLE:
|
|
127
|
+
- selector != judge. You GATE (pass/fail against a contract); you do not re-rank or
|
|
128
|
+
rewrite the worker's content yourself.
|
|
129
|
+
- "Settled a file" is not delivery. The deterministic check passing is delivery.
|
|
130
|
+
- Default to FAILING. A pass is earned by surviving the checker, not by looking plausible.
|
|
131
|
+
```
|
|
132
|
+
|
|
133
|
+
**Why this shape:** the driver's intelligence is *deciding what to check and how to steer on a
|
|
134
|
+
fail*, not re-doing the research. The expensive judgment (does the source support the claim) is gated
|
|
135
|
+
behind the cheap deterministic checker, so on most runs the LLM-judge step never fires.
|
|
136
|
+
|
|
137
|
+
---
|
|
138
|
+
|
|
139
|
+
## Sketch 3 — `research-supervisor` (fan-out → fuse → recurse)
|
|
140
|
+
|
|
141
|
+
**Status: exists as `research-then-build-driver` + `parallel-fanout-supervisor` in supervisor-lab.**
|
|
142
|
+
This sketch is the **research-specialized merge** of those two — build new only as a thin overlay via
|
|
143
|
+
`mergeAgentProfiles`; the orchestration body is identical.
|
|
144
|
+
|
|
145
|
+
**When to use:** a research question too broad for one worker — needs decomposition into disjoint
|
|
146
|
+
sub-topics, parallel grounded workers, and a fused cited synthesis. The top of a research tree.
|
|
147
|
+
|
|
148
|
+
**Tools / resources:** coordination MCP (`spawn_agent` / `await_event` / `steer_agent` /
|
|
149
|
+
`answer_question` / `stop`) over a `Scope`; `web_search` to *ground the decomposition itself*; skills
|
|
150
|
+
`dynamic-workflows`, `orchestrating-workers`, `authoring-agent-profiles` (all already in
|
|
151
|
+
`supervisor-lab/skills/`).
|
|
152
|
+
|
|
153
|
+
**System prompt (the contract):**
|
|
154
|
+
|
|
155
|
+
```
|
|
156
|
+
You are a research SUPERVISOR. You do NOT research the topic yourself — you ground the
|
|
157
|
+
decomposition, fan out grounded-researcher workers, gate them, and FUSE their cited findings.
|
|
158
|
+
|
|
159
|
+
THE SHAPE — ground → fan-out → gate → fan-in:
|
|
160
|
+
1. GROUND web_search FIRST to see what actually exists. Let the real landscape, not
|
|
161
|
+
your priors, define the sub-topics. Cite what you find.
|
|
162
|
+
2. DECOMPOSE split into DISJOINT sub-questions — non-overlapping ownership.
|
|
163
|
+
3. FAN OUT author a grounded-researcher profile per sub-question; spawn ALL of them in
|
|
164
|
+
ONE wave before awaiting any. Vary their angles. Never spawn-await-spawn.
|
|
165
|
+
4. GATE drain await_event. For each settled worker, run the deterministic checker
|
|
166
|
+
(Sketch 4). REJECT uncited or unsupported claims — a finding without a
|
|
167
|
+
source is not a finding. Steer or respawn on a fail.
|
|
168
|
+
5. FAN IN author a FINAL synthesis worker that fuses ONLY the gated findings into one
|
|
169
|
+
cited answer. Never paste raw worker dumps. Check each result for failure first.
|
|
170
|
+
6. RECURSE if a sub-question is itself large, spawn a SUB-supervisor (this profile +
|
|
171
|
+
the coordination MCP) to own it.
|
|
172
|
+
|
|
173
|
+
NON-NEGOTIABLE:
|
|
174
|
+
- Author EACH worker as a FULL profile (name, description, prompt, model, tools, skills).
|
|
175
|
+
A 2-sentence prompt with no tools is a stub, not a worker — the richness gate will flag it.
|
|
176
|
+
- Never go blind: keep pulling until every spawned worker is terminal.
|
|
177
|
+
- Settle only on gated, cited output. Budget is conserved and depth is bounded by the Scope.
|
|
178
|
+
```
|
|
179
|
+
|
|
180
|
+
**Why this shape:** the supervisor's only real lever is *the quality of the profiles it authors* —
|
|
181
|
+
that is the capability `supervisor-lab` measures (`thinProfileRatio`,
|
|
182
|
+
`bench/supervise-topology.ts`). The gate (step 4) reuses the cheap verifier so the supervisor doesn't
|
|
183
|
+
burn its budget re-judging.
|
|
184
|
+
|
|
185
|
+
---
|
|
186
|
+
|
|
187
|
+
## Sketch 4 — `dedup-verifier` (thin, cheap, mostly deterministic)
|
|
188
|
+
|
|
189
|
+
**This is the emilkowalski dedup lesson made into a profile.** It is deliberately *thin* — the
|
|
190
|
+
"taste" lives in the deterministic validator, the profile is just the lens that applies it and
|
|
191
|
+
escalates only the residual.
|
|
192
|
+
|
|
193
|
+
**When to use:** called by a driver/supervisor at a gate (Sketch 2 step 3, Sketch 3 step 4). Never
|
|
194
|
+
auto-invoked — the verifier equivalent of `disable-model-invocation: true`.
|
|
195
|
+
|
|
196
|
+
**Tools:** `shell` (to run `agent-knowledge lint` / `validate --strict`), the
|
|
197
|
+
`createResearcherValidator` call, and an LLM *only* for the one semantic check. **No `web_search`, no
|
|
198
|
+
`fs` write, no generation tools** — a verifier that can edit content is a verifier that can cheat.
|
|
199
|
+
|
|
200
|
+
**System prompt (the whole thing — it's short on purpose):**
|
|
201
|
+
|
|
202
|
+
```
|
|
203
|
+
You are a VERIFIER. You judge ONE research output against a fixed contract. You default
|
|
204
|
+
to FAILING; a pass is earned. You never edit, research, or rewrite — you only judge.
|
|
205
|
+
|
|
206
|
+
ORDER (cheapest check first, stop at the first hard fail):
|
|
207
|
+
1. DETERMINISTIC run the validator/lint: citation density >= floor, namespace match,
|
|
208
|
+
links resolve, schema valid. If any fails → FAIL with the rule name.
|
|
209
|
+
~90% of failures die here, at zero LLM cost.
|
|
210
|
+
2. SEMANTIC ONLY if (1) passes: for each cited claim, does the cited source
|
|
211
|
+
actually support it? This is the one thing a checker can't do. Be terse.
|
|
212
|
+
|
|
213
|
+
OUTPUT: a findings table (claim | source | supported? | why) + a verdict (PASS / FAIL),
|
|
214
|
+
grouped by severity. No prose. A violation is a finding.
|
|
215
|
+
|
|
216
|
+
You carry NO generation ability by design. If you find yourself wanting to fix the
|
|
217
|
+
output, STOP — that is the worker's job, not yours.
|
|
218
|
+
```
|
|
219
|
+
|
|
220
|
+
**Why thin beats heavy here:** the dedup already happened in `createResearcherValidator` and the lint
|
|
221
|
+
path. A heavy verifier that re-reasons every claim with an LLM pays full model cost to re-derive a
|
|
222
|
+
check the validator does for free, and (worse) a verifier with generation tools can drift into doing
|
|
223
|
+
the work and grading itself. Thin + deterministic-first is both cheaper *and* harder to game. Measured
|
|
224
|
+
claim to test when built: on a real run, the LLM (step 2) should fire on **< ~10–20% of gated outputs**
|
|
225
|
+
because the deterministic check (step 1) absorbs the rest — if it fires on most, the validator floor
|
|
226
|
+
is mis-set, not the verifier.
|
|
227
|
+
|
|
228
|
+
---
|
|
229
|
+
|
|
230
|
+
## supervisor-lab — status and port plan
|
|
231
|
+
|
|
232
|
+
**It exists.** `~/code/supervisor-lab` is a real, non-trivial repo
|
|
233
|
+
(`github.com/tangle-network/supervisor-lab`), not a stub. `ls ~/code | grep -i supervisor` → it's
|
|
234
|
+
there.
|
|
235
|
+
|
|
236
|
+
It already is what one might propose to "create": the product/experiment layer for supervisor agents
|
|
237
|
+
over the `agent-runtime` substrate, dependency-one-way (`supervisor-lab → agent-runtime →
|
|
238
|
+
agent-eval`). It ships:
|
|
239
|
+
|
|
240
|
+
- **the two-agent loop, already ported** — `bench/supervise-topology.ts` is exactly the "two-agent
|
|
241
|
+
loop": a supervisor mounts the coordination MCP over a live `Scope`/`Supervisor`
|
|
242
|
+
(`createSupervisor`, `serveCoordinationMcp`), authors worker `AgentProfile`s as code, spawns them,
|
|
243
|
+
drains the bus (`await_event`), steers (`steer_agent`), and grades on a real judge. It has an
|
|
244
|
+
**OFFLINE $0 scripted path** and a LIVE cli-bridge path. This is the loop to plug these sketches
|
|
245
|
+
into — it does not need to be built.
|
|
246
|
+
- **profile archetypes** — `profiles/research/{grounded-researcher,research-then-build-driver}.ts`,
|
|
247
|
+
`profiles/engineering/{parallel-fanout-supervisor,long-horizon-plan-driver,hardware-auditor}.ts`.
|
|
248
|
+
Sketches 1 and 3 above are *already these files*.
|
|
249
|
+
- **a skill layer** — `skills/{authoring-agent-profiles,orchestrating-workers,dynamic-workflows,
|
|
250
|
+
spawning-research-loops}/SKILL.md`, each in the exact emilkowalski frontmatter+rules shape.
|
|
251
|
+
- **a catalog + ingest pipeline** — `src/catalog/`, `src/ingest/skills.ts` ingests vendored
|
|
252
|
+
Claude-Code-style skill packs (the same shape as `emilkowalski/skills`) by convention.
|
|
253
|
+
- **the richness gate** — `assessAuthoredProfile` / `profileRichnessFinding` / `thinProfileRatio`,
|
|
254
|
+
the deterministic "is this a stub" check that is the dedup-verifier's first line.
|
|
255
|
+
|
|
256
|
+
### How to port the two-agent loop (it's already there — this is how to extend it)
|
|
257
|
+
|
|
258
|
+
Because the loop already exists, "porting" means **adding these four profiles + the thin-verifier gate
|
|
259
|
+
into the existing harness**, not rebuilding it:
|
|
260
|
+
|
|
261
|
+
1. **Land the profiles as catalog files.** Sketches 1 and 3 are already
|
|
262
|
+
`profiles/research/grounded-researcher.ts` and `research-then-build-driver.ts`. Sketch 2
|
|
263
|
+
(`verifier-driver`) and Sketch 4 (`dedup-verifier`) are new files under
|
|
264
|
+
`profiles/research/` — author them as `defineAgentProfile` entries, resolve their skills via the
|
|
265
|
+
`skills(...)` helper in `profiles/_shared.ts` (fails closed on an unknown skill).
|
|
266
|
+
|
|
267
|
+
2. **Wire the dedup-verifier into the gate.** In `bench/supervise-topology.ts`, the worker's
|
|
268
|
+
`runWorker` already grades on `adapter.judge`. For research tasks, replace/augment that with the
|
|
269
|
+
deterministic `createResearcherValidator` + `agent-knowledge lint` path FIRST, and only escalate to
|
|
270
|
+
the LLM semantic check on a deterministic pass. This is the dedup point made executable: the
|
|
271
|
+
existing `assessAuthoredProfile` richness gate is the *profile* check; the validator is the
|
|
272
|
+
*output* check. Both run before any LLM judge.
|
|
273
|
+
|
|
274
|
+
3. **Add a research bench arm.** `supervise-topology.ts` auto-detects domain (repo vs. answer/text).
|
|
275
|
+
A research arm is an "answer/text" task whose `adapter.judge` is the cited-output validator. Add
|
|
276
|
+
`BENCH=research` that loads a research question + namespace and grades via
|
|
277
|
+
`createResearcherValidator`. The OFFLINE scripted path already proves the harness at $0; extend
|
|
278
|
+
`scriptedRun.workerResult` to emit a cited/uncited synthesis so the offline smoke exercises the
|
|
279
|
+
verifier without spend.
|
|
280
|
+
|
|
281
|
+
4. **Run the existing experiment knob.** The whole point of the lab is the one knob: catalog/skills
|
|
282
|
+
ON vs. OFF (`CATALOG=1`). With these profiles + the thin verifier in the catalog, run
|
|
283
|
+
capability-aware vs. baseline on a hard research task and read `thinProfileRatio` + delivery rate —
|
|
284
|
+
does giving the supervisor the research archetypes + the cheap gate produce better-composed teams
|
|
285
|
+
and more *cited, gated* deliveries.
|
|
286
|
+
|
|
287
|
+
### Cross-repo note
|
|
288
|
+
|
|
289
|
+
`agent-knowledge` already owns the deterministic research checker (`createResearcherValidator`, the
|
|
290
|
+
`lint`/`validate` CLI) and the researcher profile (`researcherProfile`,
|
|
291
|
+
`multiHarnessResearcherFanout`). The thin verifier should **call those**, not reimplement them —
|
|
292
|
+
`supervisor-lab` depends on `agent-runtime`, and the research checks live in `agent-knowledge`, so the
|
|
293
|
+
research bench arm pulls `@tangle-network/agent-knowledge` for the verifier's deterministic line. That
|
|
294
|
+
keeps the dedup in one place: the validator is authored once, the verifier profile is a thin lens over
|
|
295
|
+
it.
|