@tangle-network/agent-knowledge 1.8.0 → 1.10.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -1,4 +1,4 @@
1
- import { S as SourceRecord } from '../types-BEDGlXB-.js';
1
+ import { S as SourceRecord } from '../types-C17sAROL.js';
2
2
  import { z } from 'zod';
3
3
 
4
4
  type AgentMemoryKind = 'message' | 'entity' | 'fact' | 'preference' | 'observation' | 'reasoning-trace';
@@ -170,4 +170,4 @@ interface KnowledgeRelease {
170
170
  metadata?: Record<string, unknown>;
171
171
  }
172
172
 
173
- export type { ClaimRef as C, KnowledgeGraphEdge as K, SourceRecord as S, KnowledgeGraphNode as a, KnowledgeGraph as b, KnowledgeIndex as c, KnowledgeSearchResult as d, KnowledgeEventType as e, KnowledgeEvent as f, KnowledgePage as g, KnowledgeLintFinding as h, KnowledgeWriteBlock as i, KnowledgeClaim as j, KnowledgeRelease as k, SourceRegistry as l, KnowledgeWriteParseResult as m, KnowledgeBaseCandidate as n, KnowledgeId as o, KnowledgePolicy as p, KnowledgeRelation as q, KnowledgeUnit as r, SourceAnchor as s };
173
+ export type { ClaimRef as C, KnowledgeGraphEdge as K, SourceRecord as S, KnowledgeGraphNode as a, KnowledgeGraph as b, KnowledgeIndex as c, KnowledgeSearchResult as d, SourceRegistry as e, KnowledgeEvent as f, KnowledgeEventType as g, KnowledgePage as h, KnowledgeLintFinding as i, KnowledgeWriteBlock as j, KnowledgeClaim as k, KnowledgeRelease as l, KnowledgeWriteParseResult as m, KnowledgeBaseCandidate as n, KnowledgeId as o, KnowledgePolicy as p, KnowledgeRelation as q, KnowledgeUnit as r, SourceAnchor as s };
@@ -1,4 +1,4 @@
1
- import { K as KnowledgeGraphEdge, a as KnowledgeGraphNode, b as KnowledgeGraph } from '../types-BEDGlXB-.js';
1
+ import { K as KnowledgeGraphEdge, a as KnowledgeGraphNode, b as KnowledgeGraph } from '../types-C17sAROL.js';
2
2
 
3
3
  interface KnowledgeVizNode extends KnowledgeGraphNode {
4
4
  degree: number;
@@ -0,0 +1,120 @@
1
+ # Adaptive topology: spend the LLM verifier only when it pays
2
+
3
+ The cost/quality A/B (`docs/results/cost-quality.md`) found the LLM relevance
4
+ verifier's cleanliness win is dominated by **de-duplication** — which a
5
+ deterministic content-hash / canonical-URL check captures at ~none of the LLM
6
+ premium — and that an LLM check only earns its dollar on the off-scope tail. The
7
+ production move it named was: do the cheap deterministic work first, reserve the
8
+ LLM for the ambiguous tail. `createAdaptiveResearchDriver`
9
+ (`src/adaptive-driver.ts`) is that driver, and this is its measurement.
10
+
11
+ Per candidate source the adaptive driver runs three stages, cheapest first,
12
+ stopping at the first that decides:
13
+
14
+ 1. **Dedup ($0).** Reject a source whose canonical URL (scheme/`www`/trailing
15
+ slash/tracking params stripped) or normalized-text content hash matches one
16
+ already accepted this round or in the KB.
17
+ 2. **Heuristic triage ($0).** Classify a unique survivor with host/title/length
18
+ signals only: an authoritative host (arxiv, `*.edu`, `*.gov`, official docs,
19
+ github, …) with a substantial body is **kept**; an obvious spam/listicle
20
+ title or a too-thin body is **dropped**; everything else is **ambiguous**.
21
+ 3. **LLM escalation ($).** Only ambiguous survivors reach the shipped LLM
22
+ relevance verifier (`createVerifyingResearchDriver`) — one call each.
23
+
24
+ ## Live frontier, n=5 topics (glm-5.2)
25
+
26
+ Real web-research worker fetches each topic once; the same fetched proposals
27
+ (plus one planted tracking-decorated mirror of the first source, so the dedup
28
+ stage has a real duplicate to catch) are gated through all three drivers. Cost
29
+ is the per-arm `RouterClient.usage()` diff (#36). Total spend for the run:
30
+ **$0.033**.
31
+
32
+ | topic | fetched | single admit | full-LLM admit / calls / $ | adaptive admit / LLM calls / $ |
33
+ |---|---|---|---|---|
34
+ | self-speculative decoding | 3 | 3 | 1 / 3 / $0.0027 | 2 / **0** / **$0.0000** |
35
+ | rotary position embeddings | 3 | 3 | 1 / 3 / $0.0031 | 2 / **0** / **$0.0000** |
36
+ | grouped-query attention | 7 | 7 | 3 / 7 / $0.0072 | 6 / 3 / $0.0030 |
37
+ | KV-cache quantization | 5 | 5 | 3 / 5 / $0.0052 | 4 / **0** / **$0.0000** |
38
+ | LoRA fine-tuning | 7 | 7 | 4 / 7 / $0.0079 | 6 / 3 / $0.0037 |
39
+ | **total** | **25** | **25** | **12 / 25 / $0.0261** | **20 / 6 / $0.0068** |
40
+
41
+ **Cost.** Adaptive cuts LLM verifier calls **76%** (25 → 6) and dollars **74%**
42
+ ($0.0261 → $0.0068). On 3 of the 5 topics it spent **zero** LLM calls — every
43
+ unique survivor was on an authoritative host, so the $0 stages decided
44
+ everything.
45
+
46
+ ## The honest reading: adaptive is a frontier POINT, not a free lunch
47
+
48
+ Admitted-source counts (lower = cleaner KB): **single 25, adaptive 20,
49
+ full-LLM 12**. Adaptive sits **between** the two:
50
+
51
+ - It removes **5 of the 13 sources** the full-LLM judge rejects that the
52
+ single-agent loop keeps (every one of them a real duplicate caught by the $0
53
+ dedup stage — exactly the de-dup-dominated win the cost/quality result
54
+ predicted).
55
+ - It does **NOT** match full-LLM cleanliness. The remaining 8 sources full-LLM
56
+ rejects, adaptive keeps. The cause is structural and visible in the trace: on
57
+ this topic set every non-duplicate survivor landed on an authoritative host
58
+ (arxiv / github / official docs), so the heuristic **kept** it without ever
59
+ asking the LLM — and the LLM, when full-LLM did ask it, judged several of
60
+ those same authoritative pages not-quite-on-topic and dropped them. The host
61
+ prior is coarser than the relevance judge.
62
+
63
+ So the frontier tradeoff is concrete: **adaptive recovers the deterministic
64
+ de-dup half of full-LLM's cleanliness for ~26% of full-LLM's dollars, and gives
65
+ up the relevance-judgment half.** Whether that is the right point depends on the
66
+ cost of a kept-but-marginal source. If a slightly-off-topic authoritative page
67
+ is cheap to carry, adaptive dominates. If every admitted source must clear a
68
+ relevance bar, the host heuristic is too permissive and you want the full LLM —
69
+ or a tightened heuristic.
70
+
71
+ ## Where the heuristic is weak — stated plainly
72
+
73
+ The escalation count is the diagnostic. On 3 of 5 topics it was **zero**: the
74
+ heuristic never deferred to the LLM, so on those topics adaptive is a
75
+ **pure host/title/length rule**, and its cleanliness is exactly that rule's
76
+ cleanliness — no smarter than "trust arxiv/github, drop spam." That is fine when
77
+ the worker's sources are dominated by authoritative hosts (as here), but it
78
+ means the LLM's relevance judgment is contributing nothing on those topics, by
79
+ construction. The two topics where adaptive *did* escalate (grouped-query
80
+ attention, LoRA) are where some survivors were on unknown hosts — and there the
81
+ 3 LLM calls per topic are the off-scope tail the verifier is actually for.
82
+
83
+ The heuristic would mis-route in two directions a richer worker would expose,
84
+ neither seen on this authoritative-host-heavy set:
85
+
86
+ - **False keep:** an authoritative-host page that is off-topic or low-value
87
+ (an arxiv paper on an unrelated subject) is kept without the LLM ever seeing
88
+ it. The host prior cannot catch this; only the relevance judge can.
89
+ - **False drop:** a genuinely good source on an unknown blog/host with a
90
+ spam-shaped title, or under the 400-char body floor, is dropped before the LLM
91
+ could rescue it.
92
+
93
+ ## What this changes
94
+
95
+ The deployable recommendation from the cost/quality result was "deterministic
96
+ dedup first, reserve the LLM for the tail." This driver ships that and the
97
+ measurement confirms the **cost** half cleanly (76% fewer calls, 74% cheaper)
98
+ and qualifies the **quality** half honestly: adaptive captures the de-dup
99
+ cleanliness (the dominant, free win) but not the LLM's relevance cleanliness,
100
+ because the host heuristic resolves authoritative survivors without asking. For
101
+ a worker whose sources are mostly authoritative, adaptive is the right frontier
102
+ point. For one whose junk is on-topic-looking pages on unknown hosts, the
103
+ ambiguous tail grows and adaptive converges toward full-LLM cost — which is the
104
+ correct behavior: it pays for the LLM exactly when the cheap signals can't
105
+ decide.
106
+
107
+ ## Run it
108
+
109
+ ```
110
+ # offline (controlled wiring + escalates-only-ambiguous proof, no creds)
111
+ pnpm exec vitest run tests/loops/adaptive-ab.test.ts
112
+
113
+ # live three-topology frontier (needs TANGLE_API_KEY with glm-5.2 credits)
114
+ AGENT_KNOWLEDGE_LIVE=1 \
115
+ ADAPTIVE_LIVE_GOALS="self-speculative decoding|rotary position embeddings|grouped-query attention|KV-cache quantization|LoRA fine-tuning" \
116
+ pnpm exec vitest run tests/loops/adaptive-ab.test.ts -t "three-topology"
117
+ ```
118
+
119
+ A cheap one-call glm-5.2 smoke gates the multi-topic burn (fails fast if the key
120
+ or the reasoning-token floor is broken) before any dollars are spent.
@@ -0,0 +1,97 @@
1
+ # Claim-grounding: the band where the verifier earns its dollar
2
+
3
+ `docs/results/cost-quality.md` found the relevance verifier's cleanliness win is
4
+ **dominated by de-duplication** — a deterministic content-hash captures most of it
5
+ at ~none of the LLM premium. So the open question was: is there an error band where
6
+ a verifier earns its cost — something a hash AND a relevance judge both miss?
7
+
8
+ **Yes: misattributed citations.** A source that is on-topic, unique, and real, but
9
+ whose cited CLAIM does not appear in the page (the LLM wrote a plausible sentence
10
+ and hung a real URL off it). De-dup passes it (it's unique). A relevance judge
11
+ passes it (the page is on-topic). Only checking the claim against the fetched text
12
+ catches it — and that check is **deterministic text presence, $0 inference**.
13
+
14
+ ## The mode
15
+
16
+ Each proposed source now carries the specific claim it is cited for
17
+ (`withCitedClaim` → `metadata.citedClaim`). The verifier
18
+ (`createClaimGroundingVerifier`) runs `groundClaimInText(claim, pageText)` over the
19
+ `htmlToText` output of the page the worker actually fetched — verbatim, normalized
20
+ (punctuation/whitespace-insensitive), or a ≥70% content-word overlap close
21
+ paraphrase. A claim that isn't present is rejected as **misattributed**. The oracle
22
+ is text presence, not a model call, so it composes with the LLM relevance verifier
23
+ (reject off-topic AND misattributed) or runs alone at zero inference cost.
24
+
25
+ ## Live A/B (glm-5.2, real web fetch, planted misattribution band)
26
+
27
+ Real worker (glm-5.2 query-gen → live `/v1/search` → `politeFetch` → `htmlToText`)
28
+ fetches the sources once per topic; we then plant ONE misattribution per topic (real
29
+ fetched page + a deliberately-wrong claim) and run three verifier arms over the SAME
30
+ proposals. Cost diffed per arm with the #36 `RouterClient.usage()` instrumentation.
31
+
32
+ | n=5 topics | misattributions caught | marginal $ | $/topic | per-$ caught |
33
+ |---|---|---|---|---|
34
+ | no-verifier | 0 / 5 | $0.0000 | — | — |
35
+ | relevance (LLM judge) | **4 / 5** | $0.0157 | ~$0.0031 | 254 |
36
+ | claim-grounding (text) | **5 / 5** | **$0.0000** | $0 | ∞ |
37
+
38
+ Per-topic (caught relevance / grounding): self-speculative decoding 1/1, rotary
39
+ position embeddings 1/1, grouped-query attention 1/1, **KV-cache quantization 0/1**,
40
+ LoRA 1/1. (An earlier 3-topic run missed self-speculative decoding instead — the
41
+ miss moves around; it is not a fixed topic.)
42
+
43
+ **Reading.** Claim-grounding catches every misattribution at **$0**; the relevance
44
+ judge catches most but **misses one in five at ~$0.003/topic**. The miss is the
45
+ point: the relevance verifier only ever sees the page text, never the cited claim,
46
+ so it is *structurally blind* to misattribution. It catches one only by accident —
47
+ when the fabricated claim happens to also make the page read off-topic
48
+ (e.g. a "12-billion-parameter draft transformer" claim on a rotary-embeddings page).
49
+ When the fabrication stays on-topic (the KV-cache case), the judge waves it through.
50
+
51
+ So on THIS band the verifier-per-dollar comparison inverts the cost/quality result:
52
+ there, the LLM verifier bought a dedup-shaped gain a free hash already captures —
53
+ expensive for what a cheap rule does. Here the cheap, deterministic check
54
+ **dominates** the expensive judge: it catches strictly more (5/5 vs 4/5) at strictly
55
+ less ($0 vs $0.0157). The verifier earns its dollar on misattribution; it does not on
56
+ de-duplication.
57
+
58
+ ## Why this is a real correctable band (not dedup, not relevance)
59
+
60
+ - **Not de-duplication.** Every planted source has a unique URL and unique text; a
61
+ content-hash / canonical-URL dedup keeps all of them.
62
+ - **Not generic relevance.** Every planted source is on-topic; the relevance judge
63
+ (and the offline relevance stand-in) accept them. The error is in the *claim*, not
64
+ the *topic*.
65
+ - **Executable ground truth.** The check is presence/close-paraphrase of the claim
66
+ in the fetched text — deployable in production with no oracle and no model call.
67
+
68
+ The offline arm proves the floor with a controlled 4-source pool (2 grounded, 2
69
+ misattributed): claim-grounding admits **0/2** misattributions and keeps **2/2**
70
+ grounded sources, while relevance and no-verifier both admit **2/2**.
71
+
72
+ ## Threats to validity
73
+
74
+ - **n=5 topics, 1 misattribution each.** The direction (grounding ≥ relevance caught,
75
+ at ≤ cost) is asserted in the test on every run; the magnitude is small-n. The
76
+ relevance miss-rate (1/5 here, 1/3 earlier) is an existence proof of the blind
77
+ spot, not a calibrated rate.
78
+ - **Planted misattributions, not naturally-occurring ones.** Like the cost/quality
79
+ offline floor, the misattribution is injected so the band is measurable. It models
80
+ the real LLM citation-fabrication failure but does not measure its base rate in the
81
+ wild — that needs a corpus of model-written citations checked by hand.
82
+ - **The grounding oracle is conservative.** A real paraphrase whose inflected words
83
+ differ from the page ("drafts" vs "draft") can score below 0.7 and be rejected —
84
+ a false-positive misattribution flag. `minOverlap` tunes this; the worker should
85
+ cite the page's own key terms (the `createClaimDecorator` extractor is told to).
86
+
87
+ ## Run it
88
+
89
+ ```bash
90
+ # offline floor (no creds)
91
+ pnpm exec vitest run tests/loops/claim-grounding-ab.test.ts -t "offline"
92
+
93
+ # live A/B (creds-gated). A cheap glm-5.2 smoke runs BEFORE the multi-topic burn.
94
+ AGENT_KNOWLEDGE_LIVE=1 TANGLE_API_KEY=… \
95
+ CLAIM_GROUNDING_LIVE_GOALS='self-speculative decoding|rotary position embeddings|grouped-query attention|KV-cache quantization|LoRA' \
96
+ pnpm exec vitest run tests/loops/claim-grounding-ab.test.ts -t "three verifier arms"
97
+ ```
@@ -0,0 +1,27 @@
1
+ # Cost/quality: the two-agent loop's inference premium
2
+
3
+ Live 9-topic A/B (glm-5.2, budget B ≤ 4 passes/arm), measured per arm with the
4
+ router-client instrumentation. The original A/B reported only admitted-sources at
5
+ "equal passes" — which charged the two-agent verify step as one pass while it is
6
+ actually N `verifySource` LLM calls. Pricing the calls shows what that hid.
7
+
8
+ | per topic (mean) | two-agent | single-agent | ratio |
9
+ |---|---|---|---|
10
+ | LLM chat calls | 5.4 | 1.0 | ~5.4× |
11
+ | tokens (in+out) | ~4,900 | ~530 | ~9× |
12
+ | cost (USD) | ~$0.0072 | ~$0.0013 | ~5.5× |
13
+ | latency (wall) | ~37 s | ~11 s | ~3.4× |
14
+ | cleanliness Δ (single − two admitted) | — | — | +1.56, 95% CI [0.33, 2.67] |
15
+
16
+ Per-topic Δ (single − two admitted) this run: self-speculative decoding +4,
17
+ grouped-query attention 0, rotary position embeddings +1, KV-cache quantization
18
+ **−1**, LoRA +4, ring attention +2, constitutional AI +3, transformer +2, gradient
19
+ descent **−1**. Coverage 1.00 every topic, both arms.
20
+
21
+ **Reading.** The verifier buys ~1.5–2.7 fewer junk sources for roughly **5× the
22
+ dollars, 9× the tokens, and 3× the latency** — and on two topics it admitted *more*
23
+ than the single agent (the cleanliness signal is real but noisier than the +2.3 /
24
+ +2.7 of earlier runs). The cleanliness gain is dominated by de-duplication, so the
25
+ honest production move is a deterministic content-hash / canonical-URL dedup, which
26
+ captures most of the cleanliness at ~none of this premium; reserve an LLM check for
27
+ the off-scope tail. This is the cost half the "equal passes" framing left out.
@@ -0,0 +1,295 @@
1
+ # Supervisor prompt + agent-profile ideas
2
+
3
+ Four `AgentProfile` sketches for the research/knowledge domain — a **researcher worker**, a
4
+ **verifier driver**, a **research supervisor**, and a thin **dedup-first verifier** — plus a port
5
+ plan for the existing `~/code/supervisor-lab`.
6
+
7
+ This is a *design doc*, not shipped code. Each sketch names the real primitive it composes (in
8
+ `agent-knowledge` or `supervisor-lab`) so building it is "assemble", not "invent". Where a primitive
9
+ already exists, the sketch says so and points at it — we extend, we do not fork.
10
+
11
+ ## What we borrowed from emilkowalski/skills
12
+
13
+ [`emilkowalski/skills`](https://github.com/emilkowalski/skills) is a small pack of Claude skills for
14
+ design engineering (`emil-design-eng`, `review-animations`). It is not about agents, but its *skill
15
+ shape* is worth copying, and one structural choice maps directly onto our verifier:
16
+
17
+ - **Tight frontmatter, two fields.** `name` + a one-paragraph `description` that says *when* to
18
+ invoke. Nothing else. Our profiles' `description` should read the same way: a trigger, not a
19
+ feature list.
20
+ - **Terse non-negotiable rules over prose.** `review-animations` is "The Ten Non-Negotiable
21
+ Standards" — numbered, imperative, absolute ("Animate `transform` and `opacity` only").
22
+ A worker system prompt is more legible as a short numbered contract than as paragraphs.
23
+ - **"Default to flagging; approval is earned."** The review skill is *adversarial by posture* — it
24
+ exists to reject, and its output is a findings table + a verdict. This is exactly the right posture
25
+ for a verifier, and it informs the dedup point below.
26
+ - **`disable-model-invocation: true` on the reviewer.** The review skill is not auto-invoked by the
27
+ model; it is routed in deliberately. Our verifier is the same: a driver *calls* it at a gate, the
28
+ worker never invokes it on a whim.
29
+ - **Cheap because narrow.** Each skill does one thing. The review skill carries no design *generation*
30
+ ability — it only judges. That separation (generate vs. judge) is the whole reason the judge can be
31
+ thin.
32
+
33
+ ### The dedup insight (why the verifier is a thin profile, not a heavy one)
34
+
35
+ `review-animations` is mostly a deduplicated rule set: the craft knowledge lives in `emil-design-eng`,
36
+ and the reviewer is a thin lens that *applies the same rules* to a diff. It does not re-derive taste;
37
+ it checks against a list.
38
+
39
+ Our verifier is the same. In this repo the "rules" are already deterministic code:
40
+
41
+ - `createResearcherValidator` (`src/profiles/researcher.ts:235`) — citation-density floor + namespace
42
+ check, **no LLM**.
43
+ - the `lint` / `validate --strict` CLI path — citation, link, and schema checks over markdown, **no
44
+ LLM** (`docs/architecture.md`, "The CLI ... does not call an LLM").
45
+ - `assessAuthoredProfile` / `profileRichnessFinding` in supervisor-lab's bench — a deterministic
46
+ "is this profile a stub" gate.
47
+
48
+ So the verifier profile should be **thin and cheap**: it mostly *calls the deterministic checker and
49
+ reports the verdict*, escalating to an LLM judge **only** for the residual a deterministic check can't
50
+ cover (does the cited source actually support the claim — a semantic check). A heavy verifier that
51
+ re-reasons every claim from scratch is wasted spend: the dedup already happened in the validator.
52
+ **Spend the model budget on the worker (generation is hard); keep the verifier a cheap lens (judging
53
+ against a list is easy).**
54
+
55
+ ---
56
+
57
+ ## Sketch 1 — `grounded-researcher` (leaf worker)
58
+
59
+ **Status: already exists, twice.** `agent-knowledge` ships `researcherProfile()`
60
+ (`src/profiles/researcher.ts:130`) — `tools: { web_search, fs, shell }`, propose-don't-apply, with a
61
+ matching `createResearcherValidator`. `supervisor-lab` ships
62
+ `profiles/research/grounded-researcher.ts` (web-search MCP, search→cite→synthesize). This sketch is
63
+ the **canonical shape both converge on** — build new only if you need a variant; otherwise
64
+ `mergeAgentProfiles` over the existing one.
65
+
66
+ **When to use:** one self-contained research question, spawned one-per-sub-topic by a driver. The
67
+ leaf — it does not spawn.
68
+
69
+ **Tools / resources:** `web_search` (the `tcloud mcp` stdio server, or the `web_search: true` tool),
70
+ `fs` for writing the cited dossier, `shell` for `agent-knowledge index/lint`.
71
+
72
+ **System prompt (the contract):**
73
+
74
+ ```
75
+ You are a grounded research WORKER. You answer ONE question, and every load-bearing
76
+ claim is traceable to a real source you actually retrieved.
77
+
78
+ THE LOOP — search → cite → synthesize:
79
+ 1. SEARCH issue multiple focused web_search queries; never stop at the first hit.
80
+ 2. CITE attach every load-bearing claim to its source (title + url/identifier).
81
+ A claim you cannot cite is a claim you do NOT make. Never invent a source.
82
+ 3. SYNTH lead with the answer; note where sources agree, conflict, or leave a gap.
83
+
84
+ NON-NEGOTIABLE:
85
+ - Honest "the evidence is thin / conflicting" beats a confident hallucination.
86
+ - Citation density floor is enforced downstream — under-citing is a hard fail, not a warning.
87
+ - You are the LEAF. You do not spawn sub-workers.
88
+
89
+ Return your cited synthesis as the settled output (a structured object if a schema was passed).
90
+ ```
91
+
92
+ **Why this shape:** the validator (`createResearcherValidator`) checks citation density and namespace
93
+ *deterministically*, so the prompt's job is only to make the worker *produce* citable output — the
94
+ checking is not the worker's job. That split is what keeps the verifier cheap (Sketch 4).
95
+
96
+ ---
97
+
98
+ ## Sketch 2 — `verifier-driver` (the gate, NOT the judge)
99
+
100
+ **Status: composes existing primitives.** This is `agent-runtime`'s `verify({ implement, verifier })`
101
+ combinator (`src/runtime/personify/combinators.ts:333`) wired to a research worker + a thin checker.
102
+ **Do not write a new verify-loop** — the 2-node implement→gate already exists.
103
+
104
+ **When to use:** when "did the worker actually deliver" must be proven before the result counts —
105
+ i.e. always, on a graded run. A driver that runs ONE worker behind ONE gate.
106
+
107
+ **Tools:** the coordination verbs (`spawn_agent`, `await_event`, `steer_agent`) over a `Scope`, plus
108
+ the ability to call the deterministic checker (`agent-knowledge lint` / `createResearcherValidator`).
109
+
110
+ **System prompt (the contract):**
111
+
112
+ ```
113
+ You are a VERIFIER-DRIVER. You spawn ONE research worker, then you GATE its output —
114
+ you never accept the worker's own say-so.
115
+
116
+ THE GATE — generate → check → decide:
117
+ 1. SPAWN author a grounded-researcher worker for the question and spawn it.
118
+ 2. AWAIT pull its settled output (await_event). Read the REAL output, not a summary.
119
+ 3. CHECK run the DETERMINISTIC checker first (citation density, namespace, lint,
120
+ schema). This is cheap and catches most failures. Only if it PASSES do you
121
+ escalate to the one semantic question a checker can't answer:
122
+ "does each cited source actually support the claim it's attached to?"
123
+ 4. DECIDE PASS → settle. FAIL → steer_agent with the SPECIFIC gap named
124
+ ("claim X cites source Y but Y does not say X"), or respawn a sharper worker.
125
+
126
+ NON-NEGOTIABLE:
127
+ - selector != judge. You GATE (pass/fail against a contract); you do not re-rank or
128
+ rewrite the worker's content yourself.
129
+ - "Settled a file" is not delivery. The deterministic check passing is delivery.
130
+ - Default to FAILING. A pass is earned by surviving the checker, not by looking plausible.
131
+ ```
132
+
133
+ **Why this shape:** the driver's intelligence is *deciding what to check and how to steer on a
134
+ fail*, not re-doing the research. The expensive judgment (does the source support the claim) is gated
135
+ behind the cheap deterministic checker, so on most runs the LLM-judge step never fires.
136
+
137
+ ---
138
+
139
+ ## Sketch 3 — `research-supervisor` (fan-out → fuse → recurse)
140
+
141
+ **Status: exists as `research-then-build-driver` + `parallel-fanout-supervisor` in supervisor-lab.**
142
+ This sketch is the **research-specialized merge** of those two — build new only as a thin overlay via
143
+ `mergeAgentProfiles`; the orchestration body is identical.
144
+
145
+ **When to use:** a research question too broad for one worker — needs decomposition into disjoint
146
+ sub-topics, parallel grounded workers, and a fused cited synthesis. The top of a research tree.
147
+
148
+ **Tools / resources:** coordination MCP (`spawn_agent` / `await_event` / `steer_agent` /
149
+ `answer_question` / `stop`) over a `Scope`; `web_search` to *ground the decomposition itself*; skills
150
+ `dynamic-workflows`, `orchestrating-workers`, `authoring-agent-profiles` (all already in
151
+ `supervisor-lab/skills/`).
152
+
153
+ **System prompt (the contract):**
154
+
155
+ ```
156
+ You are a research SUPERVISOR. You do NOT research the topic yourself — you ground the
157
+ decomposition, fan out grounded-researcher workers, gate them, and FUSE their cited findings.
158
+
159
+ THE SHAPE — ground → fan-out → gate → fan-in:
160
+ 1. GROUND web_search FIRST to see what actually exists. Let the real landscape, not
161
+ your priors, define the sub-topics. Cite what you find.
162
+ 2. DECOMPOSE split into DISJOINT sub-questions — non-overlapping ownership.
163
+ 3. FAN OUT author a grounded-researcher profile per sub-question; spawn ALL of them in
164
+ ONE wave before awaiting any. Vary their angles. Never spawn-await-spawn.
165
+ 4. GATE drain await_event. For each settled worker, run the deterministic checker
166
+ (Sketch 4). REJECT uncited or unsupported claims — a finding without a
167
+ source is not a finding. Steer or respawn on a fail.
168
+ 5. FAN IN author a FINAL synthesis worker that fuses ONLY the gated findings into one
169
+ cited answer. Never paste raw worker dumps. Check each result for failure first.
170
+ 6. RECURSE if a sub-question is itself large, spawn a SUB-supervisor (this profile +
171
+ the coordination MCP) to own it.
172
+
173
+ NON-NEGOTIABLE:
174
+ - Author EACH worker as a FULL profile (name, description, prompt, model, tools, skills).
175
+ A 2-sentence prompt with no tools is a stub, not a worker — the richness gate will flag it.
176
+ - Never go blind: keep pulling until every spawned worker is terminal.
177
+ - Settle only on gated, cited output. Budget is conserved and depth is bounded by the Scope.
178
+ ```
179
+
180
+ **Why this shape:** the supervisor's only real lever is *the quality of the profiles it authors* —
181
+ that is the capability `supervisor-lab` measures (`thinProfileRatio`,
182
+ `bench/supervise-topology.ts`). The gate (step 4) reuses the cheap verifier so the supervisor doesn't
183
+ burn its budget re-judging.
184
+
185
+ ---
186
+
187
+ ## Sketch 4 — `dedup-verifier` (thin, cheap, mostly deterministic)
188
+
189
+ **This is the emilkowalski dedup lesson made into a profile.** It is deliberately *thin* — the
190
+ "taste" lives in the deterministic validator, the profile is just the lens that applies it and
191
+ escalates only the residual.
192
+
193
+ **When to use:** called by a driver/supervisor at a gate (Sketch 2 step 3, Sketch 3 step 4). Never
194
+ auto-invoked — the verifier equivalent of `disable-model-invocation: true`.
195
+
196
+ **Tools:** `shell` (to run `agent-knowledge lint` / `validate --strict`), the
197
+ `createResearcherValidator` call, and an LLM *only* for the one semantic check. **No `web_search`, no
198
+ `fs` write, no generation tools** — a verifier that can edit content is a verifier that can cheat.
199
+
200
+ **System prompt (the whole thing — it's short on purpose):**
201
+
202
+ ```
203
+ You are a VERIFIER. You judge ONE research output against a fixed contract. You default
204
+ to FAILING; a pass is earned. You never edit, research, or rewrite — you only judge.
205
+
206
+ ORDER (cheapest check first, stop at the first hard fail):
207
+ 1. DETERMINISTIC run the validator/lint: citation density >= floor, namespace match,
208
+ links resolve, schema valid. If any fails → FAIL with the rule name.
209
+ ~90% of failures die here, at zero LLM cost.
210
+ 2. SEMANTIC ONLY if (1) passes: for each cited claim, does the cited source
211
+ actually support it? This is the one thing a checker can't do. Be terse.
212
+
213
+ OUTPUT: a findings table (claim | source | supported? | why) + a verdict (PASS / FAIL),
214
+ grouped by severity. No prose. A violation is a finding.
215
+
216
+ You carry NO generation ability by design. If you find yourself wanting to fix the
217
+ output, STOP — that is the worker's job, not yours.
218
+ ```
219
+
220
+ **Why thin beats heavy here:** the dedup already happened in `createResearcherValidator` and the lint
221
+ path. A heavy verifier that re-reasons every claim with an LLM pays full model cost to re-derive a
222
+ check the validator does for free, and (worse) a verifier with generation tools can drift into doing
223
+ the work and grading itself. Thin + deterministic-first is both cheaper *and* harder to game. Measured
224
+ claim to test when built: on a real run, the LLM (step 2) should fire on **< ~10–20% of gated outputs**
225
+ because the deterministic check (step 1) absorbs the rest — if it fires on most, the validator floor
226
+ is mis-set, not the verifier.
227
+
228
+ ---
229
+
230
+ ## supervisor-lab — status and port plan
231
+
232
+ **It exists.** `~/code/supervisor-lab` is a real, non-trivial repo
233
+ (`github.com/tangle-network/supervisor-lab`), not a stub. `ls ~/code | grep -i supervisor` → it's
234
+ there.
235
+
236
+ It already is what one might propose to "create": the product/experiment layer for supervisor agents
237
+ over the `agent-runtime` substrate, dependency-one-way (`supervisor-lab → agent-runtime →
238
+ agent-eval`). It ships:
239
+
240
+ - **the two-agent loop, already ported** — `bench/supervise-topology.ts` is exactly the "two-agent
241
+ loop": a supervisor mounts the coordination MCP over a live `Scope`/`Supervisor`
242
+ (`createSupervisor`, `serveCoordinationMcp`), authors worker `AgentProfile`s as code, spawns them,
243
+ drains the bus (`await_event`), steers (`steer_agent`), and grades on a real judge. It has an
244
+ **OFFLINE $0 scripted path** and a LIVE cli-bridge path. This is the loop to plug these sketches
245
+ into — it does not need to be built.
246
+ - **profile archetypes** — `profiles/research/{grounded-researcher,research-then-build-driver}.ts`,
247
+ `profiles/engineering/{parallel-fanout-supervisor,long-horizon-plan-driver,hardware-auditor}.ts`.
248
+ Sketches 1 and 3 above are *already these files*.
249
+ - **a skill layer** — `skills/{authoring-agent-profiles,orchestrating-workers,dynamic-workflows,
250
+ spawning-research-loops}/SKILL.md`, each in the exact emilkowalski frontmatter+rules shape.
251
+ - **a catalog + ingest pipeline** — `src/catalog/`, `src/ingest/skills.ts` ingests vendored
252
+ Claude-Code-style skill packs (the same shape as `emilkowalski/skills`) by convention.
253
+ - **the richness gate** — `assessAuthoredProfile` / `profileRichnessFinding` / `thinProfileRatio`,
254
+ the deterministic "is this a stub" check that is the dedup-verifier's first line.
255
+
256
+ ### How to port the two-agent loop (it's already there — this is how to extend it)
257
+
258
+ Because the loop already exists, "porting" means **adding these four profiles + the thin-verifier gate
259
+ into the existing harness**, not rebuilding it:
260
+
261
+ 1. **Land the profiles as catalog files.** Sketches 1 and 3 are already
262
+ `profiles/research/grounded-researcher.ts` and `research-then-build-driver.ts`. Sketch 2
263
+ (`verifier-driver`) and Sketch 4 (`dedup-verifier`) are new files under
264
+ `profiles/research/` — author them as `defineAgentProfile` entries, resolve their skills via the
265
+ `skills(...)` helper in `profiles/_shared.ts` (fails closed on an unknown skill).
266
+
267
+ 2. **Wire the dedup-verifier into the gate.** In `bench/supervise-topology.ts`, the worker's
268
+ `runWorker` already grades on `adapter.judge`. For research tasks, replace/augment that with the
269
+ deterministic `createResearcherValidator` + `agent-knowledge lint` path FIRST, and only escalate to
270
+ the LLM semantic check on a deterministic pass. This is the dedup point made executable: the
271
+ existing `assessAuthoredProfile` richness gate is the *profile* check; the validator is the
272
+ *output* check. Both run before any LLM judge.
273
+
274
+ 3. **Add a research bench arm.** `supervise-topology.ts` auto-detects domain (repo vs. answer/text).
275
+ A research arm is an "answer/text" task whose `adapter.judge` is the cited-output validator. Add
276
+ `BENCH=research` that loads a research question + namespace and grades via
277
+ `createResearcherValidator`. The OFFLINE scripted path already proves the harness at $0; extend
278
+ `scriptedRun.workerResult` to emit a cited/uncited synthesis so the offline smoke exercises the
279
+ verifier without spend.
280
+
281
+ 4. **Run the existing experiment knob.** The whole point of the lab is the one knob: catalog/skills
282
+ ON vs. OFF (`CATALOG=1`). With these profiles + the thin verifier in the catalog, run
283
+ capability-aware vs. baseline on a hard research task and read `thinProfileRatio` + delivery rate —
284
+ does giving the supervisor the research archetypes + the cheap gate produce better-composed teams
285
+ and more *cited, gated* deliveries.
286
+
287
+ ### Cross-repo note
288
+
289
+ `agent-knowledge` already owns the deterministic research checker (`createResearcherValidator`, the
290
+ `lint`/`validate` CLI) and the researcher profile (`researcherProfile`,
291
+ `multiHarnessResearcherFanout`). The thin verifier should **call those**, not reimplement them —
292
+ `supervisor-lab` depends on `agent-runtime`, and the research checks live in `agent-knowledge`, so the
293
+ research bench arm pulls `@tangle-network/agent-knowledge` for the verifier's deterministic line. That
294
+ keeps the dedup in one place: the validator is authored once, the verifier profile is a thin lens over
295
+ it.