@mmerterden/multi-agent-pipeline 20.20.0 → 20.21.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +25 -0
- package/README.md +24 -13
- package/README.tr.md +21 -10
- package/docs/ecosystem.md +6 -5
- package/docs/facts.json +5 -4
- package/docs/features.md +1 -0
- package/install/catalog-history.json +1 -1
- package/manifest.json +42 -39
- package/package.json +1 -1
- package/pipeline/commands/multi-agent/SKILL.md +1 -1
- package/pipeline/commands/multi-agent/route-on/SKILL.md +32 -8
- package/pipeline/commands/multi-agent/route-status/SKILL.md +12 -2
- package/pipeline/commands/multi-agent/setup/SKILL.md +2 -1
- package/pipeline/lib/model-dispatch.sh +72 -24
- package/pipeline/lib/outbound-gate.mjs +2 -0
- package/pipeline/lib/route-state.sh +118 -10
- package/pipeline/lib/toolkit-llm.sh +30 -0
- package/pipeline/lib/unattended-profile-check.mjs +23 -2
- package/pipeline/multi-agent-refs/analysis/evidence.md +3 -1
- package/pipeline/multi-agent-refs/cross-cli-contract.md +8 -3
- package/pipeline/multi-agent-refs/features/doctor.md +11 -0
- package/pipeline/multi-agent-refs/features/free-models.md +111 -0
- package/pipeline/multi-agent-refs/features/model-fallback.md +21 -15
- package/pipeline/multi-agent-refs/features/unattended-security.md +4 -2
- package/pipeline/multi-agent-refs/setup/free-models.md +22 -0
- package/pipeline/preferences-template.json +1 -0
- package/pipeline/schemas/prefs.schema.json +10 -1
- package/pipeline/schemas/route-config.schema.json +33 -8
- package/pipeline/schemas/secret-patterns.json +14 -0
- package/pipeline/scripts/bulk-read.sh +98 -12
- package/pipeline/scripts/cost-table.json +9 -1
- package/pipeline/scripts/doctor.mjs +100 -0
- package/pipeline/scripts/pre-commit-check.sh +1 -1
- package/pipeline/scripts/unattended_policy.py +20 -0
- package/pipeline/skills/.skill-manifest.json +5 -5
- package/pipeline/skills/shared/core/multi-agent/SKILL.md +2 -0
- package/pipeline/skills/shared/core/multi-agent-route-on/SKILL.md +31 -7
- package/pipeline/skills/shared/core/multi-agent-route-status/SKILL.md +12 -2
- package/pipeline/skills/shared/core/multi-agent-setup/SKILL.md +1 -0
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
|
3
3
|
"$id": "https://github.com/mmerterden/multi-agent-pipeline/pipeline/schemas/route-config.schema.json",
|
|
4
4
|
"title": "Model routing configuration",
|
|
5
|
-
"description": "Policy-driven model selection, OFF by default. Lives under prefs.global.modelRouting. Two layers, and the difference is the whole safety argument. Layer 1 chooses a rung INSIDE the Anthropic ladder and writes to the existing PHASE_MODEL_OVERRIDE seam - no new network path, no new account. Layer 2
|
|
5
|
+
"description": "Policy-driven model selection, OFF by default. Lives under prefs.global.modelRouting. Two layers, and the difference is the whole safety argument. Layer 1 chooses a rung INSIDE the Anthropic ladder and writes to the existing PHASE_MODEL_OVERRIDE seam - no new network path, no new account. Layer 2 is the `free` rung: a simple text task (a bulk read, a summary, an extraction, a label, a translation draft) goes to the multi-agent-toolkit's verified free-model chain (llm_delegate / multi-agent-toolkit-llm), whose every answer is checked against its source and discarded when it does not hold, with the Anthropic default taking over. It is reachable only where the pipeline makes the call itself (`bulk-read`, `delegate`); a subagent is dispatched by the host, so model-dispatch.sh refuses `free` and every other non-Anthropic rung at the subagent site and falls to the rule's next rung. `scope` has no `host-session` member and that absence is enforced here rather than written in prose: rewriting the host's base URL would route the user's ENTIRE session, including work that has nothing to do with this pipeline, through a third layer - breaking the subscription's auth model and silently changing which model answers. Shipping this disabled is the same reasoning as modelFallback.fableEnabled: a cost control that is on by default is not a control.",
|
|
6
6
|
"type": "object",
|
|
7
7
|
"additionalProperties": false,
|
|
8
8
|
"required": ["enabled"],
|
|
@@ -16,16 +16,16 @@
|
|
|
16
16
|
"type": "string",
|
|
17
17
|
"enum": ["manual", "task-fit", "cost-ceiling"],
|
|
18
18
|
"default": "manual",
|
|
19
|
-
"description": "manual: only the explicit rules[] apply, nothing is inferred. task-fit:
|
|
19
|
+
"description": "manual: only the explicit rules[] apply, nothing is inferred. task-fit: the matched rule's rungs are tried cheapest first (cost-table.json prices, free first). cost-ceiling: rules apply in order until the run reaches the warn threshold of budgetCeilingUsd, then the cheapest rung is tried first."
|
|
20
20
|
},
|
|
21
21
|
"scope": {
|
|
22
22
|
"type": "array",
|
|
23
23
|
"default": ["subagent"],
|
|
24
24
|
"uniqueItems": true,
|
|
25
|
-
"description": "Where routing is allowed to act. Every member names a call site the pipeline itself owns. There is deliberately no `host-session` member - see the title description.",
|
|
25
|
+
"description": "Where routing is allowed to act. Every member names a call site the pipeline itself owns: subagent (persona dispatch, Anthropic rungs only), bulk-read (bulk-read.sh), delegate (simple text tasks the skills hand to llm_delegate), research. There is deliberately no `host-session` member - see the title description.",
|
|
26
26
|
"items": {
|
|
27
27
|
"type": "string",
|
|
28
|
-
"enum": ["subagent", "bulk-read", "research"]
|
|
28
|
+
"enum": ["subagent", "bulk-read", "delegate", "research"]
|
|
29
29
|
}
|
|
30
30
|
},
|
|
31
31
|
"rules": {
|
|
@@ -42,8 +42,14 @@
|
|
|
42
42
|
"additionalProperties": false,
|
|
43
43
|
"minProperties": 1,
|
|
44
44
|
"properties": {
|
|
45
|
-
"persona": {
|
|
46
|
-
|
|
45
|
+
"persona": {
|
|
46
|
+
"type": "string"
|
|
47
|
+
},
|
|
48
|
+
"phase": {
|
|
49
|
+
"type": "integer",
|
|
50
|
+
"minimum": 0,
|
|
51
|
+
"maximum": 5
|
|
52
|
+
},
|
|
47
53
|
"taskKind": {
|
|
48
54
|
"type": "string",
|
|
49
55
|
"enum": ["bugfix", "feature", "refactor", "chore", "component"]
|
|
@@ -54,16 +60,35 @@
|
|
|
54
60
|
"type": "array",
|
|
55
61
|
"minItems": 1,
|
|
56
62
|
"description": "Rungs in descending preference. Rung NAMES are the contract; model ids are not - they live in cost-table.json and change without a config edit.",
|
|
57
|
-
"items": {
|
|
63
|
+
"items": {
|
|
64
|
+
"type": "string"
|
|
65
|
+
}
|
|
58
66
|
}
|
|
59
67
|
}
|
|
60
68
|
}
|
|
61
69
|
},
|
|
70
|
+
"free": {
|
|
71
|
+
"type": "object",
|
|
72
|
+
"additionalProperties": false,
|
|
73
|
+
"description": "Layer 2. With enabled:true, the bulk-read and delegate call sites try the free rung before the rule's Anthropic rungs. Written by route-on --free.",
|
|
74
|
+
"properties": {
|
|
75
|
+
"enabled": {
|
|
76
|
+
"type": "boolean",
|
|
77
|
+
"default": false,
|
|
78
|
+
"description": "Try the verified free-model chain first at bulk-read and delegate. Ships false."
|
|
79
|
+
},
|
|
80
|
+
"allowTraining": {
|
|
81
|
+
"type": "boolean",
|
|
82
|
+
"default": false,
|
|
83
|
+
"description": "Also use free tiers that may train on prompts (Mistral, Gemini) or whose data use is unknown. false keeps text to providers that do not (Groq) and a local Ollama."
|
|
84
|
+
}
|
|
85
|
+
}
|
|
86
|
+
},
|
|
62
87
|
"budgetCeilingUsd": {
|
|
63
88
|
"type": ["number", "null"],
|
|
64
89
|
"default": null,
|
|
65
90
|
"minimum": 0,
|
|
66
|
-
"description": "Only consulted by the cost-ceiling strategy. null means no ceiling, which is not the same as a ceiling of 0."
|
|
91
|
+
"description": "Only consulted by the cost-ceiling strategy, through cost-budget-check.mjs against the run's tracker. null means no ceiling, which is not the same as a ceiling of 0."
|
|
67
92
|
},
|
|
68
93
|
"recordDecisions": {
|
|
69
94
|
"type": "boolean",
|
|
@@ -78,6 +78,20 @@
|
|
|
78
78
|
"len": 36,
|
|
79
79
|
"suffix": ""
|
|
80
80
|
},
|
|
81
|
+
{
|
|
82
|
+
"id": "groq",
|
|
83
|
+
"prefix": "gsk_",
|
|
84
|
+
"fill": "A",
|
|
85
|
+
"len": 52,
|
|
86
|
+
"suffix": ""
|
|
87
|
+
},
|
|
88
|
+
{
|
|
89
|
+
"id": "openrouter",
|
|
90
|
+
"prefix": "sk-or-v1-",
|
|
91
|
+
"fill": "a",
|
|
92
|
+
"len": 64,
|
|
93
|
+
"suffix": ""
|
|
94
|
+
},
|
|
81
95
|
{
|
|
82
96
|
"id": "huggingface",
|
|
83
97
|
"prefix": "hf_",
|
|
@@ -91,11 +91,12 @@ if command -v jq >/dev/null 2>&1; then
|
|
|
91
91
|
fi
|
|
92
92
|
[ -n "$MODEL" ] || MODEL="${PREF_MODEL:-haiku}"
|
|
93
93
|
|
|
94
|
-
# Routing picks the rung
|
|
95
|
-
# `
|
|
96
|
-
#
|
|
97
|
-
#
|
|
98
|
-
# account that owns it,
|
|
94
|
+
# Routing picks the rung. An Anthropic rung goes to the `claude` CLI below. The
|
|
95
|
+
# `free` rung (route-on --free) goes to the multi-agent-toolkit's verified
|
|
96
|
+
# free-model chain first, and the Anthropic default answers whenever that chain
|
|
97
|
+
# escalates or is unreachable. Sending this read to a free provider puts the
|
|
98
|
+
# file's text outside the account that owns it, which is why it happens only
|
|
99
|
+
# after a user turned it on, and only to providers the data policy allows.
|
|
99
100
|
# An explicit --model still wins: a caller that named a rung asked for that rung.
|
|
100
101
|
if [ -z "${MODEL_EXPLICIT:-}" ] && [ -x "$HERE/../lib/model-dispatch.sh" ]; then
|
|
101
102
|
MODEL=$("$HERE/../lib/model-dispatch.sh" bulk-read \
|
|
@@ -104,7 +105,9 @@ fi
|
|
|
104
105
|
case "$TIMEOUT" in "" ) TIMEOUT="$PREF_TIMEOUT" ;; esac
|
|
105
106
|
case "$TIMEOUT" in ""|*[!0-9]*|0) TIMEOUT=60 ;; esac
|
|
106
107
|
|
|
107
|
-
|
|
108
|
+
FALLBACK_MODEL="${PREF_MODEL:-haiku}"
|
|
109
|
+
[ "$FALLBACK_MODEL" = "free" ] && FALLBACK_MODEL="haiku"
|
|
110
|
+
[ "$MODEL" = "free" ] || command -v claude >/dev/null 2>&1 || degrade "the claude CLI is not on PATH"
|
|
108
111
|
|
|
109
112
|
LINES=$(wc -l < "$FILE" | tr -d ' ')
|
|
110
113
|
BYTES=$(wc -c < "$FILE" | tr -d ' ')
|
|
@@ -163,7 +166,8 @@ fi
|
|
|
163
166
|
|
|
164
167
|
PROMPT_FILE="$(mktemp -t bulk-read.XXXXXX)"
|
|
165
168
|
OUT_FILE="$(mktemp -t bulk-read-out.XXXXXX)"
|
|
166
|
-
|
|
169
|
+
FREE_DIR="$(mktemp -d -t bulk-read-free.XXXXXX)"
|
|
170
|
+
trap 'rm -rf "$PROMPT_FILE" "$OUT_FILE" "$FREE_DIR"' EXIT
|
|
167
171
|
|
|
168
172
|
# The worker gets numbered lines because every claim it makes has to be
|
|
169
173
|
# addressable afterwards. `nl -ba` numbers blank lines too, so the numbers match
|
|
@@ -191,6 +195,90 @@ done
|
|
|
191
195
|
} > "$PROMPT_FILE"
|
|
192
196
|
|
|
193
197
|
START=$(date +%s)
|
|
198
|
+
TIMEOUT_BIN=""
|
|
199
|
+
for candidate in timeout gtimeout; do
|
|
200
|
+
command -v "$candidate" >/dev/null 2>&1 && { TIMEOUT_BIN="$candidate"; break; }
|
|
201
|
+
done
|
|
202
|
+
|
|
203
|
+
# The free chain gets the same persona, question and numbered lines, and the
|
|
204
|
+
# bulk-reader output schema. The toolkit verifies the answer
|
|
205
|
+
# against the numbered text before it returns it; the range check below adds
|
|
206
|
+
# what only this script knows, the file's line count. Anything short of a
|
|
207
|
+
# verified, in-range answer hands the read to the Anthropic default.
|
|
208
|
+
FREE_DONE=0
|
|
209
|
+
if [ "$MODEL" = "free" ]; then
|
|
210
|
+
FREE_REASON=""
|
|
211
|
+
_MA_TL="$(dirname "${BASH_SOURCE[0]}")/../lib/toolkit-llm.sh"
|
|
212
|
+
[ -f "$_MA_TL" ] || _MA_TL="$HOME/.claude/lib/toolkit-llm.sh"
|
|
213
|
+
FREE_SCHEMA="$HERE/../schemas/bulk-read-output.schema.json"
|
|
214
|
+
if [ -f "$_MA_TL" ] && [ -f "$FREE_SCHEMA" ]; then
|
|
215
|
+
. "$_MA_TL"
|
|
216
|
+
# The sandbox of an unattended run denies writes under ~/.claude, where the
|
|
217
|
+
# toolkit keeps its quota and usage files by default.
|
|
218
|
+
if [ "${MULTI_AGENT_UNATTENDED:-}" = "1" ] && [ -z "${MCP_TOOLKIT_INDEX_DIR:-}" ]; then
|
|
219
|
+
export MCP_TOOLKIT_INDEX_DIR="$HOME/.multi-agent-unattended/toolkit-index"
|
|
220
|
+
fi
|
|
221
|
+
nl -ba "$FILE" > "$FREE_DIR/numbered.txt"
|
|
222
|
+
FREE_PROMPT="$(awk 'BEGIN{fm=0} /^---$/{fm++; next} fm>=2{print}' "$PERSONA")
|
|
223
|
+
<question>
|
|
224
|
+
$QUESTION
|
|
225
|
+
</question>
|
|
226
|
+
The SOURCE is the file $FILE with its $LINES lines numbered."
|
|
227
|
+
ALLOW_FLAG=""
|
|
228
|
+
for prefs in "${MULTI_AGENT_PREFS:-}" "$HOME/.claude/multi-agent-preferences.json" "$HOME/.config/multi-agent-pipeline/multi-agent-preferences.json"; do
|
|
229
|
+
[ -n "$prefs" ] && [ -f "$prefs" ] || continue
|
|
230
|
+
[ "$(jq -r '.global.modelRouting.free.allowTraining // false' "$prefs" 2>/dev/null)" = "true" ] && ALLOW_FLAG="--allow-training"
|
|
231
|
+
break
|
|
232
|
+
done
|
|
233
|
+
if [ -n "$TIMEOUT_BIN" ]; then
|
|
234
|
+
"$TIMEOUT_BIN" "$TIMEOUT" bash -c '. "$1"; shift; ma_toolkit_llm "$@"' _ "$_MA_TL" delegate --task summarize \
|
|
235
|
+
--prompt "$FREE_PROMPT" --path "$FREE_DIR/numbered.txt" --schema "$FREE_SCHEMA" \
|
|
236
|
+
--max-tokens 2048 $ALLOW_FLAG > "$FREE_DIR/result.json" 2>/dev/null
|
|
237
|
+
else
|
|
238
|
+
ma_toolkit_llm delegate --task summarize --prompt "$FREE_PROMPT" --path "$FREE_DIR/numbered.txt" \
|
|
239
|
+
--schema "$FREE_SCHEMA" --max-tokens 2048 $ALLOW_FLAG > "$FREE_DIR/result.json" 2>/dev/null
|
|
240
|
+
fi
|
|
241
|
+
FREE_STATUS=$?
|
|
242
|
+
case "$FREE_STATUS" in
|
|
243
|
+
0)
|
|
244
|
+
if LINES="$LINES" python3 - "$FREE_DIR/result.json" "$OUT_FILE" <<'PYEOF'
|
|
245
|
+
import json, os, sys
|
|
246
|
+
r = json.load(open(sys.argv[1]))
|
|
247
|
+
v = r.get("value")
|
|
248
|
+
n = int(os.environ["LINES"])
|
|
249
|
+
ok = isinstance(v, dict)
|
|
250
|
+
for reg in (v or {}).get("regions") or []:
|
|
251
|
+
ok = ok and 1 <= reg.get("start", 0) <= reg.get("end", 0) <= n
|
|
252
|
+
for sym in (v or {}).get("symbols") or []:
|
|
253
|
+
ok = ok and 1 <= sym.get("line", 0) <= n
|
|
254
|
+
if not ok:
|
|
255
|
+
sys.exit(1)
|
|
256
|
+
json.dump(v, open(sys.argv[2], "w"))
|
|
257
|
+
open(sys.argv[2] + ".via", "w").write("free:%s/%s" % (r.get("provider"), r.get("model")))
|
|
258
|
+
PYEOF
|
|
259
|
+
then
|
|
260
|
+
FREE_DONE=1
|
|
261
|
+
MODEL=$(cat "$OUT_FILE.via"); rm -f "$OUT_FILE.via"
|
|
262
|
+
else
|
|
263
|
+
FREE_REASON="the free answer cited lines outside 1-$LINES"
|
|
264
|
+
fi ;;
|
|
265
|
+
3) FREE_REASON="no free provider produced a verified answer" ;;
|
|
266
|
+
124) FREE_REASON="the free chain did not answer within ${TIMEOUT}s" ;;
|
|
267
|
+
*) FREE_REASON="the free chain is unavailable (exit $FREE_STATUS)" ;;
|
|
268
|
+
esac
|
|
269
|
+
else
|
|
270
|
+
FREE_REASON="toolkit-llm.sh or the bulk-read output schema is not installed"
|
|
271
|
+
fi
|
|
272
|
+
[ -x "$HERE/log-metric.sh" ] && \
|
|
273
|
+
"$HERE/log-metric.sh" "${MULTI_AGENT_TASK_ID:-unknown}" "$PHASE" bulk_read.free \
|
|
274
|
+
outcome="$([ "$FREE_DONE" = 1 ] && echo verified || echo escalated)" reason="${FREE_REASON:-ok}" >/dev/null 2>&1
|
|
275
|
+
if [ "$FREE_DONE" != 1 ]; then
|
|
276
|
+
echo "bulk-read: free chain not used ($FREE_REASON); reading on $FALLBACK_MODEL" >&2
|
|
277
|
+
MODEL="$FALLBACK_MODEL"
|
|
278
|
+
command -v claude >/dev/null 2>&1 || degrade "the claude CLI is not on PATH"
|
|
279
|
+
fi
|
|
280
|
+
fi
|
|
281
|
+
|
|
194
282
|
# `claude -p` reads the prompt from stdin; the model flag names the rung. A
|
|
195
283
|
# non-zero exit, an empty answer and a non-JSON answer are all degradations, and
|
|
196
284
|
# each one names itself rather than falling through to a generic failure.
|
|
@@ -198,11 +286,9 @@ START=$(date +%s)
|
|
|
198
286
|
# GNU spelling would leave the budget silently unenforced on the platform most of
|
|
199
287
|
# these runs happen on - a pref that does nothing, which is the class
|
|
200
288
|
# smoke-prefs-consumed exists to catch.
|
|
201
|
-
|
|
202
|
-
|
|
203
|
-
|
|
204
|
-
done
|
|
205
|
-
if [ -n "$TIMEOUT_BIN" ]; then
|
|
289
|
+
if [ "$FREE_DONE" = 1 ]; then
|
|
290
|
+
STATUS=0
|
|
291
|
+
elif [ -n "$TIMEOUT_BIN" ]; then
|
|
206
292
|
"$TIMEOUT_BIN" "$TIMEOUT" claude -p --model "$MODEL" < "$PROMPT_FILE" > "$OUT_FILE" 2>/dev/null
|
|
207
293
|
STATUS=$?
|
|
208
294
|
else
|
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
{
|
|
2
|
-
"_readme": "Per-model unit prices in USD per million tokens. Source: Anthropic public pricing (Claude-family rows verified 2026-07-28 against the current model table; the gpt-* rows remain approximate). Update when Anthropic publishes new tiers. Rung names (fable / opus / sonnet / haiku) are the pipeline's stable identifiers and `modelId` is the wire value they currently resolve to - dispatch reads the rung, so a generation move is a `modelId` edit here plus the phase specs, never a rename of the rungs. `provider` says which layer a rung belongs to: an `anthropic` rung is reachable from every call site
|
|
2
|
+
"_readme": "Per-model unit prices in USD per million tokens. Source: Anthropic public pricing (Claude-family rows verified 2026-07-28 against the current model table; the gpt-* rows remain approximate). Update when Anthropic publishes new tiers. Rung names (fable / opus / sonnet / haiku) are the pipeline's stable identifiers and `modelId` is the wire value they currently resolve to - dispatch reads the rung, so a generation move is a `modelId` edit here plus the phase specs, never a rename of the rungs. `provider` says which layer a rung belongs to: an `anthropic` rung is reachable from every call site; `free-chain` (the `free` rung) only from bulk-read and delegate, where the multi-agent-toolkit's verified free-model chain answers and the caller's default takes over when it escalates; any other provider from no call site. model-dispatch.sh enforces this. Unknown models render USD as ' - ' and emit a footnote - never block PR-body generation. cacheReadPerMtok is the discounted rate for prompt-cache hits (~10% of inPerMtok); the renderer prices a phase's tokens_cached at this rate when the tracker records it, so resume/cache reuse is visible in the ledger.",
|
|
3
3
|
"schemaVersion": "1.1.0",
|
|
4
4
|
"prices": {
|
|
5
5
|
"fable": {
|
|
@@ -57,6 +57,14 @@
|
|
|
57
57
|
"modelId": "gpt-5.6-terra",
|
|
58
58
|
"provider": "openai",
|
|
59
59
|
"note": "Codex CLI floor tier - speed-optimised, maps the haiku rung of the Codex persona tier map (task-clarifier). Approximate; verify against OpenAI pricing before relying on totals."
|
|
60
|
+
},
|
|
61
|
+
"free": {
|
|
62
|
+
"inPerMtok": 0.0,
|
|
63
|
+
"outPerMtok": 0.0,
|
|
64
|
+
"cacheReadPerMtok": 0.0,
|
|
65
|
+
"modelId": "free-chain",
|
|
66
|
+
"provider": "free-chain",
|
|
67
|
+
"note": "Layer 2: the multi-agent-toolkit's verified free-model chain (llm_delegate / multi-agent-toolkit-llm) over free tiers - Groq, Mistral, Gemini, OpenRouter free models, Z.AI GLM Flash, local Ollama. Priced at zero because the provider bills nothing on a free tier; what it saves is reported by route-status from the toolkit's usage log at the rate of the rung it replaced."
|
|
60
68
|
}
|
|
61
69
|
}
|
|
62
70
|
}
|
|
@@ -104,6 +104,7 @@ const CHECK_IDS = [
|
|
|
104
104
|
"disk-space",
|
|
105
105
|
"worktree-residue",
|
|
106
106
|
"usage-report",
|
|
107
|
+
"free-models",
|
|
107
108
|
"xcode-skills",
|
|
108
109
|
];
|
|
109
110
|
|
|
@@ -935,6 +936,104 @@ function checkUsageReport() {
|
|
|
935
936
|
);
|
|
936
937
|
}
|
|
937
938
|
|
|
939
|
+
// route-on --free promises that simple tasks leave Claude. With no usable
|
|
940
|
+
// provider every one of them silently escalates back, which costs nothing but
|
|
941
|
+
// saves nothing either - the user believes a control is working that is not.
|
|
942
|
+
// --probe sends each configured provider one tiny request, so a dead key or an
|
|
943
|
+
// exhausted quota shows up here instead of as a week of escalations.
|
|
944
|
+
function checkFreeModels() {
|
|
945
|
+
const prefs = readJson(join(CLAUDE, "multi-agent-preferences.json"));
|
|
946
|
+
const routing = prefs?.global?.modelRouting;
|
|
947
|
+
if (routing?.enabled !== true || routing?.free?.enabled !== true) {
|
|
948
|
+
skip("free-models", "free-model routing is off");
|
|
949
|
+
return;
|
|
950
|
+
}
|
|
951
|
+
const lib = [
|
|
952
|
+
join(HERE, "..", "lib", "toolkit-llm.sh"),
|
|
953
|
+
join(CLAUDE, "lib", "toolkit-llm.sh"),
|
|
954
|
+
].find((p) => existsSync(p));
|
|
955
|
+
if (!lib) {
|
|
956
|
+
report(
|
|
957
|
+
"free-models",
|
|
958
|
+
"WARN",
|
|
959
|
+
"route-on --free is on but lib/toolkit-llm.sh is not installed, so every free task escalates to Claude",
|
|
960
|
+
`run ${hostCommand("update", HERE)}`,
|
|
961
|
+
);
|
|
962
|
+
return;
|
|
963
|
+
}
|
|
964
|
+
const train = routing.free.allowTraining === true ? ["--allow-training"] : [];
|
|
965
|
+
const run = (args) =>
|
|
966
|
+
JSON.parse(
|
|
967
|
+
execFileSync("bash", ["-c", '. "$1"; shift; ma_toolkit_llm "$@"', "_", lib, ...args], {
|
|
968
|
+
encoding: "utf8",
|
|
969
|
+
timeout: PROBE ? 180000 : 60000,
|
|
970
|
+
stdio: ["ignore", "pipe", "ignore"],
|
|
971
|
+
}),
|
|
972
|
+
);
|
|
973
|
+
let listing;
|
|
974
|
+
try {
|
|
975
|
+
listing = run(["providers", ...train]);
|
|
976
|
+
} catch {
|
|
977
|
+
report(
|
|
978
|
+
"free-models",
|
|
979
|
+
"WARN",
|
|
980
|
+
"route-on --free is on but the multi-agent-toolkit CLI could not be run (node and npx are needed), so every free task escalates to Claude",
|
|
981
|
+
"run npx -y --@mmerterden:registry=https://registry.npmjs.org -p @mmerterden/multi-agent-toolkit-mcp@latest multi-agent-toolkit-llm providers",
|
|
982
|
+
);
|
|
983
|
+
return;
|
|
984
|
+
}
|
|
985
|
+
const usable = (listing.providers || []).filter((p) => p.configured && p.allowed_by_policy);
|
|
986
|
+
const detail = (listing.providers || []).map(
|
|
987
|
+
(p) =>
|
|
988
|
+
`${p.id}: ${p.configured ? (p.allowed_by_policy ? `ready (${p.key_source})` : `configured, outside the ${listing.data_policy} policy`) : `missing - ${p.setup}`}`,
|
|
989
|
+
);
|
|
990
|
+
if (!usable.length) {
|
|
991
|
+
report(
|
|
992
|
+
"free-models",
|
|
993
|
+
"WARN",
|
|
994
|
+
`route-on --free is on but no free provider is usable under the ${listing.data_policy} data policy, so every free task escalates to Claude`,
|
|
995
|
+
'run bash "$HOME/.claude/lib/credential-store.sh" set llm-groq - and paste a free key from https://console.groq.com/keys',
|
|
996
|
+
detail,
|
|
997
|
+
);
|
|
998
|
+
return;
|
|
999
|
+
}
|
|
1000
|
+
if (!PROBE) {
|
|
1001
|
+
report("free-models", "OK", null, null, detail);
|
|
1002
|
+
return;
|
|
1003
|
+
}
|
|
1004
|
+
let probed;
|
|
1005
|
+
try {
|
|
1006
|
+
probed = run(["probe"]);
|
|
1007
|
+
} catch {
|
|
1008
|
+
report(
|
|
1009
|
+
"free-models",
|
|
1010
|
+
"WARN",
|
|
1011
|
+
"the free-provider probe could not run",
|
|
1012
|
+
"re-run with --probe once the network is reachable",
|
|
1013
|
+
detail,
|
|
1014
|
+
);
|
|
1015
|
+
return;
|
|
1016
|
+
}
|
|
1017
|
+
const dead = probed.filter((r) => usable.some((u) => u.id === r.provider) && r.outcome !== "ok");
|
|
1018
|
+
if (dead.length === usable.length) {
|
|
1019
|
+
report(
|
|
1020
|
+
"free-models",
|
|
1021
|
+
"WARN",
|
|
1022
|
+
`every usable free provider failed a probe (${dead.map((r) => `${r.provider}: ${r.outcome}`).join(", ")}), so free tasks escalate to Claude until one recovers`,
|
|
1023
|
+
"set a fresh key for the failing provider with the credential store, or add a second provider",
|
|
1024
|
+
probed.map((r) => `${r.provider}: ${r.outcome}${r.reason ? ` (${r.reason})` : ""}`),
|
|
1025
|
+
);
|
|
1026
|
+
return;
|
|
1027
|
+
}
|
|
1028
|
+
report(
|
|
1029
|
+
"free-models",
|
|
1030
|
+
"OK",
|
|
1031
|
+
null,
|
|
1032
|
+
null,
|
|
1033
|
+
probed.map((r) => `${r.provider}: ${r.outcome}${r.reason ? ` (${r.reason})` : ""}`),
|
|
1034
|
+
);
|
|
1035
|
+
}
|
|
1036
|
+
|
|
938
1037
|
// Apple's Xcode skills are the reference for iPhone Duo and window resizing,
|
|
939
1038
|
// SwiftUI, App Intents and the accessibility audits, and Claude Code reads them
|
|
940
1039
|
// from a copy the installer keeps at the installed Xcode build. A newer Xcode
|
|
@@ -1263,6 +1362,7 @@ function main() {
|
|
|
1263
1362
|
checkDiskSpace();
|
|
1264
1363
|
checkWorktreeResidue();
|
|
1265
1364
|
checkUsageReport();
|
|
1365
|
+
checkFreeModels();
|
|
1266
1366
|
checkXcodeSkills();
|
|
1267
1367
|
if (PROFILE === "server") checkServerProfile(resolved);
|
|
1268
1368
|
|
|
@@ -227,7 +227,7 @@ $folded"
|
|
|
227
227
|
|
|
228
228
|
# Model-provider and ML-hub keys. `sk-ant-` and `sk-proj-` are checked before
|
|
229
229
|
# the bare `sk-` form so the message names the provider a reader can revoke.
|
|
230
|
-
if echo "$content" | grep -qE 'sk-ant-[A-Za-z0-9_-]{32,}|sk-proj-[A-Za-z0-9_-]{32,}|pplx-[A-Za-z0-9]{32,}|hf_[A-Za-z0-9]{30,}'; then
|
|
230
|
+
if echo "$content" | grep -qE 'sk-ant-[A-Za-z0-9_-]{32,}|sk-proj-[A-Za-z0-9_-]{32,}|sk-or-v1-[A-Za-z0-9]{40,}|pplx-[A-Za-z0-9]{32,}|hf_[A-Za-z0-9]{30,}|gsk_[A-Za-z0-9]{40,}'; then
|
|
231
231
|
echo "BLOCKED: Model-provider API key in $file" >&2
|
|
232
232
|
FOUND=1
|
|
233
233
|
fi
|
|
@@ -2644,6 +2644,24 @@ def _toolkit_path_refusal(name, ti, policy, env, cwd):
|
|
|
2644
2644
|
return None
|
|
2645
2645
|
|
|
2646
2646
|
|
|
2647
|
+
def _free_routing_on(env):
|
|
2648
|
+
"""modelRouting.enabled and modelRouting.free.enabled in the preferences
|
|
2649
|
+
file. The user's own switch, read at call time: off, unreadable or absent
|
|
2650
|
+
all mean off."""
|
|
2651
|
+
env = env if env is not None else os.environ
|
|
2652
|
+
home = env.get("HOME") or os.path.expanduser("~")
|
|
2653
|
+
for path in (env.get("MULTI_AGENT_PREFS"), os.path.join(home, ".claude", "multi-agent-preferences.json")):
|
|
2654
|
+
if not path or not os.path.isfile(path):
|
|
2655
|
+
continue
|
|
2656
|
+
try:
|
|
2657
|
+
with open(path, encoding="utf-8") as fh:
|
|
2658
|
+
routing = (json.load(fh).get("global") or {}).get("modelRouting") or {}
|
|
2659
|
+
except (OSError, ValueError, AttributeError):
|
|
2660
|
+
return False
|
|
2661
|
+
return routing.get("enabled") is True and (routing.get("free") or {}).get("enabled") is True
|
|
2662
|
+
return False
|
|
2663
|
+
|
|
2664
|
+
|
|
2647
2665
|
def check_toolkit(name, ti, policy, env, depth=0, cwd=None):
|
|
2648
2666
|
"""A multi-agent-toolkit tool call, by its short name (web_goto,
|
|
2649
2667
|
ios_open_url, agent_run_steps). agent_run_steps dispatches other toolkit
|
|
@@ -2653,6 +2671,8 @@ def check_toolkit(name, ti, policy, env, depth=0, cwd=None):
|
|
|
2653
2671
|
return f"{name} arguments are not an object the guard can read"
|
|
2654
2672
|
if name.startswith("research_"):
|
|
2655
2673
|
return f"{name} sends the query to an outside service"
|
|
2674
|
+
if name == "llm_delegate" and not _free_routing_on(env):
|
|
2675
|
+
return "llm_delegate sends text to a free-model provider; unattended runs allow it only after route-on --free"
|
|
2656
2676
|
if name == "web_eval":
|
|
2657
2677
|
return "web_eval runs arbitrary JavaScript that can reach any host; refused unattended"
|
|
2658
2678
|
if name == "agent_run_steps":
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"schemaVersion": "1.0.0",
|
|
3
|
-
"generatedAt": "2026-10-
|
|
3
|
+
"generatedAt": "2026-10-12T02:17:06Z",
|
|
4
4
|
"skillCount": 228,
|
|
5
5
|
"entries": [
|
|
6
6
|
{
|
|
@@ -173,11 +173,11 @@
|
|
|
173
173
|
},
|
|
174
174
|
{
|
|
175
175
|
"path": "shared/core/multi-agent-route-on/SKILL.md",
|
|
176
|
-
"sha256": "
|
|
176
|
+
"sha256": "97c76d711454e6bc9edfd47344bbeab7f73cf72c116cca92cffacdcda7fe96b1"
|
|
177
177
|
},
|
|
178
178
|
{
|
|
179
179
|
"path": "shared/core/multi-agent-route-status/SKILL.md",
|
|
180
|
-
"sha256": "
|
|
180
|
+
"sha256": "869689dbf84edece67168a892a61b51459ff6c15cbf7dafe40c2b774f875b9f0"
|
|
181
181
|
},
|
|
182
182
|
{
|
|
183
183
|
"path": "shared/core/multi-agent-routines/SKILL.md",
|
|
@@ -213,7 +213,7 @@
|
|
|
213
213
|
},
|
|
214
214
|
{
|
|
215
215
|
"path": "shared/core/multi-agent-setup/SKILL.md",
|
|
216
|
-
"sha256": "
|
|
216
|
+
"sha256": "e3523960d5d178125302e5ceeb865902d7b12ea5d615670cddeb4d542fa48035"
|
|
217
217
|
},
|
|
218
218
|
{
|
|
219
219
|
"path": "shared/core/multi-agent-stack/SKILL.md",
|
|
@@ -269,7 +269,7 @@
|
|
|
269
269
|
},
|
|
270
270
|
{
|
|
271
271
|
"path": "shared/core/multi-agent/SKILL.md",
|
|
272
|
-
"sha256": "
|
|
272
|
+
"sha256": "a39f712426638e7955db9171c950be720f11c861621884e628c81a0b6a49de5d"
|
|
273
273
|
},
|
|
274
274
|
{
|
|
275
275
|
"path": "shared/external/accessibility-compliance-accessibility-audit/SKILL.md",
|
|
@@ -122,6 +122,8 @@ export CLAUDE_CODE_SUBAGENT_MODEL="$resolved_model"
|
|
|
122
122
|
|
|
123
123
|
**Precedence:** per-dispatch `PHASE_MODEL_OVERRIDE` > `prefs.global.modelRouting` rule > persona `preferredModel` > global default (`opus`). Routing here is Anthropic-only: features/model-fallback.md "Routing at dispatch".
|
|
124
124
|
|
|
125
|
+
**Simple text tasks (`delegate`):** before summarizing a log or document, extracting fields, labelling or drafting a translation or commit body in-session, ask `"$HOME/.claude/lib/model-dispatch.sh" delegate --default host`. `free` means call `llm_delegate`; `ESCALATE_TO_CLAUDE` or any other answer means do it in-session. Never for code, plans, reviews or triage. Task list and rules: `$HOME/.claude/multi-agent-refs/features/free-models.md`.
|
|
126
|
+
|
|
125
127
|
**Observability:** emit `phase.agent.dispatch` OTel span with `{persona, resolved_model, override_source}` attributes when `MULTI_AGENT_OTEL_SPANS=1` (see `phase-tracker.sh`).
|
|
126
128
|
|
|
127
129
|
## Skill loading and injection
|
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
name: multi-agent-route-on
|
|
3
3
|
description: "Enable policy-driven model routing: pick a strategy, a scope, and the rules that say which rung a call lands on. Use when asked to turn model routing on."
|
|
4
4
|
user-invocable: true
|
|
5
|
-
argument-hint: "[--strategy=manual|task-fit|cost-ceiling] [--scope=subagent,bulk-read,research]"
|
|
5
|
+
argument-hint: "[--free [--allow-training] | --no-free] [--strategy=manual|task-fit|cost-ceiling] [--scope=subagent,bulk-read,delegate,research]"
|
|
6
6
|
metadata:
|
|
7
7
|
language: en
|
|
8
8
|
---
|
|
@@ -24,7 +24,8 @@ Routing ships **off**. This is the command that turns it on, and it writes to
|
|
|
24
24
|
```
|
|
25
25
|
|
|
26
26
|
Ordered, first match wins. `when` matches on `persona`, `phase` (0..5) or
|
|
27
|
-
`taskKind`; `prefer` lists rungs in descending preference.
|
|
27
|
+
`taskKind`; `prefer` lists rungs in descending preference. `free` is a rung too,
|
|
28
|
+
honoured only at `bulk-read` and `delegate` and only while `--free` is on.
|
|
28
29
|
|
|
29
30
|
**Rung names are the contract; model ids are not.** `opus` is a rung here and a
|
|
30
31
|
model id in `cost-table.json`, and the second can change without anyone editing
|
|
@@ -37,18 +38,41 @@ Set rules with:
|
|
|
37
38
|
bash "$HOME/.claude/lib/route-state.sh" set-rules path/to/rules.json
|
|
38
39
|
```
|
|
39
40
|
|
|
41
|
+
## Free models: `--free`
|
|
42
|
+
|
|
43
|
+
```bash
|
|
44
|
+
bash "$HOME/.claude/lib/route-state.sh" on --free
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
Layer 2. Simple text work the pipeline does itself - a large-file read, a log or
|
|
48
|
+
document summary, field extraction, a label, a translation or commit-body draft -
|
|
49
|
+
goes to the multi-agent-toolkit's verified free-model chain first. Every answer
|
|
50
|
+
is checked against its source before it is used; when none passes, Claude does
|
|
51
|
+
the task as before. Decisions, plans, code, reviews and triage never leave Claude.
|
|
52
|
+
|
|
53
|
+
`--free` turns on `modelRouting.free`, adds `bulk-read` and `delegate` to the
|
|
54
|
+
scope, and lists which providers are usable. With none, it says exactly which key
|
|
55
|
+
to add: one free Groq key is enough. `--allow-training` also admits free tiers
|
|
56
|
+
that may train on prompts; leave it off for corporate code and tickets.
|
|
57
|
+
`--no-free` turns only the free chain off. Full contract, the task list for
|
|
58
|
+
development and analysis, and unattended behaviour:
|
|
59
|
+
`$HOME/.claude/multi-agent-refs/features/free-models.md`.
|
|
60
|
+
|
|
40
61
|
## Strategies
|
|
41
62
|
|
|
42
63
|
| Strategy | What it does |
|
|
43
64
|
|---|---|
|
|
44
65
|
| `manual` | only the explicit rules apply, nothing is inferred. The default, because a router that guesses is a router nobody can predict |
|
|
45
|
-
| `task-fit` |
|
|
46
|
-
| `cost-ceiling` |
|
|
66
|
+
| `task-fit` | the matched rule's rungs are tried cheapest first (cost-table prices, `free` first) |
|
|
67
|
+
| `cost-ceiling` | the rule's order holds until the run reaches the warn threshold of `budgetCeilingUsd`, then the cheapest rung is tried first |
|
|
68
|
+
|
|
69
|
+
`set-rules` checks every rule against the schema and the cost table and writes
|
|
70
|
+
nothing if one fails: an unknown rung, a phase outside 0-5, an unknown key.
|
|
47
71
|
|
|
48
72
|
## Scope, and the value that is not in it
|
|
49
73
|
|
|
50
74
|
`scope` names the call sites routing may act on: `subagent`, `bulk-read`,
|
|
51
|
-
`research`. Every one of them is a call **this pipeline makes itself**.
|
|
75
|
+
`delegate`, `research`. Every one of them is a call **this pipeline makes itself**.
|
|
52
76
|
|
|
53
77
|
There is no `host-session` value, and that absence is enforced by that schema
|
|
54
78
|
rather than written as advice. Routing a host session means rewriting the CLI's
|
|
@@ -61,8 +85,8 @@ Passing `--scope=host-session` is refused with that reason, not ignored.
|
|
|
61
85
|
|
|
62
86
|
On Claude Code a subagent cannot be dispatched to a non-Anthropic model: subagent
|
|
63
87
|
dispatch belongs to the host, not to us. So Phase 1, 2 and 3 personas stay inside
|
|
64
|
-
the Anthropic ladder no matter what the rules say, and
|
|
65
|
-
only at `bulk-read` and `
|
|
88
|
+
the Anthropic ladder no matter what the rules say, and the `free` rung applies
|
|
89
|
+
only at `bulk-read` and `delegate`, where the pipeline makes the call itself.
|
|
66
90
|
|
|
67
91
|
This is printed by `route-status` on every invocation instead of living in a doc,
|
|
68
92
|
because the question it answers - "routing is on, why is the reviewer still on
|
|
@@ -28,13 +28,23 @@ A disabled router with rules is deliberately not reported as "off" alone: that
|
|
|
28
28
|
reads as "unconfigured" and sends the user through `route-on`'s questions a
|
|
29
29
|
second time.
|
|
30
30
|
|
|
31
|
+
## Free models and what they saved
|
|
32
|
+
|
|
33
|
+
With `route-on --free`, status also prints `free: true` with its data policy,
|
|
34
|
+
where the free chain answers first (`bulk-read`, `delegate`), and the last seven
|
|
35
|
+
days from the toolkit's usage log: tasks answered free and verified, answers
|
|
36
|
+
served from cache, tasks handed back to Claude, and the input and output tokens
|
|
37
|
+
kept off Claude with an estimated saving at haiku rates. The estimate is labelled
|
|
38
|
+
as one: it prices the tokens at the rung the task would otherwise have used.
|
|
39
|
+
|
|
31
40
|
## The limit, printed every time
|
|
32
41
|
|
|
33
42
|
On Claude Code a subagent cannot be sent to a non-Anthropic model. Subagent
|
|
34
43
|
dispatch belongs to the host; the pipeline asks for a persona and the host
|
|
35
44
|
decides what answers. So Phase 1, 2 and 3 personas stay inside the Anthropic
|
|
36
|
-
ladder whatever the rules say, and
|
|
37
|
-
|
|
45
|
+
ladder whatever the rules say, and a free model answers only where the pipeline
|
|
46
|
+
makes the call itself - `bulk-read.sh` and the `delegate` call site - and only
|
|
47
|
+
after the toolkit verified its answer against the source.
|
|
38
48
|
|
|
39
49
|
This paragraph is output, not documentation, because the alternative is the
|
|
40
50
|
question arriving later as "routing is on but the reviewer is still on Opus, is
|
|
@@ -189,6 +189,7 @@ The Step 2 discovery summary is never the end of setup. Immediately after showin
|
|
|
189
189
|
10. `jenkins`
|
|
190
190
|
11. App Store Connect block (Step 3b; only when the target repo is iOS)
|
|
191
191
|
12. Google Play block (Step 3d; only when the target repo is Android)
|
|
192
|
+
13. Free-model keys `llm-<id>` (only after `route-on --free`): `$HOME/.claude/multi-agent-refs/setup/free-models.md`
|
|
192
193
|
|
|
193
194
|
Loop rules:
|
|
194
195
|
|