ai-consultants 2.25.0 → 2.25.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CLAUDE.md +11 -1
- package/README.md +2 -2
- package/SKILL.md +2 -2
- package/docs/COST_RATES.md +1 -1
- package/docs/cost_rates.json +1 -1
- package/docs/releases/v2.25.1.md +77 -0
- package/package.json +1 -1
- package/scripts/config.sh +1 -1
- package/scripts/consult_all.sh +4 -17
- package/scripts/lib/common.sh +37 -16
- package/scripts/lib/costs.sh +24 -6
- package/scripts/lib/routing.sh +30 -3
- package/scripts/lib/voting.sh +110 -62
- package/scripts/peer_review.sh +6 -3
- package/scripts/query_gemini.sh +8 -1
package/CLAUDE.md
CHANGED
|
@@ -6,7 +6,7 @@ AI Consultants is a multi-model AI deliberation system that queries up to 11 AI
|
|
|
6
6
|
|
|
7
7
|
**Self-Exclusion**: The invoking agent is automatically excluded from the panel to prevent self-consultation. Claude Code won't query Claude, Codex CLI won't query Codex, etc.
|
|
8
8
|
|
|
9
|
-
**Version**: 2.25.
|
|
9
|
+
**Version**: 2.25.1
|
|
10
10
|
|
|
11
11
|
## Distribution
|
|
12
12
|
|
|
@@ -801,6 +801,16 @@ curl -fsSL https://raw.githubusercontent.com/matteoscurati/ai-consultants/main/s
|
|
|
801
801
|
|
|
802
802
|
## Changelog
|
|
803
803
|
|
|
804
|
+
### v2.25.1
|
|
805
|
+
- **Six defects found by an independent review of the released v2.25.0** (Codex `gpt-5.6-sol`, effort high, read-only via the `sol-reviewer` profile). Five were reproduced locally by executing the shipped functions before accepting them; the bash 3.2 crash was verified by the reviewer against `/bin/bash 3.2.57`.
|
|
806
|
+
- **`calculate_final_score` scored every consultant as a dissenter.** `calculate_weighted_recommendation` emitted `_map_sanitize_key`'s output (spaces and punctuation stripped) as the winner, while `calculate_final_score` compared the raw `.response.approach` against it. Unanimous multiword panels scored 2/10, single-word panels 10/10. Voting now keeps original strings in bash 3.2-compatible indexed arrays. **The shipped voting tests used single-word approaches, which is exactly why this survived every prior review** — when writing a voting fixture, use a realistic multiword approach.
|
|
807
|
+
- **v2.25.0's recursive billing scan was too wide.** It was made recursive to reach `round_N/`, which also captured `peer_review/` — anonymized copies carry the original `tokens_used`, bookkeeping files take the `// 1000` fallback. Now scoped to root + `round_<digits>/`. `_is_consultant_response_file` became a shape check (object, non-empty `.consultant`, object `.response`) instead of a six-name denylist; it gates 16 call sites across voting/synthesis/peer-review/costs, and was verified to still accept error responses, which must stay counted for quorum.
|
|
808
|
+
- **`select_consultants` returned disabled and self-excluded consultants.** Eligibility lived only in `consult_all.sh`'s non-smart branch. Now applied in `routing.sh` where the output is consumed, with `declare -f` guards; custom-agent discovery was extracted to `_list_custom_api_agents` (common.sh) and is shared with `_discover_custom_api_agents`.
|
|
809
|
+
- **Consensus used connected components**, so similarity being non-transitive produced 100% for `alpha beta`/`beta gamma`/`gamma delta`. Replaced with an exact maximum-clique search with branch-and-bound. Worst case is exponential; measured 1s at 11 consultants and 4s at 30, and the panel is bounded by the roster.
|
|
810
|
+
- **The CI caught a test that passed locally for the wrong reason — read this before trusting a local green.** `test_suite.sh` asserted DeepSeek is selected for QUICK_SYNTAX without enabling it; `ENABLE_DEEPSEEK` defaults to false, so once routing honoured `ENABLE_*` the assertion only held for someone whose user config enables it. It passed on the maintainer's machine and failed on Ubuntu. **`npm test` from your own shell is not a clean signal**: `load_user_config` exports `~/.config/ai-consultants/.env` into every suite. Verify with `AI_CONSULTANTS_CONFIG_DIR=<empty dir> npm test`, which is the condition CI runs in. Note `env -i` is NOT the right harness — it strips variables the tests legitimately need and makes `test_context_optimization.sh` fail on unmodified HEAD.
|
|
811
|
+
- **`estimate_tokens ""` read stdin, and it hung the release gate for hours.** The function chose stdin-vs-argument by emptiness (`[[ -z "$text" ]]`), so an explicit empty string was indistinguishable from no argument and fell through to `cat`. `test_suite.sh` calls `estimate_tokens ""`; with stdin closed (CI, background tasks piping from /dev/null) `cat` gets EOF and returns 0 — the "empty string: 0 tokens" assertion passes by accident. With stdin an open pipe (an interactive-ish local gate run) it blocks forever. `release.sh`'s gate wedged for ~10h on exactly this, and an earlier run had left a second orphaned tree hung ~16h. Fixed at the source: decide by `$#`, not emptiness — `estimate_tokens ""` now returns 0 without touching stdin, and only a genuinely arg-less call reads it. Regression test drives it under a never-closing fifo with `run_with_timeout`, so a re-hang is a FAIL not an infinite run. **This is a third hermeticity axis after `LOG_LEVEL` and the user config: a suite can pass or hang depending on whether stdin is closed.** The lesson compounds — several "22/22 passed" reports earlier in this work were real only because their stdin happened to be closed; the gate was never as green as it read.
|
|
812
|
+
- **Deliberately unchanged**: peer review still runs after voting and synthesis, so its scores cannot alter the recommendation the docs call a "refutation gate". That is an architectural decision, not a bug fix. The same review also judged the panel premise itself weak — it measures agreement rather than correctness, and no held-out comparison against a single strong model exists in the repo.
|
|
813
|
+
|
|
804
814
|
### v2.25.0
|
|
805
815
|
- **`tokens_used` was hardcoded to 0, so the cost report priced pipeline metadata and nothing else.** `build_response_metadata` (`lib/common.sh`) emitted a literal `tokens_used: 0` and was the only writer of that field in the repo; `calculate_session_cost` (`lib/costs.sh:248`) multiplies it by the model rate. Structurally zero for all 11 consultants, both transports, since the field existed. **Do not describe this as "every session reported $0.00"** — I did, in three changelogs, and it is wrong: the pipeline files caught by the same unfiltered glob have no `tokens_used` key at all, so they took the `// 1000` fallback at the default rate (~$0.009 each). Measured against HEAD on a realistic three-consultant directory, v2.24 reports **$0.0270**, entirely phantom. The reported number tracked how many pipeline stages ran. Worse in API mode: `run_api_mode_query` (`lib/api_query.sh:154`) called `extract_token_usage`, which computes the right number (verified: extracts 85 from a live Token Plan body), `log_debug`'d it, and dropped it on the floor.
|
|
806
816
|
- **Four paths write response metadata**, which is why the fix is spread out: `process_consultant_response` (codex/cursor/gemini/kimi/minimax/qwen3), direct builder calls (claude/mistral), `run_api_consultant` (glm/deepseek/grok), and the error path. All converge on `build_response_metadata`, so that is where the new `tokens`/`tokens_source` parameters land; the builders thread them through and default to `0`/`unknown` so pre-existing 5-arg calls still work.
|
package/README.md
CHANGED
|
@@ -1,8 +1,8 @@
|
|
|
1
|
-
# AI Consultants v2.25.
|
|
1
|
+
# AI Consultants v2.25.1
|
|
2
2
|
|
|
3
3
|
> **A harness for every question.** A panel of up to 11 frontier models that writes its own playbook per question — fan out, debate to convergence, cross-examine under adversarial review, or run a tournament — and checks its work before it reaches you.
|
|
4
4
|
|
|
5
|
-
[](https://github.com/matteoscurati/ai-consultants)
|
|
6
6
|
[](LICENSE)
|
|
7
7
|
[](https://docs.anthropic.com/en/docs/claude-code/skills)
|
|
8
8
|
[](https://github.com/matteoscurati/ai-consultants)
|
package/SKILL.md
CHANGED
|
@@ -5,10 +5,10 @@ license: MIT
|
|
|
5
5
|
compatibility: Requires bash, jq, and at least 2 AI CLI tools (agy, codex, vibe, etc.). macOS and Linux.
|
|
6
6
|
metadata:
|
|
7
7
|
author: matteoscurati
|
|
8
|
-
version: 2.25.
|
|
8
|
+
version: 2.25.1
|
|
9
9
|
---
|
|
10
10
|
|
|
11
|
-
# AI Consultants v2.25.
|
|
11
|
+
# AI Consultants v2.25.1 - AI Expert Panel
|
|
12
12
|
|
|
13
13
|
**A harness for every question.** Convene a panel of AI "consultants" for coding questions — and let the engine pick *how* they deliberate: a quick read, a convergence loop, an adversarial refutation gate, a tournament, or an exhaustive sweep. Each consultant has a **configurable persona** that shapes its analysis.
|
|
14
14
|
|
package/docs/COST_RATES.md
CHANGED
package/docs/cost_rates.json
CHANGED
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
{
|
|
2
|
-
"version": "2.25.
|
|
2
|
+
"version": "2.25.1",
|
|
3
3
|
"updated": "2026-07-16",
|
|
4
4
|
"models": {
|
|
5
5
|
"_comment_units": "All rates are USD per 1,000 tokens (per-1K). lib/costs.sh computes cost as (token_count / 1000) * rate, so every value here MUST be a per-1K rate = (provider $/1M price) / 1000. Do NOT paste $/MTok figures directly.",
|
|
@@ -0,0 +1,77 @@
|
|
|
1
|
+
# Release v2.25.1
|
|
2
|
+
|
|
3
|
+
**Date:** 2026-07-21
|
|
4
|
+
**Type:** Patch — six defects found by an independent review of v2.25.0
|
|
5
|
+
**Previous:** v2.25.0
|
|
6
|
+
|
|
7
|
+
## Highlights
|
|
8
|
+
|
|
9
|
+
- **The final score said 2/10 even when every consultant agreed.** Fixed.
|
|
10
|
+
- **Peer-review artifacts were being billed as consultations** — a regression introduced by v2.25.0's own cost work.
|
|
11
|
+
- **Smart routing queried consultants you had disabled**, and billed them.
|
|
12
|
+
- Gemini API-mode calls were costed as the wrong model.
|
|
13
|
+
- Peer review crashed on the stock macOS shell.
|
|
14
|
+
- Consensus reported 100% for panels that genuinely disagreed.
|
|
15
|
+
|
|
16
|
+
## What's New
|
|
17
|
+
|
|
18
|
+
### The score was wrong on essentially every consultation
|
|
19
|
+
|
|
20
|
+
The recommendation was emitted as a key with spaces and punctuation stripped, while scoring compared each consultant's original answer against that stripped version. Nothing ever matched, so every consultant was counted as a dissenter.
|
|
21
|
+
|
|
22
|
+
| Answer | Score |
|
|
23
|
+
|---|---|
|
|
24
|
+
| `Use JWT with rotating sessions` | 2/10 |
|
|
25
|
+
| `JWT` | 10/10 |
|
|
26
|
+
|
|
27
|
+
Same panel, same unanimity, same confidence. Real answers are multiword.
|
|
28
|
+
|
|
29
|
+
The same stripping merged genuinely different options: `Use JWT + sessions` and `Use JWT sessions` collapsed into one. Both are fixed, and the voting tests now use realistic multiword answers — using single-word ones is why this survived.
|
|
30
|
+
|
|
31
|
+
### A regression from v2.25.0
|
|
32
|
+
|
|
33
|
+
v2.25.0 made the cost scan recursive so it would reach debate rounds. That also reached everything peer review writes below the response directory: anonymized copies keep the original token counts, and bookkeeping files fall back to 1000 tokens each. A run whose real cost was $0.125 reported $0.244.
|
|
34
|
+
|
|
35
|
+
Billing now covers the responses and the debate rounds and nothing else, and a response is recognised by its structure rather than by its filename.
|
|
36
|
+
|
|
37
|
+
### Disabled consultants were being queried
|
|
38
|
+
|
|
39
|
+
`ENABLE_*` and self-exclusion were applied only on the non-smart path. With `ENABLE_SMART_ROUTING=true`, a consultant you had disabled — or the agent invoking the skill — could still be sent the question and billed. Custom API agents were excluded from smart routing entirely; they are now included.
|
|
40
|
+
|
|
41
|
+
### The rest
|
|
42
|
+
|
|
43
|
+
- **Gemini API mode** sent `GEMINI_API_MODEL` but recorded `GEMINI_MODEL`, mispricing by 2.24x.
|
|
44
|
+
- **Peer review** used two bash 4 constructs under `/bin/bash` 3.2, the stock macOS shell that `doctor` reports as supported.
|
|
45
|
+
- **Consensus** grouped answers transitively, so A resembling B and B resembling C put A and C together even with nothing in common. Groups now require their members to resemble each other, which also prevents a convergence loop ending early on false agreement.
|
|
46
|
+
|
|
47
|
+
## Breaking Changes
|
|
48
|
+
|
|
49
|
+
None. No configuration key, default, or output field changed.
|
|
50
|
+
|
|
51
|
+
Two numbers will move, both because they were wrong: the final score rises on any panel that agreed on a multiword answer, and consensus falls on panels that were being merged transitively.
|
|
52
|
+
|
|
53
|
+
## Upgrade Guide
|
|
54
|
+
|
|
55
|
+
```bash
|
|
56
|
+
# If installed via git clone
|
|
57
|
+
cd ~/.claude/skills/ai-consultants && git pull
|
|
58
|
+
|
|
59
|
+
# If installed via curl | bash
|
|
60
|
+
curl -fsSL https://raw.githubusercontent.com/matteoscurati/ai-consultants/main/scripts/install.sh | bash
|
|
61
|
+
```
|
|
62
|
+
|
|
63
|
+
No configuration changes required.
|
|
64
|
+
|
|
65
|
+
## Verification
|
|
66
|
+
|
|
67
|
+
Each fix carries a regression test observed failing before the change. `npm test` covers 22 suites.
|
|
68
|
+
|
|
69
|
+
One test in this release passed locally and failed in CI: it asserted a consultant was selected without enabling it, and `ENABLE_DEEPSEEK` defaults to false — so it held only for a maintainer whose own config enabled it. Verify with the user config neutralised, which is the condition CI runs in:
|
|
70
|
+
|
|
71
|
+
```bash
|
|
72
|
+
AI_CONSULTANTS_CONFIG_DIR=$(mktemp -d) npm test
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
## Contributors
|
|
76
|
+
|
|
77
|
+
- @matteoscurati
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "ai-consultants",
|
|
3
|
-
"version": "2.25.
|
|
3
|
+
"version": "2.25.1",
|
|
4
4
|
"description": "A harness for every question - a panel of up to 11 AI models that picks its own deliberation strategy per question (convergence, adversarial verification, tournament) and checks its work before answering",
|
|
5
5
|
"bin": {
|
|
6
6
|
"ai-consultants": "bin/ai-consultants"
|
package/scripts/config.sh
CHANGED
package/scripts/consult_all.sh
CHANGED
|
@@ -46,25 +46,12 @@ source "$SCRIPT_DIR/lib/orchestration.sh"
|
|
|
46
46
|
# Discovers custom API agents from environment variables
|
|
47
47
|
# Convention: ENABLE_AGENTNAME=true with AGENTNAME_API_URL set
|
|
48
48
|
_discover_custom_api_agents() {
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
[[ "$
|
|
52
|
-
|
|
53
|
-
local agent_upper="${var#ENABLE_}"
|
|
54
|
-
|
|
55
|
-
# Skip known agents (uses helper from common.sh)
|
|
56
|
-
is_known_agent "$agent_upper" && continue
|
|
57
|
-
|
|
58
|
-
# Check if it has an API URL configured (indicates it's an API agent)
|
|
59
|
-
local url_var="${agent_upper}_API_URL"
|
|
60
|
-
[[ -z "${!url_var:-}" ]] && continue
|
|
61
|
-
|
|
62
|
-
# Add custom API agent with proper case
|
|
63
|
-
local agent_name
|
|
64
|
-
agent_name=$(to_title "$agent_upper")
|
|
49
|
+
local agent_name
|
|
50
|
+
while IFS= read -r agent_name; do
|
|
51
|
+
[[ -n "$agent_name" ]] || continue
|
|
65
52
|
SELECTED_CONSULTANTS+=("$agent_name")
|
|
66
53
|
log_debug "Discovered custom API agent: $agent_name"
|
|
67
|
-
done < <(
|
|
54
|
+
done < <(_list_custom_api_agents | sort -u)
|
|
68
55
|
}
|
|
69
56
|
|
|
70
57
|
# --- Show usage help ---
|
package/scripts/lib/common.sh
CHANGED
|
@@ -302,6 +302,23 @@ is_known_agent() {
|
|
|
302
302
|
return 1
|
|
303
303
|
}
|
|
304
304
|
|
|
305
|
+
# List enabled custom API consultants discovered from the environment.
|
|
306
|
+
# Convention: ENABLE_AGENTNAME=true plus AGENTNAME_API_URL. Known consultants
|
|
307
|
+
# and non-consultant feature flags are excluded by the shared registry.
|
|
308
|
+
_list_custom_api_agents() {
|
|
309
|
+
local var value agent_upper url_var agent_name
|
|
310
|
+
while IFS='=' read -r var value; do
|
|
311
|
+
[[ "$var" == ENABLE_* && "$value" == "true" ]] || continue
|
|
312
|
+
agent_upper="${var#ENABLE_}"
|
|
313
|
+
is_known_agent "$agent_upper" && continue
|
|
314
|
+
|
|
315
|
+
url_var="${agent_upper}_API_URL"
|
|
316
|
+
[[ -n "${!url_var:-}" ]] || continue
|
|
317
|
+
agent_name=$(to_title "$agent_upper")
|
|
318
|
+
printf '%s\n' "$agent_name"
|
|
319
|
+
done < <(env)
|
|
320
|
+
}
|
|
321
|
+
|
|
305
322
|
# =============================================================================
|
|
306
323
|
# CLI/API MODE SWITCHING (v2.6)
|
|
307
324
|
# =============================================================================
|
|
@@ -892,9 +909,16 @@ get_panic_diagnosis() {
|
|
|
892
909
|
# Uses ~4 characters per token approximation for English text
|
|
893
910
|
# Usage: estimate_tokens "text" or echo "text" | estimate_tokens
|
|
894
911
|
estimate_tokens() {
|
|
895
|
-
local text
|
|
896
|
-
|
|
897
|
-
|
|
912
|
+
local text
|
|
913
|
+
# Decide by argument COUNT, not emptiness: `estimate_tokens ""` is an
|
|
914
|
+
# explicit empty string (0 tokens), not a request to read stdin. Keying on
|
|
915
|
+
# `-z` made the two indistinguishable, so an empty-string call fell through
|
|
916
|
+
# to `cat` and blocked forever whenever stdin was an open pipe rather than
|
|
917
|
+
# closed/EOF — a hang that passed in CI (stdin is /dev/null there) and wedged
|
|
918
|
+
# local gate runs. Only a genuinely arg-less call reads stdin.
|
|
919
|
+
if [[ $# -gt 0 ]]; then
|
|
920
|
+
text="$1"
|
|
921
|
+
else
|
|
898
922
|
text=$(cat)
|
|
899
923
|
fi
|
|
900
924
|
local chars
|
|
@@ -1117,22 +1141,19 @@ _kimi_extract_content() {
|
|
|
1117
1141
|
' "$1" 2>/dev/null
|
|
1118
1142
|
}
|
|
1119
1143
|
|
|
1120
|
-
#
|
|
1121
|
-
#
|
|
1122
|
-
#
|
|
1123
|
-
#
|
|
1124
|
-
#
|
|
1125
|
-
# responses dir for consultant data must gate on this. Shared by voting.sh,
|
|
1126
|
-
# peer_review.sh, synthesize.sh.
|
|
1144
|
+
# Validate the response envelope rather than guessing from filenames. Pipeline
|
|
1145
|
+
# metadata and peer-review artifacts can use arbitrary names, while anonymous
|
|
1146
|
+
# copies deliberately remove the consultant identity. Every loop that consumes
|
|
1147
|
+
# consultant responses must gate on the actual shape. Shared by voting.sh,
|
|
1148
|
+
# peer_review.sh, synthesize.sh, and costs.sh.
|
|
1127
1149
|
_is_consultant_response_file() {
|
|
1128
1150
|
local f="$1"
|
|
1129
1151
|
[[ -f "$f" && -s "$f" ]] || return 1
|
|
1130
|
-
|
|
1131
|
-
|
|
1132
|
-
|
|
1133
|
-
|
|
1134
|
-
|
|
1135
|
-
return 0
|
|
1152
|
+
jq -e '
|
|
1153
|
+
type == "object" and
|
|
1154
|
+
(.consultant | type == "string" and length > 0) and
|
|
1155
|
+
(.response | type == "object")
|
|
1156
|
+
' "$f" >/dev/null 2>&1
|
|
1136
1157
|
}
|
|
1137
1158
|
|
|
1138
1159
|
# Extract a concise, ANSI-stripped failure reason from a consultant's captured
|
package/scripts/lib/costs.sh
CHANGED
|
@@ -266,23 +266,41 @@ is_unpriced_model() {
|
|
|
266
266
|
# before v2.25.0 made tokens_used non-zero and turned them into money:
|
|
267
267
|
# - only consultant responses (voting.json and friends carry no tokens_used
|
|
268
268
|
# and would each be priced at the `// 1000` fallback times the default rate);
|
|
269
|
-
# - debate
|
|
270
|
-
#
|
|
269
|
+
# - debate rounds live in `round_N/` subdirectories and are separate billed
|
|
270
|
+
# queries, so those directories must be included without recursing into
|
|
271
|
+
# derived peer-review trees;
|
|
271
272
|
# - a cache hit made no API call, and `<consultant>_escalated.json` is a copy
|
|
272
273
|
# of a file already counted, so both are excluded.
|
|
273
274
|
_billable_response_files() {
|
|
274
275
|
local responses_dir="$1"
|
|
275
276
|
[[ -d "$responses_dir" ]] || return 0
|
|
276
277
|
|
|
277
|
-
local f
|
|
278
|
-
|
|
278
|
+
local f round_dir round_name
|
|
279
|
+
for f in "$responses_dir"/*.json "$responses_dir"/round_*/*.json; do
|
|
279
280
|
[[ -f "$f" && -s "$f" ]] || continue
|
|
280
|
-
|
|
281
|
+
|
|
282
|
+
# Nested billable files are allowed only one level below an exact
|
|
283
|
+
# round_<digits> directory. Peer-review and other derived trees are not
|
|
284
|
+
# provider queries and must never reach the token fallback.
|
|
285
|
+
if [[ "${f%/*}" != "$responses_dir" ]]; then
|
|
286
|
+
round_dir="${f%/*}"
|
|
287
|
+
round_name="${round_dir##*/}"
|
|
288
|
+
case "$round_name" in
|
|
289
|
+
round_[0-9]*)
|
|
290
|
+
case "${round_name#round_}" in
|
|
291
|
+
*[!0-9]*) continue ;;
|
|
292
|
+
esac
|
|
293
|
+
;;
|
|
294
|
+
*) continue ;;
|
|
295
|
+
esac
|
|
296
|
+
fi
|
|
297
|
+
|
|
298
|
+
[[ "${f##*/}" == *_escalated.json ]] && continue
|
|
281
299
|
_is_consultant_response_file "$f" 2>/dev/null || continue
|
|
282
300
|
# A cache hit is a replay of an earlier response, metadata included.
|
|
283
301
|
[[ "$(jq -r '.cache_metadata.from_cache // false' "$f" 2>/dev/null)" == "true" ]] && continue
|
|
284
302
|
printf '%s\n' "$f"
|
|
285
|
-
done
|
|
303
|
+
done
|
|
286
304
|
}
|
|
287
305
|
|
|
288
306
|
# Calculate total session cost from responses
|
package/scripts/lib/routing.sh
CHANGED
|
@@ -183,10 +183,12 @@ select_consultants() {
|
|
|
183
183
|
local min_affinity="${2:-7}"
|
|
184
184
|
local max_consultants="${3:-8}"
|
|
185
185
|
|
|
186
|
-
#
|
|
186
|
+
# Known roster order matches ALL_CONSULTANTS in config.sh. Eligibility is
|
|
187
|
+
# applied here because smart-routing callers consume this output directly.
|
|
187
188
|
local consultants=("Gemini" "Codex" "Mistral" "Cursor" "Kimi" "Claude" "Qwen3" "GLM" "Grok" "DeepSeek" "MiniMax")
|
|
188
189
|
local selected=()
|
|
189
190
|
local scores=()
|
|
191
|
+
local custom_consultants=()
|
|
190
192
|
|
|
191
193
|
# Collect affinities. When capability-aware routing is on, the ELIGIBILITY
|
|
192
194
|
# filter still uses raw category affinity (same panel as before), but the
|
|
@@ -198,6 +200,14 @@ select_consultants() {
|
|
|
198
200
|
_cap_axis=$(get_category_axis "$category")
|
|
199
201
|
fi
|
|
200
202
|
for c in "${consultants[@]}"; do
|
|
203
|
+
local consultant_flag enable_var
|
|
204
|
+
consultant_flag=$(printf '%s' "$c" | tr '[:lower:]' '[:upper:]')
|
|
205
|
+
enable_var="ENABLE_${consultant_flag}"
|
|
206
|
+
[[ "${!enable_var:-true}" == "true" ]] || continue
|
|
207
|
+
if declare -f should_skip_consultant >/dev/null 2>&1 && should_skip_consultant "$consultant_flag"; then
|
|
208
|
+
continue
|
|
209
|
+
fi
|
|
210
|
+
|
|
201
211
|
local score=$(get_affinity "$category" "$c")
|
|
202
212
|
if [[ $score -ge $min_affinity ]]; then
|
|
203
213
|
local rank=$score
|
|
@@ -211,6 +221,15 @@ select_consultants() {
|
|
|
211
221
|
fi
|
|
212
222
|
done
|
|
213
223
|
|
|
224
|
+
# Custom agents have no affinity entry. Their explicit ENABLE_* + API URL
|
|
225
|
+
# configuration is the eligibility signal, so reserve panel slots for them
|
|
226
|
+
# after affinity-ranked known consultants instead of silently omitting them.
|
|
227
|
+
if declare -f _list_custom_api_agents >/dev/null 2>&1; then
|
|
228
|
+
while IFS= read -r c; do
|
|
229
|
+
[[ -n "$c" ]] && custom_consultants+=("$c")
|
|
230
|
+
done < <(_list_custom_api_agents | sort -u)
|
|
231
|
+
fi
|
|
232
|
+
|
|
214
233
|
# Sort by score (simple bubble sort)
|
|
215
234
|
local n=${#selected[@]}
|
|
216
235
|
for ((i=0; i<n-1; i++)); do
|
|
@@ -228,15 +247,23 @@ select_consultants() {
|
|
|
228
247
|
done
|
|
229
248
|
done
|
|
230
249
|
|
|
231
|
-
# Limit to
|
|
250
|
+
# Limit to the requested panel size while keeping room for configured
|
|
251
|
+
# custom agents, which are appended in stable name order.
|
|
232
252
|
local count=0
|
|
253
|
+
local known_limit=$(( max_consultants - ${#custom_consultants[@]} ))
|
|
254
|
+
[[ $known_limit -lt 0 ]] && known_limit=0
|
|
233
255
|
for c in ${selected[@]+"${selected[@]}"}; do
|
|
234
|
-
if [[ $count -ge $
|
|
256
|
+
if [[ $count -ge $known_limit ]]; then
|
|
235
257
|
break
|
|
236
258
|
fi
|
|
237
259
|
echo "$c"
|
|
238
260
|
count=$((count + 1))
|
|
239
261
|
done
|
|
262
|
+
for c in ${custom_consultants[@]+"${custom_consultants[@]}"}; do
|
|
263
|
+
[[ $count -ge $max_consultants ]] && break
|
|
264
|
+
echo "$c"
|
|
265
|
+
count=$((count + 1))
|
|
266
|
+
done
|
|
240
267
|
}
|
|
241
268
|
|
|
242
269
|
# Get the list of consultants as JSON
|
package/scripts/lib/voting.sh
CHANGED
|
@@ -135,6 +135,49 @@ _jaccard_similarity() {
|
|
|
135
135
|
fi
|
|
136
136
|
}
|
|
137
137
|
|
|
138
|
+
# Exact maximum-clique search over the approach-similarity graph. Candidates
|
|
139
|
+
# are space-separated numeric indices; each recursive level keeps only nodes
|
|
140
|
+
# adjacent to every node already chosen. The default panel is at most eleven,
|
|
141
|
+
# so this bounded backtracking stays small while avoiding order-dependent or
|
|
142
|
+
# transitive clustering. Globals are used because Bash 3.2 has no namerefs.
|
|
143
|
+
_CONSENSUS_NEIGHBORS=()
|
|
144
|
+
_CONSENSUS_MAX_CLUSTER=0
|
|
145
|
+
_find_largest_pairwise_cluster() {
|
|
146
|
+
local candidates="$1"
|
|
147
|
+
local chosen_count="$2"
|
|
148
|
+
local candidate remaining_count=0 other next_candidates neighbors
|
|
149
|
+
|
|
150
|
+
if [[ $chosen_count -gt $_CONSENSUS_MAX_CLUSTER ]]; then
|
|
151
|
+
_CONSENSUS_MAX_CLUSTER=$chosen_count
|
|
152
|
+
fi
|
|
153
|
+
|
|
154
|
+
for candidate in $candidates; do
|
|
155
|
+
remaining_count=$((remaining_count + 1))
|
|
156
|
+
done
|
|
157
|
+
[[ $((chosen_count + remaining_count)) -le $_CONSENSUS_MAX_CLUSTER ]] && return
|
|
158
|
+
|
|
159
|
+
while [[ -n "$candidates" ]]; do
|
|
160
|
+
candidate="${candidates%% *}"
|
|
161
|
+
if [[ "$candidates" == *" "* ]]; then
|
|
162
|
+
candidates="${candidates#* }"
|
|
163
|
+
else
|
|
164
|
+
candidates=""
|
|
165
|
+
fi
|
|
166
|
+
|
|
167
|
+
next_candidates=""
|
|
168
|
+
neighbors=" ${_CONSENSUS_NEIGHBORS[$candidate]:-} "
|
|
169
|
+
for other in $candidates; do
|
|
170
|
+
if [[ "$neighbors" == *" $other "* ]]; then
|
|
171
|
+
next_candidates="${next_candidates:+$next_candidates }$other"
|
|
172
|
+
fi
|
|
173
|
+
done
|
|
174
|
+
_find_largest_pairwise_cluster "$next_candidates" $((chosen_count + 1))
|
|
175
|
+
|
|
176
|
+
remaining_count=$((remaining_count - 1))
|
|
177
|
+
[[ $((chosen_count + remaining_count)) -le $_CONSENSUS_MAX_CLUSTER ]] && return
|
|
178
|
+
done
|
|
179
|
+
}
|
|
180
|
+
|
|
138
181
|
# Calculate consensus score based on keyword overlap between approaches
|
|
139
182
|
# Uses Jaccard similarity instead of exact string matching so that
|
|
140
183
|
# "JWT tokens", "JWT-based auth", and "JSON Web Tokens" are recognized as similar.
|
|
@@ -198,7 +241,7 @@ calculate_consensus_score() {
|
|
|
198
241
|
return
|
|
199
242
|
fi
|
|
200
243
|
|
|
201
|
-
# Consensus = size of the LARGEST cluster of
|
|
244
|
+
# Consensus = size of the LARGEST cluster of pairwise-similar approaches
|
|
202
245
|
# (Jaccard >= threshold), as a percentage of all approaches. This asks
|
|
203
246
|
# "does a majority agree?" rather than "is the whole panel mutually similar?"
|
|
204
247
|
# -- the latter is structurally capped low whenever any single consultant
|
|
@@ -208,37 +251,32 @@ calculate_consensus_score() {
|
|
|
208
251
|
# robust fix is a controlled-vocabulary stance field, tracked separately).
|
|
209
252
|
local threshold=15
|
|
210
253
|
|
|
211
|
-
#
|
|
212
|
-
#
|
|
213
|
-
|
|
214
|
-
|
|
215
|
-
|
|
254
|
+
# Build the undirected similarity graph once, then find its exact maximum
|
|
255
|
+
# clique. Connected components are invalid here because similarity is not
|
|
256
|
+
# transitive: A~B and B~C does not imply A~C.
|
|
257
|
+
_CONSENSUS_NEIGHBORS=()
|
|
258
|
+
_CONSENSUS_MAX_CLUSTER=0
|
|
259
|
+
local i j candidates=""
|
|
260
|
+
for ((i=0; i<total; i++)); do
|
|
261
|
+
_CONSENSUS_NEIGHBORS[$i]=""
|
|
262
|
+
candidates="${candidates:+$candidates }$i"
|
|
263
|
+
done
|
|
216
264
|
for ((i=0; i<total; i++)); do
|
|
217
265
|
for ((j=i+1; j<total; j++)); do
|
|
218
266
|
local sim
|
|
219
267
|
sim=$(_jaccard_similarity "${keyword_sets[$i]}" "${keyword_sets[$j]}")
|
|
220
268
|
if [[ $sim -ge $threshold ]]; then
|
|
221
|
-
|
|
222
|
-
|
|
223
|
-
for ((k=0; k<total; k++)); do
|
|
224
|
-
[[ ${cluster[$k]} -eq $cj ]] && cluster[$k]=$ci
|
|
225
|
-
done
|
|
226
|
-
fi
|
|
269
|
+
_CONSENSUS_NEIGHBORS[$i]="${_CONSENSUS_NEIGHBORS[$i]:+${_CONSENSUS_NEIGHBORS[$i]} }$j"
|
|
270
|
+
_CONSENSUS_NEIGHBORS[$j]="${_CONSENSUS_NEIGHBORS[$j]:+${_CONSENSUS_NEIGHBORS[$j]} }$i"
|
|
227
271
|
fi
|
|
228
272
|
done
|
|
229
273
|
done
|
|
230
274
|
|
|
231
|
-
|
|
232
|
-
|
|
233
|
-
|
|
234
|
-
|
|
235
|
-
|
|
236
|
-
for ((i=0; i<total; i++)); do
|
|
237
|
-
lbl=${cluster[$i]}
|
|
238
|
-
csize[$lbl]=$(( csize[$lbl] + 1 ))
|
|
239
|
-
[[ ${csize[$lbl]} -gt $maxc ]] && maxc=${csize[$lbl]}
|
|
240
|
-
done
|
|
241
|
-
echo $(( maxc * 100 / total ))
|
|
275
|
+
_find_largest_pairwise_cluster "$candidates" 0
|
|
276
|
+
local max_cluster=$_CONSENSUS_MAX_CLUSTER
|
|
277
|
+
_CONSENSUS_NEIGHBORS=()
|
|
278
|
+
_CONSENSUS_MAX_CLUSTER=0
|
|
279
|
+
echo $(( max_cluster * 100 / total ))
|
|
242
280
|
}
|
|
243
281
|
|
|
244
282
|
# Determine the consensus level from a score
|
|
@@ -264,51 +302,55 @@ get_consensus_level() {
|
|
|
264
302
|
# Output: JSON with the recommendation
|
|
265
303
|
calculate_weighted_recommendation() {
|
|
266
304
|
local responses_dir="$1"
|
|
267
|
-
|
|
268
|
-
|
|
269
|
-
|
|
270
|
-
map_clear "APPROACH_SUPPORTERS"
|
|
305
|
+
local -a approaches=()
|
|
306
|
+
local -a approach_weights=()
|
|
307
|
+
local -a approach_supporters=()
|
|
271
308
|
|
|
272
309
|
# Resolve the capability-weighting axis once (empty when weighting is off)
|
|
273
310
|
local _cap_axis
|
|
274
311
|
_cap_axis=$(_voting_capability_axis)
|
|
275
312
|
|
|
276
|
-
#
|
|
313
|
+
# Keep the original approach strings as indexed-array values. The generic
|
|
314
|
+
# Bash 3.2 map encodes keys as shell variable names, so using an approach as
|
|
315
|
+
# a map key loses punctuation and whitespace, cannot round-trip into the
|
|
316
|
+
# final score, and can merge distinct recommendations.
|
|
277
317
|
for f in "$responses_dir"/*.json; do
|
|
278
318
|
if _is_consultant_response_file "$f"; then
|
|
279
|
-
local consultant approach confidence
|
|
319
|
+
local consultant approach confidence weight index i
|
|
280
320
|
consultant=$(jq -r '.consultant // "unknown"' "$f" 2>/dev/null)
|
|
281
321
|
approach=$(jq -r '.response.approach // "unknown"' "$f" 2>/dev/null)
|
|
282
322
|
confidence=$(jq -r '.confidence.score // 5' "$f" 2>/dev/null)
|
|
283
323
|
|
|
284
|
-
# Add weight (capability-modulated when ENABLE_CAPABILITY_WEIGHTING)
|
|
285
324
|
weight=$(_effective_vote_weight "$consultant" "$confidence" "$_cap_axis")
|
|
286
|
-
|
|
287
|
-
|
|
325
|
+
index=-1
|
|
326
|
+
for ((i=0; i<${#approaches[@]}; i++)); do
|
|
327
|
+
if [[ "${approaches[$i]}" == "$approach" ]]; then
|
|
328
|
+
index=$i
|
|
329
|
+
break
|
|
330
|
+
fi
|
|
331
|
+
done
|
|
288
332
|
|
|
289
|
-
|
|
290
|
-
|
|
291
|
-
|
|
292
|
-
|
|
333
|
+
if [[ $index -lt 0 ]]; then
|
|
334
|
+
index=${#approaches[@]}
|
|
335
|
+
approaches+=("$approach")
|
|
336
|
+
approach_weights+=("$weight")
|
|
337
|
+
approach_supporters+=("$consultant")
|
|
293
338
|
else
|
|
294
|
-
|
|
339
|
+
approach_weights[$index]=$(( approach_weights[$index] + weight ))
|
|
340
|
+
approach_supporters[$index]="${approach_supporters[$index]},$consultant"
|
|
295
341
|
fi
|
|
296
342
|
fi
|
|
297
343
|
done
|
|
298
344
|
|
|
299
|
-
# Find the approach with the highest weight using map_keys
|
|
300
345
|
local best_approach=""
|
|
301
346
|
local best_weight=0
|
|
302
347
|
local best_supporters=""
|
|
303
|
-
local
|
|
304
|
-
|
|
305
|
-
|
|
306
|
-
|
|
307
|
-
|
|
308
|
-
|
|
309
|
-
best_weight=$weight
|
|
310
|
-
best_approach="$approach"
|
|
311
|
-
best_supporters=$(map_get "APPROACH_SUPPORTERS" "$approach")
|
|
348
|
+
local i
|
|
349
|
+
for ((i=0; i<${#approaches[@]}; i++)); do
|
|
350
|
+
if [[ ${approach_weights[$i]} -gt $best_weight ]]; then
|
|
351
|
+
best_weight=${approach_weights[$i]}
|
|
352
|
+
best_approach="${approaches[$i]}"
|
|
353
|
+
best_supporters="${approach_supporters[$i]}"
|
|
312
354
|
fi
|
|
313
355
|
done
|
|
314
356
|
|
|
@@ -398,30 +440,36 @@ calculate_final_score() {
|
|
|
398
440
|
# Usage: simple_majority_vote <json_responses_dir>
|
|
399
441
|
simple_majority_vote() {
|
|
400
442
|
local responses_dir="$1"
|
|
401
|
-
|
|
402
|
-
|
|
403
|
-
map_clear "MAJORITY_VOTES"
|
|
443
|
+
local -a approaches=()
|
|
444
|
+
local -a votes=()
|
|
404
445
|
|
|
405
446
|
for f in "$responses_dir"/*.json; do
|
|
406
447
|
if _is_consultant_response_file "$f"; then
|
|
407
|
-
local approach
|
|
448
|
+
local approach index i
|
|
408
449
|
approach=$(jq -r '.response.approach // "unknown"' "$f" 2>/dev/null)
|
|
409
|
-
|
|
410
|
-
|
|
450
|
+
index=-1
|
|
451
|
+
for ((i=0; i<${#approaches[@]}; i++)); do
|
|
452
|
+
if [[ "${approaches[$i]}" == "$approach" ]]; then
|
|
453
|
+
index=$i
|
|
454
|
+
break
|
|
455
|
+
fi
|
|
456
|
+
done
|
|
457
|
+
if [[ $index -lt 0 ]]; then
|
|
458
|
+
approaches+=("$approach")
|
|
459
|
+
votes+=(1)
|
|
460
|
+
else
|
|
461
|
+
votes[$index]=$(( votes[$index] + 1 ))
|
|
462
|
+
fi
|
|
411
463
|
fi
|
|
412
464
|
done
|
|
413
465
|
|
|
414
|
-
# Find approach with most votes using map_keys
|
|
415
466
|
local winner=""
|
|
416
467
|
local max_votes=0
|
|
417
|
-
local
|
|
418
|
-
|
|
419
|
-
|
|
420
|
-
|
|
421
|
-
|
|
422
|
-
if [[ $votes -gt $max_votes ]]; then
|
|
423
|
-
max_votes=$votes
|
|
424
|
-
winner="$approach"
|
|
468
|
+
local i
|
|
469
|
+
for ((i=0; i<${#approaches[@]}; i++)); do
|
|
470
|
+
if [[ ${votes[$i]} -gt $max_votes ]]; then
|
|
471
|
+
max_votes=${votes[$i]}
|
|
472
|
+
winner="${approaches[$i]}"
|
|
425
473
|
fi
|
|
426
474
|
done
|
|
427
475
|
|
package/scripts/peer_review.sh
CHANGED
|
@@ -141,7 +141,9 @@ run_consultant_review() {
|
|
|
141
141
|
local prompt
|
|
142
142
|
prompt=$(generate_review_prompt "$(cat "$anonymous_file")")
|
|
143
143
|
|
|
144
|
-
local timeout_var
|
|
144
|
+
local consultant_upper timeout_var
|
|
145
|
+
consultant_upper=$(to_upper "$consultant")
|
|
146
|
+
timeout_var="${consultant_upper}_TIMEOUT_SECONDS"
|
|
145
147
|
local timeout="${!timeout_var:-180}"
|
|
146
148
|
|
|
147
149
|
log_debug "Running peer review by $consultant..."
|
|
@@ -382,8 +384,9 @@ main() {
|
|
|
382
384
|
for reviewer in "${reviewers[@]}"; do
|
|
383
385
|
local review_output="$reviews_dir/review_${reviewer}.json"
|
|
384
386
|
run_consultant_review "$reviewer" "$output_dir/anonymous_responses.md" "$review_output" &
|
|
385
|
-
|
|
386
|
-
|
|
387
|
+
local review_pid=$!
|
|
388
|
+
pids+=("$review_pid")
|
|
389
|
+
log_debug "Started review by $reviewer (PID: $review_pid)"
|
|
387
390
|
done
|
|
388
391
|
|
|
389
392
|
# Wait for all reviews
|
package/scripts/query_gemini.sh
CHANGED
|
@@ -99,7 +99,14 @@ END_TIME=$(get_timestamp_ms)
|
|
|
99
99
|
LATENCY_MS=$((END_TIME - START_TIME))
|
|
100
100
|
|
|
101
101
|
# --- Configuration for response building ---
|
|
102
|
-
|
|
102
|
+
# Record the identifier actually sent to the provider so cost lookup uses the
|
|
103
|
+
# matching rate table entry. CLI and API mode intentionally use different model
|
|
104
|
+
# namespaces.
|
|
105
|
+
if is_api_mode "gemini"; then
|
|
106
|
+
MODEL_USED="$GEMINI_API_MODEL"
|
|
107
|
+
else
|
|
108
|
+
MODEL_USED="$GEMINI_MODEL"
|
|
109
|
+
fi
|
|
103
110
|
PERSONA_NAME=$(get_persona_name "$CONSULTANT_NAME")
|
|
104
111
|
|
|
105
112
|
# --- Post-processing: use shared helper ---
|