ai-consultants 2.24.0 → 2.25.1

package/.env.example

@@ -107,6 +107,9 @@ ENABLE_DEEPSEEK=false
 # CLAUDE_API_URL=https://api.anthropic.com/v1/messages
 # MISTRAL_API_URL=https://api.mistral.ai/v1/chat/completions
 # QWEN3_API_URL=https://dashscope.aliyuncs.com/api/v1/services/aigc/text-generation/generation
+#   Token Plan alternative (OpenAI-compatible, needs QWEN3_FORMAT=openai and a
+#   Token Plan key in QWEN3_API_KEY; pass the FULL path, it is used verbatim):
+#   https://token-plan.ap-southeast-1.maas.aliyuncs.com/compatible-mode/v1/chat/completions
 
 # =============================================================================
 # MODEL CONFIGURATION (June 2026 premium defaults)
@@ -148,6 +151,14 @@ CLAUDE_CMD=claude
 QWEN3_MODEL=qwen3.7-max
 QWEN3_TIMEOUT=180
 QWEN3_CMD=qwen
+# Reasoning effort. API mode ONLY - the qwen CLI has no effort flag, so in CLI
+# mode this is ignored (with a warning); set model.reasoningEffort in
+# ~/.qwen/settings.json instead. Accepted: none|minimal|low|medium|high|xhigh|max
+# (the provider's own enum). The valid subset depends on the model:
+# qwen3.8-max-preview takes all of them EXCEPT 'none' - thinking cannot be
+# disabled on it - and defaults to xhigh. An unsupported combination is
+# rejected by the provider with a 400, not silently ignored.
+# QWEN3_REASONING_EFFORT=
 
 # =============================================================================
 # API-BASED CONSULTANT CONFIGURATION
@@ -174,7 +185,12 @@ MINIMAX_MODEL=MiniMax-M2.7
 MINIMAX_TIMEOUT=180
 MINIMAX_API_URL=https://api.minimax.io/v1/chat/completions
 
-# Response formats for API-based consultants
+# Wire format for a consultant's API mode. Honored since v2.25 (previously
+# declared but never read). Values: google_ai|anthropic|qwen|openai; an
+# unrecognized value falls back to the consultant's default with a warning.
+# Change this only to reach an endpoint that speaks a different protocol than
+# the provider's own - e.g. QWEN3_FORMAT=openai for Qwen Cloud Token Plan,
+# which is OpenAI-compatible while DashScope is not. See docs/RECIPES.md.
 QWEN3_FORMAT=qwen
 GLM_FORMAT=openai
 GROK_FORMAT=openai

package/CLAUDE.md

@@ -6,7 +6,7 @@ AI Consultants is a multi-model AI deliberation system that queries up to 11 AI
 
 **Self-Exclusion**: The invoking agent is automatically excluded from the panel to prevent self-consultation. Claude Code won't query Claude, Codex CLI won't query Codex, etc.
 
-**Version**: 2.24.0
+**Version**: 2.25.1
 
 ## Distribution
 
@@ -801,6 +801,37 @@ curl -fsSL https://raw.githubusercontent.com/matteoscurati/ai-consultants/main/s
 
 ## Changelog
 
+### v2.25.1
+- **Six defects found by an independent review of the released v2.25.0** (Codex `gpt-5.6-sol`, effort high, read-only via the `sol-reviewer` profile). Five were reproduced locally by executing the shipped functions before accepting them; the bash 3.2 crash was verified by the reviewer against `/bin/bash 3.2.57`.
+- **`calculate_final_score` scored every consultant as a dissenter.** `calculate_weighted_recommendation` emitted `_map_sanitize_key`'s output (spaces and punctuation stripped) as the winner, while `calculate_final_score` compared the raw `.response.approach` against it. Unanimous multiword panels scored 2/10, single-word panels 10/10. Voting now keeps original strings in bash 3.2-compatible indexed arrays. **The shipped voting tests used single-word approaches, which is exactly why this survived every prior review** — when writing a voting fixture, use a realistic multiword approach.
+- **v2.25.0's recursive billing scan was too wide.** It was made recursive to reach `round_N/`, which also captured `peer_review/` — anonymized copies carry the original `tokens_used`, bookkeeping files take the `// 1000` fallback. Now scoped to root + `round_<digits>/`. `_is_consultant_response_file` became a shape check (object, non-empty `.consultant`, object `.response`) instead of a six-name denylist; it gates 16 call sites across voting/synthesis/peer-review/costs, and was verified to still accept error responses, which must stay counted for quorum.
+- **`select_consultants` returned disabled and self-excluded consultants.** Eligibility lived only in `consult_all.sh`'s non-smart branch. Now applied in `routing.sh` where the output is consumed, with `declare -f` guards; custom-agent discovery was extracted to `_list_custom_api_agents` (common.sh) and is shared with `_discover_custom_api_agents`.
+- **Consensus used connected components**, so similarity being non-transitive produced 100% for `alpha beta`/`beta gamma`/`gamma delta`. Replaced with an exact maximum-clique search with branch-and-bound. Worst case is exponential; measured 1s at 11 consultants and 4s at 30, and the panel is bounded by the roster.
+- **The CI caught a test that passed locally for the wrong reason — read this before trusting a local green.** `test_suite.sh` asserted DeepSeek is selected for QUICK_SYNTAX without enabling it; `ENABLE_DEEPSEEK` defaults to false, so once routing honoured `ENABLE_*` the assertion only held for someone whose user config enables it. It passed on the maintainer's machine and failed on Ubuntu. **`npm test` from your own shell is not a clean signal**: `load_user_config` exports `~/.config/ai-consultants/.env` into every suite. Verify with `AI_CONSULTANTS_CONFIG_DIR=<empty dir> npm test`, which is the condition CI runs in. Note `env -i` is NOT the right harness — it strips variables the tests legitimately need and makes `test_context_optimization.sh` fail on unmodified HEAD.
+- **`estimate_tokens ""` read stdin, and it hung the release gate for hours.** The function chose stdin-vs-argument by emptiness (`[[ -z "$text" ]]`), so an explicit empty string was indistinguishable from no argument and fell through to `cat`. `test_suite.sh` calls `estimate_tokens ""`; with stdin closed (CI, background tasks piping from /dev/null) `cat` gets EOF and returns 0 — the "empty string: 0 tokens" assertion passes by accident. With stdin an open pipe (an interactive-ish local gate run) it blocks forever. `release.sh`'s gate wedged for ~10h on exactly this, and an earlier run had left a second orphaned tree hung ~16h. Fixed at the source: decide by `$#`, not emptiness — `estimate_tokens ""` now returns 0 without touching stdin, and only a genuinely arg-less call reads it. Regression test drives it under a never-closing fifo with `run_with_timeout`, so a re-hang is a FAIL not an infinite run. **This is a third hermeticity axis after `LOG_LEVEL` and the user config: a suite can pass or hang depending on whether stdin is closed.** The lesson compounds — several "22/22 passed" reports earlier in this work were real only because their stdin happened to be closed; the gate was never as green as it read.
+- **Deliberately unchanged**: peer review still runs after voting and synthesis, so its scores cannot alter the recommendation the docs call a "refutation gate". That is an architectural decision, not a bug fix. The same review also judged the panel premise itself weak — it measures agreement rather than correctness, and no held-out comparison against a single strong model exists in the repo.
+
+### v2.25.0
+- **`tokens_used` was hardcoded to 0, so the cost report priced pipeline metadata and nothing else.** `build_response_metadata` (`lib/common.sh`) emitted a literal `tokens_used: 0` and was the only writer of that field in the repo; `calculate_session_cost` (`lib/costs.sh:248`) multiplies it by the model rate. Structurally zero for all 11 consultants, both transports, since the field existed. **Do not describe this as "every session reported $0.00"** — I did, in three changelogs, and it is wrong: the pipeline files caught by the same unfiltered glob have no `tokens_used` key at all, so they took the `// 1000` fallback at the default rate (~$0.009 each). Measured against HEAD on a realistic three-consultant directory, v2.24 reports **$0.0270**, entirely phantom. The reported number tracked how many pipeline stages ran. Worse in API mode: `run_api_mode_query` (`lib/api_query.sh:154`) called `extract_token_usage`, which computes the right number (verified: extracts 85 from a live Token Plan body), `log_debug`'d it, and dropped it on the floor.
+- **Four paths write response metadata**, which is why the fix is spread out: `process_consultant_response` (codex/cursor/gemini/kimi/minimax/qwen3), direct builder calls (claude/mistral), `run_api_consultant` (glm/deepseek/grok), and the error path. All converge on `build_response_metadata`, so that is where the new `tokens`/`tokens_source` parameters land; the builders thread them through and default to `0`/`unknown` so pre-existing 5-arg calls still work.
+- **The measured split travels in a module variable, `_API_TOKEN_SPLIT`.** `run_api_mode_query` publishes it via `set_api_token_split`; `resolve_response_tokens` reads it; the CALLER clears it via `clear_api_token_split`. The asymmetry is the whole subtlety: `resolve_response_tokens` runs inside `$(...)`, so it can *read* the variable (subshells inherit) but an assignment there dies with the subshell — it cannot clear it. Its callers are plain statements and can. **An earlier cut used a temp-file sidecar justified by the claim that some callers of `run_api_mode_query` run in a command substitution. That claim was false** — all six call sites (`query_{claude,codex,gemini,mistral,qwen3}.sh` and `run_api_consultant`) are plain statements — and it bought a cross-function file lifecycle with five cleanup sites for nothing. Caught by `/code-review max`, which also noted the false comment had been promoted to binding policy in this very file. The real subshell constraint applies to `calculate_session_cost` / `format_cost_caveats`, which ARE always `$(...)` and therefore take their input as arguments.
+- **The provider's prompt/completion SPLIT is kept, not just the total.** `extract_token_split` (api.sh) preserves it and `metadata.tokens_input`/`tokens_output` record it; `calculate_session_cost` prefers them over the 60/40 fallback. Collapsing to a total and re-splitting it 60/40 overstated API-mode cost ~2.7x on a realistic 20000-prompt/500-completion gpt-5.5 call ($0.3075 vs $0.1150) — while still labeling it `measured`, the one value that is supposed to mean "the provider's own figure". The 60/40 guess survives only for CLI mode, which has no split to record.
+- **CLI mode estimates, and says so.** No CLI exposes token counts, and CLI-first is mandatory since v2.21, so leaving CLI at 0 would have fixed almost nothing. `resolve_response_tokens` falls back to `estimate_tokens` (the pre-existing 4-chars-per-token helper at `common.sh:864`, previously unused for this) over prompt + reply. The six `process_consultant_response` call sites pass `"${#FULL_QUERY}"`; claude/mistral/`run_api_consultant` resolve inline. `metadata.tokens_source` records `measured|estimated|unknown` so a consumer can tell the difference — added to both `lib/schema.json` and `docs/JSON_SCHEMA.md`.
+- **Budget impact is narrower than it first looks.** `enforce_budget` is called four times in `consult_all.sh`. The first two (`:345`, `:451`) pass literal `0` as current cost and project from `ESTIMATED_COST`, derived from `CONTEXT_SIZE` — those always worked. Only `:755` and `:862` pass `CURRENT_COST` from `calculate_session_cost`, and that was the zero. Do not describe this as "budget enforcement never triggered"; it did, off estimates.
+- **Five `*_FORMAT` flags were dead since v2.6.** `QWEN3_`/`GLM_`/`GROK_`/`DEEPSEEK_`/`MINIMAX_FORMAT` are declared in `config.sh:174,185,195,205,216` and documented in `.env.example:178-182`, but `get_api_format()` (`common.sh:396`) hardcoded the mapping and `api_query.sh:50` is its only consumer — ten occurrences, all definitions, zero reads. Same shape as the `ENABLE_REFLECTION` removal in v2.23.0, but the opposite call: that capability was covered elsewhere, this one is not — there is no other route to a non-DashScope Qwen endpoint, so deleting the flag would have forced a Qwen-only hardcode. Now honored, with validation against `google_ai|anthropic|qwen|openai` and fallback-with-warning on anything else.
+- **Note the deliberate asymmetry**: a bad `*_FORMAT` falls back to the default; a bad `*_REASONING_EFFORT` fails. Format has a correct per-consultant default to fall back to. Effort does not — silently substituting the model's default when the user asked for `xhigh` *is* the silent-no-op bug the knob exists to avoid.
+- **The effort enum came from the API, not the docs, and the docs were wrong.** Every public write-up on `qwen3.8-max-preview` reported `low|high|xhigh`. A live 400 reports `'reasoning_effort' must be one of: 'none', 'minimal', 'low', 'medium', 'high', 'xhigh', 'max'`. The first implementation allowed five values and would have rejected `minimal`, which the model accepts. The per-model subset is intentionally NOT enforced in shell — `none` is rejected by this model at a second layer (`The value of the enable_thinking parameter is restricted to True`), which is a model fact that belongs to the provider, not to us. Hardcoding per-model facts is what produced the stale `MODEL_USED` bugs also fixed here.
+- **Qwen 3.8 is opt-in and stays that way.** It needs a Token Plan subscription (`https://token-plan.ap-southeast-1.maas.aliyuncs.com/compatible-mode/v1`, its own key, credit billing) and is not on DashScope, so swapping `config.sh:668` would break the Qwen consultant for every user without one. `test_suite.sh` now asserts the negative explicitly (`premium != qwen3.8-max-preview`) so the contract is self-documenting. `references/affinity.json` is untouched **by construction**: affinity and capability axes are keyed on the consultant name, never the model id, so changing `QWEN3_MODEL` implies no affinity change and `test_routing_parity.sh` needed no edit.
+- **No invented cost rate.** `qwen3.8-max-preview` is credit-billed with no published per-token price, so it is catalogued at 0/0 plus a new `unpriced_models` list. Doing nothing was NOT neutral: `calculate_session_cost` never calls `resolve_model_for_cost`, so an absent model falls to the `*)` arm of `get_input_cost_per_1k` (`costs.sh:163`) at 0.005/0.015 — ~4x the real qwen3.7-max input rate, reported confidently. The same fallback sits in `roster_calibrate.sh:34-35`, so a measured cost axis would have been corrupted by a model that is not billed per token at all. Run `run_calibration.sh` only after this entry exists.
+- **Verified live against Token Plan**, not just fixtures: endpoint 200, structured response through the real adapter (confidence 9/10, persona applied — not the confidence-5 stub the Gemini fence bug produced in v2.15.0), `622 measured` tokens in API mode, `311 estimated` in CLI mode via Codex, and a full `consult_all.sh` run reporting `$0.0550 [token counts estimated for 3 of 3]` where it would previously have said `0¢`.
+- **Making `tokens_used` real turned five latent mis-billings into money, all found by `/code-review max` (55 agents, 52 verified findings → 15 defects).** Every one was harmless while the field was hardcoded 0. `calculate_session_cost` globbed `*.json` unfiltered, so `voting.json`/`orchestration.json`/`synthesis.json` were each priced at the `// 1000` fallback times the default rate (measured 2.8x inflation); a cache hit replayed the original response's metadata and billed at full price ($0.52 for zero API calls); debate rounds live in `round_N/` subdirectories the non-recursive glob never saw; and `<consultant>_escalated.json` was left beside the file it had just been copied over, counting one query twice. New `_billable_response_files` is the single gate for all four — **any future loop over the responses dir should use it**, not a bare glob. `format_cost_caveats` additionally counted `unknown` (i.e. failed) responses in the "estimated for N of M" denominator, which made failures read as provider-measured.
+- **The reasoning-effort knob was a silent no-op on the exact path the docs steer users to.** `${AGENT}_REASONING_EFFORT` was resolved inside the `openai` arm of the format switch, but `config.sh:174` defaults `QWEN3_FORMAT=qwen` — so `QWEN3_USE_API=true` + `QWEN3_REASONING_EFFORT=high` against DashScope built a body with no effort field, emitted no warning, and never validated the value. Resolution is now hoisted **above** the switch so every wire format sees it, and the three formats that cannot carry it (`qwen`, `google_ai`, `anthropic`) say so. Related: `validate_reasoning_effort`'s `return 1` propagated to a bare `run_api_mode_query` call under `set -euo pipefail`, killing the query script before `exit_code=$?` — no error envelope, a 0-byte output file, and the consultant silently vanishing from the panel instead of being reported as failed. All five API-capable query scripts now use the explicit `if ! …; then` guard `run_query` already used; this also fixes the same pre-existing hole for network failures.
+- **Cataloguing the credit-billed model at 0/0 fixed the estimator and broke the calibrator.** `roster_calibrate.sh` falls back to `FALLBACK_IN/OUT` only when the catalog lookup *misses*; with a 0/0 entry the lookup now succeeds, so every qwen3.8-max-preview response cost 0.00000000 and the "cheapest = 10" rank-normalizer scored Qwen3 the cheapest consultant on the panel — which `--write` would have persisted into `affinity.json`, preferentially routing work to a model whose real consumption is invisible there. It now sources `costs.sh` and skips unpriced models from the cost axis: no data beats wrong data.
+- **Applying the review fixes introduced four more bugs, every one caught by a test rather than by reading.** Worth recording because they are all the same shape — a change that *looks* applied but is not: (1) the `roster_calibrate.sh` guard called `is_unpriced_model` in a script that did not source `costs.sh`, so it would have been a silent no-op; (2) the shared CLI effort warning fired in API mode too, where the setting is honored, until it was made self-guarding; (3) `warn_effort_ignored_in_cli` used `is_api_mode "$agent" && return 0` — an &&-list that short-circuits returns non-zero as a statement and kills the caller under `set -e`, **the exact v2.22.0 `prompt_value` bug re-created**; and (4) `build_fallback_response` used `$tokens_in` without declaring the local, so under `set -u` **every unstructured reply would have aborted for all 11 consultants**. Only (4) was caught by an existing test — `test_query_kimi.sh`, whose stub happens to return an unstructured envelope — which means the fallback path had no direct coverage at all. `test_response_tokens` now exercises all three builders at both the legacy and the full arity under `set -u`, plus the `set -e` safety of the warning helper.
+- **`${#FULL_QUERY}` is characters, `estimate_tokens` counts bytes.** The estimate summed the two, under-counting multibyte input roughly threefold (`日本語のテスト` is 7 chars, 21 bytes). `resolve_response_tokens` now takes the prompt TEXT and runs both halves through `estimate_tokens`, so the units match.
+- **Three suites were not hermetic against `LOG_LEVEL`, and it read as a flake.** One `npm test` run reported `test_api_transport.sh` failed and could not be reproduced. It is not flaky: `_log` filters by `LOG_LEVEL`, so `log_warn` is silent at `ERROR`, and any assertion matching on warning output fails for that environment and passes everywhere else — deterministic per-environment, exactly the v2.23.0 release-gate shape. Found by sweeping every suite at DEBUG/INFO/WARN/ERROR rather than by re-running and hoping. Two were pre-existing: `test_context_optimization.sh` (3 assertions, **two of them the v2.17.2 secret-exfiltration guards** — a security regression test that depended on the caller's verbosity) and `test_update_clis.sh` (2 assertions). All three now `export LOG_LEVEL=INFO` before sourcing `config.sh`, which honors an ambient value. **If you add an assertion that matches on log output, pin the level in the suite.** Note the earlier "all assertions passed" reading was wrong — it came from a `grep -A40` truncated before the summary line, not from the actual output.
+- **`npm test` = 22 suites.** New `test_api_transport.sh` (41 assertions) exists mainly as a back-compat gate: `get_api_format` for all 11 consultants and a byte-identical assertion on `build_openai_request`, which six consultants share. That assertion was written and confirmed green against the *unmodified* function before either was touched — the md5 of the default body is unchanged end to end.
+
 ### v2.24.0
 - **Stale slash commands are finally removed on install/update, on both entry points.** `scripts/install.sh` and `bin/ai-consultants install` only ever `cp`'d `.claude/commands/*.md` into `~/.claude/commands`, never pruning — so a command deleted from the repo lived on the user's machine indefinitely. Measured on a real installation: **14 `ai-consultants:*.md` installed, 4 of which this repo still ships.** The other ten were dropped by `9ae73e3` (v2.10.0, "command consolidation") and had been dead for 13 releases. Not inert — they stay invocable, and `config-features.md` tells the user to set `ENABLE_REFLECTION`, removed in v2.23.0 and now rejected by `configure --set` with a non-zero exit. New `prune_removed_commands()` in `install.sh` plus an inline equivalent in `bin/ai-consultants`; both scoped to the `ai-consultants:*.md` namespace. **Known and accepted bound**: within that namespace it also removes a user-authored command, because nothing distinguishes it from one of ours deleted upstream. Documented in `CHANGELOG.md` rather than worked around.
 - **`bin/ai-consultants install` was initially missed** — the first cut of the fix landed only in `install.sh`, leaving npx users (the path the shipped `bin` help and `docs/releases/v2.19.2.md` advertise) with exactly the stale commands the fix targeted. Caught by the `/code-review max` pass, not by tests.

package/README.md

@@ -1,8 +1,8 @@
-# AI Consultants v2.24.0
+# AI Consultants v2.25.1
 
 > **A harness for every question.** A panel of up to 11 frontier models that writes its own playbook per question — fan out, debate to convergence, cross-examine under adversarial review, or run a tournament — and checks its work before it reaches you.
 
-[![Version](https://img.shields.io/badge/version-2.24.0-blue.svg)](https://github.com/matteoscurati/ai-consultants)
+[![Version](https://img.shields.io/badge/version-2.25.1-blue.svg)](https://github.com/matteoscurati/ai-consultants)
 [![License](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)
 [![Claude Code Skill](https://img.shields.io/badge/Claude%20Code-Skill-orange.svg)](https://docs.anthropic.com/en/docs/claude-code/skills)
 [![GitHub stars](https://img.shields.io/github/stars/matteoscurati/ai-consultants?style=social)](https://github.com/matteoscurati/ai-consultants)

package/SKILL.md

@@ -5,10 +5,10 @@ license: MIT
 compatibility: Requires bash, jq, and at least 2 AI CLI tools (agy, codex, vibe, etc.). macOS and Linux.
 metadata:
   author: matteoscurati
-  version: 2.24.0
+  version: 2.25.1
 ---
 
-# AI Consultants v2.24.0 - AI Expert Panel
+# AI Consultants v2.25.1 - AI Expert Panel
 
 **A harness for every question.** Convene a panel of AI "consultants" for coding questions — and let the engine pick *how* they deliberate: a quick read, a convergence loop, an adversarial refutation gate, a tournament, or an exhaustive sweep. Each consultant has a **configurable persona** that shapes its analysis.
 

package/docs/COST_RATES.md

@@ -1,4 +1,4 @@
-# Cost Rates - AI Consultants v2.24.0
+# Cost Rates - AI Consultants v2.25.1
 
 This page documents the per-token rates used by the cost tracking system.
 

package/docs/JSON_SCHEMA.md

@@ -32,6 +32,9 @@ The schema is defined in `scripts/lib/schema.json` following JSON Schema Draft-0
   "debate": {...},
   "metadata": {
     "tokens_used": 1500,
+    "tokens_source": "measured",
+    "tokens_input": 1200,
+    "tokens_output": 300,
     "latency_ms": 2345,
     "model_version": "gemini-3.1-pro-preview",
     "timestamp": "2024-01-14T12:34:56Z"
@@ -188,6 +191,9 @@ Present only if Multi-Agent Debate is enabled (round >= 2):
 {
   "metadata": {
     "tokens_used": 1500,
+    "tokens_source": "measured",
+    "tokens_input": 1200,
+    "tokens_output": 300,
     "latency_ms": 2345,
     "model_version": "gemini-3.1-pro-preview",
     "timestamp": "2024-01-14T12:34:56Z"
@@ -197,7 +203,10 @@ Present only if Multi-Agent Debate is enabled (round >= 2):
 
 | Field | Type | Description |
 |-------|------|-------------|
-| `tokens_used` | integer | Tokens consumed (estimate) |
+| `tokens_used` | integer | Tokens consumed. Provider-reported in API mode; locally approximated in CLI mode |
+| `tokens_source` | string | `measured` \| `estimated` \| `unknown` — how `tokens_used` was obtained |
+| `tokens_input` | integer | Prompt tokens, when the provider reported them (API mode only) |
+| `tokens_output` | integer | Completion tokens, when the provider reported them (API mode only) |
 | `latency_ms` | integer | Response time in ms |
 | `model_version` | string | Exact model version |
 | `timestamp` | string | ISO 8601 timestamp |
@@ -225,6 +234,28 @@ Present only if Multi-Agent Debate is enabled (round >= 2):
 "minor" | "moderate" | "major"
 ```
 
+### metadata.tokens_source
+
+```
+"measured" | "estimated" | "unknown"
+```
+
+- `measured` — the provider's own usage figures, available in API mode.
+- `estimated` — a local 4-chars-per-token approximation over prompt + reply.
+  CLI-backed consultants report no token counts, so this is what they get; it
+  is the majority case, since CLI is the default transport.
+- `unknown` — not recorded (error responses, and callers on the pre-v2.25.0
+  metadata signature).
+
+When `tokens_input`/`tokens_output` are present, price on those rather than
+splitting `tokens_used`: output rates run several times input rates and
+consultations are large-context/short-reply, so a fixed split overstates cost
+substantially.
+
+Consumers that add up cost should read this field: a total built partly on
+estimates is not a measurement. `consult_all.sh` discloses the split on its
+session-cost line for that reason.
+
 ## Confidence Score Guidelines
 
 | Score | Meaning |
@@ -297,6 +328,7 @@ jq 'has("consultant") and has("response") and has("confidence")' output.json
   },
   "metadata": {
     "tokens_used": 1250,
+    "tokens_source": "estimated",
     "latency_ms": 1850,
     "model_version": "gemini-3.1-pro-preview",
     "timestamp": "2024-01-14T12:34:56Z"

package/docs/RECIPES.md

@@ -268,3 +268,69 @@ ai-consultants update-clis --dry-run
 For every available variable and its default, see
 [`references/configuration.md`](../references/configuration.md). The executable
 source of truth is [`scripts/config.sh`](../scripts/config.sh).
+
+## Qwen via Qwen Cloud Token Plan (opt-in, preview)
+
+`qwen3.8-max-preview` is **not** served by DashScope/ModelStudio. It is reachable
+only through a Qwen Cloud **Token Plan** subscription, which has its own base URL
+and its own API key. This is opt-in: nothing below is a default, and the panel's
+premium Qwen model remains `qwen3.7-max`.
+
+Understand the tradeoffs before enabling it:
+
+- **Preview, not GA.** Alibaba describes it as continuously evolving. No open
+  weights, no published benchmarks, no stable per-token price.
+- **Billed in prepaid credits**, not per token — and credit consumption varies
+  with reasoning depth. Cost reports therefore exclude it and say so explicitly
+  rather than estimating a figure.
+
+### API mode (the effort knob works here)
+
+```bash
+QWEN3_USE_API=true
+QWEN3_FORMAT=openai      # Token Plan is OpenAI-compatible; DashScope is not
+QWEN3_API_URL=https://token-plan.ap-southeast-1.maas.aliyuncs.com/compatible-mode/v1/chat/completions
+QWEN3_MODEL=qwen3.8-max-preview
+QWEN3_API_KEY=<your Token Plan key>
+QWEN3_REASONING_EFFORT=high    # none|minimal|low|medium|high|xhigh|max; default xhigh
+```
+
+Pass the **full** URL including `/chat/completions` — it is used verbatim. Use
+`QWEN3_API_KEY` for the Token Plan key; there is deliberately no separate
+key variable, so the key always matches whatever `QWEN3_API_URL` points at.
+
+Accepted effort values are the provider's own enum:
+`none|minimal|low|medium|high|xhigh|max`. Verified against the live endpoint on
+2026-07-21 — note this is wider than the `low|high|xhigh` reported in the public
+write-ups. The one value this model rejects is `none`: thinking is always
+enabled and cannot be disabled (`The value of the enable_thinking parameter is
+restricted to True`). Anything unsupported comes back as a 400, never as a
+silently ignored setting.
+
+### CLI mode (the effort knob does NOT work here)
+
+The `qwen` CLI has no reasoning-effort flag. Effort is a user-owned setting, so
+ai-consultants cannot deliver it per call and warns if you set
+`QWEN3_REASONING_EFFORT` in CLI mode. Configure it yourself in
+`~/.qwen/settings.json`:
+
+```json
+{
+  "modelProviders": {
+    "openai": [
+      {
+        "id": "qwen3.8-max-preview",
+        "name": "[Token Plan] qwen3.8-max-preview",
+        "baseUrl": "https://token-plan.ap-southeast-1.maas.aliyuncs.com/compatible-mode/v1",
+        "envKey": "BAILIAN_TOKEN_PLAN_API_KEY",
+        "generationConfig": { "reasoning": { "effort": "high" } }
+      }
+    ]
+  }
+}
+```
+
+**Caveat worth knowing**: if you also set `generationConfig.samplingParams` on an
+OpenAI-compatible provider, Qwen Code ships those keys to the wire verbatim and
+**skips the `reasoning` injection entirely** — so setting both silently gives you
+no effort control at all.

package/docs/cost_rates.json

@@ -1,5 +1,5 @@
 {
-  "version": "2.24.0",
+  "version": "2.25.1",
   "updated": "2026-07-16",
   "models": {
     "_comment_units": "All rates are USD per 1,000 tokens (per-1K). lib/costs.sh computes cost as (token_count / 1000) * rate, so every value here MUST be a per-1K rate = (provider $/1M price) / 1000. Do NOT paste $/MTok figures directly.",
@@ -320,8 +320,16 @@
     "grok-4.3": {
       "input": 0.00125,
       "output": 0.0025
+    },
+    "_comment_unpriced": "Credit-billed models: drawn from a prepaid subscription pool, with no published per-token price. Rated 0 so the estimator does not fabricate a figure - an absent model would otherwise fall through to default_rate (0.005/0.015) and be reported as a confident, wrong number. Every entry here must also be listed in unpriced_models, which is what makes the session report disclose the exclusion instead of implying the model was free.",
+    "qwen3.8-max-preview": {
+      "input": 0,
+      "output": 0
     }
   },
+  "unpriced_models": [
+    "qwen3.8-max-preview"
+  ],
   "consultant_fallbacks": {
     "_comment": "Premium defaults - Updated Jun 2026 (v2.17.0)",
     "claude": "claude-opus-4-8",

package/docs/releases/v2.25.0.md

@@ -0,0 +1,101 @@
+# Release v2.25.0
+
+**Date:** 2026-07-21
+**Type:** Minor — cost reporting starts working, and five settings that never did
+**Previous:** v2.24.0
+
+## Highlights
+
+- **The session cost you were shown was phantom.** Every consultant response carried a hardcoded token count of zero, so none of them contributed anything; the figure came entirely from the pipeline's own metadata files, priced at a fallback rate. A realistic three-consultant run reported $0.0270 without pricing a single consultation.
+- **Five `*_FORMAT` settings had never done anything** since v2.6, despite being documented in `.env.example`. Now honored.
+- **Opt-in Qwen Cloud Token Plan support**, including `qwen3.8-max-preview`. No default changed.
+- **New `QWEN3_REASONING_EFFORT`** for API mode — and it tells you when it cannot apply, instead of pretending.
+- **New `metadata.tokens_source`** field distinguishing a measured token count from an estimated one.
+
+## What's New
+
+### Cost reporting reflects reality
+
+`build_response_metadata` emitted `tokens_used: 0` on every response, and it was the only writer of that field. `calculate_session_cost` multiplies it by the model's rate, so every real consultation contributed exactly nothing.
+
+What you saw instead came from the same loop reaching files it should never have priced: `voting.json` and the other pipeline artifacts carry no token count, so each fell to a 1000-token fallback at the default rate. The reported cost tracked how many pipeline stages had run.
+
+In API mode the correct number was already being computed — `run_api_mode_query` extracted the provider's usage figures, logged them at debug level, and discarded them.
+
+Responses now carry a real count from two sources:
+
+| Transport | Source | Field value |
+|---|---|---|
+| API | the provider's own usage figures | `tokens_source: "measured"` |
+| CLI | local 4-chars-per-token approximation over prompt + reply | `tokens_source: "estimated"` |
+
+No CLI exposes token counts, and CLI is the default transport, so the approximation is unavoidable if the fix is to mean anything. It is labeled rather than hidden, and the session-cost line says what it rests on:
+
+```
+Session cost: $0.0550 [token counts estimated for 3 of 3]
+```
+
+### Settings that were never read
+
+`QWEN3_FORMAT`, `GLM_FORMAT`, `GROK_FORMAT`, `DEEPSEEK_FORMAT` and `MINIMAX_FORMAT` were declared and documented, but the function that picks a consultant's wire format hardcoded the mapping and never consulted them. All five now work. An unrecognized value falls back to the consultant's default with a warning, so a typo cannot produce a malformed request.
+
+This is what makes the Token Plan endpoint reachable: it is OpenAI-compatible, whereas Qwen's own DashScope endpoint is not.
+
+### Qwen Cloud Token Plan (opt-in)
+
+`qwen3.8-max-preview` is reachable only through a Token Plan subscription — separate endpoint, separate key, billed in prepaid credits. It is a preview, with no published benchmarks, no open weights, and no per-token price.
+
+Because of that it is **configuration, not a default**: the premium Qwen model remains `qwen3.7-max`, and a test asserts that explicitly. Setting it up would break nothing for users without a subscription, because nothing points at it unless you do. See the recipe in `docs/RECIPES.md`.
+
+Credit-billed models are catalogued at zero and listed under `unpriced_models`, so the session-cost line names them as excluded. Leaving such a model out of the catalog is not neutral — an unknown model falls through to a default rate roughly 4x the real Qwen premium input rate, and reports it as a confident figure.
+
+### Reasoning effort
+
+`QWEN3_REASONING_EFFORT` accepts the provider's own enum: `none|minimal|low|medium|high|xhigh|max`. Note this is wider than the `low|high|xhigh` that public write-ups report for this model — the list here was read off the API, not the documentation.
+
+**It applies in API mode only.** The `qwen` CLI has no effort flag; the level is a user-owned setting in `~/.qwen/settings.json`. Setting the variable in CLI mode logs a warning pointing at that file, rather than silently doing nothing.
+
+Which values a given model accepts is enforced by the provider, not guessed locally. `qwen3.8-max-preview`, for instance, rejects `none` because it cannot disable thinking — a 400 with a clear message beats a stale hardcoded list.
+
+### Review pass
+
+The change was put through a multi-agent review at maximum depth before release (55 agents, 52 verified findings, 15 distinct defects, all fixed). Nearly all of them were in the same class: making the token count real converted long-standing latent mis-billings into real money.
+
+Pipeline metadata files were priced as phantom consultants. Cache hits — which make no API call — were billed at full price. Debate rounds were invisible. Escalated consultants were counted twice. And the provider's measured prompt/completion split was being collapsed and re-derived with a fixed 60/40 guess, overstating API-mode cost roughly 2.7x while still reporting it as measured. All are fixed, and a single helper now gates every loop over the responses directory so the next one cannot reintroduce them piecemeal.
+
+The review also caught the reasoning-effort knob doing nothing on Qwen's *default* API configuration — the one path the documentation steers users to — which is precisely the failure the feature was designed to avoid.
+
+## Breaking Changes
+
+None in the API sense — no documented behavior is removed and defaults are byte-identical.
+
+Two behavioral changes worth knowing:
+
+- **Reported costs go from zero to real numbers.** If you tuned `MAX_SESSION_COST` against always-zero reports, revisit it. Budget projections before debate and synthesis now include actual accumulated spend; the pre-run checks were driven by a context-size estimate and already worked.
+- **Responses gained `metadata.tokens_source`, and `tokens_input`/`tokens_output` when the provider reports them.** Additive, but consumers parsing response JSON strictly should know.
+
+## Upgrade Guide
+
+```bash
+# If installed via git clone
+cd ~/.claude/skills/ai-consultants && git pull
+
+# If installed via curl | bash
+curl -fsSL https://raw.githubusercontent.com/matteoscurati/ai-consultants/main/scripts/install.sh | bash
+```
+
+No configuration changes are required. Everything new is opt-in.
+
+To use Qwen 3.8, you need a Qwen Cloud Token Plan subscription and its dedicated API key, then the five settings in the `docs/RECIPES.md` recipe.
+
+## Verification
+
+Verified against a live Token Plan subscription rather than fixtures alone: a structured response through the real adapter (confidence 9, persona applied), `622` measured tokens in API mode, `311` estimated in CLI mode, and a full consultation reporting `$0.0550` where it would previously have said `0¢`.
+
+A sweep of every suite at all four `LOG_LEVEL` values found three that were not hermetic against it — assertions matching on warning output are silent at `ERROR`, which fails deterministically for that environment while passing everywhere else. Two were pre-existing, and two of the affected assertions were the v2.17.2 secret-exfiltration guards. All three now pin the level.
+
+`npm test` covers 22 suites. The new `test_api_transport.sh` exists mainly as a back-compat gate — its byte-identical assertion on the request builder shared by six consultants was written and confirmed green against the unmodified function before anything changed.
+
+## Contributors
+
+- @matteoscurati

package/docs/releases/v2.25.1.md

@@ -0,0 +1,77 @@
+# Release v2.25.1
+
+**Date:** 2026-07-21
+**Type:** Patch — six defects found by an independent review of v2.25.0
+**Previous:** v2.25.0
+
+## Highlights
+
+- **The final score said 2/10 even when every consultant agreed.** Fixed.
+- **Peer-review artifacts were being billed as consultations** — a regression introduced by v2.25.0's own cost work.
+- **Smart routing queried consultants you had disabled**, and billed them.
+- Gemini API-mode calls were costed as the wrong model.
+- Peer review crashed on the stock macOS shell.
+- Consensus reported 100% for panels that genuinely disagreed.
+
+## What's New
+
+### The score was wrong on essentially every consultation
+
+The recommendation was emitted as a key with spaces and punctuation stripped, while scoring compared each consultant's original answer against that stripped version. Nothing ever matched, so every consultant was counted as a dissenter.
+
+| Answer | Score |
+|---|---|
+| `Use JWT with rotating sessions` | 2/10 |
+| `JWT` | 10/10 |
+
+Same panel, same unanimity, same confidence. Real answers are multiword.
+
+The same stripping merged genuinely different options: `Use JWT + sessions` and `Use JWT sessions` collapsed into one. Both are fixed, and the voting tests now use realistic multiword answers — using single-word ones is why this survived.
+
+### A regression from v2.25.0
+
+v2.25.0 made the cost scan recursive so it would reach debate rounds. That also reached everything peer review writes below the response directory: anonymized copies keep the original token counts, and bookkeeping files fall back to 1000 tokens each. A run whose real cost was $0.125 reported $0.244.
+
+Billing now covers the responses and the debate rounds and nothing else, and a response is recognised by its structure rather than by its filename.
+
+### Disabled consultants were being queried
+
+`ENABLE_*` and self-exclusion were applied only on the non-smart path. With `ENABLE_SMART_ROUTING=true`, a consultant you had disabled — or the agent invoking the skill — could still be sent the question and billed. Custom API agents were excluded from smart routing entirely; they are now included.
+
+### The rest
+
+- **Gemini API mode** sent `GEMINI_API_MODEL` but recorded `GEMINI_MODEL`, mispricing by 2.24x.
+- **Peer review** used two bash 4 constructs under `/bin/bash` 3.2, the stock macOS shell that `doctor` reports as supported.
+- **Consensus** grouped answers transitively, so A resembling B and B resembling C put A and C together even with nothing in common. Groups now require their members to resemble each other, which also prevents a convergence loop ending early on false agreement.
+
+## Breaking Changes
+
+None. No configuration key, default, or output field changed.
+
+Two numbers will move, both because they were wrong: the final score rises on any panel that agreed on a multiword answer, and consensus falls on panels that were being merged transitively.
+
+## Upgrade Guide
+
+```bash
+# If installed via git clone
+cd ~/.claude/skills/ai-consultants && git pull
+
+# If installed via curl | bash
+curl -fsSL https://raw.githubusercontent.com/matteoscurati/ai-consultants/main/scripts/install.sh | bash
+```
+
+No configuration changes required.
+
+## Verification
+
+Each fix carries a regression test observed failing before the change. `npm test` covers 22 suites.
+
+One test in this release passed locally and failed in CI: it asserted a consultant was selected without enabling it, and `ENABLE_DEEPSEEK` defaults to false — so it held only for a maintainer whose own config enabled it. Verify with the user config neutralised, which is the condition CI runs in:
+
+```bash
+AI_CONSULTANTS_CONFIG_DIR=$(mktemp -d) npm test
+```
+
+## Contributors
+
+- @matteoscurati

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "ai-consultants",
-  "version": "2.24.0",
+  "version": "2.25.1",
   "description": "A harness for every question - a panel of up to 11 AI models that picks its own deliberation strategy per question (convergence, adversarial verification, tournament) and checks its work before answering",
   "bin": {
     "ai-consultants": "bin/ai-consultants"

package/scripts/config.sh

@@ -861,4 +861,4 @@ EOF
 # VERSION
 # =============================================================================
 
-AI_CONSULTANTS_VERSION="2.24.0"
+AI_CONSULTANTS_VERSION="2.25.1"

package/scripts/consult_all.sh

@@ -46,25 +46,12 @@ source "$SCRIPT_DIR/lib/orchestration.sh"
 # Discovers custom API agents from environment variables
 # Convention: ENABLE_AGENTNAME=true with AGENTNAME_API_URL set
 _discover_custom_api_agents() {
-    while IFS='=' read -r var value; do
-        [[ "$var" != ENABLE_* ]] && continue
-        [[ "$value" != "true" ]] && continue
-
-        local agent_upper="${var#ENABLE_}"
-
-        # Skip known agents (uses helper from common.sh)
-        is_known_agent "$agent_upper" && continue
-
-        # Check if it has an API URL configured (indicates it's an API agent)
-        local url_var="${agent_upper}_API_URL"
-        [[ -z "${!url_var:-}" ]] && continue
-
-        # Add custom API agent with proper case
-        local agent_name
-        agent_name=$(to_title "$agent_upper")
+    local agent_name
+    while IFS= read -r agent_name; do
+        [[ -n "$agent_name" ]] || continue
         SELECTED_CONSULTANTS+=("$agent_name")
         log_debug "Discovered custom API agent: $agent_name"
-    done < <(env)
+    done < <(_list_custom_api_agents | sort -u)
 }
 
 # --- Show usage help ---
@@ -692,9 +679,14 @@ if is_escalation_enabled && [[ $SUCCESS_COUNT -gt 0 ]]; then
                         [[ "$new_confidence" =~ ^[0-9]+$ ]] || new_confidence=0
                         if [[ "$new_confidence" -gt "$original_confidence" ]]; then
                             cp "$escalation_file" "$response_file"
+                            # Remove the copy: leaving it makes one query look
+                            # like two to every glob over this directory -
+                            # cost (now that tokens are real) and voting alike.
+                            rm -f "$escalation_file"
                             log_success "  $consultant escalated: confidence $original_confidence → $new_confidence"
                             ESCALATED_COUNT=$((ESCALATED_COUNT + 1))
                         else
+                            rm -f "$escalation_file"
                             log_debug "  $consultant escalation did not improve confidence"
                         fi
                     fi
@@ -1092,7 +1084,7 @@ log_info "Session saved: $SESSION_ID"
 if [[ "$ENABLE_COST_TRACKING" == "true" ]]; then
     ACTUAL_COST=$(calculate_session_cost "$OUTPUT_DIR")
     track_session_cost "$SESSION_ID" "$ACTUAL_COST"
-    log_info "Session cost: $(format_cost "$ACTUAL_COST")"
+    log_info "Session cost: $(format_cost "$ACTUAL_COST")$(format_cost_caveats "$OUTPUT_DIR")"
 fi
 
 # --- Final Summary ---