ai-consultants 2.17.2 → 2.20.0

package/.claude/commands/ai-consultants:help.md

@@ -13,6 +13,7 @@ All skill details (consultants, personas, presets, strategies, features, configu
 |---------|-------------|
 | `/ai-consultants:consult` | Main consultation — ask AI consultants a coding question |
 | `/ai-consultants:debate` | Run consultation with multi-round debate |
+| `/ai-consultants:roster-audit` | Audit which consultants add distinct value vs echo the panel |
 | `/ai-consultants:help` | Show this help |
 
 ## Quick Examples

package/.claude/commands/ai-consultants:roster-audit.md

@@ -0,0 +1,38 @@
+---
+description: Audit the consultant roster for uncorrelated value — which consultants contribute a distinct approach vs echo the panel (candidates to drop or down-weight). Use when deciding roster composition or after noticing redundant consultations.
+argument-hint: [--recent N] [--threshold PCT] [dir ...]
+allowed-tools: Bash
+---
+
+# AI Consultants - Roster Audit
+
+Measure each consultant's *uncorrelated value* across past consultations: a
+consultant whose approach rarely differs from the rest (low keyword overlap) is
+redundant; one that is often distinct earns its seat on diversity. This is the
+"does it add signal the others miss?" bar, applied to the panel.
+
+**Arguments:** $ARGUMENTS
+
+## Instructions
+
+1. Run the audit (pass `$ARGUMENTS` through; with no args it audits the most
+   recent consultations under the output base):
+   ```bash
+   cd "${AI_CONSULTANTS_DIR:-$HOME/.claude/skills/ai-consultants}" && ./scripts/roster_audit.sh $ARGUMENTS
+   ```
+   - No args → the 20 most recent consultation dirs.
+   - `--recent N` → the N most recent; `--threshold PCT` → correlation cutoff (default 20%).
+   - Or pass explicit consultation output dirs; add `--json` for machine-readable output.
+
+2. **Present the table** as-is (consultant · participated · distinct · distinct% ·
+   verdict), then interpret each verdict:
+   - **unique-value** — pulls its weight on diversity; keep.
+   - **some-value** — contributes occasionally.
+   - **redundant?** — correlated with the panel; candidate to drop or down-weight
+     in `references/affinity.json` (lower its category affinities, or its
+     capability axes when capability routing is enabled).
+   - **insufficient-data** — too few samples; audit more consultations.
+
+3. If the audit prints "nothing to audit" (exit 1), tell the user no consultations
+   with ≥2 responders were found and suggest running some consultations first —
+   each `consult_all.sh` run writes an output directory the audit can read.

package/CLAUDE.md

@@ -6,7 +6,7 @@ AI Consultants is a multi-model AI deliberation system that queries up to 15 AI
 
 **Self-Exclusion**: The invoking agent is automatically excluded from the panel to prevent self-consultation. Claude Code won't query Claude, Codex CLI won't query Codex, etc.
 
-**Version**: 2.17.2
+**Version**: 2.20.0
 
 ## Distribution
 
@@ -39,6 +39,10 @@ ai-consultants/
 │   ├── config.sh               # Centralized configuration
 │   ├── doctor.sh               # Diagnostic and auto-fix tool (v2.2)
 │   ├── peer_review.sh          # Anonymous peer review (v2.2)
+│   ├── roster_audit.sh         # Roster uncorrelated-value audit (v2.20)
+│   ├── roster_calibrate.sh     # Measured capability calibration, Tier A (v2.20)
+│   ├── run_calibration.sh      # Calibration data-collection harness (v2.20)
+│   ├── taste_elo.sh            # Pairwise-judge taste Elo, Tier B (v2.20)
 │   ├── install.sh              # One-liner installer (v2.2)
 │   ├── query_*.sh              # Wrapper for each consultant
 │   ├── query_claude.sh         # Claude consultant (v2.2)
@@ -668,6 +672,10 @@ for f in scripts/*.sh scripts/lib/*.sh; do bash -n "$f" && echo "OK: $f"; done
 | `ENABLE_COST_AWARE_ROUTING` | false | Cost-based model routing (v2.3) |
 | `ENABLE_DEBATE_OPTIMIZATION` | false | Skip debate if all agree (v2.3, opt-in) |
 | `ENABLE_COMPACT_REPORT` | true | Compact report format (v2.3) |
+| `ENABLE_CAPABILITY_WEIGHTING` | false | Capability-weighted voting (v2.20, opt-in) |
+| `ENABLE_CAPABILITY_ROUTING` | false | Capability-aware panel composition (v2.20, opt-in) |
+| `CAPABILITY_WEIGHT_STRENGTH` | 10 | Vote-weight modulation: conf*(S+cap)/S (v2.20) |
+| `CAPABILITY_DEFAULT` | 5 | Fallback capability for a missing consultant/axis (v2.20) |
 | `ENABLE_BUDGET_LIMIT` | false | Budget enforcement (v2.4, opt-in) |
 | `BUDGET_ACTION` | warn | Action on budget exceeded: warn/stop (v2.4) |
 | `QWEN3_USE_API` | false | Use DashScope API instead of qwen CLI (v2.7) |
@@ -826,6 +834,48 @@ curl -fsSL https://raw.githubusercontent.com/matteoscurati/ai-consultants/main/s
 
 ## Changelog
 
+### v2.20.0
+- **Capability axes (borrowed from the delegation-kit cost/intelligence/taste table).** `references/affinity.json` → v1.1: new `capabilities` (per-consultant {intelligence, taste, cost}, 1-10), `category_axis` (category → the quality axis it stresses: taste for API_DESIGN/ARCHITECTURE/CODE_REVIEW/GENERAL, intelligence otherwise), `capability_default`. New `lib/routing.sh::get_capability` / `get_category_axis` (cached like `get_affinity`).
+- **Capability-weighted voting** (`ENABLE_CAPABILITY_WEIGHTING`, opt-in): `lib/voting.sh::_effective_vote_weight` modulates a vote by the consultant's capability on the run's axis — `confidence × (S + cap) / S` (`S = CAPABILITY_WEIGHT_STRENGTH`, default 10). Applied in `calculate_weighted_recommendation` and `calculate_final_score` (normalizer uses the same weight → 1-10 scale preserved). Axis from `QUESTION_CATEGORY`. `cost` never weights a vote (tie-break intelligence > taste > cost).
+- **Capability-aware composition** (`ENABLE_CAPABILITY_ROUTING`, opt-in): `select_consultants` keeps the raw-affinity eligibility filter but ranks eligible consultants by `affinity + (cap − capability_default)`, so under a size limit the quality axis reorders who makes the cut.
+- **Roster audit** — `scripts/roster_audit.sh` scores each consultant's "distinct approach" rate (max pairwise Jaccard of approaches < threshold) across consultations; verdicts unique-value / some-value / redundant? / insufficient-data. Wired as `doctor.sh --roster-audit` (short-circuit mode) and the `/ai-consultants:roster-audit` slash command (3 hosts). Reuses `voting.sh` keyword/Jaccard helpers.
+- **Measured calibration** — `scripts/roster_calibrate.sh` (Tier A): intelligence/taste = mean blind peer-review score sliced by `category_axis`, cost = mean observed `tokens_used`×rate (60/40 split, rank-normalized cheapest=10); emits a `capabilities` block (`--json`/`--write`). `scripts/taste_elo.sh` (Tier B): taste via pairwise-judge Elo (pluggable judge — `JUDGE_CLI` / `TASTE_JUDGE_CMD`). `scripts/run_calibration.sh` drives `references/calibration_benchmark.json` (50 questions, 20 taste / 30 intelligence) through the panel with peer review, then calibrates.
+- **Config** (`config.sh`): `ENABLE_CAPABILITY_WEIGHTING`, `ENABLE_CAPABILITY_ROUTING` (both false), `CAPABILITY_WEIGHT_STRENGTH` (10), `CAPABILITY_DEFAULT` (5). **Observability**: `consult_all.sh` records `capability{weighting_enabled, routing_enabled, axis}` in `optimization_metrics.json`.
+- **Tests**: `test_capability_weighting` (17), `test_roster_audit` (6), `test_roster_calibrate` (10), `test_taste_elo` (6) — 12 suites total. All opt-in; with flags off, existing voting/routing/parity tests pass unchanged. shellcheck clean.
+- **Docs**: `SMART_ROUTING.md` (capability + roster-audit + measured-calibration sections), `.env.example`, `references/configuration.md`, env-var table + structure list here. Website feature card added (sync on push).
+- **Seeds are subjective** (Claude/Codex grounded on the model-routing table; the rest heuristic) — the calibration workflow measures them empirically. Fully back-compat.
+
+### v2.19.2
+- **Cost tracking silently lost on a fresh install — fixed.** `lib/costs.sh::track_session_cost` wrote to `$COST_TRACKING_FILE` (`$XDG_DATA_HOME/ai-consultants/costs.json`, i.e. `~/.local/share/...`) without ever creating the parent directory. That dir doesn't exist until something makes it, and `costs.json` is the only artifact stored there — so on a fresh install every session's cost write failed with "No such file or directory" (swallowed under `set -e` in the caller) and cumulative cost tracking never accumulated. Fix: `mkdir -p "$(dirname "$COST_TRACKING_FILE")"` up front; on failure, `log_warn` + `return 0`.
+- **Function split for lock discipline**: `track_session_cost` (outer) now owns dir creation + locking and delegates the read-modify-write to a new inner `_track_session_cost_update`, which runs with the lock held and never propagates failure. Rationale documented inline: the caller in `consult_all.sh` runs under `set -e` *after* every consultant has already been queried and billed, so a bookkeeping failure must degrade to a warning, never abort the run.
+- **Concurrency**: the RMW of `costs.json` was unguarded, so parallel consultations could interleave and lose records or fail the `mv`. Added a portable `mkdir`-based lock (flock is unavailable on macOS) with a bounded wait (50 × 0.1s = 5s); if the lock stays busy it proceeds unlocked with a warning rather than blocking or aborting. Writes now go through `mktemp "${COST_TRACKING_FILE}.XXXXXX"` instead of a fixed `.tmp` sibling, so an unlocked concurrent writer can't clobber another run's temp file (lost record / failed `mv`).
+- **Corrupt-file self-heal**: a corrupt `costs.json` (truncated write, interleaved update) previously failed every future `jq` update forever and never recovered. Now `jq empty` gates the file; on failure it's moved aside to `${COST_TRACKING_FILE}.corrupt` and reset, so tracking recovers on the next session.
+- **Tests**: `scripts/test_suite.sh::test_cost_tracking_resilience` (5 assertions, wired into `main()`) — costs.json created under a missing nested parent dir (fresh-install path), second session accumulates on the existing file (0.25 → 0.75, both exact in binary float), a corrupt costs.json is reset with the new session recorded, and the corrupt original is preserved as `.corrupt`. Calls guarded with `|| true` so a regression surfaces as a FAIL assertion, not a `set -e` suite abort. 8 suites pass; the new assertions are green; shellcheck clean.
+- **No behavioral change for existing installs** whose data dir already exists; the fix only affects the first-run/absent-dir, concurrent, and corrupt-file paths. Cost bookkeeping remains best-effort by design.
+
+### v2.19.1
+- Cleanup from `/code-review max` on v2.19.0 (the workflow ran degraded under rate limits but surfaced these once it completed; all confirmed inline):
+  - **Report-table mangling fixed**: a failure reason containing a `|` (e.g. a CLI error mentioning a piped command) broke the "Diagnosed Failures" markdown row. The new `lib/common.sh::render_diagnosed_failure <entry> [console|table]` escapes `|`→`\|` in table mode. (Not a security injection — the text is the user's own CLI stderr — but a real rendering bug.)
+  - **DRY**: that helper is now the single source of the `name|reason` decode; the console log (quorum FAILED branch) and the report table both call it, so a future delimiter/encoding change touches one place instead of two.
+  - **Health gate is cache-aware**: it no longer pings a consultant whose response is already cached (Round 1 would serve it free via `check_cache`) — it keeps the cached consultant without a billed probe. Avoids the opt-in gate defeating the semantic cache.
+  - **`ping_consultant` takes the already-lowercased id** (callers compute it for the out/err paths anyway), dropping a redundant `to_lower` fork per consultant on both call sites (`doctor --live`, health gate).
+  - **Documented** the health gate's inherent pre-Round-1 startup latency (serial with the run by definition; up to `HEALTH_GATE_TIMEOUT`) in `config.sh` — it's the cost of pruning up front; opt-in + tunable.
+- Tests: `test_render_diagnosed_failure` (4 assertions incl. pipe-escaping); `test_ping_consultant` updated for the lowercased-id signature. 8 suites pass; shellcheck clean. Smoke-verified: report table renders correctly, health gate still prunes + the min-2 guard fires on the pruned count.
+
+### v2.19.0
+- **Quorum grading + "Diagnosed Failures" report** (the #1 pick from the `/workflows` CLI-reliability investigation — chosen for zero transport/billing risk over the CLI→API fallback, which the workflow's adversarial critique showed would emit persona-less confidence-5 stubs that poison voting). After the round-1 collect loop, `grade_quorum <success> <attempted> <min>` (new pure helper in `lib/common.sh`) classifies the run **MET / DEGRADED / FAILED** vs `QUORUM_MIN` (default 2). The report gains an `**Outcome**:` banner and a **## Diagnosed Failures** table that surfaces each dropped consultant's reason (the v2.18.0 `.err` capture) — so a panel that silently shrank to 2/7 is visibly DEGRADED instead of presenting as authoritative. `QUORUM_ACTION=stop` aborts below quorum; default `warn` continues with the banner. Failures are collected in `_surface_consultant_error` (DRY — same call that already logs the reason).
+- **Health gate (`ENABLE_HEALTH_GATE`, opt-in)** — before the run, ping every selected consultant **in parallel** (new `lib/common.sh::ping_consultant`, `HEALTH_GATE_TIMEOUT` default 30s) and drop the non-responsive ones, so installed-but-unauthenticated CLIs (Cursor/Kimi/stale installs) are pruned up front and the quorum/budget checks see the genuinely-working panel. Script-less custom API agents (no `query_*.sh`) return code 2 and are kept (not probeable). Opt-in because it costs one extra tiny query per consultant; it prunes, it does not switch transport.
+- **DRY**: `doctor --live` refactored to use the shared `ping_consultant`; both surfaces now share one probe implementation.
+- **Explicitly NOT done** (per the workflow synthesis): CLI→API automatic fallback (persona-loss vote poisoning; Gemini already auto-resolves), cold-start retry-timeout escalation (stretches the whole round's worst case), warm-up calls, and the transport-abstraction circuit breaker (racy on-disk state, highest blast radius).
+- **Tests**: `test_functions.sh` — `test_grade_quorum` (5 assertions: MET/DEGRADED/FAILED + boundaries) and `test_ping_consultant` (3: valid→0, no-output→fail, missing-script→2, via stub query scripts). 8 suites pass; shellcheck clean. End-to-end smoke verified: a forced CLI failure yields FAILED outcome + Diagnosed Failures table with the real reason, and the health gate prunes the dead consultant pre-run.
+
+### v2.18.0
+- **Failures are now diagnosable, not silent (Fix A).** `consult_all.sh` launched every consultant with `> /dev/null 2>&1` — discarding stderr — so a failed consultant produced only a bare `Failed`/`Empty response`, with no way to tell *not installed* vs *not authenticated* vs *transient* (this is exactly what led a real session to mis-attribute failures to a timeout). Now each consultant's stderr is captured to `$OUTPUT_DIR/<consultant>.err`, and on FAILED/EMPTY the run surfaces a one-line reason via the new `lib/common.sh::get_consultant_error_reason` helper (ANSI-stripped, prefers an explicit error line, drops orchestration status noise, falls back to "no error captured — CLI likely missing or not authenticated").
+- **`run_query` now embeds the CLI's real error in its failure log.** The per-attempt CLI stderr (`${output_file}.err`, an unpredictable mktemp path the orchestrator can't locate) was only `log_debug`'d. Its first line is now appended to the `Error (code: N)` and `All N attempts failed` warnings, so the actual reason (e.g. `401 Unauthorized`, `command not found`) reaches the captured stderr and the surfaced message.
+- **`doctor.sh --live` (Fix B): real ping per consultant.** The static checks only verify a CLI is installed (`--version`), so `doctor` reports a consultant healthy even when it errors at query time (unauthenticated) — a real source of "All systems healthy" while 3 consultants silently fail. `--live` sends a minimal real query to each *enabled* (and not self-excluded) consultant with a short timeout (`DOCTOR_LIVE_TIMEOUT`, default 45s), and reports ✓/✗ with the captured reason; failures become `doctor` issues (exit 1). Opt-in (costs one tiny query each). Standalone short-circuit mode like `--suggest-preset`.
+- **Tests**: `test_functions.sh::test_get_consultant_error_reason` (5 assertions: explicit-error line, embedded-auth reason, orchestration-noise-not-mistaken, empty/missing file). 8 suites pass; shellcheck clean (CI invocation).
+- **Not fixable in code (documented for users)**: the underlying CLI auth/install state, and cold-start/warm-up retry effects, are environment issues — `--live` *surfaces* them but can't authenticate a CLI for you.
+
 ### v2.17.2
 - **SECURITY: revert v2.17.1's `allow_absolute=true` for context files — it opened a secret-exfiltration surface (caught in `/code-review max`).** v2.17.1 fixed the macOS `/private/tmp` drop by letting `build_context.sh` accept *any* absolute context path behind `validate_file_path`'s prefix-only blocklist (`/etc /root /var/log /proc /sys /dev`). That blocklist covers no home secrets, so a context arg of `~/.ssh/id_rsa`, `~/.aws/credentials`, `~/.netrc`, or `~/.config/gh/hosts.yml` was read verbatim and **sent to the external AI providers**. It also matched literal non-canonical prefixes, so `/private/etc/master.passwd` (the real `/etc` on macOS, via the same symlink aliasing) bypassed the `/etc` rule.
 - **Fix (correct altitude — allowlist, not blocklist)**: context files are now accepted only when (a) relative and in-tree (`validate_file_path ... false` — rejects absolute and `..`), or (b) under a recognized temp root via the new `_is_temp_path` helper (`/tmp`, `/private/tmp`, `$TMPDIR`, and the `/private`-prefixed macOS alias of `$TMPDIR`; `..` rejected so a temp prefix can't traverse out). Still fixes the original macOS scratch-file regression (Claude Code writes `/private/tmp/...`) **without** widening to the whole filesystem. Verified: `/tmp`, `/private/tmp`, `$TMPDIR/...`, relative in-tree → accepted; `~/.ssh/id_rsa`, `/etc/passwd`, `/private/etc/master.passwd`, `/tmp/../etc/passwd` → rejected.

package/README.md

@@ -1,8 +1,8 @@
-# AI Consultants v2.17.2
+# AI Consultants v2.20.0
 
 > **A harness for every question.** A panel of up to 15 frontier models that writes its own playbook per question — fan out, debate to convergence, cross-examine under adversarial review, or run a tournament — and checks its work before it reaches you.
 
-[![Version](https://img.shields.io/badge/version-2.17.2-blue.svg)](https://github.com/matteoscurati/ai-consultants)
+[![Version](https://img.shields.io/badge/version-2.20.0-blue.svg)](https://github.com/matteoscurati/ai-consultants)
 [![License](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)
 [![Claude Code Skill](https://img.shields.io/badge/Claude%20Code-Skill-orange.svg)](https://docs.anthropic.com/en/docs/claude-code/skills)
 [![GitHub stars](https://img.shields.io/github/stars/matteoscurati/ai-consultants?style=social)](https://github.com/matteoscurati/ai-consultants)
@@ -574,13 +574,16 @@ Full reference (~150 vars): [`references/configuration.md`](references/configura
 Diagnose, suggest, and fix:
 
 ```bash
-ai-consultants doctor                                          # 22 health checks
+ai-consultants doctor                                          # 22 health checks (CLI installed?)
 ai-consultants doctor --fix                                    # Auto-fix common issues
 ai-consultants doctor --json                                   # JSON for automation
+ai-consultants doctor --live                                   # Real ping per consultant — catches installed-but-unauthenticated CLIs
 ai-consultants doctor --suggest-config                         # Print recommended ENABLE_* based on detected CLIs
 ai-consultants doctor --suggest-preset --question "..."        # Recommend preset + strategy for a question
 ```
 
+> When a consultant fails mid-consultation, the run surfaces the captured reason (e.g. `CLI not found`, `401 Unauthorized`) instead of a bare "Failed", so you can tell *not installed* from *not authenticated* from *transient*.
+
 ---
 
 ## How It Works

package/SKILL.md

@@ -5,10 +5,10 @@ license: MIT
 compatibility: Requires bash, jq, and at least 2 AI CLI tools (agy, codex, vibe, etc.). macOS and Linux.
 metadata:
   author: matteoscurati
-  version: 2.17.2
+  version: 2.20.0
 ---
 
-# AI Consultants v2.17.2 - AI Expert Panel
+# AI Consultants v2.20.0 - AI Expert Panel
 
 **A harness for every question.** Convene a panel of AI "consultants" for coding questions — and let the engine pick *how* they deliberate: a quick read, a convergence loop, an adversarial refutation gate, a tournament, or an exhaustive sweep. Each consultant has a **configurable persona** that shapes its analysis.
 
@@ -130,11 +130,14 @@ See [Full Configuration Reference](references/configuration.md) for all variable
 Diagnose and fix issues:
 
 ```bash
-./scripts/doctor.sh              # Full check
+./scripts/doctor.sh              # Full check (CLI installed?)
 ./scripts/doctor.sh --fix        # Auto-fix
 ./scripts/doctor.sh --json       # JSON output
+./scripts/doctor.sh --live       # Real ping per consultant — catches installed-but-unauthenticated CLIs
 ```
 
+When a consultant fails during a consultation, the run prints the captured reason (e.g. "CLI not found", "401 Unauthorized"). The run is also **graded** MET/DEGRADED/FAILED by how many responded (`QUORUM_MIN`), with a "Diagnosed Failures" section in the report. Set `ENABLE_HEALTH_GATE=true` to ping consultants and drop the dead ones *before* the run.
+
 ## Interpreting Results
 
 | Scenario | Recommendation |

package/docs/COST_RATES.md

@@ -1,4 +1,4 @@
-# Cost Rates - AI Consultants v2.17.2
+# Cost Rates - AI Consultants v2.20.0
 
 This page documents the per-token rates used by the cost tracking system.
 

package/docs/SMART_ROUTING.md

@@ -46,13 +46,21 @@ customization possible without editing bash.
 
 ```json
 {
-  "version": "1.0",
+  "version": "1.1",
   "default_score": 5,
   "general_score": 8,
+  "capability_default": 5,
   "known_consultants": ["Gemini", "Codex", "..."],
   "categories": {
     "CODE_REVIEW": { "Gemini": 7, "Codex": 10, "..." },
     "BUG_DEBUG":   { "..." }
+  },
+  "capabilities": {
+    "Gemini": { "intelligence": 8, "taste": 8, "cost": 6 },
+    "Codex":  { "intelligence": 8, "taste": 5, "cost": 4 }
+  },
+  "category_axis": {
+    "API_DESIGN": "taste", "CODE_REVIEW": "taste", "ALGORITHM": "intelligence"
   }
 }
 ```
@@ -119,6 +127,112 @@ COMPLEXITY_THRESHOLD_MEDIUM=6    # Score 4-6 = use standard models
 
 See [COST_RATES.md](COST_RATES.md) for model pricing by tier.
 
+## Capability-Aware Routing & Voting (v2.20+)
+
+Beyond category *fit* (the affinity matrix), v1.1 of `affinity.json` adds a
+per-consultant **capability** score on three axes (1-10): `intelligence`,
+`taste`, and `cost`. A `category_axis` map names the quality axis each category
+stresses — **taste** for design-shaped work (API_DESIGN, ARCHITECTURE,
+CODE_REVIEW, GENERAL), **intelligence** otherwise.
+
+Two opt-in features consume this:
+
+- **`ENABLE_CAPABILITY_WEIGHTING`** — a consultant's vote weight becomes
+  `confidence × (S + capability) / S` (`S = CAPABILITY_WEIGHT_STRENGTH`, default
+  10), so on a taste-shaped question a high-taste model's opinion counts for more.
+- **`ENABLE_CAPABILITY_ROUTING`** — panel *eligibility* still uses raw affinity,
+  but the *sort rank* becomes `affinity + (capability − CAPABILITY_DEFAULT)`, so
+  under a size limit the quality axis reorders which eligible consultants make
+  the cut.
+
+`cost` is an efficiency axis for budget-aware composition only — **never** a
+vote weight (tie-break order: intelligence > taste > cost). Scores are
+subjective, point-in-time seeds — re-derive them for your own panel. Helpers:
+`get_capability <consultant> <axis>` and `get_category_axis <category>` in
+`lib/routing.sh`.
+
+## Roster Audit — Uncorrelated Value (v2.20+)
+
+Adding a consultant only helps if it says something the others don't. The
+`scripts/roster_audit.sh` tool measures this across past consultations: a
+consultant's approach is *distinct* in a round when its keyword set has low
+Jaccard overlap (`< --threshold`, default 20%) with **every** other consultant
+that round. A consultant that is rarely distinct is correlated with the panel —
+a candidate to drop or down-weight; one that is often distinct earns its seat on
+diversity.
+
+```bash
+# audit specific consultation output dirs
+./scripts/roster_audit.sh /path/to/consultation_dir ...
+
+# or the N most recent under the consultations base
+./scripts/roster_audit.sh --recent 30
+
+# machine-readable
+./scripts/roster_audit.sh --json --recent 30
+```
+
+Read-only; reuses the `voting.sh` keyword/Jaccard machinery. Consultations with
+fewer than two responders are skipped (correlation is undefined). This is the
+model-routing "before a new model earns a row" bar (contribute >=1 thing the
+incumbents miss) applied to the consultation panel instead of a review lane.
+
+## Measured Calibration — Replacing the Heuristic Seeds (v2.20+)
+
+The `capabilities` scores in `affinity.json` ship as subjective seeds. To
+**measure** them instead of guessing, three tools derive each axis from data the
+tool already produces — sliced by the same `category_axis` that consumes them:
+
+| Tool | Derives | From |
+|---|---|---|
+| `scripts/roster_calibrate.sh` | intelligence, taste, cost (**Tier A**) | blind peer-review scores (sliced by axis) + observed `tokens_used` × catalog rate |
+| `scripts/taste_elo.sh` | taste (**Tier B**) | pairwise LLM-as-judge Elo over taste-axis answers (removes peer self-bias) |
+| `references/calibration_benchmark.json` | the question set | 50 questions, 5 per category, balanced across the two axes |
+| `scripts/run_calibration.sh` | the data | runs the benchmark through the panel with `ENABLE_PEER_REVIEW=true`, then calibrates |
+
+### Workflow
+
+```bash
+# 1. Collect data: run the benchmark through the full panel (real calls — costs money).
+./scripts/run_calibration.sh --limit 10 --dry-run   # preview
+./scripts/run_calibration.sh                         # full 50-question run
+
+# 2. Tier A: measured intelligence/taste/cost -> capabilities block
+./scripts/roster_calibrate.sh --recent 50            # print measured block
+./scripts/roster_calibrate.sh --recent 50 --write    # merge into affinity.json (keeps unmeasured cells)
+
+# 3. Tier B (optional): refine taste via pairwise-judge Elo
+./scripts/taste_elo.sh --recent 50 --write
+```
+
+Both `roster_calibrate.sh` and `taste_elo.sh` also accept explicit consultation
+dirs and `--json`, and back up `affinity.json` to `.bak` before `--write`.
+
+### How each axis is measured
+
+- **cost** — mean *observed* `$/response` (`tokens_used` × catalog rate, 60/40
+  input/output split), rank-normalized so the cheapest = 10. This is cost per
+  *task* (verbosity + completion), not per-token sticker price — a terse pricey
+  model can still rank cheap.
+- **intelligence / taste** — mean **blind peer-review** `quality_score` (already
+  1-10), sliced by `category_axis`: intelligence = performance on
+  intelligence-axis categories, taste = on taste-axis categories.
+- **taste (Tier B)** — taste has no ground truth, so `taste_elo.sh` measures it
+  *relative to a chosen judge*. Each pair of answers on a taste-axis question is
+  shown to the judge, which picks the better one on design taste only; wins feed
+  an Elo ranking. The judge is pluggable:
+  - `JUDGE_CLI` — CLI for the built-in judge (default `claude`).
+  - `TASTE_JUDGE_CMD` — external judge `cmd <ctx> <A.json> <B.json>` printing
+    `A`/`B` (used by tests and custom judges).
+
+### Companion: which consultants earn a seat
+
+`roster_audit.sh` (above) answers *who to keep*; calibration answers *how much
+each is worth per axis*. Together they make the roster data-driven: audit to
+prune redundant consultants, calibrate to score the survivors, then enable
+`ENABLE_CAPABILITY_WEIGHTING` / `ENABLE_CAPABILITY_ROUTING` to act on the
+measured scores.
+
 ## Timeout per Category
 
 Timeouts optimized based on category complexity:

package/docs/cost_rates.json

@@ -1,5 +1,5 @@
 {
-  "version": "2.17.2",
+  "version": "2.20.0",
   "updated": "2026-06-24",
   "models": {
     "_comment_units": "All rates are USD per 1,000 tokens (per-1K). lib/costs.sh computes cost as (token_count / 1000) * rate, so every value here MUST be a per-1K rate = (provider $/1M price) / 1000. Do NOT paste $/MTok figures directly.",

package/docs/releases/v2.18.0.md

@@ -0,0 +1,63 @@
+# Release v2.18.0
+
+**Date:** 2026-06-26
+**Type:** Minor — observability (diagnosable failures + live doctor)
+**Previous:** v2.17.2
+
+## Highlights
+
+- **When a consultant fails, you now see *why*** — "CLI not found", "401 Unauthorized", "Timeout", etc. — instead of a bare "Failed".
+- **`doctor.sh --live`** sends a real ping to each enabled consultant, catching CLIs that are installed but not authenticated (which the static check reports as healthy).
+
+## What's New
+
+### Failures tell you the reason
+
+Previously the orchestrator launched every consultant with its output **and stderr** sent to `/dev/null`, so a failing consultant produced only `Failed` / `Empty response` with no detail — making it impossible to distinguish *not installed* from *not authenticated* from a *transient* error without digging through temp files. (In a real session this led to a wrong diagnosis — blaming a timeout for what were actually auth failures.)
+
+Now each consultant's stderr is captured, and on failure the run prints a one-line reason:
+
+```
+  ✗ Cursor: Failed
+    ↳ Cursor: [ERROR] Cursor CLI not found (command: agent)
+  ✗ GLM: Failed
+    ↳ GLM: All 2 attempts failed: 401 Unauthorized
+```
+
+If nothing diagnostic was captured, it says so plainly ("no output and no error captured — CLI likely missing or not authenticated").
+
+### `doctor.sh --live`
+
+The normal `doctor.sh` only checks that a consultant's CLI is *installed* (`--version`). `--live` goes further: it sends a minimal real query to each **enabled** consultant and reports which actually respond.
+
+```bash
+./scripts/doctor.sh --live
+# ✓ Gemini: responded
+# ✗ Cursor: no valid response — Cursor CLI not found (command: agent)
+# ✗ Kimi: no valid response
+# Live result: 1 responding, 2 failing
+```
+
+It's opt-in (one small query per consultant) and honors `DOCTOR_LIVE_TIMEOUT` (default 45s). Failures are reported as `doctor` issues (exit 1).
+
+## Breaking Changes
+
+None.
+
+## Not fixable in code
+
+`--live` and the surfaced reasons *diagnose* a consultant that's missing or unauthenticated — but installing/authenticating that CLI is up to you. Cold-start/warm-up retry effects are also environmental.
+
+## Upgrade Guide
+
+```bash
+cd ~/.claude/skills/ai-consultants && git pull   # or: npx ai-consultants@latest install
+```
+
+## Commits
+
+- `_______` feat: diagnosable consultant failures + doctor --live (observability)
+
+## Contributors
+
+- @matteoscurati

package/docs/releases/v2.19.0.md

@@ -0,0 +1,66 @@
+# Release v2.19.0
+
+**Date:** 2026-06-26
+**Type:** Minor — reliability (quorum grading + health gate)
+**Previous:** v2.18.0
+
+## Highlights
+
+- **A consultation that quietly ran with fewer panelists no longer looks authoritative.** It's now graded **MET / DEGRADED / FAILED**, and the report lists exactly which consultants dropped and why.
+- **Opt-in health gate** prunes non-responsive consultants *before* the run, so you don't spend the full consultation on CLIs that are installed but not authenticated.
+
+These were chosen from a multi-agent investigation of the "flaky CLI" problem; the adversarial review explicitly steered us *away* from an automatic CLI→API fallback (it would emit persona-less, confidence-5 stubs that poison the vote).
+
+## What's New
+
+### Quorum grading + Diagnosed Failures
+
+After the panel responds, the run is graded by how many consultants actually answered, relative to `QUORUM_MIN` (default 2):
+
+- **MET** — everyone responded
+- **DEGRADED** — some failed, but ≥ quorum
+- **FAILED** — below quorum
+
+The report now opens with an `Outcome:` line and, when anything dropped, a **Diagnosed Failures** table:
+
+```
+**Outcome**: DEGRADED (5/7 responded, quorum=2)
+
+## Diagnosed Failures
+| Consultant | Reason |
+|------------|--------|
+| Cursor | Cursor CLI not found (command: agent) |
+| Kimi   | no output and no error captured — CLI likely missing or not authenticated |
+```
+
+`QUORUM_ACTION=stop` aborts a below-quorum run; the default `warn` continues with the banner so nothing silently passes as a full-panel result.
+
+### Health gate (opt-in)
+
+```bash
+ENABLE_HEALTH_GATE=true ai-consultants "your question"
+```
+
+Pings every selected consultant **in parallel** with a tiny real query (`HEALTH_GATE_TIMEOUT`, default 30s) and drops the ones that don't respond before the real run starts — catching installed-but-unauthenticated CLIs and stale installs. It prunes; it does not switch transport. Custom API agents without a query script are kept (not probeable).
+
+## Breaking Changes
+
+None. Both features are additive; `QUORUM_ACTION` defaults to `warn` (banner only) and the health gate is off by default.
+
+## Not included (deliberately)
+
+The investigation rejected, for now: automatic CLI→API fallback (vote-poisoning persona loss), cold-start retry-timeout escalation (stretches the whole round's worst case), warm-up calls (extra billing/OAuth contention), and a transport-abstraction circuit breaker (racy on-disk state).
+
+## Upgrade Guide
+
+```bash
+cd ~/.claude/skills/ai-consultants && git pull   # or: npx ai-consultants@latest install
+```
+
+## Commits
+
+- `_______` feat: quorum grading + diagnosed-failures report + opt-in health gate
+
+## Contributors
+
+- @matteoscurati

package/docs/releases/v2.19.1.md

@@ -0,0 +1,44 @@
+# Release v2.19.1
+
+**Date:** 2026-06-27
+**Type:** Patch — code-review cleanup for v2.19.0
+**Previous:** v2.19.0
+
+## Highlights
+
+- **Fixes the "Diagnosed Failures" report table breaking** when a consultant's error reason contains a `|`.
+- **The opt-in health gate no longer wastes a ping** on consultants whose answer is already cached.
+
+## What's New
+
+### Report table escapes pipes
+
+A failure reason containing a `|` (e.g. a CLI error that mentions a piped command) used to mangle the markdown row in the **Diagnosed Failures** table. The reason is now escaped (`|` → `\|`) in table mode. (This was a rendering bug, not a security issue — the text is your own CLI's stderr.)
+
+### Cache-aware health gate
+
+When `ENABLE_HEALTH_GATE=true` and semantic caching is on, a consultant whose response is already cached is kept **without** a ping — Round 1 would serve it for free, so probing it would defeat the cache. The gate only pings consultants it would actually run live.
+
+### Internal cleanup (from `/code-review max`)
+
+- A single `render_diagnosed_failure` helper now renders both the console log and the report table — the `name|reason` decode lives in one place.
+- `ping_consultant` takes the already-lowercased consultant id (callers compute it anyway), removing a redundant `to_lower` fork per consultant.
+- The health gate's inherent pre-run startup latency (it's serial with the consultation by design, up to `HEALTH_GATE_TIMEOUT`) is now documented in `config.sh`.
+
+## Breaking Changes
+
+None.
+
+## Upgrade Guide
+
+```bash
+cd ~/.claude/skills/ai-consultants && git pull   # or: npx ai-consultants@latest install
+```
+
+## Commits
+
+- `_______` fix: escape pipes in diagnosed-failures table + cache-aware health gate + review cleanup
+
+## Contributors
+
+- @matteoscurati

package/docs/releases/v2.19.2.md

@@ -0,0 +1,54 @@
+# Release v2.19.2
+
+**Date:** 2026-07-11
+**Type:** Patch — cost-tracking resilience (fresh-install fix + concurrency + corruption self-heal)
+**Previous:** v2.19.1
+
+## Highlights
+
+- **Fresh-install cost tracking is fixed** — sessions no longer fail to record their cost the first time you run the tool.
+- **A corrupt `costs.json` now self-heals** instead of permanently wedging every future cost update.
+- **Concurrent consultations can't lose cost records** — the update is serialized and uses a unique temp file.
+- **Cost bookkeeping can never abort a run** — every failure path degrades to a warning.
+- No configuration changes; no impact on existing installs whose data dir already exists.
+
+## What's New
+
+### Cost tracking resilience (`lib/costs.sh`)
+
+`track_session_cost` records each consultation's cost to `costs.json` under the XDG data dir (`~/.local/share/ai-consultants/`). Four problems are fixed:
+
+| Problem | Before | After |
+|---|---|---|
+| **Fresh install** | Wrote to `costs.json` without creating its parent dir → every session failed with "No such file or directory"; cost tracking silently never accumulated | `mkdir -p` the parent dir on demand |
+| **Corrupt file** | A malformed `costs.json` failed every future `jq` update, forever | Detected via `jq empty`, moved aside to `costs.json.corrupt`, and reset |
+| **Concurrency** | Unguarded read-modify-write; parallel runs could interleave and lose records | Serialized with a portable `mkdir` lock (5s bounded wait, then proceeds unlocked) |
+| **Temp file** | Fixed `.tmp` sibling shared across runs → clobbered writes / failed `mv` | Unique `mktemp` temp file per write |
+
+All failure paths now log a warning and return success: cost bookkeeping is best-effort and can never abort a consultation. This matters because the caller in `consult_all.sh` runs under `set -e` *after* every consultant has already been queried and billed — a bookkeeping hiccup must not throw away the run's results.
+
+The function is split into an outer `track_session_cost` (directory creation + lock management) and an inner `_track_session_cost_update` (the read-modify-write, run with the lock held).
+
+### Tests
+
+New `test_cost_tracking_resilience` in `scripts/test_suite.sh` (5 assertions, wired into `main()`) covers the fresh-install path (costs.json created under a missing nested parent dir), cross-session accumulation (0.25 → 0.75), and the corrupt-file self-heal including the `.corrupt` backup. All 8 suites pass; shellcheck clean.
+
+## Breaking Changes
+
+None. Backwards-compatible. Installs whose data dir already exists see no behavioral change.
+
+## Upgrade Guide
+
+```bash
+cd ~/.claude/skills/ai-consultants && git pull   # or: npx ai-consultants@latest install
+```
+
+No configuration changes required.
+
+## Commits
+
+- `_______` fix(costs): harden track_session_cost against fresh-install, concurrency, and corruption
+
+## Contributors
+
+- @matteoscurati