ai-consultants 2.19.2 → 2.20.0

package/.claude/commands/ai-consultants:help.md

@@ -13,6 +13,7 @@ All skill details (consultants, personas, presets, strategies, features, configu
 |---------|-------------|
 | `/ai-consultants:consult` | Main consultation — ask AI consultants a coding question |
 | `/ai-consultants:debate` | Run consultation with multi-round debate |
+| `/ai-consultants:roster-audit` | Audit which consultants add distinct value vs echo the panel |
 | `/ai-consultants:help` | Show this help |
 
 ## Quick Examples

package/.claude/commands/ai-consultants:roster-audit.md

@@ -0,0 +1,38 @@
+---
+description: Audit the consultant roster for uncorrelated value — which consultants contribute a distinct approach vs echo the panel (candidates to drop or down-weight). Use when deciding roster composition or after noticing redundant consultations.
+argument-hint: [--recent N] [--threshold PCT] [dir ...]
+allowed-tools: Bash
+---
+
+# AI Consultants - Roster Audit
+
+Measure each consultant's *uncorrelated value* across past consultations: a
+consultant whose approach rarely differs from the rest (low keyword overlap) is
+redundant; one that is often distinct earns its seat on diversity. This is the
+"does it add signal the others miss?" bar, applied to the panel.
+
+**Arguments:** $ARGUMENTS
+
+## Instructions
+
+1. Run the audit (pass `$ARGUMENTS` through; with no args it audits the most
+   recent consultations under the output base):
+   ```bash
+   cd "${AI_CONSULTANTS_DIR:-$HOME/.claude/skills/ai-consultants}" && ./scripts/roster_audit.sh $ARGUMENTS
+   ```
+   - No args → the 20 most recent consultation dirs.
+   - `--recent N` → the N most recent; `--threshold PCT` → correlation cutoff (default 20%).
+   - Or pass explicit consultation output dirs; add `--json` for machine-readable output.
+
+2. **Present the table** as-is (consultant · participated · distinct · distinct% ·
+   verdict), then interpret each verdict:
+   - **unique-value** — pulls its weight on diversity; keep.
+   - **some-value** — contributes occasionally.
+   - **redundant?** — correlated with the panel; candidate to drop or down-weight
+     in `references/affinity.json` (lower its category affinities, or its
+     capability axes when capability routing is enabled).
+   - **insufficient-data** — too few samples; audit more consultations.
+
+3. If the audit prints "nothing to audit" (exit 1), tell the user no consultations
+   with ≥2 responders were found and suggest running some consultations first —
+   each `consult_all.sh` run writes an output directory the audit can read.

package/CLAUDE.md

@@ -6,7 +6,7 @@ AI Consultants is a multi-model AI deliberation system that queries up to 15 AI
 
 **Self-Exclusion**: The invoking agent is automatically excluded from the panel to prevent self-consultation. Claude Code won't query Claude, Codex CLI won't query Codex, etc.
 
-**Version**: 2.19.2
+**Version**: 2.20.0
 
 ## Distribution
 
@@ -39,6 +39,10 @@ ai-consultants/
 │   ├── config.sh               # Centralized configuration
 │   ├── doctor.sh               # Diagnostic and auto-fix tool (v2.2)
 │   ├── peer_review.sh          # Anonymous peer review (v2.2)
+│   ├── roster_audit.sh         # Roster uncorrelated-value audit (v2.20)
+│   ├── roster_calibrate.sh     # Measured capability calibration, Tier A (v2.20)
+│   ├── run_calibration.sh      # Calibration data-collection harness (v2.20)
+│   ├── taste_elo.sh            # Pairwise-judge taste Elo, Tier B (v2.20)
 │   ├── install.sh              # One-liner installer (v2.2)
 │   ├── query_*.sh              # Wrapper for each consultant
 │   ├── query_claude.sh         # Claude consultant (v2.2)
@@ -668,6 +672,10 @@ for f in scripts/*.sh scripts/lib/*.sh; do bash -n "$f" && echo "OK: $f"; done
 | `ENABLE_COST_AWARE_ROUTING` | false | Cost-based model routing (v2.3) |
 | `ENABLE_DEBATE_OPTIMIZATION` | false | Skip debate if all agree (v2.3, opt-in) |
 | `ENABLE_COMPACT_REPORT` | true | Compact report format (v2.3) |
+| `ENABLE_CAPABILITY_WEIGHTING` | false | Capability-weighted voting (v2.20, opt-in) |
+| `ENABLE_CAPABILITY_ROUTING` | false | Capability-aware panel composition (v2.20, opt-in) |
+| `CAPABILITY_WEIGHT_STRENGTH` | 10 | Vote-weight modulation: conf*(S+cap)/S (v2.20) |
+| `CAPABILITY_DEFAULT` | 5 | Fallback capability for a missing consultant/axis (v2.20) |
 | `ENABLE_BUDGET_LIMIT` | false | Budget enforcement (v2.4, opt-in) |
 | `BUDGET_ACTION` | warn | Action on budget exceeded: warn/stop (v2.4) |
 | `QWEN3_USE_API` | false | Use DashScope API instead of qwen CLI (v2.7) |
@@ -826,6 +834,17 @@ curl -fsSL https://raw.githubusercontent.com/matteoscurati/ai-consultants/main/s
 
 ## Changelog
 
+### v2.20.0
+- **Capability axes (borrowed from the delegation-kit cost/intelligence/taste table).** `references/affinity.json` → v1.1: new `capabilities` (per-consultant {intelligence, taste, cost}, 1-10), `category_axis` (category → the quality axis it stresses: taste for API_DESIGN/ARCHITECTURE/CODE_REVIEW/GENERAL, intelligence otherwise), `capability_default`. New `lib/routing.sh::get_capability` / `get_category_axis` (cached like `get_affinity`).
+- **Capability-weighted voting** (`ENABLE_CAPABILITY_WEIGHTING`, opt-in): `lib/voting.sh::_effective_vote_weight` modulates a vote by the consultant's capability on the run's axis — `confidence × (S + cap) / S` (`S = CAPABILITY_WEIGHT_STRENGTH`, default 10). Applied in `calculate_weighted_recommendation` and `calculate_final_score` (normalizer uses the same weight → 1-10 scale preserved). Axis from `QUESTION_CATEGORY`. `cost` never weights a vote (tie-break intelligence > taste > cost).
+- **Capability-aware composition** (`ENABLE_CAPABILITY_ROUTING`, opt-in): `select_consultants` keeps the raw-affinity eligibility filter but ranks eligible consultants by `affinity + (cap − capability_default)`, so under a size limit the quality axis reorders who makes the cut.
+- **Roster audit** — `scripts/roster_audit.sh` scores each consultant's "distinct approach" rate (max pairwise Jaccard of approaches < threshold) across consultations; verdicts unique-value / some-value / redundant? / insufficient-data. Wired as `doctor.sh --roster-audit` (short-circuit mode) and the `/ai-consultants:roster-audit` slash command (3 hosts). Reuses `voting.sh` keyword/Jaccard helpers.
+- **Measured calibration** — `scripts/roster_calibrate.sh` (Tier A): intelligence/taste = mean blind peer-review score sliced by `category_axis`, cost = mean observed `tokens_used`×rate (60/40 split, rank-normalized cheapest=10); emits a `capabilities` block (`--json`/`--write`). `scripts/taste_elo.sh` (Tier B): taste via pairwise-judge Elo (pluggable judge — `JUDGE_CLI` / `TASTE_JUDGE_CMD`). `scripts/run_calibration.sh` drives `references/calibration_benchmark.json` (50 questions, 20 taste / 30 intelligence) through the panel with peer review, then calibrates.
+- **Config** (`config.sh`): `ENABLE_CAPABILITY_WEIGHTING`, `ENABLE_CAPABILITY_ROUTING` (both false), `CAPABILITY_WEIGHT_STRENGTH` (10), `CAPABILITY_DEFAULT` (5). **Observability**: `consult_all.sh` records `capability{weighting_enabled, routing_enabled, axis}` in `optimization_metrics.json`.
+- **Tests**: `test_capability_weighting` (17), `test_roster_audit` (6), `test_roster_calibrate` (10), `test_taste_elo` (6) — 12 suites total. All opt-in; with flags off, existing voting/routing/parity tests pass unchanged. shellcheck clean.
+- **Docs**: `SMART_ROUTING.md` (capability + roster-audit + measured-calibration sections), `.env.example`, `references/configuration.md`, env-var table + structure list here. Website feature card added (sync on push).
+- **Seeds are subjective** (Claude/Codex grounded on the model-routing table; the rest heuristic) — the calibration workflow measures them empirically. Fully back-compat.
+
 ### v2.19.2
 - **Cost tracking silently lost on a fresh install — fixed.** `lib/costs.sh::track_session_cost` wrote to `$COST_TRACKING_FILE` (`$XDG_DATA_HOME/ai-consultants/costs.json`, i.e. `~/.local/share/...`) without ever creating the parent directory. That dir doesn't exist until something makes it, and `costs.json` is the only artifact stored there — so on a fresh install every session's cost write failed with "No such file or directory" (swallowed under `set -e` in the caller) and cumulative cost tracking never accumulated. Fix: `mkdir -p "$(dirname "$COST_TRACKING_FILE")"` up front; on failure, `log_warn` + `return 0`.
 - **Function split for lock discipline**: `track_session_cost` (outer) now owns dir creation + locking and delegates the read-modify-write to a new inner `_track_session_cost_update`, which runs with the lock held and never propagates failure. Rationale documented inline: the caller in `consult_all.sh` runs under `set -e` *after* every consultant has already been queried and billed, so a bookkeeping failure must degrade to a warning, never abort the run.

package/README.md

@@ -1,8 +1,8 @@
-# AI Consultants v2.19.2
+# AI Consultants v2.20.0
 
 > **A harness for every question.** A panel of up to 15 frontier models that writes its own playbook per question — fan out, debate to convergence, cross-examine under adversarial review, or run a tournament — and checks its work before it reaches you.
 
-[![Version](https://img.shields.io/badge/version-2.19.2-blue.svg)](https://github.com/matteoscurati/ai-consultants)
+[![Version](https://img.shields.io/badge/version-2.20.0-blue.svg)](https://github.com/matteoscurati/ai-consultants)
 [![License](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)
 [![Claude Code Skill](https://img.shields.io/badge/Claude%20Code-Skill-orange.svg)](https://docs.anthropic.com/en/docs/claude-code/skills)
 [![GitHub stars](https://img.shields.io/github/stars/matteoscurati/ai-consultants?style=social)](https://github.com/matteoscurati/ai-consultants)

package/SKILL.md

@@ -5,10 +5,10 @@ license: MIT
 compatibility: Requires bash, jq, and at least 2 AI CLI tools (agy, codex, vibe, etc.). macOS and Linux.
 metadata:
   author: matteoscurati
-  version: 2.19.2
+  version: 2.20.0
 ---
 
-# AI Consultants v2.19.2 - AI Expert Panel
+# AI Consultants v2.20.0 - AI Expert Panel
 
 **A harness for every question.** Convene a panel of AI "consultants" for coding questions — and let the engine pick *how* they deliberate: a quick read, a convergence loop, an adversarial refutation gate, a tournament, or an exhaustive sweep. Each consultant has a **configurable persona** that shapes its analysis.
 

package/docs/COST_RATES.md

@@ -1,4 +1,4 @@
-# Cost Rates - AI Consultants v2.19.2
+# Cost Rates - AI Consultants v2.20.0
 
 This page documents the per-token rates used by the cost tracking system.
 

package/docs/SMART_ROUTING.md

@@ -46,13 +46,21 @@ customization possible without editing bash.
 
 ```json
 {
-  "version": "1.0",
+  "version": "1.1",
   "default_score": 5,
   "general_score": 8,
+  "capability_default": 5,
   "known_consultants": ["Gemini", "Codex", "..."],
   "categories": {
     "CODE_REVIEW": { "Gemini": 7, "Codex": 10, "..." },
     "BUG_DEBUG":   { "..." }
+  },
+  "capabilities": {
+    "Gemini": { "intelligence": 8, "taste": 8, "cost": 6 },
+    "Codex":  { "intelligence": 8, "taste": 5, "cost": 4 }
+  },
+  "category_axis": {
+    "API_DESIGN": "taste", "CODE_REVIEW": "taste", "ALGORITHM": "intelligence"
   }
 }
 ```
@@ -119,6 +127,112 @@ COMPLEXITY_THRESHOLD_MEDIUM=6    # Score 4-6 = use standard models
 
 See [COST_RATES.md](COST_RATES.md) for model pricing by tier.
 
+## Capability-Aware Routing & Voting (v2.20+)
+
+Beyond category *fit* (the affinity matrix), v1.1 of `affinity.json` adds a
+per-consultant **capability** score on three axes (1-10): `intelligence`,
+`taste`, and `cost`. A `category_axis` map names the quality axis each category
+stresses — **taste** for design-shaped work (API_DESIGN, ARCHITECTURE,
+CODE_REVIEW, GENERAL), **intelligence** otherwise.
+
+Two opt-in features consume this:
+
+- **`ENABLE_CAPABILITY_WEIGHTING`** — a consultant's vote weight becomes
+  `confidence × (S + capability) / S` (`S = CAPABILITY_WEIGHT_STRENGTH`, default
+  10), so on a taste-shaped question a high-taste model's opinion counts for more.
+- **`ENABLE_CAPABILITY_ROUTING`** — panel *eligibility* still uses raw affinity,
+  but the *sort rank* becomes `affinity + (capability − CAPABILITY_DEFAULT)`, so
+  under a size limit the quality axis reorders which eligible consultants make
+  the cut.
+
+`cost` is an efficiency axis for budget-aware composition only — **never** a
+vote weight (tie-break order: intelligence > taste > cost). Scores are
+subjective, point-in-time seeds — re-derive them for your own panel. Helpers:
+`get_capability <consultant> <axis>` and `get_category_axis <category>` in
+`lib/routing.sh`.
+
+## Roster Audit — Uncorrelated Value (v2.20+)
+
+Adding a consultant only helps if it says something the others don't. The
+`scripts/roster_audit.sh` tool measures this across past consultations: a
+consultant's approach is *distinct* in a round when its keyword set has low
+Jaccard overlap (`< --threshold`, default 20%) with **every** other consultant
+that round. A consultant that is rarely distinct is correlated with the panel —
+a candidate to drop or down-weight; one that is often distinct earns its seat on
+diversity.
+
+```bash
+# audit specific consultation output dirs
+./scripts/roster_audit.sh /path/to/consultation_dir ...
+
+# or the N most recent under the consultations base
+./scripts/roster_audit.sh --recent 30
+
+# machine-readable
+./scripts/roster_audit.sh --json --recent 30
+```
+
+Read-only; reuses the `voting.sh` keyword/Jaccard machinery. Consultations with
+fewer than two responders are skipped (correlation is undefined). This is the
+model-routing "before a new model earns a row" bar (contribute >=1 thing the
+incumbents miss) applied to the consultation panel instead of a review lane.
+
+## Measured Calibration — Replacing the Heuristic Seeds (v2.20+)
+
+The `capabilities` scores in `affinity.json` ship as subjective seeds. To
+**measure** them instead of guessing, three tools derive each axis from data the
+tool already produces — sliced by the same `category_axis` that consumes them:
+
+| Tool | Derives | From |
+|---|---|---|
+| `scripts/roster_calibrate.sh` | intelligence, taste, cost (**Tier A**) | blind peer-review scores (sliced by axis) + observed `tokens_used` × catalog rate |
+| `scripts/taste_elo.sh` | taste (**Tier B**) | pairwise LLM-as-judge Elo over taste-axis answers (removes peer self-bias) |
+| `references/calibration_benchmark.json` | the question set | 50 questions, 5 per category, balanced across the two axes |
+| `scripts/run_calibration.sh` | the data | runs the benchmark through the panel with `ENABLE_PEER_REVIEW=true`, then calibrates |
+
+### Workflow
+
+```bash
+# 1. Collect data: run the benchmark through the full panel (real calls — costs money).
+./scripts/run_calibration.sh --limit 10 --dry-run   # preview
+./scripts/run_calibration.sh                         # full 50-question run
+
+# 2. Tier A: measured intelligence/taste/cost -> capabilities block
+./scripts/roster_calibrate.sh --recent 50            # print measured block
+./scripts/roster_calibrate.sh --recent 50 --write    # merge into affinity.json (keeps unmeasured cells)
+
+# 3. Tier B (optional): refine taste via pairwise-judge Elo
+./scripts/taste_elo.sh --recent 50 --write
+```
+
+Both `roster_calibrate.sh` and `taste_elo.sh` also accept explicit consultation
+dirs and `--json`, and back up `affinity.json` to `.bak` before `--write`.
+
+### How each axis is measured
+
+- **cost** — mean *observed* `$/response` (`tokens_used` × catalog rate, 60/40
+  input/output split), rank-normalized so the cheapest = 10. This is cost per
+  *task* (verbosity + completion), not per-token sticker price — a terse pricey
+  model can still rank cheap.
+- **intelligence / taste** — mean **blind peer-review** `quality_score` (already
+  1-10), sliced by `category_axis`: intelligence = performance on
+  intelligence-axis categories, taste = on taste-axis categories.
+- **taste (Tier B)** — taste has no ground truth, so `taste_elo.sh` measures it
+  *relative to a chosen judge*. Each pair of answers on a taste-axis question is
+  shown to the judge, which picks the better one on design taste only; wins feed
+  an Elo ranking. The judge is pluggable:
+  - `JUDGE_CLI` — CLI for the built-in judge (default `claude`).
+  - `TASTE_JUDGE_CMD` — external judge `cmd <ctx> <A.json> <B.json>` printing
+    `A`/`B` (used by tests and custom judges).
+
+### Companion: which consultants earn a seat
+
+`roster_audit.sh` (above) answers *who to keep*; calibration answers *how much
+each is worth per axis*. Together they make the roster data-driven: audit to
+prune redundant consultants, calibrate to score the survivors, then enable
+`ENABLE_CAPABILITY_WEIGHTING` / `ENABLE_CAPABILITY_ROUTING` to act on the
+measured scores.
+
 ## Timeout per Category
 
 Timeouts optimized based on category complexity:

package/docs/cost_rates.json

@@ -1,5 +1,5 @@
 {
-  "version": "2.19.2",
+  "version": "2.20.0",
   "updated": "2026-06-24",
   "models": {
     "_comment_units": "All rates are USD per 1,000 tokens (per-1K). lib/costs.sh computes cost as (token_count / 1000) * rate, so every value here MUST be a per-1K rate = (provider $/1M price) / 1000. Do NOT paste $/MTok figures directly.",

package/docs/releases/v2.20.0.md

@@ -0,0 +1,65 @@
+# Release v2.20.0
+
+**Date:** 2026-07-12
+**Type:** Minor — capability-aware deliberation + measured roster calibration
+**Previous:** v2.19.2
+
+## Highlights
+
+- **Capability-aware voting & composition** — weight votes and rank the panel by each model's capability on the quality axis a question stresses (intelligence or taste). Opt-in.
+- **Roster audit** — see which consultants add distinct value vs merely echo the panel.
+- **Measured calibration** — derive the capability scores from data (blind peer review + observed cost + pairwise-judge Elo) instead of heuristics.
+- Everything new is opt-in and additive: with the flags off, behavior is identical to v2.19.2.
+
+## What's New
+
+### Capability axes (`affinity.json` v1.1)
+
+Beyond category *fit* (the affinity matrix), each consultant now has a `capabilities` score on three axes (1-10) and a `category_axis` map names the quality axis each category stresses — **taste** for API_DESIGN / ARCHITECTURE / CODE_REVIEW / GENERAL, **intelligence** otherwise.
+
+| Flag (default off) | Effect |
+|---|---|
+| `ENABLE_CAPABILITY_WEIGHTING` | A consultant's vote weight becomes `confidence × (S + capability) / S` on the run's axis (`S = CAPABILITY_WEIGHT_STRENGTH`, default 10). |
+| `ENABLE_CAPABILITY_ROUTING` | Panel *eligibility* still uses raw affinity, but eligible consultants are ranked by `affinity + capability`, so under a size limit the quality axis reorders who makes the cut. |
+
+`cost` is a composition/budget axis only — never a vote weight (tie-break order: intelligence > taste > cost).
+
+### Roster audit — uncorrelated value
+
+`scripts/roster_audit.sh` scores each consultant's *distinctiveness* across past consultations: an approach with low keyword overlap versus every other consultant that round is "distinct". A consultant rarely distinct is correlated with the panel (candidate to drop or down-weight); one often distinct earns its seat. Also available as `doctor.sh --roster-audit` and the `/ai-consultants:roster-audit` slash command. Read-only.
+
+### Measured calibration (Tier A + Tier B)
+
+Replace the heuristic capability seeds with measurements, each derived from a signal the tool already records and sliced by the same `category_axis`:
+
+- **`scripts/roster_calibrate.sh` (Tier A)** — intelligence/taste = mean **blind peer-review** score on that axis's categories; cost = mean **observed** `tokens_used` × catalog rate (60/40 split, rank-normalized, cheapest = 10 → cost per *task*, not per token).
+- **`scripts/taste_elo.sh` (Tier B)** — taste via **pairwise LLM-as-judge Elo** on taste-axis answers (removes the peer panel's self-bias). Judge is pluggable: `JUDGE_CLI` (default `claude`) or `TASTE_JUDGE_CMD`.
+- **`scripts/run_calibration.sh` + `references/calibration_benchmark.json`** — 50 questions (20 taste / 30 intelligence) run through the panel with `ENABLE_PEER_REVIEW=true`, then calibrated.
+
+Both calibrators accept `--json` and `--write` (merge measured scores into `affinity.json`, backing it up first).
+
+## Breaking Changes
+
+None. All features are opt-in; with the flags off, behavior is identical to v2.19.2. The shipped `capabilities` scores are subjective seeds (Claude/Codex grounded on the model-routing table, the rest heuristic) until you run the calibration workflow.
+
+## Upgrade Guide
+
+```bash
+cd ~/.claude/skills/ai-consultants && git pull   # or: npx ai-consultants@latest install
+```
+
+Optional — measure capability scores for your own panel:
+
+```bash
+./scripts/run_calibration.sh              # collect data (real calls; --limit to sample)
+./scripts/roster_calibrate.sh --write     # Tier A: intelligence/taste/cost -> affinity.json
+./scripts/taste_elo.sh --write            # Tier B: refine taste (optional)
+```
+
+## Commits
+
+- `_______` feat: capability-aware voting/routing + roster audit + measured calibration (v2.20.0)
+
+## Contributors
+
+- @matteoscurati

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "ai-consultants",
-  "version": "2.19.2",
+  "version": "2.20.0",
   "description": "A harness for every question - a panel of up to 15 AI models that picks its own deliberation strategy per question (convergence, adversarial verification, tournament) and checks its work before answering",
   "bin": {
     "ai-consultants": "bin/ai-consultants"

package/references/affinity.json

@@ -1,8 +1,10 @@
 {
-  "version": "1.0",
+  "version": "1.1",
   "_comment": "Affinity scores per (category, consultant). 10 = perfect match, 1 = poor match. Lookup order: (1) consultant NOT in known_consultants -> default_score (5). (2) category present in 'categories' -> categories[category][consultant] (with cell-missing fallback to default_score=5). (3) category absent -> general_score (8). NOTE the asymmetry: a known consultant in an unknown category gets 8, but in a known category with no cell gets 5. Rationale: a missing cell in a known category is a configuration gap (caught by doctor.sh) and should de-rank the consultant; an unknown category is just unmapped routing and treats all known consultants equally. To override at runtime: AFFINITY_FILE=/path/to/custom.json.",
+  "_comment_capabilities": "v1.1: per-consultant capability axes (1-10, higher=better) in 'capabilities', plus 'category_axis' mapping each question category to the quality axis it stresses. intelligence + taste are QUALITY axes: they weight votes (voting.sh, ENABLE_CAPABILITY_WEIGHTING) and rank the panel (routing.sh, ENABLE_CAPABILITY_ROUTING). cost is an EFFICIENCY axis for budget-aware composition only and is NEVER a vote weight (tie-break order: intelligence > taste > cost). Missing consultant/axis -> capability_default. Both features are OPT-IN and additive: with the flags off, behavior is identical to v1.0. These scores are SUBJECTIVE point-in-time seeds — re-derive them for your own panel (see the model-routing methodology): Claude (claude-opus-4-8) and Codex (gpt-5.5) axes are grounded on the author's model-routing table; the other consultants' axes are heuristic seeds pending measurement. Unmapped category_axis -> intelligence.",
   "default_score": 5,
   "general_score": 8,
+  "capability_default": 5,
   "known_consultants": [
     "Gemini", "Codex", "Mistral", "Kilo", "Cursor", "Aider",
     "Amp", "Kimi", "Claude", "Qwen3", "GLM", "Grok", "DeepSeek", "MiniMax"
@@ -53,5 +55,33 @@
       "Amp": 8, "Kimi": 6, "Claude": 7, "Qwen3": 8, "GLM": 10, "Grok": 7,
       "DeepSeek": 8, "MiniMax": 7
     }
+  },
+  "capabilities": {
+    "Gemini":   { "intelligence": 8, "taste": 8, "cost": 6 },
+    "Codex":    { "intelligence": 8, "taste": 5, "cost": 4 },
+    "Mistral":  { "intelligence": 6, "taste": 6, "cost": 6 },
+    "Kilo":     { "intelligence": 6, "taste": 6, "cost": 6 },
+    "Cursor":   { "intelligence": 6, "taste": 7, "cost": 6 },
+    "Aider":    { "intelligence": 6, "taste": 5, "cost": 9 },
+    "Amp":      { "intelligence": 7, "taste": 7, "cost": 5 },
+    "Kimi":     { "intelligence": 6, "taste": 6, "cost": 6 },
+    "Claude":   { "intelligence": 7, "taste": 8, "cost": 5 },
+    "Qwen3":    { "intelligence": 7, "taste": 5, "cost": 8 },
+    "GLM":      { "intelligence": 6, "taste": 5, "cost": 7 },
+    "Grok":     { "intelligence": 7, "taste": 6, "cost": 7 },
+    "DeepSeek": { "intelligence": 8, "taste": 5, "cost": 9 },
+    "MiniMax":  { "intelligence": 6, "taste": 6, "cost": 8 }
+  },
+  "category_axis": {
+    "API_DESIGN":   "taste",
+    "ARCHITECTURE": "taste",
+    "GENERAL":      "taste",
+    "CODE_REVIEW":  "taste",
+    "ALGORITHM":    "intelligence",
+    "SECURITY":     "intelligence",
+    "BUG_DEBUG":    "intelligence",
+    "DATABASE":     "intelligence",
+    "TESTING":      "intelligence",
+    "QUICK_SYNTAX": "intelligence"
   }
 }

package/references/calibration_benchmark.json

@@ -0,0 +1,65 @@
+{
+  "_comment": "Calibration benchmark for roster_calibrate.sh. ~5 questions per category across all 10 categories, balanced over the two capability axes (taste: API_DESIGN, ARCHITECTURE, CODE_REVIEW, GENERAL; intelligence: the rest). Run each through consult_all.sh with ENABLE_PEER_REVIEW=true (scripts/run_calibration.sh does this), then roster_calibrate the output dirs to derive measured intelligence/taste/cost. Questions are deliberately generic (no repo context) so the panel is scored on judgment, not codebase familiarity. Re-derive/extend for your own domain.",
+  "version": "1.0",
+  "questions": [
+    { "category": "ALGORITHM", "axis": "intelligence", "q": "Design an algorithm to find the k-th smallest element in the union of two sorted arrays in O(log(m+n)) time. Explain the invariant." },
+    { "category": "ALGORITHM", "axis": "intelligence", "q": "Maintain the running median of a stream of integers efficiently. Which data structures, and what is the per-update cost?" },
+    { "category": "ALGORITHM", "axis": "intelligence", "q": "Detect a cycle in a directed graph and return one concrete cycle. Compare an iterative and a recursive approach." },
+    { "category": "ALGORITHM", "axis": "intelligence", "q": "Deduplicate 10 billion 64-bit integers using at most 4 GB of RAM. Outline an approach and its trade-offs." },
+    { "category": "ALGORITHM", "axis": "intelligence", "q": "Implement an LRU cache with O(1) get and put. Justify the data-structure choices and the failure modes." },
+
+    { "category": "BUG_DEBUG", "axis": "intelligence", "q": "A Node.js service leaks memory slowly under sustained load. Describe a systematic method to locate the leak." },
+    { "category": "BUG_DEBUG", "axis": "intelligence", "q": "Intermittent 'connection reset' errors appear only in production behind a load balancer. How do you diagnose the root cause?" },
+    { "category": "BUG_DEBUG", "axis": "intelligence", "q": "A unit test passes locally but fails ~10% of the time in CI. How do you find and fix the flakiness?" },
+    { "category": "BUG_DEBUG", "axis": "intelligence", "q": "After a deploy, p99 latency doubled while p50 is unchanged. Where do you look, and in what order?" },
+    { "category": "BUG_DEBUG", "axis": "intelligence", "q": "A Python multiprocessing script deadlocks occasionally. Enumerate likely causes and how to confirm each." },
+
+    { "category": "SECURITY", "axis": "intelligence", "q": "A web app stores session tokens in localStorage. Assess the risk and recommend a safer alternative with reasoning." },
+    { "category": "SECURITY", "axis": "intelligence", "q": "Design a secure password-reset flow. Enumerate the attacks it must resist and how each is mitigated." },
+    { "category": "SECURITY", "axis": "intelligence", "q": "How do you prevent SSRF in a service that fetches user-supplied URLs? Cover the non-obvious bypasses." },
+    { "category": "SECURITY", "axis": "intelligence", "q": "Review a JWT setup using HS256 with one shared secret across all services. What are the risks and fixes?" },
+    { "category": "SECURITY", "axis": "intelligence", "q": "Beyond parameterized queries, what controls reduce SQL-injection risk in a large codebase?" },
+
+    { "category": "QUICK_SYNTAX", "axis": "intelligence", "q": "In Python, what is the idiomatic way to merge two dictionaries with the second overriding the first?" },
+    { "category": "QUICK_SYNTAX", "axis": "intelligence", "q": "In JavaScript, explain == vs === with a concrete gotcha example." },
+    { "category": "QUICK_SYNTAX", "axis": "intelligence", "q": "In Java, how do you remove items from a list while iterating without a ConcurrentModificationException?" },
+    { "category": "QUICK_SYNTAX", "axis": "intelligence", "q": "In Bash, how do you distinguish an unset variable from an empty one?" },
+    { "category": "QUICK_SYNTAX", "axis": "intelligence", "q": "In Go, what is the idiomatic way to convert a slice of ints to a slice of strings?" },
+
+    { "category": "DATABASE", "axis": "intelligence", "q": "A JOIN across two 10M-row tables is slow. Walk through your indexing and diagnosis steps." },
+    { "category": "DATABASE", "axis": "intelligence", "q": "When would you denormalize a schema, and what consistency trade-offs does it introduce?" },
+    { "category": "DATABASE", "axis": "intelligence", "q": "Design multi-tenant storage: shared tables vs schema-per-tenant vs database-per-tenant. Recommend with reasoning." },
+    { "category": "DATABASE", "axis": "intelligence", "q": "Implement pagination that stays correct and fast on a frequently-updated table. What breaks with OFFSET?" },
+    { "category": "DATABASE", "axis": "intelligence", "q": "How do you choose between a B-tree and a hash index? Give a concrete example favoring each." },
+
+    { "category": "TESTING", "axis": "intelligence", "q": "For a payment service, how do you decide what to unit-test vs integration-test?" },
+    { "category": "TESTING", "axis": "intelligence", "q": "Design a testing strategy for a function with heavy date/time logic across time zones and DST." },
+    { "category": "TESTING", "axis": "intelligence", "q": "How do you test code that calls a flaky third-party API without hitting it in CI?" },
+    { "category": "TESTING", "axis": "intelligence", "q": "Propose a coverage-target policy that avoids gaming the metric while still catching regressions." },
+    { "category": "TESTING", "axis": "intelligence", "q": "Design property-based tests for a JSON serializer/deserializer round-trip. What invariants do you assert?" },
+
+    { "category": "CODE_REVIEW", "axis": "taste", "q": "Critique a function that takes 8 positional boolean parameters and propose a cleaner interface." },
+    { "category": "CODE_REVIEW", "axis": "taste", "q": "A 500-line module mixes business logic, HTTP handling, and SQL. How would you restructure it, and why?" },
+    { "category": "CODE_REVIEW", "axis": "taste", "q": "What makes error handling in a library 'good'? Compare returning null, throwing, and result types." },
+    { "category": "CODE_REVIEW", "axis": "taste", "q": "Booleans are named flag, check, and status throughout a codebase. Propose naming principles for a review guide." },
+    { "category": "CODE_REVIEW", "axis": "taste", "q": "When is a comment a code smell? Give concrete guidance for a team's review checklist." },
+
+    { "category": "ARCHITECTURE", "axis": "taste", "q": "Design the module boundaries for a URL-shortener service. Where are the seams and why there?" },
+    { "category": "ARCHITECTURE", "axis": "taste", "q": "Monolith vs microservices for a 5-engineer startup shipping B2B SaaS. Recommend and defend the choice." },
+    { "category": "ARCHITECTURE", "axis": "taste", "q": "Design an event-driven architecture for order processing. Where do you place consistency boundaries?" },
+    { "category": "ARCHITECTURE", "axis": "taste", "q": "How would you structure a shared design-system component library used by six product teams?" },
+    { "category": "ARCHITECTURE", "axis": "taste", "q": "Propose a caching strategy (layers, invalidation, keys) for a read-heavy content platform." },
+
+    { "category": "API_DESIGN", "axis": "taste", "q": "Design a REST API for a commenting system with threads, replies, and reactions. Show the resource model." },
+    { "category": "API_DESIGN", "axis": "taste", "q": "Offset vs cursor pagination for a public API. Recommend one and design the request/response contract." },
+    { "category": "API_DESIGN", "axis": "taste", "q": "How should an API represent partial updates? Compare PATCH strategies and design a clean one." },
+    { "category": "API_DESIGN", "axis": "taste", "q": "Design error responses for a public JSON API so clients can handle failures programmatically." },
+    { "category": "API_DESIGN", "axis": "taste", "q": "Version a public API without breaking clients. Compare URL vs header versioning and recommend." },
+
+    { "category": "GENERAL", "axis": "taste", "q": "What principles keep a fast-growing codebase maintainable over three years?" },
+    { "category": "GENERAL", "axis": "taste", "q": "How do you decide whether to adopt a new framework or stay on the current stack?" },
+    { "category": "GENERAL", "axis": "taste", "q": "What makes a good pull-request description? Provide a template and the rationale behind each part." },
+    { "category": "GENERAL", "axis": "taste", "q": "How should a team balance shipping speed against technical debt? Give a concrete, enforceable policy." },
+    { "category": "GENERAL", "axis": "taste", "q": "What is your approach to writing documentation that developers actually read and keep current?" }
+  ]
+}

package/references/configuration.md

@@ -249,6 +249,23 @@ ENABLE_DEBATE_OPTIMIZATION=false # Skip debate if all agree
 ENABLE_COMPACT_REPORT=true   # Summaries only in reports
 ```
 
+## Capability-Aware Routing & Voting (v2.20+)
+
+Weight each consultant's vote (and rank the panel) by its capability on the
+quality axis a question stresses — **intelligence** or **taste**, per the
+`category_axis` map in [`references/affinity.json`](../references/affinity.json).
+Per-consultant scores live in that file's `capabilities` block. `cost` is a
+composition/budget axis only and is never a vote weight (tie-break order:
+intelligence > taste > cost). Both features are opt-in and additive — with the
+flags off, behavior is identical to before.
+
+```bash
+ENABLE_CAPABILITY_WEIGHTING=false  # capability-weighted voting
+ENABLE_CAPABILITY_ROUTING=false    # capability-aware panel composition
+CAPABILITY_WEIGHT_STRENGTH=10      # higher = gentler nudge: weight = conf*(S+cap)/S
+CAPABILITY_DEFAULT=5               # fallback for a missing consultant/axis
+```
+
 ## Timeouts and Retries
 
 ```bash

package/scripts/config.sh

@@ -543,6 +543,18 @@ USE_ECONOMIC_MODELS_FOR_SIMPLE="${USE_ECONOMIC_MODELS_FOR_SIMPLE:-true}"
 COMPLEXITY_THRESHOLD_SIMPLE="${COMPLEXITY_THRESHOLD_SIMPLE:-3}"
 COMPLEXITY_THRESHOLD_MEDIUM="${COMPLEXITY_THRESHOLD_MEDIUM:-6}"
 
+# --- Capability-Aware Routing & Voting (opt-in, v2.20) ---
+# Weight each consultant's vote (and rank the panel) by its capability on the
+# quality axis a question stresses — intelligence or taste, per
+# references/affinity.json 'category_axis'. Scores live in affinity.json
+# 'capabilities'. cost is a composition/budget axis only, never a vote weight
+# (tie-break: intelligence > taste > cost). Both default off -> identical to
+# prior behavior.
+ENABLE_CAPABILITY_WEIGHTING="${ENABLE_CAPABILITY_WEIGHTING:-false}"  # capability-weighted voting
+ENABLE_CAPABILITY_ROUTING="${ENABLE_CAPABILITY_ROUTING:-false}"      # capability-aware panel composition
+CAPABILITY_WEIGHT_STRENGTH="${CAPABILITY_WEIGHT_STRENGTH:-10}"       # higher = gentler nudge: weight = conf*(S+cap)/S
+CAPABILITY_DEFAULT="${CAPABILITY_DEFAULT:-5}"                        # fallback capability for a missing consultant/axis
+
 # --- Selective Context ---
 # Send only relevant files to each consultant
 ENABLE_SELECTIVE_CONTEXT="${ENABLE_SELECTIVE_CONTEXT:-false}"
@@ -887,4 +899,4 @@ EOF
 # VERSION
 # =============================================================================
 
-AI_CONSULTANTS_VERSION="2.19.2"
+AI_CONSULTANTS_VERSION="2.20.0"

package/scripts/consult_all.sh

@@ -786,6 +786,9 @@ jq -n \
     --arg orch_shape "${ORCH_SHAPE:-fixed}" \
     --argjson complexity "${QUERY_COMPLEXITY:-5}" \
     --arg intent "${QUERY_INTENT:-advise}" \
+    --arg cap_axis "$(get_category_axis "$QUESTION_CATEGORY" 2>/dev/null || echo "")" \
+    --argjson cap_weighting "${ENABLE_CAPABILITY_WEIGHTING:-false}" \
+    --argjson cap_routing "${ENABLE_CAPABILITY_ROUTING:-false}" \
     '{
         optimization_settings: {
             cache_enabled: $cache_enabled,
@@ -801,6 +804,11 @@ jq -n \
             complexity: $complexity,
             intent: $intent
         },
+        capability: {
+            weighting_enabled: $cap_weighting,
+            routing_enabled: $cap_routing,
+            axis: $cap_axis
+        },
         quality_metrics: {
             consensus_score: $consensus,
             consensus_level: $consensus_level,

package/scripts/doctor.sh

@@ -34,6 +34,7 @@ SUGGEST_CONFIG=false
 SUGGEST_PRESET=false
 LIVE_MODE=false
 QUESTION=""
+ROSTER_AUDIT=false
 
 while [[ $# -gt 0 ]]; do
     case "$1" in
@@ -65,6 +66,10 @@ while [[ $# -gt 0 ]]; do
             LIVE_MODE=true
             shift
             ;;
+        --roster-audit)
+            ROSTER_AUDIT=true
+            shift
+            ;;
         --question)
             QUESTION="${2:-}"
             shift 2
@@ -86,6 +91,8 @@ while [[ $# -gt 0 ]]; do
             echo "  --live             Send a real ping query to each enabled consultant"
             echo "                     (catches installed-but-unauthenticated CLIs that the"
             echo "                     static check reports as healthy). Costs a tiny query each."
+            echo "  --roster-audit     Audit past consultations for uncorrelated value"
+            echo "                     (which consultants add distinct signal vs echo the panel)"
             exit 0
             ;;
         *)
@@ -1183,4 +1190,12 @@ if [[ "$LIVE_MODE" == "true" ]]; then
     [[ ${#ISSUES[@]} -gt 0 ]] && exit 1 || exit 0
 fi
 
+if [[ "$ROSTER_AUDIT" == "true" ]]; then
+    # Delegate to the standalone audit tool (reuses doctor's --json flag).
+    audit_args=()
+    [[ "$JSON_OUTPUT" == "true" ]] && audit_args+=(--json)
+    "$SCRIPT_DIR/roster_audit.sh" ${audit_args[@]+"${audit_args[@]}"}
+    exit $?
+fi
+
 main