ai-consultants 2.25.2 → 3.0.0

package/docs/COST_RATES.md

@@ -1,4 +1,4 @@
-# Cost Rates - AI Consultants v2.25.2
+# Cost Rates - AI Consultants v3.0.0
 
 This page documents the per-token rates used by the cost tracking system.
 

package/docs/RECIPES.md

@@ -17,16 +17,13 @@ must be enabled.
 
 ## 1. Everyday balanced panel
 
-Use four complementary CLI consultants, synthesize their answers, and keep the
-dynamic planner available without forcing debate on every question.
+Use four complementary CLI consultants and synthesize their answers with the
+default coverage strategy.
 
 ```dotenv
 DEFAULT_PRESET=balanced
-DEFAULT_STRATEGY=majority
+DEFAULT_STRATEGY=coverage
 ENABLE_SYNTHESIS=true
-ENABLE_DEBATE=true
-ORCHESTRATION_MODE=auto
-ENABLE_DEBATE_OPTIMIZATION=true
 ```
 
 ```bash
@@ -35,7 +32,7 @@ ai-consultants "Redis or Memcached for a write-heavy session store?"
 
 ## 2. Fast and inexpensive check
 
-The `fast` preset selects two consultants, economy models, and no debate.
+The `fast` preset selects two consultants and economy models.
 
 ```bash
 ai-consultants --preset fast "Is this SQL index definition valid?"
@@ -51,57 +48,10 @@ MAX_SESSION_COST=0.10
 BUDGET_ACTION=stop
 ```
 
-## 3. Debate until the panel converges
-
-Use the dynamic convergence loop for architectural questions. It stops when
-consensus reaches the target, progress stalls, or the round cap is reached.
-
-```dotenv
-ENABLE_DEBATE=true
-ORCHESTRATION_MODE=converge
-CONVERGENCE_MAX_ROUNDS=4
-CONVERGENCE_TARGET_CONSENSUS=75
-CONVERGENCE_STALL_EPSILON=5
-ENABLE_DEBATE_OPTIMIZATION=true
-DEBATE_USE_SUMMARIES=true
-```
-
-```bash
-ai-consultants --strategy risk_averse \
-  "Should this service use an event log or mutable relational state?"
-```
-
-## 4. Exactly two debate rounds
-
-Use fixed mode when a benchmark or repeatable workflow requires a predictable
-number of calls.
-
-```dotenv
-ENABLE_DEBATE=true
-ORCHESTRATION_MODE=fixed
-DEBATE_ROUNDS=2
-ENABLE_DEBATE_OPTIMIZATION=false
-```
-
-```bash
-ai-consultants "Review this migration plan" docs/migration.md@PRIMARY
-```
-
-`DEBATE_ROUNDS=1` means initial answers only. Values `2` and `3` add one and two
-cross-critique rounds respectively.
-
-## 5. Security review with an adversarial gate
-
-The adversarial shape forces critique and enables peer review as a refutation
-gate before synthesis.
+## 3. Security review
 
 ```dotenv
 DEFAULT_STRATEGY=security_first
-ENABLE_DEBATE=true
-ORCHESTRATION_MODE=adversarial
-ENABLE_ADVERSARIAL_VERIFY=true
-ENABLE_PEER_REVIEW=true
-PEER_REVIEW_MIN_RESPONSES=3
 ```
 
 ```bash
@@ -112,34 +62,17 @@ ai-consultants --preset security \
 
 Never include credentials, tokens, or private keys in consultation context.
 
-## 6. Compare two approaches and choose one
-
-Force tournament mode when the answer must end with one winner.
-
-```bash
-ORCHESTRATION_MODE=tournament \
-ENABLE_DEBATE=true \
-ai-consultants --strategy majority \
-  "For this API, choose REST or GraphQL and justify one winner"
-```
-
-Use `--strategy compare_only` instead when you want the trade-offs without a
-recommendation.
-
-## 7. Exhaustive audit
+## 4. Compare two approaches without a recommendation
 
-Continue until a round surfaces no new approach or finding.
+Use `--strategy compare_only` when you want the trade-offs laid out without a
+single winner.
 
 ```bash
-ORCHESTRATION_MODE=exhaustive \
-ENABLE_DEBATE=true \
-CONVERGENCE_MAX_ROUNDS=4 \
-ai-consultants --preset high-stakes \
-  "Find all correctness and failure-mode risks in this worker" \
-  src/worker.ts@PRIMARY src/queue.ts@CONTEXT
+ai-consultants --strategy compare_only \
+  "For this API, compare REST and GraphQL"
 ```
 
-## 8. Reliable panel with health gate and quorum
+## 5. Reliable panel with health gate and quorum
 
 Ping selected consultants before the full run and stop when too few respond.
 The health gate adds one small call per selected consultant.
@@ -159,7 +92,7 @@ ai-consultants "Make a release recommendation from the attached evidence" \
   release-notes.md@PRIMARY test-results.md@CONTEXT
 ```
 
-## 9. Hard budget ceiling
+## 6. Hard budget ceiling
 
 Track estimated spend and stop with partial results rather than crossing the
 session limit.
@@ -179,7 +112,7 @@ ai-consultants --preset cost-capped --strategy cost_capped \
   "Propose the smallest safe fix for this regression"
 ```
 
-## 10. CLI-only operation
+## 7. CLI-only operation
 
 Keep every switchable consultant on its installed CLI. CLI authentication is
 handled by each provider's own login flow.
@@ -205,7 +138,7 @@ ai-consultants doctor --live
 `KIMI_MODEL` is passed to the Kimi CLI with `--model`, so this recipe uses K3
 even when `~/.kimi/config.toml` still names an older default model.
 
-## 11. Hybrid CLI and API panel
+## 8. Hybrid CLI and API panel
 
 Enable API transport only where needed. Gemini and MiniMax automatically choose
 API mode when their API key is present unless `*_USE_API` is explicitly set.
@@ -227,22 +160,7 @@ ENABLE_DEEPSEEK=false
 
 Keep `~/.config/ai-consultants/.env` at mode `600` because it may contain keys.
 
-## 12. Exact semantic consensus
-
-Ask the panel to choose from shared stance options so differently worded answers
-can still count as agreement. This adds roughly one LLM call per run.
-
-```dotenv
-ENABLE_STANCE_CONSENSUS=true
-STANCE_MAX_OPTIONS=5
-STANCE_TIMEOUT=60
-```
-
-```bash
-ai-consultants "Should package lockfiles always be committed for applications?"
-```
-
-## 13. Large or awkward prompts
+## 9. Large or awkward prompts
 
 Use `--query-file` to avoid shell quoting and argument-size problems. Mark the
 main files with `@PRIMARY` and supporting files with `@CONTEXT`.

package/docs/SETUP.md

@@ -62,12 +62,11 @@ That's it! You're ready to go.
 
 ## Claude Code Slash Commands
 
-AI Consultants provides 3 slash commands:
+AI Consultants provides 2 slash commands:
 
 | Command | Description |
 |---------|-------------|
 | `/ai-consultants:consult` | Main consultation - ask AI consultants a coding question |
-| `/ai-consultants:debate` | Run consultation with multi-round debate |
 | `/ai-consultants:help` | Show all commands and usage |
 
 Configuration can be managed with `ai-consultants configure`, including presets,
@@ -460,8 +459,8 @@ Presets let you quickly configure how many consultants to use.
 | `minimal` | 2 (Gemini + Codex) | Quick, cheap |
 | `balanced` | 4 (+ Mistral + Cursor) | Standard |
 | `thorough` | 4 | Comprehensive |
-| `high-stakes` | Expanded panel + debate | Critical decisions |
-| `security` | Security-focused + debate | Security reviews |
+| `high-stakes` | Expanded panel | Critical decisions |
+| `security` | Security-focused | Security reviews |
 | `cost-capped` | Budget-friendly | Low cost |
 
 ### Set Default Preset
@@ -491,7 +490,8 @@ Strategies control how consultant responses are combined.
 
 | Strategy | Description |
 |----------|-------------|
-| `majority` | Simple voting, most common answer wins (default) |
+| `coverage` | Union of every distinct point/risk across consultants, deduplicated (default) |
+| `majority` | Simple voting, most common answer wins |
 | `risk_averse` | Weight conservative responses higher |
 | `security_first` | Prioritize security-focused insights |
 | `cost_capped` | Prefer simpler, cheaper solutions |
@@ -522,33 +522,9 @@ DEFAULT_STRATEGY=risk_averse
 These examples can be prefixed to one command or saved in
 `~/.config/ai-consultants/.env`.
 
-### Dynamic debate
-
-```bash
-ENABLE_DEBATE=true \
-ORCHESTRATION_MODE=converge \
-CONVERGENCE_MAX_ROUNDS=4 \
-CONVERGENCE_TARGET_CONSENSUS=75 \
-ai-consultants --strategy risk_averse \
-  "Should this service use an event log or mutable relational state?"
-```
-
-### Fixed two-round debate
-
-```bash
-ENABLE_DEBATE=true \
-ORCHESTRATION_MODE=fixed \
-DEBATE_ROUNDS=2 \
-ENABLE_DEBATE_OPTIMIZATION=false \
-ai-consultants "Review this migration plan" docs/migration.md@PRIMARY
-```
-
 ### Security gate with a required quorum
 
 ```bash
-ENABLE_DEBATE=true \
-ORCHESTRATION_MODE=adversarial \
-ENABLE_PEER_REVIEW=true \
 ENABLE_HEALTH_GATE=true \
 QUORUM_MIN=3 \
 QUORUM_ACTION=stop \
@@ -556,9 +532,8 @@ ai-consultants --preset security --strategy security_first \
   "Find authentication bypasses" src/auth.ts@PRIMARY
 ```
 
-More scenarios—including tournament, exhaustive audit, CLI-only, hybrid API,
-hard-budget, semantic-consensus, and large-context runs—are in
-[`docs/RECIPES.md`](RECIPES.md).
+More scenarios—including CLI-only, hybrid API, hard-budget, and
+large-context runs—are in [`docs/RECIPES.md`](RECIPES.md).
 
 ---
 
@@ -573,7 +548,7 @@ ai-consultants configure
 # Set any supported parameter non-interactively
 ai-consultants configure \
   --set DEFAULT_PRESET=balanced \
-  --set ENABLE_DEBATE=true
+  --set ENABLE_SYNTHESIS=true
 
 # Guided or exhaustive review
 ai-consultants configure --interactive
@@ -643,13 +618,10 @@ MINIMAX_USE_API=false  # Default: mmx CLI mode
 
 # Defaults (v2.8)
 DEFAULT_PRESET=balanced
-DEFAULT_STRATEGY=majority
+DEFAULT_STRATEGY=coverage
 
 # Features
-ENABLE_DEBATE=false
 ENABLE_SYNTHESIS=true
-ENABLE_PANIC_MODE=auto
-ORCHESTRATION_MODE=auto
 ENABLE_HEALTH_GATE=false
 QUORUM_MIN=2
 QUORUM_ACTION=warn

package/docs/SMART_ROUTING.md

@@ -49,18 +49,10 @@ customization possible without editing bash.
   "version": "1.1",
   "default_score": 5,
   "general_score": 8,
-  "capability_default": 5,
   "known_consultants": ["Gemini", "Codex", "..."],
   "categories": {
     "CODE_REVIEW": { "Gemini": 7, "Codex": 10, "..." },
     "BUG_DEBUG":   { "..." }
-  },
-  "capabilities": {
-    "Gemini": { "intelligence": 8, "taste": 8, "cost": 6 },
-    "Codex":  { "intelligence": 8, "taste": 5, "cost": 4 }
-  },
-  "category_axis": {
-    "API_DESIGN": "taste", "CODE_REVIEW": "taste", "ALGORITHM": "intelligence"
   }
 }
 ```
@@ -127,112 +119,6 @@ COMPLEXITY_THRESHOLD_MEDIUM=6    # Score 4-6 = use standard models
 
 See [COST_RATES.md](COST_RATES.md) for model pricing by tier.
 
-## Capability-Aware Routing & Voting (v2.20+)
-
-Beyond category *fit* (the affinity matrix), v1.1 of `affinity.json` adds a
-per-consultant **capability** score on three axes (1-10): `intelligence`,
-`taste`, and `cost`. A `category_axis` map names the quality axis each category
-stresses — **taste** for design-shaped work (API_DESIGN, ARCHITECTURE,
-CODE_REVIEW, GENERAL), **intelligence** otherwise.
-
-Two opt-in features consume this:
-
-- **`ENABLE_CAPABILITY_WEIGHTING`** — a consultant's vote weight becomes
-  `confidence × (S + capability) / S` (`S = CAPABILITY_WEIGHT_STRENGTH`, default
-  10), so on a taste-shaped question a high-taste model's opinion counts for more.
-- **`ENABLE_CAPABILITY_ROUTING`** — panel *eligibility* still uses raw affinity,
-  but the *sort rank* becomes `affinity + (capability − CAPABILITY_DEFAULT)`, so
-  under a size limit the quality axis reorders which eligible consultants make
-  the cut.
-
-`cost` is an efficiency axis for budget-aware composition only — **never** a
-vote weight (tie-break order: intelligence > taste > cost). Scores are
-subjective, point-in-time seeds — re-derive them for your own panel. Helpers:
-`get_capability <consultant> <axis>` and `get_category_axis <category>` in
-`lib/routing.sh`.
-
-## Roster Audit — Uncorrelated Value (v2.20+)
-
-Adding a consultant only helps if it says something the others don't. The
-`scripts/roster_audit.sh` tool measures this across past consultations: a
-consultant's approach is *distinct* in a round when its keyword set has low
-Jaccard overlap (`< --threshold`, default 20%) with **every** other consultant
-that round. A consultant that is rarely distinct is correlated with the panel —
-a candidate to drop or down-weight; one that is often distinct earns its seat on
-diversity.
-
-```bash
-# audit specific consultation output dirs
-./scripts/roster_audit.sh /path/to/consultation_dir ...
-
-# or the N most recent under the consultations base
-./scripts/roster_audit.sh --recent 30
-
-# machine-readable
-./scripts/roster_audit.sh --json --recent 30
-```
-
-Read-only; reuses the `voting.sh` keyword/Jaccard machinery. Consultations with
-fewer than two responders are skipped (correlation is undefined). This is the
-model-routing "before a new model earns a row" bar (contribute >=1 thing the
-incumbents miss) applied to the consultation panel instead of a review lane.
-
-## Measured Calibration — Replacing the Heuristic Seeds (v2.20+)
-
-The `capabilities` scores in `affinity.json` ship as subjective seeds. To
-**measure** them instead of guessing, three tools derive each axis from data the
-tool already produces — sliced by the same `category_axis` that consumes them:
-
-| Tool | Derives | From |
-|---|---|---|
-| `scripts/roster_calibrate.sh` | intelligence, taste, cost (**Tier A**) | blind peer-review scores (sliced by axis) + observed `tokens_used` × catalog rate |
-| `scripts/taste_elo.sh` | taste (**Tier B**) | pairwise LLM-as-judge Elo over taste-axis answers (removes peer self-bias) |
-| `references/calibration_benchmark.json` | the question set | 50 questions, 5 per category, balanced across the two axes |
-| `scripts/run_calibration.sh` | the data | runs the benchmark through the panel with `ENABLE_PEER_REVIEW=true`, then calibrates |
-
-### Workflow
-
-```bash
-# 1. Collect data: run the benchmark through the full panel (real calls — costs money).
-./scripts/run_calibration.sh --limit 10 --dry-run   # preview
-./scripts/run_calibration.sh                         # full 50-question run
-
-# 2. Tier A: measured intelligence/taste/cost -> capabilities block
-./scripts/roster_calibrate.sh --recent 50            # print measured block
-./scripts/roster_calibrate.sh --recent 50 --write    # merge into affinity.json (keeps unmeasured cells)
-
-# 3. Tier B (optional): refine taste via pairwise-judge Elo
-./scripts/taste_elo.sh --recent 50 --write
-```
-
-Both `roster_calibrate.sh` and `taste_elo.sh` also accept explicit consultation
-dirs and `--json`, and back up `affinity.json` to `.bak` before `--write`.
-
-### How each axis is measured
-
-- **cost** — mean *observed* `$/response` (`tokens_used` × catalog rate, 60/40
-  input/output split), rank-normalized so the cheapest = 10. This is cost per
-  *task* (verbosity + completion), not per-token sticker price — a terse pricey
-  model can still rank cheap.
-- **intelligence / taste** — mean **blind peer-review** `quality_score` (already
-  1-10), sliced by `category_axis`: intelligence = performance on
-  intelligence-axis categories, taste = on taste-axis categories.
-- **taste (Tier B)** — taste has no ground truth, so `taste_elo.sh` measures it
-  *relative to a chosen judge*. Each pair of answers on a taste-axis question is
-  shown to the judge, which picks the better one on design taste only; wins feed
-  an Elo ranking. The judge is pluggable:
-  - `JUDGE_CLI` — CLI for the built-in judge (default `claude`).
-  - `TASTE_JUDGE_CMD` — external judge `cmd <ctx> <A.json> <B.json>` printing
-    `A`/`B` (used by tests and custom judges).
-
-### Companion: which consultants earn a seat
-
-`roster_audit.sh` (above) answers *who to keep*; calibration answers *how much
-each is worth per axis*. Together they make the roster data-driven: audit to
-prune redundant consultants, calibrate to score the survivors, then enable
-`ENABLE_CAPABILITY_WEIGHTING` / `ENABLE_CAPABILITY_ROUTING` to act on the
-measured scores.
-
 ## Timeout per Category
 
 Timeouts optimized based on category complexity:

package/docs/cost_rates.json

@@ -1,5 +1,5 @@
 {
-  "version": "2.25.2",
+  "version": "3.0.0",
   "updated": "2026-07-16",
   "models": {
     "_comment_units": "All rates are USD per 1,000 tokens (per-1K). lib/costs.sh computes cost as (token_count / 1000) * rate, so every value here MUST be a per-1K rate = (provider $/1M price) / 1000. Do NOT paste $/MTok figures directly.",

package/docs/releases/v3.0.0.md

@@ -0,0 +1,71 @@
+# Release v3.0.0
+
+**Date:** 2026-07-23
+**Type:** Major — removes the deliberation/consensus machinery; default output is now the coverage union
+**Previous:** v2.25.2
+
+## Highlights
+
+- **The panel premise was tested and the answer is task-dependent.** A held-out coverage experiment found a single strong model saturates *convergent* defect-finding (it caught 19/19 hard bugs, including real CVEs and a concurrency bug found by formal verification), while a diverse cross-vendor panel materially out-covers it on *breadth* questions — "what could go wrong?", enumerate the risks — winning as a **union of distinct answers**, not through deliberation.
+- **Default output is now the `coverage` union** — the deduplicated set of every distinct point, recommendation, risk, and edge case across the panel, preserving what only one model raised. No more voted single recommendation by default.
+- **The deliberation/consensus machinery is gone**: multi-round debate, dynamic orchestration, voting/consensus scoring, capability weighting, panic mode, peer review, stance consensus, and the capability-calibration tooling.
+- **What made the panel good is kept**: diverse parallel fan-out to all 11 consultants, category-affinity routing, doctor, cost tracking, quorum/health gate, and every adapter.
+- **Simpler and smaller**: 19 files removed (~-4,600 lines), `npm test` 22 → 15 suites, all green.
+
+## What's New
+
+### Coverage synthesis (the new default)
+
+The pipeline is now **classify → route → parallel fan-out → coverage synthesis**. Every selected consultant answers once, in parallel; the synthesizer returns the *union* of their distinct points rather than picking a winner or averaging a vote.
+
+| Strategy | Behavior |
+|---|---|
+| `coverage` (default) | Union of every distinct point/recommendation/risk across the panel; credits what only one model raised |
+| `compare_only` | Each consultant side-by-side, no synthesized union |
+| `majority` | A single blended recommendation, weighting all equally |
+| `risk_averse` / `security_first` / `cost_capped` | Emphasis variants |
+
+Override per run with `--strategy`, or persist with `DEFAULT_STRATEGY`.
+
+### Why
+
+Two independent reviews had judged the panel's premise weak: voting and consensus measure *agreement*, not correctness, and the repo had no held-out comparison against one strong model. The A/W/C coverage experiment (maintainer instrumentation, not shipped) measured, on deep 60-point rubrics across three domains:
+
+| task | single model | self-consistency (same model ×k) | diverse panel (union) |
+|---|---|---|---|
+| convergent defect-finding | 19/19 (saturates) | — | no headroom to add |
+| breadth / enumeration | 51% | 70% | **93%** |
+
+The panel wins on breadth, and it wins as the raw union with deliberation **off** — so the diverse fan-out is the value and the averaging machinery is not.
+
+## Breaking Changes
+
+- **Default output shape changed** from a voted recommendation to the coverage union.
+- **Removed features and their config** (setting any of these via `configure --set` now fails closed): `ENABLE_DEBATE`/`DEBATE_ROUNDS`, `ORCHESTRATION_MODE`/`CONVERGENCE_*`, `ENABLE_DEBATE_OPTIMIZATION`, `ENABLE_PANIC_MODE`/`PANIC_*`, `ENABLE_ADVERSARIAL_VERIFY`, `ENABLE_PEER_REVIEW`/`PEER_REVIEW_MIN_RESPONSES`, `ENABLE_CAPABILITY_WEIGHTING`/`ENABLE_CAPABILITY_ROUTING`/`CAPABILITY_*`, `ENABLE_STANCE_CONSENSUS`/`STANCE_*`.
+- **Removed outputs**: `voting.json`, `round_N/`, `panic_diagnosis.json`.
+- **Removed slash commands**: `/ai-consultants:debate`, `/ai-consultants:roster-audit` (installers prune them on update).
+- **Removed tooling**: `roster_audit.sh`, `roster_calibrate.sh`, `run_calibration.sh`, `taste_elo.sh`, `doctor --roster-audit`.
+- The affinity matrix keeps its category→consultant routing scores; the `capabilities`/`category_axis` axes are removed.
+
+## Upgrade Guide
+
+```bash
+# If installed via git clone
+cd ~/.claude/skills/ai-consultants && git pull
+
+# If installed via curl | bash
+curl -fsSL https://raw.githubusercontent.com/matteoscurati/ai-consultants/main/scripts/install.sh | bash
+
+# If installed via npx (no install step)
+npx ai-consultants@latest "your question"
+```
+
+No configuration change is required. If your `.env` sets a removed variable, delete it — the tool now fans out and returns the coverage union by default. To get a single recommendation instead, use `--strategy majority`; for a raw side-by-side, `--strategy compare_only`.
+
+## Note on the evidence
+
+The breadth result is **directional** (n=3, hand-validated) alongside the n=19 defect-finding result. The cut is a considered bet on that evidence; a statistically binding breadth claim would need n≥15–20 deep-rubric items (the experiment harness exists on the `experiment/coverage-metric` branch).
+
+## Contributors
+
+- matteoscurati

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "ai-consultants",
-  "version": "2.25.2",
+  "version": "3.0.0",
   "description": "A harness for every question - a panel of up to 11 AI models that picks its own deliberation strategy per question (convergence, adversarial verification, tournament) and checks its work before answering",
   "bin": {
     "ai-consultants": "bin/ai-consultants"
@@ -12,6 +12,7 @@
     "!scripts/test_fixtures/",
     "!scripts/lib/test_helpers.sh",
     "!scripts/release.sh",
+    "!scripts/experiment/",
     "templates/",
     ".env.example",
     ".claude/commands/",