ai-consultants 2.22.0 → 2.24.0

package/.env.example

@@ -207,10 +207,6 @@ SYNTH_MAX=10
 ENABLE_DEBATE=false
 DEBATE_ROUNDS=1              # 1 = no debate, 2-3 = with cross-critique
 
-# Self-Reflection (experimental): generate-critique-refine loop
-ENABLE_REFLECTION=false
-REFLECTION_CYCLES=1
-
 # Peer Review: anonymous evaluation
 ENABLE_PEER_REVIEW=false
 PEER_REVIEW_MIN_RESPONSES=3

package/CLAUDE.md

@@ -6,7 +6,7 @@ AI Consultants is a multi-model AI deliberation system that queries up to 11 AI
 
 **Self-Exclusion**: The invoking agent is automatically excluded from the panel to prevent self-consultation. Claude Code won't query Claude, Codex CLI won't query Codex, etc.
 
-**Version**: 2.22.0
+**Version**: 2.24.0
 
 ## Distribution
 
@@ -64,8 +64,7 @@ ai-consultants/
 │       ├── costs.sh            # Cost tracking + response limits
 │       ├── cache.sh            # Semantic caching (v2.3)
 │       ├── api_query.sh        # API mode query execution (v2.6)
-│       ├── progress.sh         # Progress bars
-│       └── reflection.sh       # Self-reflection + judge step
+│       └── progress.sh         # Progress bars
 ├── references/
 │   ├── configuration.md      # Full configuration reference
 │   └── details.md            # Presets, strategies, best practices
@@ -194,7 +193,7 @@ Functions in `lib/common.sh`:
 ### Configuration Presets
 ```bash
 # Quality Tier Presets (v2.5)
-./scripts/consult_all.sh --preset max_quality "question"  # All + premium models + debate
+./scripts/consult_all.sh --preset max_quality "question"  # 8 of 11 consultants + premium models + peer review
 ./scripts/consult_all.sh --preset medium "question"       # 4 consultants + standard models
 ./scripts/consult_all.sh --preset fast "question"         # 2 consultants + economy models
 
@@ -252,12 +251,6 @@ Process:
 3. Aggregate peer scores to identify strongest arguments
 4. De-anonymize in final report
 
-### Judge Step (Overconfidence Detection)
-Functions in `lib/reflection.sh`:
-- `judge_response()` - Evaluates single response for overconfidence
-- `heuristic_overconfidence_check()` - Fast fallback without LLM
-- `judge_all_responses()` - Batch evaluation
-
 ## v2.9 Features
 
 ### Kimi CLI Support
@@ -410,7 +403,7 @@ get_model_for_tier "claude" "economy"   # → claude-haiku-4-5
 Three new presets leverage the model tiers:
 
 ```bash
-# Maximum quality - all premium models + debate + reflection
+# Maximum quality - all premium models + debate + peer review
 ./scripts/consult_all.sh --preset max_quality "critical architecture decision"
 
 # Balanced quality - standard models, 4 consultants, light debate
@@ -726,15 +719,21 @@ Manual lint of the full repo: `npm run lint`. Bypass the hook: `git commit --no-
 
 Every version bump **must** include a release note in `docs/releases/v<VERSION>.md`. Use the template below.
 
+**The whole flow is automated by the maintainer's `ai-consultants-release` skill — run `/ai-consultants-release <VERSION>` from anywhere inside this checkout.** It orchestrates the steps below and refuses to proceed when a surface is missing.
+
+That skill deliberately lives **outside this repository**, at `~/.claude/skills/ai-consultants-release/`, and is not distributed. `scripts/install.sh` git-clones this entire repo into `~/.claude/skills/ai-consultants`, so anything under a `.claude/skills/` directory here reaches every `curl | bash` user — and release tooling that pushes tags and deploys the showcase site must not. The skill's own `references/surfaces.md` is the authoritative map of every release surface; this section is the summary.
+
+Consequence worth knowing: the skill's `finalize.sh` re-verifies the same version surfaces `scripts/release.sh` owns, and the two now live in different places with nothing making them drift together. `finalize.sh` compares its arity against that script's `SURFACES` array on every run and aborts if they disagree — so **adding or removing a surface in `scripts/release.sh` will correctly fail the next release until the skill is updated to match.** That is the tradeoff for not shipping release tooling to users.
+
 ### Steps
 
-1. **Update version** in all files: `package.json`, `scripts/config.sh` (`AI_CONSULTANTS_VERSION`), `SKILL.md` (frontmatter + title), `README.md` (title + badge), `CLAUDE.md` (`**Version**`), `docs/cost_rates.json`, `docs/COST_RATES.md` (title)
-2. **Add `## Changelog` entry** in CLAUDE.md (developer-facing, long-form, rationale + line-level commentary)
-3. **Add `CHANGELOG.md` entry** at the top of `CHANGELOG.md` (Keep a Changelog format: `## [VERSION] - YYYY-MM-DD` with `### Added/Changed/Fixed/Removed/Deprecated/Security` subsections — user-facing, one-line bullets)
-4. **Create `docs/releases/v<VERSION>.md`** using the template below (highlights + upgrade guide — for GitHub releases / users)
-5. **Update workspace sync surfaces** if applicable: `aiconsultants.sh/index.html` (`softwareVersion` schema + badge), `../CLAUDE.md` workspace guide ("Latest at time of last sync" + Recent release line)
-6. **Run tests**: `npm test` must pass before committing
-7. **Commit, push**, and optionally create a GitHub release pointing to the release note
+1. **Preflight**: `~/.claude/skills/ai-consultants-release/scripts/preflight.sh <VERSION>` — read-only gate on both repos (branch, cleanliness, origin sync, forward semver, free tag, tooling)
+2. **Bump the version**: `scripts/release.sh <VERSION>` — rewrites 9 anchored surfaces across 7 files (`package.json`, `scripts/config.sh`, `SKILL.md` ×2, `README.md` ×2, `CLAUDE.md`, `docs/cost_rates.json`, `docs/COST_RATES.md`), validates them, and runs `npm test` + `npm run lint`. Never commits, tags, or publishes.
+3. **Write the three changelogs** (hand-written — see "Why three changelog surfaces?" below): `CHANGELOG.md` entry at the top, `## Changelog` entry in this file, and `docs/releases/v<VERSION>.md`
+4. **Update workspace sync surfaces**: `../CLAUDE.md` workspace guide ("Latest at time of last sync" + Recent release line)
+5. **Finalize**: `~/.claude/skills/ai-consultants-release/scripts/finalize.sh <VERSION> --message-file … --tag-message-file …` — re-verifies all 9 version surfaces *and* the 3 doc surfaces, re-runs the gate, then commits, creates the annotated tag, and pushes both
+6. **Publishing is automatic**: pushing the `v*` tag triggers `.github/workflows/publish.yml`, which re-runs the gate, verifies the tag matches `package.json`, publishes to npm via **Trusted Publishing (OIDC — no `NPM_TOKEN`)** with provenance, and creates the GitHub release from the release note. Both terminal steps are idempotent, so a failed run can be re-run after a fix — **never move or delete a pushed tag**, and never `npm publish` from a workstation.
+7. **Sync the showcase site last**: `~/.claude/skills/ai-consultants-release/scripts/sync_site.sh <VERSION> --message-file …` waits for npm to actually serve the version, bumps `index.html` (`softwareVersion` schema + badge), then commits and pushes. Editorial copy (roster, feature cards, presets table, install commands) is hand-edited before running it. The wait enforces the npx timing rule: a subcommand advertised as `npx ai-consultants <cmd>` before publication does not error — `bin/` routes an unknown argument to `consult_all.sh`, starting a real **billable** consultation.
 
 **Why three changelog surfaces?** They serve different audiences:
 - `CLAUDE.md ## Changelog` — what the *next maintainer* needs (file/line references, rationale, latent bugs uncovered)
@@ -802,6 +801,26 @@ curl -fsSL https://raw.githubusercontent.com/matteoscurati/ai-consultants/main/s
 
 ## Changelog
 
+### v2.24.0
+- **Stale slash commands are finally removed on install/update, on both entry points.** `scripts/install.sh` and `bin/ai-consultants install` only ever `cp`'d `.claude/commands/*.md` into `~/.claude/commands`, never pruning — so a command deleted from the repo lived on the user's machine indefinitely. Measured on a real installation: **14 `ai-consultants:*.md` installed, 4 of which this repo still ships.** The other ten were dropped by `9ae73e3` (v2.10.0, "command consolidation") and had been dead for 13 releases. Not inert — they stay invocable, and `config-features.md` tells the user to set `ENABLE_REFLECTION`, removed in v2.23.0 and now rejected by `configure --set` with a non-zero exit. New `prune_removed_commands()` in `install.sh` plus an inline equivalent in `bin/ai-consultants`; both scoped to the `ai-consultants:*.md` namespace. **Known and accepted bound**: within that namespace it also removes a user-authored command, because nothing distinguishes it from one of ours deleted upstream. Documented in `CHANGELOG.md` rather than worked around.
+- **`bin/ai-consultants install` was initially missed** — the first cut of the fix landed only in `install.sh`, leaving npx users (the path the shipped `bin` help and `docs/releases/v2.19.2.md` advertise) with exactly the stale commands the fix targeted. Caught by the `/code-review max` pass, not by tests.
+- **`install.sh` gained a define-only source guard, and its first placement was destructive.** The guard (`AI_CONSULTANTS_INSTALL_DEFINE_ONLY`) exists so `test_install.sh` can source the file for its helpers without the installer running. It was placed at line 240 — *below* the argument parser (line 145) and *below* the uninstall block (line 179). A sourced script inherits the **caller's** positional parameters, so `bash scripts/test_install.sh --uninstall` had `install.sh` parse the suite's own flag and execute the real uninstall: `rm -rf` on the installed skill plus every `~/.claude/commands/ai-consultants*.md`. Reproduced in a sandboxed `HOME` with a canary file before fixing, and re-verified after. The guard and `prune_removed_commands()` now sit above the parser, with the invariant stated inline: nothing above the guard may act, and anything below it must be safe to skip. Milder instance of the same root cause: `./scripts/test_install.sh --verbose` exited 1 printing `install.sh`'s help.
+- **`scripts/release.sh` dropped from the npm tarball** (`package.json` `files` gains `!scripts/release.sh`). It is the last maintainer-only artifact in the package now that `.claude/skills/` is gone; `bin/ai-consultants` does not route it, so `npx ai-consultants release` reaches the consult fallback, not the script. **Deliberately kept** after checking their consumers: `roster_audit.sh` (backs the shipped `roster-audit` slash command and `doctor --roster-audit`), `roster_calibrate.sh` / `taste_elo.sh` / `run_calibration.sh` (documented for users in the shipped `docs/SMART_ROUTING.md`), `install-hooks.sh` (wired to a `package.json` script, and `CHANGELOG.md` records it as safe for tarball consumption).
+- **Release tooling moved out of the repository entirely** (`6a4e70d`, `95d2e29`, `384a1cd`). `scripts/install.sh` git-clones the whole repo into `~/.claude/skills/ai-consultants`, making `.claude/skills/` a distribution channel — confirmed on a real install, which carried both `release` and `verify`. Shipping tooling that pushes tags and deploys the showcase site to end users is both useless and dangerous, so the release and verify skills now live in the maintainer's `~/.claude/skills/`, and this repo has **no** `.claude/skills/` directory. Keep it that way; if a future skill must be in-tree, prune it in `install.sh` after the clone. New `.github/workflows/publish.yml` publishes to npm via **Trusted Publishing (OIDC, no `NPM_TOKEN`)** on a `v*` tag push and creates the GitHub release from `docs/releases/v<VERSION>.md`; it re-runs the full gate because `ci.yml` only covers branches and PRs, and verifies all nine version surfaces server-side (checking only `package.json` would let a hand-bumped tag publish a package whose `config.sh`/`SKILL.md`/`README.md` still report the previous version — unfixable, since npm forbids republishing).
+- **Two test-harness weaknesses fixed, both found by breaking them on purpose.** `lib/test_helpers.sh::test_summary` now fails a suite that ran zero assertions: `run_test` takes `"<name>" <function>`, and given the function alone the name absorbs it, `shift` empties the list, `"$@"` expands to nothing — no test runs and the suite exits 0. That is the v2.23.0 `test_functions.sh` fix one level up (that one scored a test by its last assertion; this one scored a suite that had none). The guard only covers suites calling `test_summary`, so `test_routing_parity.sh` got the same check inline, and `test_set_e_safety.sh` — a static lint with no assertion counter — now asserts its scan corpus is non-empty, since a lint that scans nothing reports "zero occurrences" and passes forever.
+- **`test_configure.sh`'s parameter-contract scan correctly flagged the new test hook.** It walks every `${VAR:-}` in non-test scripts and demands each be a declared config parameter; `AI_CONSULTANTS_INSTALL_DEFINE_ONLY` joins the existing internal hooks (`SKIP_GATE`, `ROOT`, `FORCE`) in its exclusion list. The test did its job on a change it had no knowledge of.
+- **New `scripts/test_install.sh`** (12 assertions, `npm test` now 21 suites): prunes what upstream dropped, keeps what it ships, ignores other tools' commands and lookalike filenames, no-ops on a fresh install (glob matches nothing) and on an absent commands directory, and asserts the define-only guard is still present so a refactor that drops it fails loudly rather than making `npm test` clone.
+- **Cosmetic, found while sweeping for dead code**: three SKIP lines in `test_context_optimization.sh` used plain `echo` with `${C_YELLOW}`, which under bash prints a literal `\033[33m` instead of colour (zsh's builtin `echo` interprets escapes, which masked it during the first check). Now `echo -e`. `C_YELLOW` itself is **not** dead despite shellcheck SC2034 — that suite sources `test_helpers.sh` and uses it; shellcheck analyses one file at a time, which is why its message says "or export if used externally".
+
+### v2.23.0
+- **`lib/reflection.sh` deleted (452 lines) and `ENABLE_REFLECTION` / `REFLECTION_CYCLES` removed.** The module was never sourced — `grep -rn 'reflection\.sh' scripts/` matched only the file's own header — so the generate-critique-refine loop had never executed. The two flags were declared at `config.sh` and exported by four presets (`high-stakes`, `max_quality`, `fast`, `_disable_all_consultants`) with no consumer anywhere. It had also aged past the codebase: `_exec_consultant` hardcoded a Gemini/Codex/Mistral case (3 of 11 consultants; the rest hit `*) return 1`) and its Gemini branch still used the pre-`agy` CLI syntax retired in v2.15.0 — so wiring it up as written would have produced partial reflection with a broken Gemini path. Removed rather than implemented because v2.16's `converge`/`adversarial` shapes already run critique-refine off measured consensus rather than a fixed cycle count. Worth noting for archaeology: the module was *maintained* in at least three review passes (v2.15.1 de-fencing, the `--model` parity change, the v2.10.9 `((var++))` sweep) — effort spent on code that never ran.
+- **The release gate had been red since 2.22.0 and nobody could see it.** `scripts/release.sh:26` sources `lib/common.sh` → `config.sh` → `load_user_config` *before* `npm test` at line 194, so `_apply_env_file`'s `export "$key=$value"` put the maintainer's entire `~/.config/ai-consultants/.env` into the test environment. Two suites were not hermetic against that, and the first aborted the run before the second could surface:
+  - **`test_user_config.sh`** — `test_xdg_fallback` (Test 6) unsets `AI_CONSULTANTS_CONFIG_DIR` in the suite's own shell (required, to assert the XDG branch) and `_reset_state` never restored it, so the unset leaked into all 22 later tests, which then read the *real* user config. Fixed by re-exporting an empty temp dir in `_reset_state`, making isolation order-independent; tests that legitimately override the dir per-invocation are unaffected. Plus the whole `*_USE_API` set unset at suite top (the guard `test_configure.sh` already carried), and `ENABLE_KIMI`/`GROK_API_URL` (Test 27) and `GEMINI_MODEL`/`PANIC_KEYWORDS` (Test 28) unset inside their subshells — those assert on *real* var names and `load_user_config` lets the environment win over the file by design, so with `config.sh` pre-sourced the file under test was never consulted.
+  - **`test_configure.sh`** — `run_clean_configure` scrubbed the nine API keys but no settings, while `configure`'s `has_explicit_value` reads any ambient value as a deliberate pin. An exported `DEFAULT_PRESET` therefore overrode the value Tests 6 and 10 had just written into the file they assert on. The scrub set is now derived from `configure --show-parameters` — configure's own contract — with a hard failure if that list comes back empty, so the scrub can never silently degrade to a no-op. **Verified pre-existing**: the same two assertions fail at `2ac18d4` (the v2.22.0 release commit) under the same conditions, via a throwaway worktree.
+- **`test_functions.sh` was under-reporting failures across the whole file.** `run_tests` scores a test by its exit status — that of its LAST command — so a failing assertion followed by a passing one printed `FAIL`, incremented nothing, and let the suite exit 0. 8 of the 13 test functions run more than one assertion and none accumulated a result. The five `assert_*` helpers now set `_ASSERT_FAILED` on every failure path and `run_tests` fails a test when either that flag or the exit status is non-zero. **Caught by trying to make the new pin fail**: the first version of `test_known_feature_flags_in_sync` printed `FAIL: ... Actual: PHANTOM_FEATURE` while the suite still reported "All tests passed". Verified against a pre-existing test by breaking the first of three assertions in `test_case_conversion` — previously green, now correctly red.
+- **`KNOWN_FEATURE_FLAGS` synced (9 → 27) and pinned.** `_discover_custom_api_agents` (`consult_all.sh:48`) walks `env` and enrolls any `ENABLE_X=true` as a consultant when `X_API_URL` is set and `X` is absent from the registry, so the 18 missing flags — `PEER_REVIEW`, `HEALTH_GATE`, `SEMANTIC_CACHE`, … — were latent phantom consultants. Measured drift: 18 missing, 0 stale. `test_known_feature_flags_in_sync` derives the expected set from `config.sh`'s top-level `ENABLE_*` declarations minus `KNOWN_CLI_AGENTS`/`KNOWN_API_AGENTS` and asserts both directions, so the next added flag fails the build. (Checked that the `^ENABLE_` grep misses nothing declared only inside functions — the sole extra hit was `AGENTNAME`, from the convention's doc comment.)
+- **Preset tables were documenting a mode that stopped being the default in v2.16.** Found by the `/code-review max` pass on the removal branch: renaming the README column from "Reflection" to "Peer Review" turned a code-enforced "No" into a false claim, because `_disable_all_consultants` exported `ENABLE_REFLECTION=false` — the line this release deletes — whereas nothing ever clears `ENABLE_PEER_REVIEW`. Combined with `consult_all.sh:285` force-enabling peer review for the adversarial shape, `--preset medium` on a SECURITY question runs a full billed peer-review round the table promised it would not. The Debate column was stale for the same reason (`DEBATE_ROUNDS` is only read under `ORCHESTRATION_MODE=fixed`). Both tables now state only what the preset pins, with the planner's role called out. Also corrected `max_quality`'s "All consultants" → 8 of 11, which contradicted the README table in the same commit.
+
 ### v2.22.0
 - Added the public `ai-consultants configure` / `config` subcommand. Automatic mode detects every current consultant, uses CLI-first transport selection with API fallback when credentials are available, persists to the XDG user config directory, preserves custom values and secrets while refreshing availability-derived `ENABLE_*` flags, backs up rewrites, and keeps files at mode 600.
 - Replaced the stale 1,092-line v2.0 configurator with a compact implementation whose accepted parameter surface is derived from `.env.example`. `--set KEY=VALUE` supports repeatable automation and participates in detection, `--interactive` reviews credentials/transports/panel selection, `--advanced` reviews every parameter, `--show-parameters` exposes the contract, and side-effect-free `--dry-run` redacts secrets.

package/README.md

@@ -1,8 +1,8 @@
-# AI Consultants v2.22.0
+# AI Consultants v2.24.0
 
 > **A harness for every question.** A panel of up to 11 frontier models that writes its own playbook per question — fan out, debate to convergence, cross-examine under adversarial review, or run a tournament — and checks its work before it reaches you.
 
-[![Version](https://img.shields.io/badge/version-2.22.0-blue.svg)](https://github.com/matteoscurati/ai-consultants)
+[![Version](https://img.shields.io/badge/version-2.24.0-blue.svg)](https://github.com/matteoscurati/ai-consultants)
 [![License](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)
 [![Claude Code Skill](https://img.shields.io/badge/Claude%20Code-Skill-orange.svg)](https://docs.anthropic.com/en/docs/claude-code/skills)
 [![GitHub stars](https://img.shields.io/github/stars/matteoscurati/ai-consultants?style=social)](https://github.com/matteoscurati/ai-consultants)
@@ -428,11 +428,19 @@ Choose the right balance of quality, speed, and cost with model quality tiers.
 
 ### Tier Presets
 
-| Preset | Tier | Agents | Debate | Reflection | Use Case |
-|--------|------|--------|--------|------------|----------|
-| `max_quality` | Premium | 8 | 3 rounds | 2 cycles + peer review | Critical decisions |
-| `medium` | Standard | 4 | 1 round | No | General questions |
-| `fast` | Economy | 2 | No | No | Quick checks |
+| Preset | Tier | Agents | Use Case |
+|--------|------|--------|----------|
+| `max_quality` | Premium | 8 of 11 | Critical decisions |
+| `medium` | Standard | 4 | General questions |
+| `fast` | Economy | 2 | Quick checks |
+
+**Deliberation depth is not fixed by the preset.** Under the default
+`ORCHESTRATION_MODE=auto` the planner picks the shape per question, so the
+`DEBATE_ROUNDS` a preset pins applies only under `ORCHESTRATION_MODE=fixed`,
+and a `SECURITY`-classified question runs the adversarial shape — including a
+peer-review round — whatever preset you chose. `max_quality` additionally
+enables peer review outright; `medium` and `fast` do not *clear* it, so an
+enabled peer review carries into them.
 
 ### Models by Tier
 
@@ -479,8 +487,8 @@ Choose how many consultants to use:
 
 | Preset | Consultants | Tier | Use Case |
 |--------|-------------|------|----------|
-| `max_quality` | 8 + debate + reflection | Premium | Critical decisions |
-| `medium` | 4 + light debate | Standard | General questions |
+| `max_quality` | 8 of 11 (+ peer review) | Premium | Critical decisions |
+| `medium` | 4 | Standard | General questions |
 | `fast` | 2 | Economy | Quick checks |
 | `minimal` | 2 (Gemini + Codex) | Default | Quick questions, low cost |
 | `balanced` | 4 (+ Mistral + Cursor) | Default | Standard consultations |
@@ -717,6 +725,14 @@ Each consultation generates:
 
 ## Changelog
 
+### v2.23.0
+
+- **`ENABLE_REFLECTION` and `REFLECTION_CYCLES` removed.** The self-reflection module they controlled was never called by anything, so the settings had never affected a consultation — v2.16's `converge` and `adversarial` shapes already run critique-refine off measured consensus. **If you set either key**, remove it: `configure --set ENABLE_REFLECTION=...` now exits non-zero, and both keys are dropped when an existing `.env` is rewritten.
+- **The release gate works again.** Two test suites failed deterministically against any real user configuration, so no release since 2.22.0 could pass its own `npm test`.
+- **`npm test` no longer swallows failures.** A failing assertion followed by a passing one inside the same test function was counted as a pass, across 8 of 13 test functions.
+- **Feature flags can no longer be enrolled as consultants.** A registry that told feature flags apart from custom API agents was 18 entries out of date; it is now complete and pinned by a test.
+- **Preset tables corrected** — they described a debate and peer-review depth the code has not enforced since v2.16, and called `high-stakes`/`max_quality` "all consultants" when they enable 5 and 8 of 11. See the [v2.23.0 release note](docs/releases/v2.23.0.md).
+
 ### v2.22.0
 
 - **`ai-consultants configure`**: a public automatic configurator. It detects your installed CLIs and API keys, picks CLI-first transports, and writes a private XDG config while preserving your existing settings and secrets. Drive it with repeatable `--set KEY=VALUE`, review it with `--interactive` / `--advanced`, or preview it with `--dry-run`. See the [v2.22.0 release note](docs/releases/v2.22.0.md).

package/SKILL.md

@@ -5,10 +5,10 @@ license: MIT
 compatibility: Requires bash, jq, and at least 2 AI CLI tools (agy, codex, vibe, etc.). macOS and Linux.
 metadata:
   author: matteoscurati
-  version: 2.22.0
+  version: 2.24.0
 ---
 
-# AI Consultants v2.22.0 - AI Expert Panel
+# AI Consultants v2.24.0 - AI Expert Panel
 
 **A harness for every question.** Convene a panel of AI "consultants" for coding questions — and let the engine pick *how* they deliberate: a quick read, a convergence loop, an adversarial refutation gate, a tournament, or an exhaustive sweep. Each consultant has a **configurable persona** that shapes its analysis.
 

package/bin/ai-consultants

@@ -185,6 +185,25 @@ EOF
       exit 1
     fi
     mkdir -p "$COMMANDS_DST"
+
+    # Prune commands this project no longer ships, before copying the current
+    # set in. Copying alone leaves a removed command installed forever: the
+    # v2.10.0 consolidation dropped ten, and config-features among them still
+    # tells users to set ENABLE_REFLECTION, which v2.23.0 removed and
+    # `configure --set` now rejects. scripts/install.sh does the same for the
+    # curl | bash path — both entry points must prune or npx users keep them.
+    pruned=0
+    for f in "$COMMANDS_DST"/ai-consultants:*.md; do
+      [ -e "$f" ] || continue
+      if [ ! -f "$COMMANDS_SRC/$(basename "$f")" ]; then
+        rm -f "$f"
+        pruned=$((pruned + 1))
+      fi
+    done
+    if [ "$pruned" -gt 0 ]; then
+      echo "Pruned $pruned slash command(s) removed in a previous release"
+    fi
+
     copied=0
     for f in "$COMMANDS_SRC"/ai-consultants:*.md; do
       [ -f "$f" ] || continue

package/docs/COST_RATES.md

@@ -1,4 +1,4 @@
-# Cost Rates - AI Consultants v2.22.0
+# Cost Rates - AI Consultants v2.24.0
 
 This page documents the per-token rates used by the cost tracking system.
 
@@ -102,14 +102,6 @@ Since v2.5, models are organized into three tiers. Use `apply_model_tier()` or `
 
 **Estimated cost:** $0.08 - $0.15
 
-### With Self-Reflection
-
-**Additional parameters:**
-- Critique + refinement cycle per consultant
-- ~500 additional tokens per cycle
-
-**Estimated cost:** $0.10 - $0.20
-
 ## Budget Management
 
 ### Configuration Variables

package/docs/RECIPES.md

@@ -102,8 +102,6 @@ ORCHESTRATION_MODE=adversarial
 ENABLE_ADVERSARIAL_VERIFY=true
 ENABLE_PEER_REVIEW=true
 PEER_REVIEW_MIN_RESPONSES=3
-ENABLE_REFLECTION=true
-REFLECTION_CYCLES=1
 ```
 
 ```bash

package/docs/cost_rates.json

@@ -1,5 +1,5 @@
 {
-  "version": "2.22.0",
+  "version": "2.24.0",
   "updated": "2026-07-16",
   "models": {
     "_comment_units": "All rates are USD per 1,000 tokens (per-1K). lib/costs.sh computes cost as (token_count / 1000) * rate, so every value here MUST be a per-1K rate = (provider $/1M price) / 1000. Do NOT paste $/MTok figures directly.",

package/docs/releases/v2.23.0.md

@@ -0,0 +1,75 @@
+# Release v2.23.0
+
+**Date:** 2026-07-20
+**Type:** Minor — removes a dead feature flag and repairs the release gate
+**Previous:** v2.22.0
+
+## Highlights
+
+- **`ENABLE_REFLECTION` / `REFLECTION_CYCLES` removed.** The self-reflection module they controlled was never called by anything, so the settings had never affected a consultation.
+- **The release gate works again.** `scripts/release.sh` could not pass its own `npm test` since 2.22.0; two test suites failed deterministically against any real user configuration.
+- **`npm test` now reports failures it used to swallow.** A failing assertion followed by a passing one inside the same test function was counted as a pass.
+- **A registry that could enroll feature flags as consultants is fixed**, and pinned by a test so it cannot drift again.
+- **Preset documentation corrected** — the tables described a deliberation depth the code has not enforced since v2.16.
+
+## What's New
+
+### Self-reflection is gone
+
+`lib/reflection.sh` implemented a generate-critique-refine loop across 452 lines. Nothing ever sourced it. `ENABLE_REFLECTION` and `REFLECTION_CYCLES` were declared in `config.sh` and set by four presets, but no consumer read them — the flags were configuration that promised behavior the tool never performed.
+
+The module had also aged past the codebase. Its `_exec_consultant` hardcoded a three-branch dispatch (Gemini/Codex/Mistral) predating both the 11-consultant roster and the CLI/API transport layer, and its Gemini branch still used the CLI syntax retired in v2.15.0 by the `agy` migration. Wiring it up as written would have produced reflection for 3 of 11 consultants with a broken Gemini path.
+
+It was removed rather than implemented because v2.16 already covers the ground: the `converge` and `adversarial` orchestration shapes run critique-refine driven by measured consensus instead of a fixed cycle count.
+
+### The release gate
+
+`scripts/release.sh` sources `lib/common.sh` — and so `config.sh`, and so `load_user_config` — before running `npm test`. Every setting in the maintainer's own `~/.config/ai-consultants/.env` was therefore exported into the test process, and two suites were not isolated against it:
+
+- **`test_user_config.sh`** — `test_xdg_fallback` unsets `AI_CONSULTANTS_CONFIG_DIR` in the suite's own shell, which it must in order to assert the XDG branch, but nothing restored it. The unset leaked into all 22 following tests, which then read the real user config. Two further tests asserted on real configuration variable names that an ambient export silently replaced.
+- **`test_configure.sh`** — its `run_clean_configure` helper scrubbed the nine API keys but no settings, while `configure`'s `has_explicit_value` reads *any* ambient value as a deliberate user pin. An exported `DEFAULT_PRESET` overrode the value the preservation tests had just written into the file they were asserting on.
+
+Both suites now derive their isolation from the contract they test rather than from a hand-listed subset: `test_user_config.sh` re-establishes an empty config directory before every test, and `test_configure.sh` builds its scrub set from `configure --show-parameters`.
+
+This is why the failure was invisible: `configure` shipped in 2.22.0, so only a maintainer who had actually run it hit the condition, and the first failing suite aborted the run before the second could be seen.
+
+### Test failures that were being swallowed
+
+`test_functions.sh` scores a test function by its exit status, which in bash is that of its last command. A failing assertion followed by a passing one printed `FAIL`, counted as a pass, left the failure total at zero and exited 0. Eight of the thirteen test functions run more than one assertion and were exposed.
+
+The `assert_*` helpers now record failures in a flag the runner consults, so a test fails if either its status or any assertion inside it did.
+
+### Phantom consultants from feature flags
+
+`_discover_custom_api_agents` walks the environment and enrolls any `ENABLE_X=true` as a consultant when an `X_API_URL` is also present and `X` is absent from `KNOWN_FEATURE_FLAGS`. That registry listed 9 entries against 27 live non-consultant flags, so 18 — `ENABLE_PEER_REVIEW` and `ENABLE_HEALTH_GATE` among them — were latent phantom consultants. All 27 are registered, and a test derives the expected set from `config.sh` so the next added flag fails the build instead of drifting silently.
+
+### Preset documentation
+
+The tier-preset tables stated a per-preset debate and peer-review depth that the code does not enforce. Under the default `ORCHESTRATION_MODE=auto` the planner chooses the shape per question: a `DEBATE_ROUNDS` value a preset pins applies only under `ORCHESTRATION_MODE=fixed`, and a `SECURITY`-classified question runs the adversarial shape — peer review included — whatever preset was chosen. `max_quality` was also described as "all consultants" while enabling 8 of the 11-consultant roster.
+
+## Breaking Changes
+
+**`ENABLE_REFLECTION` and `REFLECTION_CYCLES` are no longer accepted parameters.**
+
+No runtime behavior changes — the flags controlled nothing. What changes is the configuration surface:
+
+- `ai-consultants configure --set ENABLE_REFLECTION=...` now exits non-zero instead of accepting the value. A provisioning script that pins either key will abort after upgrading.
+- Both keys are dropped, without warning, the next time `configure` rewrites an existing user `.env`.
+
+## Upgrade Guide
+
+```bash
+# If installed via git clone
+cd ~/.claude/skills/ai-consultants && git pull
+
+# If installed via curl | bash
+curl -fsSL https://raw.githubusercontent.com/matteoscurati/ai-consultants/main/scripts/install.sh | bash
+
+# If installed via npx — nothing to do, npx resolves the published version
+```
+
+Remove `ENABLE_REFLECTION` and `REFLECTION_CYCLES` from any `.env` file or provisioning script that sets them. No other configuration changes are required.
+
+## Contributors
+
+- matteoscurati

package/docs/releases/v2.24.0.md

@@ -0,0 +1,88 @@
+# Release v2.24.0
+
+**Date:** 2026-07-20
+**Type:** Minor — installing now removes slash commands that were deleted in earlier releases
+**Previous:** v2.23.0
+
+## Highlights
+
+- Installing or updating now **removes** `ai-consultants:*.md` slash commands this project no longer ships. Previously they were only ever copied in, so a command deleted upstream stayed on your machine forever.
+- Ten such commands have been dead since v2.10.0 and are still installed on every setup predating it. One of them, `ai-consultants:config-features`, tells you to set `ENABLE_REFLECTION` — removed in v2.23.0, and now rejected by `configure --set`.
+- **Read the breaking-change note below if you wrote your own commands under the `ai-consultants:` prefix.**
+- Both entry points are fixed: `curl | bash` (`scripts/install.sh`) and npx (`ai-consultants install`).
+- `scripts/release.sh` is no longer shipped in the npm package.
+
+## What's New
+
+### Stale slash commands are pruned on install and update
+
+Both installers used to `cp` the current commands into `~/.claude/commands` and nothing else. Nothing ever removed one that had been deleted from the project, so obsolete commands accumulated and stayed invocable.
+
+Measured on a real installation: **14 `ai-consultants:*.md` files present, 4 of which this project still ships.** The other ten were removed by the v2.10.0 command consolidation, thirteen releases ago:
+
+```
+ask-experts       config-check      config-preset     config-strategy
+config-api        config-features   config-status     config-wizard
+config-budget     config-personas
+```
+
+They are not merely clutter. `config-features` instructs you to set `ENABLE_REFLECTION`, a setting v2.23.0 removed; `ai-consultants configure --set ENABLE_REFLECTION=...` now exits non-zero. Following that command's instructions fails.
+
+Both paths now prune before copying and report what they removed:
+
+```
+Pruned 10 slash command(s) removed in a previous release
+Installed 4 slash commands to /Users/you/.claude/commands
+```
+
+### `scripts/release.sh` removed from the npm package
+
+Maintainer-only version-bump automation. `bin/ai-consultants` does not route it, so it was never reachable from the CLI — `npx ai-consultants release` starts a consultation, it does not run the script.
+
+Other scripts that look maintainer-only are **deliberately kept**, having checked what depends on them: `roster_audit.sh` backs the shipped `roster-audit` slash command and `doctor --roster-audit`; `roster_calibrate.sh`, `taste_elo.sh` and `run_calibration.sh` are documented for your own use in `docs/SMART_ROUTING.md`; `install-hooks.sh` is wired to a `package.json` script.
+
+## Breaking Changes
+
+**The prune removes any `ai-consultants:*.md` this project does not ship — including a command you wrote yourself under that prefix.**
+
+There is no way to distinguish a command you authored in this namespace from one of ours that was deleted upstream: both are simply files we do not ship. The `ai-consultants:` prefix is this project's namespace, and the prune treats it as such.
+
+If you have custom commands there, **rename them out of the prefix before upgrading**:
+
+```bash
+mv ~/.claude/commands/ai-consultants:mine.md ~/.claude/commands/my-consultants:mine.md
+```
+
+Commands belonging to other tools are never considered — only the `ai-consultants:` prefix is examined.
+
+## Upgrade Guide
+
+```bash
+# npx (no install)
+npx ai-consultants "question"
+
+# refresh the slash commands (this is what prunes)
+npx ai-consultants install
+
+# git clone install
+cd ~/.claude/skills/ai-consultants && git pull
+
+# curl | bash install
+curl -fsSL https://raw.githubusercontent.com/matteoscurati/ai-consultants/main/scripts/install.sh | bash
+```
+
+No configuration changes required.
+
+Note for `git pull` users: pulling updates the checkout but does **not** re-run the installer, so it does not prune. Run `scripts/install.sh` or `ai-consultants install` afterwards to refresh the installed commands.
+
+## Commits
+
+- `315afb6` fix: harden the release and install tooling after a max code review
+- `48892e4` fix(install): prune slash commands removed in earlier releases
+- `384a1cd` chore: move the verify skill out of the distributed repo
+- `95d2e29` chore(release): move the release skill out of the distributed repo
+- `6a4e70d` chore(release): automate the release and publish pipeline
+
+## Contributors
+
+- Matteo Scurati

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "ai-consultants",
-  "version": "2.22.0",
+  "version": "2.24.0",
   "description": "A harness for every question - a panel of up to 11 AI models that picks its own deliberation strategy per question (convergence, adversarial verification, tournament) and checks its work before answering",
   "bin": {
     "ai-consultants": "bin/ai-consultants"
@@ -11,6 +11,7 @@
     "!scripts/test_*.sh",
     "!scripts/test_fixtures/",
     "!scripts/lib/test_helpers.sh",
+    "!scripts/release.sh",
     "templates/",
     ".env.example",
     ".claude/commands/",

package/references/configuration.md

@@ -164,8 +164,6 @@ ENABLE_DEBATE=false          # Multi-agent debate
 DEBATE_ROUNDS=1              # Used by ORCHESTRATION_MODE=fixed
 ENABLE_PEER_REVIEW=false     # Anonymous peer review
 PEER_REVIEW_MIN_RESPONSES=3  # Minimum panel size for peer review
-ENABLE_REFLECTION=false      # Generate -> critique -> refine
-REFLECTION_CYCLES=1
 ENABLE_PANIC_MODE=auto       # auto | always | never
 ENABLE_SMART_ROUTING=false   # Category-based consultant selection
 ENABLE_COST_TRACKING=true    # Track API usage costs

package/scripts/config.sh

@@ -374,16 +374,6 @@ PANIC_EXTRA_DEBATE_ROUNDS="${PANIC_EXTRA_DEBATE_ROUNDS:-1}"
 # Keywords that trigger panic mode when found in responses
 PANIC_KEYWORDS="${PANIC_KEYWORDS:-uncertain|maybe|not sure|possibly|unclear|depends|hard to say|difficult to determine}"
 
-# =============================================================================
-# SELF-REFLECTION (v2.0)
-# =============================================================================
-
-# Enable auto-reflection (generate-critique-refine)
-ENABLE_REFLECTION="${ENABLE_REFLECTION:-false}"
-
-# Number of reflection cycles per response
-REFLECTION_CYCLES="${REFLECTION_CYCLES:-1}"
-
 # =============================================================================
 # SMART ROUTING (v2.0)
 # =============================================================================
@@ -761,7 +751,7 @@ apply_model_tier() {
 #   high-stakes  - Broad premium panel + debate (maximum rigor)
 #
 # Quality Tiers (v2.5):
-#   max_quality  - All consultants + premium models + debate + reflection
+#   max_quality  - 8 of 11 consultants + premium models + peer review
 #   medium       - 4 consultants + standard models + light debate
 #   fast         - 2 consultants + economy models, no debate
 #
@@ -773,7 +763,7 @@ _disable_all_consultants() {
     export ENABLE_CURSOR=false ENABLE_KIMI=false ENABLE_CLAUDE=false
     export ENABLE_QWEN3=false ENABLE_GLM=false ENABLE_GROK=false
     export ENABLE_DEEPSEEK=false ENABLE_MINIMAX=false
-    export ENABLE_DEBATE=false ENABLE_REFLECTION=false
+    export ENABLE_DEBATE=false
 }
 
 # Apply a preset configuration
@@ -800,7 +790,6 @@ apply_preset() {
             export ENABLE_GEMINI=true ENABLE_CODEX=true ENABLE_MISTRAL=true
             export ENABLE_CURSOR=true ENABLE_CLAUDE=true
             export ENABLE_DEBATE=true DEBATE_ROUNDS=2
-            export ENABLE_REFLECTION=true REFLECTION_CYCLES=1
             ;;
         security)
             export ENABLE_GEMINI=true ENABLE_CODEX=true
@@ -820,7 +809,6 @@ apply_preset() {
             export ENABLE_CURSOR=true ENABLE_KIMI=true
             export ENABLE_CLAUDE=true ENABLE_QWEN3=true ENABLE_MINIMAX=true
             export ENABLE_DEBATE=true DEBATE_ROUNDS=3
-            export ENABLE_REFLECTION=true REFLECTION_CYCLES=2
             export ENABLE_PEER_REVIEW=true
             ;;
         medium)
@@ -834,7 +822,7 @@ apply_preset() {
             # Super fast - economy models, minimal consultants
             apply_model_tier "economy"
             export ENABLE_GEMINI=true ENABLE_CODEX=true
-            export ENABLE_DEBATE=false ENABLE_REFLECTION=false
+            export ENABLE_DEBATE=false
             export ENABLE_COMPACT_REPORT=true
             ;;
         *)
@@ -853,7 +841,7 @@ list_presets() {
 Available presets:
 
 Quality Tiers (v2.5):
-  max_quality  All consultants + premium models + debate + reflection
+  max_quality  8 of 11 consultants + premium models + peer review
   medium       4 consultants + standard models + light debate
   fast         2 consultants + economy models, no debate
 
@@ -873,4 +861,4 @@ EOF
 # VERSION
 # =============================================================================
 
-AI_CONSULTANTS_VERSION="2.22.0"
+AI_CONSULTANTS_VERSION="2.24.0"

package/scripts/consult_all.sh

@@ -82,7 +82,7 @@ Options:
                          balanced     - 4 models, good coverage
                          thorough     - 5 models, comprehensive
                          high-stakes  - All models + debate
-                         max_quality  - All + premium models + debate + reflection
+                         max_quality  - 8 of 11 consultants + premium models + peer review
                          medium       - 4 models + standard models + light debate
                          fast         - 2 models + economy models, no debate
                          security     - Security-focused + debate

package/scripts/install.sh

@@ -138,6 +138,51 @@ check_dependencies() {
     return 0
 }
 
+# Remove installed slash commands that this project no longer ships.
+#
+# The install step only copies files in, so a command deleted from the repo
+# would live in the user's commands directory forever. v2.10.0's command
+# consolidation dropped ten of them, and every install predating it still
+# carries all ten — including config-features, which instructs users to set
+# ENABLE_REFLECTION, a setting v2.23.0 removed and `configure --set` now
+# rejects with a non-zero exit.
+#
+# Scoped to this project's own "ai-consultants:*.md" namespace, so another
+# tool's commands are never considered. Within that namespace it removes
+# anything this repo does not ship — including a command the user authored
+# under the same prefix. That is the intended bound: the prefix is this
+# project's namespace, and there is no way to tell a user's file there from
+# one of ours that was deleted upstream.
+# Sets PRUNED_COUNT rather than echoing, so log output can't be mistaken for it.
+prune_removed_commands() {
+    local commands_dir="$1" repo_commands_dir="$2" installed
+    PRUNED_COUNT=0
+
+    [[ -d "$commands_dir" ]] || return 0
+
+    for installed in "$commands_dir"/ai-consultants:*.md; do
+        [[ -e "$installed" ]] || continue   # glob didn't match: nothing installed yet
+        if [[ ! -f "$repo_commands_dir/$(basename "$installed")" ]]; then
+            rm -f "$installed"
+            PRUNED_COUNT=$((PRUNED_COUNT + 1))
+        fi
+    done
+}
+
+# Test hook: `AI_CONSULTANTS_INSTALL_DEFINE_ONLY=1 source scripts/install.sh`
+# yields the functions above and nothing else.
+#
+# It sits HERE — above the argument parser and the uninstall block — because a
+# sourced script inherits the CALLER's positional parameters. With the guard
+# below them, `bash scripts/test_install.sh --uninstall` had install.sh parse
+# the suite's own flag and run the real uninstall: rm -rf on the user's
+# installed skill plus every ~/.claude/commands/ai-consultants*.md. Anything
+# added below this point must be safe to skip, and nothing above it may act.
+if [[ -n "${AI_CONSULTANTS_INSTALL_DEFINE_ONLY:-}" ]]; then
+    # shellcheck disable=SC2317  # exit fallback for script-mode load
+    return 0 2>/dev/null || exit 0
+fi
+
 # =============================================================================
 # ARGUMENT PARSING
 # =============================================================================
@@ -203,10 +248,6 @@ if [[ "$UNINSTALL_MODE" == "true" ]]; then
     exit 0
 fi
 
-# =============================================================================
-# INSTALL / UPDATE
-# =============================================================================
-
 print_header
 
 # Check dependencies
@@ -260,6 +301,11 @@ if [[ "$INSTALL_COMMANDS" == "true" ]]; then
     log_info "Installing slash commands..."
 
     if [[ -d "$INSTALL_DIR/.claude/commands" ]]; then
+        prune_removed_commands "$COMMANDS_DIR" "$INSTALL_DIR/.claude/commands"
+        if [[ "$PRUNED_COUNT" -gt 0 ]]; then
+            log_info "Pruned $PRUNED_COUNT slash command(s) removed in a previous release"
+        fi
+
         cp "$INSTALL_DIR"/.claude/commands/*.md "$COMMANDS_DIR/" 2>/dev/null || true
         cmd_count=$(find "$INSTALL_DIR/.claude/commands" -maxdepth 1 -name '*.md' -type f 2>/dev/null | wc -l | tr -d ' ')
         log_success "Installed $cmd_count slash commands"