amicus 1.0.0

package/skills/second-opinion/COUNCIL-DESIGN.md

@@ -0,0 +1,227 @@
+# Second Opinion v3 — "LLM Council" Design
+
+_Status: implemented (v3). v2 (2026-06-03) added the council mechanics; v3 (2026-06-10) swapped the
+transport onto the Amicus fanout/JSON engine primitives. v2 history lives in git
+(`V2-COUNCIL-DESIGN.md`, deleted at v3)._
+_Design for `SKILL.md` and `MODEL-NOTES.md` of the `second-opinion` skill._
+
+## 1. Intent
+
+Upgrade `second-opinion` by porting the best mechanics of the **LLM Council** web-app
+pattern — peer cross-review, anonymized
+ranking, aggregate scoring, and a designated chairman — into the skill, while keeping its
+existing strengths (model recommendation, sidecar orchestration, tiered accept/deny,
+reviewed-copy output, and the MODEL-NOTES self-improvement loop).
+
+This is the **"full port" (option C)**: cross-review + anonymization + aggregate scoring +
+non-Claude chairman + per-model inspectable artifacts.
+
+## 2. Core framing & principles
+
+- **Second-opinion is a *secondary* review tool.** By the time it runs, **Claude has already
+  given its opinion** in the main conversation. The skill exists to bring in *independent
+  outside* views.
+- **The council is the *non-Claude* bench by default.** Council members are models from
+  families other than the orchestrator (Gemini, DeepSeek, GPT, etc.). Claude is **not** a
+  first-opinion council member unless the optional "Claude in the council" toggle is on (§5.4)
+  — and even then it is judged but does not vote or chair.
+- **Claude's role shrinks to orchestrator:** prep material, recommend the council, anonymize,
+  drive the stages, score, present accept/deny, write files. **Claude does not synthesize the
+  verdict** — a council model chairs that (§5.3).
+- **Subject of review.** The novel cross-review step has models critique **each other's
+  reviews of your artifact** — *not* re-review the artifact. (LLM Council ranks the models'
+  own answers; here the "answers" are the reviews.)
+- **This is an executed skill, not an app.** All council logic (anonymize → rank → aggregate →
+  chair) is prose workflow Claude performs while driving the `amicus` CLI. v3 note: the *transport*
+  is now engine-native — each review wave is ONE `amicus fanout --json` call returning structured
+  run documents — but scoring, tallying, anonymization, and synthesis remain Claude's manual work.
+  No backend, no parsing code beyond reading JSON fields.
+
+## 3. What changes vs. v1
+
+| Area | v1 | v3 (this design) |
+|---|---|---|
+| Independent reviews | ✅ Phase 2 parallel sidecars | ✅ Stage 1 — now emits a **structured findings list** |
+| Cross-review | ❌ none | ⭐ **Stage 2** — anonymized peer ranking **+** per-finding adjudication |
+| Synthesis | Claude synthesizes | ⭐ **Council-model chair** synthesizes; Claude only presents |
+| Decision tiers | Claude's consensus/divergence read | ⭐ **Peer-validated** tiers (Confirmed / Contested / Singleton) |
+| Scoring | none | ⭐ Reviewer **street-cred** + per-finding **peer-confidence** |
+| Artifacts | reviewed copy + report | + per-model raw reviews, cross-review matrix, chair verdict (run folder) |
+| MODEL-NOTES | per-model quirks | + **reviewer-reliability** rolling table feeding recommendations |
+| Transport (v3) | N parallel `start` calls + prose-scraping | ⭐ one `fanout --json` wave per stage; briefings via `--prompt-file`; JSON status/summary parsing |
+
+Preserved unchanged: intake/criteria intake, the MODEL-NOTES operating rules, reviewed-copy vs
+standalone-report logic, cost guardrail, and the Stage 6 approval-gated MODEL-NOTES update.
+
+## 4. The council flow
+
+Run as ordered phases; track as todos. **Three sequential waves of model calls** (Stage 1 →
+2 → 3 each depend on the prior); within each wave, models run in parallel.
+
+### Stage 0 — Intake & prep
+- Confirm **source material**, **the analysis**, **the criteria** (ask only for what's missing).
+- Prepare material for council models per MODEL-NOTES (extract clean text from links / large /
+  marked-up sources to a small temp file; small text used as-is).
+- Pick the council: **non-Claude models, default 3 from different families** (enough voices for
+  a real cross-review + a tie-breaker). Recommend ranked-by-fit (consult MODEL-NOTES
+  reviewer-reliability), state cost, **disclose run shape up front** — "~2N+1 calls across 3
+  waves, ~X min" — and **wait for confirmation**. Honor the cost guardrail.
+- **Scale-down is explicit:** 1 model = thorough single pass (skip Stage 2 & chair); 2 = cross-
+  review works but ranking is thin; 3 = default deep council.
+- **Optional — "Claude in the council" (default off):** offer to add Claude as a *judged*
+  contributor so the bench's verdict on Claude's own take is visible. When on, Claude adds one
+  Stage-1 review to the bundle but does **not** judge (Stage 2) or chair (Stage 3). See §5.4.
+
+### Stage 1 — Independent reviews
+- All council models review **your artifact** via ONE fanout wave (see SKILL.md Stage 1 for the
+  canonical command). The wave JSON returns every leg's status + review in one parse. A red-team
+  reviewer with a distinct brief runs as a separate concurrent `amicus start --json` call alongside the wave (fanout legs share
+  one prompt by design).
+- Required structured output: a **findings list**, each finding = `id · claim · severity
+  (blocker/major/minor/nit) · location (section/quote) · rationale`, plus a short overall take.
+- Save each raw review to the run folder (§6).
+- If **"Claude in the council"** is on, Claude also produces a **fresh** review in the same
+  findings format (a new structured pass on the artifact, regardless of any upstream feedback)
+  and adds it to the bundle as one more review (§5.4).
+
+### Stage 2 — Cross-review (the headline)
+- Claude builds **one shared anonymized bundle**: all Stage 1 reviews relabeled **Review A/B/C…**,
+  with a private label↔model map Claude keeps (§5.1).
+- The **same bundle** goes to **every** council model — exactly fanout's shared-prompt model: one
+  wave call distributes it, and each judge is asked to do two things on the bundle:
+  1. **Rank** the reviews by accuracy + insight, ending with a parseable block:
+     `FINAL RANKING:` then `1. Review C` / `2. Review A` … (LLM Council's format).
+  2. **Adjudicate findings** — for each finding in the bundle: `agree | dispute | neutral` +
+     one-line reason (an "I missed this, it's valid" counts as agree). Findings are referenced
+     by **review label + finding id** (e.g. `A2` = Review A's 2nd finding) so Claude can map
+     each verdict back to the originating model and claim when tallying.
+- Each model **unknowingly ranks/adjudicates its own review too** — this is the anti-favoritism
+  mechanism, not a bug (§5.1).
+- Claude de-anonymizes for scoring/display only.
+
+### Stage 3 — Council-chair synthesis
+- A designated **non-Claude** chair (recommended + confirmed in Stage 0/launch) receives all
+  reviews + rankings + adjudications and writes the **synthesized verdict**, weighted by
+  street-cred and peer-confidence. Independent of Claude.
+- Chair selection & fallback: §5.3.
+
+### Stage 4 — Tiered decisions (peer-validated)
+- **Consensus tier** = **Confirmed** findings (peers agree) → offer one **bulk accept/deny**
+  (user may name exceptions).
+- **Judgment tier** = **Contested** (peers dispute/split) or **Singleton** (only the raiser)
+  findings → present **each individually**, showing the dissent and which model raised/disputed it.
+- Record every decision (accepted / denied / modified).
+
+### Stage 5 — Outputs
+- **Editable source** → write `<stem>-reviewed.<ext>` next to the original (accepted changes
+  only; validate structural integrity). **Fixed source** → standalone reviewed report.
+- Always write the run folder artifacts (§6).
+
+### Stage 6 — Capture lessons (compounding)
+- Reflect on failures/mitigations and briefing wording, as today.
+- **Additionally** update the per-model **reviewer-reliability** table (§7).
+- **Show the proposed MODEL-NOTES diff and get approval before writing.** Keep it tight.
+
+## 5. Key mechanics
+
+### 5.1 Anonymization (shared bundle)
+- After Stage 1, Claude assembles **one** bundle with stable labels Review A/B/C… and keeps a
+  private map (e.g., `Review A → deepseek`, `Review B → gemini`, …).
+- The identical bundle is sent to every judge. Because a model can't tell which review is its
+  own, it ranks/adjudicates all of them honestly; symmetric self-bias washes out across judges.
+- Claude only de-anonymizes when computing scores and writing the matrix/report for the user.
+- When "Claude in the council" is on, Claude's own review is anonymized into the **same** bundle
+  and judged blind by the council models. Claude never ranks/adjudicates (it holds the map) —
+  the asymmetry detailed in §5.4.
+
+### 5.2 Scoring (Claude tallies by hand — no code required)
+- **Street cred** = each model's **average rank position** across all judges' `FINAL RANKING:`
+  blocks (lower = better), exactly as LLM Council's aggregate. Surface as a small table.
+- **Per-finding peer-confidence** = qualitative tier from the adjudications:
+  - **Confirmed** — agrees clearly outweigh disputes (and ≥2 judges engaged).
+  - **Contested** — meaningful split or explicit disputes.
+  - **Singleton** — only the original raiser; others neutral/silent.
+  These tiers drive Stage 4. Claude exercises judgment at the margins; no rigid formula.
+
+### 5.3 Chair selection & fallback
+- Default: Claude **recommends a non-Claude chair** from the council each run (often the
+  strongest reasoner / best reviewer-reliability) and the user confirms at launch.
+- The chair **may** also be a Stage-1 council member (it sees the anonymized bundle + scores).
+- **Fallback order if the chair fails:** re-run → promote next-best council model →
+  **Claude chairs only as last resort, with explicit disclosure** that the verdict is no longer
+  fully independent.
+
+### 5.4 Claude in the council (optional, default off)
+Lets you see how the bench judges Claude's *own* take.
+- **Asymmetric by necessity.** Claude is the orchestrator and holds the label↔model map, so it
+  cannot judge blind. Therefore Claude **contributes a review to be judged but does not vote
+  (Stage 2) or chair (Stage 3).** The verdict stays independent.
+- **Which review: always fresh** — Claude does a new structured Stage-1 review on the artifact
+  every time it's enabled (not a formalization of upstream feedback).
+- **Readout — "How Claude's review fared":** Claude's street-cred rank among peers and the
+  Confirmed/Contested/Singleton split of its findings, reported in the matrix and report.
+- **Integrity:** when Claude presents results, it reports the bench's verdict on its own review
+  at face value — no defending or re-litigating.
+
+## 6. Outputs & naming
+One tidy run folder: `output/<stem>-council/` (or `./second-opinion/<stem>-council/` if no
+`output/` exists):
+- `review-<model>.md` ×N — raw Stage 1 reviews (plus `review-claude.md` when "Claude in the
+  council" is on)
+- `crossreview-matrix.md` — adjudication grid + street-cred table (de-anonymized)
+- `verdict.md` — the chair's synthesis
+- `report.md` — synthesis + decision log + what was applied (+ the "How Claude's review fared"
+  readout when "Claude in the council" is on)
+- `<stem>-reviewed.<ext>` — written **next to the original**, as today (editable sources only)
+- Temp extracts get a clearly-temporary name and are cleaned up at the end.
+
+## 7. MODEL-NOTES reviewer-reliability
+Add a compact rolling table consulted in Stage 0 and updated (with approval) in Stage 6:
+
+| model | runs | avg street-cred | confirm-rate | notes |
+|---|---|---|---|---|
+
+- **avg street-cred** — running average rank when peer-ranked.
+- **confirm-rate** — share of this model's findings that ended up **Confirmed** by peers.
+- Used to justify recommendations ("DeepSeek findings peer-confirm ~80% → strong default
+  reviewer"). Kept tight per the existing no-bloat rule; merge/prune rather than append.
+
+## 8. Gating, cost, degradation & failure handling
+
+- **Gating:** council is the default identity but scales down (§ Stage 0). Always disclose run
+  shape/cost and confirm before launching.
+- **Cost guardrail (unchanged):** never `o3`/`o3-pro` unless the user asks by name; warn on cost.
+- **Degradation (judge-count thresholds, unchanged):** gating counts **non-Claude judges**.
+  1 judge → single-pass (no Stage 2/chair); if the bench drops below 2 mid-run, degrade to
+  single-pass and disclose. 2 judges → Stage 2 runs but note the thin ranking. ("Claude in the
+  council" adds a *judged* review but **no** judge.)
+- **Wave-degrade rules (v3):** leg failures are read from the wave document
+  (`status: partial`, `counts`, per-leg `status`/`error`):
+  - **Stage 1:** a leg ends `error`/`timeout`/`crashed`/`aborted` → proceed when ≥2 reviews
+    survive; below 2, offer a re-run of the dead leg or a disclosed downgrade to single-pass.
+  - **Stage 2:** a judge leg dies → tally rankings/adjudications over the surviving judges and
+    disclose the reduced bench in `crossreview-matrix.md`. Tier definitions are unchanged (they
+    already count "judges engaged").
+  - **Stage 3:** chair failure uses the same fallback chain (re-run → promote next-best non-Claude
+    → Claude chairs with explicit disclosure).
+- **Run stats (v3):** `report.md` includes a per-leg table (model, status, durationMs) read from
+  the wave/run documents. The schema carries no cost data — never invent cost figures.
+- **Transient failures:** provider 502s etc. → re-run the affected leg (solo `start --json`) or
+  the wave; never present a half-finished run as an answer.
+
+## 9. Non-goals (YAGNI)
+- No web UI, API server, or persistent conversation store (LLM Council's app shell).
+- No code/backend for scoring or parsing — Claude does it inline.
+- No automatic MODEL-NOTES writes — always approval-gated.
+- Claude is **not** a council member by default; it joins only via the opt-in toggle (§5.4),
+  and even then it is judged-but-non-voting/non-chairing.
+
+## 10. Open questions
+- None blocking. Possible later refinement: a numeric peer-confidence score instead of the
+  three qualitative tiers, if tiers prove too coarse in practice.
+
+## 11. Implementation surface
+- `SKILL.md` — the Stage 0–6 council flow on the v3 transport.
+- `MODEL-NOTES.md` — reviewer-reliability table, per-model quirks, cost guardrail, Stage-2
+  briefing tips. Engine workarounds that F1/F2/F4 made obsolete were pruned at v3.
+- No other files.

package/skills/second-opinion/MODEL-NOTES.md

@@ -0,0 +1,104 @@
+# MODEL-NOTES — Operating Lessons for Amicus Models
+
+This file is the `second-opinion` skill's evolving memory of **how to actually drive each model
+well**. Read it before Stage 0 (council selection and launch); update it, with the user's
+approval, at the end of each run (Stage 6). Keep it tight — merge and prune rather than append.
+
+_Last updated: 2026-06-10 (v3 migration: engine workarounds pruned — see changelog)._
+
+## Global operating rules (all models)
+- **Council runs are headless by design** (autonomous batch work): `fanout` is headless by
+  definition; solo runs use `--no-ui`. Interactive GUI sessions are for the `sidecar` chat skill,
+  not councils.
+- **`--agent Plan`** for review/analysis — read-only, so a model can't accidentally edit the source.
+- **`--no-context`** always — council briefings are self-contained; don't drag the host
+  conversation in.
+- **`--summary-length verbose`** — the analysis IS the deliverable; don't let it get summarized away.
+- **Briefings via `--prompt-file`** (temp `_tmp-*.md` files in the run folder — `output/<stem>-council/`,
+  defined in SKILL.md Stage 0) — no size cap, no shell-quoting hazards. Never inline a briefing as
+  a CLI argument.
+- **Run in the background (`run_in_background: true`); you're notified on completion. Don't poll.**
+- **Read results from the JSON documents** (`--json`): a wave's `legs[].summary` / a run's
+  `summary` is the model's output; `status`/`error`/`counts` are ground truth for failures. Never
+  scrape stderr logs to judge success.
+- **Transient provider errors** (502s, connection drops): re-run the affected leg (solo
+  `amicus start --json`, same briefing file) or the wave — see per-model notes for
+  model-specific signals. Never present a half-finished run as an answer.
+- **Credentials:** keys live in `~/.config/amicus/.env` (legacy `~/.config/sidecar/.env` still
+  read). Configure with `amicus setup`.
+
+## Stage-2 cross-review briefing tips
+
+- **Send the same anonymized bundle to every judge** — stable labels Review A/B/C…, no model
+  names — so rankings are directly comparable (one fanout wave distributes it).
+- **Require a `FINAL RANKING:` block** at the end of the response (e.g. `1. Review C / 2. Review
+  A …`), plus a per-finding `agree | dispute | neutral` verdict with a one-line reason for each
+  finding referenced by label+id (e.g. `A2`).
+
+## Per-model notes
+
+### Gemini  (`--model gemini`)
+- **Strengths:** fast, very large context. Good for broad sweeps and long documents.
+- **Quirk:** tends to narrate intentions and pad with preamble; instruct it to emit the
+  structured output verbatim, without preamble. (Historical: its narrate-then-glob habit used to
+  trip the old headless poller; the engine handles tool-call gaps now — F1.)
+- Don't trust its self-reported version string ("I am gemini-X") as ground truth.
+
+### DeepSeek  (`--model deepseek` → via OpenRouter)
+- **Strengths:** resilient; produces strong, well-structured, well-cited critical analysis. A good
+  default reviewer and a proven chair.
+- **Quirk:** occasional transient 502 mid-run → re-run the leg.
+
+### GPT  (`--model gpt` → via OpenRouter)
+- **Strengths:** reachable via the OpenRouter key; resilient; very thorough structured critique
+  (25 findings on a 1-page framework). Cleanly separates the review criteria.
+- **Quirks:** verbose — peers dinged it for volume-over-judgment (good coverage, lower
+  discrimination); **self-ranked its own review #1** in cross-review → discount self-votes.
+
+### (others — add as used)
+- Opus / o-series etc. are reachable via amicus **if their API keys are configured**. Add notes
+  here the first time each is used.
+
+## Reviewer-reliability table
+
+Consulted in Stage 0 (council selection) and updated with approval in Stage 6.
+
+- **avg street-cred** — rolling average of this model's per-run street-cred (mean rank position
+  across judges' `FINAL RANKING:` blocks; lower = better).
+- **confirm-rate** — share of this model's findings that reached the **Confirmed** tier (agrees
+  outweigh disputes, ≥ 2 judges engaged).
+
+| model | runs | avg street-cred | confirm-rate | notes |
+| --- | --- | --- | --- | --- |
+| deepseek | 1 | 2.33 | 100% (12/12) | strong synthesis, resilient; chaired well |
+| gpt | 1 | 2.67 | 92% (23/25) | thorough but verbose; self-ranked #1 → discount; OpenRouter |
+| gemini | 1 | 3.67 | 89% (8/9) | fast, large-context; more absolute/adversarial ("blocker" inflation); ranked lowest |
+
+_Scale note: the 2026-06-04 run used a 4-review pool (Claude in-council), so street-cred is on a
+1–4 scale rather than 1–3 — treat these as run-1 baselines, not directly comparable to future
+3-model runs. Merge/prune rather than append._
+
+## Cost guardrail
+- **Never** use `o3` / `o3-pro` unless the user explicitly asks for it by name — these cost
+  roughly $10–60+ per request. Warn about cost before proceeding even when asked.
+
+## General
+- Model citations are usually real but **verify any load-bearing reference before publishing**;
+  watch for loosely-attached attributions (e.g., a real paper cited for the wrong claim).
+- Prefer models from **different families** for genuinely independent opinions.
+
+## Lessons changelog
+- **2026-06-03** — Seeded from the study-guide review (Gemini + DeepSeek). Found the (since-fixed)
+  headless poller trap and Gemini's narrate-then-glob pattern; confirmed DeepSeek's resilience and
+  its occasional transient 502.
+- **2026-06-03** — v2 council upgrade: added cross-review (Stage-2 anonymized peer ranking +
+  per-finding adjudication) and reviewer-reliability tracking.
+- **2026-06-04** — Trusst messaging-framework council (Gemini + GPT + DeepSeek + Claude-in-council;
+  DeepSeek chair). First GPT use → per-model note. First scored reviewer-reliability rows
+  (deepseek 2.33/100%, gpt 2.67/92%, gemini 3.67/89%; 1–4 scale).
+- **2026-06-10** — v3 migration: transport moved to `fanout --json` + `--prompt-file` (F4); pruned
+  the obsolete engine workarounds they replaced — the headless-poller trap + single-read/no-glob/
+  no-narration ritual + "Polling loop exited" false-alarm note (fixed by F1), the ~32 KB inline-arg
+  cap (superseded by `--prompt-file`), the absolute-path/cwd trap (fixed by F2), and the
+  GUI-hangs-on-this-machine rule (resolved 2026-06-10; headless stays the council default by
+  design). Config path updated to `~/.config/amicus/.env`.