dsh-aris-panel 0.1.0

package/skills/meta-optimize/SKILL.md

@@ -0,0 +1,437 @@
+---
+name: meta-optimize
+description: "Analyze ARIS usage logs and propose optimizations to SKILL.md files, reviewer prompts, and workflow defaults. Outer-loop harness optimization inspired by Meta-Harness (Lee et al., 2026). Use when user says \"优化技能\", \"meta optimize\", \"improve skills\", \"分析使用记录\", or wants to optimize ARIS's own harness components based on accumulated experience."
+argument-hint: "[target-skill-or-all]"
+allowed-tools: Bash(*), Read, Grep, Glob, mcp__codex__codex, mcp__codex__codex-reply
+---
+
+# Meta-Optimize: Outer-Loop Harness Optimization for ARIS
+
+Analyze accumulated usage logs and propose optimizations for: **$ARGUMENTS**
+
+## Privilege boundary — this skill is a READ-ONLY PRODUCER
+
+meta-optimize **proposes**; it does not **land**. The mutation of the skill corpus
+is the exclusive job of a separate, human-invoked skill: [`/meta-apply`](../meta-apply/SKILL.md).
+This split is structural, not advisory — it is why a missed instruction cannot let
+this loop apply its own patch (the self-acquittal failure mode):
+
+- **No `Write`/`Edit` tool.** This skill cannot edit a SKILL.md / shared-reference /
+  any corpus file with the frictionless mutators. Its only outputs are the REPORT and
+  staged patch files, written under `.aris/meta/` (a scratch area, never the corpus).
+- **No apply step.** There is no in-skill "apply the patch" path (see Step 6). The
+  producer ends by *staging* approved patches for `/meta-apply`; a human must then
+  invoke `/meta-apply` to land them. That human action is the landing gate.
+- **`Bash` writes to the corpus are filtered, not impossible — be honest about the
+  layers.** What IS fully closed: the *accidental / in-flow* self-acquittal — this skill
+  has no `Write`/`Edit` and no apply step, so an honest run cannot slip into editing the
+  corpus. Defense-in-depth: install the
+  [`corpus_write_guard`](../../templates/claude-hooks/corpus_write_guard.json) PreToolUse
+  hook (like `meta_logging.json`), which DENIES the common Bash shell-writes (`>`, `tee`,
+  `sed -i`, `cp`/`mv`, `touch`, `open(...,'w')`) to corpus paths. **This is a blacklist,
+  NOT a complete sandbox** — a *deliberately* obscured Bash write (`git apply`, `patch`,
+  `$var`/absolute paths, language file APIs) is not all caught. **Full structural
+  prevention requires either removing this skill's `Bash` or an FS sandbox** — over-built
+  for a not-yet-load-bearing producer, so deferred to when the gate carries real
+  auto-modification volume (a brick-3 trigger). The intended backstop against a deliberate
+  write is **detection, not prevention** — a corpus change with no valid/current
+  `provenance` stamp (content-hash mismatch) *would be* catchable in a pre-push integrity
+  check — but that verifier is **NOT yet built** (`provenance.py` has `content_hash` but no
+  integrity-check subcommand, and no pre-push hook runs one). So today the deliberate-write
+  case is neither prevented nor actively detected; track the integrity verifier as a
+  follow-up before this producer goes load-bearing. Its legitimate Bash writes go only to
+  `.aris/meta/`.
+
+See [`shared-references/acceptance-gate.md`](../shared-references/acceptance-gate.md):
+a loop can DRIVE (propose, review) same-model, but the ACQUITTAL that lands a change
+must be cross-model (Step 4 jury) **and** the landing must be a separate human-gated
+act (`/meta-apply`).
+
+## Context
+
+ARIS is a **research harness** — a system of skills, bridges, workflows, and artifact contracts that wraps around LLMs to orchestrate research. This skill implements a prototype **outer loop** that observes how the harness is used and proposes improvements to the harness itself (not to the research artifacts it produces).
+
+Inspired by Meta-Harness (Lee et al., 2026): the key insight is that harness design matters as much as model weights, and harness engineering can be partially automated by logging execution traces and using them to guide improvements.
+
+## What This Skill Optimizes (Harness Components)
+
+| Component | Example | Optimizable? |
+|-----------|---------|:---:|
+| SKILL.md prompts | Reviewer instructions, quality gates, step descriptions | Yes |
+| Default parameters | `difficulty: medium`, `MAX_ROUNDS: 4`, `threshold: 6/10` | Yes |
+| Convergence rules | When to stop the review loop, retry counts | Yes |
+| Workflow ordering | Skill chain sequence within a workflow | Yes |
+| Artifact schemas | What fields go in EXPERIMENT_LOG.md, idea-stage/IDEA_REPORT.md | Cautious |
+| MCP bridge config | Which reviewer model, routing rules | No (infra) |
+
+**Not optimized**: The research artifacts themselves (papers, code, experiments). That's what the regular workflows do.
+
+## Prerequisites
+
+1. **Logging must be active.** Copy `templates/claude-hooks/meta_logging.json` into your project's `.claude/settings.json` (or merge the hooks section).
+2. **Sufficient data.** At least 5 complete workflow runs logged in `.aris/meta/events.jsonl`. The skill will check and warn if insufficient.
+
+## Workflow
+
+### Step 0: Check Data Availability
+
+```bash
+EVENTS_FILE=".aris/meta/events.jsonl"
+if [ ! -f "$EVENTS_FILE" ]; then
+    echo "ERROR: No event log found at $EVENTS_FILE"
+    echo "Enable logging first: copy templates/claude-hooks/meta_logging.json into .claude/settings.json"
+    exit 1
+fi
+
+EVENT_COUNT=$(wc -l < "$EVENTS_FILE")
+SKILL_INVOCATIONS=$(grep -c '"skill_invoke"' "$EVENTS_FILE" || echo 0)
+SESSIONS=$(grep -c '"session_start"' "$EVENTS_FILE" || echo 0)
+
+echo "📊 Event log: $EVENT_COUNT events, $SKILL_INVOCATIONS skill invocations, $SESSIONS sessions"
+
+if [ "$SKILL_INVOCATIONS" -lt 5 ]; then
+    echo "⚠️  Insufficient data (<5 skill invocations). Continue using ARIS normally and re-run later."
+    exit 0
+fi
+
+# Bottleneck succession: what did the LAST cycle say was the limiting stage?
+BOTTLENECK_LOG=".aris/meta/bottleneck_log.jsonl"
+if [ -f "$BOTTLENECK_LOG" ]; then
+    echo "🧭 Prior cycle's bottleneck: $(tail -1 "$BOTTLENECK_LOG")"
+fi
+```
+
+If a prior bottleneck entry exists, open the report (Step 5) by stating whether
+that named bottleneck was **resolved** (and by which landed patches) and what it
+has now **moved to** — bottleneck SUCCESSION, not just existence, is the signal
+this ledger exists to carry.
+
+### Step 1: Analyze Usage Patterns
+
+Read `.aris/meta/events.jsonl` and compute:
+
+**Frequency analysis:**
+- Which skills are invoked most often?
+- Which slash commands do users type most?
+- What parameter overrides are most common? (These suggest bad defaults.)
+
+**Failure analysis:**
+- Which tools fail most often? In which skills?
+- What error patterns repeat? (OOM, import, compilation, timeout)
+- How many auto-debug retries per workflow run?
+
+**Convergence analysis (for auto-review-loop):**
+- Average rounds to reach threshold
+- Score trajectory shape (fast improvement? plateau? oscillation?)
+- Which review round catches the most critical issues?
+- Do users override difficulty mid-run?
+
+**Human intervention analysis:**
+- Where do users interrupt with manual prompts during workflows?
+- What manual corrections do users make most? (These indicate skill gaps.)
+
+**Model-delta analysis (harness diet):**
+- Has the session model (`session_start` events' `model` field) or the pinned
+  reviewer model changed since a skill's SKILL.md was last touched?
+  (`git log -1 --format=%cs -- skills/<skill>/SKILL.md` vs the model-bump date.)
+- A model bump is a **trigger to re-read, not evidence by itself**. For each
+  reasoning-scaffolding step or worked example in that SKILL.md, a deletion
+  proposal must cite TARGET-SPECIFIC evidence that the new model no longer
+  needs it: a capability-specific release note, or repeated observed behavior
+  in the event log (e.g. zero failures/interventions in the guarded step since
+  the bump). "The model got newer" alone never justifies a deletion.
+- **Never deletion candidates**, regardless of model: privilege boundaries,
+  acceptance/review gates, corpus- and provenance-integrity rules, output
+  contracts, and safety checks. The diet targets model-compensation scaffolding
+  only — a capability the new model has natively is pure overhead (context
+  weight, drift surface, reading cost). A harness that only ever grows is a
+  harness nobody is re-reading.
+
+**Trigger-rate analysis (optional, measured — not from the event log):**
+- The event log shows which skills were USED, not which were WANTED-but-omitted
+  — the omission failure mode (Claude Code passing over the right skill when the
+  installed list is long) is invisible to it. `tools/meta_opt/trigger_eval.py`
+  measures it directly: `claude -p` probes with paraphrased-intent queries run
+  from a neutral cwd (so the realistic long installed corpus is loaded), scored
+  as trigger / confusion(→which skill) / miss.
+- Run it when a specific skill is suspected of under- or mis-triggering, or as a
+  before/after check around a description edit:
+  `python3 tools/meta_opt/trigger_eval.py --eval-file tools/meta_opt/trigger_evals.sample.json --skills <name> --samples 2`
+- The **confusion matrix is the signal**, not just the rate: a query that keeps
+  landing on a sibling skill means the two descriptions overlap on that intent —
+  the fix is disambiguation, not "make the description pushier".
+- **Measure-only, evidence not verdict.** A low trigger rate is an INPUT to a
+  Step-2 proposal (which lands only via `/meta-apply`), never a self-applied
+  description rewrite. Trigger rate is model-dependent, so compare like with
+  like (record the probe model) and treat it as a proxy — it measures selection
+  under a query set, not the full long-list omission problem.
+
+Present findings as a structured summary table.
+
+### Step 1.5: Name the Current Bottleneck
+
+Synthesize the Step-1 analyses into **one sentence naming the single
+most-limiting pipeline stage right now** — e.g. "planning", "verification
+quality", "experiment execution reliability", "writing polish" — with the
+supporting evidence. The bottleneck always moves: when coding stops being the
+constraint, planning becomes it; when planning is solved, verification; when
+verification is automated, taste. This step exists to make the CURRENT
+constraint visible, so Step 2's ranked table reads as sub-fixes for one named
+constraint instead of scattered tweaks.
+
+Append the verdict to the append-only ledger `.aris/meta/bottleneck_log.jsonl`
+(same never-mutate discipline as `.aris/runs/<run_id>.iterations.jsonl`):
+
+```bash
+mkdir -p .aris/meta
+# json.dumps, NOT hand-interpolated shell strings: bottleneck/evidence are
+# natural language — a stray quote must not break the JSONL (or the shell).
+python3 - <<'PY'
+import json, datetime
+entry = {
+    "ts": datetime.datetime.now().astimezone().isoformat(timespec="seconds"),
+    "cycle": 3,
+    "bottleneck": "verification quality",
+    "evidence": "review rounds plateau at 6/10 while tool failures are rare",
+    "top_patch_ids": ["P1", "P2"],
+}
+with open(".aris/meta/bottleneck_log.jsonl", "a", encoding="utf-8") as fh:
+    fh.write(json.dumps(entry, ensure_ascii=False) + "\n")
+PY
+```
+
+Never edit or delete prior lines — succession history is the point.
+
+### Step 2: Identify Optimization Targets
+
+Based on Step 1, rank optimization opportunities by expected impact:
+
+```markdown
+## Optimization Opportunities (ranked)
+
+| # | Target | Signal | Proposed Change | Expected Impact |
+|---|--------|--------|-----------------|-----------------|
+| 1 | auto-review-loop default threshold | Users override to 7/10 in 60% of runs | Change default from 6/10 to 7/10 | Fewer manual overrides |
+| 2 | experiment-bridge retry count | 40% of runs hit max retries on OOM | Add OOM-specific recovery (reduce batch size) | Fewer failed experiments |
+| 3 | paper-write de-AI patterns | Users manually fix "delve" in 80% of runs | Add "delve" to default watchword list | Fewer manual edits |
+| 4 | experiment-bridge Phase-2 hand-holding steps | Model bump (session_start model changed); scaffold untouched since 2 generations ago; zero tool_failures in the steps it guards | **DELETE steps N–M — the new model does this unprompted** | Smaller harness, less drift surface |
+```
+
+The Proposed-Change column is explicitly allowed to be a **deletion** — "DELETE
+step N, new model does this for free" is a first-class optimization, ranked by
+the same impact logic as additions.
+
+If `$ARGUMENTS` specifies a target skill, focus analysis on that skill only.
+If `$ARGUMENTS` is empty or "all", analyze all skills with sufficient data.
+
+### Step 3: Generate Patch Proposals
+
+For each optimization target, generate a concrete diff:
+
+```diff
+--- a/skills/auto-review-loop/SKILL.md
++++ b/skills/auto-review-loop/SKILL.md
+@@ -15,7 +15,7 @@
+ ## Constants
+ 
+-- **SCORE_THRESHOLD = 6** — Minimum review score to accept.
++- **SCORE_THRESHOLD = 7** — Minimum review score to accept. (Raised based on usage data: 60% of users overrode to 7+.)
+```
+
+**Rules for patch generation:**
+- One patch per optimization target
+- Each patch must include a comment explaining WHY (with data from the log)
+- Patches must be minimal — change only what the data supports
+- Never change artifact schemas or MCP bridge config in v1
+- Never change behavior that would break existing user workflows
+- **Anti-self-poisoning screen** (see [`shared-references/capture-antipatterns.md`](../shared-references/capture-antipatterns.md)):
+  run a proposed patch's rationale through `tools/capture_filter.py` (resolve via
+  the canonical chain). NEVER propose a change that encodes a **negative
+  tool-capability claim** ("codex can't…", "gemini is broken") or a **one-off /
+  transient failure** as a durable rule — those harden into self-cited refusals.
+  Encode the *fix / the flag needed / the workaround*, not "X can't do Y".
+
+### Step 4: Cross-Model Review of Patches (ADVISORY pre-screen)
+
+> This review is **advisory** — it sharpens the Step-5 REPORT so the human can decide
+> what to stage. It is **not** the landing verdict. The binding cross-model jury runs
+> later, at landing, inside [`/meta-apply`](../meta-apply/SKILL.md), on the actual staged
+> diff (a producer-relayed verdict would be forgeable). Record this result as
+> `advisory_screen` only.
+
+Send each patch to GPT-5.6-Sol xhigh for adversarial review:
+
+```
+mcp__codex__codex:
+  model: gpt-5.6-sol
+  config: {"model_reasoning_effort": "xhigh"}
+  prompt: |
+    You are reviewing a proposed optimization to an ARIS SKILL.md file.
+    
+    ## Original Skill (relevant section)
+    [paste original]
+    
+    ## Proposed Patch
+    [paste diff]
+    
+    ## Evidence from Usage Log
+    [paste summary stats]
+    
+    Review this patch:
+    1. Does the evidence support the change?
+    2. Could this change hurt other use cases?
+    3. Is the change minimal and safe?
+    4. Score 1-10: should this be applied?
+    
+    If score < 7, explain what additional evidence would be needed.
+
+    === SCOPE LIMITS (these bound what you PROPOSE, never what you look for) ===
+    Report anything that is actually wrong here — including a rare-looking case, if
+    this repo actually produces it. Then keep the fix in scope:
+    1. This is a RESEARCH-WORKFLOW tool, not a security paper. Verification is
+       welcome; over-defense is not. Assume a cooperating operator on their own
+       machine — a malicious local user is NOT in the threat model.
+    2. Do NOT propose SHA / hash / content-fingerprint / digest-binding schemes.
+       Reporting a real defect in hashing code that already exists is fine.
+    3. NO speculative machinery: do not add feature flags, migration frameworks,
+       compat layers, wrappers, pins, or similar mechanisms unless evidence shows
+       a current repo defect they fix or an explicit existing invariant they must
+       preserve. "Load-bearing", "compatibility", and "not scaffolding" are labels,
+       not evidence. Point to the failing path/artifact or invariant, and check the
+       proposal's factual premises, such as whether a named package version exists.
+    4. NO corner-case obsession: exotic encodings, symlink races, RTL text and
+       millisecond races are out of scope unless you can show the case arises here.
+    5. Where a rubric or checklist is genuinely needed, do not over-mechanize
+       judgement. A clear sentence a human reads beats a scored table nobody
+       maintains.
+    Exception: code that runs remote commands, starts a network service, or installs
+    an MCP server runs on the user's machine with their credentials — trust-boundary
+    findings there are in scope and the default is strict.
+    Say plainly when something is correct. Do not manufacture findings.
+```
+
+### Step 5: Present Results
+
+Output a structured report:
+
+```markdown
+# ARIS Meta-Optimization Report
+
+**Date**: [today]
+**Data**: [N] events, [M] skill invocations, [K] sessions
+**Target**: [skill name or "all"]
+
+## Current Bottleneck
+
+**[one-phrase name]** — [one-line evidence]. Prior cycle's bottleneck: [name —
+resolved by <patch ids> / unresolved / first recorded cycle]. (Ledger:
+`.aris/meta/bottleneck_log.jsonl`)
+
+## Proposed Changes
+
+### Change 1: [title]
+- **Target**: [skill/file:line]
+- **Signal**: [what the data shows]
+- **Patch**: [diff]
+- **Reviewer Score**: [X/10]
+- **Reviewer Notes**: [summary]
+- **Status**: ✅ Recommended / ⚠️ Needs more data / ❌ Rejected
+
+### Change 2: ...
+
+## Changes NOT Made (insufficient evidence)
+- [pattern observed but too few samples]
+
+## Recommendations
+- [ ] Apply Change 1 (reviewer approved)
+- [ ] Collect more data for Change 3 (need N more runs)
+- [ ] Consider manual review of Change 2
+
+## Next Steps
+This skill only **proposes**. To land changes: tell me which to stage, then run
+`/meta-apply` (a separate, human-invoked applier that re-checks the cross-model
+verdict before mutating anything). meta-optimize never applies.
+```
+
+### Step 6: Stage approved patches for `/meta-apply` (NO in-skill apply)
+
+This skill does **not** apply anything. After the user has read the Step-5 REPORT and
+indicated which changes to land, **stage** them for the privileged applier:
+
+1. For each approved change `N`, write its unified diff to
+   `.aris/meta/pending/<NN>_<skill>.diff` and append a row to
+   `.aris/meta/pending/manifest.jsonl`:
+   `{patch: "<NN>_<skill>.diff", target: "<corpus path>", author_model: "<executor>",
+   advisory_screen: "pass|kill", advisory_reason: "<one line>"}`.
+   The `advisory_screen` (your Step-4 codex pre-review) is **advisory only** — it helps
+   the human read the REPORT. It is **NOT** the landing verdict and `/meta-apply` does not
+   trust it: a producer-written verdict would be forgeable. The binding cross-model jury
+   runs **at landing, inside `/meta-apply`,** on the actual staged diff.
+2. Tell the user: *"Staged M patches. Run `/meta-apply` to judge & land them."*
+
+The backup → **fresh jury-at-landing** → apply → **provenance stamp** → log all happen
+inside [`/meta-apply`](../meta-apply/SKILL.md). meta-optimize never touches the corpus and
+never produces the acquittal.
+
+**Never apply in this skill. Landing is `/meta-apply` + a fresh jury + a human, always.**
+
+## Key Rules
+
+- **Log-driven, not speculative.** Every proposed change must cite specific data from the event log. No "I think this would be better."
+- **Minimal patches.** Change one thing at a time. Don't rewrite entire skills — the one sanctioned large edit is a scaffolding **deletion** backed by TARGET-SPECIFIC model-delta evidence (a capability-specific release note, or repeated post-bump event-log behavior showing the scaffold is unused — the model name changing is a trigger to look, never sufficient evidence). Privilege boundaries, acceptance gates, corpus/provenance rules, output contracts, and safety checks are never deletion candidates. Deletions go through the same review + approval gates as everything else.
+- **Reviewer-gated.** Every patch goes through cross-model review before recommendation.
+- **Reversible.** Always back up before applying. Always log what changed.
+- **User-approved.** Never auto-apply. Present, explain, let the user decide.
+- **Honest about uncertainty.** If the data is insufficient, say so. Don't optimize on noise.
+- **Portable.** Optimizations should improve the skill for all users, not just one user's style. If a change seems user-specific, flag it.
+
+## Event Schema Reference
+
+The log at `.aris/meta/events.jsonl` contains JSONL records with these shapes:
+
+```jsonl
+{"ts":"...","session":"...","event":"skill_invoke","skill":"auto-review-loop","args":"difficulty: hard"}
+{"ts":"...","session":"...","event":"PostToolUse","tool":"Bash","input_summary":"pdflatex main.tex"}
+{"ts":"...","session":"...","event":"codex_call","tool":"mcp__codex__codex","input_summary":"review..."}
+{"ts":"...","session":"...","event":"tool_failure","tool":"Bash","input_summary":"python train.py"}
+{"ts":"...","session":"...","event":"slash_command","command":"/auto-review-loop","args":""}
+{"ts":"...","session":"...","event":"user_prompt","prompt_preview":"change difficulty to hard"}
+{"ts":"...","session":"...","event":"session_start","source":"startup","model":"claude-opus-4-6"}
+{"ts":"...","session":"...","event":"session_end"}
+```
+
+## Triggering
+
+This skill is NOT part of the standard W1→W1.5→W2→W3→W4 pipeline. It is a **maintenance workflow** with three trigger mechanisms:
+
+1. **Passive logging** (always on): Claude Code hooks record events to `.aris/meta/events.jsonl` automatically during normal usage. Zero user effort.
+
+2. **Automatic readiness check** (SessionEnd hook): When a Claude Code session ends, `check_ready.sh` counts skill invocations since the last `/meta-optimize` run. If ≥5 new invocations have accumulated, it prints a reminder:
+   ```
+   📊 ARIS has logged 8 skill runs since last optimization. Run /meta-optimize to check for improvement opportunities.
+   ```
+   It ALSO fires — regardless of invocation count — when the session model has
+   changed since the last optimize (compared against `.aris/meta/.last_optimize_model`):
+   ```
+   🔁 Model changed since last optimization (claude-opus-4-6 → claude-opus-4-8). Run /meta-optimize — a model bump makes existing scaffolding a deletion candidate (harness diet).
+   ```
+   Both are **suggestions only** — they do not auto-run optimization.
+
+3. **Manual trigger**: User runs `/meta-optimize` when they see the reminder or whenever they want.
+
+**After each `/meta-optimize` run**, the skill writes the current timestamp to `.aris/meta/.last_optimize` and the current session model (latest `session_start` event's `model` field) to `.aris/meta/.last_optimize_model`, so the readiness check can detect both new usage and model bumps.
+
+## Acknowledgements
+
+Inspired by [Meta-Harness](https://arxiv.org/abs/2603.28052) (Lee et al., 2026) — end-to-end optimization of model harnesses via filesystem-based experience access and agentic code search.
+
+## Output Protocols
+
+> Follow these shared protocols for all output files:
+> - **[Output Versioning Protocol](../shared-references/output-versioning.md)** — write timestamped file first, then copy to fixed name
+> - **[Output Manifest Protocol](../shared-references/output-manifest.md)** — log every output to MANIFEST.md
+> - **[Output Language Protocol](../shared-references/output-language.md)** — respect the project's language setting
+
+## Review Tracing
+
+After each `mcp__codex__codex` or `mcp__codex__codex-reply` reviewer call, save the trace following `shared-references/review-tracing.md` (Policy C — forensic; never silently skip). Use `save_trace.sh` (resolved per the chain in `shared-references/integration-contract.md` §2) or write files directly to `.aris/traces/<skill>/<date>_run<NN>/`. Respect the `--- trace:` parameter (default: `full`).

package/skills/monitor-experiment/SKILL.md

@@ -0,0 +1,140 @@
+---
+name: monitor-experiment
+description: Monitor running experiments, check progress, collect results. Use when user says "check results", "is it done", "monitor", or wants experiment output.
+argument-hint: "[server-alias or screen-name]"
+allowed-tools: Bash(ssh *), Bash(echo *), Read, Write, Edit
+---
+
+# Monitor Experiment Results
+
+> ⏱ **External cadence is appropriate here.** This skill waits on an external
+> fact (job completion / progress), so it is a natural `/loop` / `CronCreate`
+> surface: the wake reads status and self-judges only **machine-checkable**
+> completion (exit code, file exists, epoch logged) — never quality. This is
+> the additive external-wait shape in
+> [`shared-references/external-cadence.md`](../shared-references/external-cadence.md).
+> If a scheduled wait here ends in a verdict step (e.g. then audit results),
+> run that verdict **once** after the wait clears — not re-entered per tick.
+
+Monitor: $ARGUMENTS
+
+## Workflow
+
+### Step 1: Check What's Running
+
+**SSH server:**
+```bash
+ssh <server> "screen -ls"
+```
+
+**Vast.ai instance** (read `ssh_host`, `ssh_port` from `vast-instances.json`):
+```bash
+ssh -p <PORT> root@<HOST> "screen -ls"
+```
+
+Also check vast.ai instance status:
+```bash
+vastai show instances
+```
+
+**Modal** (when `gpu: modal` in CLAUDE.md):
+```bash
+modal app list         # List running/recent apps
+modal app logs <app>   # Stream logs from a running app
+```
+Modal apps auto-terminate when done — if it's not in the list, it already finished. Check results via `modal volume ls <volume>` or local output.
+
+### Step 2: Collect Output from Each Screen
+For each screen session, capture the last N lines:
+```bash
+ssh <server> "screen -S <name> -X hardcopy /tmp/screen_<name>.txt && tail -50 /tmp/screen_<name>.txt"
+```
+
+If hardcopy fails, check for log files or tee output.
+
+### Step 3: Check for JSON Result Files
+```bash
+ssh <server> "ls -lt <results_dir>/*.json 2>/dev/null | head -20"
+```
+
+If JSON results exist, fetch and parse them:
+```bash
+ssh <server> "cat <results_dir>/<latest>.json"
+```
+
+### Step 3.5: Pull W&B Metrics (when `wandb: true` in CLAUDE.md)
+
+**Skip this step entirely if `wandb` is not set or is `false` in CLAUDE.md.**
+
+Pull training curves and metrics from Weights & Biases via Python API:
+
+```bash
+# List recent runs in the project
+ssh <server> "python3 -c \"
+import wandb
+api = wandb.Api()
+runs = api.runs('<entity>/<project>', per_page=10)
+for r in runs:
+    print(f'{r.id}  {r.state}  {r.name}  {r.summary.get(\"eval/loss\", \"N/A\")}')
+\""
+
+# Pull specific metrics from a run (last 50 steps)
+ssh <server> "python3 -c \"
+import wandb, json
+api = wandb.Api()
+run = api.run('<entity>/<project>/<run_id>')
+history = list(run.scan_history(keys=['train/loss', 'eval/loss', 'eval/ppl', 'train/lr'], page_size=50))
+print(json.dumps(history[-10:], indent=2))
+\""
+
+# Pull run summary (final metrics)
+ssh <server> "python3 -c \"
+import wandb, json
+api = wandb.Api()
+run = api.run('<entity>/<project>/<run_id>')
+print(json.dumps(dict(run.summary), indent=2, default=str))
+\""
+```
+
+**What to extract:**
+- **Training loss curve** — is it converging? diverging? plateauing?
+- **Eval metrics** — loss, PPL, accuracy at latest checkpoint
+- **Learning rate** — is the schedule behaving as expected?
+- **GPU memory** — any OOM risk?
+- **Run status** — running / finished / crashed?
+
+**W&B dashboard link** (include in summary for user):
+```
+https://wandb.ai/<entity>/<project>/runs/<run_id>
+```
+
+> This gives the auto-review-loop richer signal than just screen output — training dynamics, loss curves, and metric trends over time.
+
+### Step 4: Summarize Results
+
+Present results in a comparison table:
+```
+| Experiment | Metric | Delta vs Baseline | Status |
+|-----------|--------|-------------------|--------|
+| Baseline  | X.XX   | —                 | done   |
+| Method A  | X.XX   | +Y.Y              | done   |
+```
+
+### Step 5: Interpret
+- Compare against known baselines
+- Flag unexpected results (negative delta, NaN, divergence)
+- Suggest next steps based on findings
+
+### Step 6: Feishu Notification (if configured)
+
+After results are collected, check `~/.claude/feishu.json`:
+- Send `experiment_done` notification: results summary table, delta vs baseline
+- If config absent or mode `"off"`: skip entirely (no-op)
+
+## Key Rules
+- Always show raw numbers before interpretation
+- Compare against the correct baseline (same config)
+- Note if experiments are still running (check progress bars, iteration counts)
+- If results look wrong, check training logs for errors before concluding
+- **Vast.ai cost awareness**: When monitoring vast.ai instances, report the running cost (hours * $/hr from `vast-instances.json`). If all experiments on an instance are done, remind the user to run `/vast-gpu destroy <instance_id>` to stop billing
+- **Modal cost awareness**: Modal auto-scales to zero — no idle billing. When reporting results from Modal runs, note the actual execution time and estimated cost (time * $/hr from the GPU tier used). No cleanup action needed