dsh-aris-panel 0.1.0

package/skills/skills-codex/result-to-claim/SKILL.md

@@ -0,0 +1,246 @@
+---
+name: result-to-claim
+description: Use when experiments complete to judge what claims the results support, what they don't, and what evidence is still missing. A secondary Codex agent evaluates results against intended claims and routes to next action (pivot, supplement, or confirm). Use after experiments finish — before writing the paper or running ablations.
+argument-hint: "[experiment-description-or-wandb-run]"
+allowed-tools: Bash(*), Read, Grep, Glob, Write, Edit
+---
+
+# Result-to-Claim Gate
+
+> **Codex assurance:** deterministic evidence existence can be accepted, while
+> the base semantic claim judgment records `review_independence: same-family`
+> and `acceptance_status: provisional`. Cross-family overlays may record
+> accepted; reviewer failure emits BLOCKED.
+
+Experiments produce numbers; this gate decides what those numbers *mean*. Collect results from available sources, get a secondary Codex judgment, then auto-route based on the verdict.
+
+## Context: $ARGUMENTS
+
+## When to Use
+
+- After a set of experiments completes (main results, not just sanity checks)
+- Before committing to claims in a paper or review response
+- When results are ambiguous and you need an objective second opinion
+
+## Workflow
+
+### Step 1: Collect Results
+
+Gather experiment data from whatever sources are available in the project:
+
+1. **W&B** (preferred): `wandb.Api().run("<entity>/<project>/<run_id>").history()` — metrics, training curves, comparisons
+2. **EXPERIMENT_LOG.md**: full results table with baselines and verdicts
+3. **EXPERIMENT_TRACKER.md**: check which experiments are DONE vs still running
+4. **Log files**: `ssh server "tail -100 /path/to/training.log"` if no other source
+5. **`idea-stage/docs/research_contract.md`** (legacy fallback: `docs/research_contract.md`): intended claims and experiment design
+
+Assemble the key information:
+- What experiments were run (method, dataset, config)
+- Main metrics and baseline comparisons (deltas)
+- The intended claim these experiments were designed to test
+- Any known confounds or caveats
+
+### Step 1.5: Deterministic evidence pre-check
+
+Before the reviewer call, resolve and run `evidence_check.py` per
+[`evidence-precheck.md`](../shared-references/evidence-precheck.md):
+
+```bash
+if [ -z "${ARIS_REPO:-}" ] && [ -f .aris/installed-skills-codex.txt ]; then
+  ARIS_REPO=$(awk -F'\t' '$1=="repo_root"{print $2; exit}' .aris/installed-skills-codex.txt 2>/dev/null) || true
+fi
+EVIDENCE_CHECK=""
+[ -n "${ARIS_REPO:-}" ] && [ -f "$ARIS_REPO/tools/evidence_check.py" ] && EVIDENCE_CHECK="$ARIS_REPO/tools/evidence_check.py"
+[ -z "$EVIDENCE_CHECK" ] && [ -f tools/evidence_check.py ] && EVIDENCE_CHECK="tools/evidence_check.py"
+mkdir -p .aris
+if [ -n "$EVIDENCE_CHECK" ]; then
+  python3 "$EVIDENCE_CHECK" . --batch .aris/claims.json \
+    > .aris/evidence_precheck.json 2>.aris/evidence_precheck.err || true
+else
+  echo "WARN: evidence_check.py unresolved; semantic review will still run" >&2
+fi
+```
+
+Treat `path_missing` and `value_not_found` as unsupported evidence before the
+semantic review. `verified` means only that the cited value exists; it does not
+prove the claim. Pass the pre-check JSON path to the fresh reviewer. The Codex
+reviewer's positive result remains `review_independence: same-family` and
+`acceptance_status: provisional`; a deterministic evidence check never upgrades
+a semantic claim to accepted by itself.
+
+### Step 2: Codex Judgment
+
+Send the collected results to a secondary Codex agent for objective evaluation:
+
+```text
+spawn_agent:
+  model: gpt-5.6-sol
+  reasoning_effort: ultra
+  message: |
+    RESULT-TO-CLAIM EVALUATION
+
+    I need you to judge whether experimental results support the intended claim.
+
+    Intended claim: [the claim these experiments test]
+
+    Experiments run:
+    [list experiments with method, dataset, metrics]
+
+    Results:
+    [paste key numbers, comparison deltas, significance]
+
+    Baselines:
+    [baseline numbers and sources — reproduced or from paper]
+
+    Known caveats:
+    [any confounding factors, limited datasets, missing comparisons]
+
+    Please evaluate:
+    1. claim_supported: yes | partial | no
+    2. what_results_support: what the data actually shows
+    3. what_results_dont_support: where the data falls short of the claim
+    4. missing_evidence: specific evidence gaps
+    5. suggested_claim_revision: if the claim should be strengthened, weakened, or reframed
+    6. next_experiments_needed: specific experiments to fill gaps (if any)
+    7. confidence: high | medium | low
+
+    Be honest. Do not inflate claims beyond what the data supports.
+    A single positive result on one dataset does not support a general claim.
+```
+
+### Step 3: Parse and Normalize
+
+Extract structured fields from the secondary Codex response:
+
+```markdown
+- claim_supported: yes | partial | no
+- what_results_support: "..."
+- what_results_dont_support: "..."
+- missing_evidence: "..."
+- suggested_claim_revision: "..."
+- next_experiments_needed: "..."
+- confidence: high | medium | low
+```
+
+### Step 3.5: Check Experiment Integrity (if audit exists)
+
+**Skip this step if `EXPERIMENT_AUDIT.json` does not exist.**
+
+```
+if EXPERIMENT_AUDIT.json exists:
+    read integrity_status from file
+    attach to verdict output:
+        integrity_status: pass | warn | fail
+
+    if integrity_status == "fail":
+        append to verdict: "[INTEGRITY CONCERN] — audit found issues, see EXPERIMENT_AUDIT.md"
+        downgrade confidence to "low" regardless of Codex judgment
+
+    if integrity_status == "warn":
+        append to verdict: "[INTEGRITY: WARN] — audit flagged potential issues"
+else:
+    integrity_status = "unavailable"
+    verdict is labeled "provisional — no integrity audit run"
+    (this does NOT block anything — pipeline continues normally)
+```
+
+See `shared-references/experiment-integrity.md` for the full integrity protocol.
+
+### Step 4: Route Based on Verdict
+
+#### `no` — Claim not supported
+
+1. Record postmortem in findings.md (Research Findings section):
+   - What was tested, what failed, hypotheses for why
+   - Constraints for future attempts (what NOT to try again)
+2. Update the project pipeline status in `AGENTS.md` or project notes
+3. Decide whether to pivot to next idea from IDEA_CANDIDATES.md or try an alternative approach
+
+#### `partial` — Claim partially supported
+
+1. Update the working claim to reflect what IS supported
+2. Record the gap in findings.md
+3. Design and run supplementary experiments to fill evidence gaps
+4. Re-run result-to-claim after supplementary experiments complete
+5. **Multiple rounds of `partial` on the same claim** → record analysis in findings.md, consider whether to narrow the claim scope or switch ideas
+
+#### `yes` — Claim supported
+
+1. Record confirmed claim in project notes
+2. If ablation studies are incomplete → trigger `/ablation-planner`
+3. If all evidence is in → ready for paper writing
+
+### Step 5: Update Research Wiki (if active)
+
+**Skip this step entirely if `research-wiki/` does not exist.**
+
+```
+if research-wiki/ exists:
+    # Resolve the helper (Codex chain). If unavailable, skip wiki writes; still report verdict.
+    ARIS_REPO="${ARIS_REPO:-$(awk -F'\t' '$1=="repo_root"{print $2; exit}' .aris/installed-skills-codex.txt 2>/dev/null)}"
+    WIKI_SCRIPT=""
+    [ -n "$ARIS_REPO" ] && [ -f "$ARIS_REPO/tools/research_wiki.py" ] && WIKI_SCRIPT="$ARIS_REPO/tools/research_wiki.py"
+    [ -z "$WIKI_SCRIPT" ] && [ -f tools/research_wiki.py ] && WIKI_SCRIPT="tools/research_wiki.py"
+    [ -z "$WIKI_SCRIPT" ] && [ -f ~/.codex/skills/research-wiki/research_wiki.py ] && WIKI_SCRIPT="$HOME/.codex/skills/research-wiki/research_wiki.py"
+    [ -n "$WIKI_SCRIPT" ] || echo "WARN: research_wiki.py unreachable; skipping wiki writes (verdict still reported)." >&2
+
+    # 1. Create/refresh the experiment node FIRST (verdict OWNER → --update-on-exist so a
+    #    re-judge overwrites the stale verdict). The supports/invalidates edges in #2 point
+    #    FROM exp:<id> and add_edge does NOT verify node existence, so only add them if the
+    #    experiment node was born (EXP_NODE_OK); otherwise skip the wiki edges.
+    EXP_NODE_OK=0
+    [ -n "$WIKI_SCRIPT" ] && python3 "$WIKI_SCRIPT" add_experiment research-wiki/ \
+         --slug "<exp_id>" --idea "idea:<active_idea>" \
+         --verdict "<yes|partial|no>" --confidence "<high|medium|low>" \
+         --date "<date>" --hardware "<hw>" --duration "<dur>" \
+         --metrics "<key metrics>" --reasoning "<one-line why this verdict>" \
+         --provenance "<EXPERIMENT_AUDIT.md / run dir>" --update-on-exist && EXP_NODE_OK=1
+
+    # 2. Record empirical support as EDGES ONLY, and ONLY if EXP_NODE_OK. NEVER edit a
+    #    claim page's `status`: that is the PROOF axis (verified / refuted / unproven /
+    #    sound-modulo-imports / drafted / retracted), owned by /proof-checker (the claim
+    #    birth point) — the ARIS helper REJECTS "supported"/"partial"/"invalidated".
+    if [ "$EXP_NODE_OK" = 1 ]:
+        for each claim resolved by this verdict:
+            if verdict == "yes":
+                python3 "$WIKI_SCRIPT" add_edge research-wiki/ --from "exp:<id>" --to "claim:<cid>" --type supports --evidence "<metric>"
+            elif verdict == "partial":
+                python3 "$WIKI_SCRIPT" add_edge research-wiki/ --from "exp:<id>" --to "claim:<cid>" --type supports --evidence "partial: <metric>"
+            else:
+                python3 "$WIKI_SCRIPT" add_edge research-wiki/ --from "exp:<id>" --to "claim:<cid>" --type invalidates --evidence "<why>"
+
+    # 3. Update idea outcome (raw markdown, helper-free — preserves the rich idea body)
+    Update research-wiki/ideas/<idea_id>.md:
+      - outcome: positive | mixed | negative
+      - If negative: fill "Failure / Risk Notes" and "Lessons Learned"
+      - If positive: fill "Actual Outcome" and "Reusable Components"
+
+    # 4. Rebuild + log (reflect the new edges; only if WIKI_SCRIPT resolved)
+    [ -n "$WIKI_SCRIPT" ] && python3 "$WIKI_SCRIPT" rebuild_query_pack research-wiki/
+    [ -n "$WIKI_SCRIPT" ] && python3 "$WIKI_SCRIPT" log research-wiki/ "result-to-claim: exp:<id> verdict=<verdict> for idea:<idea_id>"
+
+    # 5. Re-ideation suggestion
+    Count failed/partial ideas since last /idea-creator run.
+    If >= 3: print "💡 3+ ideas tested since last ideation. Consider re-running /idea-creator — the wiki now knows what doesn't work."
+```
+
+## Rules
+
+- **The secondary Codex agent is the judge, not the local executor.** The local executor collects evidence and routes; the reviewer agent evaluates. This prevents post-hoc rationalization.
+- Do not inflate claims beyond what the data supports. If Codex says "partial", do not round up to "yes".
+- A single positive result on one dataset does not support a general claim. Be honest about scope.
+- If `confidence` is low, treat the judgment as inconclusive and add experiments rather than committing to a claim.
+- **Fail closed if the reviewer is unavailable.** Follow the capability fallback
+  in `reviewer-routing.md` (`gpt-5.6-sol` + `ultra` → `gpt-5.6-sol` + `xhigh`
+  → `gpt-5.5` + `xhigh`), and never downgrade on timeout, rate-limit, auth,
+  transport, server, or context errors. If no allowed pair succeeds, write a
+  traced `BLOCKED` review record with the unavailable route and evidence paths, write
+  `CLAIMS_FROM_RESULTS.md` containing only `verdict: REVIEW_UNAVAILABLE`, record
+  the same in findings.md, and stop. Do not emit a local PASS/WARN substitute or
+  advance a submission-facing claim; only an explicitly non-submission
+  evidence-gathering phase may continue.
+- Always record the verdict and reasoning in findings.md, regardless of outcome.
+
+## Review Tracing
+
+After the secondary Codex judgment, save a trace following `../shared-references/review-tracing.md`. Write files directly to `.aris/traces/result-to-claim/<date>_run<NN>/` and include the prompt, raw reviewer response, parsed verdict, routing action, and whether the result is `[pending external review]`. Respect the `--- trace:` parameter when present (default: `full`).

package/skills/skills-codex/run-experiment/SKILL.md

@@ -0,0 +1,236 @@
+---
+name: "run-experiment"
+description: "Deploy and run ML experiments on local or remote GPU servers. Use when user says \"run experiment\", \"deploy to server\", \"\u8dd1\u5b9e\u9a8c\", or needs to launch training jobs."
+---
+
+# Run Experiment
+
+Deploy and run ML experiment: $ARGUMENTS
+
+## Workflow
+
+### Step 1: Detect Environment
+
+Read the project's `AGENTS.md` to determine the experiment environment:
+
+- **Local GPU**: Look for local CUDA/MPS setup info
+- **Remote server**: Look for SSH alias, conda env, code directory
+- **Vast.ai instance**: Look for `gpu: vast`, `vast_instance`, SSH host/port, remote path, and optional `auto_destroy`
+- **Modal serverless**: Look for `gpu: modal`, app/function name, image/dependency setup, and secrets
+
+If no server info is found in `AGENTS.md`, ask the user.
+
+**Environment contract** (`../shared-references/compute-env-contract.md`): before
+building or trusting any environment, read the provider's env ledger
+(`.aris/compute/<provider>.md`) — an unchanged spec hash means warm-reuse, a
+changed one means rebuild. New env → write the declarative spec first, render it
+for this provider's shape, and never declare it ready on import-success alone:
+run the seeded kernel witness, and after any rebuild/doc edit run the
+agent-follows-doc pass (a fresh subagent executes the documented invocation
+verbatim and reports doc-vs-reality divergence).
+
+### Step 2: Pre-flight Check
+
+Check GPU availability on the target machine:
+
+**Remote:**
+```bash
+ssh <server> nvidia-smi --query-gpu=index,memory.used,memory.total --format=csv,noheader
+```
+
+**Local:**
+```bash
+nvidia-smi --query-gpu=index,memory.used,memory.total --format=csv,noheader
+# or for Mac MPS:
+python -c "import torch; print('MPS available:', torch.backends.mps.is_available())"
+```
+
+Free GPU = memory.used < 500 MiB.
+
+### Step 3: Sync Code (Remote Only)
+
+Check the project's `AGENTS.md` for a `code_sync` setting. If not specified, default to `rsync`.
+
+#### Option A: rsync (default)
+
+Only sync necessary files — NOT data, checkpoints, or large files:
+```bash
+rsync -avz --include='*.py' --exclude='*' <local_src>/ <server>:<remote_dst>/
+```
+
+#### Option B: git (when `code_sync: git` is set in AGENTS.md)
+
+Push local changes to remote repo, then pull on the server:
+```bash
+# 1. Push from local
+git add -A && git commit -m "sync: experiment deployment" && git push
+
+# 2. Pull on server
+ssh <server> "cd <remote_dst> && git pull"
+```
+
+Benefits: version-tracked, multi-server sync with one push, no rsync include/exclude rules needed.
+
+#### Option C: Vast.ai instance
+
+If `gpu: vast` is configured, treat the Vast.ai machine as a remote server with an explicit lifecycle:
+
+1. Verify the instance is running and reachable.
+2. Sync code to the configured remote path.
+3. Confirm data/checkpoints are already mounted or intentionally copied.
+4. Record the instance id in the launch summary for later cleanup.
+
+Do not silently ignore a requested Vast.ai route. If Vast.ai CLI credentials or instance metadata are missing, stop and ask the user to configure them.
+
+### Step 3.5: W&B Integration (when `wandb: true` in AGENTS.md)
+
+**Skip this step entirely if `wandb` is not set or is `false` in AGENTS.md.**
+
+Before deploying, ensure the experiment scripts have W&B logging:
+
+1. **Check if wandb is already in the script** — look for `import wandb` or `wandb.init`. If present, skip to Step 4.
+
+2. **If not present, add W&B logging** to the training script:
+   ```python
+   import wandb
+   wandb.init(project=WANDB_PROJECT, name=EXP_NAME, config={...hyperparams...})
+
+   # Inside training loop:
+   wandb.log({"train/loss": loss, "train/lr": lr, "step": step})
+
+   # After eval:
+   wandb.log({"eval/loss": eval_loss, "eval/ppl": ppl, "eval/accuracy": acc})
+
+   # At end:
+   wandb.finish()
+   ```
+
+3. **Metrics to log** (add whichever apply to the experiment):
+   - `train/loss` — training loss per step
+   - `train/lr` — learning rate
+   - `eval/loss`, `eval/ppl`, `eval/accuracy` — eval metrics per epoch
+   - `gpu/memory_used` — GPU memory (via `torch.cuda.max_memory_allocated()`)
+   - `speed/samples_per_sec` — throughput
+   - Any custom metrics the experiment already computes
+
+4. **Verify wandb login on the target machine:**
+   ```bash
+   ssh <server> "wandb status"  # should show logged in
+   # If not logged in:
+   ssh <server> "wandb login <WANDB_API_KEY>"
+   ```
+
+> The W&B project name and API key come from `AGENTS.md` (see example below). The experiment name is auto-generated from the script name + timestamp.
+
+### Step 4: Deploy
+
+#### Remote (via SSH + screen)
+
+For each experiment, create a dedicated screen session with GPU binding:
+```bash
+ssh <server> "screen -dmS <exp_name> bash -c '\
+  eval \"\$(<conda_path>/conda shell.bash hook)\" && \
+  conda activate <env> && \
+  CUDA_VISIBLE_DEVICES=<gpu_id> python <script> <args> 2>&1 | tee <log_file>'"
+```
+
+#### Vast.ai instance
+
+Use the same SSH + screen pattern, but include the Vast.ai instance id, public SSH endpoint, and remote working directory in the report. If `auto_destroy: true`, write a cleanup command to the run notes before launch.
+
+Record the estimated hourly cost, expected run duration, and cleanup owner. If the command fails to start or the instance becomes unreachable, do not relaunch blindly; capture logs and ask for a rescue / second opinion before spending more GPU time.
+
+#### Modal (serverless)
+
+If `gpu: modal` is configured, deploy through Modal instead of SSH:
+
+```bash
+modal run <module_or_app>.py -- <args>
+```
+
+Before launch, verify required secrets, volumes, image dependencies, and output persistence. If Modal is requested but the project lacks Modal configuration, stop and ask the user to configure it rather than falling back to local execution.
+
+Record the Modal app/function name, GPU type, timeout, mounted volumes, and where results will be stored. If Modal reports an image, secret, or volume error, preserve the exact error and run a configuration fix before retrying.
+
+#### Local
+
+```bash
+# Linux with CUDA
+CUDA_VISIBLE_DEVICES=<gpu_id> python <script> <args> 2>&1 | tee <log_file>
+
+# Mac with MPS (PyTorch uses MPS automatically)
+python <script> <args> 2>&1 | tee <log_file>
+```
+
+For local long-running jobs, use `run_in_background: true` to keep the conversation responsive.
+
+### Step 5: Verify Launch
+
+**Remote:**
+```bash
+ssh <server> "screen -ls"
+```
+
+**Local:**
+Check process is running and GPU is allocated.
+
+### Step 6: Feishu Notification (if configured)
+
+After deployment is verified, check `~/.codex/feishu.json`:
+- Send `experiment_done` notification: which experiments launched, which GPUs, estimated time
+- If config absent or mode `"off"`: skip entirely (no-op)
+
+### Step 7: Auto-Destroy Vast.ai Instance (when `gpu: vast` and `auto_destroy: true`)
+
+Only run this after the experiment has completed and results/logs/checkpoints have been copied or otherwise persisted.
+
+1. Verify the target process has exited.
+2. Copy result files and logs to the configured durable location.
+3. Ask for confirmation unless AGENTS.md explicitly says `auto_destroy: true`.
+4. Destroy only the recorded instance id for this run.
+
+If any artifact copy fails, do not destroy the instance.
+
+## Key Rules
+
+- ALWAYS check GPU availability first — never blindly assign GPUs
+- Each experiment gets its own screen session + GPU (remote) or background process (local)
+- Use `tee` to save logs for later inspection
+- Run deployment commands with `run_in_background: true` to keep conversation responsive
+- Report back: which GPU, which screen/process, what command, estimated time
+- If multiple experiments, launch them in parallel on different GPUs
+
+## AGENTS.md Example
+
+Users should add their server info to their project's `AGENTS.md`:
+
+```markdown
+## Remote Server
+- SSH: `ssh my-gpu-server`
+- GPU: 4x A100 (80GB each)
+- Conda: `eval "$(/opt/conda/bin/conda shell.bash hook)" && conda activate research`
+- Code dir: `/home/user/experiments/`
+- code_sync: rsync          # default. Or set to "git" for git push/pull workflow
+- wandb: false              # set to "true" to auto-add W&B logging to experiment scripts
+- wandb_project: my-project # W&B project name (required if wandb: true)
+- wandb_entity: my-team     # W&B team/user (optional, uses default if omitted)
+
+## Vast.ai
+- gpu: vast
+- vast_instance: 123456
+- SSH: `ssh -p 12345 root@ssh.vast.ai`
+- Code dir: `/workspace/experiments/`
+- auto_destroy: false
+
+## Modal
+- gpu: modal
+- modal_app: `train.py`
+- modal_secrets: `wandb-secret`
+- modal_volume: `experiment-results`
+
+## Local Environment
+- Mac MPS / Linux CUDA
+- Conda env: `ml` (Python 3.10 + PyTorch)
+```
+
+> **W&B setup**: Run `wandb login` on your server once (or set `WANDB_API_KEY` env var). The skill reads project/entity from `AGENTS.md` and adds `wandb.init()` + `wandb.log()` to your training scripts automatically. Dashboard: `https://wandb.ai/<entity>/<project>`.