@research-copilot/plugin 1.1.17 → 1.1.19

package/dist/skills/auto-review-loop-minimax/SKILL.md

@@ -0,0 +1,290 @@
+---
+name: "auto-review-loop-minimax"
+description: "Autonomous multi-round research review loop using MiniMax API. Use when you want to use MiniMax instead of Codex MCP for external review. Trigger with \"auto review loop minimax\" or \"minimax review\"."
+---
+
+# Auto Review Loop (MiniMax Version): Autonomous Research Improvement
+
+Autonomously iterate: review → implement fixes → re-review, until the external reviewer gives a positive assessment or MAX_ROUNDS is reached.
+
+## Context: $ARGUMENTS
+
+## Constants
+
+- MAX_ROUNDS = 4
+- POSITIVE_THRESHOLD: score >= 6/10 AND verdict ∈ {"ready", "almost"} — both must hold, matching the operative STOP CONDITION below. Verdict vocabulary is {"ready", "almost", "not ready"}. (Earlier wording used "or" + a stale verdict set; the AND form is authoritative.)
+- REVIEW_DOC: `review-stage/AUTO_REVIEW.md` (cumulative log) *(fall back to `./AUTO_REVIEW.md` for legacy projects)*
+- REVIEWER_MODEL = `MiniMax-M3` — Model used via MiniMax API
+
+## API Configuration
+
+This skill uses MiniMax API for external review. Two methods are supported:
+
+### Method 1: MCP Tool (Primary)
+
+If `mcp__minimax-chat__minimax_chat` is available, use it:
+
+```
+mcp__minimax-chat__minimax_chat:
+  message: |
+    [Review prompt content]
+  model: "MiniMax-M3"
+  system: "You are a senior machine learning researcher..."
+```
+
+### Method 2: curl (Fallback)
+
+If MCP is not available, use curl directly:
+
+```bash
+curl -s "https://api.minimax.io/v1/chat/completions" \
+  -H "Content-Type: application/json" \
+  -H "Authorization: Bearer $MINIMAX_API_KEY" \
+  -d '{
+    "model": "MiniMax-M3",
+    "messages": [
+      {"role": "system", "content": "You are a senior ML researcher..."},
+      {"role": "user", "content": "[Review prompt]"}
+    ],
+    "max_tokens": 4096
+  }'
+```
+
+**API Key**: Read from `~/.codex/settings.json` under `env.MINIMAX_API_KEY`, or from environment variable.
+
+**Why MiniMax instead of a secondary Codex agent?** Codex CLI uses OpenAI's Responses API (`/v1/responses`) which is not supported by third-party providers. See: https://github.com/openai/codex/discussions/7782
+
+## State Persistence (Compact Recovery)
+
+Long-running loops may hit the context window limit, triggering automatic compaction. To survive this, persist state to `review-stage/REVIEW_STATE.json` after each round:
+
+```json
+{
+  "round": 2,
+  "status": "in_progress",
+  "last_score": 5.0,
+  "last_verdict": "not ready",
+  "pending_experiments": ["screen_name_1"],
+  "timestamp": "2026-03-13T21:00:00"
+}
+```
+
+**Write this file at the end of every Phase E** (after documenting the round). Overwrite each time — only the latest state matters.
+
+**On completion** (positive assessment or max rounds), set `"status": "completed"` so future invocations don't accidentally resume a finished loop.
+
+## Workflow
+
+### Initialization
+
+1. **Check for `review-stage/REVIEW_STATE.json`** *(fall back to `./REVIEW_STATE.json` if not found — legacy path)*:
+   - If neither path exists: **fresh start** (normal case)
+   - If it exists AND `status` is `"completed"`: **fresh start** (previous loop finished normally)
+   - If it exists AND `status` is `"in_progress"` AND `timestamp` is older than 24 hours: **fresh start** (stale state from a killed/abandoned run — delete the file and start over)
+   - If it exists AND `status` is `"in_progress"` AND `timestamp` is within 24 hours: **resume**
+     - Read the state file to recover `round`, `last_score`, `pending_experiments`
+     - Read `review-stage/AUTO_REVIEW.md` to restore full context of prior rounds *(fall back to `./AUTO_REVIEW.md`)*
+     - If `pending_experiments` is non-empty, check if they have completed (e.g., check screen sessions)
+     - Resume from the next round (round = saved round + 1)
+     - Log: "Recovered from context compaction. Resuming at Round N."
+2. Read project narrative documents, memory files, and any prior review documents
+3. Read recent experiment results (check output directories, logs)
+4. Identify current weaknesses and open TODOs from prior reviews
+5. Initialize round counter = 1 (unless recovered from state file)
+6. Create/update `review-stage/AUTO_REVIEW.md` with header and timestamp
+
+### Loop (repeat up to MAX_ROUNDS)
+
+#### Phase A: Review
+
+Send comprehensive context to the external reviewer.
+
+**Check MCP availability first**, then use appropriate method:
+
+**If MCP available (Primary):**
+```
+Use mcp__minimax-chat__minimax_chat tool with:
+- system: "You are a senior machine learning researcher serving as a reviewer for top-tier conferences like NeurIPS, ICML, and ICLR. Provide rigorous, constructive feedback."
+- prompt: [Full review prompt with context]
+- model: "MiniMax-M3"
+```
+
+**If MCP NOT available (Fallback):**
+```bash
+curl -s "https://api.minimax.io/v1/chat/completions" \
+  -H "Content-Type: application/json" \
+  -H "Authorization: Bearer $MINIMAX_API_KEY" \
+  -d '{
+    "model": "MiniMax-M3",
+    "messages": [
+      {
+        "role": "system",
+        "content": "You are a senior machine learning researcher serving as a reviewer for top-tier conferences like NeurIPS, ICML, and ICLR. Provide rigorous, constructive feedback."
+      },
+      {
+        "role": "user",
+        "content": "[Round N/MAX_ROUNDS of autonomous review loop]\n\n[Full research context: claims, methods, results, known weaknesses]\n[Changes since last round, if any]\n[For round 2+: Summary of previous review feedback and what was addressed]\n\nPlease act as a senior ML reviewer (NeurIPS/ICML level).\n\n1. Score this work 1-10 for a top venue\n2. List remaining critical weaknesses (ranked by severity)\n3. For each weakness, specify the MINIMUM fix (experiment, analysis, or reframing)\n4. State clearly: is this READY for submission? Yes/No/Almost\n\nBe brutally honest. If the work is ready, say so clearly."
+      }
+    ],
+    "max_tokens": 4096
+  }'
+```
+
+**Note**: Each round is a standalone API call. For round 2+, include the summary of previous reviews and changes in the prompt itself.
+
+#### Phase B: Parse Assessment
+
+**CRITICAL: Save the FULL raw response** from the external reviewer verbatim (store in a variable for Phase E). Do NOT discard or summarize — the raw text is the primary record.
+
+Then extract structured fields:
+- **Score** (numeric 1-10)
+- **Verdict** ("ready" / "almost" / "not ready")
+- **Action items** (ranked list of fixes)
+
+**STOP CONDITION**: If score >= 6 AND verdict ∈ {"ready", "almost"} (exact match — "not ready" does NOT qualify) → stop loop, document final state.
+
+#### Phase C: Implement Fixes (if not stopping)
+
+For each action item (highest priority first):
+
+1. **Code changes**: Write/modify experiment scripts, model code, analysis scripts
+2. **Run experiments**: Deploy to GPU server via SSH + screen/tmux
+3. **Analysis**: Run evaluation, collect results, update figures/tables
+4. **Documentation**: Update project notes and review document
+
+Prioritization rules:
+- Skip fixes requiring excessive compute (flag for manual follow-up)
+- Skip fixes requiring external data/models not available
+- Prefer reframing/analysis over new experiments when both address the concern
+- Always implement metric additions (cheap, high impact)
+
+#### Phase D: Wait for Results
+
+If experiments were launched:
+- Monitor remote sessions for completion
+- Collect results from output files and logs
+
+#### Phase E: Document Round
+
+Append to `review-stage/AUTO_REVIEW.md`:
+
+```markdown
+## Round N (timestamp)
+
+### Assessment (Summary)
+- Score: X/10
+- Verdict: [ready/almost/not ready]
+- Key criticisms: [bullet list]
+
+### Reviewer Raw Response
+
+<details>
+<summary>Click to expand full reviewer response</summary>
+
+[Paste the COMPLETE raw response from the external reviewer here — verbatim, unedited.
+This is the authoritative record. Do NOT truncate or paraphrase.]
+
+</details>
+
+### Actions Taken
+- [what was implemented/changed]
+
+### Results
+- [experiment outcomes, if any]
+
+### Status
+- [continuing to round N+1 / stopping]
+```
+
+**Write `review-stage/REVIEW_STATE.json`** with current round, score, verdict, and any pending experiments.
+
+Increment round counter → back to Phase A.
+
+### Termination
+
+When loop ends (positive assessment or max rounds):
+
+1. Update `review-stage/REVIEW_STATE.json` with `"status": "completed"`
+2. Write final summary to `review-stage/AUTO_REVIEW.md`
+3. Update project notes with conclusions
+4. If stopped at max rounds without positive assessment:
+   - List remaining blockers
+   - Estimate effort needed for each
+   - Suggest whether to continue manually or pivot
+
+## Key Rules
+
+- **Large file handling**: If the Write tool fails due to file size, immediately retry using Bash (`cat << 'EOF' > file`) to write in chunks. Do NOT ask the user for permission — just do it silently.
+
+- Be honest — include negative results and failed experiments
+- Do NOT hide weaknesses to game a positive score
+- Implement fixes BEFORE re-reviewing (don't just promise to fix)
+- If an experiment takes > 30 minutes, launch it and continue with other fixes while waiting
+- Document EVERYTHING — the review log should be self-contained
+- Update project notes after each round, not just at the end
+- For round 2+, always include previous review context in the prompt
+- Prefer MCP tool over curl when available (more reliable)
+
+## Prompt Template for Round 2+
+
+**MCP Method (Primary):**
+```
+mcp__minimax-chat__minimax_chat:
+  model: "MiniMax-M3"
+  system: "You are a senior machine learning researcher serving as a reviewer for top-tier conferences like NeurIPS, ICML, and ICLR. Provide rigorous, constructive feedback."
+  message: |
+    [Round N/MAX_ROUNDS of autonomous review loop]
+
+    ## Previous Review Summary (Round N-1)
+    - Previous Score: X/10
+    - Previous Verdict: [ready/almost/not ready]
+    - Previous Key Weaknesses: [list]
+
+    ## Changes Since Last Review
+    1. [Action 1]: [result]
+    2. [Action 2]: [result]
+    3. [Action 3]: [result]
+
+    ## Updated Results
+    [paste updated metrics/tables]
+
+    ## Current Research Context
+    [brief summary of claims, methods, current state]
+
+    Please re-score and re-assess:
+    1. Score this work 1-10 for a top venue
+    2. List remaining critical weaknesses (ranked by severity)
+    3. For each weakness, specify the MINIMUM fix
+    4. State clearly: is this READY for submission? Yes/No/Almost
+
+    Be brutally honest. If the work is ready, say so clearly.
+```
+
+**curl Fallback:**
+```bash
+curl -s "https://api.minimax.io/v1/chat/completions" \
+  -H "Content-Type: application/json" \
+  -H "Authorization: Bearer $MINIMAX_API_KEY" \
+  -d '{
+    "model": "MiniMax-M3",
+    "messages": [
+      {
+        "role": "system",
+        "content": "You are a senior machine learning researcher serving as a reviewer for top-tier conferences like NeurIPS, ICML, and ICLR. Provide rigorous, constructive feedback."
+      },
+      {
+        "role": "user",
+        "content": "[Round N/MAX_ROUNDS of autonomous review loop]\n\n## Previous Review Summary (Round N-1)\n- Previous Score: X/10\n- Previous Verdict: [ready/almost/not ready]\n- Previous Key Weaknesses: [list]\n\n## Changes Since Last Review\n1. [Action 1]: [result]\n2. [Action 2]: [result]\n3. [Action 3]: [result]\n\n## Updated Results\n[paste updated metrics/tables]\n\n## Current Research Context\n[brief summary of claims, methods, current state]\n\nPlease re-score and re-assess:\n1. Score this work 1-10 for a top venue\n2. List remaining critical weaknesses (ranked by severity)\n3. For each weakness, specify the MINIMUM fix\n4. State clearly: is this READY for submission? Yes/No/Almost\n\nBe brutally honest. If the work is ready, say so clearly."
+      }
+    ],
+    "max_tokens": 4096
+  }'
+```
+
+
+## Output Protocols
+
+> Follow these shared protocols for all output files:
+> - **[Output Versioning Protocol](../../shared-references/output-versioning.md)** — write timestamped file first, then copy to fixed name
+> - **[Output Manifest Protocol](../../shared-references/output-manifest.md)** — log every output to MANIFEST.md
+> - **[Output Language Protocol](../../shared-references/output-language.md)** — respect the project's language setting

package/dist/skills/benchmark-paper-template/SKILL.md

@@ -0,0 +1,142 @@
+---
+name: benchmark-paper-template
+description: Structures Benchmark and Evaluation papers using the five-pillar framework (Research Gap, Construction Pipeline, Evaluation Framework, Empirical Findings, optional Companion Method). Returns a completeness audit, a six-part Introduction logic chain, a Section 2-7 skeleton, and a pre-submission checklist. Use when writing a benchmark paper, structuring a benchmark paper, checking whether a benchmark idea is substantive, drafting a benchmark Introduction, or planning the data-construction pipeline or experiments.
+license: CC-BY-4.0
+---
+
+# Benchmark Paper Template
+
+## Overview
+
+A Benchmark paper does not win by proposing a new algorithm. It wins by defining a new evaluation dimension and shipping a construction pipeline that makes the measurement high-quality, scalable, and reproducible. This skill scaffolds the five pillars a reviewer checks, then gives you a six-part Introduction chain, a Section 2-7 skeleton, and a pre-submission checklist. Stage-specific depth lives in seven reference files under `references/`.
+
+## Core capabilities
+
+1. **Five-pillar completeness audit**: is the Research Gap articulated? Is the Construction Pipeline principled? Is the Evaluation Framework fine-grained? Do the Empirical Findings reveal capability boundaries? Is a Companion Method warranted?
+2. **Introduction six-part logic chain**: Background + Running Example, Existing-Benchmark Limitations (no more than three), Research Questions, Design Considerations, Our Proposal, Contributions.
+3. **Section skeleton for §2 to §7**: Task and Design Goals, Construction Pipeline, Optional Companion Method, Experiments organized by RQ, Discussion and Research Opportunities, Related Work with benchmark comparison table, Conclusion.
+4. **Pre-submission self-check**: four-category reviewer checklist (Introduction, Benchmark section, Experiments, Overall) with Critical, Major, Minor severity.
+
+## Benchmark paper vs technical paper
+
+| Dimension | Technical paper | Benchmark paper |
+|---|---|---|
+| Main contribution | Novel algorithm or method | Novel evaluation dimension or dataset |
+| Introduction axis | Key Idea or Mechanism | Evaluation Gap and Benchmark Design Rationale |
+| Problem definition | One-sentence goal | The problem definition IS the contribution |
+| Heaviest chapter | Method | Construction Pipeline + Evaluation Framework |
+| Experiments purpose | Prove "my method beats baselines" | Reveal "where model capability boundaries sit" |
+| Canonical Figure 1 | Method framework diagram | Running example + pipeline diagram |
+
+For technical and position papers, use the `tech-paper-template` skill. For the Introduction outline in isolation, use `intro-drafter`.
+
+## The five pillars
+
+1. **Research Gap**. What dimension of evaluation does existing work miss? Ground the gap in a concrete failure case and cite at least three prior benchmarks whose limitations you are addressing (no more than three). Exemplars: StatQA highlights missing statistical-method appropriateness; nvBench 2.0 highlights query-ambiguity blindness; VisJudge-Bench highlights the fidelity-expressiveness-aesthetics trinity in visualization evaluation.
+2. **Construction Pipeline**. How do you build high-quality, scalable, reproducible data? Three common paradigms: Reverse Synthesis (seed knowledge then instantiate), Controlled Injection (seed queries then inject targeted ambiguity or error), Adaptive Generation with Expert Validation. Specify source selection, generation, annotation, quality control, split strategy, and statistical profile. Deep dive: `references/construction-pipeline.md`.
+3. **Evaluation Framework**. Beyond a single overall score: difficulty tiers, error taxonomy, per-dimension rubrics. Explain why this taxonomy diagnoses what the gap pointed at. Deep dive: `references/benchmark-design.md`.
+4. **Empirical Findings**. Multi-angle comparisons (Human vs LLM, architecture families, error distributions) condensed into bolded *Finding X:* sentences that read like lemmas. Each Finding must be actionable for future research. Deep dive: `references/experiments.md`.
+5. **Companion Method (optional)**. A specialized model tuned for this benchmark signals that the community can act on the findings. Examples: Step-Text2Vis, VisJudge. Not mandatory, but strongly recommended for benchmarks targeting mature tasks.
+
+## Introduction six-part flowchart
+
+1. **Research Background + Running Example (Figure 1)**. Establish the task, why it matters, and one concrete example that threads through the entire paper.
+2. **Existing-Benchmark Limitations**. At most three, each specific and traceable to an evaluation blind spot. Avoid vague "is limited" phrasing.
+3. **Research Questions**. Two or three RQs covering construction quality, capability boundaries, and the human-AI gap.
+4. **Design Considerations**. What should a good benchmark for this dimension have? Quality, scale, coverage, reproducibility, contamination resistance.
+5. **Our Proposal**. One paragraph: the benchmark plus the companion method if any.
+6. **Contributions**. Typically four items: benchmark + pipeline innovation + systematic evaluation + findings or companion method.
+
+## Section skeleton
+
+- **§2 Task + Design Goals**: problem formulation, goals (G1 coverage, G2 fine-grained diagnostics, G3 reproducibility, G4 contamination resistance). See `references/benchmark-design.md`.
+- **§3 Construction Pipeline**: sources, generation, annotation protocol, QC, statistical profile. Figure 2 is the canonical pipeline diagram. See `references/construction-pipeline.md`.
+- **§4 Companion Method (optional)**: a specialized model whose training set is this benchmark.
+- **§5 Experiments**: organized by RQ. Include the Overall Performance table (typically the largest table in the paper), fine-grained analysis, a human baseline when available, and bolded *Finding X:* summaries. See `references/experiments.md`.
+- **§6 Discussion + Research Opportunities**: what the findings reveal and what comes next.
+- **§7 Related Work**: a benchmark comparison table (often labelled Table 1) is essential, either here or at the end of §1.
+
+The full section-by-section writing guide with page budgets and figure placement is in `references/paper-structure.md`.
+
+## Prompt template
+
+Paste the block below into your AI assistant with the input slots filled.
+
+```markdown
+# Role
+You are a senior researcher who has published multiple Benchmark papers at top venues (NeurIPS Datasets and Benchmarks Track, SIGMOD, VLDB, ICML, ICLR). You know what reviewers look for in Benchmark submissions and how those criteria differ from Technical papers.
+
+# Task
+I will give you the core information about a Benchmark or Evaluation paper. Audit it against the five-pillar framework, then produce a complete logic skeleton for the paper.
+
+# Five pillars (all must be addressed)
+1. Research Gap: which dimension of evaluation does existing work miss?
+2. Construction Pipeline: how is the data built at scale without losing quality?
+3. Evaluation Framework: what is the fine-grained taxonomy?
+4. Empirical Findings: what capability boundary does this reveal?
+5. Companion Method (optional): a specialized model tuned for this benchmark.
+
+# Input
+- Research area: [e.g., Text-to-SQL, Text-to-Visualization, code generation]
+- Benchmark name: [name]
+- Research gap and motivation: [the evaluation blind spot you target]
+- Construction approach: [how the data is built]
+- Evaluation framework: [metrics and taxonomy]
+- Data scale: [number of tasks, domains, difficulty tiers]
+- Key findings or insights: [one to three]
+
+# Output
+
+## Step 1: Five-pillar completeness table
+
+| Pillar | Covered? | Your content | Improvement suggestion |
+|---|---|---|---|
+| Research Gap | Y or N | ... | ... |
+| Construction Pipeline | Y or N | ... | ... |
+| Evaluation Framework | Y or N | ... | ... |
+| Empirical Findings | Y or N | ... | ... |
+| Companion Method | Y, N, or NA | ... | ... |
+
+## Step 2: Introduction six-part logic chain
+
+| Part | Your content |
+|---|---|
+| 1. Background + Running Example | ... |
+| 2. Existing-benchmark limitations (up to 3) | Limitation 1: ... | Limitation 2: ... | Limitation 3: ... |
+| 3. Research Questions | RQ1: ... | RQ2: ... | RQ3 (optional): ... |
+| 4. Design Considerations | ... |
+| 5. Our Proposal | ... |
+| 6. Contributions | 1. ... | 2. ... | 3. ... | 4. ... |
+
+## Step 3: Section outline for §2 to §7
+
+For each section, produce a one-paragraph sketch naming the figure or table that carries its weight.
+
+## Step 4: Pre-submission self-check
+
+Load `references/checklist.md` and walk the four-category checklist. Report any Critical or Major items that are unresolved.
+```
+
+## Reference exemplars
+
+- **StatQA (NeurIPS 2024)**: gap is evaluation of statistical-method appropriateness; pipeline is reverse synthesis from textbooks; finding is that LLMs often pick the statistically wrong test even when the numeric answer is computed correctly.
+- **nvBench 2.0 (NeurIPS 2025)**: gap is query-ambiguity blindness in Text-to-Visualization; pipeline is controlled ambiguity injection; finding is that LLM output quality swings dramatically with minor wording changes, while humans navigate via clarification dialogue.
+- **VisJudge-Bench (ICLR 2026)**: gap is the fidelity-expressiveness-aesthetics trinity in visualization quality; pipeline is expert-curated with adaptive generation; companion method is VisJudge, a specialized judge model trained on this benchmark.
+
+## Usage tips
+
+- Use early, at scope lock. The cheapest fix for a missing pillar is before data construction starts.
+- When the user's answer to a pillar is "we have this but it is messy", point them to the specific file in `references/` rather than trying to resolve it in one turn.
+- Do not confuse this Introduction flowchart with the technical-paper flowchart; they are structurally different. For technical papers, invoke `tech-paper-template`.
+- For pre-submission self-check, load `references/checklist.md` and walk it line by line with the user.
+
+## References
+
+- [`references/gap-analysis.md`](references/gap-analysis.md): systematic identification of the evaluation blind spot.
+- [`references/benchmark-design.md`](references/benchmark-design.md): design goals, task scope, taxonomy patterns, evaluation framework.
+- [`references/construction-pipeline.md`](references/construction-pipeline.md): the three construction paradigms, pipeline stages, quality control.
+- [`references/experiments.md`](references/experiments.md): baseline selection, RQ-driven analysis, *Finding X* pattern, case studies.
+- [`references/paper-structure.md`](references/paper-structure.md): section-by-section writing with page budgets and figure placement.
+- [`references/checklist.md`](references/checklist.md): four-category pre-submission checklist with severity classification.
+- [`references/instantiation-template.md`](references/instantiation-template.md): fillable template for instantiating this thinking model on your paper.
+- [`references/orchestrator-notes.md`](references/orchestrator-notes.md): historical notes from the earlier staged orchestrator architecture, kept for context.

package/dist/skills/benchmark-paper-template/references/benchmark-design.md

@@ -0,0 +1,180 @@
+# Benchmark Architecture Design
+
+## Contents
+
+- [Element 1: Design Goals](#element-1-design-goals)
+- [Element 2: Task Scope](#element-2-task-scope)
+- [Element 3: Taxonomy Design](#element-3-taxonomy-design)
+- [Element 4: Evaluation Framework](#element-4-evaluation-framework)
+- [Element 5: Companion Method (Optional but Highly Recommended)](#element-5-companion-method-optional-but-highly-recommended)
+- [Element 6: Contamination Mitigation Strategy](#element-6-contamination-mitigation-strategy)
+- [Element 7: Difficulty Calibration](#element-7-difficulty-calibration)
+- [Design Document Template](#design-document-template)
+
+Design the conceptual blueprint of a benchmark, its goals, scope, taxonomy, evaluation framework, and optional companion method. This is the architectural plan before any data is built.
+
+## Element 1: Design Goals
+
+Every benchmark should explicitly state its design goals. Use the G1-G4 standard framework, plus custom goals as needed:
+
+| Goal | Description | Typical Strategy |
+|------|-------------|-----------------|
+| **G1: Coverage** | Benchmark covers the breadth of the target capability | Systematic taxonomy, diverse sources, stratified sampling |
+| **G2: Fine-grained Diagnostics** | Can pinpoint WHERE models fail, not just IF | Multi-dimensional taxonomy with sub-categories and difficulty levels |
+| **G3: Scalability** | Construction is reproducible and extensible at low cost | Automated/semi-automated pipeline, minimal manual annotation |
+| **G4: Quality** | Annotations are accurate and evaluation is reliable | Multi-stage QC, inter-annotator agreement, expert validation |
+
+Ask the user: "How will the benchmark address each of these four goals? Are there any additional design goals to add?"
+
+## Element 2: Task Scope
+
+Clearly define the boundary. This pre-empts the reviewer question "why didn't you include X?"
+
+```
+Evaluates:
+- [Capability 1]: [brief description]
+- [Capability 2]: [brief description]
+- ...
+
+Does NOT evaluate (out of scope, with reason):
+- [Excluded capability 1]: [why excluded, different research question, existing benchmarks already cover it, etc.]
+- ...
+```
+
+## Element 3: Taxonomy Design
+
+The taxonomy is the backbone of fine-grained evaluation. Choose from three proven patterns:
+
+### Pattern 1: Capability × Difficulty Matrix (StatQA)
+
+```
+Categories:  [Cat1] [Cat2] [Cat3] [Cat4] [Cat5]
+                    ×
+Difficulty:  [Easy] [Hard]
+             ──────────
+             = 10 evaluation cells
+```
+
+**Best for**: Tasks where capabilities are naturally categorical with clear difficulty gradients.
+
+### Pattern 2: Phenomenon Type × Severity Spectrum (nvBench 2.0)
+
+```
+Types:    [Type1] [Type2] [Type3] [Type4]
+                  ×
+Severity: [Lv1] [Lv2] [Lv3] [Lv4] [Lv5]
+          ──────────
+          = 20 evaluation cells
+```
+
+**Best for**: Tasks studying a specific phenomenon (ambiguity, noise, bias) across intensity levels.
+
+### Pattern 3: Multi-dimensional Quality Framework (VisJudge-Bench)
+
+```
+Top Dimensions:    [Fidelity] [Expressiveness] [Aesthetics]
+                         ↓              ↓                  ↓
+Sub-dimensions:    [Sub1.1] [Sub1.2] [Sub2.1] [Sub2.2] [Sub3.1] [Sub3.2]
+                   ──────────
+                   = 6 measurable sub-dimensions
+```
+
+**Best for**: Tasks where quality has multiple independent facets that interact.
+
+Ask the user: "Which taxonomy pattern best fits the evaluation dimensions? List the initial taxonomy."
+
+## Element 4: Evaluation Framework
+
+Beyond taxonomy, define HOW each dimension is measured:
+
+| Sub-dimension | Metric | Scoring Method | Range | Automation |
+|--------------|--------|---------------|-------|-----------|
+| [SubDim 1] | e.g., Accuracy | Exact match | 0-1 | Fully auto |
+| [SubDim 2] | e.g., Relevance | LLM-as-Judge | 1-5 | Semi-auto |
+| [SubDim 3] | e.g., Naturalness | Human rating | 1-7 | Manual |
+
+Key design decisions to discuss with the user:
+
+| Decision | Options | Trade-off |
+|----------|---------|-----------|
+| **Auto vs. Human eval** | Automatic metrics / LLM-as-Judge / Human | Scale vs. reliability |
+| **Single vs. Multi-answer** | One correct answer / Multiple valid answers | Simplicity vs. real-world fidelity |
+| **LLM-as-Judge validation** | Correlation with human / Agreement rate | MUST validate if using LLM eval |
+| **Scoring granularity** | Binary / Ordinal / Continuous | Discriminability vs. annotation difficulty |
+
+## Element 5: Companion Method (Optional but Highly Recommended)
+
+A companion method answers: "This benchmark not only evaluates, it also helps IMPROVE model capability."
+
+| Component | Question | Example |
+|-----------|----------|---------|
+| **Training signal** | What data does the benchmark uniquely provide? | Step-wise reasoning paths, preference pairs, fine-grained scores |
+| **Optimization technique** | How is the signal used? | SFT, DPO, GRPO, RLHF, curriculum learning |
+| **Value proposition** | Why is this better than training without the benchmark? | Targeted improvement on the specific blind spot |
+
+Reference implementations:
+
+| Paper | Training Signal | Method | Technique |
+|-------|----------------|--------|-----------|
+| nvBench 2.0 | Step-wise reasoning paths | Step-Text2Vis | Step-DPO preference optimization |
+| VisJudge-Bench | Fine-grained quality scores | VisJudge | GRPO reinforcement learning |
+
+**This element significantly strengthens the paper.** Even a simple fine-tuning experiment showing improvement on the benchmark's target capability adds substantial value.
+
+## Element 6: Contamination Mitigation Strategy
+
+Reviewers increasingly care about data contamination. Specify how you prevent:
+
+| Contamination Type | Risk | Mitigation |
+|-------------------|------|------------|
+| **Training contamination** | Benchmark data appears in model training sets | Time-gated construction, synthetic generation, novel data sources |
+| **Prompt contamination** | Evaluation prompts leak into training | Unique prompt templates, private evaluation scripts |
+| **Contamination laundering** | Using synthetic data from contaminated models to generate "new" data | Verify data provenance, cross-validate with non-LLM sources |
+| **Benchmark saturation** | Models overfit to benchmark style over time | Design for extensibility, include version-control plan |
+
+## Element 7: Difficulty Calibration
+
+Top models should achieve only **~0.1-9% accuracy** on the hardest subset at launch (Ofir Press guideline). If top models score >50%, the benchmark may already be too easy. Include a difficulty sanity check:
+
+- What do you expect top models to score? If too high, add harder subsets.
+- Plan for benchmark longevity: include difficulty levels that challenge future models.
+
+## Design Document Template
+
+Help the user fill in this structured template:
+
+```
+Benchmark Name: [Name]
+Gap Statement: [From bench-gap-analysis]
+Research Questions: [RQ1, RQ2, RQ3]
+
+Design Goals:
+  G1 (Coverage): [Strategy]
+  G2 (Diagnostics): [Strategy]
+  G3 (Scalability): [Strategy]
+  G4 (Quality): [Strategy]
+
+Task Scope:
+  In-scope: [List]
+  Out-of-scope: [List with reasons]
+
+Taxonomy:
+  Pattern: [1/2/3]
+  Dimensions: [List]
+  Sub-dimensions: [List]
+  Total evaluation cells: [N]
+
+Evaluation Framework:
+  [Table of metrics per dimension]
+
+Companion Method (optional):
+  Training signal: [What]
+  Technique: [How]
+
+Contamination Mitigation:
+  Strategy: [How you prevent data leakage]
+
+Difficulty Calibration:
+  Expected top-model accuracy: [X%]
+  Hardest subset target: [Y%]
+```