@baimingtao/lbs-agent 1.0.0

package/extensions/skills/builtin/research/research-paper-writing/SKILL.md

@@ -0,0 +1,813 @@
+---
+name: research-paper-writing
+title: Research Paper Writing Pipeline
+description: "为 NeurIPS/ICML/ICLR 撰写 ML 论文：设计→提交。"
+version: 1.1.0
+author: Orchestra Research
+license: MIT
+dependencies: [semanticscholar, arxiv, habanero, requests, scipy, numpy, matplotlib, SciencePlots]
+platforms: [linux, macos]
+metadata:
+  hermes:
+    tags: [Research, Paper Writing, Experiments, ML, AI, NeurIPS, ICML, ICLR, ACL, AAAI, COLM, LaTeX, Citations, Statistical Analysis]
+    category: research
+    related_skills: [arxiv, ml-paper-writing, subagent-driven-development, plan]
+    requires_toolsets: [terminal, files]
+
+---
+
+# Research Paper Writing Pipeline
+
+用于产出可发表的 ML/AI 研究论文的端到端 pipeline，目标会议为 **NeurIPS, ICML, ICLR, ACL, AAAI, 和 COLM**。这个 skill 涵盖了完整的科研生命周期：实验设计、执行、监控、分析、论文撰写、审稿、修订以及提交。
+
+这**不是一个线性的 pipeline** —— 而是一个迭代的循环。结果会触发新的实验；审稿意见会触发新的分析。agent 必须能够处理这些反馈循环。
+
+<!-- ascii-guard-ignore -->
+```
+┌─────────────────────────────────────────────────────────────┐
+│                    RESEARCH PAPER PIPELINE                  │
+│                                                             │
+│  Phase 0: Project Setup ──► Phase 1: Literature Review      │
+│       │                          │                          │
+│       ▼                          ▼                          │
+│  Phase 2: Experiment     Phase 5: Paper Drafting ◄──┐      │
+│       Design                     │                   │      │
+│       │                          ▼                   │      │
+│       ▼                    Phase 6: Self-Review      │      │
+│  Phase 3: Execution &           & Revision ──────────┘      │
+│       Monitoring                 │                          │
+│       │                          ▼                          │
+│       ▼                    Phase 7: Submission               │
+│  Phase 4: Analysis ─────► (feeds back to Phase 2 or 5)     │
+│                                                             │
+└─────────────────────────────────────────────────────────────┘
+```
+<!-- ascii-guard-ignore-end -->
+
+---
+
+## When To Use This Skill
+
+Use this skill when:
+- **Starting a new research paper** from an existing codebase or idea
+- **Designing and running experiments** to support paper claims
+- **Writing or revising** any section of a research paper
+- **Preparing for submission** to a specific conference or workshop
+- **Responding to reviews** with additional experiments or revisions
+- **Converting** a paper between conference formats
+- **Writing non-empirical papers** — theory, survey, benchmark, or position papers (see [Paper Types Beyond Empirical ML](#paper-types-beyond-empirical-ml))
+- **Designing human evaluations** for NLP, HCI, or alignment research
+- **Preparing post-acceptance deliverables** — posters, talks, code releases
+
+## Core Philosophy
+
+1. **Be proactive.** 交付完整的初稿，而不是一堆问题。科学家们都很忙——产出一些具体的成果供他们审阅，然后再迭代。
+2. **Never hallucinate citations.** AI 生成的引用错误率高达 40% 左右。务必通过程序化方式获取。将无法验证的引用标记为 `[CITATION NEEDED]`。
+3. **Paper is a story, not a collection of experiments.** 每篇论文都需要能用一句话清晰表述的核心 contribution。如果你做不到这一点，这篇论文就还没有准备好。
+4. **Experiments serve claims.** 每个实验都必须明确说明它支持哪个 claim。绝不运行与论文 narrative 无关的实验。
+5. **Commit early, commit often.** 每完成一批实验、每更新一次论文草稿——都要使用具有描述性的信息进行 commit。Git log 就是实验的历史记录。
+
+### Proactivity and Collaboration
+
+**默认: 采取主动。先撰写初稿，在初稿中提出疑问。**
+
+| Confidence Level | Action |
+|-----------------|--------|
+| **High** (clear repo, obvious contribution) | 撰写完整初稿，交付，基于反馈进行迭代 |
+| **Medium** (some ambiguity) | 撰写初稿并标记不确定的地方，然后继续 |
+| **Low** (major unknowns) | 通过 `clarify` 提出 1-2 个有针对性的问题，然后再撰写初稿 |
+
+| Section | Draft Autonomously? | Flag With Draft |
+|---------|-------------------|-----------------|
+| Abstract | Yes | "Framed contribution as X — adjust if needed" |
+| Introduction | Yes | "Emphasized problem Y — correct if wrong" |
+| Methods | Yes | "Included details A, B, C — add missing pieces" |
+| Experiments | Yes | "Highlighted results 1, 2, 3 — reorder if needed" |
+| Related Work | Yes | "Cited papers X, Y, Z — add any I missed" |
+
+**Block for input only when**: target venue unclear, multiple contradictory framings, results seem incomplete, explicit request to review first.
+
+---
+
+## Phase 0: Project 配置
+
+**Goal**: 建立工作区，了解现有工作，确定 contribution。
+
+### Step 0.1: Explore the Repository
+
+```bash
+# Understand project structure
+ls -la
+find . -name "*.py" | head -30
+find . -name "*.md" -o -name "*.txt" | xargs grep -l -i "result\|conclusion\|finding"
+```
+
+Look for:
+- `README.md` — project overview and claims
+- `results/`, `outputs/`, `experiments/` — existing findings
+- `configs/` — experimental settings
+- `.bib` files — existing citations
+- Draft documents or notes
+
+### Step 0.2: Organize the Workspace
+
+Establish a consistent workspace structure:
+
+```
+workspace/
+  paper/               # LaTeX source, figures, compiled PDFs
+  experiments/         # Experiment runner scripts
+  code/                # Core method implementation
+  results/             # Raw experiment results (auto-generated)
+  tasks/               # Task/benchmark definitions
+  human_eval/          # Human evaluation materials (if needed)
+```
+
+### Step 0.3: Set Up Version Control
+
+```bash
+git init  # if not already
+git remote add origin <repo-url>
+git checkout -b paper-draft  # or main
+```
+
+**Git discipline**: Every completed experiment batch gets committed with a descriptive message. 示例:
+```
+Add Monte Carlo constrained results (5 runs, Sonnet 4.6, policy memo task)
+Add Haiku baseline comparison: autoreason vs refinement baselines at cheap model tier
+```
+
+### Step 0.4: Identify the Contribution
+
+Before writing anything, articulate:
+- **The What**: What is the single thing this paper contributes?
+- **The Why**: What evidence supports it?
+- **The So What**: Why should readers care?
+
+> Propose to the scientist: "Based on my understanding, the main contribution is: [one sentence]. The key results show [Y]. Is this the framing you want?"
+
+### Step 0.5: Create a TODO List
+
+Use the `todo` tool to create a structured project plan:
+
+```
+Research Paper TODO:
+- [ ] Define one-sentence contribution
+- [ ] Literature review (related work + baselines)
+- [ ] Design core experiments
+- [ ] Run experiments
+- [ ] Analyze results
+- [ ] Write first draft
+- [ ] Self-review (simulate reviewers)
+- [ ] Revise based on review
+- [ ] Submission prep
+```
+
+更新 this throughout the project. It serves as the persistent state across sessions.
+
+### Step 0.6: Estimate Compute Budget
+
+Before running experiments, estimate total cost and time:
+
+```
+Compute Budget Checklist:
+- [ ] API costs: (model price per token) × (estimated tokens per run) × (number of runs)
+- [ ] GPU hours: (time per experiment) × (number of experiments) × (number of seeds)
+- [ ] Human evaluation costs: (annotators) × (hours) × (hourly rate)
+- [ ] Total budget ceiling and contingency (add 30-50% for reruns)
+```
+
+Track actual spend as experiments run:
+```python
+# Simple cost tracker pattern
+import json, os
+from datetime import datetime
+
+COST_LOG = "results/cost_log.jsonl"
+
+def log_cost(experiment: str, model: str, input_tokens: int, output_tokens: int, cost_usd: float):
+    entry = {
+        "timestamp": datetime.now().isoformat(),
+        "experiment": experiment,
+        "model": model,
+        "input_tokens": input_tokens,
+        "output_tokens": output_tokens,
+        "cost_usd": cost_usd,
+    }
+    with open(COST_LOG, "a") as f:
+        f.write(json.dumps(entry) + "\n")
+```
+
+**When budget is tight**: Run pilot experiments (1-2 seeds, subset of tasks) before committing to full sweeps. Use cheaper models for debugging pipelines, then switch to target models for final runs.
+
+### Step 0.7: Multi-Author Coordination
+
+Most papers have 3-10 authors. Establish workflows early:
+
+| 工作流 | 工具 | When to Use |
+|----------|------|-------------|
+| **Overleaf** | Browser-based | 多名作者同时编辑，无 git 经验 |
+| **Git + LaTeX** | 带有 `.gitignore` (用于辅助文件) 的 `git` | 技术团队，需要基于分支的 review |
+| **Overleaf + Git sync** | Overleaf premium | 结合两者优点——实时协作且带有版本历史 |
+
+**Section ownership**: Assign each section to one primary author. Others comment but don't edit directly. Prevents merge conflicts and style inconsistency.
+
+```
+Author Coordination Checklist:
+- [ ] Agree on section ownership (who writes what)
+- [ ] Set up shared workspace (Overleaf or git repo)
+- [ ] Establish notation conventions (before anyone writes)
+- [ ] Schedule internal review rounds (not just at the end)
+- [ ] Designate one person for final formatting pass
+- [ ] Agree on figure style (colors, fonts, sizes) before creating figures
+```
+
+**LaTeX conventions to agree on early**:
+- `\method{}` macro for consistent method naming
+- Citation style: `\citet{}` vs `\citep{}` usage
+- Math notation: lowercase bold for vectors, uppercase bold for matrices, etc.
+- British vs American spelling
+
+---
+
+## Phase 1: Literature Review
+
+**Goal**: Find related work, identify baselines, gather citations.
+
+### Step 1.1: Identify Seed Papers
+
+Start from papers already referenced in the codebase:
+
+```bash
+# Via terminal:
+grep -r "arxiv\|doi\|cite" --include="*.md" --include="*.bib" --include="*.py"
+find . -name "*.bib"
+```
+
+### Step 1.2: Search for Related Work
+
+**Load the `arxiv` skill** for structured paper discovery: `skill_view("arxiv")`. It provides arXiv REST API search, Semantic Scholar citation graphs, author profiles, and BibTeX generation.
+
+Use `web_search` for broad discovery, `web_extract` for fetching specific papers:
+
+```
+# Via web_search:
+web_search("[main technique] + [application domain] site:arxiv.org")
+web_search("[baseline method] comparison ICML NeurIPS 2024")
+
+# Via web_extract (for specific papers):
+web_extract("https://arxiv.org/abs/2303.17651")
+```
+
+Additional search queries to try:
+
+```
+Search queries:
+- "[main technique] + [application domain]"
+- "[baseline method] comparison"
+- "[problem name] state-of-the-art"
+- Author names from existing citations
+```
+
+**推荐**: 安装 **Exa MCP** 用于实时学术搜索：
+```bash
+claude mcp add exa -- npx -y mcp-remote "https://mcp.exa.ai/mcp"
+```
+
+### Step 1.2b: Deepen the Search (Breadth-First, Then Depth)
+
+A flat search (one round of queries) typically misses important related work. Use an iterative **breadth-then-depth** pattern inspired by deep research pipelines:
+
+```
+Iterative Literature Search:
+
+Round 1 (Breadth): 4-6 parallel queries covering different angles
+  - "[method] + [domain]"
+  - "[problem name] state-of-the-art 2024 2025"
+  - "[baseline method] comparison"
+  - "[alternative approach] vs [your approach]"
+  → Collect papers, extract key concepts and terminology
+
+Round 2 (Depth): Generate follow-up queries from Round 1 learnings
+  - New terminology discovered in Round 1 papers
+  - Papers cited by the most relevant Round 1 results
+  - Contradictory findings that need investigation
+  → Collect papers, identify remaining gaps
+
+Round 3 (Targeted): Fill specific gaps
+  - Missing baselines identified in Rounds 1-2
+  - Concurrent work (last 6 months, same problem)
+  - Key negative results or failed approaches
+  → Stop when new queries return mostly papers you've already seen
+```
+
+**When to stop**: If a round returns >80% papers already in your collection, the search is saturated. Typically 2-3 rounds suffice. For survey papers, expect 4-5 rounds.
+
+**For agent-based workflows**: Delegate each round's queries in parallel via `delegate_task`. Collect results, deduplicate, then generate the next round's queries from the combined learnings.
+
+### Step 1.3: Verify Every Citation
+
+**NEVER generate BibTeX from memory. ALWAYS fetch programmatically.**
+
+For each citation, follow the mandatory 5-step process:
+
+```
+Citation Verification (MANDATORY per citation):
+1. SEARCH → Query Semantic Scholar or Exa MCP with specific keywords
+2. VERIFY → Confirm paper exists in 2+ sources (Semantic Scholar + arXiv/CrossRef)
+3. RETRIEVE → Get BibTeX via DOI content negotiation (programmatically, not from memory)
+4. VALIDATE → Confirm the claim you're citing actually appears in the paper
+5. ADD → Add verified BibTeX to bibliography
+If ANY step fails → mark as [CITATION NEEDED], inform scientist
+```
+
+```python
+# Fetch BibTeX via DOI
+import requests
+
+def doi_to_bibtex(doi: str) -> str:
+    response = requests.get(
+        f"https://doi.org/{doi}",
+        headers={"Accept": "application/x-bibtex"}
+    )
+    response.raise_for_status()
+    return response.text
+```
+
+If you cannot verify a citation:
+
+```latex
+\cite{PLACEHOLDER_author2024_verify_this}  % TODO: Verify this citation exists
+```
+
+**Always tell the scientist**: "I've marked [X] citations as placeholders that need verification."
+
+See [references/citation-workflow.md](references/citation-workflow.md) for complete API documentation and the full `CitationManager` class.
+
+### Step 1.4: Organize Related Work
+
+Group papers by methodology, not paper-by-paper:
+
+**Good**: "One line of work uses X's assumption [refs] whereas we use Y's assumption because..."
+**Bad**: "Smith et al. introduced X. Jones et al. introduced Y. We combine both."
+
+---
+
+## Phase 2: Experiment Design
+
+**Goal**: 设计直接支持论文 claim 的实验。每个实验都必须回答一个具体的问题。
+
+### Step 2.1: Map Claims to Experiments
+
+Create an explicit mapping:
+
+| Claim | Experiment | Expected Evidence |
+|-------|-----------|-------------------|
+| "Our method outperforms baselines" | Main comparison (Table 1) | Win rate, statistical significance |
+| "Effect is larger for weaker models" | Model scaling study | Monotonic improvement curve |
+| "Convergence requires scope constraints" | Constrained vs unconstrained | Convergence rate comparison |
+
+**Rule**: If an experiment doesn't map to a claim, don't run it.
+
+### Step 2.2: Design Baselines
+
+Strong baselines are what separates accepted papers from rejected ones. Reviewers will ask: "Did they compare against X?"
+
+Standard baseline categories:
+- **Naive baseline**: Simplest possible approach
+- **Strong baseline**: Best known existing method
+- **Ablation baselines**: Your method minus one component
+- **Compute-matched baselines**: Same compute budget, different allocation
+
+### Step 2.3: Define Evaluation Protocol
+
+Before running anything, specify:
+- **Metrics**: What you're measuring, direction symbols (higher/lower better)
+- **Aggregation**: How results are combined across runs/tasks
+- **Statistical tests**: What tests will establish significance
+- **Sample sizes**: How many runs/problems/tasks
+
+### Step 2.4: Write Experiment Scripts
+
+Follow these patterns from successful research pipelines:
+
+**Incremental saving** — save results after each step for crash recovery:
+```python
+# Save after each problem/task
+result_path = f"results/{task}/{strategy}/result.json"
+if os.path.exists(result_path):
+    continue  # Skip already-completed work
+# ... run experiment ...
+with open(result_path, 'w') as f:
+    json.dump(result, f, indent=2)
+```
+
+**Artifact preservation** — save all intermediate outputs:
+```
+results/<experiment>/
+  <task>/
+    <strategy>/
+      final_output.md          # Final result
+      history.json             # Full trajectory
+      pass_01/                 # Per-iteration artifacts
+        version_a.md
+        version_b.md
+        critic.md
+```
+
+**Separation of concerns** — keep generation, evaluation, and visualization separate:
+```
+run_experiment.py              # Core experiment runner
+run_baselines.py               # Baseline comparison
+run_comparison_judge.py        # Blind evaluation
+analyze_results.py             # Statistical analysis
+make_charts.py                 # Visualization
+```
+
+See [references/experiment-patterns.md](references/experiment-patterns.md) for complete design patterns, cron monitoring, and error recovery.
+
+### Step 2.5: Design Human Evaluation (If Applicable)
+
+Many NLP, HCI, and alignment papers require human evaluation as primary or complementary evidence. Design this before running automated experiments — human eval often has longer lead times (IRB approval, annotator recruitment).
+
+**When human evaluation is needed:**
+- Automated metrics don't capture what you care about (fluency, helpfulness, safety)
+- Your contribution is about human-facing qualities (readability, preference, trust)
+- Reviewers at NLP venues (ACL, EMNLP) expect it for generation tasks
+
+**Key design decisions:**
+
+| Decision | Options | Guidance |
+|----------|---------|----------|
+| **Annotator type** | Expert, crowdworker, end-user | Match to what your claims require |
+| **Scale** | Likert (1-5), pairwise comparison, ranking | Pairwise is more reliable than Likert for LLM outputs |
+| **Sample size** | Per annotator and total items | Power analysis or minimum 100 items, 3+ annotators |
+| **Agreement metric** | Cohen's kappa, Krippendorff's alpha, ICC | Krippendorff's alpha for >2 annotators; report raw agreement too |
+| **Platform** | Prolific, MTurk, internal team | Prolific for quality; MTurk for scale; internal for domain expertise |
+
+**Annotation guideline checklist:**
+```
+- [ ] Clear task description with examples (good AND bad)
+- [ ] Decision criteria for ambiguous cases
+- [ ] At least 2 worked examples per category
+- [ ] Attention checks / gold standard items (10-15% of total)
+- [ ] Qualification task or screening round
+- [ ] Estimated time per item and fair compensation (>= local minimum wage)
+- [ ] IRB/ethics review if required by your institution
+```
+
+**Reporting requirements** (reviewers check all of these):
+- Number of annotators and their qualifications
+- Inter-annotator agreement with specific metric and value
+- Compensation details (amount, estimated hourly rate)
+- Annotation interface description or screenshot (appendix)
+- Total annotation time
+
+See [references/human-evaluation.md](references/human-evaluation.md) for complete guide including statistical tests for human eval data, crowdsourcing quality control patterns, and IRB guidance.
+
+---
+
+## Phase 3: Experiment Execution & Monitoring
+
+**Goal**: Run experiments reliably, monitor progress, recover from failures.
+
+### Step 3.1: Launch Experiments
+
+Use `nohup` for long-running experiments:
+
+```bash
+nohup python run_experiment.py --config config.yaml > logs/experiment_01.log 2>&1 &
+echo $!  # Record the PID
+```
+
+**Parallel execution**: Run independent experiments simultaneously, but be aware of API rate limits. 4+ concurrent experiments on the same API will slow each down.
+
+### Step 3.2: Set Up Monitoring (Cron Pattern)
+
+For long-running experiments, set up periodic status checks. The cron prompt should follow this template:
+
+```
+Monitor Prompt Template:
+1. Check if process is still running: ps aux | grep <pattern>
+2. Read last 30 lines of log: tail -30 <logfile>
+3. Check for completed results: ls <result_dir>
+4. If results exist, read and report: cat <result_file>
+5. If all done, commit: git add -A && git commit -m "<descriptive message>" && git push
+6. Report in structured format (tables with key metrics)
+7. Answer the key analytical question for this experiment
+```
+
+**Silent mode**: If nothing has changed since the last check, respond with `[SILENT]` to suppress notification to the user. Only report when there's news.
+
+### Step 3.3: Handle Failures
+
+Common failure modes and recovery:
+
+| Failure | Detection | Recovery |
+|---------|-----------|----------|
+| API rate limit / credit exhaustion | 402/429 errors in logs | Wait, then re-run (scripts skip completed work) |
+| Process crash | PID gone, incomplete results | Re-run from last checkpoint |
+| Timeout on hard problems | Process stuck, no log progress | Kill and skip, note in results |
+| Wrong model ID | Errors referencing model name | Fix ID and re-run |
+
+**Key**: Scripts should always check for existing results and skip completed work. This makes re-runs safe and efficient.
+
+### Step 3.4: Commit Completed Results
+
+After each experiment batch completes:
+
+```bash
+git add -A
+git commit -m "Add <experiment name>: <key finding in 1 line>"
+git push
+```
+
+### Step 3.5: Maintain an Experiment Journal
+
+Git commits track what happened, but not the **exploration tree** — the decisions about what to try next based on what you learned. Maintain a structured experiment journal that captures this tree:
+
+```json
+// experiment_journal.jsonl — append one entry per experiment attempt
+{
+  "id": "exp_003",
+  "parent": "exp_001",
+  "timestamp": "2025-05-10T14:30:00Z",
+  "hypothesis": "Adding scope constraints will fix convergence failure from exp_001",
+  "plan": "Re-run autoreason with max_tokens=2000 and fixed structure template",
+  "config": {"model": "haiku", "strategy": "autoreason", "max_tokens": 2000},
+  "status": "completed",
+  "result_path": "results/exp_003/",
+  "key_metrics": {"win_rate": 0.85, "convergence_rounds": 3},
+  "analysis": "Scope constraints fixed convergence. Win rate jumped from 0.42 to 0.85.",
+  "next_steps": ["Try same constraints on Sonnet", "Test without structure template"],
+  "figures": ["figures/exp003_convergence.pdf"]
+}
+```
+
+**Why a journal, not just git?** Git tracks file changes. The journal tracks the reasoning: why you tried X, what you learned, and what that implies for the next experiment. When writing the paper, this tree is invaluable for the Methods section ("we observed X, which motivated Y") and for honest failure reporting.
+
+**Selecting the best path**: When the journal shows a branching tree (exp_001 → exp_002a, exp_002b, exp_003), identify the path that best supports the paper's claims. Document dead-end branches in the appendix as ablations or negative results.
+
+**Snapshot code per experiment**: Copy the experiment script after each run:
+```bash
+cp experiment.py results/exp_003/experiment_snapshot.py
+```
+This enables exact reproduction even after subsequent code changes.
+
+---
+
+## Phase 4: Result Analysis
+
+**Goal**: 提取发现，计算统计数据，找出核心故事。
+
+### Step 4.1: Aggregate Results
+
+Write analysis scripts that:
+1. Load all result files from a batch
+2. Compute per-task and aggregate metrics
+3. Generate summary tables
+
+```python
+# Standard analysis pattern
+import json, os
+from pathlib import Path
+
+results = {}
+for result_file in Path("results/").rglob("result.json"):
+    data = json.loads(result_file.read_text())
+    strategy = result_file.parent.name
+    task = result_file.parent.parent.name
+    results.setdefault(strategy, {})[task] = data
+
+# Compute aggregate metrics
+for strategy, tasks in results.items():
+    scores = [t["score"] for t in tasks.values()]
+    print(f"{strategy}: mean={np.mean(scores):.1f}, std={np.std(scores):.1f}")
+```
+
+### Step 4.2: Statistical Significance
+
+Always compute:
+- **Error bars**: Standard deviation or standard error, specify which
+- **Confidence intervals**: 95% CI for key results
+- **Pairwise tests**: McNemar's test for comparing two methods
+- **Effect sizes**: Cohen's d or h for practical significance
+
+See [references/experiment-patterns.md](references/experiment-patterns.md) for complete implementations of McNemar's test, bootstrapped CIs, and Cohen's h.
+
+### Step 4.3: Identify the Story
+
+After analysis, explicitly answer:
+1. **What is the main finding?** State it in one sentence.
+2. **What surprised you?** Unexpected results often make the best papers.
+3. **What failed?** Failed experiments can be the most informative. Honest reporting of failures strengthens the paper.
+4. **What follow-up experiments are needed?** Results often raise new questions.
+
+#### Handling Negative or Null Results
+
+When your hypothesis was wrong or results are inconclusive, you have three options:
+
+| Situation | Action | Venue Fit |
+|-----------|--------|-----------|
+| Hypothesis wrong but **why** is informative | Frame paper around the analysis of why | NeurIPS, ICML (if analysis is rigorous) |
+| Method doesn't beat baselines but **reveals something new** | Reframe contribution as understanding/analysis | ICLR (values understanding), workshop papers |
+| Clean negative result on popular claim | Write it up — the field needs to know | NeurIPS Datasets & Benchmarks, TMLR, workshops |
+| Results inconclusive, no clear story | Pivot — run different experiments or reframe | Don't force a paper that isn't there |
+
+**How to write a negative results paper:**
+- Lead with what the community believes and why it matters to test it
+- Describe your rigorous methodology (must be airtight — reviewers will scrutinize harder)
+- Present the null result clearly with statistical evidence
+- Analyze **why** the expected result didn't materialize
+- Discuss implications for the field
+
+**Venues that explicitly welcome negative results**: NeurIPS (Datasets & Benchmarks track), TMLR, ML Reproducibility Challenge, workshops at major conferences. Some workshops specifically call for negative results.
+
+### Step 4.4: Create Figures and Tables
+
+**Figures**:
+- Use vector graphics (PDF) for all plots: `plt.savefig('fig.pdf')`
+- Colorblind-safe palettes (Okabe-Ito or Paul Tol)
+- Self-contained captions — reader should understand without main text
+- No title inside figure — the caption serves this function
+
+**Tables**:
+- Use `booktabs` LaTeX package
+- Bold best value per metric
+- Include direction symbols (higher/lower better)
+- Consistent decimal precision
+
+```latex
+\usepackage{booktabs}
+\begin{tabular}{lcc}
+\toprule
+Method & Accuracy $\uparrow$ & Latency $\downarrow$ \\
+\midrule
+Baseline & 85.2 & 45ms \\
+\textbf{Ours} & \textbf{92.1} & 38ms \\
+\bottomrule
+\end{tabular}
+```
+
+### Step 4.5: Decide: More Experiments or Write?
+
+| Situation | Action |
+|-----------|--------|
+| Core claims supported, results significant | Move to Phase 5 (writing) |
+| Results inconclusive, need more data | Back to Phase 2 (design) |
+| Unexpected finding suggests new direction | Back to Phase 2 (design) |
+| Missing one ablation reviewers will ask for | Run it, then Phase 5 |
+| All experiments done but some failed | Note failures, move to Phase 5 |
+
+### Step 4.6: Write the Experiment Log (Bridge to Writeup)
+
+Before moving to paper writing, create a structured experiment log that bridges results to prose. This is the single most important connective tissue between experiments and the writeup — without it, the writing agent has to re-derive the story from raw result files.
+
+**Create `experiment_log.md`** with the following structure:
+
+```markdown
+# Experiment Log
+
+## Contribution (one sentence)
+[The paper's main claim]
+
+## Experiments Run
+
+### Experiment 1: [Name]
+- **Claim tested**: [Which paper claim this supports]
+- **Setup**: [Model, dataset, config, number of runs]
+- **Key result**: [One sentence with the number]
+- **Result files**: results/exp1/final_info.json
+- **Figures generated**: figures/exp1_comparison.pdf
+- **Surprising findings**: [Anything unexpected]
+
+### Experiment 2: [Name]
+...
+
+## Figures
+| Filename | Description | Which section it belongs in |
+|----------|-------------|---------------------------|
+| figures/main_comparison.pdf | Bar chart comparing all methods on benchmark X | Results, Figure 2 |
+| figures/ablation.pdf | Ablation removing components A, B, C | Results, Figure 3 |
+...
+
+## Failed Experiments (document for honesty)
+- [What was tried, why it failed, what it tells us]
+
+## Open Questions
+- [Anything the results raised that the paper should address]
+```
+
+**Why this matters**: When drafting, the agent (or a delegated sub-agent) can load `experiment_log.md` alongside the LaTeX template and produce a first draft grounded in actual results. Without this bridge, the writing agent must parse raw JSON/CSV files and infer the story — a common source of hallucinated or misreported numbers.
+
+**Git discipline**: Commit this log alongside the results it describes.
+
+---
+
+## Iterative Refinement: Strategy Selection
+
+Any output in this pipeline — paper drafts, experiment scripts, analysis — can be iteratively refined. The autoreason research provides empirical evidence for when each refinement strategy works and when it fails. Use this section to choose the right approach.
+
+### Quick Decision Table
+
+| Your Situation | Strategy | Why |
+|---------------|----------|-----|
+| Mid-tier model + constrained task | **Autoreason** | Sweet spot. Generation-evaluation gap is widest. Baselines actively destroy weak model outputs. |
+| Mid-tier model + open task | **Autoreason** with scope constraints added | Add fixed facts, structure, or deliverable to bound the improvement space. |
+| Frontier model + constrained task | **Autoreason** | Wins 2/3 constrained tasks even at frontier. |
+| Frontier model + unconstrained task | **Critique-and-revise** or **single pass** | Autoreason comes last. Model self-evaluates well enough. |
+| Concrete technical task (system design) | **Critique-and-revise** | Direct find-and-fix loop is more efficient. |
+| Template-filling task (one correct structure) | **Single pass** or **conservative** | Minimal decision space. Iteration adds no value. |
+| Code with test cases | **Autoreason (code variant)** | Structured analysis of *why* it failed before fixing. Recovery rate 62% vs 43%. |
+| Very weak model (Llama 8B class) | **Single pass** | Model too weak for diverse candidates. Invest in generation quality. |
+
+### The Generation-Evaluation Gap
+
+**Core insight**: Autoreason's value depends on the gap between a model's generation capability and its self-evaluation capability.
+
+```
+Model Tier        │ Generation │ Self-Eval │ Gap    │ Autoreason Value
+──────────────────┼────────────┼───────────┼────────┼─────────────────
+Weak (Llama 8B)   │ Poor       │ Poor      │ Small  │ None — can't generate diverse candidates
+Mid (Haiku 3.5)   │ Decent     │ Poor      │ LARGE  │ MAXIMUM — 42/42 perfect Borda
+Mid (Gemini Flash)│ Decent     │ Moderate  │ Large  │ High — wins 2/3
+Strong (Sonnet 4) │ Good       │ Decent    │ Medium │ Moderate — wins 3/5
+Frontier (S4.6)   │ Excellent  │ Good      │ Small  │ Only with constraints
+```
+
+This gap is structural, not temporary. As costs drop, today's frontier becomes tomorrow's mid-tier. The sweet spot moves but never disappears.
+
+### Autoreason Loop (Summary)
+
+Each pass produces three candidates from fresh, isolated agents:
+
+1. **Critic** → finds problems in incumbent A (no fixes)
+2. **Author B** → revises A based on critique
+3. **Synthesizer** → merges A and B (randomized labels)
+4. **Judge Panel** → 3 blind CoT judges rank A, B, AB via Borda count
+5. **Convergence** → A wins k=2 consecutive passes → done
+
+**Key parameters:**
+- k=2 convergence (k=1 premature, k=3 too expensive, no quality gain)
+- CoT judges always (3x faster convergence)
+- Temperature 0.8 authors, 0.3 judges
+- Conservative tiebreak: incumbent wins ties
+- Every role is a fresh agent with no shared context
+
+### Applying to Paper Drafts
+
+When refining the paper itself through autoreason:
+- **Provide ground truth to the critic**: actual experimental data, result JSONs, statistical outputs. Without this, models hallucinate fabricated ablation studies and fake confidence intervals.
+- **Use 3 working judges minimum**: A broken judge parser doesn't add noise — it prevents equilibrium entirely.
+- **Scope constrain the revision**: "Address these specific weaknesses" not "improve the paper."
+
+### Failure Modes
+
+| Failure | Detection | Fix |
+|---------|-----------|-----|
+| No convergence (A never wins) | A wins <15% over 20+ passes | Add scope constraints to the task |
+| Synthesis drift | Word counts grow unboundedly | Constrain structure and deliverable |
+| Degradation below single pass | Baselines score higher than iterated output | Switch to single pass; model may be too weak |
+| Overfitting (code) | High public-test pass, low private-test pass | Use structured analysis, not just test feedback |
+| Broken judges | Parsing failures reduce panel below 3 | Fix parser before continuing |
+
+See [references/autoreason-methodology.md](references/autoreason-methodology.md) for complete prompts, Borda scoring details, model selection guide, scope constraint design patterns, and compute budget reference.
+
+---
+
+## Phase 5: Paper Drafting
+
+**Goal**: 撰写完整的、具备发表条件的论文。
+
+### Context Management for Large Projects
+
+A paper project with 50+ experiment files, multiple result directories, and extensive literature notes can easily exceed the agent's context window. Manage this proactively:
+
+**What to load into context per drafting task:**
+
+| Drafting Task | Load Into Context | Do NOT Load |
+|---------------|------------------|-------------|
+| Writing Introduction | `experiment_log.md`, contribution statement, 5-10 most relevant paper abstracts | Raw result JSONs, full experiment scripts, all literature notes |
+| Writing Methods | Experiment configs, pseudocode, architecture description | Raw logs, results from other experiments |
+| Writing Results | `experiment_log.md`, result summary tables, figure list | Full analysis scripts, intermediate data |
+| Writing Related Work | Organized citation notes (Step 1.4 output), .bib file | Experiment files, raw PDFs |
+| Revision pass | Full paper draft, specific reviewer concerns | Everything else |
+
+**Principles:**
+- **`experiment_log.md` is the primary context bridge** — it summarizes everything needed for writing without loading raw data files (see Step 4.6)
+- **Load one section's context at a time** when delegating. A sub-agent drafting Methods doesn't need the literature review notes.
+- **Summarize, don't include raw files.** For a 200-line result JSON, load a 10-line summary table. For a 50-page related paper, load the 5-sentence abstract + your 2-line note about its relevance.
+- **For very large projects**: Create a `context/` directory with pre-compressed summaries:
+  ```
+  context/
+    contribution.md          # 1 sentence
+    experiment_summary.md    # Key results table (from experiment_log.md)
+    literature_map.md        # Organized citation notes
+    figure_inventory.md      # List of figures with descriptions
+  ```
+
+### The Narrative Principle
+
+**The single most critical insight**: Your paper is not a collection of experiments — it's a story with one clear contribution supported by evidence.
+
+Every successful ML paper centers