@research-copilot/plugin 1.1.17 → 1.1.19

package/dist/skills/benchmark-paper-template/references/orchestrator-notes.md

@@ -0,0 +1,96 @@
+# Benchmark/Evaluation Paper Writing System
+
+A systematic workflow for writing Benchmark/Evaluation papers targeting top-tier AI venues (NeurIPS, ICLR, ICML, ACL, EMNLP, CVPR).
+
+## Core Philosophy
+
+A Benchmark paper's narrative axis is fundamentally different from a Technique Paper:
+
+| Dimension | Technique Paper | Benchmark Paper |
+|-----------|----------------|-----------------|
+| Core claim | "Our method is better" | "This evaluation dimension is overlooked" |
+| Role of problem definition | Goal stated in one sentence | Problem definition IS the contribution (defining "what to evaluate", "how to measure", "what insights emerge") |
+| Introduction axis | Key Idea / Mechanism | Evaluation Gap + Our Benchmark Design |
+| Main chapter | Method / Approach (largest section) | Construction Pipeline is core; companion method is bonus. For VLDB Evaluation Papers, evaluation dimensions & perspectives are the key. |
+| Experiment goal | Prove SOTA | Reveal model capability boundaries + deep insights for future research |
+| Key figures & tables | Architecture diagram | Benchmark comparison table + Construction Pipeline figure + Multi-dimensional analysis charts |
+| Key contribution | Novel technique | New evaluation paradigm + empirical findings |
+
+**The benchmark paper's core is NOT "proposing a dataset", it is defining a new evaluation dimension, providing systematic measurement infrastructure, and revealing deep insights about model capabilities.**
+
+The Gap and the Motivation must be made explicit. The paper needs to state plainly, in its own Introduction, exactly why this benchmark or this evaluation paper needs to exist.
+
+## The Five Pillars
+
+Every excellent benchmark paper addresses these five elements:
+
+1. **Research Gap** (evaluation blind spot), a clear, defensible evaluation blind spot
+2. **Construction Pipeline** (construction methodology), systematic, scalable, high-quality data creation
+3. **Evaluation Framework** (evaluation framework detail), multi-dimensional taxonomy with fine-grained diagnostics
+4. **Empirical Findings** (empirical insights), insights beyond leaderboard numbers, with "Finding X" summaries
+5. **Companion Method** (companion method, optional but recommended), a model leveraging the benchmark to improve capability
+
+## Stage Detection & Routing
+
+```dot
+digraph bench_pipeline {
+    rankdir=LR;
+    node [shape=box, style=rounded];
+
+    S [label="START", shape=doublecircle];
+    G [label="1. Gap Analysis"];
+    D [label="2. Benchmark Design"];
+    C [label="3. Construction"];
+    E [label="4. Experiments"];
+    P [label="5. Paper Structure"];
+    K [label="6. Checklist", style="rounded,bold"];
+
+    S -> G [label="no gap statement"];
+    S -> D [label="gap clear"];
+    S -> C [label="design done"];
+    S -> E [label="data built"];
+    S -> P [label="experiments done"];
+    S -> K [label="draft complete"];
+
+    G -> D; D -> C; C -> E; E -> P; P -> K;
+}
+```
+
+When invoked, ask the user: **"Which stage is the benchmark paper currently at?"** and present:
+
+| Stage | Condition | Invoke Skill | Key Question Answered |
+|-------|-----------|-------------|----------------------|
+| 1. Gap Analysis | Still exploring why this benchmark needs to exist | `bench-gap-analysis` | Why does this benchmark need to exist? |
+| 2. Benchmark Design | Gap confirmed, currently designing the benchmark system | `bench-design` | What does the benchmark look like? |
+| 3. Construction Pipeline | Design locked, planning the data construction pipeline | `bench-construction` | How is the data built? |
+| 4. Experiment Design | Data ready or under construction, experiments need to be designed | `bench-experiments` | What experiments reveal the deepest insights? |
+| 5. Paper Structure | Experiments complete, currently writing the paper | `bench-paper-structure` | How to organize and write the paper? |
+| 6. Pre-submission Checklist | First draft complete, running pre-submission checks | `bench-checklist` | Is the paper ready to submit? |
+
+If the user describes their situation without choosing a number, infer the stage from context and confirm before routing.
+
+## Reference Exemplars
+
+This skill system draws from three published benchmark papers as recurring case studies:
+
+| Paper | Venue | Domain | Key Innovation |
+|-------|-------|--------|---------------|
+| **StatQA** | NeurIPS 2024 | Math/Statistics | Reverse synthesis: fix answer first, then generate question |
+| **nvBench 2.0** | NeurIPS 2025 | Text-to-Visualization | Controlled ambiguity injection from unambiguous seeds |
+| **VisJudge-Bench** | ICLR 2026 | Visualization Evaluation | Adaptive question generation + 3-stage expert annotation |
+
+Each sub-skill references these papers as concrete examples of successful strategies. A detailed instantiation template is available in `references/instantiation-template.md`.
+
+## Recommended Paper Structure
+
+For reference, the recommended section structure is:
+
+```
+Section 1: Introduction (1.5 pages)
+Section 2: The Proposed Benchmark (3-4 pages), core chapter
+Section 3: Specialized Model/Method (1-2 pages, optional)
+Section 4: Experiments & Empirical Findings (4-5 pages)
+Section 5: Discussion & Research Opportunities (1 page)
+Section 6: Related Work (1 page)
+Section 7: Conclusion (0.5 page)
+```

package/dist/skills/benchmark-paper-template/references/paper-structure.md

@@ -0,0 +1,209 @@
+# Benchmark Paper Structure & Writing
+
+## Contents
+
+- [Introduction: 6-Part Framework](#introduction-6-part-framework)
+- [Critical Introduction Figures & Tables](#critical-introduction-figures--tables)
+- [Main Body Section Structure](#main-body-section-structure)
+- [Figure & Table Placement Guide](#figure--table-placement-guide)
+- [Key Writing Techniques for Benchmark Papers](#key-writing-techniques-for-benchmark-papers)
+- [Appendix Planning](#appendix-planning)
+
+Organize and write a Benchmark/Evaluation paper. The Introduction is the "compressed version" of the entire paper, its narrative axis is NOT "Key Idea / Mechanism" but **"Evaluation Gap + Benchmark Design Rationale."**
+
+## Introduction: 6-Part Framework
+
+The Introduction is the "compressed version" of the entire paper. Its narrative axis is NOT "Key Idea / Mechanism" but **"Evaluation Gap + Benchmark Design Rationale."**
+
+The Introduction should follow this exact logical chain. Each part maps to approximately one paragraph:
+
+### Part 1: Background & Core Scenario (~1 paragraph)
+- Establish importance of the broader task/domain in real-world scenarios
+- Show the field has made rapid progress (cite recent advances)
+- Use a carefully designed **Running Example (Figure 1)** to illustrate the task's complexity and how human experts handle it
+- **Figure 1 is one of the most important figures in the entire paper, it must be meticulously crafted**
+- **Tone**: Authoritative, citing well-known work
+
+> **Reference**: StatQA uses a complete statistical analysis workflow; nvBench 2.0 shows multiple valid interpretations of an ambiguous query; VisJudge-Bench contrasts positive/negative examples across Fidelity/Expressiveness/Aesthetics.
+
+### Part 2: Limitations of Existing Benchmarks, The Gap (~1 paragraph)
+- Survey what existing benchmarks measure, acknowledge their contributions fairly
+- Identify the IMPLICIT ASSUMPTION they all share, make it explicit
+- Articulate the evaluation blind spot (≤3 points, specific not vague)
+- **Strongly recommended**: provide a Benchmark Comparison Table (Table 1), comparing feature dimensions item by item to make the gap self-evident
+- **This is the most important paragraph.** It should make the reader think: "I never considered that, but it's obviously important."
+
+> **Reference**: nvBench 2.0 Table 1 compares 7 benchmarks across 6 dimensions; VisJudge-Bench Table 1 compares 7 benchmarks across 3 evaluation dimensions.
+
+### Part 3: Research Questions (~1 paragraph)
+- Transform the gap into 2-3 concrete Research Questions
+- RQ coverage should span: (1) How to build an appropriate benchmark? (2) Where are the capability boundaries of current models? (3) What is the gap between models and human experts?
+- RQs anchor the entire experiment section
+
+> **Reference**: StatQA: Q1 How to evaluate? Q2 How capable are LLMs? Q3 Human vs. LLM?
+
+### Part 4: Design Considerations (~1 paragraph)
+
+<EXTREMELY-IMPORTANT>
+This is a paragraph UNIQUE to Benchmark papers. Before introducing your solution, first articulate "what properties should a good benchmark have", this provides the rationale for your design choices.
+</EXTREMELY-IMPORTANT>
+
+- State design requirements explicitly (e.g., diverse coverage, fine-grained diagnostics, scalability, quality assurance)
+- This provides the logical foundation for why your benchmark is designed the way it is
+
+> **Reference**: nvBench 2.0: "This benchmark should include diverse ambiguous queries, multiple valid outputs, reasoning paths, and broad domain coverage."
+
+### Part 5: Our Benchmark + Key Findings Preview (~1-2 paragraphs)
+- Formally introduce: benchmark name, scale, key design choices
+- Highlight construction methodology innovation in 1-2 sentences
+- Reference Figure 1 and Table 1
+- Summarize 2-3 most surprising empirical findings to entice the reader
+- Use concrete numbers: "We evaluate 15 models across 6 dimensions"
+- **Tone**: Confident but precise
+
+> **Reference**: StatQA: "Our key idea is to reverse this process by synthesizing the question Q based on target answers A."
+
+### Part 6: Contributions (~1 paragraph, bulleted list)
+Typically 3-4 contributions, each mapping to a paper section:
+
+1. We identify [the gap] and propose [Name], the first benchmark for [capability]. (→ §1-2)
+2. We design [pipeline innovation] that achieves [quality/scale metric]. (→ §2)
+3. We conduct comprehensive evaluation of [N] models and reveal [key finding]. (→ §4)
+4. (Optional) We propose [companion method] that improves [metric] by [amount]. (→ §3)
+
+## Critical Introduction Figures & Tables
+
+### Figure 1: Running Example (MUST HAVE)
+The most important figure in the paper. Requirements:
+
+- Illustrates the evaluation gap with ONE concrete example
+- Shows what existing evaluation misses vs. what yours captures
+- Self-contained, a reader should grasp the paper's core idea from this figure alone
+- Placed on page 1-2 (above the fold if possible)
+
+**Design patterns:**
+- **Split view**: "Existing evaluation" (left) vs. "Our evaluation" (right)
+- **Walkthrough**: One input → multiple evaluation dimensions highlighted
+- **Contrast**: Same model output scored by old criteria vs. new criteria
+
+### Table 1: Benchmark Comparison (MUST HAVE)
+Compare with 5-8 existing benchmarks. The rightmost column should be your key differentiator:
+
+```
+| Benchmark | Year | Scale | [Dim1] | [Dim2] | [Dim3] | [Your Key Feature] |
+|-----------|------|-------|--------|--------|--------|--------------------|
+| Prior-1   | 20XX | N     | ✓      | ✗      | ✓      | ✗                  |
+| Prior-2   | 20XX | N     | ✓      | ✓      | ✗      | ✗                  |
+| **Ours**  | 2025 | N     | **✓**  | **✓**  | **✓**  | **✓**              |
+```
+
+## Main Body Section Structure
+
+### Section 2: The Proposed Benchmark (3-4 pages)
+
+```
+2.1 Design Goals & Task Scope
+    - State G1-G4 explicitly
+    - Define in-scope vs. out-of-scope
+
+2.2 Benchmark Construction Pipeline
+    - Pipeline overview paragraph
+    - Figure 2: Pipeline flowchart (MUST HAVE)
+    - Detailed steps with input → operation → output
+    - Quality control measures
+    - Annotation protocol (if human annotation involved)
+
+2.3 Benchmark Characteristics
+    - Table 2: Dataset statistics (splits, categories, scale)
+    - Figure 3-4: Distribution charts (taxonomy, difficulty, length)
+    - Figure 5: Data examples (2-3 representative samples)
+    - Comparison with existing benchmarks on scale and coverage
+```
+
+### Section 3: Specialized Model (1-2 pages, optional)
+
+```
+3.1 Motivation, how the benchmark enables model improvement
+3.2 Method, training signal + optimization technique
+3.3 Implementation details
+```
+
+**This section answers**: "This benchmark not only evaluates, it helps improve models."
+
+### Section 4: Experiments & Empirical Findings (4-5 pages)
+
+```
+4.1 Experimental Setup
+    - Models (grouped by type), metrics, prompting, implementation
+
+4.2 Overall Performance
+    - Table 3: ALL models × ALL metrics (largest table in paper)
+    - Bold best, underline second-best
+    - Finding 1 after analysis
+
+4.3 Fine-grained Analysis (organized by RQ)
+    - RQ1 analysis → Finding 2
+    - RQ2 analysis → Finding 3
+    - RQ3 analysis → Finding 4
+    - Each with supporting figure/table
+
+4.4 Case Studies
+    - 2-4 examples with qualitative analysis
+    - Success case, failure case, surprising case
+```
+
+### Section 5: Discussion & Research Opportunities (~1 page)
+
+Based on Findings, discuss:
+- How to enhance model capability on the identified gap
+- Human-AI collaboration patterns
+- Benchmark extension directions (new modalities, languages, domains)
+- Limitations of the current benchmark
+
+### Section 6: Related Work (~1 page)
+
+Organize into 2-3 subsections:
+- [Task-specific] Benchmarks
+- [Domain] Evaluation Methodologies
+- [Related capability] in Large Language Models
+
+**Include a comparison table** (can be expanded Table 1 or a new table focused on methodology differences).
+
+**Placement note**: Related Work can go before or after Experiments. Before Experiments (like VisJudge-Bench) emphasizes positioning; after Experiments (like StatQA, nvBench 2.0) lets findings speak first.
+
+### Section 7: Conclusion (~0.5 page)
+- Restate the gap and benchmark in 1-2 sentences
+- Summarize key findings (reference Finding numbers)
+- Restate open problems and future directions
+
+## Figure & Table Placement Guide
+
+| Item | Position | Purpose | Priority |
+|------|----------|---------|----------|
+| Figure 1 | Page 1-2 | Running Example | MUST |
+| Table 1 | Page 2-3 | Benchmark Comparison | MUST |
+| Figure 2 | Page 3-4 (§2.2) | Construction Pipeline | MUST |
+| Table 2 | Page 4-5 (§2.3) | Dataset Statistics | MUST |
+| Figure 3-4 | Page 4-5 (§2.3) | Distribution Charts | HIGH |
+| Figure 5 | Page 5 (§2.3) | Data Examples | HIGH |
+| Table 3 | Page 6-7 (§4.2) | Overall Performance | MUST |
+| Figure 6+ | Page 7-8 (§4.3) | Analysis Charts | HIGH |
+
+## Key Writing Techniques for Benchmark Papers
+
+1. **"Finding X" summaries**: Bold-highlight after each major analysis (see bench-experiments)
+2. **Design rationale over description**: For every choice, explain WHY, not just WHAT
+3. **Concrete numbers**: "We evaluate 15 models across 6 dimensions" not "many models"
+4. **Active positioning**: "the FIRST benchmark to..." / "Unlike prior work, we..."
+5. **Example-driven arguments**: Always pair abstract claims with concrete examples
+6. **Appendix strategy**: Move detailed content to appendix (full results, annotation guidelines, more examples, prompts used)
+
+## Appendix Planning
+
+Standard appendix sections for benchmark papers:
+- A: Full experimental results (all models × all metrics, unabridged)
+- B: Annotation guidelines and interface screenshots
+- C: Additional data examples
+- D: Prompts used for evaluation
+- E: Extended related work or taxonomy details
+- F: Ethical considerations and datasheet