@research-copilot/plugin 1.1.17 → 1.1.19

package/dist/skills/system-profile/SKILL.md

@@ -0,0 +1,103 @@
+---
+name: system-profile
+description: "Profile a target (script, process, GPU, memory, interconnect) for performance analysis. Use when user says \"profile\", \"benchmark\", \"bottleneck\", or wants performance analysis."
+argument-hint: <target, e.g. "train.py", "gpu", "pid 1234", "vllm serving">
+---
+
+# System Profile
+
+Profile the specified target and summarize the results. Target: $ARGUMENTS
+
+## Instructions
+
+You are a profiling assistant. Based on the user's target, choose appropriate profiling strategies, **including writing instrumentation code when needed**, then run profiling, analyze results, and produce a summary.
+
+### Step 1: Determine the profiling target
+
+Parse `$ARGUMENTS` to understand what to profile. Examples:
+- A Python script or module
+- A running process (PID or service name)
+- A specific function or code block
+- An entire framework or system (e.g., "autogen", "vllm serving") — profile its end-to-end execution, identify bottlenecks across components
+- "gpu" / "interconnect" / "memory" for focused profiling
+
+If `$ARGUMENTS` is empty or unclear, ask the user.
+
+### Step 2: Choose profiling methods
+
+Select from external tools and/or code instrumentation as appropriate. Don't limit yourself to the examples below — use whatever makes sense for the target.
+
+**External tools** (check availability first):
+- CPU: `cProfile`, `py-spy`, `line_profiler`, `perf stat`, `/usr/bin/time -v`
+- Memory: `tracemalloc`, `memory_profiler`, `memray`
+- GPU: `nvidia-smi`, `nvidia-smi dmon`, `nvitop`, `torch.profiler`, `nsys`
+- Interconnect: `nvidia-smi topo -m`, `nvidia-smi nvlink`, `NCCL_DEBUG=INFO`
+- System: `strace -c`, `iostat`, `vmstat`
+
+**Code instrumentation** — when external tools are insufficient, write and insert profiling code into the target. Typical scenarios:
+- Timing specific code blocks (wall time vs CPU time)
+- Measuring CPU-GPU or GPU-GPU transfer size, frequency, and bandwidth
+- Tracking memory allocation across CPU and GPU to detect redundancy
+- Wrapping NCCL collectives to measure latency and throughput
+- Adding CUDA event timing around kernels
+
+Design the instrumentation based on what you observe in the code — don't use a fixed template.
+
+### Step 3: Key dimensions to investigate
+
+Depending on the target, focus on some or all of these:
+
+**CPU overhead**
+- Context switching (voluntary / involuntary)
+- CPU utilization: ratio of CPU time to wall time
+- Per-function execution time hotspots
+
+**Memory overhead**
+- CPU and GPU memory usage (allocated vs reserved vs peak)
+- Redundant replication: same data living on both CPU and GPU
+- Per-device allocation balance in multi-GPU setups
+
+**Interconnect & communication**
+- CPU-GPU transfer: frequency, per-transfer size, total volume, bandwidth achieved
+- GPU-GPU transfer: P2P bandwidth, NVLink vs PCIe topology impact
+- NCCL collectives: operation type, message size distribution, latency
+- Communication-to-computation ratio
+
+**GPU compute**
+- SM utilization, kernel launch overhead
+- Memory bandwidth utilization vs peak
+
+### Step 4: Instrumentation guidelines
+
+When inserting code into the target:
+1. Read and understand the target code first
+2. Prefer wrapping (decorator, context manager, standalone runner) over inline edits
+3. If inline edits are necessary, mark them clearly (e.g., `# [PROFILE]` comments)
+4. Minimize observer effect — don't instrument tight inner loops; sample instead
+5. Collect results into a structured log, don't scatter print statements
+
+### Step 5: Run profiling
+
+1. Check available tools and hardware topology
+2. Run the chosen methods, capture all output
+3. Save artifacts (flamegraphs, traces, logs) to `./profile_output/`
+
+### Step 6: Produce the report
+
+**Part A — Profiling results** (structured tables by dimension, as applicable):
+- CPU overhead table
+- Memory overhead table (with redundancy column)
+- Interconnect table (transfer type / frequency / size / latency / bandwidth)
+- Hotspots / bottleneck identification
+- Actionable recommendations ranked by expected impact
+
+**Part B — Instrumentation changelog** (MANDATORY):
+List every file that was modified or created for profiling purposes:
+
+| File | Change type | What was added/modified | Line(s) |
+|------|-------------|------------------------|---------|
+| ... | modified | ... | ... |
+| ... | created | ... | — |
+
+This allows the user to review and revert all instrumentation changes.
+Offer to clean up (remove all instrumentation) when the user is done.

package/dist/skills/tech-paper-template/SKILL.md

@@ -0,0 +1,192 @@
+---
+name: tech-paper-template
+description: >-
+  Structures a technical paper's full logical skeleton using a
+  thinking-template table (research background, limitations, key idea
+  or goal, challenges, methodology modules, contributions), positions
+  the paper as Technique or New Problem/Setting, and runs a four-point
+  self-consistency check. Use when the user is brainstorming a paper,
+  discussing progress with an advisor, or planning the paper before
+  drafting. Also use for 'paper skeleton', 'paper logic chain',
+  'thinking template', 'paper-structure planning'.
+license: CC-BY-4.0
+---
+
+# Tech Paper Template
+
+## Overview
+
+Before drafting any prose, a technical paper needs a full logical
+skeleton: the research background, the specific limitations of prior
+work, the key idea or research goal, the technical challenges that
+prevent a naive solution, the methodology modules that address each
+challenge, and the contributions that the paper will claim. This
+skill fills in that skeleton via a standardised thinking-template
+table, positions the paper type, and runs four self-consistency
+checks on the logic chain.
+
+The output is a filled-in thinking template plus a consistency
+report. It is suitable for advisor-student brainstorming sessions,
+weekly progress meetings, and the final planning step before writing
+begins. It does not draft Introduction prose (use `intro-drafter` for
+that); it operates at the logical-skeleton layer.
+
+## When to use this skill
+
+- Early brainstorming of a paper project.
+- Weekly progress meeting with an advisor or collaborator.
+- Pre-drafting planning after `idea-evaluator` returns Strong Accept.
+- The paper's logic chain feels incoherent and needs an audit.
+- The user asks for 'paper skeleton', 'paper logic chain', 'thinking
+  template', or 'paper-structure planning'.
+- The user is unsure whether their paper is Technique or New
+  Problem/Setting.
+
+## When NOT to use this skill
+
+- The paper is a benchmark paper. Use `benchmark-paper-template` (separate plugin).
+- The user needs an Introduction-specific paragraph outline. Use
+  `intro-drafter` (typically run this skill first, then
+  `intro-drafter`).
+- The user has a written draft and wants review feedback. Use
+  `pre-submission-reviewer`.
+- The idea itself is not yet vetted. Use `idea-evaluator` first.
+
+## Core procedure
+
+### Step 1: Paper-type positioning
+
+See: references/paper-types.md for the positioning criteria and
+worked examples.
+
+Decide Technique versus New Problem/Setting. In Technique, the Key
+Idea carries the narrative and Our Goal is a short bridge. In New
+Problem/Setting, Our Goal is the contribution and the Key Idea
+justifies feasibility.
+
+If the user's inputs describe a benchmark, stop and redirect to
+`benchmark-paper-template` (separate plugin).
+
+### Step 2: Fill the thinking template
+
+See: references/thinking-template.md for each template cell's content
+contract, what a strong cell looks like, and common failure modes.
+
+Fill the seven cells:
+
+1. Research background. Scenario, importance, motivation.
+2. Limitations 1 through 3 (2 is acceptable; more than 3 is not).
+3. Key idea or Our Goal. One sentence.
+4. Challenges 1 through 3 (similar cap).
+5. Methodology modules. One module per challenge.
+6. Contributions (3 or 4, each mapped to a section).
+
+If a cell is incomplete given the user's inputs, mark it as a gap
+with severity.
+
+### Step 3: Run four self-consistency checks
+
+See: references/consistency-checks.md for the detailed checking
+procedure and examples of chain breaks.
+
+Run each check:
+
+1. **Limitations to Key Idea**: does the Key Idea or Goal address
+   the stated Limitations? If not, either the Limitations are
+   wrong or the Key Idea is misaligned.
+2. **Key Idea to Challenges**: do the Challenges arise naturally
+   from implementing the Key Idea? If not, the challenges are
+   invented to justify modules rather than derived from the idea.
+3. **Challenges to Methodology**: does each methodology module
+   address one challenge? If not, there is a module without
+   justification or a challenge without a fix.
+4. **Methodology to Contributions**: do the contributions cover
+   each module or experimental result? If not, contributions are
+   vague or promising more than the paper delivers.
+
+Every failure is CRITICAL.
+
+### Step 4: Generate methodology outline
+
+See: references/thinking-template.md for the methodology-outline
+template.
+
+From the challenges, derive a methodology outline: topic sentence,
+per-module subsection names, and per-module one-sentence summary.
+This becomes the skeleton for Section 3 or 4 of the paper.
+
+### Step 5: Integrity gate
+
+Before emitting, run the checks in the Integrity gate section
+below.
+
+### Step 6: Output
+
+Emit the filled template plus the consistency report in the
+Output format below.
+
+## Integrity gate
+
+All seven bullets are **[inspection]** class: the LLM verifies each
+directly from the filled template (counting, pattern-matching, or
+comparing cells). No user-side attestation required.
+
+Before returning the filled template:
+
+1. **[inspection]** Paper-type positioning is consistent with the
+   user's actual contribution (Technique paper not shoehorned into
+   New Problem framing, or vice versa).
+2. **[inspection]** Limitations are specific and cited-able, not
+   vague.
+3. **[inspection]** Key Idea or Goal is a single sentence a
+   reviewer could quote.
+4. **[inspection]** Challenges derive from implementing the Key
+   Idea; they are not invented.
+5. **[inspection]** Methodology modules have one-to-one mapping
+   with challenges.
+6. **[inspection]** Contributions map to methodology modules and
+   to specific sections.
+7. **[inspection]** All four self-consistency checks pass.
+
+If any check fails, mark the skeleton as "needs user attention".
+
+## Output format
+
+### 1. Paper-type positioning
+- Type: <Technique Paper or New Problem/Setting Paper>
+- Rationale: <one sentence>
+
+### 2. Thinking template
+
+| Stage | Your content |
+|---|---|
+| Research background | ... |
+| Limitation 1 | ... |
+| Limitation 2 | ... |
+| Limitation 3 (if applicable) | ... |
+| Key Idea / Our Goal | ... |
+| Challenge 1 | ... |
+| Challenge 2 | ... |
+| Challenge 3 (if applicable) | ... |
+| Methodology topic sentence | ... |
+| Module A (addresses Challenge 1) | ... |
+| Module B (addresses Challenge 2) | ... |
+| Module C (addresses Challenge 3) | ... |
+| Contribution 1 | ... (Section <X>) |
+| Contribution 2 | ... (Section <Y>) |
+| Contribution 3 | ... (Section <Z>) |
+
+### 3. Self-consistency checks
+- Check 1 Limitations -> Key Idea: <pass or fail>
+- Check 2 Key Idea -> Challenges: <pass or fail>
+- Check 3 Challenges -> Methodology: <pass or fail>
+- Check 4 Methodology -> Contributions: <pass or fail>
+
+### 4. Severity summary
+- <n> CRITICAL, <m> MAJOR, <k> MINOR.
+- Top three fixes first: ...
+
+### 5. Next suggested skill
+- If all checks pass: `intro-drafter` to produce the Introduction
+  paragraph outline.
+- If checks fail: address the flagged chain breaks first.

package/dist/skills/tech-paper-template/references/consistency-checks.md

@@ -0,0 +1,166 @@
+# Four self-consistency checks
+
+## Table of contents
+
+1. Why check consistency
+2. Check 1: Limitations to Key Idea
+3. Check 2: Key Idea to Challenges
+4. Check 3: Challenges to Methodology
+5. Check 4: Methodology to Contributions
+6. Common chain breaks
+
+## 1. Why check consistency
+
+A technical paper with a clean logic chain feels inevitable to a
+reviewer: each stage flows naturally from the previous one. A paper
+with a broken chain feels patchwork, even if every individual stage
+is strong. The four consistency checks below catch the patchwork
+pattern before drafting begins, saving weeks of rewriting later.
+
+Each check detects a specific class of failure. Running all four in
+sequence catches most chain breaks.
+
+## 2. Check 1: Limitations to Key Idea
+
+### Procedure
+
+For each limitation, ask: does the Key Idea or Our Goal address
+this limitation, directly or indirectly?
+
+### Detection of failure
+
+- A limitation has no corresponding response in the Key Idea or
+  Goal. The limitation is either wrong (it is not actually a
+  limitation of the prior work, or it is a limitation the paper
+  cannot address) or the Key Idea is misaligned.
+- The Key Idea addresses a problem that is not in any Limitation
+  cell. The paper is actually solving a different problem from the
+  one it motivates in the Introduction.
+
+### Fix
+
+- Remove limitations that the paper does not address.
+- Add limitations that justify the Key Idea if they are missing.
+- In rare cases, change the Key Idea to align with the motivation.
+
+### Example of failure
+
+Limitation: "prior methods cannot scale to large schemas."
+Key Idea: "we introduce chain-of-thought prompting for schema
+linking."
+Chain break: the Key Idea does not address scale, only accuracy.
+Fix: either add a scale mechanism to the Key Idea, or replace the
+limitation with "prior methods have low accuracy".
+
+## 3. Check 2: Key Idea to Challenges
+
+### Procedure
+
+For each challenge, ask: does this challenge arise naturally from
+implementing the Key Idea?
+
+### Detection of failure
+
+- A challenge has no connection to the Key Idea. It may be a
+  generic challenge copied from another paper or a challenge
+  invented to justify a module.
+- The Key Idea, if implemented naively, would not surface this
+  challenge. The challenge is unrelated.
+
+### Fix
+
+- Remove invented challenges.
+- If a real challenge is missing, add it.
+- Sometimes the Key Idea itself is weaker than thought: the naive
+  implementation works and the challenges are artificial. In that
+  case, strengthen the Key Idea or accept that the paper's
+  contribution is smaller than imagined.
+
+### Example of failure
+
+Key Idea: "use MCTS search at inference time for SQL generation."
+Challenge: "labelling data is expensive."
+Chain break: MCTS inference-time search is label-free; data labelling
+is unrelated to the Key Idea.
+Fix: drop the labelling challenge; keep only challenges derived from
+MCTS inference (for example, reward design, search-depth trade-off).
+
+## 4. Check 3: Challenges to Methodology
+
+### Procedure
+
+For each methodology module, ask: does this module address one
+specific challenge? For each challenge, ask: is there a module that
+addresses it?
+
+### Detection of failure
+
+- Module without a challenge (the module addresses a problem the
+  paper did not motivate).
+- Challenge without a module (the paper complains about a problem
+  it does not solve).
+- One-to-many mapping (one module addresses multiple challenges)
+  without explicit justification. Sometimes this is fine; usually
+  it suggests the challenges collapse into one.
+
+### Fix
+
+- Cut modules with no challenge.
+- Add modules for orphan challenges, or drop the challenge if the
+  paper does not address it.
+- Collapse redundant challenges.
+
+### Example of failure
+
+Challenge 1: "reward signal is sparse."
+Challenge 2: "search space is exponential."
+Module A: "bias the search using prior-learned reward shapes."
+Module B: none for Challenge 2.
+Chain break: Challenge 2 is orphan.
+Fix: either add a module for Challenge 2 (perhaps a pruning
+heuristic) or drop Challenge 2.
+
+## 5. Check 4: Methodology to Contributions
+
+### Procedure
+
+For each module, ask: is this module represented in the
+Contributions cell? For each contribution, ask: is there a module
+or experimental result that delivers it?
+
+### Detection of failure
+
+- A module is not represented in the contributions. The paper does
+  the work but forgets to claim credit.
+- A contribution is not backed by a module or experiment. The paper
+  claims credit without delivery. This is the most dangerous chain
+  break; reviewers quote it in reviews.
+
+### Fix
+
+- Add missing contribution entries.
+- Remove overclaimed contributions.
+
+### Example of failure
+
+Contribution 3: "theoretical analysis of convergence."
+Module: none; no theoretical subsection in the paper.
+Chain break: contribution claims theory the paper does not provide.
+Fix: either add a theoretical module and subsection or drop
+contribution 3.
+
+## 6. Common chain breaks
+
+- **Invented challenges** (fails Check 2).
+- **Orphan modules** (fails Check 3).
+- **Vague contributions that do not map to modules** (fails Check 4).
+- **Key Idea drift**: the Key Idea as filled in the template is
+  different from the Key Idea the paper actually develops in the
+  methodology. Catch by re-reading the Key Idea cell after the
+  Methodology cell is filled.
+- **Limitation-Contribution short-circuit**: a contribution that
+  does not address any limitation; the paper solves a different
+  problem from the one it motivates.
+
+All of the above are CRITICAL gaps. Fix them before running
+`intro-drafter` or writing any Introduction prose.

package/dist/skills/tech-paper-template/references/paper-types.md

@@ -0,0 +1,127 @@
+# Paper-type positioning for technical papers
+
+## Table of contents
+
+1. Three paper-type categories
+2. Technique Paper
+3. New Problem/Setting Paper
+4. Mixed case (cross-domain technique)
+5. Positioning questions
+6. How positioning affects the template
+
+## 1. Three paper-type categories
+
+Technical papers at top venues (SIGMOD, VLDB, ICML, NeurIPS, ICLR,
+ACL) fall into three categories. Positioning the paper correctly is
+the first planning decision.
+
+- **Technique Paper**: new method or mechanism on an existing
+  problem. Key Idea carries the narrative.
+- **New Problem/Setting Paper**: a new problem formulation or a new
+  setting within a known problem. Goal or Problem Formulation
+  carries the narrative.
+- **Cross-domain Technique Paper**: a Technique Paper whose
+  contribution transplants an idea from one field to another. The
+  Key Idea carries the narrative but the Broader dimension from the
+  `idea-evaluator` framework is emphasised in the Introduction.
+
+Benchmark papers are a separate category; use
+`benchmark-paper-template` (separate plugin).
+
+## 2. Technique Paper
+
+Signals:
+
+- The problem is well-established. A named benchmark exists; prior
+  work is extensive.
+- The contribution is "we do X better" for a specific X.
+- Reviewers compare against the strongest existing method on the
+  same problem.
+- Experiments focus on dominating baselines on a shared metric.
+
+Template implications:
+
+- Key Idea cell is load-bearing.
+- Goal is a short bridge, not a contribution.
+- Challenges flow from implementing the Key Idea.
+- Contribution 1 is usually the framework or method name.
+
+## 3. New Problem/Setting Paper
+
+Signals:
+
+- The problem is a new framing the community has not studied as
+  such.
+- Prior work exists on adjacent problems but not this specific
+  framing.
+- The contribution is a new way to think about the space, not a
+  new method on an existing problem.
+- Experiments characterise the new problem rather than beating a
+  leaderboard.
+
+Template implications:
+
+- Goal cell is load-bearing. Multiple sentences may be appropriate.
+- Key Idea supports "why this definition is reasonable and
+  feasible".
+- Limitations paragraph shows why existing framings miss the
+  problem.
+- Contribution 1 is the problem formulation itself.
+
+## 4. Mixed case: cross-domain technique
+
+A paper transplanting a mature idea from one field to another is a
+Technique Paper with a Broader-dimension framing.
+
+Signals:
+
+- The technique is mature in one subfield but new to another.
+- The Introduction leans on "inspired by X from field A, we apply
+  it to field B".
+- The contribution is the transfer and the mechanism, not a new
+  technique invented from scratch.
+
+Template implications:
+
+- Key Idea includes both the source-field technique and the
+  transfer mechanism.
+- Limitations emphasise the absence of field B from field A's
+  literature.
+- Challenges usually include "transferring idea X across the
+  domain gap".
+- Contribution 1 names the transfer.
+
+Example: AFlow (ICLR 2025) transplants operator-graph search (from
+neural architecture search) into agent-workflow design.
+
+## 5. Positioning questions
+
+Ask the following in order. First yes decides:
+
+1. Is the paper's primary contribution a benchmark? If yes, stop
+   and redirect to `benchmark-paper-template` (separate plugin).
+2. Is the paper's primary contribution a new problem formulation
+   that the community has not studied? If yes, New Problem/Setting.
+3. Is the paper's primary contribution a technique transplanted
+   from another field? If yes, Mixed (Technique with cross-domain
+   framing).
+4. Default: Technique Paper.
+
+If the user's answers conflict with this order, investigate. A
+paper can legitimately have two strong framings; see the mixed-case
+discussion in the Introduction-drafter skill's paper-types
+reference.
+
+## 6. How positioning affects the template
+
+| Cell | Technique | New Problem/Setting | Mixed Cross-domain |
+|---|---|---|---|
+| Background | Scenario that shows the problem | Scenario that shows the phenomenon | Scenario that shows both fields |
+| Limitations | Specific to prior methods | Specific to prior framings | Specific to field B's absence from field A |
+| Key Idea / Goal | Key Idea sentence dominant | Goal sentence dominant, often multiple sentences | Key Idea names the transfer |
+| Challenges | Derived from implementing Key Idea | Derived from realising the new setting | Derived from domain gap |
+| Methodology | Modules that realise Key Idea | Modules that realise the setting | Modules that realise the transfer |
+| Contributions | C1 = framework or method; C2-C4 = techniques, evaluation | C1 = problem formulation; C2-C4 = framework, technique, evaluation | C1 = transfer framing; C2-C4 = method, evaluation |
+
+Apply the row that matches the positioning when filling the
+template in Step 2 of the skill's core procedure.