@research-copilot/plugin 1.1.16 → 1.1.18

package/dist/skills/idea-evaluator/references/fatal-flaws.md

@@ -0,0 +1,192 @@
+# Fatal flaws in research ideas
+
+## Table of contents
+
+1. What counts as fatal
+2. Ten canonical fatal flaws
+3. Detection rules
+4. Defense strategies
+5. Severity escalation logic
+
+## 1. What counts as fatal
+
+A fatal flaw is a problem that, if left unaddressed, causes rejection
+from a top venue regardless of how well the rest of the paper is
+executed. The bar is deliberately high: not every weakness is fatal.
+Writing an idea off as fatally flawed stops the student from
+pursuing it, so this section uses a conservative definition.
+
+A fatal flaw has three signatures: it is observable in the Idea
+description (not only discovered during experiments); it cannot be
+fixed by stronger baselines or better writing alone; and reviewers
+will flag it in the first review round.
+
+The canonical count is at most two fatal flaws per idea. If the list
+exceeds two, the idea's direction itself is wrong and a pivot is
+required.
+
+## 2. Ten canonical fatal flaws
+
+### F1: No novelty versus the closest prior work
+
+The idea replicates or barely varies a published baseline in the same
+subfield. Reviewers frame this as "dominated by prior work X".
+
+### F2: Wrong venue fit
+
+The idea's contribution matches a different venue. A systems
+contribution submitted to ICML is often rejected even if the work is
+strong; a theory contribution submitted to VLDB meets the same fate.
+
+### F3: Baseline is not the real baseline
+
+The chosen baseline is weak or outdated. Beating a 2023 baseline in
+2026 convinces no one; reviewers demand the current year's SOTA.
+
+### F4: No compelling motivation
+
+The idea's usefulness in the real world is unclear. The paper cannot
+answer "who cares and why now"; reviewers call this "not motivated".
+
+### F5: Capability mismatch
+
+The student cannot execute the idea within its lifecycle. The idea is
+valid, but the student lacks the skills, time, or resources to
+complete it, causing a missed deadline or an incomplete paper.
+
+### F6: Unverifiable claim
+
+The idea depends on an empirical claim that cannot be verified from
+the planned experiments. Example: "our method works across domains"
+but no cross-domain experiments are in scope.
+
+### F7: Ethical or data-access blocker
+
+The idea needs data or human subjects the student cannot access.
+IRB approval missing; proprietary data; privacy constraint.
+Infrastructure makes execution impossible.
+
+### F8: Overly ambitious scope
+
+The idea promises too much in the contribution list. Example: "we
+propose a benchmark, a new method, theoretical analysis, and a
+deployed system" in a single paper. Each contribution is undercut by
+the others; reviewers flag this as "unfocused".
+
+### F9: Solution hunting for a problem
+
+The idea begins with a technique the student wants to use and
+searches for a problem it fits. Often produces papers where the
+problem feels contrived and the experiments are not decisive.
+
+### F10: No failure case considered
+
+The idea treats the method as a silver bullet. No discussion of
+where it fails, under what conditions, or what the limitations are.
+Reviewers flag this as overclaiming.
+
+## 3. Detection rules
+
+### F1 detection
+
+- Ask: what does this idea add over the single closest prior work?
+- Red flag: the student cannot name a specific contribution in one
+  sentence, or the contribution is "we use a bigger model" or "we
+  combine two existing methods".
+
+### F2 detection
+
+- Ask: what is the target venue, and what are the top three papers
+  from that venue's most recent edition?
+- Red flag: the idea's contribution type does not appear in those
+  three papers.
+
+### F3 detection
+
+- Ask: what is the strongest public result on the target benchmark
+  as of the most recent three months?
+- Red flag: the student's baseline is more than 12 months old or
+  does not cite a 2026 result.
+
+### F4 detection
+
+- Ask: if this problem were solved tomorrow, who would benefit, and
+  how much?
+- Red flag: the answer is "other researchers studying this narrow
+  problem" without naming an external beneficiary.
+
+### F5 detection
+
+- Run the matching logic in references/lifecycle-capability-matching.md.
+- Red flag: two or more mismatch flags fire.
+
+### F6 detection
+
+- Ask: what experiment, if it produced a specific result, would prove
+  the main claim?
+- Red flag: the student cannot design such an experiment, or the
+  experiment is out of scope.
+
+### F7 detection
+
+- Ask: is all required data currently accessible? Is IRB approval in
+  place if human subjects are involved?
+- Red flag: any required resource is missing and cannot be secured
+  within the idea's lifecycle.
+
+### F8 detection
+
+- Count the contribution bullets in the proposed Introduction.
+- Red flag: more than four, or the bullets span distinct paper types
+  (benchmark + method + theory + system).
+
+### F9 detection
+
+- Ask: did the student encounter the problem in practice, or did they
+  start from a technique they wanted to apply?
+- Red flag: the student cannot name a concrete real-world failure
+  that motivated the work.
+
+### F10 detection
+
+- Ask: under what conditions does the method fail, and what is the
+  plan for the Limitations section?
+- Red flag: the student cannot name two failure modes.
+
+## 4. Defense strategies
+
+For each flaw, a concrete defense. If the defense cannot be executed
+within the idea's lifecycle, the flaw remains fatal.
+
+| Flaw | Defense |
+|---|---|
+| F1 | Position against the closest prior work in one sentence. Name a specific axis on which the new idea dominates |
+| F2 | Either switch the venue target to match the contribution type, or reshape the contribution to fit the original venue |
+| F3 | Identify the latest state-of-the-art and add it as the primary baseline. If unavailable, document the recency cutoff and justify |
+| F4 | Name a concrete external beneficiary (a user, a deployed system, a policy question) in the first paragraph of the Introduction |
+| F5 | Follow recovery strategies in the lifecycle-capability-matching reference. Narrow scope, partner, reframe category, or pivot |
+| F6 | Design the decisive experiment explicitly and put it at the top of the experiments plan. If infeasible, pivot the claim |
+| F7 | Secure access before proceeding. For IRB, file early. For proprietary data, secure a partnership |
+| F8 | Cut the contribution list to two or three items. Split remaining items into a follow-up paper |
+| F9 | Restart from the problem side. Interview users, run a pilot study, or document a real failure that motivates the technique |
+| F10 | Preregister two failure modes and a Limitations section before starting experiments. Include failure cases in the case-study section |
+
+## 5. Severity escalation logic
+
+After listing flaws and defenses, convert each flaw into a severity
+tag using the following logic.
+
+- **CRITICAL**: the flaw cannot be defended within the idea's lifecycle
+  given current resources, or two or more MAJOR flaws are present.
+- **MAJOR**: the flaw requires 2-4 weeks of dedicated work to defend.
+- **MINOR**: the flaw can be addressed in under a week of writing or
+  literature work.
+
+Verdict implications:
+
+- **Any CRITICAL flaw**: verdict is Reject and Pivot. Do not proceed
+  with this version of the idea.
+- **Two or more MAJOR flaws**: verdict is Accept with Revisions.
+  Defend all flaws before starting experiments.
+- **At most one MAJOR flaw and any MINOR flaws**: compatible with
+  Strong Accept, subject to other evaluation steps.

package/dist/skills/idea-evaluator/references/five-dimensions.md

@@ -0,0 +1,251 @@
+# Five-dimension scoring framework
+
+## Table of contents
+
+1. Origin and purpose
+2. Dimension 1: Higher (Effectiveness)
+3. Dimension 2: Faster (Efficiency)
+4. Dimension 3: Stronger (Robustness and Generalisation)
+5. Dimension 4: Cheaper (Data or Solution Cost)
+6. Dimension 5: Broader (Cross-Domain and Unification)
+7. Scoring rubric
+8. Summary table
+9. Common scoring failures
+
+## 1. Origin and purpose
+
+This framework comes from the methodology for generating
+incremental research ideas given a strong baseline. The insight is:
+do not hunt for new problems with a solution in hand; instead, take a
+well-defined baseline and ask on which of five axes it can be
+improved. Each axis has canonical entry strategies and concrete
+examples.
+
+The same framework flips naturally into an evaluation lens: given an
+idea, score how strongly it advances each axis versus the current
+baseline. Strong ideas usually dominate on one axis and hold their own
+on at least one more. Weak ideas are vague on four or more axes.
+
+## 2. Dimension 1: Higher (Effectiveness)
+
+### Definition
+
+Improves accuracy, quality, or effectiveness metrics over the
+strongest current baseline.
+
+### Canonical entry strategies
+
+- **Information or modality augmentation**. Does the baseline ignore
+  an input signal that would help? Example: Text-to-SQL models using
+  only schema plus query may improve significantly when fed data
+  distribution or domain documents.
+- **Feedback-driven refinement**. Can execution feedback (compiler
+  errors, runtime exceptions, test failures) drive iterative
+  self-correction? Example: LLM code-generation success rates jump
+  when errors from the execution environment are returned to the
+  agent for self-reflection.
+- **Error-driven root-cause analysis**. Run the baseline, cluster
+  failures, identify the dominant failure mode, and build a module
+  that targets it. Example: if Text-to-SQL failures concentrate on
+  complex JOIN operations, design a sub-agent that decomposes JOIN
+  logic.
+
+### How to score
+
+- 9-10: the idea proposes a principled new input signal or feedback
+  loop that no prior work exploits, with a plausible path to
+  multi-point accuracy gains.
+- 6-8: the idea combines known signals or applies a known feedback
+  mechanism in a new setting; expected gains are modest but real.
+- 3-5: the idea is a standard prompt-engineering or fine-tuning
+  variant; gains will likely be within noise of stronger baselines.
+- 1-2: the idea does not obviously advance effectiveness at all.
+
+### Grounded example
+
+Alpha-SQL (ICML 2025): introduces MCTS-based inference-time search
+for Text-to-SQL. Higher score: 8. Evidence: principled search
+mechanism that no prior open-source Text-to-SQL work had applied;
+multi-point accuracy gains on BIRD. Lift: pair with schema
+augmentation for additional signal.
+
+## 3. Dimension 2: Faster (Efficiency)
+
+### Definition
+
+Reduces wall-clock time, token cost, memory footprint, or compute
+budget while preserving effectiveness.
+
+### Canonical entry strategies
+
+- **Caching and experience reuse**. Cache successful trajectories so
+  repeated tasks do not replan from scratch. Example: an LLM agent
+  that retrieves and reuses past successful plans for similar tasks.
+- **Parallelisation and decoupling**. Break a long serial pipeline into
+  independent sub-tasks handled by specialised agents running in
+  parallel. Example: a multi-agent data-analysis framework with one
+  agent per step (ingestion, transformation, visualisation).
+- **Early exit and dynamic routing**. Route simple cases to cheap
+  models, escalate only hard cases. Example: a cascade that uses a
+  small model for filtering and only invokes a large model on
+  ambiguous instances.
+
+### How to score
+
+- 9-10: the idea identifies a large, well-quantified efficiency gap
+  and proposes a principled mechanism that closes it (more than 3x
+  speedup with no accuracy loss).
+- 6-8: the idea targets a known efficiency bottleneck with a plausible
+  mechanism; gains are 1.5x-3x.
+- 3-5: the idea mentions efficiency but lacks a specific mechanism;
+  gains will likely be marginal.
+- 1-2: the idea does not address efficiency.
+
+### Grounded example
+
+LEAD (VLDB 2026): eliminates per-iteration full-dataset inference in
+iterative instruction-tuning data selection. Faster score: 9.
+Evidence: zero-additional-inference overhead is a principled gap
+closed; replaces expensive utility-score computation with training-
+loss signal already computed.
+
+## 4. Dimension 3: Stronger (Robustness and Generalisation)
+
+### Definition
+
+Maintains performance under noise, out-of-distribution inputs, or
+cross-domain transfer.
+
+### Canonical entry strategies
+
+- **Noise tolerance and fault tolerance**. Handle malformed or
+  ambiguous inputs gracefully. Example: intent-clarification modules
+  that ask users for disambiguation instead of failing silently.
+- **Exception recovery**. Detect API failures, unexpected outputs, or
+  tool crashes and retry with alternatives. Example: an agent that
+  falls back to a simpler tool when the primary one returns an
+  unexpected response.
+- **Decoupled representations**. Separate general reasoning from
+  domain-specific knowledge so the same reasoning module transfers
+  zero-shot to new domains.
+
+### How to score
+
+- 9-10: the idea directly addresses a known brittleness (ambiguity,
+  OOD, domain shift) with a principled mechanism and plans
+  cross-domain evaluation.
+- 6-8: the idea mentions robustness with a concrete mechanism but
+  evaluates on a single domain.
+- 3-5: the idea pays lip service to robustness without a mechanism.
+- 1-2: the idea does not address robustness.
+
+### Grounded example
+
+An agent framework that decouples planning from domain-specific APIs
+and ships cross-domain evaluation: Stronger score 8. Evidence: clear
+zero-shot transfer story and principled decoupling.
+
+## 5. Dimension 4: Cheaper (Data or Solution Cost)
+
+### Definition
+
+Reduces data annotation cost, training cost, or deployment cost while
+preserving effectiveness.
+
+### Canonical entry strategies
+
+- **LLM-based data synthesis**. Use a strong foundation model to
+  synthesise labelled training data or simulate agent trajectories,
+  reducing human annotation demand.
+- **Active learning with human in the loop**. Select the smallest set
+  of samples for human review that maximises model improvement,
+  rather than annotating the whole corpus.
+- **Knowledge distillation**. Distill a large model's reasoning paths
+  into a small deployable model, preserving accuracy at lower
+  inference cost.
+
+### How to score
+
+- 9-10: the idea produces a high-quality dataset at a fraction of
+  naive-annotation cost, or deploys at an order-of-magnitude lower
+  cost than the baseline.
+- 6-8: the idea reduces cost by a factor of 2-5 through a known
+  mechanism.
+- 3-5: cost reduction is mentioned but not quantified.
+- 1-2: the idea does not consider cost.
+
+### Grounded example
+
+StatQA's reverse-synthesis pipeline (NeurIPS 2024 D&B) reduced
+annotation cost by generating questions from verified answers,
+side-stepping manual question writing. Cheaper score 8.
+
+## 6. Dimension 5: Broader (Cross-Domain and Unification)
+
+### Definition
+
+Transplants a mature idea from one domain to another, or unifies a
+fragmented set of tasks under a single framework.
+
+### Canonical entry strategies
+
+- **Cross-domain transplantation**. Take a mature technique from one
+  field and apply it to another. Example: database query optimisers
+  bring cost estimation and plan caching to LLM-agent planning.
+- **Generalisation and unification**. Unify a family of tasks (Text
+  to SQL, Text to Python, Text to Chart) under a single data-agent
+  framework with one shared underlying structure.
+
+### How to score
+
+- 9-10: transplants a mature technique in a non-obvious direction with
+  a plausible mechanism; or unifies three or more previously
+  fragmented task families.
+- 6-8: transplants a known idea; or unifies two task families.
+- 3-5: suggests a connection to another domain without concrete
+  mechanism.
+- 1-2: the idea is siloed.
+
+### Grounded example
+
+AFlow (ICLR 2025): transplanted search-based workflow generation from
+neural-architecture-search-style methods to agent pipeline design.
+Broader score 8. Evidence: crossed a domain boundary with a principled
+mechanism.
+
+## 7. Scoring rubric
+
+For each dimension, assign an integer 1-10 based on the tiers above.
+Then aggregate as follows:
+
+- Top dimension at 8+ and a second dimension at 6+: this is the
+  paper's thesis. Emphasise these two in the Introduction.
+- Three or more dimensions at 5 or below: the idea is thin. Either
+  find the dimension where the idea really shines and sharpen it, or
+  pivot.
+- All five dimensions at 4 or below: Reject and Pivot.
+- No single dimension reaches 7: the idea is vague. Ask the user to
+  name the single dominant axis before re-evaluating.
+
+## 8. Summary table
+
+| Dimension | Core goal | Entry strategies | Illustrative example |
+|---|---|---|---|
+| Higher | Accuracy and effectiveness gains | Information or modality augmentation; feedback-driven refinement; error-driven root-cause analysis | MCTS inference in Alpha-SQL |
+| Faster | Efficiency and cost reduction | Caching and experience reuse; parallelisation and decoupling; early exit and dynamic routing | Zero-inference data selection in LEAD |
+| Stronger | Robustness and generalisation | Noise and fault tolerance; exception recovery; decoupled representations | Intent-clarification agents |
+| Cheaper | Data or solution cost | LLM-based synthesis; active learning; knowledge distillation | Reverse-synthesis in StatQA |
+| Broader | Cross-domain and unification | Cross-domain transplantation; abstraction and unification | Workflow search in AFlow |
+
+## 9. Common scoring failures
+
+- **Inflation**: scoring every dimension 7 or 8 "to be generous"
+  destroys the signal. Use 5 as the default and justify upward.
+- **Deflation**: scoring every dimension 3 or 4 signals that you did
+  not understand the idea; re-read.
+- **Uncited evidence**: a score without a specific sentence from the
+  user's idea pointing at why is not a real score.
+- **Generic lift suggestions**: "try a different prompt" is not a
+  lift. Name the entry strategy from this file.
+- **Score and verdict mismatch**: a Strong Accept verdict with no
+  dimension above 7 is incoherent. Realign.

package/dist/skills/idea-evaluator/references/lifecycle-capability-matching.md

@@ -0,0 +1,177 @@
+# Idea lifecycle and student capability matching
+
+## Table of contents
+
+1. Why lifecycle matters
+2. The six-category lifecycle matrix
+3. Capability self-assessment rubric
+4. Matching logic
+5. Mismatch detection and recovery
+6. Worked examples
+
+## 1. Why lifecycle matters
+
+Every research idea has an expiry date. In fast-moving subfields like
+LLM agents or Text-to-SQL, a purely applied idea may have a 3-to-6-
+month lifecycle: if the student cannot finish coding, experiments,
+writing, and submission in that window, another group likely will,
+and the paper becomes hard to publish. In slower-moving subfields
+like foundational theory, the lifecycle stretches past 12 months, but
+the methodological depth required also stretches.
+
+Lifecycle is not the same as difficulty. A short-lifecycle idea can be
+executed by a skilled coder in 3 months; a long-lifecycle idea may
+require 18 months of theoretical work before the first paper. The
+mismatch question is always: can this student finish this idea within
+its lifecycle?
+
+The canonical rule: speed is not an excuse for low
+quality. Fast iteration and submission means efficient work with full
+quality control, not corner-cutting.
+
+## 2. The six-category lifecycle matrix
+
+| Category | Lifecycle | Suited student profile | Rationale |
+|---|---|---|---|
+| Application research | Short (3-6 months) | Strong coder, fast executor, can ship experiments quickly | Competitive, quickly obsolete fields; time-to-submission is decisive |
+| Foundational theory | Long (6-12 months) | Strong mathematical base, deep thinker | Proofs and model building require sustained work |
+| Cross-disciplinary | Medium (6-9 months) | Student with prior non-CS background | Domain expertise plus CS skills enables novel connections; often HCI venues |
+| Frontier exploration | Short-to-medium (3-9 months) | Both theory and experiment capable, self-directed | New subfields require quick experiments and deep analysis |
+| Data-intensive | Medium (6-12 months) | Strong data analysis, solid engineering | Data pipelines and model iteration are the critical path |
+| Innovative technique | Long (12+ months) | Deep base, willing to challenge existing methods | Requires cross-field innovation; longer cycles are accepted |
+
+## 3. Capability self-assessment rubric
+
+Ask the user (or infer from their input) the following four inputs.
+Record each on the scale shown.
+
+### 3.1 Weekly effective hours
+
+Effective hours mean focused, deep-work hours without meetings or
+context switches. A rough calibration:
+
+- Under 10 hours per week: top-venue publication is unlikely within a
+  typical idea lifecycle unless execution is exceptional.
+- 10-25 hours per week: feasible for medium-lifecycle ideas with
+  disciplined scope management.
+- 25-40 hours per week: feasible for short-lifecycle fast-moving
+  ideas.
+- 40+ hours per week: feasible for all lifecycles, but still bounded
+  by the skill-depth axis below.
+
+### 3.2 Skill depth
+
+Rate strongest relevant skill on a 1-5 scale:
+
+- Coding and experiment engineering (Python, PyTorch, deployment).
+- Theoretical and mathematical maturity (proofs, derivations, model
+  formulation).
+- Data engineering (pipelines, annotation tools, cleaning at scale).
+- Systems programming (low-level optimisation, concurrency).
+- Domain knowledge (the target application area).
+
+A student with 5 in coding and 2 in theory should prefer Application
+or Data-intensive categories. A student with 2 in coding and 5 in
+theory should prefer Foundational theory or Innovative technique.
+
+### 3.3 Theoretical versus applied preference
+
+On a 1-5 scale from pure-applied (1) to pure-theoretical (5). This is
+a preference signal, not an ability signal: a student may be capable
+of theory but happier shipping systems. Respect the preference; it
+drives sustainability over the lifecycle.
+
+### 3.4 Infrastructure access
+
+Binary flags:
+
+- Access to at least four large GPUs (A100 or equivalent): yes or no.
+- Access to proprietary data or annotation budget: yes or no.
+- Advisor or team weekly review cadence: yes or no.
+
+Infrastructure access shortens lifecycle fit for certain categories
+(large-GPU access is often required for data-intensive and
+application research at the frontier).
+
+## 4. Matching logic
+
+Given a proposed idea and a capability profile:
+
+1. Classify the idea into one of the six categories.
+2. Look up the lifecycle range for that category.
+3. Compute realistic execution time from capability:
+   - coding + debugging time (estimate from scope)
+   - experiment time (estimate from data and compute availability)
+   - writing time (default: 3-6 weeks for a first draft)
+   - revision time (default: 2-4 weeks)
+4. Compare realistic execution time against lifecycle. If realistic
+   time exceeds lifecycle, flag a mismatch.
+
+## 5. Mismatch detection and recovery
+
+### Detection flags
+
+- **Lifecycle-short mismatch**: realistic execution exceeds 1.3x the
+  lifecycle midpoint. High risk.
+- **Capability-depth mismatch**: the category demands a skill at level
+  4 or 5 that the student rates at 2 or below. Medium-to-high risk.
+- **Infrastructure mismatch**: the category requires GPU or data
+  access the student lacks. High risk.
+- **Preference mismatch**: the category conflicts with the student's
+  theoretical-applied preference. Medium risk; watch for drop-off.
+
+### Recovery strategies
+
+- **Narrow the scope**. Cut the idea to a well-scoped subproblem that
+  fits the capability. Example: instead of a full Text-to-SQL Agent,
+  tackle only schema-linking under ambiguity.
+- **Partner with a collaborator**. Pair the student with someone whose
+  skills complement the missing depth.
+- **Change category**. Reframe the idea to fit a more compatible
+  category. Example: reframe a foundational-theory idea as a
+  data-intensive empirical study.
+- **Extend the lifecycle**. Only feasible in slow-moving subfields;
+  dangerous in fast-moving ones.
+- **Switch the idea entirely**. If three or more mismatch flags
+  exist, the idea is a poor fit. Better to pivot than to push.
+
+## 6. Worked examples
+
+### Example A: fit
+
+- Idea: incremental improvement of Text-to-SQL accuracy via
+  MCTS-based inference-time search.
+- Category: Application research (short lifecycle, 3-6 months).
+- Student capability: 5 in coding, 3 in theory, 20 weekly hours, four
+  RTX 3090 GPUs available.
+- Matching: coding (5) meets the demand; compute (four GPUs) is
+  sufficient for BIRD-scale benchmarks; 20 weekly hours is tight but
+  feasible with focused scope.
+- Verdict: Green. Recommend a single-metric success criterion and a
+  4-month timeline.
+
+### Example B: mismatch
+
+- Idea: a new theoretical framework for query-planner convergence
+  under distribution shift.
+- Category: Foundational theory (long lifecycle, 6-12 months).
+- Student capability: 5 in coding, 2 in theory, 15 weekly hours,
+  strong applied preference.
+- Matching: theory (2) is below demand; preference is applied, not
+  theoretical. Two mismatch flags.
+- Verdict: Yellow. Reframe as an empirical study of planner behaviour
+  under distribution shift, moving to Data-intensive category.
+  Alternatively, partner with a theory-strong collaborator.
+
+### Example C: high-risk fit
+
+- Idea: a cross-disciplinary HCI study of how domain experts interact
+  with AI-generated charts.
+- Category: Cross-disciplinary (medium, 6-9 months).
+- Student capability: background in statistics, 3 in coding,
+  25 weekly hours, no IRB approval yet, no participant pool.
+- Matching: coding (3) is adequate; infrastructure (no IRB, no pool)
+  is blocking. One high-severity infrastructure mismatch.
+- Verdict: Yellow. Secure IRB and recruit participants before writing
+  any code; otherwise the 9-month window will be consumed by
+  logistics.