@bohuyeshan/openagent-labforge-core 3.11.2 → 3.11.3

package/generated/skills-bundles/full/skills/research/research-ideation/auto-claude__idea-creator/SKILL.md

@@ -0,0 +1,228 @@
+---
+name: "auto-claude/idea-creator"
+description: "Generate and rank research ideas given a broad direction. Use when user says \"找idea\", \"brainstorm ideas\", \"generate research ideas\", \"what can we work on\", or wants to explore a research area for publishable directions."
+argument-hint: ["research-direction"]
+allowed-tools: "Bash(*), Read, Write, Grep, Glob, WebSearch, WebFetch, Agent, mcp__codex__codex, mcp__codex__codex-reply"
+metadata:
+  category: "research/research-ideation"
+---
+
+# Research Idea Creator
+
+Generate publishable research ideas for: $ARGUMENTS
+
+## Overview
+
+Given a broad research direction from the user, systematically generate, validate, and rank concrete research ideas. This skill composes with `/research-lit` and `/novelty-check` to form a complete idea discovery pipeline.
+
+## Constants
+
+- **PILOT_MAX_HOURS = 2** — Skip any pilot estimated to take > 2 hours per GPU. Flag as "needs manual pilot".
+- **PILOT_TIMEOUT_HOURS = 3** — Hard timeout: kill pilots exceeding 3 hours. Collect partial results if available.
+- **MAX_PILOT_IDEAS = 3** — Pilot at most 3 ideas in parallel. Additional ideas are validated on paper only.
+- **MAX_TOTAL_GPU_HOURS = 8** — Total GPU budget for all pilots combined.
+- **REVIEWER_MODEL = `gpt-5.4`** — Model used via Codex MCP for brainstorming and review. Must be an OpenAI model (e.g., `gpt-5.4`, `o3`, `gpt-4o`).
+
+> 💡 Override via argument, e.g., `/idea-creator "topic" — pilot budget: 4h per idea, 20h total`.
+
+## Workflow
+
+### Phase 1: Landscape Survey (5-10 min)
+
+Map the research area to understand what exists and where the gaps are.
+
+1. **Scan local paper library first**: Check `papers/` and `literature/` in the project directory for existing PDFs. Read first 3 pages of relevant papers to build a baseline understanding before searching online. This avoids re-discovering what the user already knows.
+
+2. **Search recent literature** using WebSearch:
+   - Top venues in the last 2 years (NeurIPS, ICML, ICLR, ACL, EMNLP, etc.)
+   - Recent arXiv preprints (last 6 months)
+   - Use 5+ different query formulations
+   - Read abstracts and introductions of the top 10-15 papers
+
+2. **Build a landscape map**:
+   - Group papers by sub-direction / approach
+   - Identify what has been tried and what hasn't
+   - Note recurring limitations mentioned in "Future Work" sections
+   - Flag any open problems explicitly stated by multiple papers
+
+3. **Identify structural gaps**:
+   - Methods that work in domain A but haven't been tried in domain B
+   - Contradictory findings between papers (opportunity for resolution)
+   - Assumptions that everyone makes but nobody has tested
+   - Scaling regimes that haven't been explored
+   - Diagnostic questions that nobody has asked
+
+### Phase 2: Idea Generation (brainstorm with external LLM)
+
+Use the external LLM via Codex MCP for divergent thinking:
+
+```
+mcp__codex__codex:
+  model: REVIEWER_MODEL
+  config: {"model_reasoning_effort": "xhigh"}
+  prompt: |
+    You are a senior ML researcher brainstorming research ideas.
+
+    Research direction: [user's direction]
+
+    Here is the current landscape:
+    [paste landscape map from Phase 1]
+
+    Key gaps identified:
+    [paste gaps from Phase 1]
+
+    Generate 8-12 concrete research ideas. For each idea:
+    1. One-sentence summary
+    2. Core hypothesis (what you expect to find and why)
+    3. Minimum viable experiment (what's the cheapest way to test this?)
+    4. Expected contribution type: empirical finding / new method / theoretical result / diagnostic
+    5. Risk level: LOW (likely works) / MEDIUM (50-50) / HIGH (speculative)
+    6. Estimated effort: days / weeks / months
+
+    Prioritize ideas that are:
+    - Testable with moderate compute (8x RTX 3090 or less)
+    - Likely to produce a clear positive OR negative result (both are publishable)
+    - Not "apply X to Y" unless the application reveals genuinely surprising insights
+    - Differentiated from the 10-15 papers above
+
+    Be creative but grounded. A great idea is one where the answer matters regardless of which way it goes.
+```
+
+Save the threadId for follow-up.
+
+### Phase 3: First-Pass Filtering
+
+For each generated idea, quickly evaluate:
+
+1. **Feasibility check**: Can we actually run this experiment with available resources?
+   - Compute requirements (estimate GPU-hours)
+   - Data availability
+   - Implementation complexity
+   - Skip ideas requiring > 1 week of GPU time or unavailable datasets
+
+2. **Novelty quick-check**: For each idea, do 2-3 targeted searches to see if it's already been done. Full `/novelty-check` comes later for survivors.
+
+3. **Impact estimation**: Would a reviewer care about the result?
+   - "So what?" test: if the experiment succeeds, does it change how people think?
+   - Is the finding actionable or just interesting?
+
+Eliminate ideas that fail any of these. Typically 8-12 ideas reduce to 4-6.
+
+### Phase 4: Deep Validation (for top ideas)
+
+For each surviving idea, run a deeper evaluation:
+
+1. **Novelty check**: Use the `/novelty-check` workflow (multi-source search + GPT-5.4 cross-verification) for each idea
+
+2. **Critical review**: Perform a rigorous internal devil's-advocate pass:
+   - What's the strongest objection a reviewer would raise?
+   - What's the most likely failure mode?
+   - How would you rank these for a top venue submission?
+   - Which 2-3 would you actually work on?
+
+3. **Combine rankings**: Merge your assessment with GPT-5.4's ranking. Select top 2-3 ideas for pilot experiments.
+
+### Phase 5: Parallel Pilot Experiments (for top 2-3 ideas)
+
+Before committing to a full research effort, run cheap pilot experiments to get empirical signal. This is the key differentiator from paper-only validation.
+
+1. **Design pilots**: For each top idea, define the minimal experiment that would give a positive or negative signal:
+   - Single seed, small scale (e.g., small dataset subset, fewer epochs)
+   - Target: 30 min - PILOT_MAX_HOURS per pilot on 1 GPU
+   - **Estimate GPU-hours BEFORE launching.** If estimated time > PILOT_MAX_HOURS, reduce scale (fewer epochs, smaller subset) or flag as "needs manual pilot"
+   - Clear success metric defined upfront (e.g., "if metric improves by > 1%, signal is positive")
+
+2. **Deploy in parallel**: Use `/run-experiment` to launch pilots on different GPUs simultaneously:
+   ```
+   GPU 0: Pilot for Idea 1
+   GPU 1: Pilot for Idea 2
+   GPU 2: Pilot for Idea 3
+   ```
+   Use `run_in_background: true` to launch all at once.
+
+3. **Collect results**: Use `/monitor-experiment` to check progress. If any pilot exceeds PILOT_TIMEOUT_HOURS, kill it and collect partial results. Once all pilots complete (or timeout), compare:
+   - Which ideas showed positive signal?
+   - Which showed null/negative results? (eliminate or deprioritize)
+   - Any surprising findings that suggest a pivot?
+   - Total GPU-hours consumed (track against MAX_TOTAL_GPU_HOURS budget)
+
+4. **Re-rank based on empirical evidence**: Update the idea ranking using pilot results. An idea with strong pilot signal jumps ahead of a theoretically appealing but untested idea.
+
+Note: Skip this phase if the ideas are purely theoretical or if no GPU is available. Flag skipped ideas as "needs pilot validation" in the report.
+
+### Phase 6: Output — Ranked Idea Report
+
+Write a structured report to `IDEA_REPORT.md` in the project root:
+
+```markdown
+# Research Idea Report
+
+**Direction**: [user's research direction]
+**Generated**: [date]
+**Ideas evaluated**: X generated → Y survived filtering → Z piloted → W recommended
+
+## Landscape Summary
+[3-5 paragraphs on the current state of the field]
+
+## Recommended Ideas (ranked)
+
+### Idea 1: [title]
+- **Hypothesis**: [one sentence]
+- **Minimum experiment**: [concrete description]
+- **Expected outcome**: [what success/failure looks like]
+- **Novelty**: X/10 — closest work: [paper]
+- **Feasibility**: [compute, data, implementation estimates]
+- **Risk**: LOW/MEDIUM/HIGH
+- **Contribution type**: empirical / method / theory / diagnostic
+- **Pilot result**: [POSITIVE: metric +X% / NEGATIVE: no signal / SKIPPED: needs GPU]
+- **Reviewer's likely objection**: [strongest counterargument]
+- **Why we should do this**: [1-2 sentences]
+
+### Idea 2: [title]
+...
+
+## Eliminated Ideas (for reference)
+| Idea | Reason eliminated |
+|------|-------------------|
+| ... | Already done by [paper] |
+| ... | Requires > 1 week GPU time |
+| ... | Result wouldn't be interesting either way |
+
+## Pilot Experiment Results
+| Idea | GPU | Time | Key Metric | Signal |
+|------|-----|------|------------|--------|
+| Idea 1 | GPU 0 | 45 min | +2.3% CE | POSITIVE |
+| Idea 2 | GPU 1 | 30 min | -0.1% CE | NEGATIVE |
+| Idea 3 | GPU 2 | 1.5 hr | +0.8% CE | WEAK POSITIVE |
+
+## Suggested Execution Order
+1. Start with Idea 1 (positive pilot signal, lowest risk)
+2. Idea 3 as backup (weak signal, may need larger scale to confirm)
+3. Idea 2 eliminated by pilot — negative result documented
+
+## Next Steps
+- [ ] Scale up Idea 1 to full experiment (multi-seed, full dataset)
+- [ ] If confirmed, invoke /ulw-loop for full iteration
+```
+
+## Key Rules
+
+- The user provides a DIRECTION, not an idea. Your job is to generate the ideas.
+- Quantity first, quality second: brainstorm broadly, then filter ruthlessly.
+- A good negative result is just as publishable as a positive one. Prioritize ideas where the answer matters regardless of direction.
+- Don't fall in love with any idea before validating it. Be willing to kill ideas.
+- Always estimate compute cost. An idea that needs 1000 GPU-hours is not actionable for most researchers.
+- "Apply X to Y" is the lowest form of research idea. Push for deeper questions.
+- Include eliminated ideas in the report — they save future time by documenting dead ends.
+- **If the user's direction is too broad (e.g., "NLP", "computer vision", "reinforcement learning"), STOP and ask them to narrow it.** A good direction is 1-2 sentences specifying the problem, domain, and constraint — e.g., "factorized gap in discrete diffusion LMs" or "sample efficiency of offline RL with image observations". Without sufficient specificity, generated ideas will be too vague to run experiments on.
+
+## Composing with Other Skills
+
+After this skill produces the ranked report:
+```
+/idea-creator "direction"     → ranked ideas
+/novelty-check "top idea"     → deep novelty verification (already done in Phase 4, but user can re-run)
+implement                     → write code
+/run-experiment               → deploy to GPU
+/ulw-loop             → iterate until submission-ready
+```

package/generated/skills-bundles/full/skills/research/research-ideation/auto-claude__idea-discovery/SKILL.md

@@ -0,0 +1,186 @@
+---
+name: "auto-claude/idea-discovery"
+description: "Workflow 1: Full idea discovery pipeline. Orchestrates research-lit → idea-creator → novelty-check to go from a broad research direction to validated, pilot-tested ideas. Use when user says \"找idea全流程\", \"idea discovery pipeline\", \"从零开始找方向\", or wants the complete idea exploration workflow."
+argument-hint: ["research-direction"]
+allowed-tools: "Bash(*), Read, Write, Edit, Grep, Glob, WebSearch, WebFetch, Agent, Skill, mcp__codex__codex, mcp__codex__codex-reply"
+metadata:
+  category: "research/research-ideation"
+---
+
+# Workflow 1: Idea Discovery Pipeline
+
+Orchestrate a complete idea discovery workflow for: **$ARGUMENTS**
+
+## Overview
+
+This skill chains three sub-skills into a single automated pipeline:
+
+```
+/research-lit → /idea-creator → /novelty-check
+  (survey)      (brainstorm)    (verify novel)
+```
+
+Each phase builds on the previous one's output. The final deliverable is a validated `IDEA_REPORT.md` with ranked ideas, pilot results, and a suggested execution plan.
+
+## Constants
+
+- **PILOT_MAX_HOURS = 2** — Skip any pilot experiment estimated to take > 2 hours per GPU. Flag as "needs manual pilot" in the report.
+- **PILOT_TIMEOUT_HOURS = 3** — Hard timeout: kill any running pilot that exceeds 3 hours. Collect partial results if available.
+- **MAX_PILOT_IDEAS = 3** — Run pilots for at most 3 top ideas in parallel. Additional ideas are validated on paper only.
+- **MAX_TOTAL_GPU_HOURS = 8** — Total GPU budget across all pilots. If exceeded, skip remaining pilots and note in report.
+- **AUTO_PROCEED = true** — If user doesn't respond at a checkpoint, automatically proceed with the best option after presenting results. Set to `false` to always wait for explicit user confirmation.
+- **REVIEWER_MODEL = `gpt-5.4`** — Model used via Codex MCP. Must be an OpenAI model (e.g., `gpt-5.4`, `o3`, `gpt-4o`). Passed to sub-skills.
+- **ARXIV_DOWNLOAD = false** — When `true`, `/research-lit` downloads the top relevant arXiv PDFs during Phase 1. When `false` (default), only fetches metadata. Passed through to `/research-lit`.
+
+> 💡 These are defaults. Override by telling the skill, e.g., `/idea-discovery "topic" — pilot budget: 4h per idea, 20h total` or `/idea-discovery "topic" — arxiv download: true`.
+
+## Pipeline
+
+### Phase 1: Literature Survey
+
+Invoke `/research-lit` to map the research landscape:
+
+```
+/research-lit "$ARGUMENTS"
+```
+
+**What this does:**
+- Search arXiv, Google Scholar, Semantic Scholar for recent papers
+- Build a landscape map: sub-directions, approaches, open problems
+- Identify structural gaps and recurring limitations
+- Output a literature summary (saved to working notes)
+
+**🚦 Checkpoint:** Present the landscape summary to the user. Ask:
+
+```
+📚 Literature survey complete. Here's what I found:
+- [key findings, gaps, open problems]
+
+Does this match your understanding? Should I adjust the scope before generating ideas?
+(If no response, I'll proceed with the top-ranked direction.)
+```
+
+- **User approves** (or no response + AUTO_PROCEED=true) → proceed to Phase 2 with best direction.
+- **User requests changes** (e.g., "focus more on X", "ignore Y", "too broad") → refine the search with updated queries, re-run `/research-lit` with adjusted scope, and present again. Repeat until the user is satisfied.
+
+### Phase 2: Idea Generation + Filtering + Pilots
+
+Invoke `/idea-creator` with the landscape context:
+
+```
+/idea-creator "$ARGUMENTS"
+```
+
+**What this does:**
+- Brainstorm 8-12 concrete ideas via GPT-5.4 xhigh
+- Filter by feasibility, compute cost, quick novelty search
+- Deep validate top ideas (full novelty check + devil's advocate)
+- Run parallel pilot experiments on available GPUs (top 2-3 ideas)
+- Rank by empirical signal
+- Output `IDEA_REPORT.md`
+
+**🚦 Checkpoint:** Present `IDEA_REPORT.md` ranked ideas to the user. Ask:
+
+```
+💡 Generated X ideas, filtered to Y, piloted Z. Top results:
+
+1. [Idea 1] — Pilot: POSITIVE (+X%)
+2. [Idea 2] — Pilot: WEAK POSITIVE (+Y%)
+3. [Idea 3] — Pilot: NEGATIVE, eliminated
+
+Which ideas should I validate further? Or should I regenerate with different constraints?
+(If no response, I'll proceed with the top-ranked ideas.)
+```
+
+- **User picks ideas** (or no response + AUTO_PROCEED=true) → proceed to Phase 3 with top-ranked ideas.
+- **User unhappy with all ideas** → collect feedback ("what's missing?", "what direction do you prefer?"), update the prompt with user's constraints, and re-run Phase 2 (idea generation). Repeat until the user selects at least 1 idea.
+- **User wants to adjust scope** → go back to Phase 1 with refined direction.
+
+### Phase 3: Deep Novelty Verification
+
+For each top idea (positive pilot signal), run a thorough novelty check:
+
+```
+/novelty-check "[top idea 1 description]"
+/novelty-check "[top idea 2 description]"
+```
+
+**What this does:**
+- Multi-source literature search (arXiv, Scholar, Semantic Scholar)
+- Cross-verify with GPT-5.4 xhigh
+- Check for concurrent work (last 3-6 months)
+- Identify closest existing work and differentiation points
+
+**Update `IDEA_REPORT.md`** with deep novelty results. Eliminate any idea that turns out to be already published.
+
+### Phase 4: Internal Critical Review
+
+For the surviving top idea(s), perform a rigorous internal critique:
+
+- Identify the strongest reviewer objections
+- List likely failure modes and missing controls
+- Propose minimum viable improvements to de-risk the idea
+
+**Update `IDEA_REPORT.md`** with internal review notes and a revised plan.
+
+### Phase 5: Final Report
+
+Finalize `IDEA_REPORT.md` with all accumulated information:
+
+```markdown
+# Idea Discovery Report
+
+**Direction**: $ARGUMENTS
+**Date**: [today]
+**Pipeline**: research-lit → idea-creator → novelty-check → internal review
+
+## Executive Summary
+[2-3 sentences: best idea, key evidence, recommended next step]
+
+## Literature Landscape
+[from Phase 1]
+
+## Ranked Ideas
+[from Phase 2, updated with Phase 3-4 results]
+
+### 🏆 Idea 1: [title] — RECOMMENDED
+- Pilot: POSITIVE (+X%)
+- Novelty: CONFIRMED (closest: [paper], differentiation: [what's different])
+- Internal review notes: [top 2-3 risks + fixes]
+- Next step: implement full experiment → /ulw-loop
+
+### Idea 2: [title] — BACKUP
+...
+
+## Eliminated Ideas
+[ideas killed at each phase, with reasons]
+
+## Next Steps
+- [ ] Implement Idea 1
+- [ ] /run-experiment to deploy full-scale experiments
+- [ ] /ulw-loop to iterate until submission-ready
+- [ ] Or invoke /research-pipeline for the complete end-to-end flow
+```
+
+## Key Rules
+
+- **Don't skip phases.** Each phase filters and validates — skipping leads to wasted effort later.
+- **Checkpoint between phases.** Briefly summarize what was found before moving on.
+- **Kill ideas early.** It's better to kill 10 bad ideas in Phase 3 than to implement one and fail.
+- **Empirical signal > theoretical appeal.** An idea with a positive pilot outranks a "sounds great" idea without evidence.
+- **Document everything.** Dead ends are just as valuable as successes for future reference.
+- **Be honest in the critique.** Include negative results and failed pilots in the review notes.
+- **Feishu notifications are optional.** If `~/.claude/feishu.json` exists, send `checkpoint` at each phase transition and `pipeline_done` at final report. If absent/off, skip silently.
+
+## Composing with Workflow 2
+
+After this pipeline produces a validated top idea:
+
+```
+/idea-discovery "direction"         ← you are here (Workflow 1)
+implement                           ← write code for the top idea
+/run-experiment                     ← deploy full-scale experiments
+/ulw-loop "top idea"        ← Workflow 2: iterate until submission-ready
+
+Or use /research-pipeline for the full end-to-end flow.
+```