@bohuyeshan/openagent-labforge-core 3.11.2 → 3.11.3

package/generated/skills-bundles/full/skills/data-analysis/experiment-ops/auto-claude__run-experiment/SKILL.md

@@ -0,0 +1,112 @@
+---
+name: "auto-claude/run-experiment"
+description: "Deploy and run ML experiments on local or remote GPU servers. Use when user says \"run experiment\", \"deploy to server\", \"跑实验\", or needs to launch training jobs."
+argument-hint: ["experiment-description"]
+allowed-tools: "Bash(*), Read, Grep, Glob, Edit, Write, Agent"
+metadata:
+  category: "data-analysis/experiment-ops"
+---
+# Run Experiment
+
+Deploy and run ML experiment: $ARGUMENTS
+
+## Workflow
+
+### Step 1: Detect Environment
+
+Read the project's `CLAUDE.md` to determine the experiment environment:
+
+- **Local GPU**: Look for local CUDA/MPS setup info
+- **Remote server**: Look for SSH alias, conda env, code directory
+
+If no server info is found in `CLAUDE.md`, ask the user.
+
+### Step 2: Pre-flight Check
+
+Check GPU availability on the target machine:
+
+**Remote:**
+```bash
+ssh <server> nvidia-smi --query-gpu=index,memory.used,memory.total --format=csv,noheader
+```
+
+**Local:**
+```bash
+nvidia-smi --query-gpu=index,memory.used,memory.total --format=csv,noheader
+# or for Mac MPS:
+python -c "import torch; print('MPS available:', torch.backends.mps.is_available())"
+```
+
+Free GPU = memory.used < 500 MiB.
+
+### Step 3: Sync Code (Remote Only)
+
+Only sync necessary files — NOT data, checkpoints, or large files:
+```bash
+rsync -avz --include='*.py' --exclude='*' <local_src>/ <server>:<remote_dst>/
+```
+
+### Step 4: Deploy
+
+#### Remote (via SSH + screen)
+
+For each experiment, create a dedicated screen session with GPU binding:
+```bash
+ssh <server> "screen -dmS <exp_name> bash -c '\
+  eval \"\$(<conda_path>/conda shell.bash hook)\" && \
+  conda activate <env> && \
+  CUDA_VISIBLE_DEVICES=<gpu_id> python <script> <args> 2>&1 | tee <log_file>'"
+```
+
+#### Local
+
+```bash
+# Linux with CUDA
+CUDA_VISIBLE_DEVICES=<gpu_id> python <script> <args> 2>&1 | tee <log_file>
+
+# Mac with MPS (PyTorch uses MPS automatically)
+python <script> <args> 2>&1 | tee <log_file>
+```
+
+For local long-running jobs, use `run_in_background: true` to keep the conversation responsive.
+
+### Step 5: Verify Launch
+
+**Remote:**
+```bash
+ssh <server> "screen -ls"
+```
+
+**Local:**
+Check process is running and GPU is allocated.
+
+### Step 6: Feishu Notification (if configured)
+
+After deployment is verified, check `~/.claude/feishu.json`:
+- Send `experiment_done` notification: which experiments launched, which GPUs, estimated time
+- If config absent or mode `"off"`: skip entirely (no-op)
+
+## Key Rules
+
+- ALWAYS check GPU availability first — never blindly assign GPUs
+- Each experiment gets its own screen session + GPU (remote) or background process (local)
+- Use `tee` to save logs for later inspection
+- Run deployment commands with `run_in_background: true` to keep conversation responsive
+- Report back: which GPU, which screen/process, what command, estimated time
+- If multiple experiments, launch them in parallel on different GPUs
+
+## CLAUDE.md Example
+
+Users should add their server info to their project's `CLAUDE.md`:
+
+```markdown
+## Remote Server
+- SSH: `ssh my-gpu-server`
+- GPU: 4x A100 (80GB each)
+- Conda: `eval "$(/opt/conda/bin/conda shell.bash hook)" && conda activate research`
+- Code dir: `/home/user/experiments/`
+
+## Local Environment
+- Mac MPS / Linux CUDA
+- Conda env: `ml` (Python 3.10 + PyTorch)
+```

package/generated/skills-bundles/full/skills/data-analysis/optimization/auto-claude__dse-loop/SKILL.md

@@ -0,0 +1,279 @@
+---
+name: "auto-claude/dse-loop"
+description: "Autonomous design space exploration loop for computer architecture and EDA. Runs a program, analyzes results, tunes parameters, and iterates until objective is met or timeout. Use when user says \"DSE\", \"design space exploration\", \"sweep parameters\", \"optimize\", \"find best config\", or wants iterative parameter tuning."
+argument-hint: ["task-description — include program","parameters","objective","and timeout"]
+allowed-tools: "Bash(*), Read, Grep, Glob, Write, Edit, Agent"
+metadata:
+  category: "data-analysis/optimization"
+---
+# DSE Loop: Autonomous Design Space Exploration
+
+Autonomously explore a design space: run → analyze → pick next parameters → repeat, until the objective is met or timeout is reached. Designed for computer architecture and EDA problems.
+
+## Context: $ARGUMENTS
+
+## Safety Rules — READ FIRST
+
+**NEVER do any of the following:**
+- `sudo` anything
+- `rm -rf`, `rm -r`, or any recursive deletion
+- `rm` any file you did not create in this session
+- Overwrite existing source files without reading them first
+- `git push`, `git reset --hard`, or any destructive git operation
+- Kill processes you did not start
+
+**If a step requires any of the above, STOP and report to the user.**
+
+## Constants (override via $ARGUMENTS)
+
+| Constant | Default | Description |
+|----------|---------|-------------|
+| `TIMEOUT` | 2h | Total wall-clock budget. Stop exploring after this. |
+| `MAX_ITERATIONS` | 50 | Hard cap on number of design points evaluated. |
+| `PATIENCE` | 10 | Stop early if no improvement for this many consecutive iterations. |
+| `OBJECTIVE` | minimize | `minimize` or `maximize` the target metric. |
+
+Override inline: `/dse-loop "task desc — timeout: 4h, max_iterations: 100, patience: 15"`
+
+## Typical Use Cases
+
+| Problem | Program | Parameters | Objective |
+|---------|---------|-----------|-----------|
+| Microarch DSE | gem5 simulation | cache size, assoc, pipeline width, ROB size, branch predictor | maximize IPC or minimize area×delay |
+| Synthesis tuning | yosys/DC script | optimization passes, target freq, effort level | minimize area at timing closure |
+| RTL parameterization | verilator sim | data width, FIFO depth, pipeline stages, buffer sizes | meet throughput target at min area |
+| Compiler flags | gcc/llvm build + benchmark | -O levels, unroll factor, vectorization, scheduling | minimize runtime or code size |
+| Placement/routing | openroad/innovus | utilization, aspect ratio, layer config | minimize wirelength / timing |
+| Formal verification | abc/sby | bound depth, engine, timeout per property | maximize coverage in time budget |
+| Memory subsystem | cacti / ramulator | bank count, row buffer policy, scheduling | optimize bandwidth/energy |
+
+## Workflow
+
+### Phase 0: Parse Task & Setup
+
+1. **Parse $ARGUMENTS** to extract:
+   - **Program**: what to run (command, script, or Makefile target)
+   - **Parameter space**: which knobs to tune and their ranges/options (may be incomplete — see step 2)
+   - **Objective metric**: what to optimize (and how to extract it from output)
+   - **Constraints**: hard limits that must not be violated (e.g., timing must close)
+   - **Timeout**: wall-clock budget
+   - **Success criteria**: when is the result "good enough" to stop early?
+
+2. **Infer missing parameter ranges** — If the user provides parameter names but NOT ranges/options, you MUST infer them before exploring:
+
+   a. **Read the source code** — search for the parameter names in the codebase:
+      - Look for argparse/click definitions, config files, Makefile variables, module parameters, `#define`, `parameter` (SystemVerilog), `localparam`, etc.
+      - Extract defaults, types, and any comments hinting at valid values
+
+   b. **Apply domain knowledge** to set reasonable ranges:
+      | Parameter type | Inference strategy |
+      |---------------|-------------------|
+      | Cache/memory sizes | Powers of 2, typically 1KB–16MB |
+      | Associativity | Powers of 2: 1, 2, 4, 8, 16 |
+      | Pipeline width / issue width | Small integers: 1, 2, 4, 8 |
+      | Buffer/queue/FIFO depth | Powers of 2: 4, 8, 16, 32, 64 |
+      | Clock period / frequency | Based on technology node; try ±50% from default |
+      | Bound depth (BMC/formal) | Geometric: 5, 10, 20, 50, 100 |
+      | Timeout values | Geometric: 10s, 30s, 60s, 120s, 300s |
+      | Boolean/enum flags | Enumerate all options found in source |
+      | Continuous (learning rate, threshold) | Log-scale sweep: 5 points spanning 2 orders of magnitude around default |
+      | Integer counts (threads, cores) | Linear: from 1 to hardware max |
+
+   c. **Start conservative** — begin with 3-5 values per parameter. Expand range later if the best result is at a boundary.
+
+   d. **Log inferred ranges** — write the inferred parameter space to `dse_results/inferred_params.md` so the user can review:
+      ```markdown
+      # Inferred Parameter Space
+
+      | Parameter | Source | Default | Inferred Range | Reasoning |
+      |-----------|--------|---------|---------------|-----------|
+      | CACHE_SIZE | config.py:42 | 32768 | [8192, 16384, 32768, 65536, 131072] | powers of 2, ±2x from default |
+      | ASSOC | config.py:43 | 4 | [1, 2, 4, 8] | standard associativities |
+      | BMC_DEPTH | run_bmc.py:15 | 10 | [5, 10, 20, 50] | geometric, common BMC depths |
+      ```
+
+   e. **Boundary expansion** — during the search, if the best result is at the min or max of a range, automatically extend that range by one step in that direction (but log the extension).
+
+3. **Read the project** to understand:
+   - How to run the program
+   - Where results are produced (stdout, log files, reports)
+   - How to parse the objective metric from output
+   - Current/baseline configuration (if any)
+
+4. **Create working directory**: `dse_results/` in project root
+   - `dse_results/dse_log.csv` — one row per design point
+   - `dse_results/DSE_REPORT.md` — final report
+   - `dse_results/DSE_STATE.json` — state for recovery
+   - `dse_results/inferred_params.md` — inferred parameter space (if ranges were not provided)
+   - `dse_results/configs/` — config files for each run
+   - `dse_results/outputs/` — raw output for each run
+
+5. **Write a parameter extraction script** (`dse_results/parse_result.py` or similar) that takes a run's output and returns the objective metric as a number. Test it on a baseline run first.
+
+6. **Run baseline** (iteration 0): run the program with default/current parameters. Record the baseline metric. This is the point to beat.
+
+### Phase 1: Initial Exploration
+
+**Goal**: Quickly survey the space to understand which parameters matter most.
+
+**Strategy**: Latin Hypercube Sampling or structured sweep of key parameters.
+
+1. Pick 5-10 diverse design points that span the parameter ranges
+2. Run them (in parallel if independent, via background processes or sequential)
+3. Record all results in `dse_log.csv`:
+   ```
+   iteration,param1,param2,...,metric,constraint_met,timestamp,notes
+   0,default,default,...,baseline_val,yes,2026-03-13T10:00:00,baseline
+   1,val1a,val2a,...,result1,yes,2026-03-13T10:05:00,initial sweep
+   ...
+   ```
+4. Analyze: which parameters have the most impact on the objective?
+5. Narrow the search to the most sensitive parameters
+
+### Phase 2: Directed Search
+
+**Goal**: Converge toward the optimum by making informed choices.
+
+**Strategy**: Adaptive — pick the approach that fits the problem:
+
+- **Few parameters (≤3)**: Fine-grained grid search around the best region from Phase 1
+- **Many parameters (>3)**: Coordinate descent — optimize one parameter at a time, holding others at current best
+- **Binary/categorical params**: Enumerate promising combinations
+- **Continuous params**: Binary search or golden section between best neighbors
+- **Multi-objective**: Track Pareto frontier, explore along the front
+
+For each iteration:
+
+1. **Select next design point** based on results so far:
+   - Look at the trend: which direction improves the metric?
+   - Avoid re-running configurations already evaluated
+   - Balance exploration (untested regions) vs exploitation (near current best)
+
+2. **Modify parameters**: edit config file, command-line args, or source constants
+
+3. **Run the program**: execute and capture output
+
+4. **Parse results**: extract the objective metric and check constraints
+
+5. **Log to `dse_log.csv`**: append the new row
+
+6. **Check stopping conditions**:
+   - Timeout reached? → stop
+   - Max iterations reached? → stop
+   - Patience exhausted (no improvement in N iterations)? → stop
+   - Success criteria met (metric is "good enough")? → stop
+   - Constraint violation pattern detected? → adjust search bounds
+
+7. **Update `DSE_STATE.json`**:
+   ```json
+   {
+     "iteration": 15,
+     "status": "in_progress",
+     "best_metric": 1.23,
+     "best_params": {"cache_size": 32768, "assoc": 4, "pipeline_width": 2},
+     "total_iterations": 15,
+     "start_time": "2026-03-13T10:00:00",
+     "timeout": "2h",
+     "patience_counter": 3
+   }
+   ```
+
+8. **Decide next step** → back to step 1
+
+### Phase 3: Refinement (if time allows)
+
+If the search converged and there's still time budget:
+
+1. **Local perturbation**: try ±1 step on each parameter from the best point
+2. **Sensitivity analysis**: which parameters can be relaxed without hurting the metric?
+3. **Constraint boundary**: if a constraint is nearly binding, explore near-feasible points
+
+### Phase 4: Report
+
+Write `dse_results/DSE_REPORT.md`:
+
+```markdown
+# Design Space Exploration Report
+
+**Task**: [description]
+**Date**: [start] → [end]
+**Total iterations**: N
+**Wall-clock time**: X hours Y minutes
+
+## Objective
+- **Metric**: [what was optimized]
+- **Direction**: minimize / maximize
+- **Baseline**: [value]
+- **Best found**: [value] ([improvement]% better than baseline)
+
+## Best Configuration
+| Parameter | Baseline | Best |
+|-----------|----------|------|
+| param1    | default  | best_val |
+| param2    | default  | best_val |
+| ...       | ...      | ... |
+
+## Search Trajectory
+| Iteration | param1 | param2 | ... | Metric | Notes |
+|-----------|--------|--------|-----|--------|-------|
+| 0 (baseline) | ... | ... | ... | ... | baseline |
+| 1 | ... | ... | ... | ... | initial sweep |
+| ... | ... | ... | ... | ... | ... |
+| N (best) | ... | ... | ... | ... | ★ best |
+
+## Parameter Sensitivity
+- **param1**: [high/medium/low impact] — [brief explanation]
+- **param2**: [high/medium/low impact] — [brief explanation]
+
+## Pareto Frontier (if multi-objective)
+[Table or description of non-dominated points]
+
+## Stopping Reason
+[timeout / max_iterations / patience / success_criteria_met]
+
+## Recommendations
+- [actionable insights from the exploration]
+- [which parameters matter most]
+- [suggested follow-up explorations]
+```
+
+Also generate a summary plot if matplotlib is available:
+- Convergence curve (metric vs iteration)
+- Parameter sensitivity bar chart
+- Pareto frontier scatter (if multi-objective)
+
+## State Recovery
+
+If the context window compacts mid-run, the loop recovers from `DSE_STATE.json` + `dse_log.csv`:
+
+1. Read `DSE_STATE.json` for current iteration, best params, patience counter
+2. Read `dse_log.csv` for full history
+3. Resume from next iteration
+
+## Key Rules
+
+- Work AUTONOMOUSLY — do not ask the user for permission at each iteration
+- **Every run must be logged** — even failed runs, constraint violations, errors. The log is the ground truth.
+- **Never re-run an identical configuration** — check `dse_log.csv` before each run
+- **Respect the timeout** — check elapsed time before starting a new iteration. If the next run is likely to exceed the timeout, stop and report.
+- **Parse metrics programmatically** — write a parsing script, don't eyeball logs
+- **Keep raw outputs** — save each run's full output in `dse_results/outputs/iter_N/`
+- **Constraint violations are not improvements** — a design point that violates constraints is never "best", regardless of the metric
+- If a run crashes, log the error, skip that point, and continue with the next
+- If the same crash repeats 3 times with different configs, stop and report the issue
+
+## Example Invocations
+
+```
+# Minimal — just name the parameters, let the agent figure out ranges
+/dse-loop "Run gem5 mcf benchmark. Tune: L1D_SIZE, L2_SIZE, ROB_ENTRIES. Objective: maximize IPC. Timeout: 3h"
+
+# Partial — some ranges given, some not
+/dse-loop "Run make synth. Tune: CLOCK_PERIOD [5ns, 4ns, 3ns, 2ns], FLATTEN, ABC_SCRIPT. Objective: minimize area at timing closure. Timeout: 1h"
+
+# Fully specified — explicit ranges for everything
+/dse-loop "Simulate processor with FIFO_DEPTH [4,8,16,32], ISSUE_WIDTH [1,2,4], PREFETCH [on,off]. Run: make sim. Objective: max throughput/area. Timeout: 2h"
+
+# Real-world: PDAG-SFA formal verification tuning
+/dse-loop "Run python run_bmc.py. Tune: BMC_DEPTH, ENGINE, TIMEOUT_PER_PROP. Objective: maximize properties proved. Timeout: 2h"
+```

package/generated/skills-bundles/full/skills/data-analysis/statistics/auto-claude__analyze-results/SKILL.md

@@ -0,0 +1,47 @@
+---
+name: "auto-claude/analyze-results"
+description: "Analyze ML experiment results, compute statistics, generate comparison tables and insights. Use when user says \"analyze results\", \"compare\", or needs to interpret experimental data."
+argument-hint: ["results-path-or-description"]
+allowed-tools: "Bash(*), Read, Grep, Glob, Write, Edit, Agent"
+metadata:
+  category: "data-analysis/statistics"
+---
+# Analyze Experiment Results
+
+Analyze: $ARGUMENTS
+
+## Workflow
+
+### Step 1: Locate Results
+Find all relevant JSON/CSV result files:
+- Check `figures/`, `results/`, or project-specific output directories
+- Parse JSON results into structured data
+
+### Step 2: Build Comparison Table
+Organize results by:
+- **Independent variables**: model type, hyperparameters, data config
+- **Dependent variables**: primary metric (e.g., perplexity, accuracy, loss), secondary metrics
+- **Delta vs baseline**: always compute relative improvement
+
+### Step 3: Statistical Analysis
+- If multiple seeds: report mean +/- std, check reproducibility
+- If sweeping a parameter: identify trends (monotonic, U-shaped, plateau)
+- Flag outliers or suspicious results
+
+### Step 4: Generate Insights
+For each finding, structure as:
+1. **Observation**: what the data shows (with numbers)
+2. **Interpretation**: why this might be happening
+3. **Implication**: what this means for the research question
+4. **Next step**: what experiment would test the interpretation
+
+### Step 5: Update Documentation
+If findings are significant:
+- Propose updates to project notes or experiment reports
+- Draft a concise finding statement (1-2 sentences)
+
+## Output Format
+Always include:
+1. Raw data table
+2. Key findings (numbered, concise)
+3. Suggested next experiments (if any)