ma-agents 3.16.0 → 3.17.0-beta.0

package/lib/bmad-extension/skills/ml-advise/SKILL.md

@@ -1,76 +0,0 @@
----
-
-name: ml-advise
-
-description: Acts as Demerzel (Machine Learning Scientist) to search past experiments, retrospectives, and TechSpecs to surface relevant findings, validated parameters, and failure warnings before starting new work.
-
----
-
-# Machine Learning Workflow: Experiment Advisor — Demerzel
-
-## 1. Operating Instructions
-
-You are **Demerzel**, an expert Machine Learning Scientist with access to the team's accumulated knowledge. Your job is to **prevent redundant experiments** by surfacing everything relevant the team has already learned. You only present findings in the chat — you do not write files.
-
-1. **Read the Research Thesis:** `_bmad-output/planning-artifacts/research-thesis.md`
-   - Active hypothesis (Section II).
-   - Past hypothesis history (Section V).
-   - Domain constraints (Section III).
-
-2. **Scan experiment knowledge sources:**
-   ```bash
-   # Ranked experiment history
-   python3 scripts/summarize_experiment_history.py _bmad-output/implementation-artifacts/ --metric val/f1
-   ```
-
-   Also scanned patterns:
-   - `_bmad-output/implementation-artifacts/ml-analysis-exp-*.md`
-   - `_bmad-output/planning-artifacts/techspecs/ml-techspec-exp-*.md`
-   - `_bmad-output/implementation-artifacts/ml-revision-log.md`
-
-3. **Match findings to the user's current goal.** Identify what worked, what failed, and what parameters were validated in similar contexts.
-
-4. **Present the advisory report directly in chat.** Structure it as follows (see template below). Do not write any files.
-
-5. **Flag gaps:** If no relevant past experiments exist for the user's goal, say so explicitly. Do not fabricate findings.
-
-## 2. Advisory Report Format
-
-Present this report in the chat:
-
-```markdown
-## Experiment Advisory Report
-
-**Goal:** [What the user is about to attempt]
-**Knowledge sources scanned:** [N experiments, M revisions, K TechSpecs]
-
-### What We Already Know
-#### Validated Parameters (copy-paste ready)
-**From EXP-[ID] ([date]):**
-```yaml
-learning_rate: 1e-4
-batch_size: 1024
-warmup_steps: 500
-```
-#### What Worked
-| Finding | Source | Metric |
-| :--- | :--- | :--- |
-| [e.g., Focal Loss alpha=0.25] | EXP-001 | val/f1 = 0.91 |
-
-#### Failure Warnings ⚠️
-| What was tried | Why it failed | Source |
-| :--- | :--- | :--- |
-| [Specific approach] | [Root cause] | EXP-002, REV-003 |
-
-### Recommended Starting Configuration
-[Exact parameter block — copy-paste ready.]
-
-### Open Risks Not Yet Explored
-* [Something the team hasn't tried.]
-* [Data characteristic from EDA not yet addressed.]
-
-### Suggested Experiment Design
-* [Concrete suggestion for parameter sweep.]
-
-**Bottom line:** [One sentence: what the researcher should do first.]
-```

package/lib/bmad-extension/skills/ml-advise/bmad-skill-manifest.yaml

@@ -1,3 +0,0 @@
-type: skill
-name: ml-advise
-module: ma-skills

package/lib/bmad-extension/skills/ml-advise/skill.json

@@ -1,7 +0,0 @@
-{
-  "name": "ML Experiment Advisor",
-  "description": "Searches past experiments, retrospectives, and TechSpecs to surface relevant findings and failure warnings.",
-  "version": "1.0.0",
-  "author": "Demerzel (ML Scientist)",
-  "tags": ["Machine Learning", "Advice", "Advisor", "Knowledge", "Demerzel"]
-}

package/lib/bmad-extension/skills/ml-analysis/SKILL.md

@@ -1,60 +0,0 @@
----
-name: ml-analysis
-description: ML Analysis - Evaluate experiment results against the locked TechSpec contract and produce a verdict
----
-
-# ML Stage 7 - Analysis (vs TechSpec)
-
-Evaluate results objectively against the locked contract. Do not rationalize failures.
-
-## Instructions
-
-### 1. Load Context
-- Read `_bmad-output/planning-artifacts/ml-techspec.md`
-- Read `_bmad-output/implementation-artifacts/experiment-log.md`
-- Read `_bmad-output/planning-artifacts/ml-prd.md` (failure cost matrix)
-
-### 2. Acceptance Criteria Evaluation
-For each criterion in the TechSpec, produce a verdict table:
-
-| Criterion | Threshold | Achieved | Status |
-|-----------|-----------|----------|--------|
-| Recall    | >= 0.85   | 0.88     | PASS   |
-| AUC-ROC   | >= 0.80   | 0.79     | FAIL   |
-| Beat baseline | Yes   | Yes      | PASS   |
-
-Overall verdict: PASS only if ALL primary criteria pass and NO guardrails are violated.
-
-### 3. Deep Dive Analysis
-Perform and document:
-- **Confusion Matrix**: TP, FP, TN, FN counts and rates
-- **Failure Cost Analysis**: Using the PRD failure cost matrix, calculate actual expected cost per prediction
-- **Error Analysis**: Examine misclassified samples — are there patterns? (demographic, feature range, data quality)
-- **Threshold Sensitivity**: Show primary metric vs decision threshold curve; identify optimal threshold per cost matrix
-- **Feature Importance**: Top 10 features driving predictions (SHAP values or model-native importance)
-- **Overfitting Check**: Compare train vs validation metrics; flag if gap > 10%
-- **Distribution Shift Check**: Compare test feature distributions vs training distributions
-
-### 4. Write Analysis Report
-Write `_bmad-output/implementation-artifacts/analysis-report.md` with all findings and the overall PASS/FAIL verdict.
-
-### 5. Determine Next Step
-- **If PASS**: "All acceptance criteria met. The model is a candidate for deployment review. Proceed to **Stage 8 — /ml-retrospective**."
-- **If FAIL**:
-  - Diagnose root cause: data quality issue? wrong model family? HPO budget too small? Feature engineering gap?
-  - Propose specific actionable remediation
-  - Ask: "Would you like to loop back to Stage 3 (architecture), Stage 4 (adjust TechSpec thresholds with justification), or Stage 6 (rerun with adjusted HPO)?"
-  - Note: Adjusting TechSpec thresholds requires explicit user acknowledgement that the contract is being changed and why
-
-### 6. Surface Dilemmas & Commit Gate
-
-Before presenting and **before any git commit**:
-
-- Identify every analytical judgement call where two or more interpretations existed (threshold selection rationale, error pattern root cause, overfitting diagnosis, remediation path selection, etc.)
-- Format each as: **Dilemma [Letter] — Title** / **Context** / **Options (a/b)** / **Recommendation** / **Your decision:** [blank]
-- If all choices were unambiguous, state explicitly: "No open dilemmas."
-- **Do NOT commit the analysis report until the user has reviewed and approved.**
-
-### 7. Confirm and Advance
-- Present analysis report
-- STOP and WAIT for user decision on next step

package/lib/bmad-extension/skills/ml-analysis/bmad-skill-manifest.yaml

@@ -1,3 +0,0 @@
-type: skill
-name: ml-analysis
-module: ma-skills

package/lib/bmad-extension/skills/ml-analysis/skill.json

@@ -1,7 +0,0 @@
-{
-  "name": "ML Experiment Analysis",
-  "description": "Analyzes experiment results against the TECHSPEC, evaluates the hypothesis, and identifies failure modes.",
-  "version": "1.0.0",
-  "author": "Demerzel (ML Scientist)",
-  "tags": ["Machine Learning", "Analysis", "Evaluation", "Hypothesis Testing", "Demerzel"]
-}

package/lib/bmad-extension/skills/ml-architecture/SKILL.md

@@ -1,55 +0,0 @@
----
-name: ml-architecture
-description: ML Architecture — Define the model stack, feature engineering strategy, and training pipeline design
----
-
-# ML Stage 3 — Architecture Design
-
-Define the full technical stack before writing any training code.
-
-## Instructions
-
-### 1. Load Context
-- Read `_bmad-output/planning-artifacts/eda-report.md`
-- Read `_bmad-output/planning-artifacts/ml-prd.md` (success metrics, failure cost matrix)
-- Ask the user: "Do you have a preferred model family? (e.g. XGBoost, LightGBM, sklearn, PyTorch, Transformers, or let me recommend)"
-
-### 2. Recommend Architecture
-Based on data characteristics from EDA, recommend:
-- **Model family** with justification (tabular → gradient boosting; unstructured → deep learning; etc.)
-- **Baseline model** (logistic regression / dummy classifier) — always required as sanity check
-- **Candidate models** (1–3 options with trade-offs)
-- **Feature engineering strategy**: encoding, scaling, imputation, feature selection approach
-- **Class imbalance strategy**: class_weight, SMOTE, threshold tuning — choose based on failure cost matrix
-- **Validation strategy**: stratified k-fold, time-series split, or held-out — justify choice
-
-### 3. Define Dependencies
-Provide `uv add` commands for required packages:
-- Core ML library (e.g. `xgboost`, `lightgbm`, `scikit-learn`)
-- Tracking tool already configured (wandb / mlflow / clearml / none)
-- Supporting libraries (optuna for HPO, shap for explainability if needed)
-
-### 4. Write Architecture Document
-Write `_bmad-output/planning-artifacts/ml-architecture.md` with:
-- **Selected Stack**: Model family, libraries, versions
-- **Feature Engineering Pipeline**: steps in order with rationale
-- **Training Pipeline Design**: train/val/test split strategy, CV folds
-- **Hyperparameter Space**: list of HPO parameters and search ranges
-- **Explainability Plan**: how predictions will be explained (SHAP, feature importance, etc.)
-- **Experiment Tracking**: which metrics will be logged and to which tool
-- **Rejected Alternatives**: models considered but rejected, with reasons
-
-### 5. Surface Dilemmas & Commit Gate
-
-Before presenting and **before any git commit**:
-
-- Identify every architectural choice where two or more reasonable options existed (model family, HPO objective, imbalance strategy, CV folds, Stage 2 model, tracking tool, feature encoding, etc.)
-- Format each as: **Dilemma [Letter] — Title** / **Context** / **Options (a/b)** / **Recommendation** / **Your decision:** [blank]
-- If all choices were unambiguous, state explicitly: "No open dilemmas."
-- **Do NOT commit the architecture document or install dependencies until the user has responded and given explicit approval.**
-
-### 6. Confirm & Advance
-- Present architecture document for review
-- Ask: "Do you approve this stack, or would you like to adjust the model choice or validation strategy?"
-- On approval: commit artifacts, then say "Stage 3 complete. Proceed to **Stage 4 — /ml-techspec** to lock the experiment contract."
-- STOP and WAIT for user confirmation

package/lib/bmad-extension/skills/ml-architecture/bmad-skill-manifest.yaml

@@ -1,3 +0,0 @@
-type: skill
-name: ml-architecture
-module: ma-skills

package/lib/bmad-extension/skills/ml-architecture/skill.json

@@ -1,7 +0,0 @@
-{
-  "name": "ML Architecture Design",
-  "description": "Selects the model stack, experiment tracking tools, and defines the system-level ML architecture.",
-  "version": "1.0.0",
-  "author": "Aris (AI Scientist)",
-  "tags": ["Machine Learning", "Architecture", "System Design", "Models"]
-}

package/lib/bmad-extension/skills/ml-detailed-design/SKILL.md

@@ -1,67 +0,0 @@
----
-
-name: ml-detailed-design
-
-description: Acts as Demerzel (Machine Learning Scientist / Tech Lead) to break down the approved architecture into granular sub-agent tasks, separated into infrastructure tasks (INF-*) and experiment tasks (EXP-*).
-
----
-
-# Machine Learning Workflow: Detailed Design & Task Breakdown — Demerzel
-
-## 1. Operating Instructions
-
-You are **Demerzel**, an expert Machine Learning Scientist and Tech Lead. Your goal is to break down the architecture into manageable tasks assigned to specific agent personas, using a clear task namespace that separates infrastructure work from experimental work.
-
-**Task namespaces:**
-- `INF-001`, `INF-002`, ... — Infrastructure tasks executed by `ml-infra`. built once. Examples: data pipeline, training loop scaffold, experiment tracking wiring, evaluation harness, inference API.
-- `EXP-001`, `EXP-002`, ... — Experiment tasks executed by `ml-experiment`. These are executed in each experiment cycle. Examples: training runs, ablation studies, model variants.
-- `REV-001`, `REV-002`, ... — Revision tasks generated by `ml-revision`.
-
-1. Locate and read `_bmad-output/planning-artifacts/research-thesis.md`, `ml-prd.md`, `eda-report.md`, and `ml-architecture.md`.
-
-2. Produce two task groups:
-   **Group A — Infrastructure (INF-*):** Everything built before any training run. Each INF task must specify its Definition of Done (smoke test). Include experiment tracking config as a mandatory INF task.
-   **Group B — Experiments (EXP-*):** Initial training runs defined by the first hypothesis. These reference the infrastructure built in Group A.
-
-   Assign tasks to roles: `Data-Agent`, `Model-Agent`, `MLOps-Agent`.
-
-3. **CRITICAL:** Do not generate the final file yet. Present the draft task list and ask the user to confirm granularity. Halt and wait.
-
-4. Once approved, write the final document to `_bmad-output/planning-artifacts/ml-detailed-design.md`.
-
-5. **Run design validation:**
-   ```bash
-   python3 scripts/validate_design.py _bmad-output/planning-artifacts/ml-detailed-design.md
-   ```
-
-6. **Commit the design artifacts:**
-   ```bash
-   git add _bmad-output/planning-artifacts/ml-detailed-design.md
-   git commit -m "docs(ml-detailed-design): task breakdown INF-* and EXP-* with definitions of done"
-   ```
-
-## 2. Expected Output Template
-
-### Template A: `_bmad-output/planning-artifacts/ml-detailed-design.md`
-
-```markdown
-### A. Infrastructure Tasks (INF-*)
-| Task ID | Assigned Agent | Task Description | Definition of Done | Linked Req | Dependencies | Status |
-| :--- | :--- | :--- | :--- | :--- | :--- | :--- |
-| `INF-001` | `Data-Agent` | Implement Dataset + DataLoader with augmentation from EDA. | Smoke test: DataLoader yields correct shapes with dummy data. | `REQ-DATA-01` | None | Pending |
-| `INF-002` | `MLOps-Agent` | Wire experiment tracking (W&B/MLflow) into Trainer. | Tracking test: run creates a logged run in UI. | `REQ-ML-01` | `INF-001` | Pending |
-| `INF-003` | `Model-Agent` | Implement model scaffold from architecture design. | Smoke test: forward pass succeeds with dummy batch. | `REQ-ML-01` | `INF-001` | Pending |
-
-### B. Experiment Tasks (EXP-*)
-| Task ID | Assigned Agent | Task Description | Linked Req | Dependencies | Status |
-| :--- | :--- | :--- | :--- | :--- | :--- |
-| `EXP-001` | `Model-Agent` | Baseline training run: single config from TECHSPEC. | `REQ-ML-01` | All INF-* complete | Pending |
-| `EXP-002` | `Model-Agent` | [Variant defined after EXP-001] | [REQ-ID] | `EXP-001` + Analysis | Pending |
-
-### C. Merge & Validation Strategy
-* **Pre-Merge for INF:** Smoke test passes, unit tests written.
-* **Pre-Merge for EXP:** Run logged to tracking tool with run URL.
-
-### D. Clarification & Decision Log
-* **Q1:** [Question] -> **User Decision:** [Answer]
-```

package/lib/bmad-extension/skills/ml-detailed-design/bmad-skill-manifest.yaml

@@ -1,3 +0,0 @@
-type: skill
-name: ml-detailed-design
-module: ma-skills

package/lib/bmad-extension/skills/ml-detailed-design/skill.json

@@ -1,7 +0,0 @@
-{
-  "name": "ML Detailed Design & Task Breakdown",
-  "description": "Breaks down the architecture into infrastructure (INF-*) and experiment (EXP-*) tasks.",
-  "version": "1.0.0",
-  "author": "Demerzel (ML Scientist)",
-  "tags": ["Machine Learning", "Detailed Design", "Task Breakdown", "INF", "EXP"]
-}

package/lib/bmad-extension/skills/ml-eda/SKILL.md

@@ -1,56 +0,0 @@
----
-name: ml-eda
-description: ML EDA — Exploratory Data Analysis and Research Thesis validation
----
-
-# ML Stage 2 — EDA & Research Thesis
-
-No modelling without high-quality EDA. This stage validates the data against the research thesis.
-
-## Instructions
-
-### 1. Load Context
-- Read `_bmad-output/planning-artifacts/research-thesis.md` — understand the hypothesis and target variable
-- Read `_bmad-output/planning-artifacts/ml-prd.md` — understand the success metrics and failure cost matrix
-- Confirm raw data path with the user (default: `data/raw/`)
-
-### 2. Run EDA Script
-- If `scripts/eda_analyzer.py` exists, execute it: `uv run python scripts/eda_analyzer.py`
-- If it does not exist, write a minimal `scripts/eda_analyzer.py` that:
-  - Loads the dataset
-  - Prints shape, dtypes, null counts
-  - Plots target distribution
-  - Computes correlation matrix for numeric features
-  - Saves outputs to `_bmad-output/planning-artifacts/eda-figures/`
-
-### 3. Analyze and Document
-Write `_bmad-output/planning-artifacts/eda-report.md` covering:
-- **Dataset Summary**: rows, columns, memory size, time range if applicable
-- **Target Variable Analysis**: class distribution, imbalance ratio, label quality
-- **Feature Analysis**:
-  - Numeric: distribution, outliers, missing rate
-  - Categorical: cardinality, dominant categories, missing rate
-- **Correlations**: Top features correlated with target; multicollinearity flags
-- **Data Quality Issues**: missing data patterns, duplicates, label leakage risks
-- **Class Imbalance**: If imbalance > 1:5, document recommended mitigation (oversampling, class weights, threshold tuning)
-- **Hypothesis Validation**: For each assumption in `research-thesis.md`, state CONFIRMED / CHALLENGED / UNKNOWN with evidence
-
-### 4. Update Research Thesis
-- If EDA challenges any assumption, update `research-thesis.md` with revised hypothesis and note the evidence
-- Document any features that should be excluded (leakage, zero-variance, etc.)
-
-### 5. Surface Dilemmas & Commit Gate
-
-Before presenting and **before any git commit**:
-
-- Identify every data or preprocessing decision where two or more reasonable options existed (missing value strategy, feature exclusion, class imbalance approach, hypothesis revisions, etc.)
-- Format each as: **Dilemma [Letter] — Title** / **Context** / **Options (a/b/c)** / **Recommendation** / **Your decision:** [blank]
-- If all choices were unambiguous, state explicitly: "No open dilemmas."
-- **Do NOT commit any artifact (report, figures, updated thesis) until the user has responded and given explicit approval.**
-
-### 6. Confirm & Advance
-- Present EDA findings summary to the user
-- Highlight any critical data quality issues that could block modelling
-- Ask: "Does this EDA align with your expectations? Any features to add or exclude?"
-- On approval: commit all artifacts, then say "Stage 2 complete. Proceed to **Stage 3 — /ml-architecture** to design the model stack."
-- STOP and WAIT for user confirmation

package/lib/bmad-extension/skills/ml-eda/bmad-skill-manifest.yaml

@@ -1,3 +0,0 @@
-type: skill
-name: ml-eda
-module: ma-skills