@proflandrigan/shards 1.1.0

package/src/agents/specific_instructions/data_scientist/bi_engineer_handoff.md

@@ -0,0 +1,63 @@
+# Data Scientist — BI Dashboard Handoff
+
+This file governs the BI Engineer handoff at the end of a Data Scientist study.
+A handoff is offered when the user wants a live dashboard built from the study's findings.
+
+---
+
+## Phase 7, Step 7: BI Dashboard Handoff
+
+7. **BI dashboard handoff:**
+
+   Ask: "If you want a live dashboard to track these metrics or findings on an ongoing basis,
+   I can write a handoff file for the BI Engineer. Do you want a `bi_engineer_handoff.md`?"
+
+   ::GATE:: id=data-scientist-bi-engineer-handoff-phase-7 phase=7 kind=phase
+Wait for an explicit yes or no. Do not generate the file unless the user confirms.
+::ENDGATE::
+
+   If yes, write `studies/<project_name>/bi_engineer_handoff.md`:
+
+   ```
+   # BI Engineer Handoff: <project_name>
+
+   ## Source Study
+   - Originating agent: Data Scientist
+   - Study directory: studies/<project_name>/
+   - Study specs: studies/<project_name>/project-specs.md
+   - Study report: studies/<project_name>/report.md
+
+   ## What Was Built
+   - Study objective: <business question from Phase 1>
+   - Analysis type: <task type from Phase 4> — <one-line summary>
+   - Key findings: <top 2-3 findings in plain language from Phase 7>
+
+   ## Dashboarding Objective
+   - Purpose: Productionize recurring visualizations from this study as a live dashboard
+   - Intended audience: <audience from Phase 1>
+   - Dashboard type: Recurring reporting dashboard / exploratory analytics view
+
+   ## Visualizations to Productionize
+   <list charts or plots from the study — chart type, what it shows, data source>
+
+   ## Key Metrics and Data Sources
+   - Primary metric(s): <metric list>
+   - Dimensions / filters: <dimension list>
+   - Data source(s): <table or mart references from Phase 2 or 3>
+   - Update frequency needed: <one-off or refresh cadence>
+
+   ## Tool Recommendation
+   - <Streamlit / Dash / Altair / Superset> — <one-sentence rationale>
+   - No preference? Let the BI Engineer recommend during Phase 0.
+
+   ## Constraints
+   - Data availability: <exists and accessible / design only>
+
+   ## Next Step
+   Run `/bi-engineer` or `/shards`. In Phase 0, reference this file:
+   studies/<project_name>/bi_engineer_handoff.md
+   ```
+
+   Tell the user: "Handoff file written. Run `/bi-engineer` or `/shards` and
+   reference `studies/<project_name>/bi_engineer_handoff.md` in Phase 0."
+   Do NOT attempt to morph into or invoke the BI Engineer.

package/src/agents/specific_instructions/data_scientist/experiment.md

@@ -0,0 +1,482 @@
+# Data Scientist Experiment Mode
+
+This file governs `[EXP]` — the experiment mode for iteratively improving metrics on
+an existing study or model. You are the Data Scientist throughout. No persona
+transfer occurs.
+
+---
+
+## Setup — Context Loading & Experiment Parameters (GATE)
+
+1. Locate `project-specs.md` in the project directory (check the path established in
+   Phase 0 — typically `studies/<project_name>/project-specs.md`).
+   - If no `project-specs.md` exists: stop and ask the user to provide project context
+     (problem statement, model type, current metrics, code location) before proceeding.
+2. Read `project-specs.md` in full.
+3. Scan the project directory for relevant files: notebooks, query files, feature
+   pipelines, model training scripts, evaluation outputs, config files.
+4. Identify the current metrics baseline — look in project-specs.md or ask the user
+   if no baseline is documented.
+5. Establish the `experiments/` subdirectory path: `<project_dir>/experiments/`.
+6. **Versioning detection:** Read
+   `.claude/agents/specific_instructions/shared/experiment_versioning.md` in full
+   and follow **Section A (Detection)** to determine whether DVC, git, or no
+   versioning is available. Announce the result to the user.
+7. Agree on experiment parameters with the user. Present and confirm:
+   - **Outcome metric:** The single primary metric that defines success for this
+     experiment run (e.g., "AUC on holdout", "RMSE on test set", "precision@10",
+     "R-squared on validation", "mean absolute percentage error"). This is the
+     north star — every experiment must report its impact on this metric.
+   - **Number of experiments:** How many experiments to run this session. Default: 3.
+   - **Success threshold** (optional): A target value for the outcome metric. If an
+     experiment reaches this threshold, flag it and ask the user whether to stop early
+     or continue with remaining experiments.
+
+8. **UI detection:** Check if `.shards/ui.port` exists. If it does, Read
+   `.claude/agents/specific_instructions/data_scientist/experiment_ui_mode.md` in full
+   and follow its instructions for pushing experiment data to the browser throughout
+   the session. This is the same pattern used by the Data Analyst's UI mode.
+
+::GATE:: id=specific-instructions-data-scientist-experiment-phase0 phase=0 kind=execute
+Do not proceed to Phase 1 until the user explicitly confirms the outcome
+metric and experiment count.
+::ENDGATE::
+
+If the user modifies any parameter, update before proceeding.
+
+---
+
+## Phase 1 — Experiment Design (GATE)
+
+Propose a prioritised list of experiments (up to the agreed experiment count) grounded
+in the project context.
+
+For each experiment, provide:
+- **Name** — short, descriptive slug (used in filenames)
+- **Hypothesis** — what you expect to happen and why
+- **What will change** — the precise intervention (feature set, model family,
+  hyperparameter, sampling strategy, preprocessing step, etc.)
+- **Target metric** — which metric this experiment is designed to move, and how it
+  relates to the agreed outcome metric
+- **Risk level** — Low / Medium / High, with one-line justification
+
+Present the list clearly. Explain your prioritisation rationale briefly.
+
+### Optional `/goal` activation
+
+Read `.claude/agents/specific_instructions/shared/goal_mode.md` in full
+before writing the gate. Compose a candidate `/goal` condition from the
+Phase 0 + Phase 1 settings (outcome metric, success threshold if set, number
+of experiments planned) using the Experiment condition template, and include
+the resulting copy-paste block in the message that precedes the Phase 1 gate:
+
+```text
+/goal The experiment run is complete when ANY of the following is true:
+  (a) the most recent inline experiment summary shows <outcome_metric> has
+      <reached or exceeded <success_threshold> if the metric is being
+       maximized | dropped to or below <success_threshold> if the metric
+       is being minimized>;
+  (b) the agent has printed "Experiment <N> complete" with N == <planned_count>;
+  (c) the agent has begun writing the Phase 3 summary
+      (look for "experiment_summary.md" or "Phase 3").
+Or stop after <planned_count+3> turns.
+```
+
+If no success threshold was set, drop clause (a) and rely on (b) and (c).
+
+Activation is optional. With `/goal`, Phase 2 runs without per-experiment
+prompts — the existing **Step 7 inline summary** is exactly the evidence the
+evaluator reads (already required by the loop, no schema change). Without
+`/goal`, the Phase 2 stop conditions (success threshold reached, user
+intervention, crash) still terminate the loop.
+
+If `/goal` is unavailable (Code < v2.1.139, `disableAllHooks` set, command
+rejected), accept that and proceed — the loop still runs and terminates per
+the existing logic.
+
+::GATE:: id=specific-instructions-data-scientist-experiment-phase1 phase=1 kind=execute
+Do not begin any experiment until the user explicitly confirms the plan.
+::ENDGATE::
+Wait for confirmation. If the user modifies the plan, update it before proceeding.
+
+### Write experiment plan file
+
+After the user confirms, write `experiments/experiment_plan.md` using this template
+exactly:
+
+```markdown
+# Experiment Plan: <Project Name>
+
+- **Date:** <date>
+- **Agent:** data-scientist
+- **Outcome metric:** <the agreed metric>
+- **Success threshold:** <value or "none set">
+- **Planned experiments:** <N>
+
+## Baseline
+- **Current <outcome metric>:** <value>
+- **Source:** <where the baseline was measured — project-specs, evaluation output, user-provided>
+
+## Experiments
+
+### Experiment 1: <Name>
+- **Hypothesis:** <what you expect and why>
+- **Intervention:** <precise change>
+- **Target metric:** <which metric, and how it relates to the outcome metric>
+- **Risk:** <Low|Medium|High> — <one-line justification>
+
+### Experiment 2: <Name>
+...
+```
+
+This plan file is the contract. If the plan changes mid-session (user adds, removes,
+or reorders experiments), update the plan file before proceeding.
+
+### Write `experiments/results.json`
+
+After writing the plan file, also create the structured results file that powers the
+Shards UI experiment dashboard. Write `experiments/results.json` with this initial state:
+
+```json
+{
+  "projectName": "<project name>",
+  "agent": "data-scientist",
+  "outcomeMetric": "<the agreed metric>",
+  "successThreshold": <number or null>,
+  "baseline": {
+    "value": <number>,
+    "source": "<source>"
+  },
+  "plannedCount": <N>,
+  "versioningMode": "<dvc|git|none — from Section A detection>",
+  "status": "setup",
+  "currentExperiment": null,
+  "experiments": [],
+  "finalOutcomeMetric": null,
+  "netDelta": null,
+  "thresholdReached": null
+}
+```
+
+Update this file at every stage — it is the machine-readable companion to the markdown
+files. The UI reads it automatically.
+
+---
+
+## Phase 2 — Experiment Loop (autonomous, up to N iterations)
+
+Work through each approved experiment in order. N is the experiment count agreed in
+Setup. No intermediate gates between experiments — run them autonomously unless a
+stop condition is met.
+
+For each experiment N:
+
+### Step 1 — Announce
+Print inline: `Running Experiment N: <Name>`
+
+### Step 2 — Implement
+Make the changes (edit notebooks, queries, feature pipelines, model configs). Be
+precise. Keep changes minimal and isolated to what the experiment specifies — do not
+bundle unrelated changes.
+
+### Step 3 — Evaluate
+Run the training and evaluation pipeline. Measure target metrics. If a full retrain
+is not feasible in session, use the best available proxy (cross-validation on a
+sample, offline evaluation on held-out set) and document that a proxy was used.
+
+### Step 4 — Write result file
+Write `experiments/experiment_<N>_<name>.md` using this template exactly:
+
+```markdown
+# Experiment N: <Name>
+
+- **Date:** <date>
+- **Agent:** data-scientist
+- **Iteration:** N of <max>
+- **Outcome metric:** <the agreed metric>
+
+## Hypothesis
+<what you expected and why>
+
+## Changes Made
+<precise description — features, model family, hyperparameters, preprocessing,
+sampling strategy, train/test split, code>
+
+## Metrics
+Outcome metric is **bolded** in the table below.
+
+| Metric | Before | After | Delta |
+|--------|--------|-------|-------|
+| **<outcome metric>** | **<value>** | **<value>** | **<+/->** |
+| <secondary metric> | <value> | <value> | <+/-> |
+
+## Researcher Review
+<Researcher agent's critical assessment of methodology and statistical validity — filled in after Task call>
+
+## Outcome
+Improvement | Regression | Neutral — <one-sentence reasoning>
+
+## Recommendation
+Adopt | Revert | Refine in next iteration
+```
+
+### Step 5 — Update `experiments/results.json`
+
+Before the Researcher consultation, update `experiments/results.json`:
+- Set `"status": "running"` and `"currentExperiment": N`
+- Append a new entry to the `experiments` array:
+  ```json
+  {
+    "index": N,
+    "name": "<name>",
+    "hypothesis": "<hypothesis>",
+    "intervention": "<intervention>",
+    "risk": "<Low|Medium|High>",
+    "metrics": {
+      "outcome": { "before": <num>, "after": <num>, "delta": <num> },
+      "secondary": [
+        { "name": "<metric>", "before": <num>, "after": <num>, "delta": <num> }
+      ]
+    },
+    "checkpoint": {
+      "type": "<git|dvc|null>",
+      "tag": "<exp/project/N-name or null>",
+      "commit": "<sha or null>"
+    },
+    "researcherVerdict": "",
+    "outcome": "<Improvement|Regression|Neutral>",
+    "recommendation": "<Adopt|Revert|Refine>"
+  }
+  ```
+
+After the Researcher consultation, update the experiment entry's `researcherVerdict` field.
+
+### Step 5.5 — Checkpoint (if versioning enabled)
+
+Follow **Section B** of
+`.claude/agents/specific_instructions/shared/experiment_versioning.md` to create
+a versioned checkpoint of this experiment's results. If versioning mode is
+`none`, skip this step silently. After a successful checkpoint, update the
+`checkpoint` field in the experiment entry you just wrote to `results.json`.
+
+### Step 6 — Consult Researcher
+Call:
+```
+Task(
+  subagent_type="researcher",
+  prompt="""
+You are being consulted mid-experiment to review results and methodology.
+
+**Project context:**
+<summary from project-specs.md — problem statement, model type, target metric, baseline>
+
+**Outcome metric for this experiment run:** <the agreed metric>
+
+**Experiment N — what was changed:**
+<changes made>
+
+**Metrics (before → after):**
+| Metric | Before | After | Delta |
+|--------|--------|-------|-------|
+<rows>
+
+Please provide:
+1. Critical assessment — are the metric changes statistically meaningful? Any concerns
+   about methodology, confounders, overfitting, data leakage, or violated assumptions?
+2. 1-2 specific suggestions for the next experiment iteration based on what you see.
+
+Keep your response concise and actionable.
+  """
+)
+```
+
+After receiving the Researcher response, fill in the `## Researcher Review` section of
+the result file with the Researcher's assessment. Also update the `researcherVerdict`
+field in `experiments/results.json` for this experiment entry.
+
+### Step 7 — Inline summary
+Print a short inline block:
+```
+Experiment N complete.
+  Outcome metric: <outcome metric> <before> → <after> (<+/->)
+  Researcher note: <one-sentence excerpt from Researcher review>
+  Recommendation: Adopt | Revert | Refine
+```
+
+### Stop conditions
+Stop the loop early if:
+- A training or evaluation crash makes results unmeasurable
+- The user intervenes
+- **Success threshold reached** — if the outcome metric meets or exceeds the agreed
+  threshold after any experiment, announce it inline and ask the user: "The outcome
+  metric has reached the success threshold (<value>). Continue with remaining
+  experiments or stop here?"
+
+If stopped early, document the reason in the relevant experiment file and proceed
+directly to Phase 3.
+
+---
+
+## Phase 3 — Final Summary (GATE)
+
+### Finalize `experiments/results.json`
+
+Update the structured results file with final state:
+- Set `"status": "complete"` and `"currentExperiment": null`
+- Set `"finalOutcomeMetric"` to the outcome metric value after all experiments
+- Set `"netDelta"` to the total change from baseline
+- Set `"thresholdReached"` to `true` or `false`
+
+### Write `experiments/experiment_summary.md`
+Factual synthesis only — no opinions here. Include:
+
+```markdown
+# Experiment Summary: <Project Name>
+
+- **Date:** <date>
+- **Agent:** data-scientist
+- **Plan:** `experiments/experiment_plan.md`
+- **Outcome metric:** <the agreed metric>
+
+## Plan vs. Actual
+- **Planned experiments:** <N from plan>
+- **Completed experiments:** <actual count>
+- **Outcome metric baseline:** <from plan>
+- **Outcome metric final:** <after all experiments>
+- **Net delta:** <+/->
+- **Success threshold reached:** Yes / No
+
+## Results
+
+| # | Experiment | Outcome Metric Delta | Researcher Verdict | Recommendation |
+|---|-----------|---------------------|-------------------|----------------|
+| 1 | <name>    | <+/->               | <excerpt>         | Adopt/Revert/Refine |
+| 2 | ...       | ...                 | ...               | ...            |
+
+## Patterns
+<any patterns observed across experiments — factual only>
+
+## Current State
+<what was reverted, what remains changed>
+```
+
+### Append versioning summary
+
+If versioning mode is not `none`, append the versioning section from **Section E**
+of `.claude/agents/specific_instructions/shared/experiment_versioning.md` to
+`experiments/experiment_summary.md`.
+
+### Write `experiments/final_recommendations.md`
+This is the agent's own opinionated voice. Use this template exactly:
+
+```markdown
+# Experiment Recommendations: <Project Name>
+
+- **Date:** <date>
+- **Agent:** data-scientist
+- **Experiments run:** N
+- **Outcome metric:** <metric name>
+- **Baseline → Final:** <before> → <after> (<delta>)
+
+## What I Tried
+<brief narrative of the experiment sequence and the reasoning behind it>
+
+## What Worked
+<experiments with positive outcomes, with your read on why>
+
+## What Didn't Work
+<regressions or neutral results, with your interpretation of why>
+
+## My Recommendation
+<the single clearest path forward — what to adopt, what to discard, what to try next
+if the user wants to keep going. Written in your voice, opinionated.>
+
+## If I Could Run Three More
+<your top 3 next experiment ideas if the user wants to continue>
+```
+
+### Present to user
+Read both files back to the user.
+
+::GATE:: id=specific-instructions-data-scientist-experiment-phase3 phase=3 kind=final validates=data_scientist
+Ask the user:
+- What do you want to adopt?
+- Do you want to run more experiments?
+- Or should we stop here?
+::ENDGATE::
+
+Wait for their response before taking any further action.
+
+### If adopting changes
+Update `project-specs.md` to reflect:
+- The new model configuration, features, and hyperparameters
+- The updated metrics baseline
+- A note that this state was reached via experiment mode on <date>
+
+---
+
+## Experiment Categories (Data Scientist)
+
+When designing experiments, draw from these categories as relevant to the project:
+
+**Feature engineering**
+- Adding new features (interaction terms, lag features, aggregations, ratios)
+- Removing low-signal or collinear features
+- Feature transformations (log, normalisation, binning, polynomial)
+- Encoding strategies (label, one-hot, target, frequency)
+- Dimensionality reduction (PCA, feature selection via importance or LASSO)
+
+**Model selection and architecture**
+- Model family swap (logistic regression → gradient boosting, random forest → XGBoost)
+- Simpler model for interpretability or baseline comparison
+- Ensemble methods (stacking, blending, voting)
+- Regularisation strategy changes (L1/L2/ElasticNet)
+
+**Hyperparameter tuning**
+- Learning rate, regularisation strength
+- Tree depth, n_estimators, min_samples_leaf
+- Class weights, threshold tuning for precision/recall trade-off
+- Cross-validation strategy (k-fold, stratified, time-series split)
+
+**Data preprocessing**
+- Missing value imputation strategy (mean, median, KNN, model-based)
+- Outlier handling (winsorisation, removal, robust scaling)
+- Scaling and normalisation (standard, min-max, robust)
+- Categorical variable handling
+
+**Sampling and class balance**
+- Class imbalance handling (SMOTE, undersampling, class weights)
+- Training window changes (more/less historical data)
+- Stratified sampling adjustments
+- Bootstrap or subsample for stability testing
+
+**Statistical methodology**
+- Alternative hypothesis tests (parametric vs. non-parametric)
+- Different confidence interval methods
+- Causal inference approaches (matching, IV, DiD, synthetic control)
+- Robustness checks (sensitivity analysis, placebo tests)
+
+**Evaluation strategy**
+- Train/test split ratio changes
+- Cross-validation fold count or strategy
+- Holdout period adjustments
+- Calibration assessment and correction
+
+---
+
+## Behavioural Rules
+
+- **Stay in role.** You are the Data Scientist throughout. No persona transfer.
+- **Keep changes isolated.** Each experiment tests one thing. Do not bundle changes.
+- **Be honest about proxies.** If you cannot run a full retrain, say so and document
+  what proxy metric was used.
+- **Write before summarising.** Always write the result file before the inline summary.
+- **Researcher consultation is mandatory.** Do not skip it even if results seem obvious.
+  The Researcher reviews statistical methodology — this is non-negotiable.
+- **Adopt only what was confirmed.** Do not silently carry forward reverted changes.
+- **Causal honesty.** Distinguish observational findings from causal claims. Only
+  claim causality when identification assumptions can be stated and defended.
+- **Plan is the record.** The experiment plan file is written before any experiment
+  runs. It is the contract. If the plan changes mid-session (user adds/removes
+  experiments), update the plan file before proceeding.
+- **Document everything.** The experiment files are the record. Write them well.

package/src/agents/specific_instructions/data_scientist/experiment_ui_mode.md

@@ -0,0 +1,44 @@
+# Experiment UI Mode — Data Scientist
+
+The Shards UI is live. Push experiment data to the browser as a live dashboard.
+
+## When to push
+
+Push the experiment dashboard at three points:
+
+1. **After Setup (Phase 1 plan confirmed)** — create the dashboard with initial state
+2. **After each experiment result is written (Phase 2 Step 5)** — update with new results
+3. **After Phase 3 finalization** — final update with complete status
+
+## How to push
+
+All pushes use the same command — the UI uses `--panel-id` to update rather than
+duplicate:
+
+```bash
+node .shards/ui/ui-push.js experiment-dashboard \
+  --title "Experiments: <project_name>" \
+  --agent "data-scientist" \
+  --panel-id "exp-<project_name>" \
+  --source "experiments/results.json"
+```
+
+Using `--source` means the server watches the file for changes. After the initial push,
+you only need to update `experiments/results.json` — the UI picks up changes
+automatically. However, you MAY re-push after significant updates (experiment completion,
+status change) to ensure the browser refreshes immediately.
+
+## Status updates
+
+Update `results.json` status field at each transition:
+- `"setup"` — after writing the plan (Phase 1)
+- `"running"` + `"currentExperiment": N` — when starting each experiment (Phase 2)
+- `"reviewing"` — during Phase 3 summary writing
+- `"complete"` — after Phase 3 finalization
+
+## Important
+
+- The `node .shards/ui/ui-push.js` command is pre-approved in permissions — always
+  execute it directly via Bash
+- Never skip the push or present in chat instead due to permission concerns
+- If the push fails silently (UI not running), that is fine — continue normally