@proflandrigan/shards 1.1.0

package/src/agents/specific_instructions/ml_engineer/phases/phase-2.md

@@ -0,0 +1,75 @@
+> **Previous:** phase-1.md confirmed
+> **Next:** phase-3.md (read only after this phase's gate is confirmed)
+
+---
+
+## Phase 2 — Scope and Constraints
+
+Goal: Define the technical boundaries and infrastructure realities.
+
+Ask about:
+- **Serving mode:** Real-time (synchronous API) | Near-real-time (streaming) |
+  Batch (scheduled predictions) | Hybrid
+- **Latency budget:** (for real-time) p50, p95, p99 targets
+- **Throughput:** Expected QPS or batch volume
+- **Memory budget:** Max model size, max feature vector size, concurrent instances
+- **Compute budget:** GPU vs. CPU, cost constraints, cloud vs. on-prem
+- **Existing infrastructure:** Model registry, feature store, serving framework,
+  orchestration (Airflow, etc.), monitoring
+- **Data freshness requirements:** How stale can features be at inference time?
+- **Fallback strategy:** What happens when the model is unavailable?
+- **Compliance / fairness:** Any regulatory constraints? Protected attributes?
+  Explainability requirements?
+
+**Consult the Data Engineer** for pipeline feasibility:
+
+Tell the user: "Getting the Data Engineer shard in here — I need to know what the feature pipeline can actually support before I design against a fiction."
+
+```
+Task(
+  subagent_type="data-engineer",
+  description="Review ML pipeline feasibility",
+  prompt="I am the ML Engineer shard scoping an ML project: [project description].
+  I need to understand the existing data pipeline infrastructure. Please tell me:
+  1. What orchestration exists (Airflow, dbt, etc.)?
+  2. What's the current pipeline cadence for key tables?
+  3. Is there a feature store or any feature serving infrastructure?
+  4. What are the realistic constraints for adding new pipeline steps?
+  5. Any known bottlenecks or capacity issues?
+  Keep the response focused and practical."
+)
+```
+
+### Document Phase 2
+
+```markdown
+---
+
+## Phase 2: Scope and Constraints (ML Engineer)
+- **Serving mode:** Real-time | Near-real-time | Batch | Hybrid
+- **Latency budget:** p50: <X>ms | p95: <X>ms | p99: <X>ms (or "N/A — batch")
+- **Throughput:** <QPS or batch volume>
+- **Memory budget:** Model: <X>MB | Features: <X>MB | Instances: <N>
+- **Compute:** GPU | CPU — <constraints>
+- **Existing infrastructure:**
+  - Model registry: <exists | needs setup — details>
+  - Feature store: <exists | needs setup | N/A>
+  - Serving: <framework or "needs design">
+  - Orchestration: <tool and cadence>
+  - Monitoring: <exists | needs setup>
+- **Data freshness at inference:** <real-time | <X> minutes | <X> hours | daily>
+- **Fallback strategy:** <rule-based default | cached predictions | graceful degradation | TBD>
+- **Compliance / fairness:** <constraints or "none identified">
+- **Data Engineer consultation:**
+  - <summary of pipeline feasibility findings>
+```
+
+::GATE:: id=ml-engineer-phase-2 phase=2 kind=phase
+Read this section back to the user. Stop here — do not begin the next phase or output any further content. Wait for the user to explicitly confirm before proceeding. Do not interpret silence or partial agreement as confirmation.
+::ENDGATE::
+
+---
+
+## When this gate is confirmed
+
+Read `.claude/agents/specific_instructions/ml_engineer/phases/phase-3.md` in full and follow its instructions starting from Phase 3. Do not pre-read further phase files.

package/src/agents/specific_instructions/ml_engineer/phases/phase-3.md

@@ -0,0 +1,124 @@
+> **Previous:** phase-2.md confirmed
+> **Next:** phase-4.md (read only after this phase's gate is confirmed)
+
+---
+
+## Phase 3 — Data and Feature Discovery
+
+Goal: Understand what data is available for features and labels.
+
+**If productionization from study:** The Data Scientist has already completed feature
+discovery. Start by reading the study's Phase 2 (Data Discovery) and Phase 4 (Modeling
+Approach) from the study's `project-specs.md`. Present the inherited feature set to the
+user, then focus this phase on the **production-specific gap**:
+- Which study features are available at the required inference latency?
+- Which features need real-time alternatives or pre-computation?
+- Are there features the study used that cannot be productionized?
+Still consult the Data Modeller, but scope the consultation to serving-time data
+availability rather than full discovery.
+
+**Otherwise (greenfield or iteration):** proceed as below.
+
+**Consult the Data Modeller:**
+
+Tell the user: "Pulling in the Data Modeller. Feature definitions have to be grounded in actual data models, not what we hope exists."
+
+```
+Task(
+  subagent_type="data-modeller",
+  description="Explore data model for ML features",
+  prompt="I am the ML Engineer shard building an ML system for [purpose]. I need to
+  understand the data models that could source features for this system. Specifically:
+  1. What tables capture [relevant entities and events]?
+  2. What's the grain and freshness of each?
+  3. How do they relate to each other (join keys, cardinality)?
+  4. Any data quality concerns?
+  5. Which tables are available in real-time vs. batch only?
+  Focus on: [specific entities, events, or business concepts].
+  Since I'll be building feature extraction queries against these tables, please run
+  grain validation (PK uniqueness checks) and freshness checks on the key tables."
+)
+```
+
+**Greenfield handling:** Applies to greenfield and iteration projects only. If this is
+a productionization from a study, skip — the study is the data source.
+
+For greenfield and iteration: check whether the Data Modeller's response contains
+"NO DATA ENVIRONMENT DETECTED".
+
+If it does:
+1. Present the Data Modeller's response to the user.
+2. Ask:
+   "The Data Modeller found no data assets in this project. For an ML system, data
+   is the foundation of every feature and training decision.
+   - (a) Feature data exists in your warehouse — tell me what entities and events
+     are available. I'll design feature extraction from there.
+   - (b) Data exists but schema details aren't available right now — I can design
+     the feature architecture and model approach; actual queries and training will wait.
+   - (c) No data exists yet — I can produce a full ML architecture design, but
+     nothing will train or serve real predictions until data is available.
+   Which situation are we in?"
+3. Wait for the user's response before proceeding.
+   - (a): proceed with provided context.
+   - (b): proceed with caveats. Flag feature availability column in Phase 3 docs as
+     "Unverified — user-described." Add:
+     `**Data environment:** Feature data exists but inaccessible — candidates user-described, not verified.`
+   - (c): tell the user: "This will be an ML architecture design document. I can
+     define feature requirements, label definition, model architecture, and
+     infrastructure design — but the model cannot train and feature queries cannot
+     run until data exists. All feature candidates will be flagged
+     [THEORETICAL — DATA NOT AVAILABLE]. Do you want to proceed on that basis?"
+     Wait for confirmation. Add:
+     `**Data environment:** GREENFIELD — No data assets detected. Theoretical ML design only.`
+
+Present findings, then ask:
+- **Label definition:** How is the target variable defined? Where does ground truth come from?
+  Is there label delay (e.g., churn only observable 90 days later)?
+- **Feature candidates:** What signals could predict the target? Group by:
+  - User/entity attributes (demographic, account-level)
+  - Behavioral features (engagement, usage patterns, recency/frequency/monetary)
+  - Contextual features (time of day, device, location)
+  - Interaction features (user x item, user x content)
+
+  For each group, also propose 1-2 **novel derived candidates** — e.g., ratios between signals, recency-weighted aggregations, behavioral sequences, or domain-specific composites not available as raw columns. These should be presented alongside standard features with a note on engineering cost.
+- **Feature availability at inference:** For each feature group, is it available
+  at the latency required for serving?
+- **Historical depth:** How far back does the data go? Is it sufficient for training?
+- **Known biases:** Selection bias, survivorship bias, feedback loops
+
+### Document Phase 3
+
+```markdown
+---
+
+## Phase 3: Data and Feature Discovery (ML Engineer)
+- **Data Modeller consultation:**
+  - <summary of data model findings>
+- **Label definition:**
+  - Target: <variable name and definition>
+  - Ground truth source: <table or event>
+  - Label delay: <duration or "none">
+  - Label quality concerns: <issues or "none">
+- **Feature candidates:**
+  | Feature Group | Examples | Source Table(s) | Available at Inference? |
+  |--------------|---------|-----------------|----------------------|
+  | Entity attributes | <examples> | <tables> | Yes — batch | Yes — real-time | No |
+  | Behavioral | <examples> | <tables> | Yes — batch | Yes — real-time | No |
+  | Contextual | <examples> | <tables> | Yes — real-time | No |
+  | Interaction | <examples> | <tables> | Yes — batch | No |
+- **Historical depth:** <time range available>
+- **Known biases:**
+  - <bias type>: <description and mitigation>
+- **Feature-serving gap:** <features available in batch but not real-time, and impact>
+- **Data environment:** <not greenfield | Feature data exists but inaccessible — candidates user-described, not verified | GREENFIELD — No data assets detected. Theoretical ML design only>
+```
+
+::GATE:: id=ml-engineer-phase-3 phase=3 kind=phase
+Read this section back to the user. Stop here — do not begin the next phase or output any further content. Wait for the user to explicitly confirm before proceeding. Do not interpret silence or partial agreement as confirmation.
+::ENDGATE::
+
+---
+
+## When this gate is confirmed
+
+Read `.claude/agents/specific_instructions/ml_engineer/phases/phase-4.md` in full and follow its instructions starting from Phase 4. Do not pre-read further phase files.

package/src/agents/specific_instructions/ml_engineer/phases/phase-4.md

@@ -0,0 +1,279 @@
+> **Previous:** phase-3.md confirmed
+> **Next:** phase-5.md (read only after this phase's gate is confirmed)
+
+---
+
+## Phase 4 — Model Design
+
+Goal: Choose the model architecture, baselines, and candidate approaches.
+
+**If productionization from study:** The Data Scientist has already validated the
+model design from a statistical perspective. Start by reading the study's Phase 4
+(Modeling Approach) and Phase 6 (Build Log) from the study's `project-specs.md`.
+The study's candidate model is the starting point, not a blank slate. Focus this phase on:
+- Can the study's best model meet serving constraints (latency, memory, size)?
+- If not, what production-friendly alternatives achieve acceptable performance?
+- What's the production baseline? (the study's baseline may differ from a production baseline)
+Still consult the Data Scientist via Task, but frame the review as "production adaptation
+review" rather than full methodology review.
+
+**Otherwise (greenfield or iteration):** proceed as below.
+
+Ask about:
+- **Model type preferences:** Any organizational preferences or existing frameworks?
+  (scikit-learn, XGBoost, LightGBM, PyTorch, TensorFlow, etc.)
+- **Interpretability vs. performance trade-off:** Where does this sit?
+- **Ensemble acceptable?** Or must it be a single model for serving simplicity?
+- **Online learning needed?** Or batch retrain is sufficient?
+
+**Feature Registry check.** If `.shards/knowledge/features/` exists, scan for features
+whose domain tags overlap with this project's data domain (from Phase 3). For each
+relevant feature, present to the user with its SQL snippet, grain, and verification
+metadata. Ask: "Would you like to import any of these into your feature candidates?"
+
+If imported, note in feature candidates list: `(imported from Knowledge Ledger —
+verified by <agent> in <source_project>)`
+
+**Consult the Data Scientist** for methodology review:
+
+Tell the user: "Asking the Data Scientist to review the modeling approach. Statistical rigor isn't optional."
+
+
+```
+Task(
+  subagent_type="data-scientist",
+  description="Review ML model design for [project]",
+  prompt="I am the ML Engineer shard designing an ML system for [purpose].
+  Here is the model design:
+  - Task: [classification/regression/ranking/etc.]
+  - Target: [definition]
+  - Features: [summary of feature groups]
+  - Baseline: [proposed baseline]
+  - Candidates: [proposed candidate models]
+  - Evaluation: [proposed metrics]
+  Please review from a statistical and methodological perspective:
+  1. Is the target definition sound? Any leakage risk?
+  2. Are the evaluation metrics appropriate for the business problem?
+  3. Are there methodological concerns (confounding, bias, train/test contamination)?
+  4. Would you suggest a different approach or additional baselines?
+  Keep the review focused — I'll handle the systems/infrastructure side."
+)
+```
+
+**Consult the Applied ML Scientist** for methodology review. This is the default —
+AMS reviews the ML science of the proposed design (problem formulation, inductive
+bias alignment, loss function choice, evaluation metric soundness) in every Build
+project. Skip ONLY if ALL of the following hold:
+- Pure tabular data with standard feature types
+- Standard sklearn/XGBoost/LightGBM model family
+- Standard loss (MSE, log loss, cross-entropy)
+- Standard evaluation metrics (AUC, RMSE, accuracy, precision/recall)
+- No custom objectives, no architecture search, no self-supervised components
+
+If any of those criteria fail — or you're unsure — consult AMS. Non-tabular data
+structures (sequences, graphs, point clouds, images), custom objectives,
+architecture search, self-supervised pretraining, multi-task learning, or user
+requests for novel approaches all require AMS review.
+
+Tell the user: "Pulling in the Applied ML Scientist to review the methodology —
+problem formulation, inductive bias, objective alignment. Science review before
+we commit to this design."
+
+If skipping, state explicitly: "Skipping Applied ML Scientist review — this is
+standard tabular <problem type> with <model family> and <metric>. No cutting-edge
+methodology in scope." Document the skip rationale in the phase output.
+
+```
+Task(
+  subagent_type="applied-ml-scientist",
+  description="ML methodology review for <system type>",
+  prompt="I am the ML Engineer shard designing a <system>. The proposed approach is:
+  - Task type: <classification | regression | ranking | etc.>
+  - Data: <modality, scale, key characteristics>
+  - Proposed model: <architecture or approach>
+  - Objective: <loss function / evaluation metric>
+  - Constraints: <latency, memory, compute budget, interpretability>
+  - Business goal: <what the model output drives>
+
+  Please review and flag:
+  1. Is the problem formulated correctly as an ML problem?
+  2. Is there a significant mismatch between the architecture and data structure?
+  3. Are there methods from recent literature that would clearly outperform the
+     proposed approach for this specific problem?
+  4. Any red flags on the loss function or evaluation metric?
+
+  Context: <key constraints and goals from Phases 1-3>."
+)
+```
+
+Apply the Reviewer Verdict Protocol (see shared protocol — `applied-ml-scientist` row).
+
+**If the candidate model involves deep learning** — neural networks for image,
+text, audio, point cloud, or graph data, transformer variants, CNNs, RNNs, or any
+multi-layer neural approach — consult the Deep Learning Engineer:
+
+Tell the user: "This involves deep learning — I'm asking the Deep Learning Engineer
+shard to review architecture–data alignment, memory footprint, and inference
+feasibility..."
+
+```
+Task(
+  subagent_type="deep-learning-engineer",
+  description="DL architecture and production feasibility review for <project>",
+  prompt="I am the ML Engineer shard designing an ML system. I need a deep learning
+  architecture and production feasibility review.
+
+  - Task type: <classification | regression | ranking | generation | etc.>
+  - Data modality: <image | text | audio | point cloud | graph | tabular | multi-modal>
+  - Proposed architecture: <name or description>
+  - Input/output shapes: <input tensor shape> → <output tensor shape>
+  - Data scale: <N training examples, sequence length or spatial dims>
+  - Hardware: <GPU, VRAM, inference latency budget>
+  - Model size budget: <parameter ceiling or 'unconstrained'>
+  - Business goal: <what the model output drives>
+
+  Please review:
+  1. Is there a mismatch between the proposed architecture and the data structure
+     (inductive bias argument)?
+  2. Does the architecture fit the stated hardware constraints (VRAM, latency)?
+  3. Are there implementation concerns (numerical instability, known failure modes
+     for this architecture class at this data scale)?
+  4. Are there superior architectures from recent literature for this exact
+     problem type that would be worth considering before committing?
+
+  Context: <key constraints and goals from Phases 1-3>."
+)
+```
+
+Apply the Reviewer Verdict Protocol (see shared protocol — `deep-learning-engineer` row).
+
+**If Interpretability is High — consult the Data Analyst:**
+
+Tell the user: "Looping in the Data Analyst — they need to validate that these features make business sense before we serve them."
+
+```
+Task(
+  subagent_type="data-analyst",
+  description="Review feature candidates for business sense and interpretability",
+  prompt="I am the ML Engineer shard building an ML system for [purpose]. High
+  interpretability has been flagged as a requirement. Please review the feature
+  candidates to confirm they make business sense for this problem.
+
+  Feature candidates: [summary of feature groups from Phase 3]
+  Target variable: [name and definition]
+  End users of model outputs: [from Phase 1 — internal system | customer-facing | analyst | API consumer]
+  Business problem: [from Phase 1]
+  Cost of wrong predictions: [false positive / false negative impact, from Phase 1]
+
+  Please review:
+  1. Do these features align with how the business understands this problem?
+  2. Are there features that are technically valid but hard to explain to [end users]?
+  3. Are there obvious business-meaningful features that appear missing?
+  4. Any features that could undermine trust in the model if surfaced via SHAP or
+     feature importance to stakeholders?
+  Focus on interpretability and business alignment — I'll handle the systems side."
+)
+```
+
+Apply the Reviewer Verdict Protocol (see shared protocol — `data-analyst` row).
+
+**If the evaluation involves statistical inference** — A/B testing, confidence
+intervals, power analysis, significance testing, or experiment design for
+online evaluation — consult the Researcher:
+
+Tell the user: "The evaluation plan involves statistical inference — I'm asking
+the Researcher shard to validate the methodology before we commit to it."
+
+```
+Task(
+  subagent_type="researcher",
+  description="Review statistical inference methodology for ML evaluation",
+  prompt="I am the ML Engineer shard designing the evaluation strategy for an
+  ML system: [description].
+  Here is the proposed evaluation approach:
+  - Task: [what the model does]
+  - Offline metrics: [list]
+  - Online evaluation plan: [A/B test design, shadow mode, etc.]
+  - Sample size / traffic split: [N or %]
+  - Statistical test planned: [t-test, chi-squared, bootstrap, etc. or 'TBD']
+  - Confidence level: [95%, 99%, etc. or 'TBD']
+  Please review from a statistical methodology perspective:
+  1. Is the proposed statistical test appropriate for this metric type?
+  2. Is the sample size / traffic split adequate for the expected effect size?
+  3. Is the experiment design sound (randomization, control, duration)?
+  4. Are there multiple comparison issues or other statistical pitfalls?
+  5. What power analysis would you recommend?
+  Keep the review focused on statistical inference methodology."
+)
+```
+
+Apply the Reviewer Verdict Protocol (see shared protocol — `researcher` row).
+
+Define:
+- **Baseline model:** Simple, fast, interpretable. The floor to beat.
+  (logistic regression, decision tree, popularity-based, rule-based)
+- **Candidate model(s):** What to try if baseline isn't sufficient.
+- **Evaluation strategy:**
+  - Offline metrics: the model metrics (AUC, RMSE, NDCG, MAP, precision@k, etc.)
+  - Online metrics: the business metrics (conversion, engagement, revenue)
+  - Validation approach: temporal split, k-fold, stratified, group-aware
+- **Model size estimate:** Approximate parameter count, serialized size
+- **Inference cost estimate:** CPU/GPU time per prediction, batch throughput
+
+### Document Phase 4
+
+```markdown
+---
+
+## Phase 4: Model Design (ML Engineer)
+- **Data Scientist review:**
+  - Verdict: Approved | Concerns raised
+  - Tier: Proceed | Proceed with caveats
+  - Notes: <summary of methodology review>
+  - Reviewer resolution: Approved | User override — <rationale>
+- **Applied ML Scientist review:** <summary if consulted> | Skipped — <skip rationale per the criteria above>
+  - Verdict: Sound | Consider Alternatives | Revise
+  - Tier: Proceed | Proceed with caveats | Halt
+  - Reviewer resolution: Approved | Approved on resubmit | User override — <rationale> | Project stopped
+- **Deep Learning Engineer review:** N/A — not a DL approach | <summary if consulted>
+  - Verdict: DEPLOY | OPTIMIZE | REDESIGN
+  - Tier: Proceed | Proceed with caveats | Halt
+  - Reviewer resolution: Approved | Approved on resubmit | User override — <rationale> | Project stopped
+- **Data Analyst feature review:** N/A — Interpretability not High | <summary>
+  - Verdict: Aligned | Concerns raised
+  - Tier: Proceed | Proceed with caveats
+  - Reviewer resolution: Approved | User override — <rationale>
+- **Researcher review:** N/A — no statistical inference in evaluation | <summary if consulted>
+  - Verdict: Sound | Concerns | Revise
+  - Tier: Proceed | Proceed with caveats | Halt
+  - Reviewer resolution: Approved | Approved on resubmit | User override — <rationale> | Project stopped
+- **Baseline model:**
+  - Type: <model type>
+  - Rationale: <why this baseline>
+  - Expected performance: <rough estimate>
+- **Candidate model(s):**
+  - <model 1>: <type, rationale, trade-offs>
+  - <model 2>: <type, rationale, trade-offs>
+- **Evaluation strategy:**
+  - Offline metrics: <list with business interpretation>
+  - Online metrics: <list — what to measure post-deploy>
+  - Validation: <temporal split | k-fold | stratified | group-aware — rationale>
+  - Minimum threshold: <metric > value — business justification>
+- **Interpretability approach:** <SHAP | LIME | feature importance | N/A>
+- **Model size estimate:** ~<N> parameters, ~<X>MB serialized
+- **Inference cost:** ~<X>ms per prediction on <CPU/GPU>
+- **Ensemble:** Yes — <strategy> | No — single model
+- **Online learning:** Yes — <strategy> | No — batch retrain
+- **Feature Registry check:** <N> relevant features found | No features found | N/A — no features directory
+  - Imported: <title(s)> | None
+```
+
+::GATE:: id=ml-engineer-phase-4 phase=4 kind=phase
+Read this section back to the user. Stop here — do not begin the next phase or output any further content. Wait for the user to explicitly confirm before proceeding. Do not interpret silence or partial agreement as confirmation.
+::ENDGATE::
+
+---
+
+## When this gate is confirmed
+
+Read `.claude/agents/specific_instructions/ml_engineer/phases/phase-5.md` in full and follow its instructions starting from Phase 5. Do not pre-read further phase files.

package/src/agents/specific_instructions/ml_engineer/phases/phase-5.md

@@ -0,0 +1,160 @@
+> **Previous:** phase-4.md confirmed
+> **Next:** phase-6.md (read only after this phase's gate is confirmed)
+
+---
+
+## Phase 5 — Infrastructure Design
+
+Goal: Design the ML infrastructure — training pipeline, serving, monitoring.
+
+**For greenfield projects**, design the full stack:
+
+1. **Training pipeline:**
+   - Feature extraction (SQL → feature store or flat files)
+   - Train/validation/test split strategy
+   - Training orchestration (Airflow DAG, manual, CI/CD triggered)
+   - Model artifact storage (model registry, S3, etc.)
+   - Hyperparameter tuning strategy (grid, random, Bayesian, manual)
+
+2. **Serving infrastructure:**
+   - Real-time: API framework (FastAPI, Flask, gRPC), containerization, scaling
+   - Batch: scheduled job, output format and destination
+   - Feature serving: pre-computed features, real-time feature computation, caching
+   - Model loading: cold start time, warm-up, model versioning
+
+3. **Monitoring:**
+   - Model performance monitoring (prediction distribution drift, feature drift)
+   - Data quality monitoring (missing features, schema changes, staleness)
+   - System monitoring (latency, error rate, throughput, memory)
+   - Alerting thresholds and escalation paths
+   - Retraining triggers (scheduled, performance-based, drift-based)
+
+4. **Rollback and safety:**
+   - A/B testing or shadow mode plan
+   - Rollback procedure (previous model version, rule-based fallback)
+   - Circuit breaker logic
+
+**Consult the Data Engineer** for pipeline design review:
+
+Tell the user: "Getting the Data Engineer shard to review the pipeline architecture. I need to know if the orchestration and scheduling actually fits before this is finalized."
+
+```
+Task(
+  subagent_type="data-engineer",
+  description="Review ML pipeline infrastructure design for [project]",
+  prompt="I am the ML Engineer shard. I've designed the data pipeline
+  infrastructure for project [project_name]. Here is the pipeline design:
+  - Training pipeline: [feature extraction method, orchestration tool, schedule]
+  - Feature serving: [pre-computed vs. real-time, caching strategy]
+  - Storage: [artifact storage, feature store, output destinations]
+  - Orchestration: [tool, DAG structure, scheduling, dependencies]
+  - Monitoring: [data quality checks, alerting, retraining triggers]
+  Please review from a data engineering perspective:
+  1. Does the orchestration design fit existing Airflow/dbt patterns?
+  2. Is the feature extraction pipeline feasible at the designed cadence?
+  3. Are the storage choices and capacity realistic?
+  4. Does this integrate cleanly with existing data infrastructure?
+  5. Any scheduling, dependency, or resource concerns?
+  Keep the review focused and practical — I'll handle the ML-specific concerns."
+)
+```
+
+**Join path self-check (pipeline design):** Before requesting the Data Modeller
+review, trace the join path for each join in the feature pipeline following
+`.claude/agents/specific_instructions/shared/join_path_protocol.md`. Present the
+trace to the user. Include it in the DM prompt below. Grain errors in feature
+extraction become training set contamination.
+
+**Consult the Data Modeller** for pipeline data correctness review with validation:
+
+Tell the user: "Pulling in the Data Modeller to verify the pipeline design against the actual data model. Grain errors here become training errors later."
+
+```
+Task(
+  subagent_type="data-modeller",
+  description="Review ML pipeline data correctness for [project]",
+  prompt="I am the ML Engineer shard. I've designed the data pipeline for project
+  [project_name]. The project specs are at: [services|<existing_dir>]/[project_name]/project-specs.md
+
+  Here is how data flows through the pipeline:
+  - Feature extraction sources: [source tables and how they're used]
+  - Join strategy in pipeline: [key joins, grain at each stage]
+  - Pipeline stages: [ETL steps from source to training-ready dataset]
+  - Feature refresh cadence: [how often each feature group is refreshed]
+  - Serving-time data flow: [how features reach the model at inference]
+
+  Please REVIEW (not just explore) from a data model perspective:
+  1. Are the source table choices correct for these features?
+  2. Is grain handled correctly at each pipeline stage?
+  3. Are the join strategies sound given the entity relationships?
+  4. Does the data freshness cadence align with the model's requirements?
+
+  Run validation queries to verify:
+  1. PK uniqueness on each source table I'm using (confirm stated grain)
+  2. Null rates on join keys and critical feature columns
+  3. Join fan-out: run row counts before/after the key joins in my pipeline
+  4. Data freshness on each source table
+
+  Cross-reference results against the project requirements in project-specs.md
+  (especially Phase 3 feature candidates and Phase 2 data freshness requirements).
+  Keep the review focused on data correctness — the Data Engineer is reviewing
+  orchestration and infrastructure separately.
+  Return your full review with query validation results."
+)
+```
+
+Address any concerns raised by either review before finalizing the infrastructure design.
+
+**For iteration projects**, focus on what's changing and ensure backward compatibility.
+
+### Document Phase 5
+
+```markdown
+---
+
+## Phase 5: Infrastructure Design (ML Engineer)
+- **Training pipeline:**
+  - Feature extraction: <method and location>
+  - Split strategy: <temporal | random | stratified — rationale>
+  - Orchestration: <tool and schedule>
+  - Artifact storage: <registry or location>
+  - HPO strategy: <grid | random | Bayesian | manual>
+- **Serving infrastructure:**
+  - Mode: <real-time API | batch | streaming>
+  - Framework: <FastAPI | Flask | gRPC | batch job>
+  - Feature serving: <pre-computed | real-time | cache — TTL>
+  - Model loading: cold start ~<X>s, model version strategy
+  - Scaling: <horizontal | vertical — triggers>
+- **Monitoring:**
+  - Model perf: <metrics tracked, drift detection method>
+  - Data quality: <what's monitored>
+  - System: <latency, error rate, throughput — dashboards>
+  - Alerting: <thresholds and escalation>
+  - Retraining trigger: <scheduled | performance-based | drift-based — threshold>
+- **Rollback plan:**
+  - A/B testing: <plan or "N/A">
+  - Shadow mode: <plan or "N/A">
+  - Rollback: <procedure>
+  - Fallback: <rule-based default | cached predictions | error response>
+- **Data Engineer pipeline review:**
+  - Verdict: Approved | Concerns raised
+  - Notes: <summary of infrastructure review>
+  - Issues addressed: <how resolved or "none raised">
+- **Data Modeller pipeline review:**
+  - Verdict: Approved | Concerns raised
+  - Notes: <summary of data correctness review>
+  - Issues addressed: <how resolved or "none raised">
+- **Iteration-specific (if applicable):**
+  - What's changing: <features | model | infra | all>
+  - Backward compatibility: <ensured | breaking — migration plan>
+```
+
+::GATE:: id=ml-engineer-phase-5 phase=5 kind=phase
+Read this section back to the user. Stop here — do not begin the next phase or output any further content. Wait for the user to explicitly confirm before proceeding. Do not interpret silence or partial agreement as confirmation.
+::ENDGATE::
+
+---
+
+## When this gate is confirmed
+
+Read `.claude/agents/specific_instructions/ml_engineer/phases/phase-6.md` in full and follow its instructions starting from Phase 6. Do not pre-read further phase files.