wicked-garden 12.27.0

package/skills/data/pipeline/refs/templates.md

@@ -0,0 +1,175 @@
+---
+phase_relevance: ["design", "build"]
+archetype_relevance: ["*"]
+---
+# Pipeline Output Templates
+
+Standard templates for pipeline design and review documentation.
+
+## Pipeline Design Document
+
+```markdown
+## Pipeline Design: {name}
+
+**Author**: {name}
+**Date**: {date}
+**Status**: {draft/review/approved}
+
+### Requirements
+- **Source**: {system/database/API}
+- **Target**: {destination system}
+- **Frequency**: {hourly/daily/streaming}
+- **Latency SLA**: {max delay from source}
+- **Volume**: {GB/day or events/sec}
+- **Retention**: {how long to keep data}
+
+### Architecture
+
+```
+┌──────────┐    ┌───────────┐    ┌───────────┐    ┌──────────┐
+│  Source  │───►│  Extract  │───►│ Transform │───►│  Target  │
+└──────────┘    └───────────┘    └───────────┘    └──────────┘
+                     │                 │
+                     ▼                 ▼
+              ┌───────────┐    ┌───────────┐
+              │  Staging  │    │  Errors   │
+              └───────────┘    └───────────┘
+```
+
+### Data Flow
+
+#### 1. Extract
+- **Method**: {full/incremental/CDC}
+- **Source format**: {JSON/CSV/SQL}
+- **Volume**: {rows/day}
+- **Watermark**: {how to track progress}
+
+#### 2. Transform
+| Step | Operation | Input | Output |
+|------|-----------|-------|--------|
+| 1 | Parse | raw JSON | typed records |
+| 2 | Validate | typed records | valid records + errors |
+| 3 | Enrich | valid records | enriched records |
+| 4 | Aggregate | enriched records | summary tables |
+
+#### 3. Validate
+- [ ] Schema validation
+- [ ] Business rule validation
+- [ ] Referential integrity
+- [ ] Data quality checks
+
+#### 4. Load
+- **Pattern**: {append/merge/replace}
+- **Partitioning**: {by date/region/etc}
+- **Indexing**: {primary key, indexes}
+
+### Error Handling
+
+| Error Type | Detection | Response | Recovery |
+|------------|-----------|----------|----------|
+| Network | Connection timeout | Retry 3x | Alert if persists |
+| Data format | Schema validation | Log + skip | Manual review |
+| Business rule | Validation check | Dead letter queue | Reprocess |
+| System | Exception | Alert + stop | Manual restart |
+
+### Monitoring
+
+| Metric | Target | Alert Threshold |
+|--------|--------|-----------------|
+| Row count | ±5% of avg | >10% deviation |
+| Duration | <2 hours | >3 hours |
+| Error rate | <0.1% | >1% |
+| Freshness | <4 hours | >6 hours |
+
+### Operations
+
+#### Backfill Procedure
+1. Set date range parameters
+2. Run in staging mode first
+3. Validate results
+4. Run production load
+
+#### Recovery Procedure
+1. Check error logs
+2. Fix root cause
+3. Replay from last checkpoint
+4. Verify data consistency
+
+### Dependencies
+| Dependency | Type | Owner | SLA |
+|------------|------|-------|-----|
+| {system} | upstream | {team} | {SLA} |
+
+### Deployment
+- **Orchestration**: {Airflow/Dagster/Prefect}
+- **Schedule**: {cron expression}
+- **Resources**: {CPU/memory/storage}
+- **Environments**: {dev/staging/prod}
+```
+
+## Pipeline Review Report
+
+```markdown
+## Pipeline Review: {name}
+
+**Reviewed**: {date}
+**Reviewer**: {name}
+
+### Summary
+{2-3 sentence assessment of pipeline quality and readiness}
+
+### Findings
+
+#### Critical (Must Fix)
+| # | Finding | Risk | Recommendation |
+|---|---------|------|----------------|
+| 1 | {issue} | Data loss | {action} |
+
+#### High (Should Fix)
+| # | Finding | Risk | Recommendation |
+|---|---------|------|----------------|
+| 1 | {issue} | Performance | {action} |
+
+#### Medium (Consider)
+| # | Finding | Risk | Recommendation |
+|---|---------|------|----------------|
+| 1 | {issue} | Maintenance | {action} |
+
+### Code Quality Assessment
+
+| Category | Score | Notes |
+|----------|-------|-------|
+| Structure | {1-10} | {comments} |
+| Error Handling | {1-10} | {comments} |
+| Testing | {1-10} | {comments} |
+| Documentation | {1-10} | {comments} |
+| Maintainability | {1-10} | {comments} |
+
+**Overall**: {score}/50
+
+### Performance Assessment
+
+- **Current load**: {X rows in Y time}
+- **Bottleneck**: {identified}
+- **10x projection**: {will it scale?}
+- **Recommendations**: {optimizations}
+
+### Operational Readiness
+
+- [ ] Monitoring configured
+- [ ] Alerting configured
+- [ ] Runbook documented
+- [ ] Backfill tested
+- [ ] Recovery tested
+- [ ] On-call rotation assigned
+
+### Verdict
+
+**Status**: [PRODUCTION READY | NEEDS WORK | MAJOR REFACTOR]
+
+**Blocking Issues**: {count}
+**Required Actions Before Deploy**:
+1. {action}
+
+**Sign-off**: {name, date}
+```

package/skills/data/refs/analyze.md

@@ -0,0 +1,123 @@
+# Data Analysis Rubric
+
+Apply this inline. Route by `--focus` to the right analysis mode. File has been
+pre-profiled (columns / types / nulls / sample rows) by the caller.
+
+## EDA output format (default / --focus stats)
+
+```markdown
+## Data Analysis Report
+
+**Dataset**: {name}
+**Analysis Date**: {date}
+
+### Executive Summary
+{2-3 key takeaways}
+
+### Quick Stats
+- **Rows**: {count}  **Columns**: {count}  **Date Range**: {start–end}
+
+### Initial Findings
+1. {Observable pattern or anomaly}
+2. {Interesting correlation or trend}
+
+### Visualization Recommendations
+- {Chart type}: {What it shows}
+
+### Next Steps / Questions Raised
+- {Follow-up analyses or data needs}
+
+**Confidence**: {HIGH|MEDIUM|LOW}
+```
+
+## Statistical analysis patterns
+
+**Distributions**: MIN / Q1 / MEDIAN / Q3 / MAX; histogram or box-plot.
+**Categorical**: frequency + percentage breakdown; bar chart.
+**Time series**: DATE_TRUNC day/week aggregation; line chart.
+**Correlations**: CORR(a, b) spot-check.
+**Cohort/RFM**: group by signup month; recency/frequency/monetary.
+
+## Insight pattern
+
+- **Observation**: What the data shows (fact).
+- **Insight**: Why it matters (interpretation).
+- **Action**: What to do (recommendation).
+- **Confidence**: HIGH | MEDIUM | LOW (state sample size + caveats).
+
+## --focus quality (data-quality mode)
+
+| Dimension | Check |
+|-----------|-------|
+| Completeness | Null rates per column |
+| Uniqueness | Duplicate detection |
+| Validity | Type conformance + constraints |
+| Consistency | Cross-field validation |
+| Timeliness | Freshness metrics |
+
+```markdown
+## Data Quality Report
+
+**Dataset**: {name}  **Rows**: {count}  **Columns**: {count}
+
+### Quality Metrics
+| Dimension | Score | Issues |
+|-----------|-------|--------|
+| Completeness | {%} | {null columns} |
+| Uniqueness  | {%} | {duplicate rate} |
+| Validity    | {%} | {constraint violations} |
+
+### Critical Issues
+- {Issue with severity and impact}
+
+### Recommendations
+1. {Prioritized action items}
+```
+
+## --focus warehouse (warehouse/lakehouse mode)
+
+Evaluate dimensional model fit: Star vs Snowflake vs Data Vault vs Wide Table.
+Cover partitioning / clustering / materialization strategy and cost plan.
+
+```markdown
+## Warehouse Analysis: {domain}
+
+### Approach
+Pattern: {Star|Snowflake|Data Vault|Wide Table}
+Justification: {why}
+
+### Schema
+#### Fact: {fact_table}
+| Column | Type | Description | Source |
+
+#### Dimension: {dim_table}
+| Column | Type | SCD Type | Description |
+
+### Performance
+- Partitioning: {strategy}   Clustering: {columns}   Materialization: {views vs tables}
+
+### Cost Plan
+{Storage tier, compute, optimization levers}
+```
+
+## --focus ml (ML-readiness mode)
+
+Assess training data quality for ML: leakage, imbalance, feature distributions,
+label quality. See refs/ml.md for full ML pipeline rubric.
+
+## Scenarios block (--scenarios)
+
+When `--scenarios` is set, append:
+```yaml
+scenarios:
+  api:
+    - name: "{dataset}_baseline"
+      given: "data file at {path}"
+      when: "analyze --focus {focus}"
+      then: "key finding X observed"
+  perf:
+    - name: "{dataset}_large_file"
+      given: "file with {N} rows"
+      when: "profiling runs"
+      then: "completes within threshold"
+```

package/skills/data/refs/data.md

@@ -0,0 +1,95 @@
+# Data Engineering Rubric — profile / validate / quality
+
+Apply this inline for schema-level checks. Caller has pre-read the data file
+head/tail and captured columns / types / nulls.
+
+## profile
+
+Emit a structured profile with shape, per-column stats, and prioritized findings.
+
+```markdown
+## Data Profile: {name}
+
+**Rows**: {count}  **Columns**: {count}  **File**: {path}
+
+### Column Summary
+| Column | Type | Null% | Unique | Min | Max | Sample |
+|--------|------|-------|--------|-----|-----|--------|
+
+### Key Observations
+1. {High null-rate columns}
+2. {Type anomalies or mixed types}
+3. {Skew / outlier signals}
+
+### Recommendations
+1. {Prioritized action — e.g. handle nulls in col X}
+
+**Confidence**: {HIGH|MEDIUM|LOW}
+```
+
+## validate
+
+Compare actual data against the provided schema. Report breaking vs non-breaking drift.
+
+```markdown
+## Schema Validation: {name}
+
+**Schema**: {schema-path}  **Data**: {data-path}
+
+### Result: {PASS|FAIL}
+
+### Breaking Issues (must fix)
+| Column | Expected | Actual | Impact |
+|--------|----------|--------|--------|
+
+### Warnings (non-breaking)
+| Column | Note |
+|--------|------|
+
+### Recommendations
+1. {Fix breaking issues first}
+
+**Confidence**: {HIGH|MEDIUM|LOW}
+```
+
+Schema design principles to apply:
+- Explicit types (avoid variant unless necessary).
+- Clear nullability contracts.
+- Consistent naming conventions.
+- Version schemas explicitly.
+- Document breaking vs non-breaking changes.
+
+## quality
+
+Five-dimension quality assessment: Completeness, Uniqueness, Validity, Consistency, Timeliness.
+
+```markdown
+## Data Quality Report
+
+**Dataset**: {name}  **Rows**: {count}  **Columns**: {count}
+
+### Quality Metrics
+| Dimension    | Score | Issues |
+|--------------|-------|--------|
+| Completeness | {%}   | {null columns} |
+| Uniqueness   | {%}   | {duplicate rate} |
+| Validity     | {%}   | {constraint violations} |
+| Consistency  | {%}   | {cross-field failures} |
+| Timeliness   | {age} | {freshness status} |
+
+### Critical Issues
+- {Issue with severity and impact}
+
+### Recommendations
+1. {Prioritized action items}
+
+**Confidence**: {HIGH|MEDIUM|LOW}
+```
+
+## General engineering checklist
+
+- [ ] Schema-first: define schemas before processing.
+- [ ] Fail fast: validate early, fail loudly.
+- [ ] Idempotent: pipelines rerunnable without side effects.
+- [ ] Observable: emit metrics + logs at every stage.
+- [ ] Tested: data quality tests are non-negotiable.

package/skills/data/refs/ml.md

@@ -0,0 +1,118 @@
+# ML Engineering Rubric — review / pipeline
+
+Apply this inline. Caller has pre-gathered model files (for review) or
+captured task-type (for pipeline design).
+
+## review
+
+```markdown
+## ML Engineering Assessment
+
+**Model**: {name}
+**Type**: {classification|regression|ranking|other}
+**Status**: {Development|Staging|Production}
+
+### Summary
+{2-3 sentence overview}
+
+### Architecture Review
+- **Problem Type**: {clear definition}
+- **Model Type**: {algorithm/architecture}
+- **Justification**: {why this approach}
+- **Alternatives Considered**: {other options}
+
+### Data & Features
+- **Training Size**: {rows}
+- **Feature Count**: {count}
+- **Quality**: {assessment}
+
+| Check | Status | Notes |
+|-------|--------|-------|
+| No data leakage (target in features) | {pass/fail} | |
+| Label quality | {pass/fail} | |
+| Class imbalance handled | {pass/fail} | |
+| Missing values handled | {pass/fail} | |
+| Categorical encoding defined | {pass/fail} | |
+| Feature importance tracked | {pass/fail} | |
+
+### Evaluation Strategy
+- **Metrics**: {precision, recall, RMSE, etc.}
+- **Validation**: {holdout / k-fold / time-based}
+- **Success Criteria**: {minimum acceptable performance}
+
+### Deployment Readiness
+- [ ] Inference latency acceptable
+- [ ] Model size appropriate for deployment target
+- [ ] Feature pipeline reproducible in production
+- [ ] Graceful fallback if model fails
+- [ ] A/B testing framework ready
+- [ ] Monitoring and logging instrumented
+- [ ] Rollback plan documented
+
+### Recommendations
+| Priority | Recommendation | Impact | Effort |
+|----------|----------------|--------|--------|
+| P1 | {critical item} | HIGH | {S/M/L} |
+
+**Recommendation**: {APPROVE|CONDITIONAL|NEEDS REWORK}
+**Confidence**: {HIGH|MEDIUM|LOW}
+```
+
+## pipeline (training pipeline design)
+
+Problem setup → data/feature stages → model arch → evaluation → tuning → deployment → monitoring.
+
+```markdown
+## Training Pipeline: {model_name}
+
+### Problem Statement
+- **Type**: {Classification|Regression|Ranking|Other}
+- **Target**: {what is being predicted}
+- **Business Metric**: {how success is measured}
+- **Baseline**: {minimum viable baseline established?}
+
+### Data Flow
+1. **Source**: {where data comes from}
+2. **Sampling**: {train/val/test split strategy — e.g. 70/15/15}
+3. **Features**: {transformation pipeline, feature store integration}
+4. **Training**: {model training config, hyperparameter search}
+5. **Validation**: {holdout / cross-validation approach}
+6. **Registry**: {versioning scheme, metadata, promotion criteria}
+
+### Configuration (sample)
+```yaml
+data:
+  source: {path}
+  train_split: 0.7  val_split: 0.15  test_split: 0.15
+features:
+  categorical: [col1, col2]
+  numerical: [col3, col4]
+  target: label
+model:
+  type: {algorithm}
+  key_params: {}
+evaluation:
+  metric: {f1_score|rmse|ndcg}
+  threshold: {value}
+```
+
+### Monitoring Plan
+- **Drift detection**: {feature/target drift strategy}
+- **Performance tracking**: {metrics logged}
+- **Retraining trigger**: {when to retrain}
+
+### MLOps Principles
+- **Reproducible**: version data, code, and models.
+- **Monitored**: track performance continuously.
+- **Automated**: CI/CD for ML pipelines.
+- **Tested**: unit tests for features, integration tests for pipeline.
+- **Documented**: model cards for transparency.
+```
+
+## Common pitfalls
+
+- Training on future data (leakage).
+- Overfitting to validation set.
+- Ignoring class imbalance.
+- Not testing inference performance.
+- Deploying without monitoring.

package/skills/data/refs/pipeline.md

@@ -0,0 +1,92 @@
+# Data Pipeline Rubric — design / review
+
+Apply this inline for ETL/streaming pipeline architecture.
+Caller has pre-captured source / target / frequency (for design) or read pipeline
+files at `<path>` (for review).
+
+## design
+
+Cover architecture pattern, orchestrator, stage flow, quality gates, monitoring, costs, risks.
+
+```markdown
+## Pipeline Design: {name}
+
+### Architecture
+- **Pattern**: {Batch|Streaming|Hybrid}
+- **Orchestration**: {Airflow|Dagster|Prefect|Other}
+- **Storage**: {Data Lake|Warehouse|Lakehouse}
+
+### Data Flow
+1. **Source**: {description — type, API/DB/file, credentials approach}
+2. **Extract**: {method and frequency}
+3. **Transform**: {key transformations}
+4. **Load**: {destination and format}
+
+### Quality Gates
+- **Source validation**: {checks before processing}
+- **Transform validation**: {checks mid-pipeline}
+- **Load validation**: {row counts, key checks post-load}
+
+### Performance
+- **Expected volume**: {records/day}
+- **Processing time estimate**: {duration}
+- **Cost estimate**: {$/month}
+
+### Risk Assessment
+| Level | Risk | Mitigation |
+|-------|------|------------|
+| HIGH  | {critical risk} | {mitigation} |
+| MED   | {moderate risk} | {mitigation} |
+
+### Design Checklist
+- [ ] Data sources identified and accessible
+- [ ] Schema evolution strategy defined
+- [ ] Error handling and retry logic designed
+- [ ] Idempotency and reprocessing supported
+- [ ] Monitoring and alerting plan
+- [ ] Data quality checks embedded
+- [ ] Cost estimation completed
+
+**Confidence**: {HIGH|MEDIUM|LOW}
+```
+
+## review
+
+Cover code quality, idempotency, monitoring, validation, silent-loss risk.
+Emit P1/P2/P3 findings with code fixes.
+
+```markdown
+## Pipeline Review: {path}
+
+**Architecture**: {summary — pattern, orchestrator, storage}
+**Quality Score**: {score}/100
+
+### Findings
+| Priority | Finding | Impact | Effort | Fix |
+|----------|---------|--------|--------|-----|
+| P1 | {critical} | HIGH | {S/M/L} | {code snippet} |
+| P2 | {moderate} | MED  | {S/M/L} | {code snippet} |
+| P3 | {minor}    | LOW  | {S/M/L} | {note} |
+
+### Review Checklist
+- [ ] Error handling: failures managed and surfaced
+- [ ] Schema versioned and enforced
+- [ ] Monitoring: what metrics are tracked
+- [ ] Testing: data quality tests present
+- [ ] Idempotent: safe to reprocess without duplication
+- [ ] Silent-loss risk: no data drops without alert
+- [ ] Documentation: pipeline well-documented
+
+### Recommendations
+1. {Action with rationale and expected outcome}
+
+**Confidence**: {HIGH|MEDIUM|LOW}
+```
+
+## Engineering standards (apply to both)
+
+- **Schema-first**: always define schemas before processing.
+- **Fail fast**: validate early, fail loudly.
+- **Idempotent**: pipelines rerunnable.
+- **Observable**: emit metrics + logs at every stage.
+- **Tested**: data quality tests are non-negotiable.