wicked-garden 12.27.0

package/skills/data/data/refs/examples.md

@@ -0,0 +1,235 @@
+---
+phase_relevance: ["design", "build"]
+archetype_relevance: ["*"]
+---
+# Data Engineering Examples
+
+Detailed examples for data profiling, schema validation, and quality assessment.
+
+## Profiling Output Example
+
+```json
+{
+  "file": "sales.csv",
+  "rows": 10000,
+  "columns": 8,
+  "schema": [
+    {
+      "name": "order_id",
+      "type": "string",
+      "null_rate": 0.0,
+      "cardinality": 10000,
+      "sample": ["ORD001", "ORD002", "ORD003"]
+    },
+    {
+      "name": "amount",
+      "type": "decimal",
+      "null_rate": 0.02,
+      "stats": {
+        "min": 10.50,
+        "max": 9999.99,
+        "mean": 523.45,
+        "median": 299.00
+      }
+    },
+    {
+      "name": "customer_id",
+      "type": "integer",
+      "null_rate": 0.0,
+      "cardinality": 5000
+    },
+    {
+      "name": "order_date",
+      "type": "date",
+      "null_rate": 0.0,
+      "min": "2025-01-01",
+      "max": "2025-12-31"
+    }
+  ],
+  "quality_score": 95,
+  "issues": [
+    "Column 'email' has 2% null values"
+  ]
+}
+```
+
+## Schema Definition Example
+
+```json
+{
+  "columns": [
+    {
+      "name": "user_id",
+      "type": "integer",
+      "nullable": false,
+      "constraints": {
+        "unique": true,
+        "min": 1
+      }
+    },
+    {
+      "name": "email",
+      "type": "string",
+      "nullable": false,
+      "constraints": {
+        "pattern": "^[\\w\\.-]+@[\\w\\.-]+\\.\\w+$"
+      }
+    },
+    {
+      "name": "created_at",
+      "type": "datetime",
+      "nullable": false
+    },
+    {
+      "name": "status",
+      "type": "string",
+      "nullable": false,
+      "constraints": {
+        "enum": ["active", "inactive", "pending"]
+      }
+    },
+    {
+      "name": "balance",
+      "type": "decimal",
+      "nullable": true,
+      "constraints": {
+        "min": 0,
+        "precision": 2
+      }
+    }
+  ]
+}
+```
+
+## Quality Report Example
+
+```markdown
+## Data Quality Report
+
+**Dataset**: sales.csv
+**Date**: 2026-01-24
+**Rows**: 10,000
+
+### Overall Score: 87/100
+
+### Dimension Scores
+| Dimension | Score | Status |
+|-----------|-------|--------|
+| Completeness | 95 | PASS |
+| Uniqueness | 98 | PASS |
+| Validity | 85 | WARNING |
+| Consistency | 70 | FAIL |
+
+### Critical Issues
+1. **Validity**: 15% of 'phone' values don't match pattern
+2. **Consistency**: 30% of records have order_date > ship_date
+
+### Issue Details
+
+#### Issue 1: Invalid Phone Format
+- **Column**: phone
+- **Pattern**: Expected `^\\+?[0-9]{10,14}$`
+- **Violations**: 1,500 rows (15%)
+- **Samples**: "123-456", "N/A", "(555)123-4567"
+- **Root Cause**: No input validation at data entry
+
+#### Issue 2: Date Consistency
+- **Rule**: ship_date >= order_date
+- **Violations**: 3,000 rows (30%)
+- **Impact**: Analytics reports show negative fulfillment times
+- **Root Cause**: Timezone mismatch between systems
+
+### Recommendations
+1. Add validation for phone format at source
+2. Implement business rule: ship_date >= order_date
+3. Standardize timezone handling
+
+**Action Required**: Fix consistency issues before using for analytics
+```
+
+## Common Workflows
+
+### New Dataset Analysis
+
+1. **Profile** to understand structure
+```bash
+/wicked-garden:data:data profile new_data.csv
+```
+
+2. **Validate** against expected schema (if available)
+```bash
+/wicked-garden:data:data validate --schema expected.json --data new_data.csv
+```
+
+3. **Generate quality report**
+```bash
+/wicked-garden:data:data quality new_data.csv
+```
+
+4. **Document issues** via native TaskCreate
+```
+TaskCreate(
+  subject="Data Quality: Fix phone format",
+  metadata={
+    "event_type": "task",
+    "chain_id": "data-quality.root",
+    "source_agent": "data-profiler",
+    "priority": "P1"
+  }
+)
+```
+
+5. **Recommend remediation** with specific actions
+
+### Schema Migration
+
+1. Profile source data
+2. Design target schema based on profile
+3. Validate sample data against target
+4. Identify transformation requirements:
+   - Type conversions
+   - Null handling
+   - Constraint additions
+5. Document migration plan
+
+### Quality Monitoring
+
+1. Profile dataset regularly (daily/weekly)
+2. Compare metrics over time
+3. Alert on quality degradation:
+   - Null rate increase >5%
+   - New duplicate patterns
+   - Type conformance drops
+4. Track improvement initiatives via native TaskCreate
+
+## SQL Quality Queries
+
+```sql
+-- Completeness check
+SELECT
+  column_name,
+  COUNT(*) as total,
+  SUM(CASE WHEN column_name IS NULL THEN 1 ELSE 0 END) as nulls,
+  ROUND(100.0 * SUM(CASE WHEN column_name IS NULL THEN 1 ELSE 0 END) / COUNT(*), 2) as null_pct
+FROM data
+GROUP BY ALL;
+
+-- Uniqueness check
+SELECT
+  column_name,
+  COUNT(*) as total,
+  COUNT(DISTINCT column_name) as distinct_count,
+  COUNT(*) - COUNT(DISTINCT column_name) as duplicates
+FROM data
+GROUP BY ALL;
+
+-- Validity check (pattern matching)
+SELECT
+  COUNT(*) FILTER (WHERE email NOT LIKE '%@%.%') as invalid_emails
+FROM data;
+
+-- Consistency check (cross-field rules)
+SELECT
+  COUNT(*) FILTER (WHERE ship_date < order_date) as date_violations
+FROM data;
+```

package/skills/data/ml/SKILL.md

@@ -0,0 +1,125 @@
+---
+name: ml
+description: |
+  This skill should be used when working with machine learning models — architecture review,
+  training pipeline design, feature engineering, and deployment guidance.
+
+  Use when:
+  - "review this ML model"
+  - "design ML training pipeline"
+  - "how should I deploy this model"
+  - "feature engineering advice"
+  - "ML architecture guidance"
+phase_relevance: ["design", "build"]
+archetype_relevance: ["*"]
+---
+
+# ML Engineering Skill
+
+Guide machine learning model development, training, and deployment.
+
+## Quick Start
+
+### Review Model Architecture
+
+```bash
+/wicked-garden:data:ml review path/to/model/
+```
+
+Reviews: Model choice, training data quality, evaluation strategy, deployment readiness.
+
+### Design Training Pipeline
+
+```bash
+/wicked-garden:data:ml pipeline --type classification
+```
+
+Generates: Data loading, feature engineering, training config, evaluation framework.
+
+## ML Development Workflow
+
+### 1. Problem Definition
+
+- [ ] Problem type clear (classification/regression/ranking)
+- [ ] Success metric defined
+- [ ] Baseline established
+- [ ] Data availability confirmed
+- [ ] Inference requirements understood (latency/throughput)
+
+### 2. Data Assessment
+
+- [ ] Sufficient volume (>1000x features)
+- [ ] Labels accurate and consistent
+- [ ] Features available at inference time
+- [ ] No data leakage from target
+- [ ] Class balance acceptable
+
+### 3. Feature Engineering
+
+**Good features are**: Predictive, Available at inference, Clean (no leakage), Interpretable.
+
+**Common transformations**:
+- Numeric: Scaling, log transform
+- Categorical: One-hot, target encoding
+- Time: Extract components, cyclical encoding
+- Aggregations: Rolling windows, user stats
+
+### 4. Model Selection
+
+| Data Size | Structured | Recommendation |
+|-----------|------------|----------------|
+| <10K rows | Yes | Linear/Simple tree |
+| 10K-1M | Yes | GradientBoosting (XGBoost/LightGBM) |
+| >1M | Yes | Deep learning possible |
+| Any | Images/Text | Deep learning |
+
+### 5. Training & Evaluation
+
+**Split strategy**: Random (if i.i.d.), Time-based (if time series), Cross-validation (robust).
+
+**Key metrics**:
+- **Classification**: Accuracy, Precision, Recall, F1, AUC
+- **Regression**: RMSE, MAE, R²
+
+### 6. Hyperparameter Tuning
+
+- **Grid search**: Exhaustive, slow
+- **Random search**: More efficient
+- **Bayesian optimization**: Most efficient
+
+### 7. Deployment
+
+**Patterns**: Batch scoring, REST API, Streaming
+
+**Checklist**:
+- [ ] Inference latency acceptable
+- [ ] Model size appropriate
+- [ ] Feature pipeline reproducible
+- [ ] Monitoring instrumented
+- [ ] Rollback plan documented
+
+## ML Monitoring
+
+**Model Performance**: Prediction accuracy, distribution shifts, error rate by segment.
+
+**Data Quality**: Feature distributions, missing rates, cardinality changes.
+
+**System Health**: Latency (p50, p95, p99), throughput, memory.
+
+## Integration
+
+- **wicked-brain:search**: Find model code with `wicked-brain:search "model|classifier"` (FTS5 over indexed code)
+- **Native tasks**: Track experiments via TaskCreate with `metadata.event_type="task"`
+- **wicked-garden:data:analyze**: Analyze training data via DuckDB SQL
+
+## Best Practices
+
+- **Always baseline**: Start simple, measure improvement, justify complexity
+- **Avoid leakage**: Use only past data, split before processing
+- **Monitor production**: Track predictions, detect drift, plan retraining
+- **Document everything**: Architecture, features, training data, results
+
+## Reference
+
+For detailed techniques:
+- [Output Templates](refs/templates.md) - Model review report, model card

package/skills/data/ml/refs/templates.md

@@ -0,0 +1,172 @@
+---
+phase_relevance: ["design", "build"]
+archetype_relevance: ["*"]
+---
+# ML Output Templates
+
+Standard templates for ML model reviews and documentation.
+
+## Model Review Report
+
+```markdown
+## ML Model Review: {model_name}
+
+**Type**: {classification/regression/clustering/ranking}
+**Status**: {development/staging/production}
+**Reviewer**: {name}
+**Date**: {date}
+
+### Summary
+{2-3 sentence assessment of model readiness}
+
+### Assessment
+
+#### Data Quality: {score}/10
+- [ ] Sufficient volume (>1000x features)
+- [ ] Label quality verified
+- [ ] No data leakage from target
+- [ ] Class balance acceptable
+- [ ] Train/test split proper
+
+**Notes**: {observations}
+
+#### Feature Engineering: {score}/10
+- [ ] Features available at inference time
+- [ ] Proper scaling/encoding applied
+- [ ] No leakage from target
+- [ ] Feature importance analyzed
+- [ ] Missing value handling documented
+
+**Notes**: {observations}
+
+#### Model Choice: {score}/10
+- [ ] Appropriate for problem type
+- [ ] Baseline model established
+- [ ] Complexity justified by improvement
+- [ ] Alternatives considered and documented
+- [ ] Model interpretability assessed
+
+**Notes**: {observations}
+
+#### Evaluation: {score}/10
+- [ ] Train/test split strategy proper
+- [ ] Metrics aligned with business goal
+- [ ] Cross-validation performed
+- [ ] Performance acceptable for use case
+- [ ] Confidence intervals reported
+
+**Metrics**:
+| Metric | Train | Test | Target |
+|--------|-------|------|--------|
+| {metric} | {val} | {val} | {target} |
+
+**Notes**: {observations}
+
+#### Deployment Readiness: {score}/10
+- [ ] Inference latency acceptable (<{X}ms)
+- [ ] Model size appropriate
+- [ ] Feature pipeline reproducible
+- [ ] Monitoring instrumented
+- [ ] Rollback plan documented
+- [ ] A/B testing framework ready
+
+**Notes**: {observations}
+
+### Recommendations
+| Priority | Recommendation | Impact | Effort |
+|----------|----------------|--------|--------|
+| P1 | {action} | HIGH | {S/M/L} |
+| P2 | {action} | MEDIUM | {S/M/L} |
+| P3 | {action} | LOW | {S/M/L} |
+
+### Overall Assessment
+
+**Score**: {total}/50
+
+**Decision**: [APPROVE | CONDITIONAL APPROVE | NEEDS REWORK]
+
+**Conditions** (if applicable):
+1. {condition that must be met before approval}
+
+**Next Steps**:
+1. {immediate action}
+2. {follow-up action}
+```
+
+## Model Card
+
+```markdown
+# Model Card: {model_name}
+
+## Model Details
+- **Name**: {model_name}
+- **Version**: {version}
+- **Type**: {classification/regression/etc}
+- **Algorithm**: {RandomForest/XGBoost/NeuralNet/etc}
+- **Framework**: {sklearn/pytorch/tensorflow}
+- **Created**: {date}
+- **Owner**: {team/person}
+
+## Intended Use
+- **Primary use**: {description}
+- **Users**: {who will use this}
+- **Out of scope**: {what this should NOT be used for}
+
+## Training Data
+- **Source**: {data source}
+- **Size**: {rows} rows, {features} features
+- **Date range**: {start} to {end}
+- **Preprocessing**: {summary of transformations}
+
+## Evaluation
+| Metric | Value | Target |
+|--------|-------|--------|
+| {metric} | {val} | {target} |
+
+## Ethical Considerations
+- **Bias**: {known biases and mitigations}
+- **Fairness**: {fairness testing performed}
+- **Privacy**: {PII handling}
+
+## Limitations
+- {limitation 1}
+- {limitation 2}
+
+## Deployment
+- **Endpoint**: {URL/service}
+- **Latency**: p50={X}ms, p99={Y}ms
+- **Throughput**: {requests/sec}
+
+## Monitoring
+- **Dashboards**: {links}
+- **Alerts**: {configured alerts}
+- **Retraining trigger**: {criteria}
+```
+
+## Experiment Log
+
+```markdown
+## Experiment: {experiment_name}
+
+**Goal**: {what are we trying to achieve}
+**Baseline**: {current performance}
+**Target**: {goal performance}
+
+### Experiment 1: {description}
+**Changes**: {what was changed}
+**Results**:
+| Metric | Before | After | Change |
+|--------|--------|-------|--------|
+| {metric} | {val} | {val} | {+/-X%} |
+
+**Conclusion**: {SUCCESS/FAILED/INCONCLUSIVE}
+**Notes**: {observations}
+
+### Experiment 2: {description}
+...
+
+### Summary
+**Best configuration**: {description}
+**Final performance**: {metrics}
+**Next steps**: {what to try next or ship}
+```

package/skills/data/pipeline/SKILL.md

@@ -0,0 +1,136 @@
+---
+name: pipeline
+description: |
+  This skill should be used when designing or reviewing data pipelines — ETL patterns,
+  orchestration, and performance optimization for data workflows.
+
+  Use when:
+  - "design a data pipeline"
+  - "review this ETL"
+  - "optimize data processing"
+  - "how should I orchestrate this"
+  - "pipeline architecture"
+# TODO #339: When Claude Code supports 'paths' in skill frontmatter for
+# file-context auto-activation, add:
+#   paths: ["**/dags/**", "**/pipelines/**", "**/etl/**", "**/airflow/**", "**/prefect/**"]
+phase_relevance: ["design", "build"]
+archetype_relevance: ["*"]
+---
+
+# Pipeline Engineering Skill
+
+Design, review, and optimize data pipelines and ETL workflows.
+
+## Quick Start
+
+### Design a Pipeline
+
+```bash
+/wicked-garden:data:pipeline design \
+  --source "postgres://sales_db" \
+  --target "s3://data-lake/sales" \
+  --frequency daily
+```
+
+Generates: Architecture diagram, ETL logic, orchestration config, monitoring plan.
+
+### Review Existing Pipeline
+
+```bash
+/wicked-garden:data:pipeline review path/to/pipeline/
+```
+
+Analyzes: Code quality, error handling, performance, maintainability.
+
+## Pipeline Patterns
+
+### Batch ETL
+**Use when**: Regular scheduled loads, historical processing
+**Pattern**: Extract → Transform → Validate → Load
+**Tools**: Airflow, Dagster, Prefect
+
+### Streaming Pipeline
+**Use when**: Real-time processing, event-driven
+**Pattern**: Consume → Transform → Sink
+**Tools**: Kafka, Flink, Spark Streaming
+
+### Incremental Processing
+**Use when**: Large datasets, only processing changes
+**Pattern**: Watermark tracking + Merge/Upsert
+
+## Pipeline Design Checklist
+
+### Architecture
+- [ ] Source systems identified and accessible
+- [ ] Data volume estimated (GB/day)
+- [ ] Latency requirements clear
+- [ ] Target schema defined
+- [ ] Orchestration tool selected
+
+### Data Quality
+- [ ] Schema validation at source
+- [ ] Null handling strategy
+- [ ] Duplicate detection
+- [ ] Business rule validation
+
+### Error Handling
+- [ ] Transient errors: retry with backoff
+- [ ] Fatal errors: alert and stop
+- [ ] Invalid records: log separately
+- [ ] Rollback/recovery strategy
+
+### Performance
+- [ ] Parallelization strategy
+- [ ] Batch size optimized
+- [ ] Bottlenecks identified
+- [ ] Scaling plan documented
+
+### Monitoring
+- [ ] Row counts logged
+- [ ] Processing duration tracked
+- [ ] Error rates monitored
+- [ ] Data freshness SLA defined
+
+### Operations
+- [ ] Backfill procedure documented
+- [ ] Replay capability implemented
+- [ ] Config externalized
+- [ ] Secrets managed securely
+
+## Common Issues
+
+| Issue | Symptoms | Solution |
+|-------|----------|----------|
+| Fails halfway | Partial data, inconsistent state | Staging + commit pattern |
+| Duplicates | Same data loaded multiple times | Watermarks + idempotency |
+| Slow processing | Misses SLA | Profile and optimize bottlenecks |
+
+## Integration
+
+- **wicked-brain:search**: Find pipeline code with `wicked-brain:search "dag|pipeline"` (FTS5 over indexed code)
+- **Native tasks**: Track pipeline issues via TaskCreate with `metadata.event_type="task"`
+- **wicked-brain:memory**: Recall pipeline patterns
+
+## Best Practices
+
+- **Idempotency**: Same input → same output, pipelines safely rerunnable
+- **Observability**: Log row counts, track duration, emit metrics, alert on anomalies
+- **Testing**: Unit test transforms, integration test full pipeline, test error scenarios
+- **Documentation**: Clear lineage, versioned schemas, operations runbook
+
+## External Integration Discovery
+
+Pipeline engineering can leverage available integrations by capability:
+
+| Capability | Discovery Patterns | Provides |
+|------------|-------------------|----------|
+| **Warehouses** | `snowflake`, `databricks`, `bigquery` | Query execution, schema access |
+| **ETL** | `airbyte`, `fivetran`, `dbt` | Pipeline status, model metadata |
+| **Observability** | `monte-carlo`, `datadog` | Data quality metrics |
+
+Discover available integrations via capability detection. Fall back to wicked-garden:data:analyze for local file analysis via DuckDB.
+
+## Reference
+
+For detailed patterns:
+- [Output Templates](refs/templates.md) - Design doc, review report templates