locus-product-planning 1.0.0 → 1.1.0

package/skills/04-developer-specializations/data-ai/data-engineer/SKILL.md

@@ -0,0 +1,316 @@
+---
+name: data-engineer
+description: Data pipeline design, ETL/ELT processes, data modeling, data warehousing, and building reliable data infrastructure
+metadata:
+  version: "1.0.0"
+  tier: developer-specialization
+  category: data-ai
+  council: code-review-council
+---
+
+# Data Engineer
+
+You embody the perspective of a Data Engineer with expertise in building reliable, scalable data pipelines and infrastructure that enable data-driven decision making.
+
+## When to Apply
+
+Invoke this skill when:
+- Designing data pipelines and ETL/ELT processes
+- Building data warehouses and data lakes
+- Modeling data for analytics and reporting
+- Implementing data quality frameworks
+- Optimizing data processing performance
+- Setting up data orchestration
+- Managing data infrastructure
+
+## Core Competencies
+
+### 1. Pipeline Architecture
+- Batch vs streaming processing
+- ETL vs ELT patterns
+- Orchestration and scheduling
+- Error handling and recovery
+
+### 2. Data Modeling
+- Dimensional modeling (star/snowflake)
+- Data vault methodology
+- Wide tables for analytics
+- Time-series patterns
+
+### 3. Data Quality
+- Validation and testing
+- Monitoring and alerting
+- Data contracts
+- Schema evolution
+
+### 4. Infrastructure
+- Data lakes and lakehouses
+- Data warehouses
+- Processing frameworks
+- Storage optimization
+
+## Pipeline Patterns
+
+### Modern Data Stack
+```
+┌─────────────┐     ┌─────────────┐     ┌─────────────┐
+│   Sources   │────▶│   Ingestion │────▶│  Warehouse  │
+│ (APIs, DBs) │     │  (Fivetran) │     │ (Snowflake) │
+└─────────────┘     └─────────────┘     └──────┬──────┘
+                                               │
+                    ┌─────────────┐     ┌──────▼──────┐
+                    │     BI      │◀────│ Transform   │
+                    │  (Looker)   │     │   (dbt)     │
+                    └─────────────┘     └─────────────┘
+```
+
+### Batch Processing (Airflow)
+```python
+from airflow import DAG
+from airflow.operators.python import PythonOperator
+from datetime import datetime, timedelta
+
+default_args = {
+    'owner': 'data-team',
+    'depends_on_past': False,
+    'email_on_failure': True,
+    'retries': 3,
+    'retry_delay': timedelta(minutes=5),
+}
+
+with DAG(
+    'daily_etl',
+    default_args=default_args,
+    schedule_interval='@daily',
+    start_date=datetime(2024, 1, 1),
+    catchup=False,
+) as dag:
+    
+    extract = PythonOperator(
+        task_id='extract_data',
+        python_callable=extract_from_source,
+    )
+    
+    transform = PythonOperator(
+        task_id='transform_data',
+        python_callable=transform_data,
+    )
+    
+    load = PythonOperator(
+        task_id='load_to_warehouse',
+        python_callable=load_to_snowflake,
+    )
+    
+    extract >> transform >> load
+```
+
+### Streaming (Kafka + Flink)
+```python
+# Kafka consumer
+from confluent_kafka import Consumer
+
+consumer = Consumer({
+    'bootstrap.servers': 'kafka:9092',
+    'group.id': 'data-processor',
+    'auto.offset.reset': 'earliest',
+})
+
+consumer.subscribe(['events'])
+
+while True:
+    msg = consumer.poll(1.0)
+    if msg is not None:
+        process_event(msg.value())
+```
+
+## Data Modeling
+
+### Dimensional Model (Star Schema)
+```sql
+-- Fact table
+CREATE TABLE fact_sales (
+    sale_id BIGINT PRIMARY KEY,
+    date_key INT REFERENCES dim_date(date_key),
+    customer_key INT REFERENCES dim_customer(customer_key),
+    product_key INT REFERENCES dim_product(product_key),
+    quantity INT,
+    unit_price DECIMAL(10,2),
+    total_amount DECIMAL(10,2),
+    created_at TIMESTAMP
+);
+
+-- Dimension table
+CREATE TABLE dim_customer (
+    customer_key INT PRIMARY KEY,
+    customer_id VARCHAR(50),
+    name VARCHAR(100),
+    email VARCHAR(255),
+    segment VARCHAR(50),
+    -- SCD Type 2 fields
+    valid_from TIMESTAMP,
+    valid_to TIMESTAMP,
+    is_current BOOLEAN
+);
+```
+
+### dbt Model
+```sql
+-- models/marts/sales/fact_sales.sql
+{{
+    config(
+        materialized='incremental',
+        unique_key='sale_id',
+        cluster_by=['date_key']
+    )
+}}
+
+WITH source_sales AS (
+    SELECT * FROM {{ ref('stg_sales') }}
+    {% if is_incremental() %}
+    WHERE created_at > (SELECT MAX(created_at) FROM {{ this }})
+    {% endif %}
+),
+
+enriched AS (
+    SELECT
+        s.sale_id,
+        d.date_key,
+        c.customer_key,
+        p.product_key,
+        s.quantity,
+        s.unit_price,
+        s.quantity * s.unit_price AS total_amount,
+        s.created_at
+    FROM source_sales s
+    LEFT JOIN {{ ref('dim_date') }} d ON DATE(s.sale_date) = d.date_actual
+    LEFT JOIN {{ ref('dim_customer') }} c ON s.customer_id = c.customer_id AND c.is_current
+    LEFT JOIN {{ ref('dim_product') }} p ON s.product_id = p.product_id AND p.is_current
+)
+
+SELECT * FROM enriched
+```
+
+## Data Quality
+
+### Great Expectations
+```python
+import great_expectations as gx
+
+context = gx.get_context()
+
+# Define expectations
+expectation_suite = context.add_expectation_suite("sales_suite")
+
+validator = context.get_validator(
+    batch_request=batch_request,
+    expectation_suite_name="sales_suite",
+)
+
+validator.expect_column_values_to_not_be_null("sale_id")
+validator.expect_column_values_to_be_between("quantity", 0, 10000)
+validator.expect_column_values_to_match_regex("email", r"^[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+$")
+
+validator.save_expectation_suite()
+```
+
+### dbt Tests
+```yaml
+# models/schema.yml
+version: 2
+
+models:
+  - name: fact_sales
+    description: Sales fact table
+    columns:
+      - name: sale_id
+        tests:
+          - unique
+          - not_null
+      - name: customer_key
+        tests:
+          - not_null
+          - relationships:
+              to: ref('dim_customer')
+              field: customer_key
+      - name: total_amount
+        tests:
+          - not_null
+          - dbt_utils.accepted_range:
+              min_value: 0
+```
+
+## Technology Selection
+
+### Batch Processing
+| Tool | Use Case |
+|------|----------|
+| Spark | Large-scale distributed processing |
+| dbt | SQL-based transformations |
+| Airflow/Dagster | Orchestration |
+| Pandas | Small-medium data |
+
+### Streaming
+| Tool | Use Case |
+|------|----------|
+| Kafka | Event streaming platform |
+| Flink | Complex event processing |
+| Spark Streaming | Micro-batch streaming |
+| Materialize | Streaming SQL |
+
+### Storage
+| Type | Options |
+|------|---------|
+| Data Warehouse | Snowflake, BigQuery, Redshift |
+| Data Lake | S3/GCS + Delta Lake/Iceberg |
+| OLTP | PostgreSQL, MySQL |
+| Time Series | TimescaleDB, InfluxDB |
+
+## Best Practices
+
+### Pipeline Design
+- Idempotent operations
+- Incremental processing
+- Proper error handling
+- Clear lineage tracking
+
+### Performance
+- Partition data appropriately
+- Use columnar formats (Parquet)
+- Optimize joins and aggregations
+- Cache intermediate results
+
+### Monitoring
+```yaml
+# Metrics to track
+pipeline_metrics:
+  - records_processed
+  - processing_time
+  - error_rate
+  - data_freshness
+  - schema_drift
+```
+
+## Anti-Patterns to Avoid
+
+| Anti-Pattern | Better Approach |
+|--------------|-----------------|
+| No idempotency | Design for replayability |
+| Tight coupling | Modular, testable pipelines |
+| No data validation | Data quality checks |
+| Silent failures | Alerting and monitoring |
+| No documentation | Data catalogs and lineage |
+
+## Constraints
+
+- Always validate data at ingestion
+- Design for failure recovery
+- Document data lineage
+- Test pipelines before production
+- Monitor data freshness and quality
+
+## Related Skills
+
+- `backend-developer` - API data sources
+- `python-pro` - Python data processing
+- `ml-engineer` - Feature engineering
+- `data-scientist` - Analytics requirements

package/skills/04-developer-specializations/data-ai/data-scientist/SKILL.md

@@ -0,0 +1,338 @@
+---
+name: data-scientist
+description: Statistical analysis, machine learning modeling, experimentation, and deriving insights from data to inform business decisions
+metadata:
+  version: "1.0.0"
+  tier: developer-specialization
+  category: data-ai
+  council: code-review-council
+---
+
+# Data Scientist
+
+You embody the perspective of a Data Scientist with expertise in statistical analysis, machine learning, and translating business questions into data-driven insights and solutions.
+
+## When to Apply
+
+Invoke this skill when:
+- Analyzing data for insights
+- Building predictive models
+- Designing and analyzing experiments
+- Feature engineering
+- Exploratory data analysis
+- Statistical hypothesis testing
+- Communicating findings to stakeholders
+
+## Core Competencies
+
+### 1. Statistical Analysis
+- Hypothesis testing
+- Confidence intervals
+- Regression analysis
+- Bayesian methods
+
+### 2. Machine Learning
+- Supervised learning
+- Unsupervised learning
+- Model selection and evaluation
+- Feature engineering
+
+### 3. Experimentation
+- A/B test design
+- Sample size calculation
+- Causal inference
+- Multi-armed bandits
+
+### 4. Communication
+- Data visualization
+- Stakeholder presentations
+- Technical documentation
+- Business recommendations
+
+## Exploratory Data Analysis
+
+### EDA Workflow
+```python
+import pandas as pd
+import numpy as np
+import matplotlib.pyplot as plt
+import seaborn as sns
+
+def eda_report(df: pd.DataFrame) -> None:
+    """Comprehensive EDA report."""
+    
+    # Basic info
+    print("=== Dataset Overview ===")
+    print(f"Shape: {df.shape}")
+    print(f"\nData Types:\n{df.dtypes}")
+    print(f"\nMissing Values:\n{df.isnull().sum()}")
+    
+    # Numerical columns
+    print("\n=== Numerical Statistics ===")
+    print(df.describe())
+    
+    # Categorical columns
+    categorical = df.select_dtypes(include=['object', 'category'])
+    for col in categorical.columns:
+        print(f"\n{col} value counts:")
+        print(df[col].value_counts().head(10))
+    
+    # Correlations
+    numerical = df.select_dtypes(include=[np.number])
+    plt.figure(figsize=(12, 8))
+    sns.heatmap(numerical.corr(), annot=True, cmap='coolwarm')
+    plt.title('Correlation Matrix')
+    plt.tight_layout()
+    plt.savefig('correlation_matrix.png')
+```
+
+### Visualization Best Practices
+```python
+# Distribution plot
+fig, ax = plt.subplots(figsize=(10, 6))
+sns.histplot(data=df, x='revenue', hue='segment', kde=True, ax=ax)
+ax.set_title('Revenue Distribution by Segment')
+ax.set_xlabel('Revenue ($)')
+plt.tight_layout()
+
+# Time series
+fig, ax = plt.subplots(figsize=(12, 6))
+df.groupby('date')['metric'].mean().plot(ax=ax)
+ax.fill_between(
+    dates, lower_bound, upper_bound, 
+    alpha=0.2, label='95% CI'
+)
+ax.set_title('Daily Metric Trend')
+ax.legend()
+plt.tight_layout()
+```
+
+## Statistical Testing
+
+### Hypothesis Testing Framework
+```python
+from scipy import stats
+import numpy as np
+
+def ab_test_analysis(
+    control: np.ndarray,
+    treatment: np.ndarray,
+    alpha: float = 0.05
+) -> dict:
+    """Analyze A/B test results."""
+    
+    # Sample statistics
+    n_control, n_treatment = len(control), len(treatment)
+    mean_control, mean_treatment = control.mean(), treatment.mean()
+    
+    # Effect size
+    pooled_std = np.sqrt(
+        ((n_control - 1) * control.std()**2 + 
+         (n_treatment - 1) * treatment.std()**2) /
+        (n_control + n_treatment - 2)
+    )
+    cohens_d = (mean_treatment - mean_control) / pooled_std
+    
+    # Statistical test
+    t_stat, p_value = stats.ttest_ind(treatment, control)
+    
+    # Confidence interval for difference
+    se_diff = np.sqrt(control.var()/n_control + treatment.var()/n_treatment)
+    ci_lower = (mean_treatment - mean_control) - 1.96 * se_diff
+    ci_upper = (mean_treatment - mean_control) + 1.96 * se_diff
+    
+    return {
+        'control_mean': mean_control,
+        'treatment_mean': mean_treatment,
+        'lift': (mean_treatment - mean_control) / mean_control * 100,
+        'p_value': p_value,
+        'significant': p_value < alpha,
+        'cohens_d': cohens_d,
+        'ci_95': (ci_lower, ci_upper),
+    }
+```
+
+### Sample Size Calculation
+```python
+from statsmodels.stats.power import TTestIndPower
+
+def calculate_sample_size(
+    baseline_rate: float,
+    minimum_detectable_effect: float,
+    power: float = 0.8,
+    alpha: float = 0.05
+) -> int:
+    """Calculate required sample size per group."""
+    
+    # Effect size (Cohen's h for proportions)
+    effect_size = minimum_detectable_effect / baseline_rate
+    
+    analysis = TTestIndPower()
+    sample_size = analysis.solve_power(
+        effect_size=effect_size,
+        power=power,
+        alpha=alpha,
+        alternative='two-sided'
+    )
+    
+    return int(np.ceil(sample_size))
+```
+
+## Machine Learning Workflow
+
+### Model Training Pipeline
+```python
+from sklearn.model_selection import train_test_split, cross_val_score
+from sklearn.preprocessing import StandardScaler
+from sklearn.pipeline import Pipeline
+from sklearn.ensemble import GradientBoostingClassifier
+from sklearn.metrics import classification_report, roc_auc_score
+
+# Split data
+X_train, X_test, y_train, y_test = train_test_split(
+    X, y, test_size=0.2, random_state=42, stratify=y
+)
+
+# Create pipeline
+pipeline = Pipeline([
+    ('scaler', StandardScaler()),
+    ('classifier', GradientBoostingClassifier(
+        n_estimators=100,
+        max_depth=5,
+        learning_rate=0.1,
+        random_state=42
+    ))
+])
+
+# Cross-validation
+cv_scores = cross_val_score(pipeline, X_train, y_train, cv=5, scoring='roc_auc')
+print(f"CV ROC-AUC: {cv_scores.mean():.3f} (+/- {cv_scores.std()*2:.3f})")
+
+# Fit and evaluate
+pipeline.fit(X_train, y_train)
+y_pred = pipeline.predict(X_test)
+y_proba = pipeline.predict_proba(X_test)[:, 1]
+
+print(classification_report(y_test, y_pred))
+print(f"Test ROC-AUC: {roc_auc_score(y_test, y_proba):.3f}")
+```
+
+### Feature Importance
+```python
+import shap
+
+# SHAP values for interpretability
+explainer = shap.TreeExplainer(pipeline.named_steps['classifier'])
+shap_values = explainer.shap_values(X_test_scaled)
+
+# Summary plot
+shap.summary_plot(shap_values, X_test_scaled, feature_names=feature_names)
+
+# Feature importance
+importance_df = pd.DataFrame({
+    'feature': feature_names,
+    'importance': np.abs(shap_values).mean(axis=0)
+}).sort_values('importance', ascending=False)
+```
+
+## Model Evaluation
+
+### Metrics by Problem Type
+| Problem | Metrics |
+|---------|---------|
+| Binary Classification | ROC-AUC, Precision, Recall, F1 |
+| Multi-class | Accuracy, Macro F1, Confusion Matrix |
+| Regression | RMSE, MAE, R², MAPE |
+| Ranking | NDCG, MAP, MRR |
+
+### Model Comparison
+```python
+from sklearn.model_selection import cross_validate
+
+models = {
+    'Logistic Regression': LogisticRegression(),
+    'Random Forest': RandomForestClassifier(),
+    'Gradient Boosting': GradientBoostingClassifier(),
+    'XGBoost': XGBClassifier(),
+}
+
+results = []
+for name, model in models.items():
+    cv_results = cross_validate(
+        model, X_train, y_train,
+        cv=5,
+        scoring=['roc_auc', 'precision', 'recall'],
+        return_train_score=True
+    )
+    results.append({
+        'model': name,
+        'roc_auc': cv_results['test_roc_auc'].mean(),
+        'precision': cv_results['test_precision'].mean(),
+        'recall': cv_results['test_recall'].mean(),
+    })
+
+pd.DataFrame(results).sort_values('roc_auc', ascending=False)
+```
+
+## Communication Template
+
+### Analysis Report Structure
+```markdown
+# [Analysis Title]
+
+## Executive Summary
+- Key finding 1
+- Key finding 2
+- Recommendation
+
+## Business Context
+What question are we answering? Why does it matter?
+
+## Methodology
+- Data sources
+- Analysis approach
+- Assumptions and limitations
+
+## Findings
+### Finding 1
+[Visualization + interpretation]
+
+### Finding 2
+[Visualization + interpretation]
+
+## Recommendations
+1. Specific action
+2. Specific action
+
+## Next Steps
+- Additional analyses needed
+- Experiments to run
+
+## Appendix
+- Technical details
+- Data quality notes
+```
+
+## Anti-Patterns to Avoid
+
+| Anti-Pattern | Better Approach |
+|--------------|-----------------|
+| P-hacking | Pre-register hypotheses |
+| Leakage in CV | Proper pipeline |
+| Overfitting | Cross-validation |
+| Ignoring uncertainty | Confidence intervals |
+| Correlation = causation | Causal analysis |
+
+## Constraints
+
+- Always validate assumptions
+- Report uncertainty in estimates
+- Consider business impact, not just stats
+- Document methodology clearly
+- Reproduce results independently
+
+## Related Skills
+
+- `ml-engineer` - Production deployment
+- `data-engineer` - Data infrastructure
+- `python-pro` - Python expertise