tech-hub-skills 1.0.0

package/tech_hub_skills/roles/data-governance/skills/02-data-lineage/README.md

@@ -0,0 +1,129 @@
+# dg-02: Data Lineage
+
+## Overview
+
+Track end-to-end data lineage for impact analysis, root cause analysis, and regulatory compliance.
+
+## Key Capabilities
+
+- **End-to-End Lineage**: From source to consumption
+- **Impact Analysis**: Understand downstream impacts
+- **Root Cause Analysis**: Trace issues to source
+- **Column-Level Lineage**: Field-level tracking
+- **Transformation Documentation**: Track data transformations
+
+## Tools & Technologies
+
+- **Azure Purview**: Native lineage tracking
+- **OpenLineage**: Open standard for lineage
+- **Marquez**: Metadata service for lineage
+- **Spline**: Spark lineage tracking
+
+## Implementation
+
+### 1. Lineage Extraction
+
+```python
+# Extract lineage from Spark jobs
+from spline import SplineAgent
+
+def track_spark_lineage(spark_session):
+    """Enable lineage tracking for Spark"""
+    spark_session.sparkContext.setLogLevel("INFO")
+
+    # Initialize Spline agent
+    SplineAgent.builder() \
+        .appName("data-pipeline") \
+        .mode("REQUIRED") \
+        .url("http://spline-server:9090") \
+        .build()
+```
+
+### 2. Column-Level Lineage
+
+```sql
+-- Azure Purview automatically tracks column lineage
+-- Example transformation with lineage
+CREATE VIEW customer_360 AS
+SELECT
+    c.customer_id,
+    c.first_name || ' ' || c.last_name as full_name,  -- Lineage: derived
+    o.total_orders,
+    p.total_payments
+FROM customers c
+LEFT JOIN order_summary o ON c.customer_id = o.customer_id
+LEFT JOIN payment_summary p ON c.customer_id = p.customer_id;
+```
+
+### 3. Impact Analysis
+
+```python
+# Find downstream dependencies
+def get_downstream_impact(asset_id):
+    """Find all downstream assets affected by changes"""
+    lineage = client.lineage.get_lineage(
+        guid=asset_id,
+        direction="OUTPUT",
+        depth=10
+    )
+
+    downstream_assets = []
+    for entity in lineage['guidEntityMap'].values():
+        downstream_assets.append({
+            'name': entity['attributes']['name'],
+            'type': entity['typeName'],
+            'owner': entity.get('attributes', {}).get('owner')
+        })
+
+    return downstream_assets
+```
+
+### 4. OpenLineage Integration
+
+```python
+# Emit lineage events using OpenLineage
+from openlineage.client import OpenLineageClient
+from openlineage.client.run import RunEvent, RunState, Run, Job
+
+def emit_lineage_event(job_name, inputs, outputs):
+    """Emit lineage event to OpenLineage"""
+    client = OpenLineageClient(url="http://lineage-api:5000")
+
+    event = RunEvent(
+        eventType=RunState.COMPLETE,
+        eventTime="2025-01-01T00:00:00Z",
+        run=Run(runId=str(uuid.uuid4())),
+        job=Job(namespace="production", name=job_name),
+        inputs=inputs,
+        outputs=outputs
+    )
+
+    client.emit(event)
+```
+
+## Best Practices
+
+1. **Automate Collection** - Manual lineage doesn't scale
+2. **Column-Level Tracking** - For sensitive data, track field-level
+3. **Version Control** - Track lineage changes over time
+4. **Clear Visualization** - Make lineage easy to understand
+5. **Regular Validation** - Verify lineage accuracy
+
+## Cost Optimization
+
+- Use incremental lineage updates
+- Archive old lineage data after retention period
+- Cache frequently accessed lineage queries
+- Use materialized views for complex lineage
+
+## Integration
+
+**Connects with:**
+- de-02 (ETL): Track pipeline lineage
+- dg-01 (Catalog): Link assets to lineage
+- ml-02 (Feature Engineering): Track feature lineage
+- ai-02 (RAG): Track document lineage
+
+## Quick Win
+
+Start with 1 critical data pipeline, manually document lineage, validate accuracy, then automate extraction.

package/tech_hub_skills/roles/data-governance/skills/03-data-quality-framework/README.md

@@ -0,0 +1,182 @@
+# dg-03: Data Quality Framework
+
+## Overview
+
+Implement automated data quality validation, scoring, monitoring, and issue remediation workflows.
+
+## Key Capabilities
+
+- **Quality Rules Definition**: Completeness, accuracy, consistency
+- **Automated Validation**: Real-time quality checks
+- **Quality Scoring**: Quantifiable quality metrics
+- **Quality Monitoring**: Continuous quality tracking
+- **Issue Remediation**: Workflows for quality issues
+
+## Tools & Technologies
+
+- **Great Expectations**: Python data validation
+- **Soda**: Data quality as code
+- **dbt tests**: Quality tests in dbt
+- **Azure Data Quality**: Native Azure solution
+
+## Implementation
+
+### 1. Quality Rules with Great Expectations
+
+```python
+# Define quality expectations
+import great_expectations as gx
+
+def create_quality_suite(context, table_name):
+    """Create data quality test suite"""
+    suite = context.add_expectation_suite(
+        expectation_suite_name=f"{table_name}_quality_suite"
+    )
+
+    validator = context.get_validator(
+        batch_request=batch_request,
+        expectation_suite_name=suite.expectation_suite_name
+    )
+
+    # Completeness checks
+    validator.expect_column_values_to_not_be_null(column="customer_id")
+    validator.expect_column_values_to_not_be_null(column="order_date")
+
+    # Accuracy checks
+    validator.expect_column_values_to_be_between(
+        column="age",
+        min_value=0,
+        max_value=120
+    )
+
+    # Consistency checks
+    validator.expect_column_values_to_match_regex(
+        column="email",
+        regex=r"^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$"
+    )
+
+    validator.save_expectation_suite()
+    return validator
+```
+
+### 2. Quality Scoring
+
+```python
+# Calculate quality score
+def calculate_quality_score(validation_results):
+    """Calculate overall quality score"""
+    total_checks = validation_results.statistics['evaluated_expectations']
+    successful_checks = validation_results.statistics['successful_expectations']
+
+    score = (successful_checks / total_checks) * 100
+
+    # Categorize quality
+    if score >= 95:
+        quality_level = "Excellent"
+    elif score >= 85:
+        quality_level = "Good"
+    elif score >= 70:
+        quality_level = "Acceptable"
+    else:
+        quality_level = "Poor"
+
+    return {
+        'score': score,
+        'level': quality_level,
+        'total_checks': total_checks,
+        'passed_checks': successful_checks
+    }
+```
+
+### 3. Automated Monitoring
+
+```python
+# Set up quality monitoring
+def setup_quality_monitoring(checkpoint_name):
+    """Configure automated quality monitoring"""
+    checkpoint_config = {
+        "name": checkpoint_name,
+        "config_version": 1.0,
+        "template_name": "default",
+        "run_name_template": "%Y%m%d-%H%M%S",
+        "validations": [
+            {
+                "batch_request": {
+                    "datasource_name": "production_data",
+                    "data_connector_name": "default_inferred_data_connector_name",
+                    "data_asset_name": "customers"
+                },
+                "expectation_suite_name": "customers_quality_suite"
+            }
+        ],
+        "action_list": [
+            {
+                "name": "store_validation_result",
+                "action": {"class_name": "StoreValidationResultAction"}
+            },
+            {
+                "name": "send_slack_notification",
+                "action": {
+                    "class_name": "SlackNotificationAction",
+                    "slack_webhook": "${SLACK_WEBHOOK}",
+                    "notify_on": "failure"
+                }
+            }
+        ]
+    }
+
+    context.add_checkpoint(**checkpoint_config)
+```
+
+### 4. Issue Remediation Workflow
+
+```python
+# Create remediation workflow
+def create_remediation_workflow(quality_issues):
+    """Create tickets for quality issues"""
+    from azure.devops import AzureDevOpsClient
+
+    client = AzureDevOpsClient()
+
+    for issue in quality_issues:
+        work_item = {
+            'title': f"Data Quality Issue: {issue['column']}",
+            'description': issue['description'],
+            'priority': issue['severity'],
+            'assigned_to': issue['data_owner'],
+            'tags': ['data-quality', issue['table']]
+        }
+
+        client.create_work_item(
+            project='DataGovernance',
+            work_item_type='Bug',
+            fields=work_item
+        )
+```
+
+## Best Practices
+
+1. **Start Simple** - Begin with critical fields, expand coverage
+2. **Automate Everything** - Manual checks don't scale
+3. **Clear Ownership** - Assign quality issues to data owners
+4. **Threshold Alerts** - Alert on quality score drops
+5. **Historical Tracking** - Monitor quality trends over time
+
+## Cost Optimization
+
+- Run quality checks incrementally (only new/changed data)
+- Use sampling for large datasets
+- Cache validation results
+- Right-size validation compute
+
+## Integration
+
+**Connects with:**
+- de-01 (Lakehouse): Validate lakehouse data
+- de-03 (Data Quality): Engineering quality checks
+- dg-01 (Catalog): Link quality scores to assets
+- dg-02 (Lineage): Trace quality issues to source
+
+## Quick Win
+
+Implement completeness checks on 5 critical fields in your most important table. Show before/after quality scores.

package/tech_hub_skills/roles/data-governance/skills/04-access-control/README.md

@@ -0,0 +1,39 @@
+# dg-04: Access Control & Policies
+
+## Overview
+
+Implement role-based access control, column/row-level security, dynamic data masking, and access audit logging.
+
+## Key Capabilities
+
+- **RBAC**: Role-based access control
+- **Column-Level Security**: Restrict sensitive columns
+- **Row-Level Security**: Filter data by user context
+- **Dynamic Data Masking**: Auto-mask sensitive data
+- **Access Audit Logging**: Track all data access
+
+## Implementation
+
+```sql
+-- Column-level security
+CREATE VIEW customer_secure AS
+SELECT
+    customer_id,
+    CASE
+        WHEN CURRENT_USER() IN (SELECT user FROM admin_users)
+        THEN email  -- Show full email to admins
+        ELSE CONCAT(LEFT(email, 3), '***@', SPLIT_PART(email, '@', 2))  -- Mask for others
+    END as email,
+    first_name,
+    last_name
+FROM customers;
+
+-- Row-level security
+CREATE POLICY customer_region_policy ON customers
+FOR SELECT
+USING (region = current_setting('app.user_region'));
+```
+
+## Integration
+
+**Connects with:** sa-01 (PII Detection), sa-04 (IAM), dg-01 (Catalog)

package/tech_hub_skills/roles/data-governance/skills/05-master-data-management/README.md

@@ -0,0 +1,40 @@
+# dg-05: Master Data Management
+
+## Overview
+
+Entity resolution, golden record creation, data stewardship, and hierarchy management for critical business entities.
+
+## Key Capabilities
+
+- **Entity Resolution**: Match and merge duplicate entities
+- **Golden Record**: Single source of truth
+- **Data Stewardship**: Workflows for data quality
+- **Cross-Reference**: Link entities across systems
+- **Hierarchy Management**: Organizational structures
+
+## Implementation
+
+```python
+# Entity resolution
+from recordlinkage import Index, Compare
+
+def match_customers(df1, df2):
+    """Match customer records across systems"""
+    indexer = Index()
+    indexer.block('last_name')
+    candidate_pairs = indexer.index(df1, df2)
+
+    compare = Compare()
+    compare.exact('first_name', 'first_name')
+    compare.string('email', 'email', method='jarowinkler', threshold=0.85)
+    compare.numeric('age', 'age', method='linear', offset=2)
+
+    features = compare.compute(candidate_pairs, df1, df2)
+    matches = features[features.sum(axis=1) > 2.5]
+
+    return matches
+```
+
+## Integration
+
+**Connects with:** dg-01 (Catalog), dg-03 (Quality), de-02 (ETL)

package/tech_hub_skills/roles/data-governance/skills/06-compliance-privacy/README.md

@@ -0,0 +1,46 @@
+# dg-06: Compliance & Privacy
+
+## Overview
+
+GDPR compliance automation, data retention policies, right to be forgotten, consent management, and privacy impact assessments.
+
+## Key Capabilities
+
+- **GDPR Automation**: Automated compliance checks
+- **Data Retention**: Automated data lifecycle
+- **Right to be Forgotten**: Delete personal data on request
+- **Consent Management**: Track user consent
+- **Privacy Impact Assessments**: Risk assessment
+
+## Implementation
+
+```python
+# Right to be forgotten
+def delete_user_data(user_id):
+    """Delete all personal data for a user"""
+    tables = [
+        'customers', 'orders', 'payments',
+        'preferences', 'analytics_events'
+    ]
+
+    for table in tables:
+        spark.sql(f"""
+            DELETE FROM {table}
+            WHERE user_id = '{user_id}'
+        """)
+
+    # Log deletion for audit
+    log_gdpr_deletion(user_id, tables)
+
+# Data retention policy
+def apply_retention_policy():
+    """Delete data past retention period"""
+    spark.sql("""
+        DELETE FROM customer_events
+        WHERE event_date < DATE_SUB(CURRENT_DATE(), 730)  -- 2 years
+    """)
+```
+
+## Integration
+
+**Connects with:** sa-01 (PII Detection), dg-01 (Catalog), dg-04 (Access Control)

package/tech_hub_skills/roles/data-scientist/skills/01-eda-automation/README.md

@@ -0,0 +1,230 @@
+# Skill 1: Automated Exploratory Data Analysis (EDA)
+
+## 🎯 Overview
+Automated EDA with statistical profiling, visualization, and insight generation.
+
+## 🔗 Connections
+- **Data Engineer**: Provides feedback on data quality issues (de-01, de-03)
+- **ML Engineer**: Identifies promising features for modeling (ml-01, ml-02)
+- **MLOps**: Experiment tracking for EDA findings (mo-01)
+- **AI Engineer**: Generates insights for LLM context (ai-02, ai-03)
+- **Security Architect**: PII detection in datasets (sa-01)
+- **FinOps**: Cost-effective analytics compute (fo-06)
+- **DevOps**: Automated reporting pipelines (do-01)
+
+## 🛠️ Tools Included
+
+### 1. `eda_generator.py`
+Automated EDA report generation with ydata-profiling.
+
+### 2. `statistical_analyzer.py`
+Statistical tests, distributions, and correlations.
+
+### 3. `visualization_suite.py`
+Interactive visualizations with Plotly.
+
+### 4. `insight_extractor.py`
+Automated insight extraction and anomaly detection.
+
+### 5. `eda_queries.sql`
+SQL templates for common analytical queries.
+
+## 📊 Key Outputs
+- Automated profiling reports (HTML)
+- Statistical summaries
+- Correlation matrices
+- Distribution plots
+- Anomaly detection alerts
+
+## 🚀 Quick Start
+
+```python
+from eda_generator import EDAGenerator
+
+# Initialize
+eda = EDAGenerator()
+
+# Load data
+df = pd.read_csv("customer_data.csv")
+
+# Generate comprehensive report
+report = eda.generate_report(
+    df=df,
+    title="Customer Data Analysis",
+    output_file="eda_report.html"
+)
+
+# Extract key insights
+insights = eda.extract_insights(df)
+print(insights)
+```
+
+## 📚 Best Practices
+
+### Data Quality & Security (Cross-Role Integration)
+
+1. **PII Detection Before Analysis**
+   - Scan datasets for PII before profiling
+   - Mask sensitive data in reports and visualizations
+   - Track data lineage for compliance
+   - Reference: Security Architect sa-01 (PII Detection)
+
+2. **Data Quality Validation**
+   - Validate schema before EDA
+   - Check completeness, accuracy, consistency
+   - Alert Data Engineering team on quality issues
+   - Reference: Data Engineer de-03 (Data Quality)
+
+3. **Automated Quality Feedback Loop**
+   - Generate data quality scorecards
+   - Feed insights back to data pipelines
+   - Track quality improvements over time
+   - Reference: Data Engineer de-01, de-03
+
+### Cost Optimization (FinOps Integration)
+
+4. **Optimize Compute for Analysis**
+   - Use appropriate instance sizes for EDA workloads
+   - Auto-shutdown notebooks when idle
+   - Sample large datasets intelligently
+   - Monitor analysis costs per project
+   - Reference: FinOps fo-06 (Compute Optimization)
+
+5. **Efficient Data Sampling**
+   - Use stratified sampling for large datasets
+   - Profile samples before full dataset analysis
+   - Cache intermediate results
+   - Minimize data movement and storage
+   - Reference: FinOps fo-05, Data Engineer de-01
+
+### MLOps Integration
+
+6. **Track EDA Experiments**
+   - Log EDA findings in MLflow/Azure ML
+   - Version datasets used for analysis
+   - Document feature engineering insights
+   - Link EDA to downstream model experiments
+   - Reference: MLOps mo-01 (Experiment Tracking)
+
+7. **Feature Discovery Documentation**
+   - Document promising features for ML
+   - Track feature importance from EDA
+   - Share insights with ML Engineering team
+   - Maintain feature catalog
+   - Reference: ML Engineer ml-02 (Feature Engineering)
+
+### Automation & Deployment (DevOps Integration)
+
+8. **Automated EDA Pipelines**
+   - Schedule regular EDA reports for key datasets
+   - Automate anomaly detection and alerting
+   - Deploy EDA as part of data pipeline monitoring
+   - Version control EDA scripts
+   - Reference: DevOps do-01 (CI/CD), do-08 (Monitoring)
+
+9. **Reproducible Analysis**
+   - Use containerized environments
+   - Pin package versions
+   - Document analysis dependencies
+   - Enable one-click report regeneration
+   - Reference: DevOps do-03 (Containerization)
+
+### AI Integration
+
+10. **LLM-Powered Insights**
+    - Use LLMs to generate narrative insights
+    - Automate insight extraction from distributions
+    - Create natural language data summaries
+    - Reference: AI Engineer ai-01, ai-07
+
+## 💰 Cost Optimization Examples
+
+### Compute Cost Tracking
+```python
+from eda_generator import EDAGenerator
+from finops_tracker import AnalyticsCostTracker
+
+cost_tracker = AnalyticsCostTracker()
+
+# Track EDA compute costs
+@cost_tracker.track_analysis_cost
+def run_eda(dataset_path: str):
+    eda = EDAGenerator()
+    df = pd.read_csv(dataset_path)
+
+    # Smart sampling for large datasets
+    if len(df) > 1_000_000:
+        df = df.sample(n=100_000, random_state=42)  # Cost savings
+
+    report = eda.generate_report(df)
+    return report
+
+# Cost report
+report = cost_tracker.monthly_report()
+print(f"Total EDA costs: ${report.total_cost:.2f}")
+print(f"Cost per analysis: ${report.avg_cost:.2f}")
+```
+
+## 🔒 Security Best Practices
+
+### PII Masking in Reports
+```python
+from pii_detector import PIIDetector
+from eda_generator import EDAGenerator
+
+detector = PIIDetector()
+eda = EDAGenerator()
+
+def secure_eda(df: pd.DataFrame):
+    # Detect PII columns
+    pii_columns = []
+    for col in df.columns:
+        sample = df[col].astype(str).sample(min(100, len(df)))
+        if detector.contains_pii(sample.tolist()):
+            pii_columns.append(col)
+
+    # Mask PII before EDA
+    df_masked = df.copy()
+    for col in pii_columns:
+        df_masked[col] = "***MASKED***"
+
+    # Generate report on masked data
+    report = eda.generate_report(
+        df_masked,
+        title="Customer Data Analysis (PII Masked)"
+    )
+
+    return report, pii_columns
+```
+
+## 🔄 Integration Workflow
+
+### End-to-End EDA Pipeline
+```
+1. Data Ingestion (de-01)
+   ↓
+2. PII Detection (sa-01)
+   ↓
+3. Data Quality Check (de-03)
+   ↓
+4. Automated EDA (ds-01)
+   ↓
+5. Track Findings (mo-01)
+   ↓
+6. Feature Discovery (ml-02)
+   ↓
+7. Generate Insights (ai-07)
+   ↓
+8. Share Report (Automated)
+   ↓
+9. Monitor Costs (fo-06)
+```
+
+## 🎯 Quick Wins
+
+1. **Automate PII detection** - Prevent compliance violations in reports
+2. **Set up cost tracking** - Monitor analysis compute spending
+3. **Enable auto-shutdown** - Stop idle notebooks to save costs
+4. **Sample large datasets** - Faster EDA at lower cost
+5. **Track EDA experiments** - Link insights to model performance
+6. **Automate report generation** - Schedule weekly data profiling