tech-hub-skills 1.2.0 → 1.5.2

package/{tech_hub_skills → .claude}/roles/azure/skills/02-data-factory/README.md

@@ -1,264 +1,264 @@
-# Skill 02: Azure Data Factory
-
-## 🎯 Overview
-Pipeline creation, integration runtime, linked services
-
-## 🔗 Connections
-- **Data Engineer**: Data foundation and pipelines (de-01, de-02, de-03)
-- **Security Architect**: Compliance, PII detection, access control (sa-01, sa-02)
-- **ML Engineer**: Model lifecycle and serving (ml-01, ml-04)
-- **AI Engineer**: LLM integration and automation (ai-01, ai-02, ai-07)
-- **MLOps**: Experiment tracking and monitoring (mo-01, mo-03, mo-06)
-- **FinOps**: Cost optimization and tracking (fo-01, fo-07)
-- **DevOps**: CI/CD, containerization, monitoring (do-01, do-03, do-08)
-- **System Design**: Architecture patterns (sd-01)
-- **Dependencies**: de-02
-
-## 🛠️ Tools Included
-
-### 1. Primary Implementation Script
-Core implementation for azure data factory.
-
-### 2. Configuration Manager
-Manage configuration and settings for azure data factory.
-
-### 3. Integration Connector
-Connect with other Tech Hub skills and external services.
-
-### 4. Monitoring & Metrics
-Track performance, costs, and quality metrics.
-
-### 5. Automation Scripts
-Automate common workflows and tasks.
-
-## 📊 Key Metrics
-- Implementation quality score
-- Performance benchmarks
-- Cost efficiency
-- Security compliance rate
-- Integration test coverage
-
-## 🚀 Quick Start
-
-```python
-# Example implementation for Azure Data Factory
-from azure import 02_data_factory
-
-# Initialize
-service = 02DataFactoryService()
-
-# Execute
-result = service.execute(
-    config={
-        "environment": "production",
-        "enable_monitoring": True
-    }
-)
-
-print(f"Status: {result.status}")
-print(f"Metrics: {result.metrics}")
-```
-
-## 📚 Best Practices
-
-### Cost Optimization (FinOps Integration)
-
-1. **Monitor Resource Costs**
-   - Track costs per execution
-   - Set budget alerts
-   - Optimize resource utilization
-   - Reference: FinOps fo-01 (Cost Monitoring)
-
-2. **Right-size Resources**
-   - Use appropriate compute sizes
-   - Implement auto-scaling
-   - Leverage spot/reserved instances where applicable
-   - Reference: FinOps fo-06, fo-07
-
-### Security & Privacy (Security Architect Integration)
-
-3. **Implement Access Control**
-   - Use least privilege principle
-   - Enable Azure AD authentication
-   - Audit access logs
-   - Reference: Security Architect sa-02 (IAM), sa-04
-
-4. **Data Protection**
-   - Encrypt data at rest and in transit
-   - Scan for PII before processing
-   - Implement data retention policies
-   - Reference: Security Architect sa-01 (PII Detection)
-
-### Quality & Governance (Data Engineer Integration)
-
-5. **Ensure Data Quality**
-   - Validate inputs and outputs
-   - Implement quality gates
-   - Monitor data freshness
-   - Reference: Data Engineer de-03 (Data Quality)
-
-### Lifecycle Management (MLOps Integration)
-
-6. **Version Control**
-   - Version all configurations
-   - Track changes over time
-   - Enable rollback capability
-   - Reference: MLOps mo-03 (Versioning)
-
-7. **Continuous Monitoring**
-   - Track performance metrics
-   - Set up alerting
-   - Monitor for drift
-   - Reference: MLOps mo-06 (Monitoring)
-
-### Deployment & Operations (DevOps Integration)
-
-8. **Automate Deployment**
-   - Implement CI/CD pipelines
-   - Use infrastructure as code
-   - Enable blue-green deployments
-   - Reference: DevOps do-01 (CI/CD), do-03 (IaC)
-
-9. **Observability**
-   - Implement distributed tracing
-   - Set up dashboards
-   - Enable logging and metrics
-   - Reference: DevOps do-08 (Monitoring)
-
-### Azure-Specific Best Practices
-
-10. **Leverage Azure Services**
-    - Use managed services where possible
-    - Implement Azure Policy for governance
-    - Enable Azure Monitor integration
-    - Use managed identities for authentication
-
-## 💰 Cost Optimization Examples
-
-### Cost Tracking
-```python
-from finops_tracker import CostTracker
-
-tracker = CostTracker()
-
-@tracker.track_costs
-def run_operation(params):
-    # Your operation here
-    result = execute_operation(params)
-    return result
-
-# Monthly report
-report = tracker.monthly_report()
-print(f"Total cost: ${report.total_cost:.2f}")
-print(f"Cost per operation: ${report.avg_cost:.4f}")
-```
-
-## 🔒 Security Best Practices Examples
-
-### Access Control Implementation
-```python
-from azure.identity import DefaultAzureCredential
-from security_manager import AccessControl
-
-credential = DefaultAzureCredential()
-access_control = AccessControl(credential)
-
-# Validate access before operation
-@access_control.require_role("operator")
-def sensitive_operation(data):
-    # Operation logic
-    return process_data(data)
-```
-
-## 📊 Enhanced Metrics & Monitoring
-
-| Metric Category | Metric | Target | Tool |
-|-----------------|--------|--------|------|
-| **Performance** | Execution time (p95) | <5s | Azure Monitor |
-| | Success rate | >99% | Custom metrics |
-| **Cost** | Cost per operation | <$0.05 | FinOps dashboard |
-| | Resource utilization | >75% | Azure Monitor |
-| **Quality** | Error rate | <1% | App Insights |
-| | Data quality score | >95% | Quality tracker |
-| **Security** | Access violations | 0 | Security logs |
-| | Compliance score | 100% | Audit system |
-
-## 🚀 Deployment Pipeline
-
-### CI/CD Example
-```yaml
-# .github/workflows/deploy-02-data-factory.yml
-name: Deploy Azure Data Factory
-
-on:
-  push:
-    paths:
-      - 'azure/skills/02-data-factory/**'
-    branches:
-      - main
-
-jobs:
-  test:
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v3
-      - name: Run tests
-        run: pytest tests/ -v
-      - name: Security scan
-        run: python scripts/security_scan.py
-      - name: Cost validation
-        run: python scripts/validate_costs.py
-
-  deploy:
-    needs: test
-    runs-on: ubuntu-latest
-    steps:
-      - name: Deploy to Azure
-        run: |
-          az deployment group create \
-            --resource-group rg-azure \
-            --template-file infra/main.bicep
-      - name: Monitor deployment
-        run: python scripts/monitor_health.py --duration 10m
-```
-
-## 🔄 Integration Workflow
-
-### End-to-End Process
-```
-1. Input Validation
-   ↓
-2. Security Checks (sa-01, sa-02)
-   ↓
-3. Main Processing
-   ↓
-4. Quality Validation (de-03)
-   ↓
-5. Cost Tracking (fo-01)
-   ↓
-6. Monitoring & Logging (do-08)
-   ↓
-7. Output Delivery
-```
-
-## 🎯 Quick Wins
-
-1. **Enable cost tracking** - Monitor spending from day one
-2. **Implement security scanning** - Catch vulnerabilities early
-3. **Set up monitoring** - Full visibility into operations
-4. **Automate deployment** - Faster, safer releases
-5. **Add quality gates** - Prevent bad data from propagating
-6. **Enable caching** - Reduce redundant operations
-7. **Implement retries** - Improve reliability
-8. **Set up alerting** - Know about issues immediately
-
-## 🔗 Related Skills
-- de-02
-
----
-
-**Skill ID**: `02-data-factory`  
-**Complexity**: Medium  
-**Dependencies**: de-02  
-**Business Value**: High  
-**Estimated Implementation Time**: 4-8 hours
+# Skill 02: Azure Data Factory
+
+## 🎯 Overview
+Pipeline creation, integration runtime, linked services
+
+## 🔗 Connections
+- **Data Engineer**: Data foundation and pipelines (de-01, de-02, de-03)
+- **Security Architect**: Compliance, PII detection, access control (sa-01, sa-02)
+- **ML Engineer**: Model lifecycle and serving (ml-01, ml-04)
+- **AI Engineer**: LLM integration and automation (ai-01, ai-02, ai-07)
+- **MLOps**: Experiment tracking and monitoring (mo-01, mo-03, mo-06)
+- **FinOps**: Cost optimization and tracking (fo-01, fo-07)
+- **DevOps**: CI/CD, containerization, monitoring (do-01, do-03, do-08)
+- **System Design**: Architecture patterns (sd-01)
+- **Dependencies**: de-02
+
+## 🛠️ Tools Included
+
+### 1. Primary Implementation Script
+Core implementation for azure data factory.
+
+### 2. Configuration Manager
+Manage configuration and settings for azure data factory.
+
+### 3. Integration Connector
+Connect with other Tech Hub skills and external services.
+
+### 4. Monitoring & Metrics
+Track performance, costs, and quality metrics.
+
+### 5. Automation Scripts
+Automate common workflows and tasks.
+
+## 📊 Key Metrics
+- Implementation quality score
+- Performance benchmarks
+- Cost efficiency
+- Security compliance rate
+- Integration test coverage
+
+## 🚀 Quick Start
+
+```python
+# Example implementation for Azure Data Factory
+from azure import 02_data_factory
+
+# Initialize
+service = 02DataFactoryService()
+
+# Execute
+result = service.execute(
+    config={
+        "environment": "production",
+        "enable_monitoring": True
+    }
+)
+
+print(f"Status: {result.status}")
+print(f"Metrics: {result.metrics}")
+```
+
+## 📚 Best Practices
+
+### Cost Optimization (FinOps Integration)
+
+1. **Monitor Resource Costs**
+   - Track costs per execution
+   - Set budget alerts
+   - Optimize resource utilization
+   - Reference: FinOps fo-01 (Cost Monitoring)
+
+2. **Right-size Resources**
+   - Use appropriate compute sizes
+   - Implement auto-scaling
+   - Leverage spot/reserved instances where applicable
+   - Reference: FinOps fo-06, fo-07
+
+### Security & Privacy (Security Architect Integration)
+
+3. **Implement Access Control**
+   - Use least privilege principle
+   - Enable Azure AD authentication
+   - Audit access logs
+   - Reference: Security Architect sa-02 (IAM), sa-04
+
+4. **Data Protection**
+   - Encrypt data at rest and in transit
+   - Scan for PII before processing
+   - Implement data retention policies
+   - Reference: Security Architect sa-01 (PII Detection)
+
+### Quality & Governance (Data Engineer Integration)
+
+5. **Ensure Data Quality**
+   - Validate inputs and outputs
+   - Implement quality gates
+   - Monitor data freshness
+   - Reference: Data Engineer de-03 (Data Quality)
+
+### Lifecycle Management (MLOps Integration)
+
+6. **Version Control**
+   - Version all configurations
+   - Track changes over time
+   - Enable rollback capability
+   - Reference: MLOps mo-03 (Versioning)
+
+7. **Continuous Monitoring**
+   - Track performance metrics
+   - Set up alerting
+   - Monitor for drift
+   - Reference: MLOps mo-06 (Monitoring)
+
+### Deployment & Operations (DevOps Integration)
+
+8. **Automate Deployment**
+   - Implement CI/CD pipelines
+   - Use infrastructure as code
+   - Enable blue-green deployments
+   - Reference: DevOps do-01 (CI/CD), do-03 (IaC)
+
+9. **Observability**
+   - Implement distributed tracing
+   - Set up dashboards
+   - Enable logging and metrics
+   - Reference: DevOps do-08 (Monitoring)
+
+### Azure-Specific Best Practices
+
+10. **Leverage Azure Services**
+    - Use managed services where possible
+    - Implement Azure Policy for governance
+    - Enable Azure Monitor integration
+    - Use managed identities for authentication
+
+## 💰 Cost Optimization Examples
+
+### Cost Tracking
+```python
+from finops_tracker import CostTracker
+
+tracker = CostTracker()
+
+@tracker.track_costs
+def run_operation(params):
+    # Your operation here
+    result = execute_operation(params)
+    return result
+
+# Monthly report
+report = tracker.monthly_report()
+print(f"Total cost: ${report.total_cost:.2f}")
+print(f"Cost per operation: ${report.avg_cost:.4f}")
+```
+
+## 🔒 Security Best Practices Examples
+
+### Access Control Implementation
+```python
+from azure.identity import DefaultAzureCredential
+from security_manager import AccessControl
+
+credential = DefaultAzureCredential()
+access_control = AccessControl(credential)
+
+# Validate access before operation
+@access_control.require_role("operator")
+def sensitive_operation(data):
+    # Operation logic
+    return process_data(data)
+```
+
+## 📊 Enhanced Metrics & Monitoring
+
+| Metric Category | Metric | Target | Tool |
+|-----------------|--------|--------|------|
+| **Performance** | Execution time (p95) | <5s | Azure Monitor |
+| | Success rate | >99% | Custom metrics |
+| **Cost** | Cost per operation | <$0.05 | FinOps dashboard |
+| | Resource utilization | >75% | Azure Monitor |
+| **Quality** | Error rate | <1% | App Insights |
+| | Data quality score | >95% | Quality tracker |
+| **Security** | Access violations | 0 | Security logs |
+| | Compliance score | 100% | Audit system |
+
+## 🚀 Deployment Pipeline
+
+### CI/CD Example
+```yaml
+# .github/workflows/deploy-02-data-factory.yml
+name: Deploy Azure Data Factory
+
+on:
+  push:
+    paths:
+      - 'azure/skills/02-data-factory/**'
+    branches:
+      - main
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+      - name: Run tests
+        run: pytest tests/ -v
+      - name: Security scan
+        run: python scripts/security_scan.py
+      - name: Cost validation
+        run: python scripts/validate_costs.py
+
+  deploy:
+    needs: test
+    runs-on: ubuntu-latest
+    steps:
+      - name: Deploy to Azure
+        run: |
+          az deployment group create \
+            --resource-group rg-azure \
+            --template-file infra/main.bicep
+      - name: Monitor deployment
+        run: python scripts/monitor_health.py --duration 10m
+```
+
+## 🔄 Integration Workflow
+
+### End-to-End Process
+```
+1. Input Validation
+   ↓
+2. Security Checks (sa-01, sa-02)
+   ↓
+3. Main Processing
+   ↓
+4. Quality Validation (de-03)
+   ↓
+5. Cost Tracking (fo-01)
+   ↓
+6. Monitoring & Logging (do-08)
+   ↓
+7. Output Delivery
+```
+
+## 🎯 Quick Wins
+
+1. **Enable cost tracking** - Monitor spending from day one
+2. **Implement security scanning** - Catch vulnerabilities early
+3. **Set up monitoring** - Full visibility into operations
+4. **Automate deployment** - Faster, safer releases
+5. **Add quality gates** - Prevent bad data from propagating
+6. **Enable caching** - Reduce redundant operations
+7. **Implement retries** - Improve reliability
+8. **Set up alerting** - Know about issues immediately
+
+## 🔗 Related Skills
+- de-02
+
+---
+
+**Skill ID**: `02-data-factory`  
+**Complexity**: Medium  
+**Dependencies**: de-02  
+**Business Value**: High  
+**Estimated Implementation Time**: 4-8 hours