tech-hub-skills 1.0.0

package/tech_hub_skills/roles/mlops/skills/09-automated-retraining/README.md

@@ -0,0 +1,264 @@
+# Skill 09: Automated Retraining Pipelines
+
+## 🎯 Overview
+Trigger-based retraining, validation gates
+
+## 🔗 Connections
+- **Data Engineer**: Data foundation and pipelines (de-01, de-02, de-03)
+- **Security Architect**: Compliance, PII detection, access control (sa-01, sa-02)
+- **ML Engineer**: Model lifecycle and serving (ml-01, ml-04)
+- **AI Engineer**: LLM integration and automation (ai-01, ai-02, ai-07)
+- **MLOps**: Experiment tracking and monitoring (mo-01, mo-03, mo-06)
+- **FinOps**: Cost optimization and tracking (fo-01, fo-07)
+- **DevOps**: CI/CD, containerization, monitoring (do-01, do-03, do-08)
+- **System Design**: Architecture patterns (sd-01)
+- **Dependencies**: mo-06
+
+## 🛠️ Tools Included
+
+### 1. Primary Implementation Script
+Core implementation for automated retraining pipelines.
+
+### 2. Configuration Manager
+Manage configuration and settings for automated retraining pipelines.
+
+### 3. Integration Connector
+Connect with other Tech Hub skills and external services.
+
+### 4. Monitoring & Metrics
+Track performance, costs, and quality metrics.
+
+### 5. Automation Scripts
+Automate common workflows and tasks.
+
+## 📊 Key Metrics
+- Implementation quality score
+- Performance benchmarks
+- Cost efficiency
+- Security compliance rate
+- Integration test coverage
+
+## 🚀 Quick Start
+
+```python
+# Example implementation for Automated Retraining Pipelines
+from mlops import 09_automated_retraining
+
+# Initialize
+service = 09AutomatedRetrainingService()
+
+# Execute
+result = service.execute(
+    config={
+        "environment": "production",
+        "enable_monitoring": True
+    }
+)
+
+print(f"Status: {result.status}")
+print(f"Metrics: {result.metrics}")
+```
+
+## 📚 Best Practices
+
+### Cost Optimization (FinOps Integration)
+
+1. **Monitor Resource Costs**
+   - Track costs per execution
+   - Set budget alerts
+   - Optimize resource utilization
+   - Reference: FinOps fo-01 (Cost Monitoring)
+
+2. **Right-size Resources**
+   - Use appropriate compute sizes
+   - Implement auto-scaling
+   - Leverage spot/reserved instances where applicable
+   - Reference: FinOps fo-06, fo-07
+
+### Security & Privacy (Security Architect Integration)
+
+3. **Implement Access Control**
+   - Use least privilege principle
+   - Enable Azure AD authentication
+   - Audit access logs
+   - Reference: Security Architect sa-02 (IAM), sa-04
+
+4. **Data Protection**
+   - Encrypt data at rest and in transit
+   - Scan for PII before processing
+   - Implement data retention policies
+   - Reference: Security Architect sa-01 (PII Detection)
+
+### Quality & Governance (Data Engineer Integration)
+
+5. **Ensure Data Quality**
+   - Validate inputs and outputs
+   - Implement quality gates
+   - Monitor data freshness
+   - Reference: Data Engineer de-03 (Data Quality)
+
+### Lifecycle Management (MLOps Integration)
+
+6. **Version Control**
+   - Version all configurations
+   - Track changes over time
+   - Enable rollback capability
+   - Reference: MLOps mo-03 (Versioning)
+
+7. **Continuous Monitoring**
+   - Track performance metrics
+   - Set up alerting
+   - Monitor for drift
+   - Reference: MLOps mo-06 (Monitoring)
+
+### Deployment & Operations (DevOps Integration)
+
+8. **Automate Deployment**
+   - Implement CI/CD pipelines
+   - Use infrastructure as code
+   - Enable blue-green deployments
+   - Reference: DevOps do-01 (CI/CD), do-03 (IaC)
+
+9. **Observability**
+   - Implement distributed tracing
+   - Set up dashboards
+   - Enable logging and metrics
+   - Reference: DevOps do-08 (Monitoring)
+
+### Azure-Specific Best Practices
+
+10. **Leverage Azure Services**
+    - Use managed services where possible
+    - Implement Azure Policy for governance
+    - Enable Azure Monitor integration
+    - Use managed identities for authentication
+
+## 💰 Cost Optimization Examples
+
+### Cost Tracking
+```python
+from finops_tracker import CostTracker
+
+tracker = CostTracker()
+
+@tracker.track_costs
+def run_operation(params):
+    # Your operation here
+    result = execute_operation(params)
+    return result
+
+# Monthly report
+report = tracker.monthly_report()
+print(f"Total cost: ${report.total_cost:.2f}")
+print(f"Cost per operation: ${report.avg_cost:.4f}")
+```
+
+## 🔒 Security Best Practices Examples
+
+### Access Control Implementation
+```python
+from azure.identity import DefaultAzureCredential
+from security_manager import AccessControl
+
+credential = DefaultAzureCredential()
+access_control = AccessControl(credential)
+
+# Validate access before operation
+@access_control.require_role("operator")
+def sensitive_operation(data):
+    # Operation logic
+    return process_data(data)
+```
+
+## 📊 Enhanced Metrics & Monitoring
+
+| Metric Category | Metric | Target | Tool |
+|-----------------|--------|--------|------|
+| **Performance** | Execution time (p95) | <5s | Azure Monitor |
+| | Success rate | >99% | Custom metrics |
+| **Cost** | Cost per operation | <$0.05 | FinOps dashboard |
+| | Resource utilization | >75% | Azure Monitor |
+| **Quality** | Error rate | <1% | App Insights |
+| | Data quality score | >95% | Quality tracker |
+| **Security** | Access violations | 0 | Security logs |
+| | Compliance score | 100% | Audit system |
+
+## 🚀 Deployment Pipeline
+
+### CI/CD Example
+```yaml
+# .github/workflows/deploy-09-automated-retraining.yml
+name: Deploy Automated Retraining Pipelines
+
+on:
+  push:
+    paths:
+      - 'mlops/skills/09-automated-retraining/**'
+    branches:
+      - main
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+      - name: Run tests
+        run: pytest tests/ -v
+      - name: Security scan
+        run: python scripts/security_scan.py
+      - name: Cost validation
+        run: python scripts/validate_costs.py
+
+  deploy:
+    needs: test
+    runs-on: ubuntu-latest
+    steps:
+      - name: Deploy to Azure
+        run: |
+          az deployment group create \
+            --resource-group rg-mlops \
+            --template-file infra/main.bicep
+      - name: Monitor deployment
+        run: python scripts/monitor_health.py --duration 10m
+```
+
+## 🔄 Integration Workflow
+
+### End-to-End Process
+```
+1. Input Validation
+   ↓
+2. Security Checks (sa-01, sa-02)
+   ↓
+3. Main Processing
+   ↓
+4. Quality Validation (de-03)
+   ↓
+5. Cost Tracking (fo-01)
+   ↓
+6. Monitoring & Logging (do-08)
+   ↓
+7. Output Delivery
+```
+
+## 🎯 Quick Wins
+
+1. **Enable cost tracking** - Monitor spending from day one
+2. **Implement security scanning** - Catch vulnerabilities early
+3. **Set up monitoring** - Full visibility into operations
+4. **Automate deployment** - Faster, safer releases
+5. **Add quality gates** - Prevent bad data from propagating
+6. **Enable caching** - Reduce redundant operations
+7. **Implement retries** - Improve reliability
+8. **Set up alerting** - Know about issues immediately
+
+## 🔗 Related Skills
+- mo-06
+
+---
+
+**Skill ID**: `09-automated-retraining`  
+**Complexity**: Expert  
+**Dependencies**: mo-06  
+**Business Value**: High  
+**Estimated Implementation Time**: 1-2 weeks

package/tech_hub_skills/roles/platform-engineer/skills/01-internal-developer-platform/README.md

@@ -0,0 +1,153 @@
+# pe-01: Internal Developer Platform (IDP)
+
+## Overview
+
+Build developer portals using Backstage for service catalog, golden path templates, self-service provisioning, and platform documentation.
+
+## Key Capabilities
+
+- **Developer Portal**: Centralized platform UI (Backstage)
+- **Service Catalog**: All services, APIs, documentation
+- **Golden Path Templates**: Scaffolding for new services
+- **Self-Service Provisioning**: One-click infrastructure
+- **Platform Documentation**: Unified docs portal
+
+## Tools & Technologies
+
+- **Backstage**: Open-source developer portal
+- **Port**: Developer portal platform
+- **Humanitec**: Platform orchestrator
+- **Kratix**: Platform-as-a-product framework
+
+## Implementation
+
+### 1. Backstage Setup
+
+```yaml
+# app-config.yaml
+app:
+  title: Tech Hub Platform
+  baseUrl: http://localhost:3000
+
+organization:
+  name: Tech Innovation Hub
+
+backend:
+  baseUrl: http://localhost:7007
+  listen:
+    port: 7007
+  database:
+    client: pg
+    connection:
+      host: ${POSTGRES_HOST}
+      port: ${POSTGRES_PORT}
+
+catalog:
+  providers:
+    github:
+      organization: 'your-org'
+      catalogPath: '/catalog-info.yaml'
+```
+
+### 2. Service Catalog
+
+```yaml
+# catalog-info.yaml
+apiVersion: backstage.io/v1alpha1
+kind: Component
+metadata:
+  name: customer-api
+  description: Customer management API
+  annotations:
+    github.com/project-slug: your-org/customer-api
+    sonarqube.org/project-key: customer-api
+spec:
+  type: service
+  lifecycle: production
+  owner: team-platform
+  system: customer-domain
+  providesApis:
+    - customer-api
+  consumesApis:
+    - auth-api
+```
+
+### 3. Golden Path Template
+
+```yaml
+# template.yaml
+apiVersion: scaffolder.backstage.io/v1beta3
+kind: Template
+metadata:
+  name: python-fastapi-service
+  title: Python FastAPI Service
+  description: Create a new Python FastAPI microservice
+spec:
+  owner: platform-team
+  type: service
+
+  parameters:
+    - title: Service Information
+      required:
+        - name
+        - owner
+      properties:
+        name:
+          title: Service Name
+          type: string
+          description: Unique name for the service
+        owner:
+          title: Owner
+          type: string
+          description: Team that owns this service
+
+  steps:
+    - id: fetch-template
+      name: Fetch Template
+      action: fetch:template
+      input:
+        url: ./skeleton
+        values:
+          name: ${{ parameters.name }}
+          owner: ${{ parameters.owner }}
+
+    - id: publish
+      name: Publish to GitHub
+      action: publish:github
+      input:
+        repoUrl: github.com?owner=your-org&repo=${{ parameters.name }}
+
+    - id: register
+      name: Register Component
+      action: catalog:register
+      input:
+        repoContentsUrl: ${{ steps.publish.output.repoContentsUrl }}
+        catalogInfoPath: '/catalog-info.yaml'
+```
+
+## Best Practices
+
+1. **Start Small**: Begin with service catalog, add features iteratively
+2. **Golden Paths**: Create templates for 80% of use cases
+3. **Self-Service**: Minimize manual ticket workflows
+4. **Measure Adoption**: Track active users and template usage
+5. **Documentation**: Keep docs updated and searchable
+
+## Cost Optimization
+
+- Host Backstage on Kubernetes spot instances
+- Use PostgreSQL managed service (cheaper than self-hosted)
+- Cache plugin data to reduce API calls
+- Right-size backend resources
+
+## Integration
+
+**Connects with:**
+- do-01 (CI/CD): Link to deployment pipelines
+- do-02 (Kubernetes): Service deployment info
+- pe-03 (SLO): Display SLO status
+- dg-01 (Catalog): Link to data catalog
+
+## Quick Win
+
+Deploy Backstage with GitHub integration, import 5 services to catalog, show team the unified view of their services.

package/tech_hub_skills/roles/platform-engineer/skills/02-self-service-infrastructure/README.md

@@ -0,0 +1,57 @@
+# pe-02: Self-Service Infrastructure
+
+## Overview
+
+Enable developers to provision namespaces, databases, secrets, and environments through self-service automation.
+
+## Key Capabilities
+
+- **Namespace Provisioning**: Auto-create K8s namespaces
+- **Database Provisioning**: Self-service DB creation
+- **Secret Management**: Automated secret injection
+- **Resource Quotas**: Automatic quota management
+- **Environment Management**: Dev/staging/prod provisioning
+
+## Implementation
+
+```python
+# Self-service namespace provisioning
+from kubernetes import client, config
+
+def provision_namespace(team_name, environment):
+    """Create namespace with quotas and RBAC"""
+    config.load_kube_config()
+    v1 = client.CoreV1Api()
+
+    # Create namespace
+    namespace = client.V1Namespace(
+        metadata=client.V1ObjectMeta(
+            name=f"{team_name}-{environment}",
+            labels={
+                "team": team_name,
+                "environment": environment
+            }
+        )
+    )
+    v1.create_namespace(namespace)
+
+    # Apply resource quota
+    quota = client.V1ResourceQuota(
+        metadata=client.V1ObjectMeta(name="default-quota"),
+        spec=client.V1ResourceQuotaSpec(
+            hard={
+                "requests.cpu": "10",
+                "requests.memory": "20Gi",
+                "pods": "50"
+            }
+        )
+    )
+    v1.create_namespaced_resource_quota(
+        namespace=namespace.metadata.name,
+        body=quota
+    )
+```
+
+## Integration
+
+**Connects with:** do-02 (Kubernetes), sa-06 (Secrets), pe-01 (IDP)

package/tech_hub_skills/roles/platform-engineer/skills/03-slo-sli-management/README.md

@@ -0,0 +1,59 @@
+# pe-03: SLO/SLI Management
+
+## Overview
+
+Define and track Service Level Objectives (SLOs), manage error budgets, instrument SLIs, and create SLO-based alerting.
+
+## Key Capabilities
+
+- **SLO Definition**: Availability, latency, error rate targets
+- **Error Budget Management**: Track remaining error budget
+- **SLI Instrumentation**: Collect service-level indicators
+- **SLO-Based Alerting**: Alert on error budget burn
+- **SLO Dashboards**: Visualize SLO compliance
+
+## Implementation
+
+```yaml
+# SLO definition (Sloth)
+version: prometheus/v1
+service: customer-api
+slos:
+  - name: requests-availability
+    objective: 99.9
+    description: 99.9% of requests successful
+    sli:
+      events:
+        error_query: |
+          sum(rate(http_requests_total{job="customer-api",code=~"5.."}[5m]))
+        total_query: |
+          sum(rate(http_requests_total{job="customer-api"}[5m]))
+    alerting:
+      name: CustomerAPIHighErrorRate
+      labels:
+        severity: page
+      annotations:
+        summary: High error rate on customer API
+```
+
+```python
+# Error budget calculation
+def calculate_error_budget(slo_target, time_window_days=30):
+    """Calculate remaining error budget"""
+    total_minutes = time_window_days * 24 * 60
+    allowed_downtime = total_minutes * (1 - slo_target/100)
+
+    actual_downtime = get_actual_downtime(time_window_days)
+    remaining_budget = allowed_downtime - actual_downtime
+
+    return {
+        'allowed_downtime_minutes': allowed_downtime,
+        'actual_downtime_minutes': actual_downtime,
+        'remaining_budget_minutes': remaining_budget,
+        'budget_consumed_percent': (actual_downtime / allowed_downtime) * 100
+    }
+```
+
+## Integration
+
+**Connects with:** do-08 (Monitoring), pe-01 (IDP), pe-05 (Incident Management)

package/tech_hub_skills/roles/platform-engineer/skills/04-developer-experience/README.md

@@ -0,0 +1,57 @@
+# pe-04: Developer Experience
+
+## Overview
+
+Improve developer velocity through automated onboarding, documentation-as-code, CLI tools, and DORA metrics tracking.
+
+## Key Capabilities
+
+- **Automated Onboarding**: Zero-to-commit in < 1 hour
+- **Documentation-as-Code**: Docs in git, versioned
+- **Developer CLI**: Unified command-line tools
+- **DORA Metrics**: Deployment frequency, lead time, MTTR, change fail rate
+- **Feedback Collection**: Regular developer surveys
+
+## Implementation
+
+```bash
+# Developer CLI
+#!/bin/bash
+# platform-cli
+
+case $1 in
+  create-service)
+    backstage scaffold $2 --template python-fastapi
+    ;;
+  deploy)
+    kubectl apply -f k8s/ --namespace=$2
+    ;;
+  logs)
+    kubectl logs -f deployment/$2 -n $3
+    ;;
+  metrics)
+    open "https://grafana.company.com/d/dora-metrics"
+    ;;
+esac
+```
+
+```python
+# DORA metrics collection
+def calculate_dora_metrics(team_name, days=30):
+    """Calculate DORA metrics"""
+    deployments = get_deployments(team_name, days)
+    incidents = get_incidents(team_name, days)
+
+    metrics = {
+        'deployment_frequency': len(deployments) / days,
+        'lead_time_hours': sum(d.lead_time for d in deployments) / len(deployments),
+        'mttr_hours': sum(i.resolution_time for i in incidents) / len(incidents) if incidents else 0,
+        'change_fail_rate': len([d for d in deployments if d.failed]) / len(deployments)
+    }
+
+    return metrics
+```
+
+## Integration
+
+**Connects with:** pe-01 (IDP), do-01 (CI/CD), pe-05 (Incident Management)

package/tech_hub_skills/roles/platform-engineer/skills/05-incident-management/README.md

@@ -0,0 +1,73 @@
+# pe-05: Incident Management
+
+## Overview
+
+On-call management, incident response procedures, postmortem templates, runbook automation, and alert routing.
+
+## Key Capabilities
+
+- **On-Call Management**: Rotation schedules
+- **Incident Response**: Clear escalation procedures
+- **Postmortem Templates**: Blameless retrospectives
+- **Runbook Automation**: Auto-remediation
+- **Alert Routing**: Intelligent alert distribution
+
+## Implementation
+
+```yaml
+# PagerDuty incident response
+services:
+  - name: customer-api
+    escalation_policy: platform-team
+    alert_grouping: intelligent
+    incident_urgency_rule:
+      type: constant
+      urgency: high
+
+# Runbook automation
+runbooks:
+  - name: high-memory-usage
+    trigger: memory_usage > 90%
+    actions:
+      - restart_pod
+      - scale_replicas: 2
+      - notify_oncall
+```
+
+```python
+# Postmortem template generator
+def create_postmortem(incident_id):
+    """Generate postmortem document"""
+    incident = get_incident(incident_id)
+
+    template = f"""
+# Incident Postmortem: {incident.title}
+
+## Incident Summary
+- **Date**: {incident.date}
+- **Duration**: {incident.duration}
+- **Severity**: {incident.severity}
+- **Responders**: {incident.responders}
+
+## Timeline
+{incident.timeline}
+
+## Root Cause
+[To be filled]
+
+## Resolution
+{incident.resolution}
+
+## Action Items
+- [ ] TODO 1
+- [ ] TODO 2
+
+## Lessons Learned
+[To be filled]
+"""
+    return template
+```
+
+## Integration
+
+**Connects with:** pe-03 (SLO), do-08 (Monitoring), pe-01 (IDP)