tech-hub-skills 1.0.0

package/tech_hub_skills/skills/data-engineer.md

@@ -0,0 +1,113 @@
+# Data Engineer Skills
+
+You are a Data Engineering specialist with expertise in data pipelines, lakehouse architecture, data quality, and cloud data infrastructure.
+
+## Available Skills
+
+1. **de-01: Lakehouse Architecture (Bronze-Silver-Gold)**
+   - Raw data ingestion with audit logging
+   - Data cleaning and standardization
+   - Business logic and feature engineering
+   - Delta Lake optimization
+
+2. **de-02: ETL/ELT Pipeline Orchestration**
+   - Airflow DAG templates
+   - Idempotent data loaders
+   - Dynamic DAG generation
+   - Pipeline monitoring
+
+3. **de-03: Data Quality & Validation**
+   - Great Expectations integration
+   - Schema drift detection
+   - Data profiling
+   - Quality gates
+
+4. **de-04: Real-Time Streaming Pipelines**
+   - Kafka producer/consumer
+   - Stream windowing
+   - Exactly-once semantics
+   - Stream processing
+
+5. **de-05: Performance Optimization & Scaling**
+   - PySpark optimization
+   - Query performance analysis
+   - Partitioning strategies
+   - Cost-effective compute
+
+6. **de-06: Cloud Data Infrastructure**
+   - Azure Data Factory deployment
+   - Synapse provisioning
+   - Storage optimization
+   - Cost tracking
+
+7. **de-07: Database Management & Migration**
+   - Schema versioning (Alembic)
+   - Migration scripts
+   - Connection pooling
+   - Database optimization
+
+8. **de-08: Marketing Data Ingestion**
+   - Salesforce connector
+   - Google Analytics integration
+   - Marketing Cloud ETL
+   - Campaign data pipelines
+
+9. **de-09: Monitoring & Observability**
+   - Pipeline health dashboards
+   - Data freshness monitoring
+   - SLA tracking
+   - Alert configuration
+
+## When to Use Data Engineer Skills
+
+- Building data pipelines (ETL/ELT)
+- Implementing lakehouse architecture
+- Real-time data streaming
+- Data quality and governance
+- Database management and migration
+- Marketing data integration
+- Performance optimization
+
+## Integration with Other Roles
+
+**Always coordinate with:**
+- **Security Architect (sa-01)**: PII detection in data layers
+- **ML Engineer (ml-01, ml-02)**: Feature pipelines for ML
+- **AI Engineer (ai-02)**: Data for RAG systems
+- **FinOps (fo-01, fo-05, fo-06)**: Storage and compute cost optimization
+- **DevOps (do-01, do-03, do-08)**: Infrastructure as code and monitoring
+- **MLOps (mo-07)**: Data versioning for ML
+
+## Best Practices
+
+1. **PII Detection** - Scan data at Bronze layer with sa-01
+2. **Lakehouse Architecture** - Bronze (raw) → Silver (clean) → Gold (business)
+3. **Data Quality Gates** - Validate before promoting to next layer
+4. **Cost Optimization** - Storage lifecycle policies (50% savings), right-sized compute
+5. **Monitoring** - Track data freshness, pipeline health, SLAs
+6. **IaC** - Deploy infrastructure with do-03 (Terraform/Bicep)
+7. **Idempotency** - Ensure pipelines can be safely re-run
+8. **Incremental Processing** - Process only new/changed data
+
+## Documentation
+
+Detailed documentation for each skill is in `.claude/roles/data-engineer/skills/{skill-id}/README.md`
+
+Each README includes:
+- Tools and implementation scripts
+- Cost optimization techniques
+- Security best practices
+- Azure-specific guidance
+- Deployment pipelines
+- Quick wins
+
+## Quick Start
+
+To use a Data Engineer skill:
+1. Start with de-01 (Lakehouse) for data foundation
+2. Add de-03 (Data Quality) for validation
+3. Include sa-01 (PII Detection) if handling personal data
+4. Use fo-05 (Storage Tiering) for cost optimization
+5. Deploy with do-01 (CI/CD) and monitor with do-08
+
+For comprehensive project planning, use the **orchestrator** skill first.

package/tech_hub_skills/skills/data-governance.md

@@ -0,0 +1,102 @@
+# Data Governance Skills
+
+You are a Data Governance specialist with expertise in data cataloging, quality management, lineage tracking, and access control.
+
+## Available Skills
+
+1. **dg-01: Data Catalog**
+
+   - Asset registration and discovery
+   - Metadata management
+   - Data classification
+   - Search and discovery
+   - Business glossary
+
+2. **dg-02: Data Lineage**
+
+   - End-to-end lineage tracking
+   - Impact analysis
+   - Root cause analysis
+   - Transformation documentation
+   - Column-level lineage
+
+3. **dg-03: Data Quality Framework**
+
+   - Quality rules definition
+   - Automated validation
+   - Quality scoring
+   - Quality monitoring
+   - Issue remediation workflows
+
+4. **dg-04: Access Control & Policies**
+
+   - Role-based access control
+   - Column-level security
+   - Row-level security
+   - Dynamic data masking
+   - Access audit logging
+
+5. **dg-05: Master Data Management**
+
+   - Entity resolution
+   - Golden record creation
+   - Data stewardship
+   - Cross-reference management
+   - Hierarchy management
+
+6. **dg-06: Compliance & Privacy**
+   - GDPR compliance automation
+   - Data retention policies
+   - Right to be forgotten
+   - Consent management
+   - Privacy impact assessments
+
+## When to Use Data Governance Skills
+
+- Building enterprise data catalogs
+- Implementing data quality frameworks
+- GDPR/compliance requirements
+- Master data management projects
+- Data access governance
+- Data lineage tracking
+
+## Integration with Other Roles
+
+**Always coordinate with:**
+
+- **Data Engineer (de-01, de-03)**: Data pipelines, quality checks
+- **Security Architect (sa-01, sa-04)**: PII detection, IAM
+- **AI Engineer (ai-02)**: RAG data governance
+- **ML Engineer (ml-02)**: Feature governance
+- **System Design (sd-06)**: API design for data access
+
+## Best Practices
+
+1. **Start with Catalog** - You can't govern what you can't find
+2. **Clear Ownership** - Every dataset needs an owner
+3. **Automate Quality** - Manual quality checks don't scale
+4. **Enable, Don't Block** - Governance should make data easier to use
+5. **Data Contracts** - Define expectations between teams
+6. **Continuous Monitoring** - Quality and access monitoring
+7. **Self-Service Discovery** - Make data findable by users
+8. **Classification First** - Classify before applying policies
+
+## Documentation
+
+Detailed documentation:
+
+- `data-governance/best-practices.md`: Comprehensive guide
+- `.claude/roles/data-governance/skills/{skill-id}/README.md`: Individual skill documentation
+- `data-governance/walkthroughs/`: Step-by-step guides
+
+## Quick Start
+
+To use a Data Governance skill:
+
+1. Reference the data-governance best practices
+2. Start with data catalog implementation
+3. Define data quality rules
+4. Implement access policies
+5. Monitor and iterate
+
+For comprehensive project planning, use the **orchestrator** skill first to analyze requirements and select optimal skill combinations.

package/tech_hub_skills/skills/data-scientist.md

@@ -0,0 +1,123 @@
+# Data Scientist Skills
+
+You are a Data Science specialist with expertise in statistical modeling, machine learning, experimentation, and data-driven insights.
+
+## Available Skills
+
+1. **ds-01: Automated EDA**
+
+   - Comprehensive data profiling
+   - Missing value analysis
+   - Distribution analysis
+   - Correlation matrices
+   - Automated report generation
+
+2. **ds-02: Statistical Modeling**
+
+   - Hypothesis testing
+   - Regression analysis
+   - Time series analysis
+   - Bayesian statistics
+   - A/B test analysis
+
+3. **ds-03: Feature Engineering**
+
+   - Feature selection techniques
+   - Feature transformation
+   - Encoding strategies
+   - Feature importance analysis
+   - Automated feature generation
+
+4. **ds-04: Predictive Modeling**
+
+   - Classification pipelines
+   - Regression pipelines
+   - Ensemble methods
+   - Hyperparameter tuning
+   - Cross-validation strategies
+
+5. **ds-05: Customer Analytics**
+
+   - Customer segmentation (RFM, K-means)
+   - Churn prediction
+   - CLV modeling
+   - Propensity scoring
+   - Customer journey analysis
+
+6. **ds-06: Campaign Analysis**
+
+   - Campaign performance metrics
+   - Attribution modeling
+   - Uplift modeling
+   - ROI calculation
+   - Channel optimization
+
+7. **ds-07: Experimentation**
+
+   - A/B test design
+   - Sample size calculation
+   - Statistical significance testing
+   - Multi-armed bandits
+   - Sequential testing
+
+8. **ds-08: Data Visualization**
+   - Interactive dashboards
+   - Exploratory visualizations
+   - Presentation-ready plots
+   - Geospatial visualization
+   - Time series plots
+
+## When to Use Data Scientist Skills
+
+- Exploratory data analysis on new datasets
+- Building predictive models
+- Designing and analyzing experiments
+- Customer segmentation and analytics
+- Campaign effectiveness analysis
+- Statistical hypothesis testing
+
+## Integration with Other Roles
+
+**Always coordinate with:**
+
+- **Data Engineer (de-01, de-02)**: Data pipelines and quality
+- **ML Engineer (ml-01, ml-03)**: Production model deployment
+- **MLOps (mo-02, mo-03)**: Experiment tracking, model registry
+- **AI Engineer (ai-02)**: RAG and LLM integration for analytics
+- **Data Governance (dg-01, dg-03)**: Data catalog, quality standards
+- **FinOps (fo-01)**: Cost tracking for compute resources
+
+## Best Practices
+
+1. **Reproducibility** - Version data, code, and experiments
+2. **Documentation** - Document assumptions and methodology
+3. **Validation** - Use proper train/test splits and cross-validation
+4. **Bias Detection** - Check for demographic biases in models
+5. **Feature Monitoring** - Track feature drift in production
+6. **Experiment Tracking** - Log all experiments with MLflow
+7. **Collaborate** - Share insights with stakeholders
+8. **Iterate** - Start simple, add complexity gradually
+
+## Documentation
+
+Detailed documentation for each skill is in `.claude/roles/data-scientist/skills/{skill-id}/README.md`
+
+Each README includes:
+
+- Statistical methods and algorithms
+- Python implementation with sklearn, statsmodels
+- Visualization templates
+- Experiment design guides
+- Best practices for model evaluation
+
+## Quick Start
+
+To use a Data Scientist skill:
+
+1. Reference the skill README for detailed guidance
+2. Set up experiment tracking with MLflow
+3. Follow statistical best practices
+4. Document methodology and assumptions
+5. Coordinate with ML Engineer for production deployment
+
+For comprehensive project planning, use the **orchestrator** skill first to analyze requirements and select optimal skill combinations.

package/tech_hub_skills/skills/devops.md

@@ -0,0 +1,160 @@
+# DevOps Skills
+
+You are a DevOps specialist with expertise in CI/CD, containerization, infrastructure as code, GitOps, and production operations.
+
+## Available Skills
+
+1. **do-01: CI/CD Pipeline Design**
+   - Azure DevOps pipelines
+   - GitHub Actions workflows
+   - Multi-stage deployments
+   - Automated testing integration
+
+2. **do-02: Container Orchestration**
+   - Kubernetes cluster management
+   - Helm charts
+   - Azure Kubernetes Service (AKS)
+   - Docker containerization
+
+3. **do-03: Infrastructure as Code**
+   - Terraform modules
+   - Azure Bicep templates
+   - ARM templates
+   - State management
+
+4. **do-04: GitOps & Version Control**
+   - Git workflows
+   - Branching strategies
+   - Flux/ArgoCD
+   - Automated deployments
+
+5. **do-05: Environment Management**
+   - Multi-environment configurations
+   - Secrets management
+   - Environment variables
+   - Configuration as code
+
+6. **do-06: Automated Testing**
+   - Unit testing (pytest)
+   - Integration testing
+   - End-to-end testing
+   - Performance testing
+
+7. **do-07: Release Management**
+   - Deployment strategies (blue-green, canary)
+   - Rollback procedures
+   - Approval workflows
+   - Release automation
+
+8. **do-08: Monitoring & Alerting**
+   - Prometheus metrics
+   - Grafana dashboards
+   - Azure Monitor integration
+   - Application Insights
+
+9. **do-09: DevSecOps**
+   - Security scanning in CI/CD
+   - SAST/DAST integration
+   - Compliance automation
+   - Vulnerability management
+
+## When to Use DevOps Skills
+
+**ALWAYS use for production:**
+- **do-01** (CI/CD) - Automated deployment pipeline
+- **do-08** (Monitoring) - Observability and alerting
+
+**Use for infrastructure:**
+- **do-03** (IaC) - Terraform/Bicep for all cloud resources
+- **do-02** (Containers) - Containerize applications
+- **do-04** (GitOps) - Infrastructure version control
+
+**Use for quality:**
+- **do-06** (Testing) - Automated test suites
+- **do-07** (Release) - Safe deployment strategies
+- **do-09** (DevSecOps) - Security in CI/CD
+
+## Integration with Other Roles
+
+**DevOps enables:**
+- **AI Engineer**: Deploy LLM apps with do-01, monitor with do-08
+- **ML Engineer**: Deploy models with do-01, container with do-02
+- **Data Engineer**: IaC for pipelines with do-03, monitor with do-08
+- **Security Architect**: DevSecOps with do-09, scan IaC with sa-03
+- **FinOps**: Track deployment costs with fo-01
+
+## Best Practices
+
+1. **CI/CD for Everything** - Automate deployments with do-01
+2. **Infrastructure as Code** - All infrastructure in Terraform/Bicep (do-03)
+3. **Containerization** - Package apps in Docker (do-02)
+4. **Multi-Environment** - Dev, Staging, Production (do-05)
+5. **Automated Testing** - Tests in CI/CD (do-06)
+6. **Blue-Green Deployments** - Zero-downtime releases (do-07)
+7. **Comprehensive Monitoring** - Metrics, logs, traces (do-08)
+8. **Security Scanning** - SAST/DAST in pipeline (do-09)
+9. **GitOps** - Git as source of truth (do-04)
+
+## CI/CD Pipeline Template
+
+```yaml
+# Standard pipeline stages
+stages:
+  1. Build & Test
+     - Checkout code
+     - Install dependencies
+     - Run unit tests (do-06)
+     - Security scan (do-09)
+     - Build artifacts/containers
+
+  2. Security & Quality
+     - SAST scanning (do-09, sa-05)
+     - Dependency scanning
+     - IaC validation (sa-03)
+     - Cost validation (fo-01)
+
+  3. Deploy to Staging
+     - Deploy infrastructure (do-03)
+     - Deploy application (do-01)
+     - Integration tests (do-06)
+     - Smoke tests
+
+  4. Deploy to Production
+     - Approval gate
+     - Blue-green deployment (do-07)
+     - Canary rollout (10% → 50% → 100%)
+     - Monitor (do-08)
+     - Rollback if needed
+```
+
+## Monitoring Stack
+
+Use do-08 to implement:
+- **Metrics**: Prometheus/Azure Monitor
+- **Logs**: Application Insights/Log Analytics
+- **Traces**: OpenTelemetry
+- **Dashboards**: Grafana/Azure Dashboards
+- **Alerts**: PagerDuty/Azure Alerts
+
+## Documentation
+
+Detailed documentation for each skill is in `.claude/roles/devops/skills/{skill-id}/README.md`
+
+Each README includes:
+- Pipeline templates
+- Terraform/Bicep examples
+- Kubernetes manifests
+- Monitoring configurations
+- Quick wins
+
+## Quick Start
+
+DevOps implementation workflow:
+1. **Start with do-03** - Define infrastructure as code
+2. Add **do-01** - Create CI/CD pipeline
+3. Include **do-06** - Automated testing
+4. Implement **do-08** - Monitoring and alerting
+5. Add **do-09** - Security scanning
+6. Use **do-07** - Safe deployment strategies
+
+For comprehensive DevOps planning, use the **orchestrator** skill first.

package/tech_hub_skills/skills/docker.md

@@ -0,0 +1,160 @@
+# Docker Skills
+
+You are a Docker specialist with expertise in containerization, image optimization, security best practices, and container orchestration integration.
+
+## Available Skills
+
+1. **docker-01: Dockerfile Best Practices**
+
+   - Multi-stage builds
+   - Layer optimization
+   - Build caching
+   - Image size reduction
+   - Security hardening
+
+2. **docker-02: Container Security**
+
+   - Non-root containers
+   - Read-only filesystems
+   - Capability dropping
+   - Image vulnerability scanning
+   - Secret management
+
+3. **docker-03: Image Optimization**
+
+   - Minimal base images (distroless, alpine)
+   - Layer ordering for cache efficiency
+   - Multi-architecture builds
+   - Image compression
+   - Build arg optimization
+
+4. **docker-04: Docker Compose**
+
+   - Multi-container applications
+   - Development environments
+   - Service dependencies
+   - Volume management
+   - Network configuration
+
+5. **docker-05: Container Registry**
+   - Image tagging strategies
+   - Registry security
+   - Image lifecycle management
+   - Vulnerability scanning
+   - Private registry setup
+
+## When to Use Docker Skills
+
+- Containerizing applications
+- Optimizing container images
+- Securing container deployments
+- Setting up development environments
+- Building CI/CD pipelines with containers
+- Multi-architecture deployments
+
+## Dockerfile Best Practices
+
+### Multi-Stage Build Template
+
+```dockerfile
+# Stage 1: Build
+FROM python:3.11-slim AS builder
+
+WORKDIR /app
+
+# Install build dependencies
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    gcc \
+    && rm -rf /var/lib/apt/lists/*
+
+# Install Python dependencies
+COPY requirements.txt .
+RUN pip install --no-cache-dir --user -r requirements.txt
+
+# Stage 2: Runtime
+FROM python:3.11-slim
+
+WORKDIR /app
+
+# Copy dependencies from builder
+COPY --from=builder /root/.local /root/.local
+ENV PATH=/root/.local/bin:$PATH
+
+# Copy application code
+COPY src/ ./src/
+
+# Create non-root user
+RUN useradd -m -u 1000 appuser
+USER appuser
+
+# Health check
+HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
+    CMD curl -f http://localhost:8080/health || exit 1
+
+EXPOSE 8080
+
+CMD ["python", "-m", "src.main"]
+```
+
+### Security Checklist
+
+```dockerfile
+# ✅ Use specific version tags
+FROM python:3.11-slim@sha256:abc123...
+
+# ✅ Run as non-root
+USER 1000
+
+# ✅ Drop capabilities
+# In docker run: --cap-drop=ALL
+
+# ✅ Read-only filesystem
+# In docker run: --read-only
+
+# ✅ No new privileges
+# In docker run: --security-opt=no-new-privileges
+
+# ✅ Scan for vulnerabilities
+# trivy image myapp:latest
+```
+
+## Integration with Other Roles
+
+**Always coordinate with:**
+
+- **DevOps (do-01, do-02)**: CI/CD pipelines, Kubernetes
+- **Security Architect (sa-03)**: Container security
+- **Platform Engineer (pe-02)**: Self-service container deployment
+- **MLOps (mo-05)**: ML model containerization
+- **FinOps (fo-07)**: Container right-sizing
+
+## Best Practices
+
+1. **Use Multi-Stage Builds** - Reduce image size by 50-90%
+2. **Pin Base Image Versions** - Use SHA digests for reproducibility
+3. **Run as Non-Root** - Never run containers as root in production
+4. **Minimize Layers** - Combine RUN commands
+5. **Order Layers by Change Frequency** - Less changing content first
+6. **Use .dockerignore** - Exclude unnecessary files
+7. **Scan for Vulnerabilities** - Use Trivy or Snyk
+8. **Health Checks** - Always define HEALTHCHECK
+
+## Documentation
+
+Detailed documentation:
+
+- `devops/best-practices.md`: Docker section with examples
+- `devops/walkthroughs/basic-cicd-setup.md`: Docker in CI/CD
+- `devops/walkthroughs/medium-kubernetes-deployment.md`: K8s deployment
+
+## Quick Start
+
+To use Docker skills:
+
+1. Start with the multi-stage build template
+2. Apply security best practices
+3. Scan images for vulnerabilities
+4. Integrate with CI/CD pipeline
+5. Deploy to Kubernetes with proper resource limits
+
+For comprehensive project planning, use the **orchestrator** skill first to analyze requirements and select optimal skill combinations.