tech-hub-skills 1.2.0 → 1.5.2

package/{tech_hub_skills → .claude}/roles/data-governance/skills/01-data-catalog/README.md

@@ -1,112 +1,112 @@
-# dg-01: Data Catalog
-
-## Overview
-
-Build enterprise data catalogs for asset discovery, metadata management, and data classification.
-
-## Key Capabilities
-
-- **Asset Registration**: Automated discovery and registration of data assets
-- **Metadata Management**: Technical, business, and operational metadata
-- **Data Classification**: Automatic classification (PII, confidential, public)
-- **Search & Discovery**: Powerful search capabilities for data consumers
-- **Business Glossary**: Standardized business terminology
-
-## Tools & Technologies
-
-- **Azure Purview**: Enterprise data catalog
-- **DataHub**: Open-source metadata platform
-- **Amundsen**: Lyft's data discovery platform
-- **Collibra**: Data governance platform
-
-## Implementation
-
-### 1. Asset Registration
-
-```python
-# Automated asset registration
-from azure.purview.catalog import PurviewCatalogClient
-
-def register_data_asset(asset_name, asset_type, location):
-    """Register data asset in catalog"""
-    client = PurviewCatalogClient()
-
-    asset = {
-        "typeName": asset_type,
-        "attributes": {
-            "name": asset_name,
-            "qualifiedName": f"{location}/{asset_name}",
-            "location": location
-        }
-    }
-
-    return client.entity.create_or_update(entity=asset)
-```
-
-### 2. Metadata Management
-
-```python
-# Add business metadata
-def add_business_metadata(asset_id, owner, description, tags):
-    """Enrich asset with business context"""
-    metadata = {
-        "businessOwner": owner,
-        "description": description,
-        "tags": tags,
-        "certification": "certified"
-    }
-
-    return client.entity.add_business_metadata(
-        guid=asset_id,
-        business_metadata=metadata
-    )
-```
-
-### 3. Data Classification
-
-```python
-# Automatic classification
-def classify_data(asset_id):
-    """Apply automatic classification based on content"""
-    classifications = []
-
-    # Scan for PII
-    if contains_pii(asset_id):
-        classifications.append("PII")
-
-    # Scan for confidential data
-    if contains_confidential(asset_id):
-        classifications.append("Confidential")
-
-    return client.entity.add_classifications(
-        guid=asset_id,
-        classifications=classifications
-    )
-```
-
-## Best Practices
-
-1. **Automate Discovery** - Use scanners to auto-discover assets
-2. **Enrich Metadata** - Add business context, not just technical
-3. **Clear Ownership** - Every asset needs a business owner
-4. **Regular Updates** - Keep metadata fresh and relevant
-5. **User Training** - Train users on search capabilities
-
-## Cost Optimization
-
-- Use Azure Purview Standard tier for < 100k assets
-- Schedule scans during off-peak hours
-- Use incremental scans instead of full scans
-- Archive unused asset metadata
-
-## Integration
-
-**Connects with:**
-- de-01 (Lakehouse): Catalog lakehouse tables
-- sa-01 (PII Detection): Auto-classify PII data
-- dg-02 (Lineage): Link to lineage tracking
-- dg-03 (Quality): Link quality scores
-
-## Quick Win
-
-Start with top 10 critical datasets, manually catalog them with rich metadata, then expand automated discovery.
+# dg-01: Data Catalog
+
+## Overview
+
+Build enterprise data catalogs for asset discovery, metadata management, and data classification.
+
+## Key Capabilities
+
+- **Asset Registration**: Automated discovery and registration of data assets
+- **Metadata Management**: Technical, business, and operational metadata
+- **Data Classification**: Automatic classification (PII, confidential, public)
+- **Search & Discovery**: Powerful search capabilities for data consumers
+- **Business Glossary**: Standardized business terminology
+
+## Tools & Technologies
+
+- **Azure Purview**: Enterprise data catalog
+- **DataHub**: Open-source metadata platform
+- **Amundsen**: Lyft's data discovery platform
+- **Collibra**: Data governance platform
+
+## Implementation
+
+### 1. Asset Registration
+
+```python
+# Automated asset registration
+from azure.purview.catalog import PurviewCatalogClient
+
+def register_data_asset(asset_name, asset_type, location):
+    """Register data asset in catalog"""
+    client = PurviewCatalogClient()
+
+    asset = {
+        "typeName": asset_type,
+        "attributes": {
+            "name": asset_name,
+            "qualifiedName": f"{location}/{asset_name}",
+            "location": location
+        }
+    }
+
+    return client.entity.create_or_update(entity=asset)
+```
+
+### 2. Metadata Management
+
+```python
+# Add business metadata
+def add_business_metadata(asset_id, owner, description, tags):
+    """Enrich asset with business context"""
+    metadata = {
+        "businessOwner": owner,
+        "description": description,
+        "tags": tags,
+        "certification": "certified"
+    }
+
+    return client.entity.add_business_metadata(
+        guid=asset_id,
+        business_metadata=metadata
+    )
+```
+
+### 3. Data Classification
+
+```python
+# Automatic classification
+def classify_data(asset_id):
+    """Apply automatic classification based on content"""
+    classifications = []
+
+    # Scan for PII
+    if contains_pii(asset_id):
+        classifications.append("PII")
+
+    # Scan for confidential data
+    if contains_confidential(asset_id):
+        classifications.append("Confidential")
+
+    return client.entity.add_classifications(
+        guid=asset_id,
+        classifications=classifications
+    )
+```
+
+## Best Practices
+
+1. **Automate Discovery** - Use scanners to auto-discover assets
+2. **Enrich Metadata** - Add business context, not just technical
+3. **Clear Ownership** - Every asset needs a business owner
+4. **Regular Updates** - Keep metadata fresh and relevant
+5. **User Training** - Train users on search capabilities
+
+## Cost Optimization
+
+- Use Azure Purview Standard tier for < 100k assets
+- Schedule scans during off-peak hours
+- Use incremental scans instead of full scans
+- Archive unused asset metadata
+
+## Integration
+
+**Connects with:**
+- de-01 (Lakehouse): Catalog lakehouse tables
+- sa-01 (PII Detection): Auto-classify PII data
+- dg-02 (Lineage): Link to lineage tracking
+- dg-03 (Quality): Link quality scores
+
+## Quick Win
+
+Start with top 10 critical datasets, manually catalog them with rich metadata, then expand automated discovery.

package/{tech_hub_skills → .claude}/roles/data-governance/skills/02-data-lineage/README.md

@@ -1,129 +1,129 @@
-# dg-02: Data Lineage
-
-## Overview
-
-Track end-to-end data lineage for impact analysis, root cause analysis, and regulatory compliance.
-
-## Key Capabilities
-
-- **End-to-End Lineage**: From source to consumption
-- **Impact Analysis**: Understand downstream impacts
-- **Root Cause Analysis**: Trace issues to source
-- **Column-Level Lineage**: Field-level tracking
-- **Transformation Documentation**: Track data transformations
-
-## Tools & Technologies
-
-- **Azure Purview**: Native lineage tracking
-- **OpenLineage**: Open standard for lineage
-- **Marquez**: Metadata service for lineage
-- **Spline**: Spark lineage tracking
-
-## Implementation
-
-### 1. Lineage Extraction
-
-```python
-# Extract lineage from Spark jobs
-from spline import SplineAgent
-
-def track_spark_lineage(spark_session):
-    """Enable lineage tracking for Spark"""
-    spark_session.sparkContext.setLogLevel("INFO")
-
-    # Initialize Spline agent
-    SplineAgent.builder() \
-        .appName("data-pipeline") \
-        .mode("REQUIRED") \
-        .url("http://spline-server:9090") \
-        .build()
-```
-
-### 2. Column-Level Lineage
-
-```sql
--- Azure Purview automatically tracks column lineage
--- Example transformation with lineage
-CREATE VIEW customer_360 AS
-SELECT
-    c.customer_id,
-    c.first_name || ' ' || c.last_name as full_name,  -- Lineage: derived
-    o.total_orders,
-    p.total_payments
-FROM customers c
-LEFT JOIN order_summary o ON c.customer_id = o.customer_id
-LEFT JOIN payment_summary p ON c.customer_id = p.customer_id;
-```
-
-### 3. Impact Analysis
-
-```python
-# Find downstream dependencies
-def get_downstream_impact(asset_id):
-    """Find all downstream assets affected by changes"""
-    lineage = client.lineage.get_lineage(
-        guid=asset_id,
-        direction="OUTPUT",
-        depth=10
-    )
-
-    downstream_assets = []
-    for entity in lineage['guidEntityMap'].values():
-        downstream_assets.append({
-            'name': entity['attributes']['name'],
-            'type': entity['typeName'],
-            'owner': entity.get('attributes', {}).get('owner')
-        })
-
-    return downstream_assets
-```
-
-### 4. OpenLineage Integration
-
-```python
-# Emit lineage events using OpenLineage
-from openlineage.client import OpenLineageClient
-from openlineage.client.run import RunEvent, RunState, Run, Job
-
-def emit_lineage_event(job_name, inputs, outputs):
-    """Emit lineage event to OpenLineage"""
-    client = OpenLineageClient(url="http://lineage-api:5000")
-
-    event = RunEvent(
-        eventType=RunState.COMPLETE,
-        eventTime="2025-01-01T00:00:00Z",
-        run=Run(runId=str(uuid.uuid4())),
-        job=Job(namespace="production", name=job_name),
-        inputs=inputs,
-        outputs=outputs
-    )
-
-    client.emit(event)
-```
-
-## Best Practices
-
-1. **Automate Collection** - Manual lineage doesn't scale
-2. **Column-Level Tracking** - For sensitive data, track field-level
-3. **Version Control** - Track lineage changes over time
-4. **Clear Visualization** - Make lineage easy to understand
-5. **Regular Validation** - Verify lineage accuracy
-
-## Cost Optimization
-
-- Use incremental lineage updates
-- Archive old lineage data after retention period
-- Cache frequently accessed lineage queries
-- Use materialized views for complex lineage
-
-## Integration
-
-**Connects with:**
-- de-02 (ETL): Track pipeline lineage
-- dg-01 (Catalog): Link assets to lineage
-- ml-02 (Feature Engineering): Track feature lineage
-- ai-02 (RAG): Track document lineage
-
-## Quick Win
-
-Start with 1 critical data pipeline, manually document lineage, validate accuracy, then automate extraction.
+# dg-02: Data Lineage
+
+## Overview
+
+Track end-to-end data lineage for impact analysis, root cause analysis, and regulatory compliance.
+
+## Key Capabilities
+
+- **End-to-End Lineage**: From source to consumption
+- **Impact Analysis**: Understand downstream impacts
+- **Root Cause Analysis**: Trace issues to source
+- **Column-Level Lineage**: Field-level tracking
+- **Transformation Documentation**: Track data transformations
+
+## Tools & Technologies
+
+- **Azure Purview**: Native lineage tracking
+- **OpenLineage**: Open standard for lineage
+- **Marquez**: Metadata service for lineage
+- **Spline**: Spark lineage tracking
+
+## Implementation
+
+### 1. Lineage Extraction
+
+```python
+# Extract lineage from Spark jobs
+from spline import SplineAgent
+
+def track_spark_lineage(spark_session):
+    """Enable lineage tracking for Spark"""
+    spark_session.sparkContext.setLogLevel("INFO")
+
+    # Initialize Spline agent
+    SplineAgent.builder() \
+        .appName("data-pipeline") \
+        .mode("REQUIRED") \
+        .url("http://spline-server:9090") \
+        .build()
+```
+
+### 2. Column-Level Lineage
+
+```sql
+-- Azure Purview automatically tracks column lineage
+-- Example transformation with lineage
+CREATE VIEW customer_360 AS
+SELECT
+    c.customer_id,
+    c.first_name || ' ' || c.last_name as full_name,  -- Lineage: derived
+    o.total_orders,
+    p.total_payments
+FROM customers c
+LEFT JOIN order_summary o ON c.customer_id = o.customer_id
+LEFT JOIN payment_summary p ON c.customer_id = p.customer_id;
+```
+
+### 3. Impact Analysis
+
+```python
+# Find downstream dependencies
+def get_downstream_impact(asset_id):
+    """Find all downstream assets affected by changes"""
+    lineage = client.lineage.get_lineage(
+        guid=asset_id,
+        direction="OUTPUT",
+        depth=10
+    )
+
+    downstream_assets = []
+    for entity in lineage['guidEntityMap'].values():
+        downstream_assets.append({
+            'name': entity['attributes']['name'],
+            'type': entity['typeName'],
+            'owner': entity.get('attributes', {}).get('owner')
+        })
+
+    return downstream_assets
+```
+
+### 4. OpenLineage Integration
+
+```python
+# Emit lineage events using OpenLineage
+from openlineage.client import OpenLineageClient
+from openlineage.client.run import RunEvent, RunState, Run, Job
+
+def emit_lineage_event(job_name, inputs, outputs):
+    """Emit lineage event to OpenLineage"""
+    client = OpenLineageClient(url="http://lineage-api:5000")
+
+    event = RunEvent(
+        eventType=RunState.COMPLETE,
+        eventTime="2025-01-01T00:00:00Z",
+        run=Run(runId=str(uuid.uuid4())),
+        job=Job(namespace="production", name=job_name),
+        inputs=inputs,
+        outputs=outputs
+    )
+
+    client.emit(event)
+```
+
+## Best Practices
+
+1. **Automate Collection** - Manual lineage doesn't scale
+2. **Column-Level Tracking** - For sensitive data, track field-level
+3. **Version Control** - Track lineage changes over time
+4. **Clear Visualization** - Make lineage easy to understand
+5. **Regular Validation** - Verify lineage accuracy
+
+## Cost Optimization
+
+- Use incremental lineage updates
+- Archive old lineage data after retention period
+- Cache frequently accessed lineage queries
+- Use materialized views for complex lineage
+
+## Integration
+
+**Connects with:**
+- de-02 (ETL): Track pipeline lineage
+- dg-01 (Catalog): Link assets to lineage
+- ml-02 (Feature Engineering): Track feature lineage
+- ai-02 (RAG): Track document lineage
+
+## Quick Win
+
+Start with 1 critical data pipeline, manually document lineage, validate accuracy, then automate extraction.