tech-hub-skills 1.2.0 → 1.5.2

package/{tech_hub_skills → .claude}/roles/data-governance/skills/03-data-quality-framework/README.md

@@ -1,182 +1,182 @@
-# dg-03: Data Quality Framework
-
-## Overview
-
-Implement automated data quality validation, scoring, monitoring, and issue remediation workflows.
-
-## Key Capabilities
-
-- **Quality Rules Definition**: Completeness, accuracy, consistency
-- **Automated Validation**: Real-time quality checks
-- **Quality Scoring**: Quantifiable quality metrics
-- **Quality Monitoring**: Continuous quality tracking
-- **Issue Remediation**: Workflows for quality issues
-
-## Tools & Technologies
-
-- **Great Expectations**: Python data validation
-- **Soda**: Data quality as code
-- **dbt tests**: Quality tests in dbt
-- **Azure Data Quality**: Native Azure solution
-
-## Implementation
-
-### 1. Quality Rules with Great Expectations
-
-```python
-# Define quality expectations
-import great_expectations as gx
-
-def create_quality_suite(context, table_name):
-    """Create data quality test suite"""
-    suite = context.add_expectation_suite(
-        expectation_suite_name=f"{table_name}_quality_suite"
-    )
-
-    validator = context.get_validator(
-        batch_request=batch_request,
-        expectation_suite_name=suite.expectation_suite_name
-    )
-
-    # Completeness checks
-    validator.expect_column_values_to_not_be_null(column="customer_id")
-    validator.expect_column_values_to_not_be_null(column="order_date")
-
-    # Accuracy checks
-    validator.expect_column_values_to_be_between(
-        column="age",
-        min_value=0,
-        max_value=120
-    )
-
-    # Consistency checks
-    validator.expect_column_values_to_match_regex(
-        column="email",
-        regex=r"^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$"
-    )
-
-    validator.save_expectation_suite()
-    return validator
-```
-
-### 2. Quality Scoring
-
-```python
-# Calculate quality score
-def calculate_quality_score(validation_results):
-    """Calculate overall quality score"""
-    total_checks = validation_results.statistics['evaluated_expectations']
-    successful_checks = validation_results.statistics['successful_expectations']
-
-    score = (successful_checks / total_checks) * 100
-
-    # Categorize quality
-    if score >= 95:
-        quality_level = "Excellent"
-    elif score >= 85:
-        quality_level = "Good"
-    elif score >= 70:
-        quality_level = "Acceptable"
-    else:
-        quality_level = "Poor"
-
-    return {
-        'score': score,
-        'level': quality_level,
-        'total_checks': total_checks,
-        'passed_checks': successful_checks
-    }
-```
-
-### 3. Automated Monitoring
-
-```python
-# Set up quality monitoring
-def setup_quality_monitoring(checkpoint_name):
-    """Configure automated quality monitoring"""
-    checkpoint_config = {
-        "name": checkpoint_name,
-        "config_version": 1.0,
-        "template_name": "default",
-        "run_name_template": "%Y%m%d-%H%M%S",
-        "validations": [
-            {
-                "batch_request": {
-                    "datasource_name": "production_data",
-                    "data_connector_name": "default_inferred_data_connector_name",
-                    "data_asset_name": "customers"
-                },
-                "expectation_suite_name": "customers_quality_suite"
-            }
-        ],
-        "action_list": [
-            {
-                "name": "store_validation_result",
-                "action": {"class_name": "StoreValidationResultAction"}
-            },
-            {
-                "name": "send_slack_notification",
-                "action": {
-                    "class_name": "SlackNotificationAction",
-                    "slack_webhook": "${SLACK_WEBHOOK}",
-                    "notify_on": "failure"
-                }
-            }
-        ]
-    }
-
-    context.add_checkpoint(**checkpoint_config)
-```
-
-### 4. Issue Remediation Workflow
-
-```python
-# Create remediation workflow
-def create_remediation_workflow(quality_issues):
-    """Create tickets for quality issues"""
-    from azure.devops import AzureDevOpsClient
-
-    client = AzureDevOpsClient()
-
-    for issue in quality_issues:
-        work_item = {
-            'title': f"Data Quality Issue: {issue['column']}",
-            'description': issue['description'],
-            'priority': issue['severity'],
-            'assigned_to': issue['data_owner'],
-            'tags': ['data-quality', issue['table']]
-        }
-
-        client.create_work_item(
-            project='DataGovernance',
-            work_item_type='Bug',
-            fields=work_item
-        )
-```
-
-## Best Practices
-
-1. **Start Simple** - Begin with critical fields, expand coverage
-2. **Automate Everything** - Manual checks don't scale
-3. **Clear Ownership** - Assign quality issues to data owners
-4. **Threshold Alerts** - Alert on quality score drops
-5. **Historical Tracking** - Monitor quality trends over time
-
-## Cost Optimization
-
-- Run quality checks incrementally (only new/changed data)
-- Use sampling for large datasets
-- Cache validation results
-- Right-size validation compute
-
-## Integration
-
-**Connects with:**
-- de-01 (Lakehouse): Validate lakehouse data
-- de-03 (Data Quality): Engineering quality checks
-- dg-01 (Catalog): Link quality scores to assets
-- dg-02 (Lineage): Trace quality issues to source
-
-## Quick Win
-
-Implement completeness checks on 5 critical fields in your most important table. Show before/after quality scores.
+# dg-03: Data Quality Framework
+
+## Overview
+
+Implement automated data quality validation, scoring, monitoring, and issue remediation workflows.
+
+## Key Capabilities
+
+- **Quality Rules Definition**: Completeness, accuracy, consistency
+- **Automated Validation**: Real-time quality checks
+- **Quality Scoring**: Quantifiable quality metrics
+- **Quality Monitoring**: Continuous quality tracking
+- **Issue Remediation**: Workflows for quality issues
+
+## Tools & Technologies
+
+- **Great Expectations**: Python data validation
+- **Soda**: Data quality as code
+- **dbt tests**: Quality tests in dbt
+- **Azure Data Quality**: Native Azure solution
+
+## Implementation
+
+### 1. Quality Rules with Great Expectations
+
+```python
+# Define quality expectations
+import great_expectations as gx
+
+def create_quality_suite(context, table_name):
+    """Create data quality test suite"""
+    suite = context.add_expectation_suite(
+        expectation_suite_name=f"{table_name}_quality_suite"
+    )
+
+    validator = context.get_validator(
+        batch_request=batch_request,
+        expectation_suite_name=suite.expectation_suite_name
+    )
+
+    # Completeness checks
+    validator.expect_column_values_to_not_be_null(column="customer_id")
+    validator.expect_column_values_to_not_be_null(column="order_date")
+
+    # Accuracy checks
+    validator.expect_column_values_to_be_between(
+        column="age",
+        min_value=0,
+        max_value=120
+    )
+
+    # Consistency checks
+    validator.expect_column_values_to_match_regex(
+        column="email",
+        regex=r"^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$"
+    )
+
+    validator.save_expectation_suite()
+    return validator
+```
+
+### 2. Quality Scoring
+
+```python
+# Calculate quality score
+def calculate_quality_score(validation_results):
+    """Calculate overall quality score"""
+    total_checks = validation_results.statistics['evaluated_expectations']
+    successful_checks = validation_results.statistics['successful_expectations']
+
+    score = (successful_checks / total_checks) * 100
+
+    # Categorize quality
+    if score >= 95:
+        quality_level = "Excellent"
+    elif score >= 85:
+        quality_level = "Good"
+    elif score >= 70:
+        quality_level = "Acceptable"
+    else:
+        quality_level = "Poor"
+
+    return {
+        'score': score,
+        'level': quality_level,
+        'total_checks': total_checks,
+        'passed_checks': successful_checks
+    }
+```
+
+### 3. Automated Monitoring
+
+```python
+# Set up quality monitoring
+def setup_quality_monitoring(checkpoint_name):
+    """Configure automated quality monitoring"""
+    checkpoint_config = {
+        "name": checkpoint_name,
+        "config_version": 1.0,
+        "template_name": "default",
+        "run_name_template": "%Y%m%d-%H%M%S",
+        "validations": [
+            {
+                "batch_request": {
+                    "datasource_name": "production_data",
+                    "data_connector_name": "default_inferred_data_connector_name",
+                    "data_asset_name": "customers"
+                },
+                "expectation_suite_name": "customers_quality_suite"
+            }
+        ],
+        "action_list": [
+            {
+                "name": "store_validation_result",
+                "action": {"class_name": "StoreValidationResultAction"}
+            },
+            {
+                "name": "send_slack_notification",
+                "action": {
+                    "class_name": "SlackNotificationAction",
+                    "slack_webhook": "${SLACK_WEBHOOK}",
+                    "notify_on": "failure"
+                }
+            }
+        ]
+    }
+
+    context.add_checkpoint(**checkpoint_config)
+```
+
+### 4. Issue Remediation Workflow
+
+```python
+# Create remediation workflow
+def create_remediation_workflow(quality_issues):
+    """Create tickets for quality issues"""
+    from azure.devops import AzureDevOpsClient
+
+    client = AzureDevOpsClient()
+
+    for issue in quality_issues:
+        work_item = {
+            'title': f"Data Quality Issue: {issue['column']}",
+            'description': issue['description'],
+            'priority': issue['severity'],
+            'assigned_to': issue['data_owner'],
+            'tags': ['data-quality', issue['table']]
+        }
+
+        client.create_work_item(
+            project='DataGovernance',
+            work_item_type='Bug',
+            fields=work_item
+        )
+```
+
+## Best Practices
+
+1. **Start Simple** - Begin with critical fields, expand coverage
+2. **Automate Everything** - Manual checks don't scale
+3. **Clear Ownership** - Assign quality issues to data owners
+4. **Threshold Alerts** - Alert on quality score drops
+5. **Historical Tracking** - Monitor quality trends over time
+
+## Cost Optimization
+
+- Run quality checks incrementally (only new/changed data)
+- Use sampling for large datasets
+- Cache validation results
+- Right-size validation compute
+
+## Integration
+
+**Connects with:**
+- de-01 (Lakehouse): Validate lakehouse data
+- de-03 (Data Quality): Engineering quality checks
+- dg-01 (Catalog): Link quality scores to assets
+- dg-02 (Lineage): Trace quality issues to source
+
+## Quick Win
+
+Implement completeness checks on 5 critical fields in your most important table. Show before/after quality scores.

package/{tech_hub_skills → .claude}/roles/data-governance/skills/04-access-control/README.md

@@ -1,39 +1,39 @@
-# dg-04: Access Control & Policies
-
-## Overview
-
-Implement role-based access control, column/row-level security, dynamic data masking, and access audit logging.
-
-## Key Capabilities
-
-- **RBAC**: Role-based access control
-- **Column-Level Security**: Restrict sensitive columns
-- **Row-Level Security**: Filter data by user context
-- **Dynamic Data Masking**: Auto-mask sensitive data
-- **Access Audit Logging**: Track all data access
-
-## Implementation
-
-```sql
--- Column-level security
-CREATE VIEW customer_secure AS
-SELECT
-    customer_id,
-    CASE
-        WHEN CURRENT_USER() IN (SELECT user FROM admin_users)
-        THEN email  -- Show full email to admins
-        ELSE CONCAT(LEFT(email, 3), '***@', SPLIT_PART(email, '@', 2))  -- Mask for others
-    END as email,
-    first_name,
-    last_name
-FROM customers;
-
--- Row-level security
-CREATE POLICY customer_region_policy ON customers
-FOR SELECT
-USING (region = current_setting('app.user_region'));
-```
-
-## Integration
-
-**Connects with:** sa-01 (PII Detection), sa-04 (IAM), dg-01 (Catalog)
+# dg-04: Access Control & Policies
+
+## Overview
+
+Implement role-based access control, column/row-level security, dynamic data masking, and access audit logging.
+
+## Key Capabilities
+
+- **RBAC**: Role-based access control
+- **Column-Level Security**: Restrict sensitive columns
+- **Row-Level Security**: Filter data by user context
+- **Dynamic Data Masking**: Auto-mask sensitive data
+- **Access Audit Logging**: Track all data access
+
+## Implementation
+
+```sql
+-- Column-level security
+CREATE VIEW customer_secure AS
+SELECT
+    customer_id,
+    CASE
+        WHEN CURRENT_USER() IN (SELECT user FROM admin_users)
+        THEN email  -- Show full email to admins
+        ELSE CONCAT(LEFT(email, 3), '***@', SPLIT_PART(email, '@', 2))  -- Mask for others
+    END as email,
+    first_name,
+    last_name
+FROM customers;
+
+-- Row-level security
+CREATE POLICY customer_region_policy ON customers
+FOR SELECT
+USING (region = current_setting('app.user_region'));
+```
+
+## Integration
+
+**Connects with:** sa-01 (PII Detection), sa-04 (IAM), dg-01 (Catalog)

package/{tech_hub_skills → .claude}/roles/data-governance/skills/05-master-data-management/README.md

@@ -1,40 +1,40 @@
-# dg-05: Master Data Management
-
-## Overview
-
-Entity resolution, golden record creation, data stewardship, and hierarchy management for critical business entities.
-
-## Key Capabilities
-
-- **Entity Resolution**: Match and merge duplicate entities
-- **Golden Record**: Single source of truth
-- **Data Stewardship**: Workflows for data quality
-- **Cross-Reference**: Link entities across systems
-- **Hierarchy Management**: Organizational structures
-
-## Implementation
-
-```python
-# Entity resolution
-from recordlinkage import Index, Compare
-
-def match_customers(df1, df2):
-    """Match customer records across systems"""
-    indexer = Index()
-    indexer.block('last_name')
-    candidate_pairs = indexer.index(df1, df2)
-
-    compare = Compare()
-    compare.exact('first_name', 'first_name')
-    compare.string('email', 'email', method='jarowinkler', threshold=0.85)
-    compare.numeric('age', 'age', method='linear', offset=2)
-
-    features = compare.compute(candidate_pairs, df1, df2)
-    matches = features[features.sum(axis=1) > 2.5]
-
-    return matches
-```
-
-## Integration
-
-**Connects with:** dg-01 (Catalog), dg-03 (Quality), de-02 (ETL)
+# dg-05: Master Data Management
+
+## Overview
+
+Entity resolution, golden record creation, data stewardship, and hierarchy management for critical business entities.
+
+## Key Capabilities
+
+- **Entity Resolution**: Match and merge duplicate entities
+- **Golden Record**: Single source of truth
+- **Data Stewardship**: Workflows for data quality
+- **Cross-Reference**: Link entities across systems
+- **Hierarchy Management**: Organizational structures
+
+## Implementation
+
+```python
+# Entity resolution
+from recordlinkage import Index, Compare
+
+def match_customers(df1, df2):
+    """Match customer records across systems"""
+    indexer = Index()
+    indexer.block('last_name')
+    candidate_pairs = indexer.index(df1, df2)
+
+    compare = Compare()
+    compare.exact('first_name', 'first_name')
+    compare.string('email', 'email', method='jarowinkler', threshold=0.85)
+    compare.numeric('age', 'age', method='linear', offset=2)
+
+    features = compare.compute(candidate_pairs, df1, df2)
+    matches = features[features.sum(axis=1) > 2.5]
+
+    return matches
+```
+
+## Integration
+
+**Connects with:** dg-01 (Catalog), dg-03 (Quality), de-02 (ETL)

package/{tech_hub_skills → .claude}/roles/data-governance/skills/06-compliance-privacy/README.md

@@ -1,46 +1,46 @@
-# dg-06: Compliance & Privacy
-
-## Overview
-
-GDPR compliance automation, data retention policies, right to be forgotten, consent management, and privacy impact assessments.
-
-## Key Capabilities
-
-- **GDPR Automation**: Automated compliance checks
-- **Data Retention**: Automated data lifecycle
-- **Right to be Forgotten**: Delete personal data on request
-- **Consent Management**: Track user consent
-- **Privacy Impact Assessments**: Risk assessment
-
-## Implementation
-
-```python
-# Right to be forgotten
-def delete_user_data(user_id):
-    """Delete all personal data for a user"""
-    tables = [
-        'customers', 'orders', 'payments',
-        'preferences', 'analytics_events'
-    ]
-
-    for table in tables:
-        spark.sql(f"""
-            DELETE FROM {table}
-            WHERE user_id = '{user_id}'
-        """)
-
-    # Log deletion for audit
-    log_gdpr_deletion(user_id, tables)
-
-# Data retention policy
-def apply_retention_policy():
-    """Delete data past retention period"""
-    spark.sql("""
-        DELETE FROM customer_events
-        WHERE event_date < DATE_SUB(CURRENT_DATE(), 730)  -- 2 years
-    """)
-```
-
-## Integration
-
-**Connects with:** sa-01 (PII Detection), dg-01 (Catalog), dg-04 (Access Control)
+# dg-06: Compliance & Privacy
+
+## Overview
+
+GDPR compliance automation, data retention policies, right to be forgotten, consent management, and privacy impact assessments.
+
+## Key Capabilities
+
+- **GDPR Automation**: Automated compliance checks
+- **Data Retention**: Automated data lifecycle
+- **Right to be Forgotten**: Delete personal data on request
+- **Consent Management**: Track user consent
+- **Privacy Impact Assessments**: Risk assessment
+
+## Implementation
+
+```python
+# Right to be forgotten
+def delete_user_data(user_id):
+    """Delete all personal data for a user"""
+    tables = [
+        'customers', 'orders', 'payments',
+        'preferences', 'analytics_events'
+    ]
+
+    for table in tables:
+        spark.sql(f"""
+            DELETE FROM {table}
+            WHERE user_id = '{user_id}'
+        """)
+
+    # Log deletion for audit
+    log_gdpr_deletion(user_id, tables)
+
+# Data retention policy
+def apply_retention_policy():
+    """Delete data past retention period"""
+    spark.sql("""
+        DELETE FROM customer_events
+        WHERE event_date < DATE_SUB(CURRENT_DATE(), 730)  -- 2 years
+    """)
+```
+
+## Integration
+
+**Connects with:** sa-01 (PII Detection), dg-01 (Catalog), dg-04 (Access Control)