@cloudstreamsoftware/claude-tools 1.0.0 → 1.1.0

package/skills/gcp-data-engineering/medallion-architecture/gold-layer.md

@@ -0,0 +1,311 @@
+# Gold Layer - Business-Ready Aggregations
+
+> The gold layer provides pre-computed, business-aligned datasets optimized for consumption by Looker Studio, Zoho Analytics, and custom applications.
+
+## Architecture Overview
+
+```
+Silver (cleansed entities) → Gold (business metrics, dimensions, facts)
+                                   │
+                                   ├── Fact tables (events, transactions)
+                                   ├── Dimension tables (customers, products)
+                                   ├── Materialized views (common aggregations)
+                                   ├── Pre-computed KPIs (daily snapshots)
+                                   └── Consumer-specific views
+```
+
+## Materialized Views vs Tables: Decision Framework
+
+| Factor | Materialized View | Table |
+|--------|-------------------|-------|
+| Refresh | Auto (within limits) | Manual/scheduled |
+| Cost (storage) | Free (cached query) | Billed at storage rates |
+| Cost (compute) | Incremental refresh | Full rebuild each run |
+| Freshness | Near real-time (5-min lag) | Depends on schedule |
+| Complexity | Single-table aggregations only | Arbitrary SQL |
+| Joins | Limited (single source table) | Unlimited |
+| Best for | Simple rollups, dashboards | Complex multi-source models |
+
+> **COST WARNING**: Materialized views only support single-table sources with limited SQL. For multi-table joins, use scheduled dbt models instead.
+
+```sql
+-- Good candidate for materialized view: simple daily aggregation
+CREATE MATERIALIZED VIEW `project.gold.mv_daily_deal_summary`
+PARTITION BY deal_date
+CLUSTER BY owner_id
+AS
+SELECT
+  DATE(close_date) AS deal_date,
+  owner_id,
+  stage,
+  COUNT(*) AS deal_count,
+  SUM(amount) AS total_amount,
+  AVG(amount) AS avg_amount
+FROM `project.silver.zoho_deals`
+WHERE close_date IS NOT NULL
+GROUP BY 1, 2, 3;
+
+-- NOT a candidate for materialized view: multi-table join → use dbt table
+-- models/marts/fct_revenue.sql
+```
+
+## Fact/Dimension Modeling
+
+### Dimension Tables
+
+```sql
+-- models/marts/dim_customers.sql
+{{
+  config(
+    materialized='table',
+    partition_by={'field': '_loaded_date', 'data_type': 'date'},
+    cluster_by=['industry', 'region']
+  )
+}}
+
+SELECT
+  c.record_id AS customer_id,
+  c.account_name,
+  c.industry,
+  c.annual_revenue,
+  c.employee_count,
+  a.billing_city,
+  a.billing_state,
+  a.billing_country,
+  CASE
+    WHEN a.billing_country IN ('US', 'CA') THEN 'North America'
+    WHEN a.billing_country IN ('GB', 'DE', 'FR') THEN 'Europe'
+    ELSE 'Other'
+  END AS region,
+  o.owner_name AS account_owner,
+  c.created_time AS customer_since,
+  CURRENT_DATE() AS _loaded_date
+FROM {{ ref('stg_zoho__accounts') }} c
+LEFT JOIN {{ ref('int_zoho__addresses') }} a ON c.record_id = a.account_id
+LEFT JOIN {{ ref('stg_zoho__users') }} o ON c.owner_id = o.user_id
+```
+
+### Fact Tables
+
+```sql
+-- models/marts/fct_invoices.sql
+{{
+  config(
+    materialized='incremental',
+    unique_key='invoice_id',
+    partition_by={'field': 'invoice_date', 'data_type': 'date'},
+    cluster_by=['customer_id', 'status']
+  )
+}}
+
+SELECT
+  i.record_id AS invoice_id,
+  i.customer_id,
+  i.invoice_number,
+  DATE(i.invoice_date) AS invoice_date,
+  DATE(i.due_date) AS due_date,
+  i.status,
+  i.sub_total,
+  i.tax_amount,
+  i.total AS invoice_total,
+  i.balance_due,
+  DATE_DIFF(
+    COALESCE(i.paid_date, CURRENT_DATE()),
+    i.invoice_date,
+    DAY
+  ) AS days_to_payment,
+  i.currency_code,
+  i.salesperson_id,
+  CURRENT_TIMESTAMP() AS _loaded_at
+FROM {{ ref('stg_zoho__invoices') }} i
+{% if is_incremental() %}
+WHERE i._ingestion_timestamp > (SELECT MAX(_loaded_at) FROM {{ this }})
+{% endif %}
+```
+
+## Pre-Computed KPIs
+
+```sql
+-- models/marts/kpi_daily_snapshot.sql
+{{
+  config(
+    materialized='incremental',
+    unique_key='snapshot_date',
+    partition_by={'field': 'snapshot_date', 'data_type': 'date'}
+  )
+}}
+
+SELECT
+  CURRENT_DATE() AS snapshot_date,
+  -- Revenue KPIs
+  SUM(CASE WHEN status = 'paid' AND invoice_date >= DATE_TRUNC(CURRENT_DATE(), MONTH)
+      THEN invoice_total ELSE 0 END) AS mtd_revenue,
+  SUM(CASE WHEN status = 'paid' AND invoice_date >= DATE_TRUNC(CURRENT_DATE(), YEAR)
+      THEN invoice_total ELSE 0 END) AS ytd_revenue,
+  -- Pipeline KPIs
+  (SELECT SUM(amount) FROM {{ ref('fct_deals') }}
+   WHERE stage NOT IN ('Closed Won', 'Closed Lost')) AS open_pipeline_value,
+  (SELECT COUNT(*) FROM {{ ref('fct_deals') }}
+   WHERE stage NOT IN ('Closed Won', 'Closed Lost')) AS open_deal_count,
+  -- AR KPIs
+  SUM(CASE WHEN status = 'overdue' THEN balance_due ELSE 0 END) AS total_overdue,
+  COUNT(CASE WHEN status = 'overdue' THEN 1 END) AS overdue_invoice_count,
+  AVG(days_to_payment) AS avg_days_to_payment,
+  CURRENT_TIMESTAMP() AS _computed_at
+FROM {{ ref('fct_invoices') }}
+```
+
+## Refresh Strategies
+
+### Event-Driven Refresh (Recommended for Critical KPIs)
+
+```python
+# Cloud Function triggered by silver table update
+from google.cloud import bigquery
+import functions_framework
+
+@functions_framework.cloud_event
+def refresh_gold_on_silver_update(cloud_event):
+    """Triggered by BigQuery audit log when silver tables are updated."""
+    client = bigquery.Client()
+
+    # Determine which gold models need refresh
+    source_table = cloud_event.data.get("protoPayload", {}).get(
+        "resourceName", ""
+    )
+
+    refresh_map = {
+        "silver.zoho_deals": ["gold.fct_deals", "gold.kpi_daily_snapshot"],
+        "silver.zoho_invoices": ["gold.fct_invoices", "gold.kpi_daily_snapshot"],
+        "silver.zoho_contacts": ["gold.dim_customers"],
+    }
+
+    for table_suffix, targets in refresh_map.items():
+        if table_suffix in source_table:
+            for target in targets:
+                job = client.query(f"CALL `project.gold.refresh_{target.split('.')[1]}`()")
+                job.result()
+```
+
+### Scheduled Refresh (Standard Pattern)
+
+```yaml
+# Cloud Scheduler → dbt run for gold models
+# Runs every 6 hours
+schedule:
+  name: gold-layer-refresh
+  cron: "0 */6 * * *"
+  target:
+    command: "dbt run --select tag:gold --target prod"
+
+# Critical KPIs refresh more frequently
+schedule:
+  name: kpi-snapshot-refresh
+  cron: "0 * * * *"  # Every hour during business hours
+  target:
+    command: "dbt run --select kpi_daily_snapshot --target prod"
+```
+
+## Downstream Consumer Patterns
+
+### Looker Studio
+
+```sql
+-- Optimized view for Looker Studio dashboards
+-- Use BI Engine reservation for sub-second queries
+CREATE OR REPLACE VIEW `project.gold.v_looker_revenue_dashboard` AS
+SELECT
+  snapshot_date,
+  mtd_revenue,
+  ytd_revenue,
+  open_pipeline_value,
+  total_overdue,
+  avg_days_to_payment,
+  -- Pre-computed period comparisons for Looker
+  LAG(mtd_revenue, 30) OVER (ORDER BY snapshot_date) AS prior_month_revenue,
+  SAFE_DIVIDE(
+    mtd_revenue - LAG(mtd_revenue, 30) OVER (ORDER BY snapshot_date),
+    LAG(mtd_revenue, 30) OVER (ORDER BY snapshot_date)
+  ) AS revenue_mom_growth
+FROM `project.gold.kpi_daily_snapshot`
+WHERE snapshot_date >= DATE_SUB(CURRENT_DATE(), INTERVAL 13 MONTH);
+```
+
+### Zoho Analytics (via BigQuery Connector)
+
+```sql
+-- Zoho Analytics pulls from dedicated views with flattened structure
+-- Avoid nested/repeated fields - Zoho Analytics cannot handle them
+CREATE OR REPLACE VIEW `project.gold.v_zoho_analytics_deals` AS
+SELECT
+  deal_id,
+  deal_name,
+  customer_name,   -- Denormalized, no joins needed
+  owner_name,      -- Denormalized
+  stage,
+  amount,
+  FORMAT_DATE('%Y-%m-%d', close_date) AS close_date_str,  -- String dates for Zoho
+  CAST(probability AS INT64) AS probability_pct
+FROM `project.gold.fct_deals` d
+JOIN `project.gold.dim_customers` c ON d.customer_id = c.customer_id;
+```
+
+## Access Control Per Gold Table
+
+```sql
+-- Grant granular access to gold tables
+-- Finance team: revenue and invoice data only
+GRANT `roles/bigquery.dataViewer`
+ON TABLE `project.gold.fct_invoices`
+TO "group:finance@cloudstreamsoftware.com";
+
+-- Sales team: deals and pipeline
+GRANT `roles/bigquery.dataViewer`
+ON TABLE `project.gold.fct_deals`
+TO "group:sales@cloudstreamsoftware.com";
+
+-- Executives: KPI snapshots
+GRANT `roles/bigquery.dataViewer`
+ON TABLE `project.gold.kpi_daily_snapshot`
+TO "group:executives@cloudstreamsoftware.com";
+
+-- Row-level security for multi-tenant data
+CREATE OR REPLACE ROW ACCESS POLICY region_filter
+ON `project.gold.fct_deals`
+GRANT TO ("group:sales-na@cloudstreamsoftware.com")
+FILTER USING (region = 'North America');
+```
+
+## Versioning Strategy
+
+```sql
+-- Version gold tables for breaking changes
+-- v1 remains available while consumers migrate to v2
+CREATE OR REPLACE VIEW `project.gold.v1.fct_deals` AS
+SELECT deal_id, amount, stage FROM `project.gold.fct_deals`;
+
+CREATE OR REPLACE VIEW `project.gold.v2.fct_deals` AS
+SELECT deal_id, amount, stage, probability, expected_revenue
+FROM `project.gold.fct_deals`;
+
+-- Track which consumers use which version
+CREATE TABLE `project.gold._version_registry` (
+  table_name STRING,
+  version STRING,
+  consumer STRING,
+  migration_deadline DATE,
+  migrated BOOL DEFAULT FALSE
+);
+```
+
+## Best Practices
+
+1. **Denormalize for consumers** - Gold tables should minimize joins for dashboard performance
+2. **Pre-compute period comparisons** - MoM, YoY, MTD built into the table, not at query time
+3. **Flatten nested structures** - Zoho Analytics and Looker prefer flat schemas
+4. **Use BI Engine for dashboards** - Reserve capacity for sub-second Looker queries
+5. **Version breaking changes** - Never modify gold schemas without deprecation period
+6. **Restrict access by role** - Not everyone needs all gold tables
+7. **Monitor query patterns** - Use INFORMATION_SCHEMA to identify optimization opportunities
+
+> **COST TIP**: Materialized views are free to store and automatically refresh. Use them for any single-table aggregation that is queried more than once per day. The break-even is typically 2-3 queries/day vs re-running the aggregation each time.