@cloudstreamsoftware/claude-tools 1.0.0 → 1.2.0

package/skills/bigquery-patterns/SKILL.md

@@ -0,0 +1,27 @@
+---
+name: bigquery-patterns
+description: BigQuery analytical patterns for CloudStream's data engineering work. Covers schema design, query optimization, partitioning, clustering, and cost management.
+version: 1.0.0
+status: active
+introduced: 1.0.0
+lastUpdated: 2026-01-25
+activation: BigQuery queries, data warehouse design, analytical reporting, medallion architecture
+---
+
+# BigQuery Patterns
+
+This skill covers BigQuery-specific patterns for CloudStream's medallion architecture (bronze/silver/gold layers), cost optimization, and analytical query patterns.
+
+## When to Use
+- Designing BigQuery table schemas
+- Writing analytical queries
+- Optimizing query cost and performance
+- Building medallion layer transitions
+- Creating materialized views for Looker
+
+## Key Principles
+- Partition by date/timestamp for time-series data
+- Cluster by frequently filtered columns (max 4)
+- Use materialized views for expensive aggregations
+- Prefer STRUCT over JOINs for denormalized data
+- Monitor slot usage and costs with INFORMATION_SCHEMA

package/skills/bigquery-patterns/performance-optimization.md

@@ -0,0 +1,518 @@
+# BigQuery Performance Optimization Patterns
+
+> Advanced patterns for slot management, BI Engine acceleration, cost optimization, query tuning, and INFORMATION_SCHEMA monitoring at CloudStream.
+
+## Slot Usage Analysis
+
+### Understanding Slot Consumption
+
+```sql
+-- Identify top slot-consuming queries (last 7 days)
+SELECT
+  user_email,
+  job_id,
+  query,
+  total_slot_ms / 1000 AS slot_seconds,
+  total_bytes_processed / POW(1024, 3) AS gb_processed,
+  TIMESTAMP_DIFF(end_time, start_time, SECOND) AS wall_clock_seconds,
+  -- Slot efficiency: high slot usage with low wall time = good parallelism
+  SAFE_DIVIDE(total_slot_ms / 1000, TIMESTAMP_DIFF(end_time, start_time, SECOND)) AS avg_slots_used
+FROM `region-us`.INFORMATION_SCHEMA.JOBS_BY_PROJECT
+WHERE creation_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 7 DAY)
+  AND job_type = 'QUERY'
+  AND state = 'DONE'
+  AND error_result IS NULL
+ORDER BY total_slot_ms DESC
+LIMIT 25;
+```
+
+### Slot Usage by Time of Day (Capacity Planning)
+
+```sql
+-- Hourly slot usage pattern - identifies peak hours for reservation sizing
+SELECT
+  EXTRACT(HOUR FROM creation_time) AS hour_of_day,
+  EXTRACT(DAYOFWEEK FROM creation_time) AS day_of_week,
+  COUNT(*) AS query_count,
+  SUM(total_slot_ms) / 1000 / 3600 AS total_slot_hours,
+  MAX(total_slot_ms) / 1000 AS max_slot_seconds,
+  APPROX_QUANTILES(total_slot_ms / 1000, 100)[OFFSET(95)] AS p95_slot_seconds
+FROM `region-us`.INFORMATION_SCHEMA.JOBS_BY_PROJECT
+WHERE creation_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 30 DAY)
+  AND job_type = 'QUERY'
+GROUP BY 1, 2
+ORDER BY 1, 2;
+```
+
+### Slot Contention Detection
+
+```sql
+-- Detect queries that queued due to slot unavailability
+SELECT
+  job_id,
+  user_email,
+  creation_time,
+  start_time,
+  TIMESTAMP_DIFF(start_time, creation_time, SECOND) AS queue_wait_seconds,
+  total_slot_ms / 1000 AS slot_seconds,
+  SUBSTR(query, 1, 200) AS query_preview
+FROM `region-us`.INFORMATION_SCHEMA.JOBS_BY_PROJECT
+WHERE creation_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 24 HOUR)
+  AND job_type = 'QUERY'
+  AND TIMESTAMP_DIFF(start_time, creation_time, SECOND) > 5  -- Queued > 5s
+ORDER BY queue_wait_seconds DESC
+LIMIT 20;
+```
+
+## BI Engine for Sub-Second Dashboards
+
+### Reservation Setup
+
+```python
+# terraform/bi_engine.tf
+resource "google_bigquery_bi_reservation" "dashboard_acceleration" {
+  location   = "US"  # Must match dataset location
+  size       = 3     # 3 GB of BI Engine RAM
+
+  # Preferred tables are auto-detected, but you can influence priority
+  # by ensuring gold layer dashboard tables are queried frequently
+}
+```
+
+### BI Engine Monitoring
+
+```sql
+-- Check BI Engine acceleration status
+SELECT
+  project_id,
+  bi_engine_mode,   -- 'FULL', 'PARTIAL', 'DISABLED'
+  bi_engine_reasons, -- Why partial/disabled
+  total_bytes_processed,
+  query
+FROM `region-us`.INFORMATION_SCHEMA.JOBS_BY_PROJECT
+WHERE creation_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 24 HOUR)
+  AND bi_engine_statistics IS NOT NULL
+ORDER BY creation_time DESC
+LIMIT 50;
+
+-- BI Engine cache hit rate
+SELECT
+  COUNT(*) AS total_queries,
+  COUNTIF(bi_engine_mode = 'FULL') AS fully_accelerated,
+  COUNTIF(bi_engine_mode = 'PARTIAL') AS partially_accelerated,
+  COUNTIF(bi_engine_mode = 'DISABLED') AS not_accelerated,
+  ROUND(COUNTIF(bi_engine_mode = 'FULL') / COUNT(*) * 100, 1) AS full_acceleration_pct
+FROM `region-us`.INFORMATION_SCHEMA.JOBS_BY_PROJECT
+WHERE creation_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 7 DAY)
+  AND job_type = 'QUERY'
+  AND bi_engine_statistics IS NOT NULL;
+```
+
+### Optimizing for BI Engine
+
+BI Engine works best with these table patterns:
+
+| Optimization | Impact | Example |
+|-------------|--------|---------|
+| Fewer columns in query | Higher cache hit | SELECT only dashboard fields |
+| Smaller tables (< reserved GB) | Full acceleration | Pre-aggregate in gold layer |
+| Avoid complex functions | Better acceleration | Pre-compute in dbt model |
+| Use standard types | Full support | Avoid STRUCT, ARRAY in dashboard tables |
+| Cluster by dashboard filters | Better scanning | CLUSTER BY date, region |
+
+```sql
+-- Design gold layer views optimized for BI Engine
+CREATE OR REPLACE VIEW `project.gold.v_bi_revenue_dashboard` AS
+SELECT
+  -- Only columns used by Looker Studio dashboard
+  revenue_month,
+  customer_name,
+  region,
+  total_revenue,
+  invoice_count,
+  avg_days_to_pay,
+  -- Pre-computed comparisons (avoid window functions in dashboard query)
+  revenue_mom_pct,
+  revenue_yoy_pct
+FROM `project.gold.fct_monthly_revenue_enriched`
+WHERE revenue_month >= DATE_SUB(CURRENT_DATE(), INTERVAL 24 MONTH);
+-- Limit date range to fit in BI Engine reservation
+```
+
+> **COST TIP**: BI Engine costs $36.50/GB/month. A 2 GB reservation (~$73/month) can accelerate most dashboard tables if you pre-aggregate to keep gold tables small. Monitor the acceleration percentage -- below 80% means you need more capacity or smaller tables.
+
+## Cost Optimization: Flat-Rate vs On-Demand Breakeven
+
+### Breakeven Analysis
+
+```sql
+-- Calculate your on-demand cost to compare with flat-rate
+WITH monthly_usage AS (
+  SELECT
+    DATE_TRUNC(creation_time, MONTH) AS usage_month,
+    SUM(total_bytes_billed) / POW(1024, 4) AS tb_billed,
+    SUM(total_bytes_billed) / POW(1024, 4) * 6.25 AS on_demand_cost_usd,
+    SUM(total_slot_ms) / 1000 / 3600 AS total_slot_hours,
+    MAX(total_slot_ms) / 1000 AS max_query_slot_seconds
+  FROM `region-us`.INFORMATION_SCHEMA.JOBS_BY_PROJECT
+  WHERE creation_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 6 MONTH)
+    AND job_type = 'QUERY'
+  GROUP BY 1
+)
+SELECT
+  usage_month,
+  tb_billed,
+  on_demand_cost_usd,
+  total_slot_hours,
+  -- Standard Edition comparison: $0.04/slot-hour, 100 slot baseline
+  100 * 730 * 0.04 AS standard_100_slots_monthly,  -- $2,920/month
+  -- Is on-demand cheaper?
+  CASE
+    WHEN on_demand_cost_usd < 100 * 730 * 0.04 THEN 'ON_DEMAND_CHEAPER'
+    ELSE 'FLAT_RATE_CHEAPER'
+  END AS recommendation
+FROM monthly_usage
+ORDER BY usage_month DESC;
+```
+
+### Cost Optimization Decision Tree
+
+```
+Monthly BigQuery spend:
+  < $500/month  → Stay on-demand (not worth flat-rate complexity)
+  $500-$3,000   → Consider Standard Edition (100 baseline slots)
+  $3,000-$10,000 → Standard Edition with autoscaling
+  > $10,000     → Enterprise Edition (governance + autoscaling)
+```
+
+### On-Demand Cost Controls
+
+```sql
+-- Set maximum bytes billed per query (prevents runaway queries)
+-- Apply via project-level or user-level settings
+
+-- Project-level default (via API/Terraform)
+-- terraform/bigquery.tf
+resource "google_bigquery_reservation_assignment" "default" {
+  # Set custom quota per project
+}
+```
+
+```python
+# Set per-query byte limit in application code
+from google.cloud import bigquery
+
+client = bigquery.Client()
+job_config = bigquery.QueryJobConfig(
+    maximum_bytes_billed=10 * 1024**3  # 10 GB limit per query
+)
+
+# This query will fail if it would scan > 10 GB
+query_job = client.query(
+    "SELECT * FROM `project.silver.large_table`",
+    job_config=job_config
+)
+```
+
+## Query Optimization Patterns
+
+### Avoid Full Table Scans
+
+```sql
+-- BAD: No partition filter, scans entire table
+SELECT COUNT(*) FROM `project.silver.zoho_deals`;
+
+-- GOOD: Partition-pruned query
+SELECT COUNT(*)
+FROM `project.silver.zoho_deals`
+WHERE _ingestion_date >= DATE_SUB(CURRENT_DATE(), INTERVAL 7 DAY);
+
+-- Check if partition filter is being used (dry run)
+-- Use the BigQuery UI "Execution Details" or:
+SELECT
+  total_bytes_processed,
+  total_bytes_billed,
+  total_partitions_processed  -- Should be < total partitions
+FROM `region-us`.INFORMATION_SCHEMA.JOBS_BY_PROJECT
+WHERE job_id = 'your-job-id';
+```
+
+### Using Clustering Effectively
+
+```sql
+-- Clustering eliminates scanning irrelevant data blocks
+-- Column order matters: most selective first
+
+-- Table clustered by: owner_id, stage, account_id
+
+-- GOOD: Uses first cluster column (best pruning)
+SELECT * FROM `project.silver.zoho_deals`
+WHERE _ingestion_date = CURRENT_DATE()
+  AND owner_id = 'user_12345';
+
+-- GOOD: Uses first two cluster columns
+SELECT * FROM `project.silver.zoho_deals`
+WHERE _ingestion_date = CURRENT_DATE()
+  AND owner_id = 'user_12345'
+  AND stage = 'Closed Won';
+
+-- LESS EFFECTIVE: Skips first cluster column
+SELECT * FROM `project.silver.zoho_deals`
+WHERE _ingestion_date = CURRENT_DATE()
+  AND stage = 'Closed Won';  -- Skips owner_id, less block elimination
+```
+
+### Common Anti-Patterns
+
+```sql
+-- ANTI-PATTERN 1: SELECT * (scans all columns)
+SELECT * FROM `project.gold.fct_deals` WHERE stage = 'Closed Won';
+-- FIX: Select only needed columns
+SELECT deal_id, amount, close_date FROM `project.gold.fct_deals` WHERE stage = 'Closed Won';
+
+-- ANTI-PATTERN 2: Cross join / cartesian product
+SELECT * FROM table_a, table_b WHERE table_a.id = table_b.id;
+-- FIX: Use explicit JOIN
+SELECT * FROM table_a JOIN table_b ON table_a.id = table_b.id;
+
+-- ANTI-PATTERN 3: Repeated subqueries
+SELECT *, (SELECT AVG(amount) FROM deals) AS avg_amount FROM deals;
+-- FIX: Use window function or CTE
+SELECT *, AVG(amount) OVER () AS avg_amount FROM deals;
+
+-- ANTI-PATTERN 4: DISTINCT on large result sets
+SELECT DISTINCT * FROM `project.silver.zoho_deals`;
+-- FIX: Use GROUP BY on specific columns or fix deduplication upstream
+
+-- ANTI-PATTERN 5: ORDER BY without LIMIT
+SELECT * FROM `project.gold.fct_invoices` ORDER BY invoice_date DESC;
+-- FIX: Always pair ORDER BY with LIMIT
+SELECT * FROM `project.gold.fct_invoices` ORDER BY invoice_date DESC LIMIT 1000;
+```
+
+## Storage Tier Management
+
+```sql
+-- Monitor storage tiers and costs
+SELECT
+  table_schema AS dataset,
+  table_name,
+  ROUND(active_physical_bytes / POW(1024, 3), 2) AS active_gb,
+  ROUND(long_term_physical_bytes / POW(1024, 3), 2) AS long_term_gb,
+  ROUND(active_physical_bytes / POW(1024, 3) * 0.02, 2) AS active_monthly_cost,
+  ROUND(long_term_physical_bytes / POW(1024, 3) * 0.01, 2) AS long_term_monthly_cost,
+  TIMESTAMP_MILLIS(last_modified_time) AS last_modified
+FROM `project`.`region-us`.INFORMATION_SCHEMA.TABLE_STORAGE
+WHERE total_physical_bytes > 0
+ORDER BY (active_physical_bytes + long_term_physical_bytes) DESC
+LIMIT 50;
+
+-- Tables that could benefit from partition expiration
+SELECT
+  table_schema,
+  table_name,
+  ROUND(total_physical_bytes / POW(1024, 3), 2) AS total_gb,
+  TIMESTAMP_MILLIS(last_modified_time) AS last_modified,
+  CASE
+    WHEN TIMESTAMP_MILLIS(last_modified_time) < TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 180 DAY)
+    THEN 'Consider archiving or expiring old partitions'
+    ELSE 'Active'
+  END AS recommendation
+FROM `project`.`region-us`.INFORMATION_SCHEMA.TABLE_STORAGE
+WHERE total_physical_bytes > 1 * POW(1024, 3)  -- > 1 GB
+ORDER BY total_physical_bytes DESC;
+```
+
+### Storage Best Practices
+
+| Strategy | Savings | How |
+|----------|---------|-----|
+| Append-only tables | 50% after 90 days | Avoid UPDATE/DELETE on silver tables |
+| Partition expiration | Variable | Set `partition_expiration_days` on bronze |
+| Compression (Parquet source) | 60-80% | Use Parquet/Avro over CSV for loads |
+| Drop unused tables | 100% | Audit with INFORMATION_SCHEMA quarterly |
+| Time Travel reduction | Up to 7 days of storage | Set `max_time_travel_hours` (default 168h) |
+
+```sql
+-- Reduce time travel window for non-critical tables (saves storage)
+ALTER TABLE `project.bronze.zoho_raw_events`
+SET OPTIONS (max_time_travel_hours = 48);  -- 2 days instead of default 7
+```
+
+## Scheduled Queries for Materialization
+
+```sql
+-- Scheduled query: Refresh gold KPI table every hour
+-- Configure via Cloud Console or Terraform
+
+-- terraform/scheduled_queries.tf
+resource "google_bigquery_data_transfer_config" "kpi_refresh" {
+  display_name   = "Hourly KPI Refresh"
+  data_source_id = "scheduled_query"
+  schedule       = "every 1 hours"
+  location       = "US"
+
+  params = {
+    query = <<-EOT
+      CREATE OR REPLACE TABLE `project.gold.kpi_current` AS
+      SELECT
+        CURRENT_TIMESTAMP() AS refreshed_at,
+        (SELECT SUM(amount) FROM `project.silver.zoho_deals`
+         WHERE stage = 'Closed Won' AND close_date >= DATE_TRUNC(CURRENT_DATE(), MONTH)
+        ) AS mtd_closed_won,
+        (SELECT COUNT(*) FROM `project.silver.zoho_deals`
+         WHERE stage NOT IN ('Closed Won', 'Closed Lost')
+        ) AS open_deals,
+        (SELECT SUM(balance_due) FROM `project.silver.zoho_invoices`
+         WHERE status = 'overdue'
+        ) AS total_overdue_ar
+    EOT
+    destination_table_name_template = "kpi_current"
+    write_disposition              = "WRITE_TRUNCATE"
+  }
+}
+```
+
+## INFORMATION_SCHEMA Monitoring
+
+### Comprehensive Monitoring Dashboard Query
+
+```sql
+-- Daily monitoring report: cost, performance, anomalies
+WITH daily_stats AS (
+  SELECT
+    DATE(creation_time) AS query_date,
+    COUNT(*) AS total_queries,
+    SUM(total_bytes_processed) / POW(1024, 4) AS tb_processed,
+    SUM(total_bytes_billed) / POW(1024, 4) * 6.25 AS estimated_cost,
+    SUM(total_slot_ms) / 1000 / 3600 AS slot_hours,
+    COUNTIF(error_result IS NOT NULL) AS failed_queries,
+    APPROX_QUANTILES(TIMESTAMP_DIFF(end_time, start_time, SECOND), 100)[OFFSET(95)] AS p95_duration_sec,
+    MAX(total_bytes_processed) / POW(1024, 3) AS max_query_gb
+  FROM `region-us`.INFORMATION_SCHEMA.JOBS_BY_PROJECT
+  WHERE creation_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 30 DAY)
+    AND job_type = 'QUERY'
+  GROUP BY 1
+)
+SELECT
+  query_date,
+  total_queries,
+  ROUND(tb_processed, 3) AS tb_processed,
+  ROUND(estimated_cost, 2) AS estimated_cost_usd,
+  ROUND(slot_hours, 1) AS slot_hours,
+  failed_queries,
+  p95_duration_sec,
+  ROUND(max_query_gb, 1) AS max_query_gb,
+  -- Anomaly detection
+  CASE
+    WHEN estimated_cost > 2 * AVG(estimated_cost) OVER (
+      ORDER BY query_date ROWS BETWEEN 7 PRECEDING AND 1 PRECEDING
+    ) THEN 'COST_SPIKE'
+    WHEN failed_queries > 2 * AVG(failed_queries) OVER (
+      ORDER BY query_date ROWS BETWEEN 7 PRECEDING AND 1 PRECEDING
+    ) THEN 'ERROR_SPIKE'
+    ELSE 'NORMAL'
+  END AS anomaly_flag
+FROM daily_stats
+ORDER BY query_date DESC;
+```
+
+### User Attribution
+
+```sql
+-- Who is spending the most? (accountability)
+SELECT
+  user_email,
+  COUNT(*) AS query_count,
+  ROUND(SUM(total_bytes_billed) / POW(1024, 4) * 6.25, 2) AS cost_usd,
+  ROUND(SUM(total_bytes_billed) / POW(1024, 3), 1) AS gb_billed,
+  ROUND(AVG(total_slot_ms) / 1000, 1) AS avg_slot_seconds,
+  COUNTIF(total_bytes_billed > 10 * POW(1024, 3)) AS queries_over_10gb
+FROM `region-us`.INFORMATION_SCHEMA.JOBS_BY_PROJECT
+WHERE creation_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 30 DAY)
+  AND job_type = 'QUERY'
+GROUP BY 1
+ORDER BY cost_usd DESC
+LIMIT 20;
+```
+
+### Table-Level Usage Patterns
+
+```sql
+-- Most queried tables (optimization priority)
+SELECT
+  REGEXP_EXTRACT(query, r'FROM\s+`([^`]+)`') AS table_referenced,
+  COUNT(*) AS query_count,
+  SUM(total_bytes_processed) / POW(1024, 3) AS total_gb_scanned,
+  AVG(total_bytes_processed) / POW(1024, 3) AS avg_gb_per_query,
+  -- Tables with high scan per query = optimization candidates
+  CASE
+    WHEN AVG(total_bytes_processed) / POW(1024, 3) > 10 THEN 'OPTIMIZE: Add partition/cluster'
+    WHEN COUNT(*) > 100 AND AVG(total_bytes_processed) / POW(1024, 3) > 1 THEN 'CONSIDER: Materialized view'
+    ELSE 'OK'
+  END AS recommendation
+FROM `region-us`.INFORMATION_SCHEMA.JOBS_BY_PROJECT
+WHERE creation_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 30 DAY)
+  AND job_type = 'QUERY'
+  AND state = 'DONE'
+GROUP BY 1
+HAVING table_referenced IS NOT NULL
+ORDER BY total_gb_scanned DESC
+LIMIT 30;
+```
+
+### Automated Cost Alert Query
+
+```python
+# scripts/bq_cost_monitor.py
+"""Run daily to check BigQuery spending and alert if over budget."""
+
+from google.cloud import bigquery, monitoring_v3
+from datetime import datetime, timedelta
+
+DAILY_BUDGET_USD = 50.0  # Alert threshold
+
+def check_daily_cost():
+    client = bigquery.Client()
+
+    query = """
+    SELECT
+      ROUND(SUM(total_bytes_billed) / POW(1024, 4) * 6.25, 2) AS today_cost_usd
+    FROM `region-us`.INFORMATION_SCHEMA.JOBS_BY_PROJECT
+    WHERE creation_time >= TIMESTAMP_TRUNC(CURRENT_TIMESTAMP(), DAY)
+      AND job_type = 'QUERY'
+    """
+
+    result = list(client.query(query).result())[0]
+    today_cost = result.today_cost_usd or 0
+
+    if today_cost > DAILY_BUDGET_USD:
+        send_alert(
+            f"BigQuery daily cost alert: ${today_cost:.2f} "
+            f"(budget: ${DAILY_BUDGET_USD:.2f})"
+        )
+
+    # Also publish as custom metric for dashboards
+    publish_cost_metric(today_cost)
+
+def send_alert(message):
+    """Send Slack alert via webhook."""
+    import requests
+    requests.post(
+        "https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK",
+        json={"text": f":warning: {message}"}
+    )
+```
+
+## Best Practices Summary
+
+1. **Monitor INFORMATION_SCHEMA weekly** - Catch cost spikes and inefficient queries early
+2. **Reserve BI Engine for dashboard tables only** - Do not waste on ad-hoc queries
+3. **Set maximum_bytes_billed on all service accounts** - Prevent runaway queries
+4. **Use slot analysis before committing to flat-rate** - Ensure breakeven is favorable
+5. **Cluster gold tables by dashboard filter columns** - Matches Looker/Analytics query patterns
+6. **Scheduled query materialization** - Cheaper than users scanning large tables repeatedly
+7. **Audit storage quarterly** - Drop unused tables, reduce time travel on non-critical data
+8. **Alert on anomalies** - Cost spikes usually indicate a broken query or missing partition filter
+
+> **WARNING**: INFORMATION_SCHEMA queries themselves consume slots. Avoid running expensive monitoring queries more than once per hour. Cache results in a monitoring table for dashboard use.