@cloudstreamsoftware/claude-tools 1.0.0 → 1.1.0

package/skills/frontend-patterns/zoho-widget-patterns.md

@@ -0,0 +1,103 @@
+# Zoho Widget Development Patterns
+
+## Widget Initialization
+```javascript
+import React, { useEffect, useState } from 'react';
+
+function ZohoWidget() {
+  const [sdk, setSdk] = useState(null);
+  const [loading, setLoading] = useState(true);
+
+  useEffect(() => {
+    // ZOHO SDK is loaded via script tag in index.html
+    window.ZOHO.CREATOR.init()
+      .then(() => {
+        setSdk(window.ZOHO.CREATOR);
+        setLoading(false);
+      })
+      .catch(err => {
+        console.error('SDK init failed:', err);
+        setLoading(false);
+      });
+  }, []);
+
+  if (loading) return <div>Loading...</div>;
+
+  return <WidgetContent sdk={sdk} />;
+}
+```
+
+## Reading Form Data
+```javascript
+async function getRecordData(sdk, reportName, criteria) {
+  try {
+    const config = {
+      appName: 'your-app',
+      reportName: reportName,
+      criteria: criteria,
+      page: 1,
+      pageSize: 200
+    };
+
+    const response = await sdk.API.getAllRecords(config);
+    return response.data;
+  } catch (error) {
+    console.error('Failed to fetch records:', error);
+    return [];
+  }
+}
+```
+
+## Writing Form Data
+```javascript
+async function createRecord(sdk, formName, data) {
+  try {
+    const config = {
+      appName: 'your-app',
+      formName: formName,
+      data: { data: data }
+    };
+
+    const response = await sdk.API.addRecord(config);
+    return response;
+  } catch (error) {
+    console.error('Failed to create record:', error);
+    throw error;
+  }
+}
+```
+
+## Widget-to-Form Communication
+```javascript
+// Send data to parent form
+sdk.API.setFieldValue({ appName: 'app', formName: 'form', fieldName: 'Status', value: 'Complete' });
+
+// Listen for form events
+sdk.API.onFormSubmit(function(data) {
+  console.log('Form submitted:', data);
+});
+```
+
+## CORS Proxy Architecture
+Widgets cannot directly call external APIs due to CORS.
+Use a Catalyst function as proxy:
+
+```javascript
+// Widget → Catalyst Function → External API
+async function callExternalAPI(endpoint, data) {
+  const proxyURL = 'https://your-project.catalystserverless.com/server/proxy';
+  const response = await fetch(proxyURL, {
+    method: 'POST',
+    headers: { 'Content-Type': 'application/json' },
+    body: JSON.stringify({ endpoint, data })
+  });
+  return response.json();
+}
+```
+
+## Critical Constraints
+- Widgets do NOT work on published pages (use Publish API instead)
+- Maximum 50 widgets per Zoho One account
+- Widget SDK is loaded via script tag, not npm package
+- Test locally with Zoho SDK mock before deployment
+- Widget container has limited height/width - design responsively

package/skills/gcp-data-engineering/SKILL.md

@@ -0,0 +1,36 @@
+---
+name: gcp-data-engineering
+description: GCP data engineering patterns for medallion architecture (bronze/silver/gold), BigQuery, Dataflow, and dbt. Used for data pipeline design and implementation.
+version: 1.0.0
+status: active
+introduced: 1.0.0
+lastUpdated: 2026-01-25
+activation: BigQuery, Dataflow, data pipeline tasks, medallion architecture, dbt models
+---
+
+# GCP Data Engineering
+
+Patterns for CloudStream's data engineering work using Google Cloud Platform.
+
+## Medallion Architecture
+- **Bronze**: Raw ingestion, schema-on-read, minimal transformation
+- **Silver**: Cleansed, deduplicated, quality rules applied
+- **Gold**: Business-ready aggregations for reporting
+
+## Key Technologies
+- BigQuery: Data warehouse (silver + gold layers)
+- Cloud Storage: Raw file storage (bronze layer)
+- Dataflow: Streaming and batch ETL (Apache Beam)
+- dbt: SQL-based transformations (silver → gold)
+- Cloud Functions: Event-driven ingestion triggers
+- Looker Studio: Visualization on gold layer
+
+## Naming Conventions
+
+| Layer | Table Suffix | Example | Description |
+|-------|-------------|---------|-------------|
+| Bronze | `_raw` | `zoho_contacts_raw` | Raw ingested data, schema-on-read |
+| Silver | `_cleaned` | `contacts_cleaned` | Deduplicated, validated, typed |
+| Gold | `_agg` or descriptive | `customer_360`, `revenue_daily_agg` | Business aggregations |
+
+Dataset naming: `project.bronze.*`, `project.silver.*`, `project.gold.*`

package/skills/gcp-data-engineering/bigquery/performance-optimization.md

@@ -0,0 +1,337 @@
+# BigQuery Performance Optimization
+
+> Strategies for partitioning, clustering, slot management, and query optimization to minimize cost and maximize performance in CloudStream's BigQuery warehouse.
+
+## Partitioning Strategies
+
+### Time-Unit Partitioning (Recommended Default)
+
+```sql
+-- Partition by business date for time-series data
+CREATE TABLE `project.silver.zoho_deals` (
+  record_id STRING,
+  deal_name STRING,
+  amount FLOAT64,
+  close_date DATE,
+  _ingestion_date DATE
+)
+PARTITION BY _ingestion_date
+OPTIONS (
+  require_partition_filter = TRUE,
+  partition_expiration_days = 730
+);
+
+-- Partition by month for aggregated tables (reduces partition count)
+CREATE TABLE `project.gold.fct_monthly_revenue` (
+  revenue_month DATE,
+  customer_id STRING,
+  total_revenue FLOAT64
+)
+PARTITION BY DATE_TRUNC(revenue_month, MONTH);
+```
+
+### Integer-Range Partitioning (For Non-Date Keys)
+
+```sql
+-- Useful for customer ID-based sharding
+CREATE TABLE `project.gold.dim_customers_sharded` (
+  customer_shard INT64,  -- hash of customer_id mod 1000
+  customer_id STRING,
+  account_name STRING
+)
+PARTITION BY RANGE_BUCKET(customer_shard, GENERATE_ARRAY(0, 1000, 10));
+```
+
+### Ingestion-Time Partitioning (Bronze Layer Default)
+
+```sql
+-- Automatic partitioning by load time (no column needed)
+CREATE TABLE `project.bronze.zoho_raw_events` (
+  event_data STRING,
+  source STRING
+)
+PARTITION BY _PARTITIONDATE
+OPTIONS (require_partition_filter = TRUE);
+```
+
+### Partitioning Decision Matrix
+
+| Data Type | Recommended Partition | Granularity | Expiration |
+|-----------|----------------------|-------------|------------|
+| Bronze raw | `_PARTITIONDATE` | Day | 365 days |
+| Silver cleansed | `_ingestion_date` | Day | 730 days |
+| Gold facts | Business date | Day/Month | No expiry |
+| Gold KPIs | `snapshot_date` | Day | No expiry |
+| Quarantine | `quarantined_at` | Day | 90 days |
+
+> **WARNING**: Tables with 4000+ partitions hit BigQuery limits. Use MONTH granularity for tables spanning 10+ years.
+
+## Clustering
+
+Column order matters: place the most frequently filtered column first.
+
+```sql
+-- Optimal clustering for deal queries
+CREATE TABLE `project.silver.zoho_deals` (
+  ...
+)
+PARTITION BY _ingestion_date
+CLUSTER BY owner_id, stage, account_id;
+-- Query pattern: WHERE owner_id = 'x' AND stage = 'Closed Won'
+-- Clustering eliminates scanning irrelevant blocks
+
+-- Re-cluster existing table (free, runs automatically)
+ALTER TABLE `project.silver.zoho_deals`
+SET OPTIONS (clustering_columns = ['owner_id', 'stage', 'account_id']);
+```
+
+### Clustering Best Practices
+
+| Rule | Rationale |
+|------|-----------|
+| Max 4 columns | Diminishing returns beyond 4 |
+| High cardinality first | Provides best block elimination |
+| Filter columns only | Clustering helps WHERE, not SELECT |
+| String columns work | Unlike partitioning, strings are fine |
+| Re-clusters automatically | No maintenance needed after initial set |
+
+## BI Engine Reservations
+
+```sql
+-- Reserve BI Engine capacity for Looker Studio dashboards
+-- This provides sub-second query responses for cached data
+
+-- Check current BI Engine status
+SELECT * FROM `region-us`.INFORMATION_SCHEMA.BI_CAPACITIES;
+```
+
+```python
+# Terraform: Reserve BI Engine capacity
+# terraform/bi_engine.tf
+resource "google_bigquery_bi_reservation" "looker_cache" {
+  location = "us-central1"
+  size     = 2  # GB of BI Engine RAM (starts at ~$36.50/GB/month)
+}
+
+# Preferred tables for BI Engine acceleration
+resource "google_bigquery_table" "gold_kpi" {
+  # BI Engine automatically caches frequently-queried tables
+  # Prioritize gold layer dashboard tables
+}
+```
+
+> **COST TIP**: BI Engine costs ~$36.50/GB/month. Start with 1-2 GB for your most critical dashboards. Monitor cache hit rate in Cloud Monitoring.
+
+## Slot Management
+
+### On-Demand vs Editions
+
+| Factor | On-Demand | Standard Edition | Enterprise Edition |
+|--------|-----------|-------------------|--------------------|
+| Pricing | $6.25/TB scanned | $0.04/slot-hour | $0.06/slot-hour |
+| Breakeven | <50 TB/month | 50-200 TB/month | 200+ TB/month |
+| Autoscaling | Unlimited | Configurable | Configurable |
+| Baseline slots | N/A | Set minimum | Set minimum |
+| Concurrency | 2000 queries | Based on slots | Based on slots |
+| Best for | Unpredictable/low usage | Steady workloads | Heavy + governance |
+
+```sql
+-- Monitor slot utilization
+SELECT
+  period_start,
+  project_id,
+  job_type,
+  SUM(period_slot_ms) / 1000 / 3600 AS slot_hours,
+  SUM(total_bytes_processed) / POW(1024, 4) AS tb_processed
+FROM `region-us`.INFORMATION_SCHEMA.JOBS_TIMELINE_BY_PROJECT
+WHERE period_start >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 7 DAY)
+GROUP BY 1, 2, 3
+ORDER BY slot_hours DESC;
+```
+
+### Reservation Assignment
+
+```python
+# Assign reservations to different workload types
+from google.cloud import bigquery_reservation_v1
+
+client = bigquery_reservation_v1.ReservationServiceClient()
+
+# Create reservation for ETL workloads (off-peak slots)
+reservation = client.create_reservation(
+    parent="projects/cloudstream-prod/locations/us-central1",
+    reservation_id="etl-workload",
+    reservation=bigquery_reservation_v1.Reservation(
+        slot_capacity=200,  # 200 slots for ETL
+    )
+)
+
+# Create reservation for BI/dashboard queries (always available)
+reservation_bi = client.create_reservation(
+    parent="projects/cloudstream-prod/locations/us-central1",
+    reservation_id="bi-workload",
+    reservation=bigquery_reservation_v1.Reservation(
+        slot_capacity=100,  # 100 slots for dashboards
+    )
+)
+```
+
+## Materialized Views for Common Queries
+
+```sql
+-- Identify top queries to materialize
+SELECT
+  query,
+  COUNT(*) AS execution_count,
+  SUM(total_bytes_processed) / POW(1024, 3) AS total_gb_processed,
+  AVG(total_slot_ms) / 1000 AS avg_slot_seconds
+FROM `region-us`.INFORMATION_SCHEMA.JOBS_BY_PROJECT
+WHERE creation_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 30 DAY)
+  AND job_type = 'QUERY'
+  AND state = 'DONE'
+GROUP BY 1
+HAVING COUNT(*) > 10
+ORDER BY total_gb_processed DESC
+LIMIT 20;
+
+-- Create materialized view for frequently-run aggregation
+CREATE MATERIALIZED VIEW `project.gold.mv_deal_stage_summary`
+PARTITION BY DATE(_ingestion_date)
+CLUSTER BY stage
+AS
+SELECT
+  _ingestion_date,
+  stage,
+  owner_id,
+  COUNT(*) AS deal_count,
+  SUM(amount) AS total_amount,
+  AVG(amount) AS avg_amount,
+  MAX(modified_time) AS latest_update
+FROM `project.silver.zoho_deals`
+GROUP BY 1, 2, 3;
+```
+
+## Query Optimization Patterns
+
+### Avoid SELECT *
+
+```sql
+-- BAD: Scans all columns (expensive)
+SELECT * FROM `project.silver.zoho_deals` WHERE stage = 'Closed Won';
+
+-- GOOD: Only scan needed columns
+SELECT record_id, deal_name, amount, close_date
+FROM `project.silver.zoho_deals`
+WHERE stage = 'Closed Won';
+```
+
+### Use Approximate Functions
+
+```sql
+-- BAD: Exact distinct count (expensive for large tables)
+SELECT COUNT(DISTINCT customer_id) FROM `project.gold.fct_invoices`;
+
+-- GOOD: Approximate count (2% error, much faster)
+SELECT APPROX_COUNT_DISTINCT(customer_id) FROM `project.gold.fct_invoices`;
+
+-- GOOD: Approximate quantiles
+SELECT APPROX_QUANTILES(amount, 100)[OFFSET(50)] AS median_amount
+FROM `project.gold.fct_invoices`;
+```
+
+### Optimize Joins
+
+```sql
+-- BAD: Join before filter
+SELECT d.*, c.account_name
+FROM `project.gold.fct_deals` d
+JOIN `project.gold.dim_customers` c ON d.customer_id = c.customer_id;
+
+-- GOOD: Filter early, then join
+WITH filtered_deals AS (
+  SELECT record_id, customer_id, amount
+  FROM `project.gold.fct_deals`
+  WHERE close_date >= '2024-01-01'
+    AND stage = 'Closed Won'
+)
+SELECT d.*, c.account_name
+FROM filtered_deals d
+JOIN `project.gold.dim_customers` c ON d.customer_id = c.customer_id;
+```
+
+## Cost Monitoring and Budgets
+
+```sql
+-- Daily cost tracking query
+SELECT
+  DATE(creation_time) AS query_date,
+  user_email,
+  SUM(total_bytes_billed) / POW(1024, 4) AS tb_billed,
+  SUM(total_bytes_billed) / POW(1024, 4) * 6.25 AS estimated_cost_usd,
+  COUNT(*) AS query_count
+FROM `region-us`.INFORMATION_SCHEMA.JOBS_BY_PROJECT
+WHERE creation_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 30 DAY)
+  AND job_type = 'QUERY'
+GROUP BY 1, 2
+ORDER BY estimated_cost_usd DESC;
+```
+
+```yaml
+# Terraform: Budget alerts
+resource "google_billing_budget" "bigquery_budget" {
+  billing_account = var.billing_account_id
+  display_name    = "BigQuery Monthly Budget"
+
+  budget_filter {
+    services = ["services/24E6-581D-38E5"]  # BigQuery service ID
+  }
+
+  amount {
+    specified_amount {
+      currency_code = "USD"
+      units         = "500"  # $500/month budget
+    }
+  }
+
+  threshold_rules {
+    threshold_percent = 0.5   # Alert at 50%
+  }
+  threshold_rules {
+    threshold_percent = 0.8   # Alert at 80%
+  }
+  threshold_rules {
+    threshold_percent = 1.0   # Alert at 100%
+  }
+}
+```
+
+## Storage Optimization
+
+> **COST TIP**: BigQuery automatically moves data to long-term storage after 90 days of no modifications, reducing cost from $0.02/GB to $0.01/GB. Design tables to be append-only where possible.
+
+```sql
+-- Identify tables eligible for long-term pricing
+SELECT
+  table_schema,
+  table_name,
+  TIMESTAMP_MILLIS(last_modified_time) AS last_modified,
+  ROUND(size_bytes / POW(1024, 3), 2) AS size_gb,
+  CASE
+    WHEN TIMESTAMP_MILLIS(last_modified_time) < TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 90 DAY)
+    THEN 'LONG_TERM (50% discount)'
+    ELSE 'ACTIVE'
+  END AS storage_tier
+FROM `project`.`region-us`.INFORMATION_SCHEMA.TABLE_STORAGE
+ORDER BY size_bytes DESC;
+```
+
+## Best Practices Summary
+
+1. **Always partition** - Even small tables benefit from partition pruning
+2. **Cluster by filter columns** - Place highest-cardinality column first
+3. **Require partition filters** - Prevent accidental full scans
+4. **Use materialized views** - Free storage, auto-refresh, transparent to queries
+5. **Monitor with INFORMATION_SCHEMA** - Track bytes scanned, slot usage, cost
+6. **Set budget alerts** - Never be surprised by a BigQuery bill
+7. **Prefer approximate functions** - `APPROX_COUNT_DISTINCT` is 10x faster
+8. **Design for long-term storage** - Append-only tables get 50% discount after 90 days