@cloudstreamsoftware/claude-tools 1.0.0 → 1.2.0

package/skills/gcp-data-engineering/zoho-to-gcp/real-time-vs-batch.md

@@ -0,0 +1,337 @@
+# Real-Time vs Batch: Decision Framework
+
+> When to use real-time streaming, near-real-time polling, or batch extraction for Zoho-to-GCP data flows at CloudStream.
+
+## Decision Framework
+
+### Primary Decision Criteria
+
+```
+Is the data needed within seconds?
+  YES → Real-time (Webhooks → PubSub → Dataflow → BigQuery)
+  NO  →
+    Is the data needed within 15 minutes?
+      YES → Near-real-time (Catalyst Cron every 5 min)
+      NO  →
+        Is the data volume > 50K records/day?
+          YES → Batch with Dataflow (nightly bulk extract)
+          NO  → Batch with CData Sync (daily scheduled)
+```
+
+### Detailed Decision Matrix
+
+| Factor | Real-Time | Near-Real-Time | Batch |
+|--------|-----------|----------------|-------|
+| **Latency** | < 30 seconds | 5-15 minutes | 1-24 hours |
+| **Complexity** | High | Medium | Low |
+| **Monthly Cost** | $200-500+ | $50-150 | $20-80 |
+| **API Credits/Day** | Variable (webhook-driven) | ~2,000-5,000 | ~500-2,000 |
+| **Reliability** | Requires DLQ, retry logic | Simpler retry | Very reliable |
+| **Data Volume** | Any (event-driven) | Low-medium (API pagination) | Any (bulk API) |
+| **Schema Changes** | Must handle dynamically | Easier to adapt | Easiest to adapt |
+| **Monitoring** | Complex (streaming metrics) | Moderate | Simple (job success/fail) |
+
+## Real-Time: Zoho Webhooks to PubSub to Dataflow to BigQuery
+
+### Architecture
+
+```
+Zoho CRM/Books → Webhook → Cloud Function → PubSub → Dataflow → BigQuery (Bronze)
+                                                                        │
+                                                              dbt (Silver/Gold)
+```
+
+### When to Use Real-Time
+
+- Deal stage changes need instant dashboard updates
+- Invoice payments require immediate AR reconciliation
+- Customer support ticket creation triggers automation
+- Multi-system sync (Zoho ↔ other platforms) requires consistency
+
+### Implementation
+
+```python
+# Streaming pipeline: PubSub → Bronze → Silver
+# See dataflow/pipeline-patterns.md for full implementation
+
+# Key configuration for real-time Zoho ingestion:
+pipeline_options = {
+    'streaming': True,
+    'enable_streaming_engine': True,
+    'max_num_workers': 3,            # Low worker count for webhook volume
+    'autoscaling_algorithm': 'THROUGHPUT_BASED',
+    'min_num_workers': 1,            # Scale to 1 during off-hours
+}
+```
+
+### Cost Breakdown (Real-Time)
+
+| Component | Monthly Cost (Estimate) |
+|-----------|------------------------|
+| Cloud Function (webhook receiver) | $5-10 |
+| PubSub (message throughput) | $10-20 |
+| Dataflow (1-3 workers, 24/7) | $150-450 |
+| BigQuery (streaming inserts) | $10-30 |
+| **Total** | **$175-510** |
+
+> **COST WARNING**: Dataflow streaming jobs run 24/7. A single n1-standard-2 worker costs ~$50/month. Even with autoscaling to 1 worker off-peak, this is the most expensive option.
+
+## Near-Real-Time: Catalyst Cron Every 5 Minutes
+
+### Architecture
+
+```
+Catalyst Cron (every 5 min) → Zoho API → PubSub → Cloud Function → BigQuery (Bronze)
+                                    or
+Catalyst Cron (every 5 min) → Zoho API → Direct BigQuery Insert (Bronze)
+```
+
+### When to Use Near-Real-Time
+
+- Dashboard data needs to be "fresh enough" (< 15 min)
+- API credit budget allows frequent polling
+- Data volume per poll is manageable (< 500 records)
+- Real-time is overkill but daily is too stale
+
+### Implementation
+
+```python
+# Catalyst Cron configuration
+# See zoho-to-gcp/data-extraction.md for full Catalyst implementation
+
+# Key considerations:
+# - 5-minute interval = 288 API calls/day per module
+# - With 4 modules (Deals, Contacts, Accounts, Invoices) = 1,152 calls/day
+# - Each call may paginate: 500 records / 200 per page = 3 calls
+# - Worst case: ~3,500 API credits/day
+
+# Optimization: Only poll high-priority modules frequently
+POLL_SCHEDULES = {
+    'Deals': 5,          # Every 5 minutes (critical for pipeline visibility)
+    'Contacts': 15,      # Every 15 minutes
+    'Accounts': 30,      # Every 30 minutes
+    'Invoices': 60,      # Every hour (less frequent changes)
+    'Payments': 15,      # Every 15 minutes (important for AR)
+    'Tasks': 30,         # Every 30 minutes
+}
+```
+
+### Cost Breakdown (Near-Real-Time)
+
+| Component | Monthly Cost (Estimate) |
+|-----------|------------------------|
+| Catalyst Cron (Zoho plan included) | $0 (included in Zoho plan) |
+| PubSub (lower throughput) | $2-5 |
+| Cloud Function (trigger loads) | $5-10 |
+| BigQuery (batch inserts) | $5-10 |
+| Zoho API credits (3,500/day) | Included in plan |
+| **Total** | **$12-25** |
+
+## Batch: Daily CData Sync or Nightly Catalyst Cron
+
+### Architecture
+
+```
+Option A: CData Sync (scheduled) → BigQuery (Bronze) → dbt → Silver/Gold
+Option B: Catalyst Cron (nightly) → GCS (landing) → Cloud Function → BigQuery (Bronze)
+```
+
+### When to Use Batch
+
+- Historical reporting (yesterday's numbers are fine)
+- Large data volumes (full table syncs)
+- Cost sensitivity is high
+- Data does not drive real-time decisions
+- Initial data load / backfill
+
+### Implementation: CData Sync (Simplest)
+
+```yaml
+# CData Sync job configuration
+sync_job:
+  name: nightly_full_sync
+  schedule: "0 2 * * *"  # 2 AM nightly
+  source: zoho_crm
+  tables:
+    - Deals
+    - Contacts
+    - Accounts
+    - Invoices
+    - Payments
+    - Products
+  mode: incremental
+  incremental_column: Modified_Time
+  batch_size: 5000
+  retry_count: 3
+  notification_email: data-team@cloudstreamsoftware.com
+```
+
+### Implementation: Nightly Catalyst Cron (More Control)
+
+```python
+# Catalyst Cron: Nightly full extraction
+def nightly_extract(context, cronDetails):
+    """Run at 2 AM - extract all modified records from past 25 hours."""
+    # 25 hours ensures overlap and no gaps
+    since = (datetime.utcnow() - timedelta(hours=25)).isoformat()
+
+    for module in ['Deals', 'Contacts', 'Accounts', 'Invoices', 'Payments']:
+        records = fetch_all_modified(module, since)
+        upload_to_gcs(records, module)  # Land in GCS, Cloud Function loads to BQ
+
+    print(f"Nightly extract complete")
+```
+
+### Cost Breakdown (Batch)
+
+| Component | Monthly Cost (Estimate) |
+|-----------|------------------------|
+| CData Sync license | $50-200 (varies by plan) |
+| OR Catalyst Cron | $0 (included) |
+| BigQuery (load jobs, free) | $0 |
+| BigQuery (storage) | $5-20 |
+| GCS (landing zone) | $1-5 |
+| **Total (CData)** | **$56-225** |
+| **Total (Catalyst)** | **$6-25** |
+
+## Hybrid Patterns
+
+### Pattern 1: Real-Time for Critical + Batch for Everything
+
+```
+Critical Events (Deal Won, Payment Received):
+  Zoho Webhook → PubSub → Streaming Dataflow → Bronze
+
+Everything Else (Contact updates, Task changes):
+  Nightly CData Sync → Bronze
+```
+
+### Pattern 2: Near-Real-Time Primary + Weekly Full Refresh
+
+```
+Daily Operations:
+  Catalyst Cron (every 15 min) → Incremental to Bronze
+
+Weekly Reconciliation:
+  Sunday 2 AM → Full table refresh → Overwrite Bronze
+  Catches any missed incremental updates
+```
+
+### Pattern 3: Stream Events + Batch Dimensions
+
+```
+Fact/Event Data (Deals, Invoices, Payments):
+  Real-time or Near-real-time → Bronze → Silver facts
+
+Dimension Data (Contacts, Accounts, Products):
+  Daily batch → Bronze → Silver dimensions
+```
+
+## Cost Comparison Table (Monthly)
+
+| Scenario | Real-Time | Near-RT | Batch (CData) | Batch (Catalyst) |
+|----------|-----------|---------|----------------|-------------------|
+| 10K records/day | $200 | $15 | $70 | $10 |
+| 50K records/day | $350 | $40 | $100 | $20 |
+| 200K records/day | $500 | $100 | $200 | $50 |
+| 1M records/day | $700 | N/A (API limits) | $300 | N/A (API limits) |
+
+> **NOTE**: Near-real-time and Catalyst batch hit API credit limits for very high volumes (>200K records/day). Use CData Sync bulk connector or Dataflow for these scenarios.
+
+## When to Upgrade from Batch to Streaming
+
+### Upgrade Signals
+
+1. **Business asks "why isn't this updated yet?"** more than twice a week
+2. **Revenue decisions** depend on same-day deal data
+3. **Customer-facing systems** consume the data (not just internal reports)
+4. **Compliance requirements** mandate near-real-time audit trails
+5. **Multi-system orchestration** requires event-driven triggers
+
+### Migration Path
+
+```
+Phase 1: Daily batch (CData Sync)
+  └── Validate data quality, build silver/gold layers
+
+Phase 2: Near-real-time (Catalyst Cron every 15 min)
+  └── Reduce latency for critical modules only
+
+Phase 3: Real-time for critical events
+  └── Add webhooks for Deal Stage Changes, Payments
+  └── Keep batch for dimension tables
+
+Phase 4: Full real-time (if needed)
+  └── All modules via webhooks + streaming Dataflow
+  └── Keep weekly full refresh for reconciliation
+```
+
+## Monitoring Latency SLAs
+
+```sql
+-- Track data freshness SLA compliance
+CREATE TABLE `project.ops.freshness_sla` (
+  module STRING,
+  extraction_method STRING,  -- 'webhook', 'catalyst_cron', 'cdata_sync'
+  target_latency_minutes INT64,
+  actual_latency_minutes FLOAT64,
+  measured_at TIMESTAMP,
+  sla_met BOOL
+)
+PARTITION BY DATE(measured_at);
+
+-- Freshness check query (run every 15 minutes via Cloud Scheduler)
+INSERT INTO `project.ops.freshness_sla`
+SELECT
+  'Deals' AS module,
+  'catalyst_cron' AS extraction_method,
+  15 AS target_latency_minutes,
+  TIMESTAMP_DIFF(CURRENT_TIMESTAMP(), MAX(_ingestion_timestamp), MINUTE) AS actual_latency_minutes,
+  CURRENT_TIMESTAMP() AS measured_at,
+  TIMESTAMP_DIFF(CURRENT_TIMESTAMP(), MAX(_ingestion_timestamp), MINUTE) <= 15 AS sla_met
+FROM `project.bronze.zoho_deals`;
+
+-- SLA compliance dashboard
+SELECT
+  module,
+  extraction_method,
+  COUNT(*) AS total_checks,
+  COUNTIF(sla_met) AS met_count,
+  ROUND(COUNTIF(sla_met) / COUNT(*) * 100, 1) AS sla_compliance_pct,
+  ROUND(AVG(actual_latency_minutes), 1) AS avg_latency_minutes,
+  MAX(actual_latency_minutes) AS max_latency_minutes
+FROM `project.ops.freshness_sla`
+WHERE measured_at >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 7 DAY)
+GROUP BY 1, 2
+ORDER BY sla_compliance_pct ASC;
+```
+
+```yaml
+# Alert when SLA is violated
+resource "google_monitoring_alert_policy" "data_freshness_sla" {
+  display_name = "Data Freshness SLA Violation"
+  conditions {
+    display_name = "Deals data stale > 20 minutes"
+    condition_threshold {
+      filter = <<-EOT
+        metric.type="custom.googleapis.com/data_freshness/deals_latency_minutes"
+        AND resource.type="global"
+      EOT
+      comparison      = "COMPARISON_GT"
+      threshold_value = 20
+      duration        = "300s"
+    }
+  }
+  notification_channels = [google_monitoring_notification_channel.slack.id]
+}
+```
+
+## Best Practices
+
+1. **Start with batch, upgrade incrementally** - Avoid over-engineering on day one
+2. **Use hybrid patterns** - Not everything needs real-time; mix methods by priority
+3. **Monitor freshness, not just success** - A successful job that is 6 hours late is still a problem
+4. **Budget API credits** - Near-real-time can exhaust daily limits if not controlled
+5. **Keep weekly full refresh** regardless of primary method - catches edge cases
+6. **Separate critical from nice-to-have** - Only stream what truly needs sub-minute latency
+7. **Document SLA expectations** - Written agreement on what "fresh" means per dataset