@cloudstreamsoftware/claude-tools 1.0.0 → 1.2.0

package/skills/gcp-data-engineering/zoho-to-gcp/data-extraction.md

@@ -0,0 +1,543 @@
+# Zoho to GCP Data Extraction
+
+> Patterns for extracting data from Zoho CRM, Books, Creator, and Analytics into GCP using CData Sync, Catalyst Cron, webhooks, and APIs.
+
+## Extraction Method Overview
+
+| Method | Latency | Complexity | Best For |
+|--------|---------|------------|----------|
+| CData Sync | 15min-daily | Low | Bulk initial loads, daily syncs |
+| Catalyst Cron | 5min-hourly | Medium | Scheduled incremental pulls |
+| Zoho Webhooks → PubSub | Real-time | High | Critical business events |
+| Zoho Analytics Export | Daily | Low | Pre-built reports/datasets |
+| Bulk API | Hourly-daily | Medium | Large record sets (>10K) |
+
+## CData Sync Configuration
+
+CData Sync provides a managed connector from Zoho to BigQuery with minimal code.
+
+### Connection Setup
+
+```yaml
+# cdata_sync_config.yml (conceptual - configured via CData UI)
+connections:
+  zoho_crm:
+    driver: ZohoCRM
+    auth_type: OAuth
+    client_id: "${ZOHO_CLIENT_ID}"
+    client_secret: "${ZOHO_CLIENT_SECRET}"
+    refresh_token: "${ZOHO_REFRESH_TOKEN}"
+    api_domain: "https://www.zohoapis.com"
+    accounts_server: "https://accounts.zoho.com"
+
+  bigquery_target:
+    driver: BigQuery
+    project: cloudstream-prod
+    dataset: bronze
+    service_account_key: "${GCP_SA_KEY_PATH}"
+    location: us-central1
+
+sync_jobs:
+  - name: zoho_deals_daily
+    source: zoho_crm
+    target: bigquery_target
+    table: Deals
+    destination_table: zoho_deals
+    schedule: "0 2 * * *"  # 2 AM daily
+    mode: incremental
+    incremental_column: Modified_Time
+    batch_size: 5000
+
+  - name: zoho_contacts_daily
+    source: zoho_crm
+    target: bigquery_target
+    table: Contacts
+    destination_table: zoho_contacts
+    schedule: "0 2 * * *"
+    mode: incremental
+    incremental_column: Modified_Time
+
+  - name: zoho_invoices_daily
+    source: zoho_crm  # Or Zoho Books connection
+    target: bigquery_target
+    table: Invoices
+    destination_table: zoho_invoices
+    schedule: "0 3 * * *"  # After deals sync
+    mode: incremental
+    incremental_column: Modified_Time
+```
+
+### CData Sync Metadata Columns
+
+CData automatically adds these columns - preserve them in bronze:
+
+| Column | Description |
+|--------|-------------|
+| `_cdata_sync_id` | Unique sync operation ID |
+| `_cdata_sync_timestamp` | When the record was synced |
+| `_cdata_sync_operation` | INSERT, UPDATE, or DELETE |
+
+## Catalyst Cron Function API Pulls
+
+### Scheduled Extraction (Every 5 Minutes)
+
+```python
+# catalyst_cron/zoho_crm_sync/main.py
+"""Catalyst Cron function: Pull modified Zoho CRM records every 5 minutes."""
+
+import zcatalyst_sdk
+from google.cloud import bigquery, pubsub_v1
+from datetime import datetime, timedelta
+import requests
+import json
+
+# Zoho API configuration
+ZOHO_API_BASE = "https://www.zohoapis.com/crm/v5"
+MODULES = ['Deals', 'Contacts', 'Accounts', 'Tasks']
+
+def handler(context, cronDetails):
+    """Catalyst Cron entry point - runs every 5 minutes."""
+    app = zcatalyst_sdk.initialize()
+    cache = app.cache()
+
+    for module in MODULES:
+        try:
+            sync_module(app, cache, module)
+        except Exception as e:
+            print(f"Error syncing {module}: {e}")
+            # Continue with other modules even if one fails
+
+def sync_module(app, cache, module_name):
+    """Sync a single Zoho module incrementally."""
+    # Get last sync timestamp from Catalyst cache
+    segment = cache.get_segment('sync_timestamps')
+    last_sync = segment.get(f'{module_name}_last_sync')
+
+    if last_sync:
+        last_sync_time = last_sync['value']
+    else:
+        # First run: get last 24 hours
+        last_sync_time = (datetime.utcnow() - timedelta(hours=24)).strftime('%Y-%m-%dT%H:%M:%S+00:00')
+
+    # Fetch modified records from Zoho
+    access_token = get_zoho_token(app)
+    records = fetch_modified_records(access_token, module_name, last_sync_time)
+
+    if records:
+        # Publish to PubSub for GCP ingestion
+        publish_to_pubsub(records, module_name)
+
+        # Update last sync timestamp
+        segment.put(f'{module_name}_last_sync', {
+            'value': datetime.utcnow().strftime('%Y-%m-%dT%H:%M:%S+00:00')
+        })
+
+    print(f"Synced {len(records)} {module_name} records")
+
+def fetch_modified_records(token, module, since_time):
+    """Fetch records modified since last sync."""
+    headers = {"Authorization": f"Zoho-oauthtoken {token}"}
+    records = []
+    page = 1
+    has_more = True
+
+    while has_more:
+        params = {
+            'modified_since': since_time,
+            'page': page,
+            'per_page': 200,  # Max per page
+            'sort_by': 'Modified_Time',
+            'sort_order': 'asc'
+        }
+
+        response = requests.get(
+            f"{ZOHO_API_BASE}/{module}",
+            headers=headers,
+            params=params
+        )
+
+        if response.status_code == 200:
+            data = response.json()
+            records.extend(data.get('data', []))
+            has_more = data.get('info', {}).get('more_records', False)
+            page += 1
+        elif response.status_code == 204:
+            # No records modified
+            has_more = False
+        else:
+            raise Exception(f"Zoho API error: {response.status_code} - {response.text}")
+
+    return records
+
+def publish_to_pubsub(records, module_name):
+    """Publish records to PubSub for downstream processing."""
+    publisher = pubsub_v1.PublisherClient()
+    topic_path = publisher.topic_path('cloudstream-prod', 'zoho-ingestion')
+
+    for record in records:
+        message = json.dumps({
+            'module': module_name,
+            'record': record,
+            'extracted_at': datetime.utcnow().isoformat(),
+            'source': 'catalyst_cron'
+        }).encode('utf-8')
+
+        publisher.publish(
+            topic_path,
+            message,
+            module=module_name,
+            record_id=str(record.get('id', ''))
+        )
+
+def get_zoho_token(app):
+    """Get fresh OAuth token using refresh token from Catalyst vault."""
+    # Store refresh token in Catalyst Vault (not in code)
+    connector = app.connection()
+    token = connector.get_connector('zoho_crm').execute_request(
+        method='POST',
+        url='https://accounts.zoho.com/oauth/v2/token',
+        params={
+            'grant_type': 'refresh_token',
+            'refresh_token': app.vault().get_secret('zoho_refresh_token'),
+            'client_id': app.vault().get_secret('zoho_client_id'),
+            'client_secret': app.vault().get_secret('zoho_client_secret')
+        }
+    )
+    return token['access_token']
+```
+
+## Webhook-Triggered Ingestion
+
+### Zoho Flow to PubSub
+
+```python
+# cloud_function/zoho_webhook_receiver/main.py
+"""Cloud Function: Receive Zoho webhooks and publish to PubSub."""
+
+import functions_framework
+from google.cloud import pubsub_v1
+import json
+import hashlib
+import hmac
+from datetime import datetime
+
+WEBHOOK_SECRET = "your-webhook-secret"  # Store in Secret Manager
+PROJECT_ID = "cloudstream-prod"
+TOPIC_ID = "zoho-webhooks"
+
+@functions_framework.http
+def receive_webhook(request):
+    """HTTP endpoint for Zoho workflow/Flow webhooks."""
+    # Verify webhook signature
+    if not verify_signature(request):
+        return ('Unauthorized', 401)
+
+    payload = request.get_json(force=True)
+
+    # Extract event metadata
+    event = {
+        'module': payload.get('module', 'unknown'),
+        'event_type': payload.get('event', 'unknown'),  # create, update, delete
+        'record_id': payload.get('id', ''),
+        'record_data': payload.get('data', {}),
+        'triggered_at': datetime.utcnow().isoformat(),
+        'source': 'zoho_webhook'
+    }
+
+    # Publish to PubSub
+    publisher = pubsub_v1.PublisherClient()
+    topic_path = publisher.topic_path(PROJECT_ID, TOPIC_ID)
+
+    future = publisher.publish(
+        topic_path,
+        json.dumps(event).encode('utf-8'),
+        module=event['module'],
+        event_type=event['event_type'],
+        record_id=event['record_id']
+    )
+    message_id = future.result()
+
+    return (json.dumps({'status': 'ok', 'message_id': message_id}), 200)
+
+def verify_signature(request):
+    """Verify Zoho webhook HMAC signature."""
+    signature = request.headers.get('X-Zoho-Webhook-Signature', '')
+    body = request.get_data()
+    expected = hmac.new(
+        WEBHOOK_SECRET.encode(), body, hashlib.sha256
+    ).hexdigest()
+    return hmac.compare_digest(signature, expected)
+```
+
+### Zoho Flow Configuration
+
+```
+Zoho Flow Trigger:
+  Module: Deals
+  Events: Create, Update, Delete
+  Action: Webhook POST to:
+    URL: https://us-central1-cloudstream-prod.cloudfunctions.net/zoho-webhook-receiver
+    Headers:
+      Content-Type: application/json
+      X-Zoho-Webhook-Signature: ${HMAC_SHA256(payload, secret)}
+    Body: ${record_data}
+```
+
+## Zoho Analytics Export API
+
+```python
+# scripts/zoho_analytics_export.py
+"""Export datasets from Zoho Analytics for comparison/validation."""
+
+import requests
+from google.cloud import storage
+import json
+
+ZOHO_ANALYTICS_API = "https://analyticsapi.zoho.com/api"
+ORG_ID = "your-org-id"
+WORKSPACE = "your-workspace"
+
+def export_zoho_analytics_table(table_name, format='json'):
+    """Export a Zoho Analytics table/view to GCS."""
+    token = get_analytics_token()
+
+    response = requests.get(
+        f"{ZOHO_ANALYTICS_API}/{ORG_ID}/{WORKSPACE}/{table_name}",
+        headers={"Authorization": f"Zoho-oauthtoken {token}"},
+        params={
+            'ZOHO_ACTION': 'EXPORT',
+            'ZOHO_OUTPUT_FORMAT': format.upper(),
+            'ZOHO_ERROR_FORMAT': 'JSON',
+            'ZOHO_API_VERSION': '1.0'
+        }
+    )
+
+    if response.status_code == 200:
+        # Upload to GCS
+        client = storage.Client()
+        bucket = client.bucket('cloudstream-landing')
+        blob = bucket.blob(
+            f'zoho-analytics/{table_name}/{datetime.now().strftime("%Y/%m/%d")}/export.{format}'
+        )
+        blob.upload_from_string(response.content)
+        print(f"Exported {table_name} to GCS")
+    else:
+        raise Exception(f"Export failed: {response.status_code}")
+```
+
+## Bulk API vs Single Record
+
+### Decision Matrix
+
+| Factor | Bulk API | Single Record API |
+|--------|----------|-------------------|
+| Records per call | Up to 200 (insert), 100 (read) | 1 |
+| Rate limit impact | 1 API call = 1 credit | 1 API call = 1 credit |
+| Latency | Higher (batching overhead) | Lower |
+| Error handling | Partial success possible | All or nothing |
+| Best for | Initial load, daily sync | Webhooks, real-time |
+| Max daily calls | Based on plan | Based on plan |
+
+### Bulk Read Example
+
+```python
+def bulk_read_zoho_module(module, criteria=None, page=1):
+    """Read up to 200 records per page using bulk read."""
+    headers = {"Authorization": f"Zoho-oauthtoken {get_token()}"}
+
+    # Submit bulk read job
+    body = {
+        "query": {
+            "module": {"api_name": module},
+            "page": page,
+            "per_page": 200
+        }
+    }
+    if criteria:
+        body["query"]["criteria"] = criteria
+
+    response = requests.post(
+        f"{ZOHO_API_BASE}/bulk/read",
+        headers=headers,
+        json=body
+    )
+
+    job_id = response.json()['data'][0]['details']['id']
+
+    # Poll for completion
+    while True:
+        status = requests.get(
+            f"{ZOHO_API_BASE}/bulk/read/{job_id}",
+            headers=headers
+        ).json()
+
+        state = status['data'][0]['state']
+        if state == 'COMPLETED':
+            # Download result file
+            download_url = status['data'][0]['result']['download_url']
+            return requests.get(download_url, headers=headers).content
+        elif state == 'FAILED':
+            raise Exception(f"Bulk read failed: {status}")
+        time.sleep(5)
+```
+
+## Rate Limiting Awareness
+
+> **WARNING**: Zoho API rate limits vary significantly by plan. Exceeding limits returns HTTP 429 and may temporarily block API access.
+
+| Zoho Plan | API Credits/Day | Credits/Minute | Concurrent Requests |
+|-----------|----------------|----------------|---------------------|
+| Free | 5,000 | 100 | 5 |
+| Standard | 10,000 | 150 | 10 |
+| Professional | 15,000 | 200 | 15 |
+| Enterprise | 25,000 | 250 | 25 |
+| Ultimate | 50,000 | 500 | 25 |
+
+```python
+# Rate limiting implementation
+import time
+from functools import wraps
+
+class ZohoRateLimiter:
+    """Rate limiter for Zoho API calls."""
+
+    def __init__(self, calls_per_minute=100, daily_limit=10000):
+        self.calls_per_minute = calls_per_minute
+        self.daily_limit = daily_limit
+        self.minute_calls = []
+        self.daily_count = 0
+
+    def wait_if_needed(self):
+        """Block if rate limit would be exceeded."""
+        now = time.time()
+
+        # Clean old entries
+        self.minute_calls = [t for t in self.minute_calls if now - t < 60]
+
+        if len(self.minute_calls) >= self.calls_per_minute:
+            sleep_time = 60 - (now - self.minute_calls[0])
+            print(f"Rate limit approaching, sleeping {sleep_time:.1f}s")
+            time.sleep(sleep_time)
+
+        if self.daily_count >= self.daily_limit * 0.9:  # 90% threshold warning
+            print(f"WARNING: {self.daily_count}/{self.daily_limit} daily API credits used")
+
+        self.minute_calls.append(now)
+        self.daily_count += 1
+
+rate_limiter = ZohoRateLimiter(calls_per_minute=80, daily_limit=10000)  # 80% of limit
+
+def rate_limited_request(url, headers, params=None):
+    """Make a rate-limited request to Zoho API."""
+    rate_limiter.wait_if_needed()
+    response = requests.get(url, headers=headers, params=params)
+
+    if response.status_code == 429:
+        # Hit rate limit - back off significantly
+        retry_after = int(response.headers.get('Retry-After', 60))
+        print(f"Rate limited! Waiting {retry_after}s")
+        time.sleep(retry_after)
+        return rate_limited_request(url, headers, params)
+
+    return response
+```
+
+## Incremental Extraction (modified_time Filter)
+
+```python
+def incremental_extract(module, last_modified_time):
+    """Extract only records modified since last sync."""
+    headers = {"Authorization": f"Zoho-oauthtoken {get_token()}"}
+
+    all_records = []
+    page = 1
+    has_more = True
+
+    while has_more:
+        response = rate_limited_request(
+            f"{ZOHO_API_BASE}/{module}",
+            headers=headers,
+            params={
+                'modified_since': last_modified_time,  # ISO 8601 format
+                'page': page,
+                'per_page': 200,
+                'sort_by': 'Modified_Time',
+                'sort_order': 'asc',
+                'fields': 'id,Modified_Time,Created_Time,Deal_Name,Amount,Stage,Owner'
+            }
+        )
+
+        if response.status_code == 200:
+            data = response.json()
+            all_records.extend(data.get('data', []))
+            has_more = data.get('info', {}).get('more_records', False)
+            page += 1
+        elif response.status_code == 204:
+            has_more = False  # No modified records
+        else:
+            raise Exception(f"API error: {response.status_code}")
+
+    return all_records
+```
+
+## Full Refresh Patterns
+
+```python
+def full_refresh(module, destination_table):
+    """Full table replacement - use sparingly due to API credit cost."""
+    print(f"Starting full refresh for {module}")
+
+    all_records = []
+    page = 1
+    has_more = True
+
+    while has_more:
+        response = rate_limited_request(
+            f"{ZOHO_API_BASE}/{module}",
+            headers={"Authorization": f"Zoho-oauthtoken {get_token()}"},
+            params={'page': page, 'per_page': 200}
+        )
+
+        if response.status_code == 200:
+            data = response.json()
+            all_records.extend(data.get('data', []))
+            has_more = data.get('info', {}).get('more_records', False)
+            page += 1
+        else:
+            has_more = False
+
+    # Load to BigQuery with WRITE_TRUNCATE (full replace)
+    client = bigquery.Client()
+    job_config = bigquery.LoadJobConfig(
+        write_disposition=bigquery.WriteDisposition.WRITE_TRUNCATE,
+        source_format=bigquery.SourceFormat.NEWLINE_DELIMITED_JSON
+    )
+
+    # Write to temp file, then load
+    import tempfile
+    with tempfile.NamedTemporaryFile(mode='w', suffix='.jsonl') as f:
+        for record in all_records:
+            f.write(json.dumps(record) + '\n')
+        f.flush()
+
+        with open(f.name, 'rb') as source:
+            job = client.load_table_from_file(
+                source, destination_table, job_config=job_config
+            )
+            job.result()
+
+    print(f"Full refresh complete: {len(all_records)} records loaded to {destination_table}")
+```
+
+> **COST TIP**: A full refresh of 50,000 records costs 250 API credits (200 records/page). Do full refreshes weekly at most; use incremental extraction for daily syncs.
+
+## Best Practices
+
+1. **Always use incremental extraction** for daily operations - saves API credits
+2. **Store refresh tokens in Secret Manager** - never in code or environment variables
+3. **Implement rate limiting client-side** - do not rely on 429 responses
+4. **Use PubSub as buffer** - decouples extraction speed from load speed
+5. **Schedule full refreshes weekly** - catches any missed incremental updates
+6. **Monitor API credit usage** - set alerts at 70% daily consumption
+7. **Use bulk API for initial loads** - more efficient than single-record pagination
+8. **Preserve all Zoho metadata** - `Modified_Time`, `Created_Time`, `Owner` are critical for silver layer