@cloudstreamsoftware/claude-tools 1.0.0 → 1.1.0

package/skills/zoho-patterns/deluge/batch-processing.md

@@ -0,0 +1,403 @@
+# Deluge Batch Processing
+
+## Critical Constraints
+
+> **5000 STATEMENT LIMIT:** Every Deluge execution (workflow, custom function, scheduled task) is limited to 5000 statements total. Loops, conditionals, and function calls all count. Plan accordingly.
+
+> **getRecords MAX 200:** Each `getRecords` call returns at most 200 records. You MUST paginate to fetch more.
+
+> **Subforms MAX 200 ROWS:** Subforms are capped at 200 rows. For larger datasets, use related child forms.
+
+## Pagination Pattern (getRecords)
+
+### Basic Pagination
+
+```deluge
+// Fetch all records using pagination
+allRecords = List();
+page = 1;
+pageSize = 200;
+hasMore = true;
+
+while (hasMore)
+{
+    batch = zoho.creator.getRecords("app", "All_Records_Report", "", page, pageSize);
+
+    if (batch.size() > 0)
+    {
+        allRecords.addAll(batch);
+        page = page + 1;
+    }
+
+    if (batch.size() < pageSize)
+    {
+        hasMore = false;
+    }
+
+    // Safety: prevent statement limit issues
+    if (page > 25) // 25 pages * 200 = 5000 records max per execution
+    {
+        info "Reached page limit, process remaining in next run";
+        hasMore = false;
+    }
+}
+
+info "Total records fetched: " + allRecords.size();
+```
+
+### Pagination with Criteria
+
+```deluge
+// Paginate with filter criteria
+criteria = "Status == \"Pending\" && Created_Date >= \"01-Jan-2024\"";
+allPending = List();
+page = 1;
+
+while (true)
+{
+    batch = zoho.creator.getRecords("app", "All_Orders", criteria, page, 200);
+
+    if (batch.size() == 0)
+    {
+        break;
+    }
+
+    allPending.addAll(batch);
+    page = page + 1;
+
+    // Statement budget check (rough estimate)
+    if (page > 20)
+    {
+        break;
+    }
+}
+```
+
+## Statement Budget Estimation
+
+| Operation | Approximate Cost |
+|-----------|-----------------|
+| Variable assignment | 1 statement |
+| if/else check | 1 statement |
+| for-each iteration | 1 per loop |
+| getRecords call | 1 statement |
+| createRecord call | 1 statement |
+| invokeUrl call | 1 statement |
+| Map/List operations | 1 per operation |
+| String operations | 1 per operation |
+
+### Budget Calculator
+
+```deluge
+// Example: Processing 200 records with 15 operations each
+// Budget: 200 records * 15 ops = 3000 + pagination overhead (~50) = 3050
+// Available: 5000 - 3050 = 1950 statements for error handling, logging
+
+// Rule of thumb: Keep processing under 3500 statements
+// Reserve 1500 for infrastructure (pagination, error handling, reporting)
+```
+
+## Chunking Large Operations
+
+### Pattern 1: Process in Fixed Chunks
+
+```deluge
+// Process records in chunks to stay under statement limit
+CHUNK_SIZE = 50; // Conservative: allows ~80 statements per record
+allRecords = zoho.creator.getRecords("app", "Pending_Report", "", 1, 200);
+
+processedCount = 0;
+errorCount = 0;
+chunkStart = 0;
+
+while (chunkStart < allRecords.size() && chunkStart < CHUNK_SIZE)
+{
+    record = allRecords.get(chunkStart);
+
+    try
+    {
+        // Process single record (keep operations minimal)
+        updateMap = Map();
+        updateMap.put("Processed_Date", zoho.currenttime);
+        updateMap.put("Status", "Processed");
+        updateMap.put("Calculated_Value", record.get("Quantity").toLong() * record.get("Rate").toDecimal());
+
+        zoho.creator.updateRecord("app", "Orders", record.get("ID").toLong(), updateMap);
+        processedCount = processedCount + 1;
+    }
+    catch (e)
+    {
+        errorCount = errorCount + 1;
+        info "Error processing record " + record.get("ID") + ": " + e.toString();
+    }
+
+    chunkStart = chunkStart + 1;
+}
+
+info "Processed: " + processedCount + ", Errors: " + errorCount;
+```
+
+### Pattern 2: Marker-Based Chunking (Resume Between Runs)
+
+```deluge
+// Use a marker field to track progress across scheduled runs
+criteria = "Batch_Processed == false";
+batch = zoho.creator.getRecords("app", "All_Records", criteria, 1, 100);
+
+processedIds = List();
+
+for each record in batch
+{
+    try
+    {
+        // Your processing logic here
+        result = processRecord(record);
+
+        // Mark as processed
+        zoho.creator.updateRecord("app", "Records", record.get("ID").toLong(), {
+            "Batch_Processed": true,
+            "Batch_Result": result,
+            "Batch_Date": zoho.currentdate
+        });
+
+        processedIds.add(record.get("ID"));
+    }
+    catch (e)
+    {
+        // Mark as failed (don't reprocess next run without review)
+        zoho.creator.updateRecord("app", "Records", record.get("ID").toLong(), {
+            "Batch_Processed": true,
+            "Batch_Result": "ERROR: " + e.toString(),
+            "Batch_Date": zoho.currentdate
+        });
+    }
+}
+
+info "Batch complete: " + processedIds.size() + " records processed";
+// Next scheduled run will pick up remaining unprocessed records
+```
+
+## Tracking Progress
+
+### Progress Tracking Form
+
+Create a `Batch_Jobs` form with fields:
+- Job_Name (Text)
+- Total_Records (Number)
+- Processed_Count (Number)
+- Error_Count (Number)
+- Status (Dropdown: Running, Completed, Failed, Partial)
+- Started_At (DateTime)
+- Completed_At (DateTime)
+- Last_Error (Multi-line)
+
+```deluge
+// Initialize batch job tracking
+jobMap = Map();
+jobMap.put("Job_Name", "Monthly Invoice Generation");
+jobMap.put("Total_Records", totalCount);
+jobMap.put("Processed_Count", 0);
+jobMap.put("Error_Count", 0);
+jobMap.put("Status", "Running");
+jobMap.put("Started_At", zoho.currenttime);
+
+jobRecord = zoho.creator.createRecord("admin-app", "Batch_Jobs", jobMap);
+jobId = jobRecord.get("ID").toLong();
+
+// ... process records ...
+
+// Update progress periodically (every 50 records)
+if (processedCount % 50 == 0)
+{
+    zoho.creator.updateRecord("admin-app", "Batch_Jobs", jobId, {
+        "Processed_Count": processedCount,
+        "Error_Count": errorCount
+    });
+}
+
+// Final update
+zoho.creator.updateRecord("admin-app", "Batch_Jobs", jobId, {
+    "Processed_Count": processedCount,
+    "Error_Count": errorCount,
+    "Status": if(errorCount == 0, "Completed", "Partial"),
+    "Completed_At": zoho.currenttime
+});
+```
+
+## Error Accumulation and Reporting
+
+```deluge
+// Collect errors during batch, report at end
+errors = List();
+warnings = List();
+processed = 0;
+
+records = zoho.creator.getRecords("app", "Report", "", 1, 200);
+
+for each record in records
+{
+    try
+    {
+        // Validate before processing
+        if (record.get("Email") == null || record.get("Email") == "")
+        {
+            warnings.add("Record " + record.get("ID") + ": Missing email, skipped");
+            continue;
+        }
+
+        // Process
+        result = doProcessing(record);
+        processed = processed + 1;
+    }
+    catch (e)
+    {
+        errors.add({
+            "record_id": record.get("ID"),
+            "error": e.toString(),
+            "timestamp": zoho.currenttime
+        });
+    }
+}
+
+// Generate report
+report = "Batch Processing Report\n";
+report = report + "========================\n";
+report = report + "Total: " + records.size() + "\n";
+report = report + "Processed: " + processed + "\n";
+report = report + "Errors: " + errors.size() + "\n";
+report = report + "Warnings: " + warnings.size() + "\n\n";
+
+if (errors.size() > 0)
+{
+    report = report + "ERRORS:\n";
+    for each err in errors
+    {
+        report = report + "- Record " + err.get("record_id") + ": " + err.get("error") + "\n";
+    }
+}
+
+// Send report
+sendmail [
+    from: zoho.adminuserid
+    to: "admin@company.com"
+    subject: "Batch Report: " + processed + "/" + records.size() + " processed"
+    message: report
+];
+```
+
+## Scheduled Function Orchestration
+
+For processing more than ~1000 records, split across multiple scheduled runs:
+
+```deluge
+// Master scheduler: runs every 15 minutes
+// Checks if there's pending work and processes a chunk
+
+// 1. Check batch status
+statusRecords = zoho.creator.getRecords("admin-app", "Batch_Status", "Job_Name == \"daily_sync\"", 1, 1);
+
+if (statusRecords.size() > 0)
+{
+    status = statusRecords.get(0);
+    lastOffset = status.get("Last_Offset").toLong();
+    totalRecords = status.get("Total_Records").toLong();
+
+    if (lastOffset >= totalRecords)
+    {
+        info "All records processed. Resetting for tomorrow.";
+        zoho.creator.updateRecord("admin-app", "Batch_Status", status.get("ID").toLong(), {
+            "Last_Offset": 0,
+            "Status": "Idle"
+        });
+        return;
+    }
+
+    // 2. Process next chunk
+    CHUNK = 100;
+    page = (lastOffset / 200).toLong() + 1;
+    startInPage = lastOffset % 200;
+
+    records = zoho.creator.getRecords("app", "All_Records", "", page, 200);
+    processedInRun = 0;
+
+    idx = startInPage.toLong();
+    while (idx < records.size() && processedInRun < CHUNK)
+    {
+        record = records.get(idx);
+        processRecord(record);
+        processedInRun = processedInRun + 1;
+        idx = idx + 1;
+    }
+
+    // 3. Update offset
+    newOffset = lastOffset + processedInRun;
+    zoho.creator.updateRecord("admin-app", "Batch_Status", status.get("ID").toLong(), {
+        "Last_Offset": newOffset,
+        "Status": "In Progress",
+        "Last_Run": zoho.currenttime
+    });
+
+    info "Processed " + processedInRun + " records. Progress: " + newOffset + "/" + totalRecords;
+}
+```
+
+## Parallel Processing Limitations
+
+> **WARNING:** Deluge does NOT support parallel processing within a single execution. Each statement runs sequentially. For parallelism, use:
+
+1. **Multiple scheduled functions** running at different intervals
+2. **Catalyst Cron** with async operations (see catalyst/cron-batch-processing.md)
+3. **Zoho Flow** for event-driven parallel processing
+
+### Pseudo-Parallel: Multiple Scheduled Functions
+
+```
+Schedule 1 (every 15 min): Process records where ID % 3 == 0
+Schedule 2 (every 15 min, offset 5 min): Process records where ID % 3 == 1
+Schedule 3 (every 15 min, offset 10 min): Process records where ID % 3 == 2
+```
+
+```deluge
+// Partition-based processing (Schedule 1)
+criteria = "Mod(ID, 3) == 0 && Processed == false";
+myPartition = zoho.creator.getRecords("app", "Records", criteria, 1, 100);
+
+for each record in myPartition
+{
+    processAndMark(record);
+}
+```
+
+## Performance Tips
+
+1. **Minimize getRecords calls** - Fetch 200 at once, not 1 at a time
+2. **Batch updates** - Collect changes, apply in bulk where possible
+3. **Pre-filter with criteria** - Don't fetch all records then filter in code
+4. **Avoid nested loops** - O(n^2) statement consumption
+5. **Use formula fields** - Offload calculations to form-level formulas
+6. **Cache lookups** - Store repeated lookups in Maps
+
+```deluge
+// BAD: N+1 query pattern (fetches inside loop)
+records = zoho.creator.getRecords("app", "Orders", "", 1, 200);
+for each record in records
+{
+    customer = zoho.creator.getRecords("app", "Customers", "ID == " + record.get("Customer_ID"), 1, 1);
+    // This makes 200+ API calls!
+}
+
+// GOOD: Pre-fetch and cache
+customers = zoho.creator.getRecords("app", "Customers", "", 1, 200);
+customerMap = Map();
+for each cust in customers
+{
+    customerMap.put(cust.get("ID").toString(), cust);
+}
+
+records = zoho.creator.getRecords("app", "Orders", "", 1, 200);
+for each record in records
+{
+    customer = customerMap.get(record.get("Customer_ID").toString());
+    // No additional API call needed!
+}
+```