pi-cloudflare 0.1.0

package/skills/cloudflare/references/queues/gotchas.md

@@ -0,0 +1,206 @@
+# Queues Gotchas & Troubleshooting
+
+## CRITICAL: Top Production Mistakes
+
+### 1. "Entire Batch Retried After Single Error"
+
+**Problem:** Throwing uncaught error in queue handler retries the entire batch, not just the failed message  
+**Cause:** Uncaught exceptions propagate to the runtime, triggering batch-level retry  
+**Solution:** Always wrap individual message processing in try/catch and call `msg.retry()` explicitly
+
+```typescript
+// ❌ BAD: Throws error, retries entire batch
+async queue(batch: MessageBatch): Promise<void> {
+  for (const msg of batch.messages) {
+    await riskyOperation(msg.body); // If this throws, entire batch retries
+    msg.ack();
+  }
+}
+
+// ✅ GOOD: Catch per message, handle individually
+async queue(batch: MessageBatch): Promise<void> {
+  for (const msg of batch.messages) {
+    try {
+      await riskyOperation(msg.body);
+      msg.ack();
+    } catch (error) {
+      msg.retry({ delaySeconds: 60 });
+    }
+  }
+}
+```
+
+### 2. "Messages Retry Forever"
+
+**Problem:** Messages not explicitly ack'd or retry'd will auto-retry indefinitely  
+**Cause:** Runtime default behavior retries unhandled messages until `max_retries` reached  
+**Solution:** Always call `msg.ack()` or `msg.retry()` for each message. Never leave messages unhandled.
+
+```typescript
+// ❌ BAD: Skipped messages auto-retry forever
+async queue(batch: MessageBatch): Promise<void> {
+  for (const msg of batch.messages) {
+    if (shouldProcess(msg.body)) {
+      await process(msg.body);
+      msg.ack();
+    }
+    // Missing: msg.ack() for skipped messages - they will retry!
+  }
+}
+
+// ✅ GOOD: Explicitly handle all messages
+async queue(batch: MessageBatch): Promise<void> {
+  for (const msg of batch.messages) {
+    if (shouldProcess(msg.body)) {
+      await process(msg.body);
+      msg.ack();
+    } else {
+      msg.ack(); // Explicitly ack even if not processing
+    }
+  }
+}
+```
+
+## Common Errors
+
+### "Duplicate Message Processing"
+
+**Problem:** Same message processed multiple times  
+**Cause:** At-least-once delivery guarantee means duplicates are possible during retries  
+**Solution:** Design consumers to be idempotent by tracking processed message IDs in KV with expiration TTL
+
+```typescript
+async queue(batch: MessageBatch, env: Env): Promise<void> {
+  for (const msg of batch.messages) {
+    const processed = await env.PROCESSED_KV.get(msg.id);
+    if (processed) {
+      msg.ack();
+      continue;
+    }
+    
+    await processMessage(msg.body);
+    await env.PROCESSED_KV.put(msg.id, '1', { expirationTtl: 86400 });
+    msg.ack();
+  }
+}
+```
+
+### "Pull Consumer Can't Decode Messages"
+
+**Problem:** Pull consumer or dashboard shows unreadable message bodies  
+**Cause:** Messages sent with `v8` content type are only decodable by Workers push consumers  
+**Solution:** Use `json` content type for pull consumers or dashboard visibility
+
+```typescript
+// Use json for pull consumers
+await env.MY_QUEUE.send(data, { contentType: 'json' });
+
+// Use v8 only for push consumers with complex JS types
+await env.MY_QUEUE.send({ date: new Date(), tags: new Set() }, { contentType: 'v8' });
+```
+
+### "Messages Not Being Delivered"
+
+**Problem:** Messages sent but consumer not processing  
+**Cause:** Queue paused, consumer not configured, or consumer errors  
+**Solution:** Check queue status with `wrangler queues list`, verify consumer configured with `wrangler queues consumer add`, and check logs with `wrangler tail`
+
+### "High Dead Letter Queue Rate"
+
+**Problem:** Many messages ending up in DLQ  
+**Cause:** Consumer repeatedly failing to process messages after max retries  
+**Solution:** Review consumer error logs, check external dependency availability, verify message format matches expectations, or increase retry delay
+
+## Error Classification Patterns
+
+Classify errors to decide whether to retry or DLQ:
+
+```typescript
+async queue(batch: MessageBatch, env: Env): Promise<void> {
+  for (const msg of batch.messages) {
+    try {
+      await processMessage(msg.body);
+      msg.ack();
+    } catch (error) {
+      // Transient errors: retry with backoff
+      if (isRetryable(error)) {
+        const delay = Math.min(30 * (2 ** msg.attempts), 43200);
+        msg.retry({ delaySeconds: delay });
+      } 
+      // Permanent errors: ack to avoid infinite retries
+      else {
+        console.error('Permanent error, sending to DLQ:', error);
+        await env.ERROR_LOG.put(msg.id, JSON.stringify({ msg: msg.body, error: String(error) }));
+        msg.ack(); // Prevent further retries
+      }
+    }
+  }
+}
+
+function isRetryable(error: unknown): boolean {
+  if (error instanceof Response) {
+    // Retry: rate limits, timeouts, server errors
+    return error.status === 429 || error.status >= 500;
+  }
+  if (error instanceof Error) {
+    // Don't retry: validation, auth, not found
+    return !error.message.includes('validation') && 
+           !error.message.includes('unauthorized') &&
+           !error.message.includes('not found');
+  }
+  return false; // Unknown errors don't retry
+}
+```
+
+### "CPU Time Exceeded in Consumer"
+
+**Problem:** Consumer fails with CPU time limit exceeded  
+**Cause:** Consumer processing exceeding 30s default CPU time limit  
+**Solution:** Increase CPU limit in wrangler.jsonc: `{ "limits": { "cpu_ms": 300000 } }` (5 minutes max)
+
+## Content Type Decision Guide
+
+**When to use each content type:**
+
+| Content Type | Use When | Readable By | Supports |
+|--------------|----------|-------------|----------|
+| `json` (default) | Pull consumers, dashboard visibility, simple objects | All (push/pull/dashboard) | JSON-serializable types only |
+| `v8` | Push consumers only, complex JS objects | Push consumers only | Date, Map, Set, BigInt, typed arrays |
+| `text` | String-only payloads | All | Strings only |
+| `bytes` | Binary data (images, files) | All | ArrayBuffer, Uint8Array |
+
+**Decision tree:**
+1. Need to view in dashboard or use pull consumer? → Use `json`
+2. Need Date, Map, Set, or other V8 types? → Use `v8` (push consumers only)
+3. Just strings? → Use `text`
+4. Binary data? → Use `bytes`
+
+```typescript
+// Dashboard/pull: use json
+await env.QUEUE.send({ id: 123, name: 'test' }, { contentType: 'json' });
+
+// Complex JS types (push only): use v8
+await env.QUEUE.send({ 
+  created: new Date(), 
+  tags: new Set(['a', 'b']) 
+}, { contentType: 'v8' });
+```
+
+## Limits
+
+| Limit | Value | Notes |
+|-------|-------|-------|
+| Max queues | 10,000 | Per account |
+| Message size | 128 KB | Maximum per message |
+| Batch size (consumer) | 100 messages | Maximum messages per batch |
+| Batch size (sendBatch) | 100 msgs or 256 KB | Whichever limit reached first |
+| Throughput | 5,000 msgs/sec | Per queue |
+| Retention | 4-14 days | Configurable retention period |
+| Max backlog | 25 GB | Maximum queue backlog size |
+| Max delay | 12 hours (43,200s) | Maximum message delay |
+| Max retries | 100 | Maximum retry attempts |
+| CPU time default | 30s | Per consumer invocation |
+| CPU time max | 300s (5 min) | Configurable via `limits.cpu_ms` |
+| Operations per message | 3 (write + read + delete) | Base cost per message |
+| Pricing | $0.40 per 1M operations | After 1M free operations |
+| Message charging | Per 64 KB chunk | Messages charged in 64 KB increments |

package/skills/cloudflare/references/queues/patterns.md

@@ -0,0 +1,220 @@
+# Queues Patterns & Best Practices
+
+## Async Task Processing
+
+```typescript
+// Producer: Accept request, queue work
+export default {
+  async fetch(request: Request, env: Env): Promise<Response> {
+    const { userId, reportType } = await request.json();
+    await env.REPORT_QUEUE.send({ userId, reportType, requestedAt: Date.now() });
+    return Response.json({ message: 'Report queued', status: 'pending' });
+  }
+};
+
+// Consumer: Process reports
+export default {
+  async queue(batch: MessageBatch, env: Env): Promise<void> {
+    for (const msg of batch.messages) {
+      const { userId, reportType } = msg.body;
+      const report = await generateReport(userId, reportType, env);
+      await env.REPORTS_BUCKET.put(`${userId}/${reportType}.pdf`, report);
+      msg.ack();
+    }
+  }
+};
+```
+
+## Buffering API Calls
+
+```typescript
+// Producer: Queue log entries
+ctx.waitUntil(env.LOGS_QUEUE.send({
+  method: request.method,
+  url: request.url,
+  timestamp: Date.now()
+}));
+
+// Consumer: Batch write to external API
+async queue(batch: MessageBatch, env: Env): Promise<void> {
+  const logs = batch.messages.map(m => m.body);
+  await fetch(env.LOG_ENDPOINT, { method: 'POST', body: JSON.stringify({ logs }) });
+  batch.ackAll();
+}
+```
+
+## Rate Limiting Upstream
+
+```typescript
+async queue(batch: MessageBatch, env: Env): Promise<void> {
+  for (const msg of batch.messages) {
+    try {
+      await callRateLimitedAPI(msg.body);
+      msg.ack();
+    } catch (error) {
+      if (error.status === 429) {
+        const retryAfter = parseInt(error.headers.get('Retry-After') || '60');
+        msg.retry({ delaySeconds: retryAfter });
+      } else throw error;
+    }
+  }
+}
+```
+
+## Event-Driven Workflows
+
+```typescript
+// R2 event → Queue → Worker
+export default {
+  async queue(batch: MessageBatch, env: Env): Promise<void> {
+    for (const msg of batch.messages) {
+      const event = msg.body;
+      if (event.action === 'PutObject') {
+        await processNewFile(event.object.key, env);
+      } else if (event.action === 'DeleteObject') {
+        await cleanupReferences(event.object.key, env);
+      }
+      msg.ack();
+    }
+  }
+};
+```
+
+## Dead Letter Queue Pattern
+
+```typescript
+// Main queue: After max_retries, goes to DLQ automatically
+export default {
+  async queue(batch: MessageBatch, env: Env): Promise<void> {
+    for (const msg of batch.messages) {
+      try {
+        await riskyOperation(msg.body);
+        msg.ack();
+      } catch (error) {
+        console.error(`Failed after ${msg.attempts} attempts:`, error);
+      }
+    }
+  }
+};
+
+// DLQ consumer: Log and store failed messages
+export default {
+  async queue(batch: MessageBatch, env: Env): Promise<void> {
+    for (const msg of batch.messages) {
+      await env.FAILED_KV.put(msg.id, JSON.stringify(msg.body));
+      msg.ack();
+    }
+  }
+};
+```
+
+## Priority Queues
+
+High priority: `max_batch_size: 5, max_batch_timeout: 1`. Low priority: `max_batch_size: 100, max_batch_timeout: 30`.
+
+## Delayed Job Processing
+
+```typescript
+await env.EMAIL_QUEUE.send({ to, template, userId }, { delaySeconds: 3600 });
+```
+
+## Fan-out Pattern
+
+```typescript
+async fetch(request: Request, env: Env): Promise<Response> {
+  const event = await request.json();
+  
+  // Send to multiple queues for parallel processing
+  await Promise.all([
+    env.ANALYTICS_QUEUE.send(event),
+    env.NOTIFICATIONS_QUEUE.send(event),
+    env.AUDIT_LOG_QUEUE.send(event)
+  ]);
+  
+  return Response.json({ status: 'processed' });
+}
+```
+
+## Idempotency Pattern
+
+```typescript
+async queue(batch: MessageBatch, env: Env): Promise<void> {
+  for (const msg of batch.messages) {
+    // Check if already processed
+    const processed = await env.PROCESSED_KV.get(msg.id);
+    if (processed) {
+      msg.ack();
+      continue;
+    }
+    
+    await processMessage(msg.body);
+    await env.PROCESSED_KV.put(msg.id, '1', { expirationTtl: 86400 });
+    msg.ack();
+  }
+}
+```
+
+## Integration: D1 Batch Writes
+
+```typescript
+async queue(batch: MessageBatch, env: Env): Promise<void> {
+  // Collect all inserts for single D1 batch
+  const statements = batch.messages.map(msg => 
+    env.DB.prepare('INSERT INTO events (id, data, created) VALUES (?, ?, ?)')
+      .bind(msg.id, JSON.stringify(msg.body), Date.now())
+  );
+  
+  try {
+    await env.DB.batch(statements);
+    batch.ackAll();
+  } catch (error) {
+    console.error('D1 batch failed:', error);
+    batch.retryAll({ delaySeconds: 60 });
+  }
+}
+```
+
+## Integration: Workflows
+
+```typescript
+// Queue triggers Workflow for long-running tasks
+async queue(batch: MessageBatch, env: Env): Promise<void> {
+  for (const msg of batch.messages) {
+    try {
+      const instance = await env.MY_WORKFLOW.create({
+        id: msg.id,
+        params: msg.body
+      });
+      console.log('Workflow started:', instance.id);
+      msg.ack();
+    } catch (error) {
+      msg.retry({ delaySeconds: 30 });
+    }
+  }
+}
+```
+
+## Integration: Durable Objects
+
+```typescript
+// Queue distributes work to Durable Objects by ID
+async queue(batch: MessageBatch, env: Env): Promise<void> {
+  for (const msg of batch.messages) {
+    const { userId, action } = msg.body;
+    
+    // Route to user-specific DO
+    const id = env.USER_DO.idFromName(userId);
+    const stub = env.USER_DO.get(id);
+    
+    try {
+      await stub.fetch(new Request('https://do/process', {
+        method: 'POST',
+        body: JSON.stringify({ action, messageId: msg.id })
+      }));
+      msg.ack();
+    } catch (error) {
+      msg.retry({ delaySeconds: 60 });
+    }
+  }
+}
+```

package/skills/cloudflare/references/r2/README.md

@@ -0,0 +1,95 @@
+# Cloudflare R2 Object Storage
+
+S3-compatible object storage with zero egress fees, optimized for large file storage and delivery.
+
+## Overview
+
+R2 provides:
+- S3-compatible API (Workers API + S3 REST)
+- Zero egress fees globally
+- Strong consistency for writes/deletes
+- Storage classes (Standard/Infrequent Access)
+- SSE-C encryption support
+
+**Use cases:** Media storage, backups, static assets, user uploads, data lakes
+
+## Quick Start
+
+```bash
+wrangler r2 bucket create my-bucket --location=enam
+wrangler r2 object put my-bucket/file.txt --file=./local.txt
+```
+
+```typescript
+// Upload
+await env.MY_BUCKET.put(key, data, {
+  httpMetadata: { contentType: 'image/jpeg' }
+});
+
+// Download
+const object = await env.MY_BUCKET.get(key);
+if (object) return new Response(object.body);
+```
+
+## Core Operations
+
+| Method | Purpose | Returns |
+|--------|---------|---------|
+| `put(key, value, options?)` | Upload object | `R2Object \| null` |
+| `get(key, options?)` | Download object | `R2ObjectBody \| R2Object \| null` |
+| `head(key)` | Get metadata only | `R2Object \| null` |
+| `delete(keys)` | Delete object(s) | `Promise<void>` |
+| `list(options?)` | List objects | `R2Objects` |
+
+## Storage Classes
+
+- **Standard**: Frequent access, low latency reads
+- **InfrequentAccess**: 30-day minimum storage, retrieval fees, lower storage cost
+
+## Event Notifications
+
+R2 integrates with Cloudflare Queues for reactive workflows:
+
+```typescript
+// wrangler.jsonc
+{
+  "event_notifications": [{
+    "queue": "r2-notifications",
+    "actions": ["PutObject", "DeleteObject"]
+  }]
+}
+
+// Consumer
+async queue(batch: MessageBatch, env: Env) {
+  for (const message of batch.messages) {
+    const event = message.body; // { action, bucket, object, timestamps }
+    if (event.action === 'PutObject') {
+      // Process upload: thumbnail generation, virus scan, etc.
+    }
+  }
+}
+```
+
+## Reading Order
+
+**First-time users:** README → configuration.md → api.md → patterns.md  
+**Specific tasks:**
+- Setup: configuration.md
+- Client uploads: patterns.md (presigned URLs)
+- Public static site: patterns.md (public access + custom domain)
+- Processing uploads: README (event notifications) + queues reference
+- Debugging: gotchas.md
+
+## In This Reference
+
+- [configuration.md](./configuration.md) - Bindings, S3 SDK, CORS, lifecycles, token scopes
+- [api.md](./api.md) - Workers API, multipart, conditional requests, presigned URLs
+- [patterns.md](./patterns.md) - Streaming, caching, client uploads, public buckets
+- [gotchas.md](./gotchas.md) - List truncation, etag format, stream length, S3 SDK region
+
+## See Also
+
+- [workers](../workers/) - Worker runtime and fetch handlers
+- [kv](../kv/) - Metadata storage for R2 objects
+- [d1](../d1/) - Store R2 URLs in relational database
+- [queues](../queues/) - Process R2 uploads asynchronously