@mohanscodex/spectra-code 0.4.9 → 0.5.0

package/skills/cloudflare-deploy/references/r2/gotchas.md

@@ -0,0 +1,190 @@
+# R2 Gotchas & Troubleshooting
+
+## List Truncation
+
+```typescript
+// ❌ WRONG: Don't compare object count when using include
+while (listed.objects.length < options.limit) { ... }
+
+// ✅ CORRECT: Always use truncated property
+while (listed.truncated) {
+  const next = await env.MY_BUCKET.list({ cursor: listed.cursor });
+  // ...
+}
+```
+
+**Reason:** `include` with metadata may return fewer objects per page to fit metadata.
+
+## ETag Format
+
+```typescript
+// ❌ WRONG: Using etag (unquoted) in headers
+headers.set('etag', object.etag); // Missing quotes
+
+// ✅ CORRECT: Use httpEtag (quoted)
+headers.set('etag', object.httpEtag);
+```
+
+## Checksum Limits
+
+Only ONE checksum algorithm allowed per PUT:
+
+```typescript
+// ❌ WRONG: Multiple checksums
+await env.MY_BUCKET.put(key, data, { md5: hash1, sha256: hash2 }); // Error
+
+// ✅ CORRECT: Pick one
+await env.MY_BUCKET.put(key, data, { sha256: hash });
+```
+
+## Multipart Requirements
+
+- All parts must be uniform size (except last part)
+- Part numbers start at 1 (not 0)
+- Uncompleted uploads auto-abort after 7 days
+- `resumeMultipartUpload` doesn't validate uploadId existence
+
+## Conditional Operations
+
+```typescript
+// Precondition failure returns object WITHOUT body
+const object = await env.MY_BUCKET.get(key, {
+  onlyIf: { etagMatches: '"wrong"' }
+});
+
+// Check for body, not just null
+if (!object) return new Response('Not found', { status: 404 });
+if (!object.body) return new Response(null, { status: 304 }); // Precondition failed
+```
+
+## Key Validation
+
+```typescript
+// ❌ DANGEROUS: Path traversal
+const key = url.pathname.slice(1); // Could be ../../../etc/passwd
+await env.MY_BUCKET.get(key);
+
+// ✅ SAFE: Validate keys
+if (!key || key.includes('..') || key.startsWith('/')) {
+  return new Response('Invalid key', { status: 400 });
+}
+```
+
+## Storage Class Pitfalls
+
+- InfrequentAccess: 30-day minimum billing (even if deleted early)
+- Can't transition IA → Standard via lifecycle (use S3 CopyObject)
+- Retrieval fees apply for IA reads
+
+## Stream Length Requirement
+
+```typescript
+// ❌ WRONG: Streaming unknown length fails silently
+const response = await fetch(url);
+await env.MY_BUCKET.put(key, response.body); // May fail without error
+
+// ✅ CORRECT: Buffer or use Content-Length
+const data = await response.arrayBuffer();
+await env.MY_BUCKET.put(key, data);
+
+// OR: Pass Content-Length if known
+const object = await env.MY_BUCKET.put(key, request.body, {
+  httpMetadata: {
+    contentLength: parseInt(request.headers.get('content-length') || '0')
+  }
+});
+```
+
+**Reason:** R2 requires known length for streams. Unknown length may cause silent truncation.
+
+## S3 SDK Region Configuration
+
+```typescript
+// ❌ WRONG: Missing region breaks ALL S3 SDK calls
+const s3 = new S3Client({
+  endpoint: `https://${accountId}.r2.cloudflarestorage.com`,
+  credentials: { ... }
+});
+
+// ✅ CORRECT: MUST set region='auto'
+const s3 = new S3Client({
+  region: 'auto', // REQUIRED
+  endpoint: `https://${accountId}.r2.cloudflarestorage.com`,
+  credentials: { ... }
+});
+```
+
+**Reason:** S3 SDK requires region. R2 uses 'auto' as placeholder.
+
+## Local Development Limits
+
+```typescript
+// ❌ Miniflare/wrangler dev: Limited R2 support
+// - No multipart uploads
+// - No presigned URLs (requires S3 SDK + network)
+// - Memory-backed storage (lost on restart)
+
+// ✅ Use remote bindings for full features
+wrangler dev --remote
+
+// OR: Conditional logic
+if (env.ENVIRONMENT === 'development') {
+  // Fallback for local dev
+} else {
+  // Full R2 features
+}
+```
+
+## Presigned URL Expiry
+
+```typescript
+// ❌ WRONG: URL expires but no client validation
+const url = await getSignedUrl(s3, command, { expiresIn: 60 });
+// 61 seconds later: 403 Forbidden
+
+// ✅ CORRECT: Return expiry to client
+return Response.json({
+  uploadUrl: url,
+  expiresAt: new Date(Date.now() + 60000).toISOString()
+});
+```
+
+## Limits
+
+| Limit | Value |
+|-------|-------|
+| Object size | 5 TB |
+| Multipart part count | 10,000 |
+| Multipart part min size | 5 MB (except last) |
+| Batch delete | 1,000 keys |
+| List limit | 1,000 per request |
+| Key size | 1024 bytes |
+| Custom metadata | 2 KB per object |
+| Presigned URL max expiry | 7 days |
+
+## Common Errors
+
+### "Stream upload failed" / Silent Truncation
+
+**Cause:** Stream length unknown or Content-Length missing  
+**Solution:** Buffer data or pass explicit Content-Length
+
+### "Invalid credentials" / S3 SDK
+
+**Cause:** Missing `region: 'auto'` in S3Client config  
+**Solution:** Always set `region: 'auto'` for R2
+
+### "Object not found"
+
+**Cause:** Object key doesn't exist or was deleted  
+**Solution:** Verify object key correct, check if object was deleted, ensure bucket correct
+
+### "List compatibility error"
+
+**Cause:** Missing or old compatibility_date, or flag not enabled  
+**Solution:** Set `compatibility_date >= 2022-08-04` or enable `r2_list_honor_include` flag
+
+### "Multipart upload failed"
+
+**Cause:** Part sizes not uniform or incorrect part number  
+**Solution:** Ensure uniform size except final part, verify part numbers start at 1

package/skills/cloudflare-deploy/references/r2/patterns.md

@@ -0,0 +1,193 @@
+# R2 Patterns & Best Practices
+
+## Streaming Large Files
+
+```typescript
+const object = await env.MY_BUCKET.get(key);
+if (!object) return new Response('Not found', { status: 404 });
+
+const headers = new Headers();
+object.writeHttpMetadata(headers);
+headers.set('etag', object.httpEtag);
+
+return new Response(object.body, { headers });
+```
+
+## Conditional GET (304 Not Modified)
+
+```typescript
+const ifNoneMatch = request.headers.get('if-none-match');
+const object = await env.MY_BUCKET.get(key, {
+  onlyIf: { etagDoesNotMatch: ifNoneMatch?.replace(/"/g, '') || '' }
+});
+
+if (!object) return new Response('Not found', { status: 404 });
+if (!object.body) return new Response(null, { status: 304, headers: { 'etag': object.httpEtag } });
+
+return new Response(object.body, { headers: { 'etag': object.httpEtag } });
+```
+
+## Upload with Validation
+
+```typescript
+const key = url.pathname.slice(1);
+if (!key || key.includes('..')) return new Response('Invalid key', { status: 400 });
+
+const object = await env.MY_BUCKET.put(key, request.body, {
+  httpMetadata: { contentType: request.headers.get('content-type') || 'application/octet-stream' },
+  customMetadata: { uploadedAt: new Date().toISOString(), ip: request.headers.get('cf-connecting-ip') || 'unknown' }
+});
+
+return Response.json({ key: object.key, size: object.size, etag: object.httpEtag });
+```
+
+## Multipart with Progress
+
+```typescript
+const PART_SIZE = 5 * 1024 * 1024; // 5MB
+const partCount = Math.ceil(file.size / PART_SIZE);
+const multipart = await env.MY_BUCKET.createMultipartUpload(key, { httpMetadata: { contentType: file.type } });
+
+const uploadedParts: R2UploadedPart[] = [];
+try {
+  for (let i = 0; i < partCount; i++) {
+    const start = i * PART_SIZE;
+    const part = await multipart.uploadPart(i + 1, file.slice(start, start + PART_SIZE));
+    uploadedParts.push(part);
+    onProgress?.(Math.round(((i + 1) / partCount) * 100));
+  }
+  return await multipart.complete(uploadedParts);
+} catch (error) {
+  await multipart.abort();
+  throw error;
+}
+```
+
+## Batch Delete
+
+```typescript
+async function deletePrefix(prefix: string, env: Env) {
+  let cursor: string | undefined;
+  let truncated = true;
+
+  while (truncated) {
+    const listed = await env.MY_BUCKET.list({ prefix, limit: 1000, cursor });
+    if (listed.objects.length > 0) {
+      await env.MY_BUCKET.delete(listed.objects.map(o => o.key));
+    }
+    truncated = listed.truncated;
+    cursor = listed.cursor;
+  }
+}
+```
+
+## Checksum Validation & Storage Transitions
+
+```typescript
+// Upload with checksum
+const hash = await crypto.subtle.digest('SHA-256', data);
+await env.MY_BUCKET.put(key, data, { sha256: hash });
+
+// Transition storage class (requires S3 SDK)
+import { S3Client, CopyObjectCommand } from '@aws-sdk/client-s3';
+await s3.send(new CopyObjectCommand({
+  Bucket: 'my-bucket', Key: key,
+  CopySource: `/my-bucket/${key}`,
+  StorageClass: 'STANDARD_IA'
+}));
+```
+
+## Client-Side Uploads (Presigned URLs)
+
+```typescript
+import { S3Client } from '@aws-sdk/client-s3';
+import { getSignedUrl } from '@aws-sdk/s3-request-presigner';
+import { PutObjectCommand } from '@aws-sdk/client-s3';
+
+// Worker: Generate presigned upload URL
+const s3 = new S3Client({
+  region: 'auto',
+  endpoint: `https://${env.ACCOUNT_ID}.r2.cloudflarestorage.com`,
+  credentials: { accessKeyId: env.R2_ACCESS_KEY_ID, secretAccessKey: env.R2_SECRET_ACCESS_KEY }
+});
+
+const url = await getSignedUrl(s3, new PutObjectCommand({ Bucket: 'my-bucket', Key: key }), { expiresIn: 3600 });
+return Response.json({ uploadUrl: url });
+
+// Client: Upload directly
+const { uploadUrl } = await fetch('/api/upload-url').then(r => r.json());
+await fetch(uploadUrl, { method: 'PUT', body: file });
+```
+
+## Caching with Cache API
+
+```typescript
+export default {
+  async fetch(request: Request, env: Env, ctx: ExecutionContext): Promise<Response> {
+    const cache = caches.default;
+    const url = new URL(request.url);
+    const cacheKey = new Request(url.toString(), request);
+
+    // Check cache first
+    let response = await cache.match(cacheKey);
+    if (response) return response;
+
+    // Fetch from R2
+    const key = url.pathname.slice(1);
+    const object = await env.MY_BUCKET.get(key);
+    if (!object) return new Response('Not found', { status: 404 });
+
+    const headers = new Headers();
+    object.writeHttpMetadata(headers);
+    headers.set('etag', object.httpEtag);
+    headers.set('cache-control', 'public, max-age=31536000, immutable');
+
+    response = new Response(object.body, { headers });
+
+    // Cache for subsequent requests
+    ctx.waitUntil(cache.put(cacheKey, response.clone()));
+
+    return response;
+  }
+};
+```
+
+## Public Bucket with Custom Domain
+
+```typescript
+export default {
+  async fetch(request: Request, env: Env): Promise<Response> {
+    // CORS preflight
+    if (request.method === 'OPTIONS') {
+      return new Response(null, {
+        headers: {
+          'access-control-allow-origin': '*',
+          'access-control-allow-methods': 'GET, HEAD',
+          'access-control-max-age': '86400'
+        }
+      });
+    }
+
+    const key = new URL(request.url).pathname.slice(1);
+    if (!key) return Response.redirect('/index.html', 302);
+
+    const object = await env.MY_BUCKET.get(key);
+    if (!object) return new Response('Not found', { status: 404 });
+
+    const headers = new Headers();
+    object.writeHttpMetadata(headers);
+    headers.set('etag', object.httpEtag);
+    headers.set('access-control-allow-origin', '*');
+    headers.set('cache-control', 'public, max-age=31536000, immutable');
+
+    return new Response(object.body, { headers });
+  }
+};
+```
+
+## r2.dev Public URLs
+
+Enable r2.dev in dashboard for simple public access: `https://pub-${hashId}.r2.dev/${key}`  
+Or add custom domain via dashboard: `https://files.example.com/${key}`
+
+**Limitations:** No auth, bucket-level CORS, no cache override.

package/skills/cloudflare-deploy/references/r2-data-catalog/README.md

@@ -0,0 +1,149 @@
+# Cloudflare R2 Data Catalog Skill Reference
+
+Expert guidance for Cloudflare R2 Data Catalog - Apache Iceberg catalog built into R2 buckets.
+
+## Reading Order
+
+**New to R2 Data Catalog?** Start here:
+1. Read "What is R2 Data Catalog?" and "When to Use" below
+2. [configuration.md](configuration.md) - Enable catalog, create tokens
+3. [patterns.md](patterns.md) - PyIceberg setup and common patterns
+4. [api.md](api.md) - REST API reference as needed
+5. [gotchas.md](gotchas.md) - Troubleshooting when issues arise
+
+**Quick reference?** Jump to:
+- [Enable catalog on bucket](configuration.md#enable-catalog-on-bucket)
+- [PyIceberg connection pattern](patterns.md#pyiceberg-connection-pattern)
+- [Permission errors](gotchas.md#permission-errors)
+
+## What is R2 Data Catalog?
+
+R2 Data Catalog is a **managed Apache Iceberg REST catalog** built directly into R2 buckets. It provides:
+
+- **Apache Iceberg tables** - ACID transactions, schema evolution, time-travel queries
+- **Zero-egress costs** - Query from any cloud/region without data transfer fees
+- **Standard REST API** - Works with Spark, PyIceberg, Snowflake, Trino, DuckDB
+- **No infrastructure** - Fully managed, no catalog servers to run
+- **Public beta** - Available to all R2 subscribers, no extra cost beyond R2 storage
+
+### What is Apache Iceberg?
+
+Open table format for analytics datasets in object storage. Features:
+- **ACID transactions** - Safe concurrent reads/writes
+- **Metadata optimization** - Fast queries without full scans
+- **Schema evolution** - Add/rename/delete columns without rewrites
+- **Time-travel** - Query historical snapshots
+- **Partitioning** - Organize data for efficient queries
+
+## When to Use
+
+**Use R2 Data Catalog for:**
+- **Log analytics** - Store and query application/system logs
+- **Data lakes/warehouses** - Analytical datasets queried by multiple engines
+- **BI pipelines** - Aggregate data for dashboards and reports
+- **Multi-cloud analytics** - Share data across clouds without egress fees
+- **Time-series data** - Event streams, metrics, sensor data
+
+**Don't use for:**
+- **Transactional workloads** - Use D1 or external database instead
+- **Sub-second latency** - Iceberg optimized for batch/analytical queries
+- **Small datasets (<1GB)** - Setup overhead not worth it
+- **Unstructured data** - Store files directly in R2, not as Iceberg tables
+
+## Architecture
+
+```
+┌─────────────────────────────────────────────────┐
+│  Query Engines                                  │
+│  (PyIceberg, Spark, Trino, Snowflake, DuckDB)  │
+└────────────────┬────────────────────────────────┘
+                 │
+                 │ REST API (OAuth2 token)
+                 ▼
+┌─────────────────────────────────────────────────┐
+│  R2 Data Catalog (Managed Iceberg REST Catalog)│
+│  • Namespace/table metadata                     │
+│  • Transaction coordination                     │
+│  • Snapshot management                          │
+└────────────────┬────────────────────────────────┘
+                 │
+                 │ Vended credentials
+                 ▼
+┌─────────────────────────────────────────────────┐
+│  R2 Bucket Storage                              │
+│  • Parquet data files                           │
+│  • Metadata files                               │
+│  • Manifest files                               │
+└─────────────────────────────────────────────────┘
+```
+
+**Key concepts:**
+- **Catalog URI** - REST endpoint for catalog operations (e.g., `https://<account-id>.r2.cloudflarestorage.com/iceberg/<bucket>`)
+- **Warehouse** - Logical grouping of tables (typically same as bucket name)
+- **Namespace** - Schema/database containing tables (e.g., `logs`, `analytics`)
+- **Table** - Iceberg table with schema, data files, snapshots
+- **Vended credentials** - Temporary S3 credentials catalog provides for data access
+
+## Limits
+
+| Resource | Limit | Notes |
+|----------|-------|-------|
+| Namespaces per catalog | No hard limit | Organize tables logically |
+| Tables per namespace | <10,000 recommended | Performance degrades beyond this |
+| Files per table | <100,000 recommended | Run compaction regularly |
+| Snapshots per table | Configurable retention | Expire >7 days old |
+| Partitions per table | 100-1,000 optimal | Too many = slow metadata ops |
+| Table size | Same as R2 bucket | 10GB-10TB+ common |
+| API rate limits | Standard R2 API limits | Shared with R2 storage operations |
+| Target file size | 128-512 MB | After compaction |
+
+## Current Status
+
+**Public Beta** (as of Jan 2026)
+- Available to all R2 subscribers
+- No extra cost beyond standard R2 storage/operations
+- Production-ready, but breaking changes possible
+- Supports: namespaces, tables, snapshots, compaction, time-travel, table maintenance
+
+## Decision Tree: Is R2 Data Catalog Right For You?
+
+```
+Start → Need analytics on object storage data?
+         │
+         ├─ No → Use R2 directly for object storage
+         │
+         └─ Yes → Dataset >1GB with structured schema?
+                  │
+                  ├─ No → Too small, use R2 + ad-hoc queries
+                  │
+                  └─ Yes → Need ACID transactions or schema evolution?
+                           │
+                           ├─ No → Consider simpler solutions (Parquet on R2)
+                           │
+                           └─ Yes → Need multi-cloud/multi-tool access?
+                                    │
+                                    ├─ No → D1 or external DB may be simpler
+                                    │
+                                    └─ Yes → ✅ Use R2 Data Catalog
+```
+
+**Quick check:** If you answer "yes" to all:
+- Dataset >1GB and growing
+- Structured/tabular data (logs, events, metrics)
+- Multiple query tools or cloud environments
+- Need versioning, schema changes, or concurrent access
+
+→ R2 Data Catalog is a good fit.
+
+## In This Reference
+
+- **[configuration.md](configuration.md)** - Enable catalog, create API tokens, connect clients
+- **[api.md](api.md)** - REST endpoints, operations, maintenance
+- **[patterns.md](patterns.md)** - PyIceberg examples, common use cases
+- **[gotchas.md](gotchas.md)** - Troubleshooting, best practices, limitations
+
+## See Also
+
+- [Cloudflare R2 Data Catalog Docs](https://developers.cloudflare.com/r2/data-catalog/)
+- [Apache Iceberg Docs](https://iceberg.apache.org/)
+- [PyIceberg Docs](https://py.iceberg.apache.org/)