@mohanscodex/spectra-code 0.4.8 → 0.5.0

package/skills/cloudflare-deploy/references/pages-functions/configuration.md

@@ -0,0 +1,122 @@
+# Configuration
+
+## TypeScript Setup
+
+**Generate types from wrangler.jsonc** (replaces deprecated `@cloudflare/workers-types`):
+
+```bash
+npx wrangler types
+```
+
+Creates `worker-configuration.d.ts` with typed `Env` interface based on your bindings.
+
+```typescript
+// functions/api.ts
+export const onRequest: PagesFunction<Env> = async (ctx) => {
+  // ctx.env.KV, ctx.env.DB, etc. are fully typed
+  return Response.json({ ok: true });
+};
+```
+
+**Manual types** (if not using wrangler types):
+
+```typescript
+interface Env {
+  KV: KVNamespace;
+  DB: D1Database;
+  API_KEY: string;
+}
+export const onRequest: PagesFunction<Env> = async (ctx) => { /* ... */ };
+```
+
+## wrangler.jsonc
+
+```jsonc
+{
+  "$schema": "./node_modules/wrangler/config-schema.json",
+  "name": "my-pages-app",
+  "pages_build_output_dir": "./dist",
+  "compatibility_date": "2025-01-01",
+  "compatibility_flags": ["nodejs_compat"],
+  
+  "vars": { "API_URL": "https://api.example.com" },
+  "kv_namespaces": [{ "binding": "KV", "id": "abc123" }],
+  "d1_databases": [{ "binding": "DB", "database_name": "prod-db", "database_id": "xyz789" }],
+  "r2_buckets": [{ "binding": "BUCKET", "bucket_name": "my-bucket" }],
+  "durable_objects": { "bindings": [{ "name": "COUNTER", "class_name": "Counter", "script_name": "counter-worker" }] },
+  "services": [{ "binding": "AUTH", "service": "auth-worker" }],
+  "ai": { "binding": "AI" },
+  "vectorize": [{ "binding": "VECTORIZE", "index_name": "my-index" }],
+  "analytics_engine_datasets": [{ "binding": "ANALYTICS" }]
+}
+```
+
+## Environment Overrides
+
+Top-level → local dev, `env.preview` → preview, `env.production` → production
+
+```jsonc
+{
+  "vars": { "API_URL": "http://localhost:8787" },
+  "env": {
+    "production": { "vars": { "API_URL": "https://api.example.com" } }
+  }
+}
+```
+
+**Note:** If overriding `vars`, `kv_namespaces`, `d1_databases`, etc., ALL must be redefined (non-inheritable)
+
+## Local Secrets (.dev.vars)
+
+**Local dev only** - NOT deployed:
+
+```bash
+# .dev.vars (add to .gitignore)
+SECRET_KEY="my-secret-value"
+```
+
+Accessed via `ctx.env.SECRET_KEY`. Set production secrets:
+```bash
+echo "value" | npx wrangler pages secret put SECRET_KEY --project-name=my-app
+```
+
+## Static Config Files
+
+**_routes.json** - Custom routing:
+```json
+{ "version": 1, "include": ["/api/*"], "exclude": ["/static/*"] }
+```
+
+**_headers** - Static headers:
+```
+/static/*
+  Cache-Control: public, max-age=31536000
+```
+
+**_redirects** - Redirects:
+```
+/old  /new  301
+```
+
+## Local Dev & Deployment
+
+```bash
+# Dev server
+npx wrangler pages dev ./dist
+
+# With bindings
+npx wrangler pages dev ./dist --kv=KV --d1=DB=db-id --r2=BUCKET
+
+# Durable Objects (2 terminals)
+cd do-worker && npx wrangler dev
+cd pages-project && npx wrangler pages dev ./dist --do COUNTER=Counter@do-worker
+
+# Deploy
+npx wrangler pages deploy ./dist
+npx wrangler pages deploy ./dist --branch preview
+
+# Download config
+npx wrangler pages download config my-project
+```
+
+**See also:** [api.md](./api.md) for binding usage examples

package/skills/cloudflare-deploy/references/pages-functions/gotchas.md

@@ -0,0 +1,94 @@
+# Gotchas & Debugging
+
+## Error Diagnosis
+
+| Symptom | Likely Cause | Solution |
+|---------|--------------|----------|
+| **Function not invoking** | Wrong `/functions` location, wrong extension, or `_routes.json` excludes path | Check `pages_build_output_dir`, use `.js`/`.ts`, verify `_routes.json` |
+| **`ctx.env.BINDING` undefined** | Binding not configured or name mismatch | Add to `wrangler.jsonc`, verify exact name (case-sensitive), redeploy |
+| **TypeScript errors on `ctx.env`** | Missing type definition | Run `wrangler types` or define `interface Env {}` |
+| **Middleware not running** | Wrong filename/location or missing `ctx.next()` | Name exactly `_middleware.js`, export `onRequest`, call `ctx.next()` |
+| **Secrets missing in production** | `.dev.vars` not deployed | `.dev.vars` is local only - set production secrets via dashboard or `wrangler secret put` |
+| **Type mismatch on binding** | Wrong interface type | See [api.md](./api.md) bindings table for correct types |
+| **"KV key not found" but exists** | Key in wrong namespace or env | Verify namespace binding, check preview vs production env |
+| **Function times out** | Synchronous wait or missing `await` | All I/O must be async/await, use `ctx.waitUntil()` for background tasks |
+
+## Common Errors
+
+### TypeScript type errors
+
+**Problem:** `ctx.env.MY_BINDING` shows type error  
+**Cause:** No type definition for `Env`  
+**Solution:** Run `npx wrangler types` or manually define:
+```typescript
+interface Env { MY_BINDING: KVNamespace; }
+export const onRequest: PagesFunction<Env> = async (ctx) => { /* ... */ };
+```
+
+### Secrets not available in production
+
+**Problem:** `ctx.env.SECRET_KEY` is undefined in production  
+**Cause:** `.dev.vars` is local-only, not deployed  
+**Solution:** Set production secrets:
+```bash
+echo "value" | npx wrangler pages secret put SECRET_KEY --project-name=my-app
+```
+
+## Debugging
+
+```typescript
+// Console logging
+export async function onRequest(ctx) {
+  console.log('Request:', ctx.request.method, ctx.request.url);
+  const res = await ctx.next();
+  console.log('Status:', res.status);
+  return res;
+}
+```
+
+```bash
+# Stream real-time logs
+npx wrangler pages deployment tail
+npx wrangler pages deployment tail --status error
+```
+
+```jsonc
+// Source maps (wrangler.jsonc)
+{ "upload_source_maps": true }
+```
+
+## Limits
+
+| Resource | Free | Paid |
+|----------|------|------|
+| CPU time | 10ms | 50ms |
+| Memory | 128 MB | 128 MB |
+| Script size | 10 MB compressed | 10 MB compressed |
+| Env vars | 5 KB per var, 64 max | 5 KB per var, 64 max |
+| Requests | 100k/day | Unlimited ($0.50/million) |
+
+## Best Practices
+
+**Performance:** Minimize deps (cold start), use KV for cache/D1 for relational/R2 for large files, set `Cache-Control` headers, batch DB ops, handle errors gracefully
+
+**Security:** Never commit secrets (use `.dev.vars` + gitignore), validate input, sanitize before DB, implement auth middleware, set CORS headers, rate limit per-IP
+
+## Migration
+
+**Workers → Pages Functions:**
+- `export default { fetch(req, env) {} }` → `export function onRequest(ctx) { const { request, env } = ctx; }`
+- Use `_worker.js` for complex routing: `env.ASSETS.fetch(request)` for static files
+
+**Other platforms → Pages:**
+- File-based routing: `/functions/api/users.js` → `/api/users`
+- Dynamic routes: `[param]` not `:param`
+- Replace Node.js deps with Workers APIs or add `nodejs_compat` flag
+
+## Resources
+
+- [Official Docs](https://developers.cloudflare.com/pages/functions/)
+- [Workers APIs](https://developers.cloudflare.com/workers/runtime-apis/)
+- [Examples](https://github.com/cloudflare/pages-example-projects)
+- [Discord](https://discord.gg/cloudflaredev)
+
+**See also:** [configuration.md](./configuration.md) for TypeScript setup | [patterns.md](./patterns.md) for middleware/auth | [api.md](./api.md) for bindings

package/skills/cloudflare-deploy/references/pages-functions/patterns.md

@@ -0,0 +1,137 @@
+# Common Patterns
+
+## Background Tasks (waitUntil)
+
+Non-blocking tasks after response sent (analytics, cleanup, webhooks):
+
+```typescript
+export async function onRequest(ctx: EventContext<Env>) {
+  const res = Response.json({ success: true });
+  
+  ctx.waitUntil(ctx.env.KV.put('last-visit', new Date().toISOString()));
+  ctx.waitUntil(Promise.all([
+    ctx.env.ANALYTICS.writeDataPoint({ event: 'view' }),
+    fetch('https://webhook.site/...', { method: 'POST' })
+  ]));
+  
+  return res; // Returned immediately
+}
+```
+
+## Middleware & Auth
+
+```typescript
+// functions/_middleware.js (global) or functions/users/_middleware.js (scoped)
+export async function onRequest(ctx) {
+  try { return await ctx.next(); } 
+  catch (err) { return new Response(err.message, { status: 500 }); }
+}
+
+// Chained: export const onRequest = [errorHandler, auth, logger];
+
+// Auth
+async function auth(ctx: EventContext<Env>) {
+  const token = ctx.request.headers.get('authorization')?.replace('Bearer ', '');
+  if (!token) return new Response('Unauthorized', { status: 401 });
+  const session = await ctx.env.KV.get(`session:${token}`);
+  if (!session) return new Response('Invalid', { status: 401 });
+  ctx.data.user = JSON.parse(session);
+  return ctx.next();
+}
+```
+
+## CORS & Rate Limiting
+
+```typescript
+// CORS middleware
+const cors = { 'Access-Control-Allow-Origin': '*', 'Access-Control-Allow-Methods': 'GET, POST' };
+export async function onRequestOptions() { return new Response(null, { headers: cors }); }
+export async function onRequest(ctx) {
+  const res = await ctx.next();
+  Object.entries(cors).forEach(([k, v]) => res.headers.set(k, v));
+  return res;
+}
+
+// Rate limiting (KV-based)
+async function rateLimit(ctx: EventContext<Env>) {
+  const ip = ctx.request.headers.get('CF-Connecting-IP') || 'unknown';
+  const count = parseInt(await ctx.env.KV.get(`rate:${ip}`) || '0');
+  if (count >= 100) return new Response('Rate limited', { status: 429 });
+  await ctx.env.KV.put(`rate:${ip}`, (count + 1).toString(), { expirationTtl: 3600 });
+  return ctx.next();
+}
+```
+
+## Forms, Caching, Redirects
+
+```typescript
+// JSON & file upload
+export async function onRequestPost(ctx) {
+  const ct = ctx.request.headers.get('content-type') || '';
+  if (ct.includes('application/json')) return Response.json(await ctx.request.json());
+  if (ct.includes('multipart/form-data')) {
+    const file = (await ctx.request.formData()).get('file') as File;
+    await ctx.env.BUCKET.put(file.name, file.stream());
+    return Response.json({ uploaded: file.name });
+  }
+}
+
+// Cache API
+export async function onRequest(ctx) {
+  let res = await caches.default.match(ctx.request);
+  if (!res) {
+    res = new Response('Data');
+    res.headers.set('Cache-Control', 'public, max-age=3600');
+    ctx.waitUntil(caches.default.put(ctx.request, res.clone()));
+  }
+  return res;
+}
+
+// Redirects
+export async function onRequest(ctx) {
+  if (new URL(ctx.request.url).pathname === '/old') {
+    return Response.redirect(new URL('/new', ctx.request.url), 301);
+  }
+  return ctx.next();
+}
+```
+
+## Testing
+
+**Unit tests** (Vitest + cloudflare:test):
+```typescript
+import { env } from 'cloudflare:test';
+import { it, expect } from 'vitest';
+import { onRequest } from '../functions/api';
+
+it('returns JSON', async () => {
+  const req = new Request('http://localhost/api');
+  const ctx = { request: req, env, params: {}, data: {} } as EventContext;
+  const res = await onRequest(ctx);
+  expect(res.status).toBe(200);
+});
+```
+
+**Integration:** `wrangler pages dev` + Playwright/Cypress
+
+## Advanced Mode (_worker.js)
+
+Use `_worker.js` for complex routing (replaces `/functions`):
+
+```typescript
+interface Env { ASSETS: Fetcher; KV: KVNamespace; }
+
+export default {
+  async fetch(request: Request, env: Env): Promise<Response> {
+    const url = new URL(request.url);
+    if (url.pathname.startsWith('/api/')) {
+      return Response.json({ data: await env.KV.get('key') });
+    }
+    return env.ASSETS.fetch(request); // Static files
+  }
+} satisfies ExportedHandler<Env>;
+```
+
+**When:** Existing Worker, framework-generated (Next.js/SvelteKit), custom routing logic
+
+**See also:** [api.md](./api.md) for `env.ASSETS.fetch()` | [gotchas.md](./gotchas.md) for debugging

package/skills/cloudflare-deploy/references/pipelines/README.md

@@ -0,0 +1,105 @@
+# Cloudflare Pipelines
+
+ETL streaming platform for ingesting, transforming, and loading data into R2 with SQL transformations.
+
+## Overview
+
+Pipelines provides:
+- **Streams**: Durable event buffers (HTTP/Workers ingestion)
+- **Pipelines**: SQL-based transformations
+- **Sinks**: R2 destinations (Iceberg tables or Parquet/JSON files)
+
+**Status**: Open beta (Workers Paid plan)  
+**Pricing**: No charge beyond standard R2 storage/operations
+
+## Architecture
+
+```
+Data Sources → Streams → Pipelines (SQL) → Sinks → R2
+                 ↑          ↓                ↓
+            HTTP/Workers  Transform     Iceberg/Parquet
+```
+
+| Component | Purpose | Key Feature |
+|-----------|---------|-------------|
+| Streams | Event ingestion | Structured (validated) or unstructured |
+| Pipelines | Transform with SQL | Immutable after creation |
+| Sinks | Write to R2 | Exactly-once delivery |
+
+## Quick Start
+
+```bash
+# Interactive setup (recommended)
+npx wrangler pipelines setup
+```
+
+**Minimal Worker example:**
+```typescript
+interface Env {
+  STREAM: Pipeline;
+}
+
+export default {
+  async fetch(request: Request, env: Env, ctx: ExecutionContext): Promise<Response> {
+    const event = { user_id: "123", event_type: "purchase", amount: 29.99 };
+    
+    // Fire-and-forget pattern
+    ctx.waitUntil(env.STREAM.send([event]));
+    
+    return new Response('OK');
+  }
+} satisfies ExportedHandler<Env>;
+```
+
+## Which Sink Type?
+
+```
+Need SQL queries on data?
+  → R2 Data Catalog (Iceberg)
+    ✅ ACID transactions, time-travel, schema evolution
+    ❌ More setup complexity (namespace, table, catalog token)
+
+Just file storage/archival?
+  → R2 Storage (Parquet)
+    ✅ Simple, direct file access
+    ❌ No built-in SQL queries
+
+Using external tools (Spark/Athena)?
+  → R2 Storage (Parquet with partitioning)
+    ✅ Standard format, partition pruning for performance
+    ❌ Must manage schema compatibility yourself
+```
+
+## Common Use Cases
+
+- **Analytics pipelines**: Clickstream, telemetry, server logs
+- **Data warehousing**: ETL into queryable Iceberg tables
+- **Event processing**: Mobile/IoT with enrichment
+- **Ecommerce analytics**: User events, purchases, views
+
+## Reading Order
+
+**New to Pipelines?** Start here:
+1. [configuration.md](./configuration.md) - Setup streams, sinks, pipelines
+2. [api.md](./api.md) - Send events, TypeScript types, SQL functions
+3. [patterns.md](./patterns.md) - Best practices, integrations, complete example
+4. [gotchas.md](./gotchas.md) - Critical warnings, troubleshooting
+
+**Task-based routing:**
+- Setup pipeline → [configuration.md](./configuration.md)
+- Send/query data → [api.md](./api.md)
+- Implement pattern → [patterns.md](./patterns.md)
+- Debug issue → [gotchas.md](./gotchas.md)
+
+## In This Reference
+
+- [configuration.md](./configuration.md) - wrangler.jsonc bindings, schema definition, sink options, CLI commands
+- [api.md](./api.md) - Pipeline binding interface, send() method, HTTP ingest, SQL function reference
+- [patterns.md](./patterns.md) - Fire-and-forget, schema validation with Zod, integrations, performance tuning
+- [gotchas.md](./gotchas.md) - Silent validation failures, immutable pipelines, latency expectations, limits
+
+## See Also
+
+- [r2](../r2/) - R2 storage backend for sinks
+- [queues](../queues/) - Compare with Queues for async processing
+- [workers](../workers/) - Worker runtime for event ingestion

package/skills/cloudflare-deploy/references/pipelines/api.md

@@ -0,0 +1,208 @@
+# Pipelines API Reference
+
+## Pipeline Binding Interface
+
+```typescript
+// From @cloudflare/workers-types
+interface Pipeline {
+  send(data: object | object[]): Promise<void>;
+}
+
+interface Env {
+  STREAM: Pipeline;
+}
+
+export default {
+  async fetch(request: Request, env: Env): Promise<Response> {
+    // send() returns Promise<void> - no result data
+    await env.STREAM.send([event]);
+    return new Response('OK');
+  }
+} satisfies ExportedHandler<Env>;
+```
+
+**Key points:**
+- `send()` accepts single object or array
+- Always returns `Promise<void>` (no confirmation data)
+- Throws on network/validation errors (wrap in try/catch)
+- Use `ctx.waitUntil()` for fire-and-forget pattern
+
+## Writing Events
+
+### Single Event
+
+```typescript
+await env.STREAM.send([{
+  user_id: "12345",
+  event_type: "purchase",
+  product_id: "widget-001",
+  amount: 29.99
+}]);
+```
+
+### Batch Events
+
+```typescript
+const events = [
+  { user_id: "user1", event_type: "view" },
+  { user_id: "user2", event_type: "purchase", amount: 50 }
+];
+await env.STREAM.send(events);
+```
+
+**Limits:**
+- Max 1 MB per request
+- 5 MB/s per stream
+
+### Fire-and-Forget Pattern
+
+```typescript
+export default {
+  async fetch(request: Request, env: Env, ctx: ExecutionContext): Promise<Response> {
+    const event = { /* ... */ };
+    
+    // Don't block response on send
+    ctx.waitUntil(env.STREAM.send([event]));
+    
+    return new Response('OK');
+  }
+};
+```
+
+### Error Handling
+
+```typescript
+try {
+  await env.STREAM.send([event]);
+} catch (error) {
+  console.error('Pipeline send failed:', error);
+  // Log to another system, retry, or return error response
+  return new Response('Failed to track event', { status: 500 });
+}
+```
+
+## HTTP Ingest API
+
+### Endpoint Format
+
+```
+https://{stream-id}.ingest.cloudflare.com
+```
+
+Get `{stream-id}` from: `npx wrangler pipelines streams list`
+
+### Request Format
+
+**CRITICAL:** Must send array, not single object
+
+```bash
+# ✅ Correct
+curl -X POST https://{stream-id}.ingest.cloudflare.com \
+  -H "Content-Type: application/json" \
+  -d '[{"user_id": "123", "event_type": "purchase"}]'
+
+# ❌ Wrong - will fail
+curl -X POST https://{stream-id}.ingest.cloudflare.com \
+  -H "Content-Type: application/json" \
+  -d '{"user_id": "123", "event_type": "purchase"}'
+```
+
+### Authentication
+
+```bash
+curl -X POST https://{stream-id}.ingest.cloudflare.com \
+  -H "Content-Type: application/json" \
+  -H "Authorization: Bearer YOUR_API_TOKEN" \
+  -d '[{"event": "data"}]'
+```
+
+**Required permission:** Workers Pipeline Send
+
+Create token: Dashboard → Workers → API tokens → Create with Pipeline Send permission
+
+### Response Codes
+
+| Code | Meaning | Action |
+|------|---------|--------|
+| 200 | Accepted | Success |
+| 400 | Invalid format | Check JSON array, schema match |
+| 401 | Auth failed | Verify token valid |
+| 413 | Payload too large | Split into smaller batches (<1 MB) |
+| 429 | Rate limited | Back off, retry with delay |
+| 5xx | Server error | Retry with exponential backoff |
+
+## SQL Functions Quick Reference
+
+Available in `INSERT INTO sink SELECT ... FROM stream` transformations:
+
+| Function | Example | Use Case |
+|----------|---------|----------|
+| `UPPER(s)` | `UPPER(event_type)` | Normalize strings |
+| `LOWER(s)` | `LOWER(email)` | Case-insensitive matching |
+| `CONCAT(...)` | `CONCAT(user_id, '_', product_id)` | Generate composite keys |
+| `CASE WHEN ... THEN ... END` | `CASE WHEN amount > 100 THEN 'high' ELSE 'low' END` | Conditional enrichment |
+| `CAST(x AS type)` | `CAST(timestamp AS string)` | Type conversion |
+| `COALESCE(x, y)` | `COALESCE(amount, 0.0)` | Default values |
+| Math operators | `amount * 1.1`, `price / quantity` | Calculations |
+| Comparison | `amount > 100`, `status IN ('active', 'pending')` | Filtering |
+
+**String types for CAST:** `string`, `int32`, `int64`, `float32`, `float64`, `bool`, `timestamp`
+
+Full reference: [Pipelines SQL Reference](https://developers.cloudflare.com/pipelines/sql-reference/)
+
+## SQL Transform Examples
+
+### Filter Events
+
+```sql
+INSERT INTO my_sink
+SELECT * FROM my_stream
+WHERE event_type = 'purchase' AND amount > 100
+```
+
+### Select Specific Fields
+
+```sql
+INSERT INTO my_sink
+SELECT user_id, event_type, timestamp, amount
+FROM my_stream
+```
+
+### Transform and Enrich
+
+```sql
+INSERT INTO my_sink
+SELECT
+  user_id,
+  UPPER(event_type) as event_type,
+  timestamp,
+  amount * 1.1 as amount_with_tax,
+  CONCAT(user_id, '_', product_id) as unique_key,
+  CASE
+    WHEN amount > 1000 THEN 'high_value'
+    WHEN amount > 100 THEN 'medium_value'
+    ELSE 'low_value'
+  END as customer_tier
+FROM my_stream
+WHERE event_type IN ('purchase', 'refund')
+```
+
+## Querying Results (R2 Data Catalog)
+
+```bash
+export WRANGLER_R2_SQL_AUTH_TOKEN=YOUR_CATALOG_TOKEN
+
+npx wrangler r2 sql query "warehouse_name" "
+SELECT 
+  event_type,
+  COUNT(*) as event_count,
+  SUM(amount) as total_revenue
+FROM default.my_table
+WHERE event_type = 'purchase'
+  AND timestamp >= '2025-01-01'
+GROUP BY event_type
+ORDER BY total_revenue DESC
+LIMIT 100"
+```
+
+**Note:** Iceberg tables support standard SQL queries with GROUP BY, JOINs, WHERE, ORDER BY, etc.