@mohanscodex/spectra-code 0.4.8 → 0.5.0

package/skills/cloudflare-deploy/references/pipelines/configuration.md

@@ -0,0 +1,98 @@
+# Pipelines Configuration
+
+## Worker Binding
+
+```jsonc
+// wrangler.jsonc
+{
+  "pipelines": [
+    { "pipeline": "<STREAM_ID>", "binding": "STREAM" }
+  ]
+}
+```
+
+Get stream ID: `npx wrangler pipelines streams list`
+
+## Schema (Structured Streams)
+
+```json
+{
+  "fields": [
+    { "name": "user_id", "type": "string", "required": true },
+    { "name": "event_type", "type": "string", "required": true },
+    { "name": "amount", "type": "float64", "required": false },
+    { "name": "timestamp", "type": "timestamp", "required": true }
+  ]
+}
+```
+
+**Types:** `string`, `int32`, `int64`, `float32`, `float64`, `bool`, `timestamp`, `json`, `binary`, `list`, `struct`
+
+## Stream Setup
+
+```bash
+# With schema
+npx wrangler pipelines streams create my-stream --schema-file schema.json
+
+# Unstructured (no validation)
+npx wrangler pipelines streams create my-stream
+
+# List/get/delete
+npx wrangler pipelines streams list
+npx wrangler pipelines streams get <ID>
+npx wrangler pipelines streams delete <ID>
+```
+
+## Sink Configuration
+
+**R2 Data Catalog (Iceberg):**
+```bash
+npx wrangler pipelines sinks create my-sink \
+  --type r2-data-catalog \
+  --bucket my-bucket --namespace default --table events \
+  --catalog-token $TOKEN \
+  --compression zstd --roll-interval 60
+```
+
+**R2 Raw (Parquet):**
+```bash
+npx wrangler pipelines sinks create my-sink \
+  --type r2 --bucket my-bucket --format parquet \
+  --path analytics/events \
+  --partitioning "year=%Y/month=%m/day=%d" \
+  --access-key-id $KEY --secret-access-key $SECRET
+```
+
+| Option | Values | Guidance |
+|--------|--------|----------|
+| `--compression` | `zstd`, `snappy`, `gzip` | `zstd` best ratio, `snappy` fastest |
+| `--roll-interval` | Seconds | Low latency: 10-60, Query perf: 300 |
+| `--roll-size` | MB | Larger = better compression |
+
+## Pipeline Creation
+
+```bash
+npx wrangler pipelines create my-pipeline \
+  --sql "INSERT INTO my_sink SELECT * FROM my_stream WHERE event_type = 'purchase'"
+```
+
+**⚠️ Pipelines are immutable** - cannot modify SQL. Must delete/recreate.
+
+## Credentials
+
+| Type | Permission | Get From |
+|------|------------|----------|
+| Catalog token | R2 Admin Read & Write | Dashboard → R2 → API tokens |
+| R2 credentials | Object Read & Write | `wrangler r2 bucket create` output |
+| HTTP ingest token | Workers Pipeline Send | Dashboard → Workers → API tokens |
+
+## Complete Example
+
+```bash
+npx wrangler r2 bucket create my-bucket
+npx wrangler r2 bucket catalog enable my-bucket
+npx wrangler pipelines streams create my-stream --schema-file schema.json
+npx wrangler pipelines sinks create my-sink --type r2-data-catalog --bucket my-bucket ...
+npx wrangler pipelines create my-pipeline --sql "INSERT INTO my_sink SELECT * FROM my_stream"
+npx wrangler deploy
+```

package/skills/cloudflare-deploy/references/pipelines/gotchas.md

@@ -0,0 +1,80 @@
+# Pipelines Gotchas
+
+## Critical Issues
+
+### Events Silently Dropped
+
+**Most common issue.** Events accepted (HTTP 200) but never appear in sink.
+
+**Causes:**
+1. Schema validation fails - structured streams drop invalid events silently
+2. Waiting for roll interval (10-300s) - expected behavior
+
+**Solution:** Validate client-side with Zod:
+```typescript
+const EventSchema = z.object({ user_id: z.string(), amount: z.number() });
+try {
+  const validated = EventSchema.parse(rawEvent);
+  await env.STREAM.send([validated]);
+} catch (e) { /* get immediate feedback */ }
+```
+
+### Pipelines Are Immutable
+
+Cannot modify SQL after creation. Must delete and recreate.
+
+```bash
+npx wrangler pipelines delete old-pipeline
+npx wrangler pipelines create new-pipeline --sql "..."
+```
+
+**Tip:** Use version naming (`events-pipeline-v1`) and keep SQL in version control.
+
+### Worker Binding Not Found
+
+**`env.STREAM is undefined`**
+
+1. Use **stream ID** (not pipeline ID) in `wrangler.jsonc`
+2. Redeploy after adding binding
+
+```bash
+npx wrangler pipelines streams list  # Get stream ID
+npx wrangler deploy
+```
+
+## Common Errors
+
+| Error | Cause | Fix |
+|-------|-------|-----|
+| Events not in R2 | Roll interval not elapsed | Wait 10-300s, check `roll_interval` |
+| Schema validation failures | Type mismatch, missing fields | Validate client-side |
+| Rate limit (429) | >5 MB/s per stream | Batch events, request increase |
+| Payload too large (413) | >1 MB request | Split into smaller batches |
+| Cannot delete stream | Pipeline references it | Delete pipelines first |
+| Sink credential errors | Token expired | Recreate sink with new credentials |
+
+## Limits (Open Beta)
+
+| Resource | Limit |
+|----------|-------|
+| Streams/Sinks/Pipelines per account | 20 each |
+| Payload size | 1 MB |
+| Ingest rate per stream | 5 MB/s |
+| Event retention | 24 hours |
+| Recommended batch size | 100 events |
+
+## SQL Limitations
+
+- **No JOINs** - single stream per pipeline
+- **No window functions** - basic SQL only
+- **No subqueries** - must use `INSERT INTO ... SELECT ... FROM`
+- **No schema evolution** - cannot modify after creation
+
+## Debug Checklist
+
+- [ ] Stream exists: `npx wrangler pipelines streams list`
+- [ ] Pipeline healthy: `npx wrangler pipelines get <ID>`
+- [ ] SQL syntax matches schema
+- [ ] Worker redeployed after binding added
+- [ ] Waited for roll interval
+- [ ] Accepted vs processed count matches (no validation drops)

package/skills/cloudflare-deploy/references/pipelines/patterns.md

@@ -0,0 +1,87 @@
+# Pipelines Patterns
+
+## Fire-and-Forget
+
+```typescript
+export default {
+  async fetch(request, env, ctx) {
+    const event = { user_id: '...', event_type: 'page_view', timestamp: new Date().toISOString() };
+    ctx.waitUntil(env.STREAM.send([event])); // Don't block response
+    return new Response('OK');
+  }
+};
+```
+
+## Schema Validation with Zod
+
+```typescript
+import { z } from 'zod';
+
+const EventSchema = z.object({
+  user_id: z.string(),
+  event_type: z.enum(['purchase', 'view']),
+  amount: z.number().positive().optional()
+});
+
+const validated = EventSchema.parse(rawEvent); // Throws on invalid
+await env.STREAM.send([validated]);
+```
+
+**Why:** Structured streams drop invalid events silently. Client validation gives immediate feedback.
+
+## SQL Transform Patterns
+
+```sql
+-- Filter early (reduce storage)
+INSERT INTO my_sink
+SELECT user_id, event_type, amount
+FROM my_stream
+WHERE event_type = 'purchase' AND amount > 10
+
+-- Select only needed fields
+INSERT INTO my_sink
+SELECT user_id, event_type, timestamp FROM my_stream
+
+-- Enrich with CASE
+INSERT INTO my_sink
+SELECT user_id, amount,
+  CASE WHEN amount > 1000 THEN 'vip' ELSE 'standard' END as tier
+FROM my_stream
+```
+
+## Pipelines + Queues Fan-out
+
+```typescript
+await Promise.all([
+  env.ANALYTICS_STREAM.send([event]),  // Long-term storage
+  env.PROCESS_QUEUE.send(event)        // Immediate processing
+]);
+```
+
+| Need | Use |
+|------|-----|
+| Long-term storage, SQL queries | Pipelines |
+| Immediate processing, retries | Queues |
+| Both | Fan-out pattern |
+
+## Performance Tuning
+
+| Goal | Config |
+|------|--------|
+| Low latency | `--roll-interval 10` |
+| Query performance | `--roll-interval 300 --roll-size 100` |
+| Cost optimal | `--compression zstd --roll-interval 300` |
+
+## Schema Evolution
+
+Pipelines are immutable. Use versioning:
+
+```bash
+# Create v2 stream/sink/pipeline
+npx wrangler pipelines streams create events-v2 --schema-file v2.json
+
+# Dual-write during transition
+await Promise.all([env.EVENTS_V1.send([event]), env.EVENTS_V2.send([event])]);
+
+# Query across versions with UNION ALL
+```

package/skills/cloudflare-deploy/references/pulumi/README.md

@@ -0,0 +1,100 @@
+# Cloudflare Pulumi Provider
+
+Expert guidance for Cloudflare Pulumi Provider (@pulumi/cloudflare).
+
+## Overview
+
+Programmatic management of Cloudflare resources: Workers, Pages, D1, KV, R2, DNS, Queues, etc.
+
+**Packages:**
+- TypeScript/JS: `@pulumi/cloudflare`
+- Python: `pulumi-cloudflare`
+- Go: `github.com/pulumi/pulumi-cloudflare/sdk/v6/go/cloudflare`
+- .NET: `Pulumi.Cloudflare`
+
+**Version:** v6.x
+
+## Core Principles
+
+1. Use API tokens (not legacy API keys)
+2. Store accountId in stack config
+3. Match binding names across code/config
+4. Use `module: true` for ES modules
+5. Set `compatibilityDate` to lock behavior
+
+## Authentication
+
+```typescript
+import * as cloudflare from "@pulumi/cloudflare";
+
+// API Token (recommended): CLOUDFLARE_API_TOKEN env
+const provider = new cloudflare.Provider("cf", { apiToken: process.env.CLOUDFLARE_API_TOKEN });
+
+// API Key (legacy): CLOUDFLARE_API_KEY + CLOUDFLARE_EMAIL env
+const provider = new cloudflare.Provider("cf", { apiKey: process.env.CLOUDFLARE_API_KEY, email: process.env.CLOUDFLARE_EMAIL });
+
+// API User Service Key: CLOUDFLARE_API_USER_SERVICE_KEY env
+const provider = new cloudflare.Provider("cf", { apiUserServiceKey: process.env.CLOUDFLARE_API_USER_SERVICE_KEY });
+```
+
+## Setup
+
+**Pulumi.yaml:**
+```yaml
+name: my-cloudflare-app
+runtime: nodejs
+config:
+  cloudflare:apiToken:
+    value: ${CLOUDFLARE_API_TOKEN}
+```
+
+**Pulumi.<stack>.yaml:**
+```yaml
+config:
+  cloudflare:accountId: "abc123..."
+```
+
+**index.ts:**
+```typescript
+import * as pulumi from "@pulumi/pulumi";
+import * as cloudflare from "@pulumi/cloudflare";
+const accountId = new pulumi.Config("cloudflare").require("accountId");
+```
+
+## Common Resource Types
+- `Provider` - Provider config
+- `WorkerScript` - Worker
+- `WorkersKvNamespace` - KV
+- `R2Bucket` - R2
+- `D1Database` - D1
+- `Queue` - Queue
+- `PagesProject` - Pages
+- `DnsRecord` - DNS
+- `WorkerRoute` - Worker route
+- `WorkersDomain` - Custom domain
+
+## Key Properties
+- `accountId` - Required for most resources
+- `zoneId` - Required for DNS/domain
+- `name`/`title` - Resource identifier
+- `*Bindings` - Connect resources to Workers
+
+## Reading Order
+
+| Order | File | What | When to Read |
+|-------|------|------|--------------|
+| 1 | [configuration.md](./configuration.md) | Resource config for Workers/KV/D1/R2/Queues/Pages | First time setup, resource reference |
+| 2 | [patterns.md](./patterns.md) | Architecture patterns, multi-env, component resources | Building complex apps, best practices |
+| 3 | [api.md](./api.md) | Outputs, dependencies, imports, dynamic providers | Advanced features, integrations |
+| 4 | [gotchas.md](./gotchas.md) | Common errors, troubleshooting, limits | Debugging, deployment issues |
+
+## In This Reference
+- [configuration.md](./configuration.md) - Provider config, stack setup, Workers/bindings
+- [api.md](./api.md) - Resource types, Workers script, KV/D1/R2/queues/Pages
+- [patterns.md](./patterns.md) - Multi-env, secrets, CI/CD, stack management
+- [gotchas.md](./gotchas.md) - State issues, deployment failures, limits
+
+## See Also
+- [terraform](../terraform/) - Alternative IaC for Cloudflare
+- [wrangler](../wrangler/) - CLI deployment alternative
+- [workers](../workers/) - Worker runtime documentation

package/skills/cloudflare-deploy/references/pulumi/api.md

@@ -0,0 +1,200 @@
+# API & Data Sources
+
+## Outputs and Exports
+
+Export resource identifiers:
+
+```typescript
+export const kvId = kv.id;
+export const bucketName = bucket.name;
+export const workerUrl = worker.subdomain;
+export const dbId = db.id;
+```
+
+## Resource Dependencies
+
+Implicit dependencies via outputs:
+
+```typescript
+const kv = new cloudflare.WorkersKvNamespace("kv", {
+    accountId: accountId,
+    title: "my-kv",
+});
+
+// Worker depends on KV (implicit via kv.id)
+const worker = new cloudflare.WorkerScript("worker", {
+    accountId: accountId,
+    name: "my-worker",
+    content: code,
+    kvNamespaceBindings: [{name: "MY_KV", namespaceId: kv.id}], // Creates dependency
+});
+```
+
+Explicit dependencies:
+
+```typescript
+const migration = new command.local.Command("migration", {
+    create: pulumi.interpolate`wrangler d1 execute ${db.name} --file ./schema.sql`,
+}, {dependsOn: [db]});
+
+const worker = new cloudflare.WorkerScript("worker", {
+    accountId: accountId,
+    name: "worker",
+    content: code,
+    d1DatabaseBindings: [{name: "DB", databaseId: db.id}],
+}, {dependsOn: [migration]}); // Ensure migrations run first
+```
+
+## Using Outputs with API Calls
+
+```typescript
+const db = new cloudflare.D1Database("db", {accountId, name: "my-db"});
+
+db.id.apply(async (dbId) => {
+    const response = await fetch(
+        `https://api.cloudflare.com/client/v4/accounts/${accountId}/d1/database/${dbId}/query`,
+        {method: "POST", headers: {"Authorization": `Bearer ${apiToken}`, "Content-Type": "application/json"},
+         body: JSON.stringify({sql: "CREATE TABLE users (id INT)"})}
+    );
+    return response.json();
+});
+```
+
+## Custom Dynamic Providers
+
+For resources not in provider:
+
+```typescript
+import * as pulumi from "@pulumi/pulumi";
+
+class D1MigrationProvider implements pulumi.dynamic.ResourceProvider {
+    async create(inputs: any): Promise<pulumi.dynamic.CreateResult> {
+        const response = await fetch(
+            `https://api.cloudflare.com/client/v4/accounts/${inputs.accountId}/d1/database/${inputs.databaseId}/query`,
+            {method: "POST", headers: {"Authorization": `Bearer ${inputs.apiToken}`, "Content-Type": "application/json"},
+             body: JSON.stringify({sql: inputs.sql})}
+        );
+        return {id: `${inputs.databaseId}-${Date.now()}`, outs: await response.json()};
+    }
+    async update(id: string, olds: any, news: any): Promise<pulumi.dynamic.UpdateResult> {
+        if (olds.sql !== news.sql) await this.create(news);
+        return {};
+    }
+    async delete(id: string, props: any): Promise<void> {}
+}
+
+class D1Migration extends pulumi.dynamic.Resource {
+    constructor(name: string, args: any, opts?: pulumi.CustomResourceOptions) {
+        super(new D1MigrationProvider(), name, args, opts);
+    }
+}
+
+const migration = new D1Migration("migration", {
+    accountId, databaseId: db.id, apiToken, sql: "CREATE TABLE users (id INT)",
+}, {dependsOn: [db]});
+```
+
+## Data Sources
+
+**Get Zone:**
+```typescript
+const zone = cloudflare.getZone({name: "example.com"});
+const zoneId = zone.then(z => z.id);
+```
+
+**Get Accounts (via API):**
+Use Cloudflare API directly or custom dynamic resources.
+
+## Import Existing Resources
+
+```bash
+# Import worker
+pulumi import cloudflare:index/workerScript:WorkerScript my-worker <account_id>/<worker_name>
+
+# Import KV namespace
+pulumi import cloudflare:index/workersKvNamespace:WorkersKvNamespace my-kv <namespace_id>
+
+# Import R2 bucket
+pulumi import cloudflare:index/r2Bucket:R2Bucket my-bucket <account_id>/<bucket_name>
+
+# Import D1 database
+pulumi import cloudflare:index/d1Database:D1Database my-db <account_id>/<database_id>
+
+# Import DNS record
+pulumi import cloudflare:index/dnsRecord:DnsRecord my-record <zone_id>/<record_id>
+```
+
+## Secrets Management
+
+```typescript
+import * as pulumi from "@pulumi/pulumi";
+
+const config = new pulumi.Config();
+const apiKey = config.requireSecret("apiKey"); // Encrypted in state
+
+const worker = new cloudflare.WorkerScript("worker", {
+    accountId: accountId,
+    name: "my-worker",
+    content: code,
+    secretTextBindings: [{name: "API_KEY", text: apiKey}],
+});
+```
+
+Store secrets:
+```bash
+pulumi config set --secret apiKey "secret-value"
+```
+
+## Transform Pattern
+
+Modify resource args before creation:
+
+```typescript
+import {Transform} from "@pulumi/pulumi";
+
+interface BucketArgs {
+    accountId: pulumi.Input<string>;
+    transform?: {bucket?: Transform<cloudflare.R2BucketArgs>};
+}
+
+function createBucket(name: string, args: BucketArgs) {
+    const bucketArgs: cloudflare.R2BucketArgs = {
+        accountId: args.accountId,
+        name: name,
+        location: "auto",
+    };
+    const finalArgs = args.transform?.bucket?.(bucketArgs) ?? bucketArgs;
+    return new cloudflare.R2Bucket(name, finalArgs);
+}
+```
+
+## v6.x Worker Versioning Resources
+
+**Worker** - Container for versions:
+```typescript
+const worker = new cloudflare.Worker("api", {accountId, name: "api-worker"});
+export const workerId = worker.id;
+```
+
+**WorkerVersion** - Immutable code + config:
+```typescript
+const version = new cloudflare.WorkerVersion("v1", {
+    accountId, workerId: worker.id,
+    content: fs.readFileSync("./dist/worker.js", "utf8"),
+    compatibilityDate: "2025-01-01",
+});
+export const versionId = version.id;
+```
+
+**WorkersDeployment** - Active deployment with bindings:
+```typescript
+const deployment = new cloudflare.WorkersDeployment("prod", {
+    accountId, workerId: worker.id, versionId: version.id,
+    kvNamespaceBindings: [{name: "MY_KV", namespaceId: kv.id}],
+});
+```
+
+**Use:** Advanced deployments (canary, blue-green). Most apps should use `WorkerScript` (auto-versioning).
+
+---
+See: [README.md](./README.md), [configuration.md](./configuration.md), [patterns.md](./patterns.md), [gotchas.md](./gotchas.md)