devstack 0.1.0

package/vendor/skills/skills/cloudflare/references/pipelines/configuration.md

@@ -0,0 +1,98 @@
+# Pipelines Configuration
+
+## Worker Binding
+
+```jsonc
+// wrangler.jsonc
+{
+  "pipelines": [{ "pipeline": "<STREAM_ID>", "binding": "STREAM" }],
+}
+```
+
+Get stream ID: `npx wrangler pipelines streams list`
+
+## Schema (Structured Streams)
+
+```json
+{
+  "fields": [
+    { "name": "user_id", "type": "string", "required": true },
+    { "name": "event_type", "type": "string", "required": true },
+    { "name": "amount", "type": "float64", "required": false },
+    { "name": "timestamp", "type": "timestamp", "required": true }
+  ]
+}
+```
+
+**Types:** `string`, `int32`, `int64`, `float32`, `float64`, `bool`, `timestamp`, `json`, `binary`, `list`, `struct`
+
+## Stream Setup
+
+```bash
+# With schema
+npx wrangler pipelines streams create my-stream --schema-file schema.json
+
+# Unstructured (no validation)
+npx wrangler pipelines streams create my-stream
+
+# List/get/delete
+npx wrangler pipelines streams list
+npx wrangler pipelines streams get <ID>
+npx wrangler pipelines streams delete <ID>
+```
+
+## Sink Configuration
+
+**R2 Data Catalog (Iceberg):**
+
+```bash
+npx wrangler pipelines sinks create my-sink \
+  --type r2-data-catalog \
+  --bucket my-bucket --namespace default --table events \
+  --catalog-token $TOKEN \
+  --compression zstd --roll-interval 60
+```
+
+**R2 Raw (Parquet):**
+
+```bash
+npx wrangler pipelines sinks create my-sink \
+  --type r2 --bucket my-bucket --format parquet \
+  --path analytics/events \
+  --partitioning "year=%Y/month=%m/day=%d" \
+  --access-key-id $KEY --secret-access-key $SECRET
+```
+
+| Option            | Values                   | Guidance                            |
+| ----------------- | ------------------------ | ----------------------------------- |
+| `--compression`   | `zstd`, `snappy`, `gzip` | `zstd` best ratio, `snappy` fastest |
+| `--roll-interval` | Seconds                  | Low latency: 10-60, Query perf: 300 |
+| `--roll-size`     | MB                       | Larger = better compression         |
+
+## Pipeline Creation
+
+```bash
+npx wrangler pipelines create my-pipeline \
+  --sql "INSERT INTO my_sink SELECT * FROM my_stream WHERE event_type = 'purchase'"
+```
+
+**⚠️ Pipelines are immutable** - cannot modify SQL. Must delete/recreate.
+
+## Credentials
+
+| Type              | Permission            | Get From                           |
+| ----------------- | --------------------- | ---------------------------------- |
+| Catalog token     | R2 Admin Read & Write | Dashboard → R2 → API tokens        |
+| R2 credentials    | Object Read & Write   | `wrangler r2 bucket create` output |
+| HTTP ingest token | Workers Pipeline Send | Dashboard → Workers → API tokens   |
+
+## Complete Example
+
+```bash
+npx wrangler r2 bucket create my-bucket
+npx wrangler r2 bucket catalog enable my-bucket
+npx wrangler pipelines streams create my-stream --schema-file schema.json
+npx wrangler pipelines sinks create my-sink --type r2-data-catalog --bucket my-bucket ...
+npx wrangler pipelines create my-pipeline --sql "INSERT INTO my_sink SELECT * FROM my_stream"
+npx wrangler deploy
+```

package/vendor/skills/skills/cloudflare/references/pipelines/gotchas.md

@@ -0,0 +1,84 @@
+# Pipelines Gotchas
+
+## Critical Issues
+
+### Events Silently Dropped
+
+**Most common issue.** Events accepted (HTTP 200) but never appear in sink.
+
+**Causes:**
+
+1. Schema validation fails - structured streams drop invalid events silently
+2. Waiting for roll interval (10-300s) - expected behavior
+
+**Solution:** Validate client-side with Zod:
+
+```typescript
+const EventSchema = z.object({ user_id: z.string(), amount: z.number() });
+try {
+  const validated = EventSchema.parse(rawEvent);
+  await env.STREAM.send([validated]);
+} catch (e) {
+  /* get immediate feedback */
+}
+```
+
+### Pipelines Are Immutable
+
+Cannot modify SQL after creation. Must delete and recreate.
+
+```bash
+npx wrangler pipelines delete old-pipeline
+npx wrangler pipelines create new-pipeline --sql "..."
+```
+
+**Tip:** Use version naming (`events-pipeline-v1`) and keep SQL in version control.
+
+### Worker Binding Not Found
+
+**`env.STREAM is undefined`**
+
+1. Use **stream ID** (not pipeline ID) in `wrangler.jsonc`
+2. Redeploy after adding binding
+
+```bash
+npx wrangler pipelines streams list  # Get stream ID
+npx wrangler deploy
+```
+
+## Common Errors
+
+| Error                      | Cause                         | Fix                                 |
+| -------------------------- | ----------------------------- | ----------------------------------- |
+| Events not in R2           | Roll interval not elapsed     | Wait 10-300s, check `roll_interval` |
+| Schema validation failures | Type mismatch, missing fields | Validate client-side                |
+| Rate limit (429)           | >5 MB/s per stream            | Batch events, request increase      |
+| Payload too large (413)    | >1 MB request                 | Split into smaller batches          |
+| Cannot delete stream       | Pipeline references it        | Delete pipelines first              |
+| Sink credential errors     | Token expired                 | Recreate sink with new credentials  |
+
+## Limits (Open Beta)
+
+| Resource                            | Limit      |
+| ----------------------------------- | ---------- |
+| Streams/Sinks/Pipelines per account | 20 each    |
+| Payload size                        | 1 MB       |
+| Ingest rate per stream              | 5 MB/s     |
+| Event retention                     | 24 hours   |
+| Recommended batch size              | 100 events |
+
+## SQL Limitations
+
+- **No JOINs** - single stream per pipeline
+- **No window functions** - basic SQL only
+- **No subqueries** - must use `INSERT INTO ... SELECT ... FROM`
+- **No schema evolution** - cannot modify after creation
+
+## Debug Checklist
+
+- [ ] Stream exists: `npx wrangler pipelines streams list`
+- [ ] Pipeline healthy: `npx wrangler pipelines get <ID>`
+- [ ] SQL syntax matches schema
+- [ ] Worker redeployed after binding added
+- [ ] Waited for roll interval
+- [ ] Accepted vs processed count matches (no validation drops)

package/vendor/skills/skills/cloudflare/references/pipelines/patterns.md

@@ -0,0 +1,87 @@
+# Pipelines Patterns
+
+## Fire-and-Forget
+
+```typescript
+export default {
+  async fetch(request, env, ctx) {
+    const event = { user_id: "...", event_type: "page_view", timestamp: new Date().toISOString() };
+    ctx.waitUntil(env.STREAM.send([event])); // Don't block response
+    return new Response("OK");
+  },
+};
+```
+
+## Schema Validation with Zod
+
+```typescript
+import { z } from "zod";
+
+const EventSchema = z.object({
+  user_id: z.string(),
+  event_type: z.enum(["purchase", "view"]),
+  amount: z.number().positive().optional(),
+});
+
+const validated = EventSchema.parse(rawEvent); // Throws on invalid
+await env.STREAM.send([validated]);
+```
+
+**Why:** Structured streams drop invalid events silently. Client validation gives immediate feedback.
+
+## SQL Transform Patterns
+
+```sql
+-- Filter early (reduce storage)
+INSERT INTO my_sink
+SELECT user_id, event_type, amount
+FROM my_stream
+WHERE event_type = 'purchase' AND amount > 10
+
+-- Select only needed fields
+INSERT INTO my_sink
+SELECT user_id, event_type, timestamp FROM my_stream
+
+-- Enrich with CASE
+INSERT INTO my_sink
+SELECT user_id, amount,
+  CASE WHEN amount > 1000 THEN 'vip' ELSE 'standard' END as tier
+FROM my_stream
+```
+
+## Pipelines + Queues Fan-out
+
+```typescript
+await Promise.all([
+  env.ANALYTICS_STREAM.send([event]), // Long-term storage
+  env.PROCESS_QUEUE.send(event), // Immediate processing
+]);
+```
+
+| Need                           | Use             |
+| ------------------------------ | --------------- |
+| Long-term storage, SQL queries | Pipelines       |
+| Immediate processing, retries  | Queues          |
+| Both                           | Fan-out pattern |
+
+## Performance Tuning
+
+| Goal              | Config                                   |
+| ----------------- | ---------------------------------------- |
+| Low latency       | `--roll-interval 10`                     |
+| Query performance | `--roll-interval 300 --roll-size 100`    |
+| Cost optimal      | `--compression zstd --roll-interval 300` |
+
+## Schema Evolution
+
+Pipelines are immutable. Use versioning:
+
+```bash
+# Create v2 stream/sink/pipeline
+npx wrangler pipelines streams create events-v2 --schema-file v2.json
+
+# Dual-write during transition
+await Promise.all([env.EVENTS_V1.send([event]), env.EVENTS_V2.send([event])]);
+
+# Query across versions with UNION ALL
+```

package/vendor/skills/skills/cloudflare/references/pulumi/README.md

@@ -0,0 +1,113 @@
+# Cloudflare Pulumi Provider
+
+Expert guidance for Cloudflare Pulumi Provider (@pulumi/cloudflare).
+
+## Overview
+
+Programmatic management of Cloudflare resources: Workers, Pages, D1, KV, R2, DNS, Queues, etc.
+
+**Packages:**
+
+- TypeScript/JS: `@pulumi/cloudflare`
+- Python: `pulumi-cloudflare`
+- Go: `github.com/pulumi/pulumi-cloudflare/sdk/v6/go/cloudflare`
+- .NET: `Pulumi.Cloudflare`
+
+**Version:** v6.x
+
+## Core Principles
+
+1. Use API tokens (not legacy API keys)
+2. Store accountId in stack config
+3. Match binding names across code/config
+4. Use `module: true` for ES modules
+5. Set `compatibilityDate` to lock behavior
+
+## Authentication
+
+```typescript
+import * as cloudflare from "@pulumi/cloudflare";
+
+// API Token (recommended): CLOUDFLARE_API_TOKEN env
+const provider = new cloudflare.Provider("cf", { apiToken: process.env.CLOUDFLARE_API_TOKEN });
+
+// API Key (legacy): CLOUDFLARE_API_KEY + CLOUDFLARE_EMAIL env
+const provider = new cloudflare.Provider("cf", {
+  apiKey: process.env.CLOUDFLARE_API_KEY,
+  email: process.env.CLOUDFLARE_EMAIL,
+});
+
+// API User Service Key: CLOUDFLARE_API_USER_SERVICE_KEY env
+const provider = new cloudflare.Provider("cf", {
+  apiUserServiceKey: process.env.CLOUDFLARE_API_USER_SERVICE_KEY,
+});
+```
+
+## Setup
+
+**Pulumi.yaml:**
+
+```yaml
+name: my-cloudflare-app
+runtime: nodejs
+config:
+  cloudflare:apiToken:
+    value: ${CLOUDFLARE_API_TOKEN}
+```
+
+**Pulumi.<stack>.yaml:**
+
+```yaml
+config:
+  cloudflare:accountId: "abc123..."
+```
+
+**index.ts:**
+
+```typescript
+import * as pulumi from "@pulumi/pulumi";
+import * as cloudflare from "@pulumi/cloudflare";
+const accountId = new pulumi.Config("cloudflare").require("accountId");
+```
+
+## Common Resource Types
+
+- `Provider` - Provider config
+- `WorkerScript` - Worker
+- `WorkersKvNamespace` - KV
+- `R2Bucket` - R2
+- `D1Database` - D1
+- `Queue` - Queue
+- `PagesProject` - Pages
+- `DnsRecord` - DNS
+- `WorkerRoute` - Worker route
+- `WorkersDomain` - Custom domain
+
+## Key Properties
+
+- `accountId` - Required for most resources
+- `zoneId` - Required for DNS/domain
+- `name`/`title` - Resource identifier
+- `*Bindings` - Connect resources to Workers
+
+## Reading Order
+
+| Order | File                                   | What                                                  | When to Read                          |
+| ----- | -------------------------------------- | ----------------------------------------------------- | ------------------------------------- |
+| 1     | [configuration.md](./configuration.md) | Resource config for Workers/KV/D1/R2/Queues/Pages     | First time setup, resource reference  |
+| 2     | [patterns.md](./patterns.md)           | Architecture patterns, multi-env, component resources | Building complex apps, best practices |
+| 3     | [api.md](./api.md)                     | Outputs, dependencies, imports, dynamic providers     | Advanced features, integrations       |
+| 4     | [gotchas.md](./gotchas.md)             | Common errors, troubleshooting, limits                | Debugging, deployment issues          |
+
+## In This Reference
+
+- [configuration.md](./configuration.md) - Provider config, stack setup, Workers/bindings
+- [api.md](./api.md) - Resource types, Workers script, KV/D1/R2/queues/Pages
+- [patterns.md](./patterns.md) - Multi-env, secrets, CI/CD, stack management
+- [gotchas.md](./gotchas.md) - State issues, deployment failures, limits
+
+## See Also
+
+- [terraform](../terraform/) - Alternative IaC for Cloudflare
+- [wrangler](../wrangler/) - CLI deployment alternative
+- [workers](../workers/) - Worker runtime documentation

package/vendor/skills/skills/cloudflare/references/pulumi/api.md

@@ -0,0 +1,230 @@
+# API & Data Sources
+
+## Outputs and Exports
+
+Export resource identifiers:
+
+```typescript
+export const kvId = kv.id;
+export const bucketName = bucket.name;
+export const workerUrl = worker.subdomain;
+export const dbId = db.id;
+```
+
+## Resource Dependencies
+
+Implicit dependencies via outputs:
+
+```typescript
+const kv = new cloudflare.WorkersKvNamespace("kv", {
+  accountId: accountId,
+  title: "my-kv",
+});
+
+// Worker depends on KV (implicit via kv.id)
+const worker = new cloudflare.WorkerScript("worker", {
+  accountId: accountId,
+  name: "my-worker",
+  content: code,
+  kvNamespaceBindings: [{ name: "MY_KV", namespaceId: kv.id }], // Creates dependency
+});
+```
+
+Explicit dependencies:
+
+```typescript
+const migration = new command.local.Command(
+  "migration",
+  {
+    create: pulumi.interpolate`wrangler d1 execute ${db.name} --file ./schema.sql`,
+  },
+  { dependsOn: [db] }
+);
+
+const worker = new cloudflare.WorkerScript(
+  "worker",
+  {
+    accountId: accountId,
+    name: "worker",
+    content: code,
+    d1DatabaseBindings: [{ name: "DB", databaseId: db.id }],
+  },
+  { dependsOn: [migration] }
+); // Ensure migrations run first
+```
+
+## Using Outputs with API Calls
+
+```typescript
+const db = new cloudflare.D1Database("db", { accountId, name: "my-db" });
+
+db.id.apply(async dbId => {
+  const response = await fetch(
+    `https://api.cloudflare.com/client/v4/accounts/${accountId}/d1/database/${dbId}/query`,
+    {
+      method: "POST",
+      headers: { Authorization: `Bearer ${apiToken}`, "Content-Type": "application/json" },
+      body: JSON.stringify({ sql: "CREATE TABLE users (id INT)" }),
+    }
+  );
+  return response.json();
+});
+```
+
+## Custom Dynamic Providers
+
+For resources not in provider:
+
+```typescript
+import * as pulumi from "@pulumi/pulumi";
+
+class D1MigrationProvider implements pulumi.dynamic.ResourceProvider {
+  async create(inputs: any): Promise<pulumi.dynamic.CreateResult> {
+    const response = await fetch(
+      `https://api.cloudflare.com/client/v4/accounts/${inputs.accountId}/d1/database/${inputs.databaseId}/query`,
+      {
+        method: "POST",
+        headers: { Authorization: `Bearer ${inputs.apiToken}`, "Content-Type": "application/json" },
+        body: JSON.stringify({ sql: inputs.sql }),
+      }
+    );
+    return { id: `${inputs.databaseId}-${Date.now()}`, outs: await response.json() };
+  }
+  async update(id: string, olds: any, news: any): Promise<pulumi.dynamic.UpdateResult> {
+    if (olds.sql !== news.sql) await this.create(news);
+    return {};
+  }
+  async delete(id: string, props: any): Promise<void> {}
+}
+
+class D1Migration extends pulumi.dynamic.Resource {
+  constructor(name: string, args: any, opts?: pulumi.CustomResourceOptions) {
+    super(new D1MigrationProvider(), name, args, opts);
+  }
+}
+
+const migration = new D1Migration(
+  "migration",
+  {
+    accountId,
+    databaseId: db.id,
+    apiToken,
+    sql: "CREATE TABLE users (id INT)",
+  },
+  { dependsOn: [db] }
+);
+```
+
+## Data Sources
+
+**Get Zone:**
+
+```typescript
+const zone = cloudflare.getZone({ name: "example.com" });
+const zoneId = zone.then(z => z.id);
+```
+
+**Get Accounts (via API):**
+Use Cloudflare API directly or custom dynamic resources.
+
+## Import Existing Resources
+
+```bash
+# Import worker
+pulumi import cloudflare:index/workerScript:WorkerScript my-worker <account_id>/<worker_name>
+
+# Import KV namespace
+pulumi import cloudflare:index/workersKvNamespace:WorkersKvNamespace my-kv <namespace_id>
+
+# Import R2 bucket
+pulumi import cloudflare:index/r2Bucket:R2Bucket my-bucket <account_id>/<bucket_name>
+
+# Import D1 database
+pulumi import cloudflare:index/d1Database:D1Database my-db <account_id>/<database_id>
+
+# Import DNS record
+pulumi import cloudflare:index/dnsRecord:DnsRecord my-record <zone_id>/<record_id>
+```
+
+## Secrets Management
+
+```typescript
+import * as pulumi from "@pulumi/pulumi";
+
+const config = new pulumi.Config();
+const apiKey = config.requireSecret("apiKey"); // Encrypted in state
+
+const worker = new cloudflare.WorkerScript("worker", {
+  accountId: accountId,
+  name: "my-worker",
+  content: code,
+  secretTextBindings: [{ name: "API_KEY", text: apiKey }],
+});
+```
+
+Store secrets:
+
+```bash
+pulumi config set --secret apiKey "secret-value"
+```
+
+## Transform Pattern
+
+Modify resource args before creation:
+
+```typescript
+import { Transform } from "@pulumi/pulumi";
+
+interface BucketArgs {
+  accountId: pulumi.Input<string>;
+  transform?: { bucket?: Transform<cloudflare.R2BucketArgs> };
+}
+
+function createBucket(name: string, args: BucketArgs) {
+  const bucketArgs: cloudflare.R2BucketArgs = {
+    accountId: args.accountId,
+    name: name,
+    location: "auto",
+  };
+  const finalArgs = args.transform?.bucket?.(bucketArgs) ?? bucketArgs;
+  return new cloudflare.R2Bucket(name, finalArgs);
+}
+```
+
+## v6.x Worker Versioning Resources
+
+**Worker** - Container for versions:
+
+```typescript
+const worker = new cloudflare.Worker("api", { accountId, name: "api-worker" });
+export const workerId = worker.id;
+```
+
+**WorkerVersion** - Immutable code + config:
+
+```typescript
+const version = new cloudflare.WorkerVersion("v1", {
+  accountId,
+  workerId: worker.id,
+  content: fs.readFileSync("./dist/worker.js", "utf8"),
+  compatibilityDate: "2025-01-01",
+});
+export const versionId = version.id;
+```
+
+**WorkersDeployment** - Active deployment with bindings:
+
+```typescript
+const deployment = new cloudflare.WorkersDeployment("prod", {
+  accountId,
+  workerId: worker.id,
+  versionId: version.id,
+  kvNamespaceBindings: [{ name: "MY_KV", namespaceId: kv.id }],
+});
+```
+
+**Use:** Advanced deployments (canary, blue-green). Most apps should use `WorkerScript` (auto-versioning).
+
+---
+
+See: [README.md](./README.md), [configuration.md](./configuration.md), [patterns.md](./patterns.md), [gotchas.md](./gotchas.md)