@mohanscodex/spectra-code 0.4.8 → 0.5.0

package/skills/cloudflare-deploy/references/r2-sql/gotchas.md

@@ -0,0 +1,212 @@
+# R2 SQL Gotchas
+
+Limitations, troubleshooting, and common pitfalls for R2 SQL.
+
+## Critical Limitations
+
+### No Workers Binding
+
+**Cannot call R2 SQL from Workers/Pages code** - no binding exists.
+
+```typescript
+// ❌ This doesn't exist
+export default {
+  async fetch(request, env) {
+    const result = await env.R2_SQL.query("SELECT * FROM table");  // Not possible
+    return Response.json(result);
+  }
+};
+```
+
+**Solutions:**
+- HTTP API from external systems (not Workers)
+- PyIceberg/Spark via r2-data-catalog REST API
+- For Workers, use D1 or external databases
+
+### ORDER BY Limitations
+
+Can only order by:
+1. **Partition key columns** - Always supported
+2. **Aggregation functions** - Supported via shuffle strategy
+
+**Cannot order by** regular non-partition columns.
+
+```sql
+-- ✅ Valid: ORDER BY partition key
+SELECT * FROM logs.requests ORDER BY timestamp DESC LIMIT 100;
+
+-- ✅ Valid: ORDER BY aggregation
+SELECT region, SUM(amount) FROM sales.transactions
+GROUP BY region ORDER BY SUM(amount) DESC;
+
+-- ❌ Invalid: ORDER BY non-partition column
+SELECT * FROM logs.requests ORDER BY user_id;
+
+-- ❌ Invalid: ORDER BY alias (must repeat function)
+SELECT region, SUM(amount) as total FROM sales.transactions
+GROUP BY region ORDER BY total;  -- Use ORDER BY SUM(amount)
+```
+
+Check partition spec: `DESCRIBE namespace.table_name`
+
+## SQL Feature Limitations
+
+| Feature | Supported | Notes |
+|---------|-----------|-------|
+| SELECT, WHERE, GROUP BY, HAVING | ✅ | Standard support |
+| COUNT, SUM, AVG, MIN, MAX | ✅ | Standard aggregations |
+| ORDER BY partition/aggregation | ✅ | See above |
+| LIMIT | ✅ | Max 10,000 |
+| Column aliases | ❌ | No AS alias |
+| Expressions in SELECT | ❌ | No col1 + col2 |
+| ORDER BY non-partition | ❌ | Fails at runtime |
+| JOINs, subqueries, CTEs | ❌ | Denormalize at write time |
+| Window functions, UNION | ❌ | Use external engines |
+| INSERT/UPDATE/DELETE | ❌ | Use PyIceberg/Pipelines |
+| Nested columns, arrays, JSON | ❌ | Flatten at write time |
+
+**Workarounds:**
+- No JOINs: Denormalize data or use Spark/PyIceberg
+- No subqueries: Split into multiple queries
+- No aliases: Accept generated names, transform in app
+
+## Common Errors
+
+### "Column not found"
+**Cause:** Typo, column doesn't exist, or case mismatch  
+**Solution:** `DESCRIBE namespace.table_name` to check schema
+
+### "Type mismatch"
+```sql
+-- ❌ Wrong types
+WHERE status = '200'              -- string instead of integer
+WHERE timestamp > '2025-01-01'    -- missing time/timezone
+
+-- ✅ Correct types
+WHERE status = 200
+WHERE timestamp > '2025-01-01T00:00:00Z'
+```
+
+### "ORDER BY column not in partition key"
+**Cause:** Ordering by non-partition column  
+**Solution:** Use partition key, aggregation, or remove ORDER BY. Check: `DESCRIBE table`
+
+### "Token authentication failed"
+```bash
+# Check/set token
+echo $WRANGLER_R2_SQL_AUTH_TOKEN
+export WRANGLER_R2_SQL_AUTH_TOKEN=<your-token>
+
+# Or .env file
+echo "WRANGLER_R2_SQL_AUTH_TOKEN=<your-token>" > .env
+```
+
+### "Table not found"
+```sql
+-- Verify catalog and tables
+SHOW DATABASES;
+SHOW TABLES IN namespace_name;
+```
+
+Enable catalog: `npx wrangler r2 bucket catalog enable <bucket>`
+
+### "LIMIT exceeds maximum"
+Max LIMIT is 10,000. For pagination, use WHERE filters with partition keys.
+
+### "No data returned" (unexpected)
+**Debug steps:**
+1. `SELECT COUNT(*) FROM table` - verify data exists
+2. Remove WHERE filters incrementally
+3. `SELECT * FROM table LIMIT 10` - inspect actual data/types
+
+## Performance Issues
+
+### Slow Queries
+
+**Causes:** Too many partitions, large LIMIT, no filters, small files
+
+```sql
+-- ❌ Slow: No filters
+SELECT * FROM logs.requests LIMIT 10000;
+
+-- ✅ Fast: Filter on partition key
+SELECT * FROM logs.requests 
+WHERE timestamp >= '2025-01-15T00:00:00Z' AND timestamp < '2025-01-16T00:00:00Z'
+LIMIT 1000;
+
+-- ✅ Faster: Multiple filters
+SELECT * FROM logs.requests 
+WHERE timestamp >= '2025-01-15T00:00:00Z' AND status = 404 AND method = 'GET'
+LIMIT 1000;
+```
+
+**File optimization:**
+- Target Parquet size: 100-500MB compressed
+- Pipelines roll interval: 300+ sec (prod), 10 sec (dev)
+- Run compaction to merge small files
+
+### Query Timeout
+
+**Solution:** Add restrictive WHERE filters, reduce time range, query smaller intervals
+
+```sql
+-- ❌ Times out: Year-long aggregation
+SELECT status, COUNT(*) FROM logs.requests 
+WHERE timestamp >= '2024-01-01T00:00:00Z' GROUP BY status;
+
+-- ✅ Faster: Month-long aggregation
+SELECT status, COUNT(*) FROM logs.requests 
+WHERE timestamp >= '2025-01-01T00:00:00Z' AND timestamp < '2025-02-01T00:00:00Z'
+GROUP BY status;
+```
+
+## Best Practices
+
+### Partitioning
+- **Time-series:** Partition by day/hour on timestamp
+- **Avoid:** High-cardinality keys (user_id), >10,000 partitions
+
+```python
+from pyiceberg.partitioning import PartitionSpec, PartitionField
+from pyiceberg.transforms import DayTransform
+
+PartitionSpec(PartitionField(source_id=1, field_id=1000, transform=DayTransform(), name="day"))
+```
+
+### Query Writing
+- **Always use LIMIT** for early termination
+- **Filter on partition keys first** for pruning
+- **Combine filters with AND** for more pruning
+
+```sql
+-- Good
+WHERE timestamp >= '2025-01-15T00:00:00Z' AND status = 404 AND method = 'GET' LIMIT 100
+```
+
+### Type Safety
+- Quote strings: `'GET'` not `GET`
+- RFC3339 timestamps: `'2025-01-01T00:00:00Z'` not `'2025-01-01'`
+- ISO dates: `'2025-01-15'` not `'01/15/2025'`
+
+### Data Organization
+- **Pipelines:** Dev `roll_file_time: 10`, Prod `roll_file_time: 300+`
+- **Compression:** Use `zstd`
+- **Maintenance:** Compaction for small files, expire old snapshots
+
+## Debugging Checklist
+
+1. `npx wrangler r2 bucket catalog enable <bucket>` - Verify catalog
+2. `echo $WRANGLER_R2_SQL_AUTH_TOKEN` - Check token
+3. `SHOW DATABASES` - List namespaces
+4. `SHOW TABLES IN namespace` - List tables
+5. `DESCRIBE namespace.table` - Check schema
+6. `SELECT COUNT(*) FROM namespace.table` - Verify data
+7. `SELECT * FROM namespace.table LIMIT 10` - Test simple query
+8. Add filters incrementally
+
+## See Also
+
+- [api.md](api.md) - SQL syntax
+- [patterns.md](patterns.md) - Query optimization
+- [configuration.md](configuration.md) - Setup
+- [Cloudflare R2 SQL Docs](https://developers.cloudflare.com/r2-sql/)

package/skills/cloudflare-deploy/references/r2-sql/patterns.md

@@ -0,0 +1,222 @@
+# R2 SQL Patterns
+
+Common patterns, use cases, and integration examples for R2 SQL.
+
+## Wrangler CLI Query
+
+```bash
+# Basic query
+npx wrangler r2 sql query "my-bucket" "SELECT * FROM default.logs LIMIT 10"
+
+# Multi-line query
+npx wrangler r2 sql query "my-bucket" "
+  SELECT status, COUNT(*), AVG(response_time)
+  FROM logs.http_requests
+  WHERE timestamp >= '2025-01-01T00:00:00Z'
+  GROUP BY status
+  ORDER BY COUNT(*) DESC
+  LIMIT 100
+"
+
+# Use environment variable
+export R2_SQL_WAREHOUSE="my-bucket"
+npx wrangler r2 sql query "$R2_SQL_WAREHOUSE" "SELECT * FROM default.logs"
+```
+
+## HTTP API Query
+
+For programmatic access from external systems (not Workers - see gotchas.md).
+
+```bash
+curl -X POST https://api.cloudflare.com/client/v4/accounts/{account_id}/r2/sql/query \
+  -H "Authorization: Bearer <your-token>" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "warehouse": "my-bucket",
+    "query": "SELECT * FROM default.my_table WHERE status = 200 LIMIT 100"
+  }'
+```
+
+Response:
+```json
+{
+  "success": true,
+  "result": [{"user_id": "user_123", "timestamp": "2025-01-15T10:30:00Z", "status": 200}],
+  "errors": []
+}
+```
+
+## Pipelines Integration
+
+Stream data to Iceberg tables via Pipelines, then query with R2 SQL.
+
+```bash
+# Setup pipeline (select Data Catalog Table destination)
+npx wrangler pipelines setup
+
+# Key settings:
+# - Destination: Data Catalog Table
+# - Compression: zstd (recommended)
+# - Roll file time: 300+ sec (production), 10 sec (dev)
+
+# Send data to pipeline
+curl -X POST https://{stream-id}.ingest.cloudflare.com \
+  -H "Content-Type: application/json" \
+  -d '[{"user_id": "user_123", "event_type": "purchase", "timestamp": "2025-01-15T10:30:00Z", "amount": 29.99}]'
+
+# Query ingested data (wait for roll interval)
+npx wrangler r2 sql query "my-bucket" "
+  SELECT event_type, COUNT(*), SUM(amount)
+  FROM default.events
+  WHERE timestamp >= '2025-01-15T00:00:00Z'
+  GROUP BY event_type
+"
+```
+
+See [pipelines/patterns.md](../pipelines/patterns.md) for detailed setup.
+
+## PyIceberg Integration
+
+Create and populate Iceberg tables with PyIceberg, then query with R2 SQL.
+
+```python
+from pyiceberg.catalog.rest import RestCatalog
+import pyarrow as pa
+import pandas as pd
+
+# Setup catalog
+catalog = RestCatalog(
+    name="my_catalog",
+    warehouse="my-bucket",
+    uri="https://<account-id>.r2.cloudflarestorage.com/iceberg/my-bucket",
+    token="<your-token>",
+)
+catalog.create_namespace_if_not_exists("analytics")
+
+# Create table
+schema = pa.schema([
+    pa.field("user_id", pa.string(), nullable=False),
+    pa.field("event_time", pa.timestamp("us", tz="UTC"), nullable=False),
+    pa.field("page_views", pa.int64(), nullable=False),
+])
+table = catalog.create_table(("analytics", "user_metrics"), schema=schema)
+
+# Append data
+df = pd.DataFrame({
+    "user_id": ["user_1", "user_2"],
+    "event_time": pd.to_datetime(["2025-01-15 10:00:00", "2025-01-15 11:00:00"], utc=True),
+    "page_views": [10, 25],
+})
+table.append(pa.Table.from_pandas(df, schema=schema))
+```
+
+Query with R2 SQL:
+```bash
+npx wrangler r2 sql query "my-bucket" "
+  SELECT user_id, SUM(page_views)
+  FROM analytics.user_metrics
+  WHERE event_time >= '2025-01-15T00:00:00Z'
+  GROUP BY user_id
+"
+```
+
+See [r2-data-catalog/patterns.md](../r2-data-catalog/patterns.md) for advanced PyIceberg patterns.
+
+## Use Cases
+
+### Log Analytics
+```sql
+-- Error rate by endpoint
+SELECT path, COUNT(*), SUM(CASE WHEN status >= 400 THEN 1 ELSE 0 END) as errors
+FROM logs.http_requests
+WHERE timestamp BETWEEN '2025-01-01T00:00:00Z' AND '2025-01-31T23:59:59Z'
+GROUP BY path ORDER BY errors DESC LIMIT 20;
+
+-- Response time stats
+SELECT method, MIN(response_time_ms), AVG(response_time_ms), MAX(response_time_ms)
+FROM logs.http_requests WHERE timestamp >= '2025-01-15T00:00:00Z' GROUP BY method;
+
+-- Traffic by status
+SELECT status, COUNT(*) FROM logs.http_requests
+WHERE timestamp >= '2025-01-15T00:00:00Z' AND method = 'GET'
+GROUP BY status ORDER BY COUNT(*) DESC;
+```
+
+### Fraud Detection
+```sql
+-- High-value transactions
+SELECT location, COUNT(*), SUM(amount), AVG(amount)
+FROM fraud.transactions WHERE transaction_timestamp >= '2025-01-01T00:00:00Z' AND amount > 1000.0
+GROUP BY location ORDER BY SUM(amount) DESC LIMIT 20;
+
+-- Flagged transactions
+SELECT merchant_category, COUNT(*), AVG(amount) FROM fraud.transactions
+WHERE is_fraud_flag = true AND transaction_timestamp >= '2025-01-01T00:00:00Z'
+GROUP BY merchant_category HAVING COUNT(*) > 10 ORDER BY COUNT(*) DESC;
+```
+
+### Business Intelligence
+```sql
+-- Sales by department
+SELECT department, SUM(revenue), AVG(revenue), COUNT(*) FROM sales.transactions
+WHERE sale_date >= '2024-01-01' GROUP BY department ORDER BY SUM(revenue) DESC LIMIT 10;
+
+-- Product performance
+SELECT category, COUNT(DISTINCT product_id), SUM(units_sold), SUM(revenue)
+FROM sales.product_sales WHERE sale_date BETWEEN '2024-10-01' AND '2024-12-31'
+GROUP BY category ORDER BY SUM(revenue) DESC;
+```
+
+## Connecting External Engines
+
+R2 Data Catalog exposes Iceberg REST API. Connect Spark, Snowflake, Trino, DuckDB, etc.
+
+```scala
+// Apache Spark example
+val spark = SparkSession.builder()
+  .config("spark.sql.catalog.my_catalog", "org.apache.iceberg.spark.SparkCatalog")
+  .config("spark.sql.catalog.my_catalog.catalog-impl", "org.apache.iceberg.rest.RESTCatalog")
+  .config("spark.sql.catalog.my_catalog.uri", "https://<account-id>.r2.cloudflarestorage.com/iceberg/my-bucket")
+  .config("spark.sql.catalog.my_catalog.token", "<token>")
+  .getOrCreate()
+
+spark.sql("SELECT * FROM my_catalog.default.my_table LIMIT 10").show()
+```
+
+See [r2-data-catalog/patterns.md](../r2-data-catalog/patterns.md) for more engines.
+
+## Performance Optimization
+
+### Partitioning
+- **Time-series:** day/hour on timestamp
+- **Geographic:** region/country
+- **Avoid:** High-cardinality keys (user_id)
+
+```python
+from pyiceberg.partitioning import PartitionSpec, PartitionField
+from pyiceberg.transforms import DayTransform
+
+PartitionSpec(PartitionField(source_id=1, field_id=1000, transform=DayTransform(), name="day"))
+```
+
+### Query Optimization
+- **Always use LIMIT** for early termination
+- **Filter on partition keys first**
+- **Multiple filters** for better pruning
+
+```sql
+-- Better: Multiple filters on partition key
+SELECT * FROM logs.requests 
+WHERE timestamp >= '2025-01-15T00:00:00Z' AND status = 404 AND method = 'GET' LIMIT 100;
+```
+
+### File Organization
+- **Pipelines roll:** Dev 10-30s, Prod 300+s
+- **Target Parquet:** 100-500MB compressed
+
+## See Also
+
+- [api.md](api.md) - SQL syntax reference
+- [gotchas.md](gotchas.md) - Limitations and troubleshooting
+- [r2-data-catalog/patterns.md](../r2-data-catalog/patterns.md) - PyIceberg advanced patterns
+- [pipelines/patterns.md](../pipelines/patterns.md) - Streaming ingestion patterns

package/skills/cloudflare-deploy/references/realtime-sfu/README.md

@@ -0,0 +1,65 @@
+# Cloudflare Realtime SFU Reference
+
+Expert guidance for building real-time audio/video/data applications using Cloudflare Realtime SFU (Selective Forwarding Unit).
+
+## Reading Order
+
+| Task | Files | ~Tokens |
+|------|-------|---------|
+| New project | README → configuration | ~1200 |
+| Implement publish/subscribe | README → api | ~1600 |
+| Add PartyTracks | patterns (PartyTracks section) | ~800 |
+| Build presence system | patterns (DO section) | ~800 |
+| Debug connection issues | gotchas | ~700 |
+| Scale to millions | patterns (Cascading section) | ~600 |
+| Add simulcast | patterns (Advanced section) | ~500 |
+| Configure TURN | configuration (TURN section) | ~400 |
+
+## In This Reference
+
+- **[configuration.md](configuration.md)** - Setup, deployment, environment variables, Wrangler config
+- **[api.md](api.md)** - Sessions, tracks, endpoints, request/response patterns
+- **[patterns.md](patterns.md)** - Architecture patterns, use cases, integration examples
+- **[gotchas.md](gotchas.md)** - Common issues, debugging, performance, security
+
+## Quick Start
+
+Cloudflare Realtime SFU: WebRTC infrastructure on global network (310+ cities). Anycast routing, no regional constraints, pub/sub model.
+
+**Core concepts:**
+- **Sessions:** WebRTC PeerConnection to Cloudflare edge
+- **Tracks:** Audio/video/data channels you publish or subscribe to
+- **No rooms:** Build presence layer yourself via track sharing (see patterns.md)
+
+**Mental model:** Your client establishes one WebRTC session, publishes tracks (audio/video), shares track IDs via your backend, others subscribe to your tracks using track IDs + your session ID.
+
+## Choose Your Approach
+
+| Approach | When to Use | Complexity |
+|----------|-------------|------------|
+| **PartyTracks** | Production apps with device switching, React | Low - Observable-based, handles reconnections |
+| **Raw API** | Custom requirements, non-browser, learning | Medium - Full control, manual WebRTC lifecycle |
+| **RealtimeKit** | End-to-end SDK with UI components | Lowest - Managed state, React hooks |
+
+**Recommendation:** Start with PartyTracks for most production applications. See patterns.md for PartyTracks examples.
+
+## SFU vs RealtimeKit
+
+- **Realtime SFU:** WebRTC infrastructure (this reference). Build your own signaling, presence, UI.
+- **RealtimeKit:** SDK layer on top of SFU. Includes React hooks, state management, UI components. Part of Cloudflare AI platform.
+
+Use SFU directly when you need custom signaling or non-React framework. Use RealtimeKit for faster development with React.
+
+## Setup
+
+Dashboard: https://dash.cloudflare.com/?to=/:account/calls
+
+Get `CALLS_APP_ID` and `CALLS_APP_SECRET` from dashboard, then see configuration.md for deployment.
+
+## See Also
+
+- [Orange Meets Demo](https://demo.orange.cloudflare.dev/)
+- [Orange Source](https://github.com/cloudflare/orange)
+- [Calls Examples](https://github.com/cloudflare/calls-examples)
+- [API Reference](https://developers.cloudflare.com/api/resources/calls/)
+- [RealtimeKit Docs](https://developers.cloudflare.com/workers-ai/realtimekit/)

package/skills/cloudflare-deploy/references/realtime-sfu/api.md

@@ -0,0 +1,158 @@
+# API Reference
+
+## Authentication
+
+```bash
+curl -X POST 'https://rtc.live/v1/apps/${CALLS_APP_ID}/sessions/new' \
+  -H "Authorization: Bearer ${CALLS_APP_SECRET}"
+```
+
+## Core Concepts
+
+**Sessions:** PeerConnection to Cloudflare edge  
+**Tracks:** Media/data channels (audio/video/datachannel)  
+**No rooms:** Build presence via track sharing
+
+## Client Libraries
+
+**PartyTracks (Recommended):** Observable-based client library for production use. Handles device changes, network switches, ICE restarts automatically. Push/pull API with React hooks. See patterns.md for full examples.
+
+```bash
+npm install partytracks @cloudflare/calls
+```
+
+**Raw API:** Direct HTTP + WebRTC for custom requirements (documented below).
+
+## Endpoints
+
+### Create Session
+```http
+POST /v1/apps/{appId}/sessions/new
+→ {sessionId, sessionDescription}
+```
+
+### Add Track (Publish)
+```http
+POST /v1/apps/{appId}/sessions/{sessionId}/tracks/new
+Body: {
+  sessionDescription: {sdp, type: "offer"},
+  tracks: [{location: "local", trackName: "my-video"}]
+}
+→ {sessionDescription, tracks: [{trackName}]}
+```
+
+### Add Track (Subscribe)
+```http
+POST /v1/apps/{appId}/sessions/{sessionId}/tracks/new
+Body: {
+  tracks: [{
+    location: "remote",
+    trackName: "remote-track-id",
+    sessionId: "other-session-id"
+  }]
+}
+→ {sessionDescription} (server offer)
+```
+
+### Renegotiate
+```http
+PUT /v1/apps/{appId}/sessions/{sessionId}/renegotiate
+Body: {sessionDescription: {sdp, type: "answer"}}
+```
+
+### Close Tracks
+```http
+PUT /v1/apps/{appId}/sessions/{sessionId}/tracks/close
+Body: {tracks: [{trackName}]}
+→ {requiresImmediateRenegotiation: boolean}
+```
+
+### Get Session
+```http
+GET /v1/apps/{appId}/sessions/{sessionId}
+→ {sessionId, tracks: TrackMetadata[]}
+```
+
+## TypeScript Types
+
+```typescript
+interface TrackMetadata {
+  trackName: string;
+  location: "local" | "remote";
+  sessionId?: string; // For remote tracks
+  mid?: string; // WebRTC mid
+}
+```
+
+## WebRTC Flow
+
+```typescript
+// 1. Create PeerConnection
+const pc = new RTCPeerConnection({
+  iceServers: [{urls: 'stun:stun.cloudflare.com:3478'}]
+});
+
+// 2. Add tracks
+const stream = await navigator.mediaDevices.getUserMedia({video: true, audio: true});
+stream.getTracks().forEach(track => pc.addTrack(track, stream));
+
+// 3. Create offer
+const offer = await pc.createOffer();
+await pc.setLocalDescription(offer);
+
+// 4. Send to backend → Cloudflare API
+const response = await fetch('/api/new-session', {
+  method: 'POST',
+  body: JSON.stringify({sdp: offer.sdp})
+});
+
+// 5. Set remote answer
+const {sessionDescription} = await response.json();
+await pc.setRemoteDescription(sessionDescription);
+```
+
+## Publishing
+
+```typescript
+const offer = await pc.createOffer();
+await pc.setLocalDescription(offer);
+
+const res = await fetch(`/api/sessions/${sessionId}/tracks`, {
+  method: 'POST',
+  body: JSON.stringify({
+    sdp: offer.sdp,
+    tracks: [{location: 'local', trackName: 'my-video'}]
+  })
+});
+
+const {sessionDescription, tracks} = await res.json();
+await pc.setRemoteDescription(sessionDescription);
+const publishedTrackId = tracks[0].trackName; // Share with others
+```
+
+## Subscribing
+
+```typescript
+const res = await fetch(`/api/sessions/${sessionId}/tracks`, {
+  method: 'POST',
+  body: JSON.stringify({
+    tracks: [{location: 'remote', trackName: remoteTrackId, sessionId: remoteSessionId}]
+  })
+});
+
+const {sessionDescription} = await res.json();
+await pc.setRemoteDescription(sessionDescription);
+
+const answer = await pc.createAnswer();
+await pc.setLocalDescription(answer);
+
+await fetch(`/api/sessions/${sessionId}/renegotiate`, {
+  method: 'PUT',
+  body: JSON.stringify({sdp: answer.sdp})
+});
+
+pc.ontrack = (event) => {
+  const [remoteStream] = event.streams;
+  videoElement.srcObject = remoteStream;
+};
+```