@bcelep/capint 0.4.2 → 0.7.0

package/skills/cloudflare/references/r2-sql/configuration.md

@@ -0,0 +1,147 @@
+# R2 SQL Configuration
+
+Setup and configuration for R2 SQL queries.
+
+## Prerequisites
+
+- R2 bucket with Data Catalog enabled
+- API token with R2 permissions
+- Wrangler CLI installed (for CLI queries)
+
+## Enable R2 Data Catalog
+
+R2 SQL queries Apache Iceberg tables in R2 Data Catalog. Must enable catalog on bucket first.
+
+### Via Wrangler CLI
+
+```bash
+npx wrangler r2 bucket catalog enable <bucket-name>
+```
+
+Output includes:
+- **Warehouse name** - Typically same as bucket name
+- **Catalog URI** - REST endpoint for catalog operations
+
+Example output:
+```
+Catalog enabled successfully
+Warehouse: my-bucket
+Catalog URI: https://abc123.r2.cloudflarestorage.com/iceberg/my-bucket
+```
+
+### Via Dashboard
+
+1. Navigate to **R2 Object Storage** → Select your bucket
+2. Click **Settings** tab
+3. Scroll to **R2 Data Catalog** section
+4. Click **Enable**
+5. Note the **Catalog URI** and **Warehouse** name
+
+**Important:** Enabling catalog creates metadata directories in bucket but does not modify existing objects.
+
+## Create API Token
+
+R2 SQL requires API token with R2 permissions.
+
+### Required Permission
+
+**R2 Admin Read & Write** (includes R2 SQL Read permission)
+
+### Via Dashboard
+
+1. Navigate to **R2 Object Storage**
+2. Click **Manage API tokens** (top right)
+3. Click **Create API token**
+4. Select **Admin Read & Write** permission
+5. Click **Create API Token**
+6. **Copy token value** - shown only once
+
+### Permission Scope
+
+| Permission | Grants Access To |
+|------------|------------------|
+| R2 Admin Read & Write | R2 storage operations + R2 SQL queries + Data Catalog operations |
+| R2 SQL Read | SQL queries only (no storage writes) |
+
+**Note:** R2 SQL Read permission not yet available via Dashboard - use Admin Read & Write.
+
+## Configure Environment
+
+### Wrangler CLI
+
+Set environment variable for Wrangler to use:
+
+```bash
+export WRANGLER_R2_SQL_AUTH_TOKEN=<your-token>
+```
+
+Or create `.env` file in project directory:
+
+```
+WRANGLER_R2_SQL_AUTH_TOKEN=<your-token>
+```
+
+Wrangler automatically loads `.env` file when running commands.
+
+### HTTP API
+
+For programmatic access (non-Wrangler), pass token in Authorization header:
+
+```bash
+curl -X POST https://api.cloudflare.com/client/v4/accounts/{account_id}/r2/sql/query \
+  -H "Authorization: Bearer <your-token>" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "warehouse": "my-bucket",
+    "query": "SELECT * FROM default.my_table LIMIT 10"
+  }'
+```
+
+**Note:** HTTP API endpoint URL may vary - see [patterns.md](patterns.md#http-api-query) for current endpoint.
+
+## Verify Setup
+
+Test configuration by querying system tables:
+
+```bash
+# List namespaces
+npx wrangler r2 sql query "my-bucket" "SHOW DATABASES"
+
+# List tables in namespace
+npx wrangler r2 sql query "my-bucket" "SHOW TABLES IN default"
+```
+
+If successful, returns JSON array of results.
+
+## Troubleshooting
+
+### "Token authentication failed"
+
+**Cause:** Invalid or missing token
+
+**Solution:**
+- Verify `WRANGLER_R2_SQL_AUTH_TOKEN` environment variable set
+- Check token has Admin Read & Write permission
+- Create new token if expired
+
+### "Catalog not enabled on bucket"
+
+**Cause:** Data Catalog not enabled
+
+**Solution:**
+- Run `npx wrangler r2 bucket catalog enable <bucket-name>`
+- Or enable via Dashboard (R2 → bucket → Settings → R2 Data Catalog)
+
+### "Permission denied"
+
+**Cause:** Token lacks required permissions
+
+**Solution:**
+- Verify token has **Admin Read & Write** permission
+- Create new token with correct permissions
+
+## See Also
+
+- [r2-data-catalog/configuration.md](../r2-data-catalog/configuration.md) - Detailed token setup and PyIceberg connection
+- [patterns.md](patterns.md) - Query examples using configuration
+- [gotchas.md](gotchas.md) - Common configuration errors

package/skills/cloudflare/references/r2-sql/gotchas.md

@@ -0,0 +1,212 @@
+# R2 SQL Gotchas
+
+Limitations, troubleshooting, and common pitfalls for R2 SQL.
+
+## Critical Limitations
+
+### No Workers Binding
+
+**Cannot call R2 SQL from Workers/Pages code** - no binding exists.
+
+```typescript
+// ❌ This doesn't exist
+export default {
+  async fetch(request, env) {
+    const result = await env.R2_SQL.query("SELECT * FROM table");  // Not possible
+    return Response.json(result);
+  }
+};
+```
+
+**Solutions:**
+- HTTP API from external systems (not Workers)
+- PyIceberg/Spark via r2-data-catalog REST API
+- For Workers, use D1 or external databases
+
+### ORDER BY Limitations
+
+Can only order by:
+1. **Partition key columns** - Always supported
+2. **Aggregation functions** - Supported via shuffle strategy
+
+**Cannot order by** regular non-partition columns.
+
+```sql
+-- ✅ Valid: ORDER BY partition key
+SELECT * FROM logs.requests ORDER BY timestamp DESC LIMIT 100;
+
+-- ✅ Valid: ORDER BY aggregation
+SELECT region, SUM(amount) FROM sales.transactions
+GROUP BY region ORDER BY SUM(amount) DESC;
+
+-- ❌ Invalid: ORDER BY non-partition column
+SELECT * FROM logs.requests ORDER BY user_id;
+
+-- ❌ Invalid: ORDER BY alias (must repeat function)
+SELECT region, SUM(amount) as total FROM sales.transactions
+GROUP BY region ORDER BY total;  -- Use ORDER BY SUM(amount)
+```
+
+Check partition spec: `DESCRIBE namespace.table_name`
+
+## SQL Feature Limitations
+
+| Feature | Supported | Notes |
+|---------|-----------|-------|
+| SELECT, WHERE, GROUP BY, HAVING | ✅ | Standard support |
+| COUNT, SUM, AVG, MIN, MAX | ✅ | Standard aggregations |
+| ORDER BY partition/aggregation | ✅ | See above |
+| LIMIT | ✅ | Max 10,000 |
+| Column aliases | ❌ | No AS alias |
+| Expressions in SELECT | ❌ | No col1 + col2 |
+| ORDER BY non-partition | ❌ | Fails at runtime |
+| JOINs, subqueries, CTEs | ❌ | Denormalize at write time |
+| Window functions, UNION | ❌ | Use external engines |
+| INSERT/UPDATE/DELETE | ❌ | Use PyIceberg/Pipelines |
+| Nested columns, arrays, JSON | ❌ | Flatten at write time |
+
+**Workarounds:**
+- No JOINs: Denormalize data or use Spark/PyIceberg
+- No subqueries: Split into multiple queries
+- No aliases: Accept generated names, transform in app
+
+## Common Errors
+
+### "Column not found"
+**Cause:** Typo, column doesn't exist, or case mismatch  
+**Solution:** `DESCRIBE namespace.table_name` to check schema
+
+### "Type mismatch"
+```sql
+-- ❌ Wrong types
+WHERE status = '200'              -- string instead of integer
+WHERE timestamp > '2025-01-01'    -- missing time/timezone
+
+-- ✅ Correct types
+WHERE status = 200
+WHERE timestamp > '2025-01-01T00:00:00Z'
+```
+
+### "ORDER BY column not in partition key"
+**Cause:** Ordering by non-partition column  
+**Solution:** Use partition key, aggregation, or remove ORDER BY. Check: `DESCRIBE table`
+
+### "Token authentication failed"
+```bash
+# Check/set token
+echo $WRANGLER_R2_SQL_AUTH_TOKEN
+export WRANGLER_R2_SQL_AUTH_TOKEN=<your-token>
+
+# Or .env file
+echo "WRANGLER_R2_SQL_AUTH_TOKEN=<your-token>" > .env
+```
+
+### "Table not found"
+```sql
+-- Verify catalog and tables
+SHOW DATABASES;
+SHOW TABLES IN namespace_name;
+```
+
+Enable catalog: `npx wrangler r2 bucket catalog enable <bucket>`
+
+### "LIMIT exceeds maximum"
+Max LIMIT is 10,000. For pagination, use WHERE filters with partition keys.
+
+### "No data returned" (unexpected)
+**Debug steps:**
+1. `SELECT COUNT(*) FROM table` - verify data exists
+2. Remove WHERE filters incrementally
+3. `SELECT * FROM table LIMIT 10` - inspect actual data/types
+
+## Performance Issues
+
+### Slow Queries
+
+**Causes:** Too many partitions, large LIMIT, no filters, small files
+
+```sql
+-- ❌ Slow: No filters
+SELECT * FROM logs.requests LIMIT 10000;
+
+-- ✅ Fast: Filter on partition key
+SELECT * FROM logs.requests 
+WHERE timestamp >= '2025-01-15T00:00:00Z' AND timestamp < '2025-01-16T00:00:00Z'
+LIMIT 1000;
+
+-- ✅ Faster: Multiple filters
+SELECT * FROM logs.requests 
+WHERE timestamp >= '2025-01-15T00:00:00Z' AND status = 404 AND method = 'GET'
+LIMIT 1000;
+```
+
+**File optimization:**
+- Target Parquet size: 100-500MB compressed
+- Pipelines roll interval: 300+ sec (prod), 10 sec (dev)
+- Run compaction to merge small files
+
+### Query Timeout
+
+**Solution:** Add restrictive WHERE filters, reduce time range, query smaller intervals
+
+```sql
+-- ❌ Times out: Year-long aggregation
+SELECT status, COUNT(*) FROM logs.requests 
+WHERE timestamp >= '2024-01-01T00:00:00Z' GROUP BY status;
+
+-- ✅ Faster: Month-long aggregation
+SELECT status, COUNT(*) FROM logs.requests 
+WHERE timestamp >= '2025-01-01T00:00:00Z' AND timestamp < '2025-02-01T00:00:00Z'
+GROUP BY status;
+```
+
+## Best Practices
+
+### Partitioning
+- **Time-series:** Partition by day/hour on timestamp
+- **Avoid:** High-cardinality keys (user_id), >10,000 partitions
+
+```python
+from pyiceberg.partitioning import PartitionSpec, PartitionField
+from pyiceberg.transforms import DayTransform
+
+PartitionSpec(PartitionField(source_id=1, field_id=1000, transform=DayTransform(), name="day"))
+```
+
+### Query Writing
+- **Always use LIMIT** for early termination
+- **Filter on partition keys first** for pruning
+- **Combine filters with AND** for more pruning
+
+```sql
+-- Good
+WHERE timestamp >= '2025-01-15T00:00:00Z' AND status = 404 AND method = 'GET' LIMIT 100
+```
+
+### Type Safety
+- Quote strings: `'GET'` not `GET`
+- RFC3339 timestamps: `'2025-01-01T00:00:00Z'` not `'2025-01-01'`
+- ISO dates: `'2025-01-15'` not `'01/15/2025'`
+
+### Data Organization
+- **Pipelines:** Dev `roll_file_time: 10`, Prod `roll_file_time: 300+`
+- **Compression:** Use `zstd`
+- **Maintenance:** Compaction for small files, expire old snapshots
+
+## Debugging Checklist
+
+1. `npx wrangler r2 bucket catalog enable <bucket>` - Verify catalog
+2. `echo $WRANGLER_R2_SQL_AUTH_TOKEN` - Check token
+3. `SHOW DATABASES` - List namespaces
+4. `SHOW TABLES IN namespace` - List tables
+5. `DESCRIBE namespace.table` - Check schema
+6. `SELECT COUNT(*) FROM namespace.table` - Verify data
+7. `SELECT * FROM namespace.table LIMIT 10` - Test simple query
+8. Add filters incrementally
+
+## See Also
+
+- [api.md](api.md) - SQL syntax
+- [patterns.md](patterns.md) - Query optimization
+- [configuration.md](configuration.md) - Setup
+- [Cloudflare R2 SQL Docs](https://developers.cloudflare.com/r2-sql/)

package/skills/cloudflare/references/r2-sql/patterns.md

@@ -0,0 +1,222 @@
+# R2 SQL Patterns
+
+Common patterns, use cases, and integration examples for R2 SQL.
+
+## Wrangler CLI Query
+
+```bash
+# Basic query
+npx wrangler r2 sql query "my-bucket" "SELECT * FROM default.logs LIMIT 10"
+
+# Multi-line query
+npx wrangler r2 sql query "my-bucket" "
+  SELECT status, COUNT(*), AVG(response_time)
+  FROM logs.http_requests
+  WHERE timestamp >= '2025-01-01T00:00:00Z'
+  GROUP BY status
+  ORDER BY COUNT(*) DESC
+  LIMIT 100
+"
+
+# Use environment variable
+export R2_SQL_WAREHOUSE="my-bucket"
+npx wrangler r2 sql query "$R2_SQL_WAREHOUSE" "SELECT * FROM default.logs"
+```
+
+## HTTP API Query
+
+For programmatic access from external systems (not Workers - see gotchas.md).
+
+```bash
+curl -X POST https://api.cloudflare.com/client/v4/accounts/{account_id}/r2/sql/query \
+  -H "Authorization: Bearer <your-token>" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "warehouse": "my-bucket",
+    "query": "SELECT * FROM default.my_table WHERE status = 200 LIMIT 100"
+  }'
+```
+
+Response:
+```json
+{
+  "success": true,
+  "result": [{"user_id": "user_123", "timestamp": "2025-01-15T10:30:00Z", "status": 200}],
+  "errors": []
+}
+```
+
+## Pipelines Integration
+
+Stream data to Iceberg tables via Pipelines, then query with R2 SQL.
+
+```bash
+# Setup pipeline (select Data Catalog Table destination)
+npx wrangler pipelines setup
+
+# Key settings:
+# - Destination: Data Catalog Table
+# - Compression: zstd (recommended)
+# - Roll file time: 300+ sec (production), 10 sec (dev)
+
+# Send data to pipeline
+curl -X POST https://{stream-id}.ingest.cloudflare.com \
+  -H "Content-Type: application/json" \
+  -d '[{"user_id": "user_123", "event_type": "purchase", "timestamp": "2025-01-15T10:30:00Z", "amount": 29.99}]'
+
+# Query ingested data (wait for roll interval)
+npx wrangler r2 sql query "my-bucket" "
+  SELECT event_type, COUNT(*), SUM(amount)
+  FROM default.events
+  WHERE timestamp >= '2025-01-15T00:00:00Z'
+  GROUP BY event_type
+"
+```
+
+See [pipelines/patterns.md](../pipelines/patterns.md) for detailed setup.
+
+## PyIceberg Integration
+
+Create and populate Iceberg tables with PyIceberg, then query with R2 SQL.
+
+```python
+from pyiceberg.catalog.rest import RestCatalog
+import pyarrow as pa
+import pandas as pd
+
+# Setup catalog
+catalog = RestCatalog(
+    name="my_catalog",
+    warehouse="my-bucket",
+    uri="https://<account-id>.r2.cloudflarestorage.com/iceberg/my-bucket",
+    token="<your-token>",
+)
+catalog.create_namespace_if_not_exists("analytics")
+
+# Create table
+schema = pa.schema([
+    pa.field("user_id", pa.string(), nullable=False),
+    pa.field("event_time", pa.timestamp("us", tz="UTC"), nullable=False),
+    pa.field("page_views", pa.int64(), nullable=False),
+])
+table = catalog.create_table(("analytics", "user_metrics"), schema=schema)
+
+# Append data
+df = pd.DataFrame({
+    "user_id": ["user_1", "user_2"],
+    "event_time": pd.to_datetime(["2025-01-15 10:00:00", "2025-01-15 11:00:00"], utc=True),
+    "page_views": [10, 25],
+})
+table.append(pa.Table.from_pandas(df, schema=schema))
+```
+
+Query with R2 SQL:
+```bash
+npx wrangler r2 sql query "my-bucket" "
+  SELECT user_id, SUM(page_views)
+  FROM analytics.user_metrics
+  WHERE event_time >= '2025-01-15T00:00:00Z'
+  GROUP BY user_id
+"
+```
+
+See [r2-data-catalog/patterns.md](../r2-data-catalog/patterns.md) for advanced PyIceberg patterns.
+
+## Use Cases
+
+### Log Analytics
+```sql
+-- Error rate by endpoint
+SELECT path, COUNT(*), SUM(CASE WHEN status >= 400 THEN 1 ELSE 0 END) as errors
+FROM logs.http_requests
+WHERE timestamp BETWEEN '2025-01-01T00:00:00Z' AND '2025-01-31T23:59:59Z'
+GROUP BY path ORDER BY errors DESC LIMIT 20;
+
+-- Response time stats
+SELECT method, MIN(response_time_ms), AVG(response_time_ms), MAX(response_time_ms)
+FROM logs.http_requests WHERE timestamp >= '2025-01-15T00:00:00Z' GROUP BY method;
+
+-- Traffic by status
+SELECT status, COUNT(*) FROM logs.http_requests
+WHERE timestamp >= '2025-01-15T00:00:00Z' AND method = 'GET'
+GROUP BY status ORDER BY COUNT(*) DESC;
+```
+
+### Fraud Detection
+```sql
+-- High-value transactions
+SELECT location, COUNT(*), SUM(amount), AVG(amount)
+FROM fraud.transactions WHERE transaction_timestamp >= '2025-01-01T00:00:00Z' AND amount > 1000.0
+GROUP BY location ORDER BY SUM(amount) DESC LIMIT 20;
+
+-- Flagged transactions
+SELECT merchant_category, COUNT(*), AVG(amount) FROM fraud.transactions
+WHERE is_fraud_flag = true AND transaction_timestamp >= '2025-01-01T00:00:00Z'
+GROUP BY merchant_category HAVING COUNT(*) > 10 ORDER BY COUNT(*) DESC;
+```
+
+### Business Intelligence
+```sql
+-- Sales by department
+SELECT department, SUM(revenue), AVG(revenue), COUNT(*) FROM sales.transactions
+WHERE sale_date >= '2024-01-01' GROUP BY department ORDER BY SUM(revenue) DESC LIMIT 10;
+
+-- Product performance
+SELECT category, COUNT(DISTINCT product_id), SUM(units_sold), SUM(revenue)
+FROM sales.product_sales WHERE sale_date BETWEEN '2024-10-01' AND '2024-12-31'
+GROUP BY category ORDER BY SUM(revenue) DESC;
+```
+
+## Connecting External Engines
+
+R2 Data Catalog exposes Iceberg REST API. Connect Spark, Snowflake, Trino, DuckDB, etc.
+
+```scala
+// Apache Spark example
+val spark = SparkSession.builder()
+  .config("spark.sql.catalog.my_catalog", "org.apache.iceberg.spark.SparkCatalog")
+  .config("spark.sql.catalog.my_catalog.catalog-impl", "org.apache.iceberg.rest.RESTCatalog")
+  .config("spark.sql.catalog.my_catalog.uri", "https://<account-id>.r2.cloudflarestorage.com/iceberg/my-bucket")
+  .config("spark.sql.catalog.my_catalog.token", "<token>")
+  .getOrCreate()
+
+spark.sql("SELECT * FROM my_catalog.default.my_table LIMIT 10").show()
+```
+
+See [r2-data-catalog/patterns.md](../r2-data-catalog/patterns.md) for more engines.
+
+## Performance Optimization
+
+### Partitioning
+- **Time-series:** day/hour on timestamp
+- **Geographic:** region/country
+- **Avoid:** High-cardinality keys (user_id)
+
+```python
+from pyiceberg.partitioning import PartitionSpec, PartitionField
+from pyiceberg.transforms import DayTransform
+
+PartitionSpec(PartitionField(source_id=1, field_id=1000, transform=DayTransform(), name="day"))
+```
+
+### Query Optimization
+- **Always use LIMIT** for early termination
+- **Filter on partition keys first**
+- **Multiple filters** for better pruning
+
+```sql
+-- Better: Multiple filters on partition key
+SELECT * FROM logs.requests 
+WHERE timestamp >= '2025-01-15T00:00:00Z' AND status = 404 AND method = 'GET' LIMIT 100;
+```
+
+### File Organization
+- **Pipelines roll:** Dev 10-30s, Prod 300+s
+- **Target Parquet:** 100-500MB compressed
+
+## See Also
+
+- [api.md](api.md) - SQL syntax reference
+- [gotchas.md](gotchas.md) - Limitations and troubleshooting
+- [r2-data-catalog/patterns.md](../r2-data-catalog/patterns.md) - PyIceberg advanced patterns
+- [pipelines/patterns.md](../pipelines/patterns.md) - Streaming ingestion patterns

package/skills/cloudflare/references/realtime-sfu/README.md

@@ -0,0 +1,65 @@
+# Cloudflare Realtime SFU Reference
+
+Expert guidance for building real-time audio/video/data applications using Cloudflare Realtime SFU (Selective Forwarding Unit).
+
+## Reading Order
+
+| Task | Files | ~Tokens |
+|------|-------|---------|
+| New project | README → configuration | ~1200 |
+| Implement publish/subscribe | README → api | ~1600 |
+| Add PartyTracks | patterns (PartyTracks section) | ~800 |
+| Build presence system | patterns (DO section) | ~800 |
+| Debug connection issues | gotchas | ~700 |
+| Scale to millions | patterns (Cascading section) | ~600 |
+| Add simulcast | patterns (Advanced section) | ~500 |
+| Configure TURN | configuration (TURN section) | ~400 |
+
+## In This Reference
+
+- **[configuration.md](configuration.md)** - Setup, deployment, environment variables, Wrangler config
+- **[api.md](api.md)** - Sessions, tracks, endpoints, request/response patterns
+- **[patterns.md](patterns.md)** - Architecture patterns, use cases, integration examples
+- **[gotchas.md](gotchas.md)** - Common issues, debugging, performance, security
+
+## Quick Start
+
+Cloudflare Realtime SFU: WebRTC infrastructure on global network (310+ cities). Anycast routing, no regional constraints, pub/sub model.
+
+**Core concepts:**
+- **Sessions:** WebRTC PeerConnection to Cloudflare edge
+- **Tracks:** Audio/video/data channels you publish or subscribe to
+- **No rooms:** Build presence layer yourself via track sharing (see patterns.md)
+
+**Mental model:** Your client establishes one WebRTC session, publishes tracks (audio/video), shares track IDs via your backend, others subscribe to your tracks using track IDs + your session ID.
+
+## Choose Your Approach
+
+| Approach | When to Use | Complexity |
+|----------|-------------|------------|
+| **PartyTracks** | Production apps with device switching, React | Low - Observable-based, handles reconnections |
+| **Raw API** | Custom requirements, non-browser, learning | Medium - Full control, manual WebRTC lifecycle |
+| **RealtimeKit** | End-to-end SDK with UI components | Lowest - Managed state, React hooks |
+
+**Recommendation:** Start with PartyTracks for most production applications. See patterns.md for PartyTracks examples.
+
+## SFU vs RealtimeKit
+
+- **Realtime SFU:** WebRTC infrastructure (this reference). Build your own signaling, presence, UI.
+- **RealtimeKit:** SDK layer on top of SFU. Includes React hooks, state management, UI components. Part of Cloudflare AI platform.
+
+Use SFU directly when you need custom signaling or non-React framework. Use RealtimeKit for faster development with React.
+
+## Setup
+
+Dashboard: https://dash.cloudflare.com/?to=/:account/calls
+
+Get `CALLS_APP_ID` and `CALLS_APP_SECRET` from dashboard, then see configuration.md for deployment.
+
+## See Also
+
+- [Orange Meets Demo](https://demo.orange.cloudflare.dev/)
+- [Orange Source](https://github.com/cloudflare/orange)
+- [Calls Examples](https://github.com/cloudflare/calls-examples)
+- [API Reference](https://developers.cloudflare.com/api/resources/calls/)
+- [RealtimeKit Docs](https://developers.cloudflare.com/workers-ai/realtimekit/)