@mohanscodex/spectra-code 0.4.9 → 0.5.0

package/skills/cloudflare-deploy/references/r2-data-catalog/patterns.md

@@ -0,0 +1,191 @@
+# Common Patterns
+
+Practical patterns for R2 Data Catalog with PyIceberg.
+
+## PyIceberg Connection
+
+```python
+import os
+from pyiceberg.catalog.rest import RestCatalog
+from pyiceberg.exceptions import NamespaceAlreadyExistsError
+
+catalog = RestCatalog(
+    name="r2_catalog",
+    warehouse=os.getenv("R2_WAREHOUSE"),      # bucket name
+    uri=os.getenv("R2_CATALOG_URI"),          # catalog endpoint
+    token=os.getenv("R2_TOKEN"),              # API token
+)
+
+# Create namespace (idempotent)
+try:
+    catalog.create_namespace("default")
+except NamespaceAlreadyExistsError:
+    pass
+```
+
+## Pattern 1: Log Analytics Pipeline
+
+Ingest logs incrementally, query by time/level.
+
+```python
+import pyarrow as pa
+from datetime import datetime
+from pyiceberg.schema import Schema
+from pyiceberg.types import NestedField, TimestampType, StringType, IntegerType
+from pyiceberg.partitioning import PartitionSpec, PartitionField
+from pyiceberg.transforms import DayTransform
+
+# Create partitioned table (once)
+schema = Schema(
+    NestedField(1, "timestamp", TimestampType(), required=True),
+    NestedField(2, "level", StringType(), required=True),
+    NestedField(3, "service", StringType(), required=True),
+    NestedField(4, "message", StringType(), required=False),
+)
+
+partition_spec = PartitionSpec(
+    PartitionField(source_id=1, field_id=1000, transform=DayTransform(), name="day")
+)
+
+catalog.create_namespace("logs")
+table = catalog.create_table(("logs", "app_logs"), schema=schema, partition_spec=partition_spec)
+
+# Append logs (incremental)
+data = pa.table({
+    "timestamp": [datetime(2026, 1, 27, 10, 30, 0)],
+    "level": ["ERROR"],
+    "service": ["auth-service"],
+    "message": ["Failed login"],
+})
+table.append(data)
+
+# Query by time + level (leverages partitioning)
+scan = table.scan(row_filter="level = 'ERROR' AND day = '2026-01-27'")
+errors = scan.to_pandas()
+```
+
+## Pattern 2: Time-Travel Queries
+
+```python
+from datetime import datetime, timedelta
+
+table = catalog.load_table(("logs", "app_logs"))
+
+# Query specific snapshot
+snapshot_id = table.current_snapshot().snapshot_id
+data = table.scan(snapshot_id=snapshot_id).to_pandas()
+
+# Query as of timestamp (yesterday)
+yesterday_ms = int((datetime.now() - timedelta(days=1)).timestamp() * 1000)
+data = table.scan(as_of_timestamp=yesterday_ms).to_pandas()
+```
+
+## Pattern 3: Schema Evolution
+
+```python
+from pyiceberg.types import StringType
+
+table = catalog.load_table(("users", "profiles"))
+
+with table.update_schema() as update:
+    update.add_column("email", StringType(), required=False)
+    update.rename_column("name", "full_name")
+# Old readers ignore new columns, new readers see nulls for old data
+```
+
+## Pattern 4: Partitioned Tables
+
+```python
+from pyiceberg.partitioning import PartitionSpec, PartitionField
+from pyiceberg.transforms import DayTransform, IdentityTransform
+
+# Partition by day + country
+partition_spec = PartitionSpec(
+    PartitionField(source_id=1, field_id=1000, transform=DayTransform(), name="day"),
+    PartitionField(source_id=2, field_id=1001, transform=IdentityTransform(), name="country"),
+)
+table = catalog.create_table(("events", "user_events"), schema=schema, partition_spec=partition_spec)
+
+# Queries prune partitions automatically
+scan = table.scan(row_filter="country = 'US' AND day = '2026-01-27'")
+```
+
+## Pattern 5: Table Maintenance
+
+```python
+from datetime import datetime, timedelta
+
+table = catalog.load_table(("logs", "app_logs"))
+
+# Compact → expire → cleanup (in order)
+table.rewrite_data_files(target_file_size_bytes=128 * 1024 * 1024)
+seven_days_ms = int((datetime.now() - timedelta(days=7)).timestamp() * 1000)
+table.expire_snapshots(older_than=seven_days_ms, retain_last=10)
+three_days_ms = int((datetime.now() - timedelta(days=3)).timestamp() * 1000)
+table.delete_orphan_files(older_than=three_days_ms)
+```
+
+See [api.md](api.md#table-maintenance) for detailed parameters.
+
+## Pattern 6: Concurrent Writes with Retry
+
+```python
+from pyiceberg.exceptions import CommitFailedException
+import time
+
+def append_with_retry(table, data, max_retries=3):
+    for attempt in range(max_retries):
+        try:
+            table.append(data)
+            return
+        except CommitFailedException:
+            if attempt == max_retries - 1:
+                raise
+            time.sleep(2 ** attempt)
+```
+
+## Pattern 7: Upsert Simulation
+
+```python
+import pandas as pd
+import pyarrow as pa
+
+# Read → merge → overwrite (not atomic, use Spark MERGE INTO for production)
+existing = table.scan().to_pandas()
+new_data = pd.DataFrame({"id": [1, 3], "value": [100, 300]})
+merged = pd.concat([existing, new_data]).drop_duplicates(subset=["id"], keep="last")
+table.overwrite(pa.Table.from_pandas(merged))
+```
+
+## Pattern 8: DuckDB Integration
+
+```python
+import duckdb
+
+arrow_table = table.scan().to_arrow()
+con = duckdb.connect()
+con.register("logs", arrow_table)
+result = con.execute("SELECT level, COUNT(*) FROM logs GROUP BY level").fetchdf()
+```
+
+## Pattern 9: Monitor Table Health
+
+```python
+files = table.scan().plan_files()
+avg_mb = sum(f.file_size_in_bytes for f in files) / len(files) / (1024**2)
+print(f"Files: {len(files)}, Avg: {avg_mb:.1f}MB, Snapshots: {len(table.snapshots())}")
+
+if avg_mb < 10 or len(files) > 1000:
+    print("⚠️ Needs compaction")
+```
+
+## Best Practices
+
+| Area | Guideline |
+|------|-----------|
+| **Partitioning** | Use day/hour for time-series; 100-1000 partitions; avoid high cardinality |
+| **File sizes** | Target 128-512MB; compact when avg <10MB or >10k files |
+| **Schema** | Add columns as nullable (`required=False`); batch changes |
+| **Maintenance** | Compact high-write daily/weekly; expire snapshots 7-30d; cleanup orphans after |
+| **Concurrency** | Reads automatic; writes to different partitions safe; retry same partition |
+| **Performance** | Filter on partitions; select only needed columns; batch appends 100MB+ |

package/skills/cloudflare-deploy/references/r2-sql/README.md

@@ -0,0 +1,128 @@
+# Cloudflare R2 SQL Skill Reference
+
+Expert guidance for Cloudflare R2 SQL - serverless distributed query engine for Apache Iceberg tables.
+
+## Reading Order
+
+**New to R2 SQL?** Start here:
+1. Read "What is R2 SQL?" and "When to Use" below
+2. [configuration.md](configuration.md) - Enable catalog, create tokens
+3. [patterns.md](patterns.md) - Wrangler CLI and integration examples
+4. [api.md](api.md) - SQL syntax and query reference
+5. [gotchas.md](gotchas.md) - Limitations and troubleshooting
+
+**Quick reference?** Jump to:
+- [Run a query via Wrangler](patterns.md#wrangler-cli-query)
+- [SQL syntax reference](api.md#sql-syntax)
+- [ORDER BY limitations](gotchas.md#order-by-limitations)
+
+## What is R2 SQL?
+
+R2 SQL is Cloudflare's **serverless distributed analytics query engine** for querying Apache Iceberg tables in R2 Data Catalog. Features:
+
+- **Serverless** - No clusters to manage, no infrastructure
+- **Distributed** - Leverages Cloudflare's global network for parallel execution
+- **SQL interface** - Familiar SQL syntax for analytics queries
+- **Zero egress fees** - Query from any cloud/region without data transfer costs
+- **Open beta** - Free during beta (standard R2 storage costs apply)
+
+### What is Apache Iceberg?
+
+Open table format for large-scale analytics datasets in object storage:
+- **ACID transactions** - Safe concurrent reads/writes
+- **Metadata optimization** - Fast queries without full table scans
+- **Schema evolution** - Add/rename/drop columns without rewrites
+- **Partitioning** - Organize data for efficient pruning
+
+## When to Use
+
+**Use R2 SQL for:**
+- **Log analytics** - Query application/system logs with WHERE filters and aggregations
+- **BI dashboards** - Generate reports from large analytical datasets
+- **Fraud detection** - Analyze transaction patterns with GROUP BY/HAVING
+- **Multi-cloud analytics** - Query data from any cloud without egress fees
+- **Ad-hoc exploration** - Run SQL queries on Iceberg tables via Wrangler CLI
+
+**Don't use R2 SQL for:**
+- **Workers/Pages runtime** - R2 SQL has no Workers binding, use HTTP API from external systems
+- **Real-time queries (<100ms)** - Optimized for analytical batch queries, not OLTP
+- **Complex joins/CTEs** - Limited SQL feature set (no JOINs, subqueries, CTEs currently)
+- **Small datasets (<1GB)** - Setup overhead not justified
+
+## Decision Tree: Need to Query R2 Data?
+
+```
+Do you need to query structured data in R2?
+├─ YES, data is in Iceberg tables
+│  ├─ Need SQL interface? → Use R2 SQL (this reference)
+│  ├─ Need Python API? → See r2-data-catalog reference (PyIceberg)
+│  └─ Need other engine? → See r2-data-catalog reference (Spark, Trino, etc.)
+│
+├─ YES, but not in Iceberg format
+│  ├─ Streaming data? → Use Pipelines to write to Data Catalog, then R2 SQL
+│  └─ Static files? → Use PyIceberg to create Iceberg tables, then R2 SQL
+│
+└─ NO, just need object storage → Use R2 reference (not R2 SQL)
+```
+
+## Architecture Overview
+
+**Query Planner:**
+- Top-down metadata investigation with multi-layer pruning
+- Partition-level, column-level, and row-group pruning
+- Streaming pipeline - execution starts before planning completes
+- Early termination with LIMIT - stops when result complete
+
+**Query Execution:**
+- Coordinator distributes work to workers across Cloudflare network
+- Workers run Apache DataFusion for parallel query execution
+- Parquet column pruning - reads only required columns
+- Ranged reads from R2 for efficiency
+
+**Aggregation Strategies:**
+- Scatter-gather - simple aggregations (SUM, COUNT, AVG)
+- Shuffling - ORDER BY/HAVING on aggregates via hash partitioning
+
+## Quick Start
+
+```bash
+# 1. Enable R2 Data Catalog on bucket
+npx wrangler r2 bucket catalog enable my-bucket
+
+# 2. Create API token (Admin Read & Write)
+# Dashboard: R2 → Manage API tokens → Create API token
+
+# 3. Set environment variable
+export WRANGLER_R2_SQL_AUTH_TOKEN=<your-token>
+
+# 4. Run query
+npx wrangler r2 sql query "my-bucket" "SELECT * FROM default.my_table LIMIT 10"
+```
+
+## Important Limitations
+
+**CRITICAL: No Workers Binding**
+- R2 SQL cannot be called directly from Workers/Pages code
+- For programmatic access, use HTTP API from external systems
+- Or query via PyIceberg, Spark, etc. (see r2-data-catalog reference)
+
+**SQL Feature Set:**
+- No JOINs, CTEs, subqueries, window functions
+- ORDER BY supports aggregation columns (not just partition keys)
+- LIMIT max 10,000 (default 500)
+- See [gotchas.md](gotchas.md) for complete limitations
+
+## In This Reference
+
+- **[configuration.md](configuration.md)** - Enable catalog, create API tokens
+- **[api.md](api.md)** - SQL syntax, functions, operators, data types
+- **[patterns.md](patterns.md)** - Wrangler CLI, HTTP API, Pipelines, PyIceberg
+- **[gotchas.md](gotchas.md)** - Limitations, troubleshooting, performance tips
+
+## See Also
+
+- [r2-data-catalog](../r2-data-catalog/) - PyIceberg, REST API, external engines
+- [pipelines](../pipelines/) - Streaming ingestion to Iceberg tables
+- [r2](../r2/) - R2 object storage fundamentals
+- [Cloudflare R2 SQL Docs](https://developers.cloudflare.com/r2-sql/)
+- [R2 SQL Deep Dive Blog](https://blog.cloudflare.com/r2-sql-deep-dive/)

package/skills/cloudflare-deploy/references/r2-sql/api.md

@@ -0,0 +1,158 @@
+# R2 SQL API Reference
+
+SQL syntax, functions, operators, and data types for R2 SQL queries.
+
+## SQL Syntax
+
+```sql
+SELECT column_list | aggregation_function
+FROM [namespace.]table_name
+WHERE conditions
+[GROUP BY column_list]
+[HAVING conditions]
+[ORDER BY column | aggregation_function [DESC | ASC]]
+[LIMIT number]
+```
+
+## Schema Discovery
+
+```sql
+SHOW DATABASES;           -- List namespaces
+SHOW NAMESPACES;          -- Alias for SHOW DATABASES
+SHOW SCHEMAS;             -- Alias for SHOW DATABASES
+SHOW TABLES IN namespace; -- List tables in namespace
+DESCRIBE namespace.table; -- Show table schema, partition keys
+```
+
+## SELECT Clause
+
+```sql
+-- All columns
+SELECT * FROM logs.http_requests;
+
+-- Specific columns
+SELECT user_id, timestamp, status FROM logs.http_requests;
+```
+
+**Limitations:** No column aliases, expressions, or nested column access
+
+## WHERE Clause
+
+### Operators
+
+| Operator | Example |
+|----------|---------|
+| `=`, `!=`, `<`, `<=`, `>`, `>=` | `status = 200` |
+| `LIKE` | `user_agent LIKE '%Chrome%'` |
+| `BETWEEN` | `timestamp BETWEEN '2025-01-01T00:00:00Z' AND '2025-01-31T23:59:59Z'` |
+| `IS NULL`, `IS NOT NULL` | `email IS NOT NULL` |
+| `AND`, `OR` | `status = 200 AND method = 'GET'` |
+
+Use parentheses for precedence: `(status = 404 OR status = 500) AND method = 'POST'`
+
+## Aggregation Functions
+
+| Function | Description |
+|----------|-------------|
+| `COUNT(*)` | Count all rows |
+| `COUNT(column)` | Count non-null values |
+| `COUNT(DISTINCT column)` | Count unique values |
+| `SUM(column)`, `AVG(column)` | Numeric aggregations |
+| `MIN(column)`, `MAX(column)` | Min/max values |
+
+```sql
+-- Multiple aggregations with GROUP BY
+SELECT region, COUNT(*), SUM(amount), AVG(amount)
+FROM sales.transactions
+WHERE sale_date >= '2024-01-01'
+GROUP BY region;
+```
+
+## HAVING Clause
+
+Filter aggregated results (after GROUP BY):
+
+```sql
+SELECT category, SUM(amount)
+FROM sales.transactions
+GROUP BY category
+HAVING SUM(amount) > 10000;
+```
+
+## ORDER BY Clause
+
+Sort results by:
+- **Partition key columns** - Always supported
+- **Aggregation functions** - Supported via shuffle strategy
+
+```sql
+-- Order by partition key
+SELECT * FROM logs.requests ORDER BY timestamp DESC LIMIT 100;
+
+-- Order by aggregation (repeat function, aliases not supported)
+SELECT region, SUM(amount)
+FROM sales.transactions
+GROUP BY region
+ORDER BY SUM(amount) DESC;
+```
+
+**Limitations:** Cannot order by non-partition columns. See [gotchas.md](gotchas.md#order-by-limitations)
+
+## LIMIT Clause
+
+```sql
+SELECT * FROM logs.requests LIMIT 100;
+```
+
+| Setting | Value |
+|---------|-------|
+| Min | 1 |
+| Max | 10,000 |
+| Default | 500 |
+
+**Always use LIMIT** to enable early termination optimization.
+
+## Data Types
+
+| Type | SQL Literal | Example |
+|------|-------------|---------|
+| `integer` | Unquoted number | `42`, `-10` |
+| `float` | Decimal number | `3.14`, `-0.5` |
+| `string` | Single quotes | `'hello'`, `'GET'` |
+| `boolean` | Keyword | `true`, `false` |
+| `timestamp` | RFC3339 string | `'2025-01-01T00:00:00Z'` |
+| `date` | ISO 8601 date | `'2025-01-01'` |
+
+### Type Safety
+
+- Quote strings with single quotes: `'value'`
+- Timestamps must be RFC3339: `'2025-01-01T00:00:00Z'` (include timezone)
+- Dates must be ISO 8601: `'2025-01-01'` (YYYY-MM-DD)
+- No implicit conversions
+
+```sql
+-- ✅ Correct
+WHERE status = 200 AND method = 'GET' AND timestamp > '2025-01-01T00:00:00Z'
+
+-- ❌ Wrong
+WHERE status = '200'              -- string instead of integer
+WHERE timestamp > '2025-01-01'    -- missing time/timezone
+WHERE method = GET                -- unquoted string
+```
+
+## Query Result Format
+
+JSON array of objects:
+
+```json
+[
+  {"user_id": "user_123", "timestamp": "2025-01-15T10:30:00Z", "status": 200},
+  {"user_id": "user_456", "timestamp": "2025-01-15T10:31:00Z", "status": 404}
+]
+```
+
+## See Also
+
+- [patterns.md](patterns.md) - Query examples and use cases
+- [gotchas.md](gotchas.md) - SQL limitations and error handling
+- [configuration.md](configuration.md) - Setup and authentication

package/skills/cloudflare-deploy/references/r2-sql/configuration.md

@@ -0,0 +1,147 @@
+# R2 SQL Configuration
+
+Setup and configuration for R2 SQL queries.
+
+## Prerequisites
+
+- R2 bucket with Data Catalog enabled
+- API token with R2 permissions
+- Wrangler CLI installed (for CLI queries)
+
+## Enable R2 Data Catalog
+
+R2 SQL queries Apache Iceberg tables in R2 Data Catalog. Must enable catalog on bucket first.
+
+### Via Wrangler CLI
+
+```bash
+npx wrangler r2 bucket catalog enable <bucket-name>
+```
+
+Output includes:
+- **Warehouse name** - Typically same as bucket name
+- **Catalog URI** - REST endpoint for catalog operations
+
+Example output:
+```
+Catalog enabled successfully
+Warehouse: my-bucket
+Catalog URI: https://abc123.r2.cloudflarestorage.com/iceberg/my-bucket
+```
+
+### Via Dashboard
+
+1. Navigate to **R2 Object Storage** → Select your bucket
+2. Click **Settings** tab
+3. Scroll to **R2 Data Catalog** section
+4. Click **Enable**
+5. Note the **Catalog URI** and **Warehouse** name
+
+**Important:** Enabling catalog creates metadata directories in bucket but does not modify existing objects.
+
+## Create API Token
+
+R2 SQL requires API token with R2 permissions.
+
+### Required Permission
+
+**R2 Admin Read & Write** (includes R2 SQL Read permission)
+
+### Via Dashboard
+
+1. Navigate to **R2 Object Storage**
+2. Click **Manage API tokens** (top right)
+3. Click **Create API token**
+4. Select **Admin Read & Write** permission
+5. Click **Create API Token**
+6. **Copy token value** - shown only once
+
+### Permission Scope
+
+| Permission | Grants Access To |
+|------------|------------------|
+| R2 Admin Read & Write | R2 storage operations + R2 SQL queries + Data Catalog operations |
+| R2 SQL Read | SQL queries only (no storage writes) |
+
+**Note:** R2 SQL Read permission not yet available via Dashboard - use Admin Read & Write.
+
+## Configure Environment
+
+### Wrangler CLI
+
+Set environment variable for Wrangler to use:
+
+```bash
+export WRANGLER_R2_SQL_AUTH_TOKEN=<your-token>
+```
+
+Or create `.env` file in project directory:
+
+```
+WRANGLER_R2_SQL_AUTH_TOKEN=<your-token>
+```
+
+Wrangler automatically loads `.env` file when running commands.
+
+### HTTP API
+
+For programmatic access (non-Wrangler), pass token in Authorization header:
+
+```bash
+curl -X POST https://api.cloudflare.com/client/v4/accounts/{account_id}/r2/sql/query \
+  -H "Authorization: Bearer <your-token>" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "warehouse": "my-bucket",
+    "query": "SELECT * FROM default.my_table LIMIT 10"
+  }'
+```
+
+**Note:** HTTP API endpoint URL may vary - see [patterns.md](patterns.md#http-api-query) for current endpoint.
+
+## Verify Setup
+
+Test configuration by querying system tables:
+
+```bash
+# List namespaces
+npx wrangler r2 sql query "my-bucket" "SHOW DATABASES"
+
+# List tables in namespace
+npx wrangler r2 sql query "my-bucket" "SHOW TABLES IN default"
+```
+
+If successful, returns JSON array of results.
+
+## Troubleshooting
+
+### "Token authentication failed"
+
+**Cause:** Invalid or missing token
+
+**Solution:**
+- Verify `WRANGLER_R2_SQL_AUTH_TOKEN` environment variable set
+- Check token has Admin Read & Write permission
+- Create new token if expired
+
+### "Catalog not enabled on bucket"
+
+**Cause:** Data Catalog not enabled
+
+**Solution:**
+- Run `npx wrangler r2 bucket catalog enable <bucket-name>`
+- Or enable via Dashboard (R2 → bucket → Settings → R2 Data Catalog)
+
+### "Permission denied"
+
+**Cause:** Token lacks required permissions
+
+**Solution:**
+- Verify token has **Admin Read & Write** permission
+- Create new token with correct permissions
+
+## See Also
+
+- [r2-data-catalog/configuration.md](../r2-data-catalog/configuration.md) - Detailed token setup and PyIceberg connection
+- [patterns.md](patterns.md) - Query examples using configuration
+- [gotchas.md](gotchas.md) - Common configuration errors