memory-journal-mcp 7.7.0 → 8.0.0

package/skills/cloudflare/references/r2-data-catalog/gotchas.md

@@ -0,0 +1,170 @@
+# Gotchas & Troubleshooting
+
+Common problems → causes → solutions.
+
+## Permission Errors
+
+### 401 Unauthorized
+
+**Error:** `"401 Unauthorized"`  
+**Cause:** Token missing R2 Data Catalog permissions.  
+**Solution:** Use "Admin Read & Write" token (includes catalog + storage permissions). Test with `catalog.list_namespaces()`.
+
+### 403 Forbidden
+
+**Error:** `"403 Forbidden"` on data files  
+**Cause:** Token lacks storage permissions.  
+**Solution:** Token needs both R2 Data Catalog + R2 Storage Bucket Item permissions.
+
+### Token Rotation Issues
+
+**Error:** New token fails after rotation.  
+**Solution:** Create new token → test in staging → update prod → monitor 24h → revoke old.
+
+## Catalog URI Issues
+
+### 404 Not Found
+
+**Error:** `"404 Catalog not found"`  
+**Cause:** Catalog not enabled or wrong URI.  
+**Solution:** Run `wrangler r2 bucket catalog enable <bucket>`. URI must be HTTPS with `/iceberg/` and case-sensitive bucket name.
+
+### Wrong Warehouse
+
+**Error:** Cannot create/load tables.  
+**Cause:** Warehouse ≠ bucket name.  
+**Solution:** Set `warehouse="bucket-name"` to match bucket exactly.
+
+## Table and Schema Issues
+
+### Table/Namespace Already Exists
+
+**Error:** `"TableAlreadyExistsError"`  
+**Solution:** Use try/except to load existing or check first.
+
+### Namespace Not Found
+
+**Error:** Cannot create table.  
+**Solution:** Create namespace first: `catalog.create_namespace("ns")`
+
+### Schema Evolution Errors
+
+**Error:** `"422 Validation"` on schema update.  
+**Cause:** Incompatible change (required field, type shrink).  
+**Solution:** Only add nullable columns, compatible type widening (int→long, float→double).
+
+## Data and Query Issues
+
+### Empty Scan Results
+
+**Error:** Scan returns no data.  
+**Cause:** Incorrect filter or partition column.  
+**Solution:** Test without filter first: `table.scan().to_pandas()`. Verify partition column names.
+
+### Slow Queries
+
+**Error:** Performance degrades over time.  
+**Cause:** Too many small files.  
+**Solution:** Check file count, compact if >1000 or avg <10MB. See [api.md](api.md#compaction).
+
+### Type Mismatch
+
+**Error:** `"Cannot cast"` on append.  
+**Cause:** PyArrow types don't match Iceberg schema.  
+**Solution:** Cast to int64 (Iceberg default), not int32. Check `table.schema()`.
+
+## Compaction Issues
+
+### Compaction Issues
+
+**Problem:** File count unchanged or compaction takes hours.  
+**Cause:** Target size too large, or table too big for PyIceberg.  
+**Solution:** Only compact if avg <50MB. For >1TB tables, use Spark. Run during low-traffic periods.
+
+## Maintenance Issues
+
+### Snapshot/Orphan Issues
+
+**Problem:** Expiration fails or orphan cleanup deletes active data.  
+**Cause:** Too aggressive retention or wrong order.  
+**Solution:** Always expire snapshots first with `retain_last=10`, then cleanup orphans with 3+ day threshold.
+
+## Concurrency Issues
+
+### Concurrent Write Conflicts
+
+**Problem:** `CommitFailedException` with multiple writers.  
+**Cause:** Optimistic locking - simultaneous commits.  
+**Solution:** Add retry with exponential backoff (see [patterns.md](patterns.md#pattern-6-concurrent-writes-with-retry)).
+
+### Stale Metadata
+
+**Problem:** Old schema/data after external update.  
+**Cause:** Cached metadata.  
+**Solution:** Reload table: `table = catalog.load_table(("ns", "table"))`
+
+## Performance Optimization
+
+### Performance Tips
+
+**Scans:** Use `row_filter` and `selected_fields` to reduce data scanned.  
+**Partitions:** 100-1000 optimal. Avoid high cardinality (millions) or low (<10).  
+**Files:** Keep 100-500MB avg. Compact if <10MB or >10k files.
+
+## Limits
+
+| Resource         | Recommended | Impact if Exceeded  |
+| ---------------- | ----------- | ------------------- |
+| Tables/namespace | <10k        | Slow list ops       |
+| Files/table      | <100k       | Slow query planning |
+| Partitions/table | 100-1k      | Metadata overhead   |
+| Snapshots/table  | Expire >7d  | Metadata bloat      |
+
+## Common Error Messages Reference
+
+| Error Message                 | Likely Cause                     | Fix                                         |
+| ----------------------------- | -------------------------------- | ------------------------------------------- |
+| `401 Unauthorized`            | Missing/invalid token            | Check token has catalog+storage permissions |
+| `403 Forbidden`               | Token lacks storage permissions  | Add R2 Storage Bucket Item permission       |
+| `404 Not Found`               | Catalog not enabled or wrong URI | Run `wrangler r2 bucket catalog enable`     |
+| `409 Conflict`                | Table/namespace already exists   | Use try/except or load existing             |
+| `422 Unprocessable Entity`    | Schema validation failed         | Check type compatibility, required fields   |
+| `CommitFailedException`       | Concurrent write conflict        | Add retry logic with backoff                |
+| `NamespaceAlreadyExistsError` | Namespace exists                 | Use try/except or load existing             |
+| `NoSuchTableError`            | Table doesn't exist              | Check namespace+table name, create first    |
+| `TypeError: Cannot cast`      | PyArrow type mismatch            | Cast data to match Iceberg schema           |
+
+## Debugging Checklist
+
+When things go wrong, check in order:
+
+1. ✅ **Catalog enabled:** `npx wrangler r2 bucket catalog status <bucket>`
+2. ✅ **Token permissions:** Both R2 Data Catalog + R2 Storage in dashboard
+3. ✅ **Connection test:** `catalog.list_namespaces()` succeeds
+4. ✅ **URI format:** HTTPS, includes `/iceberg/`, correct bucket name
+5. ✅ **Warehouse name:** Matches bucket name exactly
+6. ✅ **Namespace exists:** Create before `create_table()`
+7. ✅ **Enable debug logging:** `logging.basicConfig(level=logging.DEBUG)`
+8. ✅ **PyIceberg version:** `pip install --upgrade pyiceberg` (≥0.5.0)
+9. ✅ **File health:** Compact if >1000 files or avg <10MB
+10. ✅ **Snapshot count:** Expire if >100 snapshots
+
+## Enable Debug Logging
+
+```python
+import logging
+logging.basicConfig(level=logging.DEBUG)
+# Now operations show HTTP requests/responses
+```
+
+## Resources
+
+- [Cloudflare Community](https://community.cloudflare.com/c/developers/workers/40)
+- [Cloudflare Discord](https://discord.cloudflare.com) - #r2 channel
+- [PyIceberg GitHub](https://github.com/apache/iceberg-python/issues)
+- [Apache Iceberg Slack](https://iceberg.apache.org/community/)
+
+## Next Steps
+
+- [patterns.md](patterns.md) - Working examples
+- [api.md](api.md) - API reference

package/skills/cloudflare/references/r2-data-catalog/patterns.md

@@ -0,0 +1,191 @@
+# Common Patterns
+
+Practical patterns for R2 Data Catalog with PyIceberg.
+
+## PyIceberg Connection
+
+```python
+import os
+from pyiceberg.catalog.rest import RestCatalog
+from pyiceberg.exceptions import NamespaceAlreadyExistsError
+
+catalog = RestCatalog(
+    name="r2_catalog",
+    warehouse=os.getenv("R2_WAREHOUSE"),      # bucket name
+    uri=os.getenv("R2_CATALOG_URI"),          # catalog endpoint
+    token=os.getenv("R2_TOKEN"),              # API token
+)
+
+# Create namespace (idempotent)
+try:
+    catalog.create_namespace("default")
+except NamespaceAlreadyExistsError:
+    pass
+```
+
+## Pattern 1: Log Analytics Pipeline
+
+Ingest logs incrementally, query by time/level.
+
+```python
+import pyarrow as pa
+from datetime import datetime
+from pyiceberg.schema import Schema
+from pyiceberg.types import NestedField, TimestampType, StringType, IntegerType
+from pyiceberg.partitioning import PartitionSpec, PartitionField
+from pyiceberg.transforms import DayTransform
+
+# Create partitioned table (once)
+schema = Schema(
+    NestedField(1, "timestamp", TimestampType(), required=True),
+    NestedField(2, "level", StringType(), required=True),
+    NestedField(3, "service", StringType(), required=True),
+    NestedField(4, "message", StringType(), required=False),
+)
+
+partition_spec = PartitionSpec(
+    PartitionField(source_id=1, field_id=1000, transform=DayTransform(), name="day")
+)
+
+catalog.create_namespace("logs")
+table = catalog.create_table(("logs", "app_logs"), schema=schema, partition_spec=partition_spec)
+
+# Append logs (incremental)
+data = pa.table({
+    "timestamp": [datetime(2026, 1, 27, 10, 30, 0)],
+    "level": ["ERROR"],
+    "service": ["auth-service"],
+    "message": ["Failed login"],
+})
+table.append(data)
+
+# Query by time + level (leverages partitioning)
+scan = table.scan(row_filter="level = 'ERROR' AND day = '2026-01-27'")
+errors = scan.to_pandas()
+```
+
+## Pattern 2: Time-Travel Queries
+
+```python
+from datetime import datetime, timedelta
+
+table = catalog.load_table(("logs", "app_logs"))
+
+# Query specific snapshot
+snapshot_id = table.current_snapshot().snapshot_id
+data = table.scan(snapshot_id=snapshot_id).to_pandas()
+
+# Query as of timestamp (yesterday)
+yesterday_ms = int((datetime.now() - timedelta(days=1)).timestamp() * 1000)
+data = table.scan(as_of_timestamp=yesterday_ms).to_pandas()
+```
+
+## Pattern 3: Schema Evolution
+
+```python
+from pyiceberg.types import StringType
+
+table = catalog.load_table(("users", "profiles"))
+
+with table.update_schema() as update:
+    update.add_column("email", StringType(), required=False)
+    update.rename_column("name", "full_name")
+# Old readers ignore new columns, new readers see nulls for old data
+```
+
+## Pattern 4: Partitioned Tables
+
+```python
+from pyiceberg.partitioning import PartitionSpec, PartitionField
+from pyiceberg.transforms import DayTransform, IdentityTransform
+
+# Partition by day + country
+partition_spec = PartitionSpec(
+    PartitionField(source_id=1, field_id=1000, transform=DayTransform(), name="day"),
+    PartitionField(source_id=2, field_id=1001, transform=IdentityTransform(), name="country"),
+)
+table = catalog.create_table(("events", "user_events"), schema=schema, partition_spec=partition_spec)
+
+# Queries prune partitions automatically
+scan = table.scan(row_filter="country = 'US' AND day = '2026-01-27'")
+```
+
+## Pattern 5: Table Maintenance
+
+```python
+from datetime import datetime, timedelta
+
+table = catalog.load_table(("logs", "app_logs"))
+
+# Compact → expire → cleanup (in order)
+table.rewrite_data_files(target_file_size_bytes=128 * 1024 * 1024)
+seven_days_ms = int((datetime.now() - timedelta(days=7)).timestamp() * 1000)
+table.expire_snapshots(older_than=seven_days_ms, retain_last=10)
+three_days_ms = int((datetime.now() - timedelta(days=3)).timestamp() * 1000)
+table.delete_orphan_files(older_than=three_days_ms)
+```
+
+See [api.md](api.md#table-maintenance) for detailed parameters.
+
+## Pattern 6: Concurrent Writes with Retry
+
+```python
+from pyiceberg.exceptions import CommitFailedException
+import time
+
+def append_with_retry(table, data, max_retries=3):
+    for attempt in range(max_retries):
+        try:
+            table.append(data)
+            return
+        except CommitFailedException:
+            if attempt == max_retries - 1:
+                raise
+            time.sleep(2 ** attempt)
+```
+
+## Pattern 7: Upsert Simulation
+
+```python
+import pandas as pd
+import pyarrow as pa
+
+# Read → merge → overwrite (not atomic, use Spark MERGE INTO for production)
+existing = table.scan().to_pandas()
+new_data = pd.DataFrame({"id": [1, 3], "value": [100, 300]})
+merged = pd.concat([existing, new_data]).drop_duplicates(subset=["id"], keep="last")
+table.overwrite(pa.Table.from_pandas(merged))
+```
+
+## Pattern 8: DuckDB Integration
+
+```python
+import duckdb
+
+arrow_table = table.scan().to_arrow()
+con = duckdb.connect()
+con.register("logs", arrow_table)
+result = con.execute("SELECT level, COUNT(*) FROM logs GROUP BY level").fetchdf()
+```
+
+## Pattern 9: Monitor Table Health
+
+```python
+files = table.scan().plan_files()
+avg_mb = sum(f.file_size_in_bytes for f in files) / len(files) / (1024**2)
+print(f"Files: {len(files)}, Avg: {avg_mb:.1f}MB, Snapshots: {len(table.snapshots())}")
+
+if avg_mb < 10 or len(files) > 1000:
+    print("⚠️ Needs compaction")
+```
+
+## Best Practices
+
+| Area             | Guideline                                                                      |
+| ---------------- | ------------------------------------------------------------------------------ |
+| **Partitioning** | Use day/hour for time-series; 100-1000 partitions; avoid high cardinality      |
+| **File sizes**   | Target 128-512MB; compact when avg <10MB or >10k files                         |
+| **Schema**       | Add columns as nullable (`required=False`); batch changes                      |
+| **Maintenance**  | Compact high-write daily/weekly; expire snapshots 7-30d; cleanup orphans after |
+| **Concurrency**  | Reads automatic; writes to different partitions safe; retry same partition     |
+| **Performance**  | Filter on partitions; select only needed columns; batch appends 100MB+         |

package/skills/cloudflare/references/r2-sql/README.md

@@ -0,0 +1,138 @@
+# Cloudflare R2 SQL Skill Reference
+
+Expert guidance for Cloudflare R2 SQL - serverless distributed query engine for Apache Iceberg tables.
+
+## Reading Order
+
+**New to R2 SQL?** Start here:
+
+1. Read "What is R2 SQL?" and "When to Use" below
+2. [configuration.md](configuration.md) - Enable catalog, create tokens
+3. [patterns.md](patterns.md) - Wrangler CLI and integration examples
+4. [api.md](api.md) - SQL syntax and query reference
+5. [gotchas.md](gotchas.md) - Limitations and troubleshooting
+
+**Quick reference?** Jump to:
+
+- [Run a query via Wrangler](patterns.md#wrangler-cli-query)
+- [SQL syntax reference](api.md#sql-syntax)
+- [ORDER BY limitations](gotchas.md#order-by-limitations)
+
+## What is R2 SQL?
+
+R2 SQL is Cloudflare's **serverless distributed analytics query engine** for querying Apache Iceberg tables in R2 Data Catalog. Features:
+
+- **Serverless** - No clusters to manage, no infrastructure
+- **Distributed** - Leverages Cloudflare's global network for parallel execution
+- **SQL interface** - Familiar SQL syntax for analytics queries
+- **Zero egress fees** - Query from any cloud/region without data transfer costs
+- **Open beta** - Free during beta (standard R2 storage costs apply)
+
+### What is Apache Iceberg?
+
+Open table format for large-scale analytics datasets in object storage:
+
+- **ACID transactions** - Safe concurrent reads/writes
+- **Metadata optimization** - Fast queries without full table scans
+- **Schema evolution** - Add/rename/drop columns without rewrites
+- **Partitioning** - Organize data for efficient pruning
+
+## When to Use
+
+**Use R2 SQL for:**
+
+- **Log analytics** - Query application/system logs with WHERE filters and aggregations
+- **BI dashboards** - Generate reports from large analytical datasets
+- **Fraud detection** - Analyze transaction patterns with GROUP BY/HAVING
+- **Multi-cloud analytics** - Query data from any cloud without egress fees
+- **Ad-hoc exploration** - Run SQL queries on Iceberg tables via Wrangler CLI
+
+**Don't use R2 SQL for:**
+
+- **Workers/Pages runtime** - R2 SQL has no Workers binding, use HTTP API from external systems
+- **Real-time queries (<100ms)** - Optimized for analytical batch queries, not OLTP
+- **Complex joins/CTEs** - Limited SQL feature set (no JOINs, subqueries, CTEs currently)
+- **Small datasets (<1GB)** - Setup overhead not justified
+
+## Decision Tree: Need to Query R2 Data?
+
+```
+Do you need to query structured data in R2?
+├─ YES, data is in Iceberg tables
+│  ├─ Need SQL interface? → Use R2 SQL (this reference)
+│  ├─ Need Python API? → See r2-data-catalog reference (PyIceberg)
+│  └─ Need other engine? → See r2-data-catalog reference (Spark, Trino, etc.)
+│
+├─ YES, but not in Iceberg format
+│  ├─ Streaming data? → Use Pipelines to write to Data Catalog, then R2 SQL
+│  └─ Static files? → Use PyIceberg to create Iceberg tables, then R2 SQL
+│
+└─ NO, just need object storage → Use R2 reference (not R2 SQL)
+```
+
+## Architecture Overview
+
+**Query Planner:**
+
+- Top-down metadata investigation with multi-layer pruning
+- Partition-level, column-level, and row-group pruning
+- Streaming pipeline - execution starts before planning completes
+- Early termination with LIMIT - stops when result complete
+
+**Query Execution:**
+
+- Coordinator distributes work to workers across Cloudflare network
+- Workers run Apache DataFusion for parallel query execution
+- Parquet column pruning - reads only required columns
+- Ranged reads from R2 for efficiency
+
+**Aggregation Strategies:**
+
+- Scatter-gather - simple aggregations (SUM, COUNT, AVG)
+- Shuffling - ORDER BY/HAVING on aggregates via hash partitioning
+
+## Quick Start
+
+```bash
+# 1. Enable R2 Data Catalog on bucket
+npx wrangler r2 bucket catalog enable my-bucket
+
+# 2. Create API token (Admin Read & Write)
+# Dashboard: R2 → Manage API tokens → Create API token
+
+# 3. Set environment variable
+export WRANGLER_R2_SQL_AUTH_TOKEN=<your-token>
+
+# 4. Run query
+npx wrangler r2 sql query "my-bucket" "SELECT * FROM default.my_table LIMIT 10"
+```
+
+## Important Limitations
+
+**CRITICAL: No Workers Binding**
+
+- R2 SQL cannot be called directly from Workers/Pages code
+- For programmatic access, use HTTP API from external systems
+- Or query via PyIceberg, Spark, etc. (see r2-data-catalog reference)
+
+**SQL Feature Set:**
+
+- No JOINs, CTEs, subqueries, window functions
+- ORDER BY supports aggregation columns (not just partition keys)
+- LIMIT max 10,000 (default 500)
+- See [gotchas.md](gotchas.md) for complete limitations
+
+## In This Reference
+
+- **[configuration.md](configuration.md)** - Enable catalog, create API tokens
+- **[api.md](api.md)** - SQL syntax, functions, operators, data types
+- **[patterns.md](patterns.md)** - Wrangler CLI, HTTP API, Pipelines, PyIceberg
+- **[gotchas.md](gotchas.md)** - Limitations, troubleshooting, performance tips
+
+## See Also
+
+- [r2-data-catalog](../r2-data-catalog/) - PyIceberg, REST API, external engines
+- [pipelines](../pipelines/) - Streaming ingestion to Iceberg tables
+- [r2](../r2/) - R2 object storage fundamentals
+- [Cloudflare R2 SQL Docs](https://developers.cloudflare.com/r2-sql/)
+- [R2 SQL Deep Dive Blog](https://blog.cloudflare.com/r2-sql-deep-dive/)