memory-journal-mcp 8.0.1 → 9.1.0

package/skills/cloudflare/references/r2-sql/SKILL.md.backup

@@ -1,512 +0,0 @@
-# Cloudflare R2 SQL Skill
-
-Guide for using Cloudflare R2 SQL - serverless distributed query engine for Apache Iceberg tables in R2 Data Catalog.
-
-## Overview
-
-R2 SQL is Cloudflare's serverless distributed analytics query engine for querying Apache Iceberg tables in R2 Data Catalog. Features:
-- Serverless - no clusters to manage
-- Distributed - leverages Cloudflare's global network
-- Zero egress fees - query from any cloud/region
-- Open beta - free during beta (standard R2 storage costs apply)
-
-## Core Concepts
-
-### Apache Iceberg Table Format
-- Open table format for large-scale analytics datasets
-- ACID transactions for reliable concurrent reads/writes
-- Schema evolution - add/rename/drop columns without rewriting data
-- Optimized metadata - avoids full table scans via indexed metadata
-- Supported by Spark, Trino, Snowflake, DuckDB, ClickHouse, PyIceberg
-
-### R2 Data Catalog
-- Managed Apache Iceberg catalog built into R2 bucket
-- Exposes standard Iceberg REST catalog interface
-- Single source of truth for table metadata
-- Tracks table state via immutable snapshots
-- Supports multiple query engines safely accessing same tables
-
-### Architecture
-**Query Planner**:
-- Top-down metadata investigation
-- Multi-layer pruning (partition-level, column-level, row-group level)
-- Streaming pipeline - execution starts before planning completes
-- Early termination - stops when result complete without full scan
-- Uses partition stats and column stats (min/max, null counts)
-
-**Query Execution**:
-- Coordinator distributes work to workers across Cloudflare network
-- Workers run Apache DataFusion for parallel query execution
-- Arrow IPC format for inter-process communication
-- Parquet column pruning - reads only required columns
-- Ranged reads from R2 for efficiency
-
-**Aggregation Strategies**:
-- Scatter-gather - for simple aggregations (sum, count, avg)
-- Shuffling - for ORDER BY/HAVING on aggregates via hash partitioning
-
-## Setup & Configuration
-
-### 1. Enable R2 Data Catalog
-
-CLI:
-```bash
-npx wrangler r2 bucket catalog enable <bucket-name>
-```
-
-Note the Warehouse name and Catalog URI from output.
-
-Dashboard:
-1. R2 Object Storage → Select bucket
-2. Settings tab → R2 Data Catalog → Enable
-3. Note Catalog URI and Warehouse name
-
-### 2. Create API Token
-
-Required permissions: R2 Admin Read & Write (includes R2 SQL Read)
-
-Dashboard:
-1. R2 Object Storage → Manage API tokens
-2. Create API token → Admin Read & Write
-3. Save token value
-
-### 3. Configure Environment
-
-```bash
-export WRANGLER_R2_SQL_AUTH_TOKEN=<your-token>
-```
-
-Or `.env` file:
-```
-WRANGLER_R2_SQL_AUTH_TOKEN=<your-token>
-```
-
-## Common Code Patterns
-
-### Wrangler CLI Query
-
-```bash
-npx wrangler r2 sql query "<warehouse-name>" "
-  SELECT * 
-  FROM namespace.table_name 
-  WHERE condition 
-  LIMIT 10"
-```
-
-### PyIceberg Setup
-
-```python
-from pyiceberg.catalog.rest import RestCatalog
-
-catalog = RestCatalog(
-    name="my_catalog",
-    warehouse="<WAREHOUSE>",
-    uri="<CATALOG_URI>",
-    token="<TOKEN>",
-)
-
-# Create namespace
-catalog.create_namespace_if_not_exists("default")
-```
-
-### Create Table
-
-```python
-import pyarrow as pa
-
-# Define schema
-df = pa.table({
-    "id": [1, 2, 3],
-    "name": ["Alice", "Bob", "Charlie"],
-    "score": [80.0, 92.5, 88.0],
-})
-
-# Create table
-table = catalog.create_table(
-    ("default", "people"),
-    schema=df.schema,
-)
-```
-
-### Append Data
-
-```python
-table.append(df)
-```
-
-### Query Table
-
-```python
-# Scan and convert to Pandas
-scanned = table.scan().to_arrow()
-print(scanned.to_pandas())
-```
-
-## SQL Reference
-
-### Query Structure
-
-```sql
-SELECT column_list | aggregation_function
-FROM table_name
-WHERE conditions
-[GROUP BY column_list]
-[HAVING conditions]
-[ORDER BY partition_key [DESC | ASC]]
-[LIMIT number]
-```
-
-### Schema Discovery
-
-```sql
--- List namespaces
-SHOW DATABASES;
-SHOW NAMESPACES;
-
--- List tables
-SHOW TABLES IN namespace_name;
-
--- Describe table
-DESCRIBE namespace_name.table_name;
-```
-
-### SELECT Patterns
-
-```sql
--- All columns
-SELECT * FROM ns.table;
-
--- Specific columns
-SELECT user_id, timestamp, status FROM ns.table;
-
--- With conditions
-SELECT * FROM ns.table 
-WHERE timestamp BETWEEN '2025-01-01T00:00:00Z' AND '2025-01-31T23:59:59Z'
-  AND status = 200
-LIMIT 100;
-
--- Complex conditions
-SELECT * FROM ns.table 
-WHERE (status = 404 OR status = 500) 
-  AND method = 'POST'
-  AND user_agent IS NOT NULL
-ORDER BY timestamp DESC;
-```
-
-### Aggregations
-
-Supported functions: COUNT(*), SUM(col), AVG(col), MIN(col), MAX(col)
-
-```sql
--- Count by group
-SELECT department, COUNT(*)
-FROM ns.sales_data
-GROUP BY department;
-
--- Multiple aggregates
-SELECT region, MIN(price), MAX(price), AVG(price)
-FROM ns.products
-GROUP BY region
-ORDER BY AVG(price) DESC;
-
--- With HAVING filter
-SELECT category, SUM(amount)
-FROM ns.sales
-WHERE sale_date >= '2024-01-01'
-GROUP BY category
-HAVING SUM(amount) > 10000
-LIMIT 10;
-```
-
-### Data Types
-
-| Type | Description | Example |
-|------|-------------|---------|
-| integer | Whole numbers | 1, 42, -10 |
-| float | Decimals | 1.5, 3.14 |
-| string | Text (quoted) | 'hello', 'GET' |
-| boolean | true/false | true, false |
-| timestamp | RFC3339 | '2025-01-01T00:00:00Z' |
-| date | YYYY-MM-DD | '2025-01-01' |
-
-### Operators
-
-Comparison: =, !=, <, <=, >, >=, LIKE, BETWEEN, IS NULL, IS NOT NULL
-Logical: AND (higher precedence), OR (lower precedence)
-
-### ORDER BY Limitations
-
-**CRITICAL**: ORDER BY only supports partition key columns
-
-```sql
--- Valid if timestamp is partition key
-SELECT * FROM ns.logs ORDER BY timestamp DESC LIMIT 100;
-
--- Invalid if column not in partition key
-SELECT * FROM ns.logs ORDER BY user_id;  -- ERROR
-```
-
-### LIMIT Defaults
-
-- Range: 1 to 10,000
-- Default: 500 if not specified
-
-## Pipelines Integration
-
-### Create Pipeline with Data Catalog Sink
-
-Schema file (`schema.json`):
-```json
-{
-  "fields": [
-    {"name": "user_id", "type": "string", "required": true},
-    {"name": "event_type", "type": "string", "required": true},
-    {"name": "amount", "type": "float64", "required": false}
-  ]
-}
-```
-
-Setup:
-```bash
-npx wrangler pipelines setup
-```
-
-Configuration:
-- Pipeline name: ecommerce
-- Enable HTTP endpoint: yes
-- Schema: Load from file → schema.json
-- Destination: Data Catalog Table
-- R2 bucket: your-bucket
-- Namespace: default
-- Table name: events
-- Catalog token: <your-token>
-- Compression: zstd
-- Roll file time: 10 seconds (dev), 300+ (prod)
-
-### Send Data to Pipeline
-
-```bash
-curl -X POST https://{stream-id}.ingest.cloudflare.com \
-  -H "Content-Type: application/json" \
-  -d '[
-    {
-      "user_id": "user_123",
-      "event_type": "purchase",
-      "amount": 29.99
-    }
-  ]'
-```
-
-## Common Use Cases
-
-### Log Analytics
-- Ingest logs via Pipelines to Iceberg table
-- Partition by day(timestamp) for efficient queries
-- Query specific time ranges with automatic pruning
-- Aggregate by status codes, endpoints, user agents
-
-```sql
-SELECT status, COUNT(*)
-FROM logs.http_requests
-WHERE timestamp BETWEEN '2025-01-01T00:00:00Z' AND '2025-01-31T23:59:59Z'
-  AND method = 'GET'
-GROUP BY status
-ORDER BY COUNT(*) DESC;
-```
-
-### Fraud Detection
-- Stream transaction events to catalog
-- Query suspicious patterns with WHERE filters
-- Aggregate by location, merchant, time windows
-
-```sql
-SELECT location, COUNT(*), AVG(amount)
-FROM fraud.transactions
-WHERE is_fraud = true
-  AND transaction_timestamp >= '2025-01-01'
-GROUP BY location
-HAVING COUNT(*) > 10;
-```
-
-### Business Intelligence
-- ETL data into partitioned Iceberg tables
-- Run analytical queries across large datasets
-- Generate reports with GROUP BY aggregations
-- No egress fees when querying from BI tools
-
-```sql
-SELECT 
-  department, 
-  SUM(revenue) as total_revenue,
-  AVG(revenue) as avg_revenue
-FROM sales.transactions
-WHERE sale_date >= '2024-01-01'
-GROUP BY department
-ORDER BY SUM(revenue) DESC
-LIMIT 10;
-```
-
-## Performance Optimization
-
-### Partitioning Strategy
-- Choose partition key based on common query patterns
-- Typical: day(timestamp), hour(timestamp), region, category
-- Enables metadata pruning to skip entire partitions
-- Required for ORDER BY optimization
-
-### Query Optimization
-- Use WHERE filters to leverage partition/column stats
-- Specify LIMIT to enable early termination
-- ORDER BY partition key columns only
-- Filter on high-selectivity columns first
-
-### Data Organization
-- Smaller files → slower queries (overhead)
-- Larger files → better compression, fewer metadata ops
-- Recommended: 100-500MB Parquet files after compression
-- Use appropriate roll intervals in Pipelines (300+ seconds for prod)
-
-### File Pruning
-Automatic at three levels:
-1. Partition-level: Skip manifests not matching query
-2. File-level: Skip Parquet files via column stats
-3. Row-group level: Skip row groups within files
-
-## Iceberg Metadata Structure
-
-```
-bucket/
-  metadata/
-    snap-{id}.avro          # Snapshot (points to manifest list)
-    {uuid}-m0.avro          # Manifest file (lists data files + stats)
-    version-hint.text       # Current metadata version
-    v{n}.metadata.json      # Table metadata (schema, snapshots)
-  data/
-    00000-0-{uuid}.parquet  # Data files
-```
-
-**Metadata hierarchy**:
-1. Table metadata JSON - schema, partition spec, snapshot log
-2. Snapshot - points to manifest list
-3. Manifest list - partition stats for each manifest
-4. Manifest files - column stats for each data file
-5. Parquet files - row group stats in footer
-
-## Limitations & Best Practices
-
-### Current Limitations (Open Beta)
-- ORDER BY only on partition key columns
-- COUNT(*) only - COUNT(column) not supported
-- No aliases in SELECT
-- No subqueries, joins, or CTEs
-- No nested column access
-- LIMIT max 10,000
-
-### Best Practices
-- Partition by time dimension for time-series data
-- Use BETWEEN for time ranges (leverages partition pruning)
-- Combine filters with AND for better pruning
-- Set appropriate LIMIT based on use case
-- Use compression (zstd recommended)
-- Monitor query performance and adjust partitioning
-
-### Type Safety
-- Quote string values: 'value'
-- Use RFC3339 for timestamps: '2025-01-01T00:00:00Z'
-- Use YYYY-MM-DD for dates: '2025-01-01'
-- No implicit type conversions
-
-## Connecting Other Engines
-
-R2 Data Catalog supports standard Iceberg REST catalog API.
-
-### Spark (Scala)
-```scala
-val spark = SparkSession.builder()
-  .config("spark.sql.catalog.my_catalog", "org.apache.iceberg.spark.SparkCatalog")
-  .config("spark.sql.catalog.my_catalog.catalog-impl", "org.apache.iceberg.rest.RESTCatalog")
-  .config("spark.sql.catalog.my_catalog.uri", catalogUri)
-  .config("spark.sql.catalog.my_catalog.token", token)
-  .config("spark.sql.catalog.my_catalog.warehouse", warehouse)
-  .getOrCreate()
-```
-
-### Snowflake
-- Create external Iceberg catalog connection
-- Configure with Catalog URI and R2 credentials
-- Query tables via SQL interface
-
-### DuckDB, Trino, ClickHouse
-- Supported via Iceberg REST catalog protocol
-- Refer to engine-specific documentation for configuration
-
-## Pricing (Future)
-
-Currently in open beta - no charges beyond standard R2 costs.
-
-Planned future pricing:
-- R2 storage: $0.015/GB-month
-- Class A operations: $4.50/million
-- Class B operations: $0.36/million
-- Catalog operations: $9.00/million (create table, get metadata, etc)
-- Compaction: $0.05/GB + $4.00/million objects processed
-- Egress: $0 (always free)
-
-30+ days notice before billing begins.
-
-## Troubleshooting
-
-### Common Errors
-
-**"ORDER BY column not in partition key"**
-- Only partition key columns can be used in ORDER BY
-- Check table partition spec with DESCRIBE
-- Remove ORDER BY or adjust table partitioning
-
-**"Token authentication failed"**
-- Verify WRANGLER_R2_SQL_AUTH_TOKEN is set
-- Ensure token has R2 Admin Read & Write + SQL Read permissions
-- Token may be expired - create new one
-
-**"Table not found"**
-- Verify namespace exists: SHOW DATABASES
-- Check table name: SHOW TABLES IN namespace
-- Ensure catalog enabled on bucket
-
-**"No data returned"**
-- Check WHERE conditions match data
-- Verify time range in BETWEEN clause
-- Try removing filters to confirm data exists
-
-### Performance Issues
-
-**Slow queries**:
-- Check partition pruning effectiveness
-- Reduce LIMIT if scanning too much data
-- Ensure filters on partition key columns
-- Review Parquet file sizes (aim for 100-500MB)
-
-**Query timeout**:
-- Add more restrictive WHERE filters
-- Reduce LIMIT
-- Consider better partitioning strategy
-
-## Resources
-
-- Docs: https://developers.cloudflare.com/r2-sql/
-- Data Catalog: https://developers.cloudflare.com/r2/data-catalog/
-- Blog: https://blog.cloudflare.com/r2-sql-deep-dive/
-- Discord: https://discord.cloudflare.com/
-
-## Key Reminders
-
-1. R2 SQL queries ONLY Apache Iceberg tables in R2 Data Catalog
-2. Enable catalog on bucket before use
-3. Create API token with R2 + catalog permissions
-4. Partition by time for time-series data
-5. ORDER BY limited to partition key columns
-6. Use LIMIT and WHERE for optimal performance
-7. Zero egress fees - query from anywhere
-8. Open beta - free during testing phase
-9. Serverless - no infrastructure management
-10. Leverage Cloudflare's global network for distributed execution

package/skills/cloudflare/references/r2-sql/api.md

@@ -1,159 +0,0 @@
-# R2 SQL API Reference
-
-SQL syntax, functions, operators, and data types for R2 SQL queries.
-
-## SQL Syntax
-
-```sql
-SELECT column_list | aggregation_function
-FROM [namespace.]table_name
-WHERE conditions
-[GROUP BY column_list]
-[HAVING conditions]
-[ORDER BY column | aggregation_function [DESC | ASC]]
-[LIMIT number]
-```
-
-## Schema Discovery
-
-```sql
-SHOW DATABASES;           -- List namespaces
-SHOW NAMESPACES;          -- Alias for SHOW DATABASES
-SHOW SCHEMAS;             -- Alias for SHOW DATABASES
-SHOW TABLES IN namespace; -- List tables in namespace
-DESCRIBE namespace.table; -- Show table schema, partition keys
-```
-
-## SELECT Clause
-
-```sql
--- All columns
-SELECT * FROM logs.http_requests;
-
--- Specific columns
-SELECT user_id, timestamp, status FROM logs.http_requests;
-```
-
-**Limitations:** No column aliases, expressions, or nested column access
-
-## WHERE Clause
-
-### Operators
-
-| Operator                        | Example                                                               |
-| ------------------------------- | --------------------------------------------------------------------- |
-| `=`, `!=`, `<`, `<=`, `>`, `>=` | `status = 200`                                                        |
-| `LIKE`                          | `user_agent LIKE '%Chrome%'`                                          |
-| `BETWEEN`                       | `timestamp BETWEEN '2025-01-01T00:00:00Z' AND '2025-01-31T23:59:59Z'` |
-| `IS NULL`, `IS NOT NULL`        | `email IS NOT NULL`                                                   |
-| `AND`, `OR`                     | `status = 200 AND method = 'GET'`                                     |
-
-Use parentheses for precedence: `(status = 404 OR status = 500) AND method = 'POST'`
-
-## Aggregation Functions
-
-| Function                     | Description           |
-| ---------------------------- | --------------------- |
-| `COUNT(*)`                   | Count all rows        |
-| `COUNT(column)`              | Count non-null values |
-| `COUNT(DISTINCT column)`     | Count unique values   |
-| `SUM(column)`, `AVG(column)` | Numeric aggregations  |
-| `MIN(column)`, `MAX(column)` | Min/max values        |
-
-```sql
--- Multiple aggregations with GROUP BY
-SELECT region, COUNT(*), SUM(amount), AVG(amount)
-FROM sales.transactions
-WHERE sale_date >= '2024-01-01'
-GROUP BY region;
-```
-
-## HAVING Clause
-
-Filter aggregated results (after GROUP BY):
-
-```sql
-SELECT category, SUM(amount)
-FROM sales.transactions
-GROUP BY category
-HAVING SUM(amount) > 10000;
-```
-
-## ORDER BY Clause
-
-Sort results by:
-
-- **Partition key columns** - Always supported
-- **Aggregation functions** - Supported via shuffle strategy
-
-```sql
--- Order by partition key
-SELECT * FROM logs.requests ORDER BY timestamp DESC LIMIT 100;
-
--- Order by aggregation (repeat function, aliases not supported)
-SELECT region, SUM(amount)
-FROM sales.transactions
-GROUP BY region
-ORDER BY SUM(amount) DESC;
-```
-
-**Limitations:** Cannot order by non-partition columns. See [gotchas.md](gotchas.md#order-by-limitations)
-
-## LIMIT Clause
-
-```sql
-SELECT * FROM logs.requests LIMIT 100;
-```
-
-| Setting | Value  |
-| ------- | ------ |
-| Min     | 1      |
-| Max     | 10,000 |
-| Default | 500    |
-
-**Always use LIMIT** to enable early termination optimization.
-
-## Data Types
-
-| Type        | SQL Literal     | Example                  |
-| ----------- | --------------- | ------------------------ |
-| `integer`   | Unquoted number | `42`, `-10`              |
-| `float`     | Decimal number  | `3.14`, `-0.5`           |
-| `string`    | Single quotes   | `'hello'`, `'GET'`       |
-| `boolean`   | Keyword         | `true`, `false`          |
-| `timestamp` | RFC3339 string  | `'2025-01-01T00:00:00Z'` |
-| `date`      | ISO 8601 date   | `'2025-01-01'`           |
-
-### Type Safety
-
-- Quote strings with single quotes: `'value'`
-- Timestamps must be RFC3339: `'2025-01-01T00:00:00Z'` (include timezone)
-- Dates must be ISO 8601: `'2025-01-01'` (YYYY-MM-DD)
-- No implicit conversions
-
-```sql
--- ✅ Correct
-WHERE status = 200 AND method = 'GET' AND timestamp > '2025-01-01T00:00:00Z'
-
--- ❌ Wrong
-WHERE status = '200'              -- string instead of integer
-WHERE timestamp > '2025-01-01'    -- missing time/timezone
-WHERE method = GET                -- unquoted string
-```
-
-## Query Result Format
-
-JSON array of objects:
-
-```json
-[
-  { "user_id": "user_123", "timestamp": "2025-01-15T10:30:00Z", "status": 200 },
-  { "user_id": "user_456", "timestamp": "2025-01-15T10:31:00Z", "status": 404 }
-]
-```
-
-## See Also
-
-- [patterns.md](patterns.md) - Query examples and use cases
-- [gotchas.md](gotchas.md) - SQL limitations and error handling
-- [configuration.md](configuration.md) - Setup and authentication