oh-my-opencode 4.19.3 → 5.0.0-beta.1

package/packages/omo-codex/plugin/skills/data-scientist/SKILL.md

@@ -0,0 +1,243 @@
+---
+name: data-scientist
+description: "Expert data processing specialist with intelligent DuckDB/Polars selection for maximum performance. Always includes numpy, never uses pandas, runs everything through uv. Triggers: 'analyze the data', 'analyze this file', 'what is in this CSV/parquet/json', 'summarize this', 'group by', 'filter rows', 'sort by', 'join these files', 'merge datasets', 'time series trend', 'last 30 days data', 'compare yesterday and today', 'distribution/histogram', 'correlation', 'clean duplicates', 'handle missing values', 'dataset larger than RAM', 'SQL query on files', 'DataFrame operations', 'chart/plot this data', DuckDB vs Polars selection, quick data exploration CLI. NOT for plain text/code inspection, configs, or tiny inline math."
+---
+
+# Data Scientist: High-Performance Data Processing Expert
+
+## Role & Expertise
+
+Performance-obsessed data scientist with expertise in:
+- Intelligent tool selection: DuckDB vs Polars based on operation characteristics
+- Zero-copy data interchange via Apache Arrow
+- Memory-efficient processing for datasets exceeding RAM
+- SQL and DataFrame API mastery for analytical workloads
+
+## Environment Setup
+
+Everything runs through **uv**. If `uv` is not on PATH, set it up first — pick the path that matches the system and run it, no manual guesswork:
+
+```bash
+bash scripts/setup-uv.sh        # macOS / Linux / WSL / Git Bash — auto-detects OS + arch, installs or updates uv to latest
+```
+
+```powershell
+powershell -ExecutionPolicy Bypass -File scripts/setup-uv.ps1   # native Windows — installs or updates uv to latest
+```
+
+Both scripts detect the platform, install uv when missing (official installer first, Homebrew/winget as fallback), upgrade it when present (`uv self update`), put it on PATH for the current shell, and verify with `uv --version`. The full per-platform matrix, PATH notes, and CI usage live in [references/uv-setup.md](references/uv-setup.md). Verify: `uv --version`.
+
+## Core Principles
+
+### ABSOLUTE RULES
+
+1. **ALWAYS include numpy** in all data processing operations (`uv run --with numpy ...`)
+2. **NEVER use pandas** - Polars and DuckDB beat it decisively on every operation; the entire skill assumes pandas is absent
+3. **ALWAYS use Python via `uv run`** for calculations and data processing
+4. **Intelligent tool selection**: Choose DuckDB or Polars based on operation types, NOT arbitrarily
+5. **Zero-copy conversions**: hand data across DuckDB and Polars through Arrow — `duckdb.sql(...).pl()`. Never call `.df()` (returns a pandas frame; crashes without pandas). Keep `pyarrow` in the package set or `.pl()` raises `ModuleNotFoundError`
+6. **Lazy evaluation**: Prefer `scan_csv`/`scan_parquet` and `.collect()` only when needed
+7. **Direct file queries**: Let DuckDB query files directly instead of loading to memory when possible
+
+### Standard Package Pattern
+
+```bash
+# Default for data tasks (numpy + pyarrow are mandatory parts of the set)
+uv run --with numpy --with duckdb --with polars --with pyarrow python -c "{code}"
+
+# With visualization (RECOMMENDED for most analysis requests)
+uv run --with numpy --with duckdb --with polars --with pyarrow --with matplotlib python -c "{code}"
+
+# Pure Polars
+uv run --with numpy --with polars python -c "{code}"
+
+# Pure DuckDB (with the Arrow handoff available)
+uv run --with numpy --with duckdb --with pyarrow python -c "{code}"
+```
+
+**When to include matplotlib:**
+- User requests visualization: "graph", "chart", "plot", "show me"
+- Exploratory data analysis (EDA): "analyze", "trends", "patterns"
+- Time-series analysis: "over time", "daily", "trends"
+- Distribution analysis: "distribution", "histogram", "statistics"
+- Comparison tasks: "compare", visual comparison implied
+- **Default to including matplotlib** when in doubt - overhead is minimal
+
+## Tool Selection Logic
+
+### Decision Tree (Apply in Order)
+
+1. **Is it a `.duckdb` file?** → **USE DUCKDB** (native format, optimal performance)
+2. **Simple one-off query without needing full data in memory?** → **USE DUCKDB** (direct file query, zero memory load)
+3. **Very heavy complex SQL query (multi-table joins, window functions)?** → **USE DUCKDB** (superior SQL optimizer)
+4. **Main operation is FILTERING?** → **USE POLARS** (typically the fastest by a wide margin — see benchmarks)
+5. **Main operation is SORTING?** → **USE POLARS** (typically the fastest)
+6. **Complex SQL JOINS needed?** → **USE DUCKDB** (stronger join engine, more join types)
+7. **Heavy GROUP BY AGGREGATIONS?** → **USE DUCKDB** (typically faster on large datasets)
+8. **Window functions with partitioning?** → **POLARS** (typically faster)
+9. **Complex TRANSFORMATIONS (pivot, melt, string ops)?** → **USE POLARS**
+10. **Dataset larger than available RAM?** → **USE POLARS** (streaming support) or **DUCKDB** (out-of-core)
+11. **Mixed operations?** → **USE HYBRID APPROACH** (leverage strengths of both)
+
+### Quick Reference
+
+```
+Simple query → DuckDB
+Heavy complex query → DuckDB
+Filter → Polars
+Sort → Polars
+Join → DuckDB
+Aggregate → DuckDB
+Window → Polars
+Transform → Polars
+Too large for RAM → Polars streaming
+Mixed operations → Hybrid
+```
+
+The exact multipliers these heuristics distill (with sources and caveats — routing heuristics, not guarantees) live in [performance-benchmarks.md](references/performance-benchmarks.md).
+
+## Essential Patterns
+
+### DuckDB Direct File Query
+
+```python
+import duckdb
+# Query file directly - no memory load
+result = duckdb.sql("""
+    SELECT category, SUM(amount) as total
+    FROM 'data.csv'
+    GROUP BY category
+""").pl()  # .pl() -> Polars via Arrow. Requires pyarrow. Never .df() (pandas).
+```
+
+### Polars Lazy Evaluation
+
+```python
+import polars as pl
+# Lazy scan - optimizes and executes once
+result = (
+    pl.scan_csv('data.csv')
+    .filter(pl.col('value') > 100)
+    .sort('value', descending=True)
+    .collect()
+)
+```
+
+### Zero-Copy DuckDB → Polars
+
+```python
+import duckdb
+# Direct conversion via Arrow (pyarrow required in the package set)
+df_polars = duckdb.sql("SELECT * FROM 'data.csv'").pl()
+```
+
+### Hybrid Approach
+
+```python
+import duckdb
+import polars as pl
+
+# Phase 1: DuckDB for joins
+joined = duckdb.sql(
+    "SELECT * FROM 'orders.csv' o "
+    "JOIN 'customers.csv' c ON o.customer_id = c.customer_id"
+).pl()
+
+# Phase 2: Polars for filtering
+filtered = joined.filter(pl.col('amount') > 100)
+
+# Phase 3: Back to DuckDB for aggregation
+duckdb.register('filtered_data', filtered)
+final = duckdb.sql('SELECT category, SUM(amount) FROM filtered_data GROUP BY category').pl()
+```
+
+## Quick Query CLI
+
+For ad-hoc data exploration, use the built-in query runner:
+
+```bash
+# SQL query (uses DuckDB)
+uv run scripts/quick-query.py data.csv "SELECT category, COUNT(*) FROM data GROUP BY category"
+
+# Filter expression — Polars SQL syntax, e.g. "amount > 100" (NOT Python: never passes through eval)
+uv run scripts/quick-query.py data.csv --filter "amount > 100"
+
+# Auto-describe (schema + stats)
+uv run scripts/quick-query.py data.parquet --describe
+```
+
+Supports CSV, Parquet, JSON, NDJSON. Cross-platform (macOS, Linux, Windows). Excel files are not read directly — export to CSV or Parquet first.
+
+## Reference Documentation
+
+For detailed guidance, consult these reference files:
+
+- **Environment setup per platform**: See [uv-setup.md](references/uv-setup.md) — install/update uv on macOS, Linux, Windows, WSL, CI; PATH fixes; `scripts/setup-uv.sh` / `scripts/setup-uv.ps1` automate it.
+- **Performance benchmarks and operation detection**: See [performance-benchmarks.md](references/performance-benchmarks.md)
+- **Integration patterns and best practices**: See [integration-patterns.md](references/integration-patterns.md)
+- **Execution templates**: See [execution-templates.md](references/execution-templates.md)
+- **Common scenarios**: See [common-scenarios.md](references/common-scenarios.md)
+
+## Quality Assurance Process
+
+### Before Execution
+1. **Analyze request** → Detect operation types (filter, join, aggregate, etc.)
+2. **Select optimal tool** → Apply decision tree based on detected operations
+3. **Verify approach** → Confirm tool selection matches the benchmark heuristics
+4. **Check package list** → Ensure numpy AND pyarrow are included
+
+### During Execution
+1. **Use lazy evaluation** when possible (Polars `scan_*`, DuckDB direct queries)
+2. **Monitor for errors** and have fallback strategy ready
+3. **Provide progress updates** for long operations
+
+### After Execution
+1. **Report performance** → Show processing time and row counts
+2. **Validate results** → Confirm output matches expectations
+3. **Document tool choice** → Explain why specific tool was selected
+
+## Activation Context
+
+**Automatic activation triggers:**
+
+### Exploratory Questions
+- "Analyze the data" / "What's in the data" / "What's in this file"
+- "Show me the data" / "Take a look at this file" / "Check the file contents"
+
+### Temporal/Historical Analysis
+- "What happened in the past N days?" / "How's last week's data?"
+- "What's the trend for the last 30 days?" / "Compare yesterday and today"
+
+### Aggregation/Summary Requests
+- "Summarize this" / "What's the total?" / "What's the average?"
+- "Show by category" / "Show statistics" / "How many?"
+
+### Filtering/Search Patterns
+- "Show only above 100" / "Find specific conditions" / "Top 10"
+
+### Comparison/Correlation
+- "Compare A and B" / "What's the difference?" / "Is there a correlation?" / "Merge two files"
+
+### Transformation/Cleaning
+- "Clean this up" / "Remove duplicates" / "Handle missing values" / "Convert format"
+
+### Technical Patterns
+- Working with CSV, Parquet, JSON, NDJSON, or `.duckdb` files
+- File paths ending in `.csv`, `.parquet`, `.json`, `.jsonl`, `.ndjson`, `.tsv`, `.duckdb`
+- Requests involving calculations or aggregations
+- Joining, filtering, sorting, or transforming datasets
+- Processing large datasets that may exceed memory
+- Comparing or analyzing data from multiple sources
+- Performance-critical data operations
+- SQL queries or DataFrame operations mentioned
+
+### When NOT to Activate
+- Simple file reading for text/code inspection (use the harness's file-read surface)
+- Non-data files (images, videos, binaries)
+- Configuration files (YAML, TOML, JSON configs) unless specifically for data analysis
+- Small inline calculations (run them directly)
+- Excel files — convert to CSV/Parquet first
+
+---
+
+**Core execution principle:** Always apply intelligent tool selection based on operation characteristics, never use pandas, and always include numpy and pyarrow in the execution environment.

package/packages/omo-codex/plugin/skills/data-scientist/agents/openai.yaml

@@ -0,0 +1,2 @@
+interface:
+  display_name: "(OmO) data-scientist"

package/packages/omo-codex/plugin/skills/data-scientist/references/common-scenarios.md

@@ -0,0 +1,176 @@
+# Common Scenarios with Tool Selection
+
+## Scenario 1: Simple Calculation
+
+**Decision: Python (always)**
+
+```python
+uv run --with numpy python -c "
+import numpy as np
+result = np.sum([1, 2, 3, 4, 5])
+print(f'Result: {result}')
+"
+```
+
+## Scenario 2: CSV Quick Analysis
+
+**Decision: DuckDB (direct query, no memory load)**
+
+```python
+uv run --with numpy --with duckdb python -c "
+import duckdb
+result = duckdb.sql('''
+    SELECT * FROM 'data.csv'
+    LIMIT 10
+''').pl()
+print(result)
+"
+```
+
+## Scenario 3: Filter + Sort on Large Dataset
+
+**Decision: Polars (128x faster filtering, 12x faster sorting)**
+
+```python
+uv run --with numpy --with polars python -c "
+import polars as pl
+result = (
+    pl.scan_csv('large.csv')
+    .filter(pl.col('value') > 1000)
+    .sort('value', descending=True)
+    .head(100)
+    .collect()
+)
+print(result)
+"
+```
+
+## Scenario 4: Multi-Table Join + Aggregation
+
+**Decision: DuckDB (best for joins and aggregations)**
+
+```python
+uv run --with numpy --with duckdb python -c "
+import duckdb
+result = duckdb.sql('''
+    SELECT
+        a.category,
+        COUNT(*) as count,
+        SUM(b.amount) as total
+    FROM 'table1.csv' a
+    JOIN 'table2.csv' b ON a.id = b.id
+    GROUP BY a.category
+    ORDER BY total DESC
+''').pl()
+print(result)
+"
+```
+
+## Scenario 5: Data Exploration
+
+**Decision: DuckDB for quick exploration**
+
+```python
+uv run --with numpy --with duckdb python -c "
+import duckdb
+
+# Show first few rows
+print('**Sample Data**')
+print(duckdb.sql('SELECT * FROM \"data.csv\" LIMIT 5').pl())
+
+# Show summary statistics
+print('\\n**Summary Statistics**')
+print(duckdb.sql('DESCRIBE SELECT * FROM \"data.csv\"').pl())
+
+# Show row count
+print('\\n**Row Count**')
+print(duckdb.sql('SELECT COUNT(*) as total_rows FROM \"data.csv\"').pl())
+"
+```
+
+## Scenario 6: Time-Series Analysis
+
+**Decision: DuckDB for aggregation + matplotlib for visualization**
+
+```python
+uv run --with numpy --with duckdb --with pyarrow --with matplotlib python -c "
+import duckdb
+import matplotlib.pyplot as plt
+
+# Aggregate by date
+result = duckdb.sql('''
+    SELECT
+        DATE_TRUNC('day', timestamp) as date,
+        COUNT(*) as count,
+        AVG(value) as avg_value
+    FROM 'timeseries.csv'
+    GROUP BY date
+    ORDER BY date
+''').pl()
+
+# Plot
+plt.figure(figsize=(12, 6))
+plt.subplot(2, 1, 1)
+plt.plot(result['date'], result['count'])
+plt.title('Daily Count')
+
+plt.subplot(2, 1, 2)
+plt.plot(result['date'], result['avg_value'])
+plt.title('Daily Average Value')
+
+plt.tight_layout()
+plt.savefig('timeseries.png')
+print('Saved to timeseries.png')
+"
+```
+
+## Scenario 7: Complex Transformation
+
+**Decision: Polars for efficient transformations**
+
+```python
+uv run --with numpy --with polars python -c "
+import polars as pl
+
+result = (
+    pl.scan_csv('data.csv')
+    .with_columns([
+        # Create new calculated columns
+        (pl.col('price') * pl.col('quantity')).alias('total'),
+        pl.col('date').str.strptime(pl.Date, '%Y-%m-%d').alias('parsed_date'),
+        pl.col('name').str.to_uppercase().alias('upper_name'),
+    ])
+    .filter(pl.col('total') > 100)
+    .select(['parsed_date', 'upper_name', 'total'])
+    .collect()
+)
+
+print(result)
+"
+```
+
+## Scenario 8: Large File Processing
+
+**Decision: Polars streaming mode**
+
+```python
+uv run --with numpy --with polars python -c "
+import polars as pl
+
+# Process file larger than RAM
+result = (
+    pl.scan_csv('huge_file.csv')
+    .filter(pl.col('active') == True)
+    .groupby('category')
+    .agg([
+        pl.count().alias('count'),
+        pl.sum('amount').alias('total'),
+        pl.mean('amount').alias('average'),
+    ])
+    .collect(streaming=True)  # Streaming mode
+)
+
+print(result)
+print(f'\\nProcessed {result[\"count\"].sum():,} rows')
+"
+```

package/packages/omo-codex/plugin/skills/data-scientist/references/execution-templates.md

@@ -0,0 +1,197 @@
+# Execution Templates
+
+## Template 1: DuckDB for Simple/Complex SQL Queries
+
+**Use when:**
+- Simple aggregation on single file
+- Complex multi-table joins
+- Heavy GROUP BY operations
+- Window functions with complex SQL logic
+- Ad-hoc exploration queries
+
+```python
+uv run --with numpy --with duckdb python -c "
+import duckdb
+
+# Simple query - direct file access
+result = duckdb.sql('''
+    SELECT
+        category,
+        COUNT(*) as count,
+        AVG(amount) as avg_amount,
+        SUM(amount) as total
+    FROM 'data.csv'
+    WHERE date >= '2024-01-01'
+    GROUP BY category
+    ORDER BY total DESC
+    LIMIT 10
+''').pl()
+
+print('**Results**')
+print(result)
+print(f'\\nProcessed {len(result)} categories')
+"
+```
+
+## Template 2: Polars for Filtering & Sorting
+
+**Use when:**
+- Primary operation is filtering large dataset
+- Sorting required
+- Chain transformations
+- Memory-efficient processing needed
+
+```python
+uv run --with numpy --with polars python -c "
+import polars as pl
+
+# Lazy evaluation for optimal performance
+result = (
+    pl.scan_csv('data.csv')  # Lazy scan
+    .filter(
+        (pl.col('amount') > 1000) &
+        (pl.col('status') == 'active') &
+        (pl.col('date') >= '2024-01-01')
+    )
+    .sort('amount', descending=True)
+    .head(100)
+    .collect()  # Execute optimized plan
+)
+
+print('**Filtered and Sorted Results**')
+print(result)
+print(f'\\nFound {len(result)} matching rows')
+"
+```
+
+## Template 3: Hybrid Approach (Best of Both)
+
+**Use when:**
+- Need joins AND heavy filtering
+- Complex SQL followed by transformations
+- Optimize different operation stages
+
+```python
+uv run --with numpy --with duckdb --with polars --with pyarrow python -c "
+import duckdb
+import polars as pl
+
+print('Phase 1: DuckDB for complex join (3x faster)')
+# DuckDB excels at joins
+joined = duckdb.sql('''
+    SELECT
+        o.order_id,
+        o.amount,
+        c.customer_id,
+        c.region,
+        p.category
+    FROM 'orders.csv' o
+    JOIN 'customers.csv' c ON o.customer_id = c.customer_id
+    JOIN 'products.csv' p ON o.product_id = p.product_id
+    WHERE o.date >= '2024-01-01'
+''').pl()  # Convert to Polars
+
+print(f'Joined {len(joined):,} rows')
+
+print('\\nPhase 2: Polars for ultra-fast filtering (128x faster)')
+# Polars excels at filtering
+filtered = (
+    joined
+    .filter(
+        (pl.col('amount') > 100) &
+        (pl.col('region').is_in(['North', 'South', 'East']))
+    )
+    .with_columns([
+        (pl.col('amount') * 1.1).alias('amount_with_tax')
+    ])
+)
+
+print(f'Filtered to {len(filtered):,} rows')
+
+print('\\nPhase 3: DuckDB for final aggregation (4x faster)')
+# Back to DuckDB for aggregation
+duckdb.register('filtered_data', filtered)
+final = duckdb.sql('''
+    SELECT
+        region,
+        category,
+        COUNT(DISTINCT customer_id) as customers,
+        SUM(amount_with_tax) as total_revenue,
+        AVG(amount_with_tax) as avg_transaction
+    FROM filtered_data
+    GROUP BY region, category
+    HAVING total_revenue > 10000
+    ORDER BY total_revenue DESC
+''').pl()
+
+print('\\n**Final Results**')
+print(final)
+"
+```
+
+## Template 4: Polars Streaming for Large Files
+
+**Use when:**
+- Dataset larger than available RAM
+- Need to process data in batches
+- Memory constraints
+
+```python
+uv run --with numpy --with polars python -c "
+import polars as pl
+
+# Streaming mode - processes data in chunks
+result = (
+    pl.scan_csv('huge_file.csv')
+    .filter(pl.col('status') == 'active')
+    .with_columns([
+        (pl.col('amount') * 1.1).alias('adjusted_amount')
+    ])
+    .groupby('category')
+    .agg([
+        pl.sum('adjusted_amount').alias('total'),
+        pl.count().alias('count')
+    ])
+    .collect(streaming=True)  # Streaming mode for large data
+)
+
+print('**Streaming Results**')
+print(result)
+print(f'\\nProcessed {result[\"count\"].sum():,} total rows')
+"
+```
+
+## Template 5: Visualization with Matplotlib
+
+**Use when:**
+- User requests charts, graphs, or plots
+- Exploratory data analysis (EDA)
+- Time-series or distribution analysis
+
+```python
+uv run --with numpy --with duckdb --with polars --with pyarrow --with matplotlib python -c "
+import duckdb
+import matplotlib.pyplot as plt
+
+# Query data
+result = duckdb.sql('''
+    SELECT
+        date,
+        SUM(amount) as total
+    FROM 'data.csv'
+    GROUP BY date
+    ORDER BY date
+''').pl()
+
+# Create visualization
+plt.figure(figsize=(10, 6))
+plt.plot(result['date'], result['total'])
+plt.xlabel('Date')
+plt.ylabel('Total Amount')
+plt.title('Daily Total Trends')
+plt.xticks(rotation=45)
+plt.tight_layout()
+plt.savefig('output.png')
+print('Chart saved to output.png')
+"
+```