oh-my-opencode 5.0.0-beta.26 → 5.0.0-beta.28

package/packages/omo-codex/plugin/skills/data-scientist/references/polars-lane.md

@@ -0,0 +1,95 @@
+# Polars lane (resident Python kernel)
+
+When the work is DataFrame-shaped, Polars is the right engine — and it runs in the resident
+Python kernel by default. Kernels rarely ship polars/pyarrow preinstalled, so inject them
+once per session; the install lands in a user cache keyed to the kernel's interpreter, and
+the interpreter itself is never mutated (run with the skill directory as cwd, or spell out
+the script's absolute path):
+
+```python
+import subprocess, sys
+site = subprocess.run(["bash", "scripts/ensure-py-deps.sh", sys.executable],
+                      capture_output=True, text=True, check=True).stdout.strip()
+sys.path.insert(0, site)
+import polars as pl
+```
+
+After this, Polars lives across cells like every other resident engine: lazy frames,
+intermediate results, and the DuckDB handoff all persist with no per-call process cost.
+
+## When Polars wins over DuckDB SQL
+
+- Expression-chain transforms: many derived columns, per-column conditional logic, string
+  pipelines — `with_columns` chains read and optimize better than nested SQL SELECTs.
+- Reshapes: `unpivot`/`pivot` beat SQL gymnastics.
+- Larger-than-RAM pipelines: the streaming engine executes lazy plans in chunks.
+- Window-heavy feature engineering with `over()`.
+
+SQL-shaped work (joins, aggregation, ad-hoc questions) stays in DuckDB; mixed pipelines hand
+off zero-copy (below) instead of forcing one engine to do everything.
+
+## Current API (1.x) — older spellings fail or warn
+
+Training data is full of the pre-1.0 API. Current names:
+
+| Use | Not |
+| --- | --- |
+| `pl.scan_csv` / `pl.scan_parquet` + `.collect()` | eager `read_*` on big files |
+| `.group_by(...)` | `.groupby(...)` |
+| `pl.len()` | `pl.count()` |
+| `.collect(engine="streaming")` | `.collect(streaming=True)` |
+| `.unpivot(...)` | `.melt(...)` |
+
+Lazy first: `scan_*` builds a plan, pushes filters and projections down to the file read, and
+executes once at `.collect()`. Eager `read_*` is for small files mutated interactively.
+
+```python
+out = (pl.scan_csv("data.csv")
+       .filter(pl.col("value") > 100)
+       .group_by("category")
+       .agg(pl.col("value").sum().alias("total"), pl.len().alias("n"))
+       .sort("total", descending=True)
+       .collect())
+```
+
+## Zero-copy handoff with DuckDB
+
+Both engines speak Arrow, so mixed pipelines pay no serialization cost — all in-kernel:
+
+```python
+import duckdb
+df = duckdb.sql("SELECT * FROM 'orders.csv' o JOIN 'items.csv' i USING (id)").pl()
+shaped = df.with_columns((pl.col("qty") * pl.col("price")).alias("rev"))
+duckdb.register("shaped", shaped)
+out = duckdb.sql("SELECT category, SUM(rev) AS total FROM shaped GROUP BY 1").pl()
+```
+
+- `.pl()` requires pyarrow — the injection above provides it; without it, it raises
+  `ModuleNotFoundError`.
+- Never `.df()`: it requires pandas (raising without it), and pandas is banned and absent.
+
+## Streaming past RAM
+
+```python
+out = (pl.scan_parquet("huge.parquet")
+       .filter(pl.col("status") == "active")
+       .group_by("region").agg(pl.len())
+       .collect(engine="streaming"))
+```
+
+Streaming executes lazy plans only — keep the plan lazy end-to-end, with no intermediate
+`.collect()` breaking it into eager pieces.
+
+## Kernel-less fallback (uv one-shot)
+
+On a harness with no persistent kernel, the same code runs as one-shots — batch several
+questions per process, since each invocation pays spawn plus imports:
+
+```bash
+uv run --with duckdb --with polars --with pyarrow python -c "
+import duckdb
+import polars as pl
+df = duckdb.sql(\"SELECT * FROM 'data.csv'\").pl()
+print(df.group_by('category').agg(pl.len()).sort('category'))
+"
+```

package/packages/omo-codex/plugin/skills/data-scientist/references/uv-setup.md

@@ -1,6 +1,6 @@
 # uv Setup — Per-Platform
 
-This skill runs every data operation through `uv run --with ...`. If `uv --version` fails, set uv up with the automated scripts or the manual commands below, then verify.
+This skill's uv lane (kernel-less harnesses, isolated one-shots) runs through `uv run --with ...`, and `scripts/ensure-py-deps.sh` uses uv as its installer. If `uv --version` fails, set uv up with the automated scripts or the manual commands below, then verify.
 
 ## Automated (recommended)
 

package/packages/omo-codex/plugin/skills/data-scientist/references/visualization.md

@@ -0,0 +1,64 @@
+# Visualization
+
+A chart exists to answer a question at a glance. Render it with matplotlib (resident in most
+Python kernels; `uv run --with matplotlib` otherwise), then look at it before delivering —
+a chart nobody inspected is not evidence.
+
+## When to chart
+
+Chart when the user asked for one, and default to charting when the answer is a shape prose
+cannot carry: a trend over time, a distribution, a comparison across many categories, a
+relationship between variables. Skip the chart when a number or a five-row table answers the
+question — decoration dilutes the answer.
+
+## Chart type follows the question
+
+| Question shape | Chart |
+| --- | --- |
+| How did X change over time? | line, datetime x-axis |
+| Which categories are biggest? | horizontal bar, sorted by value |
+| How is X distributed? | histogram (tune bin count) or box plot per group |
+| Is X related to Y? | scatter; add a trend line only when it aids the eye |
+| Composition of a whole? | stacked or 100% bar — pie only for four or fewer slices |
+| Many series over time? | small multiples over one spaghetti chart |
+
+## Quality bar — every chart
+
+- Title states the finding ("Seoul overtook Busan in March"), not the dataset name.
+- Axis labels carry units. Tick density stays readable: `fig.autofmt_xdate()` for dates,
+  rotate or abbreviate long category names.
+- Size for the medium: inline chat reads well around `figsize=(10, 6)` at default dpi;
+  documents want `dpi=150` or more at export.
+- `tight_layout()` (or `constrained_layout=True`) before saving — clipped labels are the
+  most common chart defect.
+- Few series: label lines directly, or keep the legend inside empty plot space. Many
+  series: gray the context, color only the series that answers the question.
+- The default color cycle is fine; avoid rainbow palettes and 3D. Sort categorical bars by
+  value, never alphabetically.
+
+## CJK and other non-Latin text
+
+Matplotlib's default font renders CJK as empty boxes (tofu). Set a fallback before plotting
+whenever any label or title contains CJK:
+
+```python
+import platform
+import matplotlib
+cjk = {"Darwin": "AppleGothic", "Windows": "Malgun Gothic"}.get(platform.system(), "Noto Sans CJK KR")
+matplotlib.rcParams["font.family"] = [cjk, "DejaVu Sans"]
+matplotlib.rcParams["axes.unicode_minus"] = False   # keeps the minus sign rendering
+```
+
+## Output contract
+
+1. Save a PNG next to the work: `plt.savefig(path, dpi=150, bbox_inches="tight")`.
+2. Also render inline when the surface displays rich output (kernels usually do).
+3. Report the file path together with the answer.
+
+## Visual QA — mandatory
+
+Open the produced image — kernel display, or the harness's image-reading surface — and
+check four things: labels readable and unclipped, no tofu or mojibake, nothing overlapping,
+and the chart actually shows the finding the title claims. A failed check means fix and
+re-render, not ship with a caveat. This one pass catches nearly every chart defect;
+skipping it is how tofu titles reach users.

package/packages/omo-codex/plugin/skills/data-scientist/scripts/ensure-js-deps.sh

@@ -0,0 +1,28 @@
+#!/usr/bin/env bash
+# Install @duckdb/node-api into a user-level cache (outside any repo) and print the
+# absolute import path as the ONLY stdout line. Idempotent: re-runs reuse the install.
+set -euo pipefail
+
+log() { printf '[ensure-js-deps] %s\n' "$*" >&2; }
+
+CACHE_DIR="${OMO_DATA_SCIENTIST_CACHE:-$HOME/.cache/omo-data-scientist}"
+IMPORT_PATH="$CACHE_DIR/node_modules/@duckdb/node-api/lib/index.js"
+
+if ! command -v bun >/dev/null 2>&1; then
+  log "bun is required (https://bun.sh); install it, or use the uv lane instead."
+  exit 1
+fi
+
+if [ ! -f "$IMPORT_PATH" ]; then
+  log "installing @duckdb/node-api into $CACHE_DIR"
+  mkdir -p "$CACHE_DIR"
+  [ -f "$CACHE_DIR/package.json" ] || printf '{"name":"omo-data-scientist-cache","private":true}\n' > "$CACHE_DIR/package.json"
+  (cd "$CACHE_DIR" && bun add @duckdb/node-api 1>&2)
+fi
+
+if [ ! -f "$IMPORT_PATH" ]; then
+  log "install finished but $IMPORT_PATH is missing; inspect $CACHE_DIR"
+  exit 1
+fi
+
+printf '%s\n' "$IMPORT_PATH"

package/packages/omo-codex/plugin/skills/data-scientist/scripts/ensure-py-deps.sh

@@ -0,0 +1,37 @@
+#!/usr/bin/env bash
+# Install polars + pyarrow for a given Python interpreter into a user-level cache
+# (never mutating the interpreter itself) and print the site directory as the ONLY
+# stdout line. Idempotent: re-runs reuse the install. arg1 = python executable
+# (default: python3); pass the kernel's sys.executable for kernel use.
+set -euo pipefail
+
+log() { printf '[ensure-py-deps] %s\n' "$*" >&2; }
+
+PYTHON_BIN="${1:-python3}"
+
+if ! command -v uv >/dev/null 2>&1; then
+  log "uv is required (see references/uv-setup.md); install it first."
+  exit 1
+fi
+if ! command -v "$PYTHON_BIN" >/dev/null 2>&1 && [ ! -x "$PYTHON_BIN" ]; then
+  log "python executable not found: $PYTHON_BIN"
+  exit 1
+fi
+
+TAG="$("$PYTHON_BIN" -c 'import sys; print(f"cp{sys.version_info[0]}{sys.version_info[1]}")')" \
+  || { log "not a working python interpreter: $PYTHON_BIN"; exit 1; }
+CACHE_DIR="${OMO_DATA_SCIENTIST_CACHE:-$HOME/.cache/omo-data-scientist}"
+SITE_DIR="$CACHE_DIR/py-$TAG"
+
+if [ ! -d "$SITE_DIR/polars" ] || [ ! -d "$SITE_DIR/pyarrow" ]; then
+  log "installing polars + pyarrow for $TAG into $SITE_DIR"
+  mkdir -p "$SITE_DIR"
+  uv pip install --python "$PYTHON_BIN" --target "$SITE_DIR" polars pyarrow 1>&2
+fi
+
+if [ ! -d "$SITE_DIR/polars" ]; then
+  log "install finished but $SITE_DIR/polars is missing; inspect $SITE_DIR"
+  exit 1
+fi
+
+printf '%s\n' "$SITE_DIR"

package/packages/omo-codex/plugin/skills/ulw-research/SKILL.md

@@ -161,7 +161,9 @@ Record the answer in the journal; Phase 5 opens by turning it into `design-spec.
 
 ## Phase 1 — Saturation wave
 
-Launch the entire first wave in one turn — every axis at once, as team members if you formed a team, else as background workers. Sequential launches and "start with one and see" defeat the mode.
+**When the user asked for MASS research, the wave is sized by the topic's angles, not by the roster ceiling.** "mass ulw research", "mulw research", "ulw mass research" — in any language — order over-collection that a team of 8 cannot produce. Where the harness has a dependency-graph surface (the `mass-ulw` skill and its `references/planning.md`), run collection through it instead: a 60+ node opening wave covering every angle the topic has, routed across the whole difficulty ladder in one graph — mechanical sweeps and per-item harvest batches on the cheapest tier, judgment-shaped angles a step up, cross-territory angles a step above that, and the deepest tier reserved for genuinely hairy contradictions. Each wave's EXPAND leads define the next wave's nodes, and the synthesis reduces through several parallel architect-tier nodes into ONE architect-tier reducer (the strongest reasoning tier substitutes wherever no architect tier exists). Absent that surface, hold the same shape with background workers in batched waves. Everything else in this skill still binds: the format gate, the journal, the claim graph, the convergence rules, and the delivery gates.
+
+Otherwise launch the entire first wave in one turn — every axis at once, as team members if you formed a team, else as background workers. Sequential launches and "start with one and see" defeat the mode.
 
 Scaling floor — more angles always justify more workers:
 

package/packages/omo-codex/scripts/install-dist/install-local.mjs

@@ -1,5 +1,5 @@
 #!/usr/bin/env node
-// omo-codex-install:819e258f7485f9ccdcbb64a54ac0097bf81cd8e2980d7f907017f224b1f99b98:3a0f0cc479b76911cc517d857bad53dceaaf7421ac945a5476ac1a5112c0bcf5
+// omo-codex-install:819e258f7485f9ccdcbb64a54ac0097bf81cd8e2980d7f907017f224b1f99b98:c7cf2301eb62bcb863c4e5844198424f235d9a88b5621bbf03436d1ef66c3491
 var __defProp = Object.defineProperty;
 var __returnValue = (v) => v;
 function __exportSetter(name, newValue) {
@@ -7977,7 +7977,7 @@ var package_default;
 var init_package = __esm(() => {
   package_default = {
     name: "@oh-my-opencode/omo-codex",
-    version: "5.0.0-beta.26",
+    version: "5.0.0-beta.28",
     type: "module",
     private: true,
     description: "Codex harness adapter for oh-my-openagent. Vendored Codex plugin namespace (omo) + TypeScript installer + telemetry.",

package/packages/shared-skills/skills/data-scientist/SKILL.md

@@ -1,243 +1,103 @@
 ---
 name: data-scientist
-description: "Expert data processing specialist with intelligent DuckDB/Polars selection for maximum performance. Always includes numpy, never uses pandas, runs everything through uv. Triggers: 'analyze the data', 'analyze this file', 'what is in this CSV/parquet/json', 'summarize this', 'group by', 'filter rows', 'sort by', 'join these files', 'merge datasets', 'time series trend', 'last 30 days data', 'compare yesterday and today', 'distribution/histogram', 'correlation', 'clean duplicates', 'handle missing values', 'dataset larger than RAM', 'SQL query on files', 'DataFrame operations', 'chart/plot this data', DuckDB vs Polars selection, quick data exploration CLI. NOT for plain text/code inspection, configs, or tiny inline math."
+description: "Expert data processing with a hybrid engine strategy: resident-kernel engines first - DuckDB plus a resident Python stack (Polars/numpy/matplotlib) in persistent js/py eval kernels where the harness has them, bun/uv one-shots elsewhere - and per-action placement judgment (in-memory vs streaming vs remote-in-place). Triggers: 'analyze the data', 'what is in this CSV/parquet/json', 'summarize this', 'group by', 'filter rows', 'sort by', 'join these files', 'merge datasets', 'time series trend', 'compare yesterday and today', 'distribution/histogram', 'correlation', 'clean duplicates', 'handle missing values', 'dataset larger than RAM', 'SQL query on files', 'DataFrame operations', 'chart/plot this data', DuckDB vs Polars selection, quick data exploration CLI. NOT for plain text/code inspection, configs, or tiny inline math."
 ---
 
-# Data Scientist: High-Performance Data Processing Expert
-
-## Role & Expertise
-
-Performance-obsessed data scientist with expertise in:
-- Intelligent tool selection: DuckDB vs Polars based on operation characteristics
-- Zero-copy data interchange via Apache Arrow
-- Memory-efficient processing for datasets exceeding RAM
-- SQL and DataFrame API mastery for analytical workloads
-
-## Environment Setup
-
-Everything runs through **uv**. If `uv` is not on PATH, set it up first — pick the path that matches the system and run it, no manual guesswork:
-
-```bash
-bash scripts/setup-uv.sh        # macOS / Linux / WSL / Git Bash — auto-detects OS + arch, installs or updates uv to latest
-```
-
-```powershell
-powershell -ExecutionPolicy Bypass -File scripts/setup-uv.ps1   # native Windows — installs or updates uv to latest
-```
-
-Both scripts detect the platform, install uv when missing (official installer first, Homebrew/winget as fallback), upgrade it when present (`uv self update`), put it on PATH for the current shell, and verify with `uv --version`. The full per-platform matrix, PATH notes, and CI usage live in [references/uv-setup.md](references/uv-setup.md). Verify: `uv --version`.
-
-## Core Principles
-
-### ABSOLUTE RULES
-
-1. **ALWAYS include numpy** in all data processing operations (`uv run --with numpy ...`)
-2. **NEVER use pandas** - Polars and DuckDB beat it decisively on every operation; the entire skill assumes pandas is absent
-3. **ALWAYS use Python via `uv run`** for calculations and data processing
-4. **Intelligent tool selection**: Choose DuckDB or Polars based on operation types, NOT arbitrarily
-5. **Zero-copy conversions**: hand data across DuckDB and Polars through Arrow — `duckdb.sql(...).pl()`. Never call `.df()` (returns a pandas frame; crashes without pandas). Keep `pyarrow` in the package set or `.pl()` raises `ModuleNotFoundError`
-6. **Lazy evaluation**: Prefer `scan_csv`/`scan_parquet` and `.collect()` only when needed
-7. **Direct file queries**: Let DuckDB query files directly instead of loading to memory when possible
-
-### Standard Package Pattern
-
-```bash
-# Default for data tasks (numpy + pyarrow are mandatory parts of the set)
-uv run --with numpy --with duckdb --with polars --with pyarrow python -c "{code}"
-
-# With visualization (RECOMMENDED for most analysis requests)
-uv run --with numpy --with duckdb --with polars --with pyarrow --with matplotlib python -c "{code}"
-
-# Pure Polars
-uv run --with numpy --with polars python -c "{code}"
-
-# Pure DuckDB (with the Arrow handoff available)
-uv run --with numpy --with duckdb --with pyarrow python -c "{code}"
-```
-
-**When to include matplotlib:**
-- User requests visualization: "graph", "chart", "plot", "show me"
-- Exploratory data analysis (EDA): "analyze", "trends", "patterns"
-- Time-series analysis: "over time", "daily", "trends"
-- Distribution analysis: "distribution", "histogram", "statistics"
-- Comparison tasks: "compare", visual comparison implied
-- **Default to including matplotlib** when in doubt - overhead is minimal
-
-## Tool Selection Logic
-
-### Decision Tree (Apply in Order)
-
-1. **Is it a `.duckdb` file?** → **USE DUCKDB** (native format, optimal performance)
-2. **Simple one-off query without needing full data in memory?** → **USE DUCKDB** (direct file query, zero memory load)
-3. **Very heavy complex SQL query (multi-table joins, window functions)?** → **USE DUCKDB** (superior SQL optimizer)
-4. **Main operation is FILTERING?** → **USE POLARS** (typically the fastest by a wide margin — see benchmarks)
-5. **Main operation is SORTING?** → **USE POLARS** (typically the fastest)
-6. **Complex SQL JOINS needed?** → **USE DUCKDB** (stronger join engine, more join types)
-7. **Heavy GROUP BY AGGREGATIONS?** → **USE DUCKDB** (typically faster on large datasets)
-8. **Window functions with partitioning?** → **POLARS** (typically faster)
-9. **Complex TRANSFORMATIONS (pivot, melt, string ops)?** → **USE POLARS**
-10. **Dataset larger than available RAM?** → **USE POLARS** (streaming support) or **DUCKDB** (out-of-core)
-11. **Mixed operations?** → **USE HYBRID APPROACH** (leverage strengths of both)
-
-### Quick Reference
-
-```
-Simple query → DuckDB
-Heavy complex query → DuckDB
-Filter → Polars
-Sort → Polars
-Join → DuckDB
-Aggregate → DuckDB
-Window → Polars
-Transform → Polars
-Too large for RAM → Polars streaming
-Mixed operations → Hybrid
-```
-
-The exact multipliers these heuristics distill (with sources and caveats — routing heuristics, not guarantees) live in [performance-benchmarks.md](references/performance-benchmarks.md).
-
-## Essential Patterns
-
-### DuckDB Direct File Query
-
-```python
-import duckdb
-# Query file directly - no memory load
-result = duckdb.sql("""
-    SELECT category, SUM(amount) as total
-    FROM 'data.csv'
-    GROUP BY category
-""").pl()  # .pl() -> Polars via Arrow. Requires pyarrow. Never .df() (pandas).
-```
-
-### Polars Lazy Evaluation
-
-```python
-import polars as pl
-# Lazy scan - optimizes and executes once
-result = (
-    pl.scan_csv('data.csv')
-    .filter(pl.col('value') > 100)
-    .sort('value', descending=True)
-    .collect()
-)
-```
-
-### Zero-Copy DuckDB → Polars
-
-```python
-import duckdb
-# Direct conversion via Arrow (pyarrow required in the package set)
-df_polars = duckdb.sql("SELECT * FROM 'data.csv'").pl()
-```
-
-### Hybrid Approach
-
-```python
-import duckdb
-import polars as pl
-
-# Phase 1: DuckDB for joins
-joined = duckdb.sql(
-    "SELECT * FROM 'orders.csv' o "
-    "JOIN 'customers.csv' c ON o.customer_id = c.customer_id"
-).pl()
-
-# Phase 2: Polars for filtering
-filtered = joined.filter(pl.col('amount') > 100)
-
-# Phase 3: Back to DuckDB for aggregation
-duckdb.register('filtered_data', filtered)
-final = duckdb.sql('SELECT category, SUM(amount) FROM filtered_data GROUP BY category').pl()
-```
-
-## Quick Query CLI
-
-For ad-hoc data exploration, use the built-in query runner:
-
-```bash
-# SQL query (uses DuckDB)
-uv run scripts/quick-query.py data.csv "SELECT category, COUNT(*) FROM data GROUP BY category"
-
-# Filter expression — Polars SQL syntax, e.g. "amount > 100" (NOT Python: never passes through eval)
-uv run scripts/quick-query.py data.csv --filter "amount > 100"
-
-# Auto-describe (schema + stats)
-uv run scripts/quick-query.py data.parquet --describe
-```
-
-Supports CSV, Parquet, JSON, NDJSON. Cross-platform (macOS, Linux, Windows). Excel files are not read directly — export to CSV or Parquet first.
-
-## Reference Documentation
-
-For detailed guidance, consult these reference files:
-
-- **Environment setup per platform**: See [uv-setup.md](references/uv-setup.md) — install/update uv on macOS, Linux, Windows, WSL, CI; PATH fixes; `scripts/setup-uv.sh` / `scripts/setup-uv.ps1` automate it.
-- **Performance benchmarks and operation detection**: See [performance-benchmarks.md](references/performance-benchmarks.md)
-- **Integration patterns and best practices**: See [integration-patterns.md](references/integration-patterns.md)
-- **Execution templates**: See [execution-templates.md](references/execution-templates.md)
-- **Common scenarios**: See [common-scenarios.md](references/common-scenarios.md)
-
-## Quality Assurance Process
-
-### Before Execution
-1. **Analyze request** → Detect operation types (filter, join, aggregate, etc.)
-2. **Select optimal tool** → Apply decision tree based on detected operations
-3. **Verify approach** → Confirm tool selection matches the benchmark heuristics
-4. **Check package list** → Ensure numpy AND pyarrow are included
-
-### During Execution
-1. **Use lazy evaluation** when possible (Polars `scan_*`, DuckDB direct queries)
-2. **Monitor for errors** and have fallback strategy ready
-3. **Provide progress updates** for long operations
-
-### After Execution
-1. **Report performance** → Show processing time and row counts
-2. **Validate results** → Confirm output matches expectations
-3. **Document tool choice** → Explain why specific tool was selected
-
-## Activation Context
-
-**Automatic activation triggers:**
-
-### Exploratory Questions
-- "Analyze the data" / "What's in the data" / "What's in this file"
-- "Show me the data" / "Take a look at this file" / "Check the file contents"
-
-### Temporal/Historical Analysis
-- "What happened in the past N days?" / "How's last week's data?"
-- "What's the trend for the last 30 days?" / "Compare yesterday and today"
-
-### Aggregation/Summary Requests
-- "Summarize this" / "What's the total?" / "What's the average?"
-- "Show by category" / "Show statistics" / "How many?"
-
-### Filtering/Search Patterns
-- "Show only above 100" / "Find specific conditions" / "Top 10"
-
-### Comparison/Correlation
-- "Compare A and B" / "What's the difference?" / "Is there a correlation?" / "Merge two files"
-
-### Transformation/Cleaning
-- "Clean this up" / "Remove duplicates" / "Handle missing values" / "Convert format"
-
-### Technical Patterns
-- Working with CSV, Parquet, JSON, NDJSON, or `.duckdb` files
-- File paths ending in `.csv`, `.parquet`, `.json`, `.jsonl`, `.ndjson`, `.tsv`, `.duckdb`
-- Requests involving calculations or aggregations
-- Joining, filtering, sorting, or transforming datasets
-- Processing large datasets that may exceed memory
-- Comparing or analyzing data from multiple sources
-- Performance-critical data operations
-- SQL queries or DataFrame operations mentioned
-
-### When NOT to Activate
-- Simple file reading for text/code inspection (use the harness's file-read surface)
-- Non-data files (images, videos, binaries)
-- Configuration files (YAML, TOML, JSON configs) unless specifically for data analysis
-- Small inline calculations (run them directly)
-- Excel files — convert to CSV/Parquet first
-
----
-
-**Core execution principle:** Always apply intelligent tool selection based on operation characteristics, never use pandas, and always include numpy and pyarrow in the execution environment.
+# Data Scientist: Hybrid-Engine Data Processing
+
+Answer data questions through the cheapest engine and surface that can prove the answer, and
+decide where the computation should live before touching the data.
+
+## Execution surfaces: resident kernel first
+
+A persistent REPL/eval kernel (many harnesses expose one for JavaScript and Python) is the
+default surface. Reason: each one-shot process pays roughly a second of spawn-plus-import
+overhead and re-scans the input file, while a resident connection amortizes both — after a
+one-time load, repeat queries return in milliseconds. Exploration is repeat queries, so this
+difference dominates the session.
+
+1. **JavaScript kernel (Bun)**: run `scripts/ensure-js-deps.sh` once; it prints the absolute
+   import path for `@duckdb/node-api`. Dynamic-import it, connect once, query across cells.
+2. **Python kernel**: the default surface for Python work. duckdb/numpy/matplotlib are
+   typically resident; Polars and pyarrow come from `scripts/ensure-py-deps.sh`, which
+   installs them once into a user cache keyed to the kernel's interpreter —
+   `sys.path.insert` the printed directory and import. The interpreter itself is never
+   mutated.
+3. **uv lane** (`uv run --with ...`): isolation for a heavy or crash-prone one-shot that
+   should not take the kernel down.
+4. **No kernel** (plain-shell harness): the same engines as one-shots — `bun -e` for
+   DuckDB-js, `uv run python -c` for the Python stack — batching several questions per
+   process.
+
+Per-surface patterns and pitfalls: read `references/execution-surfaces.md` before first use.
+
+## Engine selection
+
+- **DuckDB** for SQL-shaped work: direct file queries, joins, aggregation, subqueries,
+  window functions. It queries CSV/Parquet/JSON in place without loading, spills to disk
+  past its memory limit, and reads remote files with the same syntax.
+- **Polars** when the pipeline is DataFrame-shaped: expression-chain transforms, reshapes,
+  streaming datasets past RAM — resident in the Python kernel via `ensure-py-deps.sh`.
+  Read `references/polars-lane.md` — the current 1.x API differs from widely-memorized
+  older spellings.
+- **numpy** when numeric work goes beyond SQL/DataFrame aggregation: statistical tests,
+  linear algebra, FFT, random sampling.
+- **matplotlib** for every chart — read `references/visualization.md` first; it carries the
+  quality bar and a mandatory visual check.
+
+Performance folklore ("X is Nx faster at filtering") varies with data shape, cardinality,
+and hardware. When the engine choice materially matters, measure on the actual data instead
+of trusting remembered multipliers.
+
+## Placement: decide where the computation lives
+
+Probe before you compute — one cell: file size, free RAM, and (when unclear) a row count via
+a direct scan. Then place the work:
+
+- **Load into memory** when the working set stays within roughly a quarter of free RAM AND
+  the session will run repeated queries: `CREATE TABLE t AS SELECT ...` (or a collected
+  DataFrame) once, then iterate. One scan up front converts every later query from a file
+  re-scan into milliseconds.
+- **Query in place / stream** when the question is single-pass, or the data exceeds RAM:
+  DuckDB reads files directly (`FROM 'data.csv'`); past RAM, cap DuckDB's memory and let it
+  spill, or use Polars' streaming engine in the Python kernel. NEVER load a larger-than-RAM
+  dataset fully into memory — swapping stalls the whole machine, while streaming merely
+  takes longer.
+- **Query remotely, in place** when the data lives elsewhere: DuckDB reads http(s)/S3
+  Parquet and CSV with projection and predicate pushdown, so fetch the columns and rows the
+  question needs, never the whole file. When data sits on another machine you can execute
+  on, ship the query to the data and return the small result. Rule: result much smaller
+  than data — move the query; repeated local iteration planned — move a pruned copy of the
+  data once.
+
+Sizing heuristics and recipes: `references/placement.md`.
+
+## Hard rules
+
+- **NEVER use pandas.** DuckDB and Polars beat it decisively on every workload this skill
+  covers, and the environments this skill assumes do not ship it — `.df()` on a DuckDB
+  result raises unless pandas is installed; convert with `.pl()` via Arrow instead.
+- Excel files are not read directly: export to CSV or Parquet first.
+
+## Output contract
+
+Answer the question; report row counts and timing for anything heavy; then stop — no bonus
+charts, no extra exploration passes beyond what the question needed. Chart when asked, or
+when the answer is a shape (trend, distribution, comparison) that prose cannot carry — then
+follow `references/visualization.md` including its visual QA step.
+
+## References
+
+| Read | When |
+| --- | --- |
+| `references/execution-surfaces.md` | before the first query on any surface: kernel patterns, one-shot recipes, escalation rules |
+| `references/polars-lane.md` | DataFrame-shaped pipeline or data past RAM: current API, Arrow handoff, package sets |
+| `references/placement.md` | before heavy or remote work: sizing probe, memory limits, remote reads |
+| `references/visualization.md` | before any chart: type selection, quality bar, CJK fonts, visual QA |
+| `references/uv-setup.md` | uv missing or broken on this machine |
+
+## CLI fallback
+
+When no kernel or REPL surface exists, `uv run scripts/quick-query.py <file> [SQL]`
+(`--filter <polars-sql-expr>`, `--describe`) answers ad-hoc questions with zero code.
+Supports CSV, Parquet, JSON, NDJSON.