pi-eda 0.1.0

package/README.md

@@ -0,0 +1,27 @@
+# pi-eda
+
+Pi skills and subagents for interactive exploratory data analysis.
+
+- `skills/exploring-data`: iterative analysis & plotting
+- `skills/finishing-exploratory-data-analysis`: locking in analysis into a wiki-style report
+- `show` extension: renders a figure to the user in the TUI without spending tokens on it
+
+## Requires
+
+The [`pi-subagents`](https://www.npmjs.com/package/pi-subagents) package, for
+custom-agent support and the `/skill:pi-subagents` dispatch pattern the skills
+reference.
+
+## Install
+
+```bash
+pi install git:github.com/shivam-v-verma/pi-eda
+pi install npm:pi-eda
+```
+
+## Matplotlib style
+
+`skill:exploring-data` respects a matplotlib template `eda.mplstyle`,
+usually located at `~/.config/matplotlib/stylelib/eda.mplstyle`.
+
+A minimal example file `eda.mplstyle.example` is given here.

package/agents/critical-auditor.md

@@ -0,0 +1,23 @@
+---
+name: critical-auditor
+description: Confirms an EDA report/script's claims match the agreed intent, not just clarity
+tools: read, grep, find
+defaultContext: fork
+---
+
+Confirm the report's claims and the script's implementation actually
+match what was agreed -- correct semantic intent, not just clarity. If
+you have no conversation context (fork wasn't available), expect a
+written recap of the framing/decisions instead -- context-free, you're
+a naive reader and thus not able to do your job properly.
+
+- Inputs: the report, figure(s), numbered script(s), and any associated
+  summary data. You need all of the work product to evaluate correctness.
+- Check whether the report/script match what was agreed; flag drift
+  between intent and implementation.
+- Distrust rule: don't trust matching/ordering/canonicalization logic
+  verified only on one easy example -- check it was tested on nontrivial
+  cases that could actually fail.
+- Findings have no tiers: everything you report is drift between intent
+  and implementation. Never silently resolve it yourself -- report it
+  verbatim for the calling agent to surface.

package/agents/job-runner.md

@@ -0,0 +1,34 @@
+---
+name: job-runner
+description: Launches and monitors a heavy/long-running analysis script in the background
+tools: bash, read
+inheritProjectContext: true
+---
+
+You own execution and monitoring of a full-scale run that's already been
+checked and cleared to launch. You do not review or edit the script --
+that already happened before you were dispatched.
+
+## What to do
+
+- Confirm inputs before launching:
+  - The estimated runtime and peak memory from the smoke test (should be
+      in the task you were given -- if missing, ask the dispatching agent
+      for it rather than guessing).
+  - Whether `systemd-run` is available on this machine.
+- Launch the script as a background process (following the project's run
+  convention, e.g. `uv run`).
+- Cap it with a memory limit so a runaway process gets killed by the OS
+  instead of taking down the machine, sized off the estimate with
+  headroom (e.g. 2x the estimated peak):
+  - `systemd-run --scope -p MemoryMax=<budget> --user -- uv run <script>`
+      if available.
+  - Otherwise, say so explicitly and fall back to periodic `ps`/`smem`
+      checks with a manual kill.
+- Monitor progress and memory use periodically rather than blocking on it
+  synchronously -- this is in addition to the hard cap above, not a
+  replacement for it, since you want to know about a job trending toward
+  the ceiling before it gets killed.
+- Report back once it completes: success/failure, runtime, and where the
+  output landed. On failure, include the actual error, not just "it
+  failed."

package/agents/naive-reader.md

@@ -0,0 +1,31 @@
+---
+name: naive-reader
+description: Confirms an EDA report reads as a clear, complete, standalone artifact
+tools: read
+defaultContext: fresh
+---
+
+Confirm the report (with `index.md`) reads as a clear, comprehensive,
+standalone artifact -- no missing context, no unexplained gaps. Read it
+cold, as if returning in 6 months having forgotten everything.
+
+- Inputs: report file, figure(s), summary CSV(s), and `index.md` for
+  project context. You may open the script only to resolve a specific
+  confusion, never to run it.
+- Comprehension only -- you never judge whether a choice was right, only
+  whether it's explained.
+- Confirm every figure file passed to you is actually embedded in the
+  report -- an unreferenced figure is material-tier, since there isn't an
+  interpretation for it.
+
+## Severity rubric
+
+| Tier | Definition |
+| --- | --- |
+| cosmetic | wording/labeling nit, doesn't change interpretation |
+| confusing | correct and complete, but too dense/jargon-heavy to parse easily |
+| clarifying | you can't tell why/what/how from the report alone. If you had to open the script to answer your own question, that's automatic clarifying-tier |
+| material | your conclusion contradicts what the report claims, or the claim isn't backed by anything shown |
+
+Report every finding with its tier. Do not silently decide a fix for
+anything -- that's the calling agent's call, not yours.

package/agents/script-checker.md

@@ -0,0 +1,27 @@
+---
+name: script-checker
+description: Checks a script against stated intent/framing before a heavy scale-up run
+tools: read, grep, find
+inheritProjectContext: true
+---
+
+You check a script against the user's stated intent/framing before an
+expensive full-scale run. You review; you do not edit the script or run
+it.
+
+## What to check
+
+- Does each transform (groupby, join, pivot, filter, etc.) actually do
+  what the stated intent/framing says it should.
+- Any matching/ordering/canonicalization logic verified only on an
+  easy/symmetric case -- that's a red flag, not a pass. Ask whether it
+  was tested against a case that could actually fail.
+- Anything that would silently produce a plausible-but-wrong result
+  (wrong join key, off-by-one window, unintended dtype coercion).
+
+## Output
+
+- If everything checks out, say so plainly -- don't invent issues.
+- If something's off, name the exact line/transform and what's wrong
+  with it, specific enough that the calling agent can fix it without
+  guessing.

package/eda.mplstyle.example

@@ -0,0 +1,28 @@
+# Shared matplotlib style for /skill:exploring-data.
+# This is a template shown for example; include in default matplotlib 
+# style sheet dir (see README.md).
+# See plotting.md for the conventions this backs and what still needs a
+# per-script judgment call (aspect ratio, bin count, title text, etc).
+#
+# Full rcParams menu: 
+# https://github.com/matplotlib/matplotlib/blob/main/lib/matplotlib/mpl-data/matplotlibrc
+
+figure.dpi: 150
+savefig.dpi: 150
+savefig.format: png
+savefig.bbox: tight
+figure.autolayout: True
+
+axes.grid: False
+# CSS named colors: https://developer.mozilla.org/en-US/docs/Web/CSS/named-color
+axes.prop_cycle: cycler('color', ['lightsteelblue', 'darksalmon', 'darkseagreen', 'plum', 'burlywood', 'rosybrown'])
+
+font.size: 10
+axes.titlesize: 10
+axes.labelsize: 10
+legend.fontsize: 10
+xtick.labelsize: 10
+ytick.labelsize: 10
+legend.frameon: False
+
+image.cmap: Blues

package/extensions/show.ts

@@ -0,0 +1,114 @@
+/**
+ * Show Extension - Render an image to the user without spending LLM tokens on it.
+ *
+ * `show` reads an image file and renders it in the TUI via the built-in
+ * `Image` component. The LLM only ever sees a short confirmation string in
+ * `content` (what gets sent back to the model) - the image bytes live in
+ * `details`, which is TUI-only and never enters LLM context. If the model
+ * later needs to actually look at the image, it uses the normal `read` tool.
+ */
+
+import { readFile, stat } from "node:fs/promises";
+import { extname, resolve } from "node:path";
+import type { ExtensionAPI } from "@earendil-works/pi-coding-agent";
+import { Image, Text } from "@earendil-works/pi-tui";
+import { Type } from "typebox";
+
+const MEDIA_TYPES: Record<string, string> = {
+	".png": "image/png",
+	".jpg": "image/jpeg",
+	".jpeg": "image/jpeg",
+	".gif": "image/gif",
+	".webp": "image/webp",
+	".bmp": "image/bmp",
+};
+
+interface ShowDetails {
+	path: string;
+	base64: string;
+	mediaType: string;
+}
+
+export default function (pi: ExtensionAPI) {
+	pi.registerTool({
+		name: "show",
+		label: "Show",
+		description:
+			"Display an image to the user in the terminal without loading its pixels into your own context. " +
+			"Use read on the same path afterward if you need to analyze the image yourself.",
+		promptSnippet:
+			"Show an image to the user in the terminal (does not add the image to your context)",
+		promptGuidelines: [
+			"Use show to display an image the user asked to see; use read instead when you need to look at the image yourself.",
+		],
+		parameters: Type.Object({
+			path: Type.String({
+				description: "Path to the image file (png, jpg, gif, webp, bmp)",
+			}),
+		}),
+
+		async execute(_toolCallId, params, _signal, _onUpdate, ctx) {
+			const rawPath = params.path.startsWith("@")
+				? params.path.slice(1)
+				: params.path;
+			const absolutePath = resolve(ctx.cwd, rawPath);
+
+			const mediaType = MEDIA_TYPES[extname(absolutePath).toLowerCase()];
+			if (!mediaType) {
+				return {
+					content: [
+						{ type: "text", text: `Not a supported image type: ${params.path}` },
+					],
+					isError: true,
+				};
+			}
+
+			const fileStat = await stat(absolutePath).catch(() => null);
+			if (!fileStat?.isFile()) {
+				return {
+					content: [{ type: "text", text: `File not found: ${params.path}` }],
+					isError: true,
+				};
+			}
+
+			if (ctx.mode !== "tui") {
+				return {
+					content: [
+						{
+							type: "text",
+							text: `Cannot render images outside interactive TUI mode (${ctx.mode}).`,
+						},
+					],
+					isError: true,
+				};
+			}
+
+			const base64 = (await readFile(absolutePath)).toString("base64");
+
+			return {
+				content: [{ type: "text", text: `Displayed ${params.path} to the user.` }],
+				details: { path: params.path, base64, mediaType } as ShowDetails,
+			};
+		},
+
+		renderCall(args, theme, _context) {
+			return new Text(
+				theme.fg("toolTitle", theme.bold("show ")) + theme.fg("muted", args.path),
+				0,
+				0,
+			);
+		},
+
+		renderResult(result, _options, theme, _context) {
+			const details = result.details as ShowDetails | undefined;
+			if (!details) {
+				const text = result.content[0];
+				return new Text(text?.type === "text" ? text.text : "", 0, 0);
+			}
+			return new Image(details.base64, details.mediaType, theme, {
+				maxWidthCells: 80,
+				maxHeightCells: 24,
+			});
+		},
+	});
+}

package/package.json

@@ -0,0 +1,27 @@
+{
+  "name": "pi-eda",
+  "version": "0.1.0",
+  "description": "Pi tooling for interactive exploratory data analysis.",
+  "license": "MIT",
+  "author": "Shivam V Verma",
+  "keywords": ["pi-package", "pi", "eda", "exploratory-data-analysis"],
+  "files": [
+    "README.md",
+    "agents",
+    "eda.mplstyle.example",
+    "extensions",
+    "skills"
+  ],
+  "peerDependencies": {
+    "@earendil-works/pi-coding-agent": "*",
+    "@earendil-works/pi-tui": "*",
+    "typebox": "*"
+  },
+  "pi": {
+    "extensions": ["./extensions"],
+    "skills": ["./skills"],
+    "subagents": {
+      "agents": ["./agents"]
+    }
+  }
+}

package/skills/exploring-data/SKILL.md

@@ -0,0 +1,70 @@
+---
+name: exploring-data
+description: Use when exploring a dataset, generating plots, or iterating on data analysis scripts
+---
+
+> **Related skills:** Dispatch the `script-checker` with `/skill:pi-subagents`. Once a figure/analysis seems meaningful, use `/skill:finishing-exploratory-data-analysis` before writing it up or committing.
+
+# Exploring Data
+
+**Scope, small to large:**
+
+- *plot*: a chart type/category (e.g. histogram, scatter)
+- *figure*: one rendered image
+- *question*: one concept to test through one or more figures
+- *analysis*: one `.md` *report* file + its paired scripts/figs folder
+- *project*: the whole `eda/` directory, created across many
+  `/skill:exploring-data` sessions
+
+## Overview
+
+**Core principle:** Your job is to help the user surface hypotheses,
+and suggest or build the plots that interrogate them. What a confirmed pattern
+*means*, the story it tells, is the user's call: hand back a clear summary of
+what the figure shows, not a conclusion dressed as a finding.
+
+See `workflow.md` for stages of this skill. In all stages, confirm framing
+with the user. Never run a batch of steps by yourself; always interleave
+by checking for the user's insight/thoughts.
+
+A good hypothesis is usually a simple, clean one -- reachable through better
+framing of the question, not through aggressively slicing the dataset more ways
+looking for a signal.
+
+## Directory Structure
+
+Default to `eda/` at the project root (ask if a different name/location is
+wanted). See `directory-structure.md` for the full layout convention
+(scripts/figs tree, numbering rules, shared-code placement).
+
+## Core Data Practices
+
+Always, whether running a numbered script or an ad hoc check (e.g.
+`python -c "..."`):
+
+- Check input file size(s) and be aware of runtime/memory.
+- Set a timeout via the `timeout` param (seconds) in pi's `bash` tool
+  on any check or smoke test. If it times out, reduce sample size and
+  extrapolate rather than waiting for a highly accurate estimate.
+- Don't delete scripts or attempts yourself unless the user explicitly
+  asks to. If a script is paused or abandoned, leave a module/function
+  docstring giving its status and where it was left off.
+- If processing is heavy, decide with the user whether to run on the
+  full dataset or a representative subset for this pass.
+
+## Common Mistakes
+
+The most cross-cutting/devastating failures. Each reference file above has
+its own narrower table.
+
+| Mistake | Fix |
+| --- | --- |
+| Running the full dataset before a smoke test | See `checking-and-running-scripts.md` -- always sample/time on a subset first |
+| Auto-advancing to the next question without a framing check | See `pre-code-checkpoints.md` -- confirm framing before each new question, not once for the whole analysis |
+| Picking the report/subfolder name yourself instead of asking | Ask before creating it -- one name, shared by the file and its folder |
+| Launching a heavy scale-up run while the `script-checker` is still pending | See `checking-and-running-scripts.md` -- wait for it, or the expensive run finishes before a material issue could surface |
+| Deleting/pruning a failed or paused script | Leave it in place with a status docstring unless the user explicitly asks to remove it |
+| Treating an approval-sounding signal ("looks great", "keep going", "I trust your judgment") as done | Ask whether to move to `/skill:finishing-exploratory-data-analysis` -- don't auto-advance and don't stay silent |
+| Chasing a pattern that only appears after trying many slices/cuts of the data | Multiple-comparisons risk -- go back to framing, not more cuts |
+| Stating a small or non-random-sample result as a general fact | State it as a property of the sample examined ("true in this subset") until checked for generalization |
+| Writing to `index.md` or the report before `/skill:finishing-exploratory-data-analysis` | Both are created/updated together at lock-in checkpoint -- confirm scope/framing conversationally during exploration instead |

package/skills/exploring-data/checking-and-running-scripts.md

@@ -0,0 +1,97 @@
+# Checking and Running Scripts
+
+Building confidence in a script for `/skill:exploring-data`, then deciding
+how to run it. See `workflow.md` for where this fits in the full loop
+(the light/heavy fork is inlined there, each branch with its own check
+step and loop-back on findings).
+
+## Sanity Checking a Script
+
+- After each nontrivial transform (groupby, join, pivot, filter, etc.), do
+  a quick check (`.head()`, row counts, spot-check a few values) that the
+  output matches what the transform should have done -- not just once at
+  the end of the script.
+- Pick a case that could actually fail, not the first one at hand --
+  especially for canonicalization/ordering/matching logic, where an easy
+  case can pass while hiding a real bug.
+
+## Smoke Testing a Script
+
+- Run on a sampled/head subset (fixed seed) and time it before running at
+  full scale.
+- After it runs, glance at the actual output values, not just "it
+  completed in time." A fast clean run on garbage output is not a
+  passing smoke test.
+- Exception: a cached-read-and-plot script -- one that purely reads an
+  already-validated cached table and plots it, with zero computation (no
+  groupby, join, pivot, or filter). State that justification explicitly
+  instead of skipping silently.
+
+## Judging Heavy vs. Light Scale-up
+
+Goal: increase iteration speed without being reckless. Either extreme
+wastes the user's time -- blocking every quick diff on a check likely to
+pass, or committing to a large/expensive run that turns out wrong.
+
+- Judge from the smoke-test timing/size ratio whether the full-scale run
+  will be heavy (long-running or memory-heavy relative to the sample --
+  e.g. 1% took a second but full scale extrapolates to minutes).
+- Rough thresholds based on estimated time: light if under ~1 minute,
+  heavy if over ~5 minutes. In between is a judgment call.
+- State that judgment explicitly: "sample took Xs, full run estimated ~Y,
+  treating as heavy."
+
+## Light Scale-up
+
+- No `script-checker` dispatch -- reread the script yourself before
+  running: does it match the user's stated intent/framing, does each
+  nontrivial transform do what it should.
+- If the reread surfaces an issue, fix it and reread again before
+  running -- don't run first and raise the concern after.
+- This is a deliberately lower bar than the heavy path: a light-path
+  mistake is cheap to catch downstream too, since the run finishes fast
+  and you're looking at the output right away. The real second-set-of-
+  eyes gate for the analysis still happens at
+  `/skill:finishing-exploratory-data-analysis`'s lock-in checkpoint --
+  this reread isn't the last line of defense, it's a cheap intermediate
+  one.
+
+## Heavy Scale-up
+
+- Dispatch the `script-checker` before launching:
+  `subagent({agent:"script-checker", task:"<script path + stated intent/framing>"})`.
+- Do not launch until it has returned and come back clean.
+- If it's still running, wait for it (or say explicitly you're waiting)
+  -- don't launch anyway planning to fold in feedback once it lands.
+- Launching "in parallel while the `script-checker` runs in the
+  background" still counts as launching without it -- the expensive run
+  finishes and gets shown to the user before a material issue could
+  surface.
+- A `script-checker` pass only checks the script -- it never substitutes
+  for the lock-in checkpoint in
+  `/skill:finishing-exploratory-data-analysis` once a figure/question is
+  ready to show the user.
+- Two or more edits since the last `script-checker` dispatch count as one
+  nontrivial change on their own -- dispatch again at that point, don't
+  wait for it to feel big enough.
+
+### Dispatching the `job-runner`
+
+- Once clear to launch:
+  `subagent({agent:"job-runner", task:"<script path + output location + estimated runtime + estimated peak memory>"})`.
+  This frees the main session to keep iterating while it runs. Pass the
+  runtime/memory estimate from your smoke test (see "Judging Heavy vs.
+  Light Scale-up" above) rather than making it re-derive them blind.
+
+## Common Mistakes
+
+| Mistake | Fix |
+| --- | --- |
+| Running the full dataset (or an ad hoc exploratory probe) before a smoke test | Always sample/time on a subset first -- applies to one-off probes too, not just numbered scripts |
+| Silently skipping the smoke test on a "just read cache and plot" script | Only skip for zero-computation cached-read-and-plot, and state that justification explicitly |
+| Trusting a sanity check that only ran on an easy/symmetric case | Pick a case that could actually have failed, especially for matching/ordering/canonicalization logic |
+| Checking that a smoke test ran fast/clean without looking at what it produced | Glance at the actual output values -- a clean run on garbage isn't a pass |
+| Running a light-path script without rereading it first because it's "just quick" | Reread against the stated intent/framing before running -- speed doesn't waive the check, it just changes who does it |
+| Launching a heavy scale-up "while the `script-checker` runs in the background" instead of waiting for it | Confirm the `script-checker` came back clean before launching -- don't launch and plan to fold feedback in later |
+| Letting 2+ small edits pile up on a heavy-path script since the last `script-checker` dispatch because none felt big enough alone | Dispatch again once 2+ edits have landed |
+| Treating a mid-work `script-checker` pass as covering the finishing skill's lock-in checkpoint | Different gate, different artifact -- a pending or completed `script-checker` pass never substitutes for it |

package/skills/exploring-data/directory-structure.md

@@ -0,0 +1,44 @@
+# Directory Structure
+
+Layout convention for `/skill:exploring-data`.
+
+Default to `eda/` at the project root (ask if a different name/location is
+wanted).
+
+```text
+eda/
+  index.md                    # landing pad for readers -- what's been done,
+                               # links to each report below
+  customer_churn.md           # a report -- lives directly under eda/
+  customer_churn/             # paired folder, same stem as the report file
+    scripts/
+      common_data.py          # unnumbered: shared code, not an analysis step
+      01_description.py
+      02_description.py
+    figs/
+      01_description.png
+      01_description_b.png    # one script can produce many figures
+      02_description.png
+```
+
+- `index.md` is a skimmable document for first-time readers, briefly
+  describing what analyses have been done, with links to the reports.
+- A report (`<name>.md`) is the analysis's confirmed write-up -- its
+  questions, each with approach/results/figures, and a conclusion.
+- The report file and its folder must share the exact same stem -- that's
+  the only thing that pairs them, no manifest or frontmatter needed.
+- `index.md` and the report are only created/updated during
+  `/skill:finishing-exploratory-data-analysis`, and not during exploration.
+- No strict 1:1 between scripts, figures, and a report's *Questions* -- number for
+  correlation, don't enforce it.
+- Numbered scripts (`NN_description.py`) can't import each other.
+- Shared code (data loading, plotting helpers, constants) goes in an
+  unnumbered `common_<topic>.py` and gets imported from there instead --
+  never import one numbered script from another.
+
+## Common Mistakes
+
+| Mistake | Fix |
+| --- | --- |
+| Naming the report file and its folder with different stems | They must match exactly -- that's the only thing that pairs them |
+| Importing one numbered script from another | Shared code goes in `common_<topic>.py` instead |

package/skills/exploring-data/plotting.md

@@ -0,0 +1,74 @@
+# Plotting Preferences
+
+Style conventions and the figure-production process for
+`/skill:exploring-data`. Defaults below apply to every plot
+type; subsections add rules for a specific type. If a plot type has no
+subsection yet, follow Defaults and ask if unsure -- don't invent a
+per-type rule that hasn't been stated.
+
+## Process
+
+- Generate figures with matplotlib, saved as PNG, following the conventions
+  below.
+- Apply the user's `eda.mplstyle` at the top of the script
+  (`plt.style.use("eda")`) and respect user defaults.
+- Visually inspect every figure with the `read` tool before treating it as a
+  result -- never make hypotheses from a figure you haven't looked at.
+- `read` is for your own inspection. When you're ready to show the figure to
+  the user (the "Summarize the answer" handoff), use `show` to display it.
+  This will render the PNG to the user without spending your tokens
+  rereading it.
+- Check specifically for: overlapping/clipped labels, indistinguishable
+  traces/series, and rendering artifacts (banding, gaps) that don't reflect
+  real structure in the data.
+- Save a table alongside the figure if it's built from a compact aggregate
+  (roughly <=20 distinct values) rather than a full-row plot -- cheap at that
+  size, and it avoids guessing what the report will cite.
+
+## Defaults
+
+If the user has not set up `eda.mplstyle` or has not offered opinions
+on plot style, use these defaults:
+
+- Colors: `eda.mplstyle` (see the repo-root `eda.mplstyle.example`) defaults to
+  `lightsteelblue` for one color and `Blues`
+  for a colormap -- pick something else only if needed to convey a point, e.g.
+  categorical comparison.
+- Gridlines: none. Spines / ticks: matplotlib defaults -- don't add or strip
+  either.
+- Titles: Be brief -- words must earn their space. Detailed descriptions belong
+  in report prose, not the title.
+- Axis labels and legends: always present, be terse.
+- Sample size: prefer density/percentage over raw counts unless there's a
+  specific reason to show counts.
+
+## Bar-shaped Plots (Bar Charts, Histograms)
+
+- Aspect ratio: 8:3 (wide).
+- Shaded (filled) bars, not an unfilled outline/step line.
+- **Histograms only**:
+  - For continuous variables or integer variables with >= 30 bins,
+      suggest KDE smoothing with filled plots. Never start with smoothing;
+      instead, show an initial histogram without smoothing and suggest it.
+  - For integer variables/PMF plots that are graphed without smoothing,
+      ensure the number of bins aligns with the x range. Graph with shaded
+      bars only.
+
+## Square-aspect Plots (Scatter Plots, 2D Histograms)
+
+- Aspect ratio: 1:1 (square).
+- **2D Histograms only** -- include a colorbar sized to the height of
+  the histogram for visual clarity.
+
+## Common Mistakes
+
+| Mistake | Fix |
+| --- | --- |
+| Describing what a figure shows without having looked at the PNG | Always `read` the figure before writing anything about what it shows |
+| Summarizing a figure in prose only, without the user seeing it | `show` the PNG to the user as part of the handoff, then summarize |
+| Building a plot from a compact aggregate (<=20 values) with no saved table | Save the underlying table alongside the figure -- cheap at that size |
+| Inventing a per-type styling rule for a plot type with no subsection yet | Follow Defaults and ask -- don't guess a convention that hasn't been stated |
+| Adding a KDE overlay to a discrete-variable histogram | KDE implies continuous density -- shaded bars only, even at cleanup time |
+| Histogram bars show banding or an every-other-bar artifact | Try a different bin width/count, or overlay a KDE to check if the shape is real or a binning artifact |
+| Comparing multiple histograms on one axes with different bin edges per series | Use identical bin edges across all series being compared |
+| A scatter plot has so many overlapping points it's an unreadable blob | Switch to `hist2d`/hexbin (density) or drop alpha significantly, instead of a plain scatter |

package/skills/exploring-data/pre-code-checkpoints.md

@@ -0,0 +1,63 @@
+# Pre-Code Checkpoints
+
+What to do for `/skill:exploring-data` before you write any
+code for the next question.
+
+## Scope the analysis (once per analysis, before Peek at the data)
+
+- Before opening the data, check in one or two sentences what is broadly in
+  scope for this analysis and get the user's confirmation. This is a "what
+  are we even doing here", separate from and prior to the per-question framing
+  below.
+- Ask the user for a report name for this analysis -- it names both the
+  report file and its paired folder (see `directory-structure.md`).
+  Don't pick one yourself.
+- Keep it broad on purpose -- treat it as provisional, not a lock. Expect
+  it to narrow or shift as the data reveals what's actually there, well
+  before you reach conclusions -- confirm the new framing with the user
+  when it does, same as any other question (see below). `index.md` and
+  the report are written together during
+  `/skill:finishing-exploratory-data-analysis`, not during exploration.
+
+## Peek at the data (once per new dataset, before Propose/ask a question)
+
+- Load the dataset (lazily), check schema/dtypes.
+- Get null counts and basic summary stats (mean/median, min/max, std).
+- Get a feel for the actual data: `.head()`, a strided sample, or a random
+  sample (fixed seed).
+- Read a few raw rows by eye. They catch what schema/dtype checks and basic
+  stats above miss: a "numeric" column that's really a formatted string,
+  an ID convention, units, or what a free-text field actually holds.
+- Even raw rows and basic stats can miss duplicate rows.
+- Beyond that, think about what else is cheap to calculate and might
+  reveal something missed above -- e.g. sentinel/placeholder values (`-1`,
+  `0`, `"unknown"`), class imbalance, or a grain mismatch after a join.
+
+## Propose/ask a question (once per question, every time)
+
+- Confirm the framing for *this* question/figure with the user before
+  writing a script for it.
+- State the new framing in a sentence before scripting it, even if a broad
+  go-ahead was already given for the analysis as a whole.
+- This applies to every question, including one you decide to add on your
+  own after an earlier go-ahead. Approval for one question is not
+  approval for the next one the user hasn't seen proposed yet.
+- **A blanket "keep going" / "I trust your judgment" is not per-question
+  approval.** It authorizes you to keep working on framing, not to skip
+  asking about the next question's framing. Treat it the same as
+  silence: propose the next question's framing in one sentence, then
+  build it.
+- Do not build multiple questions in one pass and hand over a "finished
+  report" -- stop after each one and surface what it shows before
+  moving to the next.
+
+## Common Mistakes
+
+| Mistake | Fix |
+| --- | --- |
+| Opening the data before agreeing on a broad scope | State the analysis's scope/question area in a sentence and confirm before Peek at the data |
+| Picking the report/subfolder name yourself instead of asking | Ask before creating it -- one name, shared by the file and its folder |
+| Treating EDA as a pipeline and auto-advancing through questions without a framing check | Confirm framing per question, not once for the whole analysis |
+| Adding a follow-up figure/question under an earlier blanket go-ahead without confirming its framing | State the new framing in a sentence before scripting it |
+| Building the next two questions back-to-back and presenting a "complete report" because the user said "keep going" / "I trust your judgment" | That phrase covers continued work, not skipped per-question confirmation -- stop after each one |
+| Skipping the raw-row eyeball and stopping at summary stats | Read a few actual rows -- aggregates hide format/unit/encoding surprises |

package/skills/exploring-data/workflow.md

@@ -0,0 +1,66 @@
+# Workflow
+
+Loop/flow-control for `/skill:exploring-data`.
+
+The user drives this loop: they propose or pick the next thing to plot or
+check, you build it and return what it shows. They then interpret it
+and pick next steps or ask additional questions.
+
+This is a looped pipeline, exiting only when the user decides to
+lock-in an analysis.
+
+```dot
+digraph EDA_WORKFLOW {
+    "Start analysis" [shape=doublecircle];
+    "Scope the analysis" [shape=box, style=filled, fillcolor=lightblue];
+    "Peek at the data" [shape=box, style=filled, fillcolor=lightyellow];
+    "Propose/ask a question" [shape=box, style=filled, fillcolor=lightblue];
+    "Build script/plot" [shape=box];
+    "light or heavy?" [shape=diamond];
+    "Reread script" [shape=box];
+    "Dispatch script-checker (blocking)" [shape=box];
+    "run now, this session" [shape=box];
+    "Dispatch job-runner" [shape=box];
+    "Summarize the answer" [shape=box, style=filled, fillcolor=lightyellow];
+    "Continue analysis or lock in?" [shape=diamond, style=filled, fillcolor=lightblue];
+    "/skill:finishing-exploratory-data-analysis" [shape=doublecircle];
+
+    "Start analysis" -> "Scope the analysis";
+    "Scope the analysis" -> "Peek at the data";
+    "Peek at the data" -> "Propose/ask a question";
+    "Propose/ask a question" -> "Build script/plot";
+    "Build script/plot" -> "light or heavy?";
+    "light or heavy?" -> "Reread script" [label="light"];
+    "Reread script" -> "Build script/plot" [label="issues found"];
+    "Reread script" -> "run now, this session" [label="looks right"];
+    "light or heavy?" -> "Dispatch script-checker (blocking)" [label="heavy"];
+    "Dispatch script-checker (blocking)" -> "Build script/plot" [label="issues found"];
+    "Dispatch script-checker (blocking)" -> "Dispatch job-runner" [label="clean"];
+    "run now, this session" -> "Summarize the answer";
+    "Dispatch job-runner" -> "Summarize the answer";
+    "Summarize the answer" -> "Continue analysis or lock in?";
+    "Continue analysis or lock in?" -> "Propose/ask a question" [label="next question"];
+    "Continue analysis or lock in?" -> "/skill:finishing-exploratory-data-analysis" [label="finish analysis"];
+}
+```
+
+(blue = User turn, yellow = Agent -> User handoff, white = Agent-only)
+
+| Stage | Turn | What it looks like | See |
+| --- | --- | --- | --- |
+| Scope the analysis | User | State the broad scope/question area for this analysis and confirm it, once per analysis, before touching data | `pre-code-checkpoints.md` (Scope the analysis) |
+| Peek at the data | Agent -> User | Schema/dtypes, `.head()`, a sample, eyeball raw rows, hand back for a look | `pre-code-checkpoints.md` (Peek at the data) |
+| Propose/ask a question | User (agent may suggest) | Confirm framing for the next question, then the user picks what to plot or check | `pre-code-checkpoints.md` (Propose/ask a question) |
+| Build script/plot | Agent | Write the script/plot, smoke test on a subset before scaling | `checking-and-running-scripts.md`, `plotting.md` |
+| light or heavy? | Agent | Judge from the smoke-test timing/size ratio | `checking-and-running-scripts.md` |
+| Reread script | Agent | Light: reread the script yourself before running -- no `script-checker` dispatch, loop back to the script on issues you find | `checking-and-running-scripts.md` |
+| Dispatch script-checker (blocking) / Dispatch job-runner | Agent | Heavy: dispatch the `script-checker` and wait; loop back to the script on findings; don't launch the `job-runner` until it's clean | `checking-and-running-scripts.md` |
+| Summarize the answer | Agent -> User | Interpret the figure/stats as a summary, not a conclusion -- the user decides what it means and what's next | `SKILL.md` Core principle |
+| Lock-in checkpoint | User | Figure/analysis seems meaningful, ready to write up or commit | `/skill:finishing-exploratory-data-analysis` |
+
+## Common Mistakes
+
+| Mistake | Fix |
+| --- | --- |
+| Treating any issue in scripts (via self-reread or `script-checker`) as a note to fold in later instead of a loop back to the script | Go to "Build script/plot" and fix it -- see the graph's "issues found" edges |
+| Launching a heavy scale-up while the `script-checker` is still pending or has open findings | See `checking-and-running-scripts.md`'s Common Mistakes |

package/skills/finishing-exploratory-data-analysis/SKILL.md

@@ -0,0 +1,70 @@
+---
+name: finishing-exploratory-data-analysis
+description: Use when a figure or EDA analysis seems meaningful and ready to summarize, write into a report, or commit
+---
+
+> **Related skills:** Use `/skill:exploring-data` for the exploration/plotting phase this follows. Use `/skill:verification-before-completion` before treating any figure as a finished result. Use `/skill:pi-subagents` (`runs.all`, `context: "fresh"`/`"fork"`) for the two dispatches in `verification-dispatch.md`.
+
+# Finishing Exploratory Data Analysis
+
+## Overview
+
+Locking in a figure/analysis is a sequential, one-at-a-time gate, not a
+batch step at the end of a session.
+
+**Core principle:** never write to the report or commit from your own
+read of a figure -- confirm the story with the user first, every time. A
+broader go-ahead, or the user saying "ship it" about the figure itself,
+confirms the figure's story -- not this unrun verification step.
+
+Once the report is written or updated, don't skip straight to offering to
+commit -- run the verification dispatch (step 4) first. A completed code
+review doesn't count (see Common Mistakes).
+
+## Lock-in Checkpoint (do this once a figure/analysis seems meaningful)
+
+1. **Walk through the figures with the user.** `show` each figure and
+   re-examine it together to confirm the narrative -- no written artifact
+   yet.
+2. **Summarize and wait.** Give the user a ~4 sentence written summary
+   of the agreed narrative. Wait for confirmation before proceeding.
+3. **Write the report and `index.md`.** Once confirmed, add or update
+   the `index.md` and report (`<report_name>.md`, sibling to its paired
+   `<report_name>/` scripts/figs folder). Create them here if necessary.
+   - Writing style: clear sentences. No run-ons, reduce em/en-dash
+     (`--`/`—`) as a sentence joiner. Split into two sentences or use a
+     comma/colon instead.
+   - `index.md` contents -- a brief description (no figures) containing:
+       - *Link to report*
+       - *Brief summary*
+       - *Scope of analysis*
+       - *Dataset description*
+       - *Open questions*, if any
+   - Report contents -- at minimum:
+       - *Scope of analysis*
+       - *Data source(s) used*
+       - *Questions*, grouped when closely related:
+         - *Approach* in prose, with key equations if nontrivial
+         - *Results*, with the embedded figure(s)
+       - *Conclusion*
+4. **Reread, then dispatch verification.** See `verification-dispatch.md`
+   for the reread pass, the `naive-reader`/`critical-auditor` dispatch, and
+   the severity rubric for findings.
+5. **Integrate findings and modify the report.** Apply silent fixes per
+   the rubric; surface material-tier and `critical-auditor` findings
+   verbatim, flagged unresolved if not yet addressed. If the fix is
+   substantive, ask the user whether to rerun verification.
+6. **Ask before committing.** Explicitly ask if the user is ready to
+   commit -- don't commit on your own judgment.
+
+## Common Mistakes
+
+| Mistake | Fix |
+| --- | --- |
+| Writing to the report from your own interpretation of a figure | Confirm with a ~4 sentence summary first |
+| Putting figures/results into `index.md` | Keep it a skimmable summary of what's been done -- figures and results belong in the report files |
+| Dispatching verification before the report is written | Verification checks the written report against the figures/script -- write first (step 3), then dispatch (step 4) |
+| Stating a correlation as if it were causal | Hedge appropriately in report prose; correlation is not causation |
+| Silently patching a material-tier or `critical-auditor` finding before telling the user | Always surface these verbatim in the summary, even unresolved -- only cosmetic/confusing/non-fabricated-clarifying fixes get silently applied |
+| Treating a completed script/code review as satisfying this checkpoint's dispatch | Different gate, different artifact -- a code review checks the script, this checkpoint checks the written report. See `verification-dispatch.md` |
+| Saying an analysis is "done and reviewed" once code review passes, before the lock-in checkpoint has run | Say so explicitly ("script validated; report still needs the lock-in checkpoint") rather than a blanket "done" that could be read as covering sign-off too |

package/skills/finishing-exploratory-data-analysis/verification-dispatch.md

@@ -0,0 +1,51 @@
+# Verification Dispatch
+
+The critical checkpoint in `/skill:finishing-exploratory-data-analysis`:
+reread the report/script, then dispatch subagents to check it, before
+you offer to commit. Together the two dispatches establish trust in
+the report: that it's clear and complete as a standalone artifact
+(`naive-reader`), and that it was produced with the correct semantic
+intent (`critical-auditor`). See each agent's own definition in this
+package's `agents/` directory (`naive-reader.md`, `critical-auditor.md`)
+for what they check and how they're configured (context, tools).
+
+1. **Reread first.** Reread the report/script yourself for leftover
+   editing artifacts (draft asides, TODO notes, contradictions) before
+   dispatching -- don't hand reviewers an unread draft.
+2. **Dispatch both in parallel** (`runs.all`), every lock-in, not just
+   risky ones:
+
+   ```js
+   runs.all([
+     {key:"naive", agent:"naive-reader", task:"<report, figures, index.md>"},
+     {key:"audit", agent:"critical-auditor", task:"<report, figures, numbered scripts>"},
+   ])
+   ```
+
+   If fork context isn't available for `critical-auditor`, hand it a
+   written recap of this session's framing/decisions in the task instead
+   -- context-free it's just a second `naive-reader`.
+
+## Handling `naive-reader` findings by tier
+
+| Tier | Handling |
+| --- | --- |
+| cosmetic | fix silently, one-line changelog note |
+| confusing | fix silently (simplify/rewrite), one-line changelog note |
+| clarifying | fix silently if you actually know the answer. If closing the gap means guessing or inventing a reason, don't fabricate it -- tell the user honestly instead |
+| material | always surface, never silently patched -- this is a correctness signal wearing a comprehension-test disguise |
+
+## Handling `critical-auditor` findings
+
+Findings have no tiers: everything it reports is drift between intent
+and implementation, never silently resolved -- goes into the
+user-facing summary verbatim.
+
+## Common Mistakes
+
+| Mistake | Fix |
+| --- | --- |
+| Skipping the `naive-reader`/`critical-auditor` dispatch because it "seems fine" | Every lock-in, no exceptions -- this is the final check before commit, and "seems fine" is exactly the curse-of-knowledge blind spot it exists to catch |
+| Inventing a plausible-sounding rationale to resolve a clarifying-tier gap you don't actually know the answer to | Surface it as an open question instead -- never fabricate reasoning to make a flag disappear |
+| Letting the `critical-auditor` run with no conversation context because fork wasn't available | Hand it an explicit written recap of the framing/decisions instead -- context-free, it's just a second `naive-reader` and can't catch intent drift |
+| Treating a completed script/code review as satisfying this dispatch | Different gate, different artifact -- a code review checks the script; this dispatch checks the written report |