goad-toolkit 0.2.7__tar.gz → 0.2.9__tar.gz

goad_toolkit-0.2.9/.claude/worktrees/quizzical-solomon-b62511/.claude/settings.local.json

@@ -0,0 +1,7 @@
+{
+  "permissions": {
+    "allow": [
+      "Bash(cd /Users/rgrouls/code/courses/goad_toolkit && cat pyproject.toml)"
+    ]
+  }
+}

goad_toolkit-0.2.9/.claude/worktrees/quizzical-solomon-b62511/CHANGELOG.md

@@ -0,0 +1,152 @@
+# Changelog
+
+## 0.2.7
+
+### Added
+
+- `ScatterPlot`, `RegPlot` and `CorrelationHeatmap`: the relation plots for two-variable
+  questions, in their order of use — a scatter first, a fit once the scatter has shown which
+  kind, a correlation matrix when there are too many pairs to plot by hand.
+- `ScatterPlot` defaults `alpha` below 1, since overplotting is the standard way a scatter
+  misreports its own density.
+- `RegPlot` wraps `sns.regplot`, which is axis-level and so can draw onto a layer's axis.
+  `scatter=False` lays a fit over an existing `ScatterPlot`; `lowess` together with
+  `order != 1` raises, since both answer "what shape" and only one can apply.
+- `CorrelationHeatmap` pins the colour scale to `[-1, 1]` centred on 0 and defaults to a
+  diverging colormap, so the sign of a correlation stays readable. Non-numeric columns are
+  dropped, and a frame with nothing numeric left raises rather than drawing an empty grid.
+- docs/04 §4.4 covers the three and their layering; docs/08 §8.10 documents the seaborn
+  axis-label override — a seaborn call with `x=`/`y=` writes the column names over
+  `PlotSettings`.
+
+## 0.2.6
+
+### Added
+
+- `NullDistribution` and `NullResult`: shuffle one column, re-measure, and compare the
+  observed value against the cloud that produces. The statistic stays the caller's —
+  `NullDistribution` takes a callable and ships no ready-made statistics.
+- `NullResult.p_value` counts rather than tests. Both one-sided counts include the observed
+  arrangement, so the smallest value it can return is `1 / (n_iter + 1)`.
+- `NullPlot`: the null as a grey `HistogramPlot` with the observed value marked, its value
+  and p-value in the legend, stated as a bound (`p < …`) once the count floor is reached,
+  since an exact number there would overstate what a thousand shuffles support.
+- `VerticalLine`, the numeric counterpart to `VerticalDate` and the layer `NullPlot`
+  composes.
+- docs/06 §6.7 carries the prose, including the two things the shuffle test does not defend
+  against: the garden of forking paths, and a confounder that shuffling breaks along with
+  the association.
+
+## 0.2.5
+
+### Added
+
+- `RegexFeature` transform: one new column derived from a regular expression over a text
+  column, in `count` (occurrences per row), `has` (bool) or `extract` (first capture group,
+  NaN where nothing matched) mode. The output column is named by `feature`, because
+  `Pipeline.add(name=...)` already claims `name` for the step.
+- `mode="extract"` reports its match rate through `loguru`: it is the one mode that fails
+  silently, filling with NaN where a wrong pattern in the other two gives a visibly zero or
+  all-`False` column.
+- docs/02 §2.5 documents the shipped transform with its mode table and coverage log; §2.4
+  keeps `IsWeekend` as the write-your-own example.
+
+## 0.2.4
+
+### Added
+
+- `bernoulli`, `binomial`, `nbinom` and `beta` are default registry families. `pareto` is
+  deliberately not among them: lesson 4 has the student register it as the extensibility
+  exercise, and the class docstring shows the call.
+- `QQPlot`: sample quantiles against a frozen distribution's theoretical quantiles with a
+  `y = x` reference line, for the tail mismatches a `PlotFits` histogram is too
+  bulk-dominated to show.
+- `ECDFPlot`: one or two empirical CDFs, bin-free.
+- `fit_table`: turns a `fit()` result list into a dataframe ranked by log-likelihood — one
+  row per family with params, log-likelihood, KS statistic and p-value, and which criteria
+  it won. `FailedFit` entries carry their message and sort last, since a missing likelihood
+  is not a small one.
+
+### Fixed
+
+- Log-likelihood uses `logpmf` for discrete families and `logpdf` for continuous ones, so
+  every discrete family reports a real likelihood and the comparison between them means
+  something.
+- The lower bound for a single-parameter discrete family is `1e-3`, low enough for a
+  probability (bernoulli's `p`) as well as a count-scale rate (poisson's lambda).
+- `best_likelihood` and `best_ks` are marked independently: a KS p-value that never clears
+  the `p > 0` threshold — routine for a discrete family on a large, tied sample — leaves
+  the likelihood winner marked. An unknown `criterion` raises `ValueError` whether or not
+  a winner was found.
+
+## 0.2.3
+
+### Added
+
+- `TimeFeatures` transform: derives `date`, `hour`, `day_name`, `isoweek` and `year_week`
+  from a timestamp column, with a `features=[...]` selector. The default is the three
+  lesson 1 relies on, and an unknown feature name raises `ValueError`.
+- `DecomposePlot`: `seasonal_decompose` as the four-panel observed/trend/seasonal/residual
+  view, composing a `LinePlot` per panel.
+- `ACFPlot`: autocorrelations as bars with a Bartlett confidence band. NaN input raises,
+  since `statsmodels.acf` returns an all-NaN result instead of raising.
+
+### Changed
+
+- `statsmodels` is a dependency.
+
+## 0.2.2
+
+### Added
+
+- Analysis-method rows on exploratory versus confirmatory questions and on mechanism: which
+  of the two a question is decides what would count as evidence, and a significant result
+  with no nameable mechanism points at a confounder rather than a discovery.
+- Analysis-method rows on the unit of analysis — the unit a claim is about against the unit
+  of a row, and the number of independent units behind each group being compared — with
+  §9.3 naming pseudoreplication and showing the n=4000/n=5 bar chart.
+- An analysis-method row on how many comparisons were considered, with §9.6 on the garden
+  of forking paths: the arithmetic of fifteen tries at a one-in-twenty threshold, and the
+  three defences (write the question down, split the data, report the count).
+- A visual-critique row separating "A differs from B" from "A is the highest", since every
+  set of numbers has a maximum and a ranking claim needs the ranking to be stable.
+- A counting step at the head of the distribution method: how many observations, and how
+  many independent ones.
+
+These land in docs/09-analysis-method.md and in the matching `goad_mcp` checklists.
+
+## 0.2.1
+
+### Breaking
+
+- **`FileConfig` gained `date_column` and `date_index`.** `FileHandler.load` reads csv and
+  parquet (reader picked from the suffix), parses `date_column` as dates and, when
+  `date_index` is true, moves it into the index. The defaults (`"date"`, `True`) keep the
+  covid pipeline working unchanged, but a `FileConfig` subclass that overrode `load` to
+  support another shape can now express it as configuration instead. A file whose suffix has
+  no reader, or a `date_column` that is not in the frame, raises `ValueError`. Parquet needs
+  `pyarrow`, available as the `parquet` extra.
+- **`DistributionRegistry` is a normal class.** Constructing one no longer returns a shared
+  process-wide instance, so a registration affects only the registry it was made on. Code
+  that registered a family on one registry and relied on a separately constructed
+  `DistributionFitter()` picking it up must now pass it: `DistributionFitter(registry)`.
+
+### Added
+
+- `goad` console script: fits every family in the registry to one column of a csv or parquet
+  file and prints the fits ranked by log-likelihood (`goad data/residuals.csv residual`).
+- `DistributionFitter` accepts an optional `DistributionRegistry`.
+- `filehandler.reader_for`, `filehandler.write_frame` and `filehandler.parse_dates` as
+  reusable pieces; `FileHandler.save` writes parquet as well as csv.
+- A pytest suite under `tests/`.
+
+### Fixed
+
+- `BarWithDates.build` and `VerticalDate.build` return `(fig, ax)` like every other `build`.
+- Plots draw on `self.ax` / `self.fig` instead of module-level `plt` calls, so a layer placed
+  with `plot_on_axes` lands on the axis it was handed rather than on matplotlib's current one.
+- The `goad` console script points at a module that exists.
+
+### Changed
+
+- `__version__` is read from the installed package metadata.

goad_toolkit-0.2.9/.remember/archive.md

@@ -0,0 +1,3 @@
+# Archive
+
+(empty)

goad_toolkit-0.2.9/.remember/logs/memory-2026-08-12.log

@@ -0,0 +1,28 @@
+10:46:23 [hook] session-start: PROJECT_DIR=/Users/rgrouls/code/courses/goad_toolkit/.claude/worktrees/quizzical-solomon-b62511 PIPELINE_DIR=/Users/rgrouls/.claude/plugins/cache/claude-plugins-official/remember/0.19.0 REMEMBER_DIR=/Users/rgrouls/code/courses/goad_toolkit/.remember
+10:46:23 [hook] run-consolidation: PROJECT_DIR=/Users/rgrouls/code/courses/goad_toolkit/.claude/worktrees/quizzical-solomon-b62511 PIPELINE_DIR=/Users/rgrouls/.claude/plugins/cache/claude-plugins-official/remember/0.19.0 PYTHON=python3 REMEMBER_DIR=/Users/rgrouls/code/courses/goad_toolkit/.remember
+10:46:24 [consolidation] start
+10:46:34 [consolidation] tokens: 9+6254cache→792out ($0.013050)
+10:46:34 [consolidation] done: 1 files consolidated
+10:46:36 [hook] post-tool: PROJECT_DIR=/Users/rgrouls/code/courses/goad_toolkit/.claude/worktrees/quizzical-solomon-b62511 PIPELINE_DIR=/Users/rgrouls/.claude/plugins/cache/claude-plugins-official/remember/0.19.0 PYTHON=python3 REMEMBER_DIR=/Users/rgrouls/code/courses/goad_toolkit/.remember
+10:46:49 [hook] post-tool: PROJECT_DIR=/Users/rgrouls/code/courses/goad_toolkit/.claude/worktrees/quizzical-solomon-b62511 PIPELINE_DIR=/Users/rgrouls/.claude/plugins/cache/claude-plugins-official/remember/0.19.0 PYTHON=python3 REMEMBER_DIR=/Users/rgrouls/code/courses/goad_toolkit/.remember
+10:46:54 [hook] post-tool: PROJECT_DIR=/Users/rgrouls/code/courses/goad_toolkit/.claude/worktrees/quizzical-solomon-b62511 PIPELINE_DIR=/Users/rgrouls/.claude/plugins/cache/claude-plugins-official/remember/0.19.0 PYTHON=python3 REMEMBER_DIR=/Users/rgrouls/code/courses/goad_toolkit/.remember
+10:47:00 [hook] post-tool: PROJECT_DIR=/Users/rgrouls/code/courses/goad_toolkit/.claude/worktrees/quizzical-solomon-b62511 PIPELINE_DIR=/Users/rgrouls/.claude/plugins/cache/claude-plugins-official/remember/0.19.0 PYTHON=python3 REMEMBER_DIR=/Users/rgrouls/code/courses/goad_toolkit/.remember
+10:47:04 [hook] post-tool: PROJECT_DIR=/Users/rgrouls/code/courses/goad_toolkit/.claude/worktrees/quizzical-solomon-b62511 PIPELINE_DIR=/Users/rgrouls/.claude/plugins/cache/claude-plugins-official/remember/0.19.0 PYTHON=python3 REMEMBER_DIR=/Users/rgrouls/code/courses/goad_toolkit/.remember
+10:47:10 [hook] post-tool: PROJECT_DIR=/Users/rgrouls/code/courses/goad_toolkit/.claude/worktrees/quizzical-solomon-b62511 PIPELINE_DIR=/Users/rgrouls/.claude/plugins/cache/claude-plugins-official/remember/0.19.0 PYTHON=python3 REMEMBER_DIR=/Users/rgrouls/code/courses/goad_toolkit/.remember
+10:47:15 [hook] post-tool: PROJECT_DIR=/Users/rgrouls/code/courses/goad_toolkit/.claude/worktrees/quizzical-solomon-b62511 PIPELINE_DIR=/Users/rgrouls/.claude/plugins/cache/claude-plugins-official/remember/0.19.0 PYTHON=python3 REMEMBER_DIR=/Users/rgrouls/code/courses/goad_toolkit/.remember
+10:47:22 [hook] post-tool: PROJECT_DIR=/Users/rgrouls/code/courses/goad_toolkit/.claude/worktrees/quizzical-solomon-b62511 PIPELINE_DIR=/Users/rgrouls/.claude/plugins/cache/claude-plugins-official/remember/0.19.0 PYTHON=python3 REMEMBER_DIR=/Users/rgrouls/code/courses/goad_toolkit/.remember
+10:47:27 [hook] post-tool: PROJECT_DIR=/Users/rgrouls/code/courses/goad_toolkit/.claude/worktrees/quizzical-solomon-b62511 PIPELINE_DIR=/Users/rgrouls/.claude/plugins/cache/claude-plugins-official/remember/0.19.0 PYTHON=python3 REMEMBER_DIR=/Users/rgrouls/code/courses/goad_toolkit/.remember
+10:47:29 [hook] post-tool: PROJECT_DIR=/Users/rgrouls/code/courses/goad_toolkit/.claude/worktrees/quizzical-solomon-b62511 PIPELINE_DIR=/Users/rgrouls/.claude/plugins/cache/claude-plugins-official/remember/0.19.0 PYTHON=python3 REMEMBER_DIR=/Users/rgrouls/code/courses/goad_toolkit/.remember
+10:47:39 [hook] post-tool: PROJECT_DIR=/Users/rgrouls/code/courses/goad_toolkit/.claude/worktrees/quizzical-solomon-b62511 PIPELINE_DIR=/Users/rgrouls/.claude/plugins/cache/claude-plugins-official/remember/0.19.0 PYTHON=python3 REMEMBER_DIR=/Users/rgrouls/code/courses/goad_toolkit/.remember
+10:47:46 [hook] post-tool: PROJECT_DIR=/Users/rgrouls/code/courses/goad_toolkit/.claude/worktrees/quizzical-solomon-b62511 PIPELINE_DIR=/Users/rgrouls/.claude/plugins/cache/claude-plugins-official/remember/0.19.0 PYTHON=python3 REMEMBER_DIR=/Users/rgrouls/code/courses/goad_toolkit/.remember
+10:47:50 [hook] post-tool: PROJECT_DIR=/Users/rgrouls/code/courses/goad_toolkit/.claude/worktrees/quizzical-solomon-b62511 PIPELINE_DIR=/Users/rgrouls/.claude/plugins/cache/claude-plugins-official/remember/0.19.0 PYTHON=python3 REMEMBER_DIR=/Users/rgrouls/code/courses/goad_toolkit/.remember
+10:47:59 [hook] post-tool: PROJECT_DIR=/Users/rgrouls/code/courses/goad_toolkit/.claude/worktrees/quizzical-solomon-b62511 PIPELINE_DIR=/Users/rgrouls/.claude/plugins/cache/claude-plugins-official/remember/0.19.0 PYTHON=python3 REMEMBER_DIR=/Users/rgrouls/code/courses/goad_toolkit/.remember
+10:47:59 [hook] save-session: PROJECT_DIR=/Users/rgrouls/code/courses/goad_toolkit/.claude/worktrees/quizzical-solomon-b62511 PIPELINE_DIR=/Users/rgrouls/.claude/plugins/cache/claude-plugins-official/remember/0.19.0 PYTHON=python3 REMEMBER_DIR=/Users/rgrouls/code/courses/goad_toolkit/.remember
+10:47:59 [extract] session 16b4788a-5507-4216-98ec-06bac13a9aba
+10:47:59 [extract] 17 exchanges (1 human)
+10:47:59 [extract] 1 human msgs < 3, skip
+10:48:45 [hook] post-tool: PROJECT_DIR=/Users/rgrouls/code/courses/goad_toolkit/.claude/worktrees/quizzical-solomon-b62511 PIPELINE_DIR=/Users/rgrouls/.claude/plugins/cache/claude-plugins-official/remember/0.19.0 PYTHON=python3 REMEMBER_DIR=/Users/rgrouls/code/courses/goad_toolkit/.remember
+10:48:50 [hook] post-tool: PROJECT_DIR=/Users/rgrouls/code/courses/goad_toolkit/.claude/worktrees/quizzical-solomon-b62511 PIPELINE_DIR=/Users/rgrouls/.claude/plugins/cache/claude-plugins-official/remember/0.19.0 PYTHON=python3 REMEMBER_DIR=/Users/rgrouls/code/courses/goad_toolkit/.remember
+10:48:54 [hook] post-tool: PROJECT_DIR=/Users/rgrouls/code/courses/goad_toolkit/.claude/worktrees/quizzical-solomon-b62511 PIPELINE_DIR=/Users/rgrouls/.claude/plugins/cache/claude-plugins-official/remember/0.19.0 PYTHON=python3 REMEMBER_DIR=/Users/rgrouls/code/courses/goad_toolkit/.remember
+10:49:08 [hook] post-tool: PROJECT_DIR=/Users/rgrouls/code/courses/goad_toolkit/.claude/worktrees/quizzical-solomon-b62511 PIPELINE_DIR=/Users/rgrouls/.claude/plugins/cache/claude-plugins-official/remember/0.19.0 PYTHON=python3 REMEMBER_DIR=/Users/rgrouls/code/courses/goad_toolkit/.remember
+10:49:13 [hook] post-tool: PROJECT_DIR=/Users/rgrouls/code/courses/goad_toolkit/.claude/worktrees/quizzical-solomon-b62511 PIPELINE_DIR=/Users/rgrouls/.claude/plugins/cache/claude-plugins-official/remember/0.19.0 PYTHON=python3 REMEMBER_DIR=/Users/rgrouls/code/courses/goad_toolkit/.remember

goad_toolkit-0.2.9/.remember/recent.md

@@ -0,0 +1,7 @@
+# Recent
+
+## 2026-08-10
+Fixed 5 defects in goad_toolkit (feat/docs-and-mcp), shipped v0.2.0. Removed DistributionRegistry singleton, parameterized FileHandler.load for csv/parquet with date col support, fixed BasePlot.build return contracts. Refactored visualizer.py from module-level plt.* to instance methods (self.ax/.fig), added CLI.
+
+## Identity Candidates
+- IDENTITY CANDIDATE: Architectural preference for instance methods over module-level state and singleton patterns

goad_toolkit-0.2.9/.remember/tmp/capture-alive

@@ -0,0 +1 @@
+16b4788a-5507-4216-98ec-06bac13a9aba

goad_toolkit-0.2.9/.remember/tmp/last-save-ts

@@ -0,0 +1 @@
+1786524479

goad_toolkit-0.2.9/.remember/tmp/save-session.pid

@@ -0,0 +1 @@
+70230

{goad_toolkit-0.2.7 → goad_toolkit-0.2.9}/.remember/tmp/session-slug

@@ -4,4 +4,4 @@ project_dir=/Users/rgrouls/code/courses/goad_toolkit/.claude/worktrees/quizzical
 slug=-Users-rgrouls-code-courses-goad-toolkit--claude-worktrees-quizzical-solomon-b62511
 sessions_dir=/Users/rgrouls/.claude/projects/-Users-rgrouls-code-courses-goad-toolkit--claude-worktrees-quizzical-solomon-b62511
 memory_dir=/Users/rgrouls/code/courses/goad_toolkit/.remember
-session_id=90574e63-97cd-4c11-87e5-23b656c56fd9
+session_id=16b4788a-5507-4216-98ec-06bac13a9aba

{goad_toolkit-0.2.7/.claude/worktrees/quizzical-solomon-b62511 → goad_toolkit-0.2.9}/MCP_SERVER.md

@@ -103,13 +103,13 @@ You pin a version with `GOAD_REF` — a git ref (branch, tag, or commit). That o
 selects both the script (in the URL) and the docs it fetches (inside the script), so a whole
 cohort runs exactly the same thing and the two cannot drift apart.
 
-> **Note:** the docs and this server land in **v0.2.0**. Until that tag exists, point
-> `GOAD_REF` at the branch (`feat/docs-and-mcp`) or work from a local clone (see below).
+> **Note:** the docs and this server land in **v0.2.2**. 
+> You can also point `GOAD_REF` to a branch or work from a local clone (see below).
 
 ### Claude Code
 
 ```bash
-claude mcp add goad -e GOAD_REF=v0.2.0 -- \
+claude mcp add goad -e GOAD_REF=v0.2.2 -- \
   sh -c 'uv run --no-project https://raw.githubusercontent.com/raoulg/goad_toolkit/$GOAD_REF/goad_mcp.py'
 ```
 
@@ -123,7 +123,7 @@ Add to `.cursor/mcp.json` (per-project) or `~/.cursor/mcp.json` (global):
     "goad": {
       "command": "sh",
       "args": ["-c", "uv run --no-project https://raw.githubusercontent.com/raoulg/goad_toolkit/$GOAD_REF/goad_mcp.py"],
-      "env": { "GOAD_REF": "v0.2.0" }
+      "env": { "GOAD_REF": "v0.2.2" }
     }
   }
 }
@@ -137,7 +137,7 @@ Add to `~/.codex/config.toml`:
 [mcp_servers.goad]
 command = "sh"
 args = ["-c", "uv run --no-project https://raw.githubusercontent.com/raoulg/goad_toolkit/$GOAD_REF/goad_mcp.py"]
-env = { GOAD_REF = "v0.2.0" }
+env = { GOAD_REF = "v0.2.2" }
 ```
 
 ### Claude Desktop
@@ -151,7 +151,7 @@ app:
     "goad": {
       "command": "sh",
       "args": ["-c", "uv run --no-project https://raw.githubusercontent.com/raoulg/goad_toolkit/$GOAD_REF/goad_mcp.py"],
-      "env": { "GOAD_REF": "v0.2.0" }
+      "env": { "GOAD_REF": "v0.2.2" }
     }
   }
 }
@@ -187,7 +187,7 @@ add with the new tag:
 
 ```bash
 claude mcp remove goad -s user
-claude mcp add goad -s user -e GOAD_REF=v0.2.0 -- \
+claude mcp add goad -s user -e GOAD_REF=v0.2.2 -- \
   sh -c 'uv run --no-project https://raw.githubusercontent.com/raoulg/goad_toolkit/$GOAD_REF/goad_mcp.py'
 ```
 

{goad_toolkit-0.2.7 → goad_toolkit-0.2.9}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: goad-toolkit
-Version: 0.2.7
+Version: 0.2.9
 Summary: An extensible toolkit for Goal Oriented Analysis of Data
 Project-URL: Github, https://github.com/raoulg/goad_toolkit
 Author-email: raoul grouls <Raoul.Grouls@han.nl>

{goad_toolkit-0.2.7/.claude/worktrees/quizzical-solomon-b62511 → goad_toolkit-0.2.9}/README.md

@@ -183,6 +183,36 @@ For the [kstest](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stat
 The plots are sorted by log-likelihood, which means there is no good fit with a distribution in this case.
 ![residuals](img/distribution_fit.png)
 
+### 🎲 Is it real? The shuffle test
+
+Shuffle the labels, measure again, and see where the real number lands. No test, no
+assumptions, no table of critical values — just a picture of what your measurement does when
+the label means nothing:
+
+```python
+from goad_toolkit.analytics import NullDistribution
+from goad_toolkit.visualizer import NullPlot, PlotSettings
+
+def gap(frame):
+    means = frame.groupby("is_bot")["length"].mean()
+    return means[True] - means[False]
+
+result = NullDistribution(gap, n_iter=2000, seed=1).run(data, label="is_bot")
+settings = PlotSettings(
+    title="Mean message length: bots minus humans",
+    xlabel="difference in mean length (characters)",
+    ylabel="density under shuffled labels",
+)
+NullPlot(settings).plot(result=result)
+```
+
+![null distribution](img/null-distribution.png)
+
+The grey cloud is the statistic under shuffled labels; the line is what the real data did.
+Writing the statistic is your job — that is the claim. See
+[Models and residuals](docs/06-models-and-residuals.md) §6.7 for what the p-value can and
+cannot carry.
+
 ### 🧩 Extending with Custom Distributions
 
 You can easily register new distributions:

{goad_toolkit-0.2.7/.claude/worktrees/quizzical-solomon-b62511 → goad_toolkit-0.2.9}/docs/02-pipelines.md

@@ -58,6 +58,8 @@ input frame is safe, and the steps stay cheap.
 | `SelectDataRange` | keep rows in a date range | `start_date`, `end_date` |
 | `RollingAvg` | rolling mean, drops the leading NaNs | `column`, `window`, `rename` |
 | `ZScaler` | standardise to mean 0, std 1 | `column`, `rename` |
+| `TimeFeatures` | derive calendar columns from a timestamp | `column`, `features` |
+| `RegexFeature` | count / flag / extract a pattern in a text column | `column`, `pattern`, `feature`, `mode` |
 
 `rename=True` writes to a new column (`deaths_shifted`, `deaths_zscore`, …) instead of
 overwriting. Prefer it. An overwritten column is a step you cannot debug, and the whole
@@ -75,20 +77,17 @@ Subclass `TransformBase` and implement `transform`. That is the entire contract.
 from goad_toolkit.datatransforms import TransformBase
 import pandas as pd
 
-class TimeFeatures(TransformBase):
-    """Derive calendar features from a timestamp column."""
+class IsWeekend(TransformBase):
+    """Flag Saturday/Sunday from a timestamp column."""
 
     def transform(self, data: pd.DataFrame, column: str) -> pd.DataFrame:
         ts = pd.to_datetime(data[column])
-        data["date"] = ts.dt.date
-        data["hour"] = ts.dt.hour
-        data["day_name"] = ts.dt.day_name()
-        data["isoweek"] = ts.dt.isocalendar().week
+        data["is_weekend"] = ts.dt.dayofweek >= 5
         return data
 ```
 
 ```python
-pipeline.add(TimeFeatures, column="timestamp")
+pipeline.add(IsWeekend, column="timestamp")
 ```
 
 What the base class does for you:
@@ -106,39 +105,39 @@ Two rules for your `transform`:
 - **Name your parameters explicitly** in the signature. `def transform(self, data, column,
   window)` documents itself; `**kwargs` does not, and the validation cannot help you.
 
-## 2.5 A transform for feature enrichment
+## 2.5 `RegexFeature`: feature enrichment from text
 
-The single most useful custom transform for text data, and a good template:
+The single most useful transform for text data, and the reason `TransformBase` is worth
+subclassing at all. It ships:
 
 ```python
-class RegexFeature(TransformBase):
-    """Add a feature extracted from a text column with a regular expression."""
-
-    def transform(
-        self,
-        data: pd.DataFrame,
-        column: str,
-        pattern: str,
-        feature: str,
-        mode: str = "count",
-    ) -> pd.DataFrame:
-        text = data[column].fillna("")
-        if mode == "count":
-            data[feature] = text.str.count(pattern)
-        elif mode == "has":
-            data[feature] = text.str.contains(pattern, regex=True)
-        elif mode == "extract":
-            data[feature] = text.str.extract(pattern, expand=False)
-        else:
-            raise ValueError(f"mode must be count/has/extract, got {mode!r}")
-        return data
-```
+from goad_toolkit.datatransforms import RegexFeature
 
-```python
 pipeline.add(RegexFeature, name="url_flag",
              column="message", pattern=r"https?://\S+", feature="has_url", mode="has")
 ```
 
+Three modes, each writing one new column named by `feature`:
+
+| `mode` | writes | use for |
+|---|---|---|
+| `"count"` | how many times the pattern occurs (int) | how many URLs, how many question marks |
+| `"has"` | whether it occurs at all (bool) | flags you will group or filter on |
+| `"extract"` | the first capture group, NaN where nothing matched | pulling a value *out* of the text |
+
+`"extract"` needs exactly one capture group in `pattern`, and it is the mode worth being
+careful with. `count` and `has` fail visibly when a pattern is wrong — a column of all zeros
+or all `False` is hard to miss. Extraction fails *silently*, filling with NaN, so it reports
+its own coverage through `loguru`:
+
+```
+mentions: extracted 'addressed_to' from 92,415/627,172 rows (14.7%); 534,757 rows had no match
+```
+
+That number is the point. A pattern that matches 15% of rows may be exactly right — on IRC,
+most messages do not address anyone — or it may be silently broken. The log line makes you
+decide which, instead of finding out four notebooks later.
+
 Note the two different `name`s: `Pipeline.add(name=...)` names the *step*, and
 `TransformBase.__init__` consumes it. So the new-column parameter has to be called something
 else — `feature` here. Any transform that wants to name an output column hits this.
@@ -200,9 +199,31 @@ will not warn you. Write the steps in the order you would do them by hand.
 you would want to run again on new data. A one-off `df[df.author == "Alice"]` while you are
 poking around is a notebook cell, and should stay one.
 
-**Debugging.** When a pipeline produces something surprising, add the steps back one at a
-time rather than dismantling it into cells — the moment it becomes cells again you have lost
-the record of what you did, which is the thing you were trying to keep.
+**Debugging.** When a pipeline produces something surprising, do not dismantle it into cells:
+the moment it becomes cells again you have lost the record of what you did, which is the
+thing you were trying to keep. Ask it what it did instead.
+
+```python
+result = pipeline.apply(data, keep_intermediate=True)
+pipeline.report()
+```
+
+```
+     step  rows  columns          added  removed
+ calendar  4083        9  hour, day_name
+  weekend  4083       10      is_weekend
+   window  3961       10
+```
+
+`report()` gives one row per step: how many rows survived it, how many columns the frame had
+afterwards, and which columns that step added or removed. The step where the row count falls
+off a cliff is the one filtering more than you meant it to, and a step with an empty `added`
+is a step that silently did nothing.
+
+`pipeline.intermediate` holds the frame as it stood after each step, keyed by step name, so
+you can look at the rows themselves once `report()` has told you where to look. Both are off
+by default — keeping them costs one full copy of the frame per step — and turning them on
+does not change what `apply` returns.
 
 ---
 

{goad_toolkit-0.2.7/.claude/worktrees/quizzical-solomon-b62511 → goad_toolkit-0.2.9}/docs/03-plot-composition.md

@@ -148,12 +148,29 @@ step 3.
 | `LinePlot` | `sns.lineplot` | the primitive most compositions build on |
 | `ComparePlot` | two lines with labels | `LinePlot` twice via `plot_on` |
 | `VerticalDate` | a labelled vertical line at a date | a layer, not a plot |
+| `VerticalLine` | the same at a number | a layer, not a plot |
 | `ComparePlotDate` | `ComparePlot` + `VerticalDate` | |
+| `BarPlot` | `sns.barplot`, one bar per category | see [Five families §4.1](04-five-families.md) |
+| `GroupedBarPlot` | bars split by `hue` | two categorical variables at once |
+| `HeatmapPlot` | `sns.heatmap`, pivoting the frame if asked | when both variables have many levels |
+| `BarbellPlot` | before and after per category, joined | the change is the message |
+| `HighlightCategory` | recolours bars: grey, plus the one that matters | a layer, and it needs no help from what drew them |
+| `Annotate` | text on the plot, with an arrow to the point | a layer; the finding, written where it is read |
+| `ScatterPlot` | `sns.scatterplot` | the first plot of any relation question |
+| `RegPlot` | `sns.regplot` — `fit_reg`, `lowess`, `order` | `scatter=False` layers it over a `ScatterPlot` |
+| `CorrelationHeatmap` | `.corr()` as a heatmap, scale pinned to [-1, 1] | where to look next, not a ranking |
 | `BarWithDates` | bars with a month locator on the x axis | |
 | `ResidualPlot` | `BarWithDates` + `VerticalDate` | the residual-over-time view |
+| `DecomposePlot` | `seasonal_decompose`, one panel each | observed/trend/seasonal/residual, overrides `plot` |
+| `ACFPlot` | `acf` as bars with a confidence band | see [Five families §4.2](04-five-families.md) |
 | `HistogramPlot` | `sns.histplot`, `stat="density"` | density so a pdf can be overlaid |
 | `DistPlot` | a scipy distribution's pdf or pmf | estimates its own x-range from `ppf` |
 | `PlotFits` | histogram + fitted pdf, one panel per fit | see [Distributions](05-distributions.md) |
+| `QQPlot` | sample quantiles vs. a distribution's theoretical quantiles | the tails, not the bulk — see [Distributions §5.5](05-distributions.md) |
+| `ECDFPlot` | one or two empirical CDFs, bin-free | the right tool for comparing two samples |
+| `NullPlot` | a shuffled statistic, with the observed value marked | `HistogramPlot` + `VerticalLine`, see [Models and residuals §6.7](06-models-and-residuals.md) |
+| `ProjectionPlot` | already-fitted 2-D coordinates | takes an array, not a model — see [Five families §4.5](04-five-families.md) |
+| `ScreePlot` | variance explained per component, plus the running total | the figure a projection has to come with |
 
 `HistogramPlot` defaults to `sqrt(n)` bins capped at 50. That default is a starting point,
 not an answer — bin count changes what a histogram appears to say, and choosing it is part of
@@ -164,6 +181,45 @@ falling back to `pmf` for discrete families. Left to itself it picks an x-range
 0.1st to the 99.9th percentile, which is right for most things and wrong for anything with a
 very heavy tail — pass `x_range` explicitly there.
 
+`QQPlot` takes a sample and a *frozen* distribution and plots sorted data against the
+distribution's quantiles at matching plotting positions, with a y=x reference line —
+where `PlotFits`' histogram is dominated by the bulk of the data, this is where a tail
+mismatch actually shows up.
+
+`ECDFPlot` plots one sample, or two with `compare=`, as step functions of the empirical CDF —
+no bin width to choose, so nothing about the shape is a plotting decision.
+
+`HighlightCategory` is the clearest case for why layers beat settings. It recolours the bars
+already on the axis — grey for everything, one colour for the categories you name — so it
+works over `BarPlot`, over a bare `sns.barplot`, over pandas' `.plot.bar`, and over the
+`BasePlot` subclass you wrote yourself. None of them need to know it exists:
+
+```python
+settings = PlotSettings(title="Survival rate by deck", highlight=["D"])
+bars = BarPlot(settings)
+bars.plot(data=decks, x="deck", y="survival")
+bars.plot_on(HighlightCategory(settings))
+```
+
+`highlight`, `base_color` and `highlight_color` live on `PlotSettings` so that the call stays
+about the data, but they are only defaults — pass `categories=` at the call to override.
+Naming a category that is not on the axis raises, and lists the ones that are.
+
+`Annotate` is the other layer worth reaching for by habit. A reader with thirty seconds reads
+the annotation and not the axes, so the sentence you would have said out loud belongs on the
+plot:
+
+```python
+line.plot_on(
+    Annotate(settings),
+    text="six-day gap: the logging host was down, not the channel",
+    xy=(gap_date, gap_value),          # the point being explained
+    xytext=(label_date, label_value),  # where the text sits; an arrow joins them
+)
+```
+
+Give it `xy` alone and the text lands on the point with no arrow.
+
 ## 3.7 Writing your own plot
 
 Two questions decide the shape:
@@ -216,9 +272,11 @@ lines of code.
   `self.ax.axvline(...)`, `self.fig.suptitle(...)`. Keeping to them is what makes
   `plot_on_axes` reliable.
 
-`PlotFits` overrides `plot` rather than `build`, because it needs to know how many panels to
-create before the figure exists. Its `build` is a no-op. It is the exception, not a pattern to
-copy.
+`PlotFits` and `DecomposePlot` override `plot` rather than `build`, because both need to know
+how many panels to create before the figure exists — variable for `PlotFits` (one per fit),
+fixed at four for `DecomposePlot`. Either way, `build` alone cannot decide the panel count
+before `create_figure` runs, so both leave it as a no-op. This is the exception, not a pattern
+to copy for a plot that only ever needs one axis.
 
 ---