goad-toolkit 0.2.5__tar.gz → 0.2.8__tar.gz

goad_toolkit-0.2.8/.claude/worktrees/quizzical-solomon-b62511/.claude/settings.local.json

@@ -0,0 +1,7 @@
+{
+  "permissions": {
+    "allow": [
+      "Bash(cd /Users/rgrouls/code/courses/goad_toolkit && cat pyproject.toml)"
+    ]
+  }
+}

goad_toolkit-0.2.8/.claude/worktrees/quizzical-solomon-b62511/CHANGELOG.md

@@ -0,0 +1,152 @@
+# Changelog
+
+## 0.2.7
+
+### Added
+
+- `ScatterPlot`, `RegPlot` and `CorrelationHeatmap`: the relation plots for two-variable
+  questions, in their order of use — a scatter first, a fit once the scatter has shown which
+  kind, a correlation matrix when there are too many pairs to plot by hand.
+- `ScatterPlot` defaults `alpha` below 1, since overplotting is the standard way a scatter
+  misreports its own density.
+- `RegPlot` wraps `sns.regplot`, which is axis-level and so can draw onto a layer's axis.
+  `scatter=False` lays a fit over an existing `ScatterPlot`; `lowess` together with
+  `order != 1` raises, since both answer "what shape" and only one can apply.
+- `CorrelationHeatmap` pins the colour scale to `[-1, 1]` centred on 0 and defaults to a
+  diverging colormap, so the sign of a correlation stays readable. Non-numeric columns are
+  dropped, and a frame with nothing numeric left raises rather than drawing an empty grid.
+- docs/04 §4.4 covers the three and their layering; docs/08 §8.10 documents the seaborn
+  axis-label override — a seaborn call with `x=`/`y=` writes the column names over
+  `PlotSettings`.
+
+## 0.2.6
+
+### Added
+
+- `NullDistribution` and `NullResult`: shuffle one column, re-measure, and compare the
+  observed value against the cloud that produces. The statistic stays the caller's —
+  `NullDistribution` takes a callable and ships no ready-made statistics.
+- `NullResult.p_value` counts rather than tests. Both one-sided counts include the observed
+  arrangement, so the smallest value it can return is `1 / (n_iter + 1)`.
+- `NullPlot`: the null as a grey `HistogramPlot` with the observed value marked, its value
+  and p-value in the legend, stated as a bound (`p < …`) once the count floor is reached,
+  since an exact number there would overstate what a thousand shuffles support.
+- `VerticalLine`, the numeric counterpart to `VerticalDate` and the layer `NullPlot`
+  composes.
+- docs/06 §6.7 carries the prose, including the two things the shuffle test does not defend
+  against: the garden of forking paths, and a confounder that shuffling breaks along with
+  the association.
+
+## 0.2.5
+
+### Added
+
+- `RegexFeature` transform: one new column derived from a regular expression over a text
+  column, in `count` (occurrences per row), `has` (bool) or `extract` (first capture group,
+  NaN where nothing matched) mode. The output column is named by `feature`, because
+  `Pipeline.add(name=...)` already claims `name` for the step.
+- `mode="extract"` reports its match rate through `loguru`: it is the one mode that fails
+  silently, filling with NaN where a wrong pattern in the other two gives a visibly zero or
+  all-`False` column.
+- docs/02 §2.5 documents the shipped transform with its mode table and coverage log; §2.4
+  keeps `IsWeekend` as the write-your-own example.
+
+## 0.2.4
+
+### Added
+
+- `bernoulli`, `binomial`, `nbinom` and `beta` are default registry families. `pareto` is
+  deliberately not among them: lesson 4 has the student register it as the extensibility
+  exercise, and the class docstring shows the call.
+- `QQPlot`: sample quantiles against a frozen distribution's theoretical quantiles with a
+  `y = x` reference line, for the tail mismatches a `PlotFits` histogram is too
+  bulk-dominated to show.
+- `ECDFPlot`: one or two empirical CDFs, bin-free.
+- `fit_table`: turns a `fit()` result list into a dataframe ranked by log-likelihood — one
+  row per family with params, log-likelihood, KS statistic and p-value, and which criteria
+  it won. `FailedFit` entries carry their message and sort last, since a missing likelihood
+  is not a small one.
+
+### Fixed
+
+- Log-likelihood uses `logpmf` for discrete families and `logpdf` for continuous ones, so
+  every discrete family reports a real likelihood and the comparison between them means
+  something.
+- The lower bound for a single-parameter discrete family is `1e-3`, low enough for a
+  probability (bernoulli's `p`) as well as a count-scale rate (poisson's lambda).
+- `best_likelihood` and `best_ks` are marked independently: a KS p-value that never clears
+  the `p > 0` threshold — routine for a discrete family on a large, tied sample — leaves
+  the likelihood winner marked. An unknown `criterion` raises `ValueError` whether or not
+  a winner was found.
+
+## 0.2.3
+
+### Added
+
+- `TimeFeatures` transform: derives `date`, `hour`, `day_name`, `isoweek` and `year_week`
+  from a timestamp column, with a `features=[...]` selector. The default is the three
+  lesson 1 relies on, and an unknown feature name raises `ValueError`.
+- `DecomposePlot`: `seasonal_decompose` as the four-panel observed/trend/seasonal/residual
+  view, composing a `LinePlot` per panel.
+- `ACFPlot`: autocorrelations as bars with a Bartlett confidence band. NaN input raises,
+  since `statsmodels.acf` returns an all-NaN result instead of raising.
+
+### Changed
+
+- `statsmodels` is a dependency.
+
+## 0.2.2
+
+### Added
+
+- Analysis-method rows on exploratory versus confirmatory questions and on mechanism: which
+  of the two a question is decides what would count as evidence, and a significant result
+  with no nameable mechanism points at a confounder rather than a discovery.
+- Analysis-method rows on the unit of analysis — the unit a claim is about against the unit
+  of a row, and the number of independent units behind each group being compared — with
+  §9.3 naming pseudoreplication and showing the n=4000/n=5 bar chart.
+- An analysis-method row on how many comparisons were considered, with §9.6 on the garden
+  of forking paths: the arithmetic of fifteen tries at a one-in-twenty threshold, and the
+  three defences (write the question down, split the data, report the count).
+- A visual-critique row separating "A differs from B" from "A is the highest", since every
+  set of numbers has a maximum and a ranking claim needs the ranking to be stable.
+- A counting step at the head of the distribution method: how many observations, and how
+  many independent ones.
+
+These land in docs/09-analysis-method.md and in the matching `goad_mcp` checklists.
+
+## 0.2.1
+
+### Breaking
+
+- **`FileConfig` gained `date_column` and `date_index`.** `FileHandler.load` reads csv and
+  parquet (reader picked from the suffix), parses `date_column` as dates and, when
+  `date_index` is true, moves it into the index. The defaults (`"date"`, `True`) keep the
+  covid pipeline working unchanged, but a `FileConfig` subclass that overrode `load` to
+  support another shape can now express it as configuration instead. A file whose suffix has
+  no reader, or a `date_column` that is not in the frame, raises `ValueError`. Parquet needs
+  `pyarrow`, available as the `parquet` extra.
+- **`DistributionRegistry` is a normal class.** Constructing one no longer returns a shared
+  process-wide instance, so a registration affects only the registry it was made on. Code
+  that registered a family on one registry and relied on a separately constructed
+  `DistributionFitter()` picking it up must now pass it: `DistributionFitter(registry)`.
+
+### Added
+
+- `goad` console script: fits every family in the registry to one column of a csv or parquet
+  file and prints the fits ranked by log-likelihood (`goad data/residuals.csv residual`).
+- `DistributionFitter` accepts an optional `DistributionRegistry`.
+- `filehandler.reader_for`, `filehandler.write_frame` and `filehandler.parse_dates` as
+  reusable pieces; `FileHandler.save` writes parquet as well as csv.
+- A pytest suite under `tests/`.
+
+### Fixed
+
+- `BarWithDates.build` and `VerticalDate.build` return `(fig, ax)` like every other `build`.
+- Plots draw on `self.ax` / `self.fig` instead of module-level `plt` calls, so a layer placed
+  with `plot_on_axes` lands on the axis it was handed rather than on matplotlib's current one.
+- The `goad` console script points at a module that exists.
+
+### Changed
+
+- `__version__` is read from the installed package metadata.

{goad_toolkit-0.2.5 → goad_toolkit-0.2.8}/.claude/worktrees/quizzical-solomon-b62511/README.md

@@ -183,6 +183,36 @@ For the [kstest](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stat
 The plots are sorted by log-likelihood, which means there is no good fit with a distribution in this case.
 ![residuals](img/distribution_fit.png)
 
+### 🎲 Is it real? The shuffle test
+
+Shuffle the labels, measure again, and see where the real number lands. No test, no
+assumptions, no table of critical values — just a picture of what your measurement does when
+the label means nothing:
+
+```python
+from goad_toolkit.analytics import NullDistribution
+from goad_toolkit.visualizer import NullPlot, PlotSettings
+
+def gap(frame):
+    means = frame.groupby("is_bot")["length"].mean()
+    return means[True] - means[False]
+
+result = NullDistribution(gap, n_iter=2000, seed=1).run(data, label="is_bot")
+settings = PlotSettings(
+    title="Mean message length: bots minus humans",
+    xlabel="difference in mean length (characters)",
+    ylabel="density under shuffled labels",
+)
+NullPlot(settings).plot(result=result)
+```
+
+![null distribution](img/null-distribution.png)
+
+The grey cloud is the statistic under shuffled labels; the line is what the real data did.
+Writing the statistic is your job — that is the claim. See
+[Models and residuals](docs/06-models-and-residuals.md) §6.7 for what the p-value can and
+cannot carry.
+
 ### 🧩 Extending with Custom Distributions
 
 You can easily register new distributions:

{goad_toolkit-0.2.5 → goad_toolkit-0.2.8/.claude/worktrees/quizzical-solomon-b62511}/docs/04-five-families.md

@@ -149,9 +149,22 @@ to look.
   are not independent evidence. This is exactly why regularisation is the better tool for
   "which variables matter" — see [Models and residuals](06-models-and-residuals.md).
 
-**In GOAD:** the scatter and the fit are `BasePlot` subclasses; the interesting work is in
-[Models and residuals](06-models-and-residuals.md), because a relation you can state as a
-model is a relation whose failures you can inspect.
+**In GOAD:** `ScatterPlot`, `RegPlot` and `CorrelationHeatmap`, in that order of use.
+
+```python
+scatter = ScatterPlot(settings)
+scatter.plot(data=data, x="messages", y="length", hue="is_bot")
+scatter.plot_on(RegPlot(settings), data=data, x="messages", y="length", scatter=False)
+```
+
+The layering is the point: the scatter is drawn first and the fit goes on top of it, so the
+choice of `fit_reg`, `order` or `lowess` is made after you have seen the shape rather than
+before. `CorrelationHeatmap` is the entry point when there are too many pairs to plot — it
+tells you which scatter to make next, and nothing more than that.
+
+The interesting work is still in [Models and residuals](06-models-and-residuals.md), because
+a relation you can state as a model is a relation whose failures you can inspect — and
+§6.7 is how you check that the relation is not something a shuffle would have produced.
 
 ---
 

{goad_toolkit-0.2.5 → goad_toolkit-0.2.8}/.claude/worktrees/quizzical-solomon-b62511/docs/06-models-and-residuals.md

@@ -158,6 +158,50 @@ They compose: the residual of a functional model is what you fit a distribution
 composition is the loop from [Goal oriented analysis](01-goal-oriented-analysis.md) §1.3, and
 it is where the two halves of this library meet.
 
+## 6.7 The shuffle test
+
+§6.1 said a model is a statement about what is boring. The most boring statement available is
+*this label explains nothing*, and you can build that model without writing one down: shuffle
+the label column and measure again. Everything else stays as it was — the labels keep their
+proportions, the values keep their shape — and only the pairing between them is destroyed.
+Do it a thousand times and you have a picture of what your measurement does in a world where
+there is nothing to find.
+
+```python
+from goad_toolkit.analytics import NullDistribution
+from goad_toolkit.visualizer import NullPlot, PlotSettings
+
+def gap(frame):
+    means = frame.groupby("is_bot")["length"].mean()
+    return means[True] - means[False]
+
+result = NullDistribution(gap, n_iter=2000, seed=1).run(data, label="is_bot")
+NullPlot(PlotSettings(xlabel="difference in mean length")).plot(result=result)
+```
+
+The statistic is yours. `NullDistribution` runs the loop; deciding *what* to measure and
+*which* column to shuffle is the part that cannot be delegated, because it is the claim.
+
+Read the picture, not the number. Either the line stands clear of the grey cloud or it sits
+inside it, and that judgement needs no test, no distributional assumption, and no table of
+critical values. A finding sitting in the middle of its own null is not a weak finding; it is
+the null, and the honest write-up says so.
+
+Three things worth knowing before you quote a number from it:
+
+- **`p_value()` counts, it does not test.** It is the fraction of shuffles that matched or
+  beat what you saw, defaulting to two-sided. The observed arrangement is itself one of the
+  arrangements, so the count starts at one and the smallest reportable value is
+  `1 / (n_iter + 1)`. With 2000 shuffles, "p ≤ 0.001" is the strongest sentence available —
+  which is why `NullPlot` writes `≤` on the plot when you are at that floor.
+- **It answers one question.** Run it on the fifteenth thing you measured and it will happily
+  report the fifteenth thing as striking. The shuffle test defends against noise, not against
+  the garden of forking paths ([Analysis method](09-analysis-method.md) §9.6) — only writing
+  the question down first, or confirming on a slice you did not hunt on, does that.
+- **Shuffling the label tests the label.** If a confounder drives both the label and the
+  value, shuffling breaks the confounding too, and the test will confirm an association that
+  is real but not the mechanism you have in mind.
+
 ---
 
 Next: [Visual critique](07-visual-critique.md) — judging the picture you just made.

{goad_toolkit-0.2.5 → goad_toolkit-0.2.8/.claude/worktrees/quizzical-solomon-b62511}/docs/08-api-reference.md

@@ -218,6 +218,33 @@ winner — it just has no `best_ks` winner.
 `log_likelihood` descending, with `FailedFit` rows (their `message` kept, metrics `NaN`)
 sorted last.
 
+The shuffle test — see [Models and residuals](06-models-and-residuals.md) §6.7:
+
+```python
+@dataclass
+class NullResult:
+    observed: float
+    null: np.ndarray                     # the statistic under n_iter shuffles
+    n_iter: int
+    seed: Optional[int] = None
+
+    def p_value(self, alternative: str = "two-sided") -> float
+    quantile: float                      # property, where observed sits in the null
+
+class NullDistribution:
+    def __init__(self, statistic: Callable[[pd.DataFrame], float],
+                 n_iter: int = 1000, seed: Optional[int] = None)
+
+    def run(self, data: pd.DataFrame, label: str) -> NullResult
+```
+
+`statistic` takes a frame and returns one number; `run` calls it once on the data as given,
+then once per shuffle of the `label` column. The caller's frame is never modified.
+`alternative` is one of `"two-sided"`, `"greater"`, `"less"`. Both one-sided counts include
+the observed arrangement, so `p_value` bottoms out at `1 / (n_iter + 1)` rather than 0. A
+`label` that is not a column raises `ValueError`, as does a statistic that is not finite on
+the unshuffled data; non-finite values among the shuffles are logged, not raised.
+
 ---
 
 ## 8.7 `goad_toolkit.models`
@@ -288,14 +315,19 @@ class BasePlot(ABC):
 | `LinePlot` | `data, **kwargs` → `sns.lineplot` |
 | `ComparePlot` | `data, x, y1, y2, **kwargs` |
 | `VerticalDate` | `date: str, label: str` |
+| `VerticalLine` | `x: float, label=None, color="crimson", linestyle="--", linewidth=2, **kwargs` |
 | `ComparePlotDate` | `data, x, y1, y2, date, datelabel, **kwargs` |
 | `BarWithDates` | `data, x, y, interval: int = 1, **kwargs` |
 | `ResidualPlot` | `data, x, y, date, datelabel, interval: int = 1` |
+| `ScatterPlot` | `data, x, y, alpha=0.6, **kwargs` → `sns.scatterplot` (`hue`, `size`, `style`) |
+| `RegPlot` | `data, x, y, fit_reg=True, lowess=False, order=1, scatter=True, color="crimson", **kwargs` → `sns.regplot` |
+| `CorrelationHeatmap` | `data, columns=None, method="pearson", cmap="vlag", annot=True, fmt=".2f", **kwargs` |
 | `HistogramPlot` | `data: np.ndarray, bins=None, kde=False, color="skyblue", alpha=0.7, **kwargs` |
 | `DistPlot` | `distribution, x_range=None, samples=1000, color="crimson", linewidth=2, label=None, **kwargs` |
 | `ACFPlot` | `data, nlags=40, alpha=0.05, color="steelblue", band_color="gray", **kwargs` |
 | `QQPlot` | `data, distribution, color="steelblue", line_color="black", alpha=0.6, **kwargs` |
 | `ECDFPlot` | `data, compare=None, label="data", compare_label="compare", color="steelblue", compare_color="crimson", **kwargs` |
+| `NullPlot` | `result: NullResult, bins=None, alternative="two-sided", color="lightgrey", line_color="crimson", label=None, **kwargs` |
 
 ```python
 class PlotFits(BasePlot):
@@ -316,6 +348,17 @@ axis it was handed.
 results. `HistogramPlot` uses `stat="density"` so a pdf overlays correctly; `DistPlot` takes a
 **frozen** distribution and falls back to `pmf` when there is no `pdf`.
 
+`NullPlot` composes `HistogramPlot` and `VerticalLine`: grey cloud, coloured line. It writes
+the observed value and its p-value into the legend, using `≤` when the p-value is at the
+floor set by `n_iter`. Pass `label` to write your own.
+
+`RegPlot` wraps `sns.regplot`, the axes-level counterpart to `sns.lmplot` — `lmplot` builds
+its own figure and cannot draw onto a `BasePlot` axis. `lowess=True` with `order != 1` raises:
+both are answers to "what shape", and only one can apply. Pass `scatter=False` to layer a fit
+over an existing `ScatterPlot`. `CorrelationHeatmap` drops non-numeric columns, raises when
+nothing numeric is left, and pins the colour scale to [-1, 1] centred on 0 — override with
+`vmin` / `vmax` / `center` in `**kwargs` if you must.
+
 ---
 
 ## 8.9 `goad_toolkit.cli`
@@ -352,6 +395,10 @@ of `likelihood`, `ks`, `combined`; `--method` is passed to `scipy.stats.fit`.
 - `ACFPlot` raises `ValueError` if the data contains NaNs, rather than passing them through.
   `statsmodels.tsa.stattools.acf` does not raise on missing values — it silently returns an
   all-NaN result — so a decomposition's residual needs `.dropna()` before it reaches `ACFPlot`.
+- A seaborn function called with `x=` and `y=` as column names writes those names onto the
+  axes, overwriting the `xlabel` / `ylabel` from `PlotSettings`. `create_figure` runs first,
+  seaborn runs second, and second wins. Set the labels on the axis after plotting when the
+  column name is not the label you want a reader to see.
 - `binomial` and `nbinom` fit two shape parameters (`n`, `p`) from the sample mean and
   variance alone. If `n` is not known independently, it is not identifiable from the data —
   many `(n, p)` pairs with the same `n·p` fit about as well, so a fitted `n` should be treated

{goad_toolkit-0.2.5 → goad_toolkit-0.2.8/.claude/worktrees/quizzical-solomon-b62511}/docs/09-analysis-method.md

@@ -141,7 +141,9 @@ See [Visual critique](07-visual-critique.md).
 | Does it survive changing the parameter from stage 4? | If it does not, the finding is about the parameter | the sweep you ran |
 | If you modelled it, what does the residual look like? | Structure in the residual is an unmodelled mechanism, not noise | the residual's shape over time and its distribution |
 
-See [Models and residuals](06-models-and-residuals.md) §6.3.
+See [Models and residuals](06-models-and-residuals.md) §6.3 for the residual loop, and §6.7
+for the shuffle test — `NullDistribution` runs the resampling once you have said what to
+shuffle.
 
 ### The garden of forking paths
 

{goad_toolkit-0.2.5 → goad_toolkit-0.2.8/.claude/worktrees/quizzical-solomon-b62511}/goad_mcp.py

@@ -697,8 +697,8 @@ STAGE_ROWS: dict[AnalysisStage, list[StageRow]] = {
             "did not suggest it",
         ),
         (
-            "If you shuffled the labels and re-measured, how often would you see something "
-            "this strong?",
+            f"If you shuffled the labels and re-measured, how often would you see something "
+            f"this strong? ({_models()} §6.7)",
             "The cheapest honest check available, and it needs no distributional theory at "
             "all.",
             "what gets shuffled, and a rough frequency",

{goad_toolkit-0.2.5 → goad_toolkit-0.2.8/.claude/worktrees/quizzical-solomon-b62511}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "goad-toolkit"
-version = "0.2.5"
+version = "0.2.7"
 description = "An extensible toolkit for Goal Oriented Analysis of Data"
 readme = "README.md"
 authors = [

{goad_toolkit-0.2.5 → goad_toolkit-0.2.8/.claude/worktrees/quizzical-solomon-b62511}/src/goad_toolkit/analytics.py

@@ -1,5 +1,5 @@
 from dataclasses import dataclass
-from typing import Any, List, Optional, Tuple, Union
+from typing import Any, Callable, List, Optional, Tuple, Union
 
 import numpy as np
 import pandas as pd
@@ -148,7 +148,9 @@ class DistributionFitter:
         We will prefer distributions with higher log-likelihood values.
         """
         try:
-            log_density = dist_obj.dist.logpmf if dist_obj.is_discrete else dist_obj.dist.logpdf
+            log_density = (
+                dist_obj.dist.logpmf if dist_obj.is_discrete else dist_obj.dist.logpdf
+            )
             return np.sum(log_density(data, *params))
         except Exception as e:
             logger.warning(f"Log-likelihood calculation failed: {str(e)}")
@@ -327,6 +329,155 @@ class DistributionFitter:
             raise ValueError(f"Unknown criterion '{criterion}'")
 
 
+@dataclass
+class NullResult:
+    """A statistic, and the same statistic measured under shuffled labels.
+
+    `observed` is the number you found. `null` holds the same measurement taken
+    `n_iter` times on data where the labels were shuffled — data in which, by
+    construction, the label explains nothing. If `observed` sits inside that
+    cloud, a world where nothing is happening produces your finding routinely.
+    """
+
+    observed: float
+    null: np.ndarray
+    n_iter: int
+    seed: Optional[int] = None
+
+    def p_value(self, alternative: str = "two-sided") -> float:
+        """How often a shuffle matched or beat the observed value.
+
+        Args:
+            alternative: `"greater"` counts shuffles at or above `observed`,
+                `"less"` counts shuffles at or below it, and `"two-sided"`
+                (the default) is twice the smaller of the two, capped at 1.
+
+        Both one-sided counts include the observed arrangement itself — it is
+        one of the arrangements the labels could take — so the smallest value
+        this can return is `1 / (n_iter + 1)` rather than zero. With 1000
+        shuffles, "p < 0.001" is the strongest statement available; report it
+        that way rather than as an exact number.
+        """
+        greater = (int(np.sum(self.null >= self.observed)) + 1) / (self.n_iter + 1)
+        less = (int(np.sum(self.null <= self.observed)) + 1) / (self.n_iter + 1)
+
+        if alternative == "greater":
+            return greater
+        if alternative == "less":
+            return less
+        if alternative == "two-sided":
+            return min(1.0, 2 * min(greater, less))
+        raise ValueError(
+            f"Unknown alternative '{alternative}', "
+            "expected 'two-sided', 'greater' or 'less'"
+        )
+
+    @property
+    def quantile(self) -> float:
+        """Where `observed` sits in the null, as a fraction between 0 and 1."""
+        return float(np.mean(self.null < self.observed))
+
+    def __repr__(self) -> str:
+        return (
+            f"NullResult(observed={self.observed:.4g}, "
+            f"null_mean={float(np.mean(self.null)):.4g}, "
+            f"quantile={self.quantile:.3f}, "
+            f"p={self.p_value():.4f}, n_iter={self.n_iter})"
+        )
+
+
+class NullDistribution:
+    """Shuffle the labels, measure again, and see where the real number lands.
+
+    The cheapest honest check in the method (§9.6): it assumes nothing about the
+    shape of the data, needs no table of critical values, and answers in a
+    picture rather than a test. Pair it with `NullPlot` — a student who has seen
+    their finding sitting in the middle of the null does not need a p-value
+    explained.
+
+    The statistic is yours to write. Naming what you are measuring is the
+    thinking part; running it a thousand times is the part worth delegating.
+
+    ```python
+    def gap(frame):
+        means = frame.groupby("is_bot")["length"].mean()
+        return means[True] - means[False]
+
+    result = NullDistribution(gap, n_iter=1000, seed=42).run(data, label="is_bot")
+    ```
+    """
+
+    def __init__(
+        self,
+        statistic: Callable[[pd.DataFrame], float],
+        n_iter: int = 1000,
+        seed: Optional[int] = None,
+    ) -> None:
+        """
+        Args:
+            statistic: Takes a dataframe, returns one number. Called once on the
+                data as given, then once per shuffle.
+            n_iter: Number of shuffles. 1000 is enough to see the shape; go to
+                10_000 when the observed value sits far out in the tail.
+            seed: Passed to `numpy.random.default_rng`. Set it, or the picture
+                changes every time the cell is run.
+        """
+        if n_iter < 1:
+            raise ValueError(f"n_iter must be at least 1, got {n_iter}")
+        self.statistic = statistic
+        self.n_iter = n_iter
+        self.seed = seed
+
+    def run(self, data: pd.DataFrame, label: str) -> NullResult:
+        """Measure `statistic` on `data`, then on `n_iter` shuffles of `label`.
+
+        Shuffling one column breaks the link between it and every other column
+        while leaving both distributions exactly as they were: the labels keep
+        their proportions, the values keep their shape, and only the pairing is
+        destroyed. That is the "nothing is happening" world to compare against.
+
+        Args:
+            data: The frame to measure. It is never modified.
+            label: Column to shuffle — the one whose association with the rest
+                of the frame is in question.
+
+        Returns:
+            A `NullResult` holding the observed value and the shuffled cloud.
+        """
+        if label not in data.columns:
+            raise ValueError(
+                f"Column '{label}' is not in the data; got {list(data.columns)}"
+            )
+
+        observed = float(self.statistic(data))
+        if not np.isfinite(observed):
+            raise ValueError(
+                f"The statistic returned {observed} on the unshuffled data; "
+                "there is nothing to compare a null against."
+            )
+
+        rng = np.random.default_rng(self.seed)
+        values = data[label].to_numpy()
+        shuffled = data.copy()
+
+        null = np.empty(self.n_iter)
+        for i in range(self.n_iter):
+            # A plain array, not a Series: assigning a permuted Series realigns
+            # on the index and puts every label straight back where it started.
+            shuffled[label] = rng.permutation(values)
+            null[i] = float(self.statistic(shuffled))
+
+        if not np.all(np.isfinite(null)):
+            logger.warning(
+                f"{int(np.sum(~np.isfinite(null)))} of {self.n_iter} shuffles gave a "
+                "non-finite statistic; a small group may be emptying under shuffling."
+            )
+
+        return NullResult(
+            observed=observed, null=null, n_iter=self.n_iter, seed=self.seed
+        )
+
+
 def fit_table(results: List[Result]) -> pd.DataFrame:
     """Turn a list of fit results into a ranked dataframe: what a student hands in.
 
@@ -343,8 +494,12 @@ def fit_table(results: List[Result]) -> pd.DataFrame:
                     "distribution": result.distribution,
                     "params": result.params,
                     "log_likelihood": result.log_likelihood,
-                    "ks_stat": result.kstest.statistic if result.kstest else float("nan"),
-                    "ks_pvalue": result.kstest.p_value if result.kstest else float("nan"),
+                    "ks_stat": result.kstest.statistic
+                    if result.kstest
+                    else float("nan"),
+                    "ks_pvalue": result.kstest.p_value
+                    if result.kstest
+                    else float("nan"),
                     "best_likelihood": bool(result.best_likelihood),
                     "best_ks": bool(result.best_ks),
                     "message": result.message,