data-sampler 3.2.1__tar.gz → 3.3.1__tar.gz

{data_sampler-3.2.1 → data_sampler-3.3.1}/CHANGELOG.md

@@ -1,5 +1,33 @@
 # Changelog
 
+## v3.3.1 — 2026-07-23
+
+- **Fixed the second CI-only TUI race** (caught, again, by the release CI —
+  it blocked v3.3.0's PyPI publish): a stale `Changed` message (mount-time
+  echo or superseded edit, delivered late because every Textual widget runs
+  its own message pump) could roll an anonymizer choice back to `none` and
+  wipe its options. All Select/Input/Switch handlers now drop any `Changed`
+  whose value no longer matches the widget's current value — provably stale
+  — which also subsumes the v3.2.1 duplicate-highlight fix. v3.3.0 was never
+  published to PyPI; this release carries all of its changes.
+
+## v3.3.0 — 2026-07-23
+
+- **Two-phase narrow sampling in the DuckDB engine.** The expensive phase of a
+  sample (the per-stratum window sort, or the reservoir buffer) now runs over
+  only the stratification columns plus a stable row id — `file_row_number` for
+  single-file Parquet sources, a positional id for DataFrames — and a second
+  pass fetches the winning rows with every column. Measured on 300k rows ×
+  401 columns: stratified sampling 2.5× faster on Parquet and 9.5× faster for
+  DataFrame sources (whose wide payload never enters the SQL engine; winners
+  return as dtype-preserving pandas slices). CSV/TSV/JSON and multi-file
+  Parquet globs keep the previous single-pass shape (per-file row numbers are
+  not a global id — verification caught that a glob would otherwise be
+  silently over-sampled — and text formats must re-parse per scan anyway).
+  No behavioral change to counts, allocation, NaN strata, or determinism;
+  note that *which* rows a given seed selects changes vs v3.2.1 for
+  single-file Parquet and DataFrame sources.
+
 ## v3.2.1 — 2026-07-23
 
 - **Fixed a TUI race that only surfaced on slow machines** (caught by the

{data_sampler-3.2.1 → data_sampler-3.3.1}/PKG-INFO

@@ -1,7 +1,7 @@
 Metadata-Version: 2.4
 Name: data-sampler
-Version: 3.2.1
-Summary: Representative (stratified) sampling and column anonymization for tabular data files, with a terminal UI.
+Version: 3.3.1
+Summary: Hand someone data that looks and behaves like your production data, isn't your production data, and provably kept its statistical variety — stratified sampling + consistent anonymization, with a terminal UI.
 Project-URL: Homepage, https://github.com/aaronified/data-sampler
 Project-URL: Changelog, https://github.com/aaronified/data-sampler/blob/main/CHANGELOG.md
 Author: aaronified
@@ -55,23 +55,51 @@ Description-Content-Type: text/markdown
 
 # data-sampler
 
-Creates representative samples from data files, using stratified sampling to
-preserve the statistical variety of your data — with optional per-column
-anonymization and a colorful terminal UI.
+[![PyPI](https://img.shields.io/pypi/v/data-sampler)](https://pypi.org/project/data-sampler/)
+[![Python](https://img.shields.io/pypi/pyversions/data-sampler)](https://pypi.org/project/data-sampler/)
 
-Everything ships as a single Python package: launch the TUI with one
-function call or command, or use the sampling/anonymization functions
-directly from Python.
+**Hand someone data that looks and behaves like your production data, isn't
+your production data, and provably kept its statistical variety — in one
+command.**
+
+```sh
+data-sampler customers.xlsx 500 --suggest
+```
+
+Use it to:
+
+- **Share a realistic slice with a vendor or contractor** — names, ids,
+  emails, salaries, and dates anonymized, but every distribution, duplicate,
+  and group structure intact.
+- **Attach a repro to a bug report** without leaking the real records that
+  trigger it.
+- **Cut a 2 GB export down to 500 rows** for a demo or prototype that still
+  behaves like the real thing.
+- **Build test fixtures that mirror production skew** instead of uniform toy
+  data.
+- **Give a class or workshop realistic data** without a data-sharing
+  agreement.
+- **Pull 10,000 representative rows from a 100M-row Parquet file** without
+  loading it — the optional DuckDB engine samples out-of-core, in parallel.
+- **Prove the sample is representative**: every run produces a side-by-side
+  source-vs-sample distribution report, per column.
+
+How: stratified sampling preserves the statistical variety of your data
+(strata are detected automatically), and every anonymizer maps each unique
+original value to exactly one replacement — so repeated values stay repeated
+and the joint distributions survive anonymization. Everything ships as a
+single Python package: a colorful terminal UI for non-programmers, a headless
+CLI, and a plain Python API.
 
 ## Install
 
 ```sh
-pip install data-sampler        # once released on PyPI
-# from a clone, today:
-pip install -e ".[dev]"
+pip install data-sampler             # from PyPI
+pip install "data-sampler[large]"    # + the out-of-core DuckDB engine for huge files
 ```
 
-Requires Python 3.10+.
+Requires Python 3.10+. For development, clone the repo and
+`pip install -e ".[dev]"`.
 
 ## Terminal UI
 
@@ -394,6 +422,92 @@ The pandas path on the same file: 5.6 s total while materializing a ~0.9 GB
 frame in RAM — the engine's reservoir sampling is ~50× faster and never
 materializes the source at all.
 
+## How it scales: the algorithms
+
+Every trick used to handle millions-to-billions of rows and thousands of
+columns, in one place.
+
+### Millions (to billions) of rows
+
+- **Out-of-core execution.** With the DuckDB engine, loading, stratification,
+  and sampling run inside a vectorized, multi-threaded SQL engine with a
+  memory limit and a temp directory — it spills to disk instead of OOM-ing,
+  and only the resulting sample ever becomes a DataFrame.
+- **Reservoir sampling** for the random case: a single streaming pass with
+  O(sample size) memory, an exact row count, and `REPEATABLE(seed)`
+  reproducibility that is independent of the file format and thread count.
+- **Two-pass stratified sampling.** Pass 1 is a `GROUP BY` over the
+  stratification columns — one row per stratum, tiny regardless of source
+  size. Largest-remainder proportional allocation is computed on that tiny
+  table in numpy. Pass 2 ranks rows per stratum with
+  `row_number() OVER (PARTITION BY strata ORDER BY random())` and keeps the
+  first *allocation* rows of each, joining the allocation table with
+  `IS NOT DISTINCT FROM` so missing-value strata are sampled too. DuckDB
+  parallelizes the partitions and spills the window sort if needed.
+- **Parquet projection pushdown.** Pass 1 and the stats queries touch only
+  the columns they reference, so a wide Parquet file is never read in full.
+- **Two-phase narrow sampling.** The expensive phase (the per-stratum window
+  sort, or the reservoir buffer) runs over *only the stratification columns
+  plus a stable row id* — `file_row_number` for single-file Parquet, a
+  positional id for DataFrames — and a second pass fetches just the winning
+  rows with every column. The sort never carries the wide payload: measured
+  2.5× on a 400-column Parquet file and 9.5× for wide DataFrames (whose
+  payload never enters the SQL engine at all — winners come back as
+  dtype-preserving pandas slices). CSV/JSON and multi-file Parquet globs
+  keep the single-pass shape, which is the correct one there (text must be
+  re-parsed per scan, and per-file row numbers aren't a global id).
+- **Determinism engineering.** DuckDB's `GROUP BY` output order is
+  nondeterministic, so the stratum order is pinned with `ORDER BY … NULLS
+  LAST` before allocation (otherwise remainder ties break differently run to
+  run); seeded stratified runs drop to a single thread because `random()`
+  ordering is only reproducible that way, then restore the thread count.
+- **Row-count caching.** `count(*)` runs once per source per engine session
+  instead of once per operation.
+- **Vectorized anonymizers.** The column is dictionary-encoded once with
+  `pd.factorize`, each *unique* value gets one replacement, and the result is
+  assembled by a fancy-index gather — the in-process equivalent of a native
+  join against a mapping table. Cost scales with unique values plus one
+  vectorized pass, never a Python loop over rows: sequential IDs are an
+  `np.arange`, numeric/datetime jitter are single vectorized RNG draws.
+
+### Thousands of columns
+
+- **One scan for all scalar stats.** `DuckDBEngine.stats()` computes count,
+  distinct, min/max/mean/std, and median for *every column in a single
+  aggregate query* — one streaming pass over the data regardless of how many
+  columns there are.
+- **Sketches instead of sorts.** Distinct counts use HyperLogLog
+  (`approx_count_distinct`) and medians use `approx_quantile` — streaming
+  approximations with fixed memory per column, no per-column sort or full
+  hash table. Exact mode (`approximate=False`) exists for small data, and
+  approximate results are flagged on the stats object.
+- **`distributions=False`** skips the per-column histogram/top-k passes
+  entirely, so very wide tables get exactly one scan (this is what the CLI's
+  `--suggest` uses to pick anonymizer types).
+- **Bounded stratification search.** Candidate columns are screened in one
+  aggregate pass (HLL cardinality + average text length), then a greedy
+  fewest-categories-first selection keeps the *joint* stratum count at or
+  below the sample size — so the group count stays bounded no matter how
+  many columns the file has.
+- **Pandas-path trims.** Numeric columns skip the top-values stringification
+  (~1.9× faster stats), report histograms skip near-unique columns instead
+  of hashing millions of ids for a meaningless top-8, and the TUI computes
+  stats in a worker thread so the UI never freezes on load.
+
+### Correctness under scale
+
+Details that only bite on real data, all regression-tested: DuckDB treats NaN
+as a value rather than NULL, so every NaN-sensitive aggregate is filtered
+(`FILTER (WHERE NOT isnan(…))`); missing-value strata survive the allocation
+join via `IS NOT DISTINCT FROM`; HyperLogLog estimates are clamped to each
+column's non-null count; and columns-oriented JSON (the pandas `to_json()`
+default, which SQL engines parse as one giant row) is detected and refused
+with guidance.
+
+Known trade-offs: CSV sources are re-parsed per query (the streaming design —
+convert to Parquet for repeated work), and a seeded stratified run gives up
+multi-threading for reproducibility (unseeded runs use all cores).
+
 ## Supported formats
 
 | Format | Extensions |
@@ -420,7 +534,8 @@ Logging is controlled by `DATA_SAMPLER_LOG` (`quiet`/`info`/`verbose`) and
 `DATA_SAMPLER_LOG_FILE`. See `ROADMAP.md` for planned work and
 `TROUBLESHOOTING.md` for known failure modes.
 
-PyPI releases are manual and happen only after extensive testing.
+Releases go to [PyPI](https://pypi.org/project/data-sampler/) via a
+human-triggered, test-gated GitHub Actions workflow — see `RELEASING.md`.
 
 ---
 

{data_sampler-3.2.1 → data_sampler-3.3.1}/README.md

@@ -1,22 +1,50 @@
 # data-sampler
 
-Creates representative samples from data files, using stratified sampling to
-preserve the statistical variety of your data — with optional per-column
-anonymization and a colorful terminal UI.
+[![PyPI](https://img.shields.io/pypi/v/data-sampler)](https://pypi.org/project/data-sampler/)
+[![Python](https://img.shields.io/pypi/pyversions/data-sampler)](https://pypi.org/project/data-sampler/)
 
-Everything ships as a single Python package: launch the TUI with one
-function call or command, or use the sampling/anonymization functions
-directly from Python.
+**Hand someone data that looks and behaves like your production data, isn't
+your production data, and provably kept its statistical variety — in one
+command.**
+
+```sh
+data-sampler customers.xlsx 500 --suggest
+```
+
+Use it to:
+
+- **Share a realistic slice with a vendor or contractor** — names, ids,
+  emails, salaries, and dates anonymized, but every distribution, duplicate,
+  and group structure intact.
+- **Attach a repro to a bug report** without leaking the real records that
+  trigger it.
+- **Cut a 2 GB export down to 500 rows** for a demo or prototype that still
+  behaves like the real thing.
+- **Build test fixtures that mirror production skew** instead of uniform toy
+  data.
+- **Give a class or workshop realistic data** without a data-sharing
+  agreement.
+- **Pull 10,000 representative rows from a 100M-row Parquet file** without
+  loading it — the optional DuckDB engine samples out-of-core, in parallel.
+- **Prove the sample is representative**: every run produces a side-by-side
+  source-vs-sample distribution report, per column.
+
+How: stratified sampling preserves the statistical variety of your data
+(strata are detected automatically), and every anonymizer maps each unique
+original value to exactly one replacement — so repeated values stay repeated
+and the joint distributions survive anonymization. Everything ships as a
+single Python package: a colorful terminal UI for non-programmers, a headless
+CLI, and a plain Python API.
 
 ## Install
 
 ```sh
-pip install data-sampler        # once released on PyPI
-# from a clone, today:
-pip install -e ".[dev]"
+pip install data-sampler             # from PyPI
+pip install "data-sampler[large]"    # + the out-of-core DuckDB engine for huge files
 ```
 
-Requires Python 3.10+.
+Requires Python 3.10+. For development, clone the repo and
+`pip install -e ".[dev]"`.
 
 ## Terminal UI
 
@@ -339,6 +367,92 @@ The pandas path on the same file: 5.6 s total while materializing a ~0.9 GB
 frame in RAM — the engine's reservoir sampling is ~50× faster and never
 materializes the source at all.
 
+## How it scales: the algorithms
+
+Every trick used to handle millions-to-billions of rows and thousands of
+columns, in one place.
+
+### Millions (to billions) of rows
+
+- **Out-of-core execution.** With the DuckDB engine, loading, stratification,
+  and sampling run inside a vectorized, multi-threaded SQL engine with a
+  memory limit and a temp directory — it spills to disk instead of OOM-ing,
+  and only the resulting sample ever becomes a DataFrame.
+- **Reservoir sampling** for the random case: a single streaming pass with
+  O(sample size) memory, an exact row count, and `REPEATABLE(seed)`
+  reproducibility that is independent of the file format and thread count.
+- **Two-pass stratified sampling.** Pass 1 is a `GROUP BY` over the
+  stratification columns — one row per stratum, tiny regardless of source
+  size. Largest-remainder proportional allocation is computed on that tiny
+  table in numpy. Pass 2 ranks rows per stratum with
+  `row_number() OVER (PARTITION BY strata ORDER BY random())` and keeps the
+  first *allocation* rows of each, joining the allocation table with
+  `IS NOT DISTINCT FROM` so missing-value strata are sampled too. DuckDB
+  parallelizes the partitions and spills the window sort if needed.
+- **Parquet projection pushdown.** Pass 1 and the stats queries touch only
+  the columns they reference, so a wide Parquet file is never read in full.
+- **Two-phase narrow sampling.** The expensive phase (the per-stratum window
+  sort, or the reservoir buffer) runs over *only the stratification columns
+  plus a stable row id* — `file_row_number` for single-file Parquet, a
+  positional id for DataFrames — and a second pass fetches just the winning
+  rows with every column. The sort never carries the wide payload: measured
+  2.5× on a 400-column Parquet file and 9.5× for wide DataFrames (whose
+  payload never enters the SQL engine at all — winners come back as
+  dtype-preserving pandas slices). CSV/JSON and multi-file Parquet globs
+  keep the single-pass shape, which is the correct one there (text must be
+  re-parsed per scan, and per-file row numbers aren't a global id).
+- **Determinism engineering.** DuckDB's `GROUP BY` output order is
+  nondeterministic, so the stratum order is pinned with `ORDER BY … NULLS
+  LAST` before allocation (otherwise remainder ties break differently run to
+  run); seeded stratified runs drop to a single thread because `random()`
+  ordering is only reproducible that way, then restore the thread count.
+- **Row-count caching.** `count(*)` runs once per source per engine session
+  instead of once per operation.
+- **Vectorized anonymizers.** The column is dictionary-encoded once with
+  `pd.factorize`, each *unique* value gets one replacement, and the result is
+  assembled by a fancy-index gather — the in-process equivalent of a native
+  join against a mapping table. Cost scales with unique values plus one
+  vectorized pass, never a Python loop over rows: sequential IDs are an
+  `np.arange`, numeric/datetime jitter are single vectorized RNG draws.
+
+### Thousands of columns
+
+- **One scan for all scalar stats.** `DuckDBEngine.stats()` computes count,
+  distinct, min/max/mean/std, and median for *every column in a single
+  aggregate query* — one streaming pass over the data regardless of how many
+  columns there are.
+- **Sketches instead of sorts.** Distinct counts use HyperLogLog
+  (`approx_count_distinct`) and medians use `approx_quantile` — streaming
+  approximations with fixed memory per column, no per-column sort or full
+  hash table. Exact mode (`approximate=False`) exists for small data, and
+  approximate results are flagged on the stats object.
+- **`distributions=False`** skips the per-column histogram/top-k passes
+  entirely, so very wide tables get exactly one scan (this is what the CLI's
+  `--suggest` uses to pick anonymizer types).
+- **Bounded stratification search.** Candidate columns are screened in one
+  aggregate pass (HLL cardinality + average text length), then a greedy
+  fewest-categories-first selection keeps the *joint* stratum count at or
+  below the sample size — so the group count stays bounded no matter how
+  many columns the file has.
+- **Pandas-path trims.** Numeric columns skip the top-values stringification
+  (~1.9× faster stats), report histograms skip near-unique columns instead
+  of hashing millions of ids for a meaningless top-8, and the TUI computes
+  stats in a worker thread so the UI never freezes on load.
+
+### Correctness under scale
+
+Details that only bite on real data, all regression-tested: DuckDB treats NaN
+as a value rather than NULL, so every NaN-sensitive aggregate is filtered
+(`FILTER (WHERE NOT isnan(…))`); missing-value strata survive the allocation
+join via `IS NOT DISTINCT FROM`; HyperLogLog estimates are clamped to each
+column's non-null count; and columns-oriented JSON (the pandas `to_json()`
+default, which SQL engines parse as one giant row) is detected and refused
+with guidance.
+
+Known trade-offs: CSV sources are re-parsed per query (the streaming design —
+convert to Parquet for repeated work), and a seeded stratified run gives up
+multi-threading for reproducibility (unseeded runs use all cores).
+
 ## Supported formats
 
 | Format | Extensions |
@@ -365,7 +479,8 @@ Logging is controlled by `DATA_SAMPLER_LOG` (`quiet`/`info`/`verbose`) and
 `DATA_SAMPLER_LOG_FILE`. See `ROADMAP.md` for planned work and
 `TROUBLESHOOTING.md` for known failure modes.
 
-PyPI releases are manual and happen only after extensive testing.
+Releases go to [PyPI](https://pypi.org/project/data-sampler/) via a
+human-triggered, test-gated GitHub Actions workflow — see `RELEASING.md`.
 
 ---
 

{data_sampler-3.2.1 → data_sampler-3.3.1}/RELEASING.md

@@ -5,23 +5,19 @@ human publishes a GitHub Release (or manually dispatches the workflow). The
 workflow uses **PyPI Trusted Publishing** (OIDC), so no API token is stored in
 the repo, in CI secrets, or anywhere else.
 
-## One-time setup (PyPI side)
-
-1. Create a PyPI account at <https://pypi.org/account/register/> and enable 2FA.
-2. Because `data-sampler` has no release on PyPI yet, register a **pending
-   publisher**: <https://pypi.org/manage/account/publishing/> → "Add a new
-   pending publisher" with:
-   - PyPI project name: `data-sampler`
-   - Owner: `aaronified`
-   - Repository: `data-sampler`
-   - Workflow name: `release.yml`
-   - Environment name: `pypi`
-3. (Recommended) In GitHub: Settings → Environments → create `pypi` and add
-   yourself as a required reviewer. The publish job then waits for your
-   click-to-approve on every release.
-
-After the first successful publish, the pending publisher becomes the project's
-trusted publisher automatically.
+## One-time setup (PyPI side) — DONE (v3.2.1, 2026-07-23)
+
+Completed: the PyPI account exists, and the pending publisher (project
+`data-sampler`, owner `aaronified`, repo `data-sampler`, workflow
+`release.yml`, environment `pypi`) became the project's permanent **trusted
+publisher** with the first publish. Nothing to repeat per release.
+
+If it ever needs re-registering (new repo/owner/workflow name), manage it at
+<https://pypi.org/manage/project/data-sampler/settings/publishing/>.
+
+Still recommended: in GitHub → Settings → Environments → `pypi`, add yourself
+as a required reviewer so every publish waits for your click-to-approve
+(currently publishes run unattended once tests pass).
 
 ## Per release
 

{data_sampler-3.2.1 → data_sampler-3.3.1}/ROADMAP.md

@@ -89,9 +89,19 @@ its own commit.
   the per-column passes for a single cheap scalar pass over very wide inputs.
   `approx_count_distinct` also powers the engine's stratification-column selection.
 
+## v3.3
+
+- [x] **Block P7 — two-phase narrow sampling.** Rank/reservoir over only the
+  stratification columns + a stable row id (single-file Parquet
+  `file_row_number`; DataFrame positional id), then fetch the winning full
+  rows. 2.5× on wide Parquet, 9.5× on wide DataFrames. CSV/JSON and Parquet
+  globs keep the single-pass shape (per-file row ids are not global).
+
 ## Later
 
 - Optional PyInstaller EXE build of the TUI (replaces the old Tkinter EXE).
+- Composite filename+row-number id to extend narrow sampling to Parquet globs.
+- `--columns` output projection (sample only a subset of columns).
 - Rust/Polars-on-Arrow native engine (pyo3) as an alternative to DuckDB.
 - GPU acceleration (RAPIDS cuDF) for the aggregation-heavy paths.
 - Distributed backend (Dask / Ray Data / Spark) for multi-machine scale.

{data_sampler-3.2.1 → data_sampler-3.3.1}/TROUBLESHOOTING.md

@@ -232,6 +232,63 @@ all were fixed except one accepted divergence. Highlights worth remembering:
   *future* messages; a synchronous re-entrancy flag does not protect against
   them. Guard by making re-syncs conditional, not by flags alone.
 
+## Parquet file_row_number is per-file, not global (caught in verification)
+
+- **Symptom:** the first draft of two-phase narrow sampling used
+  `read_parquet(..., file_row_number=true)` as the row id for the winner
+  fetch. On a multi-file glob (`data/*.parquet`) the verifier's probe showed a
+  request for 1,200 rows silently returning 3,600 with hundreds of duplicates
+  — each winner's row number matched one row in *every* file.
+- **Cause:** DuckDB numbers `file_row_number` per file; it is only a global
+  row id for a single physical file. The fetch join fanned out across files.
+- **Fix:** `_narrow_scan` gates the Parquet narrow shape on `Path.is_file()`;
+  globs/multi-file datasets take the full-width single-pass shape, which never
+  relies on row numbers. Regression test:
+  `test_narrow_glob_multi_file_parquet_falls_back_and_stays_exact`. A
+  composite `filename + file_row_number` id could lift this restriction later.
+- **Lesson:** any "stable row id" assumption must be validated against
+  multi-file sources — the single-file case hides the fan-out completely.
+
+## CI-only TUI failure #2: stale Changed messages (v3.3.0 release run)
+
+- **Symptom:** the v3.3.0 publish was blocked by the same Windows-CI test one
+  assertion further along: the Select edit survived (v3.2.1's fix held) but
+  the follow-up Input edit vanished (`KeyError: 'start'`).
+- **Cause:** third mechanism in the same family. Every Textual widget runs
+  its own message pump, so a stale `Select.Changed("none")` (mount-time echo
+  or superseded edit) can be *delivered* after the user's edits were applied.
+  The handler rolled `cfg.kind` back to `none` — wiping `options` — before
+  the Input edit's assertion.
+- **Fix:** a general staleness guard (`_is_stale`) on every Changed handler
+  (Select/Input/Switch): a message whose value no longer matches its widget's
+  *current* value is provably stale and dropped — the message carrying the
+  current value is either this one or still in flight. Subsumes the earlier
+  duplicate-highlight class too. Regression test delivers stale messages
+  directly (`test_stale_changed_messages_do_not_clobber_config`).
+- **Lesson (general):** in Textual, never apply a Changed message's payload
+  without checking it still matches the widget — cross-widget delivery order
+  is unspecified, and only loaded machines (CI) expose it.
+
+## CI-only TUI failure #3: message-delivery lag (v3.3.1 release run)
+
+- **Symptom:** with both real races fixed, the release CI failed once more —
+  this time inside the new regression test itself, before any stale message
+  was delivered: `options["start"]` was asserted one `pilot.pause()` after
+  setting the Input, and on a slow runner the `Changed` had not yet been
+  *delivered* to the screen.
+- **Cause:** not an app bug. `pilot.pause()` drains the app-level queue, but
+  every Textual widget runs its own message pump — cross-widget delivery can
+  outlive a single pause. App state is eventually consistent (and the
+  `_is_stale` guard keeps it *correct* whenever it settles); only the test's
+  assert-after-one-pause pattern was racy.
+- **Fix:** a `wait_until(pilot, predicate)` test helper that polls until the
+  state converges (bounded, fails loudly on timeout), used for every
+  cross-widget assertion. Tests-only change — the packaged artifact is
+  unaffected.
+- **Lesson:** the full pattern for Textual testing: *wait_for_screen* for
+  navigation, *wait_until* for cross-widget state, and never assert
+  immediately after a single pause.
+
 ## Template
 
 - **Symptom:** what was observed.

{data_sampler-3.2.1 → data_sampler-3.3.1}/pyproject.toml

@@ -4,8 +4,8 @@ build-backend = "hatchling.build"
 
 [project]
 name = "data-sampler"
-version = "3.2.1"
-description = "Representative (stratified) sampling and column anonymization for tabular data files, with a terminal UI."
+version = "3.3.1"
+description = "Hand someone data that looks and behaves like your production data, isn't your production data, and provably kept its statistical variety — stratified sampling + consistent anonymization, with a terminal UI."
 readme = "README.md"
 license = { file = "LICENSE" }
 requires-python = ">=3.10"

{data_sampler-3.2.1 → data_sampler-3.3.1}/src/data_sampler/__init__.py

@@ -16,7 +16,7 @@ Public API::
 
 from __future__ import annotations
 
-__version__ = "3.2.1"
+__version__ = "3.3.1"
 
 from .io import SUPPORTED_EXTENSIONS, list_sheets, load_file, save_output
 from .report import (

{data_sampler-3.2.1 → data_sampler-3.3.1}/src/data_sampler/engine.py

@@ -434,6 +434,72 @@ class DuckDBEngine:
 
     # ── sampling ──────────────────────────────────────────────────────────────
 
+    # column names the two-phase narrow shape needs for itself; a source that
+    # already uses one of these falls back to the single-pass full-width shape
+    _NARROW_RESERVED = frozenset({"file_row_number", "_ds_rid", "_ds_rowid", "_rn"})
+
+    def _narrow_scan(self, source, strat_cols: list[str]):
+        """Set up a two-phase "narrow" sample: rank/sample over only
+        ``strat_cols`` plus a stable row id, then fetch the winning full rows.
+
+        Sorting or reservoir-buffering the full row payload is the dominant
+        cost on wide tables; this shape makes phase 1 width-independent.
+        Returns ``(scan_sql, rid_expr, fetch)`` where ``fetch(row_ids)``
+        materializes the winners with every column — or ``None`` when the
+        source has no cheap stable row id (CSV/TSV/JSON must re-parse the
+        whole text per scan, so single-pass full-width is already the optimal
+        shape there) or a column name collides with the shape's internals.
+        """
+        if isinstance(source, pd.DataFrame):
+            if self._NARROW_RESERVED & set(map(str, source.columns)):
+                return None
+            # only the stratification columns + a positional id enter DuckDB;
+            # the wide payload never leaves pandas
+            if strat_cols:
+                narrow = source.loc[:, list(strat_cols)].reset_index(drop=True).copy()
+            else:
+                narrow = pd.DataFrame()
+            narrow["_ds_rid"] = np.arange(len(source), dtype=np.int64)
+            self._reg += 1
+            name = f"_narrow_{self._reg}"
+            self.con.register(name, narrow)
+
+            def fetch(row_ids: np.ndarray) -> pd.DataFrame:
+                # positional take preserves the original dtypes exactly
+                return source.take(np.asarray(row_ids)).reset_index(drop=True)
+
+            return name, _quote_ident("_ds_rid"), fetch
+
+        p = Path(source)
+        if p.suffix.lower() != ".parquet":
+            return None
+        # file_row_number is numbered PER FILE: it is a valid global row id
+        # only for a single physical file. Globs / multi-file datasets
+        # (data/*.parquet) must use the full-width fallback, which never
+        # relies on it — a per-file id would fan each winner out to every
+        # file sharing that row number and silently corrupt the sample.
+        # (A composite filename+row id could lift this later.)
+        if not p.is_file():
+            return None
+        if self._NARROW_RESERVED & set(map(str, self.columns(source))):
+            return None
+        scan = f"read_parquet({_quote_str(str(p))}, file_row_number=true)"
+
+        def fetch(row_ids: np.ndarray) -> pd.DataFrame:
+            self._reg += 1
+            rid_tbl = f"_rids_{self._reg}"
+            self.con.register(
+                rid_tbl, pd.DataFrame({"rid": np.asarray(row_ids, dtype=np.int64)})
+            )
+            # ORDER BY the row id: deterministic output order for seeded runs
+            return self.con.execute(
+                f"SELECT s.* EXCLUDE (file_row_number) FROM {scan} s "
+                f"JOIN {rid_tbl} r ON s.file_row_number = r.rid "
+                f"ORDER BY s.file_row_number"
+            ).df()
+
+        return scan, "file_row_number", fetch
+
     def sample(
         self,
         source,
@@ -469,29 +535,40 @@ class DuckDBEngine:
         self._set_seed(seed)
 
         if use_random:
-            return self._reservoir(src, count, seed)
+            return self._reservoir(source, src, count, seed)
 
         if strat_cols is None:
             strat_cols = self.find_stratification_columns(source, count, exclude_columns)
         if not strat_cols:
-            result = self._reservoir(src, count, seed)
+            result = self._reservoir(source, src, count, seed)
             result.notes = ["No suitable stratification columns found; using reservoir sampling."]
             return result
 
-        return self._stratified(src, count, strat_cols, seed)
+        return self._stratified(source, src, count, strat_cols, seed)
 
-    def _reservoir(self, src: str, count: int, seed: int | None) -> SampleResult:
+    def _reservoir(self, source, src: str, count: int, seed: int | None) -> SampleResult:
         rep = f" REPEATABLE ({int(seed)})" if seed is not None else ""
-        data = self.con.execute(
-            f"SELECT * FROM {src} USING SAMPLE reservoir({int(count)} ROWS){rep}"
-        ).df()
+        narrow = self._narrow_scan(source, [])
+        if narrow is not None:
+            # phase 1: reservoir over just the row id (width-independent);
+            # phase 2: fetch the winners' full rows
+            scan, rid_expr, fetch = narrow
+            row_ids = self.con.execute(
+                f"SELECT {rid_expr} AS rid FROM {scan} "
+                f"USING SAMPLE reservoir({int(count)} ROWS){rep}"
+            ).df()["rid"].to_numpy()
+            data = fetch(row_ids)
+        else:
+            data = self.con.execute(
+                f"SELECT * FROM {src} USING SAMPLE reservoir({int(count)} ROWS){rep}"
+            ).df()
         return SampleResult(
             data=data, method="random", requested=count,
             notes=["Mode: DuckDB reservoir sampling (out-of-core)"],
         )
 
     def _stratified(
-        self, src: str, count: int, strat_cols: list[str], seed: int | None
+        self, source, src: str, count: int, strat_cols: list[str], seed: int | None
     ) -> SampleResult:
         cols_sql = ", ".join(_quote_ident(c) for c in strat_cols)
         # pass 1: stratum sizes (small — one row per stratum). ORDER BY pins
@@ -515,18 +592,37 @@ class DuckDBEngine:
             f"r.{_quote_ident(c)} IS NOT DISTINCT FROM a.{_quote_ident(c)}"
             for c in strat_cols
         )
-        query = f"""
-            WITH ranked AS (
-                SELECT *, row_number() OVER (
-                    PARTITION BY {cols_sql} ORDER BY random()
-                ) AS _rn
-                FROM {src}
-            )
-            SELECT r.* EXCLUDE (_rn)
-            FROM ranked r
-            JOIN {alloc_tbl} a ON {join_on}
-            WHERE r._rn <= a._alloc
-        """
+        narrow = self._narrow_scan(source, strat_cols)
+        if narrow is not None:
+            # two-phase: rank only the strat columns + row id (the window sort
+            # never touches the wide payload), then fetch the winners' rows
+            scan, rid_expr, fetch = narrow
+            query = f"""
+                WITH ranked AS (
+                    SELECT {rid_expr} AS _ds_rowid, {cols_sql}, row_number() OVER (
+                        PARTITION BY {cols_sql} ORDER BY random()
+                    ) AS _rn
+                    FROM {scan}
+                )
+                SELECT r._ds_rowid
+                FROM ranked r
+                JOIN {alloc_tbl} a ON {join_on}
+                WHERE r._rn <= a._alloc
+            """
+        else:
+            fetch = None
+            query = f"""
+                WITH ranked AS (
+                    SELECT *, row_number() OVER (
+                        PARTITION BY {cols_sql} ORDER BY random()
+                    ) AS _rn
+                    FROM {src}
+                )
+                SELECT r.* EXCLUDE (_rn)
+                FROM ranked r
+                JOIN {alloc_tbl} a ON {join_on}
+                WHERE r._rn <= a._alloc
+            """
         # DuckDB's random() ordering is only reproducible single-threaded, so a
         # seeded run goes single-threaded for determinism; unseeded runs keep all
         # cores (the distribution is preserved either way).
@@ -535,7 +631,11 @@ class DuckDBEngine:
             self.con.execute("PRAGMA threads=1")
             self._set_seed(seed)
         try:
-            data = self.con.execute(query).df()
+            result_df = self.con.execute(query).df()
+            if fetch is not None:
+                data = fetch(result_df["_ds_rowid"].to_numpy())
+            else:
+                data = result_df
         finally:
             if reproducible:
                 self.con.execute(f"PRAGMA threads={self.threads}")

{data_sampler-3.2.1 → data_sampler-3.3.1}/src/data_sampler/tui/app.py

@@ -527,9 +527,25 @@ class ColumnsScreen(Screen):
         self._show_detail(self.selected)
         self._sync_config_panel(self.selected)
 
+    @staticmethod
+    def _is_stale(event_value, widget_value) -> bool:
+        """Whether a Changed message no longer reflects its widget's value.
+
+        Every widget runs its own message pump, so a Changed can be delivered
+        AFTER the value it reports was superseded (mount-time echoes, rapid
+        edits, panel syncs). Applying it would roll the config back — the
+        exact class of race that reset user edits on slow CI machines. A
+        message whose value differs from the widget's current value is
+        provably stale and safe to drop: the message carrying the current
+        value is either this one or still on its way.
+        """
+        return str(event_value) != str(widget_value)
+
     def on_select_changed(self, event: Select.Changed) -> None:
         if self._syncing or self.selected is None:
             return
+        if self._is_stale(event.value, event.select.value):
+            return
         cfg = self.configs[self.selected]
         if event.select.id == "anon-kind":
             if str(event.value) == cfg.kind:
@@ -545,12 +561,16 @@ class ColumnsScreen(Screen):
         if event.switch.id == "skip-strat":
             if self._syncing or self.selected is None:
                 return
+            if self._is_stale(event.value, event.switch.value):
+                return
             self.configs[self.selected].skip_strat = event.value
             self._refresh_row(self.selected)
 
     def on_input_changed(self, event: Input.Changed) -> None:
         if self._syncing or self.selected is None:
             return
+        if self._is_stale(event.value, event.input.value):
+            return
         mapping = self._OPT_IDS.get(event.input.id or "")
         if mapping is None:
             return

{data_sampler-3.2.1 → data_sampler-3.3.1}/tests/test_engine.py

@@ -339,6 +339,134 @@ def test_row_count_cached_per_file(tmp_path, big_df):
         assert len(result.data) == 50
 
 
+# ── two-phase narrow sampling (rank narrow, fetch winners) ───────────────────
+
+def test_narrow_parquet_stratified_schema_count_distribution(tmp_path, engine, big_df):
+    p = tmp_path / "narrow.parquet"
+    big_df.to_parquet(p, index=False)
+    result = engine.sample(str(p), 500, strat_cols=["region"], seed=2)
+    assert len(result.data) == 500
+    # full schema, original order, no file_row_number/_ds_rowid leak
+    assert list(result.data.columns) == list(big_df.columns)
+    src_p = big_df["region"].value_counts(normalize=True)
+    smp_p = result.data["region"].value_counts(normalize=True)
+    for cat in src_p.index:
+        assert abs(src_p[cat] - smp_p.get(cat, 0)) < 0.05
+
+
+def test_narrow_parquet_seeded_reproducible_across_engines(tmp_path, big_df):
+    p = tmp_path / "repro.parquet"
+    big_df.to_parquet(p, index=False)
+    with DuckDBEngine(threads=4) as e1:
+        s1 = e1.sample(str(p), 300, strat_cols=["region", "tier"], seed=9)
+        r1 = e1.sample(str(p), 300, use_random=True, seed=9)
+    with DuckDBEngine(threads=4) as e2:
+        s2 = e2.sample(str(p), 300, strat_cols=["region", "tier"], seed=9)
+        r2 = e2.sample(str(p), 300, use_random=True, seed=9)
+    # stratified: identical rows AND order (fetch is ORDER BY row id)
+    assert s1.data["id"].tolist() == s2.data["id"].tolist()
+    assert sorted(r1.data["id"].tolist()) == sorted(r2.data["id"].tolist())
+
+
+def test_narrow_nan_stratum_survives_both_sources(tmp_path, engine, big_df):
+    df = big_df.copy()
+    df.loc[df.index[:2000], "region"] = None  # 10% missing stratum
+    p = tmp_path / "nanstrat.parquet"
+    df.to_parquet(p, index=False)
+    for source in (str(p), df):
+        result = engine.sample(source, 1000, strat_cols=["region"], seed=1)
+        assert len(result.data) == 1000
+        # the missing-value stratum is sampled proportionally (~100 rows)
+        nan_rows = int(result.data["region"].isna().sum())
+        assert 60 <= nan_rows <= 140
+
+
+def test_narrow_guard_reserved_column_names_fall_back(tmp_path, engine, big_df):
+    # a real column named file_row_number must not break Parquet sampling
+    decoy = big_df.copy()
+    decoy["file_row_number"] = 1
+    p = tmp_path / "decoy.parquet"
+    decoy.to_parquet(p, index=False)
+    result = engine.sample(str(p), 100, strat_cols=["region"], seed=1)
+    assert len(result.data) == 100
+    assert "file_row_number" in result.data.columns
+    assert (result.data["file_row_number"] == 1).all()
+    # same for a DataFrame column named _ds_rid
+    decoy2 = big_df.copy()
+    decoy2["_ds_rid"] = 7
+    result2 = engine.sample(decoy2, 100, strat_cols=["region"], seed=1)
+    assert len(result2.data) == 100
+    assert (result2.data["_ds_rid"] == 7).all()
+
+
+def test_narrow_hostile_strat_column_name(engine):
+    hostile = 'my col"; DROP TABLE x;--'
+    df = pd.DataFrame({hostile: (["a", "b", "c"] * 400)[:1000], "v": range(1000)})
+    result = engine.sample(df, 100, strat_cols=[hostile], seed=1)
+    assert len(result.data) == 100
+    assert set(result.data.columns) == {hostile, "v"}
+
+
+def test_narrow_dataframe_preserves_dtypes(engine):
+    # the pandas take() fetch returns slices of the original frame, so exotic
+    # dtypes (nullable string, categorical) survive exactly
+    df = pd.DataFrame(
+        {
+            "s": pd.Series((["x", "y", None] * 400)[:1000], dtype="string"),
+            "c": pd.Categorical((["g", "h"] * 500)[:1000]),
+            "grp": (["a", "b"] * 500)[:1000],
+        }
+    )
+    result = engine.sample(df, 100, strat_cols=["grp"], seed=3)
+    assert isinstance(result.data["s"].dtype, pd.StringDtype)
+    assert isinstance(result.data["c"].dtype, pd.CategoricalDtype)
+
+
+def test_narrow_wide_parquet_correctness(tmp_path, engine):
+    # 120 payload columns: all must come back, in order, exact count
+    rng = np.random.default_rng(4)
+    n = 5000
+    wide = pd.DataFrame({f"c{i:03d}": rng.normal(size=n) for i in range(120)})
+    wide.insert(0, "grp", rng.choice(["a", "b", "c", "d"], n))
+    p = tmp_path / "wide.parquet"
+    wide.to_parquet(p, index=False)
+    result = engine.sample(str(p), 200, strat_cols=["grp"], seed=5)
+    assert len(result.data) == 200
+    assert list(result.data.columns) == list(wide.columns)
+    reservoir = engine.sample(str(p), 200, use_random=True, seed=5)
+    assert len(reservoir.data) == 200
+    assert list(reservoir.data.columns) == list(wide.columns)
+
+
+def test_narrow_glob_multi_file_parquet_falls_back_and_stays_exact(tmp_path, engine, big_df):
+    """Regression (caught in verification): file_row_number is per-FILE, so the
+    narrow shape on a multi-file glob would fan winners out across files and
+    silently inflate/duplicate the sample. Globs must take the full-width
+    fallback and return exact, duplicate-free counts."""
+    for i in range(3):
+        big_df.iloc[i * 5000 : (i + 1) * 5000].to_parquet(
+            tmp_path / f"part{i}.parquet", index=False
+        )
+    glob = str(tmp_path / "*.parquet")
+    assert engine._narrow_scan(glob, ["region"]) is None  # gate: not a single file
+    strat = engine.sample(glob, 1200, strat_cols=["region"], seed=1)
+    assert len(strat.data) == 1200
+    assert not strat.data["id"].duplicated().any()
+    res = engine.sample(glob, 1200, use_random=True, seed=1)
+    assert len(res.data) == 1200
+    assert not res.data["id"].duplicated().any()
+
+
+def test_narrow_csv_keeps_single_pass_and_works(tmp_path, engine, big_df):
+    # CSV has no stable row id: the full-width single-pass shape still applies
+    p = tmp_path / "plain.csv"
+    big_df.to_csv(p, index=False)
+    assert engine._narrow_scan(str(p), ["region"]) is None
+    result = engine.sample(str(p), 300, strat_cols=["region"], seed=1)
+    assert len(result.data) == 300
+    assert set(result.data.columns) == set(big_df.columns)
+
+
 # ── module-level helpers ──────────────────────────────────────────────────────
 
 def test_should_use_engine_for_parquet(tmp_path, big_df):

{data_sampler-3.2.1 → data_sampler-3.3.1}/tests/test_tui.py

@@ -40,6 +40,22 @@ async def wait_for_screen(app, pilot, screen_type, tries=100):
     raise AssertionError(f"never reached {screen_type.__name__}; on {type(app.screen).__name__}")
 
 
+async def wait_until(pilot, predicate, what="condition", tries=150):
+    """Wait for cross-widget state to converge before asserting.
+
+    Every Textual widget runs its own message pump, so a single
+    ``pilot.pause()`` does not guarantee a posted Changed message has been
+    DELIVERED to the screen yet — assert-after-one-pause is a timing lottery
+    that only loses on slow CI runners. The app state is eventually
+    consistent; tests must wait for it.
+    """
+    for _ in range(tries):
+        if predicate():
+            return
+        await pilot.pause(0.05)
+    raise AssertionError(f"never converged: {what}")
+
+
 def test_file_screen_rejects_missing_file(tmp_path):
     async def go():
         app = DataSamplerApp()
@@ -80,11 +96,15 @@ def test_full_flow_sample_and_anonymize(csv_file, tmp_path):
             # configure: anonymize the highlighted column (first = "id")
             assert screen.selected == "id"
             screen.query_one("#anon-kind", Select).value = "sequential_id"
-            await pilot.pause()
-            assert screen.configs["id"].kind == "sequential_id"
+            await wait_until(
+                pilot, lambda: screen.configs["id"].kind == "sequential_id",
+                "anon kind applied",
+            )
             screen.query_one("#opt-seq-start", Input).value = "5000"
-            await pilot.pause()
-            assert screen.configs["id"].options["start"] == "5000"
+            await wait_until(
+                pilot, lambda: screen.configs["id"].options.get("start") == "5000",
+                "start option applied",
+            )
             # skip a column from stratification
             screen.configs["region"].skip_strat = True
             # run
@@ -139,14 +159,54 @@ def test_duplicate_row_highlight_does_not_clobber_pending_edit(csv_file):
             # simulate the late duplicate mount-time highlight arriving first
             fake = SimpleNamespace(row_key=SimpleNamespace(value="id"))
             screen.on_data_table_row_highlighted(fake)
-            await pilot.pause()
-            assert screen.configs["id"].kind == "sequential_id"
+            await wait_until(
+                pilot, lambda: screen.configs["id"].kind == "sequential_id",
+                "kind survives duplicate highlight",
+            )
             # and the widget was not reset underneath the user
             assert screen.query_one("#anon-kind", Select).value == "sequential_id"
 
     run(go())
 
 
+def test_stale_changed_messages_do_not_clobber_config(csv_file):
+    """Regression: the CI-only race that blocked the v3.3.0 release.
+
+    Widgets have independent message pumps, so a stale Select.Changed("none")
+    (mount-time echo or superseded edit) can be delivered AFTER the user set
+    "sequential_id" and AFTER their option edits. Applying it resets the kind
+    and wipes the options (KeyError: 'start' on CI). A Changed whose value no
+    longer matches the widget's current value must be dropped as stale.
+    """
+    async def go():
+        app = DataSamplerApp(path=str(csv_file))
+        async with app.run_test(size=(140, 45)) as pilot:
+            screen = await wait_for_screen(app, pilot, ColumnsScreen)
+            assert screen.selected == "id"
+            sel = screen.query_one("#anon-kind", Select)
+            sel.value = "sequential_id"
+            await wait_until(
+                pilot, lambda: screen.configs["id"].kind == "sequential_id",
+                "anon kind applied",
+            )
+            start_input = screen.query_one("#opt-seq-start", Input)
+            start_input.value = "5000"
+            await wait_until(
+                pilot, lambda: screen.configs["id"].options.get("start") == "5000",
+                "start option applied",
+            )
+            # deliver a stale Changed("none"): widget shows "sequential_id",
+            # so the handler must drop it instead of resetting the config
+            screen.on_select_changed(Select.Changed(sel, "none"))
+            assert screen.configs["id"].kind == "sequential_id"
+            assert screen.configs["id"].options["start"] == "5000"
+            # stale Input.Changed likewise (widget holds "5000", message "1")
+            screen.on_input_changed(Input.Changed(start_input, "1"))
+            assert screen.configs["id"].options["start"] == "5000"
+
+    run(go())
+
+
 def test_invalid_count_notifies_instead_of_running(csv_file):
     async def go():
         app = DataSamplerApp(path=str(csv_file))