data-sampler 3.2.1__tar.gz → 3.4.0__tar.gz

{data_sampler-3.2.1 → data_sampler-3.4.0}/CHANGELOG.md

@@ -1,5 +1,69 @@
 # Changelog
 
+## v3.4.0 — 2026-07-24
+
+- **Added optional PCA column reduction** (`data_sampler.reduce`): after
+  sampling (and anonymization, if any), the numeric block of the outgoing
+  sample can be collapsed into its first *k* principal components — so the
+  delivered file is narrow as well as short. *k* is controlled one of two
+  ways: a component count (`--reduce-components N` /
+  `reduce_columns(df, n_components=N)`, capped at the number of usable
+  numeric columns) or a variance target (`--reduce-variance R` /
+  `variance_ratio=R`: the fewest components whose cumulative
+  explained-variance ratio reaches R). Non-numeric columns are always
+  preserved; `--reduce-exclude` passes chosen numeric columns (e.g.
+  identifiers) through unchanged, `--reduce-prefix` renames the `PC*`
+  output columns when the source already has one, and the tool warns when
+  an included column looks like an identifier. Reduce flags are validated
+  before sampling starts, so a typo never costs a full out-of-core run. Missing values are mean-imputed (the row count
+  never changes), constant/all-missing columns pass through with a note, and
+  columns are z-scored by default (`--reduce-no-standardize` opts out) so
+  large-unit columns cannot dominate. The exact-SVD implementation is pure
+  numpy — no new dependency — and deterministic across runs and platforms
+  (sign-fixed singular vectors). New public API: `reduce_columns`,
+  `ReductionResult`, `format_reduction_report`.
+- **Every reduction reports its rationale**, not just the variance kept: the
+  report lists per-component and cumulative explained variance, **the groups
+  of correlated columns that move together** (connected components of the
+  |r| ≥ 0.7 correlation graph — on standardized data PCA diagonalizes
+  exactly that correlation matrix, so the groups are a faithful account of
+  what compressed), each component's top driving columns, and any
+  imputation/passthrough/identifier notes. The full labelled correlation
+  matrix is available on `ReductionResult.correlation_matrix`.
+- Wired into all three surfaces: CLI flags on both the pandas and DuckDB
+  engine paths (the reduction always runs on the small sampled frame, never
+  the full source), a `reduce` control in the TUI run bar with the rationale
+  shown on the report screen, and the Python API. Output files gain a
+  `_pca{k}` tag suffix (e.g. `data_sample_500_anon_pca3.csv`).
+
+## v3.3.1 — 2026-07-23
+
+- **Fixed the second CI-only TUI race** (caught, again, by the release CI —
+  it blocked v3.3.0's PyPI publish): a stale `Changed` message (mount-time
+  echo or superseded edit, delivered late because every Textual widget runs
+  its own message pump) could roll an anonymizer choice back to `none` and
+  wipe its options. All Select/Input/Switch handlers now drop any `Changed`
+  whose value no longer matches the widget's current value — provably stale
+  — which also subsumes the v3.2.1 duplicate-highlight fix. v3.3.0 was never
+  published to PyPI; this release carries all of its changes.
+
+## v3.3.0 — 2026-07-23
+
+- **Two-phase narrow sampling in the DuckDB engine.** The expensive phase of a
+  sample (the per-stratum window sort, or the reservoir buffer) now runs over
+  only the stratification columns plus a stable row id — `file_row_number` for
+  single-file Parquet sources, a positional id for DataFrames — and a second
+  pass fetches the winning rows with every column. Measured on 300k rows ×
+  401 columns: stratified sampling 2.5× faster on Parquet and 9.5× faster for
+  DataFrame sources (whose wide payload never enters the SQL engine; winners
+  return as dtype-preserving pandas slices). CSV/TSV/JSON and multi-file
+  Parquet globs keep the previous single-pass shape (per-file row numbers are
+  not a global id — verification caught that a glob would otherwise be
+  silently over-sampled — and text formats must re-parse per scan anyway).
+  No behavioral change to counts, allocation, NaN strata, or determinism;
+  note that *which* rows a given seed selects changes vs v3.2.1 for
+  single-file Parquet and DataFrame sources.
+
 ## v3.2.1 — 2026-07-23
 
 - **Fixed a TUI race that only surfaced on slow machines** (caught by the

{data_sampler-3.2.1 → data_sampler-3.4.0}/PKG-INFO

@@ -1,7 +1,7 @@
 Metadata-Version: 2.4
 Name: data-sampler
-Version: 3.2.1
-Summary: Representative (stratified) sampling and column anonymization for tabular data files, with a terminal UI.
+Version: 3.4.0
+Summary: Hand someone data that looks and behaves like your production data, isn't your production data, and provably kept its statistical variety — stratified sampling + consistent anonymization, with a terminal UI.
 Project-URL: Homepage, https://github.com/aaronified/data-sampler
 Project-URL: Changelog, https://github.com/aaronified/data-sampler/blob/main/CHANGELOG.md
 Author: aaronified
@@ -55,23 +55,51 @@ Description-Content-Type: text/markdown
 
 # data-sampler
 
-Creates representative samples from data files, using stratified sampling to
-preserve the statistical variety of your data — with optional per-column
-anonymization and a colorful terminal UI.
+[![PyPI](https://img.shields.io/pypi/v/data-sampler)](https://pypi.org/project/data-sampler/)
+[![Python](https://img.shields.io/pypi/pyversions/data-sampler)](https://pypi.org/project/data-sampler/)
 
-Everything ships as a single Python package: launch the TUI with one
-function call or command, or use the sampling/anonymization functions
-directly from Python.
+**Hand someone data that looks and behaves like your production data, isn't
+your production data, and provably kept its statistical variety — in one
+command.**
+
+```sh
+data-sampler customers.xlsx 500 --suggest
+```
+
+Use it to:
+
+- **Share a realistic slice with a vendor or contractor** — names, ids,
+  emails, salaries, and dates anonymized, but every distribution, duplicate,
+  and group structure intact.
+- **Attach a repro to a bug report** without leaking the real records that
+  trigger it.
+- **Cut a 2 GB export down to 500 rows** for a demo or prototype that still
+  behaves like the real thing.
+- **Build test fixtures that mirror production skew** instead of uniform toy
+  data.
+- **Give a class or workshop realistic data** without a data-sharing
+  agreement.
+- **Pull 10,000 representative rows from a 100M-row Parquet file** without
+  loading it — the optional DuckDB engine samples out-of-core, in parallel.
+- **Prove the sample is representative**: every run produces a side-by-side
+  source-vs-sample distribution report, per column.
+
+How: stratified sampling preserves the statistical variety of your data
+(strata are detected automatically), and every anonymizer maps each unique
+original value to exactly one replacement — so repeated values stay repeated
+and the joint distributions survive anonymization. Everything ships as a
+single Python package: a colorful terminal UI for non-programmers, a headless
+CLI, and a plain Python API.
 
 ## Install
 
 ```sh
-pip install data-sampler        # once released on PyPI
-# from a clone, today:
-pip install -e ".[dev]"
+pip install data-sampler             # from PyPI
+pip install "data-sampler[large]"    # + the out-of-core DuckDB engine for huge files
 ```
 
-Requires Python 3.10+.
+Requires Python 3.10+. For development, clone the repo and
+`pip install -e ".[dev]"`.
 
 ## Terminal UI
 
@@ -99,7 +127,8 @@ The TUI is a panel-based dashboard (think btop / lazydocker):
    VS Code extension). Select a column to see full stats and distribution
    bars, choose an anonymizer for it, and toggle whether it should be
    skipped when preserving statistical variety (stratification). Set the
-   sample size, output folder, optional seed, and run.
+   sample size, output folder, optional seed, an optional PCA column
+   reduction (`reduce`: N components or a variance target), and run.
 3. **Report screen** — the stratification comparison and anonymization summary
    on the left, and a **column histograms** panel on the right showing every
    column's source-vs-sample distribution (numeric columns share bin edges;
@@ -126,6 +155,11 @@ data-sampler <source> <count> [options]
 | `--anon COL=KIND[:k=v,...]` | Anonymize a column (repeatable) |
 | `-i`, `--interactive` | Guided workflow: choose an anonymizer type per column from a menu |
 | `--suggest` | Auto-assign a suggested anonymizer type to each column from its stats |
+| `--reduce-components N` | PCA column reduction: replace the numeric columns with the first `N` principal components |
+| `--reduce-variance R` | PCA column reduction: keep the fewest components whose cumulative explained variance reaches `R` (0 < R < 1) |
+| `--reduce-exclude COL[,COL]` | Numeric column(s) to keep out of the reduction, e.g. identifiers (repeatable) |
+| `--reduce-prefix PREFIX` | Name prefix for the component columns (default `PC` → `PC1`, `PC2`, …) |
+| `--reduce-no-standardize` | Skip the per-column z-scoring before PCA |
 | `--engine {auto,pandas,duckdb}` | Sampling engine (default `auto`: DuckDB for Parquet/large inputs, pandas otherwise) |
 | `--threads N` | DuckDB engine: number of threads (default: all cores) |
 | `--memory-limit SIZE` | DuckDB engine: memory limit before spilling to disk (e.g. `8GB`) |
@@ -144,6 +178,11 @@ data-sampler data.csv 100 --skip region,notes --seed 7 \
 
 # large / out-of-core: sample a Parquet file in parallel with DuckDB
 data-sampler huge.parquet 10000 --engine duckdb --threads 8 --memory-limit 8GB --suggest
+
+# narrow the sample too: collapse the numeric columns into 3 principal
+# components (or keep however many retain 90% of the variance)
+data-sampler wide.csv 500 --reduce-components 3 --reduce-exclude cust_id
+data-sampler wide.csv 500 --reduce-variance 0.9
 ```
 
 ## Python API
@@ -176,7 +215,11 @@ anon = ds.anonymize(
     seed=7,
 )
 
-ds.save_output(anon, "data.xlsx", tag="sample_500_anon")
+# optionally collapse the numeric columns into principal components
+red = ds.reduce_columns(anon, variance_ratio=0.9, exclude=["cust_id"])
+print(ds.format_reduction_report(red))   # variance kept + correlated groups
+
+ds.save_output(red.data, "data.xlsx", tag="sample_500_anon_pca")
 ```
 
 ## Try it: bundled example
@@ -344,6 +387,34 @@ category. A side-by-side distribution report is produced for every run.
 If no suitable stratification columns exist, the tool falls back to pure
 random sampling automatically.
 
+### Reducing columns (PCA)
+
+Sampling narrows the *rows*; the optional PCA step narrows the *columns* of
+the outgoing sample. It replaces the numeric block with its first *k*
+principal components (`PC1..PCk`), controlled one of two ways:
+
+- **`--reduce-components N`** — `N` components in the output (capped at the
+  number of usable numeric columns, with a note when fewer are possible);
+- **`--reduce-variance R`** — the fewest components whose cumulative
+  explained-variance ratio reaches `R` (e.g. `0.9` keeps ≥ 90 % of the
+  numeric variance).
+
+Non-numeric columns (ids, categories, text, booleans, datetimes) are always
+preserved, and `--reduce-exclude` keeps chosen numeric columns out too —
+identifiers should be excluded, since an all-unique id forms its own
+artificial component (the tool warns when it spots one). Missing values are
+mean-imputed so the row count never changes; constant columns carry no signal
+and pass through unchanged. Columns are z-scored first by default (PCA on the
+correlation matrix), so a large-unit column such as a salary cannot dominate
+the components; `--reduce-no-standardize` turns that off.
+
+Every reduction prints its rationale: the variance each component retains,
+**the groups of correlated columns** that move together (which is exactly the
+redundancy PCA collapses — on standardized data PCA diagonalizes the
+correlation matrix), and each component's top driving columns. The reduction
+runs after anonymization, on the already-sampled rows, so the stratification
+and histogram reports still describe the original columns.
+
 ## Large data: the out-of-core DuckDB engine
 
 The default pandas path loads the whole file into memory. For inputs that are
@@ -394,6 +465,97 @@ The pandas path on the same file: 5.6 s total while materializing a ~0.9 GB
 frame in RAM — the engine's reservoir sampling is ~50× faster and never
 materializes the source at all.
 
+## How it scales: the algorithms
+
+Every trick used to handle millions-to-billions of rows and thousands of
+columns, in one place.
+
+### Millions (to billions) of rows
+
+- **Out-of-core execution.** With the DuckDB engine, loading, stratification,
+  and sampling run inside a vectorized, multi-threaded SQL engine with a
+  memory limit and a temp directory — it spills to disk instead of OOM-ing,
+  and only the resulting sample ever becomes a DataFrame.
+- **Reservoir sampling** for the random case: a single streaming pass with
+  O(sample size) memory, an exact row count, and `REPEATABLE(seed)`
+  reproducibility that is independent of the file format and thread count.
+- **Two-pass stratified sampling.** Pass 1 is a `GROUP BY` over the
+  stratification columns — one row per stratum, tiny regardless of source
+  size. Largest-remainder proportional allocation is computed on that tiny
+  table in numpy. Pass 2 ranks rows per stratum with
+  `row_number() OVER (PARTITION BY strata ORDER BY random())` and keeps the
+  first *allocation* rows of each, joining the allocation table with
+  `IS NOT DISTINCT FROM` so missing-value strata are sampled too. DuckDB
+  parallelizes the partitions and spills the window sort if needed.
+- **Parquet projection pushdown.** Pass 1 and the stats queries touch only
+  the columns they reference, so a wide Parquet file is never read in full.
+- **Two-phase narrow sampling.** The expensive phase (the per-stratum window
+  sort, or the reservoir buffer) runs over *only the stratification columns
+  plus a stable row id* — `file_row_number` for single-file Parquet, a
+  positional id for DataFrames — and a second pass fetches just the winning
+  rows with every column. The sort never carries the wide payload: measured
+  2.5× on a 400-column Parquet file and 9.5× for wide DataFrames (whose
+  payload never enters the SQL engine at all — winners come back as
+  dtype-preserving pandas slices). CSV/JSON and multi-file Parquet globs
+  keep the single-pass shape, which is the correct one there (text must be
+  re-parsed per scan, and per-file row numbers aren't a global id).
+- **Determinism engineering.** DuckDB's `GROUP BY` output order is
+  nondeterministic, so the stratum order is pinned with `ORDER BY … NULLS
+  LAST` before allocation (otherwise remainder ties break differently run to
+  run); seeded stratified runs drop to a single thread because `random()`
+  ordering is only reproducible that way, then restore the thread count.
+- **Row-count caching.** `count(*)` runs once per source per engine session
+  instead of once per operation.
+- **Vectorized anonymizers.** The column is dictionary-encoded once with
+  `pd.factorize`, each *unique* value gets one replacement, and the result is
+  assembled by a fancy-index gather — the in-process equivalent of a native
+  join against a mapping table. Cost scales with unique values plus one
+  vectorized pass, never a Python loop over rows: sequential IDs are an
+  `np.arange`, numeric/datetime jitter are single vectorized RNG draws.
+
+### Thousands of columns
+
+- **One scan for all scalar stats.** `DuckDBEngine.stats()` computes count,
+  distinct, min/max/mean/std, and median for *every column in a single
+  aggregate query* — one streaming pass over the data regardless of how many
+  columns there are.
+- **Sketches instead of sorts.** Distinct counts use HyperLogLog
+  (`approx_count_distinct`) and medians use `approx_quantile` — streaming
+  approximations with fixed memory per column, no per-column sort or full
+  hash table. Exact mode (`approximate=False`) exists for small data, and
+  approximate results are flagged on the stats object.
+- **`distributions=False`** skips the per-column histogram/top-k passes
+  entirely, so very wide tables get exactly one scan (this is what the CLI's
+  `--suggest` uses to pick anonymizer types).
+- **Bounded stratification search.** Candidate columns are screened in one
+  aggregate pass (HLL cardinality + average text length), then a greedy
+  fewest-categories-first selection keeps the *joint* stratum count at or
+  below the sample size — so the group count stays bounded no matter how
+  many columns the file has.
+- **Pandas-path trims.** Numeric columns skip the top-values stringification
+  (~1.9× faster stats), report histograms skip near-unique columns instead
+  of hashing millions of ids for a meaningless top-8, and the TUI computes
+  stats in a worker thread so the UI never freezes on load.
+- **PCA column reduction on the way out.** `--reduce-components` /
+  `--reduce-variance` collapse a wide numeric block into a handful of
+  principal components *after* sampling, so the SVD runs on the small sampled
+  frame (never the full source) and the delivered file is narrow as well as
+  short.
+
+### Correctness under scale
+
+Details that only bite on real data, all regression-tested: DuckDB treats NaN
+as a value rather than NULL, so every NaN-sensitive aggregate is filtered
+(`FILTER (WHERE NOT isnan(…))`); missing-value strata survive the allocation
+join via `IS NOT DISTINCT FROM`; HyperLogLog estimates are clamped to each
+column's non-null count; and columns-oriented JSON (the pandas `to_json()`
+default, which SQL engines parse as one giant row) is detected and refused
+with guidance.
+
+Known trade-offs: CSV sources are re-parsed per query (the streaming design —
+convert to Parquet for repeated work), and a seeded stratified run gives up
+multi-threading for reproducibility (unseeded runs use all cores).
+
 ## Supported formats
 
 | Format | Extensions |
@@ -406,7 +568,8 @@ materializes the source at all.
 
 Output keeps the source format and is named
 `{stem}_sample_{count}{ext}` — with an `_anon` suffix when anonymization ran
-(e.g. `data_sample_500_anon.csv`).
+and a `_pca{k}` suffix when PCA column reduction ran
+(e.g. `data_sample_500_anon_pca3.csv`).
 
 ## Development
 
@@ -420,7 +583,8 @@ Logging is controlled by `DATA_SAMPLER_LOG` (`quiet`/`info`/`verbose`) and
 `DATA_SAMPLER_LOG_FILE`. See `ROADMAP.md` for planned work and
 `TROUBLESHOOTING.md` for known failure modes.
 
-PyPI releases are manual and happen only after extensive testing.
+Releases go to [PyPI](https://pypi.org/project/data-sampler/) via a
+human-triggered, test-gated GitHub Actions workflow — see `RELEASING.md`.
 
 ---
 

{data_sampler-3.2.1 → data_sampler-3.4.0}/README.md

@@ -1,22 +1,50 @@
 # data-sampler
 
-Creates representative samples from data files, using stratified sampling to
-preserve the statistical variety of your data — with optional per-column
-anonymization and a colorful terminal UI.
+[![PyPI](https://img.shields.io/pypi/v/data-sampler)](https://pypi.org/project/data-sampler/)
+[![Python](https://img.shields.io/pypi/pyversions/data-sampler)](https://pypi.org/project/data-sampler/)
 
-Everything ships as a single Python package: launch the TUI with one
-function call or command, or use the sampling/anonymization functions
-directly from Python.
+**Hand someone data that looks and behaves like your production data, isn't
+your production data, and provably kept its statistical variety — in one
+command.**
+
+```sh
+data-sampler customers.xlsx 500 --suggest
+```
+
+Use it to:
+
+- **Share a realistic slice with a vendor or contractor** — names, ids,
+  emails, salaries, and dates anonymized, but every distribution, duplicate,
+  and group structure intact.
+- **Attach a repro to a bug report** without leaking the real records that
+  trigger it.
+- **Cut a 2 GB export down to 500 rows** for a demo or prototype that still
+  behaves like the real thing.
+- **Build test fixtures that mirror production skew** instead of uniform toy
+  data.
+- **Give a class or workshop realistic data** without a data-sharing
+  agreement.
+- **Pull 10,000 representative rows from a 100M-row Parquet file** without
+  loading it — the optional DuckDB engine samples out-of-core, in parallel.
+- **Prove the sample is representative**: every run produces a side-by-side
+  source-vs-sample distribution report, per column.
+
+How: stratified sampling preserves the statistical variety of your data
+(strata are detected automatically), and every anonymizer maps each unique
+original value to exactly one replacement — so repeated values stay repeated
+and the joint distributions survive anonymization. Everything ships as a
+single Python package: a colorful terminal UI for non-programmers, a headless
+CLI, and a plain Python API.
 
 ## Install
 
 ```sh
-pip install data-sampler        # once released on PyPI
-# from a clone, today:
-pip install -e ".[dev]"
+pip install data-sampler             # from PyPI
+pip install "data-sampler[large]"    # + the out-of-core DuckDB engine for huge files
 ```
 
-Requires Python 3.10+.
+Requires Python 3.10+. For development, clone the repo and
+`pip install -e ".[dev]"`.
 
 ## Terminal UI
 
@@ -44,7 +72,8 @@ The TUI is a panel-based dashboard (think btop / lazydocker):
    VS Code extension). Select a column to see full stats and distribution
    bars, choose an anonymizer for it, and toggle whether it should be
    skipped when preserving statistical variety (stratification). Set the
-   sample size, output folder, optional seed, and run.
+   sample size, output folder, optional seed, an optional PCA column
+   reduction (`reduce`: N components or a variance target), and run.
 3. **Report screen** — the stratification comparison and anonymization summary
    on the left, and a **column histograms** panel on the right showing every
    column's source-vs-sample distribution (numeric columns share bin edges;
@@ -71,6 +100,11 @@ data-sampler <source> <count> [options]
 | `--anon COL=KIND[:k=v,...]` | Anonymize a column (repeatable) |
 | `-i`, `--interactive` | Guided workflow: choose an anonymizer type per column from a menu |
 | `--suggest` | Auto-assign a suggested anonymizer type to each column from its stats |
+| `--reduce-components N` | PCA column reduction: replace the numeric columns with the first `N` principal components |
+| `--reduce-variance R` | PCA column reduction: keep the fewest components whose cumulative explained variance reaches `R` (0 < R < 1) |
+| `--reduce-exclude COL[,COL]` | Numeric column(s) to keep out of the reduction, e.g. identifiers (repeatable) |
+| `--reduce-prefix PREFIX` | Name prefix for the component columns (default `PC` → `PC1`, `PC2`, …) |
+| `--reduce-no-standardize` | Skip the per-column z-scoring before PCA |
 | `--engine {auto,pandas,duckdb}` | Sampling engine (default `auto`: DuckDB for Parquet/large inputs, pandas otherwise) |
 | `--threads N` | DuckDB engine: number of threads (default: all cores) |
 | `--memory-limit SIZE` | DuckDB engine: memory limit before spilling to disk (e.g. `8GB`) |
@@ -89,6 +123,11 @@ data-sampler data.csv 100 --skip region,notes --seed 7 \
 
 # large / out-of-core: sample a Parquet file in parallel with DuckDB
 data-sampler huge.parquet 10000 --engine duckdb --threads 8 --memory-limit 8GB --suggest
+
+# narrow the sample too: collapse the numeric columns into 3 principal
+# components (or keep however many retain 90% of the variance)
+data-sampler wide.csv 500 --reduce-components 3 --reduce-exclude cust_id
+data-sampler wide.csv 500 --reduce-variance 0.9
 ```
 
 ## Python API
@@ -121,7 +160,11 @@ anon = ds.anonymize(
     seed=7,
 )
 
-ds.save_output(anon, "data.xlsx", tag="sample_500_anon")
+# optionally collapse the numeric columns into principal components
+red = ds.reduce_columns(anon, variance_ratio=0.9, exclude=["cust_id"])
+print(ds.format_reduction_report(red))   # variance kept + correlated groups
+
+ds.save_output(red.data, "data.xlsx", tag="sample_500_anon_pca")
 ```
 
 ## Try it: bundled example
@@ -289,6 +332,34 @@ category. A side-by-side distribution report is produced for every run.
 If no suitable stratification columns exist, the tool falls back to pure
 random sampling automatically.
 
+### Reducing columns (PCA)
+
+Sampling narrows the *rows*; the optional PCA step narrows the *columns* of
+the outgoing sample. It replaces the numeric block with its first *k*
+principal components (`PC1..PCk`), controlled one of two ways:
+
+- **`--reduce-components N`** — `N` components in the output (capped at the
+  number of usable numeric columns, with a note when fewer are possible);
+- **`--reduce-variance R`** — the fewest components whose cumulative
+  explained-variance ratio reaches `R` (e.g. `0.9` keeps ≥ 90 % of the
+  numeric variance).
+
+Non-numeric columns (ids, categories, text, booleans, datetimes) are always
+preserved, and `--reduce-exclude` keeps chosen numeric columns out too —
+identifiers should be excluded, since an all-unique id forms its own
+artificial component (the tool warns when it spots one). Missing values are
+mean-imputed so the row count never changes; constant columns carry no signal
+and pass through unchanged. Columns are z-scored first by default (PCA on the
+correlation matrix), so a large-unit column such as a salary cannot dominate
+the components; `--reduce-no-standardize` turns that off.
+
+Every reduction prints its rationale: the variance each component retains,
+**the groups of correlated columns** that move together (which is exactly the
+redundancy PCA collapses — on standardized data PCA diagonalizes the
+correlation matrix), and each component's top driving columns. The reduction
+runs after anonymization, on the already-sampled rows, so the stratification
+and histogram reports still describe the original columns.
+
 ## Large data: the out-of-core DuckDB engine
 
 The default pandas path loads the whole file into memory. For inputs that are
@@ -339,6 +410,97 @@ The pandas path on the same file: 5.6 s total while materializing a ~0.9 GB
 frame in RAM — the engine's reservoir sampling is ~50× faster and never
 materializes the source at all.
 
+## How it scales: the algorithms
+
+Every trick used to handle millions-to-billions of rows and thousands of
+columns, in one place.
+
+### Millions (to billions) of rows
+
+- **Out-of-core execution.** With the DuckDB engine, loading, stratification,
+  and sampling run inside a vectorized, multi-threaded SQL engine with a
+  memory limit and a temp directory — it spills to disk instead of OOM-ing,
+  and only the resulting sample ever becomes a DataFrame.
+- **Reservoir sampling** for the random case: a single streaming pass with
+  O(sample size) memory, an exact row count, and `REPEATABLE(seed)`
+  reproducibility that is independent of the file format and thread count.
+- **Two-pass stratified sampling.** Pass 1 is a `GROUP BY` over the
+  stratification columns — one row per stratum, tiny regardless of source
+  size. Largest-remainder proportional allocation is computed on that tiny
+  table in numpy. Pass 2 ranks rows per stratum with
+  `row_number() OVER (PARTITION BY strata ORDER BY random())` and keeps the
+  first *allocation* rows of each, joining the allocation table with
+  `IS NOT DISTINCT FROM` so missing-value strata are sampled too. DuckDB
+  parallelizes the partitions and spills the window sort if needed.
+- **Parquet projection pushdown.** Pass 1 and the stats queries touch only
+  the columns they reference, so a wide Parquet file is never read in full.
+- **Two-phase narrow sampling.** The expensive phase (the per-stratum window
+  sort, or the reservoir buffer) runs over *only the stratification columns
+  plus a stable row id* — `file_row_number` for single-file Parquet, a
+  positional id for DataFrames — and a second pass fetches just the winning
+  rows with every column. The sort never carries the wide payload: measured
+  2.5× on a 400-column Parquet file and 9.5× for wide DataFrames (whose
+  payload never enters the SQL engine at all — winners come back as
+  dtype-preserving pandas slices). CSV/JSON and multi-file Parquet globs
+  keep the single-pass shape, which is the correct one there (text must be
+  re-parsed per scan, and per-file row numbers aren't a global id).
+- **Determinism engineering.** DuckDB's `GROUP BY` output order is
+  nondeterministic, so the stratum order is pinned with `ORDER BY … NULLS
+  LAST` before allocation (otherwise remainder ties break differently run to
+  run); seeded stratified runs drop to a single thread because `random()`
+  ordering is only reproducible that way, then restore the thread count.
+- **Row-count caching.** `count(*)` runs once per source per engine session
+  instead of once per operation.
+- **Vectorized anonymizers.** The column is dictionary-encoded once with
+  `pd.factorize`, each *unique* value gets one replacement, and the result is
+  assembled by a fancy-index gather — the in-process equivalent of a native
+  join against a mapping table. Cost scales with unique values plus one
+  vectorized pass, never a Python loop over rows: sequential IDs are an
+  `np.arange`, numeric/datetime jitter are single vectorized RNG draws.
+
+### Thousands of columns
+
+- **One scan for all scalar stats.** `DuckDBEngine.stats()` computes count,
+  distinct, min/max/mean/std, and median for *every column in a single
+  aggregate query* — one streaming pass over the data regardless of how many
+  columns there are.
+- **Sketches instead of sorts.** Distinct counts use HyperLogLog
+  (`approx_count_distinct`) and medians use `approx_quantile` — streaming
+  approximations with fixed memory per column, no per-column sort or full
+  hash table. Exact mode (`approximate=False`) exists for small data, and
+  approximate results are flagged on the stats object.
+- **`distributions=False`** skips the per-column histogram/top-k passes
+  entirely, so very wide tables get exactly one scan (this is what the CLI's
+  `--suggest` uses to pick anonymizer types).
+- **Bounded stratification search.** Candidate columns are screened in one
+  aggregate pass (HLL cardinality + average text length), then a greedy
+  fewest-categories-first selection keeps the *joint* stratum count at or
+  below the sample size — so the group count stays bounded no matter how
+  many columns the file has.
+- **Pandas-path trims.** Numeric columns skip the top-values stringification
+  (~1.9× faster stats), report histograms skip near-unique columns instead
+  of hashing millions of ids for a meaningless top-8, and the TUI computes
+  stats in a worker thread so the UI never freezes on load.
+- **PCA column reduction on the way out.** `--reduce-components` /
+  `--reduce-variance` collapse a wide numeric block into a handful of
+  principal components *after* sampling, so the SVD runs on the small sampled
+  frame (never the full source) and the delivered file is narrow as well as
+  short.
+
+### Correctness under scale
+
+Details that only bite on real data, all regression-tested: DuckDB treats NaN
+as a value rather than NULL, so every NaN-sensitive aggregate is filtered
+(`FILTER (WHERE NOT isnan(…))`); missing-value strata survive the allocation
+join via `IS NOT DISTINCT FROM`; HyperLogLog estimates are clamped to each
+column's non-null count; and columns-oriented JSON (the pandas `to_json()`
+default, which SQL engines parse as one giant row) is detected and refused
+with guidance.
+
+Known trade-offs: CSV sources are re-parsed per query (the streaming design —
+convert to Parquet for repeated work), and a seeded stratified run gives up
+multi-threading for reproducibility (unseeded runs use all cores).
+
 ## Supported formats
 
 | Format | Extensions |
@@ -351,7 +513,8 @@ materializes the source at all.
 
 Output keeps the source format and is named
 `{stem}_sample_{count}{ext}` — with an `_anon` suffix when anonymization ran
-(e.g. `data_sample_500_anon.csv`).
+and a `_pca{k}` suffix when PCA column reduction ran
+(e.g. `data_sample_500_anon_pca3.csv`).
 
 ## Development
 
@@ -365,7 +528,8 @@ Logging is controlled by `DATA_SAMPLER_LOG` (`quiet`/`info`/`verbose`) and
 `DATA_SAMPLER_LOG_FILE`. See `ROADMAP.md` for planned work and
 `TROUBLESHOOTING.md` for known failure modes.
 
-PyPI releases are manual and happen only after extensive testing.
+Releases go to [PyPI](https://pypi.org/project/data-sampler/) via a
+human-triggered, test-gated GitHub Actions workflow — see `RELEASING.md`.
 
 ---
 

{data_sampler-3.2.1 → data_sampler-3.4.0}/RELEASING.md

@@ -5,23 +5,19 @@ human publishes a GitHub Release (or manually dispatches the workflow). The
 workflow uses **PyPI Trusted Publishing** (OIDC), so no API token is stored in
 the repo, in CI secrets, or anywhere else.
 
-## One-time setup (PyPI side)
-
-1. Create a PyPI account at <https://pypi.org/account/register/> and enable 2FA.
-2. Because `data-sampler` has no release on PyPI yet, register a **pending
-   publisher**: <https://pypi.org/manage/account/publishing/> → "Add a new
-   pending publisher" with:
-   - PyPI project name: `data-sampler`
-   - Owner: `aaronified`
-   - Repository: `data-sampler`
-   - Workflow name: `release.yml`
-   - Environment name: `pypi`
-3. (Recommended) In GitHub: Settings → Environments → create `pypi` and add
-   yourself as a required reviewer. The publish job then waits for your
-   click-to-approve on every release.
-
-After the first successful publish, the pending publisher becomes the project's
-trusted publisher automatically.
+## One-time setup (PyPI side) — DONE (v3.2.1, 2026-07-23)
+
+Completed: the PyPI account exists, and the pending publisher (project
+`data-sampler`, owner `aaronified`, repo `data-sampler`, workflow
+`release.yml`, environment `pypi`) became the project's permanent **trusted
+publisher** with the first publish. Nothing to repeat per release.
+
+If it ever needs re-registering (new repo/owner/workflow name), manage it at
+<https://pypi.org/manage/project/data-sampler/settings/publishing/>.
+
+Still recommended: in GitHub → Settings → Environments → `pypi`, add yourself
+as a required reviewer so every publish waits for your click-to-approve
+(currently publishes run unattended once tests pass).
 
 ## Per release
 

{data_sampler-3.2.1 → data_sampler-3.4.0}/ROADMAP.md

@@ -89,9 +89,19 @@ its own commit.
   the per-column passes for a single cheap scalar pass over very wide inputs.
   `approx_count_distinct` also powers the engine's stratification-column selection.
 
+## v3.3
+
+- [x] **Block P7 — two-phase narrow sampling.** Rank/reservoir over only the
+  stratification columns + a stable row id (single-file Parquet
+  `file_row_number`; DataFrame positional id), then fetch the winning full
+  rows. 2.5× on wide Parquet, 9.5× on wide DataFrames. CSV/JSON and Parquet
+  globs keep the single-pass shape (per-file row ids are not global).
+
 ## Later
 
 - Optional PyInstaller EXE build of the TUI (replaces the old Tkinter EXE).
+- Composite filename+row-number id to extend narrow sampling to Parquet globs.
+- `--columns` output projection (sample only a subset of columns).
 - Rust/Polars-on-Arrow native engine (pyo3) as an alternative to DuckDB.
 - GPU acceleration (RAPIDS cuDF) for the aggregation-heavy paths.
 - Distributed backend (Dask / Ray Data / Spark) for multi-machine scale.