data-sampler 3.3.1__tar.gz → 3.4.0__tar.gz

{data_sampler-3.3.1 → data_sampler-3.4.0}/CHANGELOG.md

@@ -1,5 +1,41 @@
 # Changelog
 
+## v3.4.0 — 2026-07-24
+
+- **Added optional PCA column reduction** (`data_sampler.reduce`): after
+  sampling (and anonymization, if any), the numeric block of the outgoing
+  sample can be collapsed into its first *k* principal components — so the
+  delivered file is narrow as well as short. *k* is controlled one of two
+  ways: a component count (`--reduce-components N` /
+  `reduce_columns(df, n_components=N)`, capped at the number of usable
+  numeric columns) or a variance target (`--reduce-variance R` /
+  `variance_ratio=R`: the fewest components whose cumulative
+  explained-variance ratio reaches R). Non-numeric columns are always
+  preserved; `--reduce-exclude` passes chosen numeric columns (e.g.
+  identifiers) through unchanged, `--reduce-prefix` renames the `PC*`
+  output columns when the source already has one, and the tool warns when
+  an included column looks like an identifier. Reduce flags are validated
+  before sampling starts, so a typo never costs a full out-of-core run. Missing values are mean-imputed (the row count
+  never changes), constant/all-missing columns pass through with a note, and
+  columns are z-scored by default (`--reduce-no-standardize` opts out) so
+  large-unit columns cannot dominate. The exact-SVD implementation is pure
+  numpy — no new dependency — and deterministic across runs and platforms
+  (sign-fixed singular vectors). New public API: `reduce_columns`,
+  `ReductionResult`, `format_reduction_report`.
+- **Every reduction reports its rationale**, not just the variance kept: the
+  report lists per-component and cumulative explained variance, **the groups
+  of correlated columns that move together** (connected components of the
+  |r| ≥ 0.7 correlation graph — on standardized data PCA diagonalizes
+  exactly that correlation matrix, so the groups are a faithful account of
+  what compressed), each component's top driving columns, and any
+  imputation/passthrough/identifier notes. The full labelled correlation
+  matrix is available on `ReductionResult.correlation_matrix`.
+- Wired into all three surfaces: CLI flags on both the pandas and DuckDB
+  engine paths (the reduction always runs on the small sampled frame, never
+  the full source), a `reduce` control in the TUI run bar with the rationale
+  shown on the report screen, and the Python API. Output files gain a
+  `_pca{k}` tag suffix (e.g. `data_sample_500_anon_pca3.csv`).
+
 ## v3.3.1 — 2026-07-23
 
 - **Fixed the second CI-only TUI race** (caught, again, by the release CI —

{data_sampler-3.3.1 → data_sampler-3.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: data-sampler
-Version: 3.3.1
+Version: 3.4.0
 Summary: Hand someone data that looks and behaves like your production data, isn't your production data, and provably kept its statistical variety — stratified sampling + consistent anonymization, with a terminal UI.
 Project-URL: Homepage, https://github.com/aaronified/data-sampler
 Project-URL: Changelog, https://github.com/aaronified/data-sampler/blob/main/CHANGELOG.md
@@ -127,7 +127,8 @@ The TUI is a panel-based dashboard (think btop / lazydocker):
    VS Code extension). Select a column to see full stats and distribution
    bars, choose an anonymizer for it, and toggle whether it should be
    skipped when preserving statistical variety (stratification). Set the
-   sample size, output folder, optional seed, and run.
+   sample size, output folder, optional seed, an optional PCA column
+   reduction (`reduce`: N components or a variance target), and run.
 3. **Report screen** — the stratification comparison and anonymization summary
    on the left, and a **column histograms** panel on the right showing every
    column's source-vs-sample distribution (numeric columns share bin edges;
@@ -154,6 +155,11 @@ data-sampler <source> <count> [options]
 | `--anon COL=KIND[:k=v,...]` | Anonymize a column (repeatable) |
 | `-i`, `--interactive` | Guided workflow: choose an anonymizer type per column from a menu |
 | `--suggest` | Auto-assign a suggested anonymizer type to each column from its stats |
+| `--reduce-components N` | PCA column reduction: replace the numeric columns with the first `N` principal components |
+| `--reduce-variance R` | PCA column reduction: keep the fewest components whose cumulative explained variance reaches `R` (0 < R < 1) |
+| `--reduce-exclude COL[,COL]` | Numeric column(s) to keep out of the reduction, e.g. identifiers (repeatable) |
+| `--reduce-prefix PREFIX` | Name prefix for the component columns (default `PC` → `PC1`, `PC2`, …) |
+| `--reduce-no-standardize` | Skip the per-column z-scoring before PCA |
 | `--engine {auto,pandas,duckdb}` | Sampling engine (default `auto`: DuckDB for Parquet/large inputs, pandas otherwise) |
 | `--threads N` | DuckDB engine: number of threads (default: all cores) |
 | `--memory-limit SIZE` | DuckDB engine: memory limit before spilling to disk (e.g. `8GB`) |
@@ -172,6 +178,11 @@ data-sampler data.csv 100 --skip region,notes --seed 7 \
 
 # large / out-of-core: sample a Parquet file in parallel with DuckDB
 data-sampler huge.parquet 10000 --engine duckdb --threads 8 --memory-limit 8GB --suggest
+
+# narrow the sample too: collapse the numeric columns into 3 principal
+# components (or keep however many retain 90% of the variance)
+data-sampler wide.csv 500 --reduce-components 3 --reduce-exclude cust_id
+data-sampler wide.csv 500 --reduce-variance 0.9
 ```
 
 ## Python API
@@ -204,7 +215,11 @@ anon = ds.anonymize(
     seed=7,
 )
 
-ds.save_output(anon, "data.xlsx", tag="sample_500_anon")
+# optionally collapse the numeric columns into principal components
+red = ds.reduce_columns(anon, variance_ratio=0.9, exclude=["cust_id"])
+print(ds.format_reduction_report(red))   # variance kept + correlated groups
+
+ds.save_output(red.data, "data.xlsx", tag="sample_500_anon_pca")
 ```
 
 ## Try it: bundled example
@@ -372,6 +387,34 @@ category. A side-by-side distribution report is produced for every run.
 If no suitable stratification columns exist, the tool falls back to pure
 random sampling automatically.
 
+### Reducing columns (PCA)
+
+Sampling narrows the *rows*; the optional PCA step narrows the *columns* of
+the outgoing sample. It replaces the numeric block with its first *k*
+principal components (`PC1..PCk`), controlled one of two ways:
+
+- **`--reduce-components N`** — `N` components in the output (capped at the
+  number of usable numeric columns, with a note when fewer are possible);
+- **`--reduce-variance R`** — the fewest components whose cumulative
+  explained-variance ratio reaches `R` (e.g. `0.9` keeps ≥ 90 % of the
+  numeric variance).
+
+Non-numeric columns (ids, categories, text, booleans, datetimes) are always
+preserved, and `--reduce-exclude` keeps chosen numeric columns out too —
+identifiers should be excluded, since an all-unique id forms its own
+artificial component (the tool warns when it spots one). Missing values are
+mean-imputed so the row count never changes; constant columns carry no signal
+and pass through unchanged. Columns are z-scored first by default (PCA on the
+correlation matrix), so a large-unit column such as a salary cannot dominate
+the components; `--reduce-no-standardize` turns that off.
+
+Every reduction prints its rationale: the variance each component retains,
+**the groups of correlated columns** that move together (which is exactly the
+redundancy PCA collapses — on standardized data PCA diagonalizes the
+correlation matrix), and each component's top driving columns. The reduction
+runs after anonymization, on the already-sampled rows, so the stratification
+and histogram reports still describe the original columns.
+
 ## Large data: the out-of-core DuckDB engine
 
 The default pandas path loads the whole file into memory. For inputs that are
@@ -493,6 +536,11 @@ columns, in one place.
   (~1.9× faster stats), report histograms skip near-unique columns instead
   of hashing millions of ids for a meaningless top-8, and the TUI computes
   stats in a worker thread so the UI never freezes on load.
+- **PCA column reduction on the way out.** `--reduce-components` /
+  `--reduce-variance` collapse a wide numeric block into a handful of
+  principal components *after* sampling, so the SVD runs on the small sampled
+  frame (never the full source) and the delivered file is narrow as well as
+  short.
 
 ### Correctness under scale
 
@@ -520,7 +568,8 @@ multi-threading for reproducibility (unseeded runs use all cores).
 
 Output keeps the source format and is named
 `{stem}_sample_{count}{ext}` — with an `_anon` suffix when anonymization ran
-(e.g. `data_sample_500_anon.csv`).
+and a `_pca{k}` suffix when PCA column reduction ran
+(e.g. `data_sample_500_anon_pca3.csv`).
 
 ## Development
 

{data_sampler-3.3.1 → data_sampler-3.4.0}/README.md

@@ -72,7 +72,8 @@ The TUI is a panel-based dashboard (think btop / lazydocker):
    VS Code extension). Select a column to see full stats and distribution
    bars, choose an anonymizer for it, and toggle whether it should be
    skipped when preserving statistical variety (stratification). Set the
-   sample size, output folder, optional seed, and run.
+   sample size, output folder, optional seed, an optional PCA column
+   reduction (`reduce`: N components or a variance target), and run.
 3. **Report screen** — the stratification comparison and anonymization summary
    on the left, and a **column histograms** panel on the right showing every
    column's source-vs-sample distribution (numeric columns share bin edges;
@@ -99,6 +100,11 @@ data-sampler <source> <count> [options]
 | `--anon COL=KIND[:k=v,...]` | Anonymize a column (repeatable) |
 | `-i`, `--interactive` | Guided workflow: choose an anonymizer type per column from a menu |
 | `--suggest` | Auto-assign a suggested anonymizer type to each column from its stats |
+| `--reduce-components N` | PCA column reduction: replace the numeric columns with the first `N` principal components |
+| `--reduce-variance R` | PCA column reduction: keep the fewest components whose cumulative explained variance reaches `R` (0 < R < 1) |
+| `--reduce-exclude COL[,COL]` | Numeric column(s) to keep out of the reduction, e.g. identifiers (repeatable) |
+| `--reduce-prefix PREFIX` | Name prefix for the component columns (default `PC` → `PC1`, `PC2`, …) |
+| `--reduce-no-standardize` | Skip the per-column z-scoring before PCA |
 | `--engine {auto,pandas,duckdb}` | Sampling engine (default `auto`: DuckDB for Parquet/large inputs, pandas otherwise) |
 | `--threads N` | DuckDB engine: number of threads (default: all cores) |
 | `--memory-limit SIZE` | DuckDB engine: memory limit before spilling to disk (e.g. `8GB`) |
@@ -117,6 +123,11 @@ data-sampler data.csv 100 --skip region,notes --seed 7 \
 
 # large / out-of-core: sample a Parquet file in parallel with DuckDB
 data-sampler huge.parquet 10000 --engine duckdb --threads 8 --memory-limit 8GB --suggest
+
+# narrow the sample too: collapse the numeric columns into 3 principal
+# components (or keep however many retain 90% of the variance)
+data-sampler wide.csv 500 --reduce-components 3 --reduce-exclude cust_id
+data-sampler wide.csv 500 --reduce-variance 0.9
 ```
 
 ## Python API
@@ -149,7 +160,11 @@ anon = ds.anonymize(
     seed=7,
 )
 
-ds.save_output(anon, "data.xlsx", tag="sample_500_anon")
+# optionally collapse the numeric columns into principal components
+red = ds.reduce_columns(anon, variance_ratio=0.9, exclude=["cust_id"])
+print(ds.format_reduction_report(red))   # variance kept + correlated groups
+
+ds.save_output(red.data, "data.xlsx", tag="sample_500_anon_pca")
 ```
 
 ## Try it: bundled example
@@ -317,6 +332,34 @@ category. A side-by-side distribution report is produced for every run.
 If no suitable stratification columns exist, the tool falls back to pure
 random sampling automatically.
 
+### Reducing columns (PCA)
+
+Sampling narrows the *rows*; the optional PCA step narrows the *columns* of
+the outgoing sample. It replaces the numeric block with its first *k*
+principal components (`PC1..PCk`), controlled one of two ways:
+
+- **`--reduce-components N`** — `N` components in the output (capped at the
+  number of usable numeric columns, with a note when fewer are possible);
+- **`--reduce-variance R`** — the fewest components whose cumulative
+  explained-variance ratio reaches `R` (e.g. `0.9` keeps ≥ 90 % of the
+  numeric variance).
+
+Non-numeric columns (ids, categories, text, booleans, datetimes) are always
+preserved, and `--reduce-exclude` keeps chosen numeric columns out too —
+identifiers should be excluded, since an all-unique id forms its own
+artificial component (the tool warns when it spots one). Missing values are
+mean-imputed so the row count never changes; constant columns carry no signal
+and pass through unchanged. Columns are z-scored first by default (PCA on the
+correlation matrix), so a large-unit column such as a salary cannot dominate
+the components; `--reduce-no-standardize` turns that off.
+
+Every reduction prints its rationale: the variance each component retains,
+**the groups of correlated columns** that move together (which is exactly the
+redundancy PCA collapses — on standardized data PCA diagonalizes the
+correlation matrix), and each component's top driving columns. The reduction
+runs after anonymization, on the already-sampled rows, so the stratification
+and histogram reports still describe the original columns.
+
 ## Large data: the out-of-core DuckDB engine
 
 The default pandas path loads the whole file into memory. For inputs that are
@@ -438,6 +481,11 @@ columns, in one place.
   (~1.9× faster stats), report histograms skip near-unique columns instead
   of hashing millions of ids for a meaningless top-8, and the TUI computes
   stats in a worker thread so the UI never freezes on load.
+- **PCA column reduction on the way out.** `--reduce-components` /
+  `--reduce-variance` collapse a wide numeric block into a handful of
+  principal components *after* sampling, so the SVD runs on the small sampled
+  frame (never the full source) and the delivered file is narrow as well as
+  short.
 
 ### Correctness under scale
 
@@ -465,7 +513,8 @@ multi-threading for reproducibility (unseeded runs use all cores).
 
 Output keeps the source format and is named
 `{stem}_sample_{count}{ext}` — with an `_anon` suffix when anonymization ran
-(e.g. `data_sample_500_anon.csv`).
+and a `_pca{k}` suffix when PCA column reduction ran
+(e.g. `data_sample_500_anon_pca3.csv`).
 
 ## Development
 

{data_sampler-3.3.1 → data_sampler-3.4.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "data-sampler"
-version = "3.3.1"
+version = "3.4.0"
 description = "Hand someone data that looks and behaves like your production data, isn't your production data, and provably kept its statistical variety — stratified sampling + consistent anonymization, with a terminal UI."
 readme = "README.md"
 license = { file = "LICENSE" }

{data_sampler-3.3.1 → data_sampler-3.4.0}/src/data_sampler/__init__.py

@@ -9,20 +9,24 @@ Public API::
     result = ds.sample(df, 500, exclude_columns=["notes"])
     print(ds.format_stratification_report(df, result))
     anon = ds.anonymize(result.data, {"name": "names", "id": ("sequential_id", {"start": 1000})})
-    ds.save_output(anon, "data.csv", tag="sample_500_anon")
+    red = ds.reduce_columns(anon, variance_ratio=0.9, exclude=["id"])  # optional PCA
+    print(ds.format_reduction_report(red))
+    ds.save_output(red.data, "data.csv", tag="sample_500_anon_pca")
 
     ds.run_tui()                                 # launch the terminal UI
 """
 
 from __future__ import annotations
 
-__version__ = "3.3.1"
+__version__ = "3.4.0"
 
 from .io import SUPPORTED_EXTENSIONS, list_sheets, load_file, save_output
+from .reduce import ReductionResult, reduce_columns
 from .report import (
     column_histogram_data,
     format_column_histograms,
     format_distribution,
+    format_reduction_report,
     format_stratification_report,
 )
 from .sampling import (
@@ -47,7 +51,10 @@ __all__ = [
     "sample",
     "stratified_sample",
     "find_stratification_columns",
+    "ReductionResult",
+    "reduce_columns",
     "format_stratification_report",
+    "format_reduction_report",
     "format_distribution",
     "column_histogram_data",
     "format_column_histograms",

{data_sampler-3.3.1 → data_sampler-3.4.0}/src/data_sampler/cli.py

@@ -13,6 +13,12 @@ which picks DuckDB for Parquet/large files) samples out-of-core in parallel:
 
     data-sampler huge.parquet 10000 --engine duckdb --threads 8 \\
         --memory-limit 8GB --suggest
+
+Optionally reduce the numeric columns of the outgoing sample to K principal
+components — or to however many retain a target share of the variance:
+
+    data-sampler data.csv 500 --reduce-components 3 --reduce-exclude id
+    data-sampler data.csv 500 --reduce-variance 0.9
 """
 
 from __future__ import annotations
@@ -24,7 +30,12 @@ from . import __version__
 from ._logging import get_logger
 from .anonymize import anonymize, make_anonymizer
 from .io import load_file, save_output
-from .report import format_column_histograms, format_stratification_report
+from .reduce import reduce_columns
+from .report import (
+    format_column_histograms,
+    format_reduction_report,
+    format_stratification_report,
+)
 from .sampling import sample
 
 log = get_logger(__name__)
@@ -41,6 +52,24 @@ def _coerce_value(value: str):
     return value
 
 
+def _positive_int(text: str) -> int:
+    """argparse type: an integer >= 1 (rejects bad values at parse time)."""
+    value = int(text)
+    if value < 1:
+        raise argparse.ArgumentTypeError(f"must be a positive integer, got {text}")
+    return value
+
+
+def _unit_interval(text: str) -> float:
+    """argparse type: a float strictly between 0 and 1."""
+    value = float(text)
+    if not 0.0 < value < 1.0:
+        raise argparse.ArgumentTypeError(
+            f"must be strictly between 0 and 1, got {text}"
+        )
+    return value
+
+
 def parse_anon_option(text: str) -> tuple[str, str, dict]:
     """Parse ``COL=KIND[:key=value,key=value]`` into (column, kind, options)."""
     col, sep, rhs = text.partition("=")
@@ -110,6 +139,46 @@ def build_parser() -> argparse.ArgumentParser:
             "stats (columns set via --anon are left as given)"
         ),
     )
+    reduce_group = parser.add_mutually_exclusive_group()
+    reduce_group.add_argument(
+        "--reduce-components", type=_positive_int, metavar="N",
+        dest="reduce_components",
+        help=(
+            "PCA column reduction: replace the numeric columns with the first "
+            "N principal components (non-numeric columns are preserved; N is "
+            "capped at the number of usable numeric columns)"
+        ),
+    )
+    reduce_group.add_argument(
+        "--reduce-variance", type=_unit_interval, metavar="R",
+        dest="reduce_variance",
+        help=(
+            "PCA column reduction: keep the fewest principal components whose "
+            "cumulative explained-variance ratio reaches R (0 < R < 1)"
+        ),
+    )
+    parser.add_argument(
+        "--reduce-exclude", action="append", default=[], metavar="COL[,COL...]",
+        dest="reduce_exclude",
+        help=(
+            "numeric column(s) to exclude from PCA reduction and pass through "
+            "unchanged, e.g. identifiers (repeatable)"
+        ),
+    )
+    parser.add_argument(
+        "--reduce-prefix", default="PC", metavar="PREFIX", dest="reduce_prefix",
+        help=(
+            "name prefix for the principal-component columns (default PC → "
+            "PC1, PC2, …); change it if the source already has such a column"
+        ),
+    )
+    parser.add_argument(
+        "--reduce-no-standardize", action="store_true", dest="reduce_no_standardize",
+        help=(
+            "PCA column reduction: skip the per-column z-scoring (by default "
+            "columns are standardized so large-unit columns cannot dominate)"
+        ),
+    )
     parser.add_argument(
         "--engine", choices=("auto", "pandas", "duckdb"), default="auto",
         help=(
@@ -136,6 +205,65 @@ def build_parser() -> argparse.ArgumentParser:
     return parser
 
 
+def _reduce_requested(args) -> bool:
+    return args.reduce_components is not None or args.reduce_variance is not None
+
+
+def _validate_reduce_args(args, parser, columns) -> list[str]:
+    """Validate the reduce flags as soon as the source's columns are known —
+    BEFORE sampling, so a typo never costs a full (possibly out-of-core) run.
+
+    Returns the parsed ``--reduce-exclude`` column list.
+    """
+    exclude = [
+        c.strip() for chunk in args.reduce_exclude for c in chunk.split(",") if c.strip()
+    ]
+    if not _reduce_requested(args):
+        if exclude:
+            parser.error(
+                "--reduce-exclude requires --reduce-components or --reduce-variance"
+            )
+        if args.reduce_no_standardize:
+            parser.error(
+                "--reduce-no-standardize requires --reduce-components or "
+                "--reduce-variance"
+            )
+        return exclude
+    # compare stringified: Excel sources can carry non-string column labels
+    names = {str(c) for c in columns}
+    unknown = [c for c in exclude if c not in names]
+    if unknown:
+        parser.error(
+            f"--reduce-exclude: column(s) not found in file: {', '.join(unknown)}"
+        )
+    return exclude
+
+
+def _apply_reduction(args, parser, data, exclude):
+    """Run the optional PCA column reduction on the outgoing sample.
+
+    Returns ``(data, tag_suffix)`` — unchanged when no reduction was requested.
+    """
+    if not _reduce_requested(args):
+        return data, ""
+    try:
+        reduction = reduce_columns(
+            data,
+            n_components=args.reduce_components,
+            variance_ratio=args.reduce_variance,
+            exclude=exclude,
+            standardize=not args.reduce_no_standardize,
+            prefix=args.reduce_prefix,
+            seed=args.seed,
+        )
+    except ValueError as exc:
+        parser.error(f"column reduction: {exc}")
+    print()
+    print(format_reduction_report(reduction))
+    suffix = f"_pca{reduction.n_components}" if reduction.n_components else ""
+    return reduction.data, suffix
+
+
 def _build_engine_spec(args, parser, engine, cols) -> dict:
     """Build the anonymizer spec for the engine path (validates against ``cols``,
     supports explicit --anon and --suggest via the engine's approximate stats)."""
@@ -189,6 +317,7 @@ def _run_with_engine(args, parser, skip) -> int:
             parser.error(
                 f"--skip: column(s) not found in file: {', '.join(unknown_skip)}"
             )
+        reduce_exclude = _validate_reduce_args(args, parser, cols)
         spec = _build_engine_spec(args, parser, engine, cols)
         result = engine.sample(
             args.source, args.count,
@@ -213,7 +342,8 @@ def _run_with_engine(args, parser, skip) -> int:
             parser.error(f"anonymization failed: {exc}")
         print(f"Anonymized columns: {', '.join(spec)}")
 
-    tag = f"sample_{args.count}" + ("_anon" if spec else "")
+    data, reduce_tag = _apply_reduction(args, parser, data, reduce_exclude)
+    tag = f"sample_{args.count}" + ("_anon" if spec else "") + reduce_tag
     out_path = save_output(data, args.source, tag, output_folder=args.outdir)
     print(f"\nSampled {len(data):,} rows.")
     print(f"Output saved to: {out_path}")
@@ -288,6 +418,7 @@ def main(argv: list[str] | None = None) -> int:
     unknown_skip = [c for c in skip if c not in df.columns]
     if unknown_skip:
         parser.error(f"--skip: column(s) not found in file: {', '.join(unknown_skip)}")
+    reduce_exclude = _validate_reduce_args(args, parser, df.columns)
 
     # parse explicit --anon flags into (column, kind, options) triples
     anon_specs: list[tuple[str, str, dict]] = []
@@ -352,7 +483,8 @@ def main(argv: list[str] | None = None) -> int:
             parser.error(f"anonymization failed: {exc}")
         print(f"Anonymized columns: {', '.join(spec)}")
 
-    tag = f"sample_{args.count}" + ("_anon" if spec else "")
+    data, reduce_tag = _apply_reduction(args, parser, data, reduce_exclude)
+    tag = f"sample_{args.count}" + ("_anon" if spec else "") + reduce_tag
     out_path = save_output(data, args.source, tag, output_folder=args.outdir)
     print(f"\nSampled {len(data)} rows.")
     print(f"Output saved to: {out_path}")