data-sampler 3.4.0__tar.gz → 3.5.0__tar.gz

{data_sampler-3.4.0 → data_sampler-3.5.0}/CHANGELOG.md

@@ -1,5 +1,71 @@
 # Changelog
 
+## v3.5.0 — 2026-07-24
+
+- **Names anonymizer is now gender- and ethnicity-aware.** The bundled library
+  (`_names.py`) is regrouped into real, hand-checked names keyed by
+  `<ethnicity>_<gender>` across **33 ethnic groups** (multiple Indian
+  regions/cultures, Chinese, Japanese, Korean, Vietnamese, Anglo, Italian,
+  French, German, Russian, Polish, Scandinavian, Greek, Turkish, Persian,
+  Arab-Levantine, Sudanese, Algerian, Moroccan, Ethiopian, Rwandan, Yoruba,
+  Igbo, Ghanaian-Akan, Hispanic, Brazilian, Filipino, Indonesian). Surnames are
+  grouped too — gendered where real (Russian Ivanov/Ivanova, Polish -ski/-ska)
+  and with distinctly-female additions (North-Indian Devi/Kumari) over unisex
+  bases (Kumar/Das). `NameAnonymizer(gender=…, ethnicity=…)` fixes a gender
+  (`male`/`female`/`third`/`undisclosed`) and/or an ethnic group; **third
+  gender** mixes given names within an ethnicity, **undisclosed** draws from any
+  ethnicity.
+- **Gender/ethnicity can be read from another column** and mapped:
+  `NameAnonymizer(gender_column=…, ethnicity_column=…)` with per-value
+  `gender_map` / `ethnicity_map` overrides. Values in any encoding (M/F, 0/1 per
+  ISO-5218, `male`/`female`, other languages) are auto-detected
+  (`suggest_gender_mapping` / `suggest_ethnicity_mapping`) with a manual
+  fallback. `randomize_gender=True` reassigns genders and rewrites the gender
+  column so both fields are anonymized together.
+- **Custom name libraries.** `export_names_library()` writes the current library
+  as an editable module; `load_names_library(path=…)` (or the
+  `DATA_SAMPLER_NAMES` env var) activates one for the session; and
+  `install_names_library(path=…)` installs one permanently into the package.
+- **Report screen shows a "reproduce this in Python" snippet** — the exact
+  `ds.load_file / sample / anonymize / reduce_columns / save_output` calls that
+  reproduce the run — and it's included in the saved report `.txt`. Every TUI
+  capability now maps to the public API.
+- **Remote inputs.** `load_file` (and the TUI file screen) accept `http(s)` URLs
+  (e.g. GitHub raw links). The DuckDB engine loads `httpfs` on demand so it can
+  sample remote `s3://` / `https://` **Parquet with HTTP range requests** — a
+  multi-GB remote file is sampled out-of-core without downloading it whole.
+- **TUI restyle** — btop-inspired rounded, quiet-until-focused inputs / selects
+  / switches and outlined (non-solid) buttons; the columns table's per-stat
+  `mean`/`median`/`mode`/`sd` columns; the report save clarifies it includes the
+  histograms.
+
+## v3.4.1 — 2026-07-24
+
+- **TUI columns table now breaks the single summary cell into separate
+  `mean` / `median` / `mode` / `sd` columns.** Numeric columns fill all four;
+  non-numeric columns show `mode` (their most frequent value) with dashes for
+  the numeric-only stats. `ColumnStats` gained a `mode` field, populated for
+  every column kind by `compute_column_stats`.
+- **Bulk configuration via multi-select on the columns screen.** Ctrl-click
+  toggles individual rows, shift-click selects a contiguous range (keyboard:
+  `space` toggles the cursor row, `x` clears the selection). Any anonymizer
+  choice, stratification-skip, or reduction-skip then applies to every
+  selected column at once; the panel title shows the selection count.
+- **Columns can now be excluded from the PCA reduction from the TUI.** A new
+  “skip from reduction (PCA)” switch (and the `d` key) sets a per-column
+  `skip_reduce` flag that is passed to `reduce_columns(exclude=…)`, so chosen
+  numeric columns pass through the reduction unchanged. The columns table
+  gained a `reduce` column showing each column's candidacy / skip state, and
+  excluded columns are listed in the run report.
+- **Undo / redo for column configuration** (`ctrl+z` / `ctrl+y`), keeping well
+  over the last ten steps. Bursts of edits to one option field coalesce into a
+  single undo step.
+- **File screen: refresh the directory browser** with `ctrl+r` or the new
+  ⟳ refresh button, so files created after launch appear without restarting.
+- **Report screen: save the report + column histograms to a text file** with
+  `ctrl+s` or the 💾 save .txt button; it writes `<sample-name>_report.txt`
+  next to the sample output.
+
 ## v3.4.0 — 2026-07-24
 
 - **Added optional PCA column reduction** (`data_sampler.reduce`): after

{data_sampler-3.4.0 → data_sampler-3.5.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: data-sampler
-Version: 3.4.0
+Version: 3.5.0
 Summary: Hand someone data that looks and behaves like your production data, isn't your production data, and provably kept its statistical variety — stratified sampling + consistent anonymization, with a terminal UI.
 Project-URL: Homepage, https://github.com/aaronified/data-sampler
 Project-URL: Changelog, https://github.com/aaronified/data-sampler/blob/main/CHANGELOG.md
@@ -75,6 +75,10 @@ Use it to:
   trigger it.
 - **Cut a 2 GB export down to 500 rows** for a demo or prototype that still
   behaves like the real thing.
+- **Narrow a wide table, not just shorten it** — optionally collapse the
+  numeric columns into a handful of principal components (PCA) on the way out,
+  so the shared sample is short *and* narrow, with a report of how much
+  variance each component keeps and which columns moved together.
 - **Build test fixtures that mirror production skew** instead of uniform toy
   data.
 - **Give a class or workshop realistic data** without a data-sharing
@@ -103,6 +107,20 @@ Requires Python 3.10+. For development, clone the repo and
 
 ## Terminal UI
 
+*Columns screen — per-column stats (mean / median / mode / sd), anonymizer,
+stratify and reduce config, with multi-select for bulk edits:*
+
+![data-sampler columns screen](docs/img/tui-columns.png)
+
+*File screen — type a path or pick one from the directory browser:*
+
+![data-sampler file screen](docs/img/tui-file.png)
+
+*Report screen — stratification report on the left, source-vs-sample column
+histograms on the right:*
+
+![data-sampler report screen](docs/img/tui-report.png)
+
 ```sh
 data-sampler            # no arguments → opens the TUI
 data-sampler-tui        # explicit TUI entry point
@@ -120,24 +138,35 @@ data_sampler.run_tui("data.csv")       # pre-load a file
 
 The TUI is a panel-based dashboard (think btop / lazydocker):
 
-1. **File screen** — type a path or pick a file from the directory browser;
-   Excel files take an optional sheet name.
+1. **File screen** — type a path or pick a file from the directory browser
+   (**`ctrl+r`** or the **⟳ refresh** button re-scans the folder so files
+   created since launch show up); Excel files take an optional sheet name.
 2. **Columns screen** — every column with its type, missing %, unique count,
-   distribution sparkline, and summary (modelled after the Data Wrangler
-   VS Code extension). Select a column to see full stats and distribution
-   bars, choose an anonymizer for it, and toggle whether it should be
-   skipped when preserving statistical variety (stratification). Set the
-   sample size, output folder, optional seed, an optional PCA column
-   reduction (`reduce`: N components or a variance target), and run.
+   distribution sparkline, and per-stat **mean / median / mode / sd** columns
+   (modelled after the Data Wrangler VS Code extension). Select a column to
+   see full stats and distribution bars, choose an anonymizer for it, and
+   toggle whether it should be skipped when preserving statistical variety
+   (stratification) or excluded from the PCA reduction. **Multi-select** rows
+   to configure them in bulk: **ctrl-click** toggles individual rows,
+   **shift-click** selects a range (**space** toggles the cursor row from the
+   keyboard, **`x`** clears) — then any anonymizer / skip / reduce choice
+   applies to every selected column at once. **`ctrl+z` / `ctrl+y`** undo and
+   redo column-config changes (at least the last ten steps). Set the sample
+   size, output folder, optional seed, an optional PCA column reduction
+   (`reduce`: N components or a variance target), and run.
 3. **Report screen** — the stratification comparison and anonymization summary
    on the left, and a **column histograms** panel on the right showing every
    column's source-vs-sample distribution (numeric columns share bin edges;
    others use the source's top categories) so you can see at a glance how well
-   the sample preserved each column. The output path is shown too.
+   the sample preserved each column. The output path is shown too, and
+   **`ctrl+s`** (or **💾 save .txt**) writes the report and histograms to a
+   `*_report.txt` beside the sample.
 
-Key bindings: `ctrl+r` run sample, `a` auto-suggest anonymizer types,
-`s` toggle stratification skip,
-`escape` back, `ctrl+q` quit.
+Key bindings — columns screen: `ctrl+r` run sample, `a` auto-suggest
+anonymizer types, `s` toggle stratification skip, `d` toggle reduction skip,
+`space` select row, `x` clear selection, `ctrl+z`/`ctrl+y` undo/redo,
+`escape` back. Report screen: `ctrl+s` save report to text, `n` new file.
+Everywhere: `ctrl+q` quit.
 
 ## CLI (headless)
 
@@ -191,6 +220,9 @@ data-sampler wide.csv 500 --reduce-variance 0.9
 import data_sampler as ds
 
 df = ds.load_file("data.xlsx", sheet="Sheet2")
+# …or straight from a URL (GitHub raw, S3, etc.); large remote Parquet can be
+# sampled out-of-core with the DuckDB engine, without downloading it whole:
+# df = ds.load_file("https://raw.githubusercontent.com/owner/repo/main/data.csv")
 
 # Data Wrangler-style column stats
 for s in ds.compute_stats(df):
@@ -215,6 +247,18 @@ anon = ds.anonymize(
     seed=7,
 )
 
+# gender- and ethnicity-aware names: fix them, or read them from other columns
+gendered = ds.anonymize(result.data, {
+    "name": ds.NameAnonymizer(gender="female", ethnicity="chinese"),
+    # or map per row from existing columns (values auto-detected + overridable):
+    # "name": ds.NameAnonymizer(gender_column="sex", ethnicity_column="origin"),
+}, seed=7)
+
+# bring your own names: export the library, edit it, load it back
+ds.export_names_library("my_names.py")          # editable copy of the current library
+ds.load_names_library(path="my_names.py")       # activate it for this session
+# ds.install_names_library("my_names.py")        # …or install permanently
+
 # optionally collapse the numeric columns into principal components
 red = ds.reduce_columns(anon, variance_ratio=0.9, exclude=["cust_id"])
 print(ds.format_reduction_report(red))   # variance kept + correlated groups
@@ -333,7 +377,7 @@ Missing values are left as missing. All anonymizers accept a seed (via
 
 | Kind | Replaces values with | Options (defaults) |
 | --- | --- | --- |
-| `names` | Realistic names from a bundled library of first, middle, and last names | `style`: `first_last`, `first_middle_last`, `last_first`, `first`, `last` |
+| `names` | Realistic names from a bundled library grouped by ethnicity + gender (33 groups) | `style`: `first_last`, `first_middle_last`, `last_first`, `first`, `last`; `gender`: `male`/`female`/`third`/`undisclosed`; `ethnicity`; or `gender_column`/`ethnicity_column` (+ `gender_map`/`ethnicity_map`, `randomize_gender`) |
 | `sequential_id` | `start`, `start+interval`, ... in order of first appearance | `start` (1), `interval` (1), `prefix` (`""`), `width` (0, zero-pads) |
 | `numeric_jitter` | A random number within ±`pct` of the original | `pct` (0.2 = ±20 %), `round_to` (decimal places) |
 | `datetime_jitter` | A date/time shifted by a random offset within ±`max_delta` | `max_delta` (`"7D"`; any `pandas.Timedelta` string), `unit` (`"s"`; jitter resolution) |
@@ -581,10 +625,11 @@ python -m build        # build the wheel + sdist into dist/
 
 Logging is controlled by `DATA_SAMPLER_LOG` (`quiet`/`info`/`verbose`) and
 `DATA_SAMPLER_LOG_FILE`. See `ROADMAP.md` for planned work and
-`TROUBLESHOOTING.md` for known failure modes.
+[`docs/TROUBLESHOOTING.md`](docs/TROUBLESHOOTING.md) for known failure modes.
 
 Releases go to [PyPI](https://pypi.org/project/data-sampler/) via a
-human-triggered, test-gated GitHub Actions workflow — see `RELEASING.md`.
+human-triggered, test-gated GitHub Actions workflow — see
+[`docs/RELEASING.md`](docs/RELEASING.md).
 
 ---
 

{data_sampler-3.4.0 → data_sampler-3.5.0}/README.md

@@ -20,6 +20,10 @@ Use it to:
   trigger it.
 - **Cut a 2 GB export down to 500 rows** for a demo or prototype that still
   behaves like the real thing.
+- **Narrow a wide table, not just shorten it** — optionally collapse the
+  numeric columns into a handful of principal components (PCA) on the way out,
+  so the shared sample is short *and* narrow, with a report of how much
+  variance each component keeps and which columns moved together.
 - **Build test fixtures that mirror production skew** instead of uniform toy
   data.
 - **Give a class or workshop realistic data** without a data-sharing
@@ -48,6 +52,20 @@ Requires Python 3.10+. For development, clone the repo and
 
 ## Terminal UI
 
+*Columns screen — per-column stats (mean / median / mode / sd), anonymizer,
+stratify and reduce config, with multi-select for bulk edits:*
+
+![data-sampler columns screen](docs/img/tui-columns.png)
+
+*File screen — type a path or pick one from the directory browser:*
+
+![data-sampler file screen](docs/img/tui-file.png)
+
+*Report screen — stratification report on the left, source-vs-sample column
+histograms on the right:*
+
+![data-sampler report screen](docs/img/tui-report.png)
+
 ```sh
 data-sampler            # no arguments → opens the TUI
 data-sampler-tui        # explicit TUI entry point
@@ -65,24 +83,35 @@ data_sampler.run_tui("data.csv")       # pre-load a file
 
 The TUI is a panel-based dashboard (think btop / lazydocker):
 
-1. **File screen** — type a path or pick a file from the directory browser;
-   Excel files take an optional sheet name.
+1. **File screen** — type a path or pick a file from the directory browser
+   (**`ctrl+r`** or the **⟳ refresh** button re-scans the folder so files
+   created since launch show up); Excel files take an optional sheet name.
 2. **Columns screen** — every column with its type, missing %, unique count,
-   distribution sparkline, and summary (modelled after the Data Wrangler
-   VS Code extension). Select a column to see full stats and distribution
-   bars, choose an anonymizer for it, and toggle whether it should be
-   skipped when preserving statistical variety (stratification). Set the
-   sample size, output folder, optional seed, an optional PCA column
-   reduction (`reduce`: N components or a variance target), and run.
+   distribution sparkline, and per-stat **mean / median / mode / sd** columns
+   (modelled after the Data Wrangler VS Code extension). Select a column to
+   see full stats and distribution bars, choose an anonymizer for it, and
+   toggle whether it should be skipped when preserving statistical variety
+   (stratification) or excluded from the PCA reduction. **Multi-select** rows
+   to configure them in bulk: **ctrl-click** toggles individual rows,
+   **shift-click** selects a range (**space** toggles the cursor row from the
+   keyboard, **`x`** clears) — then any anonymizer / skip / reduce choice
+   applies to every selected column at once. **`ctrl+z` / `ctrl+y`** undo and
+   redo column-config changes (at least the last ten steps). Set the sample
+   size, output folder, optional seed, an optional PCA column reduction
+   (`reduce`: N components or a variance target), and run.
 3. **Report screen** — the stratification comparison and anonymization summary
    on the left, and a **column histograms** panel on the right showing every
    column's source-vs-sample distribution (numeric columns share bin edges;
    others use the source's top categories) so you can see at a glance how well
-   the sample preserved each column. The output path is shown too.
+   the sample preserved each column. The output path is shown too, and
+   **`ctrl+s`** (or **💾 save .txt**) writes the report and histograms to a
+   `*_report.txt` beside the sample.
 
-Key bindings: `ctrl+r` run sample, `a` auto-suggest anonymizer types,
-`s` toggle stratification skip,
-`escape` back, `ctrl+q` quit.
+Key bindings — columns screen: `ctrl+r` run sample, `a` auto-suggest
+anonymizer types, `s` toggle stratification skip, `d` toggle reduction skip,
+`space` select row, `x` clear selection, `ctrl+z`/`ctrl+y` undo/redo,
+`escape` back. Report screen: `ctrl+s` save report to text, `n` new file.
+Everywhere: `ctrl+q` quit.
 
 ## CLI (headless)
 
@@ -136,6 +165,9 @@ data-sampler wide.csv 500 --reduce-variance 0.9
 import data_sampler as ds
 
 df = ds.load_file("data.xlsx", sheet="Sheet2")
+# …or straight from a URL (GitHub raw, S3, etc.); large remote Parquet can be
+# sampled out-of-core with the DuckDB engine, without downloading it whole:
+# df = ds.load_file("https://raw.githubusercontent.com/owner/repo/main/data.csv")
 
 # Data Wrangler-style column stats
 for s in ds.compute_stats(df):
@@ -160,6 +192,18 @@ anon = ds.anonymize(
     seed=7,
 )
 
+# gender- and ethnicity-aware names: fix them, or read them from other columns
+gendered = ds.anonymize(result.data, {
+    "name": ds.NameAnonymizer(gender="female", ethnicity="chinese"),
+    # or map per row from existing columns (values auto-detected + overridable):
+    # "name": ds.NameAnonymizer(gender_column="sex", ethnicity_column="origin"),
+}, seed=7)
+
+# bring your own names: export the library, edit it, load it back
+ds.export_names_library("my_names.py")          # editable copy of the current library
+ds.load_names_library(path="my_names.py")       # activate it for this session
+# ds.install_names_library("my_names.py")        # …or install permanently
+
 # optionally collapse the numeric columns into principal components
 red = ds.reduce_columns(anon, variance_ratio=0.9, exclude=["cust_id"])
 print(ds.format_reduction_report(red))   # variance kept + correlated groups
@@ -278,7 +322,7 @@ Missing values are left as missing. All anonymizers accept a seed (via
 
 | Kind | Replaces values with | Options (defaults) |
 | --- | --- | --- |
-| `names` | Realistic names from a bundled library of first, middle, and last names | `style`: `first_last`, `first_middle_last`, `last_first`, `first`, `last` |
+| `names` | Realistic names from a bundled library grouped by ethnicity + gender (33 groups) | `style`: `first_last`, `first_middle_last`, `last_first`, `first`, `last`; `gender`: `male`/`female`/`third`/`undisclosed`; `ethnicity`; or `gender_column`/`ethnicity_column` (+ `gender_map`/`ethnicity_map`, `randomize_gender`) |
 | `sequential_id` | `start`, `start+interval`, ... in order of first appearance | `start` (1), `interval` (1), `prefix` (`""`), `width` (0, zero-pads) |
 | `numeric_jitter` | A random number within ±`pct` of the original | `pct` (0.2 = ±20 %), `round_to` (decimal places) |
 | `datetime_jitter` | A date/time shifted by a random offset within ±`max_delta` | `max_delta` (`"7D"`; any `pandas.Timedelta` string), `unit` (`"s"`; jitter resolution) |
@@ -526,10 +570,11 @@ python -m build        # build the wheel + sdist into dist/
 
 Logging is controlled by `DATA_SAMPLER_LOG` (`quiet`/`info`/`verbose`) and
 `DATA_SAMPLER_LOG_FILE`. See `ROADMAP.md` for planned work and
-`TROUBLESHOOTING.md` for known failure modes.
+[`docs/TROUBLESHOOTING.md`](docs/TROUBLESHOOTING.md) for known failure modes.
 
 Releases go to [PyPI](https://pypi.org/project/data-sampler/) via a
-human-triggered, test-gated GitHub Actions workflow — see `RELEASING.md`.
+human-triggered, test-gated GitHub Actions workflow — see
+[`docs/RELEASING.md`](docs/RELEASING.md).
 
 ---
 

{data_sampler-3.4.0 → data_sampler-3.5.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "data-sampler"
-version = "3.4.0"
+version = "3.5.0"
 description = "Hand someone data that looks and behaves like your production data, isn't your production data, and provably kept its statistical variety — stratified sampling + consistent anonymization, with a terminal UI."
 readme = "README.md"
 license = { file = "LICENSE" }

{data_sampler-3.4.0 → data_sampler-3.5.0}/src/data_sampler/__init__.py

@@ -18,7 +18,7 @@ Public API::
 
 from __future__ import annotations
 
-__version__ = "3.4.0"
+__version__ = "3.5.0"
 
 from .io import SUPPORTED_EXTENSIONS, list_sheets, load_file, save_output
 from .reduce import ReductionResult, reduce_columns
@@ -60,9 +60,16 @@ __all__ = [
     "format_column_histograms",
     "anonymize",
     "make_anonymizer",
+    "NameAnonymizer",
+    "suggest_gender_mapping",
+    "suggest_ethnicity_mapping",
     "AnonymizationPlan",
     "suggest_type",
     "TYPE_OPTIONS",
+    "export_names_library",
+    "load_names_library",
+    "install_names_library",
+    "name_ethnicities",
     "run_tui",
 ]
 
@@ -70,13 +77,30 @@ __all__ = [
 def __getattr__(name: str):
     # Lazy imports: keep pandas-only workflows fast and let the core API work
     # even while optional pieces (TUI) are not yet importable.
-    if name in ("anonymize", "make_anonymizer"):
+    if name in (
+        "anonymize", "make_anonymizer", "NameAnonymizer",
+        "suggest_gender_mapping", "suggest_ethnicity_mapping",
+    ):
         # importlib, not `from . import anonymize`: the fromlist form calls
         # hasattr(package, "anonymize"), which re-enters this __getattr__.
         import importlib
 
         _anon = importlib.import_module(".anonymize", __name__)
         return getattr(_anon, name)
+    if name in (
+        "export_names_library", "load_names_library",
+        "install_names_library", "name_ethnicities",
+    ):
+        import importlib
+
+        _n = importlib.import_module("._names", __name__)
+        _alias = {
+            "export_names_library": "export_library",
+            "load_names_library": "load_library",
+            "install_names_library": "install_library",
+            "name_ethnicities": "ETHNICITIES",
+        }[name]
+        return getattr(_n, _alias)
     if name in ("AnonymizationPlan", "suggest_type", "TYPE_OPTIONS"):
         import importlib