data-sampler 3.4.1__tar.gz → 3.5.0__tar.gz

{data_sampler-3.4.1 → data_sampler-3.5.0}/CHANGELOG.md

@@ -1,5 +1,44 @@
 # Changelog
 
+## v3.5.0 — 2026-07-24
+
+- **Names anonymizer is now gender- and ethnicity-aware.** The bundled library
+  (`_names.py`) is regrouped into real, hand-checked names keyed by
+  `<ethnicity>_<gender>` across **33 ethnic groups** (multiple Indian
+  regions/cultures, Chinese, Japanese, Korean, Vietnamese, Anglo, Italian,
+  French, German, Russian, Polish, Scandinavian, Greek, Turkish, Persian,
+  Arab-Levantine, Sudanese, Algerian, Moroccan, Ethiopian, Rwandan, Yoruba,
+  Igbo, Ghanaian-Akan, Hispanic, Brazilian, Filipino, Indonesian). Surnames are
+  grouped too — gendered where real (Russian Ivanov/Ivanova, Polish -ski/-ska)
+  and with distinctly-female additions (North-Indian Devi/Kumari) over unisex
+  bases (Kumar/Das). `NameAnonymizer(gender=…, ethnicity=…)` fixes a gender
+  (`male`/`female`/`third`/`undisclosed`) and/or an ethnic group; **third
+  gender** mixes given names within an ethnicity, **undisclosed** draws from any
+  ethnicity.
+- **Gender/ethnicity can be read from another column** and mapped:
+  `NameAnonymizer(gender_column=…, ethnicity_column=…)` with per-value
+  `gender_map` / `ethnicity_map` overrides. Values in any encoding (M/F, 0/1 per
+  ISO-5218, `male`/`female`, other languages) are auto-detected
+  (`suggest_gender_mapping` / `suggest_ethnicity_mapping`) with a manual
+  fallback. `randomize_gender=True` reassigns genders and rewrites the gender
+  column so both fields are anonymized together.
+- **Custom name libraries.** `export_names_library()` writes the current library
+  as an editable module; `load_names_library(path=…)` (or the
+  `DATA_SAMPLER_NAMES` env var) activates one for the session; and
+  `install_names_library(path=…)` installs one permanently into the package.
+- **Report screen shows a "reproduce this in Python" snippet** — the exact
+  `ds.load_file / sample / anonymize / reduce_columns / save_output` calls that
+  reproduce the run — and it's included in the saved report `.txt`. Every TUI
+  capability now maps to the public API.
+- **Remote inputs.** `load_file` (and the TUI file screen) accept `http(s)` URLs
+  (e.g. GitHub raw links). The DuckDB engine loads `httpfs` on demand so it can
+  sample remote `s3://` / `https://` **Parquet with HTTP range requests** — a
+  multi-GB remote file is sampled out-of-core without downloading it whole.
+- **TUI restyle** — btop-inspired rounded, quiet-until-focused inputs / selects
+  / switches and outlined (non-solid) buttons; the columns table's per-stat
+  `mean`/`median`/`mode`/`sd` columns; the report save clarifies it includes the
+  histograms.
+
 ## v3.4.1 — 2026-07-24
 
 - **TUI columns table now breaks the single summary cell into separate

{data_sampler-3.4.1 → data_sampler-3.5.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: data-sampler
-Version: 3.4.1
+Version: 3.5.0
 Summary: Hand someone data that looks and behaves like your production data, isn't your production data, and provably kept its statistical variety — stratified sampling + consistent anonymization, with a terminal UI.
 Project-URL: Homepage, https://github.com/aaronified/data-sampler
 Project-URL: Changelog, https://github.com/aaronified/data-sampler/blob/main/CHANGELOG.md
@@ -75,6 +75,10 @@ Use it to:
   trigger it.
 - **Cut a 2 GB export down to 500 rows** for a demo or prototype that still
   behaves like the real thing.
+- **Narrow a wide table, not just shorten it** — optionally collapse the
+  numeric columns into a handful of principal components (PCA) on the way out,
+  so the shared sample is short *and* narrow, with a report of how much
+  variance each component keeps and which columns moved together.
 - **Build test fixtures that mirror production skew** instead of uniform toy
   data.
 - **Give a class or workshop realistic data** without a data-sharing
@@ -103,6 +107,20 @@ Requires Python 3.10+. For development, clone the repo and
 
 ## Terminal UI
 
+*Columns screen — per-column stats (mean / median / mode / sd), anonymizer,
+stratify and reduce config, with multi-select for bulk edits:*
+
+![data-sampler columns screen](docs/img/tui-columns.png)
+
+*File screen — type a path or pick one from the directory browser:*
+
+![data-sampler file screen](docs/img/tui-file.png)
+
+*Report screen — stratification report on the left, source-vs-sample column
+histograms on the right:*
+
+![data-sampler report screen](docs/img/tui-report.png)
+
 ```sh
 data-sampler            # no arguments → opens the TUI
 data-sampler-tui        # explicit TUI entry point
@@ -202,6 +220,9 @@ data-sampler wide.csv 500 --reduce-variance 0.9
 import data_sampler as ds
 
 df = ds.load_file("data.xlsx", sheet="Sheet2")
+# …or straight from a URL (GitHub raw, S3, etc.); large remote Parquet can be
+# sampled out-of-core with the DuckDB engine, without downloading it whole:
+# df = ds.load_file("https://raw.githubusercontent.com/owner/repo/main/data.csv")
 
 # Data Wrangler-style column stats
 for s in ds.compute_stats(df):
@@ -226,6 +247,18 @@ anon = ds.anonymize(
     seed=7,
 )
 
+# gender- and ethnicity-aware names: fix them, or read them from other columns
+gendered = ds.anonymize(result.data, {
+    "name": ds.NameAnonymizer(gender="female", ethnicity="chinese"),
+    # or map per row from existing columns (values auto-detected + overridable):
+    # "name": ds.NameAnonymizer(gender_column="sex", ethnicity_column="origin"),
+}, seed=7)
+
+# bring your own names: export the library, edit it, load it back
+ds.export_names_library("my_names.py")          # editable copy of the current library
+ds.load_names_library(path="my_names.py")       # activate it for this session
+# ds.install_names_library("my_names.py")        # …or install permanently
+
 # optionally collapse the numeric columns into principal components
 red = ds.reduce_columns(anon, variance_ratio=0.9, exclude=["cust_id"])
 print(ds.format_reduction_report(red))   # variance kept + correlated groups
@@ -344,7 +377,7 @@ Missing values are left as missing. All anonymizers accept a seed (via
 
 | Kind | Replaces values with | Options (defaults) |
 | --- | --- | --- |
-| `names` | Realistic names from a bundled library of first, middle, and last names | `style`: `first_last`, `first_middle_last`, `last_first`, `first`, `last` |
+| `names` | Realistic names from a bundled library grouped by ethnicity + gender (33 groups) | `style`: `first_last`, `first_middle_last`, `last_first`, `first`, `last`; `gender`: `male`/`female`/`third`/`undisclosed`; `ethnicity`; or `gender_column`/`ethnicity_column` (+ `gender_map`/`ethnicity_map`, `randomize_gender`) |
 | `sequential_id` | `start`, `start+interval`, ... in order of first appearance | `start` (1), `interval` (1), `prefix` (`""`), `width` (0, zero-pads) |
 | `numeric_jitter` | A random number within ±`pct` of the original | `pct` (0.2 = ±20 %), `round_to` (decimal places) |
 | `datetime_jitter` | A date/time shifted by a random offset within ±`max_delta` | `max_delta` (`"7D"`; any `pandas.Timedelta` string), `unit` (`"s"`; jitter resolution) |
@@ -592,10 +625,11 @@ python -m build        # build the wheel + sdist into dist/
 
 Logging is controlled by `DATA_SAMPLER_LOG` (`quiet`/`info`/`verbose`) and
 `DATA_SAMPLER_LOG_FILE`. See `ROADMAP.md` for planned work and
-`TROUBLESHOOTING.md` for known failure modes.
+[`docs/TROUBLESHOOTING.md`](docs/TROUBLESHOOTING.md) for known failure modes.
 
 Releases go to [PyPI](https://pypi.org/project/data-sampler/) via a
-human-triggered, test-gated GitHub Actions workflow — see `RELEASING.md`.
+human-triggered, test-gated GitHub Actions workflow — see
+[`docs/RELEASING.md`](docs/RELEASING.md).
 
 ---
 

{data_sampler-3.4.1 → data_sampler-3.5.0}/README.md

@@ -20,6 +20,10 @@ Use it to:
   trigger it.
 - **Cut a 2 GB export down to 500 rows** for a demo or prototype that still
   behaves like the real thing.
+- **Narrow a wide table, not just shorten it** — optionally collapse the
+  numeric columns into a handful of principal components (PCA) on the way out,
+  so the shared sample is short *and* narrow, with a report of how much
+  variance each component keeps and which columns moved together.
 - **Build test fixtures that mirror production skew** instead of uniform toy
   data.
 - **Give a class or workshop realistic data** without a data-sharing
@@ -48,6 +52,20 @@ Requires Python 3.10+. For development, clone the repo and
 
 ## Terminal UI
 
+*Columns screen — per-column stats (mean / median / mode / sd), anonymizer,
+stratify and reduce config, with multi-select for bulk edits:*
+
+![data-sampler columns screen](docs/img/tui-columns.png)
+
+*File screen — type a path or pick one from the directory browser:*
+
+![data-sampler file screen](docs/img/tui-file.png)
+
+*Report screen — stratification report on the left, source-vs-sample column
+histograms on the right:*
+
+![data-sampler report screen](docs/img/tui-report.png)
+
 ```sh
 data-sampler            # no arguments → opens the TUI
 data-sampler-tui        # explicit TUI entry point
@@ -147,6 +165,9 @@ data-sampler wide.csv 500 --reduce-variance 0.9
 import data_sampler as ds
 
 df = ds.load_file("data.xlsx", sheet="Sheet2")
+# …or straight from a URL (GitHub raw, S3, etc.); large remote Parquet can be
+# sampled out-of-core with the DuckDB engine, without downloading it whole:
+# df = ds.load_file("https://raw.githubusercontent.com/owner/repo/main/data.csv")
 
 # Data Wrangler-style column stats
 for s in ds.compute_stats(df):
@@ -171,6 +192,18 @@ anon = ds.anonymize(
     seed=7,
 )
 
+# gender- and ethnicity-aware names: fix them, or read them from other columns
+gendered = ds.anonymize(result.data, {
+    "name": ds.NameAnonymizer(gender="female", ethnicity="chinese"),
+    # or map per row from existing columns (values auto-detected + overridable):
+    # "name": ds.NameAnonymizer(gender_column="sex", ethnicity_column="origin"),
+}, seed=7)
+
+# bring your own names: export the library, edit it, load it back
+ds.export_names_library("my_names.py")          # editable copy of the current library
+ds.load_names_library(path="my_names.py")       # activate it for this session
+# ds.install_names_library("my_names.py")        # …or install permanently
+
 # optionally collapse the numeric columns into principal components
 red = ds.reduce_columns(anon, variance_ratio=0.9, exclude=["cust_id"])
 print(ds.format_reduction_report(red))   # variance kept + correlated groups
@@ -289,7 +322,7 @@ Missing values are left as missing. All anonymizers accept a seed (via
 
 | Kind | Replaces values with | Options (defaults) |
 | --- | --- | --- |
-| `names` | Realistic names from a bundled library of first, middle, and last names | `style`: `first_last`, `first_middle_last`, `last_first`, `first`, `last` |
+| `names` | Realistic names from a bundled library grouped by ethnicity + gender (33 groups) | `style`: `first_last`, `first_middle_last`, `last_first`, `first`, `last`; `gender`: `male`/`female`/`third`/`undisclosed`; `ethnicity`; or `gender_column`/`ethnicity_column` (+ `gender_map`/`ethnicity_map`, `randomize_gender`) |
 | `sequential_id` | `start`, `start+interval`, ... in order of first appearance | `start` (1), `interval` (1), `prefix` (`""`), `width` (0, zero-pads) |
 | `numeric_jitter` | A random number within ±`pct` of the original | `pct` (0.2 = ±20 %), `round_to` (decimal places) |
 | `datetime_jitter` | A date/time shifted by a random offset within ±`max_delta` | `max_delta` (`"7D"`; any `pandas.Timedelta` string), `unit` (`"s"`; jitter resolution) |
@@ -537,10 +570,11 @@ python -m build        # build the wheel + sdist into dist/
 
 Logging is controlled by `DATA_SAMPLER_LOG` (`quiet`/`info`/`verbose`) and
 `DATA_SAMPLER_LOG_FILE`. See `ROADMAP.md` for planned work and
-`TROUBLESHOOTING.md` for known failure modes.
+[`docs/TROUBLESHOOTING.md`](docs/TROUBLESHOOTING.md) for known failure modes.
 
 Releases go to [PyPI](https://pypi.org/project/data-sampler/) via a
-human-triggered, test-gated GitHub Actions workflow — see `RELEASING.md`.
+human-triggered, test-gated GitHub Actions workflow — see
+[`docs/RELEASING.md`](docs/RELEASING.md).
 
 ---
 

{data_sampler-3.4.1 → data_sampler-3.5.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "data-sampler"
-version = "3.4.1"
+version = "3.5.0"
 description = "Hand someone data that looks and behaves like your production data, isn't your production data, and provably kept its statistical variety — stratified sampling + consistent anonymization, with a terminal UI."
 readme = "README.md"
 license = { file = "LICENSE" }

{data_sampler-3.4.1 → data_sampler-3.5.0}/src/data_sampler/__init__.py

@@ -18,7 +18,7 @@ Public API::
 
 from __future__ import annotations
 
-__version__ = "3.4.1"
+__version__ = "3.5.0"
 
 from .io import SUPPORTED_EXTENSIONS, list_sheets, load_file, save_output
 from .reduce import ReductionResult, reduce_columns
@@ -60,9 +60,16 @@ __all__ = [
     "format_column_histograms",
     "anonymize",
     "make_anonymizer",
+    "NameAnonymizer",
+    "suggest_gender_mapping",
+    "suggest_ethnicity_mapping",
     "AnonymizationPlan",
     "suggest_type",
     "TYPE_OPTIONS",
+    "export_names_library",
+    "load_names_library",
+    "install_names_library",
+    "name_ethnicities",
     "run_tui",
 ]
 
@@ -70,13 +77,30 @@ __all__ = [
 def __getattr__(name: str):
     # Lazy imports: keep pandas-only workflows fast and let the core API work
     # even while optional pieces (TUI) are not yet importable.
-    if name in ("anonymize", "make_anonymizer"):
+    if name in (
+        "anonymize", "make_anonymizer", "NameAnonymizer",
+        "suggest_gender_mapping", "suggest_ethnicity_mapping",
+    ):
         # importlib, not `from . import anonymize`: the fromlist form calls
         # hasattr(package, "anonymize"), which re-enters this __getattr__.
         import importlib
 
         _anon = importlib.import_module(".anonymize", __name__)
         return getattr(_anon, name)
+    if name in (
+        "export_names_library", "load_names_library",
+        "install_names_library", "name_ethnicities",
+    ):
+        import importlib
+
+        _n = importlib.import_module("._names", __name__)
+        _alias = {
+            "export_names_library": "export_library",
+            "load_names_library": "load_library",
+            "install_names_library": "install_library",
+            "name_ethnicities": "ETHNICITIES",
+        }[name]
+        return getattr(_n, _alias)
     if name in ("AnonymizationPlan", "suggest_type", "TYPE_OPTIONS"):
         import importlib