fauxdata-cli 0.2.1__tar.gz → 0.2.2__tar.gz

{fauxdata_cli-0.2.1 → fauxdata_cli-0.2.2}/.gitignore

@@ -3,3 +3,4 @@ tmp/
 __pycache__/
 *.egg-info/
 .coverage
+/share.png

{fauxdata_cli-0.2.1 → fauxdata_cli-0.2.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: fauxdata-cli
-Version: 0.2.1
+Version: 0.2.2
 Summary: CLI for generating and validating fake datasets
 Project-URL: Homepage, https://aborruso.github.io/fauxdata/
 Project-URL: Repository, https://github.com/aborruso/fauxdata
@@ -13,7 +13,7 @@ Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
 Classifier: Programming Language :: Python :: 3.13
 Requires-Python: >=3.11
-Requires-Dist: pointblank>=1.1.0
+Requires-Dist: pointblank>=1.2.0
 Requires-Dist: polars>=1.0
 Requires-Dist: pyfiglet>=1.0
 Requires-Dist: pyyaml>=6.0
@@ -169,12 +169,12 @@ validation:
 
 | Type | Description | Options |
 |------|-------------|---------|
-| `int` | Integer | `min`, `max`, `unique` |
-| `float` | Floating point | `min`, `max`, `precision` |
+| `int` | Integer | `min`, `max`, `unique`, `values` |
+| `float` | Floating point | `min`, `max`, `precision`, `unique`, `values` |
 | `string` | Text | `preset`, `values`, `pattern`, `unique`, `min_length`, `max_length` |
 | `bool` | Boolean | — |
-| `date` | Date | `min`, `max` (ISO format) |
-| `datetime` | Datetime | `min`, `max` (ISO format) |
+| `date` | Date | `min`, `max` (ISO format), `unique` |
+| `datetime` | Datetime | `min`, `max` (ISO format), `unique` |
 
 ### Float precision
 
@@ -269,7 +269,7 @@ fauxdata generate schemas/people.yml --out - --format jsonl | wc -l
 | `--rows` | `-r` | from schema | Number of rows to generate |
 | `--out` | `-o` | from schema | Output path — use `-` for stdout |
 | `--format` | `-f` | from schema | Output format: `csv`, `parquet`, `json`, `jsonl`, `jsonlines` |
-| `--seed` | `-s` | from schema | Random seed for reproducibility |
+| `--seed` | `-s` | from schema | Random seed for reproducibility: same seed, same data, on the same pointblank version |
 | `--validate` | `-v` | off | Run validation rules after generating |
 | `--json` | | off | Print only a JSON summary to stdout (see [Using fauxdata with AI agents](#using-fauxdata-with-ai-agents)) |
 
@@ -277,7 +277,7 @@ When `--out -` is used, all output messages are suppressed and only data is writ
 
 ### `fauxdata infer DATASET`
 
-Build a YAML schema from a **real** dataset. fauxdata inspects the table (via pointblank's schema inference) and writes a schema with inferred ranges, categorical value sets, string presets (email, url…), uniqueness, null rates, and string lengths — plus matching `validation:` rules. Feed the result back to `fauxdata generate` to produce synthetic data that mirrors the real shape, without ever shipping the real values.
+Build a YAML schema from a **real** dataset. fauxdata inspects the table (via pointblank's schema inference) and writes a schema with inferred ranges, categorical value sets, string presets (email, url…), uniqueness, null rates, and string lengths — plus matching `validation:` rules. Feed the result back to `fauxdata generate` to produce synthetic data with the same columns, ranges and value sets, without copying the real rows. Each column is generated independently, so correlations between columns, rules such as `end_date > start_date`, and category frequencies are not preserved.
 
 ```
 fauxdata infer real_data.csv                                   # -> real_data.yml
@@ -296,7 +296,7 @@ fauxdata infer big.parquet --sample-size 10000 --out - | head  # stream to stdou
 | `--detect-presets` / `--no-detect-presets` | | on | Match string columns to known presets (email, url…) |
 | `--sample-size` | | all rows | Sample N rows before analysis (for very large tables) |
 
-> **Notes.** Low-cardinality columns (≤ `--categorical-threshold` distinct values) are frozen to their exact source values via `values:` — desirable for categories like `status`, but it means sensitive low-cardinality columns are reproduced verbatim. Raise the threshold or edit the schema if that's not what you want. Also, a column inferred as `unique` over a narrow integer range cannot generate many more rows than the source; widen `min`/`max` or drop `unique` to amplify.
+> **Notes.** Low-cardinality columns (≤ `--categorical-threshold` distinct values) are frozen to their exact source values via `values:` — desirable for categories like `status`, but it means sensitive low-cardinality columns are reproduced verbatim. Raise the threshold or edit the schema if that's not what you want. This also applies to columns whose name suggests a place, company or job (`region`, `city`, `company`...). Personal presets are never frozen, so a low-cardinality column such as `product_name` still gets people's names: use `--no-detect-presets` or edit the schema. Also, a column inferred as `unique` over a narrow integer range cannot generate many more rows than the source; widen `min`/`max` or drop `unique` to amplify.
 
 ### `fauxdata validate DATASET SCHEMA`
 

{fauxdata_cli-0.2.1 → fauxdata_cli-0.2.2}/README.md

@@ -145,12 +145,12 @@ validation:
 
 | Type | Description | Options |
 |------|-------------|---------|
-| `int` | Integer | `min`, `max`, `unique` |
-| `float` | Floating point | `min`, `max`, `precision` |
+| `int` | Integer | `min`, `max`, `unique`, `values` |
+| `float` | Floating point | `min`, `max`, `precision`, `unique`, `values` |
 | `string` | Text | `preset`, `values`, `pattern`, `unique`, `min_length`, `max_length` |
 | `bool` | Boolean | — |
-| `date` | Date | `min`, `max` (ISO format) |
-| `datetime` | Datetime | `min`, `max` (ISO format) |
+| `date` | Date | `min`, `max` (ISO format), `unique` |
+| `datetime` | Datetime | `min`, `max` (ISO format), `unique` |
 
 ### Float precision
 
@@ -245,7 +245,7 @@ fauxdata generate schemas/people.yml --out - --format jsonl | wc -l
 | `--rows` | `-r` | from schema | Number of rows to generate |
 | `--out` | `-o` | from schema | Output path — use `-` for stdout |
 | `--format` | `-f` | from schema | Output format: `csv`, `parquet`, `json`, `jsonl`, `jsonlines` |
-| `--seed` | `-s` | from schema | Random seed for reproducibility |
+| `--seed` | `-s` | from schema | Random seed for reproducibility: same seed, same data, on the same pointblank version |
 | `--validate` | `-v` | off | Run validation rules after generating |
 | `--json` | | off | Print only a JSON summary to stdout (see [Using fauxdata with AI agents](#using-fauxdata-with-ai-agents)) |
 
@@ -253,7 +253,7 @@ When `--out -` is used, all output messages are suppressed and only data is writ
 
 ### `fauxdata infer DATASET`
 
-Build a YAML schema from a **real** dataset. fauxdata inspects the table (via pointblank's schema inference) and writes a schema with inferred ranges, categorical value sets, string presets (email, url…), uniqueness, null rates, and string lengths — plus matching `validation:` rules. Feed the result back to `fauxdata generate` to produce synthetic data that mirrors the real shape, without ever shipping the real values.
+Build a YAML schema from a **real** dataset. fauxdata inspects the table (via pointblank's schema inference) and writes a schema with inferred ranges, categorical value sets, string presets (email, url…), uniqueness, null rates, and string lengths — plus matching `validation:` rules. Feed the result back to `fauxdata generate` to produce synthetic data with the same columns, ranges and value sets, without copying the real rows. Each column is generated independently, so correlations between columns, rules such as `end_date > start_date`, and category frequencies are not preserved.
 
 ```
 fauxdata infer real_data.csv                                   # -> real_data.yml
@@ -272,7 +272,7 @@ fauxdata infer big.parquet --sample-size 10000 --out - | head  # stream to stdou
 | `--detect-presets` / `--no-detect-presets` | | on | Match string columns to known presets (email, url…) |
 | `--sample-size` | | all rows | Sample N rows before analysis (for very large tables) |
 
-> **Notes.** Low-cardinality columns (≤ `--categorical-threshold` distinct values) are frozen to their exact source values via `values:` — desirable for categories like `status`, but it means sensitive low-cardinality columns are reproduced verbatim. Raise the threshold or edit the schema if that's not what you want. Also, a column inferred as `unique` over a narrow integer range cannot generate many more rows than the source; widen `min`/`max` or drop `unique` to amplify.
+> **Notes.** Low-cardinality columns (≤ `--categorical-threshold` distinct values) are frozen to their exact source values via `values:` — desirable for categories like `status`, but it means sensitive low-cardinality columns are reproduced verbatim. Raise the threshold or edit the schema if that's not what you want. This also applies to columns whose name suggests a place, company or job (`region`, `city`, `company`...). Personal presets are never frozen, so a low-cardinality column such as `product_name` still gets people's names: use `--no-detect-presets` or edit the schema. Also, a column inferred as `unique` over a narrow integer range cannot generate many more rows than the source; widen `min`/`max` or drop `unique` to amplify.
 
 ### `fauxdata validate DATASET SCHEMA`
 

{fauxdata_cli-0.2.1 → fauxdata_cli-0.2.2}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "fauxdata-cli"
-version = "0.2.1"
+version = "0.2.2"
 description = "CLI for generating and validating fake datasets"
 readme = "README.md"
 requires-python = ">=3.11"
@@ -13,7 +13,7 @@ classifiers = [
     "Operating System :: OS Independent",
 ]
 dependencies = [
-    "pointblank>=1.1.0",
+    "pointblank>=1.2.0",
     "typer>=0.12",
     "rich>=13",
     "pyfiglet>=1.0",

{fauxdata_cli-0.2.1 → fauxdata_cli-0.2.2}/src/fauxdata/__init__.py

@@ -1,3 +1,3 @@
 """fauxdata - CLI for generating and validating fake datasets."""
 
-__version__ = "0.2.1"
+__version__ = "0.2.2"

{fauxdata_cli-0.2.1 → fauxdata_cli-0.2.2}/src/fauxdata/commands/infer.py

@@ -31,6 +31,11 @@ _DTYPE_MAP = {
     "Date": "date",
 }
 
+# Presets pointblank matches on the column name alone, skipping categorical detection.
+# Personal presets (name, phone, address...) are left out on purpose: restoring their
+# source values would copy real people's data into the schema.
+_CATEGORICAL_PRESETS = {"state", "city", "country", "country_code_2", "currency_code", "company", "job"}
+
 
 def _col_type_from_dtype(dtype: str) -> str:
     """Map a pointblank/polars dtype name to a fauxdata col_type."""
@@ -83,6 +88,19 @@ def _field_to_spec(field) -> dict:
     return spec
 
 
+def _restore_categorical(spec: dict, series: pl.Series, threshold: int | float) -> dict:
+    """Replace a name-matched preset with the source values on a low-cardinality column."""
+    if spec.get("preset") not in _CATEGORICAL_PRESETS or spec.get("unique"):
+        return spec
+    limit = threshold if isinstance(threshold, int) else int(threshold * series.len())
+    values = series.drop_nulls().unique().sort().to_list()
+    if len(values) > limit:
+        return spec
+    del spec["preset"]
+    spec["values"] = values
+    return spec
+
+
 def _build_validation(columns: list[tuple[str, dict]]) -> list[dict]:
     """Derive validation rules from the inferred column specs."""
     rules: list[dict] = []
@@ -160,14 +178,18 @@ def run(
     if not json_out:
         rprint(f"  Loaded [bold]{len(df)}[/bold] rows, [bold]{len(df.columns)}[/bold] columns")
 
+    threshold = _threshold(categorical_threshold)
     pb_schema = pb.schema_from_tbl(
         df,
-        categorical_threshold=_threshold(categorical_threshold),
+        categorical_threshold=threshold,
         detect_presets=detect_presets,
         sample_size=sample_size,
     )
 
-    columns = [(col_name, _field_to_spec(field)) for col_name, field in pb_schema.columns]
+    columns = [
+        (col_name, _restore_categorical(_field_to_spec(field), df[col_name], threshold))
+        for col_name, field in pb_schema.columns
+    ]
 
     schema_name = name or Path(dataset_path).stem
     out_rows = rows if rows is not None else len(df)

{fauxdata_cli-0.2.1 → fauxdata_cli-0.2.2}/tests/test_infer.py

@@ -220,3 +220,70 @@ def test_infer_roundtrip_nulls_and_low_cardinality_int(tmp_path):
     df = generate_dataset(schema, rows=500, seed=3)
     all_passed, results = validate_dataset(df, schema)
     assert all_passed, [r for r in results if not r["ok"]]
+
+
+# --- name-matched presets on categorical columns ---
+
+def _infer_spec(tmp_path, column, values, **kwargs):
+    src = tmp_path / "real.csv"
+    pl.DataFrame({column: values}).write_csv(src)
+    out = tmp_path / "inferred.yml"
+    infer_mod.run(dataset_path=str(src), out=str(out), **kwargs)
+    return yaml.safe_load(out.read_text()), out
+
+
+@pytest.mark.parametrize("column", ["region", "state", "city", "country", "currency", "company", "job"])
+def test_infer_low_cardinality_keeps_source_values(tmp_path, column):
+    """A column matched to a preset only by its name stays categorical when values repeat."""
+    source = ["nord", "centro", "sud"] * 20
+    schema, out = _infer_spec(tmp_path, column, source)
+    spec = schema["columns"][column]
+    assert "preset" not in spec
+    assert sorted(spec["values"]) == ["centro", "nord", "sud"]
+
+    loaded = load_schema(out)
+    df = generate_dataset(loaded, rows=200, seed=1)
+    assert set(df[column].to_list()) <= {"nord", "centro", "sud"}
+    all_passed, results = validate_dataset(df, loaded)
+    assert all_passed, [r for r in results if not r["ok"]]
+
+
+@pytest.mark.parametrize("column,preset", [("name", "name"), ("first_name", "first_name"), ("phone", "phone_number"), ("address", "address")])
+def test_infer_personal_preset_never_frozen(tmp_path, column, preset):
+    """Personal presets are kept, so real people's values never reach the schema."""
+    schema, out = _infer_spec(tmp_path, column, ["Mario Rossi", "Anna Bianchi"] * 20)
+    assert schema["columns"][column]["preset"] == preset
+    assert "Mario Rossi" not in out.read_text()
+
+
+def test_infer_high_cardinality_keeps_preset(tmp_path):
+    schema, _ = _infer_spec(tmp_path, "city", [f"city {i % 30}" for i in range(60)])
+    assert schema["columns"]["city"]["preset"] == "city"
+
+
+def test_infer_unique_column_keeps_preset(tmp_path):
+    schema, _ = _infer_spec(tmp_path, "city", [f"city {i}" for i in range(10)])
+    spec = schema["columns"]["city"]
+    assert spec["preset"] == "city"
+    assert spec["unique"] is True
+
+
+def test_infer_fractional_threshold_applies_to_preset_columns(tmp_path):
+    values = [f"r{i % 8}" for i in range(100)]
+    assert "values" in _infer_spec(tmp_path, "region", values, categorical_threshold=0.1)[0]["columns"]["region"]
+    assert "preset" in _infer_spec(tmp_path, "region", values, categorical_threshold=0.05)[0]["columns"]["region"]
+
+
+def test_infer_no_detect_presets_unchanged(tmp_path):
+    schema, _ = _infer_spec(tmp_path, "region", ["a", "b"] * 10, detect_presets=False)
+    assert sorted(schema["columns"]["region"]["values"]) == ["a", "b"]
+
+
+def test_infer_categorical_with_nulls_keeps_nullable(tmp_path):
+    schema, out = _infer_spec(tmp_path, "region", ["nord", None, "sud", "nord"] * 10)
+    spec = schema["columns"]["region"]
+    assert sorted(spec["values"]) == ["nord", "sud"]
+    assert spec["nullable"] is True
+    loaded = load_schema(out)
+    all_passed, results = validate_dataset(generate_dataset(loaded, rows=200, seed=1), loaded)
+    assert all_passed, [r for r in results if not r["ok"]]