fauxdata-cli 0.2.0__tar.gz → 0.2.2__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (41) hide show
  1. {fauxdata_cli-0.2.0 → fauxdata_cli-0.2.2}/.gitignore +1 -0
  2. {fauxdata_cli-0.2.0 → fauxdata_cli-0.2.2}/PKG-INFO +14 -14
  3. {fauxdata_cli-0.2.0 → fauxdata_cli-0.2.2}/README.md +12 -12
  4. {fauxdata_cli-0.2.0 → fauxdata_cli-0.2.2}/pyproject.toml +5 -2
  5. {fauxdata_cli-0.2.0 → fauxdata_cli-0.2.2}/src/fauxdata/__init__.py +1 -1
  6. {fauxdata_cli-0.2.0 → fauxdata_cli-0.2.2}/src/fauxdata/commands/agent.py +1 -1
  7. {fauxdata_cli-0.2.0 → fauxdata_cli-0.2.2}/src/fauxdata/commands/infer.py +24 -2
  8. {fauxdata_cli-0.2.0 → fauxdata_cli-0.2.2}/src/fauxdata/validator.py +1 -2
  9. {fauxdata_cli-0.2.0 → fauxdata_cli-0.2.2}/tests/test_generator.py +9 -0
  10. {fauxdata_cli-0.2.0 → fauxdata_cli-0.2.2}/tests/test_infer.py +67 -0
  11. fauxdata_cli-0.2.0/.python-version +0 -1
  12. fauxdata_cli-0.2.0/LOG.md +0 -84
  13. fauxdata_cli-0.2.0/RELEASING.md +0 -83
  14. fauxdata_cli-0.2.0/docs/deployment.md +0 -90
  15. fauxdata_cli-0.2.0/docs/future-ideas.md +0 -8
  16. fauxdata_cli-0.2.0/docs/index.html +0 -1037
  17. fauxdata_cli-0.2.0/docs/share.png +0 -0
  18. fauxdata_cli-0.2.0/tasks/todo.md +0 -90
  19. fauxdata_cli-0.2.0/uv.lock +0 -1338
  20. {fauxdata_cli-0.2.0 → fauxdata_cli-0.2.2}/LICENSE +0 -0
  21. {fauxdata_cli-0.2.0 → fauxdata_cli-0.2.2}/schemas/events.yml +0 -0
  22. {fauxdata_cli-0.2.0 → fauxdata_cli-0.2.2}/schemas/orders.yml +0 -0
  23. {fauxdata_cli-0.2.0 → fauxdata_cli-0.2.2}/schemas/people.yml +0 -0
  24. {fauxdata_cli-0.2.0 → fauxdata_cli-0.2.2}/src/fauxdata/commands/__init__.py +0 -0
  25. {fauxdata_cli-0.2.0 → fauxdata_cli-0.2.2}/src/fauxdata/commands/generate.py +0 -0
  26. {fauxdata_cli-0.2.0 → fauxdata_cli-0.2.2}/src/fauxdata/commands/init.py +0 -0
  27. {fauxdata_cli-0.2.0 → fauxdata_cli-0.2.2}/src/fauxdata/commands/preview.py +0 -0
  28. {fauxdata_cli-0.2.0 → fauxdata_cli-0.2.2}/src/fauxdata/commands/validate.py +0 -0
  29. {fauxdata_cli-0.2.0 → fauxdata_cli-0.2.2}/src/fauxdata/generator.py +0 -0
  30. {fauxdata_cli-0.2.0 → fauxdata_cli-0.2.2}/src/fauxdata/jsonmode.py +0 -0
  31. {fauxdata_cli-0.2.0 → fauxdata_cli-0.2.2}/src/fauxdata/main.py +0 -0
  32. {fauxdata_cli-0.2.0 → fauxdata_cli-0.2.2}/src/fauxdata/output.py +0 -0
  33. {fauxdata_cli-0.2.0 → fauxdata_cli-0.2.2}/src/fauxdata/schema.py +0 -0
  34. {fauxdata_cli-0.2.0 → fauxdata_cli-0.2.2}/tests/__init__.py +0 -0
  35. {fauxdata_cli-0.2.0 → fauxdata_cli-0.2.2}/tests/conftest.py +0 -0
  36. {fauxdata_cli-0.2.0 → fauxdata_cli-0.2.2}/tests/test_agent.py +0 -0
  37. {fauxdata_cli-0.2.0 → fauxdata_cli-0.2.2}/tests/test_cli.py +0 -0
  38. {fauxdata_cli-0.2.0 → fauxdata_cli-0.2.2}/tests/test_new_fields.py +0 -0
  39. {fauxdata_cli-0.2.0 → fauxdata_cli-0.2.2}/tests/test_output.py +0 -0
  40. {fauxdata_cli-0.2.0 → fauxdata_cli-0.2.2}/tests/test_schema.py +0 -0
  41. {fauxdata_cli-0.2.0 → fauxdata_cli-0.2.2}/tests/test_validator.py +0 -0
@@ -3,3 +3,4 @@ tmp/
3
3
  __pycache__/
4
4
  *.egg-info/
5
5
  .coverage
6
+ /share.png
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.5
2
2
  Name: fauxdata-cli
3
- Version: 0.2.0
3
+ Version: 0.2.2
4
4
  Summary: CLI for generating and validating fake datasets
5
5
  Project-URL: Homepage, https://aborruso.github.io/fauxdata/
6
6
  Project-URL: Repository, https://github.com/aborruso/fauxdata
@@ -13,7 +13,7 @@ Classifier: Programming Language :: Python :: 3.11
13
13
  Classifier: Programming Language :: Python :: 3.12
14
14
  Classifier: Programming Language :: Python :: 3.13
15
15
  Requires-Python: >=3.11
16
- Requires-Dist: pointblank>=1.0.1
16
+ Requires-Dist: pointblank>=1.2.0
17
17
  Requires-Dist: polars>=1.0
18
18
  Requires-Dist: pyfiglet>=1.0
19
19
  Requires-Dist: pyyaml>=6.0
@@ -93,7 +93,7 @@ fauxdata init --name orders
93
93
 
94
94
  ## Schema format
95
95
 
96
- A schema is a YAML file that describes the structure of your dataset. Here is a realistic example for a people dataset:
96
+ A schema is a YAML file that describes the structure of your dataset. Here is a realistic example for a people dataset, adapted from [`schemas/people.yml`](https://github.com/aborruso/fauxdata/blob/main/schemas/people.yml):
97
97
 
98
98
  ```yaml
99
99
  name: people
@@ -169,12 +169,12 @@ validation:
169
169
 
170
170
  | Type | Description | Options |
171
171
  |------|-------------|---------|
172
- | `int` | Integer | `min`, `max`, `unique` |
173
- | `float` | Floating point | `min`, `max`, `precision` |
172
+ | `int` | Integer | `min`, `max`, `unique`, `values` |
173
+ | `float` | Floating point | `min`, `max`, `precision`, `unique`, `values` |
174
174
  | `string` | Text | `preset`, `values`, `pattern`, `unique`, `min_length`, `max_length` |
175
175
  | `bool` | Boolean | — |
176
- | `date` | Date | `min`, `max` (ISO format) |
177
- | `datetime` | Datetime | `min`, `max` (ISO format) |
176
+ | `date` | Date | `min`, `max` (ISO format), `unique` |
177
+ | `datetime` | Datetime | `min`, `max` (ISO format), `unique` |
178
178
 
179
179
  ### Float precision
180
180
 
@@ -269,7 +269,7 @@ fauxdata generate schemas/people.yml --out - --format jsonl | wc -l
269
269
  | `--rows` | `-r` | from schema | Number of rows to generate |
270
270
  | `--out` | `-o` | from schema | Output path — use `-` for stdout |
271
271
  | `--format` | `-f` | from schema | Output format: `csv`, `parquet`, `json`, `jsonl`, `jsonlines` |
272
- | `--seed` | `-s` | from schema | Random seed for reproducibility |
272
+ | `--seed` | `-s` | from schema | Random seed for reproducibility: same seed, same data, on the same pointblank version |
273
273
  | `--validate` | `-v` | off | Run validation rules after generating |
274
274
  | `--json` | | off | Print only a JSON summary to stdout (see [Using fauxdata with AI agents](#using-fauxdata-with-ai-agents)) |
275
275
 
@@ -277,7 +277,7 @@ When `--out -` is used, all output messages are suppressed and only data is writ
277
277
 
278
278
  ### `fauxdata infer DATASET`
279
279
 
280
- Build a YAML schema from a **real** dataset. fauxdata inspects the table (via pointblank's schema inference) and writes a schema with inferred ranges, categorical value sets, string presets (email, url…), uniqueness, null rates, and string lengths — plus matching `validation:` rules. Feed the result back to `fauxdata generate` to produce synthetic data that mirrors the real shape, without ever shipping the real values.
280
+ Build a YAML schema from a **real** dataset. fauxdata inspects the table (via pointblank's schema inference) and writes a schema with inferred ranges, categorical value sets, string presets (email, url…), uniqueness, null rates, and string lengths — plus matching `validation:` rules. Feed the result back to `fauxdata generate` to produce synthetic data with the same columns, ranges and value sets, without copying the real rows. Each column is generated independently, so correlations between columns, rules such as `end_date > start_date`, and category frequencies are not preserved.
281
281
 
282
282
  ```
283
283
  fauxdata infer real_data.csv # -> real_data.yml
@@ -296,7 +296,7 @@ fauxdata infer big.parquet --sample-size 10000 --out - | head # stream to stdou
296
296
  | `--detect-presets` / `--no-detect-presets` | | on | Match string columns to known presets (email, url…) |
297
297
  | `--sample-size` | | all rows | Sample N rows before analysis (for very large tables) |
298
298
 
299
- > **Notes.** Low-cardinality columns (≤ `--categorical-threshold` distinct values) are frozen to their exact source values via `values:` — desirable for categories like `status`, but it means sensitive low-cardinality columns are reproduced verbatim. Raise the threshold or edit the schema if that's not what you want. Also, a column inferred as `unique` over a narrow integer range cannot generate many more rows than the source; widen `min`/`max` or drop `unique` to amplify.
299
+ > **Notes.** Low-cardinality columns (≤ `--categorical-threshold` distinct values) are frozen to their exact source values via `values:` — desirable for categories like `status`, but it means sensitive low-cardinality columns are reproduced verbatim. Raise the threshold or edit the schema if that's not what you want. This also applies to columns whose name suggests a place, company or job (`region`, `city`, `company`...). Personal presets are never frozen, so a low-cardinality column such as `product_name` still gets people's names: use `--no-detect-presets` or edit the schema. Also, a column inferred as `unique` over a narrow integer range cannot generate many more rows than the source; widen `min`/`max` or drop `unique` to amplify.
300
300
 
301
301
  ### `fauxdata validate DATASET SCHEMA`
302
302
 
@@ -365,13 +365,13 @@ fauxdata validate data.csv schema.yml --json
365
365
 
366
366
  ## Example schemas
367
367
 
368
- Three ready-to-use schemas are included in `schemas/`:
368
+ Three ready-to-use schemas are included in [`schemas/`](https://github.com/aborruso/fauxdata/tree/main/schemas):
369
369
 
370
370
  | Schema | Domain | Columns |
371
371
  |--------|--------|---------|
372
- | `people.yml` | Personal data | id, name, email, age, city, country_code, active, signup_date, score |
373
- | `orders.yml` | E-commerce | order_id, customer_id, product, amount, status, created_at |
374
- | `events.yml` | Analytics | event_id, user_id, event_type, timestamp, ip, user_agent, session_duration |
372
+ | [`people.yml`](https://github.com/aborruso/fauxdata/blob/main/schemas/people.yml) | Personal data | id, name, email, age, city, country_code, active, signup_date, score |
373
+ | [`orders.yml`](https://github.com/aborruso/fauxdata/blob/main/schemas/orders.yml) | E-commerce | order_id, customer_id, product, amount, status, created_at |
374
+ | [`events.yml`](https://github.com/aborruso/fauxdata/blob/main/schemas/events.yml) | Analytics | event_id, user_id, event_type, timestamp, ip, user_agent, session_duration |
375
375
 
376
376
  ---
377
377
 
@@ -69,7 +69,7 @@ fauxdata init --name orders
69
69
 
70
70
  ## Schema format
71
71
 
72
- A schema is a YAML file that describes the structure of your dataset. Here is a realistic example for a people dataset:
72
+ A schema is a YAML file that describes the structure of your dataset. Here is a realistic example for a people dataset, adapted from [`schemas/people.yml`](https://github.com/aborruso/fauxdata/blob/main/schemas/people.yml):
73
73
 
74
74
  ```yaml
75
75
  name: people
@@ -145,12 +145,12 @@ validation:
145
145
 
146
146
  | Type | Description | Options |
147
147
  |------|-------------|---------|
148
- | `int` | Integer | `min`, `max`, `unique` |
149
- | `float` | Floating point | `min`, `max`, `precision` |
148
+ | `int` | Integer | `min`, `max`, `unique`, `values` |
149
+ | `float` | Floating point | `min`, `max`, `precision`, `unique`, `values` |
150
150
  | `string` | Text | `preset`, `values`, `pattern`, `unique`, `min_length`, `max_length` |
151
151
  | `bool` | Boolean | — |
152
- | `date` | Date | `min`, `max` (ISO format) |
153
- | `datetime` | Datetime | `min`, `max` (ISO format) |
152
+ | `date` | Date | `min`, `max` (ISO format), `unique` |
153
+ | `datetime` | Datetime | `min`, `max` (ISO format), `unique` |
154
154
 
155
155
  ### Float precision
156
156
 
@@ -245,7 +245,7 @@ fauxdata generate schemas/people.yml --out - --format jsonl | wc -l
245
245
  | `--rows` | `-r` | from schema | Number of rows to generate |
246
246
  | `--out` | `-o` | from schema | Output path — use `-` for stdout |
247
247
  | `--format` | `-f` | from schema | Output format: `csv`, `parquet`, `json`, `jsonl`, `jsonlines` |
248
- | `--seed` | `-s` | from schema | Random seed for reproducibility |
248
+ | `--seed` | `-s` | from schema | Random seed for reproducibility: same seed, same data, on the same pointblank version |
249
249
  | `--validate` | `-v` | off | Run validation rules after generating |
250
250
  | `--json` | | off | Print only a JSON summary to stdout (see [Using fauxdata with AI agents](#using-fauxdata-with-ai-agents)) |
251
251
 
@@ -253,7 +253,7 @@ When `--out -` is used, all output messages are suppressed and only data is writ
253
253
 
254
254
  ### `fauxdata infer DATASET`
255
255
 
256
- Build a YAML schema from a **real** dataset. fauxdata inspects the table (via pointblank's schema inference) and writes a schema with inferred ranges, categorical value sets, string presets (email, url…), uniqueness, null rates, and string lengths — plus matching `validation:` rules. Feed the result back to `fauxdata generate` to produce synthetic data that mirrors the real shape, without ever shipping the real values.
256
+ Build a YAML schema from a **real** dataset. fauxdata inspects the table (via pointblank's schema inference) and writes a schema with inferred ranges, categorical value sets, string presets (email, url…), uniqueness, null rates, and string lengths — plus matching `validation:` rules. Feed the result back to `fauxdata generate` to produce synthetic data with the same columns, ranges and value sets, without copying the real rows. Each column is generated independently, so correlations between columns, rules such as `end_date > start_date`, and category frequencies are not preserved.
257
257
 
258
258
  ```
259
259
  fauxdata infer real_data.csv # -> real_data.yml
@@ -272,7 +272,7 @@ fauxdata infer big.parquet --sample-size 10000 --out - | head # stream to stdou
272
272
  | `--detect-presets` / `--no-detect-presets` | | on | Match string columns to known presets (email, url…) |
273
273
  | `--sample-size` | | all rows | Sample N rows before analysis (for very large tables) |
274
274
 
275
- > **Notes.** Low-cardinality columns (≤ `--categorical-threshold` distinct values) are frozen to their exact source values via `values:` — desirable for categories like `status`, but it means sensitive low-cardinality columns are reproduced verbatim. Raise the threshold or edit the schema if that's not what you want. Also, a column inferred as `unique` over a narrow integer range cannot generate many more rows than the source; widen `min`/`max` or drop `unique` to amplify.
275
+ > **Notes.** Low-cardinality columns (≤ `--categorical-threshold` distinct values) are frozen to their exact source values via `values:` — desirable for categories like `status`, but it means sensitive low-cardinality columns are reproduced verbatim. Raise the threshold or edit the schema if that's not what you want. This also applies to columns whose name suggests a place, company or job (`region`, `city`, `company`...). Personal presets are never frozen, so a low-cardinality column such as `product_name` still gets people's names: use `--no-detect-presets` or edit the schema. Also, a column inferred as `unique` over a narrow integer range cannot generate many more rows than the source; widen `min`/`max` or drop `unique` to amplify.
276
276
 
277
277
  ### `fauxdata validate DATASET SCHEMA`
278
278
 
@@ -341,13 +341,13 @@ fauxdata validate data.csv schema.yml --json
341
341
 
342
342
  ## Example schemas
343
343
 
344
- Three ready-to-use schemas are included in `schemas/`:
344
+ Three ready-to-use schemas are included in [`schemas/`](https://github.com/aborruso/fauxdata/tree/main/schemas):
345
345
 
346
346
  | Schema | Domain | Columns |
347
347
  |--------|--------|---------|
348
- | `people.yml` | Personal data | id, name, email, age, city, country_code, active, signup_date, score |
349
- | `orders.yml` | E-commerce | order_id, customer_id, product, amount, status, created_at |
350
- | `events.yml` | Analytics | event_id, user_id, event_type, timestamp, ip, user_agent, session_duration |
348
+ | [`people.yml`](https://github.com/aborruso/fauxdata/blob/main/schemas/people.yml) | Personal data | id, name, email, age, city, country_code, active, signup_date, score |
349
+ | [`orders.yml`](https://github.com/aborruso/fauxdata/blob/main/schemas/orders.yml) | E-commerce | order_id, customer_id, product, amount, status, created_at |
350
+ | [`events.yml`](https://github.com/aborruso/fauxdata/blob/main/schemas/events.yml) | Analytics | event_id, user_id, event_type, timestamp, ip, user_agent, session_duration |
351
351
 
352
352
  ---
353
353
 
@@ -1,6 +1,6 @@
1
1
  [project]
2
2
  name = "fauxdata-cli"
3
- version = "0.2.0"
3
+ version = "0.2.2"
4
4
  description = "CLI for generating and validating fake datasets"
5
5
  readme = "README.md"
6
6
  requires-python = ">=3.11"
@@ -13,7 +13,7 @@ classifiers = [
13
13
  "Operating System :: OS Independent",
14
14
  ]
15
15
  dependencies = [
16
- "pointblank>=1.0.1",
16
+ "pointblank>=1.2.0",
17
17
  "typer>=0.12",
18
18
  "rich>=13",
19
19
  "pyfiglet>=1.0",
@@ -46,3 +46,6 @@ build-backend = "hatchling.build"
46
46
 
47
47
  [tool.hatch.build.targets.wheel]
48
48
  packages = ["src/fauxdata"]
49
+
50
+ [tool.hatch.build.targets.sdist]
51
+ only-include = ["src", "tests", "schemas", "README.md", "LICENSE", "pyproject.toml"]
@@ -1,3 +1,3 @@
1
1
  """fauxdata - CLI for generating and validating fake datasets."""
2
2
 
3
- __version__ = "0.2.0"
3
+ __version__ = "0.2.2"
@@ -139,7 +139,7 @@ Notes:
139
139
 
140
140
  {presets}
141
141
 
142
- Presets are locale-aware and coherent within a row (the email matches the name, the city matches the country). Known limit: `unique: true` is not enforced on person and address presets such as `name`, `email`, `user_name`, `city` (pointblank issue #441). Validate with `rows_distinct` if uniqueness matters.
142
+ Presets are locale-aware and coherent within a row (the email matches the name, the city matches the country).
143
143
 
144
144
  ## Validation rules
145
145
 
@@ -31,6 +31,11 @@ _DTYPE_MAP = {
31
31
  "Date": "date",
32
32
  }
33
33
 
34
+ # Presets pointblank matches on the column name alone, skipping categorical detection.
35
+ # Personal presets (name, phone, address...) are left out on purpose: restoring their
36
+ # source values would copy real people's data into the schema.
37
+ _CATEGORICAL_PRESETS = {"state", "city", "country", "country_code_2", "currency_code", "company", "job"}
38
+
34
39
 
35
40
  def _col_type_from_dtype(dtype: str) -> str:
36
41
  """Map a pointblank/polars dtype name to a fauxdata col_type."""
@@ -83,6 +88,19 @@ def _field_to_spec(field) -> dict:
83
88
  return spec
84
89
 
85
90
 
91
+ def _restore_categorical(spec: dict, series: pl.Series, threshold: int | float) -> dict:
92
+ """Replace a name-matched preset with the source values on a low-cardinality column."""
93
+ if spec.get("preset") not in _CATEGORICAL_PRESETS or spec.get("unique"):
94
+ return spec
95
+ limit = threshold if isinstance(threshold, int) else int(threshold * series.len())
96
+ values = series.drop_nulls().unique().sort().to_list()
97
+ if len(values) > limit:
98
+ return spec
99
+ del spec["preset"]
100
+ spec["values"] = values
101
+ return spec
102
+
103
+
86
104
  def _build_validation(columns: list[tuple[str, dict]]) -> list[dict]:
87
105
  """Derive validation rules from the inferred column specs."""
88
106
  rules: list[dict] = []
@@ -160,14 +178,18 @@ def run(
160
178
  if not json_out:
161
179
  rprint(f" Loaded [bold]{len(df)}[/bold] rows, [bold]{len(df.columns)}[/bold] columns")
162
180
 
181
+ threshold = _threshold(categorical_threshold)
163
182
  pb_schema = pb.schema_from_tbl(
164
183
  df,
165
- categorical_threshold=_threshold(categorical_threshold),
184
+ categorical_threshold=threshold,
166
185
  detect_presets=detect_presets,
167
186
  sample_size=sample_size,
168
187
  )
169
188
 
170
- columns = [(col_name, _field_to_spec(field)) for col_name, field in pb_schema.columns]
189
+ columns = [
190
+ (col_name, _restore_categorical(_field_to_spec(field), df[col_name], threshold))
191
+ for col_name, field in pb_schema.columns
192
+ ]
171
193
 
172
194
  schema_name = name or Path(dataset_path).stem
173
195
  out_rows = rows if rows is not None else len(df)
@@ -80,8 +80,7 @@ def _add_rule(v: pb.Validate, rule: ValidationRule) -> None:
80
80
 
81
81
  def _extract_results(v: pb.Validate, rules: list[ValidationRule]) -> list[dict]:
82
82
  """Extract per-step results as a list of dicts."""
83
- # validation_info, not get_dataframe_report(): the report fails on list-valued
84
- # columns (rows_distinct) in pointblank 1.0.1
83
+ # validation_info carries active and eval_error, used to flag steps that could not run
85
84
  steps = {s.i: s for s in v.validation_info}
86
85
  results = []
87
86
  # Build a flat list of steps (one per column for multi-column rules)
@@ -103,6 +103,15 @@ def test_generate_float_precision_stays_in_range():
103
103
  assert df["f"].max() <= 9.999
104
104
 
105
105
 
106
+ def test_generate_unique_coherent_presets():
107
+ """unique is enforced on coherent presets (pointblank >= 1.1.0, posit-dev/pointblank#441)."""
108
+ schema = _make_schema([ColumnSchema(name="name", col_type="string", preset="name", unique=True),
109
+ ColumnSchema(name="email", col_type="string", preset="email", unique=True)])
110
+ df = generate_dataset(schema, rows=2000, seed=1)
111
+ assert df["name"].n_unique() == 2000
112
+ assert df["email"].n_unique() == 2000
113
+
114
+
106
115
  def test_generate_float_precision_unique():
107
116
  """A unique float column stays unique after rounding."""
108
117
  schema = _make_schema([ColumnSchema(name="f", col_type="float", min=0.0, max=1.0,
@@ -220,3 +220,70 @@ def test_infer_roundtrip_nulls_and_low_cardinality_int(tmp_path):
220
220
  df = generate_dataset(schema, rows=500, seed=3)
221
221
  all_passed, results = validate_dataset(df, schema)
222
222
  assert all_passed, [r for r in results if not r["ok"]]
223
+
224
+
225
+ # --- name-matched presets on categorical columns ---
226
+
227
+ def _infer_spec(tmp_path, column, values, **kwargs):
228
+ src = tmp_path / "real.csv"
229
+ pl.DataFrame({column: values}).write_csv(src)
230
+ out = tmp_path / "inferred.yml"
231
+ infer_mod.run(dataset_path=str(src), out=str(out), **kwargs)
232
+ return yaml.safe_load(out.read_text()), out
233
+
234
+
235
+ @pytest.mark.parametrize("column", ["region", "state", "city", "country", "currency", "company", "job"])
236
+ def test_infer_low_cardinality_keeps_source_values(tmp_path, column):
237
+ """A column matched to a preset only by its name stays categorical when values repeat."""
238
+ source = ["nord", "centro", "sud"] * 20
239
+ schema, out = _infer_spec(tmp_path, column, source)
240
+ spec = schema["columns"][column]
241
+ assert "preset" not in spec
242
+ assert sorted(spec["values"]) == ["centro", "nord", "sud"]
243
+
244
+ loaded = load_schema(out)
245
+ df = generate_dataset(loaded, rows=200, seed=1)
246
+ assert set(df[column].to_list()) <= {"nord", "centro", "sud"}
247
+ all_passed, results = validate_dataset(df, loaded)
248
+ assert all_passed, [r for r in results if not r["ok"]]
249
+
250
+
251
+ @pytest.mark.parametrize("column,preset", [("name", "name"), ("first_name", "first_name"), ("phone", "phone_number"), ("address", "address")])
252
+ def test_infer_personal_preset_never_frozen(tmp_path, column, preset):
253
+ """Personal presets are kept, so real people's values never reach the schema."""
254
+ schema, out = _infer_spec(tmp_path, column, ["Mario Rossi", "Anna Bianchi"] * 20)
255
+ assert schema["columns"][column]["preset"] == preset
256
+ assert "Mario Rossi" not in out.read_text()
257
+
258
+
259
+ def test_infer_high_cardinality_keeps_preset(tmp_path):
260
+ schema, _ = _infer_spec(tmp_path, "city", [f"city {i % 30}" for i in range(60)])
261
+ assert schema["columns"]["city"]["preset"] == "city"
262
+
263
+
264
+ def test_infer_unique_column_keeps_preset(tmp_path):
265
+ schema, _ = _infer_spec(tmp_path, "city", [f"city {i}" for i in range(10)])
266
+ spec = schema["columns"]["city"]
267
+ assert spec["preset"] == "city"
268
+ assert spec["unique"] is True
269
+
270
+
271
+ def test_infer_fractional_threshold_applies_to_preset_columns(tmp_path):
272
+ values = [f"r{i % 8}" for i in range(100)]
273
+ assert "values" in _infer_spec(tmp_path, "region", values, categorical_threshold=0.1)[0]["columns"]["region"]
274
+ assert "preset" in _infer_spec(tmp_path, "region", values, categorical_threshold=0.05)[0]["columns"]["region"]
275
+
276
+
277
+ def test_infer_no_detect_presets_unchanged(tmp_path):
278
+ schema, _ = _infer_spec(tmp_path, "region", ["a", "b"] * 10, detect_presets=False)
279
+ assert sorted(schema["columns"]["region"]["values"]) == ["a", "b"]
280
+
281
+
282
+ def test_infer_categorical_with_nulls_keeps_nullable(tmp_path):
283
+ schema, out = _infer_spec(tmp_path, "region", ["nord", None, "sud", "nord"] * 10)
284
+ spec = schema["columns"]["region"]
285
+ assert sorted(spec["values"]) == ["nord", "sud"]
286
+ assert spec["nullable"] is True
287
+ loaded = load_schema(out)
288
+ all_passed, results = validate_dataset(generate_dataset(loaded, rows=200, seed=1), loaded)
289
+ assert all_passed, [r for r in results if not r["ok"]]
@@ -1 +0,0 @@
1
- 3.12
fauxdata_cli-0.2.0/LOG.md DELETED
@@ -1,84 +0,0 @@
1
- # Log
2
-
3
- ## 2026-10-11 — 0.2.0 (non rilasciata): pointblank 1.0.1, `precision` nativa, `col_vals_str_len`, fix, comando `agent` e `--json`
4
-
5
- - Bump `pointblank>=1.0.1` (da 0.25.0). 103/103 test invariati; schemi inclusi PASS; stesso seed → CSV identici byte per byte tra 0.25 e 1.0.1. Effetto collaterale: da 0.27 pointblank richiede `anthropic[bedrock]` → 20 pacchetti in più installati (anthropic, boto3, pydantic, …), non usati da fauxdata.
6
- - `precision` float: ora nativa (`pb.float_field(precision=)`), unicità garantita dopo l'arrotondamento. Limiti stretti verso l'interno sulla griglia dei decimali (`max: 9.999`, `precision: 2` → `9.99`) perché l'arrotondamento non sfori il range; errore chiaro se nessun valore ci sta. Documentata nel README (prima non lo era).
7
- - Nuova regola `col_vals_str_len` (`column`, `min`/`max`, null ignorati). `infer` la emette solo per stringhe libere: su preset/pattern/values il generatore ignora le lunghezze.
8
- - Fix: le regole sui valori (`between`, `gt/lt/ge/le`, `regex`, `in_set`) ignorano i null (`na_pass=True`; `in_set` via `None` nel set). Prima una colonna col 20% di null falliva 195/1000 righe.
9
- - Fix: `infer --categorical-threshold` da CLI arrivava come float (anche il default) e pointblank lo leggeva come frazione → ogni colonna stringa senza preset diventava categorica con i valori reali copiati in `values:`. Ora gli interi ≥ 1 passano come conteggio.
10
- - Fix: `values` sulle colonne `int`/`float` ora rispettato in generazione (passato come `allowed`); prima `infer` lo emetteva per gli int a bassa cardinalità e il generatore lo ignorava (403/500 fallimenti in round-trip).
11
- - Fix: `rows_distinct` esclude le righe con null nelle colonne controllate (come SQL UNIQUE); prima i null contavano come duplicati tra loro.
12
- - Bug di pointblank (non nostro): `unique=True` ignorato sui preset coerenti (`email`, `name`, `city`, …); segnalato in https://github.com/posit-dev/pointblank/issues/441. Non corretto: `unique` + `nullable` su range stretto non generabile.
13
- - Nuovo comando `fauxdata agent`: guida Markdown per agenti AI (flusso, output JSON, exit code, schema, tipi, preset, regole, esempio); liste generate dalle costanti di `schema.py`.
14
- - Nuovo flag `--json` su `generate`, `validate`, `infer`, `preview`: stdout contiene solo un oggetto JSON; errori come `{"error": ...}` con exit 1. Sezione README "Using fauxdata with AI agents".
15
- - Fix: un passo su una colonna assente dal dataset risultava `ok: true` con 0/0 righe; ora campo `evaluated` (da `validation_info`), `ok: false`, `NOT RUN` in tabella. `get_dataframe_report()` non usabile: fallisce con `rows_distinct`/`rows_complete` + `columns_subset` (segnalato in https://github.com/posit-dev/pointblank/issues/442). Chiude la issue #1 di fauxdata (output strutturato → `--json`).
16
- - Sito `docs/index.html`: badge pointblank 1.0+, "six commands", righe `infer` e `agent`, `--json`.
17
- - 143/143 test, coverage 87.05%.
18
-
19
- ## 2026-07-04 — v0.1.6
20
-
21
- - Release `0.1.6`: include il fix issue #2 (`precision` float). Build con `uv build`, tag `v0.1.6`.
22
- - Deploy PyPI **manuale** (`uv publish`): nessun workflow di release automatico.
23
-
24
- ## 2026-07-04 — fix issue #2: `precision` float ignorato
25
-
26
- - Bug: l'opzione `precision` dei float veniva letta nello schema ma mai applicata (`pb.float_field` non ha il parametro). Valori generati con precisione piena.
27
- - Fix in `generator.py`: dopo `pb.generate_dataset`, arrotondo con `pl.col(...).round(precision)` le colonne `float` con `precision` impostato.
28
- - Verifica: reproducer della issue → output atteso (`27.74, 169.64, …`). Test `test_generate_float_precision`. 103/103 pass.
29
-
30
- ## 2026-06-12 — pointblank 0.25 + comando `infer`
31
-
32
- - Bump `pointblank>=0.25` (uscito su PyPI). Schemi inclusi (people/orders/events) verificati: tutti PASS.
33
- - Fase B regressioni: `rows_distinct` ora include righe con null (correttezza migliorata), lazy frame ok. fauxdata passa sempre DataFrame eager → nessun impatto.
34
- - Nuovo comando **`fauxdata infer DATASET`**: da tabella reale (csv/parquet/json/jsonl) genera schema YAML (`columns:` + `validation:`) via `pb.schema_from_tbl`. Flag: `--out/-o`, `--name/-n`, `--rows/-r`, `--format/-f`, `--categorical-threshold`, `--detect-presets/--no-detect-presets`, `--sample-size`.
35
- - `ColumnSchema`: nuovi campi `min_length`/`max_length` per stringhe (round-trip fedele); passati a `pb.string_field`. Validazione parsing (negativi, min>max).
36
- - infer CSV: `try_parse_dates=True` → date/datetime tipizzati invece che congelati come categorici.
37
- - generate: errore chiaro su conflitto unique+range stretto ("too narrow a range") invece del traceback criptico di pointblank.
38
- - Limite noto: colonne a bassa cardinalità congelate ai valori reali (possibile leak su colonne sensibili) — documentato nel README.
39
- - Test: +14 (`tests/test_infer.py` + length in `test_new_fields.py`). 102/102 pass, coverage 83.2%.
40
-
41
- - `init`: aggiunto `--description`, `--rows`, `--format`, `--yes`; questionary diventa fallback
42
- - Tutti i comandi: `epilog` con esempi in `--help`
43
- - `generate`: aggiunto `--dry-run` (mostra piano senza scrivere)
44
- - `generate`: output su successo mostra `output_path`, `format`, `rows` come chiave: valore
45
- - `init`: output su successo aggiunge `schema_path: <file>`
46
- - Fix venv: ricreato con `uv venv --clear` (cartella rinominata da `real_fake_datasets`)
47
-
48
- ## 2026-03-06 — v0.1.3
49
-
50
- - Add Python classifiers to pyproject.toml (3.11, 3.12, 3.13, MIT) — fixes pyversions badge
51
- - Fix version tests to read `__version__` dynamically instead of hardcoded string
52
-
53
- ## 2026-03-06 — v0.1.2
54
-
55
- - Bump to 0.1.2 and publish to PyPI
56
-
57
- ## 2026-03-06 (feature)
58
-
59
- - `--version` / `-V` flag nel CLI (`fauxdata --version` → `fauxdata 0.1.1`)
60
- - Coverage threshold 80% in pytest config (`--cov-fail-under=80`); attuale: 83.76%
61
- - Nuovo campo `pattern` in ColumnSchema: genera stringhe che matchano un regex via pointblank
62
- - Nuovo campo `null_probability` in ColumnSchema: controllo granulare dei null (0.0–1.0), con validazione in parsing
63
- - Rimossa dipendenza `faker` (non usata, pointblank gestisce tutto)
64
- - Fix generator: `null_probability=None` non passato a pointblank (causa TypeError)
65
- - Test aggiornati: 79/79 pass
66
-
67
- ## 2026-03-06 (tests)
68
-
69
- - Add pytest test suite: 65 tests, 100% pass, 0.44s
70
- - `tests/test_schema.py`: unit tests for YAML schema parsing (valid/invalid cases)
71
- - `tests/test_output.py`: unit tests for export functions (all formats, stdout, errors)
72
- - `tests/test_generator.py`: integration tests for generation (types, seed, unique, presets)
73
- - `tests/test_validator.py`: integration tests for validation rules (pass/fail scenarios)
74
- - `tests/test_cli.py`: CLI smoke tests via `typer.testing.CliRunner`
75
- - Add `[dependency-groups] dev` in `pyproject.toml` (pytest, pytest-cov); config via `[tool.pytest.ini_options]`
76
-
77
- ## 2026-03-06
78
-
79
- - Initial implementation of `fauxdata` CLI
80
- - Stack: pointblank 0.22 (native generation + validation), polars, typer, rich, pyfiglet, questionary
81
- - Commands: `init`, `generate`, `validate`, `preview`
82
- - Example schemas: `people.yml`, `orders.yml`, `events.yml`
83
- - All schemas generate and validate cleanly (all rules PASS)
84
- - `locale` field at schema level maps to pointblank `country=` param
@@ -1,83 +0,0 @@
1
- # Releasing
2
-
3
- Procedura per pubblicare una nuova release di `fauxdata` su PyPI.
4
- Il deploy **non è automatico**: nessun workflow GitHub Actions. Ogni step è manuale.
5
-
6
- ## Versioning
7
-
8
- `MAJOR.MINOR.PATCH` (SemVer).
9
-
10
- - **subrelease / patch** (`0.1.5` → `0.1.6`): bugfix, nessuna modifica di API.
11
- - **minor** (`0.1.x` → `0.2.0`): nuove feature retrocompatibili.
12
- - **major** (`0.x` → `1.0.0`): breaking change.
13
-
14
- ## Step
15
-
16
- 1. **Aggiorna la versione** nei due file (devono restare allineati):
17
- - `pyproject.toml` → `version = "X.Y.Z"`
18
- - `src/fauxdata/__init__.py` → `__version__ = "X.Y.Z"`
19
-
20
- 2. **Aggiorna `LOG.md`**: nuova sezione in cima con data `YYYY-MM-DD` e le modifiche.
21
-
22
- 3. **Test**: devono passare tutti.
23
-
24
- ```bash
25
- uv run pytest tests/ -q
26
- uv run fauxdata --version # deve stampare la nuova versione
27
- ```
28
-
29
- 4. **Build**:
30
-
31
- ```bash
32
- rm -rf dist/
33
- uv build
34
- ls dist/ # fauxdata_cli-X.Y.Z-py3-none-any.whl + .tar.gz
35
- ```
36
-
37
- 5. **Aggiorna la CLI installata localmente** (`fauxdata` è installato come `uv tool`):
38
-
39
- ```bash
40
- uv tool install --reinstall . # reinstalla dalla working copy
41
- fauxdata --version # deve stampare la nuova versione
42
- ```
43
-
44
- 6. **Commit + tag + push**:
45
-
46
- ```bash
47
- git add pyproject.toml src/fauxdata/__init__.py LOG.md
48
- git commit -m "chore: vX.Y.Z — <sintesi>"
49
- git tag -a vX.Y.Z -m "vX.Y.Z — <sintesi>"
50
- git push origin main
51
- git push origin vX.Y.Z
52
- ```
53
-
54
- 7. **Pubblica su PyPI** (richiede token PyPI — step da eseguire a mano):
55
-
56
- ```bash
57
- uv publish # oppure: twine upload dist/*
58
- ```
59
-
60
- Il token si passa via `UV_PUBLISH_TOKEN` o `--token`. Non committare mai il token.
61
-
62
- 8. **Verifica / aggiorna da PyPI** (il pacchetto su PyPI è `fauxdata-cli`):
63
-
64
- ```bash
65
- uvx fauxdata@X.Y.Z --version # verifica al volo la versione pubblicata
66
-
67
- pip install --upgrade fauxdata-cli # aggiorna un'installazione pip esistente
68
- # oppure, se installato come uv tool:
69
- uv tool upgrade fauxdata-cli
70
- ```
71
-
72
- 9. **GitHub Release**: dal tag già pushato (il tag da solo **non** crea la release).
73
- **Titolo e note devono essere in inglese.**
74
-
75
- ```bash
76
- gh release create vX.Y.Z --title "vX.Y.Z" --notes "<changelog in English>"
77
- ```
78
-
79
- ## Note
80
-
81
- - Se una release chiude una issue, referenziarla nel commit/tag (`#N`) e chiudere la issue con un commento che linka al commit.
82
- - La versione è testata dinamicamente (`test_cli_version` legge `__version__`), quindi il bump non rompe i test.
83
- - **Tag ≠ GitHub Release**: `git push origin vX.Y.Z` crea solo il tag; la pagina di release va creata a parte con `gh release create` (step 9).
@@ -1,90 +0,0 @@
1
- # Deployment rules
2
-
3
- ## Pre-release checklist (always)
4
-
5
- 1. Run tests locally — must all pass:
6
-
7
- ```bash
8
- uv run pytest
9
- ```
10
-
11
- Coverage must stay above 80%. If it drops, fix before proceeding.
12
-
13
- 2. Bump version in **both**:
14
- - `src/fauxdata/__init__.py` → `__version__ = "X.Y.Z"`
15
- - `pyproject.toml` → `version = "X.Y.Z"`
16
-
17
- 3. Update `LOG.md` with a summary of changes under a new date heading.
18
-
19
- ---
20
-
21
- ## GitHub release (tag + release notes)
22
-
23
- ```bash
24
- # Create and push annotated tag
25
- git tag -a vX.Y.Z -m "vX.Y.Z"
26
- git push origin vX.Y.Z
27
- ```
28
-
29
- Then create a GitHub release via `gh`:
30
-
31
- ```bash
32
- gh release create vX.Y.Z \
33
- --title "vX.Y.Z" \
34
- --notes "$(cat <<'EOF'
35
- ## What's new
36
-
37
- - Short bullet list of user-facing changes
38
- - Include new fields, commands, bug fixes
39
-
40
- ## Breaking changes
41
-
42
- - List any breaking changes here (or remove section if none)
43
-
44
- ## Installation
45
-
46
- \`\`\`bash
47
- pip install fauxdata-cli==X.Y.Z
48
- \`\`\`
49
-
50
- Full changelog: https://github.com/aborruso/fauxdata/commits/vX.Y.Z
51
- EOF
52
- )"
53
- ```
54
-
55
- Release notes style: **concise, nerd-friendly, technical**. List the actual changes with enough detail that a developer understands what changed and why.
56
-
57
- ---
58
-
59
- ## PyPI publish (via twine)
60
-
61
- ```bash
62
- # Build
63
- uv build
64
-
65
- # Check the dist
66
- twine check dist/*
67
-
68
- # Publish
69
- twine upload dist/*
70
- ```
71
-
72
- Requires `~/.pypirc` configured with PyPI token, or set `TWINE_USERNAME`/`TWINE_PASSWORD` env vars.
73
-
74
- ---
75
-
76
- ## Order of operations
77
-
78
- ```
79
- uv run pytest # must pass 100%
80
- bump version # __init__.py + pyproject.toml
81
- update LOG.md
82
- git commit + git push
83
- git tag + git push tag
84
- gh release create # with release notes
85
- uv build
86
- twine check dist/*
87
- twine upload dist/*
88
- ```
89
-
90
- Never publish to PyPI without a corresponding GitHub release.
@@ -1,8 +0,0 @@
1
- # Future ideas
2
-
3
- Ideas from the pointblank 0.26-1.0.1 releases, not yet planned (2026-10-11).
4
-
5
- - Related tables: generate linked datasets (e.g. people and orders) and check foreign keys with `col_vals_in_table()` (pointblank 0.27).
6
- - Coded missing values: a schema field for sentinel values like `-99` or `N/A`, generated and validated via `MissingSpec` and `col_pct_missing()` (pointblank 0.26).
7
- - Schema export: let `infer` also write JSON Schema, Frictionless, dbt or ODCS via `export_contract()` (pointblank 0.26/0.27).
8
- - Fidelity report: compare the real dataset with the synthetic one using `DataScan.compare()` (PSI, Kolmogorov-Smirnov) to show how close the shapes are (pointblank 0.27).