fauxdata-cli 0.2.0__tar.gz → 0.2.2__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {fauxdata_cli-0.2.0 → fauxdata_cli-0.2.2}/.gitignore +1 -0
- {fauxdata_cli-0.2.0 → fauxdata_cli-0.2.2}/PKG-INFO +14 -14
- {fauxdata_cli-0.2.0 → fauxdata_cli-0.2.2}/README.md +12 -12
- {fauxdata_cli-0.2.0 → fauxdata_cli-0.2.2}/pyproject.toml +5 -2
- {fauxdata_cli-0.2.0 → fauxdata_cli-0.2.2}/src/fauxdata/__init__.py +1 -1
- {fauxdata_cli-0.2.0 → fauxdata_cli-0.2.2}/src/fauxdata/commands/agent.py +1 -1
- {fauxdata_cli-0.2.0 → fauxdata_cli-0.2.2}/src/fauxdata/commands/infer.py +24 -2
- {fauxdata_cli-0.2.0 → fauxdata_cli-0.2.2}/src/fauxdata/validator.py +1 -2
- {fauxdata_cli-0.2.0 → fauxdata_cli-0.2.2}/tests/test_generator.py +9 -0
- {fauxdata_cli-0.2.0 → fauxdata_cli-0.2.2}/tests/test_infer.py +67 -0
- fauxdata_cli-0.2.0/.python-version +0 -1
- fauxdata_cli-0.2.0/LOG.md +0 -84
- fauxdata_cli-0.2.0/RELEASING.md +0 -83
- fauxdata_cli-0.2.0/docs/deployment.md +0 -90
- fauxdata_cli-0.2.0/docs/future-ideas.md +0 -8
- fauxdata_cli-0.2.0/docs/index.html +0 -1037
- fauxdata_cli-0.2.0/docs/share.png +0 -0
- fauxdata_cli-0.2.0/tasks/todo.md +0 -90
- fauxdata_cli-0.2.0/uv.lock +0 -1338
- {fauxdata_cli-0.2.0 → fauxdata_cli-0.2.2}/LICENSE +0 -0
- {fauxdata_cli-0.2.0 → fauxdata_cli-0.2.2}/schemas/events.yml +0 -0
- {fauxdata_cli-0.2.0 → fauxdata_cli-0.2.2}/schemas/orders.yml +0 -0
- {fauxdata_cli-0.2.0 → fauxdata_cli-0.2.2}/schemas/people.yml +0 -0
- {fauxdata_cli-0.2.0 → fauxdata_cli-0.2.2}/src/fauxdata/commands/__init__.py +0 -0
- {fauxdata_cli-0.2.0 → fauxdata_cli-0.2.2}/src/fauxdata/commands/generate.py +0 -0
- {fauxdata_cli-0.2.0 → fauxdata_cli-0.2.2}/src/fauxdata/commands/init.py +0 -0
- {fauxdata_cli-0.2.0 → fauxdata_cli-0.2.2}/src/fauxdata/commands/preview.py +0 -0
- {fauxdata_cli-0.2.0 → fauxdata_cli-0.2.2}/src/fauxdata/commands/validate.py +0 -0
- {fauxdata_cli-0.2.0 → fauxdata_cli-0.2.2}/src/fauxdata/generator.py +0 -0
- {fauxdata_cli-0.2.0 → fauxdata_cli-0.2.2}/src/fauxdata/jsonmode.py +0 -0
- {fauxdata_cli-0.2.0 → fauxdata_cli-0.2.2}/src/fauxdata/main.py +0 -0
- {fauxdata_cli-0.2.0 → fauxdata_cli-0.2.2}/src/fauxdata/output.py +0 -0
- {fauxdata_cli-0.2.0 → fauxdata_cli-0.2.2}/src/fauxdata/schema.py +0 -0
- {fauxdata_cli-0.2.0 → fauxdata_cli-0.2.2}/tests/__init__.py +0 -0
- {fauxdata_cli-0.2.0 → fauxdata_cli-0.2.2}/tests/conftest.py +0 -0
- {fauxdata_cli-0.2.0 → fauxdata_cli-0.2.2}/tests/test_agent.py +0 -0
- {fauxdata_cli-0.2.0 → fauxdata_cli-0.2.2}/tests/test_cli.py +0 -0
- {fauxdata_cli-0.2.0 → fauxdata_cli-0.2.2}/tests/test_new_fields.py +0 -0
- {fauxdata_cli-0.2.0 → fauxdata_cli-0.2.2}/tests/test_output.py +0 -0
- {fauxdata_cli-0.2.0 → fauxdata_cli-0.2.2}/tests/test_schema.py +0 -0
- {fauxdata_cli-0.2.0 → fauxdata_cli-0.2.2}/tests/test_validator.py +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.5
|
|
2
2
|
Name: fauxdata-cli
|
|
3
|
-
Version: 0.2.
|
|
3
|
+
Version: 0.2.2
|
|
4
4
|
Summary: CLI for generating and validating fake datasets
|
|
5
5
|
Project-URL: Homepage, https://aborruso.github.io/fauxdata/
|
|
6
6
|
Project-URL: Repository, https://github.com/aborruso/fauxdata
|
|
@@ -13,7 +13,7 @@ Classifier: Programming Language :: Python :: 3.11
|
|
|
13
13
|
Classifier: Programming Language :: Python :: 3.12
|
|
14
14
|
Classifier: Programming Language :: Python :: 3.13
|
|
15
15
|
Requires-Python: >=3.11
|
|
16
|
-
Requires-Dist: pointblank>=1.0
|
|
16
|
+
Requires-Dist: pointblank>=1.2.0
|
|
17
17
|
Requires-Dist: polars>=1.0
|
|
18
18
|
Requires-Dist: pyfiglet>=1.0
|
|
19
19
|
Requires-Dist: pyyaml>=6.0
|
|
@@ -93,7 +93,7 @@ fauxdata init --name orders
|
|
|
93
93
|
|
|
94
94
|
## Schema format
|
|
95
95
|
|
|
96
|
-
A schema is a YAML file that describes the structure of your dataset. Here is a realistic example for a people dataset:
|
|
96
|
+
A schema is a YAML file that describes the structure of your dataset. Here is a realistic example for a people dataset, adapted from [`schemas/people.yml`](https://github.com/aborruso/fauxdata/blob/main/schemas/people.yml):
|
|
97
97
|
|
|
98
98
|
```yaml
|
|
99
99
|
name: people
|
|
@@ -169,12 +169,12 @@ validation:
|
|
|
169
169
|
|
|
170
170
|
| Type | Description | Options |
|
|
171
171
|
|------|-------------|---------|
|
|
172
|
-
| `int` | Integer | `min`, `max`, `unique` |
|
|
173
|
-
| `float` | Floating point | `min`, `max`, `precision` |
|
|
172
|
+
| `int` | Integer | `min`, `max`, `unique`, `values` |
|
|
173
|
+
| `float` | Floating point | `min`, `max`, `precision`, `unique`, `values` |
|
|
174
174
|
| `string` | Text | `preset`, `values`, `pattern`, `unique`, `min_length`, `max_length` |
|
|
175
175
|
| `bool` | Boolean | — |
|
|
176
|
-
| `date` | Date | `min`, `max` (ISO format) |
|
|
177
|
-
| `datetime` | Datetime | `min`, `max` (ISO format) |
|
|
176
|
+
| `date` | Date | `min`, `max` (ISO format), `unique` |
|
|
177
|
+
| `datetime` | Datetime | `min`, `max` (ISO format), `unique` |
|
|
178
178
|
|
|
179
179
|
### Float precision
|
|
180
180
|
|
|
@@ -269,7 +269,7 @@ fauxdata generate schemas/people.yml --out - --format jsonl | wc -l
|
|
|
269
269
|
| `--rows` | `-r` | from schema | Number of rows to generate |
|
|
270
270
|
| `--out` | `-o` | from schema | Output path — use `-` for stdout |
|
|
271
271
|
| `--format` | `-f` | from schema | Output format: `csv`, `parquet`, `json`, `jsonl`, `jsonlines` |
|
|
272
|
-
| `--seed` | `-s` | from schema | Random seed for reproducibility |
|
|
272
|
+
| `--seed` | `-s` | from schema | Random seed for reproducibility: same seed, same data, on the same pointblank version |
|
|
273
273
|
| `--validate` | `-v` | off | Run validation rules after generating |
|
|
274
274
|
| `--json` | | off | Print only a JSON summary to stdout (see [Using fauxdata with AI agents](#using-fauxdata-with-ai-agents)) |
|
|
275
275
|
|
|
@@ -277,7 +277,7 @@ When `--out -` is used, all output messages are suppressed and only data is writ
|
|
|
277
277
|
|
|
278
278
|
### `fauxdata infer DATASET`
|
|
279
279
|
|
|
280
|
-
Build a YAML schema from a **real** dataset. fauxdata inspects the table (via pointblank's schema inference) and writes a schema with inferred ranges, categorical value sets, string presets (email, url…), uniqueness, null rates, and string lengths — plus matching `validation:` rules. Feed the result back to `fauxdata generate` to produce synthetic data
|
|
280
|
+
Build a YAML schema from a **real** dataset. fauxdata inspects the table (via pointblank's schema inference) and writes a schema with inferred ranges, categorical value sets, string presets (email, url…), uniqueness, null rates, and string lengths — plus matching `validation:` rules. Feed the result back to `fauxdata generate` to produce synthetic data with the same columns, ranges and value sets, without copying the real rows. Each column is generated independently, so correlations between columns, rules such as `end_date > start_date`, and category frequencies are not preserved.
|
|
281
281
|
|
|
282
282
|
```
|
|
283
283
|
fauxdata infer real_data.csv # -> real_data.yml
|
|
@@ -296,7 +296,7 @@ fauxdata infer big.parquet --sample-size 10000 --out - | head # stream to stdou
|
|
|
296
296
|
| `--detect-presets` / `--no-detect-presets` | | on | Match string columns to known presets (email, url…) |
|
|
297
297
|
| `--sample-size` | | all rows | Sample N rows before analysis (for very large tables) |
|
|
298
298
|
|
|
299
|
-
> **Notes.** Low-cardinality columns (≤ `--categorical-threshold` distinct values) are frozen to their exact source values via `values:` — desirable for categories like `status`, but it means sensitive low-cardinality columns are reproduced verbatim. Raise the threshold or edit the schema if that's not what you want. Also, a column inferred as `unique` over a narrow integer range cannot generate many more rows than the source; widen `min`/`max` or drop `unique` to amplify.
|
|
299
|
+
> **Notes.** Low-cardinality columns (≤ `--categorical-threshold` distinct values) are frozen to their exact source values via `values:` — desirable for categories like `status`, but it means sensitive low-cardinality columns are reproduced verbatim. Raise the threshold or edit the schema if that's not what you want. This also applies to columns whose name suggests a place, company or job (`region`, `city`, `company`...). Personal presets are never frozen, so a low-cardinality column such as `product_name` still gets people's names: use `--no-detect-presets` or edit the schema. Also, a column inferred as `unique` over a narrow integer range cannot generate many more rows than the source; widen `min`/`max` or drop `unique` to amplify.
|
|
300
300
|
|
|
301
301
|
### `fauxdata validate DATASET SCHEMA`
|
|
302
302
|
|
|
@@ -365,13 +365,13 @@ fauxdata validate data.csv schema.yml --json
|
|
|
365
365
|
|
|
366
366
|
## Example schemas
|
|
367
367
|
|
|
368
|
-
Three ready-to-use schemas are included in `schemas
|
|
368
|
+
Three ready-to-use schemas are included in [`schemas/`](https://github.com/aborruso/fauxdata/tree/main/schemas):
|
|
369
369
|
|
|
370
370
|
| Schema | Domain | Columns |
|
|
371
371
|
|--------|--------|---------|
|
|
372
|
-
| `people.yml` | Personal data | id, name, email, age, city, country_code, active, signup_date, score |
|
|
373
|
-
| `orders.yml` | E-commerce | order_id, customer_id, product, amount, status, created_at |
|
|
374
|
-
| `events.yml` | Analytics | event_id, user_id, event_type, timestamp, ip, user_agent, session_duration |
|
|
372
|
+
| [`people.yml`](https://github.com/aborruso/fauxdata/blob/main/schemas/people.yml) | Personal data | id, name, email, age, city, country_code, active, signup_date, score |
|
|
373
|
+
| [`orders.yml`](https://github.com/aborruso/fauxdata/blob/main/schemas/orders.yml) | E-commerce | order_id, customer_id, product, amount, status, created_at |
|
|
374
|
+
| [`events.yml`](https://github.com/aborruso/fauxdata/blob/main/schemas/events.yml) | Analytics | event_id, user_id, event_type, timestamp, ip, user_agent, session_duration |
|
|
375
375
|
|
|
376
376
|
---
|
|
377
377
|
|
|
@@ -69,7 +69,7 @@ fauxdata init --name orders
|
|
|
69
69
|
|
|
70
70
|
## Schema format
|
|
71
71
|
|
|
72
|
-
A schema is a YAML file that describes the structure of your dataset. Here is a realistic example for a people dataset:
|
|
72
|
+
A schema is a YAML file that describes the structure of your dataset. Here is a realistic example for a people dataset, adapted from [`schemas/people.yml`](https://github.com/aborruso/fauxdata/blob/main/schemas/people.yml):
|
|
73
73
|
|
|
74
74
|
```yaml
|
|
75
75
|
name: people
|
|
@@ -145,12 +145,12 @@ validation:
|
|
|
145
145
|
|
|
146
146
|
| Type | Description | Options |
|
|
147
147
|
|------|-------------|---------|
|
|
148
|
-
| `int` | Integer | `min`, `max`, `unique` |
|
|
149
|
-
| `float` | Floating point | `min`, `max`, `precision` |
|
|
148
|
+
| `int` | Integer | `min`, `max`, `unique`, `values` |
|
|
149
|
+
| `float` | Floating point | `min`, `max`, `precision`, `unique`, `values` |
|
|
150
150
|
| `string` | Text | `preset`, `values`, `pattern`, `unique`, `min_length`, `max_length` |
|
|
151
151
|
| `bool` | Boolean | — |
|
|
152
|
-
| `date` | Date | `min`, `max` (ISO format) |
|
|
153
|
-
| `datetime` | Datetime | `min`, `max` (ISO format) |
|
|
152
|
+
| `date` | Date | `min`, `max` (ISO format), `unique` |
|
|
153
|
+
| `datetime` | Datetime | `min`, `max` (ISO format), `unique` |
|
|
154
154
|
|
|
155
155
|
### Float precision
|
|
156
156
|
|
|
@@ -245,7 +245,7 @@ fauxdata generate schemas/people.yml --out - --format jsonl | wc -l
|
|
|
245
245
|
| `--rows` | `-r` | from schema | Number of rows to generate |
|
|
246
246
|
| `--out` | `-o` | from schema | Output path — use `-` for stdout |
|
|
247
247
|
| `--format` | `-f` | from schema | Output format: `csv`, `parquet`, `json`, `jsonl`, `jsonlines` |
|
|
248
|
-
| `--seed` | `-s` | from schema | Random seed for reproducibility |
|
|
248
|
+
| `--seed` | `-s` | from schema | Random seed for reproducibility: same seed, same data, on the same pointblank version |
|
|
249
249
|
| `--validate` | `-v` | off | Run validation rules after generating |
|
|
250
250
|
| `--json` | | off | Print only a JSON summary to stdout (see [Using fauxdata with AI agents](#using-fauxdata-with-ai-agents)) |
|
|
251
251
|
|
|
@@ -253,7 +253,7 @@ When `--out -` is used, all output messages are suppressed and only data is writ
|
|
|
253
253
|
|
|
254
254
|
### `fauxdata infer DATASET`
|
|
255
255
|
|
|
256
|
-
Build a YAML schema from a **real** dataset. fauxdata inspects the table (via pointblank's schema inference) and writes a schema with inferred ranges, categorical value sets, string presets (email, url…), uniqueness, null rates, and string lengths — plus matching `validation:` rules. Feed the result back to `fauxdata generate` to produce synthetic data
|
|
256
|
+
Build a YAML schema from a **real** dataset. fauxdata inspects the table (via pointblank's schema inference) and writes a schema with inferred ranges, categorical value sets, string presets (email, url…), uniqueness, null rates, and string lengths — plus matching `validation:` rules. Feed the result back to `fauxdata generate` to produce synthetic data with the same columns, ranges and value sets, without copying the real rows. Each column is generated independently, so correlations between columns, rules such as `end_date > start_date`, and category frequencies are not preserved.
|
|
257
257
|
|
|
258
258
|
```
|
|
259
259
|
fauxdata infer real_data.csv # -> real_data.yml
|
|
@@ -272,7 +272,7 @@ fauxdata infer big.parquet --sample-size 10000 --out - | head # stream to stdou
|
|
|
272
272
|
| `--detect-presets` / `--no-detect-presets` | | on | Match string columns to known presets (email, url…) |
|
|
273
273
|
| `--sample-size` | | all rows | Sample N rows before analysis (for very large tables) |
|
|
274
274
|
|
|
275
|
-
> **Notes.** Low-cardinality columns (≤ `--categorical-threshold` distinct values) are frozen to their exact source values via `values:` — desirable for categories like `status`, but it means sensitive low-cardinality columns are reproduced verbatim. Raise the threshold or edit the schema if that's not what you want. Also, a column inferred as `unique` over a narrow integer range cannot generate many more rows than the source; widen `min`/`max` or drop `unique` to amplify.
|
|
275
|
+
> **Notes.** Low-cardinality columns (≤ `--categorical-threshold` distinct values) are frozen to their exact source values via `values:` — desirable for categories like `status`, but it means sensitive low-cardinality columns are reproduced verbatim. Raise the threshold or edit the schema if that's not what you want. This also applies to columns whose name suggests a place, company or job (`region`, `city`, `company`...). Personal presets are never frozen, so a low-cardinality column such as `product_name` still gets people's names: use `--no-detect-presets` or edit the schema. Also, a column inferred as `unique` over a narrow integer range cannot generate many more rows than the source; widen `min`/`max` or drop `unique` to amplify.
|
|
276
276
|
|
|
277
277
|
### `fauxdata validate DATASET SCHEMA`
|
|
278
278
|
|
|
@@ -341,13 +341,13 @@ fauxdata validate data.csv schema.yml --json
|
|
|
341
341
|
|
|
342
342
|
## Example schemas
|
|
343
343
|
|
|
344
|
-
Three ready-to-use schemas are included in `schemas
|
|
344
|
+
Three ready-to-use schemas are included in [`schemas/`](https://github.com/aborruso/fauxdata/tree/main/schemas):
|
|
345
345
|
|
|
346
346
|
| Schema | Domain | Columns |
|
|
347
347
|
|--------|--------|---------|
|
|
348
|
-
| `people.yml` | Personal data | id, name, email, age, city, country_code, active, signup_date, score |
|
|
349
|
-
| `orders.yml` | E-commerce | order_id, customer_id, product, amount, status, created_at |
|
|
350
|
-
| `events.yml` | Analytics | event_id, user_id, event_type, timestamp, ip, user_agent, session_duration |
|
|
348
|
+
| [`people.yml`](https://github.com/aborruso/fauxdata/blob/main/schemas/people.yml) | Personal data | id, name, email, age, city, country_code, active, signup_date, score |
|
|
349
|
+
| [`orders.yml`](https://github.com/aborruso/fauxdata/blob/main/schemas/orders.yml) | E-commerce | order_id, customer_id, product, amount, status, created_at |
|
|
350
|
+
| [`events.yml`](https://github.com/aborruso/fauxdata/blob/main/schemas/events.yml) | Analytics | event_id, user_id, event_type, timestamp, ip, user_agent, session_duration |
|
|
351
351
|
|
|
352
352
|
---
|
|
353
353
|
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
[project]
|
|
2
2
|
name = "fauxdata-cli"
|
|
3
|
-
version = "0.2.
|
|
3
|
+
version = "0.2.2"
|
|
4
4
|
description = "CLI for generating and validating fake datasets"
|
|
5
5
|
readme = "README.md"
|
|
6
6
|
requires-python = ">=3.11"
|
|
@@ -13,7 +13,7 @@ classifiers = [
|
|
|
13
13
|
"Operating System :: OS Independent",
|
|
14
14
|
]
|
|
15
15
|
dependencies = [
|
|
16
|
-
"pointblank>=1.0
|
|
16
|
+
"pointblank>=1.2.0",
|
|
17
17
|
"typer>=0.12",
|
|
18
18
|
"rich>=13",
|
|
19
19
|
"pyfiglet>=1.0",
|
|
@@ -46,3 +46,6 @@ build-backend = "hatchling.build"
|
|
|
46
46
|
|
|
47
47
|
[tool.hatch.build.targets.wheel]
|
|
48
48
|
packages = ["src/fauxdata"]
|
|
49
|
+
|
|
50
|
+
[tool.hatch.build.targets.sdist]
|
|
51
|
+
only-include = ["src", "tests", "schemas", "README.md", "LICENSE", "pyproject.toml"]
|
|
@@ -139,7 +139,7 @@ Notes:
|
|
|
139
139
|
|
|
140
140
|
{presets}
|
|
141
141
|
|
|
142
|
-
Presets are locale-aware and coherent within a row (the email matches the name, the city matches the country).
|
|
142
|
+
Presets are locale-aware and coherent within a row (the email matches the name, the city matches the country).
|
|
143
143
|
|
|
144
144
|
## Validation rules
|
|
145
145
|
|
|
@@ -31,6 +31,11 @@ _DTYPE_MAP = {
|
|
|
31
31
|
"Date": "date",
|
|
32
32
|
}
|
|
33
33
|
|
|
34
|
+
# Presets pointblank matches on the column name alone, skipping categorical detection.
|
|
35
|
+
# Personal presets (name, phone, address...) are left out on purpose: restoring their
|
|
36
|
+
# source values would copy real people's data into the schema.
|
|
37
|
+
_CATEGORICAL_PRESETS = {"state", "city", "country", "country_code_2", "currency_code", "company", "job"}
|
|
38
|
+
|
|
34
39
|
|
|
35
40
|
def _col_type_from_dtype(dtype: str) -> str:
|
|
36
41
|
"""Map a pointblank/polars dtype name to a fauxdata col_type."""
|
|
@@ -83,6 +88,19 @@ def _field_to_spec(field) -> dict:
|
|
|
83
88
|
return spec
|
|
84
89
|
|
|
85
90
|
|
|
91
|
+
def _restore_categorical(spec: dict, series: pl.Series, threshold: int | float) -> dict:
|
|
92
|
+
"""Replace a name-matched preset with the source values on a low-cardinality column."""
|
|
93
|
+
if spec.get("preset") not in _CATEGORICAL_PRESETS or spec.get("unique"):
|
|
94
|
+
return spec
|
|
95
|
+
limit = threshold if isinstance(threshold, int) else int(threshold * series.len())
|
|
96
|
+
values = series.drop_nulls().unique().sort().to_list()
|
|
97
|
+
if len(values) > limit:
|
|
98
|
+
return spec
|
|
99
|
+
del spec["preset"]
|
|
100
|
+
spec["values"] = values
|
|
101
|
+
return spec
|
|
102
|
+
|
|
103
|
+
|
|
86
104
|
def _build_validation(columns: list[tuple[str, dict]]) -> list[dict]:
|
|
87
105
|
"""Derive validation rules from the inferred column specs."""
|
|
88
106
|
rules: list[dict] = []
|
|
@@ -160,14 +178,18 @@ def run(
|
|
|
160
178
|
if not json_out:
|
|
161
179
|
rprint(f" Loaded [bold]{len(df)}[/bold] rows, [bold]{len(df.columns)}[/bold] columns")
|
|
162
180
|
|
|
181
|
+
threshold = _threshold(categorical_threshold)
|
|
163
182
|
pb_schema = pb.schema_from_tbl(
|
|
164
183
|
df,
|
|
165
|
-
categorical_threshold=
|
|
184
|
+
categorical_threshold=threshold,
|
|
166
185
|
detect_presets=detect_presets,
|
|
167
186
|
sample_size=sample_size,
|
|
168
187
|
)
|
|
169
188
|
|
|
170
|
-
columns = [
|
|
189
|
+
columns = [
|
|
190
|
+
(col_name, _restore_categorical(_field_to_spec(field), df[col_name], threshold))
|
|
191
|
+
for col_name, field in pb_schema.columns
|
|
192
|
+
]
|
|
171
193
|
|
|
172
194
|
schema_name = name or Path(dataset_path).stem
|
|
173
195
|
out_rows = rows if rows is not None else len(df)
|
|
@@ -80,8 +80,7 @@ def _add_rule(v: pb.Validate, rule: ValidationRule) -> None:
|
|
|
80
80
|
|
|
81
81
|
def _extract_results(v: pb.Validate, rules: list[ValidationRule]) -> list[dict]:
|
|
82
82
|
"""Extract per-step results as a list of dicts."""
|
|
83
|
-
# validation_info,
|
|
84
|
-
# columns (rows_distinct) in pointblank 1.0.1
|
|
83
|
+
# validation_info carries active and eval_error, used to flag steps that could not run
|
|
85
84
|
steps = {s.i: s for s in v.validation_info}
|
|
86
85
|
results = []
|
|
87
86
|
# Build a flat list of steps (one per column for multi-column rules)
|
|
@@ -103,6 +103,15 @@ def test_generate_float_precision_stays_in_range():
|
|
|
103
103
|
assert df["f"].max() <= 9.999
|
|
104
104
|
|
|
105
105
|
|
|
106
|
+
def test_generate_unique_coherent_presets():
|
|
107
|
+
"""unique is enforced on coherent presets (pointblank >= 1.1.0, posit-dev/pointblank#441)."""
|
|
108
|
+
schema = _make_schema([ColumnSchema(name="name", col_type="string", preset="name", unique=True),
|
|
109
|
+
ColumnSchema(name="email", col_type="string", preset="email", unique=True)])
|
|
110
|
+
df = generate_dataset(schema, rows=2000, seed=1)
|
|
111
|
+
assert df["name"].n_unique() == 2000
|
|
112
|
+
assert df["email"].n_unique() == 2000
|
|
113
|
+
|
|
114
|
+
|
|
106
115
|
def test_generate_float_precision_unique():
|
|
107
116
|
"""A unique float column stays unique after rounding."""
|
|
108
117
|
schema = _make_schema([ColumnSchema(name="f", col_type="float", min=0.0, max=1.0,
|
|
@@ -220,3 +220,70 @@ def test_infer_roundtrip_nulls_and_low_cardinality_int(tmp_path):
|
|
|
220
220
|
df = generate_dataset(schema, rows=500, seed=3)
|
|
221
221
|
all_passed, results = validate_dataset(df, schema)
|
|
222
222
|
assert all_passed, [r for r in results if not r["ok"]]
|
|
223
|
+
|
|
224
|
+
|
|
225
|
+
# --- name-matched presets on categorical columns ---
|
|
226
|
+
|
|
227
|
+
def _infer_spec(tmp_path, column, values, **kwargs):
|
|
228
|
+
src = tmp_path / "real.csv"
|
|
229
|
+
pl.DataFrame({column: values}).write_csv(src)
|
|
230
|
+
out = tmp_path / "inferred.yml"
|
|
231
|
+
infer_mod.run(dataset_path=str(src), out=str(out), **kwargs)
|
|
232
|
+
return yaml.safe_load(out.read_text()), out
|
|
233
|
+
|
|
234
|
+
|
|
235
|
+
@pytest.mark.parametrize("column", ["region", "state", "city", "country", "currency", "company", "job"])
|
|
236
|
+
def test_infer_low_cardinality_keeps_source_values(tmp_path, column):
|
|
237
|
+
"""A column matched to a preset only by its name stays categorical when values repeat."""
|
|
238
|
+
source = ["nord", "centro", "sud"] * 20
|
|
239
|
+
schema, out = _infer_spec(tmp_path, column, source)
|
|
240
|
+
spec = schema["columns"][column]
|
|
241
|
+
assert "preset" not in spec
|
|
242
|
+
assert sorted(spec["values"]) == ["centro", "nord", "sud"]
|
|
243
|
+
|
|
244
|
+
loaded = load_schema(out)
|
|
245
|
+
df = generate_dataset(loaded, rows=200, seed=1)
|
|
246
|
+
assert set(df[column].to_list()) <= {"nord", "centro", "sud"}
|
|
247
|
+
all_passed, results = validate_dataset(df, loaded)
|
|
248
|
+
assert all_passed, [r for r in results if not r["ok"]]
|
|
249
|
+
|
|
250
|
+
|
|
251
|
+
@pytest.mark.parametrize("column,preset", [("name", "name"), ("first_name", "first_name"), ("phone", "phone_number"), ("address", "address")])
|
|
252
|
+
def test_infer_personal_preset_never_frozen(tmp_path, column, preset):
|
|
253
|
+
"""Personal presets are kept, so real people's values never reach the schema."""
|
|
254
|
+
schema, out = _infer_spec(tmp_path, column, ["Mario Rossi", "Anna Bianchi"] * 20)
|
|
255
|
+
assert schema["columns"][column]["preset"] == preset
|
|
256
|
+
assert "Mario Rossi" not in out.read_text()
|
|
257
|
+
|
|
258
|
+
|
|
259
|
+
def test_infer_high_cardinality_keeps_preset(tmp_path):
|
|
260
|
+
schema, _ = _infer_spec(tmp_path, "city", [f"city {i % 30}" for i in range(60)])
|
|
261
|
+
assert schema["columns"]["city"]["preset"] == "city"
|
|
262
|
+
|
|
263
|
+
|
|
264
|
+
def test_infer_unique_column_keeps_preset(tmp_path):
|
|
265
|
+
schema, _ = _infer_spec(tmp_path, "city", [f"city {i}" for i in range(10)])
|
|
266
|
+
spec = schema["columns"]["city"]
|
|
267
|
+
assert spec["preset"] == "city"
|
|
268
|
+
assert spec["unique"] is True
|
|
269
|
+
|
|
270
|
+
|
|
271
|
+
def test_infer_fractional_threshold_applies_to_preset_columns(tmp_path):
|
|
272
|
+
values = [f"r{i % 8}" for i in range(100)]
|
|
273
|
+
assert "values" in _infer_spec(tmp_path, "region", values, categorical_threshold=0.1)[0]["columns"]["region"]
|
|
274
|
+
assert "preset" in _infer_spec(tmp_path, "region", values, categorical_threshold=0.05)[0]["columns"]["region"]
|
|
275
|
+
|
|
276
|
+
|
|
277
|
+
def test_infer_no_detect_presets_unchanged(tmp_path):
|
|
278
|
+
schema, _ = _infer_spec(tmp_path, "region", ["a", "b"] * 10, detect_presets=False)
|
|
279
|
+
assert sorted(schema["columns"]["region"]["values"]) == ["a", "b"]
|
|
280
|
+
|
|
281
|
+
|
|
282
|
+
def test_infer_categorical_with_nulls_keeps_nullable(tmp_path):
|
|
283
|
+
schema, out = _infer_spec(tmp_path, "region", ["nord", None, "sud", "nord"] * 10)
|
|
284
|
+
spec = schema["columns"]["region"]
|
|
285
|
+
assert sorted(spec["values"]) == ["nord", "sud"]
|
|
286
|
+
assert spec["nullable"] is True
|
|
287
|
+
loaded = load_schema(out)
|
|
288
|
+
all_passed, results = validate_dataset(generate_dataset(loaded, rows=200, seed=1), loaded)
|
|
289
|
+
assert all_passed, [r for r in results if not r["ok"]]
|
|
@@ -1 +0,0 @@
|
|
|
1
|
-
3.12
|
fauxdata_cli-0.2.0/LOG.md
DELETED
|
@@ -1,84 +0,0 @@
|
|
|
1
|
-
# Log
|
|
2
|
-
|
|
3
|
-
## 2026-10-11 — 0.2.0 (non rilasciata): pointblank 1.0.1, `precision` nativa, `col_vals_str_len`, fix, comando `agent` e `--json`
|
|
4
|
-
|
|
5
|
-
- Bump `pointblank>=1.0.1` (da 0.25.0). 103/103 test invariati; schemi inclusi PASS; stesso seed → CSV identici byte per byte tra 0.25 e 1.0.1. Effetto collaterale: da 0.27 pointblank richiede `anthropic[bedrock]` → 20 pacchetti in più installati (anthropic, boto3, pydantic, …), non usati da fauxdata.
|
|
6
|
-
- `precision` float: ora nativa (`pb.float_field(precision=)`), unicità garantita dopo l'arrotondamento. Limiti stretti verso l'interno sulla griglia dei decimali (`max: 9.999`, `precision: 2` → `9.99`) perché l'arrotondamento non sfori il range; errore chiaro se nessun valore ci sta. Documentata nel README (prima non lo era).
|
|
7
|
-
- Nuova regola `col_vals_str_len` (`column`, `min`/`max`, null ignorati). `infer` la emette solo per stringhe libere: su preset/pattern/values il generatore ignora le lunghezze.
|
|
8
|
-
- Fix: le regole sui valori (`between`, `gt/lt/ge/le`, `regex`, `in_set`) ignorano i null (`na_pass=True`; `in_set` via `None` nel set). Prima una colonna col 20% di null falliva 195/1000 righe.
|
|
9
|
-
- Fix: `infer --categorical-threshold` da CLI arrivava come float (anche il default) e pointblank lo leggeva come frazione → ogni colonna stringa senza preset diventava categorica con i valori reali copiati in `values:`. Ora gli interi ≥ 1 passano come conteggio.
|
|
10
|
-
- Fix: `values` sulle colonne `int`/`float` ora rispettato in generazione (passato come `allowed`); prima `infer` lo emetteva per gli int a bassa cardinalità e il generatore lo ignorava (403/500 fallimenti in round-trip).
|
|
11
|
-
- Fix: `rows_distinct` esclude le righe con null nelle colonne controllate (come SQL UNIQUE); prima i null contavano come duplicati tra loro.
|
|
12
|
-
- Bug di pointblank (non nostro): `unique=True` ignorato sui preset coerenti (`email`, `name`, `city`, …); segnalato in https://github.com/posit-dev/pointblank/issues/441. Non corretto: `unique` + `nullable` su range stretto non generabile.
|
|
13
|
-
- Nuovo comando `fauxdata agent`: guida Markdown per agenti AI (flusso, output JSON, exit code, schema, tipi, preset, regole, esempio); liste generate dalle costanti di `schema.py`.
|
|
14
|
-
- Nuovo flag `--json` su `generate`, `validate`, `infer`, `preview`: stdout contiene solo un oggetto JSON; errori come `{"error": ...}` con exit 1. Sezione README "Using fauxdata with AI agents".
|
|
15
|
-
- Fix: un passo su una colonna assente dal dataset risultava `ok: true` con 0/0 righe; ora campo `evaluated` (da `validation_info`), `ok: false`, `NOT RUN` in tabella. `get_dataframe_report()` non usabile: fallisce con `rows_distinct`/`rows_complete` + `columns_subset` (segnalato in https://github.com/posit-dev/pointblank/issues/442). Chiude la issue #1 di fauxdata (output strutturato → `--json`).
|
|
16
|
-
- Sito `docs/index.html`: badge pointblank 1.0+, "six commands", righe `infer` e `agent`, `--json`.
|
|
17
|
-
- 143/143 test, coverage 87.05%.
|
|
18
|
-
|
|
19
|
-
## 2026-07-04 — v0.1.6
|
|
20
|
-
|
|
21
|
-
- Release `0.1.6`: include il fix issue #2 (`precision` float). Build con `uv build`, tag `v0.1.6`.
|
|
22
|
-
- Deploy PyPI **manuale** (`uv publish`): nessun workflow di release automatico.
|
|
23
|
-
|
|
24
|
-
## 2026-07-04 — fix issue #2: `precision` float ignorato
|
|
25
|
-
|
|
26
|
-
- Bug: l'opzione `precision` dei float veniva letta nello schema ma mai applicata (`pb.float_field` non ha il parametro). Valori generati con precisione piena.
|
|
27
|
-
- Fix in `generator.py`: dopo `pb.generate_dataset`, arrotondo con `pl.col(...).round(precision)` le colonne `float` con `precision` impostato.
|
|
28
|
-
- Verifica: reproducer della issue → output atteso (`27.74, 169.64, …`). Test `test_generate_float_precision`. 103/103 pass.
|
|
29
|
-
|
|
30
|
-
## 2026-06-12 — pointblank 0.25 + comando `infer`
|
|
31
|
-
|
|
32
|
-
- Bump `pointblank>=0.25` (uscito su PyPI). Schemi inclusi (people/orders/events) verificati: tutti PASS.
|
|
33
|
-
- Fase B regressioni: `rows_distinct` ora include righe con null (correttezza migliorata), lazy frame ok. fauxdata passa sempre DataFrame eager → nessun impatto.
|
|
34
|
-
- Nuovo comando **`fauxdata infer DATASET`**: da tabella reale (csv/parquet/json/jsonl) genera schema YAML (`columns:` + `validation:`) via `pb.schema_from_tbl`. Flag: `--out/-o`, `--name/-n`, `--rows/-r`, `--format/-f`, `--categorical-threshold`, `--detect-presets/--no-detect-presets`, `--sample-size`.
|
|
35
|
-
- `ColumnSchema`: nuovi campi `min_length`/`max_length` per stringhe (round-trip fedele); passati a `pb.string_field`. Validazione parsing (negativi, min>max).
|
|
36
|
-
- infer CSV: `try_parse_dates=True` → date/datetime tipizzati invece che congelati come categorici.
|
|
37
|
-
- generate: errore chiaro su conflitto unique+range stretto ("too narrow a range") invece del traceback criptico di pointblank.
|
|
38
|
-
- Limite noto: colonne a bassa cardinalità congelate ai valori reali (possibile leak su colonne sensibili) — documentato nel README.
|
|
39
|
-
- Test: +14 (`tests/test_infer.py` + length in `test_new_fields.py`). 102/102 pass, coverage 83.2%.
|
|
40
|
-
|
|
41
|
-
- `init`: aggiunto `--description`, `--rows`, `--format`, `--yes`; questionary diventa fallback
|
|
42
|
-
- Tutti i comandi: `epilog` con esempi in `--help`
|
|
43
|
-
- `generate`: aggiunto `--dry-run` (mostra piano senza scrivere)
|
|
44
|
-
- `generate`: output su successo mostra `output_path`, `format`, `rows` come chiave: valore
|
|
45
|
-
- `init`: output su successo aggiunge `schema_path: <file>`
|
|
46
|
-
- Fix venv: ricreato con `uv venv --clear` (cartella rinominata da `real_fake_datasets`)
|
|
47
|
-
|
|
48
|
-
## 2026-03-06 — v0.1.3
|
|
49
|
-
|
|
50
|
-
- Add Python classifiers to pyproject.toml (3.11, 3.12, 3.13, MIT) — fixes pyversions badge
|
|
51
|
-
- Fix version tests to read `__version__` dynamically instead of hardcoded string
|
|
52
|
-
|
|
53
|
-
## 2026-03-06 — v0.1.2
|
|
54
|
-
|
|
55
|
-
- Bump to 0.1.2 and publish to PyPI
|
|
56
|
-
|
|
57
|
-
## 2026-03-06 (feature)
|
|
58
|
-
|
|
59
|
-
- `--version` / `-V` flag nel CLI (`fauxdata --version` → `fauxdata 0.1.1`)
|
|
60
|
-
- Coverage threshold 80% in pytest config (`--cov-fail-under=80`); attuale: 83.76%
|
|
61
|
-
- Nuovo campo `pattern` in ColumnSchema: genera stringhe che matchano un regex via pointblank
|
|
62
|
-
- Nuovo campo `null_probability` in ColumnSchema: controllo granulare dei null (0.0–1.0), con validazione in parsing
|
|
63
|
-
- Rimossa dipendenza `faker` (non usata, pointblank gestisce tutto)
|
|
64
|
-
- Fix generator: `null_probability=None` non passato a pointblank (causa TypeError)
|
|
65
|
-
- Test aggiornati: 79/79 pass
|
|
66
|
-
|
|
67
|
-
## 2026-03-06 (tests)
|
|
68
|
-
|
|
69
|
-
- Add pytest test suite: 65 tests, 100% pass, 0.44s
|
|
70
|
-
- `tests/test_schema.py`: unit tests for YAML schema parsing (valid/invalid cases)
|
|
71
|
-
- `tests/test_output.py`: unit tests for export functions (all formats, stdout, errors)
|
|
72
|
-
- `tests/test_generator.py`: integration tests for generation (types, seed, unique, presets)
|
|
73
|
-
- `tests/test_validator.py`: integration tests for validation rules (pass/fail scenarios)
|
|
74
|
-
- `tests/test_cli.py`: CLI smoke tests via `typer.testing.CliRunner`
|
|
75
|
-
- Add `[dependency-groups] dev` in `pyproject.toml` (pytest, pytest-cov); config via `[tool.pytest.ini_options]`
|
|
76
|
-
|
|
77
|
-
## 2026-03-06
|
|
78
|
-
|
|
79
|
-
- Initial implementation of `fauxdata` CLI
|
|
80
|
-
- Stack: pointblank 0.22 (native generation + validation), polars, typer, rich, pyfiglet, questionary
|
|
81
|
-
- Commands: `init`, `generate`, `validate`, `preview`
|
|
82
|
-
- Example schemas: `people.yml`, `orders.yml`, `events.yml`
|
|
83
|
-
- All schemas generate and validate cleanly (all rules PASS)
|
|
84
|
-
- `locale` field at schema level maps to pointblank `country=` param
|
fauxdata_cli-0.2.0/RELEASING.md
DELETED
|
@@ -1,83 +0,0 @@
|
|
|
1
|
-
# Releasing
|
|
2
|
-
|
|
3
|
-
Procedura per pubblicare una nuova release di `fauxdata` su PyPI.
|
|
4
|
-
Il deploy **non è automatico**: nessun workflow GitHub Actions. Ogni step è manuale.
|
|
5
|
-
|
|
6
|
-
## Versioning
|
|
7
|
-
|
|
8
|
-
`MAJOR.MINOR.PATCH` (SemVer).
|
|
9
|
-
|
|
10
|
-
- **subrelease / patch** (`0.1.5` → `0.1.6`): bugfix, nessuna modifica di API.
|
|
11
|
-
- **minor** (`0.1.x` → `0.2.0`): nuove feature retrocompatibili.
|
|
12
|
-
- **major** (`0.x` → `1.0.0`): breaking change.
|
|
13
|
-
|
|
14
|
-
## Step
|
|
15
|
-
|
|
16
|
-
1. **Aggiorna la versione** nei due file (devono restare allineati):
|
|
17
|
-
- `pyproject.toml` → `version = "X.Y.Z"`
|
|
18
|
-
- `src/fauxdata/__init__.py` → `__version__ = "X.Y.Z"`
|
|
19
|
-
|
|
20
|
-
2. **Aggiorna `LOG.md`**: nuova sezione in cima con data `YYYY-MM-DD` e le modifiche.
|
|
21
|
-
|
|
22
|
-
3. **Test**: devono passare tutti.
|
|
23
|
-
|
|
24
|
-
```bash
|
|
25
|
-
uv run pytest tests/ -q
|
|
26
|
-
uv run fauxdata --version # deve stampare la nuova versione
|
|
27
|
-
```
|
|
28
|
-
|
|
29
|
-
4. **Build**:
|
|
30
|
-
|
|
31
|
-
```bash
|
|
32
|
-
rm -rf dist/
|
|
33
|
-
uv build
|
|
34
|
-
ls dist/ # fauxdata_cli-X.Y.Z-py3-none-any.whl + .tar.gz
|
|
35
|
-
```
|
|
36
|
-
|
|
37
|
-
5. **Aggiorna la CLI installata localmente** (`fauxdata` è installato come `uv tool`):
|
|
38
|
-
|
|
39
|
-
```bash
|
|
40
|
-
uv tool install --reinstall . # reinstalla dalla working copy
|
|
41
|
-
fauxdata --version # deve stampare la nuova versione
|
|
42
|
-
```
|
|
43
|
-
|
|
44
|
-
6. **Commit + tag + push**:
|
|
45
|
-
|
|
46
|
-
```bash
|
|
47
|
-
git add pyproject.toml src/fauxdata/__init__.py LOG.md
|
|
48
|
-
git commit -m "chore: vX.Y.Z — <sintesi>"
|
|
49
|
-
git tag -a vX.Y.Z -m "vX.Y.Z — <sintesi>"
|
|
50
|
-
git push origin main
|
|
51
|
-
git push origin vX.Y.Z
|
|
52
|
-
```
|
|
53
|
-
|
|
54
|
-
7. **Pubblica su PyPI** (richiede token PyPI — step da eseguire a mano):
|
|
55
|
-
|
|
56
|
-
```bash
|
|
57
|
-
uv publish # oppure: twine upload dist/*
|
|
58
|
-
```
|
|
59
|
-
|
|
60
|
-
Il token si passa via `UV_PUBLISH_TOKEN` o `--token`. Non committare mai il token.
|
|
61
|
-
|
|
62
|
-
8. **Verifica / aggiorna da PyPI** (il pacchetto su PyPI è `fauxdata-cli`):
|
|
63
|
-
|
|
64
|
-
```bash
|
|
65
|
-
uvx fauxdata@X.Y.Z --version # verifica al volo la versione pubblicata
|
|
66
|
-
|
|
67
|
-
pip install --upgrade fauxdata-cli # aggiorna un'installazione pip esistente
|
|
68
|
-
# oppure, se installato come uv tool:
|
|
69
|
-
uv tool upgrade fauxdata-cli
|
|
70
|
-
```
|
|
71
|
-
|
|
72
|
-
9. **GitHub Release**: dal tag già pushato (il tag da solo **non** crea la release).
|
|
73
|
-
**Titolo e note devono essere in inglese.**
|
|
74
|
-
|
|
75
|
-
```bash
|
|
76
|
-
gh release create vX.Y.Z --title "vX.Y.Z" --notes "<changelog in English>"
|
|
77
|
-
```
|
|
78
|
-
|
|
79
|
-
## Note
|
|
80
|
-
|
|
81
|
-
- Se una release chiude una issue, referenziarla nel commit/tag (`#N`) e chiudere la issue con un commento che linka al commit.
|
|
82
|
-
- La versione è testata dinamicamente (`test_cli_version` legge `__version__`), quindi il bump non rompe i test.
|
|
83
|
-
- **Tag ≠ GitHub Release**: `git push origin vX.Y.Z` crea solo il tag; la pagina di release va creata a parte con `gh release create` (step 9).
|
|
@@ -1,90 +0,0 @@
|
|
|
1
|
-
# Deployment rules
|
|
2
|
-
|
|
3
|
-
## Pre-release checklist (always)
|
|
4
|
-
|
|
5
|
-
1. Run tests locally — must all pass:
|
|
6
|
-
|
|
7
|
-
```bash
|
|
8
|
-
uv run pytest
|
|
9
|
-
```
|
|
10
|
-
|
|
11
|
-
Coverage must stay above 80%. If it drops, fix before proceeding.
|
|
12
|
-
|
|
13
|
-
2. Bump version in **both**:
|
|
14
|
-
- `src/fauxdata/__init__.py` → `__version__ = "X.Y.Z"`
|
|
15
|
-
- `pyproject.toml` → `version = "X.Y.Z"`
|
|
16
|
-
|
|
17
|
-
3. Update `LOG.md` with a summary of changes under a new date heading.
|
|
18
|
-
|
|
19
|
-
---
|
|
20
|
-
|
|
21
|
-
## GitHub release (tag + release notes)
|
|
22
|
-
|
|
23
|
-
```bash
|
|
24
|
-
# Create and push annotated tag
|
|
25
|
-
git tag -a vX.Y.Z -m "vX.Y.Z"
|
|
26
|
-
git push origin vX.Y.Z
|
|
27
|
-
```
|
|
28
|
-
|
|
29
|
-
Then create a GitHub release via `gh`:
|
|
30
|
-
|
|
31
|
-
```bash
|
|
32
|
-
gh release create vX.Y.Z \
|
|
33
|
-
--title "vX.Y.Z" \
|
|
34
|
-
--notes "$(cat <<'EOF'
|
|
35
|
-
## What's new
|
|
36
|
-
|
|
37
|
-
- Short bullet list of user-facing changes
|
|
38
|
-
- Include new fields, commands, bug fixes
|
|
39
|
-
|
|
40
|
-
## Breaking changes
|
|
41
|
-
|
|
42
|
-
- List any breaking changes here (or remove section if none)
|
|
43
|
-
|
|
44
|
-
## Installation
|
|
45
|
-
|
|
46
|
-
\`\`\`bash
|
|
47
|
-
pip install fauxdata-cli==X.Y.Z
|
|
48
|
-
\`\`\`
|
|
49
|
-
|
|
50
|
-
Full changelog: https://github.com/aborruso/fauxdata/commits/vX.Y.Z
|
|
51
|
-
EOF
|
|
52
|
-
)"
|
|
53
|
-
```
|
|
54
|
-
|
|
55
|
-
Release notes style: **concise, nerd-friendly, technical**. List the actual changes with enough detail that a developer understands what changed and why.
|
|
56
|
-
|
|
57
|
-
---
|
|
58
|
-
|
|
59
|
-
## PyPI publish (via twine)
|
|
60
|
-
|
|
61
|
-
```bash
|
|
62
|
-
# Build
|
|
63
|
-
uv build
|
|
64
|
-
|
|
65
|
-
# Check the dist
|
|
66
|
-
twine check dist/*
|
|
67
|
-
|
|
68
|
-
# Publish
|
|
69
|
-
twine upload dist/*
|
|
70
|
-
```
|
|
71
|
-
|
|
72
|
-
Requires `~/.pypirc` configured with PyPI token, or set `TWINE_USERNAME`/`TWINE_PASSWORD` env vars.
|
|
73
|
-
|
|
74
|
-
---
|
|
75
|
-
|
|
76
|
-
## Order of operations
|
|
77
|
-
|
|
78
|
-
```
|
|
79
|
-
uv run pytest # must pass 100%
|
|
80
|
-
bump version # __init__.py + pyproject.toml
|
|
81
|
-
update LOG.md
|
|
82
|
-
git commit + git push
|
|
83
|
-
git tag + git push tag
|
|
84
|
-
gh release create # with release notes
|
|
85
|
-
uv build
|
|
86
|
-
twine check dist/*
|
|
87
|
-
twine upload dist/*
|
|
88
|
-
```
|
|
89
|
-
|
|
90
|
-
Never publish to PyPI without a corresponding GitHub release.
|
|
@@ -1,8 +0,0 @@
|
|
|
1
|
-
# Future ideas
|
|
2
|
-
|
|
3
|
-
Ideas from the pointblank 0.26-1.0.1 releases, not yet planned (2026-10-11).
|
|
4
|
-
|
|
5
|
-
- Related tables: generate linked datasets (e.g. people and orders) and check foreign keys with `col_vals_in_table()` (pointblank 0.27).
|
|
6
|
-
- Coded missing values: a schema field for sentinel values like `-99` or `N/A`, generated and validated via `MissingSpec` and `col_pct_missing()` (pointblank 0.26).
|
|
7
|
-
- Schema export: let `infer` also write JSON Schema, Frictionless, dbt or ODCS via `export_contract()` (pointblank 0.26/0.27).
|
|
8
|
-
- Fidelity report: compare the real dataset with the synthetic one using `DataScan.compare()` (PSI, Kolmogorov-Smirnov) to show how close the shapes are (pointblank 0.27).
|