dataforger 0.1.2__tar.gz → 0.1.3__tar.gz

{dataforger-0.1.2/src/dataforger.egg-info → dataforger-0.1.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: dataforger
-Version: 0.1.2
+Version: 0.1.3
 Summary: Generate fake data from Python or the command line.
 Author-email: Karmanshi Shrivastava <karmanshishrivastava20@gmail.com>
 License: MIT License
@@ -98,9 +98,12 @@ dataforger --rows 100 --field name:name --field email:email --field age:age --fo
 | --- | --- | --- |
 | `--field NAME:TYPE` | Output column name and the type of value to generate. Repeat to add fields. | Prompts for fields if omitted |
 | `--rows COUNT` | Number of records. Must be a positive integer. | `10` |
-| `--format {csv,json}` | Output file format. | `json` |
-| `--output PATH` | Path to the output file. | `data.json` or `data.csv`, based on format |
+| `--format {csv,json}` | Output format; in dataset mode, exports saved records. | `json` |
+| `--output PATH` | Path for generated or exported records. | `data.json` or `data.csv`, based on format |
 | `--seed INTEGER` | Seed for repeatable output. | Random |
+| `--dataset KEY` | Reuse or create a persistent JSON dataset instead of writing an output file. | |
+| `--replace-dataset` | Regenerate and replace an existing persistent dataset; requires `--dataset`. | |
+| `--data-dir PATH` | Directory for persistent datasets. | `~/.dataforger/datasets` |
 
 `NAME` is the column/key to write in the output; choose any non-empty name.
 `TYPE` must be one of the supported field types below. For example,
@@ -160,3 +163,101 @@ Its parameters are `fields` (a mapping of output column names to supported
 types), `rows` (positive record count), and optional `seed` (integer for
 repeatable values). `write_data` takes `records`, `output` (path), and
 `format` (`"csv"` or `"json"`).
+
+### Persistent datasets
+
+Persistent datasets let separate CLI runs, Python calls, and REST requests
+reuse exactly the same records. Use a unique key for each dataset:
+
+```console
+dataforger --dataset customers --field id:uuid --field name:name --rows 100 --seed 42
+```
+
+Repeat the same command to load the saved records rather than generating new
+ones. By default, a key cannot be reused with different fields, row counts, or
+seed; Dataforger reports a conflict instead of silently changing the fixture.
+Use `--replace-dataset` to explicitly regenerate it. Dataset keys are limited
+to letters, numbers, dots, underscores, and hyphens.
+To retrieve a dataset without supplying its generation parameters, omit
+`--field`; use `--output` and `--format` to export the saved records:
+
+```console
+dataforger --dataset customers --format csv --output customers.csv
+```
+
+Datasets are stored as JSON files in `~/.dataforger/datasets` by default. Set
+`DATAFORGER_DATA_DIR` to share another location across CLI and Python usage,
+or pass `--data-dir PATH` to the CLI. The API server uses the same configured
+directory; `dataforger --serve --data-dir PATH` sets it for that server.
+Writes are atomic and serialized across threads and processes on the same
+machine. For containerized deployments, mount a persistent volume at the
+configured directory.
+
+The Python API exposes the same operations:
+
+```python
+from dataforger import get_dataset, get_or_create_dataset, replace_dataset
+
+customers = get_or_create_dataset(
+    "customers",
+    {"id": "uuid", "name": "name"},
+    rows=100,
+    seed=42,
+)
+same_customers = get_dataset("customers")
+replacement = replace_dataset(
+    "customers",
+    {"id": "uuid", "name": "name"},
+    rows=100,
+    seed=43,
+)
+```
+
+`get_or_create_dataset` returns an existing dataset only when the generation
+parameters match. It raises `DatasetConflictError` when they differ and
+`DatasetNotFoundError` is raised when `get_dataset` cannot find a key.
+`replace_dataset` explicitly generates and stores a replacement. All three
+functions accept an optional `directory` argument to override the default
+store directory.
+
+## HTTP API
+
+Start the built-in HTTP API with:
+
+```console
+dataforger --serve
+```
+
+The server listens on `127.0.0.1:8000` by default. Use `--host` and `--port`
+to change the bind address and port. The API has no authentication, so keep it
+on localhost unless you put it behind an appropriately secured service.
+
+Generate records with a JSON `POST` request to `/api/v1/generate`:
+
+```console
+curl -X POST http://127.0.0.1:8000/api/v1/generate ^
+  -H "Content-Type: application/json" ^
+  -d "{\"fields\":{\"name\":\"name\",\"email\":\"email\",\"age\":\"age\"},\"rows\":2,\"seed\":42}"
+```
+
+`fields` is a mapping of output names to supported field types, `rows` is
+optional (defaults to `10`, maximum `10000`), and `seed` is optional. The
+request body is limited to 1 MiB and `fields` to 100 entries. Responses use
+JSON; successful generation returns `{"count": 2, "data": [...]}`, and invalid
+requests return a JSON `error` with an appropriate HTTP status.
+
+`GET /api/v1/fields` lists supported field types. `GET /health` returns the
+server health status.
+
+Persistent datasets use:
+
+* `POST /api/v1/datasets` with `key`, `fields`, and optional `rows` and `seed`
+  to create or reuse a dataset. Reusing the key with different generation
+  parameters returns `409 Conflict`.
+* `GET /api/v1/datasets/{key}` to retrieve a saved dataset.
+* `PUT /api/v1/datasets/{key}` with `fields` and optional `rows` and `seed` to
+  explicitly generate or replace the dataset.
+
+These routes use the same JSON file store as the Python and CLI interfaces.
+The API has no authentication; keep it bound to localhost unless protected by
+an appropriately secured service.

dataforger-0.1.3/README.md

@@ -0,0 +1,214 @@
+# dataforger
+
+Generate realistic fake data from Python or the command line, and save it as
+CSV or JSON. Dataforger uses only the Python standard library and has no
+third-party runtime dependencies.
+
+Built-in names, streets, companies, and email domains are stored in
+`src/dataforger/data.json`. Country names and their associated cities are
+stored in `src/dataforger/countries.json`; when generating a record with both
+country and city fields, the city is selected from that country's list.
+
+## Install
+
+Requires Python 3.10 or later.
+
+From a checkout of this repository:
+
+```console
+python -m pip install .
+```
+
+After publishing the package to PyPI:
+
+```console
+python -m pip install dataforger
+```
+
+## Publishing to PyPI
+
+The GitHub Actions workflow runs the tests on Python 3.10 through 3.13 and
+builds wheel and source distributions on pushes and pull requests. To publish,
+configure a PyPI trusted publisher for this GitHub repository, using workflow
+`.github/workflows/publish.yml` and the GitHub environment `pypi`. Then create
+and publish a GitHub Release; the workflow publishes its tested distributions
+to PyPI using trusted publishing.
+
+## Command line
+
+Run `dataforger` without field options for an interactive prompt. Or specify
+all options directly:
+
+```console
+dataforger --rows 100 --field name:name --field email:email --field age:age --format csv --output users.csv --seed 42
+```
+
+### CLI options
+
+| Option | Description | Default |
+| --- | --- | --- |
+| `--field NAME:TYPE` | Output column name and the type of value to generate. Repeat to add fields. | Prompts for fields if omitted |
+| `--rows COUNT` | Number of records. Must be a positive integer. | `10` |
+| `--format {csv,json}` | Output format; in dataset mode, exports saved records. | `json` |
+| `--output PATH` | Path for generated or exported records. | `data.json` or `data.csv`, based on format |
+| `--seed INTEGER` | Seed for repeatable output. | Random |
+| `--dataset KEY` | Reuse or create a persistent JSON dataset instead of writing an output file. | |
+| `--replace-dataset` | Regenerate and replace an existing persistent dataset; requires `--dataset`. | |
+| `--data-dir PATH` | Directory for persistent datasets. | `~/.dataforger/datasets` |
+
+`NAME` is the column/key to write in the output; choose any non-empty name.
+`TYPE` must be one of the supported field types below. For example,
+`--field first_name:first_name` creates a column named `first_name` using
+generated first names, while `--field given:first_name` creates the same kind
+of values under the column name `given`.
+
+### Supported field types
+
+| Type | Generated value |
+| --- | --- |
+| `name` | Full name |
+| `first_name` | First name |
+| `last_name` | Last name |
+| `email` | Email address using a reserved example domain |
+| `phone` | Phone-like number |
+| `address` | Street address including a generated city |
+| `city` | City from the bundled country-to-city data |
+| `country` | Country from the bundled country-to-city data |
+| `company` | Company name |
+| `uuid` | UUID |
+| `date` | Date between 1970-01-01 and today |
+| `age` | Integer from 0 to 100 |
+| `integer` | Integer from 0 to 100,000 |
+| `float` | Decimal value from 0 to 100,000 |
+| `boolean` | `true` or `false` |
+
+When a record requests both `country` and `city`, the city is selected from
+that country's city list. If it also requests `address`, its city matches
+those fields.
+
+## Python API
+
+```python
+from dataforger import generate_data, write_data
+
+records = generate_data(
+    {
+        "first_name": "first_name",
+        "last_name": "last_name",
+        "country": "country",
+        "city": "city",
+        "email": "email",
+        "age": "age",
+    },
+    rows=100,
+    seed=42,
+)
+
+write_data(records, "users.json", "json")
+write_data(records, "users.csv", "csv")
+```
+
+`generate_data` returns a list of dictionaries. Pass the resulting records,
+an output path, and either `"csv"` or `"json"` to `write_data`.
+Its parameters are `fields` (a mapping of output column names to supported
+types), `rows` (positive record count), and optional `seed` (integer for
+repeatable values). `write_data` takes `records`, `output` (path), and
+`format` (`"csv"` or `"json"`).
+
+### Persistent datasets
+
+Persistent datasets let separate CLI runs, Python calls, and REST requests
+reuse exactly the same records. Use a unique key for each dataset:
+
+```console
+dataforger --dataset customers --field id:uuid --field name:name --rows 100 --seed 42
+```
+
+Repeat the same command to load the saved records rather than generating new
+ones. By default, a key cannot be reused with different fields, row counts, or
+seed; Dataforger reports a conflict instead of silently changing the fixture.
+Use `--replace-dataset` to explicitly regenerate it. Dataset keys are limited
+to letters, numbers, dots, underscores, and hyphens.
+To retrieve a dataset without supplying its generation parameters, omit
+`--field`; use `--output` and `--format` to export the saved records:
+
+```console
+dataforger --dataset customers --format csv --output customers.csv
+```
+
+Datasets are stored as JSON files in `~/.dataforger/datasets` by default. Set
+`DATAFORGER_DATA_DIR` to share another location across CLI and Python usage,
+or pass `--data-dir PATH` to the CLI. The API server uses the same configured
+directory; `dataforger --serve --data-dir PATH` sets it for that server.
+Writes are atomic and serialized across threads and processes on the same
+machine. For containerized deployments, mount a persistent volume at the
+configured directory.
+
+The Python API exposes the same operations:
+
+```python
+from dataforger import get_dataset, get_or_create_dataset, replace_dataset
+
+customers = get_or_create_dataset(
+    "customers",
+    {"id": "uuid", "name": "name"},
+    rows=100,
+    seed=42,
+)
+same_customers = get_dataset("customers")
+replacement = replace_dataset(
+    "customers",
+    {"id": "uuid", "name": "name"},
+    rows=100,
+    seed=43,
+)
+```
+
+`get_or_create_dataset` returns an existing dataset only when the generation
+parameters match. It raises `DatasetConflictError` when they differ and
+`DatasetNotFoundError` is raised when `get_dataset` cannot find a key.
+`replace_dataset` explicitly generates and stores a replacement. All three
+functions accept an optional `directory` argument to override the default
+store directory.
+
+## HTTP API
+
+Start the built-in HTTP API with:
+
+```console
+dataforger --serve
+```
+
+The server listens on `127.0.0.1:8000` by default. Use `--host` and `--port`
+to change the bind address and port. The API has no authentication, so keep it
+on localhost unless you put it behind an appropriately secured service.
+
+Generate records with a JSON `POST` request to `/api/v1/generate`:
+
+```console
+curl -X POST http://127.0.0.1:8000/api/v1/generate ^
+  -H "Content-Type: application/json" ^
+  -d "{\"fields\":{\"name\":\"name\",\"email\":\"email\",\"age\":\"age\"},\"rows\":2,\"seed\":42}"
+```
+
+`fields` is a mapping of output names to supported field types, `rows` is
+optional (defaults to `10`, maximum `10000`), and `seed` is optional. The
+request body is limited to 1 MiB and `fields` to 100 entries. Responses use
+JSON; successful generation returns `{"count": 2, "data": [...]}`, and invalid
+requests return a JSON `error` with an appropriate HTTP status.
+
+`GET /api/v1/fields` lists supported field types. `GET /health` returns the
+server health status.
+
+Persistent datasets use:
+
+* `POST /api/v1/datasets` with `key`, `fields`, and optional `rows` and `seed`
+  to create or reuse a dataset. Reusing the key with different generation
+  parameters returns `409 Conflict`.
+* `GET /api/v1/datasets/{key}` to retrieve a saved dataset.
+* `PUT /api/v1/datasets/{key}` with `fields` and optional `rows` and `seed` to
+  explicitly generate or replace the dataset.
+
+These routes use the same JSON file store as the Python and CLI interfaces.
+The API has no authentication; keep it bound to localhost unless protected by
+an appropriately secured service.

{dataforger-0.1.2 → dataforger-0.1.3}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "dataforger"
-version = "0.1.2"
+version = "0.1.3"
 description = "Generate fake data from Python or the command line."
 readme = "README.md"
 license = { file = "LICENSE" }

dataforger-0.1.3/src/dataforger/__init__.py

@@ -0,0 +1,23 @@
+from dataforger.core import FIELD_TYPES, generate_data, write_data
+from dataforger.datasets import (
+    Dataset,
+    DatasetConflictError,
+    DatasetIntegrityError,
+    DatasetNotFoundError,
+    get_dataset,
+    get_or_create_dataset,
+    replace_dataset,
+)
+
+__all__ = [
+    "FIELD_TYPES",
+    "Dataset",
+    "DatasetConflictError",
+    "DatasetIntegrityError",
+    "DatasetNotFoundError",
+    "generate_data",
+    "get_dataset",
+    "get_or_create_dataset",
+    "replace_dataset",
+    "write_data",
+]

dataforger-0.1.3/src/dataforger/api.py

@@ -0,0 +1,284 @@
+import json
+import logging
+from http import HTTPStatus
+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
+from pathlib import Path
+from typing import Any
+from urllib.parse import unquote, urlsplit
+
+from dataforger.core import FIELD_TYPES, generate_data
+from dataforger.datasets import (
+    Dataset,
+    DatasetConflictError,
+    DatasetIntegrityError,
+    DatasetNotFoundError,
+    get_dataset,
+    get_or_create_dataset,
+    replace_dataset,
+)
+
+MAX_REQUEST_BYTES = 1_048_576
+MAX_ROWS = 10_000
+MAX_FIELDS = 100
+_LOGGER = logging.getLogger(__name__)
+
+
+class _ApiServer(ThreadingHTTPServer):
+    def __init__(
+        self,
+        address: tuple[str, int],
+        handler: type[BaseHTTPRequestHandler],
+        data_directory: str | Path | None = None,
+    ) -> None:
+        self.data_directory = data_directory
+        super().__init__(address, handler)
+
+
+class _ApiHandler(BaseHTTPRequestHandler):
+    server_version = "DataforgerAPI/1.0"
+
+    def do_GET(self) -> None:
+        path = urlsplit(self.path).path
+        if path == "/health":
+            self._send_json(HTTPStatus.OK, {"status": "ok"})
+        elif path == "/api/v1/fields":
+            self._send_json(HTTPStatus.OK, {"fields": list(FIELD_TYPES)})
+        elif path.startswith("/api/v1/datasets/"):
+            key = unquote(path.removeprefix("/api/v1/datasets/"))
+            try:
+                dataset = get_dataset(key, getattr(self.server, "data_directory", None))
+            except DatasetNotFoundError as error:
+                self._send_error(HTTPStatus.NOT_FOUND, str(error))
+            except ValueError as error:
+                self._send_error(HTTPStatus.BAD_REQUEST, str(error))
+            except DatasetIntegrityError:
+                _LOGGER.exception("Persistent dataset failed its integrity check.")
+                self._send_error(
+                    HTTPStatus.INTERNAL_SERVER_ERROR,
+                    "Stored dataset is corrupt or unsupported.",
+                )
+            except OSError:
+                _LOGGER.exception("Unable to read persistent dataset.")
+                self._send_error(
+                    HTTPStatus.INTERNAL_SERVER_ERROR,
+                    "Unable to access dataset storage.",
+                )
+            else:
+                self._send_json(HTTPStatus.OK, _dataset_payload(dataset))
+        else:
+            self._send_error(HTTPStatus.NOT_FOUND, "Endpoint not found.")
+
+    def do_POST(self) -> None:
+        path = urlsplit(self.path).path
+        if path not in ("/api/v1/generate", "/api/v1/datasets"):
+            self._send_error(HTTPStatus.NOT_FOUND, "Endpoint not found.")
+            return
+
+        request = self._read_json_request()
+        if request is None:
+            return
+
+        allowed_properties = {"fields", "rows", "seed"}
+        if path == "/api/v1/datasets":
+            allowed_properties.add("key")
+        unexpected = set(request) - allowed_properties
+        if unexpected:
+            self._send_error(
+                HTTPStatus.BAD_REQUEST,
+                f"Unexpected request properties: {', '.join(sorted(unexpected))}.",
+            )
+            return
+
+        try:
+            fields, rows, seed = _parse_generation_options(request)
+        except ValueError as error:
+            self._send_error(HTTPStatus.BAD_REQUEST, str(error))
+            return
+
+        if path == "/api/v1/generate":
+            try:
+                records = generate_data(fields, rows=rows, seed=seed)
+            except ValueError as error:
+                self._send_error(HTTPStatus.BAD_REQUEST, str(error))
+                return
+            self._send_json(
+                HTTPStatus.OK,
+                {"count": len(records), "data": records},
+            )
+            return
+
+        key = request.get("key")
+        if not isinstance(key, str):
+            self._send_error(HTTPStatus.BAD_REQUEST, "'key' must be a string.")
+            return
+        try:
+            dataset = get_or_create_dataset(
+                key, fields, rows, seed, getattr(self.server, "data_directory", None)
+            )
+        except DatasetConflictError as error:
+            self._send_error(HTTPStatus.CONFLICT, str(error))
+        except ValueError as error:
+            self._send_error(HTTPStatus.BAD_REQUEST, str(error))
+        except DatasetIntegrityError:
+            _LOGGER.exception("Persistent dataset failed its integrity check.")
+            self._send_error(
+                HTTPStatus.INTERNAL_SERVER_ERROR,
+                "Stored dataset is corrupt or unsupported.",
+            )
+        except OSError:
+            _LOGGER.exception("Unable to create or load persistent dataset.")
+            self._send_error(
+                HTTPStatus.INTERNAL_SERVER_ERROR, "Unable to access dataset storage."
+            )
+        else:
+            self._send_json(HTTPStatus.OK, _dataset_payload(dataset))
+
+    def do_PUT(self) -> None:
+        path = urlsplit(self.path).path
+        prefix = "/api/v1/datasets/"
+        if not path.startswith(prefix):
+            self._send_error(HTTPStatus.NOT_FOUND, "Endpoint not found.")
+            return
+
+        key = unquote(path.removeprefix(prefix))
+        request = self._read_json_request()
+        if request is None:
+            return
+        unexpected = set(request) - {"fields", "rows", "seed"}
+        if unexpected:
+            self._send_error(
+                HTTPStatus.BAD_REQUEST,
+                f"Unexpected request properties: {', '.join(sorted(unexpected))}.",
+            )
+            return
+
+        try:
+            fields, rows, seed = _parse_generation_options(request)
+            dataset = replace_dataset(
+                key, fields, rows, seed, getattr(self.server, "data_directory", None)
+            )
+        except ValueError as error:
+            self._send_error(HTTPStatus.BAD_REQUEST, str(error))
+        except DatasetIntegrityError:
+            _LOGGER.exception("Persistent dataset failed its integrity check.")
+            self._send_error(
+                HTTPStatus.INTERNAL_SERVER_ERROR,
+                "Stored dataset is corrupt or unsupported.",
+            )
+        except OSError:
+            _LOGGER.exception("Unable to replace persistent dataset.")
+            self._send_error(
+                HTTPStatus.INTERNAL_SERVER_ERROR, "Unable to access dataset storage."
+            )
+        else:
+            self._send_json(HTTPStatus.OK, _dataset_payload(dataset))
+
+    def _read_json_request(self) -> dict[str, Any] | None:
+        content_length = self.headers.get("Content-Length")
+        if content_length is None:
+            self._send_error(
+                HTTPStatus.LENGTH_REQUIRED, "Content-Length header is required."
+            )
+            return None
+
+        try:
+            body_length = int(content_length)
+        except ValueError:
+            self._send_error(HTTPStatus.BAD_REQUEST, "Invalid Content-Length header.")
+            return None
+
+        if body_length < 0:
+            self._send_error(HTTPStatus.BAD_REQUEST, "Invalid Content-Length header.")
+            return None
+        if body_length > MAX_REQUEST_BYTES:
+            self._send_error(
+                HTTPStatus.REQUEST_ENTITY_TOO_LARGE, "Request body is too large."
+            )
+            return None
+
+        body = self.rfile.read(body_length)
+        if len(body) != body_length:
+            self._send_error(HTTPStatus.BAD_REQUEST, "Incomplete request body.")
+            return None
+
+        try:
+            request: Any = json.loads(body)
+        except (UnicodeDecodeError, json.JSONDecodeError):
+            self._send_error(HTTPStatus.BAD_REQUEST, "Request body must be valid JSON.")
+            return None
+
+        if not isinstance(request, dict):
+            self._send_error(
+                HTTPStatus.BAD_REQUEST, "Request body must be a JSON object."
+            )
+            return None
+        return request
+
+    def _send_error(self, status: HTTPStatus, message: str) -> None:
+        self._send_json(status, {"error": message})
+
+    def _send_json(self, status: HTTPStatus, payload: dict[str, object]) -> None:
+        body = json.dumps(payload, ensure_ascii=False).encode("utf-8")
+        self.send_response(status)
+        self.send_header("Content-Type", "application/json; charset=utf-8")
+        self.send_header("Content-Length", str(len(body)))
+        self.end_headers()
+        self.wfile.write(body)
+
+
+def _parse_generation_options(
+    request: dict[str, Any],
+) -> tuple[dict[str, str], int, int | None]:
+    fields = request.get("fields")
+    if not isinstance(fields, dict) or not fields:
+        raise ValueError("'fields' must be a non-empty JSON object.")
+    if len(fields) > MAX_FIELDS:
+        raise ValueError(f"'fields' cannot contain more than {MAX_FIELDS} entries.")
+    if any(
+        not isinstance(name, str) or not isinstance(value, str)
+        for name, value in fields.items()
+    ):
+        raise ValueError("'fields' must map string names to string field types.")
+
+    rows = request.get("rows", 10)
+    if (
+        isinstance(rows, bool)
+        or not isinstance(rows, int)
+        or not 1 <= rows <= MAX_ROWS
+    ):
+        raise ValueError(f"'rows' must be an integer from 1 to {MAX_ROWS}.")
+
+    seed = request.get("seed")
+    if seed is not None and (isinstance(seed, bool) or not isinstance(seed, int)):
+        raise ValueError("'seed' must be an integer or null.")
+    return fields, rows, seed
+
+
+def _dataset_payload(dataset: Dataset) -> dict[str, object]:
+    return {
+        "key": dataset.key,
+        "fields": dataset.fields,
+        "rows": dataset.rows,
+        "seed": dataset.seed,
+        "created_at": dataset.created_at,
+        "count": len(dataset.data),
+        "data": dataset.data,
+    }
+
+
+def serve(
+    host: str = "127.0.0.1",
+    port: int = 8000,
+    data_directory: str | Path | None = None,
+) -> None:
+    """Serve the Dataforger HTTP API until interrupted."""
+    if not host:
+        raise ValueError("host must not be empty.")
+    if isinstance(port, bool) or not isinstance(port, int) or not 1 <= port <= 65535:
+        raise ValueError("port must be an integer from 1 to 65535.")
+
+    server = _ApiServer((host, port), _ApiHandler, data_directory)
+    try:
+        server.serve_forever()
+    finally:
+        server.server_close()