fakerforge 0.1.0__tar.gz → 0.2.0__tar.gz

{fakerforge-0.1.0 → fakerforge-0.2.0}/CHANGELOG.md

@@ -1,5 +1,11 @@
 # Changelog
 
+## 0.2.0 - 2026-10-05
+
+- Add a dependency graph and use it for declarative schema order. Parallel groups are reported and execution stays single-threaded.
+- Add `FakerForge.load` and `FakerForge.plan` for JSON projects, and for YAML when PyYAML is installed. Planning validates providers and dependencies and does not generate rows.
+- Add `FakerForge.iter_batches`. Concatenating chunks matches `generate` for the same seed. `generate` still returns the full in-memory result.
+
 ## 0.1.0 - 2026-09-29
 
 First public release. Requires Python 3.10 or newer.

{fakerforge-0.1.0 → fakerforge-0.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: fakerforge
-Version: 0.1.0
+Version: 0.2.0
 Summary: Synthetic data generation built on the Faker library.
 Author: Shardul
 License-Expression: MIT
@@ -23,6 +23,8 @@ Provides-Extra: numpy
 Requires-Dist: numpy>=1.24; extra == "numpy"
 Provides-Extra: pandas
 Requires-Dist: pandas>=2; extra == "pandas"
+Provides-Extra: yaml
+Requires-Dist: PyYAML>=6; extra == "yaml"
 Provides-Extra: dev
 Requires-Dist: pytest>=8.0; extra == "dev"
 Requires-Dist: ruff>=0.8.0; extra == "dev"
@@ -219,6 +221,27 @@ rows = Generator(fake).generate(schema, count=3)
 
 The source distribution includes `examples/quickstart.py` and `docs/architecture.md`.
 
+## Project plan and batches
+
+`load` reads a project document. `plan` checks providers and dependencies and reports generation order, row counts, and tables that can be generated together. It does not draw rows. `estimated_bytes_per_row` is a placeholder of 64 bytes per column, not a measured size.
+
+`iter_batches` yields chunks. Concatenating them matches `generate` for the same seed and schema. `generate` still returns the full in-memory dataset. Execution stays single-threaded. The dependency graph names parallel groups for later releases.
+
+JSON works with the base install. YAML needs `pip install "fakerforge[yaml]"`. Documents are parsed as data and are not executed. Distribution draws stay reproducible for a given backend. NumPy and the standard library do not emit the same series, so reproducibility includes whether NumPy is installed.
+
+```python
+from fakerforge import FakerForge
+
+forge = FakerForge(seed=42)
+project = forge.load("examples/projects/minimal/project.json")
+plan = forge.plan(project)
+
+for batch in forge.iter_batches(project.tables, batch_size=2):
+    print(batch.table, len(batch))
+```
+
+`ARCHITECTURE_AUDIT.md` describes the current architecture and the roadmap.
+
 ## This release
 
 | Piece | What 0.1 provides |

{fakerforge-0.1.0 → fakerforge-0.2.0}/README.md

@@ -186,6 +186,27 @@ rows = Generator(fake).generate(schema, count=3)
 
 The source distribution includes `examples/quickstart.py` and `docs/architecture.md`.
 
+## Project plan and batches
+
+`load` reads a project document. `plan` checks providers and dependencies and reports generation order, row counts, and tables that can be generated together. It does not draw rows. `estimated_bytes_per_row` is a placeholder of 64 bytes per column, not a measured size.
+
+`iter_batches` yields chunks. Concatenating them matches `generate` for the same seed and schema. `generate` still returns the full in-memory dataset. Execution stays single-threaded. The dependency graph names parallel groups for later releases.
+
+JSON works with the base install. YAML needs `pip install "fakerforge[yaml]"`. Documents are parsed as data and are not executed. Distribution draws stay reproducible for a given backend. NumPy and the standard library do not emit the same series, so reproducibility includes whether NumPy is installed.
+
+```python
+from fakerforge import FakerForge
+
+forge = FakerForge(seed=42)
+project = forge.load("examples/projects/minimal/project.json")
+plan = forge.plan(project)
+
+for batch in forge.iter_batches(project.tables, batch_size=2):
+    print(batch.table, len(batch))
+```
+
+`ARCHITECTURE_AUDIT.md` describes the current architecture and the roadmap.
+
 ## This release
 
 | Piece | What 0.1 provides |

{fakerforge-0.1.0 → fakerforge-0.2.0}/docs/architecture.md

@@ -12,6 +12,7 @@ DistributionEngine       seeded uniform, normal, log-normal, categorical
     └── FakerForge.number / FakerForge.categorical
 
 DatasetSchema / DatabaseSchema
+    └── DependencyGraph      stable topological order and parallel groups
     └── FakerForge.generate  validates order, then fills rows
             ├── derived fields and date dependencies
             ├── foreign keys sampled from parent rows
@@ -19,6 +20,9 @@ DatasetSchema / DatabaseSchema
                     ├── frame is a pandas.DataFrame when pandas is installed
                     └── validate() runs DatasetCheck subclasses
 
+FakerForge.load / plan   project document, no row draws
+FakerForge.iter_batches  same row function, fixed-size chunks
+
 Schema (Field...)
     └── Generator            calls provider methods on a FakerForge
 
@@ -26,6 +30,10 @@ Constraint subclasses
     └── Validator            checks a finished record against constraints
 ```
 
-`DistributionEngine` keeps its own random stream so schema generation can sample numbers without consuming Faker provider draws. NumPy is optional.
+`DistributionEngine` keeps its own random stream so schema generation can sample numbers without consuming Faker provider draws. NumPy is optional. Each backend is deterministic, and the two backends do not emit the same series.
+
+`FakerForge.load` reads a project document (`project.name`, optional `project.seed`, and `tables`). `FakerForge.plan` checks providers and dependencies and returns generation order, row counts, and parallel groups. It does not draw rows. `estimated_bytes_per_row` is a placeholder of 64 bytes per column.
+
+`FakerForge.iter_batches` yields fixed-size chunks through the same row function as `generate`. `generate` still materializes every row. `DependencyGraph` is the topological sort used by declarative schemas.
 
-Finance is the first built-in domain provider. Many-to-many relationships are still later work.
+Finance is the first built-in domain provider. Many-to-many relationships are still later work. The roadmap is in `ARCHITECTURE_AUDIT.md`.

{fakerforge-0.1.0 → fakerforge-0.2.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "fakerforge"
-version = "0.1.0"
+version = "0.2.0"
 description = "Synthetic data generation built on the Faker library."
 readme = "README.md"
 requires-python = ">=3.10"
@@ -31,6 +31,7 @@ dependencies = [
 [project.optional-dependencies]
 numpy = ["numpy>=1.24"]
 pandas = ["pandas>=2"]
+yaml = ["PyYAML>=6"]
 dev = [
     "pytest>=8.0",
     "ruff>=0.8.0",

{fakerforge-0.1.0 → fakerforge-0.2.0}/src/fakerforge/__init__.py

@@ -2,6 +2,6 @@
 
 from fakerforge.faker import FakerForge
 
-__version__ = "0.1.0"
+__version__ = "0.2.0"
 
 __all__ = ["FakerForge", "__version__"]

fakerforge-0.2.0/src/fakerforge/core/__init__.py

@@ -0,0 +1,14 @@
+"""Project documents and generation plans."""
+
+from fakerforge.core.errors import FakerForgeError, PlanError
+from fakerforge.core.plan import ESTIMATED_BYTES_PER_COLUMN, GenerationPlan
+from fakerforge.core.project import Project, load_project
+
+__all__ = [
+    "ESTIMATED_BYTES_PER_COLUMN",
+    "FakerForgeError",
+    "GenerationPlan",
+    "PlanError",
+    "Project",
+    "load_project",
+]

fakerforge-0.2.0/src/fakerforge/core/errors.py

@@ -0,0 +1,36 @@
+"""Errors for project documents and generation plans.
+
+``SchemaError`` and ``GenerationError`` keep their original bases.
+``isinstance`` checks against ``ValueError`` and ``RuntimeError`` stay valid.
+"""
+
+
+class FakerForgeError(Exception):
+    """Base for project loading and planning failures."""
+
+
+class PlanError(FakerForgeError):
+    """Raised when a project document or dependency graph is invalid.
+
+    Args:
+        message: Human-readable failure.
+        project: Project name, when it is known.
+        table: Table name, when the failure is in one table.
+        column: Column name, when the failure is in one column.
+        dependency: Related table, column, or cycle description.
+    """
+
+    def __init__(
+        self,
+        message: str,
+        *,
+        project: str | None = None,
+        table: str | None = None,
+        column: str | None = None,
+        dependency: str | None = None,
+    ) -> None:
+        self.project = project
+        self.table = table
+        self.column = column
+        self.dependency = dependency
+        super().__init__(message)

fakerforge-0.2.0/src/fakerforge/core/plan.py

@@ -0,0 +1,39 @@
+"""Dry-run summary of a project."""
+
+from collections.abc import Mapping
+from dataclasses import dataclass
+from typing import Any
+
+# Placeholder allowance used until a measured size exists. Not a benchmark.
+ESTIMATED_BYTES_PER_COLUMN = 64
+
+
+@dataclass(frozen=True, slots=True)
+class GenerationPlan:
+    """Tables, row counts, and parallel groups for a project.
+
+    ``estimated_bytes_per_row`` multiplies
+    :data:`ESTIMATED_BYTES_PER_COLUMN` by the column count. That figure is
+    a placeholder, not a measurement of generated data or Parquet size.
+
+    Building a plan does not draw provider values or distribution samples.
+
+    Args:
+        project: Project name from the document.
+        seed: Seed recorded on the project. ``None`` when the document
+            omitted it.
+        order: Table names in generation order.
+        row_counts: Requested rows for each table.
+        parallel_groups: Tables that do not depend on each other. Names
+            inside a group stay in declaration order.
+        estimated_bytes_per_row: Placeholder bytes for one row of each table.
+        total_rows: Sum of the requested row counts.
+    """
+
+    project: str
+    seed: Any
+    order: tuple[str, ...]
+    row_counts: Mapping[str, int]
+    parallel_groups: tuple[tuple[str, ...], ...]
+    estimated_bytes_per_row: Mapping[str, int]
+    total_rows: int

fakerforge-0.2.0/src/fakerforge/core/project.py

@@ -0,0 +1,247 @@
+"""Project documents in memory, JSON, and YAML."""
+
+import json
+from collections.abc import Mapping
+from importlib import import_module
+from pathlib import Path
+from types import MappingProxyType
+from typing import Any
+
+from fakerforge.core.errors import PlanError
+from fakerforge.core.plan import ESTIMATED_BYTES_PER_COLUMN, GenerationPlan
+from fakerforge.graph.dag import DependencyGraph
+from fakerforge.schema.dataset import DatabaseSchema, SchemaError
+from fakerforge.schema.generate import _resolve_providers
+
+# Same values ``FakerForge`` accepts, without importing that module.
+Seed = int | float | str | bytes | bytearray | None
+
+_TOP_KEYS = frozenset({"project", "tables"})
+_PROJECT_KEYS = frozenset({"name", "seed"})
+
+
+class Project:
+    """A named relational schema plus the seed recorded for it.
+
+    The ``tables`` mapping uses the same shape as ``FakerForge.generate``:
+    each table has ``rows`` and ``fields``. Loading a document does not
+    generate rows and does not reseed a forge.
+
+    Args:
+        name: Project name.
+        seed: Seed from the document. ``None`` when the document omitted it.
+        tables: Relational schema mapping.
+    """
+
+    def __init__(
+        self,
+        name: str,
+        seed: Seed,
+        tables: Mapping[str, Any],
+    ) -> None:
+        if not isinstance(name, str) or name == "":
+            raise PlanError("project name must be a non-empty string")
+        if not isinstance(tables, Mapping):
+            raise PlanError("tables must be a mapping", project=name)
+        self.name = name
+        self.seed = seed
+        self.tables = dict(tables)
+
+    def schema(self) -> DatabaseSchema:
+        """Parse ``tables`` into a relational schema.
+
+        Circular dependencies raise :class:`PlanError`. Other schema
+        problems raise :class:`~fakerforge.schema.SchemaError`.
+
+        Returns:
+            The parsed database schema.
+        """
+        try:
+            return DatabaseSchema.from_dict(self.tables)
+        except SchemaError as exc:
+            circular = _circular_error(self.name, exc)
+            if circular is not None:
+                raise circular from exc
+            raise
+
+    def build_dependency_graph(self) -> DependencyGraph:
+        """Return the table and column dependency graph.
+
+        Returns:
+            A graph whose table order matches the schema generation order.
+
+        Raises:
+            PlanError: The tables contain a circular dependency.
+            SchemaError: The tables are otherwise invalid.
+        """
+        return DependencyGraph.from_database(self.schema())
+
+    def validate_dependency_graph(self) -> DependencyGraph:
+        """Return the graph when every table can be ordered.
+
+        Returns:
+            The dependency graph.
+
+        Raises:
+            PlanError: A table or field cycle is present.
+            SchemaError: The tables are otherwise invalid.
+        """
+        graph = self.build_dependency_graph()
+        if graph.cycle:
+            raise PlanError(
+                f"circular table dependency: {', '.join(graph.cycle)}",
+                project=self.name,
+                dependency=", ".join(graph.cycle),
+            )
+        return graph
+
+    def generation_plan(self, forge: Any) -> GenerationPlan:
+        """Validate providers and dependencies without generating rows.
+
+        Args:
+            forge: FakerForge instance whose registered providers are checked.
+
+        Returns:
+            Row counts, generation order, and parallel groups.
+
+        Raises:
+            PlanError: The document has a circular dependency.
+            SchemaError: A field, provider, or row count is invalid.
+        """
+        graph = self.validate_dependency_graph()
+        database = self.schema()
+        for table in database.in_generation_order():
+            _resolve_providers(forge, table.dataset, table.rows)
+        counts = {table.name: table.rows for table in database.tables}
+        estimates = {
+            table.name: ESTIMATED_BYTES_PER_COLUMN * len(table.dataset.columns)
+            for table in database.tables
+        }
+        return GenerationPlan(
+            project=self.name,
+            seed=self.seed,
+            order=graph.order,
+            row_counts=MappingProxyType(counts),
+            parallel_groups=graph.levels,
+            estimated_bytes_per_row=MappingProxyType(estimates),
+            total_rows=sum(counts[name] for name in graph.order),
+        )
+
+
+def load_project(source: str | Path | Mapping[str, Any]) -> Project:
+    """Load a project from a mapping or a JSON or YAML file.
+
+    YAML requires PyYAML (``pip install 'fakerforge[yaml]'``). Files are
+    parsed as data. They are not executed.
+
+    Args:
+        source: A project mapping, or a path to ``.json``, ``.yaml``, or
+            ``.yml``.
+
+    Returns:
+        The parsed project. Tables are not validated until
+        :meth:`Project.schema` or :meth:`Project.generation_plan`.
+
+    Raises:
+        PlanError: The document, path, or YAML dependency is invalid.
+    """
+    document = _read_document(source)
+    if not isinstance(document, Mapping):
+        raise PlanError("project document must be a mapping")
+    unknown = sorted(set(document) - _TOP_KEYS)
+    if unknown:
+        raise PlanError(f"unknown project keys {unknown}")
+    if "project" not in document or "tables" not in document:
+        raise PlanError("project document requires 'project' and 'tables'")
+    meta = document["project"]
+    if not isinstance(meta, Mapping):
+        raise PlanError("project must be a mapping")
+    unknown_meta = sorted(str(key) for key in set(meta) - _PROJECT_KEYS)
+    if unknown_meta:
+        raise PlanError(f"unknown keys in project: {unknown_meta}")
+    name = meta.get("name")
+    if not isinstance(name, str) or name == "":
+        raise PlanError("project name must be a non-empty string")
+    seed = _parse_seed(meta["seed"], name) if "seed" in meta else None
+    tables = document["tables"]
+    if not isinstance(tables, Mapping):
+        raise PlanError("tables must be a mapping", project=name)
+    return Project(name=name, seed=seed, tables=tables)
+
+
+def _read_document(source: str | Path | Mapping[str, Any]) -> Any:
+    if isinstance(source, Mapping):
+        return source
+    path = Path(source)
+    if not path.is_file():
+        raise PlanError(f"project file {path} does not exist")
+    text = path.read_text(encoding="utf-8")
+    suffix = path.suffix.lower()
+    if suffix == ".json":
+        try:
+            return json.loads(text)
+        except json.JSONDecodeError as exc:
+            raise PlanError(f"invalid JSON in {path}: {exc.msg}") from exc
+    if suffix in {".yaml", ".yml"}:
+        return _read_yaml(path, text)
+    raise PlanError(f"project file {path} must be JSON (.json) or YAML (.yaml, .yml)")
+
+
+def _read_yaml(path: Path, text: str) -> Any:
+    yaml = _import_yaml()
+    if yaml is None:
+        raise PlanError(
+            "YAML projects require PyYAML. "
+            "Install it with: pip install 'fakerforge[yaml]'"
+        )
+    try:
+        loaded = yaml.safe_load(text)
+    except Exception as exc:
+        if type(exc).__module__.split(".")[0] != "yaml":
+            raise
+        raise PlanError(f"invalid YAML in {path}") from exc
+    return loaded
+
+
+def _import_yaml() -> Any:
+    try:
+        return import_module("yaml")
+    except ImportError:
+        return None
+
+
+def _parse_seed(value: object, project: str) -> Seed:
+    if value is None or isinstance(value, (str, bytes, bytearray)):
+        return value
+    if isinstance(value, bool) or not isinstance(value, (int, float)):
+        raise PlanError(
+            "project seed must be an int, float, str, or null",
+            project=project,
+        )
+    return value
+
+
+def _circular_error(project: str, exc: SchemaError) -> PlanError | None:
+    circular = [item for item in exc.errors if "circular" in item]
+    if not circular:
+        return None
+    message = circular[0]
+    dependency = message.split(":", 1)[1].strip() if ":" in message else message
+    table = _table_name(message)
+    return PlanError(
+        message,
+        project=project,
+        table=table,
+        dependency=dependency,
+    )
+
+
+def _table_name(message: str) -> str | None:
+    marker = "table '"
+    if marker not in message:
+        return None
+    start = message.index(marker) + len(marker)
+    end = message.find("'", start)
+    if end == -1:
+        return None
+    return message[start:end]

{fakerforge-0.1.0 → fakerforge-0.2.0}/src/fakerforge/faker.py

@@ -1,12 +1,16 @@
 """Faker-compatible entry point."""
 
-from collections.abc import Mapping, Sequence
+from collections.abc import Iterator, Mapping, Sequence
+from pathlib import Path
 from typing import Any
 
 from faker import Faker
 from faker.providers import BaseProvider
 
+from fakerforge.core.plan import GenerationPlan
+from fakerforge.core.project import Project, load_project
 from fakerforge.distributions.engine import DistributionEngine
+from fakerforge.generation.batch import Batch, iter_batches
 from fakerforge.providers.finance import FinanceProvider
 from fakerforge.schema.dataset import DatabaseSchema, DatasetSchema
 from fakerforge.schema.generate import generate_dataset
@@ -178,6 +182,77 @@ class FakerForge:
         """
         return generate_dataset(self, schema, rows=rows)
 
+    def load(self, source: str | Path | Mapping[str, Any]) -> Project:
+        """Load a project document.
+
+        The document has ``project.name``, an optional ``project.seed``, and
+        ``tables`` in the same shape ``generate`` accepts for a relational
+        schema. JSON and YAML files are parsed as data. This method does not
+        generate rows and does not reseed this instance.
+
+        Args:
+            source: A project mapping, or a path to ``.json``, ``.yaml``, or
+                ``.yml``. YAML requires ``pip install 'fakerforge[yaml]'``.
+
+        Returns:
+            The loaded project. Call :meth:`plan` to validate it.
+
+        Raises:
+            PlanError: The document or file is invalid.
+        """
+        return load_project(source)
+
+    def plan(self, project: Project) -> GenerationPlan:
+        """Validate a project and describe how it would be generated.
+
+        The plan reports row counts, generation order, and tables that do
+        not depend on each other. It checks that providers are registered.
+        It does not draw values. ``estimated_bytes_per_row`` is a placeholder
+        of 64 bytes per column, not a measured size.
+
+        Args:
+            project: Project returned by :meth:`load`.
+
+        Returns:
+            The generation plan.
+
+        Raises:
+            PlanError: The project has a circular dependency.
+            SchemaError: A table, field, or provider is invalid.
+        """
+        return project.generation_plan(self)
+
+    def iter_batches(
+        self,
+        schema: Mapping[str, Any] | DatasetSchema | DatabaseSchema,
+        *,
+        rows: int | Mapping[str, int] | None = None,
+        batch_size: int = 1000,
+    ) -> Iterator[Batch]:
+        """Yield generated rows in chunks of at most ``batch_size``.
+
+        Concatenating the chunks matches :meth:`generate` for the same
+        seed and schema. This method does not keep earlier chunks. Use
+        :meth:`generate` when the full in-memory result is required.
+
+        Args:
+            schema: A field mapping, dataset, table mapping, or database.
+            rows: Row count for one table, or per-table counts. Relational
+                tables can store ``rows`` on each table instead.
+            batch_size: Maximum rows in each chunk. The default is 1000.
+
+        Yields:
+            :class:`~fakerforge.generation.Batch` objects in generation
+            order.
+
+        Raises:
+            ValueError: ``rows`` or ``batch_size`` is invalid.
+            SchemaError: The schema or its providers are invalid.
+            GenerationError: A unique value or foreign key cannot be filled
+                without inventing a value.
+        """
+        return iter_batches(self, schema, rows=rows, batch_size=batch_size)
+
     def add_provider(self, provider: type[BaseProvider] | BaseProvider) -> None:
         """Register a Faker provider on the underlying instance.
 

fakerforge-0.2.0/src/fakerforge/generation/__init__.py

@@ -0,0 +1,6 @@
+"""Batched generation and foreign-key pools."""
+
+from fakerforge.generation.batch import Batch, iter_batches
+from fakerforge.generation.keys import KeyPool
+
+__all__ = ["Batch", "KeyPool", "iter_batches"]