synthbench 0.1.0__tar.gz

synthbench-0.1.0/.gitignore

@@ -0,0 +1,13 @@
+.planning/
+.pytest_cache/
+.ruff_cache/
+.venv/
+.cache/
+*.pyc
+quality_reports/
+
+# Build artifacts
+dist/
+build/
+*.egg-info/
+.coverage

synthbench-0.1.0/CHANGELOG.md

@@ -0,0 +1,5 @@
+# Changelog
+
+## 0.1.0 — 2026-05-06
+
+Initial public release on PyPI.

synthbench-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Jan Teichert-Kluge
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

synthbench-0.1.0/PKG-INFO

@@ -0,0 +1,127 @@
+Metadata-Version: 2.4
+Name: synthbench
+Version: 0.1.0
+Summary: Synthetic datasets for ML benchmarking with controllable complexity, configurable corruptions, and full provenance.
+Project-URL: Homepage, https://github.com/JanTeichertKluge/synth-bench
+Project-URL: Documentation, https://JanTeichertKluge.github.io/synth-bench
+Project-URL: Repository, https://github.com/JanTeichertKluge/synth-bench.git
+Project-URL: Issues, https://github.com/JanTeichertKluge/synth-bench/issues
+Project-URL: Changelog, https://github.com/JanTeichertKluge/synth-bench/releases
+Author-email: Jan Teichert-Kluge <janteiklu@gmail.com>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: benchmarking,data-generating-process,dataset-generation,machine-learning,reproducibility,synthetic-data
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Typing :: Typed
+Requires-Python: >=3.12
+Requires-Dist: numpy>=2.0
+Requires-Dist: scikit-learn>=1.5
+Requires-Dist: scipy>=1.12
+Provides-Extra: dev
+Requires-Dist: pre-commit>=3.5; extra == 'dev'
+Requires-Dist: pytest-cov>=7.0; extra == 'dev'
+Requires-Dist: pytest>=8.0; extra == 'dev'
+Requires-Dist: ruff>=0.5; extra == 'dev'
+Provides-Extra: docs
+Requires-Dist: matplotlib>=3.7; extra == 'docs'
+Requires-Dist: mkdocs-jupyter>=0.26; extra == 'docs'
+Requires-Dist: mkdocs-material>=9.5; extra == 'docs'
+Requires-Dist: mkdocs>=1.6; extra == 'docs'
+Requires-Dist: mkdocstrings[python]>=1.0; extra == 'docs'
+Requires-Dist: openml>=0.14; extra == 'docs'
+Requires-Dist: pandas>=2.0; extra == 'docs'
+Provides-Extra: io
+Requires-Dist: pyarrow>=14.0; extra == 'io'
+Provides-Extra: neural
+Requires-Dist: torch>=2.1; extra == 'neural'
+Description-Content-Type: text/markdown
+
+<div align="center">
+  <img src="icon.png" alt="synthbench" width="420">
+</div>
+
+---
+
+synthbench is a small Python library for generating synthetic datasets that are actually useful for benchmarking. You control the signal complexity, add noise or missing data on top, and get back a dataset with full provenance so you know exactly what you generated and why. Every result is reproducible from a single integer seed.
+
+It covers eight DGP families, five corruptors, metadata enrichment (Bayes error, effective rank), Parquet/CSV serialization, and sweep helpers for running ablation grids.
+
+## Installation
+
+```bash
+pip install synthbench
+```
+
+For Parquet support:
+
+```bash
+pip install "synthbench[io]"
+```
+
+For `RandomNeuralDGP` (needs PyTorch):
+
+```bash
+pip install "synthbench[neural]"
+```
+
+## Basic usage
+
+```python
+from synthbench import BenchPipeline, LinearDGP, MissingDataCorruptor
+
+pipeline = BenchPipeline(
+    LinearDGP(complexity="medium", task_type="classification"),
+    corruptors=[MissingDataCorruptor(proportion=0.1, mechanism="mar")],
+)
+result = pipeline.run(n_samples=500, n_features=10, random_state=42)
+
+print(result.X.shape)                     # (500, 10)
+print(result.metadata["bayes_error"])     # empirical difficulty estimate
+print(result.metadata["effective_rank"])  # feature space dimensionality
+```
+
+## What it does
+
+**Data-generating processes** — Linear, Polynomial, Tree, Friedman (variants 1/2/3), Additive, Sparse, Geometric, and RandomNeural. Each takes a `complexity` parameter and records ground-truth feature importances alongside the data.
+
+**Corruptors** — MeasurementNoise, Outlier, MissingData, Collinearity, and Categorical corruptors for the feature matrix, plus `LabelNoiseCorruptor` for flipping labels or injecting regression noise. They chain together in a canonical order and track how much signal they degrade.
+
+**Metadata** — every result carries `bayes_error`, `effective_rank`, corruptor parameters, and version provenance. Enough to reconstruct the generating pipeline from scratch.
+
+**Sweeps** — `severity_sweep` and `difficulty_sweep` for single-axis ablations, and `experiment_grid` for full factorial runs across sample size, complexity, and severity. Seeds are derived hierarchically so cells are independent but deterministic.
+
+**Named suites** — `BenchSuite("easy-classification").run()` returns a labelled dict of results for a curated collection. Good for quick sanity checks or as a shared benchmark baseline.
+
+**Serialization** — `to_parquet` / `from_parquet` and `to_csv` / `from_csv` round-trip everything including metadata. `BenchPipeline.from_metadata` reconstructs and re-runs the pipeline for bit-identical replay.
+
+## Ablation example
+
+```python
+from synthbench import LinearDGP, OutlierCorruptor, experiment_grid
+
+grid = experiment_grid(
+    LinearDGP,
+    OutlierCorruptor,
+    n_samples_list=[200, 500, 1000],
+    complexities=["low", "medium", "high"],
+    severities=["low", "medium", "high"],
+    n_features=10,
+    random_state=0,
+    task_type="classification",
+)
+
+result = grid[(500, "high", "medium")]
+print(result.metadata["bayes_error"])
+```
+
+## Docs
+
+Full reference at [JanTeichertKluge.github.io/synth-bench](https://JanTeichertKluge.github.io/synth-bench).

synthbench-0.1.0/README.md

@@ -0,0 +1,81 @@
+<div align="center">
+  <img src="icon.png" alt="synthbench" width="420">
+</div>
+
+---
+
+synthbench is a small Python library for generating synthetic datasets that are actually useful for benchmarking. You control the signal complexity, add noise or missing data on top, and get back a dataset with full provenance so you know exactly what you generated and why. Every result is reproducible from a single integer seed.
+
+It covers eight DGP families, five corruptors, metadata enrichment (Bayes error, effective rank), Parquet/CSV serialization, and sweep helpers for running ablation grids.
+
+## Installation
+
+```bash
+pip install synthbench
+```
+
+For Parquet support:
+
+```bash
+pip install "synthbench[io]"
+```
+
+For `RandomNeuralDGP` (needs PyTorch):
+
+```bash
+pip install "synthbench[neural]"
+```
+
+## Basic usage
+
+```python
+from synthbench import BenchPipeline, LinearDGP, MissingDataCorruptor
+
+pipeline = BenchPipeline(
+    LinearDGP(complexity="medium", task_type="classification"),
+    corruptors=[MissingDataCorruptor(proportion=0.1, mechanism="mar")],
+)
+result = pipeline.run(n_samples=500, n_features=10, random_state=42)
+
+print(result.X.shape)                     # (500, 10)
+print(result.metadata["bayes_error"])     # empirical difficulty estimate
+print(result.metadata["effective_rank"])  # feature space dimensionality
+```
+
+## What it does
+
+**Data-generating processes** — Linear, Polynomial, Tree, Friedman (variants 1/2/3), Additive, Sparse, Geometric, and RandomNeural. Each takes a `complexity` parameter and records ground-truth feature importances alongside the data.
+
+**Corruptors** — MeasurementNoise, Outlier, MissingData, Collinearity, and Categorical corruptors for the feature matrix, plus `LabelNoiseCorruptor` for flipping labels or injecting regression noise. They chain together in a canonical order and track how much signal they degrade.
+
+**Metadata** — every result carries `bayes_error`, `effective_rank`, corruptor parameters, and version provenance. Enough to reconstruct the generating pipeline from scratch.
+
+**Sweeps** — `severity_sweep` and `difficulty_sweep` for single-axis ablations, and `experiment_grid` for full factorial runs across sample size, complexity, and severity. Seeds are derived hierarchically so cells are independent but deterministic.
+
+**Named suites** — `BenchSuite("easy-classification").run()` returns a labelled dict of results for a curated collection. Good for quick sanity checks or as a shared benchmark baseline.
+
+**Serialization** — `to_parquet` / `from_parquet` and `to_csv` / `from_csv` round-trip everything including metadata. `BenchPipeline.from_metadata` reconstructs and re-runs the pipeline for bit-identical replay.
+
+## Ablation example
+
+```python
+from synthbench import LinearDGP, OutlierCorruptor, experiment_grid
+
+grid = experiment_grid(
+    LinearDGP,
+    OutlierCorruptor,
+    n_samples_list=[200, 500, 1000],
+    complexities=["low", "medium", "high"],
+    severities=["low", "medium", "high"],
+    n_features=10,
+    random_state=0,
+    task_type="classification",
+)
+
+result = grid[(500, "high", "medium")]
+print(result.metadata["bayes_error"])
+```
+
+## Docs
+
+Full reference at [JanTeichertKluge.github.io/synth-bench](https://JanTeichertKluge.github.io/synth-bench).

synthbench-0.1.0/pyproject.toml

@@ -0,0 +1,99 @@
+[build-system]
+requires = ["hatchling>=1.27"]
+build-backend = "hatchling.build"
+
+[project]
+name = "synthbench"
+version = "0.1.0"
+description = "Synthetic datasets for ML benchmarking with controllable complexity, configurable corruptions, and full provenance."
+readme = "README.md"
+requires-python = ">=3.12"
+license = "MIT"
+license-files = ["LICENSE"]
+authors = [
+    { name = "Jan Teichert-Kluge", email = "janteiklu@gmail.com" },
+]
+keywords = [
+    "benchmarking",
+    "synthetic-data",
+    "machine-learning",
+    "dataset-generation",
+    "reproducibility",
+    "data-generating-process",
+]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Science/Research",
+    "Intended Audience :: Developers",
+    "Operating System :: OS Independent",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Topic :: Scientific/Engineering",
+    "Topic :: Scientific/Engineering :: Artificial Intelligence",
+    "Typing :: Typed",
+]
+dependencies = [
+    "numpy>=2.0",
+    "scikit-learn>=1.5",
+    "scipy>=1.12",
+]
+
+[project.urls]
+Homepage = "https://github.com/JanTeichertKluge/synth-bench"
+Documentation = "https://JanTeichertKluge.github.io/synth-bench"
+Repository = "https://github.com/JanTeichertKluge/synth-bench.git"
+Issues = "https://github.com/JanTeichertKluge/synth-bench/issues"
+Changelog = "https://github.com/JanTeichertKluge/synth-bench/releases"
+
+[project.optional-dependencies]
+io = ["pyarrow>=14.0"]
+neural = ["torch>=2.1"]
+docs = [
+    "mkdocs>=1.6",
+    "mkdocs-material>=9.5",
+    "mkdocstrings[python]>=1.0",
+    "mkdocs-jupyter>=0.26",
+    "openml>=0.14",
+    "matplotlib>=3.7",
+    "pandas>=2.0",
+]
+dev = [
+    "pytest>=8.0",
+    "pytest-cov>=7.0",
+    "ruff>=0.5",
+    "pre-commit>=3.5",
+]
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/synthbench"]
+
+[tool.hatch.build.targets.sdist]
+exclude = [
+    ".coverage",
+    ".gitattributes",
+    ".pre-commit-config.yaml",
+    "uv.lock",
+    ".github/",
+    "docs/",
+    "mkdocs.yml",
+]
+
+[tool.ruff]
+line-length = 88
+target-version = "py312"
+
+[tool.ruff.lint]
+select = ["E", "W", "F", "I", "UP", "B"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+markers = [
+    "neural: marks tests requiring torch (deselect with '-m not neural')",
+]
+
+[tool.coverage.report]
+fail_under = 90
+omit = [
+    "src/synthbench/dgps/neural.py",
+]

synthbench-0.1.0/src/synthbench/__init__.py

@@ -0,0 +1,77 @@
+from __future__ import annotations
+
+from synthbench._version import __version__
+from synthbench.corruptors import (
+    BaseCorruptor,
+    BaseLabelCorruptor,
+    CategoricalCorruptor,
+    CollinearityCorruptor,
+    LabelNoiseCorruptor,
+    MeasurementNoiseCorruptor,
+    MissingDataCorruptor,
+    OutlierCorruptor,
+)
+from synthbench.dgps import (
+    AdditiveDGP,
+    BaseDGP,
+    FriedmanDGP,
+    GeometricDGP,
+    LinearDGP,
+    PolynomialDGP,
+    SparseDGP,
+    TreeDGP,
+)
+from synthbench.pipeline import BenchPipeline, BenchResult
+from synthbench.suite import BenchSuite
+from synthbench.sweeps import difficulty_sweep, experiment_grid, severity_sweep
+
+__all__ = [
+    "__version__",
+    "AdditiveDGP",
+    "BaseDGP",
+    "BaseCorruptor",
+    "BaseLabelCorruptor",
+    "BenchPipeline",
+    "BenchResult",
+    "BenchSuite",
+    "CategoricalCorruptor",
+    "CollinearityCorruptor",
+    "difficulty_sweep",
+    "experiment_grid",
+    "FriedmanDGP",
+    "GeometricDGP",
+    "LabelNoiseCorruptor",
+    "LinearDGP",
+    "MeasurementNoiseCorruptor",
+    "MissingDataCorruptor",
+    "OutlierCorruptor",
+    "PolynomialDGP",
+    "severity_sweep",
+    "SparseDGP",
+    "TreeDGP",
+    # RandomNeuralDGP is available only with synthbench[neural] (torch required).
+    # It is exposed lazily via __getattr__ below so that importing synthbench
+    # does NOT unconditionally load torch into sys.modules.
+    "RandomNeuralDGP",
+]
+
+# Names that are only available with the neural extra (torch).
+_NEURAL_NAMES = {"RandomNeuralDGP"}
+
+
+def __getattr__(name: str):
+    """Lazy loader for optional neural symbols (requires synthbench[neural])."""
+    if name in _NEURAL_NAMES:
+        try:
+            from synthbench.dgps.neural import RandomNeuralDGP
+        except ImportError as exc:
+            raise ImportError(
+                f"{name} requires PyTorch. "
+                "Install it with: pip install synthbench[neural]"
+            ) from exc
+        # Cache in module namespace so subsequent lookups are O(1).
+        import synthbench as _self
+
+        setattr(_self, name, RandomNeuralDGP)
+        return RandomNeuralDGP
+    raise AttributeError(f"module 'synthbench' has no attribute {name!r}")

synthbench-0.1.0/src/synthbench/_seed.py

@@ -0,0 +1,43 @@
+from __future__ import annotations
+
+import numpy as np
+from numpy.random import SeedSequence
+
+
+def derive_seeds(master_state: int | SeedSequence, n: int) -> list[int]:
+    """Derive *n* independent integer seeds from a master state.
+
+    Uses ``numpy.random.SeedSequence.spawn`` so that child seeds are
+    statistically independent and fully reproducible from the master state.
+    The global numpy RNG state is never touched.
+
+    Parameters
+    ----------
+    master_state:
+        Either a plain integer that seeds the root ``SeedSequence``, or an
+        already-constructed ``SeedSequence`` (e.g. from a parent context).
+    n:
+        Number of independent child seeds to derive.
+
+    Returns
+    -------
+    list[int]
+        Plain Python ints (JSON-serializable, not numpy dtypes).
+    """
+    if isinstance(master_state, SeedSequence):
+        seq = master_state
+    else:
+        seq = SeedSequence(int(master_state))
+
+    children = seq.spawn(n)
+    # generate_state(1)[0] gives a single uint64 from the child sequence
+    return [int(child.generate_state(1)[0]) for child in children]
+
+
+def make_rng(seed: int) -> np.random.RandomState:
+    """Return a local ``numpy.random.RandomState`` seeded with *seed*.
+
+    Use ``sklearn.utils.check_random_state`` for public APIs that accept
+    ``int | None | RandomState``.
+    """
+    return np.random.RandomState(seed)

synthbench-0.1.0/src/synthbench/_version.py

@@ -0,0 +1,8 @@
+from __future__ import annotations
+
+try:
+    from importlib.metadata import PackageNotFoundError, version
+
+    __version__ = version("synthbench")
+except PackageNotFoundError:
+    __version__ = "0.0.0+dev"

synthbench-0.1.0/src/synthbench/corruptors/__init__.py

@@ -0,0 +1,23 @@
+from __future__ import annotations
+
+from synthbench.corruptors.base import BaseCorruptor
+from synthbench.corruptors.categorical import CategoricalCorruptor
+from synthbench.corruptors.collinearity import CollinearityCorruptor
+from synthbench.corruptors.label_base import (
+    BaseLabelCorruptor,
+)
+from synthbench.corruptors.label_noise import LabelNoiseCorruptor
+from synthbench.corruptors.measurement_noise import MeasurementNoiseCorruptor
+from synthbench.corruptors.missing_data import MissingDataCorruptor
+from synthbench.corruptors.outlier import OutlierCorruptor
+
+__all__ = [
+    "BaseCorruptor",
+    "BaseLabelCorruptor",
+    "CategoricalCorruptor",
+    "CollinearityCorruptor",
+    "LabelNoiseCorruptor",
+    "MeasurementNoiseCorruptor",
+    "MissingDataCorruptor",
+    "OutlierCorruptor",
+]

synthbench-0.1.0/src/synthbench/corruptors/base.py

@@ -0,0 +1,78 @@
+from __future__ import annotations
+
+from abc import ABC, abstractmethod
+
+import numpy as np
+
+_CORRUPTOR_REGISTRY: dict[str, type] = {}
+
+
+class BaseCorruptor(ABC):
+    """Abstract base class for all corruptors.
+
+    Concrete subclasses must:
+    - Declare ``key="some_key"`` in their class signature to be auto-registered.
+    - Implement [corrupt][synthbench.corruptors.base.BaseCorruptor.corrupt].
+
+    Corruptors transform X only; y is never passed in or mutated.
+
+    Example::
+
+        class CollinearityCorruptor(BaseCorruptor, key="collinearity"):
+            def corrupt(self, X, metadata, random_state):
+                ...
+    """
+
+    def __init_subclass__(cls, key: str | None = None, **kwargs: object) -> None:
+        super().__init_subclass__(**kwargs)
+        if key is not None:
+            if key in _CORRUPTOR_REGISTRY:
+                raise ValueError(
+                    f"Duplicate corruptor key '{key}': already registered by "
+                    f"{_CORRUPTOR_REGISTRY[key].__qualname__}"
+                )
+            _CORRUPTOR_REGISTRY[key] = cls
+
+    @abstractmethod
+    def corrupt(
+        self,
+        X: np.ndarray,
+        metadata: dict,
+        random_state: int,
+    ) -> tuple[np.ndarray, dict]:
+        """Apply a structural transformation to X.
+
+        Parameters
+        ----------
+        X:
+            Feature matrix of shape (n_samples, n_features). Must not be
+            modified in-place; return a new array.
+        metadata:
+            The BenchResult metadata dict produced by the DGP. Corruptors
+            may add keys (e.g. effective_feature_importances) but must not
+            remove or overwrite existing keys set by the DGP.
+        random_state:
+            Integer seed. Each corruptor derives its own RNG from this value
+            so that results are fully reproducible.
+
+        Returns
+        -------
+        X_corrupted : np.ndarray
+            Transformed feature matrix, same shape as X.
+        updated_metadata : dict
+            Metadata dict with any corruptor-specific fields added.
+        """
+
+    def get_params(self) -> dict[str, object]:
+        """Return the corruptor's current configuration as a plain dict.
+
+        Concrete corruptors should override this to return their __init__
+        parameters. Used by BenchPipeline to record component provenance.
+        """
+        raise NotImplementedError(
+            f"{type(self).__qualname__}.get_params() is not implemented. "
+            "Concrete corruptor subclasses must override this method."
+        )
+
+
+__all__ = ["BaseCorruptor", "_CORRUPTOR_REGISTRY"]