chronobench 0.1.1__tar.gz

chronobench-0.1.1/LICENSE

@@ -0,0 +1,19 @@
+Copyright (c) 2026 KU Leuven, DTAI Research Group
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

chronobench-0.1.1/PKG-INFO

@@ -0,0 +1,199 @@
+Metadata-Version: 2.4
+Name: chronobench
+Version: 0.1.1
+Author-email: Louis Carpentier <louis.carpentier@kuleuven.be>
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: tqdm
+Requires-Dist: tqdm; extra == "tqdm"
+Provides-Extra: simple-example
+Requires-Dist: scikit-learn>=1.3; extra == "simple-example"
+Dynamic: license-file
+
+Chronobench
+===========
+
+A registry-driven ML experiment benchmarking framework. You register your own data loaders, models, and metrics (or refer to library ones by import path) and write one training loop — Chronobench handles configuration, parameter sweeps, parallelism, and CSV output. Your domain code stays in its own modules and never needs to import Chronobench or change to add an experiment.
+
+## Installation
+
+Install Chronobench as follows:
+```bash
+pip install git+https://github.com/LouisCarpentier42/Chronobench
+```
+
+## Quick Start
+
+Keep your components in plain modules (here `loaders.py`), then write a `script.py` that registers them, defines a `runner`, and calls `chronobench.main()`:
+
+```python
+import chronobench as cb
+from loaders import IrisLoader, BreastCancerLoader
+
+cb.data_loaders.register(iris=IrisLoader, breast_cancer=BreastCancerLoader)
+
+@cb.runner
+def runner(data_loader, model, metrics) -> dict:
+    X_train, X_test, y_train, y_test = data_loader.load()
+    model.fit(X_train, y_train)
+    y_pred = model.predict(X_test)
+    return {m.func.__name__: m(y_test, y_pred) for m in metrics}
+
+if __name__ == "__main__":
+    cb.main()
+```
+
+Experiment TOML entries pick an implementation by registered `name` or by dotted `target` import path:
+
+```toml
+[[data]]
+name = "iris"                              # a registered name
+test_size = 0.2
+
+[[models]]
+target = "sklearn.svm.SVC"                 # an import path - no registration
+kernel = "linear"
+
+[[metrics]]
+target = "sklearn.metrics.accuracy_score"  # an import path — no registration
+```
+
+Then run it from the command line:
+
+```bash
+python script.py my_experiment
+```
+
+See [`examples/simple/`](examples/simple/) for a complete working example with scikit-learn.
+
+## Concepts
+
+### Registering components
+
+Chronobench keeps three registries — `cb.data_loaders`, `cb.models`, and `cb.metrics` — plus one `runner`. A *factory* is any callable that produces a component: a class, or a function defined in your script. Register factories by short name:
+
+```python
+cb.data_loaders.register(<short-name>=<object>)
+```
+
+Registration is optional: anything importable can be referenced by its `target` path instead (see below), so in practice you register the components for which you want a short, stable name — typically your own project code — and refer to library components by import path.
+
+Because `data_loader` and `model` are whatever the factory returns, you are not constrained to object-oriented wrappers — returning raw data (e.g., a tuple of arrays), a file path, or a config dict is equally valid as long as `runner` knows how to use it. The `runner` returns a dict whose keys become CSV columns.
+
+### `name` vs `target`
+
+Each TOML entry selects its implementation with **exactly one** of:
+
+- `name` — a name you registered in the script. Short, refactor-safe, and validated (an unknown name fails fast with the list of available names). Best for your own, frequently used components.
+- `target` — a dotted import path (e.g. `"sklearn.svm.SVC"`). Resolved by import, so any library class or function works with no wrapper and no registration. Best for third-party components.
+
+All other keys in the entry are forwarded to the factory as `**kwargs`.
+
+### Factories return lists or single instances
+
+Each factory may return a single instance or a list of instances. A single instance produces one job variant, a list produces one variant per element, each crossed with every other-side variant. This allows you to easily run multiple datasets and/or models in the same experiment without writing any loops or separate config files.
+
+
+A factory may return a **list** of instances. A single `[[data]]` (or `[[models]]`) entry then fans out into one variant per element, each crossed with every model (resp. data) variant — i.e. one CSV row per `(item × other-side-config)`. Each row is labeled by the instance's `name` attribute when present, otherwise by the entry label plus an index.
+
+```python
+def all_datasets(test_size=0.2, random_state=None):
+    return [
+        IrisLoader(test_size=test_size, random_state=random_state),
+        BreastCancerLoader(test_size=test_size, random_state=random_state),
+    ]
+
+cb.data_loaders.register(all=all_datasets)   # one [[data]] entry -> two dataset rows
+```
+
+To instead keep a single job that internally manages several splits (e.g. a tuning + evaluation set in one folder), just return one object whose `load()` exposes all splits and let `runner` use them.
+
+### Metrics: scoring functions without wrappers
+
+A metric is a callable the `runner` invokes with data (e.g. `metric(y_true, y_pred)`). You can use a class whose instance is that callable, but a **bare scoring function works directly** — by `target` (no registration) or by a registered `name`:
+
+```toml
+[[metrics]]
+target = "sklearn.metrics.fbeta_score"
+beta = 1.0
+average = "macro"   # bound into the function; remaining y_true/y_pred come from the runner
+```
+
+When the config kwargs don't satisfy every required parameter (here `fbeta_score` still needs `y_true`/`y_pred`), Chronobench binds the given arguments with `functools.partial` and lets the `runner` supply the rest, instead of calling the function immediately. The result is a bound function, so name the CSV column from it in your runner (e.g., `metric.func.__name__`). Note that a bare function uses *its own* defaults (e.g., `fbeta_score` defaults to `average="binary"`, which fails on multiclass data), so set what you need in the TOML.
+
+To register such a function under a short name instead, the optional convenience is:
+
+```python
+from sklearn.metrics import accuracy_score, fbeta_score
+cb.metrics.register(accuracy=accuracy_score, fbeta=fbeta_score)
+```
+
+### Context and environment injection
+
+Factories are called with their TOML `**kwargs`. Chronobench also passes `environment` and/or `context` **only when the factory declares them as explicit parameters**, so plain and library classes work untouched while advanced factories can opt in:
+
+```python
+def make_model(context, n_layers=2):     # receives the cross-initializer context
+    return MyModel(n_classes=context.data_loader.n_classes, n_layers=n_layers)
+```
+
+`environment` is the parsed `environment.toml`. `context` is a `Context` dataclass built progressively across stages:
+
+| Stage | Populated `context` fields |
+|---|---|
+| data loaders | none |
+| models | `data_loader`, `data_loader_name`, `data_loader_kwargs` |
+| metrics | above + `model`, `model_name`, `model_kwargs` |
+
+> `name`, `target`, `context`, and `environment` are reserved keys and must not be used as TOML parameter names.
+
+### Configuration files
+
+**`environment.toml`** — global settings, required keys:
+
+```toml
+path-to-results     = "./results"
+path-to-experiments = "./experiments"
+n-jobs              = 4
+```
+
+**`experiments/<name>.toml`** — one file per experiment, with `[[data]]`, `[[models]]`, and `[[metrics]]` arrays-of-tables. `[[data]]` and `[[models]]` entries produce job variants (their Cartesian product is the final job list); `[[metrics]]` entries are collected into a single flat list used for every job.
+
+### Sweep parameters
+
+Add `sweep.<key> = [v1, v2, ...]` to any entry to generate a parameter grid. The framework expands all sweep keys within a block via Cartesian product, then takes the product of data configs × model configs to produce the final job list. Sweep expansion on `[[metrics]]` entries only adds more metric instances per job — it does not create additional jobs.
+
+```toml
+[[data]]
+name = "iris"
+sweep.random_state = [0, 1, 2]   # 3 variants
+
+[[models]]  # 2 × 2 = 4 model variants
+target = "sklearn.ensemble.RandomForestClassifier"
+sweep.criterion = ["gini", "entropy"]  # 2 variants
+sweep.max_depth  = [5, 10]             # 2 variants
+
+[[metrics]]
+target = "sklearn.metrics.fbeta_score"
+average = "macro"
+sweep.beta = [0.5, 1.0, 2.0]  # 3 variants
+```
+
+This produces 3 × 4 = 12 jobs total. For each job, 3 metrics are computed ($F_\beta$ for all $\beta \in \{0.5, 1.0, 2.0\}$).
+
+### Results
+
+Results are written to:
+
+```
+{path-to-results}/raw/{experiment_name}/YYYYMMDD-HHMMSS.csv
+```
+
+Columns are the union of all keys returned by `runner()`, plus `Dataset` and `Model` metadata columns. If a key appears in both data loader and model kwargs, it is prefixed with `Dataset.` / `Model.` to avoid collision.
+
+Run `python script.py --help` for the full list of options.
+
+### Roadmap
+
+This project is in early development, and many changes are still expected. However, the main interface should remain stable. Go to the [issue page](https://github.com/LouisCarpentier42/Chronobench/issues) for an overview of the tasks we plan to add in the near future.

chronobench-0.1.1/README.md

@@ -0,0 +1,186 @@
+Chronobench
+===========
+
+A registry-driven ML experiment benchmarking framework. You register your own data loaders, models, and metrics (or refer to library ones by import path) and write one training loop — Chronobench handles configuration, parameter sweeps, parallelism, and CSV output. Your domain code stays in its own modules and never needs to import Chronobench or change to add an experiment.
+
+## Installation
+
+Install Chronobench as follows:
+```bash
+pip install git+https://github.com/LouisCarpentier42/Chronobench
+```
+
+## Quick Start
+
+Keep your components in plain modules (here `loaders.py`), then write a `script.py` that registers them, defines a `runner`, and calls `chronobench.main()`:
+
+```python
+import chronobench as cb
+from loaders import IrisLoader, BreastCancerLoader
+
+cb.data_loaders.register(iris=IrisLoader, breast_cancer=BreastCancerLoader)
+
+@cb.runner
+def runner(data_loader, model, metrics) -> dict:
+    X_train, X_test, y_train, y_test = data_loader.load()
+    model.fit(X_train, y_train)
+    y_pred = model.predict(X_test)
+    return {m.func.__name__: m(y_test, y_pred) for m in metrics}
+
+if __name__ == "__main__":
+    cb.main()
+```
+
+Experiment TOML entries pick an implementation by registered `name` or by dotted `target` import path:
+
+```toml
+[[data]]
+name = "iris"                              # a registered name
+test_size = 0.2
+
+[[models]]
+target = "sklearn.svm.SVC"                 # an import path - no registration
+kernel = "linear"
+
+[[metrics]]
+target = "sklearn.metrics.accuracy_score"  # an import path — no registration
+```
+
+Then run it from the command line:
+
+```bash
+python script.py my_experiment
+```
+
+See [`examples/simple/`](examples/simple/) for a complete working example with scikit-learn.
+
+## Concepts
+
+### Registering components
+
+Chronobench keeps three registries — `cb.data_loaders`, `cb.models`, and `cb.metrics` — plus one `runner`. A *factory* is any callable that produces a component: a class, or a function defined in your script. Register factories by short name:
+
+```python
+cb.data_loaders.register(<short-name>=<object>)
+```
+
+Registration is optional: anything importable can be referenced by its `target` path instead (see below), so in practice you register the components for which you want a short, stable name — typically your own project code — and refer to library components by import path.
+
+Because `data_loader` and `model` are whatever the factory returns, you are not constrained to object-oriented wrappers — returning raw data (e.g., a tuple of arrays), a file path, or a config dict is equally valid as long as `runner` knows how to use it. The `runner` returns a dict whose keys become CSV columns.
+
+### `name` vs `target`
+
+Each TOML entry selects its implementation with **exactly one** of:
+
+- `name` — a name you registered in the script. Short, refactor-safe, and validated (an unknown name fails fast with the list of available names). Best for your own, frequently used components.
+- `target` — a dotted import path (e.g. `"sklearn.svm.SVC"`). Resolved by import, so any library class or function works with no wrapper and no registration. Best for third-party components.
+
+All other keys in the entry are forwarded to the factory as `**kwargs`.
+
+### Factories return lists or single instances
+
+Each factory may return a single instance or a list of instances. A single instance produces one job variant, a list produces one variant per element, each crossed with every other-side variant. This allows you to easily run multiple datasets and/or models in the same experiment without writing any loops or separate config files.
+
+
+A factory may return a **list** of instances. A single `[[data]]` (or `[[models]]`) entry then fans out into one variant per element, each crossed with every model (resp. data) variant — i.e. one CSV row per `(item × other-side-config)`. Each row is labeled by the instance's `name` attribute when present, otherwise by the entry label plus an index.
+
+```python
+def all_datasets(test_size=0.2, random_state=None):
+    return [
+        IrisLoader(test_size=test_size, random_state=random_state),
+        BreastCancerLoader(test_size=test_size, random_state=random_state),
+    ]
+
+cb.data_loaders.register(all=all_datasets)   # one [[data]] entry -> two dataset rows
+```
+
+To instead keep a single job that internally manages several splits (e.g. a tuning + evaluation set in one folder), just return one object whose `load()` exposes all splits and let `runner` use them.
+
+### Metrics: scoring functions without wrappers
+
+A metric is a callable the `runner` invokes with data (e.g. `metric(y_true, y_pred)`). You can use a class whose instance is that callable, but a **bare scoring function works directly** — by `target` (no registration) or by a registered `name`:
+
+```toml
+[[metrics]]
+target = "sklearn.metrics.fbeta_score"
+beta = 1.0
+average = "macro"   # bound into the function; remaining y_true/y_pred come from the runner
+```
+
+When the config kwargs don't satisfy every required parameter (here `fbeta_score` still needs `y_true`/`y_pred`), Chronobench binds the given arguments with `functools.partial` and lets the `runner` supply the rest, instead of calling the function immediately. The result is a bound function, so name the CSV column from it in your runner (e.g., `metric.func.__name__`). Note that a bare function uses *its own* defaults (e.g., `fbeta_score` defaults to `average="binary"`, which fails on multiclass data), so set what you need in the TOML.
+
+To register such a function under a short name instead, the optional convenience is:
+
+```python
+from sklearn.metrics import accuracy_score, fbeta_score
+cb.metrics.register(accuracy=accuracy_score, fbeta=fbeta_score)
+```
+
+### Context and environment injection
+
+Factories are called with their TOML `**kwargs`. Chronobench also passes `environment` and/or `context` **only when the factory declares them as explicit parameters**, so plain and library classes work untouched while advanced factories can opt in:
+
+```python
+def make_model(context, n_layers=2):     # receives the cross-initializer context
+    return MyModel(n_classes=context.data_loader.n_classes, n_layers=n_layers)
+```
+
+`environment` is the parsed `environment.toml`. `context` is a `Context` dataclass built progressively across stages:
+
+| Stage | Populated `context` fields |
+|---|---|
+| data loaders | none |
+| models | `data_loader`, `data_loader_name`, `data_loader_kwargs` |
+| metrics | above + `model`, `model_name`, `model_kwargs` |
+
+> `name`, `target`, `context`, and `environment` are reserved keys and must not be used as TOML parameter names.
+
+### Configuration files
+
+**`environment.toml`** — global settings, required keys:
+
+```toml
+path-to-results     = "./results"
+path-to-experiments = "./experiments"
+n-jobs              = 4
+```
+
+**`experiments/<name>.toml`** — one file per experiment, with `[[data]]`, `[[models]]`, and `[[metrics]]` arrays-of-tables. `[[data]]` and `[[models]]` entries produce job variants (their Cartesian product is the final job list); `[[metrics]]` entries are collected into a single flat list used for every job.
+
+### Sweep parameters
+
+Add `sweep.<key> = [v1, v2, ...]` to any entry to generate a parameter grid. The framework expands all sweep keys within a block via Cartesian product, then takes the product of data configs × model configs to produce the final job list. Sweep expansion on `[[metrics]]` entries only adds more metric instances per job — it does not create additional jobs.
+
+```toml
+[[data]]
+name = "iris"
+sweep.random_state = [0, 1, 2]   # 3 variants
+
+[[models]]  # 2 × 2 = 4 model variants
+target = "sklearn.ensemble.RandomForestClassifier"
+sweep.criterion = ["gini", "entropy"]  # 2 variants
+sweep.max_depth  = [5, 10]             # 2 variants
+
+[[metrics]]
+target = "sklearn.metrics.fbeta_score"
+average = "macro"
+sweep.beta = [0.5, 1.0, 2.0]  # 3 variants
+```
+
+This produces 3 × 4 = 12 jobs total. For each job, 3 metrics are computed ($F_\beta$ for all $\beta \in \{0.5, 1.0, 2.0\}$).
+
+### Results
+
+Results are written to:
+
+```
+{path-to-results}/raw/{experiment_name}/YYYYMMDD-HHMMSS.csv
+```
+
+Columns are the union of all keys returned by `runner()`, plus `Dataset` and `Model` metadata columns. If a key appears in both data loader and model kwargs, it is prefixed with `Dataset.` / `Model.` to avoid collision.
+
+Run `python script.py --help` for the full list of options.
+
+### Roadmap
+
+This project is in early development, and many changes are still expected. However, the main interface should remain stable. Go to the [issue page](https://github.com/LouisCarpentier42/Chronobench/issues) for an overview of the tasks we plan to add in the near future.

chronobench-0.1.1/pyproject.toml

@@ -0,0 +1,66 @@
+[project]
+name = "chronobench"
+version = "0.1.1"
+authors = [
+    {name = "Louis Carpentier", email = "louis.carpentier@kuleuven.be"}
+]
+readme = "README.md"
+requires-python = ">=3.11"
+dependencies = []
+
+[project.optional-dependencies]
+tqdm = [
+    "tqdm",
+]
+simple-example = [
+    "scikit-learn>=1.3",
+]
+
+[dependency-groups]
+dev = [
+    "mypy>=1.19.1",
+    "pre-commit>=4.5.1",
+    "python-dotenv>=1.2.2",
+    "ruff>=0.14.11",
+    "numpydoc>=1.5.0",
+]
+test = [
+    "pytest>=9.0.3",
+    "pytest-cov>=6.0.0",
+]
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.uv]
+package = true
+
+[tool.numpydoc_validation]
+checks = [
+    "all",
+    "SA01",  # See Also section not required
+    "EX01",  # Examples section not required
+]
+exclude = [
+    '\._',  # Ignore private objects
+]
+
+[[tool.mypy.overrides]]
+module = [
+    "sklearn.*",
+    "tqdm.*",
+    "aeon.*",
+    "pytest.*",
+]
+ignore_missing_imports = true
+
+[tool.pytest.ini_options]
+addopts = "--cov=chronobench --cov-report=term-missing --cov-report=html"
+
+[tool.coverage.run]
+source = ["src/chronobench"]
+omit = ["src/chronobench/__init__.py"]
+
+[tool.coverage.report]
+show_missing = true
+skip_covered = false

chronobench-0.1.1/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

chronobench-0.1.1/src/chronobench/__init__.py

@@ -0,0 +1,16 @@
+"""
+Chronobench: a registry-driven framework for benchmarking machine learning experiments.
+"""
+
+from ._main import main
+from .context import Context
+from .registry import data_loaders, metrics, models, runner
+
+__all__ = [
+    "main",
+    "Context",
+    "data_loaders",
+    "models",
+    "metrics",
+    "runner",
+]

chronobench-0.1.1/src/chronobench/_main.py

@@ -0,0 +1,61 @@
+"""
+Entry point that dispatches CLI arguments to the appropriate execution mode.
+"""
+
+from . import registry
+from .modes.dry_run import main as main_dry_run
+from .modes.run_many import main as main_run_many
+from .utility.loading import load_environment, load_experiment_dict
+from .utility.parsing_arguments import parse_args
+
+__all__ = ["main"]
+
+
+def main():
+    """
+    Run chronobench from the command line using the registered components.
+
+    This is the primary public API. Before calling it, register your data
+    loaders, models, and metrics on the module-level registries and mark your
+    training loop with :func:`chronobench.runner`::
+
+        import chronobench as cb
+        from src.loaders import IrisLoader
+
+        cb.data_loaders.register(iris=IrisLoader)
+
+        @cb.runner
+        def runner(data_loader, model, metrics):
+            ...
+
+        cb.main()
+
+    Experiment TOML entries select an implementation with either a registered
+    ``name`` or a dotted ``target`` import path, so stock library classes (e.g.
+    ``sklearn.svm.SVC``) can be used without a wrapper. ``main()`` reads the
+    registries and the registered runner directly; nothing is passed to it.
+    """
+    args = parse_args()
+    environment = load_environment(args.environment)
+    experiments = load_experiment_dict(environment, args.experiments)
+
+    if args.dry_run:
+        if args.debug:
+            print("Warning: --debug has no effect in dry-run mode.")
+        main_dry_run(
+            experiments=experiments,
+            data_loaders=registry.data_loaders,
+            models=registry.models,
+            metrics=registry.metrics,
+            environment=environment,
+        )
+    else:
+        main_run_many(
+            experiments=experiments,
+            debug=args.debug,
+            data_loaders=registry.data_loaders,
+            models=registry.models,
+            metrics=registry.metrics,
+            runner=registry.get_runner(),
+            environment=environment,
+        )

chronobench-0.1.1/src/chronobench/context.py

@@ -0,0 +1,42 @@
+"""
+Context dataclass passed to user-supplied initializer callbacks.
+"""
+
+from dataclasses import dataclass
+
+
+@dataclass
+class Context:
+    """
+    Cross-initializer context passed progressively to each callback.
+
+    The context is populated in stages as initialization proceeds:
+
+    - ``initialize_data_loaders`` receives ``Context()`` (all fields ``None``).
+    - ``initialize_models`` receives a ``Context`` with the data-loader fields set.
+    - ``initialize_metrics`` receives a fully populated ``Context``.
+
+    Attributes
+    ----------
+    data_loader : object, optional
+        The instantiated data loader for the current job.
+    data_loader_name : str, optional
+        The name of the current data loader as defined in the experiment
+        configuration.
+    data_loader_kwargs : dict, optional
+        The keyword arguments used to instantiate the current data loader.
+    model : object, optional
+        The instantiated model for the current job.
+    model_name : str, optional
+        The name of the current model as defined in the experiment
+        configuration.
+    model_kwargs : dict, optional
+        The keyword arguments used to instantiate the current model.
+    """
+
+    data_loader: object | None = None
+    data_loader_name: str | None = None
+    data_loader_kwargs: dict | None = None
+    model: object | None = None
+    model_name: str | None = None
+    model_kwargs: dict | None = None