srbf 0.1.0__tar.gz

srbf-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 Paul Saegert
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

srbf-0.1.0/PKG-INFO

@@ -0,0 +1,106 @@
+Metadata-Version: 2.4
+Name: srbf
+Version: 0.1.0
+Summary: srbf: the symbolic-regression evaluation framework (engine, adapters, benchmarks, metrics) carved from flash-ansr.
+Author: Paul Saegert
+License-Expression: MIT
+Project-URL: Repository, https://github.com/psaegert/srbf
+Project-URL: Issues, https://github.com/psaegert/srbf/issues
+Requires-Python: >=3.12
+Description-Content-Type: text/markdown
+License-File: LICENSE
+License-File: THIRD_PARTY_LICENSES
+Requires-Dist: flash-ansr~=0.6
+Requires-Dist: simplipy>=0.3.0
+Requires-Dist: numpy>=1.24
+Requires-Dist: pandas>=1.3
+Requires-Dist: scikit-learn>=1.0
+Requires-Dist: torch>=2.0
+Requires-Dist: tqdm
+Requires-Dist: pyyaml
+Requires-Dist: zss
+Requires-Dist: editdistance
+Provides-Extra: baselines
+Requires-Dist: sympy>=1.10; extra == "baselines"
+Requires-Dist: pysr; extra == "baselines"
+Requires-Dist: omegaconf; extra == "baselines"
+Provides-Extra: dev
+Requires-Dist: pre-commit; extra == "dev"
+Requires-Dist: pytest; extra == "dev"
+Requires-Dist: pytest-cov; extra == "dev"
+Requires-Dist: mypy; extra == "dev"
+Requires-Dist: flake8; extra == "dev"
+Requires-Dist: srbf[baselines]; extra == "dev"
+Dynamic: license-file
+
+# srbf: Symbolic Regression Benchmark Framework
+
+`srbf` evaluates symbolic-regression models on shared benchmarks with shared metrics. It is the
+evaluation framework carved out of [flash-ansr](https://github.com/psaegert/flash-ansr): the
+evaluation engine, model adapters, benchmarks, and metrics. It depends one-way on `flash-ansr`
+(`srbf` imports `flash-ansr`; `flash-ansr` never imports `srbf`).
+
+**Built for contributions.** Developers of SR methods add their model by opening a PR with an
+**adapter** (two methods) plus install instructions. The built-in adapters (`flash_ansr`, `pysr`,
+`nesymres`, `e2e`, `skeleton_pool`, `brute_force`) are reference examples, not a closed set. See the
+[adapter contribution guide](docs/adapters.md).
+
+> **Status: 0.1, cleanly-carved eval.** The engine seam (`srbf.eval.core` Protocols +
+> `srbf.eval.engine`) is model-agnostic, but every built-in adapter imports `flash-ansr` and the
+> adapter set is a registry edited by PR. A plugin `register_adapter()` entry-point and raw-dataset
+> (`(X, y)` CSV/parquet) ingestion are planned follow-ons.
+
+## Install
+
+```bash
+pip install srbf                 # engine + metrics + the pip-installable adapters (flash-ansr, PySR)
+pip install "srbf[baselines]"    # + pip baseline deps (sympy, pysr, omegaconf)
+```
+
+`srbf` pulls in `flash-ansr` and `simplipy` automatically. The unpackaged research baselines
+(NeSymReS, E2E) are provisioned out-of-band; see [docs/models.md](docs/models.md).
+
+## Quickstart
+
+```bash
+# 1. point srbf at a tree holding configs/, data/, and models/ (your srbf checkout works)
+export FLASH_ANSR_ROOT=$(pwd)
+
+# 2. get a model to evaluate (flash-ansr's CLI ships with srbf)
+flash_ansr install psaegert/flash-ansr-v23.0-3M
+
+# 3. fetch a benchmark + build its skeleton pool   (see docs/benchmarks.md)
+#    ...then run an evaluation:
+srbf run -c configs/evaluation/scaling/v23.0-3M_val.yaml --limit 50 -v
+```
+
+Outputs land under `results/evaluation/.../*.pkl`, one row per evaluated dataset with flat metric
+columns. Run programmatically instead:
+
+```python
+from srbf import build_evaluation_run
+
+plan = build_evaluation_run(config="configs/evaluation/scaling/v23.0-3M_val.yaml")
+plan.engine.run(limit=plan.remaining, output_path=plan.output_path)
+```
+
+## Documentation
+
+| Guide | What it covers |
+|---|---|
+| [Running evaluations](docs/running.md) | the `srbf run` CLI, config anatomy (data_source / model_adapter / runner / experiments), outputs, resume |
+| [Benchmarks & datasets](docs/benchmarks.md) | fetching FastSRB, building skeleton pools with `flash_ansr import-data`, custom sets |
+| [Models & provisioning](docs/models.md) | installing/patching the built-in models; the `model_adapter` block per type |
+| [**Adding your model**](docs/adapters.md) | the adapter protocol + registry, and the PR flow to contribute a new SR method |
+
+## Development
+
+```bash
+pip install -e ".[dev]"
+pre-commit run --all-files
+pytest tests
+```
+
+## License
+
+MIT (see [LICENSE](LICENSE)). Third-party attributions in [THIRD_PARTY_LICENSES](THIRD_PARTY_LICENSES).

srbf-0.1.0/README.md

@@ -0,0 +1,71 @@
+# srbf: Symbolic Regression Benchmark Framework
+
+`srbf` evaluates symbolic-regression models on shared benchmarks with shared metrics. It is the
+evaluation framework carved out of [flash-ansr](https://github.com/psaegert/flash-ansr): the
+evaluation engine, model adapters, benchmarks, and metrics. It depends one-way on `flash-ansr`
+(`srbf` imports `flash-ansr`; `flash-ansr` never imports `srbf`).
+
+**Built for contributions.** Developers of SR methods add their model by opening a PR with an
+**adapter** (two methods) plus install instructions. The built-in adapters (`flash_ansr`, `pysr`,
+`nesymres`, `e2e`, `skeleton_pool`, `brute_force`) are reference examples, not a closed set. See the
+[adapter contribution guide](docs/adapters.md).
+
+> **Status: 0.1, cleanly-carved eval.** The engine seam (`srbf.eval.core` Protocols +
+> `srbf.eval.engine`) is model-agnostic, but every built-in adapter imports `flash-ansr` and the
+> adapter set is a registry edited by PR. A plugin `register_adapter()` entry-point and raw-dataset
+> (`(X, y)` CSV/parquet) ingestion are planned follow-ons.
+
+## Install
+
+```bash
+pip install srbf                 # engine + metrics + the pip-installable adapters (flash-ansr, PySR)
+pip install "srbf[baselines]"    # + pip baseline deps (sympy, pysr, omegaconf)
+```
+
+`srbf` pulls in `flash-ansr` and `simplipy` automatically. The unpackaged research baselines
+(NeSymReS, E2E) are provisioned out-of-band; see [docs/models.md](docs/models.md).
+
+## Quickstart
+
+```bash
+# 1. point srbf at a tree holding configs/, data/, and models/ (your srbf checkout works)
+export FLASH_ANSR_ROOT=$(pwd)
+
+# 2. get a model to evaluate (flash-ansr's CLI ships with srbf)
+flash_ansr install psaegert/flash-ansr-v23.0-3M
+
+# 3. fetch a benchmark + build its skeleton pool   (see docs/benchmarks.md)
+#    ...then run an evaluation:
+srbf run -c configs/evaluation/scaling/v23.0-3M_val.yaml --limit 50 -v
+```
+
+Outputs land under `results/evaluation/.../*.pkl`, one row per evaluated dataset with flat metric
+columns. Run programmatically instead:
+
+```python
+from srbf import build_evaluation_run
+
+plan = build_evaluation_run(config="configs/evaluation/scaling/v23.0-3M_val.yaml")
+plan.engine.run(limit=plan.remaining, output_path=plan.output_path)
+```
+
+## Documentation
+
+| Guide | What it covers |
+|---|---|
+| [Running evaluations](docs/running.md) | the `srbf run` CLI, config anatomy (data_source / model_adapter / runner / experiments), outputs, resume |
+| [Benchmarks & datasets](docs/benchmarks.md) | fetching FastSRB, building skeleton pools with `flash_ansr import-data`, custom sets |
+| [Models & provisioning](docs/models.md) | installing/patching the built-in models; the `model_adapter` block per type |
+| [**Adding your model**](docs/adapters.md) | the adapter protocol + registry, and the PR flow to contribute a new SR method |
+
+## Development
+
+```bash
+pip install -e ".[dev]"
+pre-commit run --all-files
+pytest tests
+```
+
+## License
+
+MIT (see [LICENSE](LICENSE)). Third-party attributions in [THIRD_PARTY_LICENSES](THIRD_PARTY_LICENSES).

srbf-0.1.0/THIRD_PARTY_LICENSES

@@ -0,0 +1,84 @@
+Third-party licenses
+====================
+
+Flash-ANSR itself is distributed under the MIT License (see LICENSE).
+Portions of this project are derived from or interoperate with third-party
+work, whose licenses and notices are reproduced below.
+
+
+-------------------------------------------------------------------------------
+Neural Symbolic Regression that Scales (NeSymReS)
+-------------------------------------------------------------------------------
+
+The function `load_nesymres` in `src/srbf/compat/nesymres.py` is largely
+based on code from the "Neural Symbolic Regression that Scales" repository:
+https://github.com/SymposiumOrganization/NeuralSymbolicRegressionThatScales
+
+That project is distributed under the MIT License:
+
+    MIT License
+
+    Copyright (c) 2021 SymposiumOrganization
+
+    Permission is hereby granted, free of charge, to any person obtaining a copy
+    of this software and associated documentation files (the "Software"), to deal
+    in the Software without restriction, including without limitation the rights
+    to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+    copies of the Software, and to permit persons to whom the Software is
+    furnished to do so, subject to the following conditions:
+
+    The above copyright notice and this permission notice shall be included in all
+    copies or substantial portions of the Software.
+
+    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+    AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+    SOFTWARE.
+
+
+-------------------------------------------------------------------------------
+Fast Symbolic Regression Benchmarking (FastSRB)
+-------------------------------------------------------------------------------
+
+The FastSRB benchmark sampling code in `src/srbf/benchmarks/fastsrb.py`
+(and the `FastSRBSource` evaluation data source in
+`src/srbf/eval/data_sources.py`) is translated and adapted from the Julia
+FastSRB benchmarking code by Viktor Martinek:
+https://github.com/viktmar/FastSRB
+
+    @misc{martinek2025fastsymbolicregressionbenchmarking,
+          title={Fast Symbolic Regression Benchmarking},
+          author={Viktor Martinek},
+          year={2025},
+          eprint={2508.14481},
+          archivePrefix={arXiv},
+          primaryClass={cs.LG},
+          url={https://arxiv.org/abs/2508.14481},
+    }
+
+That project is distributed under the MIT License:
+
+    MIT License
+
+    Copyright (c) 2025 Viktor Martinek
+
+    Permission is hereby granted, free of charge, to any person obtaining a copy
+    of this software and associated documentation files (the "Software"), to deal
+    in the Software without restriction, including without limitation the rights
+    to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+    copies of the Software, and to permit persons to whom the Software is
+    furnished to do so, subject to the following conditions:
+
+    The above copyright notice and this permission notice shall be included in all
+    copies or substantial portions of the Software.
+
+    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+    AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+    SOFTWARE.

srbf-0.1.0/pyproject.toml

@@ -0,0 +1,67 @@
+# srbf: the symbolic-regression evaluation framework, carved from flash-ansr.
+# Internal self-imports are srbf.* ; the §5 contract imports stay flash_ansr.* (one-way dependency;
+# flash-ansr must NEVER import srbf).
+[project]
+name = "srbf"
+description = "srbf: the symbolic-regression evaluation framework (engine, adapters, benchmarks, metrics) carved from flash-ansr."
+authors = [{ name = "Paul Saegert" }]
+readme = "README.md"
+requires-python = ">=3.12"
+version = "0.1.0"
+license = "MIT"
+license-files = ["LICEN[CS]E*", "THIRD_PARTY_LICENSES*"]
+
+dependencies = [
+    "flash-ansr~=0.6",   # one-way dep; CI-tested 0.6.x ceiling (flash-ansr never imports srbf).
+    "simplipy>=0.3.0",   # DIRECT: eval/baselines/benchmarks all import it
+    "numpy>=1.24",
+    "pandas>=1.3",       # runtime: baselines imported top-level by eval/model_adapters.py
+    "scikit-learn>=1.0", # runtime: baselines BaseEstimator, same reason
+    "torch>=2.0",        # direct in provenance/model_adapters/token_prediction/baselines
+    "tqdm",
+    "pyyaml",            # benchmarks/fastsrb.py
+    "zss",               # moved out of flash-ansr core (eval-only)
+    "editdistance",      # moved out of flash-ansr core (eval-only)
+]
+
+[project.optional-dependencies]
+# pip-installable third-party baselines whose imports are LAZY (not needed to `import srbf`).
+baselines = [
+    "sympy>=1.10",       # E2E + NeSymReS adapters (lazy)
+    "pysr",              # PySRAdapter (lazy; also needs Julia at runtime)
+    "omegaconf",         # NeSymReS adapter glue (compat/nesymres.py)
+]
+dev = ["pre-commit", "pytest", "pytest-cov", "mypy", "flake8", "srbf[baselines]"]
+
+[project.scripts]
+srbf = "srbf.__main__:main"   # carries ONLY the eval `run` command
+
+[project.urls]
+Repository = "https://github.com/psaegert/srbf"
+Issues = "https://github.com/psaegert/srbf/issues"
+
+# NB: the NeSymReS/E2E vendored model trees + weights are NOT pip deps and never enter the wheel;
+# they are provisioned by a git-clone bench-setup flow using scripts/patch_nesymres.py.
+
+[build-system]
+requires = ["setuptools>=68", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[tool.setuptools.packages.find]
+where = ["src"]   # src/srbf/ ; the out-of-band nesymres clone tree is not under src/, so it stays out of the wheel
+
+[tool.setuptools.package-data]
+srbf = ["py.typed"]
+
+[tool.mypy]
+# Mirror flash-ansr's mypy settings (this code was carved from flash_ansr.eval and type-checked clean there).
+no_implicit_optional = false
+disallow_untyped_defs = true
+disallow_incomplete_defs = true
+explicit_package_bases = true
+exclude = "(.venv|tests/|src/srbf/compat/nesymres.py)"   # vendored nesymres types unavailable (carried over from flash-ansr)
+ignore_missing_imports = true
+
+[tool.flake8]
+ignore = ["E501", "W503"]
+per-file-ignores = "__init__.py:F401"

srbf-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

srbf-0.1.0/src/srbf/__init__.py

@@ -0,0 +1,10 @@
+"""srbf: the symbolic-regression evaluation framework, carved from flash-ansr.
+
+Engine + model adapters + benchmarks + metrics for evaluating symbolic-regression models.
+Depends one-way on flash-ansr (srbf imports flash-ansr; flash-ansr never imports srbf).
+"""
+from srbf.eval.evaluation import Evaluation
+from srbf.eval.run_config import EvaluationRunPlan, build_evaluation_run
+
+__all__ = ["Evaluation", "EvaluationRunPlan", "build_evaluation_run"]
+__version__ = "0.1.0"

srbf-0.1.0/src/srbf/__main__.py

@@ -0,0 +1,96 @@
+"""srbf command-line interface: the ``run`` subcommand carved from flash-ansr.
+
+flash-ansr keeps the rest of its CLI (train / benchmark / import-data / install / ...); only
+``run`` is evaluation-bound and lives here. The eval imports are ``srbf.eval.*``; the
+flash-ansr ``utils`` imports are the cross-repo contract (srbf depends one-way on flash-ansr).
+"""
+import argparse
+
+
+def main(argv: list[str] | None = None) -> None:
+    parser = argparse.ArgumentParser(description="srbf: symbolic-regression evaluation framework")
+    subparsers = parser.add_subparsers(dest="command_name", required=True)
+
+    run_parser = subparsers.add_parser("run", help="Run an evaluation from a unified config")
+    run_parser.add_argument('-c', '--config', type=str, required=True, help='Path to the evaluation run config file')
+    run_parser.add_argument('-n', '--limit', type=int, default=None, help='Override the sample limit specified in the config')
+    run_parser.add_argument('-o', '--output-file', type=str, default=None, help='Override the output file path from the config')
+    run_parser.add_argument('--save-every', type=int, default=None, help='Override periodic save frequency')
+    run_parser.add_argument('--no-resume', action='store_true', help='Ignore previous results even if the output file exists')
+    run_parser.add_argument('--experiment', type=str, default=None, help='Name of the experiment defined in the config to execute')
+    run_parser.add_argument('-v', '--verbose', action='store_true', help='Print a progress bar')
+
+    args = parser.parse_args(argv)
+
+    match args.command_name:
+        case 'run':
+            from srbf.eval.run_config import build_evaluation_run, EvaluationRunPlan
+            from flash_ansr.utils.config_io import load_config
+            from flash_ansr.utils.paths import substitute_root_path
+
+            config_path = substitute_root_path(args.config)
+            if args.verbose:
+                print(f"Running evaluation plan from {config_path}")
+
+            raw_config = load_config(config_path)
+            experiment_map = raw_config.get("experiments") if isinstance(raw_config, dict) else None
+
+            from srbf.eval.provenance import collect_provenance, format_provenance
+            base_prov = collect_provenance(config_path, None)
+            print(format_provenance(base_prov), flush=True)
+
+            def _execute_plan(plan: EvaluationRunPlan, experiment_name: str | None = None) -> None:
+                label = f"[{experiment_name}] " if experiment_name else ""
+                if plan.completed or plan.engine is None:
+                    if args.verbose:
+                        target = plan.total_limit or 'configured'
+                        print(f"{label}Evaluation already completed ({plan.existing_results}/{target}). Nothing to do.")
+                    return
+
+                plan.engine.run(
+                    limit=plan.remaining,
+                    save_every=plan.save_every,
+                    output_path=plan.output_path,
+                    verbose=args.verbose,
+                    progress=args.verbose,
+                    meta={**base_prov, "experiment": experiment_name},
+                )
+
+                if args.verbose:
+                    total = plan.engine.result_store.size
+                    destination = plan.output_path or 'memory'
+                    print(f"{label}Evaluation finished with {total} samples (saved to {destination}).")
+
+            if experiment_map and args.experiment is None:
+                experiment_names = list(experiment_map.keys())
+                if args.verbose:
+                    count = len(experiment_names)
+                    print(f"No --experiment provided; running all {count} experiments defined in config.")
+                for experiment_name in experiment_names:
+                    if args.verbose:
+                        print(f"--> {experiment_name}")
+                    plan = build_evaluation_run(
+                        config=config_path,
+                        limit_override=args.limit,
+                        output_override=args.output_file,
+                        save_every_override=args.save_every,
+                        resume=None if not args.no_resume else False,
+                        experiment=experiment_name,
+                    )
+                    _execute_plan(plan, experiment_name)
+            else:
+                plan = build_evaluation_run(
+                    config=config_path,
+                    limit_override=args.limit,
+                    output_override=args.output_file,
+                    save_every_override=args.save_every,
+                    resume=None if not args.no_resume else False,
+                    experiment=args.experiment,
+                )
+                _execute_plan(plan, args.experiment)
+        case _:
+            parser.print_help()
+
+
+if __name__ == "__main__":
+    main()

srbf-0.1.0/src/srbf/baselines/__init__.py

@@ -0,0 +1,7 @@
+from .skeleton_pool_model import SkeletonPoolModel
+from .brute_force_model import BruteForceModel
+
+__all__ = [
+    "SkeletonPoolModel",
+    "BruteForceModel",
+]