mdbench 0.1.0__py3-none-any.whl

mdbench-0.1.0.dist-info/METADATA

@@ -0,0 +1,237 @@
+Metadata-Version: 2.4
+Name: mdbench
+Version: 0.1.0
+Summary: A benchmark for discovering scientific mechanisms
+Requires-Python: >=3.12
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy>=1.24
+Requires-Dist: PyYAML>=6.0
+Requires-Dist: nd2py>=3.2.3
+Requires-Dist: openai>=1.0
+Requires-Dist: google-genai>=1.0
+Requires-Dist: requests>=2.28
+Requires-Dist: python-dotenv>=1.0
+Requires-Dist: sympy>=1.13
+Requires-Dist: scipy>=1.14
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0; extra == "dev"
+Requires-Dist: sphinx>=7.0; extra == "dev"
+Requires-Dist: sphinx-book-theme>=1.1; extra == "dev"
+Requires-Dist: myst-parser>=3.0; extra == "dev"
+Requires-Dist: sphinx-autodoc-typehints>=2.0; extra == "dev"
+Requires-Dist: build>=1.2; extra == "dev"
+Dynamic: license-file
+
+# MDBench
+
+[简体中文](README.zh-CN.md)
+
+MDBench evaluates whether an AI system can recover scientific laws and the
+mechanisms that produce them from equations or observations.
+
+## What mechanism discovery means
+
+MDBench treats a phenomenological equation as the observable consequence of
+several simple, mutually consistent relationships. The phenomenological law
+describes *what* variables do; a mechanism explains *why* through physical
+relationships, assumptions, and intermediate variables.
+
+For example, Kepler's third law for a circular orbit follows from gravitation,
+Newton's second law, and uniform circular motion. See
+[`problems/demo_problem.yaml`](problems/demo_problem.yaml).
+
+Each mechanism relationship uses `variable = formula`, where the formula must
+be parseable by [nd2py](https://pypi.org/project/nd2py/). Explicit relationships
+form a DAG:
+
+```text
+a = f1(x)
+b = f2(x, a)
+y = f3(x, a, b)
+```
+
+Implicit systems are also supported. Relationships are collected until the
+unknown variables form a closed system, then solved symbolically or with a
+numerical root finder:
+
+```text
+a = f1(x, a, b)
+b = f2(x, a, b)
+y = f3(x, a, b)
+```
+
+All variables are declared under `variable_description` as `target`, `inputs`,
+`intermediates`, or `auxiliary_inputs`. The latter are external variables used
+only by the mechanism and eliminated from the final law. The original
+relationships remain in `Problem.mechanism`; executable solution steps are
+stored in `Problem.solution`.
+
+## Tasks and evaluation
+
+MDBench provides three tasks:
+
+1. **Symbolic regression:** `(X, y) → phenomenological equation`.
+2. **Mechanism explanation:** phenomenological equation → mechanism equations.
+3. **Mechanism discovery:** `(X, y) → mechanism equations`.
+
+Mechanism evaluation reports independent metrics and deliberately has no
+overall score:
+
+- **Prediction accuracy:** for symbolic regression and mechanism discovery,
+  Pearson correlation, R², MAE, RMSE, sMAPE, and tolerance accuracy on public
+  training data (feedback) or train/ID/OOD data (final).
+- **Derived-equation equivalence:** final-only SymPy, numeric, and LLM
+  cross-check against the private phenomenological equation.
+- **Mechanism fundamentality:** LLM assessment dominated by the least
+  fundamental submitted relationship; no reference answer is required.
+- **Ground-truth structure recovery:** soft formula-AST and dependency-graph
+  matching against the reference mechanism. Variable names and numeric literal
+  values are ignored.
+- **Mechanism description complexity:** reference-free mean, maximum, and total
+  nd2py AST nodes; lower values describe simpler submitted relationships.
+
+Install MDBench with Python 3.12 or newer:
+
+```bash
+pip install -e ".[dev]"
+mdbench --help
+```
+
+The Sphinx documentation lives in [`docs/`](docs/). Build it with:
+
+```bash
+cd docs
+make html
+```
+
+## Commands
+
+All lifecycle commands accept one or more YAML files or directories through
+`--problems`; the default is `./problems`.
+
+### Validate problems
+
+Checks schemas, variable usage, units, sampling specifications, explicit and
+implicit equation solving, and derivation of the target law:
+
+```bash
+mdbench validate
+mdbench validate --problems problems/demo_problem.yaml
+```
+
+An optional LLM check evaluates whether every relationship is sufficiently
+fundamental. API or response failures are reported directly and do not fall
+back to heuristics.
+
+```bash
+mdbench validate --check-fundamentality \
+  --llm-provider deepseek --llm-model deepseek-v4-flash
+```
+
+### Generate synthetic data
+
+Creates reproducible train, ID-test, and OOD-test splits:
+
+```bash
+mdbench synthetic --problems problems/ --output-dir data/synthetic_data/
+```
+
+Each NPZ stores the three arrays, their row order in `variables`, and a JSON
+`generation_config` containing the seed and sample counts. Auxiliary inputs are
+generated here and may be hidden later during task preparation.
+
+### Prepare tasks
+
+Synthetic data must already exist. Answers are private by default:
+
+```bash
+mdbench prepare \
+  --problems problems/ \
+  --synthetic-data-dir data/synthetic_data/ \
+  --task mechanism_discovery \
+  --format directory
+```
+
+Use `--save-answer` to include answers and test splits, `--reveal-auxiliary` to
+expose auxiliary inputs in mechanism tasks, and `--force` to approve planned
+overwrites. Existing directories are never cleared; redundant files are
+reported. `--format directory` writes flat files, while `--format file` packs
+the same logical artifacts into one NPZ.
+
+### Evaluate submissions
+
+A submission may be an inline formula, semicolon-separated mechanism equations,
+or a plain-text file with one equation per non-empty line. JSON and YAML
+submissions are intentionally unsupported.
+
+```bash
+mdbench evaluate \
+  --evaluation-mode feedback \
+  --problem data/problem/PREPARED_TASK \
+  --submission submission.txt \
+  --verbose
+```
+
+Feedback mode uses only the public task and training data. Benchmark operators
+run final evaluation with `--evaluation-mode final --answer answer.json`, which
+also enables hidden ID/OOD tests and reference-mechanism recovery. For Agent
+runs, copy only the prepared public task into an isolated temporary working
+directory and require the Agent to remain there. Without source problem YAML or
+private answer artifacts, the other lifecycle commands and final evaluation
+cannot access the material they require. `--verbose` prints concise equation
+chains for explicit or implicit solution steps.
+
+Fundamentality scoring automatically uses the configured external model and
+prints its provider and model:
+
+```bash
+mdbench evaluate \
+  --evaluation-mode feedback \
+  --problem data/problem/PREPARED_TASK \
+  --submission submission.txt \
+  --llm-provider deepseek \
+  --llm-model deepseek-v4-flash
+```
+
+Standalone entry points with equivalent behavior are available in `scripts/`:
+
+```text
+validate_problem_main.py   validate problem definitions
+synthetic_data_main.py     generate synthetic datasets
+prepare_problem_main.py    prepare public/private task artifacts
+evaluate_result_main.py    evaluate a submission
+```
+
+`scripts/visualize_mechanism_main.py` renders a solved mechanism as DOT, SVG,
+PNG, or PDF. Non-DOT formats require Graphviz.
+
+## Directory conventions
+
+```text
+problems/                  source problem YAML files
+data/
+  synthetic_data/         generated train/ID/OOD datasets
+  problem/                prepared benchmark tasks
+src/
+  core/                   dependency-light data models
+  features/               project-specific I/O, solving, validation, sampling
+  metrics/                formula and mechanism metrics
+  utils/                  reusable utilities and LLM clients
+scripts/                   standalone command entry points
+tests/                     unit tests and validation fixtures
+```
+
+A prepared directory contains:
+
+```text
+problem.json               public task description
+data_train.npy             public training data
+answer.json                optional private answer
+data_id_test.npy           optional private ID test data
+data_ood_test.npy          optional private OOD test data
+```
+
+Without `--save-answer`, only `problem.json` and `data_train.npy` are written.
+Public interchange types remain simple: units are `Dict[str, int | float]`,
+formulas are nd2py-compatible strings, and arrays use NumPy formats.

mdbench-0.1.0.dist-info/RECORD

@@ -0,0 +1,57 @@
+mdbench-0.1.0.dist-info/licenses/LICENSE,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+src/evaluate_result.py,sha256=aWul9urABSXmiajwZp6AFQOh57jEu_98dTIAutYugHI,28411
+src/prepare_problem.py,sha256=vUfKsBtH9bQPjGTzX0JmfLSCXWDy7I6OjTaRvXwfbRA,11224
+src/synthetic_data.py,sha256=kGqyswRojqK3d67gA0nVek-y-2_DHU6OKP56NrRh3_E,9661
+src/validate_problem.py,sha256=FLxHY9PRCCqjpFoCsN_ewKyTqjrj5IrzJoD0wHT-f8U,14072
+src/visualize_mechanism.py,sha256=TqsEXf0d382NFakIKmZilbo8JeoYiEMpRJ7w3VE1Y3I,1998
+src/cli/run.py,sha256=A_SMsBRnTKzURKkrz7Ixtsrpnkm06jcN0QMOO9Oi4RI,2596
+src/core/__init__.py,sha256=Me6enC0oja7gfcJ5ucJk6ZrBjHyb24DVEUda6VnazZw,338
+src/core/problem.py,sha256=IG2GGZR-PMJoB5ppUkducdtsU7X7ExOdMSBJU8UZsOs,2443
+src/core/solution.py,sha256=6k2vAuf54JWGcAKOmDWvtE9ALjemuhReGDn_xV2xLlw,743
+src/features/__init__.py,sha256=k10Bq7c3CPBGR1rAfNnGhGzM1itOu5QrY7njPQ-0CXI,619
+src/features/answer.py,sha256=DTmtvHXdcyivG-x_-GxXS4OL4m2myEaRACjihIhLSck,1891
+src/features/evaluation/__init__.py,sha256=mnxEa35jljr-F-IXdUJUmXYOU0PtW5NBXJzz4SpsTrM,378
+src/features/evaluation/evaluation_package.py,sha256=B6Q25B3Sf8rdBTUeIWr2Ue_B8vK6xoBlwyxexGNDx48,4190
+src/features/evaluation/mechanism_trace.py,sha256=fCUmU_-6t5wX7v61nI1LPbVV2_sxKsIeBxs0GObRgMA,6361
+src/features/io/__init__.py,sha256=U1glfV1AqnlXRMmcffqayuf-nwljU3LA2Q777MyaPZw,377
+src/features/io/load_problem.py,sha256=Tkzxp2tNPgV0ZuRJiXD1c4lv4uWQ4vRxR35YsPFaayU,6516
+src/features/io/load_submission.py,sha256=SMg18vH95Z5EDAO3yorrPGLB9ego3YrVVNRlHZVTdOs,6576
+src/features/io/solve_mechanism_equations.py,sha256=8KWQ7Jr5I7A1XBndDgJO3bsWaRpKI9-95Rg2TxKI9MQ,11482
+src/features/sampling/__init__.py,sha256=NmGkqRIGhy75yNnaQ7XFOeSwDbYgKlgbV_aulGHjJbs,114
+src/features/sampling/range_inferrer.py,sha256=XMfOaVnCwJrsKTC5w29LQPd6HckDxhZ7-eyL75q3jZg,2156
+src/features/units/__init__.py,sha256=rmrfBBY-3r78uNWk3dnTWjQJZFIZo1t6uY498uOnR34,160
+src/features/units/unit_inference.py,sha256=8QfcxUVDv9T0JmbtGxczteAcXpiNaF3oUWHBAuF_o08,5376
+src/features/validation/__init__.py,sha256=2Zm0jthm9go76W7mCmtHfv-6ZDUM-h_w7C3MqqvbpzI,412
+src/features/validation/mechanism_derivation.py,sha256=RGoG8p7UPlEeNCfM5DPw2Gt7KGOZiSt8KPILhxWUKek,3258
+src/features/validation/mechanism_fundamentality.py,sha256=lAJXjURaPLVZkLosS35GkBLoxvZaT6LYqq6nxxO-lYg,8512
+src/features/visualization/__init__.py,sha256=RM-_qtlcERQdcEVIKTq-tHFocZhPxbojP5BxytW1Otc,140
+src/features/visualization/mechanism_graph.py,sha256=9T_zCh-AEqEfEdrmolVWZEAsH-n-puCOZw37S1ZnECg,9100
+src/metrics/__init__.py,sha256=vzbkDBNNr0OGTiDsZni0CqkvPNBrOcHSmT3LtrLjgUw,678
+src/metrics/formula_similarity.py,sha256=hZEGfIRY-PwrJzZiOcEp81yjgsl0hT_eCQISbH67Blk,1859
+src/metrics/hybrid_formula_similarity.py,sha256=qG7WqDv-RTBxoYIZbpheJaRDKY7ZAyuPiQiljEomCqQ,8776
+src/metrics/mechanism_fundamentality.py,sha256=qW4tpWf1aPk_610W4OwP-yY2MxMw32zf1d6R42Leqo8,4243
+src/metrics/mechanism_similarity.py,sha256=KX_xhhUvCwHbtnRdMaESeN4NkzaNBgH5GtS0X0_Hn5U,9030
+src/metrics/mechanism_simplicity.py,sha256=GzHbOCbg80Fc7DEOhC7Uda5SRMAJY81scZJ7Snll7Ow,1273
+src/utils/__init__.py,sha256=ILszvDXBot7dBzGfPFPln9YFmCIzUXGyL3EYjLxQ3yo,387
+src/utils/console.py,sha256=JqMIcENpaNS2S8j8XIo5D8ZZI82qRL7MAw4y_uT9rrI,574
+src/utils/lazy_loader.py,sha256=WhWJUchf3LRHG-LTjdSPPKrG_P1LJWBSUE70arRugDI,1670
+src/utils/log_exception.py,sha256=5o_HOCt7otDk9KMwg4JxxkBWYpzddpnlZV4XC4JWITg,406
+src/utils/logger.py,sha256=ZzmbZ4YZCEhIJWKD1ppe571AkbRWvduS-qOJz-ThNWw,8347
+src/utils/path_utils.py,sha256=qXa-l4U_5Rzb_NK5nxBWk7UvLo1SM_kEfe0nNsi5nc8,1043
+src/utils/tag2ansi.py,sha256=HuUkO8qAspRRBUsyutlZxpdHwoWMmUVjvLPVa8_wdog,4693
+src/utils/unit_parser.py,sha256=s-uRfMr7bdyX8qdsN94QahSpIchHAMvMHypNlcJaGos,1942
+src/utils/llm/__init__.py,sha256=OA3Kp9fGbO9FFsVks7FXUiOrm9GA-yO11cu3BVZ7i0Q,914
+src/utils/llm/core.py,sha256=u-5bdVQ17Bm2zcbUPos-SuznGd1anF1PnjiKitYZ96o,2032
+src/utils/llm/deepseek_api.py,sha256=tutynP_-l9o17gQXoa2sDWz_dhVEz6SU9h3-imnzZRE,4459
+src/utils/llm/gemini_api.py,sha256=gW8qq1ZgmsE9SVKfm0FJRr3pqrMGbl7PkVY5DdVA6tQ,3904
+src/utils/llm/llm_api.py,sha256=Y4B9fh_XARWuMEMDeQroplZ_1QGe035gmro2Zhs8G7k,6932
+src/utils/llm/manual_api.py,sha256=Xlrk4wrgeUCkpWX69Zr8mLiiCgB7fYf9EFeuVxJUE9g,2636
+src/utils/llm/openai_api.py,sha256=nPt_VAgjYgg1PHFniJseZG3sRS69D2-W3FI7qibbmv4,14472
+src/utils/llm/openrouter_api.py,sha256=yUIPoxJqrurTwVC_fR29t3b5hP6mq8v61XOJ0PvW3O0,5745
+src/utils/llm/siliconflow_api.py,sha256=gumGbL-VOKuWkBs-pLJxB2kzOd2pXskK7L23XDjElYY,7529
+src/utils/llm/tool_call_mixin.py,sha256=IF0vCSPDscnr_CRP3kshLAJXghf4fBf8wsqk6BJL3iE,3573
+mdbench-0.1.0.dist-info/METADATA,sha256=sgpANKfB8ZYFFZrmhSbEQHDp1QpLbf0J07HLVc6Eg3s,8125
+mdbench-0.1.0.dist-info/WHEEL,sha256=YVMoNqKzERt-wjUZwJ33xBGAwnFl-4cqbYkTtWa4itE,91
+mdbench-0.1.0.dist-info/entry_points.txt,sha256=2rnn_P0AwJrl1aN4Tl9kN1n42ajNVJU0u8kzp1lphd4,44
+mdbench-0.1.0.dist-info/top_level.txt,sha256=74rtVfumQlgAPzR5_2CgYN24MB0XARCg0t-gzk6gTrM,4
+mdbench-0.1.0.dist-info/RECORD,,

mdbench-0.1.0.dist-info/WHEEL

@@ -0,0 +1,5 @@
+Wheel-Version: 1.0
+Generator: setuptools (84.0.0)
+Root-Is-Purelib: true
+Tag: py3-none-any
+

mdbench-0.1.0.dist-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+mdbench = src.cli.run:cli

mdbench-0.1.0.dist-info/top_level.txt

@@ -0,0 +1 @@
+src

src/cli/run.py

@@ -0,0 +1,69 @@
+"""Command-line interface for all benchmark lifecycle operations."""
+from __future__ import annotations
+import argparse
+
+
+def get_parser() -> argparse.ArgumentParser:
+    preliminary_parser = argparse.ArgumentParser(add_help=False)
+    preliminary_parser.add_argument(
+        "command",
+        nargs="?",
+        choices=("validate", "synthetic", "prepare", "evaluate"),
+    )
+    tmp_args, _ = preliminary_parser.parse_known_args()
+
+    parser = argparse.ArgumentParser(
+        prog="mdbench",
+        description="MDBench toolkit",
+        epilog="Run 'mdbench <command> --help' for command-specific options.",
+    )
+    commands = parser.add_subparsers(dest="command", required=True)
+    if tmp_args.command == "validate":
+        from ..validate_problem import get_parser as update_parser
+        update_parser(commands.add_parser("validate"))
+    elif tmp_args.command == "synthetic":
+        from ..synthetic_data import get_parser as update_parser
+        update_parser(commands.add_parser("synthetic"))
+    elif tmp_args.command == "prepare":
+        from ..prepare_problem import get_parser as update_parser
+        update_parser(commands.add_parser("prepare"))
+    elif tmp_args.command == "evaluate":
+        from ..evaluate_result import get_parser as update_parser
+        update_parser(commands.add_parser("evaluate"))
+    else:
+        commands.add_parser("validate", help="Validate benchmark problem files")
+        commands.add_parser("synthetic", help="Generate synthetic benchmark data")
+        commands.add_parser("prepare", help="Prepare benchmark task packages")
+        commands.add_parser("evaluate", help="Evaluate benchmark submissions")
+    return parser
+
+
+def main(args) -> int:
+    if args.command == "validate":
+        from ..validate_problem import main as command_main
+        return command_main(args)
+    elif args.command == "synthetic":
+        from ..synthetic_data import main as command_main
+        return command_main(args)
+    elif args.command == "prepare":
+        from ..prepare_problem import main as command_main
+        return command_main(args)
+    elif args.command == "evaluate":
+        from ..evaluate_result import main as command_main
+        return command_main(args)
+    else:
+        raise ValueError(f"Unknown command: {args.command}")
+
+
+def cli() -> int:
+    """Parse command-line arguments and dispatch the selected command."""
+    from ..utils.logger import config_logger
+
+    config_logger()
+    parser = get_parser()
+    args = parser.parse_args()
+    return main(args)
+
+
+if __name__ == "__main__":
+    raise SystemExit(cli())

src/core/__init__.py

@@ -0,0 +1,8 @@
+# Copyright (c) 2026-present, Yumeow. Licensed under the MIT License.
+from .problem import SI, UNIT, VariableSpec, ConstantSpec, MechanismItem, Problem
+from .solution import SolutionFunction, SolutionItem
+
+__all__ = [
+    "SI", "UNIT", "VariableSpec", "ConstantSpec", "MechanismItem", "Problem",
+    "SolutionFunction", "SolutionItem",
+]

src/core/problem.py

@@ -0,0 +1,81 @@
+# Copyright (c) 2026-present, Yumeow. Licensed under the MIT License.
+from typing import List, Dict, Literal, get_args
+from dataclasses import dataclass, field
+from .solution import SolutionItem
+
+SI = Literal['kg', 'm', 's', 'A', 'K', 'mol', 'cd']
+
+class UNIT:
+    def __init__(self, unit_dict: Dict[SI, int | float]):
+        self.unit_dict = unit_dict
+
+    def __repr__(self):
+        return f"UNIT({self.unit_dict})"
+
+    def __str__(self):
+        if not self.unit_dict:
+            return "1 (dimensionless)"
+        return " ".join(
+            name if exponent == 1 else f"{name}^{exponent:g}"
+            for name, exponent in sorted(self.unit_dict.items(), key=lambda x: get_args(SI).index(x[0]))
+        )
+    
+    def __eq__(self, other):
+        if not isinstance(other, UNIT):
+            return NotImplemented
+        return self.unit_dict == other.unit_dict
+
+    def to_dict(self) -> Dict[str, int | float]:
+        """Return the dependency-free interchange representation."""
+        return dict(self.unit_dict)
+
+@dataclass
+class VariableSpec:
+    name: str
+    description: str
+    unit: UNIT | None
+    sampling: Dict[str, float | str] | None = field(default=None, kw_only=True)
+
+
+@dataclass
+class ConstantSpec(VariableSpec):
+    name: str
+    description: str
+    unit: UNIT
+    value: int | float
+
+
+@dataclass
+class MechanismItem:
+    variable: str
+    formula: str  # An nd2py-compatible right-hand-side expression.
+    formula_description: str
+
+    @property
+    def equation(self) -> str:
+        """Return the complete mechanism equation."""
+        return f"{self.variable} = {self.formula}"
+
+
+@dataclass
+class Problem:
+    problem_name: str
+    problem_description: str
+    phenomenological_formula: str  # nd2py-compatible right-hand side.
+    target_variable: VariableSpec
+    input_variables: List[VariableSpec]
+    intermediate_variables: List[VariableSpec]
+    mechanism: List[MechanismItem]
+    auxiliary_input_variables: List[VariableSpec] = field(default_factory=list)
+    constants: List[ConstantSpec] = field(default_factory=list)
+    solution: List[SolutionItem] = field(default_factory=list)
+
+    @property
+    def all_variables(self) -> List[VariableSpec]:
+        """All non-constant variables in their schema order."""
+        return [
+            self.target_variable,
+            *self.input_variables,
+            *self.intermediate_variables,
+            *self.auxiliary_input_variables,
+        ]

src/core/solution.py

@@ -0,0 +1,25 @@
+"""Executable solution steps derived from a problem's mechanism equations."""
+from __future__ import annotations
+import numpy as np
+from dataclasses import dataclass
+from typing import Callable
+
+SolutionFunction = Callable[
+    [dict[str, np.ndarray]], 
+    list[np.ndarray]
+]
+
+
+@dataclass
+class SolutionItem:
+    """One ordered mechanism-solving step.
+
+    A single-variable step is an ordinary causal evaluation. Multiple
+    ``variables`` denote values that must be obtained simultaneously from one
+    coupled equation system. ``formulas`` is aligned with ``variables`` when a
+    closed form exists, and is empty when numerical solving is required.
+    """
+
+    variables: list[str]
+    formulas: list[str]
+    function: SolutionFunction