benchmax 0.2.0__tar.gz → 0.2.2.dev0__tar.gz

benchmax-0.2.2.dev0/PKG-INFO

@@ -0,0 +1,109 @@
+Metadata-Version: 2.4
+Name: benchmax
+Version: 0.2.2.dev0
+Summary: Platform-independent runtime for grouped LLM environments
+Author: benchmax Authors
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Requires-Python: ==3.12.*
+Requires-Dist: cloudpickle>=3.0.0
+Requires-Dist: openai>=2.15.0
+Requires-Dist: packaging>=24.0
+Provides-Extra: harbor
+Requires-Dist: harbor<0.19,>=0.18.0; extra == 'harbor'
+Description-Content-Type: text/markdown
+
+# benchmax
+
+benchmax envs is where you define datasets, how to execute the rollout, and scoring each rollout.
+
+for installation and project setup, start with the [main readme](../../README.md#get-started). working environments live in [`examples/`](../../examples/).
+
+## choose an environment
+
+all benchmax environments implement the same dataset and rollout contracts. choose the adapter based on who owns the agent loop:
+
+| environment | use it when | what it provides |
+| --- | --- | --- |
+| [`BaseEnv`](src/benchmax/envs/base/README.md) | default environment to extend - runs a simple loop with the option to make tool calls | chat completions, tool dispatch, turn limits, and reward hooks |
+| [`HarborEnv`](src/benchmax/envs/harbor/README.md) | you already have a Harbor task or harness | Harbor agents, sandboxes, verifiers, and RewardKit integration |
+| [`Environment`](src/benchmax/envs/README.md) | extend `Environment` if you need custom behavior not covered by `BaseEnv` and `HarborEnv` | the fundamental dataset, group execution, and outcome contracts |
+
+most custom environments should extend `BaseEnv`. most Harbor users configure `HarborEnv` directly rather than subclassing it.
+
+## architecture
+
+an environment defines its dataset and how a group of rollouts runs against each example.
+
+```text
+Environment
+├── create_dataset(split, base_dir, max_examples)
+│   └── Dataset
+│       └── Example(id, payload)
+└── run_group(requests)
+    ├── run_rollout(request) × group_size → RolloutAttempt × group_size
+    ├── adapter-specific scoring
+    ├── optional group-relative scoring
+    └── RolloutOutcome(rewards, termination_reason, error)
+```
+
+every environment follows this shape. the environment decides what an example contains, how each attempt runs, which tools are available, and how the result is scored.
+
+## datasets
+
+`create_dataset` receives a `train` or `eval` split and returns a fixed, ordered `Dataset` of `Example` objects.
+
+each example contains:
+
+- a stable id used to identify the datapoint across runs;
+- an environment-owned payload consumed by its rollout implementation.
+
+the optional `max_examples` argument limits how many examples are returned. when the data source supports it, the environment should stop loading once it reaches that limit.
+
+`JsonlDataset`, Harbor datasets, and custom datasets all produce the same fundamental `Dataset` type. the trainer and validation flow do not depend on the source file format.
+
+## tools
+
+`BaseEnv` exposes OpenAI-compatible function tools through `list_tools` and executes them through `run_tool`. an environment can provide no tools, one tool, or a collection of stateful tools.
+
+with `HarborEnv`, the Harbor agent and harness define the available tools and how they interact with the sandbox. benchmax does not convert Harbor tools into `BaseEnv` tools.
+
+## execution and scoring
+
+`run_group` receives multiple rollout requests for the same example, runs them concurrently, waits for all siblings, and returns one `RolloutOutcome` for each request.
+
+successful scoring hooks return their named reward components. operational failures return no rewards and do not cancel successful siblings; the trainer treats absent components as zero. partial attempts that reach a context, output, turn, or tool limit can still be scored.
+
+- `BaseEnv` runs the model and tool loop, then passes the transcript and example payload to `compute_reward`. `compute_group_rewards` can score the completed sibling group.
+- `HarborEnv` runs the configured Harbor agent and sandbox, then preserves its verifier or RewardKit reward components.
+
+### helpers
+
+`benchmax.rewards` provides deterministic text helpers, model judges, rubrics, ranking, adaptive rubrics, and diversity scoring for `BaseEnv` and direct `Environment` implementations. see the [rewards guide](src/benchmax/rewards/README.md).
+
+Harbor environments normally use their harness verifier and RewardKit instead of benchmax reward helpers.
+
+## bundling
+
+a bundle contains the environment class, its constructor arguments, the project-local source it needs, and its declared remote dependencies.
+
+```text
+environment class + constructor arguments
+                 + local source
+                 + dependency metadata
+                           │
+                           ▼
+                    portable bundle
+```
+
+benchmax creates the portable artifact so the same environment can be loaded outside the author's checkout. `castform` handles uploading the bundle, validating it remotely, and using it for training.
+
+## further reading
+
+- [base environment guide](src/benchmax/envs/base/README.md)
+- [harbor environment guide](src/benchmax/envs/harbor/README.md)
+- [reward helpers](src/benchmax/rewards/README.md)
+- [examples](../../examples/)
+- [development instructions](../../README.md#development)
+
+apache 2.0 © 2026 CGFT Inc.

benchmax-0.2.2.dev0/README.md

@@ -0,0 +1,94 @@
+# benchmax
+
+benchmax envs is where you define datasets, how to execute the rollout, and scoring each rollout.
+
+for installation and project setup, start with the [main readme](../../README.md#get-started). working environments live in [`examples/`](../../examples/).
+
+## choose an environment
+
+all benchmax environments implement the same dataset and rollout contracts. choose the adapter based on who owns the agent loop:
+
+| environment | use it when | what it provides |
+| --- | --- | --- |
+| [`BaseEnv`](src/benchmax/envs/base/README.md) | default environment to extend - runs a simple loop with the option to make tool calls | chat completions, tool dispatch, turn limits, and reward hooks |
+| [`HarborEnv`](src/benchmax/envs/harbor/README.md) | you already have a Harbor task or harness | Harbor agents, sandboxes, verifiers, and RewardKit integration |
+| [`Environment`](src/benchmax/envs/README.md) | extend `Environment` if you need custom behavior not covered by `BaseEnv` and `HarborEnv` | the fundamental dataset, group execution, and outcome contracts |
+
+most custom environments should extend `BaseEnv`. most Harbor users configure `HarborEnv` directly rather than subclassing it.
+
+## architecture
+
+an environment defines its dataset and how a group of rollouts runs against each example.
+
+```text
+Environment
+├── create_dataset(split, base_dir, max_examples)
+│   └── Dataset
+│       └── Example(id, payload)
+└── run_group(requests)
+    ├── run_rollout(request) × group_size → RolloutAttempt × group_size
+    ├── adapter-specific scoring
+    ├── optional group-relative scoring
+    └── RolloutOutcome(rewards, termination_reason, error)
+```
+
+every environment follows this shape. the environment decides what an example contains, how each attempt runs, which tools are available, and how the result is scored.
+
+## datasets
+
+`create_dataset` receives a `train` or `eval` split and returns a fixed, ordered `Dataset` of `Example` objects.
+
+each example contains:
+
+- a stable id used to identify the datapoint across runs;
+- an environment-owned payload consumed by its rollout implementation.
+
+the optional `max_examples` argument limits how many examples are returned. when the data source supports it, the environment should stop loading once it reaches that limit.
+
+`JsonlDataset`, Harbor datasets, and custom datasets all produce the same fundamental `Dataset` type. the trainer and validation flow do not depend on the source file format.
+
+## tools
+
+`BaseEnv` exposes OpenAI-compatible function tools through `list_tools` and executes them through `run_tool`. an environment can provide no tools, one tool, or a collection of stateful tools.
+
+with `HarborEnv`, the Harbor agent and harness define the available tools and how they interact with the sandbox. benchmax does not convert Harbor tools into `BaseEnv` tools.
+
+## execution and scoring
+
+`run_group` receives multiple rollout requests for the same example, runs them concurrently, waits for all siblings, and returns one `RolloutOutcome` for each request.
+
+successful scoring hooks return their named reward components. operational failures return no rewards and do not cancel successful siblings; the trainer treats absent components as zero. partial attempts that reach a context, output, turn, or tool limit can still be scored.
+
+- `BaseEnv` runs the model and tool loop, then passes the transcript and example payload to `compute_reward`. `compute_group_rewards` can score the completed sibling group.
+- `HarborEnv` runs the configured Harbor agent and sandbox, then preserves its verifier or RewardKit reward components.
+
+### helpers
+
+`benchmax.rewards` provides deterministic text helpers, model judges, rubrics, ranking, adaptive rubrics, and diversity scoring for `BaseEnv` and direct `Environment` implementations. see the [rewards guide](src/benchmax/rewards/README.md).
+
+Harbor environments normally use their harness verifier and RewardKit instead of benchmax reward helpers.
+
+## bundling
+
+a bundle contains the environment class, its constructor arguments, the project-local source it needs, and its declared remote dependencies.
+
+```text
+environment class + constructor arguments
+                 + local source
+                 + dependency metadata
+                           │
+                           ▼
+                    portable bundle
+```
+
+benchmax creates the portable artifact so the same environment can be loaded outside the author's checkout. `castform` handles uploading the bundle, validating it remotely, and using it for training.
+
+## further reading
+
+- [base environment guide](src/benchmax/envs/base/README.md)
+- [harbor environment guide](src/benchmax/envs/harbor/README.md)
+- [reward helpers](src/benchmax/rewards/README.md)
+- [examples](../../examples/)
+- [development instructions](../../README.md#development)
+
+apache 2.0 © 2026 CGFT Inc.

{benchmax-0.2.0 → benchmax-0.2.2.dev0}/pyproject.toml

@@ -1,9 +1,9 @@
 [project]
 name = "benchmax"
-version = "0.2.0"
+version = "0.2.2.dev0"
 description = "Platform-independent runtime for grouped LLM environments"
 readme = "README.md"
-authors = [{ name = "BenchMax Authors" }]
+authors = [{ name = "benchmax Authors" }]
 requires-python = "==3.12.*"
 dependencies = [
     "cloudpickle>=3.0.0",

benchmax-0.2.2.dev0/src/benchmax/auth.py

@@ -0,0 +1,187 @@
+"""Explicit, call-time authentication for model requests.
+
+benchmax defines only the runtime contract. Platform packages and execution
+runtimes provide concrete credential sources and bind injected credentials.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import contextvars
+import threading
+from collections.abc import AsyncIterator, Iterator, Mapping
+from contextlib import contextmanager
+from contextvars import ContextVar
+from dataclasses import dataclass, field
+from typing import Protocol, runtime_checkable
+
+import httpx
+
+__all__ = [
+    "InjectedAuth",
+    "ModelAuth",
+    "ModelRequestContext",
+    "RequestModelAuth",
+    "StaticBearerAuth",
+    "bind_model_auth",
+]
+
+
+@dataclass(frozen=True, slots=True)
+class ModelRequestContext:
+    """Identity of the model request about to be authorized."""
+
+    base_url: str
+    model: str
+    rollout_id: str
+
+
+@runtime_checkable
+class ModelAuth(Protocol):
+    """Return headers immediately before each model HTTP request."""
+
+    async def headers_for_request(
+        self,
+        context: ModelRequestContext,
+    ) -> Mapping[str, str]: ...
+
+
+@dataclass(frozen=True, slots=True)
+class StaticBearerAuth:
+    """Explicit bearer authentication for providers with a stable API key."""
+
+    token: str = field(repr=False)
+
+    def __post_init__(self) -> None:
+        if not isinstance(self.token, str) or not self.token:
+            raise ValueError("bearer token must be a non-empty string")
+
+    async def headers_for_request(
+        self,
+        context: ModelRequestContext,
+    ) -> Mapping[str, str]:
+        del context
+        return {"Authorization": f"Bearer {self.token}"}
+
+
+class RequestModelAuth(httpx.Auth):
+    """Apply a :class:`ModelAuth` immediately before each HTTP request.
+
+    Both sync and async OpenAI-compatible clients use this adapter, so model
+    credential selection never falls back to an SDK environment variable or a
+    separate token resolver. When a sync client runs inside an active event
+    loop, auth is resolved in a context-preserving helper thread; custom
+    providers used there must not depend on primitives bound to another loop.
+    """
+
+    def __init__(self, auth: ModelAuth, context: ModelRequestContext) -> None:
+        if not isinstance(auth, ModelAuth):
+            raise TypeError("request auth must implement ModelAuth")
+        self._auth = auth
+        self._context = context
+
+    def sync_auth_flow(
+        self,
+        request: httpx.Request,
+    ) -> Iterator[httpx.Request]:
+        headers = _resolve_headers_sync(self._auth, self._context)
+        for name, value in headers.items():
+            request.headers[name] = value
+        yield request
+
+    async def async_auth_flow(
+        self,
+        request: httpx.Request,
+    ) -> AsyncIterator[httpx.Request]:
+        headers = await self._auth.headers_for_request(self._context)
+        for name, value in headers.items():
+            request.headers[name] = value
+        yield request
+
+
+def _resolve_headers_sync(
+    auth: ModelAuth,
+    context: ModelRequestContext,
+) -> Mapping[str, str]:
+    """Resolve async ``ModelAuth`` from a synchronous HTTP client.
+
+    RAG search backends expose synchronous embedding callables. When one is
+    invoked from an async environment tool, its event loop is already running;
+    resolve the auth coroutine in a context-preserving helper thread rather
+    than attempting a nested event loop.
+    """
+
+    async def resolve() -> Mapping[str, str]:
+        return await auth.headers_for_request(context)
+
+    try:
+        asyncio.get_running_loop()
+    except RuntimeError:
+        return asyncio.run(resolve())
+
+    copied_context = contextvars.copy_context()
+    result: list[Mapping[str, str]] = []
+    failure: list[BaseException] = []
+
+    def run() -> None:
+        try:
+            result.append(copied_context.run(lambda: asyncio.run(resolve())))
+        except BaseException as error:  # propagate the original auth failure
+            failure.append(error)
+
+    thread = threading.Thread(target=run, daemon=True)
+    thread.start()
+    thread.join()
+    if failure:
+        raise failure[0]
+    if not result:
+        raise RuntimeError("model authentication did not return headers")
+    return result[0]
+
+
+_BOUND_MODEL_AUTH: ContextVar[Mapping[str, ModelAuth] | None] = ContextVar(
+    "benchmax_bound_model_auth",
+    default=None,
+)
+
+
+@dataclass(frozen=True, slots=True)
+class InjectedAuth:
+    """Serializable reference to authentication supplied by the runtime."""
+
+    name: str
+
+    def __post_init__(self) -> None:
+        if not isinstance(self.name, str) or not self.name.strip():
+            raise ValueError("injected auth name must be a non-empty string")
+
+    async def headers_for_request(
+        self,
+        context: ModelRequestContext,
+    ) -> Mapping[str, str]:
+        providers = _BOUND_MODEL_AUTH.get()
+        provider = providers.get(self.name) if providers is not None else None
+        if provider is None:
+            raise RuntimeError(f"No runtime model-auth provider was injected for {self.name!r}.")
+        if isinstance(provider, InjectedAuth):
+            raise RuntimeError(
+                f"Injected model-auth provider {self.name!r} cannot reference another InjectedAuth."
+            )
+        return await provider.headers_for_request(context)
+
+
+@contextmanager
+def bind_model_auth(providers: Mapping[str, ModelAuth]) -> Iterator[None]:
+    """Bind runtime providers for the current async execution context."""
+
+    normalized = dict(providers)
+    for name, provider in normalized.items():
+        if not isinstance(name, str) or not name.strip():
+            raise ValueError("model-auth provider names must be non-empty strings")
+        if not isinstance(provider, ModelAuth):
+            raise TypeError(f"model-auth provider {name!r} does not implement ModelAuth")
+    token = _BOUND_MODEL_AUTH.set(normalized)
+    try:
+        yield
+    finally:
+        _BOUND_MODEL_AUTH.reset(token)

{benchmax-0.2.0 → benchmax-0.2.2.dev0}/src/benchmax/bundle.py

@@ -8,7 +8,6 @@ import io
 import json
 import logging
 import pickle
-import re
 import site
 import sys
 import threading
@@ -20,11 +19,10 @@ from types import CodeType, ModuleType
 from typing import Any
 
 import cloudpickle
-from packaging.requirements import InvalidRequirement, Requirement
-from packaging.utils import canonicalize_name
-
 from benchmax.envs.environment import Environment
 from benchmax.envs.shared_types import RolloutAttempt
+from packaging.requirements import InvalidRequirement, Requirement
+from packaging.utils import canonicalize_name
 
 logger = logging.getLogger(__name__)
 
@@ -34,7 +32,7 @@ class BundlingError(Exception):
 
 
 class IncompatibleRuntimeError(Exception):
-    """Bundle metadata is incompatible with the current BenchMax runtime."""
+    """Bundle metadata is incompatible with the current benchmax runtime."""
 
 
 class IncompatiblePythonError(IncompatibleRuntimeError):
@@ -42,7 +40,7 @@ class IncompatiblePythonError(IncompatibleRuntimeError):
 
 
 class IncompatibleBenchmaxError(IncompatibleRuntimeError):
-    """Loader's BenchMax major.minor doesn't match the bundle's benchmax_version."""
+    """Loader's benchmax major.minor doesn't match the bundle's benchmax_version."""
 
 
 # register_pickle_by_value mutates process-global state; serialize against races.
@@ -72,9 +70,7 @@ class BundleMetadata:
             value = getattr(self, name)
             if not isinstance(value, str) or not value.strip():
                 raise ValueError(f"{name} must be a non-empty string")
-        if self.env_class_source is not None and not isinstance(
-            self.env_class_source, str
-        ):
+        if self.env_class_source is not None and not isinstance(self.env_class_source, str):
             raise TypeError("env_class_source must be a string or None")
 
     def to_json_bytes(self) -> bytes:
@@ -93,7 +89,7 @@ class BundleMetadata:
         ).encode("utf-8")
 
     @classmethod
-    def from_json_bytes(cls, data: bytes) -> "BundleMetadata":
+    def from_json_bytes(cls, data: bytes) -> BundleMetadata:
         try:
             d = json.loads(data.decode("utf-8"))
         except (AttributeError, UnicodeDecodeError, json.JSONDecodeError) as exc:
@@ -104,7 +100,7 @@ class BundleMetadata:
         if unknown:
             raise ValueError(
                 f"bundle metadata has unsupported keys {unknown}; "
-                "re-bundle with a current BenchMax release"
+                "re-bundle with a current benchmax release"
             )
         try:
             pip_dependencies = d["pip_dependencies"]
@@ -155,7 +151,7 @@ def bundle_digest(bundle: Bundle) -> str:
 
 
 def validate_bundle_compatibility(metadata: BundleMetadata) -> None:
-    """Reject metadata built for a different Python or BenchMax runtime.
+    """Reject metadata built for a different Python or benchmax runtime.
 
     This check never reads the pickle or installs environment dependencies, so
     execution runtimes can call it before performing either higher-risk step.
@@ -174,20 +170,20 @@ def validate_bundle_compatibility(metadata: BundleMetadata) -> None:
     current_benchmax = _benchmax_version()
     if metadata.benchmax_version == "unknown" or current_benchmax == "unknown":
         raise IncompatibleBenchmaxError(
-            "Cannot verify BenchMax compatibility because the bundle or "
-            "runtime version is unknown. Install BenchMax as a versioned package."
+            "Cannot verify benchmax compatibility because the bundle or "
+            "runtime version is unknown. Install benchmax as a versioned package."
         )
     bundle_series = _version_major_minor(metadata.benchmax_version)
     current_series = _version_major_minor(current_benchmax)
     if bundle_series is None or current_series is None:
         raise IncompatibleBenchmaxError(
-            f"Cannot parse BenchMax versions (bundle {metadata.benchmax_version}, "
+            f"Cannot parse benchmax versions (bundle {metadata.benchmax_version}, "
             f"runtime {current_benchmax}); expected major.minor[.patch]."
         )
     if bundle_series != current_series:
         raise IncompatibleBenchmaxError(
-            f"Bundle was packaged with BenchMax {metadata.benchmax_version} "
-            f"but this runtime uses BenchMax {current_benchmax}; "
+            f"Bundle was packaged with benchmax {metadata.benchmax_version} "
+            f"but this runtime uses benchmax {current_benchmax}; "
             "major.minor versions must match."
         )
 
@@ -238,7 +234,7 @@ def dump_bundle(
     benchmax_version = _benchmax_version()
     if benchmax_version == "unknown":
         raise BundlingError(
-            "Cannot determine the BenchMax package version; install BenchMax as "
+            "Cannot determine the benchmax package version; install benchmax as "
             "a versioned package before creating a bundle"
         )
     local_modules = local_modules or []
@@ -249,17 +245,14 @@ def dump_bundle(
         for mod in local_modules:
             if not isinstance(mod, ModuleType):
                 raise BundlingError(
-                    f"local_modules must contain module objects, got "
-                    f"{type(mod).__name__}: {mod!r}"
+                    f"local_modules must contain module objects, got {type(mod).__name__}: {mod!r}"
                 )
             cloudpickle.register_pickle_by_value(mod)
         try:
             try:
                 pickled = cloudpickle.dumps((env_class, constructor_args))
             except Exception as e:
-                raise BundlingError(
-                    f"Failed to serialize {env_class.__name__}: {e}"
-                ) from e
+                raise BundlingError(f"Failed to serialize {env_class.__name__}: {e}") from e
         finally:
             for mod in local_modules:
                 try:
@@ -283,9 +276,7 @@ def dump_bundle(
                     registered.append(mod)
                 for _ in range(10):
                     pending = [
-                        m
-                        for m in _unregistered_local_refs(pickled, project_roots)
-                        if m not in seen
+                        m for m in _unregistered_local_refs(pickled, project_roots) if m not in seen
                     ]
                     if not pending:
                         break
@@ -375,8 +366,7 @@ def load_bundle(
     *,
     instantiate: bool = True,
 ) -> (
-    Environment[Any, RolloutAttempt]
-    | tuple[type[Environment[Any, RolloutAttempt]], dict[str, Any]]
+    Environment[Any, RolloutAttempt] | tuple[type[Environment[Any, RolloutAttempt]], dict[str, Any]]
 ):
     """Unpickle and (optionally) instantiate.
 
@@ -389,7 +379,7 @@ def load_bundle(
             If False, return ``(env_class, constructor_args)``.
 
     Raises:
-        IncompatibleRuntimeError: bundle's Python or BenchMax version differs.
+        IncompatibleRuntimeError: bundle's Python or benchmax version differs.
         BundlingError: corrupt bytes or a class that does not implement Environment.
     """
     validate_bundle_compatibility(bundle.metadata)
@@ -407,8 +397,7 @@ def load_bundle(
     env_class, constructor_args = payload
     if not (isinstance(env_class, type) and issubclass(env_class, Environment)):
         raise BundlingError(
-            f"Unpickled class is {type(env_class).__name__}, "
-            "not an Environment implementation."
+            f"Unpickled class is {type(env_class).__name__}, not an Environment implementation."
         )
     if not isinstance(constructor_args, dict):
         raise BundlingError(
@@ -470,8 +459,7 @@ def _ensure_safe_python_version() -> None:
     v = sys.version_info
     if (v.major, v.minor) == (3, 13):
         raise BundlingError(
-            f"Python {v.major}.{v.minor}.{v.micro} is unsupported. "
-            "Use Python 3.12 or >= 3.14."
+            f"Python {v.major}.{v.minor}.{v.micro} is unsupported. Use Python 3.12 or >= 3.14."
         )
 
 
@@ -485,9 +473,7 @@ def unregistered_local_refs(pickled: bytes) -> list[str]:
 
     refs = _referenced_modules(pickled)
     project_roots = tuple(
-        root
-        for name in refs
-        if (root := _project_root_for_module_name(name)) is not None
+        root for name in refs if (root := _project_root_for_module_name(name)) is not None
     )
     return _unregistered_local_refs(pickled, project_roots)
 
@@ -641,9 +627,7 @@ def _imports_from_code(
     instructions = tuple(dis.get_instructions(code))
     names.update(_literal_dynamic_imports(instructions))
     for index, instruction in enumerate(instructions):
-        if instruction.opname != "IMPORT_NAME" or not isinstance(
-            instruction.argval, str
-        ):
+        if instruction.opname != "IMPORT_NAME" or not isinstance(instruction.argval, str):
             continue
         level = 0
         fromlist: object = None
@@ -699,14 +683,8 @@ def _literal_dynamic_imports(
             start -= 1
         call_setup = instructions[start + 1 : call_index]
         uses_import_callable = any(
-            (
-                item.opname == "LOAD_GLOBAL"
-                and item.argval in {"__import__", "import_module"}
-            )
-            or (
-                item.opname in {"LOAD_ATTR", "LOAD_METHOD"}
-                and item.argval == "import_module"
-            )
+            (item.opname == "LOAD_GLOBAL" and item.argval in {"__import__", "import_module"})
+            or (item.opname in {"LOAD_ATTR", "LOAD_METHOD"} and item.argval == "import_module")
             for item in call_setup
         )
         if not uses_import_callable:
@@ -715,9 +693,7 @@ def _literal_dynamic_imports(
             (
                 item.argval
                 for item in call_setup
-                if item.opname == "LOAD_CONST"
-                and isinstance(item.argval, str)
-                and item.argval
+                if item.opname == "LOAD_CONST" and isinstance(item.argval, str) and item.argval
             ),
             None,
         )
@@ -812,8 +788,7 @@ def _module_has_declared_distribution(
     top_level = module_name.partition(".")[0]
     distributions = importlib_metadata.packages_distributions().get(top_level, ())
     return any(
-        canonicalize_name(distribution) in declared_distributions
-        for distribution in distributions
+        canonicalize_name(distribution) in declared_distributions for distribution in distributions
     )
 
 
@@ -862,7 +837,7 @@ def _referenced_modules(pickled: bytes) -> set[str]:
         def __init__(self, *a: Any, **kw: Any) -> None:
             pass
 
-        def __call__(self, *a: Any, **kw: Any) -> "_Stub":
+        def __call__(self, *a: Any, **kw: Any) -> _Stub:
             return self
 
         def __reduce__(self) -> tuple:
@@ -964,8 +939,7 @@ def _module_is_project_local(
         module = _module_from_spec(module_name, spec)
 
     return any(
-        not _is_site_package_path(path)
-        and any(path.is_relative_to(root) for root in project_roots)
+        not _is_site_package_path(path) and any(path.is_relative_to(root) for root in project_roots)
         for path in _module_source_paths(module)
     )
 

{benchmax-0.2.0 → benchmax-0.2.2.dev0}/src/benchmax/envs/README.md

@@ -11,9 +11,8 @@ harbor/              Concrete optional adapter over native Harbor configs
 
 Most custom environments inherit [`BaseEnv`](base/env.py). It supplies the
 conversation loop and optional tool dispatch. Subclasses own dataset semantics,
-stable identity, and rewards. Every environment explicitly declares its
-complete `reward_keys`; operational failures return that shape with all values
-zero and do not cancel or distort siblings.
+stable identity, and rewards. Operational failures return no rewards and do not
+cancel or distort siblings.
 
 Custom rollout loops can inherit `Environment` directly. `HarborEnv` follows
 this path because Harbor owns the complete harness loop. Most Harbor users only