benchmax 0.2.0__tar.gz → 0.2.2__tar.gz

benchmax-0.2.2/PKG-INFO

@@ -0,0 +1,111 @@
+Metadata-Version: 2.4
+Name: benchmax
+Version: 0.2.2
+Summary: Platform-independent runtime for grouped LLM environments
+Author: benchmax Authors
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Requires-Python: ==3.12.*
+Requires-Dist: cloudpickle>=3.0.0
+Requires-Dist: openai>=2.15.0
+Requires-Dist: packaging>=24.0
+Provides-Extra: harbor
+Requires-Dist: harbor<0.19,>=0.18.0; extra == 'harbor'
+Description-Content-Type: text/markdown
+
+# benchmax
+
+benchmax envs is where you define datasets, how to execute the rollout, and scoring each rollout.
+
+for installation and project setup, start with the [main readme](../../README.md#get-started). working environments live in [`examples/`](../../examples/).
+
+## choose an environment
+
+all benchmax environments implement the same dataset and rollout contracts. choose the adapter based on who owns the agent loop:
+
+| environment | use it when | what it provides |
+| --- | --- | --- |
+| [`BaseEnv`](src/benchmax/envs/base/README.md) | default environment to extend - runs a simple loop with the option to make tool calls | chat completions, tool dispatch, turn limits, and reward hooks |
+| [`HarborEnv`](src/benchmax/envs/harbor/README.md) | you already have a Harbor task or harness | Harbor agents, sandboxes, verifiers, and RewardKit integration |
+| [`Environment`](src/benchmax/envs/README.md) | extend `Environment` if you need custom behavior not covered by `BaseEnv` and `HarborEnv` | the fundamental dataset, group execution, and outcome contracts |
+
+most custom environments should extend `BaseEnv`. most Harbor users configure `HarborEnv` directly rather than subclassing it.
+
+## architecture
+
+an environment defines its dataset and how a group of rollouts runs against each example.
+
+```text
+Environment
+├── create_dataset(split, base_dir, max_examples)
+│   └── Dataset
+│       └── Example(id, payload)
+└── run_group(requests)
+    ├── run_rollout(request) × group_size → RolloutAttempt × group_size
+    ├── adapter-specific scoring
+    ├── optional group-relative scoring
+    └── RolloutOutcome(rewards, termination_reason, error)
+```
+
+every environment follows this shape. the environment decides what an example contains, how each attempt runs, which tools are available, and how the result is scored.
+
+## datasets
+
+`create_dataset` receives a `train` or `eval` split and returns a fixed, ordered `Dataset` of `Example` objects.
+
+each example contains:
+
+- a stable id used to identify the datapoint across runs;
+- an environment-owned payload consumed by its rollout implementation.
+
+the optional `max_examples` argument limits how many examples are returned. when the data source supports it, the environment should stop loading once it reaches that limit.
+
+`JsonlDataset`, Harbor datasets, and custom datasets all produce the same fundamental `Dataset` type. the trainer and validation flow do not depend on the source file format.
+
+## tools
+
+`BaseEnv` exposes OpenAI-compatible function tools through `list_tools` and executes them through `run_tool`. an environment can provide no tools, one tool, or a collection of stateful tools.
+
+with `HarborEnv`, the Harbor agent and harness define the available tools and how they interact with the sandbox. benchmax does not convert Harbor tools into `BaseEnv` tools.
+
+## execution and scoring
+
+`run_group` receives multiple rollout requests for the same example, runs them concurrently, waits for all siblings, and returns one `RolloutOutcome` for each request.
+
+successful scoring hooks return their named reward components. operational failures return no rewards and do not cancel successful siblings; the trainer treats absent components as zero. partial attempts that reach a context, output, turn, or tool limit can still be scored.
+
+- `BaseEnv` runs the model and tool loop, then passes the transcript and example payload to `compute_reward`. `compute_group_rewards` can score the completed sibling group.
+- `HarborEnv` runs the configured Harbor agent and sandbox, then preserves its verifier or RewardKit reward components.
+
+### helpers
+
+`benchmax.rewards` provides deterministic text helpers, model judges, rubrics, ranking, adaptive rubrics, and diversity scoring for `BaseEnv` and direct `Environment` implementations. see the [rewards guide](src/benchmax/rewards/README.md).
+
+Harbor environments normally use their harness verifier and RewardKit instead of benchmax reward helpers.
+
+## bundling
+
+a bundle contains the environment class, its constructor arguments, the project-local source it needs, and its declared remote dependencies.
+
+```text
+environment class + constructor arguments
+                 + local source
+                 + dependency metadata
+                           │
+                           ▼
+                    portable bundle
+```
+
+benchmax creates the portable artifact so the same environment can be loaded outside the author's checkout. `castform` handles uploading the bundle, validating it remotely, and using it for training.
+
+Remote runtimes install `pip_dependencies` without enabling prereleases globally. If a dependency graph needs a prerelease, list that package explicitly even when it would normally be transitive. For example, use `pip_dependencies=["parent-package==1.0.0", "transitive-package==2.0.0rc1"]`.
+
+## further reading
+
+- [base environment guide](src/benchmax/envs/base/README.md)
+- [harbor environment guide](src/benchmax/envs/harbor/README.md)
+- [reward helpers](src/benchmax/rewards/README.md)
+- [examples](../../examples/)
+- [development instructions](../../README.md#development)
+
+apache 2.0 © 2026 CGFT Inc.

benchmax-0.2.2/README.md

@@ -0,0 +1,96 @@
+# benchmax
+
+benchmax envs is where you define datasets, how to execute the rollout, and scoring each rollout.
+
+for installation and project setup, start with the [main readme](../../README.md#get-started). working environments live in [`examples/`](../../examples/).
+
+## choose an environment
+
+all benchmax environments implement the same dataset and rollout contracts. choose the adapter based on who owns the agent loop:
+
+| environment | use it when | what it provides |
+| --- | --- | --- |
+| [`BaseEnv`](src/benchmax/envs/base/README.md) | default environment to extend - runs a simple loop with the option to make tool calls | chat completions, tool dispatch, turn limits, and reward hooks |
+| [`HarborEnv`](src/benchmax/envs/harbor/README.md) | you already have a Harbor task or harness | Harbor agents, sandboxes, verifiers, and RewardKit integration |
+| [`Environment`](src/benchmax/envs/README.md) | extend `Environment` if you need custom behavior not covered by `BaseEnv` and `HarborEnv` | the fundamental dataset, group execution, and outcome contracts |
+
+most custom environments should extend `BaseEnv`. most Harbor users configure `HarborEnv` directly rather than subclassing it.
+
+## architecture
+
+an environment defines its dataset and how a group of rollouts runs against each example.
+
+```text
+Environment
+├── create_dataset(split, base_dir, max_examples)
+│   └── Dataset
+│       └── Example(id, payload)
+└── run_group(requests)
+    ├── run_rollout(request) × group_size → RolloutAttempt × group_size
+    ├── adapter-specific scoring
+    ├── optional group-relative scoring
+    └── RolloutOutcome(rewards, termination_reason, error)
+```
+
+every environment follows this shape. the environment decides what an example contains, how each attempt runs, which tools are available, and how the result is scored.
+
+## datasets
+
+`create_dataset` receives a `train` or `eval` split and returns a fixed, ordered `Dataset` of `Example` objects.
+
+each example contains:
+
+- a stable id used to identify the datapoint across runs;
+- an environment-owned payload consumed by its rollout implementation.
+
+the optional `max_examples` argument limits how many examples are returned. when the data source supports it, the environment should stop loading once it reaches that limit.
+
+`JsonlDataset`, Harbor datasets, and custom datasets all produce the same fundamental `Dataset` type. the trainer and validation flow do not depend on the source file format.
+
+## tools
+
+`BaseEnv` exposes OpenAI-compatible function tools through `list_tools` and executes them through `run_tool`. an environment can provide no tools, one tool, or a collection of stateful tools.
+
+with `HarborEnv`, the Harbor agent and harness define the available tools and how they interact with the sandbox. benchmax does not convert Harbor tools into `BaseEnv` tools.
+
+## execution and scoring
+
+`run_group` receives multiple rollout requests for the same example, runs them concurrently, waits for all siblings, and returns one `RolloutOutcome` for each request.
+
+successful scoring hooks return their named reward components. operational failures return no rewards and do not cancel successful siblings; the trainer treats absent components as zero. partial attempts that reach a context, output, turn, or tool limit can still be scored.
+
+- `BaseEnv` runs the model and tool loop, then passes the transcript and example payload to `compute_reward`. `compute_group_rewards` can score the completed sibling group.
+- `HarborEnv` runs the configured Harbor agent and sandbox, then preserves its verifier or RewardKit reward components.
+
+### helpers
+
+`benchmax.rewards` provides deterministic text helpers, model judges, rubrics, ranking, adaptive rubrics, and diversity scoring for `BaseEnv` and direct `Environment` implementations. see the [rewards guide](src/benchmax/rewards/README.md).
+
+Harbor environments normally use their harness verifier and RewardKit instead of benchmax reward helpers.
+
+## bundling
+
+a bundle contains the environment class, its constructor arguments, the project-local source it needs, and its declared remote dependencies.
+
+```text
+environment class + constructor arguments
+                 + local source
+                 + dependency metadata
+                           │
+                           ▼
+                    portable bundle
+```
+
+benchmax creates the portable artifact so the same environment can be loaded outside the author's checkout. `castform` handles uploading the bundle, validating it remotely, and using it for training.
+
+Remote runtimes install `pip_dependencies` without enabling prereleases globally. If a dependency graph needs a prerelease, list that package explicitly even when it would normally be transitive. For example, use `pip_dependencies=["parent-package==1.0.0", "transitive-package==2.0.0rc1"]`.
+
+## further reading
+
+- [base environment guide](src/benchmax/envs/base/README.md)
+- [harbor environment guide](src/benchmax/envs/harbor/README.md)
+- [reward helpers](src/benchmax/rewards/README.md)
+- [examples](../../examples/)
+- [development instructions](../../README.md#development)
+
+apache 2.0 © 2026 CGFT Inc.

{benchmax-0.2.0 → benchmax-0.2.2}/pyproject.toml

@@ -1,9 +1,9 @@
 [project]
 name = "benchmax"
-version = "0.2.0"
+version = "0.2.2"
 description = "Platform-independent runtime for grouped LLM environments"
 readme = "README.md"
-authors = [{ name = "BenchMax Authors" }]
+authors = [{ name = "benchmax Authors" }]
 requires-python = "==3.12.*"
 dependencies = [
     "cloudpickle>=3.0.0",

benchmax-0.2.2/src/benchmax/auth.py

@@ -0,0 +1,236 @@
+"""Explicit, call-time authentication for model requests.
+
+benchmax defines only the runtime contract. Platform packages and execution
+runtimes provide concrete credential sources and bind injected credentials.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import contextvars
+import threading
+from collections.abc import AsyncIterator, Iterator, Mapping
+from contextlib import contextmanager
+from contextvars import ContextVar
+from dataclasses import dataclass, field
+from typing import Protocol, runtime_checkable
+
+import httpx
+from openai import AsyncOpenAI, OpenAI
+
+__all__ = [
+    "InjectedAuth",
+    "ModelAuth",
+    "ModelRequestContext",
+    "RequestModelAuth",
+    "StaticBearerAuth",
+    "bind_model_auth",
+    "create_async_openai_client",
+    "create_openai_client",
+]
+
+
+@dataclass(frozen=True, slots=True)
+class ModelRequestContext:
+    """Identity of the model request about to be authorized."""
+
+    base_url: str
+    model: str
+    rollout_id: str
+
+
+@runtime_checkable
+class ModelAuth(Protocol):
+    """Return headers immediately before each model HTTP request."""
+
+    async def headers_for_request(
+        self,
+        context: ModelRequestContext,
+    ) -> Mapping[str, str]: ...
+
+
+@dataclass(frozen=True, slots=True)
+class StaticBearerAuth:
+    """Explicit bearer authentication for providers with a stable API key."""
+
+    token: str = field(repr=False)
+
+    def __post_init__(self) -> None:
+        if not isinstance(self.token, str) or not self.token:
+            raise ValueError("bearer token must be a non-empty string")
+
+    async def headers_for_request(
+        self,
+        context: ModelRequestContext,
+    ) -> Mapping[str, str]:
+        del context
+        return {"Authorization": f"Bearer {self.token}"}
+
+
+class RequestModelAuth(httpx.Auth):
+    """Apply a :class:`ModelAuth` immediately before each HTTP request.
+
+    Both sync and async OpenAI-compatible clients use this adapter, so model
+    credential selection never falls back to an SDK environment variable or a
+    separate token resolver. When a sync client runs inside an active event
+    loop, auth is resolved in a context-preserving helper thread; custom
+    providers used there must not depend on primitives bound to another loop.
+    """
+
+    def __init__(self, auth: ModelAuth, context: ModelRequestContext) -> None:
+        if not isinstance(auth, ModelAuth):
+            raise TypeError("request auth must implement ModelAuth")
+        self._auth = auth
+        self._context = context
+
+    def sync_auth_flow(
+        self,
+        request: httpx.Request,
+    ) -> Iterator[httpx.Request]:
+        headers = _resolve_headers_sync(self._auth, self._context)
+        for name, value in headers.items():
+            request.headers[name] = value
+        yield request
+
+    async def async_auth_flow(
+        self,
+        request: httpx.Request,
+    ) -> AsyncIterator[httpx.Request]:
+        headers = await self._auth.headers_for_request(self._context)
+        for name, value in headers.items():
+            request.headers[name] = value
+        yield request
+
+
+def create_openai_client(
+    *,
+    model: str,
+    base_url: str,
+    auth: ModelAuth,
+    request_id: str,
+    max_retries: int = 2,
+) -> OpenAI:
+    """Create a synchronous OpenAI-compatible client with explicit auth."""
+
+    context = ModelRequestContext(
+        base_url=base_url,
+        model=model,
+        rollout_id=request_id,
+    )
+    return OpenAI(
+        base_url=base_url,
+        api_key="benchmax-explicit-auth",
+        http_client=httpx.Client(auth=RequestModelAuth(auth, context)),
+        max_retries=max_retries,
+    )
+
+
+def create_async_openai_client(
+    *,
+    model: str,
+    base_url: str,
+    auth: ModelAuth,
+    request_id: str,
+    max_retries: int = 2,
+) -> AsyncOpenAI:
+    """Create an asynchronous OpenAI-compatible client with explicit auth."""
+
+    context = ModelRequestContext(
+        base_url=base_url,
+        model=model,
+        rollout_id=request_id,
+    )
+    return AsyncOpenAI(
+        base_url=base_url,
+        api_key="benchmax-explicit-auth",
+        http_client=httpx.AsyncClient(auth=RequestModelAuth(auth, context)),
+        max_retries=max_retries,
+    )
+
+
+def _resolve_headers_sync(
+    auth: ModelAuth,
+    context: ModelRequestContext,
+) -> Mapping[str, str]:
+    """Resolve async ``ModelAuth`` from a synchronous HTTP client.
+
+    RAG search backends expose synchronous embedding callables. When one is
+    invoked from an async environment tool, its event loop is already running;
+    resolve the auth coroutine in a context-preserving helper thread rather
+    than attempting a nested event loop.
+    """
+
+    async def resolve() -> Mapping[str, str]:
+        return await auth.headers_for_request(context)
+
+    try:
+        asyncio.get_running_loop()
+    except RuntimeError:
+        return asyncio.run(resolve())
+
+    copied_context = contextvars.copy_context()
+    result: list[Mapping[str, str]] = []
+    failure: list[BaseException] = []
+
+    def run() -> None:
+        try:
+            result.append(copied_context.run(lambda: asyncio.run(resolve())))
+        except BaseException as error:  # propagate the original auth failure
+            failure.append(error)
+
+    thread = threading.Thread(target=run, daemon=True)
+    thread.start()
+    thread.join()
+    if failure:
+        raise failure[0]
+    if not result:
+        raise RuntimeError("model authentication did not return headers")
+    return result[0]
+
+
+_BOUND_MODEL_AUTH: ContextVar[Mapping[str, ModelAuth] | None] = ContextVar(
+    "benchmax_bound_model_auth",
+    default=None,
+)
+
+
+@dataclass(frozen=True, slots=True)
+class InjectedAuth:
+    """Serializable reference to authentication supplied by the runtime."""
+
+    name: str
+
+    def __post_init__(self) -> None:
+        if not isinstance(self.name, str) or not self.name.strip():
+            raise ValueError("injected auth name must be a non-empty string")
+
+    async def headers_for_request(
+        self,
+        context: ModelRequestContext,
+    ) -> Mapping[str, str]:
+        providers = _BOUND_MODEL_AUTH.get()
+        provider = providers.get(self.name) if providers is not None else None
+        if provider is None:
+            raise RuntimeError(f"No runtime model-auth provider was injected for {self.name!r}.")
+        if isinstance(provider, InjectedAuth):
+            raise RuntimeError(
+                f"Injected model-auth provider {self.name!r} cannot reference another InjectedAuth."
+            )
+        return await provider.headers_for_request(context)
+
+
+@contextmanager
+def bind_model_auth(providers: Mapping[str, ModelAuth]) -> Iterator[None]:
+    """Bind runtime providers for the current async execution context."""
+
+    normalized = dict(providers)
+    for name, provider in normalized.items():
+        if not isinstance(name, str) or not name.strip():
+            raise ValueError("model-auth provider names must be non-empty strings")
+        if not isinstance(provider, ModelAuth):
+            raise TypeError(f"model-auth provider {name!r} does not implement ModelAuth")
+    token = _BOUND_MODEL_AUTH.set(normalized)
+    try:
+        yield
+    finally:
+        _BOUND_MODEL_AUTH.reset(token)