hud-python 0.6.1__tar.gz → 0.6.3__tar.gz

{hud_python-0.6.1 → hud_python-0.6.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: hud-python
-Version: 0.6.1
+Version: 0.6.3
 Summary: SDK for the HUD platform.
 Project-URL: Homepage, https://github.com/hud-evals/hud-python
 Project-URL: Bug Tracker, https://github.com/hud-evals/hud-python/issues

hud_python-0.6.3/cookbooks/fireworks-rl-training/README.md

@@ -0,0 +1,114 @@
+# Fireworks RL Training
+
+Direct Fireworks Training API loop over the same arithmetic preview task used by
+`cookbooks/rl-training`.
+
+This does **not** use Fireworks native datasets or RFT jobs. It follows the
+Training API service path from the Fireworks docs:
+
+1. `FiretitanServiceClient.from_firetitan_config(...)`
+2. `create_deployment_sampler(...)` for high-parallel rollouts
+3. local grading of HUD-style multiplication tasks
+4. `forward_backward_custom(...)` + `optim_step(...)`
+5. `save_weights_for_sampler(...)` + sampler refresh
+
+References:
+
+- Fireworks Training API introduction: https://docs.fireworks.ai/fine-tuning/training-api/introduction
+- Training and sampling lifecycle: https://docs.fireworks.ai/fine-tuning/training-api/training-and-sampling
+- Loss functions / GRPO reference: https://docs.fireworks.ai/fine-tuning/training-api/loss-functions
+
+## Setup
+
+The repo-level `.env` is loaded automatically. It must contain:
+
+```bash
+FIREWORKS_API_KEY=...
+FIREWORKS_ACCOUNT_ID=...
+```
+
+Install the isolated cookbook environment:
+
+```bash
+uv sync --pre
+```
+
+## Calibrate task difficulty first
+
+Calibration defaults to Fireworks' OpenAI-compatible inference API, so it does
+**not** create a trainer, provision a Training API deployment, or call
+`optim_step`. This is the cheap way to tune task difficulty before paying for a
+Training API run.
+
+The calibration model is separate from the training base model because the
+`lorenss` key currently exposes only a small serverless inference catalog (no
+Qwen3 8B deployment). Override it with `--inference-model` if you have a closer
+deployed model.
+
+```bash
+uv run train.py --calibrate-only --groups-per-step 8 --rollouts-per-prompt 8 --parallelism 32
+```
+
+The goal is a reward distribution with variance. If reward is all zero, make the
+task easier:
+
+```bash
+uv run train.py --calibrate-only --min-a 10 --max-a 99 --min-b 2 --max-b 9
+```
+
+If reward is all one, make the task harder:
+
+```bash
+uv run train.py --calibrate-only --min-a 1000 --max-a 9999 --min-b 11 --max-b 99
+```
+
+The current defaults are calibrated for the visible `gpt-oss-120b` inference
+model on the `lorenss` key: 2-digit by 1-digit multiplication with a direct
+"reply only with the integer" prompt. A 32-rollout calibration gave a non-trivial
+baseline (`reward_mean ~= 0.22`, `reward_std ~= 0.42`), while the original
+3-digit by 2-digit range was all-zero.
+
+## Train
+
+Once calibration has non-trivial rewards:
+
+```bash
+uv run train.py --steps 5 --groups-per-step 8 --rollouts-per-prompt 8 --parallelism 32
+```
+
+This uses the direct Training API managed service path. If you want calibration
+to go through the managed deployment sampler too, pass
+`--calibration-backend managed`; this provisions the same resources as training.
+
+### Current Fireworks preview account blocker
+
+On the `lorenss` preview account, trainer creation currently fails before the
+first train step with:
+
+```text
+failed to ensure FIREWORKS_API_KEY secret: unkey inference api id is not configured
+```
+
+This happens even with `create_deployment=False`, so it is an account/control
+plane provisioning issue rather than a problem in the rollout or loss code. Once
+Fireworks enables the missing Unkey inference API config for the account, the
+same `uv run train.py ...` command should proceed to trainer startup and the
+first `forward_backward_custom(...)` call.
+
+Metrics are written to:
+
+- `runs/fireworks-rl-preview/metrics.jsonl`
+- `runs/fireworks-rl-preview/reward_loss.png` if `matplotlib` is installed
+
+## Notes
+
+- Defaults use Qwen 3 8B full-parameter training:
+  - `accounts/fireworks/models/qwen3-8b`
+  - `Qwen/Qwen3-8B`
+  - `accounts/fireworks/trainingShapes/qwen3-8b-128k`
+- LoRA can be tested with `--lora-rank N`, but the validated Qwen3 8B training
+  shape currently rejects LoRA mode on the `lorenss` preview account.
+- The first checkpoint sync happens after step 0 and subsequent rollouts sample
+  the updated weights through the same deployment.
+- `--keep-trainer` and `--keep-deployment` are available for debugging. By
+  default the trainer is cleaned up and the deployment scales to zero on exit.

hud_python-0.6.3/cookbooks/fireworks-rl-training/pyproject.toml

@@ -0,0 +1,19 @@
+[project]
+name = "fireworks-rl-training"
+version = "0.1.0"
+description = "Direct Fireworks Training API RL loop over HUD-style arithmetic tasks"
+requires-python = ">=3.11,<3.13"
+dependencies = [
+    "fireworks-ai[training]",
+    "hud-python",
+    "matplotlib",
+    "python-dotenv",
+    "torch>=2",
+    "transformers>=4.55",
+]
+
+[tool.uv]
+package = false
+
+[tool.uv.sources]
+hud-python = { path = "../..", editable = true }

{hud_python-0.6.1 → hud_python-0.6.3}/cookbooks/rl-training/README.md

@@ -18,12 +18,13 @@ each `optim_step` closes the on-policy loop.
 
 ## Run
 
-Needs `HUD_API_KEY` (from your environment or `.env`). List the trainable
-gateway models on your account, pick one, and set it as the `MODEL` constant at
-the top of `simple_train.py` / `ppo_custom_loss.py`:
+Needs `HUD_API_KEY` (from your environment or `.env`). List the gateway models
+on your account, pick a trainable one (the **Trainable** column marks them), and
+set it as the `MODEL` constant at the top of `simple_train.py` /
+`ppo_custom_loss.py`:
 
 ```bash
-hud models
+hud models list          # Name | Model (API) | ID | Provider | Agent | Trainable
 ```
 
 **Train on a deployed taskset (the real flow).** You've built a taskset and

{hud_python-0.6.1 → hud_python-0.6.3}/hud/cli/eval.py

@@ -5,6 +5,7 @@ Config Override Order: CLI arguments > .hud_eval.toml > defaults
 
 from __future__ import annotations
 
+import ast
 import asyncio
 import logging
 import os
@@ -665,13 +666,46 @@ def _build_agent(cfg: EvalConfig) -> Any:
     return cast("Any", cfg.agent_type.cls)(config=config)
 
 
+def _python_defines_environment(path: Path) -> bool:
+    """Return True when ``path`` constructs a v6 :class:`~hud.environment.Environment`."""
+    try:
+        tree = ast.parse(path.read_text(encoding="utf-8"))
+    except (OSError, SyntaxError):
+        return False
+    for node in ast.walk(tree):
+        if not isinstance(node, ast.Call):
+            continue
+        callee = node.func
+        callee_name = (
+            callee.id
+            if isinstance(callee, ast.Name)
+            else callee.attr
+            if isinstance(callee, ast.Attribute)
+            else None
+        )
+        if callee_name == "Environment":
+            return True
+    return False
+
+
 def _spawn_target(source: Path) -> Path:
-    """The path the ``LocalRuntime`` provider serves: the source itself for ``.py``
-    files and directories, the surrounding directory for JSON/JSONL data files
-    (the env's ``.py`` source lives next to the tasks file)."""
+    """The path the ``LocalRuntime`` provider serves.
+
+    Directories and env-defining ``.py`` files are served as-is. Task-only
+    sources (``tasks.py`` importing from ``env.py``) resolve to a sibling
+    ``env.py`` or the containing directory. JSON/JSONL data files use the
+    surrounding directory (the env source lives next to the tasks file).
+    """
     resolved = source.resolve()
-    if resolved.is_dir() or resolved.suffix == ".py":
+    if resolved.is_dir():
+        return resolved
+    if resolved.suffix != ".py":
+        return resolved.parent
+    if _python_defines_environment(resolved):
         return resolved
+    env_py = resolved.parent / "env.py"
+    if env_py.is_file():
+        return env_py
     return resolved.parent
 
 

{hud_python-0.6.1 → hud_python-0.6.3}/hud/cli/init.py

@@ -76,8 +76,8 @@ def init_command(
         None,
         "--preset",
         "-p",
-        help="Starter preset to download from GitHub (e.g. blank, coding, browser, "
-        "deepresearch, rubrics, remote-browser). Omit for an interactive picker; in a "
+        help="Starter preset to download from GitHub (e.g. blank, browser, "
+        "deepresearch, cua, autonomous-businesses, verilog). Omit for an interactive picker; in a "
         "non-interactive shell, omitting it writes the minimal local scaffold.",
     ),
 ) -> None:
@@ -89,7 +89,7 @@ def init_command(
 
     Examples:
         hud init my-env                  # interactive picker (or local scaffold)
-        hud init my-env --preset coding  # download the coding starter
+        hud init my-env --preset browser  # download the browser starter
         hud init my-env --dir envs       # create ./envs/my-env[/not dim]
     """
     hud_console = HUDConsole()

{hud_python-0.6.1 → hud_python-0.6.3}/hud/cli/models.py

@@ -56,14 +56,18 @@ def list_models(
     table = Table()
     table.add_column("Name", style="cyan")
     table.add_column("Model (API)", style="green")
+    table.add_column("ID", style="blue", no_wrap=True)
     table.add_column("Provider", style="yellow")
     table.add_column("Agent", style="magenta")
+    table.add_column("Trainable", style="green", justify="center")
     for model in models_list:
         table.add_row(
             model.name or model.id or "-",
             model.model_name or model.id or "-",
+            model.id or "-",
             model.provider.name or "-",
             model.sdk_agent_type or "-",
+            "✓" if model.is_trainable else "",
         )
     console.print(table)
     console.print(f"\n[dim]Gateway: {settings.hud_gateway_url}[/dim]")

{hud_python-0.6.1 → hud_python-0.6.3}/hud/cli/templates.py

@@ -13,7 +13,7 @@ COPY . .
 
 # Serve the Environment's control channel (tcp JSON-RPC) on 8765.
 EXPOSE 8765
-CMD ["uv", "run", "python", "-m", "hud", "dev", "env:env", "--host", "0.0.0.0", "--port", "8765"]
+CMD ["uv", "run", "hud", "serve", "env:env", "--host", "0.0.0.0", "--port", "8765"]
 """
 
 # fmt: off
@@ -78,7 +78,7 @@ async def count(sentence: str, letter: str):
 
 
 # =============================================================================
-# TEST - run with: python env.py
+# TEST - run with: uv run python env.py
 # =============================================================================
 
 async def test():
@@ -136,7 +136,6 @@ version = "0.1.0"
 requires-python = ">=3.11"
 dependencies = ["hud-python"]
 
-[build-system]
-requires = ["hatchling"]
-build-backend = "hatchling.build"
+[tool.uv]
+package = false
 """

{hud_python-0.6.1 → hud_python-0.6.3}/hud/cli/tests/test_deploy.py

@@ -48,7 +48,7 @@ class TestResolveEnvironmentName:
 
     def test_entrypoint_disambiguates_subagent(self, tmp_path: Path) -> None:
         (tmp_path / "Dockerfile").write_text(
-            'CMD ["hud", "dev", "env:env", "--port", "8765"]\n', encoding="utf-8"
+            'CMD ["hud", "serve", "env:env", "--port", "8765"]\n', encoding="utf-8"
         )
         (tmp_path / "env.py").write_text('env = Environment("trace-explorer")\n', encoding="utf-8")
         (tmp_path / "verify.py").write_text(

{hud_python-0.6.1 → hud_python-0.6.3}/hud/cli/tests/test_eval_config.py

@@ -237,3 +237,32 @@ def test_eval_max_steps_lands_in_agent_config() -> None:
     )
     agent = eval_mod._build_agent(cfg)
     assert agent.config.max_steps == 17
+
+
+def test_spawn_target_serves_single_file_env(tmp_path: Path) -> None:
+    env_py = tmp_path / "tasks.py"
+    env_py.write_text(
+        'from hud import Environment\nenv = Environment(name="demo")\n',
+        encoding="utf-8",
+    )
+    assert eval_mod._spawn_target(env_py) == env_py.resolve()
+
+
+def test_spawn_target_resolves_split_tasks_layout(tmp_path: Path) -> None:
+    (tmp_path / "env.py").write_text(
+        'from hud.environment import Environment\nenv = Environment(name="demo")\n',
+        encoding="utf-8",
+    )
+    tasks_py = tmp_path / "tasks.py"
+    tasks_py.write_text("from env import env\n\ntasks = []\n", encoding="utf-8")
+    assert eval_mod._spawn_target(tasks_py) == (tmp_path / "env.py").resolve()
+
+
+def test_spawn_target_json_uses_parent_directory(tmp_path: Path) -> None:
+    tasks_json = tmp_path / "tasks.json"
+    tasks_json.write_text("[]", encoding="utf-8")
+    assert eval_mod._spawn_target(tasks_json) == tmp_path.resolve()
+
+
+def test_spawn_target_directory_is_served_as_is(tmp_path: Path) -> None:
+    assert eval_mod._spawn_target(tmp_path) == tmp_path.resolve()

{hud_python-0.6.1 → hud_python-0.6.3}/hud/cli/tests/test_init.py

@@ -29,6 +29,14 @@ def test_init_scaffolds_a_runnable_package(tmp_path: Path) -> None:
     assert (target / "tasks.py").read_text().startswith('"""')
     assert 'name = "my-cool-env"' in (target / "pyproject.toml").read_text()
 
+    pyproject = (target / "pyproject.toml").read_text()
+    assert "package = false" in pyproject
+    assert "[build-system]" not in pyproject
+
+    dockerfile = (target / "Dockerfile.hud").read_text()
+    assert 'CMD ["uv", "run", "hud", "serve"' in dockerfile
+    assert '"dev"' not in dockerfile
+
 
 def test_init_refuses_to_clobber_nonempty_directory(tmp_path: Path) -> None:
     target = tmp_path / "taken"

{hud_python-0.6.1 → hud_python-0.6.3}/hud/eval/job.py

@@ -38,17 +38,20 @@ class Job:
     name: str
     runs: list[Run] = field(default_factory=list)
     group: int = 1
+    #: Platform taskset id this job runs, when it's a synced taskset
+    #: (``Taskset.from_api``). Links the job to that taskset on the platform.
+    taskset_id: str | None = None
 
     @classmethod
-    async def start(cls, name: str, *, group: int = 1) -> Job:
+    async def start(cls, name: str, *, group: int = 1, taskset_id: str | None = None) -> Job:
         """Open a job spanning multiple scheduler calls.
 
         A scheduler call mints its own job by default; pass a started job as
         ``job=`` to ``Task.run`` / ``Taskset.run`` to accumulate every run of a
         longer arc — a training session, a chat conversation — under one id.
         """
-        job = cls(id=uuid.uuid4().hex, name=name, group=group)
-        await job_enter(job.id, name=name, group=group)
+        job = cls(id=uuid.uuid4().hex, name=name, group=group, taskset_id=taskset_id)
+        await job_enter(job.id, name=name, group=group, taskset_id=taskset_id)
         return job
 
     @property
@@ -79,21 +82,42 @@ def _reporting_enabled() -> bool:
     return bool(settings.telemetry_enabled and settings.api_key)
 
 
-async def job_enter(job_id: str, *, name: str, group: int) -> None:
-    """Register a batch job with the platform."""
+async def job_enter(job_id: str, *, name: str, group: int, taskset_id: str | None = None) -> None:
+    """Register a batch job with the platform.
+
+    ``taskset_id`` links the job to a synced taskset (set when running
+    ``Taskset.from_api``); ``None`` for ad-hoc/local tasksets. The platform
+    creates no taskset on its own — remote rollouts carry the scenario inline.
+    """
     if not _reporting_enabled():
         return
-    await _report(f"/trace/job/{job_id}/enter", {"name": name, "group": group})
+    await _report(
+        f"/trace/job/{job_id}/enter",
+        {"name": name, "group": group, "taskset_id": taskset_id},
+    )
     from hud.settings import settings
 
     logger.info("job: %s/jobs/%s", settings.hud_web_url, job_id)
 
 
-async def trace_enter(trace_id: str, *, job_id: str | None, group_id: str | None) -> None:
-    """Report that one rollout started."""
+async def trace_enter(
+    trace_id: str,
+    *,
+    job_id: str | None,
+    group_id: str | None,
+    model: str | None = None,
+) -> None:
+    """Report that one rollout started.
+
+    ``model`` is the model string the agent will sample (when known); the
+    platform resolves it and attributes the trace immediately on enter.
+    """
     if not _reporting_enabled():
         return
-    await _report(f"/trace/{trace_id}/enter", {"job_id": job_id, "group_id": group_id})
+    await _report(
+        f"/trace/{trace_id}/enter",
+        {"job_id": job_id, "group_id": group_id, "model": model},
+    )
 
 
 async def trace_exit(run: Run) -> None:

{hud_python-0.6.1 → hud_python-0.6.3}/hud/eval/run.py

@@ -295,8 +295,15 @@ async def rollout(
         job_id = uuid.uuid4().hex
         await job_enter(job_id, name=task.id, group=1)
     trace_id = trace_id or uuid.uuid4().hex
+    # Report the model the agent will sample so the platform attributes the
+    # trace to it on enter. Only LLM tool agents carry an inference-model slug
+    # (``config.model``); robot/other agents have none. Local import avoids an
+    # eval<->agents import cycle.
+    from hud.agents.tool_agent import ToolAgent
+
+    agent_model = agent.config.model if isinstance(agent, ToolAgent) else None
     with set_trace_context(trace_id):
-        await trace_enter(trace_id, job_id=job_id, group_id=group_id)
+        await trace_enter(trace_id, job_id=job_id, group_id=group_id, model=agent_model)
         run: Run | None = None
         _phase = "provisioning"
 

{hud_python-0.6.1 → hud_python-0.6.3}/hud/eval/taskset.py

@@ -59,6 +59,17 @@ class Taskset:
         self.origin = origin
         self.tasks: dict[str, Task] = self._index_by_slug(list(tasks))
 
+    @property
+    def api_id(self) -> str | None:
+        """The platform taskset id when loaded via :meth:`from_api`, else None.
+
+        Threaded into the job so a remote run of a synced taskset links to it;
+        ad-hoc/file/module tasksets have none and create no taskset.
+        """
+        if self.origin and self.origin.startswith("api:"):
+            return self.origin[len("api:") :]
+        return None
+
     @classmethod
     def from_file(cls, path: str | Path) -> Taskset:
         """Load a taskset from ``.py`` source, a directory, or JSON/JSONL data.
@@ -242,8 +253,13 @@ class Taskset:
             expanded.extend((task, group_id) for _ in range(group))
 
         if job is None:
-            job = Job(id=uuid.uuid4().hex, name=_job_name(self.name, task_list, group), group=group)
-            await job_enter(job.id, name=job.name, group=group)
+            job = Job(
+                id=uuid.uuid4().hex,
+                name=_job_name(self.name, task_list, group),
+                group=group,
+                taskset_id=self.api_id,
+            )
+            await job_enter(job.id, name=job.name, group=group, taskset_id=self.api_id)
         job_id = job.id
 
         # Placement is chosen once for the batch: HostedRuntime delegates the

{hud_python-0.6.1 → hud_python-0.6.3}/hud/utils/gateway.py

@@ -35,6 +35,7 @@ class GatewayModelInfo(BaseModel):
     name: str | None = None
     model_name: str | None = None
     sdk_agent_type: str | None = None
+    is_trainable: bool = False
     provider: GatewayProviderInfo = Field(default_factory=GatewayProviderInfo)
 
 

{hud_python-0.6.1 → hud_python-0.6.3}/hud/version.py

@@ -4,4 +4,4 @@ Version information for the HUD SDK.
 
 from __future__ import annotations
 
-__version__ = "0.6.1"
+__version__ = "0.6.3"

{hud_python-0.6.1 → hud_python-0.6.3}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "hud-python"
-version = "0.6.1"
+version = "0.6.3"
 description = "SDK for the HUD platform."
 readme = "README.md"
 requires-python = ">=3.11, <3.13"