nomad-harness 0.1.0.dev0__tar.gz → 0.1.0.dev1__tar.gz

{nomad_harness-0.1.0.dev0 → nomad_harness-0.1.0.dev1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: nomad-harness
-Version: 0.1.0.dev0
+Version: 0.1.0.dev1
 Summary: Action runtime connecting frontier models to simulated and physical embodiments.
 Project-URL: Repository, https://github.com/Lambda-Robotics-Inc/Nomad_Harness_Interface
 Project-URL: Issues, https://github.com/Lambda-Robotics-Inc/Nomad_Harness_Interface/issues
@@ -24,13 +24,14 @@ Requires-Dist: anthropic; extra == 'anthropic'
 Provides-Extra: dev
 Requires-Dist: jsonschema>=4.18; extra == 'dev'
 Requires-Dist: mypy>=1.10; extra == 'dev'
+Requires-Dist: openai<4,>=3.24; extra == 'dev'
 Requires-Dist: pytest>=8.0; extra == 'dev'
 Requires-Dist: ruff>=0.5; extra == 'dev'
 Requires-Dist: types-pyyaml; extra == 'dev'
 Provides-Extra: gemini
 Requires-Dist: google-genai; extra == 'gemini'
 Provides-Extra: openai
-Requires-Dist: openai; extra == 'openai'
+Requires-Dist: openai<4,>=3.24; extra == 'openai'
 Provides-Extra: robosuite
 Requires-Dist: imageio[ffmpeg]; extra == 'robosuite'
 Requires-Dist: mujoco==3.3.7; extra == 'robosuite'
@@ -44,7 +45,7 @@ Description-Content-Type: text/markdown
 
 Nomad connects a model provider to a simulator or robot adapter through explicit observation, action, and execution-feedback contracts. The first milestone is a working simulation package: change the model or simulation backend without rewriting the agent loop.
 
-**Status: implementation brief, Day 2 complete.** This README defines the work to build v0.1. The core contracts, offline runtime, and fake backend/model from [the first engineering task](#the-first-engineering-task) are implemented and tested, as is the robosuite `Lift` adapter with its privileged-state scripted baseline (Day 2 of [the delivery plan](#8-seven-day-delivery-plan)). Everything else below (provider adapters, Isaac Lab, CLI, evaluation runner) is still a proposed contract, not implemented functionality. No package release, benchmark result, or hardware support is claimed here.
+**Status: implementation brief, Day 3 complete.** This README defines the work to build v0.1. The core contracts, offline runtime, and fake backend/model from [the first engineering task](#the-first-engineering-task) are implemented and tested, as are the robosuite `Lift` adapter with its privileged-state scripted baseline (Day 2 of [the delivery plan](#8-seven-day-delivery-plan)) and the OpenAI provider adapter with a model-driven robosuite rollout (Day 3). Everything else below (Anthropic and Gemini adapters, Isaac Lab, CLI, evaluation runner) is still a proposed contract, not implemented functionality. Development pre-releases are on PyPI as [`nomad-harness`](https://pypi.org/project/nomad-harness/) (install with `pip install --pre nomad-harness`); no benchmark result or hardware support is claimed here.
 
 **For the implementation partner:** start with [the first engineering task](#the-first-engineering-task), follow [the seven-day delivery plan](#8-seven-day-delivery-plan), and use [the release acceptance checklist](#10-release-acceptance-checklist) as the definition of done.
 
@@ -56,7 +57,8 @@ Nomad connects a model provider to a simulator or robot adapter through explicit
 | `FakeEmbodiment` (deterministic cube lift) and `FakeModel` (scripted) | Implemented, offline tests |
 | `Robosuite` adapter: Panda `Lift`, absolute base-frame OSC, RGB/proprioception, video | Implemented, simulator tests (robosuite 1.5.2, mujoco 3.3.7) |
 | `ScriptedLiftPolicy` baseline (privileged object state) | Implemented; succeeds on robosuite `Lift` and the fake world |
-| Isaac Lab, OpenAI, Anthropic, Gemini adapters; `nomad` CLI; evaluation runner | Not started |
+| `OpenAI` adapter: Responses API, strict tool calling, RGB + proprioception context | Implemented; offline tests on recorded responses, opt-in live test (openai 3.24) |
+| Isaac Lab, Anthropic, Gemini adapters; `nomad` CLI; evaluation runner | Not started |
 
 Run the offline demo (Python 3.10+; no API keys, simulator, or GPU):
 
@@ -76,26 +78,42 @@ pytest -m robosuite                                    # adapter acceptance test
 
 The baseline reads the simulator's cube position, so its runs are labeled `privileged_state` and are not comparable with model runs that see only images and proprioception. On headless Linux, set `MUJOCO_GL=egl` or `MUJOCO_GL=osmesa` for offscreen rendering.
 
+Run an OpenAI model on robosuite `Lift` (spends API credits). The model sees the front camera image and robot state only (`rgb_proprio`); success comes from robosuite's own predicate. The key is read from `OPENAI_API_KEY` and never written to traces; the model must be one your account can access.
+
+```bash
+python -m pip install -e ".[dev,robosuite,openai]"
+export OPENAI_API_KEY=...                  # better: source it from a private file outside the repo
+export NOMAD_OPENAI_MODEL=your-model-id
+python examples/quickstart.py                      # the minimal script shown in section 1
+python examples/robosuite_openai_lift.py --seed 0   # with options; add --reasoning-effort high
+NOMAD_OPENAI_LIVE=1 pytest -m openai_live          # live multimodal proposal test (two calls)
+```
+
+Plain `pytest` and CI never call the API: the adapter is tested offline against responses recorded from the live API, and the live test runs only with `NOMAD_OPENAI_LIVE=1`.
+
+**Pilot, not a benchmark.** On 2026-10-06, `gpt-6.1-sol` (default reasoning effort, history window 8, 30-decision budget) lifted the cube on seeds 0–3 in 18, 9, 8, and 14 decisions; seed 4 exhausted the budget after re-grasping six times at the same position. Rerun with `--reasoning-effort high`, seed 4 succeeded in 8 decisions. That is five seeds plus one rerun: too few for a success rate or for any claim about reasoning effort. The [experiment matrix](#initial-experiment-matrix) calls for at least 10 seeds per configuration.
+
 ## 1. Product goal and first release
 
-The developer experience should eventually be:
+The developer experience is this complete script ([`examples/quickstart.py`](examples/quickstart.py)), implemented for OpenAI and robosuite:
 
 ```python
-# Target API: implement this behavior before advertising it as a quickstart.
 import os
 
 from nomad import Agent, RunConfig
-from nomad.models import OpenAI
 from nomad.embodiments import Robosuite
+from nomad.models import OpenAI
 
 with Robosuite(task="Lift", robot="Panda") as env:
     agent = Agent(
-        model=OpenAI(model=os.environ["NOMAD_MODEL"]),
+        model=OpenAI(model=os.environ["NOMAD_OPENAI_MODEL"]),
         embodiment=env,
-        config=RunConfig(max_decisions=30, max_wall_time_s=180),
+        config=RunConfig(max_decisions=30, max_wall_time_s=600),
     )
     result = agent.run("Lift the cube off the table.")
-    print(result.status, result.trace_dir)
+
+print(result.status, result.outcome.reason)
+print("trace:", result.trace_dir)
 ```
 
 The provider reads its API key from the environment. Model identifiers are supplied by the user and checked against actual account access; do not hard-code an assumed future model name.
@@ -339,7 +357,14 @@ See the official [environment registry](https://isaac-sim.github.io/IsaacLab/mai
 
 ### Model providers
 
-Implement OpenAI first, then Anthropic and Gemini. Each adapter must:
+Implement OpenAI first, then Anthropic and Gemini. OpenAI is implemented (`nomad.models.OpenAI`); its shared pieces are meant to be reused by the others:
+
+- `nomad.models._tools` turns the proposal schema into one strict tool per action plus `finish`, and turns tool calls back into an unvalidated raw proposal. Flat per-action schemas fit every provider's strict-schema subset better than one schema with actions in an `anyOf`.
+- `nomad.models._encoding` renders the context as system text, user text, and images, with camera poses in the robot-state frame and an image-free summary for traces.
+- Each provider adds only a schema converter for its JSON Schema dialect (`nomad.models.openai.schema`) and a thin adapter. The runtime keeps the loop: provider agent loops and automatic tool runners are not used.
+- An `OpenAICompatible` adapter over Chat Completions (vLLM, Ollama, OpenRouter) can reuse the same pieces later.
+
+Each adapter must:
 
 - Accept a user-provided model identifier, key through environment configuration, a timeout, and an explicit call budget.
 - Encode the same canonical context into the provider's image/text request format.
@@ -509,9 +534,9 @@ A second simulator alone is not evidence of universal embodiment transfer. Add a
 
 - [ ] A fresh checkout can run the offline example using documented commands.
 - [ ] Importing `nomad` requires no simulator installation and starts no application.
-- [ ] robosuite scripted and model-driven episodes save valid traces and outcomes.
+- [x] robosuite scripted and model-driven episodes save valid traces and outcomes.
 - [ ] Isaac Lab scripted and model-driven episodes use the same runtime contract, if listed as supported.
-- [ ] Each advertised provider passes a live multimodal structured-proposal test using an accessible model.
+- [ ] Each advertised provider passes a live multimodal structured-proposal test using an accessible model. (OpenAI passes; Anthropic and Gemini pending.)
 - [ ] Adapter tests confirm units, frames, rotation composition, gripper mapping, and total-displacement semantics.
 - [ ] Invalid and stale commands cannot reach execution; timeouts and partial failures have explicit outcomes.
 - [ ] Task success comes from a defined evaluator, and model-visible privileged state is labeled.

{nomad_harness-0.1.0.dev0 → nomad_harness-0.1.0.dev1}/README.md

@@ -4,7 +4,7 @@
 
 Nomad connects a model provider to a simulator or robot adapter through explicit observation, action, and execution-feedback contracts. The first milestone is a working simulation package: change the model or simulation backend without rewriting the agent loop.
 
-**Status: implementation brief, Day 2 complete.** This README defines the work to build v0.1. The core contracts, offline runtime, and fake backend/model from [the first engineering task](#the-first-engineering-task) are implemented and tested, as is the robosuite `Lift` adapter with its privileged-state scripted baseline (Day 2 of [the delivery plan](#8-seven-day-delivery-plan)). Everything else below (provider adapters, Isaac Lab, CLI, evaluation runner) is still a proposed contract, not implemented functionality. No package release, benchmark result, or hardware support is claimed here.
+**Status: implementation brief, Day 3 complete.** This README defines the work to build v0.1. The core contracts, offline runtime, and fake backend/model from [the first engineering task](#the-first-engineering-task) are implemented and tested, as are the robosuite `Lift` adapter with its privileged-state scripted baseline (Day 2 of [the delivery plan](#8-seven-day-delivery-plan)) and the OpenAI provider adapter with a model-driven robosuite rollout (Day 3). Everything else below (Anthropic and Gemini adapters, Isaac Lab, CLI, evaluation runner) is still a proposed contract, not implemented functionality. Development pre-releases are on PyPI as [`nomad-harness`](https://pypi.org/project/nomad-harness/) (install with `pip install --pre nomad-harness`); no benchmark result or hardware support is claimed here.
 
 **For the implementation partner:** start with [the first engineering task](#the-first-engineering-task), follow [the seven-day delivery plan](#8-seven-day-delivery-plan), and use [the release acceptance checklist](#10-release-acceptance-checklist) as the definition of done.
 
@@ -16,7 +16,8 @@ Nomad connects a model provider to a simulator or robot adapter through explicit
 | `FakeEmbodiment` (deterministic cube lift) and `FakeModel` (scripted) | Implemented, offline tests |
 | `Robosuite` adapter: Panda `Lift`, absolute base-frame OSC, RGB/proprioception, video | Implemented, simulator tests (robosuite 1.5.2, mujoco 3.3.7) |
 | `ScriptedLiftPolicy` baseline (privileged object state) | Implemented; succeeds on robosuite `Lift` and the fake world |
-| Isaac Lab, OpenAI, Anthropic, Gemini adapters; `nomad` CLI; evaluation runner | Not started |
+| `OpenAI` adapter: Responses API, strict tool calling, RGB + proprioception context | Implemented; offline tests on recorded responses, opt-in live test (openai 3.24) |
+| Isaac Lab, Anthropic, Gemini adapters; `nomad` CLI; evaluation runner | Not started |
 
 Run the offline demo (Python 3.10+; no API keys, simulator, or GPU):
 
@@ -36,26 +37,42 @@ pytest -m robosuite                                    # adapter acceptance test
 
 The baseline reads the simulator's cube position, so its runs are labeled `privileged_state` and are not comparable with model runs that see only images and proprioception. On headless Linux, set `MUJOCO_GL=egl` or `MUJOCO_GL=osmesa` for offscreen rendering.
 
+Run an OpenAI model on robosuite `Lift` (spends API credits). The model sees the front camera image and robot state only (`rgb_proprio`); success comes from robosuite's own predicate. The key is read from `OPENAI_API_KEY` and never written to traces; the model must be one your account can access.
+
+```bash
+python -m pip install -e ".[dev,robosuite,openai]"
+export OPENAI_API_KEY=...                  # better: source it from a private file outside the repo
+export NOMAD_OPENAI_MODEL=your-model-id
+python examples/quickstart.py                      # the minimal script shown in section 1
+python examples/robosuite_openai_lift.py --seed 0   # with options; add --reasoning-effort high
+NOMAD_OPENAI_LIVE=1 pytest -m openai_live          # live multimodal proposal test (two calls)
+```
+
+Plain `pytest` and CI never call the API: the adapter is tested offline against responses recorded from the live API, and the live test runs only with `NOMAD_OPENAI_LIVE=1`.
+
+**Pilot, not a benchmark.** On 2026-10-06, `gpt-6.1-sol` (default reasoning effort, history window 8, 30-decision budget) lifted the cube on seeds 0–3 in 18, 9, 8, and 14 decisions; seed 4 exhausted the budget after re-grasping six times at the same position. Rerun with `--reasoning-effort high`, seed 4 succeeded in 8 decisions. That is five seeds plus one rerun: too few for a success rate or for any claim about reasoning effort. The [experiment matrix](#initial-experiment-matrix) calls for at least 10 seeds per configuration.
+
 ## 1. Product goal and first release
 
-The developer experience should eventually be:
+The developer experience is this complete script ([`examples/quickstart.py`](examples/quickstart.py)), implemented for OpenAI and robosuite:
 
 ```python
-# Target API: implement this behavior before advertising it as a quickstart.
 import os
 
 from nomad import Agent, RunConfig
-from nomad.models import OpenAI
 from nomad.embodiments import Robosuite
+from nomad.models import OpenAI
 
 with Robosuite(task="Lift", robot="Panda") as env:
     agent = Agent(
-        model=OpenAI(model=os.environ["NOMAD_MODEL"]),
+        model=OpenAI(model=os.environ["NOMAD_OPENAI_MODEL"]),
         embodiment=env,
-        config=RunConfig(max_decisions=30, max_wall_time_s=180),
+        config=RunConfig(max_decisions=30, max_wall_time_s=600),
     )
     result = agent.run("Lift the cube off the table.")
-    print(result.status, result.trace_dir)
+
+print(result.status, result.outcome.reason)
+print("trace:", result.trace_dir)
 ```
 
 The provider reads its API key from the environment. Model identifiers are supplied by the user and checked against actual account access; do not hard-code an assumed future model name.
@@ -299,7 +316,14 @@ See the official [environment registry](https://isaac-sim.github.io/IsaacLab/mai
 
 ### Model providers
 
-Implement OpenAI first, then Anthropic and Gemini. Each adapter must:
+Implement OpenAI first, then Anthropic and Gemini. OpenAI is implemented (`nomad.models.OpenAI`); its shared pieces are meant to be reused by the others:
+
+- `nomad.models._tools` turns the proposal schema into one strict tool per action plus `finish`, and turns tool calls back into an unvalidated raw proposal. Flat per-action schemas fit every provider's strict-schema subset better than one schema with actions in an `anyOf`.
+- `nomad.models._encoding` renders the context as system text, user text, and images, with camera poses in the robot-state frame and an image-free summary for traces.
+- Each provider adds only a schema converter for its JSON Schema dialect (`nomad.models.openai.schema`) and a thin adapter. The runtime keeps the loop: provider agent loops and automatic tool runners are not used.
+- An `OpenAICompatible` adapter over Chat Completions (vLLM, Ollama, OpenRouter) can reuse the same pieces later.
+
+Each adapter must:
 
 - Accept a user-provided model identifier, key through environment configuration, a timeout, and an explicit call budget.
 - Encode the same canonical context into the provider's image/text request format.
@@ -469,9 +493,9 @@ A second simulator alone is not evidence of universal embodiment transfer. Add a
 
 - [ ] A fresh checkout can run the offline example using documented commands.
 - [ ] Importing `nomad` requires no simulator installation and starts no application.
-- [ ] robosuite scripted and model-driven episodes save valid traces and outcomes.
+- [x] robosuite scripted and model-driven episodes save valid traces and outcomes.
 - [ ] Isaac Lab scripted and model-driven episodes use the same runtime contract, if listed as supported.
-- [ ] Each advertised provider passes a live multimodal structured-proposal test using an accessible model.
+- [ ] Each advertised provider passes a live multimodal structured-proposal test using an accessible model. (OpenAI passes; Anthropic and Gemini pending.)
 - [ ] Adapter tests confirm units, frames, rotation composition, gripper mapping, and total-displacement semantics.
 - [ ] Invalid and stale commands cannot reach execution; timeouts and partial failures have explicit outcomes.
 - [ ] Task success comes from a defined evaluator, and model-visible privileged state is labeled.

nomad_harness-0.1.0.dev1/examples/quickstart.py

@@ -0,0 +1,24 @@
+"""Minimal example: an OpenAI model lifts a cube in robosuite.
+
+pip install -e '.[robosuite,openai]'
+export OPENAI_API_KEY=...
+export NOMAD_OPENAI_MODEL=...   # a model your account can access
+python examples/quickstart.py
+"""
+
+import os
+
+from nomad import Agent, RunConfig
+from nomad.embodiments import Robosuite
+from nomad.models import OpenAI
+
+with Robosuite(task="Lift", robot="Panda") as env:
+    agent = Agent(
+        model=OpenAI(model=os.environ["NOMAD_OPENAI_MODEL"]),
+        embodiment=env,
+        config=RunConfig(max_decisions=30, max_wall_time_s=600),
+    )
+    result = agent.run("Lift the cube off the table.")
+
+print(result.status, result.outcome.reason)
+print("trace:", result.trace_dir)

nomad_harness-0.1.0.dev1/examples/robosuite_openai_lift.py

@@ -0,0 +1,104 @@
+"""OpenAI model rollout on robosuite Lift: the model sees only images and proprioception.
+
+An OpenAI model drives the Panda through the Nomad runtime with canonical
+``move_ee``/``set_gripper`` actions, one decision at a time, from the front
+camera image and robot state (``rgb_proprio``: no simulator object state).
+Success is decided by robosuite's own Lift predicate, never by the model.
+Every run writes config.json, trace.jsonl, frames/, metrics.json, and video.mp4,
+and its outcome is reported whether or not the cube was lifted.
+
+Spends API credits. The model must be one your account can access:
+
+    pip install -e '.[robosuite,openai]'
+    export OPENAI_API_KEY=...            # never written to traces
+    python examples/robosuite_openai_lift.py --model "$NOMAD_OPENAI_MODEL" [--seed 0]
+"""
+
+from __future__ import annotations
+
+import argparse
+import os
+import sys
+from pathlib import Path
+
+from nomad import Agent, ObservationMode, RunConfig, RunResult, RunStatus
+from nomad.core import MissingDependencyError, ModelError
+from nomad.embodiments import Robosuite
+from nomad.models import OpenAI
+
+GOAL = "Lift the cube off the table."
+
+
+def print_summary(result: RunResult) -> None:
+    m = result.metrics
+    tokens = (
+        f"{m.input_tokens} in / {m.output_tokens} out"
+        if m.input_tokens is not None and m.output_tokens is not None
+        else "unknown"
+    )
+    print(f"status:       {result.status} ({result.stop_reason})")
+    print(f"outcome:      {result.outcome.status}: {result.outcome.reason}")
+    print(f"evidence:     {result.outcome.evidence}")
+    print(
+        f"decisions:    {m.decisions} "
+        f"(completed {m.actions_completed}, rejected {m.rejections}, "
+        f"failed {m.execution_failures}, provider errors {m.provider_errors})"
+    )
+    print(f"model:        {m.provider_calls} calls, {m.model_latency_s:.1f} s, tokens {tokens}")
+    print(f"sim time:     {m.sim_time_s:.2f} s (wall {m.wall_time_s:.1f} s)")
+    if result.error:
+        print(f"error:        {result.error}")
+    if result.trace_dir is not None:
+        video = result.trace_dir / "video.mp4"
+        print(f"trace:        {result.trace_dir}")
+        print(f"video:        {video if video.exists() else 'not recorded'}")
+
+
+def main(argv: list[str] | None = None) -> int:
+    parser = argparse.ArgumentParser(description=__doc__.split("\n\n")[0])
+    parser.add_argument(
+        "--model",
+        default=os.environ.get("NOMAD_OPENAI_MODEL"),
+        help="OpenAI model ID (default: $NOMAD_OPENAI_MODEL)",
+    )
+    parser.add_argument("--seed", type=int, default=0)
+    parser.add_argument("--max-decisions", type=int, default=30)
+    parser.add_argument("--max-wall-time", type=float, default=600.0, help="seconds")
+    parser.add_argument("--reasoning-effort", default=None, help="e.g. low, medium, high")
+    parser.add_argument("--history-window", type=int, default=8, help="0 disables feedback")
+    parser.add_argument("--trace-root", type=Path, default=Path("runs"))
+    parser.add_argument("--no-video", action="store_true", help="skip video.mp4")
+    args = parser.parse_args(argv)
+    if not args.model:
+        parser.error("pass --model or set NOMAD_OPENAI_MODEL")
+
+    config = RunConfig(
+        seed=args.seed,
+        observation_mode=ObservationMode.RGB_PROPRIO,
+        max_decisions=args.max_decisions,
+        max_provider_calls=args.max_decisions,
+        max_wall_time_s=args.max_wall_time,
+        history_window=args.history_window,
+        trace_root=args.trace_root,
+        save_video=not args.no_video,
+    )
+    try:
+        model = OpenAI(
+            model=args.model,
+            reasoning_effort=args.reasoning_effort,
+            max_calls=args.max_decisions,
+        )
+        env = Robosuite(task="Lift", robot="Panda")
+    except (MissingDependencyError, ModelError) as exc:
+        print(f"error: {exc}", file=sys.stderr)
+        return 2
+
+    with model, env:
+        result = Agent(model, env, config).run(GOAL)
+
+    print_summary(result)
+    return 0 if result.status is RunStatus.SUCCESS else 1
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

{nomad_harness-0.1.0.dev0 → nomad_harness-0.1.0.dev1}/pyproject.toml

@@ -5,7 +5,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "nomad-harness"
-version = "0.1.0.dev0"
+version = "0.1.0.dev1"
 description = "Action runtime connecting frontier models to simulated and physical embodiments."
 readme = "README.md"
 requires-python = ">=3.10"
@@ -39,12 +39,15 @@ dev = [
     "mypy>=1.10",
     "jsonschema>=4.18",
     "types-PyYAML",
+    # The OpenAI adapter is type-checked and tested offline in CI.
+    "openai>=3.24,<4",
 ]
 # Integration extras. Version pins are set when each adapter is validated
 # against a concrete release. robosuite 1.5.2 does not bound mujoco, and newer
 # mujoco releases break it, so both are pinned to the tested pair.
 robosuite = ["robosuite==1.5.2", "mujoco==3.3.7", "numpy", "imageio[ffmpeg]"]
-openai = ["openai"]
+# Request and response handling measured against openai 3.24.0 (Responses API).
+openai = ["openai>=3.24,<4"]
 anthropic = ["anthropic"]
 gemini = ["google-genai"]
 
@@ -56,6 +59,7 @@ testpaths = ["tests"]
 addopts = "-ra --strict-markers"
 markers = [
     "robosuite: needs robosuite and mujoco installed (pip install -e '.[robosuite]')",
+    "openai_live: calls the live OpenAI API; opt in with NOMAD_OPENAI_LIVE=1",
 ]
 
 [tool.ruff]

{nomad_harness-0.1.0.dev0 → nomad_harness-0.1.0.dev1}/src/nomad/__init__.py

@@ -29,7 +29,7 @@ from nomad.core import (
 )
 from nomad.runtime import Runtime
 
-__version__ = "0.1.0.dev0"
+__version__ = "0.1.0.dev1"
 
 __all__ = [
     "Action",

{nomad_harness-0.1.0.dev0 → nomad_harness-0.1.0.dev1}/src/nomad/context.py

@@ -22,14 +22,54 @@ Conventions:
 - set_gripper: closure 0 is fully open, 1 is fully closed. It is a setting, not a force.
 - duration_s is the maximum execution time; motion that does not converge reports a timeout.
 
+Limits:
+{limits}
+
 Rules:
-- Respond with exactly one action per decision, matching the provided schema.
+- Respond with exactly one action per decision, using the provided action tools or schema.
 - Requests outside the limits are rejected, not clipped; read the feedback and adjust.
-- Set request_finish to true when you believe the task is complete. An independent check
-  decides success; requesting finish does not make the task succeed.
+- When you believe the task is complete, request finish (the finish tool, or request_finish
+  set to true). An independent check decides success; requesting finish does not make the
+  task succeed.
 """
 
 
+def describe_limits(manifest: EmbodimentManifest) -> str:
+    """The manifest's action limits and workspace as instruction lines.
+
+    JSON Schema cannot express norm or workspace limits, so the model would
+    otherwise only learn them from rejections.
+    """
+    lines: list[str] = []
+    actions = manifest.actions
+    if actions.move_ee is not None:
+        spec = actions.move_ee
+        lines.append(
+            f"- move_ee: frame in {list(spec.frames)}; "
+            f"|translation_m| <= {spec.max_translation_norm_m:g} m; "
+            f"|rotation_vector_rad| <= {spec.max_rotation_norm_rad:g} rad; "
+            f"duration_s <= {spec.max_duration_s:g} s."
+        )
+    if actions.set_gripper is not None:
+        grip = actions.set_gripper
+        low, high = grip.range
+        lines.append(
+            f"- set_gripper: closure in [{low:g}, {high:g}]; "
+            f"duration_s <= {grip.max_duration_s:g} s."
+        )
+    if manifest.workspace is not None:
+        ws = manifest.workspace
+        bounds = ", ".join(
+            f"{axis} [{lo:g}, {hi:g}]"
+            for axis, lo, hi in zip("xyz", ws.min_m, ws.max_m, strict=True)
+        )
+        lines.append(
+            f"- The end-effector target (current position + translation_m) must stay inside "
+            f"the '{ws.frame}' workspace: {bounds} m."
+        )
+    return "\n".join(lines)
+
+
 class ContextBuilder:
     """Turns observations and feedback into :class:`AgentContext` records.
 
@@ -61,6 +101,7 @@ class ContextBuilder:
             robot=manifest.robot,
             task=manifest.task,
             pose_frame=manifest.pose_frame,
+            limits=describe_limits(manifest),
         )
 
     def record(self, entry: HistoryEntry) -> None:

{nomad_harness-0.1.0.dev0 → nomad_harness-0.1.0.dev1}/src/nomad/core/model_io.py

@@ -60,6 +60,11 @@ class ModelResponse(Record):
 
     ``raw_proposal`` is untrusted: the runtime parses and validates it. Adapters
     must not pre-validate it into a :class:`~nomad.core.actions.Proposal`.
+
+    ``provider_details`` is optional JSON-compatible metadata for the trace,
+    e.g. the provider's response ID, finish reason, retry count, and a summary
+    of the request actually sent. Images are referenced by observation ID and
+    camera, never embedded, and credentials must never appear.
     """
 
     status: ModelResponseStatus
@@ -69,6 +74,7 @@ class ModelResponse(Record):
     raw_proposal: dict[str, Any] | str | None = None
     usage: Usage | None = None
     error: str | None = None
+    provider_details: dict[str, Any] | None = None
 
     @model_validator(mode="after")
     def _payload_matches_status(self) -> ModelResponse:

{nomad_harness-0.1.0.dev0 → nomad_harness-0.1.0.dev1}/src/nomad/core/observations.py

@@ -55,6 +55,10 @@ class Observation(Record):
     ``privileged`` may carry simulator-only state (object poses, etc.) for
     baselines and debugging. The context builder strips it unless the run is
     explicitly in ``privileged_state`` mode.
+
+    ``frame_poses`` maps declared manifest frames to their pose in ``world`` at
+    capture time (``world_from_<frame>``). It relates world-frame camera poses
+    to the robot-state pose frame; frames whose pose is unknown are omitted.
     """
 
     id: Annotated[str, Field(min_length=1)]
@@ -63,4 +67,5 @@ class Observation(Record):
     capture_wall_s: FiniteFloat
     frames: dict[str, CameraFrame]
     robot_state: RobotState
+    frame_poses: dict[str, Pose] = Field(default_factory=dict)
     privileged: dict[str, Any] | None = None

{nomad_harness-0.1.0.dev0 → nomad_harness-0.1.0.dev1}/src/nomad/embodiments/robosuite/_imports.py

@@ -8,7 +8,8 @@ from typing import NamedTuple
 
 from nomad.core.errors import MissingDependencyError
 
-INSTALL_HINT = "pip install -e '.[robosuite]'"
+# The installed-package form; from a source checkout, `pip install -e '.[robosuite]'` also works.
+INSTALL_HINT = "pip install 'nomad-harness[robosuite]'"
 
 
 class Backend(NamedTuple):

{nomad_harness-0.1.0.dev0 → nomad_harness-0.1.0.dev1}/src/nomad/embodiments/robosuite/adapter.py

@@ -154,6 +154,7 @@ class Robosuite:
                 for stream, camera in self.config.cameras.items()
             },
             robot_state=self._robot_state(),
+            frame_poses={self._manifest.pose_frame: self._world_from_base()},
             privileged=self._privileged_state(),
         )
 

{nomad_harness-0.1.0.dev0 → nomad_harness-0.1.0.dev1}/src/nomad/embodiments/robosuite/config.py

@@ -17,6 +17,7 @@ from pydantic import Field, PositiveInt
 from nomad.core.errors import ManifestError, NomadError
 from nomad.core.manifest import EmbodimentManifest, RgbStreamSpec
 from nomad.core.types import PositiveFloat, Record
+from nomad.embodiments.robosuite._imports import INSTALL_HINT
 
 # Versions the adapter's conversions and defaults were measured against. robosuite
 # declares ``mujoco>=3.3.0`` without an upper bound, but newer mujoco releases break
@@ -148,6 +149,6 @@ def check_versions(installed: Mapping[str, str | None], *, allow_untested: bool)
         found = ", ".join(f"{n} {v}" for n, v in mismatched.items())
         raise UnsupportedVersionError(
             f"the robosuite adapter is validated with {wanted}, found {found}. Install the "
-            "tested versions with `pip install -e '.[robosuite]'`, or set "
+            f"tested versions with `{INSTALL_HINT}`, or set "
             "RobosuiteConfig(allow_untested_versions=True) to try anyway."
         )

{nomad_harness-0.1.0.dev0 → nomad_harness-0.1.0.dev1}/src/nomad/models/__init__.py

@@ -5,6 +5,7 @@ that a missing SDK only fails when that adapter is requested.
 """
 
 from nomad.models.fake import FakeFailure, FakeModel
+from nomad.models.openai import OpenAI, OpenAIConfig
 from nomad.models.scripted import ScriptedLiftPolicy
 
-__all__ = ["FakeFailure", "FakeModel", "ScriptedLiftPolicy"]
+__all__ = ["FakeFailure", "FakeModel", "OpenAI", "OpenAIConfig", "ScriptedLiftPolicy"]