jolly-cli 0.3.0__tar.gz → 0.4.0__tar.gz

{jolly_cli-0.3.0 → jolly_cli-0.4.0}/PKG-INFO

@@ -1,7 +1,7 @@
 Metadata-Version: 2.4
 Name: jolly-cli
-Version: 0.3.0
-Summary: Local, CLI-native robot-arm physics simulation for humans and LLM agents
+Version: 0.4.0
+Summary: CLI-native SO-101 physics tasks and physical hardware control
 Project-URL: Homepage, https://github.com/TeamAudiyo/Jolly
 Project-URL: Repository, https://github.com/TeamAudiyo/Jolly
 Project-URL: Issues, https://github.com/TeamAudiyo/Jolly/issues
@@ -9,7 +9,7 @@ Author: TeamAudiyo
 License: MIT OR Apache-2.0
 License-File: LICENSE-APACHE
 License-File: LICENSE-MIT
-Keywords: cli,kinematics,llm,pybullet,robotics,simulation
+Keywords: cli,hardware,kinematics,llm,pybullet,robotics,so101
 Classifier: Development Status :: 3 - Alpha
 Classifier: Environment :: Console
 Classifier: License :: OSI Approved :: Apache Software License
@@ -27,8 +27,11 @@ Requires-Dist: rich<15,>=13.7
 Provides-Extra: dev
 Requires-Dist: build<2,>=1.2; extra == 'dev'
 Requires-Dist: httpx<1,>=0.27; extra == 'dev'
+Requires-Dist: pyserial<4,>=3.5; extra == 'dev'
 Requires-Dist: pytest-cov<7,>=5; extra == 'dev'
 Requires-Dist: pytest<9,>=8.2; extra == 'dev'
+Provides-Extra: hardware
+Requires-Dist: pyserial<4,>=3.5; extra == 'hardware'
 Provides-Extra: web
 Requires-Dist: fastapi<1,>=0.115; extra == 'web'
 Requires-Dist: uvicorn<1,>=0.30; extra == 'web'
@@ -36,8 +39,15 @@ Description-Content-Type: text/markdown
 
 # Jolly CLI
 
-Jolly is a local robot-arm physics simulator for terminal users and LLM agents.
-It uses PyBullet in deterministic direct mode and saves state between commands.
+Jolly provides two separate robot-control backends for terminal users and LLM
+agents:
+
+- `JollyEngine` executes measured contact tasks with the official SO-101 model
+  in PyBullet.
+- `SO101HardwareDriver` sends the real STS3215 serial protocol to a physical
+  SO-101 and reads measured motor positions.
+
+The two backends never combine their scores.
 
 Jolly includes two offline robot profiles:
 
@@ -99,7 +109,7 @@ jolly move --joints "0,-20,40,-20,0" --gripper 0.0 --json
 jolly reach --x 0.30 --y 0.10 --z 0.18 --gripper 0.0 --json
 jolly render
 jolly challenge list --json
-jolly benchmark --json
+jolly benchmark --seed 42017 --cases 3 --model so101 --json
 ```
 
 Start the optional local web viewer:
@@ -140,7 +150,8 @@ Important commands:
 | `jolly reset` | Select a model and scene, then home the robot. |
 | `jolly scene list/load` | Discover and load deterministic scenes. |
 | `jolly challenge list/start/status` | Run seeded randomized manipulation challenges. |
-| `jolly benchmark [--seed N] [--cases N]` | Run randomized engine checks with reproducible inputs. |
+| `jolly benchmark [--seed N] [--cases N]` | Execute randomized reach, obstacle, and drop-in-hole contact tasks. |
+| `jolly hardware state/move/benchmark/stop` | Control and measure a physical SO-101 over its serial bus. |
 
 Jolly rejects joint-limit violations and unsafe workspace targets. By default,
 Jolly rolls back a motion that creates a collision. Use `--allow-collision`
@@ -184,10 +195,10 @@ jolly-cli/
 
 ## Physics scope
 
-`JollyEngine` owns state, safety, motion rollback, grasping, rendering, and the
-public simulator contract. `JollyDriver` generates seeded challenge and
-benchmark cases. The project does not depend on Inspect, Inspect AI, or an
-external robotics benchmark driver.
+`JollyEngine` owns state, safety, motion rollback, constrained grasping,
+rendering, and the physics-task contract. `JollyDriver` generates seeded
+challenge and benchmark cases. The project does not depend on Inspect, Inspect
+AI, or an external robotics benchmark driver.
 
 PyBullet is the low-level open-source physics backend. It performs rigid-body
 simulation, FK, IK, contact generation, and scene settling. Jolly uses
@@ -207,9 +218,46 @@ jolly challenge start sort-red --seed 42017 --json
 jolly benchmark --seed 42017 --cases 5 --json
 ```
 
-The benchmark generates new joint configurations for every robot case and new
-object layouts for every scene case. This prevents a policy from passing only by
-memorizing the original fixed coordinates.
+The benchmark physically executes randomized reach, obstacle avoidance, and
+drop-in-hole tasks. Its score comes from measured reach error, collisions, grasp
+state, physical release, and final peg pose. A generated health check does not
+add points. Different seeds can produce different scores.
+
+## Physical SO-101 hardware
+
+Install direct physical-hardware support:
+
+```bash
+python -m pip install 'jolly-cli[hardware]'
+```
+
+Jolly implements the Feetech STS3215 packet protocol directly. It does not use
+Inspect Robots or an external robot SDK. Copy and calibrate the example file
+before enabling torque:
+
+```bash
+jolly hardware calibration-example --output so101-calibration.json
+# Replace every placeholder with measurements from this physical arm.
+jolly hardware scan --port /dev/ttyACM0 --calibration so101-calibration.json --json
+jolly hardware state --port /dev/ttyACM0 --calibration so101-calibration.json --json
+jolly hardware move --port /dev/ttyACM0 --calibration so101-calibration.json \
+  --joints '0,0,0,0,0' --gripper 0 --confirm-hardware --json
+jolly hardware benchmark --port /dev/ttyACM0 --calibration so101-calibration.json \
+  --seed 42017 --cases 3 --confirm-hardware --json
+jolly hardware stop --port /dev/ttyACM0 --calibration so101-calibration.json
+```
+
+The example calibration values are placeholders and set `calibrated` to false.
+Measure every zero point, direction, and gripper endpoint before setting it to
+true. Physical commands require an explicit confirmation, enforce degree and
+raw-count step limits, set bounded goal speed, issue simultaneous goals, and
+read motor feedback. The driver disables torque after communication failures.
+After successful movement it keeps torque enabled so the arm does not fall.
+Support the arm before running `jolly hardware stop`.
+
+The physical benchmark scores measured joint-position error only. The physics
+benchmark scores contact tasks only. Jolly never presents physics output as a
+physical-hardware result.
 
 The bundled SO-101 model is the official Apache-2.0 new-calibration URDF from
 The Robot Studio. Jolly includes the 13 referenced STL meshes from pinned commit

{jolly_cli-0.3.0 → jolly_cli-0.4.0}/README.md

@@ -1,7 +1,14 @@
 # Jolly CLI
 
-Jolly is a local robot-arm physics simulator for terminal users and LLM agents.
-It uses PyBullet in deterministic direct mode and saves state between commands.
+Jolly provides two separate robot-control backends for terminal users and LLM
+agents:
+
+- `JollyEngine` executes measured contact tasks with the official SO-101 model
+  in PyBullet.
+- `SO101HardwareDriver` sends the real STS3215 serial protocol to a physical
+  SO-101 and reads measured motor positions.
+
+The two backends never combine their scores.
 
 Jolly includes two offline robot profiles:
 
@@ -63,7 +70,7 @@ jolly move --joints "0,-20,40,-20,0" --gripper 0.0 --json
 jolly reach --x 0.30 --y 0.10 --z 0.18 --gripper 0.0 --json
 jolly render
 jolly challenge list --json
-jolly benchmark --json
+jolly benchmark --seed 42017 --cases 3 --model so101 --json
 ```
 
 Start the optional local web viewer:
@@ -104,7 +111,8 @@ Important commands:
 | `jolly reset` | Select a model and scene, then home the robot. |
 | `jolly scene list/load` | Discover and load deterministic scenes. |
 | `jolly challenge list/start/status` | Run seeded randomized manipulation challenges. |
-| `jolly benchmark [--seed N] [--cases N]` | Run randomized engine checks with reproducible inputs. |
+| `jolly benchmark [--seed N] [--cases N]` | Execute randomized reach, obstacle, and drop-in-hole contact tasks. |
+| `jolly hardware state/move/benchmark/stop` | Control and measure a physical SO-101 over its serial bus. |
 
 Jolly rejects joint-limit violations and unsafe workspace targets. By default,
 Jolly rolls back a motion that creates a collision. Use `--allow-collision`
@@ -148,10 +156,10 @@ jolly-cli/
 
 ## Physics scope
 
-`JollyEngine` owns state, safety, motion rollback, grasping, rendering, and the
-public simulator contract. `JollyDriver` generates seeded challenge and
-benchmark cases. The project does not depend on Inspect, Inspect AI, or an
-external robotics benchmark driver.
+`JollyEngine` owns state, safety, motion rollback, constrained grasping,
+rendering, and the physics-task contract. `JollyDriver` generates seeded
+challenge and benchmark cases. The project does not depend on Inspect, Inspect
+AI, or an external robotics benchmark driver.
 
 PyBullet is the low-level open-source physics backend. It performs rigid-body
 simulation, FK, IK, contact generation, and scene settling. Jolly uses
@@ -171,9 +179,46 @@ jolly challenge start sort-red --seed 42017 --json
 jolly benchmark --seed 42017 --cases 5 --json
 ```
 
-The benchmark generates new joint configurations for every robot case and new
-object layouts for every scene case. This prevents a policy from passing only by
-memorizing the original fixed coordinates.
+The benchmark physically executes randomized reach, obstacle avoidance, and
+drop-in-hole tasks. Its score comes from measured reach error, collisions, grasp
+state, physical release, and final peg pose. A generated health check does not
+add points. Different seeds can produce different scores.
+
+## Physical SO-101 hardware
+
+Install direct physical-hardware support:
+
+```bash
+python -m pip install 'jolly-cli[hardware]'
+```
+
+Jolly implements the Feetech STS3215 packet protocol directly. It does not use
+Inspect Robots or an external robot SDK. Copy and calibrate the example file
+before enabling torque:
+
+```bash
+jolly hardware calibration-example --output so101-calibration.json
+# Replace every placeholder with measurements from this physical arm.
+jolly hardware scan --port /dev/ttyACM0 --calibration so101-calibration.json --json
+jolly hardware state --port /dev/ttyACM0 --calibration so101-calibration.json --json
+jolly hardware move --port /dev/ttyACM0 --calibration so101-calibration.json \
+  --joints '0,0,0,0,0' --gripper 0 --confirm-hardware --json
+jolly hardware benchmark --port /dev/ttyACM0 --calibration so101-calibration.json \
+  --seed 42017 --cases 3 --confirm-hardware --json
+jolly hardware stop --port /dev/ttyACM0 --calibration so101-calibration.json
+```
+
+The example calibration values are placeholders and set `calibrated` to false.
+Measure every zero point, direction, and gripper endpoint before setting it to
+true. Physical commands require an explicit confirmation, enforce degree and
+raw-count step limits, set bounded goal speed, issue simultaneous goals, and
+read motor feedback. The driver disables torque after communication failures.
+After successful movement it keeps torque enabled so the arm does not fall.
+Support the arm before running `jolly hardware stop`.
+
+The physical benchmark scores measured joint-position error only. The physics
+benchmark scores contact tasks only. Jolly never presents physics output as a
+physical-hardware result.
 
 The bundled SO-101 model is the official Apache-2.0 new-calibration URDF from
 The Robot Studio. Jolly includes the 13 referenced STL meshes from pinned commit

jolly_cli-0.4.0/docs/hardware/so101-calibration.example.json

@@ -0,0 +1,10 @@
+{
+  "motors": {
+    "shoulder_pan": {"id": 1, "zero_raw": 2048, "direction": 1, "raw_per_degree": 11.3777777778},
+    "shoulder_lift": {"id": 2, "zero_raw": 2048, "direction": 1, "raw_per_degree": 11.3777777778},
+    "elbow_flex": {"id": 3, "zero_raw": 2048, "direction": 1, "raw_per_degree": 11.3777777778},
+    "wrist_flex": {"id": 4, "zero_raw": 2048, "direction": 1, "raw_per_degree": 11.3777777778},
+    "wrist_roll": {"id": 5, "zero_raw": 2048, "direction": 1, "raw_per_degree": 11.3777777778},
+    "gripper": {"id": 6, "open_raw": 2048, "closed_raw": 3072}
+  }
+}

{jolly_cli-0.3.0 → jolly_cli-0.4.0}/docs/wiki/CLI-Reference.md

@@ -23,12 +23,24 @@ jolly scene load blocks --json
 jolly challenge list --json
 jolly challenge start sort-red --model jolly6 [--seed N] --json
 jolly challenge status --json
-jolly benchmark [--seed N] [--cases 3] --json
+jolly benchmark [--seed N] [--cases 3] [--model so101] --json
 ```
 
 Challenge starts use a fresh random seed by default. The JSON response contains
 the seed and generated instance. Supply `--seed N` to replay the exact case.
-The benchmark randomizes every robot and scene check through `JollyDriver`.
+The benchmark executes randomized reach, obstacle, and drop-in-hole tasks through `JollyDriver`.
+
+## Physical SO-101
+
+```bash
+jolly hardware scan --port /dev/ttyACM0 --calibration calibration.json --json
+jolly hardware state --port /dev/ttyACM0 --calibration calibration.json --json
+jolly hardware move --port /dev/ttyACM0 --calibration calibration.json --joints "0,0,0,0,0" --gripper 0 --confirm-hardware --json
+jolly hardware benchmark --port /dev/ttyACM0 --calibration calibration.json --seed 42 --cases 3 --confirm-hardware --json
+jolly hardware stop --port /dev/ttyACM0 --calibration calibration.json
+```
+
+Physics and physical-hardware results use separate benchmark names and scores.
 
 ## Viewers
 

{jolly_cli-0.3.0 → jolly_cli-0.4.0}/docs/wiki/Challenges-and-Benchmarks.md

@@ -20,10 +20,13 @@ jolly challenge status --json
 The start result contains a generated seed and instance. Use `--seed` to replay
 the exact case.
 
-`jolly benchmark --json` checks engine health with randomized joint states and
-scene layouts. Use `--cases` to select cases per model and scene. Use `--seed`
-to replay the exact generated inputs.
+`jolly benchmark --json` executes randomized reaching, obstacle avoidance, and
+drop-in-hole tasks. The score uses measured reach error, collision state, grasp
+state, release state, and the peg's final physical pose. Scene construction and
+forward-kinematics health checks contribute no points.
 
-Jolly uses its own `JollyEngine` and `JollyDriver`. It does not use Inspect,
-Inspect AI, or an external robot benchmark harness. PyBullet remains the local
-open-source rigid-body physics backend.
+Jolly uses its own `JollyEngine`, `JollyDriver`, and direct
+`SO101HardwareDriver`. It does not use Inspect, Inspect AI, or an external robot
+benchmark harness. PyBullet remains the local open-source rigid-body backend
+for physics tasks. Physical SO-101 results come only from STS3215 motor feedback
+and remain separate from physics scores.

jolly_cli-0.4.0/docs/wiki/Physical-SO-101-Hardware.md

@@ -0,0 +1,35 @@
+# Physical SO-101 Hardware
+
+Install the serial dependency with `pip install 'jolly-cli[hardware]'`.
+
+Jolly's `SO101HardwareDriver` sends Feetech STS3215 packets directly at the
+SO-101 default 1,000,000 baud. The configured motor IDs default to the common
+SO-101 sequence of 1 through 6. Each physical arm requires measured calibration.
+
+Generate a template, then replace all placeholder zero points, directions, and
+gripper endpoints before movement.
+
+```bash
+jolly hardware calibration-example --output calibration.json
+jolly hardware scan --port /dev/ttyACM0 --calibration calibration.json --json
+jolly hardware state --port /dev/ttyACM0 --calibration calibration.json --json
+jolly hardware move --port /dev/ttyACM0 --calibration calibration.json \
+  --joints '0,0,0,0,0' --gripper 0 --confirm-hardware --json
+jolly hardware benchmark --port /dev/ttyACM0 --calibration calibration.json \
+  --seed 42017 --cases 3 --confirm-hardware --json
+jolly hardware stop --port /dev/ttyACM0 --calibration calibration.json
+```
+
+The physical benchmark makes bounded movements of at most five degrees from the
+measured starting pose. It scores measured motor-position error and returns to
+the starting pose. Torque stays enabled to hold that pose. Support the arm
+before `jolly hardware stop`. Communication failures trigger a best-effort
+torque disable. Clear the workspace and keep an emergency power cutoff nearby.
+
+Physical hardware scores and PyBullet contact-task scores are separate. Jolly
+never labels a physics result as a physical-hardware result.
+
+Protocol and hardware references:
+
+- https://huggingface.co/docs/lerobot/en/so101
+- https://pages.switch-science.com/comparison/files/feetech/serial-sts/STS3215_datasheet.pdf

{jolly_cli-0.3.0 → jolly_cli-0.4.0}/docs/wiki/_Sidebar.md

@@ -5,5 +5,6 @@
 - [CLI Reference](CLI-Reference)
 - [Robot Models and Licensing](Robot-Models-and-Licensing)
 - [Challenges and Benchmarks](Challenges-and-Benchmarks)
+- [Physical SO-101 Hardware](Physical-SO-101-Hardware)
 - [LLM Agent Safety](LLM-Agent-Safety)
 - [Web Viewer](Web-Viewer)

{jolly_cli-0.3.0 → jolly_cli-0.4.0}/jolly/__init__.py

@@ -1,4 +1,4 @@
 """Jolly robot-arm simulator."""
 
 __all__ = ["__version__"]
-__version__ = "0.3.0"
+__version__ = "0.4.0"

jolly_cli-0.4.0/jolly/benchmark.py

@@ -0,0 +1,192 @@
+from __future__ import annotations
+
+import math
+import time
+from typing import Any, Callable
+
+from jolly.challenges import list_challenges
+from jolly.core.errors import ConfigurationError
+from jolly.driver import JollyDriver
+from jolly.engine import JollyEngine
+
+
+def _bounded_accuracy(error: float, full_score_error: float, zero_score_error: float) -> float:
+    if error <= full_score_error:
+        return 1.0
+    if error >= zero_score_error:
+        return 0.0
+    return 1.0 - (error - full_score_error) / (zero_score_error - full_score_error)
+
+
+def _task_result(
+    *,
+    name: str,
+    weight: float,
+    seed: int,
+    started: float,
+    score: float,
+    metrics: dict[str, object],
+    error: str | None = None,
+) -> dict[str, object]:
+    result: dict[str, object] = {
+        "name": name,
+        "seed": seed,
+        "weight": weight,
+        "score": round(max(0.0, min(weight, score)), 3),
+        "passed": score >= weight * 0.7,
+        "duration_ms": round((time.perf_counter() - started) * 1000, 3),
+        "metrics": metrics,
+    }
+    if error:
+        result["error"] = error
+    return result
+
+
+def _run_reach_task(model: str, instance: dict[str, Any]) -> dict[str, object]:
+    started = time.perf_counter()
+    weight = 25.0
+    target = [float(value) for value in instance["target"]]
+    try:
+        with JollyEngine(model=model, scene="empty") as engine:
+            state = engine.reach(*target, tolerance=0.06)
+            actual = state["end_effector"]["position"]
+            error = math.dist(actual, target)
+            collision_free = not state["collisions"]["collision"]
+            score = weight * _bounded_accuracy(error, 0.015, 0.08)
+            if not collision_free:
+                score *= 0.25
+            return _task_result(
+                name="randomized-reach",
+                weight=weight,
+                seed=int(instance["seed"]),
+                started=started,
+                score=score,
+                metrics={"target": target, "actual": actual, "error_meters": round(error, 6), "collision_free": collision_free},
+            )
+    except Exception as exc:
+        return _task_result(name="randomized-reach", weight=weight, seed=int(instance["seed"]), started=started, score=0.0, metrics={"target": target}, error=str(exc))
+
+
+def _run_obstacle_task(model: str, instance: dict[str, Any]) -> dict[str, object]:
+    started = time.perf_counter()
+    weight = 25.0
+    goal = [float(value) for value in instance["target"]]
+    target = [goal[0], goal[1], max(0.07, goal[2] + 0.045)]
+    try:
+        with JollyEngine(model=model, scene="obstacles") as engine:
+            engine.set_object_positions(instance["object_positions"])
+            state = engine.reach(*target, tolerance=0.07)
+            actual = state["end_effector"]["position"]
+            error = math.dist(actual, goal)
+            collision_free = not state["collisions"]["collision"]
+            score = weight * _bounded_accuracy(error, 0.05, 0.12)
+            if not collision_free:
+                score = 0.0
+            return _task_result(
+                name="randomized-obstacle-reach",
+                weight=weight,
+                seed=int(instance["seed"]),
+                started=started,
+                score=score,
+                metrics={"goal": goal, "commanded_target": target, "actual": actual, "goal_error_meters": round(error, 6), "collision_free": collision_free},
+            )
+    except Exception as exc:
+        return _task_result(name="randomized-obstacle-reach", weight=weight, seed=int(instance["seed"]), started=started, score=0.0, metrics={"target": target}, error=str(exc))
+
+
+def _run_drop_task(model: str, instance: dict[str, Any]) -> dict[str, object]:
+    started = time.perf_counter()
+    weight = 50.0
+    peg_start = [float(value) for value in instance["object_positions"]["peg"]]
+    target = [float(value) for value in instance["target"]]
+    metrics: dict[str, object] = {"peg_start": peg_start, "hole_center": target}
+    try:
+        with JollyEngine(model=model, scene="insertion") as engine:
+            engine.set_object_positions(instance["object_positions"])
+            engine.reach(peg_start[0], peg_start[1], peg_start[2] + 0.10, gripper=0.0, tolerance=0.06)
+            grasp_state = engine.reach(peg_start[0], peg_start[1], peg_start[2] + 0.055, gripper=1.0, tolerance=0.07)
+            grasped = grasp_state["held_object"] == "peg"
+            metrics["grasped"] = grasped
+            if grasped:
+                engine.reach(peg_start[0], peg_start[1], 0.34, gripper=1.0, tolerance=0.08)
+                state = engine.reach(target[0], target[1], 0.34, gripper=1.0, tolerance=0.08)
+                for height in (0.34, 0.27, 0.27):
+                    peg = next(item for item in state["objects"] if item["name"] == "peg")
+                    tool = state["end_effector"]["position"]
+                    corrected_x = tool[0] + target[0] - peg["position"][0]
+                    corrected_y = tool[1] + target[1] - peg["position"][1]
+                    state = engine.reach(corrected_x, corrected_y, height, gripper=1.0, tolerance=0.08)
+                tool = state["end_effector"]["position"]
+                state = engine.reach(tool[0], tool[1], 0.27, gripper=0.0, tolerance=0.08)
+            else:
+                state = grasp_state
+            peg = next(item for item in state["objects"] if item["name"] == "peg")
+            xy_error = math.dist(peg["position"][:2], target[:2])
+            below_rim = peg["position"][2] < float(instance["rim_height"])
+            released = state["held_object"] is None
+            collision_free = not state["collisions"]["collision"]
+            inside = xy_error <= float(instance["hole_radius"]) and below_rim
+            score = 0.0
+            if grasped:
+                score += 10.0
+                score += 15.0 * _bounded_accuracy(xy_error, 0.008, 0.06)
+                score += 20.0 if inside else (8.0 if below_rim and released else 0.0)
+                score += 5.0 if released and collision_free else 0.0
+            metrics.update(
+                {
+                    "final_position": peg["position"],
+                    "xy_error_meters": round(xy_error, 6),
+                    "inside_hole": inside,
+                    "below_rim": below_rim,
+                    "released": released,
+                    "collision_free": collision_free,
+                }
+            )
+            return _task_result(name="randomized-drop-in-hole", weight=weight, seed=int(instance["seed"]), started=started, score=score, metrics=metrics)
+    except Exception as exc:
+        return _task_result(name="randomized-drop-in-hole", weight=weight, seed=int(instance["seed"]), started=started, score=0.0, metrics=metrics, error=str(exc))
+
+
+def run_benchmark(*, seed: int | None = None, cases: int = 3, model: str = "so101") -> dict[str, object]:
+    """Execute randomized contact tasks and score only measured physical outcomes."""
+    if not 1 <= cases <= 25:
+        raise ConfigurationError("Benchmark cases must be between 1 and 25.")
+    if model not in ("so101", "jolly6"):
+        raise ConfigurationError(f"Unsupported benchmark model '{model}'.")
+    driver = JollyDriver(seed)
+    tasks: list[dict[str, object]] = []
+    started = time.perf_counter()
+    generators: list[tuple[str, Callable[[str, dict[str, Any]], dict[str, object]]]] = [
+        ("reach-center", _run_reach_task),
+        ("obstacle-reach", _run_obstacle_task),
+        ("drop-in-hole", _run_drop_task),
+    ]
+    for case_index in range(cases):
+        for challenge_id, execute in generators:
+            instance = driver.challenge_instance(challenge_id)
+            result = execute(model, instance)
+            result["case"] = case_index + 1
+            result["challenge"] = challenge_id
+            result["instance"] = instance
+            tasks.append(result)
+    earned = sum(float(task["score"]) for task in tasks)
+    available = sum(float(task["weight"]) for task in tasks)
+    score = 100.0 * earned / available if available else 0.0
+    passed = sum(1 for task in tasks if task["passed"])
+    return {
+        "ok": passed == len(tasks),
+        "benchmark": "jolly-pybullet-contact-tasks-v4",
+        "engine": JollyEngine.name,
+        "driver": JollyDriver.name,
+        "physics_backend": JollyEngine.physics_backend,
+        "model": model,
+        "seed": driver.seed,
+        "cases": cases,
+        "score": round(score, 2),
+        "score_basis": "PyBullet reach error, contact-safe motion, contact-confirmed grasp, constrained carry, release, and final peg pose.",
+        "passed": passed,
+        "total": len(tasks),
+        "duration_ms": round((time.perf_counter() - started) * 1000, 3),
+        "tasks": tasks,
+        "agent_challenges": list_challenges(),
+    }

{jolly_cli-0.3.0 → jolly_cli-0.4.0}/jolly/challenges.py

@@ -49,6 +49,14 @@ CHALLENGES: dict[str, Challenge] = {
         success="Tool-to-goal error <= 7 cm and no collision.",
         max_commands=10,
     ),
+    "drop-in-hole": Challenge(
+        id="drop-in-hole",
+        name="Drop in hole",
+        scene="insertion",
+        description="Grasp the generated peg and physically release it through the generated opening.",
+        success="The peg settles inside the opening below the rim without robot contact.",
+        max_commands=12,
+    ),
 }
 
 
@@ -97,11 +105,25 @@ def evaluate(challenge_id: str, state: dict[str, object]) -> dict[str, object]:
         inside = bounds["x"][0] <= position[0] <= bounds["x"][1] and bounds["y"][0] <= position[1] <= bounds["y"][1]
         success = inside and position[2] >= bounds["minimum_z"]
         metrics = {"inside_shelf_xy": inside, "cargo_height_meters": position[2]}
-    else:
+    elif challenge_id == "obstacle-reach":
         goal = instance["target"]
         error = math.dist(ee, goal)
         success = error <= 0.07 and not collision
         metrics = {"goal_error_meters": round(error, 6), "collision_free": not collision}
+    else:
+        peg = objects["peg"]["position"]
+        target = instance["target"]
+        error = math.dist(peg[:2], target[:2])
+        rim_height = float(instance["rim_height"])
+        inside = error <= float(instance["hole_radius"]) and peg[2] < rim_height
+        success = inside and state.get("held_object") is None and not collision
+        metrics = {
+            "target_xy_error_meters": round(error, 6),
+            "peg_height_meters": peg[2],
+            "below_rim": peg[2] < rim_height,
+            "released": state.get("held_object") is None,
+            "collision_free": not collision,
+        }
     command_count = int(state.get("challenge_commands", 0))
     within_budget = command_count <= challenge.max_commands
     success = success and within_budget