jolly-cli 0.3.0__tar.gz → 0.5.0__tar.gz

{jolly_cli-0.3.0 → jolly_cli-0.5.0}/PKG-INFO

@@ -1,7 +1,7 @@
 Metadata-Version: 2.4
 Name: jolly-cli
-Version: 0.3.0
-Summary: Local, CLI-native robot-arm physics simulation for humans and LLM agents
+Version: 0.5.0
+Summary: CLI-native SO-101 physics tasks and physical hardware control
 Project-URL: Homepage, https://github.com/TeamAudiyo/Jolly
 Project-URL: Repository, https://github.com/TeamAudiyo/Jolly
 Project-URL: Issues, https://github.com/TeamAudiyo/Jolly/issues
@@ -9,7 +9,7 @@ Author: TeamAudiyo
 License: MIT OR Apache-2.0
 License-File: LICENSE-APACHE
 License-File: LICENSE-MIT
-Keywords: cli,kinematics,llm,pybullet,robotics,simulation
+Keywords: cli,hardware,kinematics,llm,pybullet,robotics,so101
 Classifier: Development Status :: 3 - Alpha
 Classifier: Environment :: Console
 Classifier: License :: OSI Approved :: Apache Software License
@@ -27,8 +27,11 @@ Requires-Dist: rich<15,>=13.7
 Provides-Extra: dev
 Requires-Dist: build<2,>=1.2; extra == 'dev'
 Requires-Dist: httpx<1,>=0.27; extra == 'dev'
+Requires-Dist: pyserial<4,>=3.5; extra == 'dev'
 Requires-Dist: pytest-cov<7,>=5; extra == 'dev'
 Requires-Dist: pytest<9,>=8.2; extra == 'dev'
+Provides-Extra: hardware
+Requires-Dist: pyserial<4,>=3.5; extra == 'hardware'
 Provides-Extra: web
 Requires-Dist: fastapi<1,>=0.115; extra == 'web'
 Requires-Dist: uvicorn<1,>=0.30; extra == 'web'
@@ -36,8 +39,15 @@ Description-Content-Type: text/markdown
 
 # Jolly CLI
 
-Jolly is a local robot-arm physics simulator for terminal users and LLM agents.
-It uses PyBullet in deterministic direct mode and saves state between commands.
+Jolly provides two separate robot-control backends for terminal users and LLM
+agents:
+
+- `JollyEngine` executes explicit operator controls with the official SO-101
+  model in PyBullet, then measures the final physical state.
+- `SO101HardwareDriver` sends the real STS3215 serial protocol to a physical
+  SO-101 and reads measured motor positions.
+
+The two backends never combine their results.
 
 Jolly includes two offline robot profiles:
 
@@ -48,20 +58,30 @@ The package also accepts scenes, challenges, an engine benchmark, a native
 PyBullet viewer, and an optional local web viewer. No cloud service, MCP server,
 API key, or network connection is required at runtime.
 
-## See Jolly in action
+## See Jolly 0.5.0 in action
+
+### Interactive SO-101 simulator
+
+![Jolly web viewer executing a Cartesian reach in the insertion scene](docs/assets/jolly-web-demo.gif)
 
-### Local web simulator
+[Watch the full WebM recording](docs/assets/jolly-web-demo.webm) ·
+[Open the full-resolution still](docs/assets/jolly-so101-viewer.png)
 
-![Jolly web viewer rendering the official SO-101 CAD model](docs/assets/jolly-so101-viewer.png)
+The recording uses the official SO-101 CAD meshes in a real PyBullet world. It
+executes a Cartesian reach in the insertion scene. The viewport is not a
+diagram or a hand-drawn robot substitute.
 
-The viewport above is a real PyBullet render of the official SO-101 CAD meshes.
-It is not a diagram or a hand-drawn robot substitute.
+### Operator-controlled randomized pick-and-place
 
-### Terminal-native control
+![Jolly 0.5.0 terminal demo showing explicit pick-and-place controls](docs/assets/jolly-terminal-pick-place-v050.gif)
 
-![Jolly terminal CLI demo](docs/assets/jolly-terminal-demo.gif)
+[Watch the MP4 recording](docs/assets/jolly-terminal-pick-place-v050.mp4)
 
-[Watch the MP4 recording](docs/assets/jolly-terminal-demo.mp4)
+The terminal recording shows every control: move above the peg, move down,
+close the gripper, lift, move sideways, lower, and release. `benchmark score`
+does not move the robot and does not generate a numeric score. It reports PASS
+only from the final measured peg pose, release state, collision state, and
+command budget.
 
 ## Install
 
@@ -99,7 +119,9 @@ jolly move --joints "0,-20,40,-20,0" --gripper 0.0 --json
 jolly reach --x 0.30 --y 0.10 --z 0.18 --gripper 0.0 --json
 jolly render
 jolly challenge list --json
-jolly benchmark --json
+jolly benchmark start --seed 42017 --model so101 --json
+# Issue explicit jolly reach controls using the generated peg and hole coordinates.
+jolly benchmark score --json
 ```
 
 Start the optional local web viewer:
@@ -140,7 +162,8 @@ Important commands:
 | `jolly reset` | Select a model and scene, then home the robot. |
 | `jolly scene list/load` | Discover and load deterministic scenes. |
 | `jolly challenge list/start/status` | Run seeded randomized manipulation challenges. |
-| `jolly benchmark [--seed N] [--cases N]` | Run randomized engine checks with reproducible inputs. |
+| `jolly benchmark start/score` | Generate a randomized pick-and-place task, accept explicit controls, then measure PASS or FAIL. |
+| `jolly hardware state/move/benchmark/stop` | Control and measure a physical SO-101 over its serial bus. |
 
 Jolly rejects joint-limit violations and unsafe workspace targets. By default,
 Jolly rolls back a motion that creates a collision. Use `--allow-collision`
@@ -184,15 +207,15 @@ jolly-cli/
 
 ## Physics scope
 
-`JollyEngine` owns state, safety, motion rollback, grasping, rendering, and the
-public simulator contract. `JollyDriver` generates seeded challenge and
-benchmark cases. The project does not depend on Inspect, Inspect AI, or an
-external robotics benchmark driver.
+`JollyEngine` owns state, safety, motion rollback, constrained grasping,
+rendering, and the physics-task contract. `JollyDriver` generates seeded
+challenge and benchmark cases. The project does not depend on Inspect, Inspect
+AI, or an external robotics benchmark driver.
 
 PyBullet is the low-level open-source physics backend. It performs rigid-body
 simulation, FK, IK, contact generation, and scene settling. Jolly uses
 deterministic state restoration across short CLI processes.
-The grasp helper attaches a nearby graspable object while the gripper is closed.
+The grasp helper attaches one contacted graspable object while the gripper is closed.
 This helper makes terminal pick-and-place repeatable. It is not a soft-contact or
 motor-current model.
 
@@ -204,12 +227,53 @@ The result includes the seed and full instance so results stay auditable. Omit
 
 ```bash
 jolly challenge start sort-red --seed 42017 --json
-jolly benchmark --seed 42017 --cases 5 --json
+jolly benchmark start --seed 42017 --model so101 --json
+# Read the generated instance, then issue each reach control yourself.
+jolly benchmark score --json
+```
+
+Benchmark start generates the randomized peg and hole but executes no motion.
+Only explicit `jolly reach` or `jolly move` commands can change the attempt.
+Benchmark score executes no motion and returns no numeric score. PASS requires
+the released peg to settle inside the generated hole without collision and
+within the command budget. The JSON result includes every requested control and
+its measured tool, gripper, grasp, and collision state.
+
+## Physical SO-101 hardware
+
+Install direct physical-hardware support:
+
+```bash
+python -m pip install 'jolly-cli[hardware]'
+```
+
+Jolly implements the Feetech STS3215 packet protocol directly. It does not use
+Inspect Robots or an external robot SDK. Copy and calibrate the example file
+before enabling torque:
+
+```bash
+jolly hardware calibration-example --output so101-calibration.json
+# Replace every placeholder with measurements from this physical arm.
+jolly hardware scan --port /dev/ttyACM0 --calibration so101-calibration.json --json
+jolly hardware state --port /dev/ttyACM0 --calibration so101-calibration.json --json
+jolly hardware move --port /dev/ttyACM0 --calibration so101-calibration.json \
+  --joints '0,0,0,0,0' --gripper 0 --confirm-hardware --json
+jolly hardware benchmark --port /dev/ttyACM0 --calibration so101-calibration.json \
+  --seed 42017 --cases 3 --confirm-hardware --json
+jolly hardware stop --port /dev/ttyACM0 --calibration so101-calibration.json
 ```
 
-The benchmark generates new joint configurations for every robot case and new
-object layouts for every scene case. This prevents a policy from passing only by
-memorizing the original fixed coordinates.
+The example calibration values are placeholders and set `calibrated` to false.
+Measure every zero point, direction, and gripper endpoint before setting it to
+true. Physical commands require an explicit confirmation, enforce degree and
+raw-count step limits, set bounded goal speed, issue simultaneous goals, and
+read motor feedback. The driver disables torque after communication failures.
+After successful movement it keeps torque enabled so the arm does not fall.
+Support the arm before running `jolly hardware stop`.
+
+The physical benchmark scores measured joint-position error only. The PyBullet
+operator benchmark returns a measured PASS or FAIL without a numeric score.
+Jolly never presents physics output as a physical-hardware result.
 
 The bundled SO-101 model is the official Apache-2.0 new-calibration URDF from
 The Robot Studio. Jolly includes the 13 referenced STL meshes from pinned commit

{jolly_cli-0.3.0 → jolly_cli-0.5.0}/README.md

@@ -1,7 +1,14 @@
 # Jolly CLI
 
-Jolly is a local robot-arm physics simulator for terminal users and LLM agents.
-It uses PyBullet in deterministic direct mode and saves state between commands.
+Jolly provides two separate robot-control backends for terminal users and LLM
+agents:
+
+- `JollyEngine` executes explicit operator controls with the official SO-101
+  model in PyBullet, then measures the final physical state.
+- `SO101HardwareDriver` sends the real STS3215 serial protocol to a physical
+  SO-101 and reads measured motor positions.
+
+The two backends never combine their results.
 
 Jolly includes two offline robot profiles:
 
@@ -12,20 +19,30 @@ The package also accepts scenes, challenges, an engine benchmark, a native
 PyBullet viewer, and an optional local web viewer. No cloud service, MCP server,
 API key, or network connection is required at runtime.
 
-## See Jolly in action
+## See Jolly 0.5.0 in action
+
+### Interactive SO-101 simulator
+
+![Jolly web viewer executing a Cartesian reach in the insertion scene](docs/assets/jolly-web-demo.gif)
 
-### Local web simulator
+[Watch the full WebM recording](docs/assets/jolly-web-demo.webm) ·
+[Open the full-resolution still](docs/assets/jolly-so101-viewer.png)
 
-![Jolly web viewer rendering the official SO-101 CAD model](docs/assets/jolly-so101-viewer.png)
+The recording uses the official SO-101 CAD meshes in a real PyBullet world. It
+executes a Cartesian reach in the insertion scene. The viewport is not a
+diagram or a hand-drawn robot substitute.
 
-The viewport above is a real PyBullet render of the official SO-101 CAD meshes.
-It is not a diagram or a hand-drawn robot substitute.
+### Operator-controlled randomized pick-and-place
 
-### Terminal-native control
+![Jolly 0.5.0 terminal demo showing explicit pick-and-place controls](docs/assets/jolly-terminal-pick-place-v050.gif)
 
-![Jolly terminal CLI demo](docs/assets/jolly-terminal-demo.gif)
+[Watch the MP4 recording](docs/assets/jolly-terminal-pick-place-v050.mp4)
 
-[Watch the MP4 recording](docs/assets/jolly-terminal-demo.mp4)
+The terminal recording shows every control: move above the peg, move down,
+close the gripper, lift, move sideways, lower, and release. `benchmark score`
+does not move the robot and does not generate a numeric score. It reports PASS
+only from the final measured peg pose, release state, collision state, and
+command budget.
 
 ## Install
 
@@ -63,7 +80,9 @@ jolly move --joints "0,-20,40,-20,0" --gripper 0.0 --json
 jolly reach --x 0.30 --y 0.10 --z 0.18 --gripper 0.0 --json
 jolly render
 jolly challenge list --json
-jolly benchmark --json
+jolly benchmark start --seed 42017 --model so101 --json
+# Issue explicit jolly reach controls using the generated peg and hole coordinates.
+jolly benchmark score --json
 ```
 
 Start the optional local web viewer:
@@ -104,7 +123,8 @@ Important commands:
 | `jolly reset` | Select a model and scene, then home the robot. |
 | `jolly scene list/load` | Discover and load deterministic scenes. |
 | `jolly challenge list/start/status` | Run seeded randomized manipulation challenges. |
-| `jolly benchmark [--seed N] [--cases N]` | Run randomized engine checks with reproducible inputs. |
+| `jolly benchmark start/score` | Generate a randomized pick-and-place task, accept explicit controls, then measure PASS or FAIL. |
+| `jolly hardware state/move/benchmark/stop` | Control and measure a physical SO-101 over its serial bus. |
 
 Jolly rejects joint-limit violations and unsafe workspace targets. By default,
 Jolly rolls back a motion that creates a collision. Use `--allow-collision`
@@ -148,15 +168,15 @@ jolly-cli/
 
 ## Physics scope
 
-`JollyEngine` owns state, safety, motion rollback, grasping, rendering, and the
-public simulator contract. `JollyDriver` generates seeded challenge and
-benchmark cases. The project does not depend on Inspect, Inspect AI, or an
-external robotics benchmark driver.
+`JollyEngine` owns state, safety, motion rollback, constrained grasping,
+rendering, and the physics-task contract. `JollyDriver` generates seeded
+challenge and benchmark cases. The project does not depend on Inspect, Inspect
+AI, or an external robotics benchmark driver.
 
 PyBullet is the low-level open-source physics backend. It performs rigid-body
 simulation, FK, IK, contact generation, and scene settling. Jolly uses
 deterministic state restoration across short CLI processes.
-The grasp helper attaches a nearby graspable object while the gripper is closed.
+The grasp helper attaches one contacted graspable object while the gripper is closed.
 This helper makes terminal pick-and-place repeatable. It is not a soft-contact or
 motor-current model.
 
@@ -168,12 +188,53 @@ The result includes the seed and full instance so results stay auditable. Omit
 
 ```bash
 jolly challenge start sort-red --seed 42017 --json
-jolly benchmark --seed 42017 --cases 5 --json
+jolly benchmark start --seed 42017 --model so101 --json
+# Read the generated instance, then issue each reach control yourself.
+jolly benchmark score --json
+```
+
+Benchmark start generates the randomized peg and hole but executes no motion.
+Only explicit `jolly reach` or `jolly move` commands can change the attempt.
+Benchmark score executes no motion and returns no numeric score. PASS requires
+the released peg to settle inside the generated hole without collision and
+within the command budget. The JSON result includes every requested control and
+its measured tool, gripper, grasp, and collision state.
+
+## Physical SO-101 hardware
+
+Install direct physical-hardware support:
+
+```bash
+python -m pip install 'jolly-cli[hardware]'
+```
+
+Jolly implements the Feetech STS3215 packet protocol directly. It does not use
+Inspect Robots or an external robot SDK. Copy and calibrate the example file
+before enabling torque:
+
+```bash
+jolly hardware calibration-example --output so101-calibration.json
+# Replace every placeholder with measurements from this physical arm.
+jolly hardware scan --port /dev/ttyACM0 --calibration so101-calibration.json --json
+jolly hardware state --port /dev/ttyACM0 --calibration so101-calibration.json --json
+jolly hardware move --port /dev/ttyACM0 --calibration so101-calibration.json \
+  --joints '0,0,0,0,0' --gripper 0 --confirm-hardware --json
+jolly hardware benchmark --port /dev/ttyACM0 --calibration so101-calibration.json \
+  --seed 42017 --cases 3 --confirm-hardware --json
+jolly hardware stop --port /dev/ttyACM0 --calibration so101-calibration.json
 ```
 
-The benchmark generates new joint configurations for every robot case and new
-object layouts for every scene case. This prevents a policy from passing only by
-memorizing the original fixed coordinates.
+The example calibration values are placeholders and set `calibrated` to false.
+Measure every zero point, direction, and gripper endpoint before setting it to
+true. Physical commands require an explicit confirmation, enforce degree and
+raw-count step limits, set bounded goal speed, issue simultaneous goals, and
+read motor feedback. The driver disables torque after communication failures.
+After successful movement it keeps torque enabled so the arm does not fall.
+Support the arm before running `jolly hardware stop`.
+
+The physical benchmark scores measured joint-position error only. The PyBullet
+operator benchmark returns a measured PASS or FAIL without a numeric score.
+Jolly never presents physics output as a physical-hardware result.
 
 The bundled SO-101 model is the official Apache-2.0 new-calibration URDF from
 The Robot Studio. Jolly includes the 13 referenced STL meshes from pinned commit

jolly_cli-0.5.0/docs/hardware/so101-calibration.example.json

@@ -0,0 +1,10 @@
+{
+  "motors": {
+    "shoulder_pan": {"id": 1, "zero_raw": 2048, "direction": 1, "raw_per_degree": 11.3777777778},
+    "shoulder_lift": {"id": 2, "zero_raw": 2048, "direction": 1, "raw_per_degree": 11.3777777778},
+    "elbow_flex": {"id": 3, "zero_raw": 2048, "direction": 1, "raw_per_degree": 11.3777777778},
+    "wrist_flex": {"id": 4, "zero_raw": 2048, "direction": 1, "raw_per_degree": 11.3777777778},
+    "wrist_roll": {"id": 5, "zero_raw": 2048, "direction": 1, "raw_per_degree": 11.3777777778},
+    "gripper": {"id": 6, "open_raw": 2048, "closed_raw": 3072}
+  }
+}

{jolly_cli-0.3.0 → jolly_cli-0.5.0}/docs/wiki/CLI-Reference.md

@@ -23,12 +23,30 @@ jolly scene load blocks --json
 jolly challenge list --json
 jolly challenge start sort-red --model jolly6 [--seed N] --json
 jolly challenge status --json
-jolly benchmark [--seed N] [--cases 3] --json
+jolly benchmark start [--seed N] [--model so101] --json
+# Issue explicit jolly reach or jolly move controls.
+jolly benchmark score --json
 ```
 
 Challenge starts use a fresh random seed by default. The JSON response contains
 the seed and generated instance. Supply `--seed N` to replay the exact case.
-The benchmark randomizes every robot and scene check through `JollyDriver`.
+Benchmark start generates a randomized drop-in-hole task through `JollyDriver`
+without moving the robot. Benchmark score measures only the state produced by
+the intervening explicit controls and returns PASS or FAIL without a numeric
+score.
+
+## Physical SO-101
+
+```bash
+jolly hardware scan --port /dev/ttyACM0 --calibration calibration.json --json
+jolly hardware state --port /dev/ttyACM0 --calibration calibration.json --json
+jolly hardware move --port /dev/ttyACM0 --calibration calibration.json --joints "0,0,0,0,0" --gripper 0 --confirm-hardware --json
+jolly hardware benchmark --port /dev/ttyACM0 --calibration calibration.json --seed 42 --cases 3 --confirm-hardware --json
+jolly hardware stop --port /dev/ttyACM0 --calibration calibration.json
+```
+
+PyBullet and physical-hardware results use separate benchmark names. The
+PyBullet operator benchmark has no numeric score.
 
 ## Viewers
 

jolly_cli-0.5.0/docs/wiki/Challenges-and-Benchmarks.md

@@ -0,0 +1,36 @@
+# Challenges and Benchmarks
+
+Jolly includes local scene templates:
+
+- `empty`: clear workspace for kinematics.
+- `blocks`: three blocks and two target pads.
+- `shelf`: floor-to-shelf manipulation.
+- `obstacles`: collision-aware reaching.
+
+Agent challenges include `reach-center`, `sort-red`, `shelf-load`, and
+`obstacle-reach`. `JollyDriver` generates a new target and scene layout for each
+start. Each challenge has a command budget and explicit success metrics.
+
+```bash
+jolly challenge start sort-red --model jolly6 --json
+# Run move and reach commands.
+jolly challenge status --json
+```
+
+The start result contains a generated seed and instance. Use `--seed` to replay
+the exact case.
+
+`jolly benchmark start --json` generates a randomized peg and hole but does not
+move the robot. The operator must issue each `jolly reach` or `jolly move`
+control to approach, descend, grasp, lift, move sideways, lower, and release.
+
+`jolly benchmark score --json` does not move the robot and returns no numeric
+score. It returns PASS only when the released peg settles inside the generated
+hole without collision and within the command budget. The result includes the
+full explicit control history and measured final state.
+
+Jolly uses its own `JollyEngine`, `JollyDriver`, and direct
+`SO101HardwareDriver`. It does not use Inspect, Inspect AI, or an external robot
+benchmark harness. PyBullet remains the local open-source rigid-body backend
+for physics tasks. Physical SO-101 results come only from STS3215 motor feedback
+and remain separate from the PyBullet PASS or FAIL result.

jolly_cli-0.5.0/docs/wiki/Physical-SO-101-Hardware.md

@@ -0,0 +1,35 @@
+# Physical SO-101 Hardware
+
+Install the serial dependency with `pip install 'jolly-cli[hardware]'`.
+
+Jolly's `SO101HardwareDriver` sends Feetech STS3215 packets directly at the
+SO-101 default 1,000,000 baud. The configured motor IDs default to the common
+SO-101 sequence of 1 through 6. Each physical arm requires measured calibration.
+
+Generate a template, then replace all placeholder zero points, directions, and
+gripper endpoints before movement.
+
+```bash
+jolly hardware calibration-example --output calibration.json
+jolly hardware scan --port /dev/ttyACM0 --calibration calibration.json --json
+jolly hardware state --port /dev/ttyACM0 --calibration calibration.json --json
+jolly hardware move --port /dev/ttyACM0 --calibration calibration.json \
+  --joints '0,0,0,0,0' --gripper 0 --confirm-hardware --json
+jolly hardware benchmark --port /dev/ttyACM0 --calibration calibration.json \
+  --seed 42017 --cases 3 --confirm-hardware --json
+jolly hardware stop --port /dev/ttyACM0 --calibration calibration.json
+```
+
+The physical benchmark makes bounded movements of at most five degrees from the
+measured starting pose. It scores measured motor-position error and returns to
+the starting pose. Torque stays enabled to hold that pose. Support the arm
+before `jolly hardware stop`. Communication failures trigger a best-effort
+torque disable. Clear the workspace and keep an emergency power cutoff nearby.
+
+Physical hardware scores and PyBullet operator PASS or FAIL results are separate. Jolly
+never labels a physics result as a physical-hardware result.
+
+Protocol and hardware references:
+
+- https://huggingface.co/docs/lerobot/en/so101
+- https://pages.switch-science.com/comparison/files/feetech/serial-sts/STS3215_datasheet.pdf

{jolly_cli-0.3.0 → jolly_cli-0.5.0}/docs/wiki/Web-Viewer.md

@@ -1,5 +1,10 @@
 # Web Viewer
 
+![Jolly web viewer executing a Cartesian reach in the insertion scene](../assets/jolly-web-demo.gif)
+
+[Watch the full WebM recording](../assets/jolly-web-demo.webm) or
+[open the full-resolution still](../assets/jolly-so101-viewer.png).
+
 Start the local viewer:
 
 ```bash
@@ -11,7 +16,7 @@ Open `http://127.0.0.1:8765`. API documentation is available at
 
 The viewer provides:
 
-- An interactive projected 3D arm scene.
+- An interactive PyBullet render of the official SO-101 CAD meshes.
 - Robot and scene selectors.
 - Joint positions and limits.
 - End-effector pose.
@@ -19,5 +24,8 @@ The viewer provides:
 - Scene objects.
 - A safe local command console for state, reset, move, and reach.
 
+The recording executes a Cartesian reach in the insertion scene through the
+local command console.
+
 The API writes the same persistent state as the CLI. The default bind is
 loopback-only. Do not use `--unsafe-public` on an untrusted network.

{jolly_cli-0.3.0 → jolly_cli-0.5.0}/docs/wiki/_Sidebar.md

@@ -5,5 +5,6 @@
 - [CLI Reference](CLI-Reference)
 - [Robot Models and Licensing](Robot-Models-and-Licensing)
 - [Challenges and Benchmarks](Challenges-and-Benchmarks)
+- [Physical SO-101 Hardware](Physical-SO-101-Hardware)
 - [LLM Agent Safety](LLM-Agent-Safety)
 - [Web Viewer](Web-Viewer)

{jolly_cli-0.3.0 → jolly_cli-0.5.0}/jolly/__init__.py

@@ -1,4 +1,4 @@
 """Jolly robot-arm simulator."""
 
 __all__ = ["__version__"]
-__version__ = "0.3.0"
+__version__ = "0.5.0"

jolly_cli-0.5.0/jolly/benchmark.py

@@ -0,0 +1,85 @@
+from __future__ import annotations
+
+from typing import Any
+
+from jolly.challenges import evaluate
+from jolly.core.errors import ConfigurationError
+
+BENCHMARK_ID = "jolly-operator-pick-place-v5"
+BENCHMARK_CHALLENGE = "drop-in-hole"
+BENCHMARK_METADATA_KEYS = (
+    "active_challenge",
+    "challenge_seed",
+    "challenge_instance",
+    "active_benchmark",
+    "benchmark_controls",
+)
+
+CONTROL_WORKFLOW = [
+    "Inspect the generated peg and hole coordinates in the start result or with 'jolly state --json'.",
+    "Move above the peg with 'jolly reach' and an open gripper.",
+    "Move down until the gripper contacts the peg, then close the gripper.",
+    "Lift the grasped peg.",
+    "Move the peg horizontally above the generated hole.",
+    "Lower the peg into the opening and release it.",
+    "Run 'jolly benchmark score' to measure the final physical state.",
+]
+
+
+def benchmark_control_history(
+    previous: dict[str, object],
+    *,
+    source: str,
+    command: str,
+    requested: dict[str, object],
+    result: dict[str, object],
+) -> list[object] | None:
+    """Return the control history with one measured explicit motion appended."""
+    if previous.get("active_benchmark") != BENCHMARK_ID:
+        return None
+    existing = previous.get("benchmark_controls", [])
+    controls = list(existing) if isinstance(existing, list) else []
+    controls.append(
+        {
+            "index": len(controls) + 1,
+            "source": source,
+            "command": command,
+            "requested": requested,
+            "measured": {
+                "tool_position": result["end_effector"]["position"],
+                "gripper": result["gripper"],
+                "held_object": result["held_object"],
+                "collision": result["collisions"]["collision"],
+            },
+        }
+    )
+    return controls
+
+
+def score_operator_benchmark(state: dict[str, Any]) -> dict[str, object]:
+    """Measure an operator-controlled pick-and-place attempt without moving the robot."""
+    if state.get("active_benchmark") != BENCHMARK_ID:
+        raise ConfigurationError("No operator benchmark is active. Run 'jolly benchmark start'.")
+    if state.get("active_challenge") != BENCHMARK_CHALLENGE:
+        raise ConfigurationError("The active benchmark is not the drop-in-hole pick-and-place task.")
+    controls = state.get("benchmark_controls")
+    if not isinstance(controls, list):
+        raise ConfigurationError("The active benchmark has no control history. Start it again.")
+    evaluation = evaluate(BENCHMARK_CHALLENGE, state)
+    success = bool(evaluation["success"])
+    return {
+        "ok": True,
+        "benchmark": BENCHMARK_ID,
+        "engine": "JollyEngine",
+        "physics_backend": "PyBullet",
+        "mode": "operator-controlled",
+        "seed": evaluation["seed"],
+        "outcome": "PASS" if success else "FAIL",
+        "success": success,
+        "control_count": len(controls),
+        "controls": controls,
+        "instance": evaluation["instance"],
+        "measurements": evaluation["metrics"],
+        "score": None,
+        "score_basis": "No synthetic numeric score. PASS requires the released peg to settle inside the generated hole without collision and within the command budget.",
+    }

{jolly_cli-0.3.0 → jolly_cli-0.5.0}/jolly/challenges.py

@@ -49,6 +49,14 @@ CHALLENGES: dict[str, Challenge] = {
         success="Tool-to-goal error <= 7 cm and no collision.",
         max_commands=10,
     ),
+    "drop-in-hole": Challenge(
+        id="drop-in-hole",
+        name="Drop in hole",
+        scene="insertion",
+        description="Grasp the generated peg and physically release it through the generated opening.",
+        success="The peg settles inside the opening below the rim without robot contact.",
+        max_commands=12,
+    ),
 }
 
 
@@ -97,11 +105,25 @@ def evaluate(challenge_id: str, state: dict[str, object]) -> dict[str, object]:
         inside = bounds["x"][0] <= position[0] <= bounds["x"][1] and bounds["y"][0] <= position[1] <= bounds["y"][1]
         success = inside and position[2] >= bounds["minimum_z"]
         metrics = {"inside_shelf_xy": inside, "cargo_height_meters": position[2]}
-    else:
+    elif challenge_id == "obstacle-reach":
         goal = instance["target"]
         error = math.dist(ee, goal)
         success = error <= 0.07 and not collision
         metrics = {"goal_error_meters": round(error, 6), "collision_free": not collision}
+    else:
+        peg = objects["peg"]["position"]
+        target = instance["target"]
+        error = math.dist(peg[:2], target[:2])
+        rim_height = float(instance["rim_height"])
+        inside = error <= float(instance["hole_radius"]) and peg[2] < rim_height
+        success = inside and state.get("held_object") is None and not collision
+        metrics = {
+            "target_xy_error_meters": round(error, 6),
+            "peg_height_meters": peg[2],
+            "below_rim": peg[2] < rim_height,
+            "released": state.get("held_object") is None,
+            "collision_free": not collision,
+        }
     command_count = int(state.get("challenge_commands", 0))
     within_budget = command_count <= challenge.max_commands
     success = success and within_budget