tabletask 0.1.0__tar.gz → 0.2.1__tar.gz

tabletask-0.2.1/.gitignore

@@ -0,0 +1,19 @@
+# Python
+__pycache__/
+*.pyc
+*.egg-info/
+.venv/
+.ruff_cache/
+.ipynb_checkpoints/
+dist/
+
+# Outputs: pictures, exports, model runs
+build/
+*.mp4
+MUJOCO_LOG.TXT
+
+# Temporary files left by mesh tools
+tetgen-tmpfile*
+
+# Local files that are never published
+.local/

tabletask-0.2.1/PKG-INFO

@@ -0,0 +1,105 @@
+Metadata-Version: 2.5
+Name: tabletask
+Version: 0.2.1
+Summary: Tabletop manipulation tasks as JSON, sampled and graded in MuJoCo
+Project-URL: Homepage, https://table-task.com
+Project-URL: Source, https://github.com/omarrayyann/tabletask
+Author-email: Omar Rayyan <olr7742@nyu.edu>
+License-Expression: MIT
+Keywords: benchmark,manipulation,mujoco,robotics,simulation
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.11
+Requires-Dist: msgspec
+Requires-Dist: mujoco
+Requires-Dist: numpy
+Requires-Dist: pillow
+Requires-Dist: scipy
+Provides-Extra: all
+Requires-Dist: coacd; extra == 'all'
+Requires-Dist: daqp; extra == 'all'
+Requires-Dist: mink; extra == 'all'
+Requires-Dist: trimesh; extra == 'all'
+Provides-Extra: control
+Requires-Dist: daqp; extra == 'control'
+Requires-Dist: mink; extra == 'control'
+Provides-Extra: dev
+Requires-Dist: ruff; extra == 'dev'
+Provides-Extra: importing
+Requires-Dist: coacd; extra == 'importing'
+Requires-Dist: trimesh; extra == 'importing'
+Provides-Extra: rtx
+Requires-Dist: ovrtx; extra == 'rtx'
+Requires-Dist: ovstage; extra == 'rtx'
+Requires-Dist: usd-core; extra == 'rtx'
+Description-Content-Type: text/markdown
+
+## tabletask
+
+A minimal, object-centric format to author tabletop manipulation tasks in simulation, from simple to complex.
+
+![tasks](https://github.com/user-attachments/assets/4aa5a007-2f82-4546-a281-3325b335b4a6)
+
+A task says where things start and what must become true:
+
+```json
+{
+  "id": "robolab-banana_in_bowl",
+  "instruction": "Pick up the banana and place it in the bowl.",
+  "start": [
+    ["on", "banana", "table"],
+    ["on", "bowl", "table"]
+  ],
+  "goal": [
+    ["inside", "banana", "bowl"]
+  ]
+}
+```
+
+![regions: a drawer, a mug handle, a conveyor belt, a crate, the compartments of a tea drawer, the slots of a dish rack, a pouring cup and the pegs of a mug tree](https://github.com/user-attachments/assets/8742a8dd-6889-49b4-808f-4ada43a58ecf)
+
+Objects carry named regions that tasks place things in and check against.
+
+
+### Installation
+
+```sh
+pip install tabletask
+pip install "tabletask[control]"
+pip install "tabletask[rtx]"
+```
+
+
+### Example
+
+```python
+import tabletask
+
+env = tabletask.make("robolab-banana_in_bowl")
+obs, info = env.reset(seed=0)
+
+for _ in range(600):
+    action = [*obs["robot"]["joint_position"], 0.0]
+    obs, reward, terminated, truncated, info = env.step(action)
+    if terminated or truncated:
+        break
+
+print("success:", info["success"])
+env.close()
+```
+
+Several environments at once, each in its own process and renderer:
+
+```python
+envs = tabletask.make("robolab-banana_in_bowl", num_envs=8, gpus=[0])
+observations, infos = envs.reset(episodes=range(8))
+observations, rewards, terminated, truncated, infos = envs.step(actions)
+envs.close()
+```
+
+
+### Acknowledgments
+
+Tasks are ported from [ABC](https://github.com/amazon-far/abc) and
+[RoboLab](https://github.com/NVlabs/RoboLab). The robot models come from
+[MuJoCo Menagerie](https://github.com/google-deepmind/mujoco_menagerie).

tabletask-0.2.1/README.md

@@ -0,0 +1,69 @@
+## tabletask
+
+A minimal, object-centric format to author tabletop manipulation tasks in simulation, from simple to complex.
+
+![tasks](https://github.com/user-attachments/assets/4aa5a007-2f82-4546-a281-3325b335b4a6)
+
+A task says where things start and what must become true:
+
+```json
+{
+  "id": "robolab-banana_in_bowl",
+  "instruction": "Pick up the banana and place it in the bowl.",
+  "start": [
+    ["on", "banana", "table"],
+    ["on", "bowl", "table"]
+  ],
+  "goal": [
+    ["inside", "banana", "bowl"]
+  ]
+}
+```
+
+![regions: a drawer, a mug handle, a conveyor belt, a crate, the compartments of a tea drawer, the slots of a dish rack, a pouring cup and the pegs of a mug tree](https://github.com/user-attachments/assets/8742a8dd-6889-49b4-808f-4ada43a58ecf)
+
+Objects carry named regions that tasks place things in and check against.
+
+
+### Installation
+
+```sh
+pip install tabletask
+pip install "tabletask[control]"
+pip install "tabletask[rtx]"
+```
+
+
+### Example
+
+```python
+import tabletask
+
+env = tabletask.make("robolab-banana_in_bowl")
+obs, info = env.reset(seed=0)
+
+for _ in range(600):
+    action = [*obs["robot"]["joint_position"], 0.0]
+    obs, reward, terminated, truncated, info = env.step(action)
+    if terminated or truncated:
+        break
+
+print("success:", info["success"])
+env.close()
+```
+
+Several environments at once, each in its own process and renderer:
+
+```python
+envs = tabletask.make("robolab-banana_in_bowl", num_envs=8, gpus=[0])
+observations, infos = envs.reset(episodes=range(8))
+observations, rewards, terminated, truncated, infos = envs.step(actions)
+envs.close()
+```
+
+
+### Acknowledgments
+
+Tasks are ported from [ABC](https://github.com/amazon-far/abc) and
+[RoboLab](https://github.com/NVlabs/RoboLab). The robot models come from
+[MuJoCo Menagerie](https://github.com/google-deepmind/mujoco_menagerie).

tabletask-0.2.1/examples/README.md

@@ -0,0 +1,293 @@
+# Examples
+
+Three ways to drive a task. These scripts live next to `tabletask/` and are not part
+of the installed package:
+
+- `agent.py`: a vision-language model through its provider's API.
+- `pi.py`: pi-0.5 through openpi.
+- `teleop.py`: your own hand, through an iOS device.
+
+Run them from the repository root (the directory containing `tabletask/`).
+
+## Vision-language model
+
+Load a task, attach Droid, and let a model control it through camera observations
+and Cartesian motion tools.
+
+### Install
+
+```sh
+python -m pip install -e ".[control]"   # from the repository root
+```
+
+`mink` and `daqp` provide Droid's Cartesian controller. The API clients use only
+Python's standard library, so no provider SDK is needed. On a headless Linux machine:
+
+```sh
+export MUJOCO_GL=egl
+```
+
+### Run
+
+Anthropic:
+
+```sh
+python -m examples.agent robolab-cube_in_the_bowl \
+  --api anthropic --model claude-fable-5-1 --reasoning medium --ask-key
+```
+
+`--ask-key` reads the API key from a hidden prompt. The key stays in the process's
+memory; it is not echoed, added to shell history, or written to the run files.
+
+OpenAI and Gemini:
+
+```sh
+python -m examples.agent robolab-cube_in_the_bowl \
+  --api openai --model gpt-6-astra --reasoning medium --ask-key
+
+python -m examples.agent robolab-cube_in_the_bowl \
+  --api gemini --model gemini-3.8-flash --reasoning medium --ask-key
+```
+
+Model names are passed through unchanged, and your account must have access to the
+model. `--reasoning` is optional; omit it for models without that setting.
+
+Without `--ask-key`, the key is read from the environment:
+
+| `--api` | Environment variable | Endpoint |
+|---|---|---|
+| `anthropic` | `ANTHROPIC_API_KEY` | Messages |
+| `openai` | `OPENAI_API_KEY` | Responses |
+| `gemini` | `GEMINI_API_KEY` | generateContent |
+
+**Stop with Ctrl+C.** The runner stops sending requests, closes the environment,
+and records `stop_reason: "interrupted"`. A request already sent may still complete
+and be billed. HTTP errors stop the run; requests are not retried.
+
+### From Python
+
+With the provider's key in the environment:
+
+```python
+import tabletask
+from tabletask.robots import Droid
+from examples.agent import run
+
+env = tabletask.make("robolab-cube_in_the_bowl", robot=Droid(action_space="cartesian_pose"))
+env.reset(0)
+
+try:
+    result = run(
+        env,
+        api="anthropic",
+        model="claude-fable-5-1",
+        reasoning="medium",
+        max_calls=30,
+    )
+    print(result)
+finally:
+    env.close()
+```
+
+`run()` also takes `api_key=` for a key obtained at runtime and `base_url=` for a
+compatible endpoint. Each call to `run()` starts a new conversation. The caller owns
+the environment and closes it.
+
+### Options
+
+Pass another task id or task file to change the task; `--episode 1` selects another
+seeded start.
+
+| CLI option | Python argument | Default | Meaning |
+|---|---|---|---|
+| `--max-calls` | `max_calls` | `57` | Maximum model requests |
+| `--seconds` | `seconds` | `45` | Maximum simulated motion time, seconds |
+| `--chunk-steps` | `chunk_steps` | whole move | Control steps before the model sees new images |
+| `--image-size` | `image_size` | `224` | Width and height of each camera image, pixels |
+| `--max-tokens` | `max_tokens` | `4096` | Maximum output tokens per response |
+| `--out` | `out` | `build/runs/<timestamp>/` | New directory for results |
+
+Without `--chunk-steps`, each move runs to completion before the next observation.
+`--chunk-steps 8` (0.4 simulated seconds) or `--chunk-steps 16` (0.8 s) stops a move
+early so the model can continue toward the same target or change course. Physics is
+paused while the model responds.
+
+The task's own time limit and `always` conditions also apply. The full conversation,
+including images, reasoning items and thought signatures, is resent on every call.
+Request size grows with each turn, so a call limit is not a cost limit.
+
+### Observations and actions
+
+Each request contains the task instruction, external and wrist RGB images, joint
+positions, gripper closure, the world pose of the grasp point, and the remaining
+budgets. **Object coordinates and goal conditions are not sent.** The model has no
+code, filesystem or scene-query tools.
+
+It has two motion tools:
+
+```python
+move_to(targets={"z": 1.05, "gripper": 0}, note="Move above the object, fingers open.")
+move_by(deltas={"dz": -0.05}, note="Lower 5 cm towards the cube.")
+```
+
+| `move_to` field | Meaning |
+|---|---|
+| `x`, `y`, `z` | Absolute world position, metres; z up |
+| `roll`, `pitch`, `yaw` | Radians, relative to the grasp orientation at the start |
+| `gripper` | **0 open, 1 closed**, thresholded at 0.5 |
+
+`move_by` takes offsets `dx`, `dy`, `dz` (metres) and `droll`, `dpitch`, `dyaw`
+(radians) from the measured pose, plus the same absolute `gripper`. It lets a model
+move in a direction it can see without knowing the world coordinate of the target.
+
+Omitted position and angle fields keep their measured values; an omitted gripper keeps its last command. Orientation is
+`Rz(yaw) Ry(pitch) Rx(roll) R_initial`. The gripper command applies during the
+whole move, so close the gripper in a separate call after arriving.
+
+Motion is interpolated at 0.15 m/s and 0.75 rad/s and tracked by Droid's Mink IK at
+20 Hz. A gripper change takes at least 0.6 s, and every move ends with 0.2 s of
+settling. Moves longer than 10 s are rejected. An IK error stops the move and reports
+how many steps ran; paths are not checked for collisions in advance.
+`chunk_complete` means every planned command was applied, not that the target was
+reached; check the images and measured pose.
+
+`done` and `give_up` end the episode. Success is evaluated by tabletask, not by the
+model calling `done`.
+
+### Results and dry runs
+
+Each run writes:
+
+| File | Contents |
+|---|---|
+| `config.json` | Task, episode, model and settings |
+| `001_external.png`, `001_wrist.png`, ... | Images sent with each call |
+| `calls.jsonl` | Observation, actions, execution feedback, latency and token usage per call |
+| `result.json` | Success, stop reason, call count, simulated time and wall time |
+
+`build/runs/` is gitignored. `--out` must not exist yet, so earlier runs are never
+overwritten. API keys are never written to these files.
+
+To check the cameras and controller **without calling a model**:
+
+```sh
+python -m examples.agent robolab-cube_in_the_bowl --dry-run
+```
+
+The dry run lifts the gripper 2 cm and lowers it again. It does not attempt the task.
+
+See `python -m examples.agent --help` for all options. The code is split into
+`api.py` (provider clients), `tools.py` (observations and motion tools) and
+`agent.py` (episode loop and logging).
+
+## pi-0.5
+
+pi-0.5 is a vision-language-action policy: it takes the two camera images and the
+arm's joint positions and returns joint-position actions at 15 Hz. Install the openpi
+client with `pip install -e <openpi>/packages/openpi-client`, then serve the checkpoint
+from an openpi checkout:
+
+```sh
+uv run scripts/serve_policy.py policy:checkpoint \
+  --policy.config=pi05_droid_jointpos \
+  --policy.dir=gs://openpi-assets/checkpoints/pi05_droid_jointpos
+```
+
+Then play the task:
+
+```sh
+python -m examples.pi robolab-banana_in_the_named_bowl --episodes 20 --out build/pi
+```
+
+Each episode writes frames and a `result.json` to its own directory under `--out`,
+and `summary.json` holds the success rate. `--chunk` is the number of actions
+executed from each inference before querying again; openpi's DROID client uses 8.
+
+Two details the adapter must get right:
+
+- **Actions are offsets from the joint positions sent with the query**, and every
+  action in a chunk is relative to that same pose. They are neither absolute angles
+  nor per-step deltas; treating them as absolute angles drives every joint toward
+  zero. `pi05_droid` (without `jointpos`) differs again: it outputs joint velocities.
+- **Images are resized with openpi's `resize_with_pad`** to 224 x 224 and sent under
+  DROID's key names (exterior camera, then wrist).
+
+## iOS device
+
+Move an iOS device to move the gripper, one episode after another. Poses are streamed
+over your network by [Teledex](https://github.com/omarrayyann/TeleDex) from the
+device's ARKit tracking.
+
+### Install
+
+```sh
+python -m pip install teledex mujoco        # add mink daqp for --robot droid
+```
+
+Install the app from the [App Store](https://apps.apple.com/ae/app/mujoco-ar/id6612039501).
+
+### Run
+
+```sh
+python -m examples.teleop robolab-banana_in_bowl --episode 6
+```
+
+By default the robot is the floating `Gripper`, which has no arm, so every pose is reachable; pass
+`--robot droid` to drive the full arm.
+
+Scan the QR code printed in the terminal with the app to connect. A window shows the
+`dashboard` camera, over the robot's shoulder, with the task instruction across the top. With
+the full arm, only the gripper and the link that carries it are drawn. When the first
+pose arrives, the phone's pose is mapped to the gripper's current pose, so nothing jumps.
+
+Motion is in world axes regardless of how you hold the phone. Forward is the direction
+the phone's camera faced when the app started tracking, so start the app facing the
+same way as the robot (world +x).
+
+| Phone | Robot |
+|---|---|
+| Move the device | The gripper follows, `--scale` metres per metre (default 2.0) |
+| Rotate the device | The gripper rotates with it |
+| Toggle | Fingers: **on closed, off open** |
+| Button A (hold) | **Clutch**: the robot holds still while you reposition the phone |
+| Button B | **Drop this episode** and start the next one, without counting it |
+
+Use the clutch for long reaches: hold the button, bring the phone back to a comfortable
+pose, release, and continue. The phone vibrates when the clutch engages and when an
+episode ends.
+
+When an episode ends (the goal is met, an `always` condition is violated, or time runs
+out), the terminal prints the outcome, the next episode starts (`--episode` 6, then 7,
+8, ...) and the phone is mapped to the gripper again. Success is evaluated by
+tabletask. Close the window or press Ctrl+C to stop; nothing is recorded.
+
+### Options
+
+| Option | Default | Meaning |
+|---|---|---|
+| `--robot` | `gripper` | `gripper` (the floating `Gripper`, no arm) or `droid` (the full arm) |
+| `--episode` | `0` | First episode to play |
+| `--scale` | `2.0` | Metres of gripper motion per metre of phone motion |
+| `--port` | `8888` | Port the phone connects to |
+| `--camera` | `dashboard` | Camera shown: a scene camera (`top`, `preview`) or a robot camera (`wrist`, `external`) |
+| `--width` | `1280` | Window width in pixels |
+| `--no-view` | off | Do not open the window |
+| `--debug` | off | Print every goal check at each start and whenever one changes; draw the goal regions |
+
+### From Python
+
+```python
+import tabletask
+from tabletask.robots import Droid
+from examples.teleop import teleoperate
+
+env = tabletask.make("robolab-banana_in_bowl", robot=Droid(action_space="cartesian_pose"))
+env.reset(6)
+try:
+    solved, played = teleoperate(env, scale=2.0)
+finally:
+    env.close()
+```
+
+`teleoperate` accepts `phone=`: any object with `start`, `anchor_to`, `target`,
+`clutching`, `restarting`, `buzz` and `stop`, so it can run without a device.

{tabletask-0.1.0 → tabletask-0.2.1}/pyproject.toml

@@ -4,12 +4,11 @@ build-backend = "hatchling.build"
 
 [project]
 name = "tabletask"
-version = "0.1.0"
+dynamic = ["version"]
 description = "Tabletop manipulation tasks as JSON, sampled and graded in MuJoCo"
 readme = "README.md"
 requires-python = ">=3.11"
 license = "MIT"
-license-files = ["LICENSE"]
 authors = [{ name = "Omar Rayyan", email = "olr7742@nyu.edu" }]
 keywords = ["robotics", "manipulation", "mujoco", "benchmark", "simulation"]
 classifiers = [
@@ -21,7 +20,9 @@ dependencies = ["msgspec", "mujoco", "numpy", "pillow", "scipy"]
 [project.optional-dependencies]
 control = ["mink", "daqp"]        # Cartesian control
 importing = ["trimesh", "coacd"]  # turning a mesh into an asset
-all = ["mink", "daqp", "trimesh", "coacd"]
+rtx = ["ovrtx", "ovstage", "usd-core"]  # path-traced rendering on an NVIDIA RTX GPU
+all = ["tabletask[control,importing]"]
+dev = ["ruff"]
 
 [project.urls]
 Homepage = "https://table-task.com"
@@ -30,11 +31,23 @@ Source = "https://github.com/omarrayyann/tabletask"
 [project.scripts]
 tabletask = "tabletask.__main__:main"
 
+[tool.hatch.version]
+path = "tabletask/__init__.py"
+
 [tool.hatch.build.targets.wheel]
 packages = ["tabletask"]
 
 [tool.hatch.build.targets.wheel.force-include]
-"tasks" = "tabletask/library/tasks"
+"tasks" = "tabletask/data/tasks"
 
 [tool.hatch.build.targets.sdist]
-include = ["tabletask", "tasks", "README.md", "AGENTS.md", "LICENSE"]
+include = ["tabletask", "tasks", "README.md"]
+
+[tool.ruff]
+line-length = 100
+
+[tool.ruff.lint]
+select = ["E", "W", "F", "I", "B", "UP", "SIM", "C4", "RUF", "PIE", "PERF", "ISC"]
+ignore = [
+  "B905",  # zip() without strict=: the lengths here match by construction
+]

tabletask-0.2.1/tabletask/__init__.py

@@ -0,0 +1,109 @@
+"""Tabletop manipulation tasks written as JSON, sampled and graded in MuJoCo."""
+
+from .env import Env
+from .inspection import preview as _preview
+from .language.errors import SamplingError, UnsupportedError
+from .language.format import dumps, json_schema, load, tasks
+from .language.grammar import words
+from .language.kernel import relation, term
+from .language.schema import Task
+from .sampling import instantiate
+from .vector import Envs
+
+__version__ = "0.2.1"
+
+__all__ = [
+    "Env",
+    "Envs",
+    "SamplingError",
+    "Task",
+    "UnsupportedError",
+    "dumps",
+    "instantiate",
+    "json_schema",
+    "load",
+    "make",
+    "preview",
+    "relation",
+    "tasks",
+    "term",
+    "words",
+]
+
+
+def make(
+    source,
+    *,
+    num_envs=None,
+    robot="droid",
+    gpus=None,
+    renderer=None,
+    reward="success",
+    scene=None,
+    bindings=None,
+    monitor_interval=0.01,
+    backend="mujoco",
+) -> Env | Envs:
+    """Make an environment for a task, or several that run in parallel.
+
+    Args:
+        source: A task id from `tasks()`, a path to a task file, a JSON string, or a dict.
+        num_envs: If given, make this many environments, each in its own process, and
+            return them as one `Envs`.
+        robot: The robot that acts in the scene: "droid", "gripper", a `Robot`, or None.
+        gpus: With `num_envs`, CUDA GPU indices the environments render on, in turn.
+        renderer: None to rasterise images, or "rtx" to path-trace them.
+        reward: "success" for 1.0 on success, or "subtasks" for 1.0 at each new milestone.
+        scene: A scene XML file to use in place of the task's own.
+        bindings: Assets for abstract objects, as {name: {"asset": path, "properties": {...}}}.
+        monitor_interval: How often, in simulated seconds, task conditions are checked.
+        backend: The simulator; only "mujoco" is provided.
+
+    Returns:
+        An `Env`, or with `num_envs` an `Envs`, ready to `reset()`.
+    """
+    if num_envs is not None:
+        return Envs(
+            source,
+            num_envs,
+            robot=robot,
+            gpus=gpus,
+            renderer=renderer,
+            reward=reward,
+            scene=scene,
+            bindings=bindings,
+            monitor_interval=monitor_interval,
+            backend=backend,
+        )
+    if gpus is not None:
+        raise ValueError("gpus go with num_envs")
+    env = Env(load(source), scene, bindings, monitor_interval=monitor_interval, backend=backend)
+    env.renderer = renderer
+    env.reward = reward
+    if isinstance(robot, str):
+        robot = _robot(robot)
+    return env if robot is None else env.add(robot)
+
+
+def _robot(name):
+    from . import robots
+
+    names = {"droid": "Droid", "gripper": "Gripper"}
+    if name not in names:
+        raise ValueError(f"unknown robot {name!r}; choose from {sorted(names)}")
+    return getattr(robots, names[name])()
+
+
+def preview(source, samples=10, out=None, **kwargs):
+    """Render a grid of sampled starts for a task.
+
+    Args:
+        source: A task id, task file, JSON string or dict.
+        samples: How many starts to draw, from episodes 0, 1, ...
+        out: Where to save the picture, if anywhere.
+        **kwargs: `columns`, `renderer="rtx"`, or picture options such as `width`.
+
+    Returns:
+        The grid as a PIL image.
+    """
+    return _preview(load(source), samples, out, **kwargs)