commandagi 0.3.0__tar.gz

commandagi-0.3.0/.gitignore

@@ -0,0 +1,65 @@
+# deps
+node_modules/
+.pnpm-store/
+
+# build
+dist/
+.next/
+out/
+.turbo/
+.vercel/
+*.tsbuildinfo
+
+# env
+.env
+.env.*
+!.env.example
+!*.example
+.dev.vars
+.wrangler/
+
+# secrets that travel with the .env cascade but are never committed
+# (e.g. the Sign in with Apple .p8 private key — see secrets/README or lib/auth.ts)
+secrets/
+*.p8
+
+# misc
+.DS_Store
+*.log
+coverage/
+.idea/
+.vscode/*
+!.vscode/extensions.json
+
+# OpenNext build output
+.open-next/
+
+# editor/python noise
+*.swp
+__pycache__/
+
+# Claude Code scheduler runtime lock (machine-local, never commit)
+scheduled_tasks.lock
+.claude/scheduled_tasks.lock
+.old-env/
+
+# local prod backups (email/calendar migration 2026-07-03) — never commit
+/backup/
+
+# research scratch — machine-local instance-id markers, peek scripts, env captures
+/.*_iid
+/peek.sh
+# ARC-AGI-3 game downloads. These belong to the model service and live in
+# services/model/environment_files/ (ignored there too) — a copy at the REPO ROOT only ever appears
+# when the harness is run from the wrong cwd, and one did: a stray tu93/ sat here duplicating the
+# submodule's copy, differing solely in date_downloaded. Deleted 2026-08-13; the rule stays so the
+# next wrong-cwd run cannot quietly repopulate the index.
+/environment_files/
+
+# staged asset-library manifests — regenerable output of scripts/stage-{editor,public}-library.mjs
+scratch/editor-library/
+scratch/public-library/
+
+# Stager RUN ARTIFACTS — `--results` output, one per invocation. Not source: a machine that ran
+# the stager should not be able to commit its log by accident.
+*.results.json

commandagi-0.3.0/PKG-INFO

@@ -0,0 +1,159 @@
+Metadata-Version: 2.5
+Name: commandagi
+Version: 0.3.0
+Summary: Python SDK for CommandAGI — launch cloud computers and 3D robot simulations and control them programmatically.
+Project-URL: Homepage, https://commandagi.com
+Project-URL: Documentation, https://github.com/CommandAGI/commandagi/blob/main/docs/ROBOT_DEVELOPER_API.md
+Author: CommandAGI
+License: MIT
+Keywords: agents,computer-use,physics,reinforcement-learning,robotics,simulation
+Requires-Python: >=3.9
+Requires-Dist: requests>=2.28
+Requires-Dist: websocket-client>=1.5
+Provides-Extra: gym
+Requires-Dist: gymnasium>=0.29; extra == 'gym'
+Requires-Dist: numpy>=1.23; extra == 'gym'
+Requires-Dist: pillow>=9.0; extra == 'gym'
+Provides-Extra: lerobot
+Requires-Dist: numpy>=1.23; extra == 'lerobot'
+Requires-Dist: pillow>=9.0; extra == 'lerobot'
+Requires-Dist: pyarrow>=14.0; extra == 'lerobot'
+Provides-Extra: vision
+Requires-Dist: numpy>=1.23; extra == 'vision'
+Requires-Dist: pillow>=9.0; extra == 'vision'
+Description-Content-Type: text/markdown
+
+# CommandAGI Python SDK
+
+Launch real cloud **computers** and **3D robot simulations** and control them from Python — stream
+the robot's camera, send actions, run episodes. No agent required: you drive.
+
+```bash
+pip install commandagi          # + `pip install commandagi[vision]` for numpy frames
+```
+
+## Robot testing in a 3D world
+
+```python
+from commandagi import CommandAGI
+
+cagi = CommandAGI(api_key="cagi_...")          # or set COMMANDAGI_API_KEY
+
+with cagi.launch("simulation/warehouse") as world:
+    obs = world.observe()                      # JPEG bytes from the robot's head camera
+    for _ in range(20):
+        obs = world.step("turn", dir="left")   # act, then get the next frame
+    world.reset()                              # robot back to the episode start
+# leaving the block stops the world and releases the cloud VM
+```
+
+`launch()` provisions a real GCE VM running a 3D physics world, waits until it's streaming, and gives
+you a `World`. Built-in scenes: `simulation/warehouse`, `simulation/house-on-fire`,
+`simulation/school` (a mobile robot in each).
+
+### The control vocabulary
+
+| World kind  | actions                                                                   |
+| ----------- | ------------------------------------------------------------------------- |
+| robot / sim | `move(speed)`, `back(speed)`, `turn(dir, rate)`, `stop`, `reset`          |
+| computer    | `click(x, y)`, `type(text)`, `key(key)`, `move(x, y)`, `scroll(x, y, dy)` |
+
+```python
+world.act("move", speed=0.8)        # fire-and-forget
+obs = world.step("move", speed=0.8) # act + return the next observation (settles 0.8s)
+obs = world.observe(fresh=True)     # wait for a frame newer than now
+arr = world.observe_array()         # HxWx3 uint8 numpy (needs commandagi[vision])
+for frame in world.stream():        # live generator of frames
+    ...
+```
+
+## Simulator instances (morphology-agnostic robots)
+
+The simulator is **morphology-agnostic**: a robot is just a set of named actuators and sites, driven
+by one small **generic** control vocabulary — no `drive`/`gripper`, just `ctrl` / `actuator` / `ik`
+/ `trajectory` / `describe`. Spin up your own instance, choose who can watch or add robots, and
+populate it with one or many robots on a single session.
+
+```python
+from commandagi import CommandAGI
+
+cagi = CommandAGI(api_key="cagi_...")
+
+sim = cagi.launch_sim(scene="the-matrix", visibility="private", title="demo")
+print("instance:", sim.id, "session:", sim.session_id)
+
+# Who can do what:
+sim.grant("user_teammate", capability="viewer")    # may watch the stream
+sim.grant("user_buddy",    capability="operator")  # may also launch robots into the world
+
+# Add robots (each becomes a embodiment on sim.session_id):
+rover = sim.join_robot(kind="rover")   # -> {robotId, embodimentId, sessionId}
+arm   = sim.join_robot(kind="arm")
+
+cagi.sims()             # list instances visible to you
+cagi.get_sim(sim.id)    # rehydrate a SimInstance
+sim.view()              # instance metadata + attached embodiments
+sim.stop()              # release it (or use `with cagi.launch_sim(...) as sim:`)
+```
+
+### Generic robot control
+
+`World` exposes the morphology-agnostic vocabulary (address a specific robot in a multi-robot embodiment
+with `robot_id`):
+
+```python
+world = cagi.connect_world(sim.session_id, rover["embodimentId"], kind="robot")
+
+desc = world.describe()                                   # actuators, sites, objects (best-effort)
+world.ctrl({"left_wheel": 1.0, "right_wheel": 1.0})       # set actuator targets directly
+world.actuator("left_wheel", 0.0)                          # one named actuator
+world.ik(target=[0.3, 0.0, 0.4], site="ee", relative=False)  # inverse kinematics to a point
+world.trajectory([{"left_wheel": 1.0}, {"left_wheel": 0.0}])  # follow waypoints
+frame = world.observe()                                    # camera frame, as before
+```
+
+> `describe()` is best-effort: the runtime answers a describe request over the session channel, but
+> there is currently no synchronous describe HTTP endpoint — if nothing echoes back it returns `{}`,
+> and the autonomous agent also obtains descriptions server-side via `/agent/robot-act`.
+
+## Autonomous agents over many robots
+
+One agent can drive **many** robots in a single session. `RobotAgent` loops perceive → reason → act:
+each step it gathers every embodiment's description + a fresh camera frame, calls `/agent/robot-act` with
+all embodiments, and applies the returned tool calls (`ctrl`/`actuator`/`ik`/`trajectory`) back to the
+addressed embodiment — until a `done` call or `max_steps`.
+
+```python
+from commandagi import CommandAGI
+from commandagi.agent import RobotAgent, attach_robots
+
+cagi = CommandAGI(api_key="cagi_...")
+sim = cagi.launch_sim(scene="warehouse")
+
+embodiments = attach_robots(cagi, sim, kinds=["rover", "arm"])   # two robots, one session
+
+with RobotAgent(cagi, sim.session_id, embodiments, goal="bring the red box to the arm") as agent:
+    result = agent.run(max_steps=25)        # blocks; prints reasoning + applied calls each step
+print("done:", result["done"], "in", result["steps"], "steps")
+
+sim.stop()
+```
+
+A full runnable script lives in [`examples/sim_agent.py`](examples/sim_agent.py).
+
+## Computers too
+
+```python
+with cagi.launch("computer/software-engineer") as pc:
+    pc.act("type", text="hello")
+    pc.act("key", key="Return")
+    screenshot = pc.observe()       # PNG bytes of the live Ubuntu desktop
+```
+
+## Auth
+
+Create an API key with an `operator` scope (dashboard → API keys, or `POST /me/api-keys`). Pass it to
+`CommandAGI(api_key=...)` or set `COMMANDAGI_API_KEY`. Point at another environment with
+`COMMANDAGI_BASE_URL` (e.g. `https://api-dev.commandagi.com`).
+
+Full HTTP + WebSocket reference (what the SDK wraps): [`docs/platform/ROBOT_DEVELOPER_API.md`](../../docs/platform/ROBOT_DEVELOPER_API.md).

commandagi-0.3.0/README.md

@@ -0,0 +1,134 @@
+# CommandAGI Python SDK
+
+Launch real cloud **computers** and **3D robot simulations** and control them from Python — stream
+the robot's camera, send actions, run episodes. No agent required: you drive.
+
+```bash
+pip install commandagi          # + `pip install commandagi[vision]` for numpy frames
+```
+
+## Robot testing in a 3D world
+
+```python
+from commandagi import CommandAGI
+
+cagi = CommandAGI(api_key="cagi_...")          # or set COMMANDAGI_API_KEY
+
+with cagi.launch("simulation/warehouse") as world:
+    obs = world.observe()                      # JPEG bytes from the robot's head camera
+    for _ in range(20):
+        obs = world.step("turn", dir="left")   # act, then get the next frame
+    world.reset()                              # robot back to the episode start
+# leaving the block stops the world and releases the cloud VM
+```
+
+`launch()` provisions a real GCE VM running a 3D physics world, waits until it's streaming, and gives
+you a `World`. Built-in scenes: `simulation/warehouse`, `simulation/house-on-fire`,
+`simulation/school` (a mobile robot in each).
+
+### The control vocabulary
+
+| World kind  | actions                                                                   |
+| ----------- | ------------------------------------------------------------------------- |
+| robot / sim | `move(speed)`, `back(speed)`, `turn(dir, rate)`, `stop`, `reset`          |
+| computer    | `click(x, y)`, `type(text)`, `key(key)`, `move(x, y)`, `scroll(x, y, dy)` |
+
+```python
+world.act("move", speed=0.8)        # fire-and-forget
+obs = world.step("move", speed=0.8) # act + return the next observation (settles 0.8s)
+obs = world.observe(fresh=True)     # wait for a frame newer than now
+arr = world.observe_array()         # HxWx3 uint8 numpy (needs commandagi[vision])
+for frame in world.stream():        # live generator of frames
+    ...
+```
+
+## Simulator instances (morphology-agnostic robots)
+
+The simulator is **morphology-agnostic**: a robot is just a set of named actuators and sites, driven
+by one small **generic** control vocabulary — no `drive`/`gripper`, just `ctrl` / `actuator` / `ik`
+/ `trajectory` / `describe`. Spin up your own instance, choose who can watch or add robots, and
+populate it with one or many robots on a single session.
+
+```python
+from commandagi import CommandAGI
+
+cagi = CommandAGI(api_key="cagi_...")
+
+sim = cagi.launch_sim(scene="the-matrix", visibility="private", title="demo")
+print("instance:", sim.id, "session:", sim.session_id)
+
+# Who can do what:
+sim.grant("user_teammate", capability="viewer")    # may watch the stream
+sim.grant("user_buddy",    capability="operator")  # may also launch robots into the world
+
+# Add robots (each becomes a embodiment on sim.session_id):
+rover = sim.join_robot(kind="rover")   # -> {robotId, embodimentId, sessionId}
+arm   = sim.join_robot(kind="arm")
+
+cagi.sims()             # list instances visible to you
+cagi.get_sim(sim.id)    # rehydrate a SimInstance
+sim.view()              # instance metadata + attached embodiments
+sim.stop()              # release it (or use `with cagi.launch_sim(...) as sim:`)
+```
+
+### Generic robot control
+
+`World` exposes the morphology-agnostic vocabulary (address a specific robot in a multi-robot embodiment
+with `robot_id`):
+
+```python
+world = cagi.connect_world(sim.session_id, rover["embodimentId"], kind="robot")
+
+desc = world.describe()                                   # actuators, sites, objects (best-effort)
+world.ctrl({"left_wheel": 1.0, "right_wheel": 1.0})       # set actuator targets directly
+world.actuator("left_wheel", 0.0)                          # one named actuator
+world.ik(target=[0.3, 0.0, 0.4], site="ee", relative=False)  # inverse kinematics to a point
+world.trajectory([{"left_wheel": 1.0}, {"left_wheel": 0.0}])  # follow waypoints
+frame = world.observe()                                    # camera frame, as before
+```
+
+> `describe()` is best-effort: the runtime answers a describe request over the session channel, but
+> there is currently no synchronous describe HTTP endpoint — if nothing echoes back it returns `{}`,
+> and the autonomous agent also obtains descriptions server-side via `/agent/robot-act`.
+
+## Autonomous agents over many robots
+
+One agent can drive **many** robots in a single session. `RobotAgent` loops perceive → reason → act:
+each step it gathers every embodiment's description + a fresh camera frame, calls `/agent/robot-act` with
+all embodiments, and applies the returned tool calls (`ctrl`/`actuator`/`ik`/`trajectory`) back to the
+addressed embodiment — until a `done` call or `max_steps`.
+
+```python
+from commandagi import CommandAGI
+from commandagi.agent import RobotAgent, attach_robots
+
+cagi = CommandAGI(api_key="cagi_...")
+sim = cagi.launch_sim(scene="warehouse")
+
+embodiments = attach_robots(cagi, sim, kinds=["rover", "arm"])   # two robots, one session
+
+with RobotAgent(cagi, sim.session_id, embodiments, goal="bring the red box to the arm") as agent:
+    result = agent.run(max_steps=25)        # blocks; prints reasoning + applied calls each step
+print("done:", result["done"], "in", result["steps"], "steps")
+
+sim.stop()
+```
+
+A full runnable script lives in [`examples/sim_agent.py`](examples/sim_agent.py).
+
+## Computers too
+
+```python
+with cagi.launch("computer/software-engineer") as pc:
+    pc.act("type", text="hello")
+    pc.act("key", key="Return")
+    screenshot = pc.observe()       # PNG bytes of the live Ubuntu desktop
+```
+
+## Auth
+
+Create an API key with an `operator` scope (dashboard → API keys, or `POST /me/api-keys`). Pass it to
+`CommandAGI(api_key=...)` or set `COMMANDAGI_API_KEY`. Point at another environment with
+`COMMANDAGI_BASE_URL` (e.g. `https://api-dev.commandagi.com`).
+
+Full HTTP + WebSocket reference (what the SDK wraps): [`docs/platform/ROBOT_DEVELOPER_API.md`](../../docs/platform/ROBOT_DEVELOPER_API.md).

commandagi-0.3.0/commandagi/__init__.py

@@ -0,0 +1,26 @@
+"""CommandAGI Python SDK.
+
+Launch cloud computers and 3D robot simulations and control them programmatically.
+
+    from commandagi import CommandAGI
+    cagi = CommandAGI(api_key="cagi_...")
+    with cagi.launch("simulation/warehouse") as world:
+        obs = world.observe()
+        obs = world.step("move", speed=0.8)
+"""
+from .agent import RobotAgent, attach_robots
+from .bridge import RobotBridge
+from .client import COMPUTERS, SIMULATIONS, CommandAGI, CommandAGIError, SimInstance, World
+
+__all__ = [
+    "CommandAGI",
+    "World",
+    "SimInstance",
+    "RobotAgent",
+    "attach_robots",
+    "RobotBridge",
+    "CommandAGIError",
+    "SIMULATIONS",
+    "COMPUTERS",
+]
+__version__ = "0.3.0"

commandagi-0.3.0/commandagi/agent.py

@@ -0,0 +1,172 @@
+"""Autonomous multi-robot agent runner.
+
+One agent, many robots. :class:`RobotAgent` drives a whole set of robot devices in a single thread
+toward a natural-language goal: each step it gathers every device's world description and a fresh
+camera frame, asks the platform's reasoning model (``POST /agent/robot-act``) what to do next, and
+applies the returned tool calls back to the right device over the generic control vocabulary
+(``ctrl`` / ``actuator`` / ``ik`` / ``trajectory``). It stops when the model emits a ``done`` call or
+``max_steps`` is reached.
+
+    from commandagi import CommandAGI
+    from commandagi.agent import RobotAgent, attach_robots
+
+    cagi = CommandAGI(api_key="cagi_…")
+    sim = cagi.launch_sim(scene="warehouse")
+    devices = attach_robots(cagi, sim, kinds=["rover", "arm"])   # two robots, one thread
+
+    agent = RobotAgent(cagi, sim.thread_id, devices, goal="bring the red box to the arm")
+    result = agent.run(max_steps=25)                              # blocks; prints each step
+    print("done:", result["done"], "in", result["steps"], "steps")
+    sim.stop()
+"""
+from __future__ import annotations
+
+import base64
+from typing import Optional
+
+from .client import CommandAGI, World
+
+# Maps a returned tool name to how its call is applied on a World. ``done`` is terminal (handled
+# specially). Anything else is forwarded as a raw action, so the runner keeps working if the API
+# grows the vocabulary.
+_GENERIC_TOOLS = {"ctrl", "actuator", "ik", "trajectory", "describe"}
+
+
+def attach_robots(client: CommandAGI, sim, kinds: list[str]) -> list[dict]:
+    """Spawn several robots into one :class:`~commandagi.client.SimInstance` and return their devices.
+
+    Returns a list of ``{deviceId, robotId, kind}`` — one per requested kind — all on
+    ``sim.thread_id``. This is the convenience wrapper for building a multi-robot thread to hand to
+    :class:`RobotAgent`.
+    """
+    devices: list[dict] = []
+    for kind in kinds:
+        info = sim.join_robot(kind=kind)
+        devices.append({"deviceId": info["deviceId"], "robotId": info.get("robotId", ""), "kind": kind})
+    return devices
+
+
+def _frame_data_url(frame: bytes) -> str:
+    mime = "image/png" if frame[:8] == b"\x89PNG\r\n\x1a\n" else "image/jpeg"
+    return f"data:{mime};base64," + base64.b64encode(frame).decode()
+
+
+class RobotAgent:
+    """An autonomous runner that drives many robot devices in one thread toward ``goal``.
+
+    Parameters
+    ----------
+    client:
+        A :class:`~commandagi.client.CommandAGI` client.
+    thread_id:
+        The thread the robot devices live on (e.g. ``sim.thread_id``).
+    devices:
+        A list of ``{deviceId, robotId?, kind?}`` describing the robots to drive (e.g. the output of
+        :func:`attach_robots`).
+    goal:
+        The natural-language objective handed to ``/agent/robot-act`` each step.
+    model:
+        Optional model id override for the reasoning call.
+    """
+
+    def __init__(self, client: CommandAGI, thread_id: str, devices: list[dict], goal: str,
+                 *, model: Optional[str] = None):
+        self.client = client
+        self.thread_id = thread_id
+        self.goal = goal
+        self.model = model
+        self._device_meta = {d["deviceId"]: d for d in devices}
+        # One control channel per device, all on the same thread. The agent addresses robots by
+        # deviceId; a single World per device carries its observe()/describe()/control.
+        self.worlds: dict[str, World] = {
+            d["deviceId"]: client.connect_world(thread_id, d["deviceId"], kind="robot")
+            for d in devices
+        }
+        self.history: list[dict] = []
+
+    # ── per-step pieces ────────────────────────────────────────────────────────
+    def _gather(self) -> list[dict]:
+        """Collect ``{deviceId, world, camera}`` for every device (description + a fresh frame)."""
+        payload: list[dict] = []
+        for device_id, world in self.worlds.items():
+            try:
+                description = world.describe(robot_id=self._device_meta[device_id].get("robotId", ""))
+            except Exception:
+                description = {}
+            try:
+                camera = _frame_data_url(world.observe(timeout=15.0))
+            except Exception:
+                camera = None
+            payload.append({"deviceId": device_id, "world": description, "camera": camera})
+        return payload
+
+    def _apply(self, call: dict) -> None:
+        """Apply one returned tool call to the addressed device's :class:`World`."""
+        tool = call.get("tool")
+        device_id = call.get("deviceId")
+        robot_id = call.get("robotId", "")
+        world = self.worlds.get(device_id)
+        if world is None or tool in (None, "done"):
+            return
+        if tool == "ctrl":
+            world.ctrl(call.get("targets", {}), robot_id=robot_id)
+        elif tool == "actuator":
+            world.actuator(call.get("name", ""), call.get("value", 0.0), robot_id=robot_id)
+        elif tool == "ik":
+            world.ik(call.get("target", []), site=call.get("site"),
+                     relative=bool(call.get("relative", False)), robot_id=robot_id)
+        elif tool == "trajectory":
+            world.trajectory(call.get("waypoints", []), robot_id=robot_id)
+        elif tool == "describe":
+            world.describe(robot_id=robot_id)
+        else:
+            # Unknown/extended tool: forward generically with whatever fields came back.
+            extra = {k: v for k, v in call.items() if k not in ("tool", "deviceId", "robotId")}
+            world.act(tool, robotId=robot_id, **extra)
+
+    # ── main loop ──────────────────────────────────────────────────────────────
+    def step(self) -> dict:
+        """Run a single perceive→reason→act cycle. Returns ``{reasoning, calls, done}``."""
+        devices = self._gather()
+        res = self.client.robot_act(self.goal, devices, model=self.model, history=self.history or None)
+        reasoning = res.get("reasoning", "")
+        calls = res.get("calls", []) or []
+        done = any(c.get("tool") == "done" for c in calls)
+        for call in calls:
+            self._apply(call)
+        self.history.append({"reasoning": reasoning, "calls": calls})
+        return {"reasoning": reasoning, "calls": calls, "done": done}
+
+    def run(self, max_steps: int = 50, *, verbose: bool = True) -> dict:
+        """Loop :meth:`step` until a ``done`` call or ``max_steps``.
+
+        Returns ``{done, steps, history}``. With ``verbose`` (default), prints the reasoning and the
+        applied calls each step.
+        """
+        steps = 0
+        done = False
+        for i in range(max_steps):
+            out = self.step()
+            steps = i + 1
+            if verbose:
+                print(f"[step {steps}] {out['reasoning']}")
+                for call in out["calls"]:
+                    print(f"    → {call.get('tool')} on {call.get('deviceId')} {call.get('robotId', '')}".rstrip())
+            if out["done"]:
+                done = True
+                break
+        return {"done": done, "steps": steps, "history": self.history}
+
+    def close(self) -> None:
+        """Close every device control channel (does not stop the thread). Safe to call twice."""
+        for world in self.worlds.values():
+            try:
+                world._ws.close()
+            except Exception:
+                pass
+
+    def __enter__(self) -> "RobotAgent":
+        return self
+
+    def __exit__(self, *_exc) -> None:
+        self.close()