abstract-client 0.2.2__tar.gz → 0.3.1__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -1,9 +1,9 @@
1
- abstract-client (Abstract's robot API from Python, import name atoms)
1
+ abstract-client (Abstract's robot API from Python, import name abstract)
2
2
  Copyright 2026 Abstract
3
3
 
4
4
  Licensed under the Apache License, Version 2.0 (LICENSE).
5
5
 
6
- The harness (atoms.AtomsClient) reproduces the calls of dreamlabs-client's DreamClient (policy,
6
+ The harness (abstract.RobotClient) reproduces the calls of dreamlabs-client's DreamClient (policy,
7
7
  set_observation, get_action, reset_episode, refill_due, close, last_timing), and its frame
8
8
  window, refill pacing and slow-network warning are ported from dreamlabs-client
9
9
  (https://github.com/dreamlabs-co/dreamlabs), first through the TypeScript harness
@@ -0,0 +1,324 @@
1
+ Metadata-Version: 2.5
2
+ Name: abstract-client
3
+ Version: 0.3.1
4
+ Summary: Abstract's robot API from Python: action chunks from served robot models, sync and asyncio.
5
+ Project-URL: Homepage, https://abstract.inc
6
+ Project-URL: Documentation, https://docs.abstract.inc
7
+ License-Expression: Apache-2.0
8
+ License-File: LICENSE
9
+ License-File: NOTICE
10
+ Classifier: Intended Audience :: Developers
11
+ Classifier: Intended Audience :: Science/Research
12
+ Classifier: Programming Language :: Python :: 3
13
+ Classifier: Programming Language :: Python :: 3.10
14
+ Classifier: Programming Language :: Python :: 3.11
15
+ Classifier: Programming Language :: Python :: 3.12
16
+ Classifier: Programming Language :: Python :: 3.13
17
+ Classifier: Programming Language :: Python :: 3.14
18
+ Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
19
+ Classifier: Typing :: Typed
20
+ Requires-Python: >=3.10
21
+ Requires-Dist: httpx2<3,>=2.12
22
+ Provides-Extra: quic
23
+ Requires-Dist: aioquic<2,>=1.3; extra == 'quic'
24
+ Description-Content-Type: text/markdown
25
+
26
+ # abstract.client
27
+
28
+ Abstract's robot API from Python. A `Client` holds your key and its connections and is safe to
29
+ share between threads; each `Policy` is one episode of one served robot model doing one task, one
30
+ action chunk per `predict()`.
31
+
32
+ ```sh
33
+ uv add abstract-client
34
+ ```
35
+
36
+ ### Quickstart (0.3.0)
37
+
38
+ ```python
39
+ import abstract
40
+
41
+ checkpoint = "allenai/molmoact2"
42
+ embodiment = "yam"
43
+ client = abstract.client(ABSTRACT_API_KEY, checkpoint, embodiment)
44
+
45
+ client.set_observation(frames, observation)
46
+ action = client.get_action()
47
+ client.reset_episode()
48
+ client.close()
49
+ ```
50
+
51
+ `ABSTRACT_API_KEY` is your key, `frames` maps each camera to JPEG bytes, and `observation`
52
+ holds `prompt` (the task) and `state` (measured alongside the frames). The YAM camera and
53
+ state fields are shown below. The returned `Chunk` contains targets, never hardware motion;
54
+ reset ends the model's episode without repositioning the robot.
55
+
56
+ `abstract.client()` checks the account's unmetered model catalogue before configuring a
57
+ `RobotClient`. The checkpoint must be available and its `robot` must match `embodiment`.
58
+ An optional URDF layer converts poses to another arm's joint geometry; it does not establish
59
+ policy transfer. Omit `api_key` to use the environment or saved
60
+ CLI credentials; connection options such as `base_url` and `timeout` pass through to the
61
+ existing client. Context managers and the lower-level `abstract.Client` interface remain supported.
62
+
63
+ The `abstract.client(...)` factory has existed since 0.2.2. Version 0.3.0 moves the full SDK
64
+ into `abstract`, including the lower-level clients and policy types. Publish that version before
65
+ advertising the renamed advanced interfaces. `import atoms`, the 0.2 name, still resolves in 0.3
66
+ with a `DeprecationWarning`: its top-level names only, `AtomsClient` being `abstract.RobotClient`.
67
+
68
+ This is all a robot with its own stack needs. The `abstract` command and the robot station are
69
+ in `abstract-atoms`, built on it (`apps/atoms`).
70
+
71
+ ```python
72
+ import abstract
73
+
74
+ client = abstract.Client() # the key: api_key=, else ABSTRACT_API_KEY, else `abstract login`'s file
75
+
76
+ with client.connect("allenai/molmoact2", "fold the towel") as policy:
77
+ frames = {"overhead": top_jpeg, "wrist_l": left_jpeg, "wrist_r": right_jpeg}
78
+ while not done:
79
+ state = {
80
+ "left_eef_xyz": [x, y, z], # metres, in the left arm's base frame
81
+ "left_eef_rot6": rot6, # the rotation matrix's first two columns
82
+ "left_grip": [g], # 0 closed, 1 open
83
+ "right_eef_xyz": [x, y, z],
84
+ "right_eef_rot6": rot6,
85
+ "right_grip": [g],
86
+ }
87
+ chunk = policy.predict(frames, state)
88
+ # Row k is a pose for t + k * chunk.dt; your arm's inverse kinematics turns each into
89
+ # joint angles. The arm held still while predict() ran, so every row runs; a loop that
90
+ # keeps the arm moving meanwhile skips the rows it ran, as the RobotClient loop below.
91
+ for left, right in zip(chunk.left, chunk.right):
92
+ robot.command(left, right)
93
+ # The next call sends policy.frames_needed JPEGs per camera, oldest first.
94
+ frames = latest_frames(policy.frames_needed)
95
+ # Leaving the block ends the episode now; the server would after 60 s idle.
96
+ ```
97
+
98
+ A stateful model takes the video since its last observation: after the first call,
99
+ send `policy.frames_needed` frames per camera.
100
+
101
+ ### Native robot checkpoints
102
+
103
+ These interfaces require `abstract-client` 0.2.0 or later.
104
+
105
+ `client.get_models()` is the availability source; an SDK adapter does not mean a checkpoint
106
+ is enabled on the account. These native contracts are separate from the YAM example:
107
+
108
+ | Model | Observation | Returned commands |
109
+ | --------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ |
110
+ | `verne/v0.5` | The YAM example's poses and cameras, one frame per camera and then four; not trained on the YAM | 18 pose rows per arm at 30 Hz |
111
+ | `runway/praxis-1` | The YAM example's poses and cameras at 16:10, one frame per camera and then 21: one at the end of each row just run. Capture them yourself and call `policy.predict()`; `RobotClient`'s window also counts the pause between chunks. | 21 pose rows per arm at 15 Hz |
112
+ | `xiaomi/xiaomi-robotics-1` | RoboDojo ARX X5 world-frame poses, normalized grippers (0 closed, 1 open), and `left_joint_position` / `right_joint_position` with six measured radians each. Cameras retain `overhead`, `wrist_l`, `wrist_r`. | `chunk.left` / `chunk.right`: 30 end-effector targets at 25 Hz, in the same RoboDojo world frame. |
113
+ | `black-forest-labs/flux-3-action` | DROID Franka `joint_position` (seven radians), `gripper_position` (one closed fraction: 0 open, 1 closed), and cameras `exterior_image_0_left`, `exterior_image_1_left`, `wrist_image_left`. | `chunk.joints`: 32 rows of seven absolute joint targets plus gripper at 15 Hz; `chunk.action_space == "joint_position"`. `left` and `right` are empty. |
114
+
115
+ Use `chunk.row_count` for either command format. `elapsed_rows`, sync/async policies, retries
116
+ and the `RobotClient` observation window work with both. Policies sharing one observation
117
+ window must use the same state and camera contract. The bimanual station does not drive these native contracts.
118
+
119
+ ### `abstract.RobotClient`: Dream Labs' DreamClient on this API
120
+
121
+ `abstract.RobotClient` keeps that window for you and paces the requests, with the calls of Dream Labs'
122
+ `DreamClient` (Apache-2.0; see `NOTICE`): bind the key, choose the model, push every camera tick,
123
+ pull a chunk when it says so.
124
+
125
+ ```python
126
+ from concurrent.futures import ThreadPoolExecutor
127
+
128
+ robot = abstract.RobotClient() # the key, as abstract.Client finds it
129
+ robot.policy({"model": "allenai/molmoact2"})
130
+ with robot, ThreadPoolExecutor(1) as worker: # the request runs while the robot moves
131
+ chunk, row, pending = None, 0, None
132
+ ran = ran_at_ask = 0 # rows the robot has run, and how many when the request went out
133
+ while not done:
134
+ frames = {"video.overhead": top, "video.wrist_l": left, "video.wrist_r": right}
135
+ robot.set_observation(frames, {"prompt": "fold the towel", "state": read_state()})
136
+ rows_left = len(chunk.left) - row if chunk else 0
137
+ if pending is None and robot.refill_due(row, rows_left):
138
+ pending, ran_at_ask = worker.submit(robot.get_action), ran
139
+ if pending is not None and pending.done():
140
+ chunk, pending = pending.result(), None
141
+ # Skip the rows the robot ran while this chunk was computed (at most those already
142
+ # due, chunk.elapsed_rows). A robot that held still meanwhile starts at row 0.
143
+ row = min(chunk.elapsed_rows, ran - ran_at_ask)
144
+ if chunk and row < len(chunk.left):
145
+ robot.command(chunk.left[row], chunk.right[row])
146
+ row, ran = row + 1, ran + 1
147
+ wait_for_next_tick()
148
+ ```
149
+
150
+ #### Several models on one robot
151
+
152
+ DreamClient runs one model. `RobotClient` runs as many as you open on the same camera ticks, each
153
+ with its own episode, frames (a stateful model takes several per camera, MolmoAct2 one), pacing and timing.
154
+ One drives the arms; the others answer the same observations, to compare them.
155
+
156
+ ```python
157
+ robot.policies(
158
+ {"molmo": {"model": "allenai/molmoact2"}, "pi": {"model": "physical-intelligence/pi0.5"}}
159
+ )
160
+ robot.set_observation(frames, observation) # one tick, every model
161
+ answers = robot.get_actions() # every model at once: {name: Chunk, or the error it raised}
162
+ drive = robot["molmo"] # one model: get_action(), refill_due(), last, mode, last_timing
163
+ ```
164
+
165
+ | DreamClient | abstract.RobotClient |
166
+ | ----------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
167
+ | `DreamClient(api_key)` | `RobotClient(api_key)`, or `RobotClient(client=...)` |
168
+ | `get_models()` | `get_models()`: served models and your fine-tunes, with their rates |
169
+ | `policy({"checkpoint": ...})` | `policy({"model": ...})`; `execution_horizon`, `action_hz`, `sync_above_ms`, `timeout_ms` as in DreamClient |
170
+ | `set_observation(frames, observation, timestamp=)` | the same: `video.<camera>` frames, `prompt` and `state` (a dict or `state.<name>` keys) |
171
+ | `get_action()` returns joint chunks | `get_action()` returns a `Chunk` in the selected model's action space: poses in `left` / `right`, native joint commands in `joints` |
172
+ | `reset_episode()`, `refill_due()`, `close()` | the same |
173
+ | `mode`, `execution_horizon`, `action_hz`, `last_timing`, `checkpoint`, `session_id` | the same, `model` for `checkpoint` |
174
+ | `obs_window`, `obs_fps` | none: each answer says how many frames the next observation carries |
175
+ | `DreamSlowNetworkWarning` | `abstract.SlowNetworkWarning` |
176
+ | one model per client | `policies({name: config})`, `get_actions()`, `robot[name]` |
177
+
178
+ ### Optional URDF FK and IK (0.3.0)
179
+
180
+ The solver is not in this package: install `abstract-atoms[ik]` after release, or
181
+ `uv pip install -e ./apps/atoms/client -e './apps/atoms[ik]'` from this checkout. Configure each physical arm's URDF, joint order, grasp point and frames:
182
+
183
+ ```python
184
+ from abstract_robots.ik import RobotIK
185
+
186
+ ik = RobotIK.from_file("robot.json")
187
+ client = abstract.client(checkpoint="allenai/molmoact2", embodiment="yam", ik=ik)
188
+ client.set_joint_observation(frames, measured_joints, grippers, prompt="fold the towel")
189
+ chunk = client.get_action()
190
+ # Measure again after inference; solving is local and commands no hardware. The arm held still
191
+ # during this call; one that kept moving passes start= the rows it ran (at most elapsed_rows).
192
+ joints = client.to_joints(chunk, current_measured_joints)
193
+ ```
194
+
195
+ See [the configuration and driver guide](../robots/IK.md) for full examples, limits, geometry checks and
196
+ validation scope. Different arm geometries/joint counts are supported locally. Model embodiment
197
+ checks remain; correct IK does not prove policy transfer. The Python station uses the same solver
198
+ through `--ik robot.json`, with `observe_joints()` / `command_joints()` drivers.
199
+
200
+ DreamClient's seam smoothing (`commanded`) is not implemented. Dataset and fine-tuning commands
201
+ and the station's Live View remain in `abstract-atoms`.
202
+
203
+ `abstract.AsyncClient` is native asyncio with the same arguments, rules and calls
204
+ (`await policy.predict(...)`). Cancelling a call cancels its request, but the server may already
205
+ be running it, so the next `predict()` asks for that answer first, as after a lost connection.
206
+
207
+ Connections are HTTPX2's (Pydantic's maintained HTTPX, as the OpenAI and Anthropic SDKs use).
208
+ Proxies come from `HTTPS_PROXY` and `NO_PROXY`, and TLS is checked against the operating system's
209
+ trust store, so a CA your IT installed works as is. For a private CA, a client certificate or a
210
+ proxy in code, configure HTTPX2 and pass it in; the client you pass stays yours to close:
211
+
212
+ ```python
213
+ import ssl
214
+
215
+ import httpx2
216
+
217
+ tls = ssl.create_default_context(cafile="lab-ca.pem")
218
+ tls.load_cert_chain("robot.pem", "robot.key")
219
+ client = abstract.Client(http_client=httpx2.Client(verify=tls, proxy="http://proxy.lab:3128"))
220
+ ```
221
+
222
+ ### Direct sessions and the time profile
223
+
224
+ ```python
225
+ with client.connect("allenai/molmoact2", "fold the towel") as policy:
226
+ chunk = policy.predict(frames, state)
227
+ chunk.stages # this chunk's milliseconds: total, client, network, server and its parts
228
+ print(policy.profile) # the episode: each stage's p50, p95 and max, and what happened on the way
229
+ ```
230
+
231
+ Where the API names an engine for a model, an episode's chunks go straight to that engine and
232
+ no longer cross the API. There is nothing to set: the API holds a block of chunks on your
233
+ account and signs a ticket for it, and each `predict()` posts to the engine under that ticket.
234
+ The same `Chunk`, the same errors, an observation resent under its own id and never run or
235
+ billed twice. Your one choice stays `transport`:
236
+
237
+ - `"http"`: HTTPS to the engine's host.
238
+ - `"quic"`: the engine's own QUIC socket, which the client reaches by UDP hole punching. An
239
+ episode's first chunk goes over HTTPS; the punch then runs in the background, and the chunks
240
+ that follow travel over QUIC. Where a network blocks UDP or a NAT does not allow the punch,
241
+ or the connection is later lost, the episode keeps to HTTPS without a word.
242
+
243
+ What happens on its own:
244
+
245
+ - The next ticket is asked for in the background while chunks remain, so `predict()` does not
246
+ wait on it.
247
+ - An engine that fails twice in a row, or stops answering for 10 s after it has answered, is
248
+ left: the API names the next one and the episode continues there, and when none is left it
249
+ continues through the API. A model without episode memory moves unnoticed, its `step`
250
+ restarting where the new engine is; a model that keeps its episode on the GPU raises
251
+ `SessionExpiredError`, as it does when its replica is lost.
252
+ - A model the API names no engine for runs through the API as ever, and so do FLUX, whose
253
+ content filters the API applies to every chunk, and your fine-tunes. A client asks about a
254
+ model once, not once an episode. `AsyncClient` always goes through the API.
255
+
256
+ What to know:
257
+
258
+ - A direct episode's observations and answers are not stored with your account's trajectories.
259
+ - The engine's hosts end in `modal.run`: a firewall that allows `api.abstract.inc` alone sends
260
+ the first episode back through the API after about four seconds, and the next ones at once.
261
+ - To punch, the client asks a public STUN server (`stun.cloudflare.com`, then
262
+ `stun.l.google.com`) what address its UDP socket is seen at. That request carries nothing
263
+ but the question; the server sees the machine's public address.
264
+
265
+ `policy.profile` is kept for every episode. `client` is the time to build the request, `server`
266
+ the engine's from receipt to answer (`decode`, `queue`, `prefill` and `denoise` are its parts,
267
+ `queue` counted from receipt), and `network` the rest: both ways, and the API when the chunk
268
+ crossed it. `policy.profile.summary()` returns the same as a dict, with which way each chunk
269
+ travelled (`api`, `https`, `quic`) and `events`: a ticket opened or renewed, a QUIC path punched
270
+ or given up for HTTPS, a move to another engine, a return to the API.
271
+
272
+ ### Transports
273
+
274
+ `transport` chooses how the calls travel. `"http"` is the default and what everything above
275
+ describes. `"quic"` sends the same calls to the same routes over HTTP/3, on one QUIC connection
276
+ that every policy on the client shares; it is opt-in and needs the `quic` extra:
277
+
278
+ ```sh
279
+ uv add 'abstract-client[quic]'
280
+ ```
281
+
282
+ ```python
283
+ client = abstract.Client(transport="quic") # or ABSTRACT_TRANSPORT=quic; every client takes it
284
+ ```
285
+
286
+ The API sees the same request either way, so chunks are billed the same and a call that gets no
287
+ answer is sent again under the same rules. What differs is the network's side:
288
+
289
+ - QUIC needs outbound UDP to the API on port 443. Where a network blocks it, calls fail with
290
+ `APIConnectionError` or `APITimeoutError`; nothing switches to `"http"` on its own.
291
+ - No proxy can carry it. On a machine set to use one (`HTTPS_PROXY`, `ALL_PROXY` or the system's)
292
+ the client raises `ConfigError` rather than go around the proxy, and `http_client=` cannot be
293
+ combined with it.
294
+ - TLS is checked against certifi's CA bundle, not the operating system's trust store: a CA your
295
+ IT installed is not trusted, and there is no setting for a private CA or a client certificate.
296
+ - A connection that hears nothing for 20 s is closed and the next call opens another. While a
297
+ call waits for its answer the client pings every 5 s, so a model's cold start does not end it.
298
+
299
+ It runs aioquic's QUIC stack as it comes, with no tuning, and has not been measured against
300
+ `"http"`: choose it to compare the two on your own network, not on a promise that it is faster.
301
+ `client.transport` says which one a client uses.
302
+
303
+ `connect(model, task, scaling={"seed": 7})` (or `scaling` in a `RobotClient` config) sends the
304
+ API's `scaling` with every observation, so a seedable model replays its noise; the engine's reports
305
+ (`scaling`, `cache`, `context`) come back in `chunk.raw`.
306
+
307
+ Each chunk is billed and can wake a GPU. A call that gets no answer is sent again with the same
308
+ bytes and `observation_id`, so the server answers it from its record and never bills it twice.
309
+ Refusals the server marks retryable are sent again after its `Retry-After`, up to `max_retries`;
310
+ `timeout` bounds a whole call, resends included, and `connect_timeout` opening a connection.
311
+
312
+ Every error is an `abstract.AbstractError`. A refusal is an `abstract.APIStatusError` subclass named
313
+ for its status (`AuthenticationError`, `PaymentRequiredError`, `RateLimitError`, ...) with the
314
+ server's `code`, `retryable`, `retry_after` and `request_id`; a refused or answered
315
+ `get_action` also carries its reply's `trace_id`, which a replay keeps. `APIConnectionError` and
316
+ `APITimeoutError` mean no answer came: the next `predict()` asks for it first. The `abstract.client`
317
+ logger reports resends; it never logs the key.
318
+
319
+ Python 3.10 or later; its one dependency is `httpx2` (`aioquic` with the `quic` extra). Tests: `pytest apps/atoms` (the contract
320
+ tests need Bun and the engine's packages: `uv run --project apps/engine pytest apps/atoms`).
321
+
322
+ Version 0.3 replaces `import atoms` with `import abstract`; the buffered harness is
323
+ `abstract.RobotClient` (previously `AtomsClient`). `abstract.client(...)` is the recommended
324
+ entry point. The Python station uses this same implementation; there is no separate YAM runner.
@@ -0,0 +1,299 @@
1
+ # abstract.client
2
+
3
+ Abstract's robot API from Python. A `Client` holds your key and its connections and is safe to
4
+ share between threads; each `Policy` is one episode of one served robot model doing one task, one
5
+ action chunk per `predict()`.
6
+
7
+ ```sh
8
+ uv add abstract-client
9
+ ```
10
+
11
+ ### Quickstart (0.3.0)
12
+
13
+ ```python
14
+ import abstract
15
+
16
+ checkpoint = "allenai/molmoact2"
17
+ embodiment = "yam"
18
+ client = abstract.client(ABSTRACT_API_KEY, checkpoint, embodiment)
19
+
20
+ client.set_observation(frames, observation)
21
+ action = client.get_action()
22
+ client.reset_episode()
23
+ client.close()
24
+ ```
25
+
26
+ `ABSTRACT_API_KEY` is your key, `frames` maps each camera to JPEG bytes, and `observation`
27
+ holds `prompt` (the task) and `state` (measured alongside the frames). The YAM camera and
28
+ state fields are shown below. The returned `Chunk` contains targets, never hardware motion;
29
+ reset ends the model's episode without repositioning the robot.
30
+
31
+ `abstract.client()` checks the account's unmetered model catalogue before configuring a
32
+ `RobotClient`. The checkpoint must be available and its `robot` must match `embodiment`.
33
+ An optional URDF layer converts poses to another arm's joint geometry; it does not establish
34
+ policy transfer. Omit `api_key` to use the environment or saved
35
+ CLI credentials; connection options such as `base_url` and `timeout` pass through to the
36
+ existing client. Context managers and the lower-level `abstract.Client` interface remain supported.
37
+
38
+ The `abstract.client(...)` factory has existed since 0.2.2. Version 0.3.0 moves the full SDK
39
+ into `abstract`, including the lower-level clients and policy types. Publish that version before
40
+ advertising the renamed advanced interfaces. `import atoms`, the 0.2 name, still resolves in 0.3
41
+ with a `DeprecationWarning`: its top-level names only, `AtomsClient` being `abstract.RobotClient`.
42
+
43
+ This is all a robot with its own stack needs. The `abstract` command and the robot station are
44
+ in `abstract-atoms`, built on it (`apps/atoms`).
45
+
46
+ ```python
47
+ import abstract
48
+
49
+ client = abstract.Client() # the key: api_key=, else ABSTRACT_API_KEY, else `abstract login`'s file
50
+
51
+ with client.connect("allenai/molmoact2", "fold the towel") as policy:
52
+ frames = {"overhead": top_jpeg, "wrist_l": left_jpeg, "wrist_r": right_jpeg}
53
+ while not done:
54
+ state = {
55
+ "left_eef_xyz": [x, y, z], # metres, in the left arm's base frame
56
+ "left_eef_rot6": rot6, # the rotation matrix's first two columns
57
+ "left_grip": [g], # 0 closed, 1 open
58
+ "right_eef_xyz": [x, y, z],
59
+ "right_eef_rot6": rot6,
60
+ "right_grip": [g],
61
+ }
62
+ chunk = policy.predict(frames, state)
63
+ # Row k is a pose for t + k * chunk.dt; your arm's inverse kinematics turns each into
64
+ # joint angles. The arm held still while predict() ran, so every row runs; a loop that
65
+ # keeps the arm moving meanwhile skips the rows it ran, as the RobotClient loop below.
66
+ for left, right in zip(chunk.left, chunk.right):
67
+ robot.command(left, right)
68
+ # The next call sends policy.frames_needed JPEGs per camera, oldest first.
69
+ frames = latest_frames(policy.frames_needed)
70
+ # Leaving the block ends the episode now; the server would after 60 s idle.
71
+ ```
72
+
73
+ A stateful model takes the video since its last observation: after the first call,
74
+ send `policy.frames_needed` frames per camera.
75
+
76
+ ### Native robot checkpoints
77
+
78
+ These interfaces require `abstract-client` 0.2.0 or later.
79
+
80
+ `client.get_models()` is the availability source; an SDK adapter does not mean a checkpoint
81
+ is enabled on the account. These native contracts are separate from the YAM example:
82
+
83
+ | Model | Observation | Returned commands |
84
+ | --------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ |
85
+ | `verne/v0.5` | The YAM example's poses and cameras, one frame per camera and then four; not trained on the YAM | 18 pose rows per arm at 30 Hz |
86
+ | `runway/praxis-1` | The YAM example's poses and cameras at 16:10, one frame per camera and then 21: one at the end of each row just run. Capture them yourself and call `policy.predict()`; `RobotClient`'s window also counts the pause between chunks. | 21 pose rows per arm at 15 Hz |
87
+ | `xiaomi/xiaomi-robotics-1` | RoboDojo ARX X5 world-frame poses, normalized grippers (0 closed, 1 open), and `left_joint_position` / `right_joint_position` with six measured radians each. Cameras retain `overhead`, `wrist_l`, `wrist_r`. | `chunk.left` / `chunk.right`: 30 end-effector targets at 25 Hz, in the same RoboDojo world frame. |
88
+ | `black-forest-labs/flux-3-action` | DROID Franka `joint_position` (seven radians), `gripper_position` (one closed fraction: 0 open, 1 closed), and cameras `exterior_image_0_left`, `exterior_image_1_left`, `wrist_image_left`. | `chunk.joints`: 32 rows of seven absolute joint targets plus gripper at 15 Hz; `chunk.action_space == "joint_position"`. `left` and `right` are empty. |
89
+
90
+ Use `chunk.row_count` for either command format. `elapsed_rows`, sync/async policies, retries
91
+ and the `RobotClient` observation window work with both. Policies sharing one observation
92
+ window must use the same state and camera contract. The bimanual station does not drive these native contracts.
93
+
94
+ ### `abstract.RobotClient`: Dream Labs' DreamClient on this API
95
+
96
+ `abstract.RobotClient` keeps that window for you and paces the requests, with the calls of Dream Labs'
97
+ `DreamClient` (Apache-2.0; see `NOTICE`): bind the key, choose the model, push every camera tick,
98
+ pull a chunk when it says so.
99
+
100
+ ```python
101
+ from concurrent.futures import ThreadPoolExecutor
102
+
103
+ robot = abstract.RobotClient() # the key, as abstract.Client finds it
104
+ robot.policy({"model": "allenai/molmoact2"})
105
+ with robot, ThreadPoolExecutor(1) as worker: # the request runs while the robot moves
106
+ chunk, row, pending = None, 0, None
107
+ ran = ran_at_ask = 0 # rows the robot has run, and how many when the request went out
108
+ while not done:
109
+ frames = {"video.overhead": top, "video.wrist_l": left, "video.wrist_r": right}
110
+ robot.set_observation(frames, {"prompt": "fold the towel", "state": read_state()})
111
+ rows_left = len(chunk.left) - row if chunk else 0
112
+ if pending is None and robot.refill_due(row, rows_left):
113
+ pending, ran_at_ask = worker.submit(robot.get_action), ran
114
+ if pending is not None and pending.done():
115
+ chunk, pending = pending.result(), None
116
+ # Skip the rows the robot ran while this chunk was computed (at most those already
117
+ # due, chunk.elapsed_rows). A robot that held still meanwhile starts at row 0.
118
+ row = min(chunk.elapsed_rows, ran - ran_at_ask)
119
+ if chunk and row < len(chunk.left):
120
+ robot.command(chunk.left[row], chunk.right[row])
121
+ row, ran = row + 1, ran + 1
122
+ wait_for_next_tick()
123
+ ```
124
+
125
+ #### Several models on one robot
126
+
127
+ DreamClient runs one model. `RobotClient` runs as many as you open on the same camera ticks, each
128
+ with its own episode, frames (a stateful model takes several per camera, MolmoAct2 one), pacing and timing.
129
+ One drives the arms; the others answer the same observations, to compare them.
130
+
131
+ ```python
132
+ robot.policies(
133
+ {"molmo": {"model": "allenai/molmoact2"}, "pi": {"model": "physical-intelligence/pi0.5"}}
134
+ )
135
+ robot.set_observation(frames, observation) # one tick, every model
136
+ answers = robot.get_actions() # every model at once: {name: Chunk, or the error it raised}
137
+ drive = robot["molmo"] # one model: get_action(), refill_due(), last, mode, last_timing
138
+ ```
139
+
140
+ | DreamClient | abstract.RobotClient |
141
+ | ----------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
142
+ | `DreamClient(api_key)` | `RobotClient(api_key)`, or `RobotClient(client=...)` |
143
+ | `get_models()` | `get_models()`: served models and your fine-tunes, with their rates |
144
+ | `policy({"checkpoint": ...})` | `policy({"model": ...})`; `execution_horizon`, `action_hz`, `sync_above_ms`, `timeout_ms` as in DreamClient |
145
+ | `set_observation(frames, observation, timestamp=)` | the same: `video.<camera>` frames, `prompt` and `state` (a dict or `state.<name>` keys) |
146
+ | `get_action()` returns joint chunks | `get_action()` returns a `Chunk` in the selected model's action space: poses in `left` / `right`, native joint commands in `joints` |
147
+ | `reset_episode()`, `refill_due()`, `close()` | the same |
148
+ | `mode`, `execution_horizon`, `action_hz`, `last_timing`, `checkpoint`, `session_id` | the same, `model` for `checkpoint` |
149
+ | `obs_window`, `obs_fps` | none: each answer says how many frames the next observation carries |
150
+ | `DreamSlowNetworkWarning` | `abstract.SlowNetworkWarning` |
151
+ | one model per client | `policies({name: config})`, `get_actions()`, `robot[name]` |
152
+
153
+ ### Optional URDF FK and IK (0.3.0)
154
+
155
+ The solver is not in this package: install `abstract-atoms[ik]` after release, or
156
+ `uv pip install -e ./apps/atoms/client -e './apps/atoms[ik]'` from this checkout. Configure each physical arm's URDF, joint order, grasp point and frames:
157
+
158
+ ```python
159
+ from abstract_robots.ik import RobotIK
160
+
161
+ ik = RobotIK.from_file("robot.json")
162
+ client = abstract.client(checkpoint="allenai/molmoact2", embodiment="yam", ik=ik)
163
+ client.set_joint_observation(frames, measured_joints, grippers, prompt="fold the towel")
164
+ chunk = client.get_action()
165
+ # Measure again after inference; solving is local and commands no hardware. The arm held still
166
+ # during this call; one that kept moving passes start= the rows it ran (at most elapsed_rows).
167
+ joints = client.to_joints(chunk, current_measured_joints)
168
+ ```
169
+
170
+ See [the configuration and driver guide](../robots/IK.md) for full examples, limits, geometry checks and
171
+ validation scope. Different arm geometries/joint counts are supported locally. Model embodiment
172
+ checks remain; correct IK does not prove policy transfer. The Python station uses the same solver
173
+ through `--ik robot.json`, with `observe_joints()` / `command_joints()` drivers.
174
+
175
+ DreamClient's seam smoothing (`commanded`) is not implemented. Dataset and fine-tuning commands
176
+ and the station's Live View remain in `abstract-atoms`.
177
+
178
+ `abstract.AsyncClient` is native asyncio with the same arguments, rules and calls
179
+ (`await policy.predict(...)`). Cancelling a call cancels its request, but the server may already
180
+ be running it, so the next `predict()` asks for that answer first, as after a lost connection.
181
+
182
+ Connections are HTTPX2's (Pydantic's maintained HTTPX, as the OpenAI and Anthropic SDKs use).
183
+ Proxies come from `HTTPS_PROXY` and `NO_PROXY`, and TLS is checked against the operating system's
184
+ trust store, so a CA your IT installed works as is. For a private CA, a client certificate or a
185
+ proxy in code, configure HTTPX2 and pass it in; the client you pass stays yours to close:
186
+
187
+ ```python
188
+ import ssl
189
+
190
+ import httpx2
191
+
192
+ tls = ssl.create_default_context(cafile="lab-ca.pem")
193
+ tls.load_cert_chain("robot.pem", "robot.key")
194
+ client = abstract.Client(http_client=httpx2.Client(verify=tls, proxy="http://proxy.lab:3128"))
195
+ ```
196
+
197
+ ### Direct sessions and the time profile
198
+
199
+ ```python
200
+ with client.connect("allenai/molmoact2", "fold the towel") as policy:
201
+ chunk = policy.predict(frames, state)
202
+ chunk.stages # this chunk's milliseconds: total, client, network, server and its parts
203
+ print(policy.profile) # the episode: each stage's p50, p95 and max, and what happened on the way
204
+ ```
205
+
206
+ Where the API names an engine for a model, an episode's chunks go straight to that engine and
207
+ no longer cross the API. There is nothing to set: the API holds a block of chunks on your
208
+ account and signs a ticket for it, and each `predict()` posts to the engine under that ticket.
209
+ The same `Chunk`, the same errors, an observation resent under its own id and never run or
210
+ billed twice. Your one choice stays `transport`:
211
+
212
+ - `"http"`: HTTPS to the engine's host.
213
+ - `"quic"`: the engine's own QUIC socket, which the client reaches by UDP hole punching. An
214
+ episode's first chunk goes over HTTPS; the punch then runs in the background, and the chunks
215
+ that follow travel over QUIC. Where a network blocks UDP or a NAT does not allow the punch,
216
+ or the connection is later lost, the episode keeps to HTTPS without a word.
217
+
218
+ What happens on its own:
219
+
220
+ - The next ticket is asked for in the background while chunks remain, so `predict()` does not
221
+ wait on it.
222
+ - An engine that fails twice in a row, or stops answering for 10 s after it has answered, is
223
+ left: the API names the next one and the episode continues there, and when none is left it
224
+ continues through the API. A model without episode memory moves unnoticed, its `step`
225
+ restarting where the new engine is; a model that keeps its episode on the GPU raises
226
+ `SessionExpiredError`, as it does when its replica is lost.
227
+ - A model the API names no engine for runs through the API as ever, and so do FLUX, whose
228
+ content filters the API applies to every chunk, and your fine-tunes. A client asks about a
229
+ model once, not once an episode. `AsyncClient` always goes through the API.
230
+
231
+ What to know:
232
+
233
+ - A direct episode's observations and answers are not stored with your account's trajectories.
234
+ - The engine's hosts end in `modal.run`: a firewall that allows `api.abstract.inc` alone sends
235
+ the first episode back through the API after about four seconds, and the next ones at once.
236
+ - To punch, the client asks a public STUN server (`stun.cloudflare.com`, then
237
+ `stun.l.google.com`) what address its UDP socket is seen at. That request carries nothing
238
+ but the question; the server sees the machine's public address.
239
+
240
+ `policy.profile` is kept for every episode. `client` is the time to build the request, `server`
241
+ the engine's from receipt to answer (`decode`, `queue`, `prefill` and `denoise` are its parts,
242
+ `queue` counted from receipt), and `network` the rest: both ways, and the API when the chunk
243
+ crossed it. `policy.profile.summary()` returns the same as a dict, with which way each chunk
244
+ travelled (`api`, `https`, `quic`) and `events`: a ticket opened or renewed, a QUIC path punched
245
+ or given up for HTTPS, a move to another engine, a return to the API.
246
+
247
+ ### Transports
248
+
249
+ `transport` chooses how the calls travel. `"http"` is the default and what everything above
250
+ describes. `"quic"` sends the same calls to the same routes over HTTP/3, on one QUIC connection
251
+ that every policy on the client shares; it is opt-in and needs the `quic` extra:
252
+
253
+ ```sh
254
+ uv add 'abstract-client[quic]'
255
+ ```
256
+
257
+ ```python
258
+ client = abstract.Client(transport="quic") # or ABSTRACT_TRANSPORT=quic; every client takes it
259
+ ```
260
+
261
+ The API sees the same request either way, so chunks are billed the same and a call that gets no
262
+ answer is sent again under the same rules. What differs is the network's side:
263
+
264
+ - QUIC needs outbound UDP to the API on port 443. Where a network blocks it, calls fail with
265
+ `APIConnectionError` or `APITimeoutError`; nothing switches to `"http"` on its own.
266
+ - No proxy can carry it. On a machine set to use one (`HTTPS_PROXY`, `ALL_PROXY` or the system's)
267
+ the client raises `ConfigError` rather than go around the proxy, and `http_client=` cannot be
268
+ combined with it.
269
+ - TLS is checked against certifi's CA bundle, not the operating system's trust store: a CA your
270
+ IT installed is not trusted, and there is no setting for a private CA or a client certificate.
271
+ - A connection that hears nothing for 20 s is closed and the next call opens another. While a
272
+ call waits for its answer the client pings every 5 s, so a model's cold start does not end it.
273
+
274
+ It runs aioquic's QUIC stack as it comes, with no tuning, and has not been measured against
275
+ `"http"`: choose it to compare the two on your own network, not on a promise that it is faster.
276
+ `client.transport` says which one a client uses.
277
+
278
+ `connect(model, task, scaling={"seed": 7})` (or `scaling` in a `RobotClient` config) sends the
279
+ API's `scaling` with every observation, so a seedable model replays its noise; the engine's reports
280
+ (`scaling`, `cache`, `context`) come back in `chunk.raw`.
281
+
282
+ Each chunk is billed and can wake a GPU. A call that gets no answer is sent again with the same
283
+ bytes and `observation_id`, so the server answers it from its record and never bills it twice.
284
+ Refusals the server marks retryable are sent again after its `Retry-After`, up to `max_retries`;
285
+ `timeout` bounds a whole call, resends included, and `connect_timeout` opening a connection.
286
+
287
+ Every error is an `abstract.AbstractError`. A refusal is an `abstract.APIStatusError` subclass named
288
+ for its status (`AuthenticationError`, `PaymentRequiredError`, `RateLimitError`, ...) with the
289
+ server's `code`, `retryable`, `retry_after` and `request_id`; a refused or answered
290
+ `get_action` also carries its reply's `trace_id`, which a replay keeps. `APIConnectionError` and
291
+ `APITimeoutError` mean no answer came: the next `predict()` asks for it first. The `abstract.client`
292
+ logger reports resends; it never logs the key.
293
+
294
+ Python 3.10 or later; its one dependency is `httpx2` (`aioquic` with the `quic` extra). Tests: `pytest apps/atoms` (the contract
295
+ tests need Bun and the engine's packages: `uv run --project apps/engine pytest apps/atoms`).
296
+
297
+ Version 0.3 replaces `import atoms` with `import abstract`; the buffered harness is
298
+ `abstract.RobotClient` (previously `AtomsClient`). `abstract.client(...)` is the recommended
299
+ entry point. The Python station uses this same implementation; there is no separate YAM runner.