vjepa-byor 0.1.0__tar.gz

vjepa_byor-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Ampyard
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

vjepa_byor-0.1.0/PKG-INFO

@@ -0,0 +1,203 @@
+Metadata-Version: 2.4
+Name: vjepa-byor
+Version: 0.1.0
+Summary: Bring Your Own Robot: fine-tune V-JEPA 2-AC to predict what your robot's actions will do.
+License-Expression: MIT
+License-File: LICENSE
+Requires-Dist: pillow>=10
+Requires-Dist: torch>=2.4
+Requires-Dist: numpy ; extra == 'ros2'
+Requires-Dist: pyyaml ; extra == 'ros2'
+Requires-Dist: pytest>=8 ; extra == 'testing'
+Requires-Dist: einops>=0.7 ; extra == 'vjepa'
+Requires-Dist: timm>=1.0 ; extra == 'vjepa'
+Requires-Python: >=3.11
+Project-URL: Homepage, https://github.com/ampyard/vjepa-byor
+Project-URL: Repository, https://github.com/ampyard/vjepa-byor
+Project-URL: Issues, https://github.com/ampyard/vjepa-byor/issues
+Provides-Extra: ros2
+Provides-Extra: testing
+Provides-Extra: vjepa
+Description-Content-Type: text/markdown
+
+# vjepa-byor
+
+**Bring Your Own Robot:** fine-tune V-JEPA 2-AC to predict what your robot's actions will do.
+
+## What is a rig?
+
+A **rig** is your robot setup: the actuators that move, the controller that drives them, and the
+camera that watches the result. A desk-sized XY stage with stepper motors, an ESP32 and a webcam is a
+rig, and so is a simulated one. "Robot" is the public-facing word; "rig" is the term used in the code
+(`Rig`, `RigState`, `--rig`).
+
+The library has no hardware code of its own. To use it with your robot, you describe your rig to it:
+which actions it can take, what its position looks like, how to talk to it, and how to see it.
+
+## Adding your rig
+
+A rig plugin implements the protocols in `vjepa_byor.interfaces`, bundles them in a `vjepa_byor.Rig`,
+and registers it under the `vjepa_byor.rigs` entry-point group:
+
+```toml
+[project.entry-points."vjepa_byor.rigs"]
+my-rig = "my_rig_package:make_rig"
+```
+
+The rig's name must match the name it is registered under: `make_rig` above returns
+`Rig(name="my-rig", ...)`, otherwise `load_rig("my-rig")` refuses it.
+
+There are two ways to talk to the hardware:
+
+- **Text commands** (serial, WiFi, Bluetooth): give the rig a `CommandCodec`, which turns an action into
+  a command line and the reply back into a state, and a `Transport` that carries the lines.
+- **Action tensors** (ROS 2, a vendor SDK, ...): give the rig an `ActionTransport` and no codec. Its
+  `execute(action, timeout)` carries out the move and returns the state the rig reports.
+
+A ROS 2 robot needs no plugin: the built-in `ros2` rig is described by a YAML file
+(`--rig ros2 --rig-config robot.yaml`, see `vjepa_byor.ros2.config`). Install with
+`pip install vjepa-byor[ros2]` in a venv created with `--system-site-packages` from a sourced ROS 2
+environment, since `rclpy` is not on PyPI. Without `rclpy`, `available_rigs()` does not list `ros2`.
+
+Validate it without hardware by subclassing `vjepa_byor.testing.conformance.RigConformanceTests`
+(install with `pip install vjepa-byor[testing]`) and backing the rig with a
+`vjepa_byor.testing.FakeTransport` (or `FakeActionTransport`) that answers like your device.
+
+## Trying it without hardware
+
+The library includes `sim`, a simulated XY stage with a rendered top-down camera, so collection,
+fine-tuning and planning run end to end with nothing attached. The examples below use it; with your own
+rig, use its registered name in place of `sim`.
+
+```
+pip install vjepa-byor
+vjepa-byor collect --rig sim --out data/sim-try --num-samples 200 --settle-time 0
+```
+
+- **Actions:** 8 compass directions × 3 step sizes (1, 5 and 20 mm by default), as `[dx_mm, dy_mm]`
+- **State:** `XYState(x_mm, y_mm)`, reported after every move; travel is 200 × 150 mm, starting in the middle
+- **V-JEPA mapping:** millimetres to metres in the end effector's x and y slots: `[dx, dy, 0, 0, 0, 0, 0]`
+- **Wire format:** the JSON-line codec over an in-memory transport; out-of-range moves are refused with
+  `{"error": "out of range"}` and the carriage stays put
+- **Camera:** 256 × 256 RGB `uint8` by default: an orange puck on a gridded mat, with a red and a blue
+  landmark block. Anti-aliased, so a 1 mm move changes the photo; deterministic, so a position always
+  gives the same photo
+
+Options (`--rig-config sim.json`): `step_sizes_mm`, `travel_mm`, `start_mm`, `resolution` (`[height, width]`),
+`reply_timeout`.
+
+```python
+from vjepa_byor import load_rig
+
+with load_rig("sim") as rig:
+    print(rig.execute(rig.action_space.by_label("NE 5mm")))   # XYState(x_mm=105.0, y_mm=80.0)
+    photo = rig.camera.capture()                               # uint8 (3, 256, 256)
+```
+
+It is built only from the library's public building blocks (`vjepa_byor.sim`), so it doubles as an
+example of a rig.
+
+### The sim as a ROS 2 robot
+
+`python -m vjepa_byor.sim.ros2_bridge` runs the sim as an ordinary ROS 2 node (needs a sourced ROS 2
+environment and `vjepa-byor[ros2]`):
+
+```bash
+python -m vjepa_byor.sim.ros2_bridge --ros-args -p start_mm:="[40.0, 100.0]"
+ros2 topic pub --once /rig/cmd geometry_msgs/msg/Vector3Stamped "{vector: {x: 0.02, y: -0.02}}"
+ros2 topic echo /rig/state
+```
+
+| Name | Type | |
+|---|---|---|
+| `/camera/image_raw` | `sensor_msgs/Image` | `rgb8`, 256 × 256 |
+| `/rig/state` | `sensor_msgs/JointState` | joints `x`, `y`, positions in metres |
+| `/rig/cmd` (subscribed) | `geometry_msgs/Vector3Stamped` | relative move, `vector.x` / `vector.y` in metres |
+| `/rig/reset` (service) | `std_srvs/Trigger` | carriage back to `start_mm` |
+
+Photo and state share one `header.stamp` and go out at `rate_hz` (10) and after every command, including
+a refused one, which leaves the position unchanged. Parameters: `travel_mm`, `start_mm`, `resolution`,
+`rate_hz`, `camera_frame`, `stage_frame`.
+
+## Collecting a fine-tuning dataset
+
+`vjepa-byor collect` drives any registered rig through short episodes of random actions and records every
+transition as (before photo, rig state, action, after photo):
+
+```
+vjepa-byor collect --rig sim --out data/sim-run1 \
+    --num-samples 1000 --steps-per-episode 20 --seed 0 --settle-time 0.5
+vjepa-byor collect --rig sim --out data/sim-run1 --num-samples 2000 --resume   # adds 1000 more
+vjepa-byor collect --verify data/sim-run1
+```
+
+`--num-samples` is the target size of the whole dataset, so a resumed run only collects what is missing.
+`--rig-config rig.json` passes options to the rig's factory, and `--state-bounds 0:200,0:150` tells the
+random walk the limits of the state tensor so it steers away from them (without bounds it learns from
+rejections). Ctrl-C finishes the step in flight and stops cleanly; a second Ctrl-C aborts.
+
+The rig needs a camera (`Rig.camera`). Since the interfaces have no "home" or "read position" command,
+the collector learns where the rig is from its reply to a first, unrecorded move.
+
+### Dataset layout
+
+```
+<dataset>/
+├── meta.json         # schema_version, rig, action vocabulary, resolution [height, width], ...
+├── manifest.jsonl    # one sample per line, append-only
+├── events.jsonl      # rejections, failures, syncs, resumptions
+├── splits.json       # episode -> train / val / test (created on first read)
+└── frames/ep000000/000.png, 001.png, ...   # lossless PNGs exactly as captured
+```
+
+Step *k* of an episode has frame *k* as its before photo and frame *k+1* as its after photo, so photos are
+stored once. The full format contract is in the docstring of `vjepa_byor.data.storage`.
+
+Each manifest line is a sample:
+
+| Field | Meaning |
+|---|---|
+| `sample_id`, `episode_id`, `step_index` | e.g. `ep000012_s0003`, `ep000012`, `3` |
+| `before_frame`, `after_frame` | PNG paths relative to the dataset root |
+| `state_before`, `state_after` | the rig's reported state, as `RigState.to_dict()` |
+| `state_before_tensor`, `state_after_tensor` | the same, as `RigState.as_tensor()` |
+| `action_index`, `action_name`, `action` | index into the vocabulary, its label, the action tensor |
+| `command` | the wire line sent to the rig |
+| `status` | `ok`, or `rejected` if the rig reported an error (the state is then unchanged) |
+| `changed` | `state_after != state_before`, from the reported states |
+| `resolution`, `timestamp`, `error` | `[height, width]`, ISO 8601 UTC, the rig's error message if rejected |
+
+Read it back with `vjepa_byor.data.CollectedDataset(root, split="train", drop_no_change=False)`; splits are by
+episode and never change once written. `vjepa_byor.data.export(root, out, split="train")` copies a split into
+a new self-contained dataset.
+
+Plugin authors can check the collector drives their rig by adding
+`vjepa_byor.testing.collector_conformance.CollectorConformanceTests` next to `RigConformanceTests`; it needs
+`rig.camera`, and `vjepa_byor.testing.fake_camera.MarkerCamera` can draw the emulated position.
+
+## Fine-tuning the predictor
+
+Only V-JEPA 2-AC's action-conditioned predictor is fine-tuned; the ViT-g encoder stays frozen. Install the
+model code's dependencies with `pip install vjepa-byor[vjepa]`. The weights are not bundled (they have their
+own license): download `vjepa2-ac-vitg.pt` (about 11 GB) from `https://dl.fbaipublicfiles.com/vjepa2/` to
+`~/.cache/torch/hub/checkpoints/`, or pass `--checkpoint PATH`.
+
+```
+vjepa-byor encode   --data data/sim-run1                        # frozen encoder, once per frame
+vjepa-byor finetune --data data/sim-run1 --out checkpoints/sim  # train split; best epoch on val is kept
+vjepa-byor eval     --data data/sim-run1 --finetuned checkpoints/sim --split test
+```
+
+- **encode** writes `<dataset>/latents/latents.pt`: every frame's layer-normalised latents (256 tokens × 1408,
+  bf16). A rerun after `collect --resume` encodes only the new frames.
+- **finetune** trains on the latents, so the encoder is never in GPU memory. The objective is upstream's
+  teacher-forced step: before latents + mapped action + mapped state → after latents, L1 loss. bf16 autocast,
+  AdamW, warmup + cosine, activation checkpointing. `--lora-rank 16` trains LoRA adapters instead of every
+  weight; `--max-vram-gb 8` caps GPU memory to check a run fits an 8 GB card. The rig's `VJepaMapping` comes
+  from the rig the dataset was collected with (`--rig` / `--rig-config` to override); a mapping that is not
+  7-D gets fresh, custom-sized action and state input layers.
+- **eval** reports the mean L1 error of the fine-tuned predictor, the original one and a "no change" baseline
+  (the before latents), over all samples and over those where the rig moved.
+
+In code, `vjepa_byor.predictor.ActionPredictor(encoder, predictor, rig.mapping).predict_basket(photo, state,
+rig.action_space)` predicts the next-state latents for every action in one batch.

vjepa_byor-0.1.0/README.md

@@ -0,0 +1,181 @@
+# vjepa-byor
+
+**Bring Your Own Robot:** fine-tune V-JEPA 2-AC to predict what your robot's actions will do.
+
+## What is a rig?
+
+A **rig** is your robot setup: the actuators that move, the controller that drives them, and the
+camera that watches the result. A desk-sized XY stage with stepper motors, an ESP32 and a webcam is a
+rig, and so is a simulated one. "Robot" is the public-facing word; "rig" is the term used in the code
+(`Rig`, `RigState`, `--rig`).
+
+The library has no hardware code of its own. To use it with your robot, you describe your rig to it:
+which actions it can take, what its position looks like, how to talk to it, and how to see it.
+
+## Adding your rig
+
+A rig plugin implements the protocols in `vjepa_byor.interfaces`, bundles them in a `vjepa_byor.Rig`,
+and registers it under the `vjepa_byor.rigs` entry-point group:
+
+```toml
+[project.entry-points."vjepa_byor.rigs"]
+my-rig = "my_rig_package:make_rig"
+```
+
+The rig's name must match the name it is registered under: `make_rig` above returns
+`Rig(name="my-rig", ...)`, otherwise `load_rig("my-rig")` refuses it.
+
+There are two ways to talk to the hardware:
+
+- **Text commands** (serial, WiFi, Bluetooth): give the rig a `CommandCodec`, which turns an action into
+  a command line and the reply back into a state, and a `Transport` that carries the lines.
+- **Action tensors** (ROS 2, a vendor SDK, ...): give the rig an `ActionTransport` and no codec. Its
+  `execute(action, timeout)` carries out the move and returns the state the rig reports.
+
+A ROS 2 robot needs no plugin: the built-in `ros2` rig is described by a YAML file
+(`--rig ros2 --rig-config robot.yaml`, see `vjepa_byor.ros2.config`). Install with
+`pip install vjepa-byor[ros2]` in a venv created with `--system-site-packages` from a sourced ROS 2
+environment, since `rclpy` is not on PyPI. Without `rclpy`, `available_rigs()` does not list `ros2`.
+
+Validate it without hardware by subclassing `vjepa_byor.testing.conformance.RigConformanceTests`
+(install with `pip install vjepa-byor[testing]`) and backing the rig with a
+`vjepa_byor.testing.FakeTransport` (or `FakeActionTransport`) that answers like your device.
+
+## Trying it without hardware
+
+The library includes `sim`, a simulated XY stage with a rendered top-down camera, so collection,
+fine-tuning and planning run end to end with nothing attached. The examples below use it; with your own
+rig, use its registered name in place of `sim`.
+
+```
+pip install vjepa-byor
+vjepa-byor collect --rig sim --out data/sim-try --num-samples 200 --settle-time 0
+```
+
+- **Actions:** 8 compass directions × 3 step sizes (1, 5 and 20 mm by default), as `[dx_mm, dy_mm]`
+- **State:** `XYState(x_mm, y_mm)`, reported after every move; travel is 200 × 150 mm, starting in the middle
+- **V-JEPA mapping:** millimetres to metres in the end effector's x and y slots: `[dx, dy, 0, 0, 0, 0, 0]`
+- **Wire format:** the JSON-line codec over an in-memory transport; out-of-range moves are refused with
+  `{"error": "out of range"}` and the carriage stays put
+- **Camera:** 256 × 256 RGB `uint8` by default: an orange puck on a gridded mat, with a red and a blue
+  landmark block. Anti-aliased, so a 1 mm move changes the photo; deterministic, so a position always
+  gives the same photo
+
+Options (`--rig-config sim.json`): `step_sizes_mm`, `travel_mm`, `start_mm`, `resolution` (`[height, width]`),
+`reply_timeout`.
+
+```python
+from vjepa_byor import load_rig
+
+with load_rig("sim") as rig:
+    print(rig.execute(rig.action_space.by_label("NE 5mm")))   # XYState(x_mm=105.0, y_mm=80.0)
+    photo = rig.camera.capture()                               # uint8 (3, 256, 256)
+```
+
+It is built only from the library's public building blocks (`vjepa_byor.sim`), so it doubles as an
+example of a rig.
+
+### The sim as a ROS 2 robot
+
+`python -m vjepa_byor.sim.ros2_bridge` runs the sim as an ordinary ROS 2 node (needs a sourced ROS 2
+environment and `vjepa-byor[ros2]`):
+
+```bash
+python -m vjepa_byor.sim.ros2_bridge --ros-args -p start_mm:="[40.0, 100.0]"
+ros2 topic pub --once /rig/cmd geometry_msgs/msg/Vector3Stamped "{vector: {x: 0.02, y: -0.02}}"
+ros2 topic echo /rig/state
+```
+
+| Name | Type | |
+|---|---|---|
+| `/camera/image_raw` | `sensor_msgs/Image` | `rgb8`, 256 × 256 |
+| `/rig/state` | `sensor_msgs/JointState` | joints `x`, `y`, positions in metres |
+| `/rig/cmd` (subscribed) | `geometry_msgs/Vector3Stamped` | relative move, `vector.x` / `vector.y` in metres |
+| `/rig/reset` (service) | `std_srvs/Trigger` | carriage back to `start_mm` |
+
+Photo and state share one `header.stamp` and go out at `rate_hz` (10) and after every command, including
+a refused one, which leaves the position unchanged. Parameters: `travel_mm`, `start_mm`, `resolution`,
+`rate_hz`, `camera_frame`, `stage_frame`.
+
+## Collecting a fine-tuning dataset
+
+`vjepa-byor collect` drives any registered rig through short episodes of random actions and records every
+transition as (before photo, rig state, action, after photo):
+
+```
+vjepa-byor collect --rig sim --out data/sim-run1 \
+    --num-samples 1000 --steps-per-episode 20 --seed 0 --settle-time 0.5
+vjepa-byor collect --rig sim --out data/sim-run1 --num-samples 2000 --resume   # adds 1000 more
+vjepa-byor collect --verify data/sim-run1
+```
+
+`--num-samples` is the target size of the whole dataset, so a resumed run only collects what is missing.
+`--rig-config rig.json` passes options to the rig's factory, and `--state-bounds 0:200,0:150` tells the
+random walk the limits of the state tensor so it steers away from them (without bounds it learns from
+rejections). Ctrl-C finishes the step in flight and stops cleanly; a second Ctrl-C aborts.
+
+The rig needs a camera (`Rig.camera`). Since the interfaces have no "home" or "read position" command,
+the collector learns where the rig is from its reply to a first, unrecorded move.
+
+### Dataset layout
+
+```
+<dataset>/
+├── meta.json         # schema_version, rig, action vocabulary, resolution [height, width], ...
+├── manifest.jsonl    # one sample per line, append-only
+├── events.jsonl      # rejections, failures, syncs, resumptions
+├── splits.json       # episode -> train / val / test (created on first read)
+└── frames/ep000000/000.png, 001.png, ...   # lossless PNGs exactly as captured
+```
+
+Step *k* of an episode has frame *k* as its before photo and frame *k+1* as its after photo, so photos are
+stored once. The full format contract is in the docstring of `vjepa_byor.data.storage`.
+
+Each manifest line is a sample:
+
+| Field | Meaning |
+|---|---|
+| `sample_id`, `episode_id`, `step_index` | e.g. `ep000012_s0003`, `ep000012`, `3` |
+| `before_frame`, `after_frame` | PNG paths relative to the dataset root |
+| `state_before`, `state_after` | the rig's reported state, as `RigState.to_dict()` |
+| `state_before_tensor`, `state_after_tensor` | the same, as `RigState.as_tensor()` |
+| `action_index`, `action_name`, `action` | index into the vocabulary, its label, the action tensor |
+| `command` | the wire line sent to the rig |
+| `status` | `ok`, or `rejected` if the rig reported an error (the state is then unchanged) |
+| `changed` | `state_after != state_before`, from the reported states |
+| `resolution`, `timestamp`, `error` | `[height, width]`, ISO 8601 UTC, the rig's error message if rejected |
+
+Read it back with `vjepa_byor.data.CollectedDataset(root, split="train", drop_no_change=False)`; splits are by
+episode and never change once written. `vjepa_byor.data.export(root, out, split="train")` copies a split into
+a new self-contained dataset.
+
+Plugin authors can check the collector drives their rig by adding
+`vjepa_byor.testing.collector_conformance.CollectorConformanceTests` next to `RigConformanceTests`; it needs
+`rig.camera`, and `vjepa_byor.testing.fake_camera.MarkerCamera` can draw the emulated position.
+
+## Fine-tuning the predictor
+
+Only V-JEPA 2-AC's action-conditioned predictor is fine-tuned; the ViT-g encoder stays frozen. Install the
+model code's dependencies with `pip install vjepa-byor[vjepa]`. The weights are not bundled (they have their
+own license): download `vjepa2-ac-vitg.pt` (about 11 GB) from `https://dl.fbaipublicfiles.com/vjepa2/` to
+`~/.cache/torch/hub/checkpoints/`, or pass `--checkpoint PATH`.
+
+```
+vjepa-byor encode   --data data/sim-run1                        # frozen encoder, once per frame
+vjepa-byor finetune --data data/sim-run1 --out checkpoints/sim  # train split; best epoch on val is kept
+vjepa-byor eval     --data data/sim-run1 --finetuned checkpoints/sim --split test
+```
+
+- **encode** writes `<dataset>/latents/latents.pt`: every frame's layer-normalised latents (256 tokens × 1408,
+  bf16). A rerun after `collect --resume` encodes only the new frames.
+- **finetune** trains on the latents, so the encoder is never in GPU memory. The objective is upstream's
+  teacher-forced step: before latents + mapped action + mapped state → after latents, L1 loss. bf16 autocast,
+  AdamW, warmup + cosine, activation checkpointing. `--lora-rank 16` trains LoRA adapters instead of every
+  weight; `--max-vram-gb 8` caps GPU memory to check a run fits an 8 GB card. The rig's `VJepaMapping` comes
+  from the rig the dataset was collected with (`--rig` / `--rig-config` to override); a mapping that is not
+  7-D gets fresh, custom-sized action and state input layers.
+- **eval** reports the mean L1 error of the fine-tuned predictor, the original one and a "no change" baseline
+  (the before latents), over all samples and over those where the rig moved.
+
+In code, `vjepa_byor.predictor.ActionPredictor(encoder, predictor, rig.mapping).predict_basket(photo, state,
+rig.action_space)` predicts the next-state latents for every action in one batch.

vjepa_byor-0.1.0/pyproject.toml

@@ -0,0 +1,35 @@
+[project]
+name = "vjepa-byor"
+version = "0.1.0"
+description = "Bring Your Own Robot: fine-tune V-JEPA 2-AC to predict what your robot's actions will do."
+readme = "README.md"
+license = "MIT"
+license-files = ["LICENSE"]
+requires-python = ">=3.11"
+dependencies = [
+    "pillow>=10",
+    "torch>=2.4",
+]
+
+[project.scripts]
+vjepa-byor = "vjepa_byor.cli:main"
+
+[project.optional-dependencies]
+vjepa = [
+    "einops>=0.7",
+    "timm>=1.0",
+]
+testing = ["pytest>=8"]
+ros2 = [
+    "numpy",
+    "pyyaml",
+]
+
+[project.urls]
+Homepage = "https://github.com/ampyard/vjepa-byor"
+Repository = "https://github.com/ampyard/vjepa-byor"
+Issues = "https://github.com/ampyard/vjepa-byor/issues"
+
+[build-system]
+requires = ["uv_build>=0.8,<0.14"]
+build-backend = "uv_build"

vjepa_byor-0.1.0/pyproject.toml.orig

@@ -0,0 +1,32 @@
+[project]
+name = "vjepa-byor"
+version = "0.1.0"
+description = "Bring Your Own Robot: fine-tune V-JEPA 2-AC to predict what your robot's actions will do."
+readme = "README.md"
+license = "MIT"
+license-files = ["LICENSE"]
+requires-python = ">=3.11"
+dependencies = [
+    "pillow>=10",
+    "torch>=2.4",
+]
+
+[project.scripts]
+vjepa-byor = "vjepa_byor.cli:main"
+
+[project.optional-dependencies]
+# Meta's V-JEPA 2-AC model code (loaded through torch.hub) needs these
+vjepa = ["einops>=0.7", "timm>=1.0"]
+testing = ["pytest>=8"]
+# The built-in ros2 rig and ROS 2 tools: their pip-installable dependencies. rclpy and the message
+# packages come from a sourced ROS 2 environment (not PyPI); unpinned so ROS's own numpy satisfies it
+ros2 = ["numpy", "pyyaml"]
+
+[project.urls]
+Homepage = "https://github.com/ampyard/vjepa-byor"
+Repository = "https://github.com/ampyard/vjepa-byor"
+Issues = "https://github.com/ampyard/vjepa-byor/issues"
+
+[build-system]
+requires = ["uv_build>=0.8,<0.14"]
+build-backend = "uv_build"

vjepa_byor-0.1.0/src/vjepa_byor/__init__.py

@@ -0,0 +1,39 @@
+"""vjepa-byor — Bring Your Own Robot: fine-tune V-JEPA 2-AC to predict what your robot's actions will do."""
+
+from vjepa_byor.interfaces import (
+    ActionSpace,
+    ActionTransport,
+    Camera,
+    CodecError,
+    CommandCodec,
+    RigError,
+    RigState,
+    Transport,
+    VJepaMapping,
+)
+from vjepa_byor.registry import (
+    RigNotFoundError,
+    available_rigs,
+    load_rig,
+    register_rig,
+    unregister_rig,
+)
+from vjepa_byor.rig import Rig
+
+__all__ = [
+    "ActionSpace",
+    "ActionTransport",
+    "Camera",
+    "CodecError",
+    "CommandCodec",
+    "Rig",
+    "RigError",
+    "RigNotFoundError",
+    "RigState",
+    "Transport",
+    "VJepaMapping",
+    "available_rigs",
+    "load_rig",
+    "register_rig",
+    "unregister_rig",
+]

vjepa_byor-0.1.0/src/vjepa_byor/actions/__init__.py

@@ -0,0 +1,19 @@
+"""Generic building blocks for a rig's actions, state, V-JEPA mapping and wire format.
+
+Rig plugins combine these with their own settings; none of them is specific to
+one piece of hardware.
+"""
+
+from vjepa_byor.actions.codecs import JsonLineCodec
+from vjepa_byor.actions.state import VectorState
+from vjepa_byor.actions.vjepa_mapping import VJEPA_AC_DIM, SlotMapping
+from vjepa_byor.actions.vocabulary import COMPASS, DiscreteActionSpace
+
+__all__ = [
+    "COMPASS",
+    "VJEPA_AC_DIM",
+    "DiscreteActionSpace",
+    "JsonLineCodec",
+    "SlotMapping",
+    "VectorState",
+]

vjepa_byor-0.1.0/src/vjepa_byor/actions/codecs/__init__.py

@@ -0,0 +1,5 @@
+"""Reusable wire formats for :class:`~vjepa_byor.interfaces.CommandCodec`."""
+
+from vjepa_byor.actions.codecs.json_line import JsonLineCodec
+
+__all__ = ["JsonLineCodec"]

vjepa_byor-0.1.0/src/vjepa_byor/actions/codecs/json_line.py

@@ -0,0 +1,69 @@
+"""A JSON-per-line wire format, for rigs whose controller can parse JSON (or are simulated).
+
+Commands and replies are single-line JSON objects::
+
+    {"action": [5, -5]}             command: one action tensor
+    {"state": {"x": 30, "y": 15}}   reply: the new state, as RigState.to_dict()
+    {"error": "out of range"}       reply: the rig refused or failed
+"""
+
+from __future__ import annotations
+
+import json
+import math
+from typing import Any
+
+import torch
+
+from vjepa_byor.interfaces import CodecError, RigError, RigState
+
+
+def _number(value: float) -> int | float:
+    return int(value) if float(value).is_integer() else value
+
+
+class JsonLineCodec:
+    """Implements :class:`~vjepa_byor.interfaces.CommandCodec` for any state type and action size."""
+
+    def __init__(self, state_type: type[RigState], action_dim: int) -> None:
+        self.state_type = state_type
+        self.action_dim = action_dim
+
+    @staticmethod
+    def _parse(line: str) -> dict[str, Any]:
+        try:
+            data = json.loads(line)
+        except json.JSONDecodeError as e:
+            raise CodecError(f"not JSON: {line!r}") from e
+        if not isinstance(data, dict):
+            raise CodecError(f"not a JSON object: {line!r}")
+        return data
+
+    def encode_action(self, action: torch.Tensor) -> str:
+        if action.shape != (self.action_dim,):
+            raise ValueError(f"expected an action of shape ({self.action_dim},), got {tuple(action.shape)}")
+        return json.dumps({"action": [_number(v) for v in action.tolist()]})
+
+    def decode_action(self, line: str) -> torch.Tensor:
+        values = self._parse(line).get("action")
+        if not (
+            isinstance(values, list)
+            and len(values) == self.action_dim
+            and all(isinstance(v, int | float) and not isinstance(v, bool) and math.isfinite(v) for v in values)
+        ):
+            raise CodecError(f"bad command: {line!r}")
+        return torch.tensor([float(v) for v in values])
+
+    def encode_state(self, state: RigState) -> str:
+        return json.dumps({"state": state.to_dict()})
+
+    def decode_state(self, line: str) -> RigState:
+        data = self._parse(line)
+        if "error" in data:
+            raise RigError(str(data["error"]))
+        if not isinstance(data.get("state"), dict):
+            raise CodecError(f"bad reply: {line!r}")
+        try:
+            return self.state_type.from_dict(data["state"])
+        except (KeyError, TypeError, ValueError) as e:
+            raise CodecError(f"bad state in reply {line!r}: {e}") from e

vjepa_byor-0.1.0/src/vjepa_byor/actions/state.py

@@ -0,0 +1,47 @@
+"""Rig state schema: a base class for states that are a few named numbers."""
+
+from __future__ import annotations
+
+from dataclasses import asdict, dataclass, fields
+from typing import Any, Self
+
+import torch
+
+
+@dataclass(frozen=True)
+class VectorState:
+    """Base for rig states made of numeric fields; implements :class:`~vjepa_byor.interfaces.RigState`.
+
+    Subclass it as a frozen dataclass; the fields, in declaration order, become
+    the state tensor::
+
+        @dataclass(frozen=True)
+        class XYState(VectorState):
+            x_mm: float
+            y_mm: float
+    """
+
+    @classmethod
+    def field_names(cls) -> list[str]:
+        return [f.name for f in fields(cls)]
+
+    def as_tensor(self) -> torch.Tensor:
+        return torch.tensor([float(getattr(self, name)) for name in self.field_names()])
+
+    def to_dict(self) -> dict[str, Any]:
+        return asdict(self)
+
+    @classmethod
+    def from_dict(cls, data: dict[str, Any]) -> Self:
+        missing = [name for name in cls.field_names() if name not in data]
+        if missing:
+            raise ValueError(f"{cls.__name__} is missing {', '.join(missing)}")
+        return cls(**{name: data[name] for name in cls.field_names()})
+
+    @classmethod
+    def from_tensor(cls, tensor: torch.Tensor) -> Self:
+        """Inverse of :meth:`as_tensor`; values come back as Python floats."""
+        names = cls.field_names()
+        if tensor.shape != (len(names),):
+            raise ValueError(f"{cls.__name__} needs a tensor of shape ({len(names)},), got {tuple(tensor.shape)}")
+        return cls(**dict(zip(names, tensor.tolist())))