vlastudio 0.2.0__tar.gz

vlastudio-0.2.0/.gitignore

@@ -0,0 +1,52 @@
+ckpt/
+ckpt
+/checkpoints/
+results/
+hub/
+assets/robot/
+policy/*gc*
+data/
+\.gradio
+core\.*
+data
+__pycache__/
+.*__pycache__/.*
+\.local/
+tmp_scripts/*
+scripts/*
+!scripts/dataset_visualizer.py
+!scripts/*example*.sh
+!scripts/mp4_to_gif.py
+!scripts/push_to_hub.py
+!scripts/download_dataset.py
+!scripts/evaluate_model.py
+logs/
+vla_backup/
+.vscode/
+eval_real_old.py
+outputs/
+.idea/
+.cursor/
+gyx/*
+wyb/*
+num2words/
+.venv/
+.venv-*/
+checkpoints/*
+configs/policy/*gc*
+configs/action_manager/so101*
+benchmark/simplerenv/ManiSkill2_real2sim/*
+benchmark/simplerenv/ManiSkill2_real2sim
+policy/*pose*
+policy/*nostate*
+*bilora*
+configs/task/local/
+
+# Local packaging validation environments
+.runtime-test/
+.wheel-test/
+.sdist-test/
+.packaging-outside/
+dist/
+/final-results/
+results/*

vlastudio-0.2.0/LICENSE

@@ -0,0 +1,12 @@
+You are free to:
+Share — copy and redistribute the material in any medium or format
+Adapt — remix, transform, and build upon the material
+The licensor cannot revoke these freedoms as long as you follow the license terms.
+Under the following terms:
+Attribution — You must give appropriate credit , provide a link to the license, and indicate if changes were made . You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.
+NonCommercial — You may not use the material for commercial purposes .
+No additional restrictions — You may not apply legal terms or technological measures that legally restrict others from doing anything the license permits.
+Notices:
+You do not have to comply with the license for elements of the material in the public domain or where your use is permitted by an applicable exception or limitation .
+
+No warranties are given. The license may not give you all of the permissions necessary for your intended use. For example, other rights such as publicity, privacy, or moral rights may limit how you use the material.

vlastudio-0.2.0/PKG-INFO

@@ -0,0 +1,474 @@
+Metadata-Version: 2.5
+Name: vlastudio
+Version: 0.2.0
+Summary: Extensible robotics policy training and deployment
+Project-URL: Homepage, https://github.com/WwZzz/VLAStudio
+Project-URL: Documentation, https://github.com/WwZzz/VLAStudio#readme
+Project-URL: Repository, https://github.com/WwZzz/VLAStudio
+Project-URL: Issues, https://github.com/WwZzz/VLAStudio/issues
+Author: VLAStudio contributors
+License: You are free to:
+        Share — copy and redistribute the material in any medium or format
+        Adapt — remix, transform, and build upon the material
+        The licensor cannot revoke these freedoms as long as you follow the license terms.
+        Under the following terms:
+        Attribution — You must give appropriate credit , provide a link to the license, and indicate if changes were made . You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.
+        NonCommercial — You may not use the material for commercial purposes .
+        No additional restrictions — You may not apply legal terms or technological measures that legally restrict others from doing anything the license permits.
+        Notices:
+        You do not have to comply with the license for elements of the material in the public domain or where your use is permitted by an applicable exception or limitation .
+        
+        No warranties are given. The license may not give you all of the permissions necessary for your intended use. For example, other rights such as publicity, privacy, or moral rights may limit how you use the material.
+License-File: LICENSE
+Keywords: imitation-learning,robotics,vision-language-action
+Classifier: Development Status :: 4 - Beta
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: <3.13,>=3.10
+Requires-Dist: accelerate<=1.12.0,>=1.0.1
+Requires-Dist: av<16,>=12
+Requires-Dist: diffusers<=0.35.1,>=0.30.3
+Requires-Dist: dm-control<=1.0.34,>=1.0.21
+Requires-Dist: editables<1,>=0.3
+Requires-Dist: einops==0.8.1
+Requires-Dist: filelock<4,>=3.16
+Requires-Dist: h5py==3.12.1
+Requires-Dist: huggingface-hub<1,>=0.29
+Requires-Dist: imageio-ffmpeg==0.6.0
+Requires-Dist: imageio==2.37.0
+Requires-Dist: loguru<1,>=0.7
+Requires-Dist: matplotlib==3.9.4
+Requires-Dist: mujoco<=3.3.6,>=3.2.0
+Requires-Dist: numpy==1.26.4
+Requires-Dist: opencv-python<=4.10.0.84,>=4.9
+Requires-Dist: packaging>=24
+Requires-Dist: pillow<13,>=11.3
+Requires-Dist: platformdirs<5,>=4
+Requires-Dist: protobuf==5.29.6
+Requires-Dist: psutil==6.1.1
+Requires-Dist: pyarrow>=14
+Requires-Dist: pyyaml<7,>=6
+Requires-Dist: requests==2.32.5
+Requires-Dist: scipy==1.14.1
+Requires-Dist: tensorboardx==2.6.4
+Requires-Dist: timm<=1.0.15,>=0.9.16
+Requires-Dist: tomli>=2; python_version < '3.11'
+Requires-Dist: torch<2.8,>=2.4
+Requires-Dist: torchdata==0.9.0
+Requires-Dist: torchvision<0.23,>=0.19
+Requires-Dist: tqdm<5,>=4.67
+Requires-Dist: transformers<=4.53.2,>=4.40.1
+Requires-Dist: websockets==13.1
+Provides-Extra: act
+Provides-Extra: aloha
+Provides-Extra: dev
+Requires-Dist: build>=1; extra == 'dev'
+Requires-Dist: hatchling; extra == 'dev'
+Requires-Dist: pytest>=8; extra == 'dev'
+Provides-Extra: remote-eval
+Provides-Extra: serve-http
+Requires-Dist: fastapi==0.115.6; extra == 'serve-http'
+Requires-Dist: uvicorn==0.34.0; extra == 'serve-http'
+Description-Content-Type: text/markdown
+
+<div align="center">
+  <img src='https://raw.githubusercontent.com/WwZzz/myfigs/refs/heads/master/vlastudio.png'  width="200"/>
+<h1> VLAStudio: A Modular Learning Playground for Robotics
+</h1>
+
+
+</div>
+
+
+<table>
+<caption><em>demos generated by policies trained in VLAStudio</em></caption>
+<tr>
+<td align="center">ACT - RLBench</td>
+<td align="center">Diffusion Policy - ALOHA-Sim</td>
+<td align="center">ACT - Square</td>
+<td align="center">FlowMatching - Transport</td>
+<td align="center">SmolVLA - MetaWorld</td>
+</tr>
+<tr>
+<td align="center"><img src="https://raw.githubusercontent.com/WwZzz/myfigs/refs/heads/master/act_rlbench.gif" height="200"></td>
+<td align="center"><img src="https://raw.githubusercontent.com/WwZzz/myfigs/refs/heads/master/diffusion_aloha_transfer.gif" height="200"></td>
+<td align="center"><img src="https://raw.githubusercontent.com/WwZzz/myfigs/refs/heads/master/act_robomimic_square.gif" height="200"></td>
+<td align="center"><img src="https://raw.githubusercontent.com/WwZzz/myfigs/refs/heads/master/flowmatching_transport.gif" height="200"></td>
+<td align="center"><img src="https://raw.githubusercontent.com/WwZzz/myfigs/refs/heads/master/fig_smolvla_mw41.gif" height="200"></td>
+</tr>
+<tr>
+<td align="center">Koch - Pick & Place -Inference</td>
+<td align="center">SO101 - Fold Tower -Inference</td>
+<td align="center">BC_MLP - AdroitHandDoor</td>
+<td align="center">Pi0 - LIBERO</td>
+<td align="center">ACT - RoboTwin</td>
+</tr>
+<tr>
+<td align="center"><img src="https://raw.githubusercontent.com/WwZzz/myfigs/refs/heads/master/fig_koch.gif" height="200"></td>
+<td align="center"><img src="https://raw.githubusercontent.com/WwZzz/myfigs/refs/heads/master/so101_fold.gif" height="200"></td>
+<td align="center"><img src="https://raw.githubusercontent.com/WwZzz/myfigs/refs/heads/master/fig_door.gif" height="200"></td>
+<td align="center"><img src="https://raw.githubusercontent.com/WwZzz/myfigs/refs/heads/master/pi0_libero.gif" height="200"></td>
+<td align="center"><img src="https://raw.githubusercontent.com/WwZzz/myfigs/refs/heads/master/robotwin_act.gif" height="200"></td>
+</tr>
+</table>
+
+
+VLAStudio is a Python library for robot learning. Write your own `run.py` to load demonstrations, train imitation-learning and vision-language-action policies, evaluate in simulation, and deploy on real robots. Compose these steps with ordinary Python functions and run your file with `python run.py`.
+
+- **Train and fine-tune** ACT, Diffusion Policy, SmolVLA, Pi0/Pi0.5, and other policies with a shared Python API.
+- **Combine data sources** through HDF5, LeRobot, and RLDS loaders and shared data processing.
+- **Evaluate and deploy** locally or through a policy server, with simulation benchmarks and real robot integrations.
+- **Extend the library** with your own policy, dataset, simulator, or robot without rewriting the pipeline.
+- **Choose your own Python environment.** Install ACT, Diffusion Policy and ALOHA-Sim with pip; use uv virtual environments and requirements files for other models.
+
+## 🛠️ Installation
+
+### Install the default environment
+
+With Python 3.10–3.12 in an activated environment:
+
+```bash
+python -m pip install vlastudio
+```
+
+For development or to run the repository examples:
+
+Clone the main branch, then install with pip in a uv virtual environment:
+
+```bash
+git clone https://github.com/WwZzz/VLAStudio.git
+cd VLAStudio
+python -m pip install uv
+python -m uv venv --python 3.10 --seed .venv
+source .venv/bin/activate
+python -m pip install -e .
+```
+
+That is the complete Python dependency installation for **ACT, Diffusion Policy,
+shared HDF5/LeRobot parquet and video processing, and ALOHA-Sim**. No additional
+model or simulator package installation is needed for the ACT/DP ALOHA examples.
+Provide demonstrations before training; the policies may download pretrained
+ResNet weights on first use. GPU execution needs a compatible NVIDIA driver,
+and headless rendering needs the host's OpenGL/EGL libraries.
+
+On Windows, activate with `.venv\Scripts\Activate.ps1`. Use `python -m pip install .`
+for a non-editable installation. Without a checkout, install the main branch:
+
+```bash
+python -m pip install "git+https://github.com/WwZzz/VLAStudio.git@main"
+```
+
+`import vlastudio` itself loads only Python's standard library. It never imports
+Torch, starts a simulator, creates an environment, or installs packages.
+
+### Other models: a separate environment and a requirements file
+
+For example, install SmolVLA and LIBERO in a separate environment:
+
+```bash
+deactivate
+python -m uv venv --python 3.10 --seed .venv-smolvla
+source .venv-smolvla/bin/activate
+python -m pip install -e . -r requirements/smolvla.txt -r requirements/libero.txt
+```
+
+Install the project and selected requirements in the **same pip invocation** so
+the resolver considers their versions together. The model files share a base
+and select model-specific versions. Simulator assets are prepared separately;
+follow the corresponding example before evaluation.
+
+The checkout also supports `python -m uv sync --python 3.10` after installing uv
+in the environment; its lock includes the default ACT/DP/ALOHA packages. Use
+ordinary `python run.py` after activation. For optional models, follow their
+requirements-file instructions instead of syncing the default lock.
+
+`load_policy`, `load_dataset`, and `load_env` report missing or incompatible
+dependencies and print an external pip installation command. For a read-only check:
+
+```python
+import vlastudio as vla
+
+plan = vla.environment_plan(policy="act", environment="aloha_sim")
+print(plan["issues"])
+print(plan["commands"]["install_current"])
+```
+
+See the [environment guide](docs/environments.md) for other models, Python versions,
+CUDA wheels, and custom dependency metadata. Installation is performed only by
+your terminal's pip/uv commands; workflow code uses the installed environment.
+
+## 🚀 Write Your Own run.py
+
+### ACT on ALOHA-Sim
+
+Prepare the demonstrations at `data/sim_transfer_cube_scripted`, or use a task YAML with your own dataset path. The built-in starting point is [sim_transfer_cube_scripted.yaml](src/vlastudio/configs/task/sim_transfer_cube_scripted.yaml).
+
+Save this as `run.py` in your project:
+
+```python
+import vlastudio as vla
+
+
+def main():
+    dataset = vla.load_dataset("sim_transfer_cube_scripted")
+    policy = vla.load_policy("act")
+    result = vla.train(
+        policy, dataset, "default", output_dir="checkpoints/act_aloha"
+    )
+    benchmark = vla.load_env("aloha_transfer")
+    evaluation = benchmark.evaluate(result.policy, output_dir="results/act_aloha")
+    print(evaluation.metrics)
+
+
+if __name__ == "__main__":
+    main()
+```
+
+```bash
+python run.py
+```
+
+The script uses your current Python environment. Paths resolve relative to your working directory; pass your own YAML paths or built-in names to the loaders. You can freely add preprocessing, logging, evaluation logic, or other library calls to this file.
+
+The [ACT example](examples/01_train_and_eval_act_on_aloha) provides the same workflow:
+
+```bash
+python examples/01_train_and_eval_act_on_aloha/train_and_evaluate.py
+```
+
+### Diffusion Policy
+
+The default installation above already includes Diffusion Policy and ALOHA-Sim. Write `run_dp.py`:
+
+```python
+import vlastudio as vla
+
+
+def main():
+    dataset = vla.load_dataset("sim_transfer_cube_scripted")
+    policy = vla.load_policy("diffusion_policy")
+    vla.train(
+        policy, dataset, "dp", output_dir="checkpoints/dp_aloha",
+        overrides={"training.max_steps": 200000, "training.save_steps": 10000},
+    )
+    vla.load_env("aloha_transfer").evaluate(policy, output_dir="results/dp_aloha")
+
+
+if __name__ == "__main__":
+    main()
+```
+
+```bash
+python run_dp.py
+```
+
+### SmolVLA on MetaWorld
+
+In a separate Python 3.10 environment, run `python -m pip install -e . -r requirements/smolvla.txt`. The built-in task uses the LeRobot MetaWorld dataset. Provide access to that dataset and the pretrained policy weights.
+
+Write `train_smolvla.py`:
+
+```python
+import vlastudio as vla
+
+
+def main():
+    dataset = vla.load_dataset("metaworld")
+    policy = vla.load_policy("smolvla")
+    vla.train(
+        policy, dataset, "smolvla", output_dir="checkpoints/smolvla_metaworld",
+        overrides={"training.max_steps": 100000,
+                   "training.per_device_train_batch_size": 64},
+    )
+
+
+if __name__ == "__main__":
+    main()
+```
+
+```bash
+python train_smolvla.py
+```
+
+Use the policy-server workflow below to evaluate with a simulator in another environment.
+
+### Runnable examples
+
+Each example shows its own environment setup and Python files. Run from the repository root after installing the required dependencies.
+
+| Goal | Example |
+| --- | --- |
+| Train and evaluate ACT | [01 · ACT on ALOHA](examples/01_train_and_eval_act_on_aloha) |
+| Serve a policy and evaluate remotely | [02 · Remote inference](examples/02_remote_inference) |
+| Write a custom policy | [03 · Custom policy](examples/03_customize_policy) |
+| Train and evaluate across tasks | [04 · Multitask policy](examples/04_multitask_policy) |
+| Fine-tune Pi0.5 | [05 · Pi0.5](examples/05_finetune_vla_pi05) |
+| Write and control a custom robot | [06 · Custom robot](examples/06_custom_robot) |
+| Integrate a custom simulator | [07 · LIBERO Plus](examples/07_custom_sim_env_libero_plus) |
+| Train and evaluate FastWAM | [08 · FastWAM on LIBERO](examples/08_fastwam_libero) |
+
+## Overview
+
+![framework](https://raw.githubusercontent.com/WwZzz/myfigs/refs/heads/master/fig_1_ilstudio.png)
+
+## 🌐 Policy Server
+
+Write `serve.py` in the model environment:
+
+```python
+import vlastudio as vla
+
+
+if __name__ == "__main__":
+    policy = vla.load_policy("smolvla", checkpoint="checkpoints/smolvla_metaworld")
+    vla.serve(policy, address="127.0.0.1:5000")
+```
+
+In a separate simulation environment, run `python -m pip install -e . -r requirements/metaworld.txt`, then write `evaluate.py`:
+
+```python
+import vlastudio as vla
+
+
+if __name__ == "__main__":
+    policy = vla.connect_policy("127.0.0.1:5000")
+    benchmark = vla.load_env("metaworld.easy")
+    result = benchmark.evaluate(
+        policy, output_dir="results/smolvla_metaworld", num_rollout=10, batch_size=10
+    )
+    print(result.metrics)
+```
+
+Start `python serve.py` in the model environment, wait for the server to listen, then start `python evaluate.py` in the simulation environment. The server handles inference while the client handles observations and actions. See [Example 02](examples/02_remote_inference) for runnable files.
+
+## 🤖 Real Robots and Data Collection
+
+For SO101, create a dedicated Python 3.10 environment and install the model and motor SDK together:
+
+```bash
+python -m pip install -e . -r requirements/smolvla.txt "lerobot[feetech]==0.3.3"
+```
+
+Customize serial ports, camera devices, IDs, and calibration paths in your robot and teleoperator YAMLs. Starting points: [so101.yaml](src/vlastudio/configs/robot/so101.yaml), [so101_leader.yaml](src/vlastudio/configs/teleop/so101_leader.yaml).
+
+Write `collect.py`:
+
+```python
+import vlastudio as vla
+
+
+if __name__ == "__main__":
+    vla.collect(
+        "so101", teleop="so101_leader", output_dir="data/so101_pick_place",
+        dataset_format="hdf5", task="pick up the cube",
+    )
+```
+
+Write `infer.py` to use an already running policy server:
+
+```python
+import vlastudio as vla
+
+
+if __name__ == "__main__":
+    policy = vla.connect_policy("127.0.0.1:5000")
+    vla.infer("so101", policy)
+```
+
+Run `python collect.py` to collect demonstrations or `python infer.py` to deploy. Pass custom YAML paths instead of built-in names for your own robot. See the [SO101 guide](src/vlastudio/deploy/robot/so101/README.md).
+
+## 🎨 Policy Gallery
+
+| **Policy**            | **Reference**                                                                                          |
+| --------------------- | ------------------------------------------------------------------------------------------------------ |
+| ACT                   | [[1]](https://arxiv.org/abs/2304.13705)                                                                |
+| Diffusion Policy      | [[2]](https://arxiv.org/abs/2303.04137)                                                                |
+| Qwen2VL+DP            | [[3]](https://arxiv.org/abs/2308.12966) + [[2]](https://arxiv.org/abs/2303.04137)                          |
+| Qwen2.5VL+DP          | [[3]](https://arxiv.org/abs/2308.12966) + [[2]](https://arxiv.org/abs/2303.04137)                          |
+| DiVLA                 | [[4]](https://arxiv.org/html/2412.03293v1)                                                                |
+| OpenVLA               | [[5]](https://arxiv.org/abs/2406.09246)                                                      |
+| OpenVLA-OFT               | [[6]](https://arxiv.org/abs/2502.19645)                                                      |
+| Pi0                   | [[7]](https://arxiv.org/html/2410.24164v1)                                                             |
+| Pi0.5                   | [[8]](https://arxiv.org/abs/2504.16054)                                                             |
+| MLP                   | -                                                                                                      |
+| ResNet                | [[9]](https://arxiv.org/abs/1512.03385)                                                                |
+| SmolVLA               | [[10]](https://arxiv.org/abs/2506.01844)                                                                |
+| Octo                  | [[11]](https://octo-models.github.io/)    |
+| FlowMatching                  |  -    |
+| GR00T| [[12]](https://arxiv.org/abs/2503.14734)|
+| X-VLA                 | [LeRobot Docs](https://huggingface.co/docs/lerobot/main/en/xvla) |
+
+## 🏆 Simulation Benchmark Gallery
+
+| Environment | Parallel rollouts | Need Vulkan |
+|------------|--------|------------|
+| aloha_sim | ✅ | ❌ |
+| gymnasium_robotics | ✅ | ❌ |
+| libero | ✅ | ❌ |
+| metaworld | ✅ | ❌ |
+| pandagym | ✅ | ❌ |
+| robomimic | ✅ | ❌ |
+| simplerenv | ✅ | ✅ |
+| robotwin | ❌ | ✅ |
+| rlbench | ❌ | ❌ |
+| calvin (Under Testing) | ❌ | ❌ |
+| bigym | ✅ | ❌ |
+| behavior1k | ❌ | ❌ |
+
+## 🦾 Robot Gallery
+- SO101
+- BimanualSO101
+- Kochv1.1
+- AgilexAloha (CobotMagic)
+
+
+## 📊 Datasets and Extensions
+
+VLAStudio provides HDF5, LeRobot, and RLDS loaders. Data processing and batch collation adapt each source to the selected policy. Your `run.py` can select built-in names or your own task and policy YAML files:
+
+```python
+import vlastudio as vla
+
+dataset = vla.load_dataset("path/to/task.yaml")
+policy = vla.load_policy("path/to/policy.yaml")
+```
+
+Built-in configurations live in [src/vlastudio/configs](src/vlastudio/configs). User configurations can live anywhere.
+
+Start from the examples above to write your own components. A policy supplies `load_model` and a model with `select_action`; optional data processors, collators, and trainers customize training. The [developer guide](docs/package_runtime.md) describes package and runtime internals.
+
+## 🩹 Troubleshooting
+
+- **Missing or conflicting dependencies:** inspect `vla.environment_plan(...)` and use its pip command in the intended environment, or install the corresponding requirements file.
+- **Headless MuJoCo on Linux:** set `os.environ["MUJOCO_GL"] = "egl"` near the start of your Python file. CPU rendering needs system OSMesa libraries and the `osmesa` backend; a desktop display can use `glfw`.
+- **Old CMake projects fail with “Compatibility with CMake < 3.5 has been removed”:** set `CMAKE_POLICY_VERSION_MINIMUM=3.5` in the terminal before running pip for that optional simulator.
+- **Model or simulator resources are missing:** Python dependencies, pretrained weights, datasets, and native simulator assets are separate setup steps. Follow the relevant example's README.
+
+## 📫 Contact
+
+- Wang Z. <zwang@stu.xmu.edu.cn>
+- Yuxin G. <yuxinxmu@163.com>
+
+## 🙏 Acknowledgements
+This repo is built on the open source codebases below. Thanks to the authors' wonderful contributions.
+
+- [Lerobot](https://github.com/huggingface/lerobot/tree/main)
+
+- [ACT](https://github.com/MarkFzp/act-plus-plus/)
+
+- [Octo-pytorch](https://github.com/emb-ai/octo-pytorch/)
+
+- [Openpi](https://github.com/Physical-Intelligence/openpi)
+
+- [OpenVLA](https://github.com/openvla/openvla/)
+
+- [DiVLA](https://diffusion-vla.github.io/)
+
+- [RoboTwin](https://robotwin-platform.github.io)
+
+- [RLBench](https://github.com/stepjam/RLBench/)
+
+- [StarVLA](https://github.com/starVLA/starVLA/)