@r1z4x/tezgah 1.0.0

package/skills/ai-research/18-multimodal/openvla-oft/references/config-troubleshooting.md

@@ -0,0 +1,97 @@
+<!-- vendored from orchestra-research/AI-research-SKILLs@773a52944ba4747a18bd4ae9ade53fff041adcbc 18-multimodal/openvla-oft/references/config-troubleshooting.md; MIT (c) Orchestra Research -->
+# Configuration and Troubleshooting
+
+## Core files map
+
+Training:
+- `vla-scripts/finetune.py`
+
+Server deployment:
+- `vla-scripts/deploy.py`
+
+LIBERO evaluation:
+- `experiments/robot/libero/run_libero_eval.py`
+
+ALOHA evaluation:
+- `experiments/robot/aloha/run_aloha_eval.py`
+
+Action/policy utilities:
+- `experiments/robot/openvla_utils.py`
+
+Platform constants:
+- `prismatic/vla/constants.py`
+
+## High-risk configuration matrix
+
+| Area | Required consistency | Typical failure if mismatched |
+|------|----------------------|-------------------------------|
+| Action head mode | `use_l1_regression` vs `use_diffusion` | Wrong head loading, unstable or invalid action generation |
+| FiLM usage | `use_film` in train/eval/deploy | Reduced language grounding, degraded policy quality |
+| Image streams | `num_images_in_input` across train/eval/deploy | Shape mismatch or strong performance drop |
+| Proprio input | `use_proprio` parity | State conditioning mismatch, action drift |
+| LoRA rank | `lora_rank` parity | Adapter loading errors or wrong effective model |
+| Crop behavior | `image_aug` in training implies `center_crop=True` in eval/deploy | Significant success-rate drop |
+| Action chunk | `num_open_loop_steps` close to `NUM_ACTIONS_CHUNK` | Latency/success tradeoff shifts, lower success |
+| Un-normalization key | `unnorm_key` present in checkpoint stats | Bad action scale or assertion failures |
+
+## Constants behavior notes
+
+`prismatic/vla/constants.py` auto-selects constants by command-line text (`libero`, `aloha`, `bridge`).
+
+Implications:
+- If command path does not include expected platform tokens, constants may default to LIBERO.
+- For custom entrypoints or renamed scripts, verify selected platform constants in logs.
+
+Expected defaults:
+- LIBERO: `NUM_ACTIONS_CHUNK=8`, `ACTION_DIM=7`, `PROPRIO_DIM=8`
+- ALOHA: `NUM_ACTIONS_CHUNK=25`, `ACTION_DIM=14`, `PROPRIO_DIM=14`
+
+## Sanity checks before long runs
+
+Check package versions:
+
+```bash
+python -c "import torch, transformers, peft; print('torch', torch.__version__); print('transformers', transformers.__version__); print('peft', peft.__version__)"
+```
+
+Check detected constants in launch logs:
+- `Using LIBERO constants: ...` or `Using ALOHA constants: ...`
+
+Dry-run one short evaluation before full benchmark:
+
+```bash
+python experiments/robot/libero/run_libero_eval.py \
+  --pretrained_checkpoint moojink/openvla-7b-oft-finetuned-libero-spatial \
+  --task_suite_name libero_spatial \
+  --num_trials_per_task 2 \
+  --seed 7
+```
+
+## Frequent failures and precise fixes
+
+**Failure: `Action un-norm key ... not found in VLA norm_stats`**
+- Cause: wrong `unnorm_key` or dataset stats not bundled with checkpoint.
+- Fix: use dataset-specific key and verify checkpoint directory contains normalization artifacts.
+
+**Failure: Large performance drop after moving from H100 to A100**
+- Cause: merged adapter/model artifact mismatch across hardware/runtime stack.
+- Fix: re-merge LoRA on target machine, then evaluate with same runtime flags.
+
+**Failure: Poor LIBERO performance despite good training loss**
+- Cause: eval config mismatch (`center_crop`, `num_images_in_input`, chunk settings).
+- Fix: align eval with paper-style inference defaults and verify constants output.
+
+**Failure: ALOHA client cannot query server**
+- Cause: bad `vla_server_url`, networking, or server not running on `8777`.
+- Fix: ensure `vla-scripts/deploy.py` is active, verify endpoint from client, check firewall and DNS.
+
+**Failure: ALOHA ROS import error with `libp11-kit` / `libffi`**
+- Cause: binary dependency mismatch in client conda environment.
+- Fix: `conda install -c conda-forge libffi`
+
+## Decision hints for key training flags
+
+- Prefer `use_l1_regression=True` for the default paper-style OFT/OFT+ runs.
+- Enable `use_film=True` when tasks require stronger language grounding.
+- Keep `use_diffusion=False` unless intentionally exploring diffusion action heads.
+- Keep `image_aug=True` in training and `center_crop=True` in eval/deploy for consistency.

package/skills/ai-research/18-multimodal/openvla-oft/references/libero-workflow.md

@@ -0,0 +1,115 @@
+<!-- vendored from orchestra-research/AI-research-SKILLs@773a52944ba4747a18bd4ae9ade53fff041adcbc 18-multimodal/openvla-oft/references/libero-workflow.md; MIT (c) Orchestra Research -->
+# LIBERO Workflow
+
+## Scope
+
+Use this guide for OpenVLA-OFT setup, evaluation, and fine-tuning on LIBERO simulation task suites.
+
+Task suite names used by evaluator:
+- `libero_spatial`
+- `libero_object`
+- `libero_goal`
+- `libero_10`
+
+## 1) Setup and dependencies
+
+```bash
+conda create -n openvla-oft python=3.10 -y
+conda activate openvla-oft
+pip3 install torch torchvision torchaudio
+pip install -e .
+
+git clone https://github.com/Lifelong-Robot-Learning/LIBERO.git
+pip install -e LIBERO
+pip install -r experiments/robot/libero/libero_requirements.txt
+```
+
+Optional dataset download from docs:
+
+```bash
+git clone git@hf.co:datasets/openvla/modified_libero_rlds
+```
+
+## 2) Evaluate official checkpoints
+
+Example for LIBERO-Spatial:
+
+```bash
+python experiments/robot/libero/run_libero_eval.py \
+  --pretrained_checkpoint moojink/openvla-7b-oft-finetuned-libero-spatial \
+  --task_suite_name libero_spatial \
+  --center_crop True \
+  --num_trials_per_task 50 \
+  --seed 7
+```
+
+Common changes:
+- `--task_suite_name libero_object|libero_goal|libero_10`
+- `--num_trials_per_task` for shorter sanity runs
+- `--use_wandb True --wandb_project ... --wandb_entity ...`
+
+## 3) Fine-tune on LIBERO RLDS
+
+Base recipe (paper-style command):
+
+```bash
+torchrun --standalone --nnodes 1 --nproc-per-node 8 vla-scripts/finetune.py \
+  --vla_path openvla/openvla-7b \
+  --data_root_dir /PATH/TO/RLDS/DATASETS/DIR/ \
+  --dataset_name libero_spatial_no_noops \
+  --run_root_dir /YOUR/CHECKPOINTS/AND/LOG/DIR/ \
+  --use_l1_regression True \
+  --use_diffusion False \
+  --use_film False \
+  --num_images_in_input 2 \
+  --use_proprio True \
+  --batch_size 8 \
+  --learning_rate 5e-4 \
+  --num_steps_before_decay 100000 \
+  --max_steps 150005 \
+  --save_freq 10000 \
+  --save_latest_checkpoint_only False \
+  --image_aug True \
+  --lora_rank 32 \
+  --wandb_entity YOUR_WANDB_ENTITY \
+  --wandb_project YOUR_WANDB_PROJECT
+```
+
+Replace `dataset_name` with one of:
+- `libero_spatial_no_noops`
+- `libero_object_no_noops`
+- `libero_goal_no_noops`
+- `libero_10_no_noops`
+
+## 4) Selection and validation strategy
+
+Suggested checkpoint strategy:
+- Evaluate 50k, 100k, and 150k checkpoints.
+- Keep the best checkpoint per suite by actual task success, not only train loss.
+
+Reason: docs report LIBERO-Goal may peak earlier than other suites.
+
+Validation checks:
+- Confirm `center_crop=True` during eval if trained with `image_aug=True`.
+- Confirm `num_open_loop_steps` matches `NUM_ACTIONS_CHUNK`.
+- Confirm `unnorm_key` exists in `model.norm_stats`.
+
+## 5) LoRA merge for deployment
+
+Use this when serving or evaluating on different hardware:
+
+```bash
+python vla-scripts/merge_lora_weights_and_save.py \
+  --base_checkpoint openvla/openvla-7b \
+  --lora_finetuned_checkpoint_dir /PATH/TO/CHECKPOINT_DIR
+```
+
+If performance drops after migrating to a different GPU family:
+- Re-merge on target machine.
+- Re-run eval with matched runtime flags.
+
+## 6) Logging locations
+
+- Default local logs: `experiments/logs/`
+- Training checkpoints: under `run_root_dir`
+- W&B (if enabled): user-defined entity/project

package/skills/ai-research/18-multimodal/openvla-oft/references/paper-and-checkpoints.md

@@ -0,0 +1,84 @@
+<!-- vendored from orchestra-research/AI-research-SKILLs@773a52944ba4747a18bd4ae9ade53fff041adcbc 18-multimodal/openvla-oft/references/paper-and-checkpoints.md; MIT (c) Orchestra Research -->
+# OpenVLA-OFT Paper and Checkpoints
+
+## Paper identity
+
+- Title: Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success
+- Authors: Moo Jin Kim, Chelsea Finn, Percy Liang
+- Year: 2025
+- ArXiv: https://arxiv.org/abs/2502.19645
+- Project page: https://openvla-oft.github.io/
+- Summary video: https://youtu.be/T3Zkkr_NTSA
+
+## What OpenVLA-OFT changes
+
+OpenVLA-OFT adapts OpenVLA for robot action generation with:
+- LoRA-based fine-tuning on VLA policies.
+- Continuous action prediction through dedicated action heads.
+- Optional FiLM conditioning for stronger language grounding (called OFT+ in ALOHA setup).
+- Multi-image and proprio input support via configurable model components.
+
+## Compute requirements from official docs
+
+Inference:
+- LIBERO tasks: about 16 GB VRAM.
+- ALOHA tasks: about 18 GB VRAM.
+
+Training:
+- 1 to 8 GPUs, roughly 27 GB to 80 GB VRAM depending on batch size, feature toggles, and precision.
+
+## Reproduction-sensitive environment notes
+
+For reported LIBERO numbers, docs recommend:
+- Python 3.10.14
+- PyTorch 2.2.0
+- OpenVLA-OFT custom Transformers fork (`transformers-openvla-oft`)
+- NVIDIA A100 when matching paper setup
+
+If reproduction diverges, check:
+- Different GPU architecture
+- Dependency drift (`torch`, `transformers`, `peft`)
+- Inference mismatches (`center_crop`, action chunk settings, and un-normalization keys)
+
+## Official LIBERO checkpoints
+
+Task-specific:
+- `moojink/openvla-7b-oft-finetuned-libero-spatial`
+- `moojink/openvla-7b-oft-finetuned-libero-object`
+- `moojink/openvla-7b-oft-finetuned-libero-goal`
+- `moojink/openvla-7b-oft-finetuned-libero-10`
+
+Combined training across all four suites:
+- `moojink/openvla-7b-oft-finetuned-libero-spatial-object-goal-10`
+
+## Reported comparison note
+
+The repository documentation reports comparable average success across four suites between:
+- task-specific policies: 97.1%
+- combined policy: 96.8%
+
+Treat these as reference values tied to official setup and seeds.
+
+## Model mode selection: OFT vs OFT+
+
+Typical defaults:
+- OFT (LIBERO): `use_film=False`, `num_images_in_input=2`, `use_proprio=True`.
+- OFT+ (ALOHA): `use_film=True`, `num_images_in_input=3`, `use_proprio=True`.
+
+Always match training and inference flags for:
+- `use_l1_regression` / `use_diffusion`
+- `use_film`
+- `num_images_in_input`
+- `use_proprio`
+- `lora_rank`
+
+## Citation block
+
+```bibtex
+@article{kim2025fine,
+  title={Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success},
+  author={Kim, Moo Jin and Finn, Chelsea and Liang, Percy},
+  journal={arXiv preprint arXiv:2502.19645},
+  year={2025}
+}
+```