gvhmr 1.0.0__tar.gz

gvhmr-1.0.0/.gitignore

@@ -0,0 +1,202 @@
+.vscode
+.hydra
+inputs
+outputs
+
+# machine-local GVHMR config (written by `gvhmr config init` / scripts/install.sh)
+/gvhmr.toml
+
+# ultralytics auto-downloads detector weights into the CWD (= repo root) — never commit them
+# (committed model-data assets live under gvhmr/**, which this root-anchored pattern doesn't touch)
+/*.pt
+
+# W&B run/sweep logs (gvhmr sweep agents run from the repo root; training logs live under outputs/)
+/wandb/
+
+# All file or folders start with tmp will be ignored
+tmp*  
+
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+.pybuilder/
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+#   For a library or package, you might want to ignore these files since the code is
+#   intended to run in multiple environments; otherwise, check them in:
+# .python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# poetry
+#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
+#poetry.lock
+
+# pdm
+#   Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
+#pdm.lock
+#   pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
+#   in version control.
+#   https://pdm.fming.dev/#use-with-ide
+.pdm.toml
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# pytype static type analyzer
+.pytype/
+
+# Cython debug symbols
+cython_debug/
+
+# 
+.DS_Store/
+
+# PyCharm
+#  JetBrains specific template is maintained in a separate JetBrains.gitignore that can
+#  be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
+#  and can be added to the global gitignore or merged into this file.  For a more nuclear
+#  option (not recommended) you can uncomment the following to ignore the entire idea folder.
+#.idea/
+
+# torchsparse
+torchsparse
+
+# tensorboard
+tensorboard
+
+# glove
+glove
+
+# videos
+videos/*
+
+# scene-aware camera backend: vendored model clones (set up via scripts/setup_scene_aware.sh)
+third-party/
+
+# local tooling / OS cruft
+.claude/
+.DS_Store
+*.bak
+# Local machine config (asset paths + model defaults; machine-specific)
+gvhmr.toml

gvhmr-1.0.0/LICENSE

@@ -0,0 +1,16 @@
+Copyright 2022-2023 3D Vision Group at the State Key Lab of CAD&CG,  
+Zhejiang University. All Rights Reserved. 
+
+For more information see <https://github.com/zju3dv/GVHMR> 
+If you use this software, please cite the corresponding publications   
+listed on the above website. 
+
+Permission to use, copy, modify and distribute this software and its 
+documentation for educational, research and non-profit purposes only. 
+Any modification based on this work must be open-source and prohibited 
+for commercial use. 
+You must retain, in the source form of any derivative works that you  
+distribute, all copyright, patent, trademark, and attribution notices  
+from the source form of this work. 
+ 
+For commercial uses of this software, please send email to xwzhou@zju.edu.cn

gvhmr-1.0.0/PKG-INFO

@@ -0,0 +1,326 @@
+Metadata-Version: 2.4
+Name: gvhmr
+Version: 1.0.0
+Summary: GVHMR: World-Grounded Human Motion Recovery via Gravity-View Coordinates (training & inference)
+Project-URL: Homepage, https://zju3dv.github.io/gvhmr
+Project-URL: Repository, https://github.com/ryanrudes/gvhmr
+Project-URL: Upstream, https://github.com/zju3dv/GVHMR
+Project-URL: Paper, https://arxiv.org/abs/2409.06662
+Author: Zehong Shen
+Maintainer: Ryan Rudes
+License: Copyright 2022-2023 3D Vision Group at the State Key Lab of CAD&CG,  
+        Zhejiang University. All Rights Reserved. 
+        
+        For more information see <https://github.com/zju3dv/GVHMR> 
+        If you use this software, please cite the corresponding publications   
+        listed on the above website. 
+        
+        Permission to use, copy, modify and distribute this software and its 
+        documentation for educational, research and non-profit purposes only. 
+        Any modification based on this work must be open-source and prohibited 
+        for commercial use. 
+        You must retain, in the source form of any derivative works that you  
+        distribute, all copyright, patent, trademark, and attribution notices  
+        from the source form of this work. 
+         
+        For commercial uses of this software, please send email to xwzhou@zju.edu.cn
+License-File: LICENSE
+Keywords: human-motion-recovery,pose-estimation,pytorch,siggraph,smpl
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.11
+Requires-Dist: av>=12
+Requires-Dist: chumpy
+Requires-Dist: einops>=0.8
+Requires-Dist: ffmpeg-python
+Requires-Dist: huggingface-hub>=0.23
+Requires-Dist: hydra-colorlog>=1.2
+Requires-Dist: hydra-core>=1.3
+Requires-Dist: hydra-zen>=0.13
+Requires-Dist: imageio>=2.34
+Requires-Dist: joblib
+Requires-Dist: matplotlib
+Requires-Dist: moderngl>=5.10
+Requires-Dist: numpy>=1.26
+Requires-Dist: omegaconf>=2.3
+Requires-Dist: opencv-python>=4.9
+Requires-Dist: pytorch-lightning>=2.3
+Requires-Dist: requests>=2.28
+Requires-Dist: rich>=13
+Requires-Dist: scikit-image
+Requires-Dist: scipy>=1.11
+Requires-Dist: smplx>=0.1.28
+Requires-Dist: termcolor
+Requires-Dist: timm>=1.0.9
+Requires-Dist: torch>=2.4
+Requires-Dist: torchvision>=0.19
+Requires-Dist: tqdm
+Requires-Dist: trimesh
+Requires-Dist: typer>=0.12
+Provides-Extra: app
+Requires-Dist: gradio>=4; extra == 'app'
+Provides-Extra: cpu
+Requires-Dist: torch; extra == 'cpu'
+Requires-Dist: torchvision; extra == 'cpu'
+Provides-Extra: cu124
+Requires-Dist: torch<2.8,>=2.4; extra == 'cu124'
+Requires-Dist: torchvision<0.23,>=0.19; extra == 'cu124'
+Provides-Extra: cu126
+Requires-Dist: torch<2.8,>=2.4; extra == 'cu126'
+Requires-Dist: torchvision<0.23,>=0.19; extra == 'cu126'
+Provides-Extra: cu128
+Requires-Dist: torch<2.8,>=2.4; extra == 'cu128'
+Requires-Dist: torchvision<0.23,>=0.19; extra == 'cu128'
+Provides-Extra: dev
+Requires-Dist: ipdb; extra == 'dev'
+Requires-Dist: pre-commit; extra == 'dev'
+Requires-Dist: pyright>=1.1.380; extra == 'dev'
+Requires-Dist: pytest-cov; extra == 'dev'
+Requires-Dist: pytest>=8; extra == 'dev'
+Requires-Dist: ruff>=0.6; extra == 'dev'
+Provides-Extra: dpvo
+Requires-Dist: numba; extra == 'dpvo'
+Requires-Dist: pypose; extra == 'dpvo'
+Provides-Extra: notebook
+Requires-Dist: ipdb; extra == 'notebook'
+Requires-Dist: ipython; extra == 'notebook'
+Requires-Dist: jupyter; extra == 'notebook'
+Provides-Extra: preproc
+Requires-Dist: cython-bbox; extra == 'preproc'
+Requires-Dist: lapx; extra == 'preproc'
+Requires-Dist: pycolmap>=0.6; extra == 'preproc'
+Requires-Dist: ultralytics>=8.2; extra == 'preproc'
+Requires-Dist: yacs; extra == 'preproc'
+Provides-Extra: render
+Requires-Dist: pytorch3d; extra == 'render'
+Provides-Extra: rtmpose
+Requires-Dist: onnxruntime; extra == 'rtmpose'
+Requires-Dist: rtmlib; extra == 'rtmpose'
+Provides-Extra: train
+Requires-Dist: tensorboard; extra == 'train'
+Requires-Dist: wandb-workspaces; extra == 'train'
+Requires-Dist: wandb>=0.17; extra == 'train'
+Provides-Extra: vis
+Requires-Dist: viser; extra == 'vis'
+Requires-Dist: wis3d; extra == 'vis'
+Description-Content-Type: text/markdown
+
+# GVHMR: World-Grounded Human Motion Recovery via Gravity-View Coordinates
+
+### [Project Page](https://zju3dv.github.io/gvhmr) | [Paper](https://arxiv.org/abs/2409.06662)
+
+> World-Grounded Human Motion Recovery via Gravity-View Coordinates
+> [Zehong Shen](https://zehongs.github.io/)<sup>\*</sup>,
+[Huaijin Pi](https://phj128.github.io/)<sup>\*</sup>,
+[Yan Xia](https://isshikihugh.github.io/scholar),
+[Zhi Cen](https://scholar.google.com/citations?user=Xyy-uFMAAAAJ),
+[Sida Peng](https://pengsida.net/)<sup>†</sup>,
+[Zechen Hu](https://zju3dv.github.io/gvhmr),
+[Hujun Bao](http://www.cad.zju.edu.cn/home/bao/),
+[Ruizhen Hu](https://csse.szu.edu.cn/staff/ruizhenhu/),
+[Xiaowei Zhou](https://xzhou.me/)
+> SIGGRAPH Asia 2024
+
+<p align="center">
+    <img src=docs/example_video/project_teaser.gif alt="animated" />
+</p>
+
+Feed it a video, get back **SMPL/SMPL-X human motion in both the camera frame and the world frame** —
+plus rendered overlay videos to see the result.
+
+> [!NOTE]
+> This is a **modernized fork** of [`zju3dv/GVHMR`](https://github.com/zju3dv/GVHMR) by
+> [@ryanrudes](https://github.com/ryanrudes): one-command install (`uv` + a Typer/Rich `gvhmr` CLI),
+> **Apple-Silicon (MPS) support** end-to-end (including a real-time moderngl mesh renderer), automatic
+> checkpoint fetching, swappable preprocessing models, scene-aware **metric** cameras
+> (`--camera dust3r|vggt` — Mac-friendly alternatives to CUDA-only DPVO), skeleton-overlay exports, a
+> CPU test suite, and re-training tooling. **The released model's behaviour is preserved** — the default
+> inference path is golden-guarded byte-identical to upstream.
+
+## Quick start
+
+```bash
+git clone https://github.com/ryanrudes/gvhmr && cd gvhmr
+scripts/install.sh                                   # detects your platform/GPU, installs, fetches checkpoints
+bin/gvhmr demo docs/example_video/tennis.mp4 -s      # recover motion from the bundled example video
+```
+
+The installer detects macOS vs Linux+NVIDIA vs CPU, picks the matching torch build, records the
+choices, and hands off to the **wizard**, which walks you through every optional component (RTMPose,
+DPVO, the DUSt3R/VGGT scene cameras, 3D visualization, …), asset locations, and the checkpoint fetch.
+From then on you interact only with the `gvhmr` CLI (via `bin/gvhmr`, or `gvhmr` after
+`source .venv/bin/activate`), never with uv: `bin/gvhmr config init` re-runs the wizard;
+`bin/gvhmr env sync` re-applies the recorded environment if it ever drifts. Two notes, and everything
+else just works:
+
+- **Body models are registration-gated** (their license forbids redistribution): sign up at
+  [SMPL](https://smpl.is.tue.mpg.de/) and [SMPL-X](https://smpl-x.is.tue.mpg.de/), then
+  `bin/gvhmr download` prints exactly where to put the files.
+- On **Linux + NVIDIA**, torch must match your CUDA driver — `scripts/install.sh` picks the right build
+  for you; if you sync manually, see [docs/INSTALL.md](docs/INSTALL.md#cuda--gpu-linux). macOS needs
+  nothing special.
+
+Run `bin/gvhmr info` anytime for a full diagnostic (device, installed features, checkpoint status, and
+the exact fix for anything missing). Full install guide: [docs/INSTALL.md](docs/INSTALL.md).
+
+Prefer not to install? Try the upstream-hosted
+[Colab](https://colab.research.google.com/drive/1N9WSchizHv2bfQqkE9Wuiegw_OT7mtGj?usp=sharing) or
+[HuggingFace Space](https://huggingface.co/spaces/LittleFrog/GVHMR).
+
+## Python library
+
+GVHMR ships a HuggingFace-style Python API (in `gvhmr/inference/`) over the exact same pipeline as the
+CLI — same code paths, byte-identical results, but a clean object you can loop over videos with.
+
+```bash
+pip install "gvhmr[preproc]"    # base + preprocessing (add cu124/cu126/cu128 on Linux+CUDA; macOS = MPS)
+gvhmr auth smpl                 # one-time: your MPI login, to fetch the gated SMPL/SMPL-X body models
+```
+
+```python
+import gvhmr
+
+# one-liner (caches a default pipeline; a loop of videos loads weights once)
+result = gvhmr.recover("dance.mp4")
+
+# or load once, reuse across videos
+pipe = gvhmr.pipeline("human-motion-recovery", device="cuda")
+result = pipe("dance.mp4", static_camera=True, flip_test=True)
+
+result.smpl_params_world      # world-frame SMPL params (global_orient/body_pose/betas/transl)
+result.joints_world           # (L, 24, 3) world-frame joints
+result.render("overlay.mp4")  # in-cam ∥ world overlay video
+result.save_npz("dance.npz")  # portable SMPL params + intrinsics
+```
+
+Body models are registration-gated by MPI and **never redistributed** — `gvhmr auth smpl` (or
+`$SMPLX_USER`/`$SMPLX_PW`) fetches them from the official source with *your* account. There's also a
+tensor-level "power path" (`gvhmr.GVHMR.from_pretrained(...).predict(...)`) for your own preprocessing,
+and `save_pretrained` / `push_to_hub` for sharing. Full guide: [docs/LIBRARY.md](docs/LIBRARY.md).
+
+## Usage
+
+The `gvhmr` command (Typer + Rich) is the main entry point — `gvhmr --help` for the full menu. (Below,
+`gvhmr` means `bin/gvhmr` from the repo root, or plain `gvhmr` in an activated venv.)
+
+```bash
+gvhmr demo VIDEO.mp4 -s                       # single video, static camera (fastest)
+gvhmr demo VIDEO.mp4                          # moving camera (SimpleVO, rotation only)
+gvhmr demo VIDEO.mp4 --camera vggt            # moving camera with world translation (any device)
+gvhmr demo-folder DIR -o outputs/batch -s     # every video in a folder
+gvhmr info                                    # environment & asset diagnostic
+gvhmr config init                             # wizard: asset locations, models, managed environment
+gvhmr env sync                                # re-apply the recorded environment (fixes any drift)
+gvhmr bench                                   # inference latency benchmark
+```
+
+Each run writes to `outputs/demo/<video>/`: the recovered motion (`hmr4d_results.pt` — SMPL-X
+parameters in camera and world frames), the camera-view render (`1_incam.mp4`), the world-view render
+(`2_global.mp4`), and a side-by-side of the two. `--no-render` skips the videos (motion only);
+`--render-scale 1` renders full-resolution overlays (default 0.5 for speed); `--verbose` saves
+intermediate overlays (detection boxes, 2D keypoints).
+
+**Cameras.** `-s/--static-cam` skips visual odometry entirely. For moving cameras the default is
+SimpleVO (rotation only); to recover the camera's **translation** too — e.g. a following/tracking
+shot — use a scene-aware **metric** camera: `--camera vggt` or `--camera dust3r` (run
+`scripts/setup_scene_aware.sh` once; works on Apple Silicon / CPU / CUDA), or classic
+`--camera dpvo` (CUDA only; `scripts/setup_dpvo.sh`).
+
+**Accuracy.** Add `--flip-test` (mirror-averaging TTA — the benchmark-time setting) and pass the true
+focal length `--f_mm` if you know it (phone metadata is read automatically when present). Evidence and
+measurement methodology: [docs/ACCURACY.md](docs/ACCURACY.md).
+
+**Skeleton exports.** `--skeleton` writes a world-frame skeleton-only video, `--skeleton-overlay` draws
+the 24-joint skeleton on top of the mesh videos, `--skeleton-joints legs,left_arm` restricts to a
+subset (groups or joint names/indices).
+
+**Device.** Auto-selected (CUDA → MPS → CPU); override with `GVHMR_DEVICE=cpu|mps|cuda`. Everything
+except CUDA-only DPVO runs on Apple Silicon, including rendering.
+
+### Swapping models
+
+The detector, 2D-pose estimator, feature backbone, and camera are each a **Hydra config group** — pick
+implementations by name, bundle choices into a committable *recipe*, or tweak any knob:
+
+```bash
+gvhmr demo clip.mp4 --detector yolo26x --pose2d rtmpose --camera dust3r   # pick implementations
+gvhmr demo clip.mp4 --recipe accurate                                     # a committable bundle
+gvhmr demo clip.mp4 --set detector.conf=0.4                               # tweak a knob
+```
+
+Every YOLO family×size is a preset; RTMPose (`gvhmr config set extras preproc,rtmpose && gvhmr env
+sync`) is a verified alternative 2D-pose backend; the feature backbone is learned conditioning, so
+swapping it needs a retrain (the tooling for that exists — `gvhmr extract-features` + `gvhmr train`).
+Machine-local defaults (asset locations + model versions + the managed environment) live in one
+readable config file managed by **`gvhmr config init`**. Full guide:
+[docs/CONFIGURATION.md](docs/CONFIGURATION.md); roadmap: [docs/EXTENSIBILITY.md](docs/EXTENSIBILITY.md).
+
+## Reproduce the paper
+
+```bash
+gvhmr eval                    # 3DPW + EMDB + RICH: auto-fetches the eval packs, runs the paper
+                              # protocol, prints your numbers next to the paper's (verified to match)
+gvhmr eval 3dpw --json m.json # one dataset; optionally dump metrics for tracking
+gvhmr eval all --ckpt outputs/my_run/checkpoints/last.ckpt    # evaluate your own training run
+gvhmr eval 3dpw --detector yolo26x        # benchmark a PREPROCESSING swap (auto-fetches raw 3DPW,
+                              # regenerates boxes/keypoints/features; see docs/EVAL.md)
+gvhmr sweep run 3dpw --detectors all      # W&B sweep comparing every detector preset on the benchmark
+
+# Train (the released ckpt used 2×4090 for 420 epochs)
+gvhmr train exp=gvhmr/mixed/mixed
+```
+
+`gvhmr eval` wraps the canonical Lightning test tasks (`gvhmr train global/task=gvhmr/test_* …` still
+works) — the only manual step is the registration-gated body models, and the command tells you exactly
+which files it needs. Details: [docs/EVAL.md](docs/EVAL.md); training: [docs/TRAINING.md](docs/TRAINING.md)
+(training doesn't apply the test-time postprocessing, so its global metrics differ from the benchmark).
+
+## Development
+
+```bash
+uv sync --extra dev        # test/lint/type tooling  (or scripts/install.sh --dev)
+uv run pre-commit install  # once — auto-formats staged files on commit
+make check                 # the required CI gates locally: ruff format --check + pytest
+make fmt                   # format the whole tree
+```
+
+The test suite is a CPU/MPS characterization net — no GPU, checkpoints, or datasets needed. Start with
+[`AGENTS.md`](AGENTS.md) for architecture, conventions, and the behaviour-preservation landmines.
+
+## Documentation
+
+| Doc | What's in it |
+|---|---|
+| [docs/INSTALL.md](docs/INSTALL.md) | full install guide: CUDA matrix, extras, weights, troubleshooting |
+| [docs/CONFIGURATION.md](docs/CONFIGURATION.md) | the config file + swapping/configuring every pipeline stage |
+| [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) | code map: data flow, packages, the 151-dim latent |
+| [docs/TRAINING.md](docs/TRAINING.md) | training/eval on any device, dataset packs, backbone retrains |
+| [docs/ACCURACY.md](docs/ACCURACY.md) | test-time accuracy levers + how they're measured |
+| [docs/PERFORMANCE.md](docs/PERFORMANCE.md) | profiling & latency notes (CPU vs MPS, render scale) |
+| [docs/EXTENSIBILITY.md](docs/EXTENSIBILITY.md) | the swappable-everything roadmap & rationale |
+| [docs/BEHAVIOR.md](docs/BEHAVIOR.md) / [docs/PROVENANCE.md](docs/PROVENANCE.md) | behaviour-preservation contract; what was vendored/renamed vs upstream |
+
+# Citation
+
+If you find this code useful for your research, please use the following BibTeX entry.
+
+```bibtex
+@inproceedings{shen2024gvhmr,
+  title={World-Grounded Human Motion Recovery via Gravity-View Coordinates},
+  author={Shen, Zehong and Pi, Huaijin and Xia, Yan and Cen, Zhi and Peng, Sida and Hu, Zechen and Bao, Hujun and Hu, Ruizhen and Zhou, Xiaowei},
+  booktitle={SIGGRAPH Asia Conference Proceedings},
+  year={2024}
+}
+```
+
+# Acknowledgement
+
+We thank the authors of
+[WHAM](https://github.com/yohanshin/WHAM),
+[4D-Humans](https://github.com/shubham-goel/4D-Humans),
+and [ViTPose-Pytorch](https://github.com/gpastal24/ViTPose-Pytorch) for their great works, without which our project/code would not be possible.
+
+Upstream repository: [zju3dv/GVHMR](https://github.com/zju3dv/GVHMR).