texture3d 0.2.0__tar.gz

texture3d-0.2.0/PKG-INFO

@@ -0,0 +1,362 @@
+Metadata-Version: 2.4
+Name: texture3d
+Version: 0.2.0
+Summary: Texture a triangle mesh from posed RGB-D frames: one keyframe per face with seam leveling
+Author: Sai Kiran
+License: MIT
+Keywords: texturing,rgbd,mesh,texture-mapping,3d-reconstruction
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Topic :: Multimedia :: Graphics :: 3D Modeling
+Classifier: Topic :: Scientific/Engineering :: Image Processing
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+Requires-Dist: numpy>=1.24
+Requires-Dist: scipy>=1.10
+Requires-Dist: opencv-python>=4.5
+Requires-Dist: trimesh>=4.0
+Requires-Dist: pillow>=9.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7; extra == "dev"
+Requires-Dist: ruff>=0.4; extra == "dev"
+
+# texture3d
+
+Texture an aligned triangle mesh from posed RGB-D frames — one photograph per face, seams levelled, one atlas out.
+
+[![Python 3.9+](https://img.shields.io/badge/python-3.9%2B-blue.svg)](#installation)
+[![License: MIT](https://img.shields.io/badge/license-MIT-green.svg)](#license)
+[![Version 0.2.0](https://img.shields.io/badge/version-0.2.0-informational.svg)](https://github.com/saikiran321/texture3d)
+
+Not on PyPI yet. Install from this repo. The texturing library is trimesh-based and does not depend on Open3D; reconstruction (optional, same repo) does.
+
+![A grey untextured scan of a room, plus three posed photographs of the same room, equals the scan rendered with the photographs applied as a texture](docs/figures/purpose.png)
+
+*A scanner gives you geometry; a camera gives you posed photos. `texture3d` puts each photo onto the faces it sees best, levels the seams, and packs one atlas. `apt0_seq0`: 977k faces, 1000 frames, 58 keyframes, ~2 min on a laptop.*
+
+## Quick start
+
+```bash
+python3 -m venv .venv
+.venv/bin/pip install -e ".[dev]"
+```
+
+```python
+import texture3d
+
+frames = texture3d.load_dataset("path/to/dataset")  # directory with dataset.json
+result = texture3d.bake("mesh.ply", frames)
+result.export("textured_mesh.obj")  # or .glb
+print(result.stats)
+```
+
+```bash
+.venv/bin/texture3d path/to/dataset --mesh mesh.ply --out textured/run
+```
+
+You need a triangle mesh in the same world frame as the camera poses, plus RGB-D frames. See [Dataset layout](#dataset-layout). Without a `dataset.json` folder, build `RGBDFrame` objects yourself ([Usage](#usage)).
+
+## Table of contents
+
+- [Why](#why)
+- [Features](#features)
+- [Installation](#installation)
+- [Usage](#usage)
+- [CLI](#cli)
+- [Configuration](#configuration)
+- [How it works](#how-it-works)
+- [RGB-D reconstruction](#rgb-d-reconstruction)
+- [Dataset layout](#dataset-layout)
+- [Testing](#testing)
+- [Limitations](#limitations)
+- [License](#license)
+
+## Why
+
+Averaging every view onto a texel blurs photograph detail as soon as poses or geometry are off by a few millimetres. **texture3d** assigns each face one keyframe (the view that sees it best), cuts connected same-view faces out as patches, then hides the seams with a global colour-offset solve and a local Poisson blend. The result is a single-material textured mesh (OBJ or GLB). Input UVs are ignored; output UVs follow the labelling, with vertices split along patch borders.
+
+![Three views of the same stairs: the photograph, the reconstruction's per-vertex colour averaged over all frames, and the texture3d result; zoomed crops show the averaged version is blurred while texture3d keeps the carpet weave and shoe detail](docs/figures/why.png)
+
+*Left: a captured frame. Middle: the TSDF reconstruction's per-vertex colour, i.e. every frame averaged — the carpet weave and the shoes smear. Right: `texture3d`, one photograph per face. Bottom row is the boxed region at native resolution.*
+
+This repo also ships an Open3D TSDF reconstruction pipeline for turning a `dataset.json` sequence into `mesh.ply`. You can texture that mesh, or an untextured STL/OBJ from another engine, as long as it is aligned to the same poses.
+
+**Status:** 0.2.0 — end-to-end usable, public API may still change.
+
+## Features
+
+- **One keyframe per face** — sharpness-weighted view scoring, outlier rejection on mean face colour, MRF labelling with Potts smoothness
+- **Seam leveling** — global least-squares offsets in YCbCr, then a Poisson blend along a configurable border strip
+- **Single atlas** — MaxRects packing (shelf packer above 4,000 patches), optional unsharp mask
+- **Unseen-face fill** — sub-pixel faces inherit a neighbour's view instead of speckling the empty colour
+- **Dataset-agnostic inputs** — `RGBDFrame` + `PinholeCamera`, or `load_dataset()` for the `dataset.json` layout
+- **OBJ and GLB export** via trimesh; `result.stats` reports coverage and per-stage timings
+
+## Installation
+
+**Prerequisites:** Python 3.9 or newer. Texturing needs NumPy, SciPy, OpenCV, trimesh, and Pillow (pulled in by the package). Reconstruction additionally needs Open3D 0.18 and NumPy 1.x (`requirements.txt`).
+
+### Texturing library + CLI
+
+```bash
+git clone https://github.com/saikiran321/texture3d.git
+cd texture3d
+python3 -m venv .venv
+.venv/bin/pip install -e .
+```
+
+Dev extras (pytest, ruff):
+
+```bash
+.venv/bin/pip install -e ".[dev]"
+```
+
+### Reconstruction extras
+
+```bash
+.venv/bin/pip install -r requirements.txt
+```
+
+Open3D 0.18 is built against the NumPy 1.x ABI. Keep `numpy>=1.24,<2` in that environment; NumPy 2.x can segfault during TSDF integrate.
+
+<!-- TODO: add a PyPI one-liner when the package is published -->
+
+## Usage
+
+### From a `dataset.json` directory
+
+```python
+import texture3d
+
+frames = texture3d.load_dataset("assets/datasets/apt0_seq0")
+result = texture3d.bake(
+    "assets/reconstructions/apt0_seq0/mesh.ply",
+    frames,
+)
+result.export("textured/apt0_seq0/textured_mesh.obj")
+print(result.stats["textured_fraction"], result.stats["stage_seconds"])
+```
+
+`BakeResult` has `.mesh` (textured `trimesh.Trimesh`), `.texture` (uint8 RGB atlas), and `.stats`.
+
+### From frames you already have in memory
+
+```python
+import numpy as np
+import texture3d
+from texture3d import BakeConfig, PinholeCamera, RGBDFrame
+
+camera = PinholeCamera(width=640, height=480, fx=582.871, fy=582.871, cx=320.0, cy=240.0)
+pose_c2w = np.eye(4)  # camera-to-world, OpenCV convention (+z forward, +y down)
+rgb = ...             # HxWx3 uint8, RGB
+depth_m = ...         # HxW float32 meters, 0 = invalid
+
+frames = [
+    RGBDFrame(pose_c2w=pose_c2w, camera=camera, rgb=rgb, depth=depth_m, depth_scale=1.0),
+]
+cfg = BakeConfig()
+cfg.keyframes.rgbd_frame_gap = 1
+result = texture3d.bake("mesh.ply", frames, cfg)
+result.export("textured_mesh.glb")
+```
+
+Poses are 4×4 **camera-to-world**. Integer depth images (and integer files) are multiplied by `depth_scale` (default `1e-3` for millimetre PNGs). Float arrays are treated as metres already.
+
+### Tuning
+
+```python
+from texture3d import BakeConfig
+
+cfg = BakeConfig()
+cfg.keyframes.rgbd_frame_gap = 5          # 1 keyframe per N frames
+cfg.mapping.data_smoothness_ratio = 0.05  # lower → larger patches
+cfg.mapping.sharpness_weight = 0.0        # disable unsharp mask
+cfg.mapping.local_leveling_strip_width = 30
+cfg.mapping.fill_unseen = True
+```
+
+## CLI
+
+```
+texture3d DATASET_DIR --mesh mesh.ply [--out DIR] [--format obj|glb]
+```
+
+| Flag | Default | Meaning |
+| --- | --- | --- |
+| `--mesh` | `reconstructions/<dataset-name>/mesh.ply` | Mesh to texture (`.ply` / `.obj` / `.stl`) |
+| `--out` | `textured/<dataset-name>` | Output directory |
+| `--format` | `obj` | `obj` or `glb` |
+| `--gap` | `5` | One keyframe per N frames (`1` = all) |
+| `--outlier` | `0.06` | Drop views below this colour likelihood (`0` = off) |
+| `--data-ratio` | `0.1` | Data cost vs smoothness; lower = larger patches |
+| `--no-global-leveling` | off | Skip the global colour-offset solve |
+| `--no-local-leveling` | off | Skip the Poisson border blend |
+| `--strip-width` | `20` | Local leveling strip width in pixels |
+| `--sharpness` | `0.5` | Unsharp-mask weight (`0` = off) |
+| `--max-texture` | `8192` | Largest atlas edge in pixels |
+| `--no-fill-unseen` | off | Leave sub-pixel faces in the empty colour |
+
+Writes `textured_mesh.<format>`, `config.json`, and `summary.json`. If the default mesh path does not exist, pass `--mesh` explicitly.
+
+```bash
+.venv/bin/texture3d assets/datasets/apt0_seq0 \
+  --mesh assets/reconstructions/apt0_seq0/mesh.ply \
+  --out textured/apt0_seq0
+```
+
+On a 977k-face mesh with 58 keyframes (640×480, 18-core Mac) this run was about **120 s** (scoring 8 s, labelling 9 s, global leveling 15 s, local leveling 82 s, atlas 1 s), **85%** of faces textured (the rest is the TSDF back shell no camera saw), 4096 atlas.
+
+`run_texturing.py` is a thin shim around the same CLI.
+
+## Configuration
+
+`BakeConfig` / `KeyframeConfig` / `MappingConfig` (`texture3d.config`):
+
+| Name | Type | Default | Description |
+| --- | --- | --- | --- |
+| `keyframes.rgbd_frame_gap` | int | `5` | Least-blurry frame in every window of this many frames |
+| `keyframes.closest_pose_translation` | float | `0.05` | Skip a candidate within this many metres of the last keyframe |
+| `keyframes.closest_pose_rotation_angle` | float | `0.09` | Same, for relative rotation (radians) |
+| `mapping.outlier_threshold` | float | `0.06` | Gaussian colour likelihood; `0` disables |
+| `mapping.data_smoothness_ratio` | float | `0.1` | Data cost scale vs Potts smoothness |
+| `mapping.global_seam_leveling` | bool | `True` | Least-squares colour offsets per (vertex, patch) |
+| `mapping.local_seam_leveling` | bool | `True` | Poisson blend along patch borders |
+| `mapping.local_leveling_strip_width` | int | `20` | Blend strip width in pixels |
+| `mapping.sharpness_weight` | float | `0.5` | Unsharp mask; `0` disables |
+| `mapping.max_texture_size` | int | `8192` | Max atlas edge (power of two) |
+| `mapping.empty_color` | RGB tuple | `(255, 255, 255)` | Texels no patch covers |
+| `mapping.scoring_workers` | int | `4` | Views scored in parallel |
+| `mapping.fill_unseen` | bool | `True` | Neighbour view for sub-pixel faces |
+| `mapping.exact_packing_limit` | int | `4000` | MaxRects up to this many patches; shelf packer beyond. `0` = always MaxRects |
+
+## How it works
+
+```mermaid
+flowchart LR
+  frames[RGB-D frames] --> kf[Keyframes]
+  mesh[Triangle mesh] --> topo[Topology]
+  kf --> score[Score views]
+  topo --> score
+  score --> label[Label faces]
+  label --> patches[Patches + seams]
+  patches --> level[Seam leveling]
+  level --> atlas[Pack atlas]
+  atlas --> out[Textured mesh]
+```
+
+1. **Keyframes** — least blurry frame in each `--gap` window (Crete metric); near-duplicate poses are skipped.
+2. **Topology** — incidence, edge adjacency, face normals smoothed over neighbours within 25°.
+3. **Scoring** — depth-tested face maps; quality is blurred Sobel magnitude times squared cosine to the normal. Colour outliers are dropped.
+4. **Labelling** — MRF over the face graph (views + “none”), loopy belief propagation.
+5. **Patches** — connected same-view faces, 2 px border in that view; seam vertices where patches meet.
+6. **Leveling** — global Tikhonov-regularised offsets (λ = 0.1), then a 20 px Poisson strip.
+7. **Atlas** — pack, copy levelled pixels, write per-corner UVs, optional sharpen.
+
+![Four panels: the per-face keyframe label map, the raw bake with visible seams, the levelled bake, and the top-left corner of the packed atlas where the largest patches (stairs, wall, door) land; a zoomed strip shows the wall mottling disappearing after leveling](docs/figures/how.png)
+
+*Stages 4–7 on one view of `apt0_seq0`. Column 4 is the atlas itself: patches are packed largest-first, so the top-left corner holds whole surfaces (stairs, picture wall, door) at full resolution; the long tail of small patches fills the rows below. 48 % of the 4096² atlas is used.*
+
+Faces no camera sees stay the empty colour (typical TSDF back shell). `result.stats["stage_seconds"]` times each stage.
+
+To check a bake visually, render the textured mesh back from dataset poses and compare with the photos:
+
+![Posed RGB-D frames and camera trajectory go in; the mesh textured by texture3d is rendered back from the same poses](docs/figures/pipeline.png)
+
+```bash
+# input/output check above
+.venv/bin/python tools/make_readme_figure.py assets/datasets/apt0_seq0 textured/apt0_seq0 \
+  --frames 100,350,600,850 --out docs/figures/pipeline.png
+# purpose / why / how figures used in this README
+.venv/bin/python tools/make_story_figures.py assets/datasets/apt0_seq0 \
+  assets/reconstructions/apt0_seq0/mesh.ply textured/apt0_seq0 --out docs/figures
+```
+
+Package layout: `src/texture3d/` is the installable library (`bake`, `camera`, `io`, `mapping/`). `rgbd_recon/` is reconstruction only.
+
+## RGB-D reconstruction
+
+Optional. Fuses a `dataset.json` sequence with Open3D TSDF (OpenCV depth cleanup first).
+
+```bash
+.venv/bin/pip install -r requirements.txt
+.venv/bin/python run_reconstruction.py assets/datasets/apt0_seq0
+.venv/bin/python run_reconstruction.py assets/datasets/apt0_seq0 --stride 3 --voxel 0.02
+```
+
+Writes to `reconstructions/<name>/` by default: `mesh.ply`, `point_cloud.ply`, `trajectory_tum.txt`, `config.json`, `summary.json`.
+
+`create_rgbd_dataset.py` cuts contiguous windows from a BundleFusion-style capture into that layout. Three `apt0` windows used while developing:
+
+| Subset | Source frames |
+| --- | --- |
+| `apt0_seq0` | 0–999 |
+| `apt0_seq1` | 3780–4779 |
+| `apt0_seq2` | 7560–8559 |
+
+Those sequences are not in the git tree (`assets/` is gitignored). Point `--output-root` / dataset paths at your own copy.
+
+## Dataset layout
+
+`texture3d.load_dataset(dir)` expects `dir/dataset.json`:
+
+```json
+{
+  "image_size": {"width": 640, "height": 480},
+  "intrinsics": {
+    "color_matrix_3x3": [[fx, 0, cx], [0, fy, cy], [0, 0, 1]],
+    "depth_matrix_3x3": null
+  },
+  "depth": {"scale_to_meters": 0.001},
+  "frames": [
+    {"frame_id": "000000", "rgb": "000000/rgb.jpg", "depth": "000000/depth.png", "pose": "000000/pose.txt"}
+  ]
+}
+```
+
+Paths are relative to the dataset directory. `pose.txt` is a 4×4 camera-to-world matrix. Depth PNGs are typically uint16 millimetres.
+
+## Testing
+
+```bash
+.venv/bin/pip install -e ".[dev]"
+.venv/bin/python -m pytest -q
+```
+
+15 tests: rasterisers, labelling, atlas packing, seam-leveling helpers, and a synthetic checker-plane bake. No GPU required.
+
+```bash
+.venv/bin/ruff check src tests
+```
+
+## Limitations
+
+- Mesh and cameras must already share a world frame. This library does not estimate poses or reconstruct geometry (use `rgbd_recon` or another engine).
+- Input UVs are discarded.
+- Unseen faces (back of a TSDF shell, or never projected) stay `empty_color` unless `fill_unseen` can borrow a neighbour.
+- Not published to PyPI; install from git.
+- Measured timings are from one Mac; local leveling dominates on dense meshes.
+
+## Contributing
+
+Issues and PRs are welcome.
+
+```bash
+git clone https://github.com/saikiran321/texture3d.git
+cd texture3d
+python3 -m venv .venv
+.venv/bin/pip install -e ".[dev]"
+.venv/bin/python -m pytest -q
+```
+
+<!-- TODO: add CONTRIBUTING.md / CODE_OF_CONDUCT.md if you want a formal process -->
+
+## License
+
+MIT — see `pyproject.toml`.
+
+<!-- TODO: add a LICENSE file at the repo root so GitHub detects the license automatically -->
+
+—
+
+[Sai Kiran](https://github.com/saikiran321)