fabriks 1.0.0__tar.gz

fabriks-1.0.0/.gitignore

@@ -0,0 +1,156 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+**/__pycache__
+*.py[cod]
+*$py.class
+*.pyc
+*.DS_Store
+db_like.db
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+pip-wheel-metadata/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+.pybuilder/
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+#   For a library or package, you might want to ignore these files since the code is
+#   intended to run in multiple environments; otherwise, check them in:
+# .python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# pytype static type analyzer
+.pytype/
+
+# Cython debug symbols
+cython_debug/
+
+# static files generated from Django application using `collectstatic`
+media
+export
+static_collected
+data
+token.temp
+.source.swp
+
+# fabriks scratch: collections written by hand while poking at the format
+/scratch/
+
+# What the examples write
+examples/out/

fabriks-1.0.0/PKG-INFO

@@ -0,0 +1,378 @@
+Metadata-Version: 2.5
+Name: fabriks
+Version: 1.0.0
+Summary: a level-of-detail mesh wire format: octree-partitioned surfaces written to any object store
+Author-email: jhnnsrs <jhnnsrs@gmail.com>
+License-Expression: MIT
+Requires-Python: >=3.11
+Requires-Dist: fast-simplification>=0.1.7
+Requires-Dist: numpy>=1.26
+Requires-Dist: pyarrow>=12
+Requires-Dist: scipy>=1.11
+Requires-Dist: shapely>=2.0
+Requires-Dist: trimesh>=4.0
+Provides-Extra: build
+Requires-Dist: uv>=0.7.12; extra == 'build'
+Provides-Extra: complete
+Requires-Dist: meshoptimizer>=0.2; extra == 'complete'
+Requires-Dist: obstore>=0.10; extra == 'complete'
+Provides-Extra: meshopt
+Requires-Dist: meshoptimizer>=0.2; extra == 'meshopt'
+Provides-Extra: obstore
+Requires-Dist: obstore>=0.10; extra == 'obstore'
+Description-Content-Type: text/markdown
+
+# fabriks
+
+[![PyPI version](https://badge.fury.io/py/fabriks.svg)](https://pypi.org/project/fabriks/)
+[![Maintenance](https://img.shields.io/badge/Maintained%3F-yes-green.svg)](https://pypi.org/project/fabriks/)
+![Maintainer](https://img.shields.io/badge/maintainer-jhnnsrs-blue)
+[![PyPI pyversions](https://img.shields.io/pypi/pyversions/fabriks.svg)](https://pypi.python.org/pypi/fabriks/)
+[![PyPI status](https://img.shields.io/pypi/status/fabriks.svg)](https://pypi.python.org/pypi/fabriks/)
+
+**A level-of-detail wire format for meshes.** fabriks turns a pile of surfaces into an
+octree-partitioned, self-describing tree of Parquet files, so a viewer fetches the detail the
+view actually needs instead of the whole thing.
+
+It is a *serializer*. It has no client, no network code and no opinion about where the bytes
+go: you hand it a store, and the same tree lands on a local disk or in an S3 prefix.
+
+## Why
+
+A segmentation of a large volume is tens of thousands of surfaces and hundreds of megabytes of
+triangles. A viewer that wants to draw it has two bad options — download everything, or ask a
+server to prepare something. fabriks takes the third: **partition once, at write time**, into a
+structure a dumb object store can serve and a renderer can plan against.
+
+- **An octree of cells.** Level 0 is full detail; each coarser level has one cell per eight
+  finer ones and a quarter of the faces. A renderer picks a level per cell, per frame.
+- **Two catalogs, answering opposite questions.** `cells.parquet` is the spatial index — read
+  once at mount, it decides which cells to fetch at which level without opening any geometry.
+  `objects.parquet` is the identity index, inverted — it answers *"where is segment 4711?"*
+  with a set of cell keys, making isolation and picking a lookup rather than a scan.
+- **No cracks between levels.** Vertices on cell faces are pinned, so a fine cell drawn next to
+  a coarse one meets it exactly. That property is what makes drawing a single object out of the
+  collection viable, and it is the writer's whole job.
+
+## Install
+
+```bash
+pip install fabriks              # everything needed to build and read collections
+pip install 'fabriks[obstore]'   # + obstore, to write to S3 and friends
+pip install 'fabriks[meshopt]'   # + the optional MESHOPT blob codec
+```
+
+Everything that takes part in building a collection is a dependency rather than an extra:
+trimesh cuts a mesh at the cell planes and `fast-simplification` makes the coarse levels.
+scipy and shapely are named explicitly because trimesh does not require them and the clipper
+does — `slice_mesh_plane` imports `scipy.spatial.cKDTree` and `trimesh.path.polygons` (an
+unguarded `from shapely import ops`) in its body, verified by blocking each in turn.
+
+What is genuinely optional is what a *consumer* would otherwise need code for: reaching a
+remote store, and decoding a compressed blob.
+
+fabriks ships a `py.typed` marker, so your type checker sees its annotations: vertices are
+`NDArray[np.float64]`, faces `NDArray[np.int64]`, and the three pluggable pieces — stores,
+codecs and simplifiers — are structural protocols you can satisfy without importing a base
+class. The package itself is checked with basedpyright in strict mode.
+
+## Examples
+
+Three runnable scripts in [`examples/`](examples/), in order — trimesh to the format, the
+format back to trimesh, and what the level-of-detail machinery actually buys:
+
+```bash
+uv run python examples/01_trimesh_to_fabriks.py
+uv run python examples/02_fabriks_to_trimesh.py
+uv run python examples/03_level_of_detail.py
+```
+
+## Writing
+
+```python
+import trimesh
+from obstore.store import LocalStore
+import fabriks
+
+objects = {
+    7: trimesh.creation.icosphere(radius=18.0).apply_translation([200, 160, 60]),
+    3: trimesh.creation.box(extents=[40, 24, 16]).apply_translation([90, 70, 40]),
+}
+
+manifest = fabriks.write_meshes(
+    objects,
+    LocalStore("/data"),          # or S3Store(...), or fabriks.DirectoryStore("/data")
+    prefix="my-collection",
+    cell_size=(128, 128, 64),     # in voxels, in the same component order as the vertices
+    levels=3,
+)
+```
+
+Objects are keyed by the id they carry in whatever they were extracted from — a label volume's
+instance ids, say — and those ids are written through unchanged. Each value is a
+`trimesh.Trimesh`, a `fabriks.Mesh`, or a plain `(vertices, faces)` pair of arrays.
+
+To inspect or check the frames before spending the writes, build and write in two steps:
+
+```python
+collection = fabriks.build_collection(objects, cell_size=(128, 128, 64))
+collection.cell_catalog      # pyarrow.Table -- the spatial index
+collection.object_catalog    # pyarrow.Table -- the identity index
+collection.shards            # [(level, pyarrow.Table)]
+collection.write(store, "my-collection")
+```
+
+## The tree it writes
+
+```
+my-collection/
+  fabriks.json                  <- the manifest, written LAST
+  catalog/cells.parquet        <- one row per (level, cell)
+  catalog/objects.parquet      <- one row per object
+  level=0/part-00000.parquet   <- the geometry, finest level
+  level=1/part-00000.parquet
+  level=2/part-00000.parquet
+```
+
+**The manifest lands last, and that is the completion protocol.** A prefix has no atomic
+"upload finished" flag: a `PutObject` either happened or it did not, but a tree is a sequence
+of writes that can stop anywhere. So every file the manifest names is written before the
+manifest is, and a prefix without one is an *interrupted write* rather than a collection —
+which turns "this thing is half written" from something a renderer discovers into something
+opening it rejects.
+
+**The geometry lands before the catalog that points into it.** Each level part is written with
+one Parquet row group per byte-budgeted run of cells, and the cell catalog records, per cell,
+the part and row group holding it — facts about bytes that do not exist until the bytes do. The
+manifest then records each file's length, because that is the one thing a reader cannot
+discover: fabriks asks a store for `put`/`get`/`list` and nothing more, and a Parquet footer
+lives at the end of a file you have to be able to seek to.
+
+## Reading and planning
+
+```python
+collection = fabriks.open_collection(LocalStore("/data"), "my-collection")
+
+collection.grid.cell_size      # (128, 128, 64)
+collection.encoding.codec      # 'NONE'
+collection.cells               # the whole spatial index, {(level, cell): CellEntry}
+
+# Which cells, at which level, for this view? Answered from the catalog alone.
+camera = fabriks.Camera.perspective((0, 0, 500), fov_y=0.8, viewport_height=1080)
+plan = collection.plan(camera=camera, pixel_budget=1.0)
+
+# Each cell costs the row group holding it, not the level it came from. `read_cells` goes
+# further: it reads a row group once however many of the planned cells share it.
+for cell in collection.read_cells([(entry.level, entry.cell) for entry in plan]):
+    draw(cell.vertices, cell.faces)
+
+collection.release()           # drop the cached levels; the catalogs stay
+
+# Or spend the budget in voxels, with no camera at all.
+collection.plan(error_budget=0.5)
+
+# One object, reassembled across every cell that holds a piece of it.
+mesh = collection.object_mesh(7)
+
+# One object out of a shared cell, with its indices re-based.
+piece = collection.read_cell(0, 3).object_mesh(7)
+```
+
+The planner descends from the coarsest level, keeps a cell when its LOD error fits the budget,
+and otherwise descends into the children `child_mask` names — so descending costs no listing
+and no second query. It also takes a query `box`, `frustum` planes, an `objects` filter
+resolved through the object catalog, and a `max_cells` cap that **degrades detail rather than
+dropping geometry**: running out of budget gives you a coarser cell, never a hole. Every
+`CellEntry` also carries `blob_bytes`, so a plan can be budgeted in bytes before a single fetch.
+
+## Reading without blocking
+
+A frame is forty cells, and forty sequential round trips to an object store is not a frame:
+
+```python
+collection = await fabriks.aopen_collection(S3Store(...), "my-collection")
+plan = collection.plan(camera=camera)
+cells = await collection.aread_cells([(e.level, e.cell) for e in plan], concurrency=16)
+```
+
+The work arrives in two waves, because the second cannot be known without the first: the
+footers of every part the plan touches, then the byte span of every row group it needs — each
+wave issued at once. Only then is anything parsed, and the decode goes to a worker thread.
+
+pyarrow's reader is synchronous and fabriks does not pretend otherwise; what is asynchronous is
+the part that is actually I/O. A store carrying `get_async` / `get_range_async` (obstore does)
+has them used directly; one without them has its sync methods run in a thread, which for a
+network round trip overlaps just as well.
+
+## Stores
+
+fabriks asks a store for three methods — `put(path, data)`, `get(path)`, `list(prefix)` — and
+uses a fourth if it is there: `get_range(path, start=..., length=...)`. That is deliberately the
+shape [obstore][obstore] already has, so its `S3Store`, `LocalStore`, `GCSStore`, `AzureStore`
+and `MemoryStore` all work **as they are**, with no adapter and without obstore being a
+dependency of fabriks.
+
+`get_range` is the optional one because it is what a hand-rolled store is likeliest to be
+missing, and its absence has to degrade rather than fail: without it, reading a cell falls back
+to fetching its whole level part and slicing — correct, and exactly what fabriks did before the
+locator existed. With it, a cell costs its row group.
+
+For a plain path and no dependencies, `fabriks.DirectoryStore("/data")` does the same job;
+`fabriks.MemoryStore()` is there for tests. Both implement `get_range`.
+
+## Simplification
+
+Each coarser level is the same surfaces with fewer triangles, and which algorithm does that
+reduction is named the way a codec is — a value out of a small vocabulary:
+
+```python
+fabriks.build_collection(objects, cell_size=(128, 128, 64))                  # QUADRIC, the default
+fabriks.build_collection(objects, cell_size=..., simplifier="GREEDY")
+fabriks.build_collection(objects, cell_size=..., simplifier=fabriks.GreedyEdgeCollapse())  # configured
+fabriks.build_collection(objects, cell_size=..., decimation=fabriks.Decimation.half())
+```
+
+| Backend | What it is |
+| --- | --- |
+| `"QUADRIC"` (`QuadricSimplifier`) | **The default**, backed by [fast-simplification][fs]. Quadric edge collapse run with `preserve_border=True`, which pins every vertex on the cut curve at *exactly* its input position while letting interior vertices move to the shape-optimal spot. That split is what the format wants: the cut curve is the only thing a neighbour shares, so it is the only thing that must not move. |
+| `"GREEDY"` (`GreedyEdgeCollapse`) | Shortest-edge collapse in pure numpy. Lower quality and a much looser error estimate, but it pins only *this* level's cell planes, so it reduces harder on heavily cut objects. |
+
+The trade is real and worth knowing before you pick. `preserve_border` is all-or-nothing: after
+a coarse cell welds its children, the topological boundary still contains the level-0 seams
+*interior* to that cell, and those get pinned too even though nothing across a face depends on
+them. On a heavily cut object that can be most of the boundary, and the collapse then falls
+short of its budget — measured on one box cut into 34 fragments, the quadric backend stopped at
+74% where the greedy collapse reached 9%. fabriks warns when a level misses its budget and names
+which cause the numbers support.
+
+[fs]: https://github.com/pyvista/fast-simplification
+
+`Decimation` controls how much survives each level — `quarter()` (the default), `half()`,
+`eighth()`, or `custom(ratio)` — plus `floor_faces`, the smallest budget any one object's piece
+is given. The ratio and the name written into `encoding.decimation` are **required to agree**:
+declaring `QUARTER` while reducing by half would be a claim about the geometry that nothing
+downstream could test.
+
+Bring your own by implementing one method — `fabriks.Simplifier` is a structural protocol, so
+there is nothing to inherit:
+
+```python
+class MySimplifier:
+    name = "mine"
+    uses_fixed_mask = True
+    def simplify(self, vertices, faces, *, fixed, target_faces) -> fabriks.Simplified: ...
+```
+
+## Checking one
+
+Two of the declarations above are kept by the writer or not at all, and no renderer can see
+whether they were — it only ever looks at one cell. Given the whole collection they are
+perfectly checkable, so fabriks ships the thing that checks:
+
+```python
+report = fabriks.verify(collection, tier="geometry")
+if not report:
+    print(report)          # every failed check, with examples
+```
+
+Three tiers, because they cost very different amounts:
+
+| tier | reads | answers |
+| --- | --- | --- |
+| `structure` | the manifest and two catalogs | do the files exist, are the recorded lengths right, does `child_mask` name the children that exist, do the locators point inside real row groups, do the two catalogs agree |
+| `blobs` | every cell | does every blob decode to the counts its row claims, do indices stay inside their vertex array, is every vertex inside its own cell box |
+| `geometry` | every level, compared | `boundary: LOCKED` — are on-plane vertices held fixed across levels; `decimation` — is a coarse level actually smaller; is `lod_error` a real bound on deviation from the level-0 surface |
+
+`structure` reads two small files and no geometry, which makes it cheap enough to run at
+registration time — the point in a pipeline where rejecting a bad collection is free.
+
+Nothing raises. A verifier that stops at the first problem tells you about one thing when you
+wanted all of them, so every check runs and the report carries the lot.
+
+## Blob encoding
+
+Two independent knobs, and **both default to `NONE`** — a blob is then the raw little-endian
+layout the format describes, which a consumer reads out of the Parquet column and uploads to
+the GPU with nothing in between:
+
+```python
+fabriks.build_collection(objects, cell_size=...)                              # NONE / NONE
+fabriks.build_collection(objects, cell_size=..., compression="ZSTD")          # smaller on disk
+fabriks.build_collection(objects, cell_size=..., codec="MESHOPT")             # needs [meshopt]
+```
+
+| | what it is | measured on the demo scene |
+| --- | --- | --- |
+| `codec: NONE` | positions are 6 bytes a vertex, indices 4 bytes an index | 366 KB |
+| `compression: ZSTD` | each blob is a zstd frame; its length comes from the row's vertex/index count, since the framing carries none | 355 KB |
+| `codec: MESHOPT` | glTF's `EXT_meshopt_compression`, needing a decoder on the reading side | 237 KB |
+
+The honest summary: ZSTD buys ~3% because the Parquet file is already zstd-compressed around
+the blobs, so it is mostly redundant. MESHOPT buys 35% and costs the consumer a decoder.
+`NONE`/`NONE` costs nothing and needs nothing, which is why it is the default.
+
+`codec: MESHOPT` with `compression: ZSTD` is refused rather than merely discouraged: the ZSTD
+framing derives a blob's uncompressed length from the row's counts, and a meshopt blob has no
+fixed size per element, so the pair is undecodable.
+
+## Coordinates
+
+**fabriks addresses components by position, never by name.** Vertices, `cell_size` and the
+`bbox_*` columns are component 0, 1 and 2 — and nothing in the writer, the octree or the
+planner asks what those components mean. Feed them in whatever order your data already has,
+as long as you feed them *consistently*, and you get the same octree either way.
+
+That matters because meshes usually come out of marching cubes over a `(z, y, x)` array, and
+the imaging stack around them is `(z, y, x)` throughout. There is no house convention to
+transpose into first:
+
+```python
+# vertices from a (z, y, x) volume, and the chunk shape of that same volume
+fabriks.write_meshes(objects, store, prefix=key, cell_size=(64, 128, 128))
+```
+
+The `x`/`y`/`z` in the `bbox_min_x` / `bbox_max_z` column names are **labels for slots 0, 1
+and 2**, fixed by the Parquet schema a server checks with a `DESCRIBE`. They are not a claim
+about which physical axis each slot holds, and the manifest makes no such claim either: naming
+these axes says how the collection relates to the image it came from or the coordinate graph it
+sits in, which is knowledge the layer that owns that coordinate system has and fabriks does not.
+A field here would be a claim nothing in the format could check, use or contradict.
+
+Two consequences worth knowing:
+
+- **An order mistake here cannot misplace geometry.** Clipping and quantization read the same
+  `cell_size`, so a mismatched one gives you a differently *shaped* octree — cells that fit
+  the data's anisotropy less well, so the tree narrows a fetch less — while every vertex still
+  decodes exactly where it started. The failure mode is efficiency, not correctness.
+- **`cell_size` is still worth matching to your source array's chunk shape.** In whatever
+  order that shape is in. A cell that matches the chunking means a viewer fetching image
+  chunks and mesh cells pulls the same regions, and nothing about the meshes themselves can
+  reveal it.
+
+## What the format promises, and what it does not
+
+Two declarations are true by construction and unverifiable by anything downstream, so they are
+kept by the writer or not at all:
+
+- **`boundary: LOCKED`** — every object is cut *once*, at the level-0 planes; coarser levels
+  are assembled by welding children and decimated with on-plane vertices held fixed. See
+  `fabriks/geometry.py` for the argument, including the residual case that follows from 65535
+  being odd.
+- **`decimation: QUARTER`** — level `L` targets `(1/4)**L` of the level-0 face count. When a
+  collection cannot reach it — usually a cell size small relative to the objects, so every cut
+  vertex is pinned — fabriks warns and says which cause the numbers support.
+
+- **`sortKey: MORTON`** — rows within a level are written in ascending Morton order. That is
+  what makes a row group a spatially compact set of cells rather than an arbitrary one, and so
+  what makes a row group the right unit for a reader to fetch.
+
+What it does *not* do: it holds every object in memory and builds one shard per level, so it is
+sized for thousands of objects rather than millions; and `lod_error` is an upper bound rather
+than a measured Hausdorff distance.
+
+The byte format is documented in `fabriks/codecs/`, how space is divided in `fabriks/octree.py`,
+and the tree layout in `fabriks/manifest.py`. The three pluggable pieces are packages with the
+same shape — a protocol module and one module per implementation: `fabriks/stores/`,
+`fabriks/codecs/` and `fabriks/simplifiers/`.
+
+[obstore]: https://developmentseed.org/obstore/