macrodata-refiner 0.2.2__tar.gz → 0.3.1__tar.gz

{macrodata_refiner-0.2.2 → macrodata_refiner-0.3.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: macrodata-refiner
-Version: 0.2.2
+Version: 0.3.1
 Summary: Refiner by Macrodata Labs, a data processing framework for Machine Learning large scale datasets
 Author: Macrodata Labs
 License-Expression: Apache-2.0
@@ -12,35 +12,61 @@ Classifier: Programming Language :: Python :: 3.12
 Requires-Python: >=3.10
 Description-Content-Type: text/markdown
 License-File: LICENSE
+Requires-Dist: aiohttp
 Requires-Dist: cloudpickle==3.1.2
-Requires-Dist: fsspec
+Requires-Dist: fsspec[http]
 Requires-Dist: httpx
 Requires-Dist: loguru
-Requires-Dist: opentelemetry-exporter-otlp-proto-http
-Requires-Dist: opentelemetry-sdk
 Requires-Dist: numpy
-Requires-Dist: psutil
 Requires-Dist: orjson
+Requires-Dist: packaging
 Requires-Dist: pyarrow
 Requires-Dist: msgspec>=0.20.0
+Requires-Dist: pydantic>=2.0.0
 Provides-Extra: video
 Requires-Dist: av; extra == "video"
-Provides-Extra: robotics
-Requires-Dist: macrodata-refiner[video]; extra == "robotics"
-Requires-Dist: huggingface-hub>=1.4.1; extra == "robotics"
-Requires-Dist: hf>=1.7.1; extra == "robotics"
+Requires-Dist: pillow; extra == "video"
+Provides-Extra: hf
+Requires-Dist: datasets>=3.0.0; extra == "hf"
+Requires-Dist: huggingface-hub>=1.4.1; extra == "hf"
+Requires-Dist: hf>=1.7.1; extra == "hf"
+Provides-Extra: hand-tracking
+Requires-Dist: macrodata-refiner[hf]; extra == "hand-tracking"
+Requires-Dist: macrodata-refiner[video]; extra == "hand-tracking"
+Requires-Dist: ego-vision[models]>=0.1.25; extra == "hand-tracking"
 Provides-Extra: text
 Requires-Dist: warcio; extra == "text"
+Provides-Extra: hdf5
+Requires-Dist: h5py; extra == "hdf5"
+Provides-Extra: zarr
+Requires-Dist: zarr<3,>=2.18; extra == "zarr"
+Requires-Dist: numcodecs<0.16; extra == "zarr"
+Provides-Extra: mcap
+Requires-Dist: av; extra == "mcap"
+Requires-Dist: mcap; extra == "mcap"
+Requires-Dist: mcap-protobuf-support; extra == "mcap"
+Requires-Dist: mcap-ros2-support; extra == "mcap"
+Requires-Dist: pillow; extra == "mcap"
 Provides-Extra: s3
 Requires-Dist: s3fs; extra == "s3"
+Provides-Extra: tensorflow
+Requires-Dist: tensorflow; extra == "tensorflow"
+Provides-Extra: tfds
+Requires-Dist: macrodata-refiner[tensorflow]; extra == "tfds"
+Requires-Dist: tensorflow-datasets; extra == "tfds"
 Provides-Extra: testing
-Requires-Dist: macrodata-refiner[robotics]; extra == "testing"
-Requires-Dist: macrodata-refiner[text]; extra == "testing"
-Requires-Dist: macrodata-refiner[s3]; extra == "testing"
+Requires-Dist: macrodata-refiner[all]; extra == "testing"
 Requires-Dist: pytest>=8.0.0; extra == "testing"
 Requires-Dist: pytest-cov>=5.0.0; extra == "testing"
 Provides-Extra: all
-Requires-Dist: macrodata-refiner[testing]; extra == "all"
+Requires-Dist: macrodata-refiner[hdf5]; extra == "all"
+Requires-Dist: macrodata-refiner[hf]; extra == "all"
+Requires-Dist: macrodata-refiner[mcap]; extra == "all"
+Requires-Dist: macrodata-refiner[video]; extra == "all"
+Requires-Dist: macrodata-refiner[zarr]; extra == "all"
+Requires-Dist: macrodata-refiner[text]; extra == "all"
+Requires-Dist: macrodata-refiner[s3]; extra == "all"
+Requires-Dist: macrodata-refiner[tfds]; extra == "all"
 Dynamic: license-file
 
 <p align="center">
@@ -49,9 +75,10 @@ Dynamic: license-file
 
 <h1 align="center">Macrodata Refiner</h1>
 
-Refiner is an open-source engine for turning raw, unstructured, and multimodal data into **high-quality datasets** for large model training.
+Refiner is an open-source engine for turning raw robotics and multimodal data into **high-quality datasets** for model training.
 
-It replaces the brittle scripts and stitched-together data tooling that teams still use for training data work, while offering much better support for multimodal data, robotics workflows, and model-based processing.
+It gives training-data teams one pipeline model for multimodal data, robotics
+workflows, and model-based processing.
 
 It also plugs into the Macrodata platform, which gives you visibility into what is happening to your data while pipelines run: job and shard lifecycle, logs, metrics, manifests, and pipeline behavior. The same code can run locally for development and then scale out through Macrodata's elastic serverless cloud.
 
@@ -90,7 +117,7 @@ import refiner as mdr
             pad_frames=5,
         )
     )
-    .write_lerobot("hf://buckets/macrodata/test_bucket/aloha_motion")
+    .write_lerobot("hf://buckets/acme-robotics/aloha_motion")
     .launch_cloud(
         name="motion_trim",
         num_workers=4,
@@ -98,7 +125,7 @@ import refiner as mdr
 )
 ```
 
-Need cloud GPUs? See [Launchers](docs/launchers.md) for the GPU-specific cloud options.
+Need cloud GPUs? See [Resources, GPUs, and Services](docs/running-pipelines/resources-gpus-and-services.md).
 
 ### Local example
 
@@ -137,31 +164,33 @@ def add_preview(row):
 
 - training-data-first pipeline primitives instead of generic ETL abstractions
 - multimodal processing, with robotics support today
-- a lot of built-in readers, transforms, sinks, and lifecycle/runtime machinery so you do not have to rebuild the same scaffolding in scripts
+- built-in readers, transforms, sinks, and runtime machinery for common dataset work
 - access to any storage backend supported by `fsspec` (S3, GCP, Hugging Face, etc.)
 - local execution for development and elastic cloud execution for large runs
-- built-in observability through the Macrodata platform, so you can inspect how your data is changing instead of debugging blindly after the fact
+- built-in observability through the Macrodata platform for job state, logs, metrics, and manifests
 
 ## Docs
 
-Getting started:
+Start here:
 
-- [Pipeline basics](docs/pipeline-basics.md)
-- [Launchers](docs/launchers.md)
-- [CLI](docs/cli.md)
+- [Docs index](docs/index.md)
+- [Quickstart](docs/quickstart.md)
+- [Running pipelines](docs/running-pipelines/index.md)
 
-Core concepts:
+Build a dataset:
 
-- [Reading and writing data](docs/reading-and-writing.md)
-- [Transforms](docs/transforms.md)
-- [Expressions](docs/expressions.md)
-- [In-process debugging](docs/in-process-debugging.md)
-- [Task pipelines](docs/task-pipelines.md)
+- [Reading data](docs/reading-data/index.md)
+- [Episode data](docs/episode-data/index.md)
+- [Transforms](docs/transforms/index.md)
+- [Episode operations](docs/episode-operations/index.md)
+- [Writing data](docs/writing-data/index.md)
+- [Examples](docs/examples/index.md)
 
-Modalities and platform:
+Operate jobs:
 
-- [Robotics](docs/robotics.md)
-- [Observability](docs/observability.md)
+- [Platform](docs/platform/index.md)
+- [CLI](docs/cli/index.md)
+- [Reference](docs/reference/index.md)
 
 ## Community
 

{macrodata_refiner-0.2.2 → macrodata_refiner-0.3.1}/README.md

@@ -4,9 +4,10 @@
 
 <h1 align="center">Macrodata Refiner</h1>
 
-Refiner is an open-source engine for turning raw, unstructured, and multimodal data into **high-quality datasets** for large model training.
+Refiner is an open-source engine for turning raw robotics and multimodal data into **high-quality datasets** for model training.
 
-It replaces the brittle scripts and stitched-together data tooling that teams still use for training data work, while offering much better support for multimodal data, robotics workflows, and model-based processing.
+It gives training-data teams one pipeline model for multimodal data, robotics
+workflows, and model-based processing.
 
 It also plugs into the Macrodata platform, which gives you visibility into what is happening to your data while pipelines run: job and shard lifecycle, logs, metrics, manifests, and pipeline behavior. The same code can run locally for development and then scale out through Macrodata's elastic serverless cloud.
 
@@ -45,7 +46,7 @@ import refiner as mdr
             pad_frames=5,
         )
     )
-    .write_lerobot("hf://buckets/macrodata/test_bucket/aloha_motion")
+    .write_lerobot("hf://buckets/acme-robotics/aloha_motion")
     .launch_cloud(
         name="motion_trim",
         num_workers=4,
@@ -53,7 +54,7 @@ import refiner as mdr
 )
 ```
 
-Need cloud GPUs? See [Launchers](docs/launchers.md) for the GPU-specific cloud options.
+Need cloud GPUs? See [Resources, GPUs, and Services](docs/running-pipelines/resources-gpus-and-services.md).
 
 ### Local example
 
@@ -92,31 +93,33 @@ def add_preview(row):
 
 - training-data-first pipeline primitives instead of generic ETL abstractions
 - multimodal processing, with robotics support today
-- a lot of built-in readers, transforms, sinks, and lifecycle/runtime machinery so you do not have to rebuild the same scaffolding in scripts
+- built-in readers, transforms, sinks, and runtime machinery for common dataset work
 - access to any storage backend supported by `fsspec` (S3, GCP, Hugging Face, etc.)
 - local execution for development and elastic cloud execution for large runs
-- built-in observability through the Macrodata platform, so you can inspect how your data is changing instead of debugging blindly after the fact
+- built-in observability through the Macrodata platform for job state, logs, metrics, and manifests
 
 ## Docs
 
-Getting started:
+Start here:
 
-- [Pipeline basics](docs/pipeline-basics.md)
-- [Launchers](docs/launchers.md)
-- [CLI](docs/cli.md)
+- [Docs index](docs/index.md)
+- [Quickstart](docs/quickstart.md)
+- [Running pipelines](docs/running-pipelines/index.md)
 
-Core concepts:
+Build a dataset:
 
-- [Reading and writing data](docs/reading-and-writing.md)
-- [Transforms](docs/transforms.md)
-- [Expressions](docs/expressions.md)
-- [In-process debugging](docs/in-process-debugging.md)
-- [Task pipelines](docs/task-pipelines.md)
+- [Reading data](docs/reading-data/index.md)
+- [Episode data](docs/episode-data/index.md)
+- [Transforms](docs/transforms/index.md)
+- [Episode operations](docs/episode-operations/index.md)
+- [Writing data](docs/writing-data/index.md)
+- [Examples](docs/examples/index.md)
 
-Modalities and platform:
+Operate jobs:
 
-- [Robotics](docs/robotics.md)
-- [Observability](docs/observability.md)
+- [Platform](docs/platform/index.md)
+- [CLI](docs/cli/index.md)
+- [Reference](docs/reference/index.md)
 
 ## Community
 

{macrodata_refiner-0.2.2 → macrodata_refiner-0.3.1}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "macrodata-refiner"
-version = "0.2.2"
+version = "0.3.1"
 description = "Refiner by Macrodata Labs, a data processing framework for Machine Learning large scale datasets"
 readme = "README.md"
 license = "Apache-2.0"
@@ -16,47 +16,80 @@ authors = [
 ]
 requires-python = ">=3.10"
 dependencies = [
+    "aiohttp",
     "cloudpickle==3.1.2",
-    "fsspec",
+    "fsspec[http]",
     "httpx",
     "loguru",
-    "opentelemetry-exporter-otlp-proto-http",
-    "opentelemetry-sdk",
     "numpy",
-    "psutil",
     "orjson",
+    "packaging",
     "pyarrow",
     "msgspec>=0.20.0",
+    "pydantic>=2.0.0",
 ]
 
 [project.optional-dependencies]
 video = [
     "av",
+    "pillow",
 ]
-robotics = [
-    "macrodata-refiner[video]",
+hf = [
+    "datasets>=3.0.0",
     "huggingface-hub>=1.4.1",
     "hf>=1.7.1",
 ]
+hand_tracking = [
+    "macrodata-refiner[hf]",
+    "macrodata-refiner[video]",
+    "ego-vision[models]>=0.1.25",
+]
 text = [
     "warcio",
 ]
+hdf5 = [
+    "h5py",
+]
+zarr = [
+    "zarr>=2.18,<3",
+    "numcodecs<0.16",
+]
+mcap = [
+    "av",
+    "mcap",
+    "mcap-protobuf-support",
+    "mcap-ros2-support",
+    "pillow",
+]
 s3 = [
     "s3fs",
 ]
+tensorflow = [
+    "tensorflow",
+]
+tfds = [
+    "macrodata-refiner[tensorflow]",
+    "tensorflow-datasets",
+]
 testing = [
-    "macrodata-refiner[robotics]",
-    "macrodata-refiner[text]",
-    "macrodata-refiner[s3]",
+    "macrodata-refiner[all]",
     "pytest>=8.0.0",
     "pytest-cov>=5.0.0",
 ]
 all = [
-    "macrodata-refiner[testing]",
+    "macrodata-refiner[hdf5]",
+    "macrodata-refiner[hf]",
+    "macrodata-refiner[mcap]",
+    "macrodata-refiner[video]",
+    "macrodata-refiner[zarr]",
+    "macrodata-refiner[text]",
+    "macrodata-refiner[s3]",
+    "macrodata-refiner[tfds]",
 ]
 
 [project.scripts]
 macrodata = "refiner.cli.main:main"
+mdr = "refiner.cli.main:main"
 
 [build-system]
 requires = ["setuptools>=77", "wheel"]
@@ -70,7 +103,7 @@ refiner = ["py.typed"]
 
 [dependency-groups]
 dev = [
-    "macrodata-refiner[all]",
+    "macrodata-refiner[testing]",
     "pre-commit>=4.0.0",
     "ruff>=0.14.10",
     "ty>=0.0.7",

{macrodata_refiner-0.2.2 → macrodata_refiner-0.3.1}/src/macrodata_refiner.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: macrodata-refiner
-Version: 0.2.2
+Version: 0.3.1
 Summary: Refiner by Macrodata Labs, a data processing framework for Machine Learning large scale datasets
 Author: Macrodata Labs
 License-Expression: Apache-2.0
@@ -12,35 +12,61 @@ Classifier: Programming Language :: Python :: 3.12
 Requires-Python: >=3.10
 Description-Content-Type: text/markdown
 License-File: LICENSE
+Requires-Dist: aiohttp
 Requires-Dist: cloudpickle==3.1.2
-Requires-Dist: fsspec
+Requires-Dist: fsspec[http]
 Requires-Dist: httpx
 Requires-Dist: loguru
-Requires-Dist: opentelemetry-exporter-otlp-proto-http
-Requires-Dist: opentelemetry-sdk
 Requires-Dist: numpy
-Requires-Dist: psutil
 Requires-Dist: orjson
+Requires-Dist: packaging
 Requires-Dist: pyarrow
 Requires-Dist: msgspec>=0.20.0
+Requires-Dist: pydantic>=2.0.0
 Provides-Extra: video
 Requires-Dist: av; extra == "video"
-Provides-Extra: robotics
-Requires-Dist: macrodata-refiner[video]; extra == "robotics"
-Requires-Dist: huggingface-hub>=1.4.1; extra == "robotics"
-Requires-Dist: hf>=1.7.1; extra == "robotics"
+Requires-Dist: pillow; extra == "video"
+Provides-Extra: hf
+Requires-Dist: datasets>=3.0.0; extra == "hf"
+Requires-Dist: huggingface-hub>=1.4.1; extra == "hf"
+Requires-Dist: hf>=1.7.1; extra == "hf"
+Provides-Extra: hand-tracking
+Requires-Dist: macrodata-refiner[hf]; extra == "hand-tracking"
+Requires-Dist: macrodata-refiner[video]; extra == "hand-tracking"
+Requires-Dist: ego-vision[models]>=0.1.25; extra == "hand-tracking"
 Provides-Extra: text
 Requires-Dist: warcio; extra == "text"
+Provides-Extra: hdf5
+Requires-Dist: h5py; extra == "hdf5"
+Provides-Extra: zarr
+Requires-Dist: zarr<3,>=2.18; extra == "zarr"
+Requires-Dist: numcodecs<0.16; extra == "zarr"
+Provides-Extra: mcap
+Requires-Dist: av; extra == "mcap"
+Requires-Dist: mcap; extra == "mcap"
+Requires-Dist: mcap-protobuf-support; extra == "mcap"
+Requires-Dist: mcap-ros2-support; extra == "mcap"
+Requires-Dist: pillow; extra == "mcap"
 Provides-Extra: s3
 Requires-Dist: s3fs; extra == "s3"
+Provides-Extra: tensorflow
+Requires-Dist: tensorflow; extra == "tensorflow"
+Provides-Extra: tfds
+Requires-Dist: macrodata-refiner[tensorflow]; extra == "tfds"
+Requires-Dist: tensorflow-datasets; extra == "tfds"
 Provides-Extra: testing
-Requires-Dist: macrodata-refiner[robotics]; extra == "testing"
-Requires-Dist: macrodata-refiner[text]; extra == "testing"
-Requires-Dist: macrodata-refiner[s3]; extra == "testing"
+Requires-Dist: macrodata-refiner[all]; extra == "testing"
 Requires-Dist: pytest>=8.0.0; extra == "testing"
 Requires-Dist: pytest-cov>=5.0.0; extra == "testing"
 Provides-Extra: all
-Requires-Dist: macrodata-refiner[testing]; extra == "all"
+Requires-Dist: macrodata-refiner[hdf5]; extra == "all"
+Requires-Dist: macrodata-refiner[hf]; extra == "all"
+Requires-Dist: macrodata-refiner[mcap]; extra == "all"
+Requires-Dist: macrodata-refiner[video]; extra == "all"
+Requires-Dist: macrodata-refiner[zarr]; extra == "all"
+Requires-Dist: macrodata-refiner[text]; extra == "all"
+Requires-Dist: macrodata-refiner[s3]; extra == "all"
+Requires-Dist: macrodata-refiner[tfds]; extra == "all"
 Dynamic: license-file
 
 <p align="center">
@@ -49,9 +75,10 @@ Dynamic: license-file
 
 <h1 align="center">Macrodata Refiner</h1>
 
-Refiner is an open-source engine for turning raw, unstructured, and multimodal data into **high-quality datasets** for large model training.
+Refiner is an open-source engine for turning raw robotics and multimodal data into **high-quality datasets** for model training.
 
-It replaces the brittle scripts and stitched-together data tooling that teams still use for training data work, while offering much better support for multimodal data, robotics workflows, and model-based processing.
+It gives training-data teams one pipeline model for multimodal data, robotics
+workflows, and model-based processing.
 
 It also plugs into the Macrodata platform, which gives you visibility into what is happening to your data while pipelines run: job and shard lifecycle, logs, metrics, manifests, and pipeline behavior. The same code can run locally for development and then scale out through Macrodata's elastic serverless cloud.
 
@@ -90,7 +117,7 @@ import refiner as mdr
             pad_frames=5,
         )
     )
-    .write_lerobot("hf://buckets/macrodata/test_bucket/aloha_motion")
+    .write_lerobot("hf://buckets/acme-robotics/aloha_motion")
     .launch_cloud(
         name="motion_trim",
         num_workers=4,
@@ -98,7 +125,7 @@ import refiner as mdr
 )
 ```
 
-Need cloud GPUs? See [Launchers](docs/launchers.md) for the GPU-specific cloud options.
+Need cloud GPUs? See [Resources, GPUs, and Services](docs/running-pipelines/resources-gpus-and-services.md).
 
 ### Local example
 
@@ -137,31 +164,33 @@ def add_preview(row):
 
 - training-data-first pipeline primitives instead of generic ETL abstractions
 - multimodal processing, with robotics support today
-- a lot of built-in readers, transforms, sinks, and lifecycle/runtime machinery so you do not have to rebuild the same scaffolding in scripts
+- built-in readers, transforms, sinks, and runtime machinery for common dataset work
 - access to any storage backend supported by `fsspec` (S3, GCP, Hugging Face, etc.)
 - local execution for development and elastic cloud execution for large runs
-- built-in observability through the Macrodata platform, so you can inspect how your data is changing instead of debugging blindly after the fact
+- built-in observability through the Macrodata platform for job state, logs, metrics, and manifests
 
 ## Docs
 
-Getting started:
+Start here:
 
-- [Pipeline basics](docs/pipeline-basics.md)
-- [Launchers](docs/launchers.md)
-- [CLI](docs/cli.md)
+- [Docs index](docs/index.md)
+- [Quickstart](docs/quickstart.md)
+- [Running pipelines](docs/running-pipelines/index.md)
 
-Core concepts:
+Build a dataset:
 
-- [Reading and writing data](docs/reading-and-writing.md)
-- [Transforms](docs/transforms.md)
-- [Expressions](docs/expressions.md)
-- [In-process debugging](docs/in-process-debugging.md)
-- [Task pipelines](docs/task-pipelines.md)
+- [Reading data](docs/reading-data/index.md)
+- [Episode data](docs/episode-data/index.md)
+- [Transforms](docs/transforms/index.md)
+- [Episode operations](docs/episode-operations/index.md)
+- [Writing data](docs/writing-data/index.md)
+- [Examples](docs/examples/index.md)
 
-Modalities and platform:
+Operate jobs:
 
-- [Robotics](docs/robotics.md)
-- [Observability](docs/observability.md)
+- [Platform](docs/platform/index.md)
+- [CLI](docs/cli/index.md)
+- [Reference](docs/reference/index.md)
 
 ## Community