macrodata-refiner 0.2.1__tar.gz → 0.3.0__tar.gz

macrodata_refiner-0.3.0/PKG-INFO

@@ -0,0 +1,196 @@
+Metadata-Version: 2.4
+Name: macrodata-refiner
+Version: 0.3.0
+Summary: Refiner by Macrodata Labs, a data processing framework for Machine Learning large scale datasets
+Author: Macrodata Labs
+License-Expression: Apache-2.0
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: cloudpickle==3.1.2
+Requires-Dist: fsspec[http]
+Requires-Dist: httpx
+Requires-Dist: loguru
+Requires-Dist: numpy
+Requires-Dist: orjson
+Requires-Dist: packaging
+Requires-Dist: pyarrow
+Requires-Dist: msgspec>=0.20.0
+Requires-Dist: pydantic>=2.0.0
+Provides-Extra: video
+Requires-Dist: av; extra == "video"
+Requires-Dist: pillow; extra == "video"
+Provides-Extra: hf
+Requires-Dist: datasets>=3.0.0; extra == "hf"
+Requires-Dist: huggingface-hub>=1.4.1; extra == "hf"
+Requires-Dist: hf>=1.7.1; extra == "hf"
+Provides-Extra: egocentric
+Requires-Dist: macrodata-refiner[hf]; extra == "egocentric"
+Requires-Dist: macrodata-refiner[video]; extra == "egocentric"
+Requires-Dist: ego-vision[models]>=0.1.8; extra == "egocentric"
+Provides-Extra: text
+Requires-Dist: warcio; extra == "text"
+Provides-Extra: hdf5
+Requires-Dist: h5py; extra == "hdf5"
+Provides-Extra: zarr
+Requires-Dist: zarr<3,>=2.18; extra == "zarr"
+Requires-Dist: numcodecs<0.16; extra == "zarr"
+Provides-Extra: mcap
+Requires-Dist: av; extra == "mcap"
+Requires-Dist: mcap; extra == "mcap"
+Requires-Dist: mcap-protobuf-support; extra == "mcap"
+Requires-Dist: mcap-ros2-support; extra == "mcap"
+Requires-Dist: pillow; extra == "mcap"
+Provides-Extra: s3
+Requires-Dist: s3fs; extra == "s3"
+Provides-Extra: tensorflow
+Requires-Dist: tensorflow; extra == "tensorflow"
+Provides-Extra: tfds
+Requires-Dist: macrodata-refiner[tensorflow]; extra == "tfds"
+Requires-Dist: tensorflow-datasets; extra == "tfds"
+Provides-Extra: testing
+Requires-Dist: macrodata-refiner[hdf5]; extra == "testing"
+Requires-Dist: macrodata-refiner[hf]; extra == "testing"
+Requires-Dist: macrodata-refiner[mcap]; extra == "testing"
+Requires-Dist: macrodata-refiner[video]; extra == "testing"
+Requires-Dist: macrodata-refiner[zarr]; extra == "testing"
+Requires-Dist: macrodata-refiner[text]; extra == "testing"
+Requires-Dist: macrodata-refiner[s3]; extra == "testing"
+Requires-Dist: macrodata-refiner[tfds]; extra == "testing"
+Requires-Dist: pytest>=8.0.0; extra == "testing"
+Requires-Dist: pytest-cov>=5.0.0; extra == "testing"
+Provides-Extra: all
+Requires-Dist: macrodata-refiner[testing]; extra == "all"
+Dynamic: license-file
+
+<p align="center">
+  <img src="https://macrodata.co/logo.svg" alt="Macrodata" width="180">
+</p>
+
+<h1 align="center">Macrodata Refiner</h1>
+
+Refiner is an open-source engine for turning raw robotics and multimodal data into **high-quality datasets** for model training.
+
+It gives training-data teams one pipeline model for multimodal data, robotics
+workflows, and model-based processing.
+
+It also plugs into the Macrodata platform, which gives you visibility into what is happening to your data while pipelines run: job and shard lifecycle, logs, metrics, manifests, and pipeline behavior. The same code can run locally for development and then scale out through Macrodata's elastic serverless cloud.
+
+## Quickstart
+
+Install:
+
+```bash
+pip install macrodata-refiner
+```
+
+Create a Macrodata API key:
+
+- https://macrodata.co/settings/api-keys
+
+Log in:
+
+```bash
+macrodata login
+```
+
+### Cloud example
+
+Launch a robotics pipeline on Macrodata Cloud.
+
+This requires a valid API key.
+
+```python
+import refiner as mdr
+
+(
+    mdr.read_lerobot("hf://datasets/macrodata/aloha_static_battery_ep005_009")
+    .map(
+        mdr.robotics.motion_trim(
+            threshold=0.001,
+            pad_frames=5,
+        )
+    )
+    .write_lerobot("hf://buckets/acme-robotics/aloha_motion")
+    .launch_cloud(
+        name="motion_trim",
+        num_workers=4,
+    )
+)
+```
+
+Need cloud GPUs? See [Resources, GPUs, and Services](docs/running-pipelines/resources-gpus-and-services.md).
+
+### Local example
+
+Launch a local pipeline:
+
+```python
+import refiner as mdr
+
+def add_preview(row):
+    return row.update(
+        preview=" ".join(row["text"].split()[:20]),
+    )
+
+(
+    mdr.read_jsonl("input/*.jsonl")
+    .filter(mdr.col("lang") == "en")
+    .with_columns(
+        text=mdr.col("text").str.strip(),
+        text_len=mdr.col("text").str.len(),
+    )
+    .map(add_preview)
+    .write_parquet("s3://my-bucket/english-cleanup/")
+    .launch_local(
+        name="english-cleanup",
+        num_workers=2,
+    )
+)
+```
+
+`pip install` gives you:
+
+- the Python package as `refiner`
+- the CLI as `macrodata`
+
+## Batteries included
+
+- training-data-first pipeline primitives instead of generic ETL abstractions
+- multimodal processing, with robotics support today
+- built-in readers, transforms, sinks, and runtime machinery for common dataset work
+- access to any storage backend supported by `fsspec` (S3, GCP, Hugging Face, etc.)
+- local execution for development and elastic cloud execution for large runs
+- built-in observability through the Macrodata platform for job state, logs, metrics, and manifests
+
+## Docs
+
+Start here:
+
+- [Docs index](docs/index.md)
+- [Quickstart](docs/quickstart.md)
+- [Running pipelines](docs/running-pipelines/index.md)
+
+Build a dataset:
+
+- [Reading data](docs/reading-data/index.md)
+- [Episode data](docs/episode-data/index.md)
+- [Transforms](docs/transforms/index.md)
+- [Episode operations](docs/episode-operations/index.md)
+- [Writing data](docs/writing-data/index.md)
+- [Examples](docs/examples/index.md)
+
+Operate jobs:
+
+- [Platform](docs/platform/index.md)
+- [CLI](docs/cli/index.md)
+- [Reference](docs/reference/index.md)
+
+## Community
+
+- join the Macrodata Discord: https://discord.gg/S8kZtmBR2x

{macrodata_refiner-0.2.1 → macrodata_refiner-0.3.0}/README.md

@@ -4,9 +4,10 @@
 
 <h1 align="center">Macrodata Refiner</h1>
 
-Refiner is an open-source engine for turning raw, unstructured, and multimodal data into **high-quality datasets** for large model training.
+Refiner is an open-source engine for turning raw robotics and multimodal data into **high-quality datasets** for model training.
 
-It replaces the brittle scripts and stitched-together data tooling that teams still use for training data work, while offering much better support for multimodal data, robotics workflows, and model-based processing.
+It gives training-data teams one pipeline model for multimodal data, robotics
+workflows, and model-based processing.
 
 It also plugs into the Macrodata platform, which gives you visibility into what is happening to your data while pipelines run: job and shard lifecycle, logs, metrics, manifests, and pipeline behavior. The same code can run locally for development and then scale out through Macrodata's elastic serverless cloud.
 
@@ -45,7 +46,7 @@ import refiner as mdr
             pad_frames=5,
         )
     )
-    .write_lerobot("hf://buckets/macrodata/test_bucket/aloha_motion")
+    .write_lerobot("hf://buckets/acme-robotics/aloha_motion")
     .launch_cloud(
         name="motion_trim",
         num_workers=4,
@@ -53,6 +54,8 @@ import refiner as mdr
 )
 ```
 
+Need cloud GPUs? See [Resources, GPUs, and Services](docs/running-pipelines/resources-gpus-and-services.md).
+
 ### Local example
 
 Launch a local pipeline:
@@ -90,31 +93,33 @@ def add_preview(row):
 
 - training-data-first pipeline primitives instead of generic ETL abstractions
 - multimodal processing, with robotics support today
-- a lot of built-in readers, transforms, sinks, and lifecycle/runtime machinery so you do not have to rebuild the same scaffolding in scripts
+- built-in readers, transforms, sinks, and runtime machinery for common dataset work
 - access to any storage backend supported by `fsspec` (S3, GCP, Hugging Face, etc.)
 - local execution for development and elastic cloud execution for large runs
-- built-in observability through the Macrodata platform, so you can inspect how your data is changing instead of debugging blindly after the fact
+- built-in observability through the Macrodata platform for job state, logs, metrics, and manifests
 
 ## Docs
 
-Getting started:
+Start here:
 
-- [Pipeline basics](docs/pipeline-basics.md)
-- [Launchers](docs/launchers.md)
-- [CLI](docs/cli.md)
+- [Docs index](docs/index.md)
+- [Quickstart](docs/quickstart.md)
+- [Running pipelines](docs/running-pipelines/index.md)
 
-Core concepts:
+Build a dataset:
 
-- [Reading and writing data](docs/reading-and-writing.md)
-- [Transforms](docs/transforms.md)
-- [Expressions](docs/expressions.md)
-- [In-process debugging](docs/in-process-debugging.md)
-- [Task pipelines](docs/task-pipelines.md)
+- [Reading data](docs/reading-data/index.md)
+- [Episode data](docs/episode-data/index.md)
+- [Transforms](docs/transforms/index.md)
+- [Episode operations](docs/episode-operations/index.md)
+- [Writing data](docs/writing-data/index.md)
+- [Examples](docs/examples/index.md)
 
-Modalities and platform:
+Operate jobs:
 
-- [Robotics](docs/robotics.md)
-- [Observability](docs/observability.md)
+- [Platform](docs/platform/index.md)
+- [CLI](docs/cli/index.md)
+- [Reference](docs/reference/index.md)
 
 ## Community
 

{macrodata_refiner-0.2.1 → macrodata_refiner-0.3.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "macrodata-refiner"
-version = "0.2.1"
+version = "0.3.0"
 description = "Refiner by Macrodata Labs, a data processing framework for Machine Learning large scale datasets"
 readme = "README.md"
 license = "Apache-2.0"
@@ -16,24 +16,79 @@ authors = [
 ]
 requires-python = ">=3.10"
 dependencies = [
-    "av",
     "cloudpickle==3.1.2",
-    "fsspec",
+    "fsspec[http]",
     "httpx",
     "loguru",
-    "huggingface-hub>=1.4.1",
-    "opentelemetry-exporter-otlp-proto-http",
-    "opentelemetry-sdk",
     "numpy",
-    "psutil",
     "orjson",
+    "packaging",
     "pyarrow",
     "msgspec>=0.20.0",
+    "pydantic>=2.0.0",
+]
+
+[project.optional-dependencies]
+video = [
+    "av",
+    "pillow",
+]
+hf = [
+    "datasets>=3.0.0",
+    "huggingface-hub>=1.4.1",
     "hf>=1.7.1",
 ]
+egocentric = [
+    "macrodata-refiner[hf]",
+    "macrodata-refiner[video]",
+    "ego-vision[models]>=0.1.8",
+]
+text = [
+    "warcio",
+]
+hdf5 = [
+    "h5py",
+]
+zarr = [
+    "zarr>=2.18,<3",
+    "numcodecs<0.16",
+]
+mcap = [
+    "av",
+    "mcap",
+    "mcap-protobuf-support",
+    "mcap-ros2-support",
+    "pillow",
+]
+s3 = [
+    "s3fs",
+]
+tensorflow = [
+    "tensorflow",
+]
+tfds = [
+    "macrodata-refiner[tensorflow]",
+    "tensorflow-datasets",
+]
+testing = [
+    "macrodata-refiner[hdf5]",
+    "macrodata-refiner[hf]",
+    "macrodata-refiner[mcap]",
+    "macrodata-refiner[video]",
+    "macrodata-refiner[zarr]",
+    "macrodata-refiner[text]",
+    "macrodata-refiner[s3]",
+    "macrodata-refiner[tfds]",
+    "pytest>=8.0.0",
+    "pytest-cov>=5.0.0",
+]
+all = [
+    "macrodata-refiner[testing]",
+]
 
 [project.scripts]
 macrodata = "refiner.cli.main:main"
+mdr = "refiner.cli.main:main"
 
 [build-system]
 requires = ["setuptools>=77", "wheel"]
@@ -47,9 +102,8 @@ refiner = ["py.typed"]
 
 [dependency-groups]
 dev = [
+    "macrodata-refiner[all]",
     "pre-commit>=4.0.0",
-    "pytest>=8.0.0",
-    "pytest-cov>=5.0.0",
     "ruff>=0.14.10",
     "ty>=0.0.7",
 ]

macrodata_refiner-0.3.0/src/macrodata_refiner.egg-info/PKG-INFO

@@ -0,0 +1,196 @@
+Metadata-Version: 2.4
+Name: macrodata-refiner
+Version: 0.3.0
+Summary: Refiner by Macrodata Labs, a data processing framework for Machine Learning large scale datasets
+Author: Macrodata Labs
+License-Expression: Apache-2.0
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: cloudpickle==3.1.2
+Requires-Dist: fsspec[http]
+Requires-Dist: httpx
+Requires-Dist: loguru
+Requires-Dist: numpy
+Requires-Dist: orjson
+Requires-Dist: packaging
+Requires-Dist: pyarrow
+Requires-Dist: msgspec>=0.20.0
+Requires-Dist: pydantic>=2.0.0
+Provides-Extra: video
+Requires-Dist: av; extra == "video"
+Requires-Dist: pillow; extra == "video"
+Provides-Extra: hf
+Requires-Dist: datasets>=3.0.0; extra == "hf"
+Requires-Dist: huggingface-hub>=1.4.1; extra == "hf"
+Requires-Dist: hf>=1.7.1; extra == "hf"
+Provides-Extra: egocentric
+Requires-Dist: macrodata-refiner[hf]; extra == "egocentric"
+Requires-Dist: macrodata-refiner[video]; extra == "egocentric"
+Requires-Dist: ego-vision[models]>=0.1.8; extra == "egocentric"
+Provides-Extra: text
+Requires-Dist: warcio; extra == "text"
+Provides-Extra: hdf5
+Requires-Dist: h5py; extra == "hdf5"
+Provides-Extra: zarr
+Requires-Dist: zarr<3,>=2.18; extra == "zarr"
+Requires-Dist: numcodecs<0.16; extra == "zarr"
+Provides-Extra: mcap
+Requires-Dist: av; extra == "mcap"
+Requires-Dist: mcap; extra == "mcap"
+Requires-Dist: mcap-protobuf-support; extra == "mcap"
+Requires-Dist: mcap-ros2-support; extra == "mcap"
+Requires-Dist: pillow; extra == "mcap"
+Provides-Extra: s3
+Requires-Dist: s3fs; extra == "s3"
+Provides-Extra: tensorflow
+Requires-Dist: tensorflow; extra == "tensorflow"
+Provides-Extra: tfds
+Requires-Dist: macrodata-refiner[tensorflow]; extra == "tfds"
+Requires-Dist: tensorflow-datasets; extra == "tfds"
+Provides-Extra: testing
+Requires-Dist: macrodata-refiner[hdf5]; extra == "testing"
+Requires-Dist: macrodata-refiner[hf]; extra == "testing"
+Requires-Dist: macrodata-refiner[mcap]; extra == "testing"
+Requires-Dist: macrodata-refiner[video]; extra == "testing"
+Requires-Dist: macrodata-refiner[zarr]; extra == "testing"
+Requires-Dist: macrodata-refiner[text]; extra == "testing"
+Requires-Dist: macrodata-refiner[s3]; extra == "testing"
+Requires-Dist: macrodata-refiner[tfds]; extra == "testing"
+Requires-Dist: pytest>=8.0.0; extra == "testing"
+Requires-Dist: pytest-cov>=5.0.0; extra == "testing"
+Provides-Extra: all
+Requires-Dist: macrodata-refiner[testing]; extra == "all"
+Dynamic: license-file
+
+<p align="center">
+  <img src="https://macrodata.co/logo.svg" alt="Macrodata" width="180">
+</p>
+
+<h1 align="center">Macrodata Refiner</h1>
+
+Refiner is an open-source engine for turning raw robotics and multimodal data into **high-quality datasets** for model training.
+
+It gives training-data teams one pipeline model for multimodal data, robotics
+workflows, and model-based processing.
+
+It also plugs into the Macrodata platform, which gives you visibility into what is happening to your data while pipelines run: job and shard lifecycle, logs, metrics, manifests, and pipeline behavior. The same code can run locally for development and then scale out through Macrodata's elastic serverless cloud.
+
+## Quickstart
+
+Install:
+
+```bash
+pip install macrodata-refiner
+```
+
+Create a Macrodata API key:
+
+- https://macrodata.co/settings/api-keys
+
+Log in:
+
+```bash
+macrodata login
+```
+
+### Cloud example
+
+Launch a robotics pipeline on Macrodata Cloud.
+
+This requires a valid API key.
+
+```python
+import refiner as mdr
+
+(
+    mdr.read_lerobot("hf://datasets/macrodata/aloha_static_battery_ep005_009")
+    .map(
+        mdr.robotics.motion_trim(
+            threshold=0.001,
+            pad_frames=5,
+        )
+    )
+    .write_lerobot("hf://buckets/acme-robotics/aloha_motion")
+    .launch_cloud(
+        name="motion_trim",
+        num_workers=4,
+    )
+)
+```
+
+Need cloud GPUs? See [Resources, GPUs, and Services](docs/running-pipelines/resources-gpus-and-services.md).
+
+### Local example
+
+Launch a local pipeline:
+
+```python
+import refiner as mdr
+
+def add_preview(row):
+    return row.update(
+        preview=" ".join(row["text"].split()[:20]),
+    )
+
+(
+    mdr.read_jsonl("input/*.jsonl")
+    .filter(mdr.col("lang") == "en")
+    .with_columns(
+        text=mdr.col("text").str.strip(),
+        text_len=mdr.col("text").str.len(),
+    )
+    .map(add_preview)
+    .write_parquet("s3://my-bucket/english-cleanup/")
+    .launch_local(
+        name="english-cleanup",
+        num_workers=2,
+    )
+)
+```
+
+`pip install` gives you:
+
+- the Python package as `refiner`
+- the CLI as `macrodata`
+
+## Batteries included
+
+- training-data-first pipeline primitives instead of generic ETL abstractions
+- multimodal processing, with robotics support today
+- built-in readers, transforms, sinks, and runtime machinery for common dataset work
+- access to any storage backend supported by `fsspec` (S3, GCP, Hugging Face, etc.)
+- local execution for development and elastic cloud execution for large runs
+- built-in observability through the Macrodata platform for job state, logs, metrics, and manifests
+
+## Docs
+
+Start here:
+
+- [Docs index](docs/index.md)
+- [Quickstart](docs/quickstart.md)
+- [Running pipelines](docs/running-pipelines/index.md)
+
+Build a dataset:
+
+- [Reading data](docs/reading-data/index.md)
+- [Episode data](docs/episode-data/index.md)
+- [Transforms](docs/transforms/index.md)
+- [Episode operations](docs/episode-operations/index.md)
+- [Writing data](docs/writing-data/index.md)
+- [Examples](docs/examples/index.md)
+
+Operate jobs:
+
+- [Platform](docs/platform/index.md)
+- [CLI](docs/cli/index.md)
+- [Reference](docs/reference/index.md)
+
+## Community
+
+- join the Macrodata Discord: https://discord.gg/S8kZtmBR2x