openbot-data 0.0.1.post1__tar.gz → 0.0.2__tar.gz

{openbot_data-0.0.1.post1 → openbot_data-0.0.2}/.gitignore

@@ -11,3 +11,8 @@ venv/
 build/
 dist/
 *.egg-info/
+
+node_modules/
+.wrangler/
+coverage/
+.coverage

openbot_data-0.0.2/CHANGELOG.md

@@ -0,0 +1,38 @@
+# Changelog
+
+## Unreleased
+
+## 0.0.2
+
+### Added
+- Single-source package versioning with CI and release-tag validation.
+- Versioned, reproducible `openbot.dataset_manifest.v1` output with relative paths,
+  dataset fingerprints, and optional SHA-256 checksums.
+- Read-only local LeRobot v2.1/v3 discovery for episodes and video streams.
+- Deterministic dataset audit findings, duplicate detection, and `audit` CLI command.
+- Runnable local preflight demo, developer documentation, API/CLI reference, and
+  shipped LeRobot acceptance fixtures.
+- Acceptance scenarios for every documented finding code and a local Python
+  3.9–3.12 test-matrix command.
+
+### Changed
+- `inspect` now supports `--format auto|video|lerobot` and optional checksums.
+- Unreadable media now continues through the specific FPS, duration, dimensions,
+  and preview-decode audit rules instead of stopping at the generic finding.
+
+## 0.0.1.post2
+
+### Added
+- `export_catalog()` Python API and `openbot-data catalog` CLI command for JSON/CSV catalog export.
+- SEO-optimized README and `pyproject.toml` keywords (`robot data`, `ego data`, `egocentric data`, `embodied AI`, etc.).
+- GitHub Actions CI workflow for Python 3.9–3.12.
+- This changelog.
+
+## 0.0.1.post1
+
+### Added
+- Initial preview release.
+- Video directory scanning and metadata extraction.
+- Preview frame extraction.
+- Manifest and report generation.
+- `openbot-data` CLI with `scan`, `inspect`, and `version` commands.

openbot_data-0.0.2/PKG-INFO

@@ -0,0 +1,245 @@
+Metadata-Version: 2.4
+Name: openbot-data
+Version: 0.0.2
+Summary: Inspect robot video datasets and generate manifests, previews, and catalogs.
+Project-URL: Homepage, https://openbot.ai/data
+Project-URL: Documentation, https://openbot.ai/api/docs#data-recipe
+Project-URL: Repository, https://github.com/openbotai/openbot-data
+Project-URL: Issues, https://github.com/openbotai/openbot-data/issues
+Author-email: OpenBot <hello@openbot.ai>
+License: MIT
+License-File: LICENSE
+Keywords: data-cleaning,dataset,dataset-catalog,ego data,egocentric data,embodied AI,embodied-ai,failure-analysis,robot data,robot dataset,robot learning,robot-video,robotics,teleoperation data,wrist-camera
+Classifier: Development Status :: 2 - Pre-Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Scientific/Engineering :: Image Processing
+Requires-Python: >=3.9
+Requires-Dist: numpy>=1.24.0
+Requires-Dist: opencv-python>=4.8.0
+Requires-Dist: tqdm>=4.65.0
+Requires-Dist: typer>=0.9.0
+Provides-Extra: dev
+Requires-Dist: build>=1.0.0; extra == 'dev'
+Requires-Dist: jsonschema>=4.18.0; extra == 'dev'
+Requires-Dist: mypy<1.19,>=1.11.0; extra == 'dev'
+Requires-Dist: pytest>=7.0.0; extra == 'dev'
+Requires-Dist: ruff>=0.6.0; extra == 'dev'
+Requires-Dist: twine>=6.2.0; extra == 'dev'
+Provides-Extra: lerobot
+Requires-Dist: pyarrow>=17.0.0; extra == 'lerobot'
+Description-Content-Type: text/markdown
+
+# OpenBot Data
+
+> Inspect, catalog, and prepare robot video data for embodied AI and robot learning.
+
+[Library documentation](docs/README.md) · [API reference](docs/api-reference.md) ·
+[Data product](https://openbot.ai/data) · [Source repository](https://github.com/openbotai/openbot-data)
+
+OpenBot Data is an early Python toolkit for working with **robot data** — especially
+**egocentric video** and **teleoperation data** collected from wrist cameras,
+head-mounted cameras, and robot demonstrators.
+
+Use it to scan video directories, extract preview frames, build dataset manifests,
+and export a searchable **robot dataset catalog** (JSON/CSV) before training or
+evaluation.
+
+## Install
+
+```bash
+pip install openbot-data
+```
+
+Requires Python 3.9+.
+
+## Runnable demo
+
+Clone the repository, install the package, and run the complete local preflight
+against a video directory or local LeRobot repository:
+
+```bash
+pip install -e .
+python examples/local_preflight.py ./robot_videos --out ./openbot-preflight --integrity sample
+```
+
+The demo uses the public Python API and writes
+`inspection/metadata/manifest.json`, `inspection/metadata/report.json`, and
+`audit.json`. Add `--format lerobot` for a local LeRobot v2.1/v3 repository,
+`--integrity full` to decode every frame, or `--no-checksum` to skip SHA-256
+duplicate detection. The demo discovers and hashes the dataset once, then shares
+one immutable `DatasetSnapshot` across both renderers.
+
+## What it does
+
+- **Scan robot video directories** — recursively find `.mp4`, `.mov`, `.avi`, `.mkv`, `.webm` files.
+- **Read video metadata** — duration, fps, resolution, frame count, file size, validity.
+- **Extract preview frames** — uniformly sample frames for quick human inspection.
+- **Generate manifests** — `manifest.json` with per-video metadata and preview paths.
+- **Generate quality reports** — `report.json` with aggregate duration, size, and resolutions.
+- **Export dataset catalogs** — JSON or CSV for dataset registries, documentation, and SEO-friendly catalog pages.
+- **Discover local LeRobot data** — read-only episode and video-stream discovery for v2.1/v3 layouts.
+- **Audit before training or upload** — stable error/warning codes without an uncalibrated quality score.
+
+## CLI
+
+### Scan videos
+
+```bash
+openbot-data scan ./robot_videos
+openbot-data scan ./robot_videos --output scan.json
+```
+
+### Inspect a dataset
+
+```bash
+openbot-data inspect ./robot_videos --out ./openbot_dataset
+openbot-data inspect ./lerobot_dataset --format lerobot --out ./openbot_dataset
+openbot-data audit ./robot_videos --out ./audit.json --fail-on error
+```
+
+Output structure:
+
+```text
+openbot_dataset/
+  previews/
+  metadata/
+    manifest.json
+    report.json
+```
+
+### Export a robot dataset catalog
+
+```bash
+# JSON catalog for dataset registries and documentation
+openbot-data catalog ./robot_videos --out ./catalog.json --format json
+
+# CSV catalog for spreadsheets or Hugging Face dataset cards
+openbot-data catalog ./robot_videos --out ./catalog.csv --format csv
+```
+
+The catalog is designed to be checked into GitHub or linked from a dataset registry
+page such as [OpenBot.ai Datasets](https://openbot.ai/datasets).
+
+## Python API
+
+```python
+from openbot_data import (
+    audit_dataset,
+    export_catalog,
+    inspect_dataset,
+    prepare_dataset,
+    read_lerobot,
+    scan_directory,
+    schema_path,
+)
+
+# Scan a directory
+scan = scan_directory("./robot_videos")
+print(scan["valid_videos"])
+
+# Inspect and extract previews
+result = inspect_dataset(
+    video_dir="./robot_videos",
+    output_dir="./openbot_dataset",
+)
+print(result["manifest_path"])
+print(result["report_path"])
+
+# Export a catalog for documentation / SEO
+export_catalog(
+    video_dir="./robot_videos",
+    output_path="./catalog.json",
+    fmt="json",
+)
+
+# Discover/hash once and reuse the immutable snapshot across renderers
+snapshot = prepare_dataset("./robot_videos", checksum="sha256", integrity="sample")
+inspection = inspect_dataset("./robot_videos", "./inspection", snapshot=snapshot)
+audit = audit_dataset("./robot_videos", snapshot=snapshot)
+
+# Discover episodes and camera streams in a local LeRobot repository
+lerobot = read_lerobot("./lerobot_dataset")
+
+# Packaged JSON Schemas are stable independently of the package version
+with schema_path("manifest") as manifest_schema:
+    print(manifest_schema)
+```
+
+## Use cases
+
+- Prepare **teleoperation data** collected from ALOHA, Mobile ALOHA, VR, or SpaceMouse setups.
+- Inspect **egocentric video datasets** before converting to LeRobot, RLDS, or HDF5.
+- Build a **robot dataset catalog** that links back to your project website or Hugging Face collection.
+- Validate video integrity before running robot policy training or VLA pre-training.
+
+## Development
+
+```bash
+pip install -e ".[dev]"
+python scripts/check_version.py
+pytest
+scripts/test_matrix.sh
+python -m build
+python -m twine check dist/*
+```
+
+The local matrix script requires [`uv`](https://docs.astral.sh/uv/) and tests
+Python 3.9–3.12 without requiring a repository CI workflow.
+
+`VERSION` is the package version source of truth. To release, update `VERSION`
+and `CHANGELOG.md`, verify locally, then publish a GitHub Release whose tag is
+`v<version>`. The release workflow validates the tag before publishing to PyPI.
+
+## Status
+
+OpenBot Data is an early local Python toolkit. Hosted upload, annotation, review,
+export, billing, Workers, and Container infrastructure belong to the main
+[OpenBot](https://github.com/openbotai/OpenBot) product repository. Use the
+separate [`openbot-sdk`](https://github.com/openbotai/openbot-sdk) package to call
+the hosted API.
+
+## Roadmap
+
+- [x] Video scanning and metadata extraction
+- [x] Preview frame extraction
+- [x] Manifest and report generation
+- [x] JSON/CSV catalog export
+- [x] [`0.0.2`: local dataset preflight](docs/version-0.0.2.md), including a
+      versioned manifest, deterministic audit findings, and local LeRobot discovery
+- [ ] Later: RLDS / HDF5 ingestion helpers
+- [ ] Later: calibrated quality evaluation after a labeled validation set exists
+
+Audit codes are documented in [`docs/audit-findings.md`](docs/audit-findings.md).
+Canonical manifests use explicit `path_base`/`path_bases` fields; they never
+serialize private source paths. Symlinks are skipped by default, and even when
+`follow_symlinks=True` a target outside the dataset root is rejected.
+
+## License
+
+MIT
+
+## Citation
+
+If you use OpenBot Data in research or production, cite the project repository:
+
+```bibtex
+@software{openbot_data,
+  title = {OpenBot Data},
+  author = {OpenBot},
+  year = {2026},
+  url = {https://github.com/openbotai/openbot-data}
+}
+```
+
+## Related
+
+- [OpenBot.ai](https://openbot.ai) — Robot dataset catalog and policy evaluation platform.
+- [OpenBot.ai Datasets](https://openbot.ai/datasets) — Searchable index of egocentric and robot datasets.

openbot_data-0.0.2/README.md

@@ -0,0 +1,206 @@
+# OpenBot Data
+
+> Inspect, catalog, and prepare robot video data for embodied AI and robot learning.
+
+[Library documentation](docs/README.md) · [API reference](docs/api-reference.md) ·
+[Data product](https://openbot.ai/data) · [Source repository](https://github.com/openbotai/openbot-data)
+
+OpenBot Data is an early Python toolkit for working with **robot data** — especially
+**egocentric video** and **teleoperation data** collected from wrist cameras,
+head-mounted cameras, and robot demonstrators.
+
+Use it to scan video directories, extract preview frames, build dataset manifests,
+and export a searchable **robot dataset catalog** (JSON/CSV) before training or
+evaluation.
+
+## Install
+
+```bash
+pip install openbot-data
+```
+
+Requires Python 3.9+.
+
+## Runnable demo
+
+Clone the repository, install the package, and run the complete local preflight
+against a video directory or local LeRobot repository:
+
+```bash
+pip install -e .
+python examples/local_preflight.py ./robot_videos --out ./openbot-preflight --integrity sample
+```
+
+The demo uses the public Python API and writes
+`inspection/metadata/manifest.json`, `inspection/metadata/report.json`, and
+`audit.json`. Add `--format lerobot` for a local LeRobot v2.1/v3 repository,
+`--integrity full` to decode every frame, or `--no-checksum` to skip SHA-256
+duplicate detection. The demo discovers and hashes the dataset once, then shares
+one immutable `DatasetSnapshot` across both renderers.
+
+## What it does
+
+- **Scan robot video directories** — recursively find `.mp4`, `.mov`, `.avi`, `.mkv`, `.webm` files.
+- **Read video metadata** — duration, fps, resolution, frame count, file size, validity.
+- **Extract preview frames** — uniformly sample frames for quick human inspection.
+- **Generate manifests** — `manifest.json` with per-video metadata and preview paths.
+- **Generate quality reports** — `report.json` with aggregate duration, size, and resolutions.
+- **Export dataset catalogs** — JSON or CSV for dataset registries, documentation, and SEO-friendly catalog pages.
+- **Discover local LeRobot data** — read-only episode and video-stream discovery for v2.1/v3 layouts.
+- **Audit before training or upload** — stable error/warning codes without an uncalibrated quality score.
+
+## CLI
+
+### Scan videos
+
+```bash
+openbot-data scan ./robot_videos
+openbot-data scan ./robot_videos --output scan.json
+```
+
+### Inspect a dataset
+
+```bash
+openbot-data inspect ./robot_videos --out ./openbot_dataset
+openbot-data inspect ./lerobot_dataset --format lerobot --out ./openbot_dataset
+openbot-data audit ./robot_videos --out ./audit.json --fail-on error
+```
+
+Output structure:
+
+```text
+openbot_dataset/
+  previews/
+  metadata/
+    manifest.json
+    report.json
+```
+
+### Export a robot dataset catalog
+
+```bash
+# JSON catalog for dataset registries and documentation
+openbot-data catalog ./robot_videos --out ./catalog.json --format json
+
+# CSV catalog for spreadsheets or Hugging Face dataset cards
+openbot-data catalog ./robot_videos --out ./catalog.csv --format csv
+```
+
+The catalog is designed to be checked into GitHub or linked from a dataset registry
+page such as [OpenBot.ai Datasets](https://openbot.ai/datasets).
+
+## Python API
+
+```python
+from openbot_data import (
+    audit_dataset,
+    export_catalog,
+    inspect_dataset,
+    prepare_dataset,
+    read_lerobot,
+    scan_directory,
+    schema_path,
+)
+
+# Scan a directory
+scan = scan_directory("./robot_videos")
+print(scan["valid_videos"])
+
+# Inspect and extract previews
+result = inspect_dataset(
+    video_dir="./robot_videos",
+    output_dir="./openbot_dataset",
+)
+print(result["manifest_path"])
+print(result["report_path"])
+
+# Export a catalog for documentation / SEO
+export_catalog(
+    video_dir="./robot_videos",
+    output_path="./catalog.json",
+    fmt="json",
+)
+
+# Discover/hash once and reuse the immutable snapshot across renderers
+snapshot = prepare_dataset("./robot_videos", checksum="sha256", integrity="sample")
+inspection = inspect_dataset("./robot_videos", "./inspection", snapshot=snapshot)
+audit = audit_dataset("./robot_videos", snapshot=snapshot)
+
+# Discover episodes and camera streams in a local LeRobot repository
+lerobot = read_lerobot("./lerobot_dataset")
+
+# Packaged JSON Schemas are stable independently of the package version
+with schema_path("manifest") as manifest_schema:
+    print(manifest_schema)
+```
+
+## Use cases
+
+- Prepare **teleoperation data** collected from ALOHA, Mobile ALOHA, VR, or SpaceMouse setups.
+- Inspect **egocentric video datasets** before converting to LeRobot, RLDS, or HDF5.
+- Build a **robot dataset catalog** that links back to your project website or Hugging Face collection.
+- Validate video integrity before running robot policy training or VLA pre-training.
+
+## Development
+
+```bash
+pip install -e ".[dev]"
+python scripts/check_version.py
+pytest
+scripts/test_matrix.sh
+python -m build
+python -m twine check dist/*
+```
+
+The local matrix script requires [`uv`](https://docs.astral.sh/uv/) and tests
+Python 3.9–3.12 without requiring a repository CI workflow.
+
+`VERSION` is the package version source of truth. To release, update `VERSION`
+and `CHANGELOG.md`, verify locally, then publish a GitHub Release whose tag is
+`v<version>`. The release workflow validates the tag before publishing to PyPI.
+
+## Status
+
+OpenBot Data is an early local Python toolkit. Hosted upload, annotation, review,
+export, billing, Workers, and Container infrastructure belong to the main
+[OpenBot](https://github.com/openbotai/OpenBot) product repository. Use the
+separate [`openbot-sdk`](https://github.com/openbotai/openbot-sdk) package to call
+the hosted API.
+
+## Roadmap
+
+- [x] Video scanning and metadata extraction
+- [x] Preview frame extraction
+- [x] Manifest and report generation
+- [x] JSON/CSV catalog export
+- [x] [`0.0.2`: local dataset preflight](docs/version-0.0.2.md), including a
+      versioned manifest, deterministic audit findings, and local LeRobot discovery
+- [ ] Later: RLDS / HDF5 ingestion helpers
+- [ ] Later: calibrated quality evaluation after a labeled validation set exists
+
+Audit codes are documented in [`docs/audit-findings.md`](docs/audit-findings.md).
+Canonical manifests use explicit `path_base`/`path_bases` fields; they never
+serialize private source paths. Symlinks are skipped by default, and even when
+`follow_symlinks=True` a target outside the dataset root is rejected.
+
+## License
+
+MIT
+
+## Citation
+
+If you use OpenBot Data in research or production, cite the project repository:
+
+```bibtex
+@software{openbot_data,
+  title = {OpenBot Data},
+  author = {OpenBot},
+  year = {2026},
+  url = {https://github.com/openbotai/openbot-data}
+}
+```
+
+## Related
+
+- [OpenBot.ai](https://openbot.ai) — Robot dataset catalog and policy evaluation platform.
+- [OpenBot.ai Datasets](https://openbot.ai/datasets) — Searchable index of egocentric and robot datasets.

openbot_data-0.0.2/VERSION

@@ -0,0 +1 @@
+0.0.2

openbot_data-0.0.2/docs/README.md

@@ -0,0 +1,25 @@
+# OpenBot Data documentation
+
+OpenBot Data is a local, open-source Python library. It inspects robot-video
+directories and local LeRobot v2.1/v3 repositories before training or upload. It
+does not call the Hosted OpenBot API.
+
+## Start here
+
+- [Getting started](getting-started.md): installation, CLI, Python workflow, and
+  the runnable demo.
+- [API reference](api-reference.md): public Python functions, CLI commands,
+  parameters, return values, and failure behavior.
+- [Audit finding codes](audit-findings.md): stable machine-readable validation
+  codes and severities.
+- [0.0.2 release contract](version-0.0.2.md): scope and acceptance criteria.
+- [Canonical manifest](examples/manifest.json) and
+  [canonical audit](examples/audit.json): complete JSON examples without local
+  machine paths.
+
+## Capability boundary
+
+This repository owns local discovery, media inspection, preview generation,
+deterministic manifests, audits, and catalog export. Uploads, API keys,
+asynchronous processing, review, hosted artifacts, retention, and billing belong
+to the main OpenBot service and are called through `openbot-sdk`.