openbot-data 0.0.2__tar.gz → 0.0.3__tar.gz

openbot_data-0.0.3/CHANGELOG.md

@@ -0,0 +1,103 @@
+# Changelog
+
+## 0.0.3 - 2026-07-28
+
+### Added
+- Explicit immutable LeRobot v2.1 and v3.0 adapters, including read-only v2.1
+  migration guidance and direct stable-contract validation without importing
+  the full LeRobot package.
+- Layered collect-all metadata, schema, data, media, alignment, statistics, and
+  provenance validation with declared capability coverage.
+- Portable `openbot.dataset_snapshot.v1` identities and strict semantic
+  `openbot.dataset_diff.v1` classification.
+- Revision-pinned Hugging Face resolution with hard byte/shard/episode/media
+  budgets and explicit metadata/sample/full coverage.
+- Built-in LeRobot core, common training, Hub publication, ACT, and SmolVLA
+  readiness profiles plus strict policy-config overrides and deterministic
+  JSON/Markdown gates.
+- Actionable-location triage and raw advisory evidence for flatline actions,
+  zero-variance state, duplicate episodes, and camera/task imbalance, including
+  non-executing synchronized idle-trim plans and no aggregate dataset score.
+- Deterministic repair plans, copy-on-write allowlisted metadata repair, and
+  verified receipts with stale-source and partial-destination protection.
+- Four-way merge compatibility plans and post-operation verification for the
+  pinned official `lerobot-edit-dataset` merge.
+- Optional pinned official LeRobot loader smoke tests in repair and merge CLI
+  verification.
+- Pinned `lerobot-doctor==0.2.0` and `robovet==0.2.2` outcome-comparison
+  evidence for the critical stale-frame-counter workflow.
+- Python 3.9–3.12 CI and local matrix coverage for the Parquet and Hub extras,
+  preventing optional-feature tests from being silently skipped.
+- A separate Python 3.12 conformance job that creates and finalizes a real
+  shared-shard video dataset with `lerobot[dataset]==0.6.0`, then proves full
+  audit, ACT/SmolVLA readiness, quantile/unknown-field compatibility, and
+  official first/middle/last loader access.
+- `build_catalog_evidence()` and the `catalog-evidence` CLI command for a
+  score-free, versioned handoff from one prepared local audit to OpenBot
+  Catalog.
+- The packaged `catalog-evidence-v1` JSON Schema, deterministic cross-language
+  typed-tree fingerprint, portable source identity, coverage, evidence maturity,
+  facts, findings, and unresolved-check reporting.
+- A versioned `openbot.dataset_catalog.v1` JSON catalog schema and shared
+  immutable-snapshot rendering across inspect, audit, and catalog output.
+- Structured findings for malformed episode lengths, duplicate indexes, invalid
+  video relations and paths, segment bounds/overlap/duration, and explicit
+  symlink skip/broken states.
+
+### Changed
+- V3 data-shard episode ranges are validated as dataset-global offsets, matching
+  the official multi-shard LeRobot 0.6.0 writer and merge tool.
+- Canonical video features are no longer incorrectly required as data-Parquet
+  columns; LeRobot v3 stores them in `videos/**`.
+- Snapshot Schema accepts the official writer's `names: null` standard-column
+  contract while retaining strict component and top-level fingerprints.
+- `sample` integrity now probes deterministic start, middle, and end positions
+  while preserving the existing manifest v1 decoded-count projection and
+  fingerprint behavior.
+- LeRobot v2.1 and v3 discovery now checks every declared camera for every
+  episode and validates shared-video segment bounds with a one-frame duration
+  tolerance.
+- Caller-supplied snapshots are validated against the requested root, resolved
+  input format, checksum coverage, integrity coverage, and symlink policy before
+  any renderer writes output.
+
+### Documentation
+- Defined the complete `0.0.3` feature and release contract, including the
+  canonical P0 feature map, public artifacts and commands, implementation
+  checkpoint, integrity/coverage semantics, uniform exit classes, scope
+  boundaries, implementation order, and release acceptance criteria.
+
+## 0.0.2
+
+### Added
+- Single-source package versioning with CI and release-tag validation.
+- Versioned, reproducible `openbot.dataset_manifest.v1` output with relative paths,
+  dataset fingerprints, and optional SHA-256 checksums.
+- Read-only local LeRobot v2.1/v3 discovery for episodes and video streams.
+- Deterministic dataset audit findings, duplicate detection, and `audit` CLI command.
+- Runnable local preflight demo, developer documentation, API/CLI reference, and
+  shipped LeRobot acceptance fixtures.
+- Acceptance scenarios for every documented finding code and a local Python
+  3.9–3.12 test-matrix command.
+
+### Changed
+- `inspect` now supports `--format auto|video|lerobot` and optional checksums.
+- Unreadable media now continues through the specific FPS, duration, dimensions,
+  and preview-decode audit rules instead of stopping at the generic finding.
+
+## 0.0.1.post2
+
+### Added
+- `export_catalog()` Python API and `openbot-data catalog` CLI command for JSON/CSV catalog export.
+- SEO-optimized README and `pyproject.toml` keywords (`robot data`, `ego data`, `egocentric data`, `embodied AI`, etc.).
+- GitHub Actions CI workflow for Python 3.9–3.12.
+- This changelog.
+
+## 0.0.1.post1
+
+### Added
+- Initial preview release.
+- Video directory scanning and metadata extraction.
+- Preview frame extraction.
+- Manifest and report generation.
+- `openbot-data` CLI with `scan`, `inspect`, and `version` commands.

{openbot_data-0.0.2 → openbot_data-0.0.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: openbot-data
-Version: 0.0.2
+Version: 0.0.3
 Summary: Inspect robot video datasets and generate manifests, previews, and catalogs.
 Project-URL: Homepage, https://openbot.ai/data
 Project-URL: Documentation, https://openbot.ai/api/docs#data-recipe
@@ -22,17 +22,19 @@ Classifier: Programming Language :: Python :: 3.12
 Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
 Classifier: Topic :: Scientific/Engineering :: Image Processing
 Requires-Python: >=3.9
+Requires-Dist: jsonschema>=4.18.0
 Requires-Dist: numpy>=1.24.0
 Requires-Dist: opencv-python>=4.8.0
 Requires-Dist: tqdm>=4.65.0
 Requires-Dist: typer>=0.9.0
 Provides-Extra: dev
 Requires-Dist: build>=1.0.0; extra == 'dev'
-Requires-Dist: jsonschema>=4.18.0; extra == 'dev'
 Requires-Dist: mypy<1.19,>=1.11.0; extra == 'dev'
-Requires-Dist: pytest>=7.0.0; extra == 'dev'
+Requires-Dist: pytest>=8.2.0; extra == 'dev'
 Requires-Dist: ruff>=0.6.0; extra == 'dev'
 Requires-Dist: twine>=6.2.0; extra == 'dev'
+Provides-Extra: hub
+Requires-Dist: huggingface-hub>=0.27.0; extra == 'hub'
 Provides-Extra: lerobot
 Requires-Dist: pyarrow>=17.0.0; extra == 'lerobot'
 Description-Content-Type: text/markdown
@@ -58,7 +60,9 @@ evaluation.
 pip install openbot-data
 ```
 
-Requires Python 3.9+.
+PyPI currently installs the released `0.0.2` package. The source tree is the
+`0.0.3` release candidate: its P0 interfaces and local release gates pass, but it
+has not been published. Requires Python 3.9+.
 
 ## Runnable demo
 
@@ -87,6 +91,13 @@ one immutable `DatasetSnapshot` across both renderers.
 - **Export dataset catalogs** — JSON or CSV for dataset registries, documentation, and SEO-friendly catalog pages.
 - **Discover local LeRobot data** — read-only episode and video-stream discovery for v2.1/v3 layouts.
 - **Audit before training or upload** — stable error/warning codes without an uncalibrated quality score.
+- **Snapshot and diff datasets** — portable component fingerprints and semantic change classes.
+- **Gate ACT and SmolVLA readiness** — distinguish `READY`, `BLOCKED`, and incomplete `PARTIAL` evidence.
+- **Audit pinned Hub revisions** — resolve branches/tags to immutable commits under hard download budgets.
+- **Repair conservatively** — plan and apply only unambiguous derived metadata changes to a new copy.
+- **Verify official merges** — check compatibility before merge, then reconcile lineage, loader, audit, and diff evidence.
+- **Hand off score-free Catalog evidence** — emit versioned facts, coverage,
+  findings, and unresolved checks for server-side evaluation.
 
 ## CLI
 
@@ -128,11 +139,59 @@ openbot-data catalog ./robot_videos --out ./catalog.csv --format csv
 The catalog is designed to be checked into GitHub or linked from a dataset registry
 page such as [OpenBot.ai Datasets](https://openbot.ai/datasets).
 
+### Build Catalog evidence from a completed local audit
+
+```bash
+openbot-data catalog-evidence ./robot_videos \
+  --dataset-id org/dataset \
+  --checked-at 2026-07-28T12:00:00Z \
+  --out ./catalog-evidence.json
+```
+
+The artifact contains no Overall score or normalized dimension score. OpenBot
+Catalog remains responsible for evaluation, review, and publication.
+
+### Snapshot, diff, and readiness
+
+```bash
+openbot-data snapshot ./lerobot_dataset \
+  --format lerobot --integrity full --out ./dataset.snapshot.json
+
+openbot-data diff ./baseline.snapshot.json ./dataset.snapshot.json \
+  --out ./dataset.diff.json --fail-on breaking
+
+openbot-data readiness ./lerobot_dataset \
+  --profile lerobot-smolvla --integrity full \
+  --out ./readiness.json --markdown ./readiness.md
+```
+
+### Repair and merge verification
+
+```bash
+openbot-data repair plan ./dataset --integrity full --out ./repair.plan.json
+openbot-data repair apply ./dataset \
+  --plan ./repair.plan.json --output ./dataset.repaired
+
+openbot-data merge-check ./dataset-a ./dataset-b --out ./merge.plan.json
+openbot-data verify-merge ./merged \
+  --input ./dataset-a.snapshot.json \
+  --input ./dataset-b.snapshot.json \
+  --operation-record ./official-operation.json \
+  --out ./merge.receipt.json
+```
+
+Physical merge stays with `lerobot-edit-dataset` from
+`lerobot[dataset]==0.6.0`; OpenBot only plans and verifies it.
+
 ## Python API
 
 ```python
 from openbot_data import (
     audit_dataset,
+    build_catalog_evidence,
+    build_dataset_snapshot,
+    diff_dataset_snapshots,
+    evaluate_dataset_readiness,
     export_catalog,
     inspect_dataset,
     prepare_dataset,
@@ -168,6 +227,27 @@ audit = audit_dataset("./robot_videos", snapshot=snapshot)
 # Discover episodes and camera streams in a local LeRobot repository
 lerobot = read_lerobot("./lerobot_dataset")
 
+# Produce score-free evidence for server-side Catalog evaluation
+evidence = build_catalog_evidence(
+    "./robot_videos",
+    dataset_id="org/dataset",
+    checked_at="2026-07-28T12:00:00Z",
+    output_path="./catalog-evidence.json",
+)
+
+portable = build_dataset_snapshot(
+    "./lerobot_dataset",
+    input_format="lerobot",
+    integrity="full",
+)
+readiness = evaluate_dataset_readiness(
+    "./lerobot_dataset",
+    profile="lerobot-act",
+    integrity="full",
+    dataset_snapshot=portable,
+)
+change = diff_dataset_snapshots(portable, portable)
+
 # Packaged JSON Schemas are stable independently of the package version
 with schema_path("manifest") as manifest_schema:
     print(manifest_schema)
@@ -200,11 +280,14 @@ and `CHANGELOG.md`, verify locally, then publish a GitHub Release whose tag is
 
 ## Status
 
-OpenBot Data is an early local Python toolkit. Hosted upload, annotation, review,
-export, billing, Workers, and Container infrastructure belong to the main
-[OpenBot](https://github.com/openbotai/OpenBot) product repository. Use the
-separate [`openbot-sdk`](https://github.com/openbotai/openbot-sdk) package to call
-the hosted API.
+The released PyPI package is `0.0.2`; the current source version is `0.0.3` and
+its local release gate passes. No tag, GitHub Release, or PyPI publication has
+been performed. OpenBot Data remains an early local Python toolkit. Hosted upload,
+annotation, review, export, billing, Workers, and Container infrastructure
+belong to the main [OpenBot](https://github.com/openbotai/OpenBot) product
+repository. Use the separate
+[`openbot-sdk`](https://github.com/openbotai/openbot-sdk) package to call the
+hosted API.
 
 ## Roadmap
 
@@ -214,10 +297,20 @@ the hosted API.
 - [x] JSON/CSV catalog export
 - [x] [`0.0.2`: local dataset preflight](docs/version-0.0.2.md), including a
       versioned manifest, deterministic audit findings, and local LeRobot discovery
-- [ ] Later: RLDS / HDF5 ingestion helpers
-- [ ] Later: calibrated quality evaluation after a labeled validation set exists
+- [x] [`0.0.3`: LeRobot compatibility and dataset change control](docs/version-0.0.3.md),
+      with correctness closure, explicit v2.1/v3 adapters, layered validation,
+      portable snapshots, semantic diff, revision-pinned Hub audit, readiness
+      profiles, evidence triage, conservative repair, and merge verification.
+      P0 interfaces, the package version, clean install, Python matrix,
+      artifacts, official conformance, and local release checks pass. Public
+      release remains a separate explicit action.
+- [ ] Later: read-only robomimic/HDF5 and RLDS/Open X adapters
+- [ ] Later: advanced episode-quality signals and any aggregate score only after
+      labeled downstream validation exists
 
 Audit codes are documented in [`docs/audit-findings.md`](docs/audit-findings.md).
+The research and differentiation decisions behind `0.0.3` are documented in
+[`docs/reference-libraries.md`](docs/reference-libraries.md).
 Canonical manifests use explicit `path_base`/`path_bases` fields; they never
 serialize private source paths. Symlinks are skipped by default, and even when
 `follow_symlinks=True` a target outside the dataset root is rejected.

{openbot_data-0.0.2 → openbot_data-0.0.3}/README.md

@@ -19,7 +19,9 @@ evaluation.
 pip install openbot-data
 ```
 
-Requires Python 3.9+.
+PyPI currently installs the released `0.0.2` package. The source tree is the
+`0.0.3` release candidate: its P0 interfaces and local release gates pass, but it
+has not been published. Requires Python 3.9+.
 
 ## Runnable demo
 
@@ -48,6 +50,13 @@ one immutable `DatasetSnapshot` across both renderers.
 - **Export dataset catalogs** — JSON or CSV for dataset registries, documentation, and SEO-friendly catalog pages.
 - **Discover local LeRobot data** — read-only episode and video-stream discovery for v2.1/v3 layouts.
 - **Audit before training or upload** — stable error/warning codes without an uncalibrated quality score.
+- **Snapshot and diff datasets** — portable component fingerprints and semantic change classes.
+- **Gate ACT and SmolVLA readiness** — distinguish `READY`, `BLOCKED`, and incomplete `PARTIAL` evidence.
+- **Audit pinned Hub revisions** — resolve branches/tags to immutable commits under hard download budgets.
+- **Repair conservatively** — plan and apply only unambiguous derived metadata changes to a new copy.
+- **Verify official merges** — check compatibility before merge, then reconcile lineage, loader, audit, and diff evidence.
+- **Hand off score-free Catalog evidence** — emit versioned facts, coverage,
+  findings, and unresolved checks for server-side evaluation.
 
 ## CLI
 
@@ -89,11 +98,59 @@ openbot-data catalog ./robot_videos --out ./catalog.csv --format csv
 The catalog is designed to be checked into GitHub or linked from a dataset registry
 page such as [OpenBot.ai Datasets](https://openbot.ai/datasets).
 
+### Build Catalog evidence from a completed local audit
+
+```bash
+openbot-data catalog-evidence ./robot_videos \
+  --dataset-id org/dataset \
+  --checked-at 2026-07-28T12:00:00Z \
+  --out ./catalog-evidence.json
+```
+
+The artifact contains no Overall score or normalized dimension score. OpenBot
+Catalog remains responsible for evaluation, review, and publication.
+
+### Snapshot, diff, and readiness
+
+```bash
+openbot-data snapshot ./lerobot_dataset \
+  --format lerobot --integrity full --out ./dataset.snapshot.json
+
+openbot-data diff ./baseline.snapshot.json ./dataset.snapshot.json \
+  --out ./dataset.diff.json --fail-on breaking
+
+openbot-data readiness ./lerobot_dataset \
+  --profile lerobot-smolvla --integrity full \
+  --out ./readiness.json --markdown ./readiness.md
+```
+
+### Repair and merge verification
+
+```bash
+openbot-data repair plan ./dataset --integrity full --out ./repair.plan.json
+openbot-data repair apply ./dataset \
+  --plan ./repair.plan.json --output ./dataset.repaired
+
+openbot-data merge-check ./dataset-a ./dataset-b --out ./merge.plan.json
+openbot-data verify-merge ./merged \
+  --input ./dataset-a.snapshot.json \
+  --input ./dataset-b.snapshot.json \
+  --operation-record ./official-operation.json \
+  --out ./merge.receipt.json
+```
+
+Physical merge stays with `lerobot-edit-dataset` from
+`lerobot[dataset]==0.6.0`; OpenBot only plans and verifies it.
+
 ## Python API
 
 ```python
 from openbot_data import (
     audit_dataset,
+    build_catalog_evidence,
+    build_dataset_snapshot,
+    diff_dataset_snapshots,
+    evaluate_dataset_readiness,
     export_catalog,
     inspect_dataset,
     prepare_dataset,
@@ -129,6 +186,27 @@ audit = audit_dataset("./robot_videos", snapshot=snapshot)
 # Discover episodes and camera streams in a local LeRobot repository
 lerobot = read_lerobot("./lerobot_dataset")
 
+# Produce score-free evidence for server-side Catalog evaluation
+evidence = build_catalog_evidence(
+    "./robot_videos",
+    dataset_id="org/dataset",
+    checked_at="2026-07-28T12:00:00Z",
+    output_path="./catalog-evidence.json",
+)
+
+portable = build_dataset_snapshot(
+    "./lerobot_dataset",
+    input_format="lerobot",
+    integrity="full",
+)
+readiness = evaluate_dataset_readiness(
+    "./lerobot_dataset",
+    profile="lerobot-act",
+    integrity="full",
+    dataset_snapshot=portable,
+)
+change = diff_dataset_snapshots(portable, portable)
+
 # Packaged JSON Schemas are stable independently of the package version
 with schema_path("manifest") as manifest_schema:
     print(manifest_schema)
@@ -161,11 +239,14 @@ and `CHANGELOG.md`, verify locally, then publish a GitHub Release whose tag is
 
 ## Status
 
-OpenBot Data is an early local Python toolkit. Hosted upload, annotation, review,
-export, billing, Workers, and Container infrastructure belong to the main
-[OpenBot](https://github.com/openbotai/OpenBot) product repository. Use the
-separate [`openbot-sdk`](https://github.com/openbotai/openbot-sdk) package to call
-the hosted API.
+The released PyPI package is `0.0.2`; the current source version is `0.0.3` and
+its local release gate passes. No tag, GitHub Release, or PyPI publication has
+been performed. OpenBot Data remains an early local Python toolkit. Hosted upload,
+annotation, review, export, billing, Workers, and Container infrastructure
+belong to the main [OpenBot](https://github.com/openbotai/OpenBot) product
+repository. Use the separate
+[`openbot-sdk`](https://github.com/openbotai/openbot-sdk) package to call the
+hosted API.
 
 ## Roadmap
 
@@ -175,10 +256,20 @@ the hosted API.
 - [x] JSON/CSV catalog export
 - [x] [`0.0.2`: local dataset preflight](docs/version-0.0.2.md), including a
       versioned manifest, deterministic audit findings, and local LeRobot discovery
-- [ ] Later: RLDS / HDF5 ingestion helpers
-- [ ] Later: calibrated quality evaluation after a labeled validation set exists
+- [x] [`0.0.3`: LeRobot compatibility and dataset change control](docs/version-0.0.3.md),
+      with correctness closure, explicit v2.1/v3 adapters, layered validation,
+      portable snapshots, semantic diff, revision-pinned Hub audit, readiness
+      profiles, evidence triage, conservative repair, and merge verification.
+      P0 interfaces, the package version, clean install, Python matrix,
+      artifacts, official conformance, and local release checks pass. Public
+      release remains a separate explicit action.
+- [ ] Later: read-only robomimic/HDF5 and RLDS/Open X adapters
+- [ ] Later: advanced episode-quality signals and any aggregate score only after
+      labeled downstream validation exists
 
 Audit codes are documented in [`docs/audit-findings.md`](docs/audit-findings.md).
+The research and differentiation decisions behind `0.0.3` are documented in
+[`docs/reference-libraries.md`](docs/reference-libraries.md).
 Canonical manifests use explicit `path_base`/`path_bases` fields; they never
 serialize private source paths. Symlinks are skipped by default, and even when
 `follow_symlinks=True` a target outside the dataset root is rejected.

openbot_data-0.0.3/VERSION

@@ -0,0 +1 @@
+0.0.3

openbot_data-0.0.3/docs/README.md

@@ -0,0 +1,36 @@
+# OpenBot Data documentation
+
+OpenBot Data is a local, open-source Python library. It inspects robot-video
+directories and local LeRobot v2.1/v3 repositories before training or upload. It
+does not call the Hosted OpenBot API.
+
+## Start here
+
+- [Getting started](getting-started.md): installation, CLI, Python workflow, and
+  the runnable demo.
+- [API reference](api-reference.md): public Python functions, CLI commands,
+  parameters, return values, and failure behavior.
+- [Audit finding codes](audit-findings.md): stable machine-readable validation
+  codes and severities.
+- [0.0.2 release contract](version-0.0.2.md): scope and acceptance criteria.
+- [0.0.3 feature and release contract](version-0.0.3.md): the complete required
+  feature map, public artifacts and commands, LeRobot compatibility, readiness,
+  remediation, merge safety, snapshots, semantic diff, current implementation
+  checkpoint, and release acceptance criteria.
+- [Reference libraries and differentiation](reference-libraries.md):
+  LeRobot, `lerobot-doctor`, `robovet`, and adjacent-library decisions.
+- [Canonical manifest](examples/manifest.json) and
+  [canonical audit](examples/audit.json): released v1 examples without local
+  machine paths.
+- [0.0.3 artifact examples](examples/v0.0.3/README.md): generated snapshot,
+  diff, readiness, Catalog evidence, repair, and merge contracts.
+
+## Capability boundary
+
+This repository owns local and revision-pinned Hub preflight, media inspection,
+deterministic manifests/audits/snapshots/diffs, readiness gates, conservative
+copy-on-write metadata repair, official-operation verification, catalog export,
+and the score-free Catalog evidence handoff. Catalog scoring, candidate review,
+publication, uploads, API keys, asynchronous processing, hosted artifacts,
+retention, and billing belong to the main OpenBot service and are called through
+`openbot-sdk`.