episodevault 0.0.2__tar.gz

episodevault-0.0.2/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 [AUTHOR]
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

episodevault-0.0.2/PKG-INFO

@@ -0,0 +1,152 @@
+Metadata-Version: 2.4
+Name: episodevault
+Version: 0.0.2
+Summary: Version control and lineage tracking for robot training episode data
+License: MIT
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pyarrow>=14.0
+Requires-Dist: pandas>=2.0
+Requires-Dist: duckdb>=0.10
+Requires-Dist: deltalake>=0.17
+Requires-Dist: click>=8.1
+Requires-Dist: rich>=13.0
+Requires-Dist: pydantic>=2.0
+Requires-Dist: xxhash>=3.4
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0; extra == "dev"
+Requires-Dist: pytest-cov>=5.0; extra == "dev"
+Requires-Dist: ruff>=0.4; extra == "dev"
+Requires-Dist: mypy>=1.10; extra == "dev"
+Dynamic: license-file
+
+# EpisodeVault: find out exactly why your robot model regressed.
+
+[![PyPI](https://img.shields.io/pypi/v/episodevault)](https://pypi.org/project/episodevault/)
+[![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
+[![LeRobot v3](https://img.shields.io/badge/LeRobot-v3-green.svg)](https://github.com/huggingface/lerobot)
+
+---
+
+## The problem
+
+Every robotics ML engineer has retrained a model and watched performance drop with no clear cause. DVC tracks which files changed. MLflow tracks which hyperparameters ran. Nobody tracks what changed at the episode level, which tasks dropped out, which quality metrics shifted, which task distribution moved between v1 and v2 of your dataset.
+
+EpisodeVault fills that gap.
+
+---
+
+## What EpisodeVault does
+
+Run `episodevault diff v1.0 v2.0` and get this:
+
+```
+Dataset diff: v1.0 → v2.0
+────────────────────────────────────────────────────
+Episodes added:    +0
+Episodes removed:  -7
+
+Distribution shift:
+  factory_pick                     2 → 6  ↑ 200%  ⚠️
+  kitchen_grasp                    4 → 1  ↓ 75%  ⚠️
+
+Quality metrics:
+  avg episode length:    3.7s → 3.0s  ↓
+  success_rate:          0.88 → 0.38  ↓
+  camera_sync_score:     1.00 → 1.00  →
+
+Regression candidates (ranked by magnitude; correlate with your eval):
+  - 'kitchen_grasp' episodes dropped 75% (4 → 1). Restore from prior
+    version if this task is in your eval benchmark.
+  - Success rate fell 50% (0.88 → 0.38). New episodes may contain failed
+    demonstrations. Run score_lerobot_episodes to identify low-quality additions.
+```
+
+---
+
+## Install
+
+```bash
+pip install episodevault
+```
+
+Requires Python 3.10+. Key dependencies: `pyarrow`, `pandas`, `duckdb`, `click`, `rich`, `pydantic`.
+
+---
+
+## Quickstart
+
+```bash
+# Start tracking a local LeRobot dataset
+episodevault track ./my_dataset
+
+# Snapshot the current state with a message
+episodevault commit -m "added 500 kitchen episodes"
+
+# Compare two snapshots
+episodevault diff v1.0 v2.0
+
+# Find what dataset a model was trained on and diff against the prior version
+episodevault blame model_v3
+```
+
+`track` initializes a `.episodevault/` store inside your dataset directory. `commit` snapshots the episode manifest (not raw sensor data -- fast). `diff` computes task distribution shift and quality deltas between any two versions. `blame` looks up which dataset version trained a given model and diffs it against the version before.
+
+---
+
+## Python API
+
+Log a training run from your training script so `blame` can trace it back:
+
+```python
+import episodevault as ev
+
+ev.log_training_run(
+    model_version="model_v3",
+    dataset_version="v2.0",
+    framework="lerobot"
+)
+```
+
+One call. That's all `blame` needs.
+
+---
+
+## Compatibility
+
+Tested against real HuggingFace LeRobot v3 datasets:
+
+| Dataset          | Robot  | Format     | Episodes | Parse time | Status |
+|------------------|--------|------------|----------|------------|--------|
+| aloha_pencil     | aloha  | LeRobot v3 | 25       | 0.33s      | OK     |
+| aloha_shrimp     | aloha  | LeRobot v3 | 18       | 0.38s      | OK     |
+| so100_stacking   | so100  | LeRobot v3 | 56       | 0.65s      | OK     |
+| aloha_cabinet    | aloha  | LeRobot v3 | 85       | 2.65s      | OK     |
+
+Parse time is for the episode manifest only. Raw sensor data (video, joint trajectories) is never loaded.
+
+---
+
+## How it works
+
+- Parses episode manifests (`meta/episodes/`, `meta/tasks.parquet`, `meta/info.json`) without loading raw sensor data -- sub-second parse regardless of frame count or video size.
+- Snapshots manifests into a version store on every `commit` -- diff and time travel are built in from the start.
+- Diff engine computes task distribution shift and quality deltas between any two snapshots -- regression candidates are ranked by a normalized severity score and the top few are surfaced, not asserted as proven causes.
+
+
+---
+
+## Credits
+
+- [HuggingFace LeRobot](https://github.com/huggingface/lerobot) team for the v3 dataset format that EpisodeVault parses.
+- Berkeley AutoLab (Kaiyuan Chen et al.) for [Robo-DM / fog_x](https://github.com/BerkeleyAutomation/fog_x), prior work on robot dataset management.
+- [score_lerobot_episodes](https://github.com/RobotData/score-lerobot-episodes) by RobotData for quality signal methodology.
+- [Evidently AI](https://github.com/evidentlyai/evidently) for drift detection methodology that informed the distribution shift logic.
+
+---
+
+## License
+
+MIT. See [LICENSE](LICENSE).

episodevault-0.0.2/README.md

@@ -0,0 +1,129 @@
+# EpisodeVault: find out exactly why your robot model regressed.
+
+[![PyPI](https://img.shields.io/pypi/v/episodevault)](https://pypi.org/project/episodevault/)
+[![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
+[![LeRobot v3](https://img.shields.io/badge/LeRobot-v3-green.svg)](https://github.com/huggingface/lerobot)
+
+---
+
+## The problem
+
+Every robotics ML engineer has retrained a model and watched performance drop with no clear cause. DVC tracks which files changed. MLflow tracks which hyperparameters ran. Nobody tracks what changed at the episode level, which tasks dropped out, which quality metrics shifted, which task distribution moved between v1 and v2 of your dataset.
+
+EpisodeVault fills that gap.
+
+---
+
+## What EpisodeVault does
+
+Run `episodevault diff v1.0 v2.0` and get this:
+
+```
+Dataset diff: v1.0 → v2.0
+────────────────────────────────────────────────────
+Episodes added:    +0
+Episodes removed:  -7
+
+Distribution shift:
+  factory_pick                     2 → 6  ↑ 200%  ⚠️
+  kitchen_grasp                    4 → 1  ↓ 75%  ⚠️
+
+Quality metrics:
+  avg episode length:    3.7s → 3.0s  ↓
+  success_rate:          0.88 → 0.38  ↓
+  camera_sync_score:     1.00 → 1.00  →
+
+Regression candidates (ranked by magnitude; correlate with your eval):
+  - 'kitchen_grasp' episodes dropped 75% (4 → 1). Restore from prior
+    version if this task is in your eval benchmark.
+  - Success rate fell 50% (0.88 → 0.38). New episodes may contain failed
+    demonstrations. Run score_lerobot_episodes to identify low-quality additions.
+```
+
+---
+
+## Install
+
+```bash
+pip install episodevault
+```
+
+Requires Python 3.10+. Key dependencies: `pyarrow`, `pandas`, `duckdb`, `click`, `rich`, `pydantic`.
+
+---
+
+## Quickstart
+
+```bash
+# Start tracking a local LeRobot dataset
+episodevault track ./my_dataset
+
+# Snapshot the current state with a message
+episodevault commit -m "added 500 kitchen episodes"
+
+# Compare two snapshots
+episodevault diff v1.0 v2.0
+
+# Find what dataset a model was trained on and diff against the prior version
+episodevault blame model_v3
+```
+
+`track` initializes a `.episodevault/` store inside your dataset directory. `commit` snapshots the episode manifest (not raw sensor data -- fast). `diff` computes task distribution shift and quality deltas between any two versions. `blame` looks up which dataset version trained a given model and diffs it against the version before.
+
+---
+
+## Python API
+
+Log a training run from your training script so `blame` can trace it back:
+
+```python
+import episodevault as ev
+
+ev.log_training_run(
+    model_version="model_v3",
+    dataset_version="v2.0",
+    framework="lerobot"
+)
+```
+
+One call. That's all `blame` needs.
+
+---
+
+## Compatibility
+
+Tested against real HuggingFace LeRobot v3 datasets:
+
+| Dataset          | Robot  | Format     | Episodes | Parse time | Status |
+|------------------|--------|------------|----------|------------|--------|
+| aloha_pencil     | aloha  | LeRobot v3 | 25       | 0.33s      | OK     |
+| aloha_shrimp     | aloha  | LeRobot v3 | 18       | 0.38s      | OK     |
+| so100_stacking   | so100  | LeRobot v3 | 56       | 0.65s      | OK     |
+| aloha_cabinet    | aloha  | LeRobot v3 | 85       | 2.65s      | OK     |
+
+Parse time is for the episode manifest only. Raw sensor data (video, joint trajectories) is never loaded.
+
+---
+
+## How it works
+
+- Parses episode manifests (`meta/episodes/`, `meta/tasks.parquet`, `meta/info.json`) without loading raw sensor data -- sub-second parse regardless of frame count or video size.
+- Snapshots manifests into a version store on every `commit` -- diff and time travel are built in from the start.
+- Diff engine computes task distribution shift and quality deltas between any two snapshots -- regression candidates are ranked by a normalized severity score and the top few are surfaced, not asserted as proven causes.
+
+
+---
+
+## Credits
+
+- [HuggingFace LeRobot](https://github.com/huggingface/lerobot) team for the v3 dataset format that EpisodeVault parses.
+- Berkeley AutoLab (Kaiyuan Chen et al.) for [Robo-DM / fog_x](https://github.com/BerkeleyAutomation/fog_x), prior work on robot dataset management.
+- [score_lerobot_episodes](https://github.com/RobotData/score-lerobot-episodes) by RobotData for quality signal methodology.
+- [Evidently AI](https://github.com/evidentlyai/evidently) for drift detection methodology that informed the distribution shift logic.
+
+---
+
+## License
+
+MIT. See [LICENSE](LICENSE).

episodevault-0.0.2/episodevault/__init__.py

@@ -0,0 +1,19 @@
+from episodevault.api import log_training_run
+from episodevault.diff.engine import diff
+from episodevault.models import DatasetManifest, EpisodeManifest, EpisodeQuality
+from episodevault.parsers.lerobot import parse as parse_lerobot
+from episodevault.store.lineage_store import LineageStore
+from episodevault.store.version_store import VersionStore
+
+__all__ = [
+    "parse_lerobot",
+    "VersionStore",
+    "LineageStore",
+    "diff",
+    "log_training_run",
+    "DatasetManifest",
+    "EpisodeManifest",
+    "EpisodeQuality",
+]
+
+__version__ = "0.1.0"

episodevault-0.0.2/episodevault/api.py

@@ -0,0 +1,23 @@
+from __future__ import annotations
+
+from pathlib import Path
+from typing import Any
+
+from episodevault.store.lineage_store import LineageStore
+
+
+def log_training_run(
+    model_version: str,
+    dataset_version: str,
+    dataset_path: str | Path = ".",
+    framework: str = "lerobot",
+    **extras: Any,
+) -> None:
+    store_path = Path(dataset_path) / ".episodevault"
+    lineage = LineageStore(store_path)
+    lineage.log_training_run(
+        model_version=model_version,
+        dataset_version=dataset_version,
+        framework=framework,
+        **extras,
+    )

episodevault-0.0.2/episodevault/cli/main.py

@@ -0,0 +1,178 @@
+from __future__ import annotations
+
+import sys
+from pathlib import Path
+
+import click
+from rich.console import Console
+from rich.table import Table
+
+from episodevault.diff.engine import diff as compute_diff
+from episodevault.parsers.lerobot import parse as parse_lerobot
+from episodevault.store.lineage_store import LineageStore
+from episodevault.store.version_store import VersionStore
+
+_console = Console(force_terminal=True)
+
+def _resolve_store(dataset_path: Path) -> Path:
+    return dataset_path / ".episodevault"
+
+
+@click.group()
+def cli() -> None:
+    pass
+
+
+@cli.command()
+@click.argument("dataset_path")
+def track(dataset_path: str) -> None:
+    if "/" in dataset_path and not Path(dataset_path).exists():
+        local_name = dataset_path.replace("/", "__")
+        _console.print(
+            f"[red]error:[/red] '{dataset_path}' looks like a HuggingFace repo ID, "
+            "not a local path."
+        )
+        _console.print(
+            f"Download first with:\n"
+            f"  huggingface-cli download --repo-type dataset {dataset_path} "
+            f"--local-dir ./{local_name}"
+        )
+        sys.exit(1)
+
+    root = Path(dataset_path)
+    if not root.exists():
+        _console.print(f"[red]error:[/red] path '{dataset_path}' does not exist.")
+        sys.exit(1)
+
+    store_path = _resolve_store(root)
+    store_path.mkdir(parents=True, exist_ok=True)
+    (store_path / ".gitignore").write_text("*.parquet\n")
+    _console.print(f"[green]Tracking[/green] {root.resolve()}")
+    _console.print(f"Store initialised at {store_path}")
+
+
+@cli.command()
+@click.argument("dataset_path", type=click.Path(exists=True))
+@click.option("-m", "--message", required=True, help="Commit message")
+def commit(dataset_path: str, message: str) -> None:
+    root = Path(dataset_path)
+    store_path = _resolve_store(root)
+
+    if not store_path.exists():
+        _console.print(
+            "[red]error:[/red] dataset not tracked. Run `episodevault track` first."
+        )
+        sys.exit(1)
+
+    _console.print("Parsing dataset…")
+    try:
+        manifest = parse_lerobot(root)
+    except (FileNotFoundError, ValueError) as exc:
+        _console.print(f"[red]error:[/red] {exc}")
+        sys.exit(1)
+
+    store = VersionStore(store_path)
+    version_id = store.commit(manifest, message)
+
+    _console.print(f"[green]Committed[/green] {version_id}  {message}")
+    _console.print(
+        f"  {manifest.total_episodes} episodes · "
+        f"{len(manifest.tasks)} tasks · "
+        f"{manifest.robot_type}"
+    )
+
+
+@cli.command(name="diff")
+@click.argument("version_before")
+@click.argument("version_after")
+@click.argument("dataset_path", type=click.Path(exists=True), default=".")
+def diff_cmd(version_before: str, version_after: str, dataset_path: str) -> None:
+    root = Path(dataset_path)
+    store_path = _resolve_store(root)
+    store = VersionStore(store_path)
+
+    try:
+        manifest_before = store.read_version(version_before)
+        manifest_after = store.read_version(version_after)
+    except KeyError as exc:
+        _console.print(f"[red]error:[/red] {exc}")
+        sys.exit(1)
+
+    object.__setattr__(manifest_before, "_version_id", version_before)
+    object.__setattr__(manifest_after, "_version_id", version_after)
+
+    result = compute_diff(manifest_before, manifest_after)
+    _console.print(result.format())
+
+
+@cli.command()
+@click.argument("model_version")
+@click.argument("dataset_path", type=click.Path(exists=True), default=".")
+def blame(model_version: str, dataset_path: str) -> None:
+    root = Path(dataset_path)
+    store_path = _resolve_store(root)
+    lineage = LineageStore(store_path)
+    store = VersionStore(store_path)
+
+    dataset_version = lineage.dataset_version_for_model(model_version)
+    if dataset_version is None:
+        _console.print(
+            f"[red]error:[/red] No training run found for model '{model_version}'. "
+            "Log runs with ev.log_training_run() in your training script."
+        )
+        sys.exit(1)
+
+    versions = store.list_versions()
+    version_ids = [v["version_id"] for v in versions]
+
+    if dataset_version not in version_ids:
+        _console.print(
+            f"[yellow]warning:[/yellow] Model '{model_version}' trained on "
+            f"dataset version '{dataset_version}' which is not in the store."
+        )
+        sys.exit(1)
+
+    idx = version_ids.index(dataset_version)
+    _console.print(
+        f"[bold]{model_version}[/bold] was trained on dataset version "
+        f"[bold]{dataset_version}[/bold]"
+    )
+
+    if idx == 0:
+        _console.print("No prior version to diff against.")
+        return
+
+    prior_version = version_ids[idx - 1]
+    _console.print(f"Diffing {prior_version} → {dataset_version}:\n")
+
+    manifest_before = store.read_version(prior_version)
+    manifest_after = store.read_version(dataset_version)
+    result = compute_diff(manifest_before, manifest_after)
+    _console.print(result.format())
+
+
+@cli.command()
+@click.argument("dataset_path", type=click.Path(exists=True), default=".")
+def log(dataset_path: str) -> None:
+    root = Path(dataset_path)
+    store_path = _resolve_store(root)
+    store = VersionStore(store_path)
+    versions = store.list_versions()
+
+    if not versions:
+        _console.print("No commits yet.")
+        return
+
+    table = Table(show_header=True, header_style="bold")
+    table.add_column("Version")
+    table.add_column("Episodes", justify="right")
+    table.add_column("Message")
+
+    for v in reversed(versions):
+        table.add_row(
+            str(v["version_id"]),
+            str(v["total_episodes"]),
+            str(v["commit_message"]),
+        )
+
+    _console.print(table)