roboaudit 0.2.2__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- roboaudit-0.2.2/LICENSE +21 -0
- roboaudit-0.2.2/PKG-INFO +503 -0
- roboaudit-0.2.2/README.md +484 -0
- roboaudit-0.2.2/pyproject.toml +35 -0
- roboaudit-0.2.2/setup.cfg +4 -0
- roboaudit-0.2.2/src/roboaudit/__init__.py +3 -0
- roboaudit-0.2.2/src/roboaudit/__main__.py +120 -0
- roboaudit-0.2.2/src/roboaudit/audit.py +197 -0
- roboaudit-0.2.2/src/roboaudit/checks12.py +603 -0
- roboaudit-0.2.2/src/roboaudit/disposition.py +123 -0
- roboaudit-0.2.2/src/roboaudit/reader.py +362 -0
- roboaudit-0.2.2/src/roboaudit/report.py +201 -0
- roboaudit-0.2.2/src/roboaudit/scoring.py +28 -0
- roboaudit-0.2.2/src/roboaudit/share.py +222 -0
- roboaudit-0.2.2/src/roboaudit.egg-info/PKG-INFO +503 -0
- roboaudit-0.2.2/src/roboaudit.egg-info/SOURCES.txt +18 -0
- roboaudit-0.2.2/src/roboaudit.egg-info/dependency_links.txt +1 -0
- roboaudit-0.2.2/src/roboaudit.egg-info/entry_points.txt +2 -0
- roboaudit-0.2.2/src/roboaudit.egg-info/requires.txt +4 -0
- roboaudit-0.2.2/src/roboaudit.egg-info/top_level.txt +1 -0
roboaudit-0.2.2/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 RoboAudit contributors
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
roboaudit-0.2.2/PKG-INFO
ADDED
|
@@ -0,0 +1,503 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: roboaudit
|
|
3
|
+
Version: 0.2.2
|
|
4
|
+
Summary: Health-check CLI for LeRobot robot datasets: 100-point score, per-episode dispositions (exclude / review / keep), and an auditable report.
|
|
5
|
+
License: MIT
|
|
6
|
+
Keywords: lerobot,robotics,dataset,data-quality,audit
|
|
7
|
+
Classifier: Programming Language :: Python :: 3
|
|
8
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
9
|
+
Classifier: Operating System :: OS Independent
|
|
10
|
+
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
|
|
11
|
+
Requires-Python: >=3.10
|
|
12
|
+
Description-Content-Type: text/markdown
|
|
13
|
+
License-File: LICENSE
|
|
14
|
+
Requires-Dist: numpy>=1.24
|
|
15
|
+
Requires-Dist: opencv-python-headless>=4.8
|
|
16
|
+
Requires-Dist: pyarrow>=12
|
|
17
|
+
Requires-Dist: jinja2>=3.0
|
|
18
|
+
Dynamic: license-file
|
|
19
|
+
|
|
20
|
+
# RoboAudit
|
|
21
|
+
|
|
22
|
+
**Before you train on it, audit it.** RoboAudit is a health-check CLI for
|
|
23
|
+
[LeRobot](https://github.com/huggingface/lerobot) robot datasets: give it a
|
|
24
|
+
dataset folder and it returns a **100-point score with a grade**, a list of
|
|
25
|
+
what is wrong and where, and episode indexes that sort episodes into
|
|
26
|
+
keep / review / exclude for training-data selection.
|
|
27
|
+
|
|
28
|
+

|
|
29
|
+
|
|
30
|
+
> **Why it exists**
|
|
31
|
+
>
|
|
32
|
+
> 1. **Planted-defect test: 43/43 detected** — the comparable open-source
|
|
33
|
+
> auditor caught 22 at verdict level.\*
|
|
34
|
+
> 2. **Audits legacy v2.1 datasets directly, without conversion** — with
|
|
35
|
+
> LeRobot 0.7.0, the official loader refuses v2.1 datasets
|
|
36
|
+
> (BackwardCompatibilityError); RoboAudit's own parser reads them.
|
|
37
|
+
> 3. **Not just a score:** exclude / review / keep episode indexes for
|
|
38
|
+
> selecting which episodes to train on.
|
|
39
|
+
>
|
|
40
|
+
> \* The defect suite was built by the RoboAudit authors — a home-field
|
|
41
|
+
> test, so read it as one. Full comparison in
|
|
42
|
+
> [How it compares](#how-it-compares) below.
|
|
43
|
+
|
|
44
|
+
## Quick start
|
|
45
|
+
|
|
46
|
+
```bash
|
|
47
|
+
git clone https://github.com/parkjb0826/roboaudit
|
|
48
|
+
cd roboaudit
|
|
49
|
+
python3 -m venv .venv && source .venv/bin/activate # recommended
|
|
50
|
+
pip install .
|
|
51
|
+
roboaudit /path/to/lerobot_dataset --out ./audit_out
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
> ⚠️ `--out` must point **outside** the dataset folder. If it points
|
|
55
|
+
> inside, the run is refused (exit code 2) — this protects the original
|
|
56
|
+
> data from ever being written into.
|
|
57
|
+
|
|
58
|
+
That writes `report.html` (the report card), `result.json`,
|
|
59
|
+
`clean_index.json`, and `review_index.json` into `./audit_out`. No robot,
|
|
60
|
+
no GPU — CPU only. Getting to the first command takes about a minute;
|
|
61
|
+
the audit itself takes minutes, not seconds — measured on this
|
|
62
|
+
project's (small, shared) VM CPU: 152–180 s for 50–75-episode datasets
|
|
63
|
+
with one camera decoded, and 296 s for a 50-episode dataset with two
|
|
64
|
+
cameras decoded. Details (Python version, optional ffmpeg, sampling
|
|
65
|
+
mode, exit codes) are in [Install](#install) and [Usage](#usage) below.
|
|
66
|
+
|
|
67
|
+
## Report cards from real datasets
|
|
68
|
+
|
|
69
|
+

|
|
70
|
+
|
|
71
|
+
Real audits of three public datasets — not demos, not synthetic data
|
|
72
|
+
(full method and caveats in [Measured results](#measured-results-not-marketing-numbers)).
|
|
73
|
+
One caveat up front: these three datasets were also used to calibrate
|
|
74
|
+
the disposition thresholds, so read their grades as in-sample results:
|
|
75
|
+
|
|
76
|
+
| Dataset | Format | Episodes | Score | Exclude / Review | One-line verdict |
|
|
77
|
+
|---|---|---|---|---|---|
|
|
78
|
+
| `lerobot/aloha_sim_transfer_cube_scripted` | v3.0 | 50 | **79 / C** | 0 / 2 | Two episodes sit just past a range-band edge (measured at 0.28–0.44 of the exclusion margin) — sent to a human, not dropped |
|
|
79
|
+
| `00ri/so100_battery` | v2.1 | 50 | **84 / B** | 0 / 5 | Five "shake" flags that look like busy episodes rather than camera shake — all review, none auto-dropped; both declared cameras audited |
|
|
80
|
+
| `sreetz-nv/so101_teleop_vials_rack_left` (partial copy) | v3.0 | 75 | **44 / F** | 7 / 8 | Audited from a **partial copy** (a 166 MB download covering only one of the two declared cameras): the missing camera is a property of this copy — the upstream dataset may be complete. The audit is flagged **incomplete** instead of passing silently; 15 jump findings split by measured strength |
|
|
81
|
+
|
|
82
|
+
## How it compares
|
|
83
|
+
|
|
84
|
+

|
|
85
|
+
|
|
86
|
+
Head-to-head with RDA (Robot Data Audit, the comparable open-source
|
|
87
|
+
auditor on PyPI), run on the same data in its full mode (`--full`, every
|
|
88
|
+
check enabled) but without a platform profile:
|
|
89
|
+
|
|
90
|
+
| | RoboAudit | RDA 0.12.4 |
|
|
91
|
+
|---|---|---|
|
|
92
|
+
| Planted defects (43) reaching a verdict | **43** | 22 (15 exclude + 7 correctly pinpointed as unverifiable) |
|
|
93
|
+
| Recorded as a finding only (no verdict) | 0 | 2 |
|
|
94
|
+
| Outside the tool's check set (no verdict, no finding) | **0** | 19 — check types RDA does not implement (state jumps, range violations, dead dimensions, duplicates, instruction problems, episode boundaries) |
|
|
95
|
+
| False-positive verdicts on clean data (40 episodes) | 0 | 0 |
|
|
96
|
+
| Behaviour on the 3 real datasets above | Borderline findings → `review` | All-pass on the two complete datasets; refused a verdict on the partial copy |
|
|
97
|
+
|
|
98
|
+
Read this fairly: the defect suite is RoboAudit's home field, RDA was run
|
|
99
|
+
without a platform profile (its joint-limit check needs one), and RDA has
|
|
100
|
+
its own separate blind test. On real data RDA is the more conservative
|
|
101
|
+
design — it records risk signals at metric level without letting them
|
|
102
|
+
change verdicts. RoboAudit's difference is breadth of checks that *do*
|
|
103
|
+
reach a verdict, plus dispositions a human can overrule.
|
|
104
|
+
|
|
105
|
+
## Demo
|
|
106
|
+
|
|
107
|
+
A fixed-data demo lives in [`demo/`](demo/): pick one of the three
|
|
108
|
+
pre-audited datasets above and browse its stored score summary and full
|
|
109
|
+
HTML report. The demo files are prepared and verified locally; the demo
|
|
110
|
+
itself is not deployed yet. Arbitrary uploads are deliberately not
|
|
111
|
+
supported — auditing is CPU-heavy (minutes per dataset), so a public
|
|
112
|
+
demo needs a queue and abuse controls first.
|
|
113
|
+
|
|
114
|
+
---
|
|
115
|
+
|
|
116
|
+
The idea is simple: robot training is expensive, and a dataset with missing
|
|
117
|
+
videos, NaN actions, reversed timestamps, or broken instructions wastes that
|
|
118
|
+
money silently. RoboAudit is the cheap inspection you run *before* training.
|
|
119
|
+
|
|
120
|
+
- Reads **LeRobot v2.1 and v3.0** with its own parser — no conversion step.
|
|
121
|
+
v2.1 is read directly, even though with LeRobot 0.7.0 the official
|
|
122
|
+
loader refuses v2.1 datasets (BackwardCompatibilityError).
|
|
123
|
+
- **12 rule/threshold checks** (structure, video, action, instruction).
|
|
124
|
+
No AI inside the checks: every deduction traces back to a rule and a
|
|
125
|
+
measured value, so a score can always be explained.
|
|
126
|
+
- **Original data is read-only.** RoboAudit never modifies the dataset
|
|
127
|
+
(verified by SHA-256 before/after in its own test suite).
|
|
128
|
+
- No robot, no GPU. CPU only.
|
|
129
|
+
|
|
130
|
+
## Install
|
|
131
|
+
|
|
132
|
+
```bash
|
|
133
|
+
git clone https://github.com/parkjb0826/roboaudit
|
|
134
|
+
cd roboaudit
|
|
135
|
+
python3 -m venv .venv && source .venv/bin/activate # recommended
|
|
136
|
+
pip install .
|
|
137
|
+
```
|
|
138
|
+
|
|
139
|
+
(Not yet on PyPI — install from the cloned repository for now.)
|
|
140
|
+
|
|
141
|
+
Requires Python ≥ 3.10 (developed and verified on Linux with
|
|
142
|
+
Python 3.12; other platforms are untested).
|
|
143
|
+
Runtime dependencies: `numpy`, `opencv-python-headless`, `pyarrow`, `jinja2`.
|
|
144
|
+
|
|
145
|
+
Optional: if a system `ffmpeg` binary is on your `PATH`, it is used as a
|
|
146
|
+
fallback decoder for video codecs OpenCV cannot read (e.g. AV1).
|
|
147
|
+
|
|
148
|
+
## Usage
|
|
149
|
+
|
|
150
|
+
Basic audit — writes `report.html`, `result.json`, `clean_index.json`, and
|
|
151
|
+
`review_index.json` into the output folder:
|
|
152
|
+
|
|
153
|
+
```bash
|
|
154
|
+
roboaudit /path/to/lerobot_dataset --out ./audit_out
|
|
155
|
+
```
|
|
156
|
+
|
|
157
|
+
The argument is the dataset **root folder on local disk** — RoboAudit
|
|
158
|
+
does not download anything, so fetch the dataset first (e.g. from the
|
|
159
|
+
Hugging Face Hub). The root is the folder that contains `meta/info.json`:
|
|
160
|
+
|
|
161
|
+
```
|
|
162
|
+
my_dataset/ ← pass this folder
|
|
163
|
+
├── meta/
|
|
164
|
+
│ ├── info.json
|
|
165
|
+
│ ├── episodes/chunk-000/file-000.parquet
|
|
166
|
+
│ └── tasks.parquet
|
|
167
|
+
├── data/chunk-000/file-000.parquet (shared across episodes)
|
|
168
|
+
└── videos/observation.images.top/chunk-000/file-000.mp4
|
|
169
|
+
```
|
|
170
|
+
|
|
171
|
+
That is the v3.0 layout. The v2.1 layout differs — per-episode files
|
|
172
|
+
(`data/chunk-000/episode_000000.parquet`,
|
|
173
|
+
`videos/chunk-000/<camera_key>/episode_000000.mp4`, `meta/episodes.jsonl`)
|
|
174
|
+
— and is read directly as well. If `meta/info.json` is missing, the
|
|
175
|
+
command stops with exit code 2 (see [Troubleshooting](#troubleshooting)).
|
|
176
|
+
|
|
177
|
+
Sampling mode for very large datasets — video statistics are computed from
|
|
178
|
+
an evenly spaced sample of at most N frames per episode instead of every
|
|
179
|
+
frame. Faster, but approximate (see *Limitations*):
|
|
180
|
+
|
|
181
|
+
```bash
|
|
182
|
+
roboaudit /path/to/big_dataset --out ./audit_out --sample-frames 60
|
|
183
|
+
```
|
|
184
|
+
|
|
185
|
+
Exit codes are part of the interface, so the tool can gate a pipeline:
|
|
186
|
+
|
|
187
|
+
| Code | Meaning |
|
|
188
|
+
|---|---|
|
|
189
|
+
| 0 | Audit complete — every check covered its full scope (regardless of how many defects were found) |
|
|
190
|
+
| 1 | Unexpected runtime failure |
|
|
191
|
+
| 2 | Input error — not a LeRobot dataset folder, or `--out` points inside the dataset folder (refused, to protect the original data) |
|
|
192
|
+
| 3 | Audit finished but **incomplete** — some check ran partially or errored; the report says which and why |
|
|
193
|
+
|
|
194
|
+
**Exit code 0 means the audit ran to completion with full coverage. It
|
|
195
|
+
does NOT mean the dataset passed** — a complete audit can still find
|
|
196
|
+
critical defects and score an F. If you gate a pipeline on RoboAudit,
|
|
197
|
+
check the score and findings in `result.json` too, and treat exit
|
|
198
|
+
code 3 (incomplete) as "the indexes are not final" (see
|
|
199
|
+
[Score and dispositions](#score-and-dispositions)).
|
|
200
|
+
|
|
201
|
+
## Outputs
|
|
202
|
+
|
|
203
|
+
| File | Contents |
|
|
204
|
+
|---|---|
|
|
205
|
+
| `report.html` | Human-readable report card: score and grade, top-3 deduction factors, per-check run status and findings, per-camera decode statistics, scope notices |
|
|
206
|
+
| `result.json` | Machine-readable full result, including a **run manifest** (tool/rules versions, thresholds actually applied, input fingerprint, decode backends) to help reproduce runs and compare their settings and results |
|
|
207
|
+
| `clean_index.json` | The episode index file: `clean_episodes` (episodes assigned `keep`), plus the `excluded_episodes` and `review_episodes` lists with the checks that flagged each one |
|
|
208
|
+
| `review_index.json` | Borderline episodes a human should look at before deciding — these are **not** in `clean_episodes` (see below) |
|
|
209
|
+
|
|
210
|
+
Episode indices everywhere are the dataset's own `episode_index`
|
|
211
|
+
values from its metadata. The three lists are disjoint and together
|
|
212
|
+
cover every episode: an episode appears in exactly one of
|
|
213
|
+
`clean_episodes`, `review_episodes`, or `excluded_episodes`. Excluded
|
|
214
|
+
episodes appear in neither the keep list nor the review list — they
|
|
215
|
+
are recorded, with the checks that flagged them, in the
|
|
216
|
+
`excluded_episodes` field of `clean_index.json` and in `result.json`.
|
|
217
|
+
|
|
218
|
+
### Using the indexes in training code
|
|
219
|
+
|
|
220
|
+
```python
|
|
221
|
+
import json
|
|
222
|
+
|
|
223
|
+
index = json.load(open("audit_out/clean_index.json"))
|
|
224
|
+
result = json.load(open("audit_out/result.json"))
|
|
225
|
+
|
|
226
|
+
# 1. If the audit was incomplete, the indexes are not final — a check
|
|
227
|
+
# that could not run may have missed defects (see report.html):
|
|
228
|
+
if result["incomplete_audit"]:
|
|
229
|
+
raise SystemExit("Incomplete audit: indexes are not final")
|
|
230
|
+
|
|
231
|
+
keep = set(index["clean_episodes"]) # episode_index values with no finding
|
|
232
|
+
review = {e["episode_index"] for e in index["review_episodes"]}
|
|
233
|
+
|
|
234
|
+
# 2. Decide on the review episodes explicitly — do not drop them by
|
|
235
|
+
# accident. Inspect them first, then name the ones you accept:
|
|
236
|
+
approved_after_review = set() # e.g. {3, 17}, once you have looked
|
|
237
|
+
|
|
238
|
+
selected = sorted(keep | approved_after_review)
|
|
239
|
+
|
|
240
|
+
# 3. Pass the selection to your training set — with LeRobot, the
|
|
241
|
+
# `episodes=` argument of LeRobotDataset takes exactly this list:
|
|
242
|
+
# ds = LeRobotDataset("you/dataset", episodes=selected)
|
|
243
|
+
train_episodes = [ep for ep in my_episode_ids if ep in set(selected)]
|
|
244
|
+
```
|
|
245
|
+
|
|
246
|
+
`review_episodes` and `excluded_episodes` in the same file are lists of
|
|
247
|
+
`{"episode_index": …, "checks": […]}` objects, so you can see *why*
|
|
248
|
+
each episode was held back before deciding what to do with it.
|
|
249
|
+
|
|
250
|
+
## Share an audit case (opt-in)
|
|
251
|
+
|
|
252
|
+
RoboAudit's disposition thresholds (such as the 1.5× borderline margin)
|
|
253
|
+
are working values, not final constants: they were calibrated on
|
|
254
|
+
synthetic injections plus three real datasets and have not yet
|
|
255
|
+
been validated against many adjudicated real-world cases. The honest
|
|
256
|
+
way to validate them is to collect many real audit cases — which is
|
|
257
|
+
why RoboAudit has an **opt-in** case sharing mechanism.
|
|
258
|
+
|
|
259
|
+
**Nothing is ever sent automatically, and sharing is off by default.**
|
|
260
|
+
An audit run alone creates no shareable file. Only if you explicitly ask
|
|
261
|
+
for one do you get a `share_bundle.json`:
|
|
262
|
+
|
|
263
|
+
```bash
|
|
264
|
+
# from an existing audit result
|
|
265
|
+
roboaudit share path/to/audit_out/result.json
|
|
266
|
+
|
|
267
|
+
# or at audit time
|
|
268
|
+
roboaudit /path/to/dataset --out ./audit_out --share-bundle
|
|
269
|
+
```
|
|
270
|
+
|
|
271
|
+
The bundle is a **summary that omits the fields listed below**, built
|
|
272
|
+
by whitelist extraction from `result.json`. It contains only:
|
|
273
|
+
|
|
274
|
+
- tool and rules versions; dataset format (v2.1 / v3.0)
|
|
275
|
+
- episode, frame, camera and task **counts** (counts only)
|
|
276
|
+
- score and grade; per-check finding counts and severity counts
|
|
277
|
+
- per-check **margin-ratio distributions** (min / median / max and a
|
|
278
|
+
histogram — ratios to thresholds, never raw measurements)
|
|
279
|
+
- disposition totals (exclude / review / keep) and run-status totals
|
|
280
|
+
- whether sampling mode was used
|
|
281
|
+
|
|
282
|
+
It **never** contains: dataset names, file or folder paths, file names,
|
|
283
|
+
task/instruction text (nor hashes of it — a hash can be a tracking
|
|
284
|
+
clue), video frames or images, raw sensor/action values, finding detail
|
|
285
|
+
text or evidence values, camera/feature key names, robot type, user or
|
|
286
|
+
machine information, or the dataset input fingerprint. Each bundle
|
|
287
|
+
states this in its own `about` field, so you can verify before sharing.
|
|
288
|
+
(This was verified mechanically: bundles generated from three real
|
|
289
|
+
audits were grepped for 33 forbidden strings — dataset names, real task
|
|
290
|
+
sentences, paths, key names — with 0 hits.)
|
|
291
|
+
|
|
292
|
+
Each bundle also carries an empty **adjudication template**: one entry
|
|
293
|
+
per flagged episode where you may record `agree`, `false_positive`, or
|
|
294
|
+
`unsure` after inspecting the episode yourself. Human-adjudicated cases
|
|
295
|
+
are the most valuable ones — they are what turns a working
|
|
296
|
+
threshold into a validated one.
|
|
297
|
+
|
|
298
|
+
To submit a case, open a GitHub issue with the **Audit case** template
|
|
299
|
+
and paste the bundle. The template asks you to confirm you checked the
|
|
300
|
+
file first. Submitted bundles are aggregated by the maintainers with
|
|
301
|
+
`tools/collect_cases.py` (included in this repository) into per-check
|
|
302
|
+
finding frequencies, margin distributions, and false-positive tallies.
|
|
303
|
+
|
|
304
|
+
## Score and dispositions
|
|
305
|
+
|
|
306
|
+
The score starts at 100, and **every recorded finding deducts by its
|
|
307
|
+
severity: critical −10, warning −3, info −1**. The score is floored
|
|
308
|
+
at 0. Most findings are recorded per affected episode — one per
|
|
309
|
+
episode for each problem a check finds — so the same defect in many
|
|
310
|
+
episodes deducts many times. A problem that is a property of the
|
|
311
|
+
dataset as a whole (a declared dimension that mismatches in every
|
|
312
|
+
episode, a missing feature declaration) is recorded **once**, not once
|
|
313
|
+
per episode. Because deductions accumulate per finding, the score is
|
|
314
|
+
not normalised by dataset size — compare scores within one dataset or
|
|
315
|
+
across versions of it (see *Limitations*). Grades: A ≥ 90, B ≥ 80,
|
|
316
|
+
C ≥ 70, D ≥ 60, F < 60.
|
|
317
|
+
|
|
318
|
+
Each episode also gets a disposition:
|
|
319
|
+
|
|
320
|
+
- **exclude** — a clear violation (missing file, NaN, structural break, or a
|
|
321
|
+
measurement at least 1.5× past its threshold). Recorded in the
|
|
322
|
+
`excluded_episodes` field of `clean_index.json` and in `result.json`;
|
|
323
|
+
in neither the keep list nor the review list.
|
|
324
|
+
- **review** — borderline: a real finding, but too close to the threshold to
|
|
325
|
+
condemn automatically. Listed in `review_index.json`, and **not** in
|
|
326
|
+
`clean_episodes`. RoboAudit's design rule is *when in doubt, ask a
|
|
327
|
+
human instead of silently dropping data*.
|
|
328
|
+
- **keep** — no finding was recorded for this episode by the checks
|
|
329
|
+
that ran. Listed in `clean_index.json` (`clean_episodes`).
|
|
330
|
+
|
|
331
|
+
Score and disposition are computed independently, so the two do not
|
|
332
|
+
scale together. The score deducts per recorded finding by severity;
|
|
333
|
+
a disposition instead depends on how far past its threshold a
|
|
334
|
+
measurement sits (margin ratio ≥ 1.5× → exclude, below that → review;
|
|
335
|
+
info findings are always review, and findings with no measurable
|
|
336
|
+
margin — a missing file, a NaN — exclude at warning/critical
|
|
337
|
+
severity). Dataset-level findings deduct from the score but do not by
|
|
338
|
+
themselves change any episode's disposition. In the sreetz report
|
|
339
|
+
card, for example, the seven excluded episodes carry warning-level
|
|
340
|
+
jump findings (−3 each): the −56 total behind the 44 is 15 jump
|
|
341
|
+
warnings (−45) plus one dataset-level critical for the missing
|
|
342
|
+
camera (−10) and one info note (−1).
|
|
343
|
+
|
|
344
|
+
Separately, if any check could not cover its full scope (e.g. a camera's
|
|
345
|
+
files are missing, some videos failed to decode), the audit is flagged
|
|
346
|
+
**incomplete** in the report and via exit code 3. A failed check never
|
|
347
|
+
masquerades as a pass.
|
|
348
|
+
|
|
349
|
+
**Incomplete audits still write `clean_index.json`** — read it with the
|
|
350
|
+
flag in mind. There, `clean_episodes` means "no finding was recorded
|
|
351
|
+
by the checks that could run"; it does **not** mean the episode passed
|
|
352
|
+
the checks that could not run. In the sreetz report card above, the
|
|
353
|
+
partial copy we audited was missing one declared camera entirely, so
|
|
354
|
+
no episode was ever checked against it — yet 60 episodes are listed
|
|
355
|
+
as clean. Check
|
|
356
|
+
`incomplete_audit` in `result.json` (or the exit code) before treating
|
|
357
|
+
the list as final.
|
|
358
|
+
|
|
359
|
+
## The 12 checks
|
|
360
|
+
|
|
361
|
+
| Group | Checks |
|
|
362
|
+
|---|---|
|
|
363
|
+
| Structure | S1 metadata & file integrity · S2 dimension/schema consistency · S3 time axis (monotonic timestamps, episode boundaries) |
|
|
364
|
+
| Video | V1 black / frozen frames · V2 shake & blur · V3 episode length anomalies |
|
|
365
|
+
| Action | A1 out-of-range values · A2 abrupt jumps · A3 dead (never-moving) dimensions · A4 duplicate episodes |
|
|
366
|
+
| Instruction | L1 missing or identical task text · L2 no success/failure labels |
|
|
367
|
+
|
|
368
|
+
## Measured results (not marketing numbers)
|
|
369
|
+
|
|
370
|
+
On three real public datasets (audited on a small shared-CPU VM, roughly
|
|
371
|
+
2–5 minutes each):
|
|
372
|
+
|
|
373
|
+
| Dataset | Format | Episodes | Score | Notes |
|
|
374
|
+
|---|---|---|---|---|
|
|
375
|
+
| `lerobot/aloha_sim_transfer_cube_scripted` | v3.0 | 50 | **79 / C** | 2 borderline episodes sent to review |
|
|
376
|
+
| `00ri/so100_battery` | v2.1 | 50 | **84 / B** | 5 borderline episodes sent to review; both cameras audited |
|
|
377
|
+
| `sreetz-nv/so101_teleop_vials_rack_left` (partial copy) | v3.0 | 75 | **44 / F** | Partial copy (166 MB, one of the two declared cameras): audit flagged **incomplete** — that camera's files are absent from this copy; the upstream dataset may differ |
|
|
378
|
+
|
|
379
|
+
On a defect-injection suite built by the RoboAudit authors (43 known defects
|
|
380
|
+
planted across both formats — a home-field test, so read it as such),
|
|
381
|
+
RoboAudit detected **43/43** with **0 false positives** on 40 clean synthetic episodes. The
|
|
382
|
+
comparable open-source tool RDA (Robot Data Audit, PyPI) reached a verdict
|
|
383
|
+
on 22 of the same 43 (15 exclude + 7 unverifiable), recorded 2 more as
|
|
384
|
+
findings only, and the remaining 19 fell outside its check set — it
|
|
385
|
+
implements no checks for action-state jumps, instruction problems,
|
|
386
|
+
episode boundaries, and similar defect types. On real
|
|
387
|
+
data, RDA's conservative all-pass behaviour was the more cautious one;
|
|
388
|
+
RoboAudit's own early exclusions on real data were themselves borderline,
|
|
389
|
+
which is exactly why the `review` disposition now exists.
|
|
390
|
+
|
|
391
|
+
## Limitations — read this before trusting a score
|
|
392
|
+
|
|
393
|
+
- **The score is a relative indicator, not an absolute quality measure.**
|
|
394
|
+
It compares well inside one dataset or across versions of the same
|
|
395
|
+
dataset. A 44/F dataset may simply be a teleoperation recording with
|
|
396
|
+
frequent discontinuities, not "bad data".
|
|
397
|
+
- **No guarantee about training outcomes or robot safety.** RoboAudit
|
|
398
|
+
checks structural and statistical properties of data. It does not prove
|
|
399
|
+
that training on the kept episodes will perform better, and an A2
|
|
400
|
+
"review" verdict explicitly means the tool *cannot* distinguish a
|
|
401
|
+
recording defect from a robot's normal operating characteristics.
|
|
402
|
+
- **Disposition thresholds are working values, not yet validated at
|
|
403
|
+
scale.** The borderline margin (1.5×) was calibrated on synthetic
|
|
404
|
+
injections plus three real datasets (175 episodes). Sensitivity was
|
|
405
|
+
measured on those datasets, not assumed: varying the A1 boundary
|
|
406
|
+
constant (0.15–0.35) changed no disposition at all, and the
|
|
407
|
+
injected-defect dispositions were identical at margins 1.5 and 1.7 —
|
|
408
|
+
but borderline episodes do move with the margin (at 1.3, two battery
|
|
409
|
+
episodes would flip from review to exclude and sreetz exclusions
|
|
410
|
+
would rise from 7 to 11; at 1.7, sreetz exclusions would fall to 4).
|
|
411
|
+
Treat the constants as working values, not constants of nature.
|
|
412
|
+
- **Sampling mode is approximate.** Boundary-case frozen segments can be
|
|
413
|
+
missed; use a full audit for a final verdict.
|
|
414
|
+
- One representative set of statistics is computed per declared camera;
|
|
415
|
+
every declared camera *is* decoded and checked, but cross-camera
|
|
416
|
+
synchronisation is not yet measured.
|
|
417
|
+
|
|
418
|
+
## Troubleshooting
|
|
419
|
+
|
|
420
|
+
- **Exit code 2 — the error says `meta/info.json` is missing.** You
|
|
421
|
+
pointed the command at the wrong folder. Pass the dataset root (the
|
|
422
|
+
folder containing `meta/info.json`), not a parent or child folder,
|
|
423
|
+
and make sure the dataset is downloaded locally first — RoboAudit
|
|
424
|
+
reads local folders only.
|
|
425
|
+
- **Exit code 2 — `--out` refused.** The output folder must be outside
|
|
426
|
+
the dataset folder. This is deliberate: it makes it impossible to
|
|
427
|
+
write audit output into the original data.
|
|
428
|
+
- **Exit code 3 — incomplete audit.** One or more checks ran partially
|
|
429
|
+
or errored; `report.html` and `result.json` say which and why. The
|
|
430
|
+
usual causes are missing files or videos that failed to decode. All
|
|
431
|
+
four outputs are still written, but read the incomplete-audit note
|
|
432
|
+
in [Score and dispositions](#score-and-dispositions) before using
|
|
433
|
+
the indexes.
|
|
434
|
+
- **Videos fail to decode.** RoboAudit decodes with OpenCV first and
|
|
435
|
+
falls back to a system `ffmpeg` binary when one is on your `PATH`
|
|
436
|
+
(needed for codecs OpenCV cannot read, e.g. AV1). If many episodes
|
|
437
|
+
fail to decode, install ffmpeg (e.g. `apt install ffmpeg` or
|
|
438
|
+
`brew install ffmpeg`) and re-run. The per-camera decode statistics
|
|
439
|
+
in the report show how many episodes each path read.
|
|
440
|
+
|
|
441
|
+
## License
|
|
442
|
+
|
|
443
|
+
MIT — see [LICENSE](LICENSE).
|
|
444
|
+
|
|
445
|
+
---
|
|
446
|
+
|
|
447
|
+
## 한국어 요약
|
|
448
|
+
|
|
449
|
+
**학습시키기 전에, 감사부터.** RoboAudit는 LeRobot 로봇 데이터셋의
|
|
450
|
+
건강검진 도구입니다. 데이터셋 폴더를 넣으면 100점 만점 점수와 등급,
|
|
451
|
+
결함 목록, 그리고 에피소드별 처분(제외/보류/유지)을 돌려줍니다.
|
|
452
|
+
|
|
453
|
+
핵심 3가지:
|
|
454
|
+
|
|
455
|
+
1. **심어 둔 고장 43개 전부 검출** — 비슷한 공개 감사 도구는 판정까지
|
|
456
|
+
간 것이 22개였습니다. (단, 시험장은 RoboAudit 제작진이 만든 것이라
|
|
457
|
+
홈그라운드 시험으로 읽어 주세요. 근거는 위 영문 비교 섹션.)
|
|
458
|
+
2. **구 포맷 v2.1도 변환 없이 직접 감사** — LeRobot 0.7.0 기준 공식
|
|
459
|
+
로더가 로딩을 거부하는(BackwardCompatibilityError) 형식입니다.
|
|
460
|
+
3. **점수만이 아니라** 제외/보류/유지 에피소드 목록까지 나와 학습에
|
|
461
|
+
쓸 에피소드를 고를 수 있습니다.
|
|
462
|
+
|
|
463
|
+
실제 공개 데이터셋 성적표 (전부 실측):
|
|
464
|
+
|
|
465
|
+
| 데이터셋 | 형식 | 점수 | 제외/보류 | 한 줄 소견 |
|
|
466
|
+
|---|---|---|---|---|
|
|
467
|
+
| `lerobot/aloha_sim_transfer_cube_scripted` | v3.0 | **79 / C** | 0 / 2 | 범위 경계를 살짝 넘은 2건은 자동 제외가 아니라 사람 확인(보류)으로 |
|
|
468
|
+
| `00ri/so100_battery` | v2.1 | **84 / B** | 0 / 5 | 흔들림으로 잡힌 5건이 실제로는 동작 많은 에피소드로 보여 전부 보류; 카메라 2대 모두 감사 |
|
|
469
|
+
| `sreetz-nv/so101_teleop_vials_rack_left` (부분 복사본) | v3.0 | **44 / F** | 7 / 8 | 부분 복사본(166MB, 선언된 카메라 2대 중 1대만 다운로드)으로 감사 — 없는 카메라는 이 복사본의 속성이고 원본 데이터셋은 완전할 수 있음; "불완전한 감사"로 명시; 점프 15건은 측정 강도로 제외/보류 분리 |
|
|
470
|
+
|
|
471
|
+
- v2.1·v3.0 형식을 자체 파서로 직접 읽습니다 (LeRobot 0.7.0 기준 공식
|
|
472
|
+
로더가 거부하는 v2.1도 변환 없이 읽음).
|
|
473
|
+
- 검사는 규칙+임계값 12종뿐이라 감점 이유를 전부 설명할 수 있고, 원본
|
|
474
|
+
데이터는 읽기만 하며 절대 수정하지 않습니다.
|
|
475
|
+
- 애매한 에피소드는 자동으로 버리지 않고 `review_index.json`으로 분리해
|
|
476
|
+
사람이 확인하게 합니다. 검사가 일부라도 실패하면 "불완전한 감사"라고
|
|
477
|
+
성적표와 종료 코드(3)로 명시합니다.
|
|
478
|
+
- 종료 코드 0은 "감사가 전체 범위까지 끝까지 돌았다"는 뜻일 뿐,
|
|
479
|
+
데이터셋이 "합격"했다는 뜻이 아닙니다. 불완전한 감사에서도
|
|
480
|
+
`clean_index.json`은 나오지만, 그 유지 목록은 "돌릴 수 있었던
|
|
481
|
+
검사에서 발견이 없었다"는 뜻으로 읽어야 합니다.
|
|
482
|
+
- 제외된 에피소드는 유지 목록(`clean_episodes`)에도 보류 목록에도
|
|
483
|
+
들어가지 않고, `clean_index.json`의 `excluded_episodes` 항목과
|
|
484
|
+
`result.json`에 걸린 검사 코드와 함께 기록됩니다. 보류 에피소드도
|
|
485
|
+
유지 목록에는 들어가지 않습니다.
|
|
486
|
+
- 라이선스는 MIT입니다.
|
|
487
|
+
- 점수는 절대 품질이 아니라 상대 지표입니다. 학습 성능이나 로봇 안전성을
|
|
488
|
+
보증하지 않으며, 처분 기준값(경계 1.5배)은 아직 잠정값입니다.
|
|
489
|
+
|
|
490
|
+
설치와 사용법은 위 영문 섹션을 따르세요: 저장소를 clone하고
|
|
491
|
+
`pip install .` 한 뒤 `roboaudit <데이터셋 폴더> --out <출력 폴더>`.
|
|
492
|
+
|
|
493
|
+
**사례 공유 (선택 참여, 기본 꺼짐)**: 처분 기준값(경계 1.5배 등)은 아직
|
|
494
|
+
잠정값이라, 여러 실제 데이터셋의 감사 사례가 쌓여야 검증할 수 있습니다.
|
|
495
|
+
`roboaudit share <result.json>` 또는 감사 시 `--share-bundle`을 명시한
|
|
496
|
+
경우에만 요약 파일(`share_bundle.json`)이 만들어집니다. 자동
|
|
497
|
+
전송은 절대 없으며, 제출은 본인이 GitHub 이슈에 직접 붙이는 수동
|
|
498
|
+
행동입니다. 요약에는 버전·형식·개수·점수·검사별 발견 건수와 margin
|
|
499
|
+
분포·처분 집계만 들어가고, 데이터셋 이름·경로·파일명·과제 원문(해시
|
|
500
|
+
포함)·원본 측정값·카메라 키 이름은 화이트리스트 방식이라 구조적으로
|
|
501
|
+
들어갈 수 없습니다. bundle 안의 빈 판정 칸에 직접 확인한 에피소드의
|
|
502
|
+
오탐 여부를 적어 제출하면, 사람이 판정한 사례가 쌓여 기준값 검증의
|
|
503
|
+
재료가 됩니다.
|