roboaudit 0.2.2__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 RoboAudit contributors
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
@@ -0,0 +1,503 @@
1
+ Metadata-Version: 2.4
2
+ Name: roboaudit
3
+ Version: 0.2.2
4
+ Summary: Health-check CLI for LeRobot robot datasets: 100-point score, per-episode dispositions (exclude / review / keep), and an auditable report.
5
+ License: MIT
6
+ Keywords: lerobot,robotics,dataset,data-quality,audit
7
+ Classifier: Programming Language :: Python :: 3
8
+ Classifier: License :: OSI Approved :: MIT License
9
+ Classifier: Operating System :: OS Independent
10
+ Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
11
+ Requires-Python: >=3.10
12
+ Description-Content-Type: text/markdown
13
+ License-File: LICENSE
14
+ Requires-Dist: numpy>=1.24
15
+ Requires-Dist: opencv-python-headless>=4.8
16
+ Requires-Dist: pyarrow>=12
17
+ Requires-Dist: jinja2>=3.0
18
+ Dynamic: license-file
19
+
20
+ # RoboAudit
21
+
22
+ **Before you train on it, audit it.** RoboAudit is a health-check CLI for
23
+ [LeRobot](https://github.com/huggingface/lerobot) robot datasets: give it a
24
+ dataset folder and it returns a **100-point score with a grade**, a list of
25
+ what is wrong and where, and episode indexes that sort episodes into
26
+ keep / review / exclude for training-data selection.
27
+
28
+ ![RoboAudit pipeline: a LeRobot dataset (v2.1 or v3.0) goes through 12 rule-based checks in four groups — structure, video, action, instruction — and comes out as a 100-point score with a grade, top-3 deduction factors, and four output files: report.html, result.json, clean_index.json, and review_index.json, plus an optional opt-in share bundle.](docs/img/flow.svg)
29
+
30
+ > **Why it exists**
31
+ >
32
+ > 1. **Planted-defect test: 43/43 detected** — the comparable open-source
33
+ > auditor caught 22 at verdict level.\*
34
+ > 2. **Audits legacy v2.1 datasets directly, without conversion** — with
35
+ > LeRobot 0.7.0, the official loader refuses v2.1 datasets
36
+ > (BackwardCompatibilityError); RoboAudit's own parser reads them.
37
+ > 3. **Not just a score:** exclude / review / keep episode indexes for
38
+ > selecting which episodes to train on.
39
+ >
40
+ > \* The defect suite was built by the RoboAudit authors — a home-field
41
+ > test, so read it as one. Full comparison in
42
+ > [How it compares](#how-it-compares) below.
43
+
44
+ ## Quick start
45
+
46
+ ```bash
47
+ git clone https://github.com/parkjb0826/roboaudit
48
+ cd roboaudit
49
+ python3 -m venv .venv && source .venv/bin/activate # recommended
50
+ pip install .
51
+ roboaudit /path/to/lerobot_dataset --out ./audit_out
52
+ ```
53
+
54
+ > ⚠️ `--out` must point **outside** the dataset folder. If it points
55
+ > inside, the run is refused (exit code 2) — this protects the original
56
+ > data from ever being written into.
57
+
58
+ That writes `report.html` (the report card), `result.json`,
59
+ `clean_index.json`, and `review_index.json` into `./audit_out`. No robot,
60
+ no GPU — CPU only. Getting to the first command takes about a minute;
61
+ the audit itself takes minutes, not seconds — measured on this
62
+ project's (small, shared) VM CPU: 152–180 s for 50–75-episode datasets
63
+ with one camera decoded, and 296 s for a 50-episode dataset with two
64
+ cameras decoded. Details (Python version, optional ffmpeg, sampling
65
+ mode, exit codes) are in [Install](#install) and [Usage](#usage) below.
66
+
67
+ ## Report cards from real datasets
68
+
69
+ ![Report cards for three real public datasets: aloha transfer cube (sim) scores 79 out of 100, grade C, with 0 episodes excluded and 2 sent to review; so100_battery scores 84, grade B, with 0 excluded and 5 in review; so101_teleop_vials (partial copy) scores 44, grade F, flagged as an incomplete audit, with 7 excluded and 8 in review.](docs/img/report_cards.svg)
70
+
71
+ Real audits of three public datasets — not demos, not synthetic data
72
+ (full method and caveats in [Measured results](#measured-results-not-marketing-numbers)).
73
+ One caveat up front: these three datasets were also used to calibrate
74
+ the disposition thresholds, so read their grades as in-sample results:
75
+
76
+ | Dataset | Format | Episodes | Score | Exclude / Review | One-line verdict |
77
+ |---|---|---|---|---|---|
78
+ | `lerobot/aloha_sim_transfer_cube_scripted` | v3.0 | 50 | **79 / C** | 0 / 2 | Two episodes sit just past a range-band edge (measured at 0.28–0.44 of the exclusion margin) — sent to a human, not dropped |
79
+ | `00ri/so100_battery` | v2.1 | 50 | **84 / B** | 0 / 5 | Five "shake" flags that look like busy episodes rather than camera shake — all review, none auto-dropped; both declared cameras audited |
80
+ | `sreetz-nv/so101_teleop_vials_rack_left` (partial copy) | v3.0 | 75 | **44 / F** | 7 / 8 | Audited from a **partial copy** (a 166 MB download covering only one of the two declared cameras): the missing camera is a property of this copy — the upstream dataset may be complete. The audit is flagged **incomplete** instead of passing silently; 15 jump findings split by measured strength |
81
+
82
+ ## How it compares
83
+
84
+ ![Bar chart of the planted-defect test with 43 known defects: RoboAudit reached a verdict on all 43; RDA 0.12.4 reached a verdict on 22 (15 exclude plus 7 unverifiable), recorded 2 more as findings only, and the remaining 19 fell outside its check set. The defect suite was built by the RoboAudit authors — a home-field test.](docs/img/detection_compare.svg)
85
+
86
+ Head-to-head with RDA (Robot Data Audit, the comparable open-source
87
+ auditor on PyPI), run on the same data in its full mode (`--full`, every
88
+ check enabled) but without a platform profile:
89
+
90
+ | | RoboAudit | RDA 0.12.4 |
91
+ |---|---|---|
92
+ | Planted defects (43) reaching a verdict | **43** | 22 (15 exclude + 7 correctly pinpointed as unverifiable) |
93
+ | Recorded as a finding only (no verdict) | 0 | 2 |
94
+ | Outside the tool's check set (no verdict, no finding) | **0** | 19 — check types RDA does not implement (state jumps, range violations, dead dimensions, duplicates, instruction problems, episode boundaries) |
95
+ | False-positive verdicts on clean data (40 episodes) | 0 | 0 |
96
+ | Behaviour on the 3 real datasets above | Borderline findings → `review` | All-pass on the two complete datasets; refused a verdict on the partial copy |
97
+
98
+ Read this fairly: the defect suite is RoboAudit's home field, RDA was run
99
+ without a platform profile (its joint-limit check needs one), and RDA has
100
+ its own separate blind test. On real data RDA is the more conservative
101
+ design — it records risk signals at metric level without letting them
102
+ change verdicts. RoboAudit's difference is breadth of checks that *do*
103
+ reach a verdict, plus dispositions a human can overrule.
104
+
105
+ ## Demo
106
+
107
+ A fixed-data demo lives in [`demo/`](demo/): pick one of the three
108
+ pre-audited datasets above and browse its stored score summary and full
109
+ HTML report. The demo files are prepared and verified locally; the demo
110
+ itself is not deployed yet. Arbitrary uploads are deliberately not
111
+ supported — auditing is CPU-heavy (minutes per dataset), so a public
112
+ demo needs a queue and abuse controls first.
113
+
114
+ ---
115
+
116
+ The idea is simple: robot training is expensive, and a dataset with missing
117
+ videos, NaN actions, reversed timestamps, or broken instructions wastes that
118
+ money silently. RoboAudit is the cheap inspection you run *before* training.
119
+
120
+ - Reads **LeRobot v2.1 and v3.0** with its own parser — no conversion step.
121
+ v2.1 is read directly, even though with LeRobot 0.7.0 the official
122
+ loader refuses v2.1 datasets (BackwardCompatibilityError).
123
+ - **12 rule/threshold checks** (structure, video, action, instruction).
124
+ No AI inside the checks: every deduction traces back to a rule and a
125
+ measured value, so a score can always be explained.
126
+ - **Original data is read-only.** RoboAudit never modifies the dataset
127
+ (verified by SHA-256 before/after in its own test suite).
128
+ - No robot, no GPU. CPU only.
129
+
130
+ ## Install
131
+
132
+ ```bash
133
+ git clone https://github.com/parkjb0826/roboaudit
134
+ cd roboaudit
135
+ python3 -m venv .venv && source .venv/bin/activate # recommended
136
+ pip install .
137
+ ```
138
+
139
+ (Not yet on PyPI — install from the cloned repository for now.)
140
+
141
+ Requires Python ≥ 3.10 (developed and verified on Linux with
142
+ Python 3.12; other platforms are untested).
143
+ Runtime dependencies: `numpy`, `opencv-python-headless`, `pyarrow`, `jinja2`.
144
+
145
+ Optional: if a system `ffmpeg` binary is on your `PATH`, it is used as a
146
+ fallback decoder for video codecs OpenCV cannot read (e.g. AV1).
147
+
148
+ ## Usage
149
+
150
+ Basic audit — writes `report.html`, `result.json`, `clean_index.json`, and
151
+ `review_index.json` into the output folder:
152
+
153
+ ```bash
154
+ roboaudit /path/to/lerobot_dataset --out ./audit_out
155
+ ```
156
+
157
+ The argument is the dataset **root folder on local disk** — RoboAudit
158
+ does not download anything, so fetch the dataset first (e.g. from the
159
+ Hugging Face Hub). The root is the folder that contains `meta/info.json`:
160
+
161
+ ```
162
+ my_dataset/ ← pass this folder
163
+ ├── meta/
164
+ │ ├── info.json
165
+ │ ├── episodes/chunk-000/file-000.parquet
166
+ │ └── tasks.parquet
167
+ ├── data/chunk-000/file-000.parquet (shared across episodes)
168
+ └── videos/observation.images.top/chunk-000/file-000.mp4
169
+ ```
170
+
171
+ That is the v3.0 layout. The v2.1 layout differs — per-episode files
172
+ (`data/chunk-000/episode_000000.parquet`,
173
+ `videos/chunk-000/<camera_key>/episode_000000.mp4`, `meta/episodes.jsonl`)
174
+ — and is read directly as well. If `meta/info.json` is missing, the
175
+ command stops with exit code 2 (see [Troubleshooting](#troubleshooting)).
176
+
177
+ Sampling mode for very large datasets — video statistics are computed from
178
+ an evenly spaced sample of at most N frames per episode instead of every
179
+ frame. Faster, but approximate (see *Limitations*):
180
+
181
+ ```bash
182
+ roboaudit /path/to/big_dataset --out ./audit_out --sample-frames 60
183
+ ```
184
+
185
+ Exit codes are part of the interface, so the tool can gate a pipeline:
186
+
187
+ | Code | Meaning |
188
+ |---|---|
189
+ | 0 | Audit complete — every check covered its full scope (regardless of how many defects were found) |
190
+ | 1 | Unexpected runtime failure |
191
+ | 2 | Input error — not a LeRobot dataset folder, or `--out` points inside the dataset folder (refused, to protect the original data) |
192
+ | 3 | Audit finished but **incomplete** — some check ran partially or errored; the report says which and why |
193
+
194
+ **Exit code 0 means the audit ran to completion with full coverage. It
195
+ does NOT mean the dataset passed** — a complete audit can still find
196
+ critical defects and score an F. If you gate a pipeline on RoboAudit,
197
+ check the score and findings in `result.json` too, and treat exit
198
+ code 3 (incomplete) as "the indexes are not final" (see
199
+ [Score and dispositions](#score-and-dispositions)).
200
+
201
+ ## Outputs
202
+
203
+ | File | Contents |
204
+ |---|---|
205
+ | `report.html` | Human-readable report card: score and grade, top-3 deduction factors, per-check run status and findings, per-camera decode statistics, scope notices |
206
+ | `result.json` | Machine-readable full result, including a **run manifest** (tool/rules versions, thresholds actually applied, input fingerprint, decode backends) to help reproduce runs and compare their settings and results |
207
+ | `clean_index.json` | The episode index file: `clean_episodes` (episodes assigned `keep`), plus the `excluded_episodes` and `review_episodes` lists with the checks that flagged each one |
208
+ | `review_index.json` | Borderline episodes a human should look at before deciding — these are **not** in `clean_episodes` (see below) |
209
+
210
+ Episode indices everywhere are the dataset's own `episode_index`
211
+ values from its metadata. The three lists are disjoint and together
212
+ cover every episode: an episode appears in exactly one of
213
+ `clean_episodes`, `review_episodes`, or `excluded_episodes`. Excluded
214
+ episodes appear in neither the keep list nor the review list — they
215
+ are recorded, with the checks that flagged them, in the
216
+ `excluded_episodes` field of `clean_index.json` and in `result.json`.
217
+
218
+ ### Using the indexes in training code
219
+
220
+ ```python
221
+ import json
222
+
223
+ index = json.load(open("audit_out/clean_index.json"))
224
+ result = json.load(open("audit_out/result.json"))
225
+
226
+ # 1. If the audit was incomplete, the indexes are not final — a check
227
+ # that could not run may have missed defects (see report.html):
228
+ if result["incomplete_audit"]:
229
+ raise SystemExit("Incomplete audit: indexes are not final")
230
+
231
+ keep = set(index["clean_episodes"]) # episode_index values with no finding
232
+ review = {e["episode_index"] for e in index["review_episodes"]}
233
+
234
+ # 2. Decide on the review episodes explicitly — do not drop them by
235
+ # accident. Inspect them first, then name the ones you accept:
236
+ approved_after_review = set() # e.g. {3, 17}, once you have looked
237
+
238
+ selected = sorted(keep | approved_after_review)
239
+
240
+ # 3. Pass the selection to your training set — with LeRobot, the
241
+ # `episodes=` argument of LeRobotDataset takes exactly this list:
242
+ # ds = LeRobotDataset("you/dataset", episodes=selected)
243
+ train_episodes = [ep for ep in my_episode_ids if ep in set(selected)]
244
+ ```
245
+
246
+ `review_episodes` and `excluded_episodes` in the same file are lists of
247
+ `{"episode_index": …, "checks": […]}` objects, so you can see *why*
248
+ each episode was held back before deciding what to do with it.
249
+
250
+ ## Share an audit case (opt-in)
251
+
252
+ RoboAudit's disposition thresholds (such as the 1.5× borderline margin)
253
+ are working values, not final constants: they were calibrated on
254
+ synthetic injections plus three real datasets and have not yet
255
+ been validated against many adjudicated real-world cases. The honest
256
+ way to validate them is to collect many real audit cases — which is
257
+ why RoboAudit has an **opt-in** case sharing mechanism.
258
+
259
+ **Nothing is ever sent automatically, and sharing is off by default.**
260
+ An audit run alone creates no shareable file. Only if you explicitly ask
261
+ for one do you get a `share_bundle.json`:
262
+
263
+ ```bash
264
+ # from an existing audit result
265
+ roboaudit share path/to/audit_out/result.json
266
+
267
+ # or at audit time
268
+ roboaudit /path/to/dataset --out ./audit_out --share-bundle
269
+ ```
270
+
271
+ The bundle is a **summary that omits the fields listed below**, built
272
+ by whitelist extraction from `result.json`. It contains only:
273
+
274
+ - tool and rules versions; dataset format (v2.1 / v3.0)
275
+ - episode, frame, camera and task **counts** (counts only)
276
+ - score and grade; per-check finding counts and severity counts
277
+ - per-check **margin-ratio distributions** (min / median / max and a
278
+ histogram — ratios to thresholds, never raw measurements)
279
+ - disposition totals (exclude / review / keep) and run-status totals
280
+ - whether sampling mode was used
281
+
282
+ It **never** contains: dataset names, file or folder paths, file names,
283
+ task/instruction text (nor hashes of it — a hash can be a tracking
284
+ clue), video frames or images, raw sensor/action values, finding detail
285
+ text or evidence values, camera/feature key names, robot type, user or
286
+ machine information, or the dataset input fingerprint. Each bundle
287
+ states this in its own `about` field, so you can verify before sharing.
288
+ (This was verified mechanically: bundles generated from three real
289
+ audits were grepped for 33 forbidden strings — dataset names, real task
290
+ sentences, paths, key names — with 0 hits.)
291
+
292
+ Each bundle also carries an empty **adjudication template**: one entry
293
+ per flagged episode where you may record `agree`, `false_positive`, or
294
+ `unsure` after inspecting the episode yourself. Human-adjudicated cases
295
+ are the most valuable ones — they are what turns a working
296
+ threshold into a validated one.
297
+
298
+ To submit a case, open a GitHub issue with the **Audit case** template
299
+ and paste the bundle. The template asks you to confirm you checked the
300
+ file first. Submitted bundles are aggregated by the maintainers with
301
+ `tools/collect_cases.py` (included in this repository) into per-check
302
+ finding frequencies, margin distributions, and false-positive tallies.
303
+
304
+ ## Score and dispositions
305
+
306
+ The score starts at 100, and **every recorded finding deducts by its
307
+ severity: critical −10, warning −3, info −1**. The score is floored
308
+ at 0. Most findings are recorded per affected episode — one per
309
+ episode for each problem a check finds — so the same defect in many
310
+ episodes deducts many times. A problem that is a property of the
311
+ dataset as a whole (a declared dimension that mismatches in every
312
+ episode, a missing feature declaration) is recorded **once**, not once
313
+ per episode. Because deductions accumulate per finding, the score is
314
+ not normalised by dataset size — compare scores within one dataset or
315
+ across versions of it (see *Limitations*). Grades: A ≥ 90, B ≥ 80,
316
+ C ≥ 70, D ≥ 60, F < 60.
317
+
318
+ Each episode also gets a disposition:
319
+
320
+ - **exclude** — a clear violation (missing file, NaN, structural break, or a
321
+ measurement at least 1.5× past its threshold). Recorded in the
322
+ `excluded_episodes` field of `clean_index.json` and in `result.json`;
323
+ in neither the keep list nor the review list.
324
+ - **review** — borderline: a real finding, but too close to the threshold to
325
+ condemn automatically. Listed in `review_index.json`, and **not** in
326
+ `clean_episodes`. RoboAudit's design rule is *when in doubt, ask a
327
+ human instead of silently dropping data*.
328
+ - **keep** — no finding was recorded for this episode by the checks
329
+ that ran. Listed in `clean_index.json` (`clean_episodes`).
330
+
331
+ Score and disposition are computed independently, so the two do not
332
+ scale together. The score deducts per recorded finding by severity;
333
+ a disposition instead depends on how far past its threshold a
334
+ measurement sits (margin ratio ≥ 1.5× → exclude, below that → review;
335
+ info findings are always review, and findings with no measurable
336
+ margin — a missing file, a NaN — exclude at warning/critical
337
+ severity). Dataset-level findings deduct from the score but do not by
338
+ themselves change any episode's disposition. In the sreetz report
339
+ card, for example, the seven excluded episodes carry warning-level
340
+ jump findings (−3 each): the −56 total behind the 44 is 15 jump
341
+ warnings (−45) plus one dataset-level critical for the missing
342
+ camera (−10) and one info note (−1).
343
+
344
+ Separately, if any check could not cover its full scope (e.g. a camera's
345
+ files are missing, some videos failed to decode), the audit is flagged
346
+ **incomplete** in the report and via exit code 3. A failed check never
347
+ masquerades as a pass.
348
+
349
+ **Incomplete audits still write `clean_index.json`** — read it with the
350
+ flag in mind. There, `clean_episodes` means "no finding was recorded
351
+ by the checks that could run"; it does **not** mean the episode passed
352
+ the checks that could not run. In the sreetz report card above, the
353
+ partial copy we audited was missing one declared camera entirely, so
354
+ no episode was ever checked against it — yet 60 episodes are listed
355
+ as clean. Check
356
+ `incomplete_audit` in `result.json` (or the exit code) before treating
357
+ the list as final.
358
+
359
+ ## The 12 checks
360
+
361
+ | Group | Checks |
362
+ |---|---|
363
+ | Structure | S1 metadata & file integrity · S2 dimension/schema consistency · S3 time axis (monotonic timestamps, episode boundaries) |
364
+ | Video | V1 black / frozen frames · V2 shake & blur · V3 episode length anomalies |
365
+ | Action | A1 out-of-range values · A2 abrupt jumps · A3 dead (never-moving) dimensions · A4 duplicate episodes |
366
+ | Instruction | L1 missing or identical task text · L2 no success/failure labels |
367
+
368
+ ## Measured results (not marketing numbers)
369
+
370
+ On three real public datasets (audited on a small shared-CPU VM, roughly
371
+ 2–5 minutes each):
372
+
373
+ | Dataset | Format | Episodes | Score | Notes |
374
+ |---|---|---|---|---|
375
+ | `lerobot/aloha_sim_transfer_cube_scripted` | v3.0 | 50 | **79 / C** | 2 borderline episodes sent to review |
376
+ | `00ri/so100_battery` | v2.1 | 50 | **84 / B** | 5 borderline episodes sent to review; both cameras audited |
377
+ | `sreetz-nv/so101_teleop_vials_rack_left` (partial copy) | v3.0 | 75 | **44 / F** | Partial copy (166 MB, one of the two declared cameras): audit flagged **incomplete** — that camera's files are absent from this copy; the upstream dataset may differ |
378
+
379
+ On a defect-injection suite built by the RoboAudit authors (43 known defects
380
+ planted across both formats — a home-field test, so read it as such),
381
+ RoboAudit detected **43/43** with **0 false positives** on 40 clean synthetic episodes. The
382
+ comparable open-source tool RDA (Robot Data Audit, PyPI) reached a verdict
383
+ on 22 of the same 43 (15 exclude + 7 unverifiable), recorded 2 more as
384
+ findings only, and the remaining 19 fell outside its check set — it
385
+ implements no checks for action-state jumps, instruction problems,
386
+ episode boundaries, and similar defect types. On real
387
+ data, RDA's conservative all-pass behaviour was the more cautious one;
388
+ RoboAudit's own early exclusions on real data were themselves borderline,
389
+ which is exactly why the `review` disposition now exists.
390
+
391
+ ## Limitations — read this before trusting a score
392
+
393
+ - **The score is a relative indicator, not an absolute quality measure.**
394
+ It compares well inside one dataset or across versions of the same
395
+ dataset. A 44/F dataset may simply be a teleoperation recording with
396
+ frequent discontinuities, not "bad data".
397
+ - **No guarantee about training outcomes or robot safety.** RoboAudit
398
+ checks structural and statistical properties of data. It does not prove
399
+ that training on the kept episodes will perform better, and an A2
400
+ "review" verdict explicitly means the tool *cannot* distinguish a
401
+ recording defect from a robot's normal operating characteristics.
402
+ - **Disposition thresholds are working values, not yet validated at
403
+ scale.** The borderline margin (1.5×) was calibrated on synthetic
404
+ injections plus three real datasets (175 episodes). Sensitivity was
405
+ measured on those datasets, not assumed: varying the A1 boundary
406
+ constant (0.15–0.35) changed no disposition at all, and the
407
+ injected-defect dispositions were identical at margins 1.5 and 1.7 —
408
+ but borderline episodes do move with the margin (at 1.3, two battery
409
+ episodes would flip from review to exclude and sreetz exclusions
410
+ would rise from 7 to 11; at 1.7, sreetz exclusions would fall to 4).
411
+ Treat the constants as working values, not constants of nature.
412
+ - **Sampling mode is approximate.** Boundary-case frozen segments can be
413
+ missed; use a full audit for a final verdict.
414
+ - One representative set of statistics is computed per declared camera;
415
+ every declared camera *is* decoded and checked, but cross-camera
416
+ synchronisation is not yet measured.
417
+
418
+ ## Troubleshooting
419
+
420
+ - **Exit code 2 — the error says `meta/info.json` is missing.** You
421
+ pointed the command at the wrong folder. Pass the dataset root (the
422
+ folder containing `meta/info.json`), not a parent or child folder,
423
+ and make sure the dataset is downloaded locally first — RoboAudit
424
+ reads local folders only.
425
+ - **Exit code 2 — `--out` refused.** The output folder must be outside
426
+ the dataset folder. This is deliberate: it makes it impossible to
427
+ write audit output into the original data.
428
+ - **Exit code 3 — incomplete audit.** One or more checks ran partially
429
+ or errored; `report.html` and `result.json` say which and why. The
430
+ usual causes are missing files or videos that failed to decode. All
431
+ four outputs are still written, but read the incomplete-audit note
432
+ in [Score and dispositions](#score-and-dispositions) before using
433
+ the indexes.
434
+ - **Videos fail to decode.** RoboAudit decodes with OpenCV first and
435
+ falls back to a system `ffmpeg` binary when one is on your `PATH`
436
+ (needed for codecs OpenCV cannot read, e.g. AV1). If many episodes
437
+ fail to decode, install ffmpeg (e.g. `apt install ffmpeg` or
438
+ `brew install ffmpeg`) and re-run. The per-camera decode statistics
439
+ in the report show how many episodes each path read.
440
+
441
+ ## License
442
+
443
+ MIT — see [LICENSE](LICENSE).
444
+
445
+ ---
446
+
447
+ ## 한국어 요약
448
+
449
+ **학습시키기 전에, 감사부터.** RoboAudit는 LeRobot 로봇 데이터셋의
450
+ 건강검진 도구입니다. 데이터셋 폴더를 넣으면 100점 만점 점수와 등급,
451
+ 결함 목록, 그리고 에피소드별 처분(제외/보류/유지)을 돌려줍니다.
452
+
453
+ 핵심 3가지:
454
+
455
+ 1. **심어 둔 고장 43개 전부 검출** — 비슷한 공개 감사 도구는 판정까지
456
+ 간 것이 22개였습니다. (단, 시험장은 RoboAudit 제작진이 만든 것이라
457
+ 홈그라운드 시험으로 읽어 주세요. 근거는 위 영문 비교 섹션.)
458
+ 2. **구 포맷 v2.1도 변환 없이 직접 감사** — LeRobot 0.7.0 기준 공식
459
+ 로더가 로딩을 거부하는(BackwardCompatibilityError) 형식입니다.
460
+ 3. **점수만이 아니라** 제외/보류/유지 에피소드 목록까지 나와 학습에
461
+ 쓸 에피소드를 고를 수 있습니다.
462
+
463
+ 실제 공개 데이터셋 성적표 (전부 실측):
464
+
465
+ | 데이터셋 | 형식 | 점수 | 제외/보류 | 한 줄 소견 |
466
+ |---|---|---|---|---|
467
+ | `lerobot/aloha_sim_transfer_cube_scripted` | v3.0 | **79 / C** | 0 / 2 | 범위 경계를 살짝 넘은 2건은 자동 제외가 아니라 사람 확인(보류)으로 |
468
+ | `00ri/so100_battery` | v2.1 | **84 / B** | 0 / 5 | 흔들림으로 잡힌 5건이 실제로는 동작 많은 에피소드로 보여 전부 보류; 카메라 2대 모두 감사 |
469
+ | `sreetz-nv/so101_teleop_vials_rack_left` (부분 복사본) | v3.0 | **44 / F** | 7 / 8 | 부분 복사본(166MB, 선언된 카메라 2대 중 1대만 다운로드)으로 감사 — 없는 카메라는 이 복사본의 속성이고 원본 데이터셋은 완전할 수 있음; "불완전한 감사"로 명시; 점프 15건은 측정 강도로 제외/보류 분리 |
470
+
471
+ - v2.1·v3.0 형식을 자체 파서로 직접 읽습니다 (LeRobot 0.7.0 기준 공식
472
+ 로더가 거부하는 v2.1도 변환 없이 읽음).
473
+ - 검사는 규칙+임계값 12종뿐이라 감점 이유를 전부 설명할 수 있고, 원본
474
+ 데이터는 읽기만 하며 절대 수정하지 않습니다.
475
+ - 애매한 에피소드는 자동으로 버리지 않고 `review_index.json`으로 분리해
476
+ 사람이 확인하게 합니다. 검사가 일부라도 실패하면 "불완전한 감사"라고
477
+ 성적표와 종료 코드(3)로 명시합니다.
478
+ - 종료 코드 0은 "감사가 전체 범위까지 끝까지 돌았다"는 뜻일 뿐,
479
+ 데이터셋이 "합격"했다는 뜻이 아닙니다. 불완전한 감사에서도
480
+ `clean_index.json`은 나오지만, 그 유지 목록은 "돌릴 수 있었던
481
+ 검사에서 발견이 없었다"는 뜻으로 읽어야 합니다.
482
+ - 제외된 에피소드는 유지 목록(`clean_episodes`)에도 보류 목록에도
483
+ 들어가지 않고, `clean_index.json`의 `excluded_episodes` 항목과
484
+ `result.json`에 걸린 검사 코드와 함께 기록됩니다. 보류 에피소드도
485
+ 유지 목록에는 들어가지 않습니다.
486
+ - 라이선스는 MIT입니다.
487
+ - 점수는 절대 품질이 아니라 상대 지표입니다. 학습 성능이나 로봇 안전성을
488
+ 보증하지 않으며, 처분 기준값(경계 1.5배)은 아직 잠정값입니다.
489
+
490
+ 설치와 사용법은 위 영문 섹션을 따르세요: 저장소를 clone하고
491
+ `pip install .` 한 뒤 `roboaudit <데이터셋 폴더> --out <출력 폴더>`.
492
+
493
+ **사례 공유 (선택 참여, 기본 꺼짐)**: 처분 기준값(경계 1.5배 등)은 아직
494
+ 잠정값이라, 여러 실제 데이터셋의 감사 사례가 쌓여야 검증할 수 있습니다.
495
+ `roboaudit share <result.json>` 또는 감사 시 `--share-bundle`을 명시한
496
+ 경우에만 요약 파일(`share_bundle.json`)이 만들어집니다. 자동
497
+ 전송은 절대 없으며, 제출은 본인이 GitHub 이슈에 직접 붙이는 수동
498
+ 행동입니다. 요약에는 버전·형식·개수·점수·검사별 발견 건수와 margin
499
+ 분포·처분 집계만 들어가고, 데이터셋 이름·경로·파일명·과제 원문(해시
500
+ 포함)·원본 측정값·카메라 키 이름은 화이트리스트 방식이라 구조적으로
501
+ 들어갈 수 없습니다. bundle 안의 빈 판정 칸에 직접 확인한 에피소드의
502
+ 오탐 여부를 적어 제출하면, 사람이 판정한 사례가 쌓여 기준값 검증의
503
+ 재료가 됩니다.