@trillboards/edge-sdk 0.2.106 → 0.2.107
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/deploy/detector/bench/litert-twin-parity.json +27 -1
- package/deploy/detector/bench/venue_adjudication.py +790 -0
- package/deploy/detector/bench/verify_litert_twin_parity.py +34 -10
- package/deploy/detector/trillboards_detector/labels.py +23 -1
- package/dist/EdgeAgent.js +1 -1
- package/dist/diagnose.d.ts +1 -1
- package/dist/diagnose.js +1 -1
- package/package.json +1 -1
|
@@ -20,7 +20,18 @@
|
|
|
20
20
|
"its scores agree to 7.2e-08, but row-wise the two disagree and lexsorting by box does",
|
|
21
21
|
"not collapse it. The model returns ZERO detections above 0.25 on 40 real production",
|
|
22
22
|
"frames, so there is no above-threshold sample to compare and row-level parity is",
|
|
23
|
-
"UNVERIFIED. It is not published. Settle it on a corpus the model actually fires on."
|
|
23
|
+
"UNVERIFIED. It is not published. Settle it on a corpus the model actually fires on.",
|
|
24
|
+
"",
|
|
25
|
+
"2026-08-26: the 22nd twin, gh_fall_tonlong_v2, is the first whose LiteRT input is",
|
|
26
|
+
"NHWC [1,640,640,3]. The line above saying an ultralytics detect export is NOT NHWC",
|
|
27
|
+
"described the TOOLCHAIN the first 21 were exported with, not LiteRT: onnx2tf",
|
|
28
|
+
"transposes to channels-last by design and ultralytics 8.3.145 leaves it that way.",
|
|
29
|
+
"verify_litert_twin_parity.py used to ASSERT NCHW and refused a correct twin; it now",
|
|
30
|
+
"reads the input layout off the graph -- the same rule it already applied to outputs",
|
|
31
|
+
"-- and feeds each runtime the SAME IMAGE in the layout that runtime declares. The",
|
|
32
|
+
"warning that motivated the assertion still stands: transposing because 'LiteRT is",
|
|
33
|
+
"channels-last' feeds a rotated image to a model that wanted NCHW, and every residual",
|
|
34
|
+
"then reads as export noise."
|
|
24
35
|
],
|
|
25
36
|
"measured_at": "2026-08-25",
|
|
26
37
|
"tolerances": {
|
|
@@ -308,6 +319,21 @@
|
|
|
308
319
|
"reason": "ok",
|
|
309
320
|
"onnx": "weapon_gunknife_11n.onnx",
|
|
310
321
|
"litert": "weapon_gunknife_11n.tflite"
|
|
322
|
+
},
|
|
323
|
+
{
|
|
324
|
+
"layout": "anchor_4_plus_nc",
|
|
325
|
+
"imgsz": 640,
|
|
326
|
+
"box_scale_applied": 640.0,
|
|
327
|
+
"box_residual_px": 0.0007171630859375,
|
|
328
|
+
"box_residual_unscaled": 646.9068156480789,
|
|
329
|
+
"cls_residual": 1.3665612286217765e-07,
|
|
330
|
+
"pass": true,
|
|
331
|
+
"reason": "ok",
|
|
332
|
+
"onnx": "gh_fall_tonlong_v2.onnx",
|
|
333
|
+
"litert": "gh_fall_tonlong_v2.tflite",
|
|
334
|
+
"litert_input_layout": "nhwc",
|
|
335
|
+
"measured_at": "2026-08-26",
|
|
336
|
+
"note": "The 22nd twin, added 2026-08-26, and the FIRST whose LiteRT input is NHWC. gh_fall_tonlong had no parity measurement at all until now: upstream ships `Violence` at slot 0 and `violence` at slot 6, normalize_class_name folds them to one name and ModelVocabulary refuses at construction, so neither format could be probed. Both were re-exported from ok.pt (sha256 1e12d89198e28d7c650ba86886c511fecd9f70b9d7f8bf985e76704690e58dd9) with the slots disambiguated by INDEX -- violence_slot0 / violence_slot6 -- which preserves upstream truth and asserts no semantics we have not measured."
|
|
311
337
|
}
|
|
312
338
|
]
|
|
313
339
|
}
|
|
@@ -0,0 +1,790 @@
|
|
|
1
|
+
#!/usr/bin/env python3
|
|
2
|
+
"""Re-adjudicate every screened detector against a VENUE-MATCHED negative set.
|
|
3
|
+
|
|
4
|
+
WHY THIS EXISTS
|
|
5
|
+
---------------
|
|
6
|
+
The 2026-08-25 screen scored 96 artifacts across 35 capabilities against 1,500
|
|
7
|
+
"production" frames. 1,494 of those 1,500 came from one screen type,
|
|
8
|
+
`taxi_rideshare_tv` -- rideshare cabin interiors, 44% of them darker than mean
|
|
9
|
+
pixel 50. Its own report says three times "publishable pending a venue-matched
|
|
10
|
+
negative set". So every "no model clears the bar" verdict and every published
|
|
11
|
+
model's operating threshold describes a car at night, and we sell a CCTV product.
|
|
12
|
+
|
|
13
|
+
The negative set here is the Phase-3 quiet-hour corpus: real fixed cameras in an
|
|
14
|
+
industrial logistics yard, a hospital hand-wash basin, a big-box retail aisle and
|
|
15
|
+
an office/lab room. The positive controls are unchanged -- only the negative half
|
|
16
|
+
of every number moves, which is exactly the variable under test.
|
|
17
|
+
|
|
18
|
+
THE TWO RULES THIS FILE ENFORCES
|
|
19
|
+
--------------------------------
|
|
20
|
+
1. "Quiet" is UNEVENTFUL, not EMPTY. The industrial footage contains people
|
|
21
|
+
working and a running forklift. A forklift frame is a POSITIVE for forklift and
|
|
22
|
+
a negative only for other classes, so validity is decided PER CLASS PER BUCKET
|
|
23
|
+
and a rate on an invalid cell is reported as a FIRING rate, never a false-alarm
|
|
24
|
+
rate. Where `scripts/sensing/quiet-hour-fleet-plan.json` declares a class, that
|
|
25
|
+
declaration wins; everything else falls back to the hand grading, and a class
|
|
26
|
+
with neither is reported `no_absence_declaration`.
|
|
27
|
+
2. Never blended. Every cell carries its own bucket, its own n and its own exact
|
|
28
|
+
Clopper-Pearson interval. A pooled rate over four scene types describes no
|
|
29
|
+
promise we sell. Each cell ALSO carries a box-level count, because
|
|
30
|
+
Clopper-Pearson assumes independent trials and the retail bucket violates that
|
|
31
|
+
at 0.22 s between frames -- `source_span_hours` in the composition is what
|
|
32
|
+
stops a frame count being read as an independence claim.
|
|
33
|
+
|
|
34
|
+
SCORED AS A CASCADE GATE, NOT A DETECTOR. The ranking inverts: what matters is
|
|
35
|
+
calls removed against events lost at the production empty-frame prior, not raw
|
|
36
|
+
precision. `cctv-detector-tier.md` is the authority on why.
|
|
37
|
+
"""
|
|
38
|
+
from __future__ import annotations
|
|
39
|
+
|
|
40
|
+
import argparse
|
|
41
|
+
import glob
|
|
42
|
+
import json
|
|
43
|
+
import math
|
|
44
|
+
import os
|
|
45
|
+
from collections import Counter
|
|
46
|
+
|
|
47
|
+
import numpy as np
|
|
48
|
+
|
|
49
|
+
# --- sellable capability -> the class names that would carry it ---------------
|
|
50
|
+
# Verbatim from the 2026-08-25 screen so the two adjudications are comparable;
|
|
51
|
+
# changing it would make the before/after a different question.
|
|
52
|
+
CAPS = {
|
|
53
|
+
"hard hat present": ["helmet", "hardhat", "hard_hat", "hat", "Helmet", "Hardhat", "Hard_hat"],
|
|
54
|
+
"hard hat ABSENT": ["no-helmet", "no_helmet", "no helmet", "no-hardhat", "nohat", "No-Helmet", "NO-Hardhat", "No_Helmet"],
|
|
55
|
+
"hi-vis vest present": ["vest", "safety vest", "safety_vest", "safety-vest", "Vest", "Safety Vest"],
|
|
56
|
+
"hi-vis vest ABSENT": ["no-vest", "no vest", "no_vest", "novest", "no-safety vest", "no_safety_vest", "No-Vest", "NO-Safety Vest"],
|
|
57
|
+
"face mask present": ["mask", "Mask", "with_mask", "face-mask-medical"],
|
|
58
|
+
"face mask ABSENT": ["no-mask", "no_mask", "without_mask", "NO-Mask", "No-Mask"],
|
|
59
|
+
"gloves present": ["gloves", "glove", "Glove", "Gloves", "glove_hand", "gloved_hand"],
|
|
60
|
+
"gloves ABSENT": ["no-gloves", "no-glove", "no_glove", "No-Glove", "bare_hand"],
|
|
61
|
+
"goggles / eye pro": ["goggles", "glass", "glasses", "Glass", "Goggles"],
|
|
62
|
+
"safety boots": ["boots", "Boots", "safety shoes", "safety-boot", "shoes"],
|
|
63
|
+
"apron": ["Apron", "apron"],
|
|
64
|
+
"hairnet": ["Hairnet", "hairnet"],
|
|
65
|
+
"fire": ["fire", "Fire"],
|
|
66
|
+
"smoke": ["smoke", "Smoke"],
|
|
67
|
+
"smoking (cigarette)": ["Cigarette", "cigarette", "smoking"],
|
|
68
|
+
"gun": ["gun", "Gun", "pistol", "handgun", "rifle", "shotgun", "Pistol", "Handgun", "Shotgun", "firearm"],
|
|
69
|
+
"knife": ["knife", "Knife"],
|
|
70
|
+
"weapon (generic)": ["weapon", "Weapon"],
|
|
71
|
+
"person down / fall": ["fall", "Fall", "fallen", "Fallen", "fall-detected", "Fall-Detected", "down"],
|
|
72
|
+
"forklift": ["forklift", "Forklift", "forklift-partial", "Forklift-Partial", "Yoto", "yoto"],
|
|
73
|
+
"heavy machinery": ["machinery", "machine", "excavator", "excavators", "wheel loader", "wheel_loader",
|
|
74
|
+
"dozer", "backhoe_loader", "mobile_crane", "tower_crane", "compactor",
|
|
75
|
+
"concrete_mixer_truck", "grader", "dump truck", "dump_truck"],
|
|
76
|
+
"pallet": ["pallet", "Pallet"],
|
|
77
|
+
"person / head count": ["person", "Person", "human", "Human", "head", "Head", "worker", "Worker"],
|
|
78
|
+
"vehicle / queue": ["vehicle", "vehicle queue", "Vehicle"],
|
|
79
|
+
"shopping cart": ["cart", "Cart", "trolley", "shopping_cart"],
|
|
80
|
+
"spill": ["Spill", "spill"],
|
|
81
|
+
"shoplifting / theft": ["shoplifting", "theft", "Shoplifting", "Theft"],
|
|
82
|
+
"door": ["door", "Door"],
|
|
83
|
+
"window": ["window", "Window"],
|
|
84
|
+
"phone use": ["phone", "phone_call", "Phone", "cell phone"],
|
|
85
|
+
# `mouse` is carried from the 2026-08-25 table so the two adjudications stay
|
|
86
|
+
# comparable, and it is SEMANTICALLY WRONG for COCO: category 74 is a COMPUTER
|
|
87
|
+
# mouse. It cannot win here -- it would have to clear >=50% recall on n_rodent,
|
|
88
|
+
# a set of rodent photographs -- but if a COCO-vocabulary model ever takes the
|
|
89
|
+
# `rodent` row, that is the reason and the row is void.
|
|
90
|
+
"rodent": ["mouse", "Mouse", "rat", "rodent"],
|
|
91
|
+
"shelf gap / OOS": ["void", "Void", "gap", "empty"],
|
|
92
|
+
"traffic cone": ["safety cone", "Safety Cone", "safety_cone"],
|
|
93
|
+
"fire extinguisher": ["fire_extinguisher", "Fire_Extinguisher"],
|
|
94
|
+
"ladder": ["ladder", "Ladder"],
|
|
95
|
+
}
|
|
96
|
+
|
|
97
|
+
# --- class name -> positive-control set(s) that should make that class fire ----
|
|
98
|
+
# 'PROD' is dropped: it meant the 1,500 rideshare frames, and a positive control
|
|
99
|
+
# drawn from the domain under test is exactly the confound this run removes.
|
|
100
|
+
CLASS_CONTROL = {
|
|
101
|
+
"fire": ["fire_cctv", "fire_generic"], "smoke": ["fire_cctv", "fire_generic"],
|
|
102
|
+
"helmet": ["helmet", "n_heavy"], "hardhat": ["helmet", "n_heavy"], "hard_hat": ["helmet", "n_heavy"],
|
|
103
|
+
"hat": ["helmet", "ppe_full"],
|
|
104
|
+
"vest": ["helmet", "n_heavy", "n_cone", "ppe_full"], "safety vest": ["helmet", "n_heavy", "n_cone", "ppe_full"],
|
|
105
|
+
"safety_vest": ["helmet", "n_heavy", "n_cone", "ppe_full"], "safety-vest": ["helmet", "n_heavy", "n_cone", "ppe_full"],
|
|
106
|
+
"mask": ["ppe_full"], "with_mask": ["ppe_full"], "face-mask-medical": ["ppe_full"],
|
|
107
|
+
"goggles": ["ppe_full"], "glass": ["ppe_full"], "glasses": ["ppe_full"],
|
|
108
|
+
"boots": ["ppe_full"], "safety shoes": ["ppe_full"], "safety-boot": ["ppe_full"], "shoes": ["ppe_full"],
|
|
109
|
+
"gloves": ["ppe_full", "wc_glove"], "glove": ["ppe_full", "wc_glove"], "glove_hand": ["wc_glove", "ppe_full"],
|
|
110
|
+
"apron": ["wc_apron"], "Apron": ["wc_apron"], "hairnet": ["n_hairnet"], "Hairnet": ["n_hairnet"],
|
|
111
|
+
"safety cone": ["n_cone"], "safety_cone": ["n_cone"],
|
|
112
|
+
"fire_extinguisher": ["n_extinguisher"], "ladder": ["n_ladder"],
|
|
113
|
+
"person": ["helmet", "ppe_full", "n_crowd"], "worker": ["ppe_full", "helmet", "n_crowd"],
|
|
114
|
+
"human": ["helmet", "ppe_full", "n_crowd"], "head": ["helmet", "n_crowd"],
|
|
115
|
+
"forklift": ["forklift"], "forklift-partial": ["forklift"], "Forklift-Partial": ["forklift"], "yoto": ["forklift"],
|
|
116
|
+
"pallet": ["n_pallet"], "machinery": ["n_heavy"], "machine": ["n_heavy"],
|
|
117
|
+
"excavator": ["n_heavy"], "excavators": ["n_heavy"], "dozer": ["n_heavy"],
|
|
118
|
+
"wheel loader": ["n_heavy"], "wheel_loader": ["n_heavy"], "backhoe_loader": ["n_heavy"],
|
|
119
|
+
"mobile_crane": ["n_heavy"], "tower_crane": ["n_heavy"], "compactor": ["n_heavy"],
|
|
120
|
+
"concrete_mixer_truck": ["n_heavy"], "grader": ["n_heavy"], "dump truck": ["n_heavy"], "dump_truck": ["n_heavy"],
|
|
121
|
+
"vehicle": ["n_traffic"], "vehicle queue": ["n_traffic"],
|
|
122
|
+
"gun": ["gun_clean", "weapon_cctv"], "pistol": ["gun_clean", "weapon_cctv"], "handgun": ["gun_clean", "weapon_cctv"],
|
|
123
|
+
"rifle": ["gun_clean"], "shotgun": ["gun_clean"], "weapon": ["weapon_cctv", "gun_clean"],
|
|
124
|
+
"firearm": ["gun_clean", "weapon_cctv"], "knife": ["knife_cctv", "weapon_cctv", "n_knife"],
|
|
125
|
+
"fall": ["fall_cctv"], "fallen": ["fall_cctv"], "fall-detected": ["fall_cctv"],
|
|
126
|
+
"Fall-Detected": ["fall_cctv"], "down": ["fall_cctv"],
|
|
127
|
+
"door": ["n_door"], "window": ["n_window"], "phone": ["n_phone"], "cell phone": ["n_phone"],
|
|
128
|
+
"phone_call": ["n_phone"], "mouse": ["n_rodent"], "void": ["n_shelfgap"],
|
|
129
|
+
"cart": ["wc_cart", "n_trolley"], "trolley": ["wc_cart", "n_trolley"], "shopping_cart": ["wc_cart", "n_trolley"],
|
|
130
|
+
"spill": ["wc_spill", "n_spill"], "Spill": ["wc_spill", "n_spill"],
|
|
131
|
+
"cigarette": ["n_smoking"], "Cigarette": ["n_smoking"], "smoking": ["n_smoking"],
|
|
132
|
+
}
|
|
133
|
+
|
|
134
|
+
def normalise_class(name: str) -> str:
|
|
135
|
+
"""`labels.py::normalize_class_name`: lower, and `-`/`_` folded to space.
|
|
136
|
+
|
|
137
|
+
The control lookup MUST go through this. A case-sensitive lookup misses
|
|
138
|
+
`Fire_Extinguisher` against a table keyed `fire_extinguisher` and the
|
|
139
|
+
capability then reports UNINFORMATIVE -- "no usable positive control exists"
|
|
140
|
+
-- when the control is sitting right there. That reads as a finding about the
|
|
141
|
+
world and is a finding about a dict.
|
|
142
|
+
"""
|
|
143
|
+
return name.strip().lower().replace("_", " ").replace("-", " ")
|
|
144
|
+
|
|
145
|
+
|
|
146
|
+
#: CLASS_CONTROL re-keyed by normalised name, built once.
|
|
147
|
+
CLASS_CONTROL_N = {}
|
|
148
|
+
for _k, _v in CLASS_CONTROL.items():
|
|
149
|
+
CLASS_CONTROL_N.setdefault(normalise_class(_k), []).extend(_v)
|
|
150
|
+
CLASS_CONTROL_N = {k: sorted(set(v), key=v.index) for k, v in CLASS_CONTROL_N.items()}
|
|
151
|
+
|
|
152
|
+
|
|
153
|
+
# --- capability -> the plan file's own class, where one covers it -------------
|
|
154
|
+
# The plan declares `expectedOnQuiet` for nine classes. Where it speaks, it wins.
|
|
155
|
+
CAP_TO_PLAN_CLASS = {
|
|
156
|
+
"person / head count": "person_count",
|
|
157
|
+
"person down / fall": "person_on_the_ground",
|
|
158
|
+
"spill": "spill_on_floor",
|
|
159
|
+
"fire": "fire_or_smoke_visible",
|
|
160
|
+
"smoke": "fire_or_smoke_visible",
|
|
161
|
+
"forklift": "forklift_present",
|
|
162
|
+
"hard hat ABSENT": "person_missing_required_ppe",
|
|
163
|
+
"hi-vis vest ABSENT": "person_missing_required_ppe",
|
|
164
|
+
"face mask ABSENT": "person_missing_required_ppe",
|
|
165
|
+
"gloves ABSENT": "person_missing_required_ppe",
|
|
166
|
+
"phone use": "person_using_phone",
|
|
167
|
+
}
|
|
168
|
+
|
|
169
|
+
# --- capability -> the hand-grading key, for classes the plan does not cover ---
|
|
170
|
+
CAP_TO_GRADE_CLASS = {
|
|
171
|
+
"hard hat present": "hard_hat", "hi-vis vest present": "hi_vis_vest",
|
|
172
|
+
"face mask present": "face_mask", "gloves present": "gloves",
|
|
173
|
+
"goggles / eye pro": "goggles", "safety boots": "safety_boots",
|
|
174
|
+
"apron": "apron", "hairnet": "hairnet", "smoking (cigarette)": "cigarette",
|
|
175
|
+
"gun": "gun", "knife": "knife", "weapon (generic)": "weapon_generic",
|
|
176
|
+
"heavy machinery": "heavy_machinery", "pallet": "pallet",
|
|
177
|
+
"vehicle / queue": "vehicle", "shopping cart": "shopping_cart",
|
|
178
|
+
"shoplifting / theft": "shoplifting", "door": "door", "window": "window",
|
|
179
|
+
"rodent": "rodent", "shelf gap / OOS": "shelf_gap",
|
|
180
|
+
"traffic cone": "traffic_cone", "fire extinguisher": "fire_extinguisher",
|
|
181
|
+
"ladder": "ladder",
|
|
182
|
+
}
|
|
183
|
+
|
|
184
|
+
|
|
185
|
+
# --- what the 2026-08-25 taxi-negative screen decided, for the before/after ----
|
|
186
|
+
# Historical fact, embedded so the comparison is reproducible from this file
|
|
187
|
+
# alone; the source `ranking.json` was a scratch artifact and is not committed.
|
|
188
|
+
PRIOR_VERDICT = {
|
|
189
|
+
'hard hat present': ('PUBLISH', 'ppe_vyra_14cls_v8m'),
|
|
190
|
+
'hard hat ABSENT': ('NOT SCREENABLE HERE', None),
|
|
191
|
+
'hi-vis vest present': ('NO MODEL CLEARS BAR', 'ppe_boots_goggles_v8s'),
|
|
192
|
+
'hi-vis vest ABSENT': ('NOT SCREENABLE HERE', None),
|
|
193
|
+
'face mask present': ('NO MODEL CLEARS BAR', 'ppe_mask_3cls_v8n'),
|
|
194
|
+
'face mask ABSENT': ('NOT SCREENABLE HERE', None),
|
|
195
|
+
'gloves present': ('NO MODEL CLEARS BAR', 'ppe_sh17_17cls_11x'),
|
|
196
|
+
'gloves ABSENT': ('NOT SCREENABLE HERE', None),
|
|
197
|
+
'goggles / eye pro': ('NO MODEL CLEARS BAR', 'ppe_vyra_14cls_v8m'),
|
|
198
|
+
'safety boots': ('NO MODEL CLEARS BAR', 'ppe_sh17_17cls_11x'),
|
|
199
|
+
'apron': ('UNINFORMATIVE', None),
|
|
200
|
+
'hairnet': ('UNINFORMATIVE', None),
|
|
201
|
+
'fire': ('NO MODEL CLEARS BAR', 'fire_smoke_weslleyskah_11n'),
|
|
202
|
+
'smoke': ('AMBIGUOUS NEGATIVE SET', 'fire_smoke_weslleyskah_11n'),
|
|
203
|
+
'smoking (cigarette)': ('UNINFORMATIVE', None),
|
|
204
|
+
'gun': ('NO MODEL CLEARS BAR', 'gh_weapon_activeshooter_11'),
|
|
205
|
+
'knife': ('NO MODEL CLEARS BAR', 'gh_weapon_nihal_11s'),
|
|
206
|
+
'weapon (generic)': ('NO MODEL CLEARS BAR', 'weapon_generic_v8m'),
|
|
207
|
+
'person down / fall': ('PUBLISH', 'gh_fall_tonlong'),
|
|
208
|
+
'forklift': ('PUBLISH', 'gh_forklift_combine'),
|
|
209
|
+
'heavy machinery': ('PUBLISH', 'ppe_construction_17cls_v8n'),
|
|
210
|
+
'pallet': ('NO MODEL CLEARS BAR', 'gh_forklift_pallet_11s'),
|
|
211
|
+
'person / head count': ('NOT SCREENABLE HERE', None),
|
|
212
|
+
'vehicle / queue': ('AMBIGUOUS NEGATIVE SET', 'gh_ppe_vinayak_l'),
|
|
213
|
+
'shopping cart': ('NO CANDIDATE', None),
|
|
214
|
+
'spill': ('UNINFORMATIVE', None),
|
|
215
|
+
'shoplifting / theft': ('UNINFORMATIVE', None),
|
|
216
|
+
'door': ('NOT SCREENABLE HERE', None),
|
|
217
|
+
'window': ('NOT SCREENABLE HERE', None),
|
|
218
|
+
'phone use': ('NOT SCREENABLE HERE', None),
|
|
219
|
+
'rodent': ('NO MODEL CLEARS BAR', 'rodent_pulkit'),
|
|
220
|
+
'shelf gap / OOS': ('NO MODEL CLEARS BAR', 'hf_shelfgap'),
|
|
221
|
+
'traffic cone': ('AMBIGUOUS NEGATIVE SET', 'gh_ppe_vinayak_l'),
|
|
222
|
+
'fire extinguisher': ('PUBLISH', 'ppe_killuminati_19cls_v8s'),
|
|
223
|
+
'ladder': ('UNINFORMATIVE', None),
|
|
224
|
+
}
|
|
225
|
+
|
|
226
|
+
#: Artifacts the shipped service REFUSES to load, and what replaces them. An
|
|
227
|
+
#: adjudication that names an unloadable artifact is worse than one that omits
|
|
228
|
+
#: it: the row looks shippable and the model cannot be published.
|
|
229
|
+
#: `gh_fall_tonlong` declares `Violence` at slot 0 and `violence` at slot 6;
|
|
230
|
+
#: `labels.py::normalize_class_name` folds them to one name and ModelVocabulary
|
|
231
|
+
#: raises at construction. `gh_fall_tonlong_v2` is the same weights re-exported
|
|
232
|
+
#: with the slots disambiguated by index, so the numbers are unchanged and only
|
|
233
|
+
#: the name the table prints moves.
|
|
234
|
+
SUPERSEDED = {"gh_fall_tonlong": "gh_fall_tonlong_v2"}
|
|
235
|
+
|
|
236
|
+
|
|
237
|
+
GRID = np.round(np.arange(0.05, 0.9751, 0.0125), 4)
|
|
238
|
+
QUIET_MAX_FIRE = 0.01 # the 2026-08-25 bar: <1% firing on valid negatives
|
|
239
|
+
RECALL_RETENTION_BAR = 0.50 # "keeps >=50% of positive-control detections"
|
|
240
|
+
OFF_TARGET_BAR = 0.10 # "<=10% mean firing on control sets built for other things"
|
|
241
|
+
PRODUCTION_EMPTY_PRIOR = 0.847 # cctv-detector-tier.md's production frame prior
|
|
242
|
+
|
|
243
|
+
|
|
244
|
+
def clopper_pearson(k: int, n: int, alpha: float = 0.05):
|
|
245
|
+
"""Exact binomial interval. A rate on 14 frames and a rate on 480 are not the
|
|
246
|
+
same claim, and only the interval says so."""
|
|
247
|
+
if n == 0:
|
|
248
|
+
return [None, None]
|
|
249
|
+
from math import inf
|
|
250
|
+
def beta_ppf(p, a, b):
|
|
251
|
+
# bisection on the regularised incomplete beta; no scipy on the box
|
|
252
|
+
if a <= 0:
|
|
253
|
+
return 0.0
|
|
254
|
+
if b <= 0:
|
|
255
|
+
return 1.0
|
|
256
|
+
lo, hi = 0.0, 1.0
|
|
257
|
+
for _ in range(200):
|
|
258
|
+
mid = (lo + hi) / 2
|
|
259
|
+
if _betainc(a, b, mid) < p:
|
|
260
|
+
lo = mid
|
|
261
|
+
else:
|
|
262
|
+
hi = mid
|
|
263
|
+
return (lo + hi) / 2
|
|
264
|
+
lo = 0.0 if k == 0 else beta_ppf(alpha / 2, k, n - k + 1)
|
|
265
|
+
hi = 1.0 if k == n else beta_ppf(1 - alpha / 2, k + 1, n - k)
|
|
266
|
+
return [round(lo, 6), round(hi, 6)]
|
|
267
|
+
|
|
268
|
+
|
|
269
|
+
def _betainc(a: float, b: float, x: float) -> float:
|
|
270
|
+
"""Regularised incomplete beta via the continued fraction (Lentz)."""
|
|
271
|
+
if x <= 0:
|
|
272
|
+
return 0.0
|
|
273
|
+
if x >= 1:
|
|
274
|
+
return 1.0
|
|
275
|
+
lbeta = math.lgamma(a) + math.lgamma(b) - math.lgamma(a + b)
|
|
276
|
+
front = math.exp(math.log(x) * a + math.log(1 - x) * b - lbeta) / a
|
|
277
|
+
f, c, d = 1.0, 1.0, 0.0
|
|
278
|
+
for i in range(0, 300):
|
|
279
|
+
m = i // 2
|
|
280
|
+
if i == 0:
|
|
281
|
+
num = 1.0
|
|
282
|
+
elif i % 2 == 0:
|
|
283
|
+
num = (m * (b - m) * x) / ((a + 2 * m - 1) * (a + 2 * m))
|
|
284
|
+
else:
|
|
285
|
+
num = -((a + m) * (a + b + m) * x) / ((a + 2 * m) * (a + 2 * m + 1))
|
|
286
|
+
d = 1.0 + num * d
|
|
287
|
+
d = 1e-30 if abs(d) < 1e-30 else d
|
|
288
|
+
d = 1.0 / d
|
|
289
|
+
c = 1.0 + num / c
|
|
290
|
+
c = 1e-30 if abs(c) < 1e-30 else c
|
|
291
|
+
f *= c * d
|
|
292
|
+
if abs(1.0 - c * d) < 1e-12:
|
|
293
|
+
break
|
|
294
|
+
r = front * (f - 1.0)
|
|
295
|
+
if x < (a + 1) / (a + b + 2):
|
|
296
|
+
return r
|
|
297
|
+
# `_betainc(b, a, 1 - x)` is the SAME CALL when a == b and x == 0.5, so the
|
|
298
|
+
# reflection recursed forever -- and `beta_ppf` starts its bisection at exactly
|
|
299
|
+
# 0.5, so any Clopper-Pearson interval with symmetric beta parameters (midpoint
|
|
300
|
+
# successes in an odd-sized sample) raised RecursionError instead of returning.
|
|
301
|
+
# I(0.5; a, a) = 0.5 by symmetry; take it directly.
|
|
302
|
+
if a == b and abs(x - 0.5) < 1e-15:
|
|
303
|
+
return 0.5
|
|
304
|
+
return 1.0 - _betainc(b, a, 1 - x)
|
|
305
|
+
|
|
306
|
+
|
|
307
|
+
def main() -> None:
|
|
308
|
+
ap = argparse.ArgumentParser()
|
|
309
|
+
ap.add_argument("--out-dir", required=True, help="directory of <alias>.npz + _index.json")
|
|
310
|
+
ap.add_argument("--combo", required=True, help="combo.json written by vm_prep2.py")
|
|
311
|
+
ap.add_argument("--grading", required=True, help="venue absence grading json")
|
|
312
|
+
ap.add_argument("--plan", required=True, help="scripts/sensing/quiet-hour-fleet-plan.json")
|
|
313
|
+
ap.add_argument("--control-quality",
|
|
314
|
+
default="trillboard-api/scripts/ai/eval-baselines/positive-control-quality-2026-08-25.json",
|
|
315
|
+
help="contact-sheet grades for the positive control sets; DISCARDED/POOR/CONTAMINATED are refused")
|
|
316
|
+
ap.add_argument("--out", required=True)
|
|
317
|
+
ap.add_argument("--markdown", help="also write the runbook table here")
|
|
318
|
+
args = ap.parse_args()
|
|
319
|
+
|
|
320
|
+
combo = json.load(open(args.combo))
|
|
321
|
+
grading = json.load(open(args.grading))
|
|
322
|
+
plan = json.load(open(args.plan))
|
|
323
|
+
ctlq = json.load(open(args.control_quality))
|
|
324
|
+
index = json.load(open(os.path.join(args.out_dir, "_index.json")))
|
|
325
|
+
|
|
326
|
+
meta = combo["meta"]
|
|
327
|
+
buckets = np.array([m.get("bucket") or "" for m in meta])
|
|
328
|
+
sets = np.array([m["set"] for m in meta])
|
|
329
|
+
is_neg = sets == "venue_src"
|
|
330
|
+
pos_sets = sorted({m["pos_set"] for m in meta if m.get("pos_set")})
|
|
331
|
+
|
|
332
|
+
# Positive-control sets whose contact-sheet grade forbids citing them.
|
|
333
|
+
# CONTAMINATED belongs here and was missed on the first pass: `wc_spill` is
|
|
334
|
+
# graded CONTAMINATED ("search pulled 'oil spill' — an overturned truck — a
|
|
335
|
+
# cathedral corridor, building exteriors and abstract cream textures"), and
|
|
336
|
+
# citing it would have given `spill` a recall number off a set that mostly
|
|
337
|
+
# does not contain a spill. Read the grades, do not enumerate the bad ones
|
|
338
|
+
# from memory.
|
|
339
|
+
UNUSABLE_GRADES = ("DISCARDED", "POOR", "CONTAMINATED")
|
|
340
|
+
unusable = {k for k, v in ctlq.items()
|
|
341
|
+
if isinstance(v, dict) and v.get("grade") in UNUSABLE_GRADES}
|
|
342
|
+
|
|
343
|
+
# ---- composition, stated rather than assumed ----------------------------
|
|
344
|
+
# `source_span_hours` is what keeps the frame count honest. 480 industrial
|
|
345
|
+
# frames spread over 10.77 h are 81 s apart and independent; 400 retail frames
|
|
346
|
+
# drawn from 72 SECONDS are ~5 frames per second of one aisle. Same n, very
|
|
347
|
+
# different evidence, and only the span says which you are holding.
|
|
348
|
+
src_index = {}
|
|
349
|
+
src_path = os.path.join(os.path.dirname(args.combo), "srcframes_index.json")
|
|
350
|
+
if os.path.exists(src_path):
|
|
351
|
+
for r in json.load(open(src_path)):
|
|
352
|
+
src_index.setdefault(r["bucket"], {})[r["box"]] = r.get("src_duration_sec") or 0
|
|
353
|
+
|
|
354
|
+
comp = {"negative_set": {}, "positive_controls": {}}
|
|
355
|
+
for b in sorted(set(buckets[is_neg].tolist())):
|
|
356
|
+
m = is_neg & (buckets == b)
|
|
357
|
+
boxes = sorted({meta[i]["box"] for i in np.where(m)[0]})
|
|
358
|
+
span = sum(src_index.get(b, {}).values()) / 3600.0
|
|
359
|
+
comp["negative_set"][b] = {
|
|
360
|
+
"frames": int(m.sum()), "boxes": len(boxes),
|
|
361
|
+
"source_span_hours": round(span, 4),
|
|
362
|
+
"seconds_between_frames": (round(span * 3600 / int(m.sum()), 2)
|
|
363
|
+
if m.sum() and span else None),
|
|
364
|
+
"corpora": sorted({meta[i]["corpus"] for i in np.where(m)[0]}),
|
|
365
|
+
"box_ids": boxes,
|
|
366
|
+
}
|
|
367
|
+
for s in pos_sets:
|
|
368
|
+
m = sets == ("pos:" + s)
|
|
369
|
+
g = (ctlq.get(s) or {})
|
|
370
|
+
comp["positive_controls"][s] = {
|
|
371
|
+
"frames": int(m.sum()), "grade": g.get("grade"),
|
|
372
|
+
"domain": g.get("domain"), "citable": s not in unusable,
|
|
373
|
+
}
|
|
374
|
+
|
|
375
|
+
def validity(cap: str, bucket: str):
|
|
376
|
+
"""Is this bucket a valid NEGATIVE for this capability?
|
|
377
|
+
|
|
378
|
+
Returns (verdict, source). `valid` and `weak` may carry a false-alarm
|
|
379
|
+
rate; `invalid` and `none` may carry only a firing rate.
|
|
380
|
+
"""
|
|
381
|
+
pc = CAP_TO_PLAN_CLASS.get(cap)
|
|
382
|
+
if pc:
|
|
383
|
+
# the plan declares three buckets; retail and office are the two
|
|
384
|
+
# halves of its `indoor_retail_and_office`, so its declaration
|
|
385
|
+
# covers both. Splitting them is a refinement, not a contradiction.
|
|
386
|
+
pb = {"indoor_retail": "indoor_retail_and_office",
|
|
387
|
+
"indoor_office_lab": "indoor_retail_and_office"}.get(bucket, bucket)
|
|
388
|
+
d = ((plan["classes"].get(pc) or {}).get("expectedOnQuiet") or {}).get(pb)
|
|
389
|
+
if d == "never":
|
|
390
|
+
return "valid", f"plan:{pc}=never"
|
|
391
|
+
if d == "possible":
|
|
392
|
+
return "invalid", f"plan:{pc}=possible"
|
|
393
|
+
gk = CAP_TO_GRADE_CLASS.get(cap)
|
|
394
|
+
if gk:
|
|
395
|
+
d = ((grading.get("per_class") or {}).get(gk) or {}).get(bucket)
|
|
396
|
+
if d == "absent":
|
|
397
|
+
return "valid", f"graded:{gk}=absent"
|
|
398
|
+
if d == "not_observed":
|
|
399
|
+
return "weak", f"graded:{gk}=not_observed"
|
|
400
|
+
if d in ("PRESENT", "AMBIGUOUS"):
|
|
401
|
+
return "invalid", f"graded:{gk}={d}"
|
|
402
|
+
return "none", "no_absence_declaration"
|
|
403
|
+
|
|
404
|
+
models = {}
|
|
405
|
+
for alias, info in index.items():
|
|
406
|
+
if not info.get("ok"):
|
|
407
|
+
continue
|
|
408
|
+
p = os.path.join(args.out_dir, alias + ".npz")
|
|
409
|
+
if not os.path.exists(p):
|
|
410
|
+
continue
|
|
411
|
+
d = np.load(p)
|
|
412
|
+
ms = d["max_scores"]
|
|
413
|
+
names = info.get("names")
|
|
414
|
+
if names:
|
|
415
|
+
names = {int(k): v for k, v in names.items()}
|
|
416
|
+
else: # RF-DETR: COCO category ids
|
|
417
|
+
coco = json.load(open(os.path.join(os.path.dirname(args.combo), "coco91.json")))
|
|
418
|
+
names = {int(k): v for k, v in coco.items()}
|
|
419
|
+
models[alias] = {"info": info, "ms": ms, "names": names,
|
|
420
|
+
"lat_p50_ms": float(np.percentile(d["lat"], 50))}
|
|
421
|
+
|
|
422
|
+
# ---- the empty-frame prior, MEASURED on venue footage -------------------
|
|
423
|
+
# Every `vlm_calls_saved` in this file is reweighted to 0.847 empty frames,
|
|
424
|
+
# which came off the RIDESHARE fleet. A parked cab with nobody in it is empty
|
|
425
|
+
# most of the time; a shop is not. The gate's own person detector says so:
|
|
426
|
+
# `rfdetr_medium` (100% rollout, the artifact actually deciding this today)
|
|
427
|
+
# sees a person in half to nearly all venue frames depending on the bucket.
|
|
428
|
+
#
|
|
429
|
+
# Reported, never substituted into the gate arithmetic above, because
|
|
430
|
+
# "detector sees no person" is a proxy for "empty" and not a label -- these
|
|
431
|
+
# frames carry no person ground truth (`person_count` is `possible` in all
|
|
432
|
+
# three plan buckets, so there is no valid negative for it here either).
|
|
433
|
+
# It is a MEASURED CEILING on what a person-gated cascade could save in a
|
|
434
|
+
# venue, and it is far below what the taxi prior implies.
|
|
435
|
+
venue_prior = {"_what": "share of venue frames with NO person detected by "
|
|
436
|
+
"rfdetr_medium, per bucket, at three confidences",
|
|
437
|
+
"_caveat": "a detector verdict, not a label. Reported beside the "
|
|
438
|
+
"carried 0.847 taxi prior, never substituted for it.",
|
|
439
|
+
"_source": None, "by_bucket": {}}
|
|
440
|
+
rf = models.get("PUB__rfdetr_medium") or models.get("rfdetr_medium")
|
|
441
|
+
if rf is not None:
|
|
442
|
+
venue_prior["_source"] = ("PUB__rfdetr_medium" if "PUB__rfdetr_medium" in models
|
|
443
|
+
else "rfdetr_medium")
|
|
444
|
+
person_idx = next((i for i, n in rf["names"].items() if n == "person"), None)
|
|
445
|
+
if person_idx is not None:
|
|
446
|
+
col = rf["ms"][:, person_idx]
|
|
447
|
+
for b in sorted(set(buckets[is_neg].tolist())):
|
|
448
|
+
m = is_neg & (buckets == b)
|
|
449
|
+
venue_prior["by_bucket"][b] = {
|
|
450
|
+
"n": int(m.sum()),
|
|
451
|
+
"empty_at_0.25": float((col[m] < 0.25).mean()),
|
|
452
|
+
"empty_at_0.50": float((col[m] < 0.50).mean()),
|
|
453
|
+
"empty_at_0.70": float((col[m] < 0.70).mean()),
|
|
454
|
+
}
|
|
455
|
+
allm = is_neg
|
|
456
|
+
venue_prior["all_venue"] = {
|
|
457
|
+
"n": int(allm.sum()),
|
|
458
|
+
"empty_at_0.50": float((col[allm] < 0.50).mean()),
|
|
459
|
+
"taxi_prior_carried_in_gate_arithmetic": PRODUCTION_EMPTY_PRIOR,
|
|
460
|
+
}
|
|
461
|
+
|
|
462
|
+
# ---- control: do the VM re-exports describe the SHIPPED artifacts? -----
|
|
463
|
+
# Four specialists and rfdetr are run twice, once from a re-export of the
|
|
464
|
+
# upstream weights and once from `pubonnx/` -- the exact bytes in
|
|
465
|
+
# sensing_model_registry. A re-export of the same .pt is NOT guaranteed to be
|
|
466
|
+
# the same artifact (fp32 export moved YOLOE's recall 28 points, §3), so the
|
|
467
|
+
# pair decides whether this whole adjudication may be read as describing what
|
|
468
|
+
# is deployed. Compared on the SAME frames through the SAME decode: only the
|
|
469
|
+
# artifact differs.
|
|
470
|
+
repro = {}
|
|
471
|
+
for alias, M in models.items():
|
|
472
|
+
if not alias.startswith("PUB__"):
|
|
473
|
+
continue
|
|
474
|
+
base = alias[len("PUB__"):]
|
|
475
|
+
other = models.get(base)
|
|
476
|
+
if other is None:
|
|
477
|
+
repro[base] = {"paired": False,
|
|
478
|
+
"why": "no re-export of this artifact on the box"}
|
|
479
|
+
continue
|
|
480
|
+
a, b = M["ms"], other["ms"]
|
|
481
|
+
if a.shape != b.shape:
|
|
482
|
+
repro[base] = {"paired": False, "why": f"shape {a.shape} vs {b.shape}"}
|
|
483
|
+
continue
|
|
484
|
+
d = np.abs(a - b)
|
|
485
|
+
# a firing decision is what we actually consume, so disagreement is
|
|
486
|
+
# counted at the threshold, not only as a residual
|
|
487
|
+
dis = {}
|
|
488
|
+
for t in (0.25, 0.50, 0.70):
|
|
489
|
+
dis[f"decision_disagreement_{t:.2f}"] = float(((a >= t) != (b >= t)).mean())
|
|
490
|
+
repro[base] = {"paired": True, "n_frames": int(a.shape[0]),
|
|
491
|
+
"n_classes": int(a.shape[1]),
|
|
492
|
+
"max_abs_score_delta": float(d.max()),
|
|
493
|
+
"mean_abs_score_delta": float(d.mean()), **dis}
|
|
494
|
+
|
|
495
|
+
rows = []
|
|
496
|
+
for cap, wanted in CAPS.items():
|
|
497
|
+
cands = []
|
|
498
|
+
for alias, M in models.items():
|
|
499
|
+
if alias in SUPERSEDED and SUPERSEDED[alias] in models:
|
|
500
|
+
continue
|
|
501
|
+
hit = [(ci, cn) for ci, cn in M["names"].items() if cn in wanted]
|
|
502
|
+
if not hit:
|
|
503
|
+
continue
|
|
504
|
+
ci, cn = hit[0]
|
|
505
|
+
col = M["ms"][:, ci]
|
|
506
|
+
|
|
507
|
+
# -------- negative side, per bucket, never blended ---------------
|
|
508
|
+
by_bucket, valid_mask = {}, np.zeros(len(col), dtype=bool)
|
|
509
|
+
for b in sorted(set(buckets[is_neg].tolist())):
|
|
510
|
+
m = is_neg & (buckets == b)
|
|
511
|
+
v, src = validity(cap, b)
|
|
512
|
+
sc = col[m]
|
|
513
|
+
k50 = int((sc >= 0.50).sum())
|
|
514
|
+
# A BOX-LEVEL view beside the frame-level one. Clopper-Pearson
|
|
515
|
+
# assumes independent trials and the retail bucket flatly violates
|
|
516
|
+
# that -- 400 frames drawn 0.22 s apart from one aisle. Reporting
|
|
517
|
+
# how many distinct SOURCE BOXES fire, out of how many exist, gives
|
|
518
|
+
# the reader the conservative unit without pretending there is one
|
|
519
|
+
# correct n. Neither number is "the" rate; the pair is.
|
|
520
|
+
fb = [meta[i]["box"] for i in np.where(m)[0]]
|
|
521
|
+
fired_boxes = {fb[t] for t in range(len(fb)) if sc[t] >= 0.50}
|
|
522
|
+
by_bucket[b] = {
|
|
523
|
+
"n": int(m.sum()), "validity": v, "validity_source": src,
|
|
524
|
+
"fire_050": float((sc >= 0.50).mean()),
|
|
525
|
+
"fire_050_ci95": clopper_pearson(k50, int(m.sum())),
|
|
526
|
+
"ci95_assumes_independent_frames": True,
|
|
527
|
+
"boxes_firing_050": len(fired_boxes),
|
|
528
|
+
"boxes": len(set(fb)),
|
|
529
|
+
"max_score": float(sc.max()) if sc.size else 0.0,
|
|
530
|
+
"metric": ("false_alarm_rate" if v in ("valid", "weak")
|
|
531
|
+
else "firing_rate_only"),
|
|
532
|
+
}
|
|
533
|
+
if v in ("valid", "weak"):
|
|
534
|
+
valid_mask |= m
|
|
535
|
+
|
|
536
|
+
n_valid = int(valid_mask.sum())
|
|
537
|
+
# THE THRESHOLD MUST HOLD IN EVERY VALID BUCKET, NOT IN THEIR UNION.
|
|
538
|
+
# Choosing it from the pooled mask is the "never blended" rule broken by
|
|
539
|
+
# the tool that states it: the hospital sink is 320 frames and the
|
|
540
|
+
# industrial yard 480, so a class firing 3% on the sink and 0% elsewhere
|
|
541
|
+
# pools to 0.8% and passes, having failed the advertised bar for that
|
|
542
|
+
# scene type. `quiet_fire_rate` is correspondingly the WORST bucket's
|
|
543
|
+
# rate at that threshold, never the average.
|
|
544
|
+
quiet_t, quiet_fire, quiet_by_bucket = None, None, None
|
|
545
|
+
valid_names = [b for b, v in by_bucket.items()
|
|
546
|
+
if v["validity"] in ("valid", "weak")]
|
|
547
|
+
if n_valid and valid_names:
|
|
548
|
+
cols = {b: col[is_neg & (buckets == b)] for b in valid_names}
|
|
549
|
+
for t in GRID:
|
|
550
|
+
rates = {b: float((sc >= t).mean()) for b, sc in cols.items()}
|
|
551
|
+
if max(rates.values()) < QUIET_MAX_FIRE:
|
|
552
|
+
quiet_t = float(t)
|
|
553
|
+
quiet_fire = max(rates.values())
|
|
554
|
+
quiet_by_bucket = {b: r for b, r in sorted(rates.items())}
|
|
555
|
+
break
|
|
556
|
+
|
|
557
|
+
# -------- positive side: same artifact, same preprocessing -------
|
|
558
|
+
ctl_names = CLASS_CONTROL_N.get(normalise_class(cn), [])
|
|
559
|
+
best_ctl, best_rate, best_at_qt = None, None, None
|
|
560
|
+
per_ctl = {}
|
|
561
|
+
for s in ctl_names:
|
|
562
|
+
if s in unusable or ("pos:" + s) not in combo["ranges"]:
|
|
563
|
+
continue
|
|
564
|
+
m = sets == ("pos:" + s)
|
|
565
|
+
if not m.sum():
|
|
566
|
+
continue
|
|
567
|
+
r25 = float((col[m] >= 0.25).mean())
|
|
568
|
+
rqt = float((col[m] >= quiet_t).mean()) if quiet_t is not None else None
|
|
569
|
+
per_ctl[s] = {"n": int(m.sum()), "rate_025": r25, "rate_at_quiet_t": rqt}
|
|
570
|
+
if best_rate is None or r25 > best_rate:
|
|
571
|
+
best_ctl, best_rate, best_at_qt = s, r25, rqt
|
|
572
|
+
|
|
573
|
+
# -------- off-target: control sets built for OTHER things --------
|
|
574
|
+
off = {}
|
|
575
|
+
for s in pos_sets:
|
|
576
|
+
if s in ctl_names or s in unusable:
|
|
577
|
+
continue
|
|
578
|
+
m = sets == ("pos:" + s)
|
|
579
|
+
if not m.sum() or quiet_t is None:
|
|
580
|
+
continue
|
|
581
|
+
off[s] = float((col[m] >= quiet_t).mean())
|
|
582
|
+
off_mean = (sum(off.values()) / len(off)) if off else None
|
|
583
|
+
off_worst = max(off.items(), key=lambda kv: kv[1]) if off else None
|
|
584
|
+
# Keep the mean, the worst and anything materially firing; drop the
|
|
585
|
+
# long tail of zeros. 24 control sets x 290 candidate rows of "0.0" is
|
|
586
|
+
# ~7,000 lines of committed noise, and the diagnostic question --
|
|
587
|
+
# WHICH other thing does this model confuse for its own class -- is
|
|
588
|
+
# answered by the non-zero entries alone.
|
|
589
|
+
off_notable = {k: v for k, v in off.items() if v >= 0.05}
|
|
590
|
+
|
|
591
|
+
# -------- cascade gate, at the production empty-frame prior ------
|
|
592
|
+
gate = None
|
|
593
|
+
if quiet_t is not None and best_rate is not None:
|
|
594
|
+
recall = best_at_qt or 0.0
|
|
595
|
+
gate = {
|
|
596
|
+
"empty_prior": PRODUCTION_EMPTY_PRIOR,
|
|
597
|
+
"empty_prior_provenance": (
|
|
598
|
+
"cctv-detector-tier.md's production frame prior, measured on the "
|
|
599
|
+
"rideshare fleet. NOT measured for a venue; it is carried here only "
|
|
600
|
+
"so the two adjudications are comparable."),
|
|
601
|
+
"operating_conf": quiet_t,
|
|
602
|
+
"vlm_calls_saved": PRODUCTION_EMPTY_PRIOR * (1.0 - quiet_fire),
|
|
603
|
+
"events_lost": 1.0 - recall,
|
|
604
|
+
"total_calls_removed": (PRODUCTION_EMPTY_PRIOR * (1.0 - quiet_fire)
|
|
605
|
+
+ (1.0 - PRODUCTION_EMPTY_PRIOR) * (1.0 - recall)),
|
|
606
|
+
}
|
|
607
|
+
|
|
608
|
+
cands.append({
|
|
609
|
+
"model": alias, "class": cn, "class_index": int(ci),
|
|
610
|
+
# `M["info"]`, NOT the bare `info` -- that name leaks out of the
|
|
611
|
+
# loop that BUILT `models` and still points at whichever index
|
|
612
|
+
# entry it saw last, so every candidate read as shipped_bytes
|
|
613
|
+
# True. Python's loop-variable scope, caught by an elision that
|
|
614
|
+
# elided nothing.
|
|
615
|
+
"shipped_bytes": bool(M["info"].get("shipped_bytes")),
|
|
616
|
+
"by_bucket": by_bucket,
|
|
617
|
+
"n_valid_negatives": n_valid,
|
|
618
|
+
"quiet_t": quiet_t,
|
|
619
|
+
"quiet_fire_rate": quiet_fire,
|
|
620
|
+
"quiet_fire_rate_is": "the WORST valid bucket at quiet_t, not a pooled rate",
|
|
621
|
+
"quiet_fire_by_bucket": quiet_by_bucket,
|
|
622
|
+
"best_control": best_ctl, "control_n": (per_ctl.get(best_ctl) or {}).get("n"),
|
|
623
|
+
"control_rate_025": best_rate, "control_rate_at_quiet_t": best_at_qt,
|
|
624
|
+
"per_control": per_ctl,
|
|
625
|
+
"off_target_mean": off_mean,
|
|
626
|
+
"off_target_n_sets": len(off),
|
|
627
|
+
"off_target_worst": ({"set": off_worst[0], "rate": off_worst[1]} if off_worst else None),
|
|
628
|
+
"off_target_notable": off_notable,
|
|
629
|
+
"gate": gate,
|
|
630
|
+
"lat_p50_ms": M["lat_p50_ms"],
|
|
631
|
+
})
|
|
632
|
+
|
|
633
|
+
# Tie-break TOWARD the shipped bytes. A specialist appears twice -- once
|
|
634
|
+
# as `PUB__<id>` (the artifact in sensing_model_registry) and once as a
|
|
635
|
+
# re-export -- and they agree to ~1e-06, so an arbitrary tie-break would
|
|
636
|
+
# name whichever the dict happened to yield. Naming the deployed artifact
|
|
637
|
+
# is the useful half of the pair.
|
|
638
|
+
cands.sort(key=lambda c: (-(c["control_rate_at_quiet_t"] or 0), c["quiet_t"] or 1,
|
|
639
|
+
not c["shipped_bytes"], c["model"]))
|
|
640
|
+
# Per-bucket detail for the candidates the decision could turn on -- the
|
|
641
|
+
# top five and every shipped artifact -- and summary rows for the rest.
|
|
642
|
+
# All 290 candidate rows at full detail are 795 KB of committed JSON, 45%
|
|
643
|
+
# of it per-bucket cells for models that lost by 60 points. The verdict
|
|
644
|
+
# stays auditable: every candidate keeps its quiet threshold, its control
|
|
645
|
+
# recall and its valid-negative count, which is everything `decide` reads.
|
|
646
|
+
for rank, c in enumerate(cands):
|
|
647
|
+
if rank >= 5 and not c["shipped_bytes"]:
|
|
648
|
+
c["by_bucket"] = {"_elided": "summary row; rerun the tool for "
|
|
649
|
+
"per-bucket cells on every candidate"}
|
|
650
|
+
c["per_control"] = {}
|
|
651
|
+
c["gate"] = None
|
|
652
|
+
|
|
653
|
+
now = decide(cap, cands)
|
|
654
|
+
was, was_model = PRIOR_VERDICT.get(cap, (None, None))
|
|
655
|
+
rows.append({"capability": cap, "n_candidates": len(cands),
|
|
656
|
+
"prior_decision_taxi_negatives": was,
|
|
657
|
+
"prior_model": was_model,
|
|
658
|
+
"changed": bool(was is not None and was != now["decision"]),
|
|
659
|
+
"decision": now, "candidates": cands})
|
|
660
|
+
|
|
661
|
+
json.dump({
|
|
662
|
+
"schema": "trillboards.venue-adjudication/1",
|
|
663
|
+
"negative_set": "venue-phase3-source-2026-08-26",
|
|
664
|
+
"supersedes": "the 1,500-frame `production` negative set of 2026-08-25, "
|
|
665
|
+
"1,494 of which were taxi_rideshare_tv cabin interiors",
|
|
666
|
+
"bar": {
|
|
667
|
+
"quiet_max_fire": QUIET_MAX_FIRE,
|
|
668
|
+
"recall_retention": RECALL_RETENTION_BAR,
|
|
669
|
+
"off_target_mean": OFF_TARGET_BAR,
|
|
670
|
+
"why": "identical to the 2026-08-25 screen so the only variable is the negative set",
|
|
671
|
+
},
|
|
672
|
+
"composition": comp,
|
|
673
|
+
"export_reproducibility": repro,
|
|
674
|
+
"venue_empty_frame_prior": venue_prior,
|
|
675
|
+
"capabilities": rows,
|
|
676
|
+
}, open(args.out, "w"), indent=1)
|
|
677
|
+
|
|
678
|
+
if args.markdown:
|
|
679
|
+
with open(args.markdown, "w") as fh:
|
|
680
|
+
fh.write(render_markdown(rows, comp))
|
|
681
|
+
|
|
682
|
+
print(json.dumps({"capabilities": len(rows),
|
|
683
|
+
"models_adjudicated": len(models),
|
|
684
|
+
"negative_frames": int(is_neg.sum()),
|
|
685
|
+
"buckets": dict(Counter(buckets[is_neg].tolist()))}, indent=1))
|
|
686
|
+
|
|
687
|
+
|
|
688
|
+
def render_markdown(rows: list, comp: dict) -> str:
|
|
689
|
+
"""The runbook table, emitted rather than hand-typed.
|
|
690
|
+
|
|
691
|
+
Hand-copying a 35-row table into prose is how a runbook starts disagreeing
|
|
692
|
+
with its own artifact -- this file already carries a twin count that was
|
|
693
|
+
wrong in nine places for a day.
|
|
694
|
+
"""
|
|
695
|
+
out = []
|
|
696
|
+
out.append(" capability was (taxi negatives) now (venue negatives) model / why")
|
|
697
|
+
out.append(" " + "-" * 108)
|
|
698
|
+
for r in rows:
|
|
699
|
+
d = r["decision"]
|
|
700
|
+
was = r["prior_decision_taxi_negatives"] or "-"
|
|
701
|
+
now = d["decision"]
|
|
702
|
+
mark = "*" if r["changed"] else " "
|
|
703
|
+
who = d.get("model") or d.get("best_effort") or ""
|
|
704
|
+
extra = ""
|
|
705
|
+
if now == "PUBLISH":
|
|
706
|
+
c = next((c for c in r["candidates"] if c["model"] == who), None)
|
|
707
|
+
if c:
|
|
708
|
+
extra = (" conf %.3f, keeps %.0f%% of %s (n=%s), <1%% on %d venue negs"
|
|
709
|
+
% (c["quiet_t"] or 0, 100 * (c["control_rate_at_quiet_t"] or 0),
|
|
710
|
+
c["best_control"], c["control_n"], c["n_valid_negatives"]))
|
|
711
|
+
out.append(" %s%-21s %-22s %-22s %s%s"
|
|
712
|
+
% (mark, r["capability"], _short(was), _short(now), who, extra))
|
|
713
|
+
out.append("")
|
|
714
|
+
out.append(" * = the verdict moved when the negative set changed.")
|
|
715
|
+
out.append("")
|
|
716
|
+
out.append(" negative set composition (span is the source video the frames came from --")
|
|
717
|
+
out.append(" a frame count is not an independence claim):")
|
|
718
|
+
for b, v in sorted(comp["negative_set"].items(),
|
|
719
|
+
key=lambda kv: -kv[1]["source_span_hours"]):
|
|
720
|
+
out.append(" %-26s %4d frames %2d boxes %6.2f h span 1 per %7s s %s"
|
|
721
|
+
% (b, v["frames"], v["boxes"], v["source_span_hours"],
|
|
722
|
+
v["seconds_between_frames"], ",".join(v["corpora"])))
|
|
723
|
+
return "\n".join(out) + "\n"
|
|
724
|
+
|
|
725
|
+
|
|
726
|
+
def _short(v: str) -> str:
|
|
727
|
+
return {"NOT SCREENABLE HERE": "NOT SCREENABLE",
|
|
728
|
+
"AMBIGUOUS NEGATIVE SET": "AMBIGUOUS NEGSET",
|
|
729
|
+
"NO MODEL CLEARS BAR": "NO MODEL CLEARS BAR"}.get(v, v)
|
|
730
|
+
|
|
731
|
+
|
|
732
|
+
def decide(cap: str, cands: list) -> dict:
|
|
733
|
+
"""PUBLISH / NO MODEL CLEARS BAR / NOT SCREENABLE, with the reason.
|
|
734
|
+
|
|
735
|
+
Same ladder as the 2026-08-25 screen. Only the negative set differs.
|
|
736
|
+
"""
|
|
737
|
+
if not cands:
|
|
738
|
+
return {"decision": "NO CANDIDATE",
|
|
739
|
+
"why": "no sourced artifact declares a class for this capability"}
|
|
740
|
+
if all(c["n_valid_negatives"] == 0 for c in cands):
|
|
741
|
+
return {"decision": "NOT SCREENABLE HERE",
|
|
742
|
+
"why": "no bucket of the venue corpus is a valid negative for this class -- "
|
|
743
|
+
"either the plan file declares it `possible` everywhere, or the corpus "
|
|
744
|
+
"was hand-graded as containing it. Firing here can be correct, so no "
|
|
745
|
+
"false-alarm rate exists to compute."}
|
|
746
|
+
live = [c for c in cands if (c["control_rate_025"] or 0) > 0]
|
|
747
|
+
if not live:
|
|
748
|
+
if all(c["best_control"] is None for c in cands):
|
|
749
|
+
return {"decision": "UNINFORMATIVE",
|
|
750
|
+
"why": "no usable positive control exists for this class, so no firing "
|
|
751
|
+
"rate here is a finding"}
|
|
752
|
+
return {"decision": "NO MODEL CLEARS BAR",
|
|
753
|
+
"why": "no candidate fires on its positive control"}
|
|
754
|
+
retained = [c for c in live if (c["control_rate_at_quiet_t"] or 0) >= RECALL_RETENTION_BAR]
|
|
755
|
+
if not retained:
|
|
756
|
+
best = max(live, key=lambda c: (c["control_rate_at_quiet_t"] or 0))
|
|
757
|
+
r = best["control_rate_at_quiet_t"]
|
|
758
|
+
return {"decision": "NO MODEL CLEARS BAR", "best_effort": best["model"],
|
|
759
|
+
"why": "candidates are alive but collapse when forced quiet: the best (%s, "
|
|
760
|
+
"class '%s') keeps only %s of its positive-control detections at the "
|
|
761
|
+
"confidence needed for <1%% firing on the venue negatives"
|
|
762
|
+
% (best["model"], best["class"],
|
|
763
|
+
("%.0f%%" % (100 * r)) if r is not None else "n/a")}
|
|
764
|
+
ok = [c for c in retained
|
|
765
|
+
if c["off_target_mean"] is not None and c["off_target_mean"] <= OFF_TARGET_BAR]
|
|
766
|
+
if not ok:
|
|
767
|
+
b = min(retained, key=lambda c: (c["off_target_mean"] if c["off_target_mean"] is not None else 9))
|
|
768
|
+
w = b["off_target_worst"] or {}
|
|
769
|
+
return {"decision": "NO MODEL CLEARS BAR", "best_effort": b["model"],
|
|
770
|
+
"why": "candidates stay quiet on venue footage but are indiscriminate "
|
|
771
|
+
"elsewhere: the best (%s) still fires on %.0f%% of off-target control "
|
|
772
|
+
"frames at its operating confidence (worst: %.0f%% of %s)"
|
|
773
|
+
% (b["model"], 100 * (b["off_target_mean"] or 0),
|
|
774
|
+
100 * w.get("rate", 0), w.get("set", "?"))}
|
|
775
|
+
win = ok[0]
|
|
776
|
+
return {"decision": "PUBLISH", "model": win["model"], "class": win["class"],
|
|
777
|
+
"operating_conf": win["quiet_t"],
|
|
778
|
+
"control": win["best_control"], "control_n": win["control_n"],
|
|
779
|
+
"n_valid_negatives": win["n_valid_negatives"],
|
|
780
|
+
"gate": win["gate"],
|
|
781
|
+
"why": "fires on %s (%s frames) at %.0f%%, keeps %.0f%% of that at conf %.3f, "
|
|
782
|
+
"where it fires on <1%% of %d hand-declared venue negatives"
|
|
783
|
+
% (win["best_control"], win["control_n"], 100 * (win["control_rate_025"] or 0),
|
|
784
|
+
100 * (win["control_rate_at_quiet_t"] or 0), win["quiet_t"] or 0,
|
|
785
|
+
win["n_valid_negatives"]),
|
|
786
|
+
"runner_up": ok[1]["model"] if len(ok) > 1 else None}
|
|
787
|
+
|
|
788
|
+
|
|
789
|
+
if __name__ == "__main__":
|
|
790
|
+
main()
|
|
@@ -143,22 +143,44 @@ def main() -> int:
|
|
|
143
143
|
out_details = interp.get_output_details()
|
|
144
144
|
shape = tuple(int(v) for v in in_detail["shape"])
|
|
145
145
|
|
|
146
|
-
#
|
|
147
|
-
#
|
|
148
|
-
#
|
|
149
|
-
|
|
150
|
-
|
|
151
|
-
|
|
146
|
+
# READ THE INPUT LAYOUT OFF THE GRAPH, the same rule this file already applies
|
|
147
|
+
# to outputs. It used to assert NCHW and refuse anything else, on the reasoning
|
|
148
|
+
# that "the LiteRT export of an ultralytics detect model is NOT NHWC". That is
|
|
149
|
+
# true of the toolchain the 21 committed twins were exported with and it is NOT
|
|
150
|
+
# a property of LiteRT: `onnx2tf` transposes to channels-last by design, and an
|
|
151
|
+
# export made with ultralytics 8.3.145 declares [1,640,640,3]. The assertion
|
|
152
|
+
# fired on a real, correct twin (gh_fall_tonlong, 2026-08-26).
|
|
153
|
+
#
|
|
154
|
+
# The warning it carried is still right and is why this branches on the DECLARED
|
|
155
|
+
# shape rather than on a flag: transposing "because LiteRT is channels-last"
|
|
156
|
+
# feeds a rotated image to a model that wanted NCHW, and every residual then
|
|
157
|
+
# looks like export noise. Both runtimes must receive the SAME IMAGE, each in
|
|
158
|
+
# the layout its own graph declares — never the same buffer.
|
|
159
|
+
if len(shape) != 4:
|
|
160
|
+
raise ParityError(f"expected a 4-D [1,3,H,W] or [1,H,W,3] input, got {shape}")
|
|
161
|
+
if shape[1] == 3 and shape[3] != 3:
|
|
162
|
+
litert_layout, imgsz = "nchw", shape[2]
|
|
163
|
+
elif shape[3] == 3 and shape[1] != 3:
|
|
164
|
+
litert_layout, imgsz = "nhwc", shape[1]
|
|
165
|
+
elif shape[1] == 3 and shape[3] == 3:
|
|
166
|
+
# [1,3,H,3] is ambiguous and no real detector input is 3 px wide.
|
|
167
|
+
raise ParityError(f"input {shape} is ambiguous between NCHW and NHWC")
|
|
168
|
+
else:
|
|
169
|
+
raise ParityError(f"no channel axis of size 3 in {shape}")
|
|
152
170
|
|
|
153
171
|
if args.image:
|
|
154
172
|
from PIL import Image
|
|
155
173
|
|
|
156
174
|
pixels = np.asarray(Image.open(args.image).convert("RGB").resize((imgsz, imgsz)))
|
|
157
|
-
|
|
175
|
+
nchw = (pixels.astype(np.float32) / 255.0).transpose(2, 0, 1)[None]
|
|
158
176
|
else:
|
|
159
|
-
|
|
177
|
+
nchw = np.random.default_rng(args.seed).random((1, 3, imgsz, imgsz)).astype(np.float32)
|
|
178
|
+
|
|
179
|
+
# ONNX always gets NCHW; LiteRT gets whatever it declared, from the same pixels.
|
|
180
|
+
x = nchw
|
|
181
|
+
x_litert = nchw if litert_layout == "nchw" else nchw.transpose(0, 2, 3, 1).copy()
|
|
160
182
|
|
|
161
|
-
interp.set_tensor(in_detail["index"],
|
|
183
|
+
interp.set_tensor(in_detail["index"], x_litert)
|
|
162
184
|
interp.invoke()
|
|
163
185
|
litert_out = interp.get_tensor(out_details[0]["index"])
|
|
164
186
|
|
|
@@ -173,11 +195,13 @@ def main() -> int:
|
|
|
173
195
|
result["reason"] = why
|
|
174
196
|
result["onnx"] = Path(args.onnx).name
|
|
175
197
|
result["litert"] = Path(args.litert).name
|
|
198
|
+
result["litert_input_layout"] = litert_layout
|
|
176
199
|
|
|
177
200
|
if args.json:
|
|
178
201
|
print(json.dumps(result, indent=2))
|
|
179
202
|
else:
|
|
180
|
-
print(f"{'PASS' if ok else 'FAIL'} {result['onnx']} layout={layout}
|
|
203
|
+
print(f"{'PASS' if ok else 'FAIL'} {result['onnx']} layout={layout} "
|
|
204
|
+
f"litert_in={litert_layout} {why}")
|
|
181
205
|
print(f" box x{result['box_scale_applied']:g}: {result['box_residual_px']:.4g} px"
|
|
182
206
|
f" (unscaled {result['box_residual_unscaled']:.4g})")
|
|
183
207
|
print(f" cls : {result['cls_residual']:.4g}")
|
|
@@ -523,7 +523,28 @@ PPE_KILLUMINATI_19CLS_V8S = ModelVocabulary(
|
|
|
523
523
|
#: this artifact as the fixture that exercises `_verify_class_axis_width`, where
|
|
524
524
|
#: `4 + 7 = 11` matches its real `[1, 11, 8400]` head and `4 + 6 = 10` would
|
|
525
525
|
#: reject it.
|
|
526
|
-
|
|
526
|
+
#: Person Down Detector — published 2026-08-27 by
|
|
527
|
+
#: scripts/sensing/publish-sensing-model.js. The class list is the artifact's own
|
|
528
|
+
#: (graph metadata key 'names'); the footprint is MEASURED, 817.3594 MiB after a
|
|
529
|
+
#: probe inference via rusage-maxrss, not an estimate from the file size.
|
|
530
|
+
#: Coordinate space MEASURED, not assumed: max|box channel| = 1.5467 over 94 real frames; / imgsz 640 = 0.0024. pixels at >= 0.2, normalized at <= 0.02, refused in between.
|
|
531
|
+
FALL_1CLS_11X_CLASS_NAMES: "MappingProxyType[int, str]" = MappingProxyType({
|
|
532
|
+
0: "fall detected",
|
|
533
|
+
})
|
|
534
|
+
|
|
535
|
+
FALL_1CLS_11X = ModelVocabulary(
|
|
536
|
+
model_name="fall_1cls_11x",
|
|
537
|
+
slot_names=FALL_1CLS_11X_CLASS_NAMES,
|
|
538
|
+
logit_slots=1,
|
|
539
|
+
approx_ram_bytes=857063451,
|
|
540
|
+
onnx_variant=VariantContract(
|
|
541
|
+
output_layout="anchor_4_plus_nc",
|
|
542
|
+
box_units="normalized",
|
|
543
|
+
box_scale_to_pixels=640.0,
|
|
544
|
+
),
|
|
545
|
+
)
|
|
546
|
+
|
|
547
|
+
BUILTIN_VOCABULARIES: "tuple[ModelVocabulary, ...]" = (RFDETR_MEDIUM, GH_FORKLIFT_COMBINE, PPE_VYRA_14CLS_V8M, PPE_CONSTRUCTION_17CLS_V8N, PPE_KILLUMINATI_19CLS_V8S, FALL_1CLS_11X,)
|
|
527
548
|
|
|
528
549
|
|
|
529
550
|
class ClassIndex:
|
|
@@ -617,6 +638,7 @@ class ClassIndex:
|
|
|
617
638
|
|
|
618
639
|
|
|
619
640
|
__all__ = [
|
|
641
|
+
"FALL_1CLS_11X",
|
|
620
642
|
"PPE_KILLUMINATI_19CLS_V8S",
|
|
621
643
|
"PPE_CONSTRUCTION_17CLS_V8N",
|
|
622
644
|
"PPE_VYRA_14CLS_V8M",
|
package/dist/EdgeAgent.js
CHANGED
|
@@ -44,7 +44,7 @@ const edge_ads_1 = require("@trillboards/edge-ads");
|
|
|
44
44
|
const CommandRouter_1 = require("./CommandRouter");
|
|
45
45
|
const config_1 = require("./config");
|
|
46
46
|
const dataDir_1 = require("./dataDir");
|
|
47
|
-
const SDK_VERSION = '0.2.
|
|
47
|
+
const SDK_VERSION = '0.2.107';
|
|
48
48
|
class EdgeAgent extends events_1.EventEmitter {
|
|
49
49
|
constructor(config, deps) {
|
|
50
50
|
super();
|
package/dist/diagnose.d.ts
CHANGED
|
@@ -31,7 +31,7 @@ export interface DiagnosticReport {
|
|
|
31
31
|
skipped: number;
|
|
32
32
|
};
|
|
33
33
|
}
|
|
34
|
-
declare const SDK_VERSION = "0.2.
|
|
34
|
+
declare const SDK_VERSION = "0.2.107";
|
|
35
35
|
declare function checkNodeVersion(): CheckResult;
|
|
36
36
|
declare function checkPlatform(): CheckResult;
|
|
37
37
|
declare function checkMemory(): CheckResult;
|
package/dist/diagnose.js
CHANGED
|
@@ -63,7 +63,7 @@ const localNetworkAccess_1 = require("./localNetworkAccess");
|
|
|
63
63
|
// ---------------------------------------------------------------------------
|
|
64
64
|
// SDK version — mirrors package.json; kept in sync by CI
|
|
65
65
|
// ---------------------------------------------------------------------------
|
|
66
|
-
const SDK_VERSION = '0.2.
|
|
66
|
+
const SDK_VERSION = '0.2.107';
|
|
67
67
|
exports._SDK_VERSION = SDK_VERSION;
|
|
68
68
|
// ---------------------------------------------------------------------------
|
|
69
69
|
// ANSI helpers (no external deps)
|
package/package.json
CHANGED