@trillboards/edge-sdk 0.2.89 → 0.2.91

package/deploy/detector/README.md

@@ -51,12 +51,44 @@ what is resident on that box right now.
 
 ```
 labels.py     one committed ModelVocabulary per model (slots -> class names,
-              logit width, measured RAM). ClassIndex is the UNION over them.
+              logit width, measured RAM, and the VariantContract naming what
+              its ONNX output MEANS). ClassIndex is the UNION over them.
+decode.py     the three output layouts and the arithmetic each needs, plus the
+              only IoU/NMS in the Python tree. NUMPY ONLY — no ORT, no Pillow.
 registry.py   name -> session map, the class index, lazy load, LRU eviction
               under a RAM budget, and the merge. STDLIB ONLY.
 loader.py     the poll loop + the real ORT session factory and probe frame.
 ```
 
+### Output layouts
+
+Three, and **which one an artifact uses is DECLARED, never sniffed**. The
+declaration is a `VariantContract` on the model's vocabulary — the box-side copy
+of that artifact's `(model_id, format)` row in `model_platform_variants`
+(`output_layout`, `box_units`, `box_scale_to_pixels`) — and `_read_graph_spec`
+CHECKS the loaded graph against it and refuses on disagreement.
+
+| layout | shape | needs NMS | preprocessing |
+|---|---|---|---|
+| `rfdetr_dets_labels` | `dets [1,Q,4]` + `labels [1,Q,C]` | no (set prediction) | ImageNet-normalised **stretch** |
+| `anchor_4_plus_nc` | `[1, 4+nc, A]`, scores already sigmoid | **yes** | Ultralytics **letterbox**, 114-grey, /255 |
+| `end2end_n_6` | `[1, N, 6]` = x1,y1,x2,y2,score,cls | no (already suppressed) | Ultralytics **letterbox**, 114-grey, /255 |
+
+Sniffing is forbidden because `[1, 300, 6]` and `[1, 6, 8400]` are both rank-3
+with a 6 in them: a reader that gets one backwards reads x1,y1,x2,y2,score,cls
+as four box channels and two classes, with no exception and a plausible table.
+
+**`box_units` is a property of the FORMAT, not of the model.** Ultralytics wraps
+a `task=detect`, `end2end=False` TFLite export in `_NormalizeCoords`, dividing
+the box channels by (W, H). Measured across all 17 committed LiteRT twins: 16 of
+17 have ONNX boxes in pixels 0..640 and LiteRT boxes in 0..1, agreeing to
+0.00037 px after multiplying by 640, with class scores agreeing to 5.5e-06.
+`queue_supermarket`, the one `end2end_n_6` twin, is pixels on both sides. Every
+other check passes on such a pair — same shape, same dtype, both load, both run
+— so a decoder that assumes pixels draws every box 640x too small in silence.
+The two preprocessors are likewise NOT interchangeable and `/health` reports
+which one ran, next to `output_layout`.
+
 * **A class routes to the FIRST catalogue model that names it.** Adding a
   specialist can never move an existing class off the model it was scored on —
   `person` stays on `rfdetr_medium` even after a PPE model that also emits it.
@@ -181,12 +213,13 @@ python3 -m unittest discover -s . -p "test_*.py" -v
 
 Runs in `guards-fast`. `test_registry.py` covers the whole residency policy —
 routing, lazy load, eviction, and every refusal — with FAKE sessions, so it runs
-there too. **Everything except the post-processing cases needs
-nothing but the standard library** — those are the only ones that touch numpy,
-and they skip themselves with a stated reason when it is absent. That invariant
-is the durable claim; the counts (45 of 60 on 2026-08-24) are a snapshot and go
-stale the next time a case is added, so derive them by running it rather than
-reading them here.
+there too. **The wire contract and the residency policy need nothing but the
+standard library**; `test_decode.py` needs numpy and nothing else (`decode.py`
+imports no ORT, deliberately, so the arithmetic is exercisable on any runner
+that can `import numpy`), and `test_session.py` needs onnxruntime too because
+`session.py` imports it. Each skips itself with a stated reason. That layering
+is the durable claim; any count is a snapshot and goes stale the next time a
+case is added, so derive it by running rather than reading it here.
 
 ```bash
 # The real client against the real service, over loopback.
@@ -218,3 +251,39 @@ uv pip install --python /tmp/rfdetr-ref/bin/python rfdetr onnx onnxruntime pillo
 Runs both pipelines on one image and fails on any disagreement. A documented
 manual run, not a CI step — it needs a checkpoint download and ~3 GB of
 dependencies.
+
+`bench/verify_detector_decode.py` is the same control for the two YOLO layouts,
+against `ultralytics` itself, in three arms that each hold one variable:
+
+```bash
+uv venv /tmp/yolo-ref --python 3.11
+uv pip install --python /tmp/yolo-ref/bin/python ultralytics onnxruntime pillow ai-edge-litert
+
+# decode: SAME raw tensor, our decode vs ultralytics NMS + scale_boxes.
+/tmp/yolo-ref/bin/python bench/verify_detector_decode.py --arm decode \
+  --models-dir /path/to/onnx --frames-dir /path/to/frames --tolerance-px 0.0
+
+# pipeline: the whole service path vs YOLO(...).predict, resize kernel held constant.
+/tmp/yolo-ref/bin/python bench/verify_detector_decode.py --arm pipeline \
+  --models-dir /path/to/onnx --frames-dir /path/to/frames --tolerance-px 0.001
+
+# litert: the same weights through both runtimes, in PIXELS, per the DECLARED scale.
+/tmp/yolo-ref/bin/python bench/verify_detector_decode.py --arm litert \
+  --models-dir /path/to/onnx --frames-dir /path/to/frames \
+  --litert-manifest /path/to/litert/MANIFEST.json --tolerance-px 0.01
+```
+
+Two traps this bench cost, both of them the instrument rather than the decode,
+and both now closed in the file:
+
+* **The reference must be the ONNX, not the `.pt`** — Ultralytics letterboxes a
+  checkpoint to 640x480 while the graph is fixed 640x640, so the two arms
+  disagree by up to 0.25 confidence in both directions and it reads as a decode
+  bug.
+* **Hold the resize kernel constant too.** Pillow's BILINEAR low-passes on
+  downscale, cv2's `INTER_LINEAR` does not, and on one 1200x900 frame that moved
+  a class from 0.18 to 0.51 — 20 of 378 presence decisions apart. The
+  `pipeline` arm therefore letterboxes ONCE and feeds both sides those bytes,
+  and it feeds the reference the pixels that came back OUT of the JPEG rather
+  than the canvas that went in, because JPEG at quality 100 is still lossy
+  enough to move a borderline box.

package/deploy/detector/bench/run_candidate.py

@@ -85,8 +85,10 @@ sys.path.insert(0, str(Path(__file__).resolve().parent.parent))  # noqa: E402
 
 from trillboards_detector.errors import ModelContractError  # noqa: E402
 from trillboards_detector.labels import (  # noqa: E402
+    BOX_UNITS_PIXELS,
     BUILTIN_VOCABULARIES,
     ModelVocabulary,
+    VariantContract,
     normalize_class_name,
 )
 from trillboards_detector.registry import read_process_rss  # noqa: E402
@@ -497,13 +499,24 @@ class ShippedRunner:
     """Inference through `OnnxDetector` — the class the connector actually runs."""
 
     path_kind = "shipped"
-    preprocessing = "session.preprocess (imagenet-normalised stretch)"
     variant_format = "onnx"
-    layout = "rfdetr_dets_labels"
 
     def __init__(self, model_path: Path, vocabulary: ModelVocabulary):
         self.detector = OnnxDetector(model_path, vocabulary)
         self.vocabulary = vocabulary
+        # READ OFF THE LOADED GRAPH, never a constant. These two feed the run
+        # file's `variant.output_layout` / `variant.end2end` and its
+        # `preprocessing`, and this class stopped being RF-DETR-only the moment
+        # the service learned the two Ultralytics layouts — a hardcoded
+        # `rfdetr_dets_labels` would publish a score card whose declared layout
+        # is wrong for every YOLO artifact, which is the exact defect the
+        # declaration exists to prevent.
+        self.layout = self.detector.spec.output_layout
+        self.preprocessing = (
+            "session.preprocess (imagenet-normalised stretch)"
+            if self.layout == "rfdetr_dets_labels"
+            else "session.preprocess_letterbox (ultralytics letterbox, 114-grey, /255)"
+        )
         self.width = self.detector.spec.width
         self.height = self.detector.spec.height
         self.providers = list(self.detector.spec.providers)
@@ -575,7 +588,9 @@ class Frame:
         self._image = None
 
 
-def synthesise_vocabulary(name: str, class_names: "dict[int, str]") -> ModelVocabulary:
+def synthesise_vocabulary(
+    name: str, class_names: "dict[int, str]", output_layout: str
+) -> ModelVocabulary:
     """A ModelVocabulary over a model's OWN names, canonicalised the service's way.
 
     Synthesised rather than committed so the loadability check is a FAIR one: it
@@ -594,7 +609,35 @@ def synthesise_vocabulary(name: str, class_names: "dict[int, str]") -> ModelVoca
             continue
         slots[idx] = canonical
     return ModelVocabulary(
-        model_name=name, slot_names=slots, logit_slots=width, approx_ram_bytes=1
+        model_name=name,
+        slot_names=slots,
+        logit_slots=width,
+        approx_ram_bytes=1,
+        # THE LAYOUT IS DERIVED FROM THE GRAPH, and it is the only half
+        # `_read_graph_spec` reads — so the probe declares what it just measured
+        # rather than a guess, and an artifact whose layout this build cannot name
+        # fails `VariantContract` here and is recorded as not loadable, with the
+        # reason, which is the right answer to "would the service load this?".
+        #
+        # THE UNITS ARE THE IDENTITY, AND THAT IS THE MEASUREMENT, not a claim
+        # about the artifact. They cannot be known here — `classify_box_units`
+        # needs a forward pass, which happens later and lands in the run file's
+        # `variant` block — and one caller of this function (`ShippedRunner`)
+        # DOES decode with them. `pixels`/1.0 is the only pair that scales
+        # nothing, so the graph's own box magnitudes reach `box_abs_max`
+        # untouched and the classifier sees what the model actually emitted.
+        # Declaring the other pair would multiply by imgsz the very quantity
+        # being measured, and the verdict would then be a function of the guess.
+        #
+        # This is right for ONNX specifically: measured across all 17 committed
+        # twins, every ONNX side is pixels and only the LiteRT sides are
+        # normalised — and `.tflite` never reaches here, it routes to
+        # `LitertRunner`. A normalised ONNX artifact would still be REPORTED
+        # correctly by `classify_box_units`, because the identity leaves its
+        # 0..1 boxes as 0..1; what it would lose is per-class detections on
+        # padded frames, which is the signal to read the units verdict before
+        # trusting that candidate's score.
+        onnx_variant=VariantContract(output_layout, BOX_UNITS_PIXELS, 1.0),
     )
 
 
@@ -613,6 +656,7 @@ def probe_loadability(model_path: Path, class_names: "dict[int, str]") -> dict:
                     f"the detector service builds an onnxruntime InferenceSession and this is a "
                     f"{model_path.suffix} artifact: {str(exc).splitlines()[0][:200]}"
                 ),
+                "output_layout": None,
             }
 
     opts = ort.SessionOptions()
@@ -620,15 +664,28 @@ def probe_loadability(model_path: Path, class_names: "dict[int, str]") -> dict:
     session = ort.InferenceSession(
         str(model_path), sess_options=opts, providers=["CPUExecutionProvider"]
     )
+    # DERIVED ONCE AND RETURNED, because the only other way for a caller to learn
+    # it is to open a second session — and the caller that needs it sits between
+    # the RSS baseline and `rss_after_session`, where a transient session is
+    # exactly the contamination `introspect_in_subprocess` exists to avoid.
+    layout = classify_output_layout([list(o.shape) for o in session.get_outputs()])
     try:
-        vocabulary = synthesise_vocabulary("probe-only", class_names)
+        vocabulary = synthesise_vocabulary("probe-only", class_names, layout)
     except ValueError as exc:
-        return {"shipped_service_loadable": False, "refusal": f"vocabulary: {exc}"}
+        return {
+            "shipped_service_loadable": False,
+            "refusal": f"vocabulary: {exc}",
+            "output_layout": layout,
+        }
     try:
         _read_graph_spec(session, vocabulary)
     except ModelContractError as exc:
-        return {"shipped_service_loadable": False, "refusal": str(exc)}
-    return {"shipped_service_loadable": True, "refusal": None}
+        return {
+            "shipped_service_loadable": False,
+            "refusal": str(exc),
+            "output_layout": layout,
+        }
+    return {"shipped_service_loadable": True, "refusal": None, "output_layout": layout}
 
 
 def introspect_in_subprocess(model_path: Path) -> dict:
@@ -759,7 +816,10 @@ def main() -> int:
     elif model_path.suffix.lower() == ".tflite":
         runner = LitertRunner(model_path, declared_names)
     elif loadability["shipped_service_loadable"]:
-        runner = ShippedRunner(model_path, synthesise_vocabulary(args.candidate, declared_names))
+        runner = ShippedRunner(
+            model_path,
+            synthesise_vocabulary(args.candidate, declared_names, loadability["output_layout"]),
+        )
     else:
         runner = HarnessYoloRunner(model_path, declared_names)