@trillboards/edge-sdk 0.2.89 → 0.2.92

package/deploy/detector/README.md

@@ -51,12 +51,44 @@ what is resident on that box right now.
 
 ```
 labels.py     one committed ModelVocabulary per model (slots -> class names,
-              logit width, measured RAM). ClassIndex is the UNION over them.
+              logit width, measured RAM, and the VariantContract naming what
+              its ONNX output MEANS). ClassIndex is the UNION over them.
+decode.py     the three output layouts and the arithmetic each needs, plus the
+              only IoU/NMS in the Python tree. NUMPY ONLY — no ORT, no Pillow.
 registry.py   name -> session map, the class index, lazy load, LRU eviction
               under a RAM budget, and the merge. STDLIB ONLY.
 loader.py     the poll loop + the real ORT session factory and probe frame.
 ```
 
+### Output layouts
+
+Three, and **which one an artifact uses is DECLARED, never sniffed**. The
+declaration is a `VariantContract` on the model's vocabulary — the box-side copy
+of that artifact's `(model_id, format)` row in `model_platform_variants`
+(`output_layout`, `box_units`, `box_scale_to_pixels`) — and `_read_graph_spec`
+CHECKS the loaded graph against it and refuses on disagreement.
+
+| layout | shape | needs NMS | preprocessing |
+|---|---|---|---|
+| `rfdetr_dets_labels` | `dets [1,Q,4]` + `labels [1,Q,C]` | no (set prediction) | ImageNet-normalised **stretch** |
+| `anchor_4_plus_nc` | `[1, 4+nc, A]`, scores already sigmoid | **yes** | Ultralytics **letterbox**, 114-grey, /255 |
+| `end2end_n_6` | `[1, N, 6]` = x1,y1,x2,y2,score,cls | no (already suppressed) | Ultralytics **letterbox**, 114-grey, /255 |
+
+Sniffing is forbidden because `[1, 300, 6]` and `[1, 6, 8400]` are both rank-3
+with a 6 in them: a reader that gets one backwards reads x1,y1,x2,y2,score,cls
+as four box channels and two classes, with no exception and a plausible table.
+
+**`box_units` is a property of the FORMAT, not of the model.** Ultralytics wraps
+a `task=detect`, `end2end=False` TFLite export in `_NormalizeCoords`, dividing
+the box channels by (W, H). Measured across all 17 committed LiteRT twins: 16 of
+17 have ONNX boxes in pixels 0..640 and LiteRT boxes in 0..1, agreeing to
+0.00037 px after multiplying by 640, with class scores agreeing to 5.5e-06.
+`queue_supermarket`, the one `end2end_n_6` twin, is pixels on both sides. Every
+other check passes on such a pair — same shape, same dtype, both load, both run
+— so a decoder that assumes pixels draws every box 640x too small in silence.
+The two preprocessors are likewise NOT interchangeable and `/health` reports
+which one ran, next to `output_layout`.
+
 * **A class routes to the FIRST catalogue model that names it.** Adding a
   specialist can never move an existing class off the model it was scored on —
   `person` stays on `rfdetr_medium` even after a PPE model that also emits it.
@@ -181,12 +213,13 @@ python3 -m unittest discover -s . -p "test_*.py" -v
 
 Runs in `guards-fast`. `test_registry.py` covers the whole residency policy —
 routing, lazy load, eviction, and every refusal — with FAKE sessions, so it runs
-there too. **Everything except the post-processing cases needs
-nothing but the standard library** — those are the only ones that touch numpy,
-and they skip themselves with a stated reason when it is absent. That invariant
-is the durable claim; the counts (45 of 60 on 2026-08-24) are a snapshot and go
-stale the next time a case is added, so derive them by running it rather than
-reading them here.
+there too. **The wire contract and the residency policy need nothing but the
+standard library**; `test_decode.py` needs numpy and nothing else (`decode.py`
+imports no ORT, deliberately, so the arithmetic is exercisable on any runner
+that can `import numpy`), and `test_session.py` needs onnxruntime too because
+`session.py` imports it. Each skips itself with a stated reason. That layering
+is the durable claim; any count is a snapshot and goes stale the next time a
+case is added, so derive it by running rather than reading it here.
 
 ```bash
 # The real client against the real service, over loopback.
@@ -218,3 +251,39 @@ uv pip install --python /tmp/rfdetr-ref/bin/python rfdetr onnx onnxruntime pillo
 Runs both pipelines on one image and fails on any disagreement. A documented
 manual run, not a CI step — it needs a checkpoint download and ~3 GB of
 dependencies.
+
+`bench/verify_detector_decode.py` is the same control for the two YOLO layouts,
+against `ultralytics` itself, in three arms that each hold one variable:
+
+```bash
+uv venv /tmp/yolo-ref --python 3.11
+uv pip install --python /tmp/yolo-ref/bin/python ultralytics onnxruntime pillow ai-edge-litert
+
+# decode: SAME raw tensor, our decode vs ultralytics NMS + scale_boxes.
+/tmp/yolo-ref/bin/python bench/verify_detector_decode.py --arm decode \
+  --models-dir /path/to/onnx --frames-dir /path/to/frames --tolerance-px 0.0
+
+# pipeline: the whole service path vs YOLO(...).predict, resize kernel held constant.
+/tmp/yolo-ref/bin/python bench/verify_detector_decode.py --arm pipeline \
+  --models-dir /path/to/onnx --frames-dir /path/to/frames --tolerance-px 0.001
+
+# litert: the same weights through both runtimes, in PIXELS, per the DECLARED scale.
+/tmp/yolo-ref/bin/python bench/verify_detector_decode.py --arm litert \
+  --models-dir /path/to/onnx --frames-dir /path/to/frames \
+  --litert-manifest /path/to/litert/MANIFEST.json --tolerance-px 0.01
+```
+
+Two traps this bench cost, both of them the instrument rather than the decode,
+and both now closed in the file:
+
+* **The reference must be the ONNX, not the `.pt`** — Ultralytics letterboxes a
+  checkpoint to 640x480 while the graph is fixed 640x640, so the two arms
+  disagree by up to 0.25 confidence in both directions and it reads as a decode
+  bug.
+* **Hold the resize kernel constant too.** Pillow's BILINEAR low-passes on
+  downscale, cv2's `INTER_LINEAR` does not, and on one 1200x900 frame that moved
+  a class from 0.18 to 0.51 — 20 of 378 presence decisions apart. The
+  `pipeline` arm therefore letterboxes ONCE and feeds both sides those bytes,
+  and it feeds the reference the pixels that came back OUT of the JPEG rather
+  than the canvas that went in, because JPEG at quality 100 is still lossy
+  enough to move a borderline box.

package/deploy/detector/bench/box_units.py

@@ -0,0 +1,71 @@
+"""What a box coordinate MEANS, decided from a forward pass — STDLIB ONLY.
+
+Split out of `run_candidate.py` so the `detector-service` CI lane can reach it.
+That lane runs `python3 -m unittest discover` with NO pip install, and
+`run_candidate.py` imports `trillboards_detector.session` and therefore
+onnxruntime at module scope — so a test importing it does not skip, it errors
+the whole lane during discovery. The decision lives here in stdlib; the forward
+pass that produces `box_abs_max` stays there.
+
+Same split, and the same reason, as `bench/twin_parity_contract.py`.
+"""
+
+from __future__ import annotations
+
+
+#: LiteRT boxes in 0..1, agreeing to 0.00266 px after multiplying by 640, while
+#: their CLASS channels agree to 1.2e-07. `queue_supermarket`, the one end2end
+#: twin, agrees on both sides — end2end heads emit pixels everywhere.
+#:
+#: Every check a publish gate would otherwise run PASSES on that pair: same
+#: output shape, same dtype, loads, runs, class scores identical. An Android
+#: post-processor reusing the ONNX decode draws every box 640x too small and
+#: nothing reports an error. So the units are MEASURED from a forward pass, per
+#: artifact, and recorded — never carried across from the sibling format.
+#:
+#: The discriminator is a ratio, max|box channel| / imgsz, and it is decisive by
+#: two and a half orders of magnitude. Measured over 119 ONNX artifacts the ratio
+#: spans 0.988..1.167; a normalised head sits at ~1/640 = 0.0016. The dead zone
+#: below is EMPTY across every artifact in the tree, and an artifact landing in it
+#: is refused rather than guessed at — that is the whole point of the field.
+BOX_UNITS_PIXEL_RATIO_MIN = 0.2
+BOX_UNITS_NORMALIZED_RATIO_MAX = 0.02
+
+
+def classify_box_units(box_abs_max: float, imgsz: int) -> dict:
+    """pixels, normalized, or REFUSE — never a guess.
+
+    Returns the verdict plus the evidence, because a bare enum in a score card is
+    a claim and the ratio is what makes it checkable.
+    """
+    # `<= 0.0` MATTERS AS MUCH AS `is None`, and it is the case that actually
+    # happens. A model whose classification head is dead still emits box rows —
+    # `forklift_akdve_11n` has healthy boxes (2.93-636 px) and a max class score
+    # of 0.0063 — so nothing is admitted at threshold, `box_abs_max` stays at its
+    # 0.0 seed, and the ratio 0.0/imgsz lands under BOX_UNITS_NORMALIZED_RATIO_MAX.
+    # The old predicate then returned a CONFIDENT `normalized` with
+    # scale_to_pixels=640 from zero observations, and that declaration is exactly
+    # what `session.py` now trusts absolutely: the layout and units are DECLARED
+    # by the variant contract and never re-derived from the tensor. So a dead
+    # artifact published a coordinate-space claim no frame supports.
+    #
+    # The refusal string below already described this case. It just could not be
+    # reached, which is the whole shape of the defect this function exists to
+    # prevent — a check that runs, passes, and measures nothing.
+    if imgsz <= 0 or box_abs_max is None or box_abs_max <= 0.0:
+        return {"units": None, "ratio": None, "scale_to_pixels": None,
+                "refusal": "no box channels were observed, so the coordinate space is unmeasured"}
+    ratio = box_abs_max / imgsz
+    if ratio >= BOX_UNITS_PIXEL_RATIO_MIN:
+        return {"units": "pixels", "ratio": ratio, "scale_to_pixels": 1.0, "refusal": None}
+    if ratio <= BOX_UNITS_NORMALIZED_RATIO_MAX:
+        return {"units": "normalized", "ratio": ratio, "scale_to_pixels": float(imgsz), "refusal": None}
+    return {
+        "units": None, "ratio": ratio, "scale_to_pixels": None,
+        "refusal": (
+            f"max|box|/imgsz = {ratio:.4f} falls between {BOX_UNITS_NORMALIZED_RATIO_MAX} and "
+            f"{BOX_UNITS_PIXEL_RATIO_MIN}, where neither pixels nor normalised is credible. "
+            "Measured over 119 artifacts this band is empty; refusing rather than picking one, "
+            "because the wrong pick is silent and scales every box by 640."
+        ),
+    }

package/deploy/detector/bench/run_candidate.py

@@ -82,11 +82,19 @@ import time
 from pathlib import Path
 
 sys.path.insert(0, str(Path(__file__).resolve().parent.parent))  # noqa: E402
+sys.path.insert(0, str(Path(__file__).resolve().parent))  # noqa: E402  (bench siblings)
 
+from box_units import (  # noqa: E402  (sibling module, stdlib-only on purpose)
+    BOX_UNITS_NORMALIZED_RATIO_MAX,
+    BOX_UNITS_PIXEL_RATIO_MIN,
+    classify_box_units,
+)
 from trillboards_detector.errors import ModelContractError  # noqa: E402
 from trillboards_detector.labels import (  # noqa: E402
+    BOX_UNITS_PIXELS,
     BUILTIN_VOCABULARIES,
     ModelVocabulary,
+    VariantContract,
     normalize_class_name,
 )
 from trillboards_detector.registry import read_process_rss  # noqa: E402
@@ -105,23 +113,7 @@ SCHEMA = "detector-candidate-run/2"
 #: Ultralytics' TFLite export path wraps a `task=detect`, `end2end=False` model in
 #: `_NormalizeCoords`, dividing the four box channels by (W, H). Measured here on
 #: all 17 committed LiteRT twins: 16 of 17 have ONNX boxes in pixels 0..640 and
-#: LiteRT boxes in 0..1, agreeing to 0.00266 px after multiplying by 640, while
-#: their CLASS channels agree to 1.2e-07. `queue_supermarket`, the one end2end
-#: twin, agrees on both sides — end2end heads emit pixels everywhere.
-#:
-#: Every check a publish gate would otherwise run PASSES on that pair: same
-#: output shape, same dtype, loads, runs, class scores identical. An Android
-#: post-processor reusing the ONNX decode draws every box 640x too small and
-#: nothing reports an error. So the units are MEASURED from a forward pass, per
-#: artifact, and recorded — never carried across from the sibling format.
-#:
-#: The discriminator is a ratio, max|box channel| / imgsz, and it is decisive by
-#: two and a half orders of magnitude. Measured over 119 ONNX artifacts the ratio
-#: spans 0.988..1.167; a normalised head sits at ~1/640 = 0.0016. The dead zone
-#: below is EMPTY across every artifact in the tree, and an artifact landing in it
-#: is refused rather than guessed at — that is the whole point of the field.
-BOX_UNITS_PIXEL_RATIO_MIN = 0.2
-BOX_UNITS_NORMALIZED_RATIO_MAX = 0.02
+
 
 #: Warm-up forward passes before any timing is kept. ORT's first run pays for
 #: arena allocation and kernel selection; folding that into a p50 over 134 frames
@@ -269,31 +261,6 @@ def classify_output_layout(output_shapes: "list[list]") -> str:
     return f"unknown_rank_{len(shape)}"
 
 
-def classify_box_units(box_abs_max: float, imgsz: int) -> dict:
-    """pixels, normalized, or REFUSE — never a guess.
-
-    Returns the verdict plus the evidence, because a bare enum in a score card is
-    a claim and the ratio is what makes it checkable.
-    """
-    if imgsz <= 0 or box_abs_max is None:
-        return {"units": None, "ratio": None, "scale_to_pixels": None,
-                "refusal": "no box channels were observed, so the coordinate space is unmeasured"}
-    ratio = box_abs_max / imgsz
-    if ratio >= BOX_UNITS_PIXEL_RATIO_MIN:
-        return {"units": "pixels", "ratio": ratio, "scale_to_pixels": 1.0, "refusal": None}
-    if ratio <= BOX_UNITS_NORMALIZED_RATIO_MAX:
-        return {"units": "normalized", "ratio": ratio, "scale_to_pixels": float(imgsz), "refusal": None}
-    return {
-        "units": None, "ratio": ratio, "scale_to_pixels": None,
-        "refusal": (
-            f"max|box|/imgsz = {ratio:.4f} falls between {BOX_UNITS_NORMALIZED_RATIO_MAX} and "
-            f"{BOX_UNITS_PIXEL_RATIO_MIN}, where neither pixels nor normalised is credible. "
-            "Measured over 119 artifacts this band is empty; refusing rather than picking one, "
-            "because the wrong pick is silent and scales every box by 640."
-        ),
-    }
-
-
 def letterbox(image, width: int, height: int):
     """Ultralytics' resize: scale to fit, pad the remainder 114-grey, centred.
 
@@ -497,13 +464,24 @@ class ShippedRunner:
     """Inference through `OnnxDetector` — the class the connector actually runs."""
 
     path_kind = "shipped"
-    preprocessing = "session.preprocess (imagenet-normalised stretch)"
     variant_format = "onnx"
-    layout = "rfdetr_dets_labels"
 
     def __init__(self, model_path: Path, vocabulary: ModelVocabulary):
         self.detector = OnnxDetector(model_path, vocabulary)
         self.vocabulary = vocabulary
+        # READ OFF THE LOADED GRAPH, never a constant. These two feed the run
+        # file's `variant.output_layout` / `variant.end2end` and its
+        # `preprocessing`, and this class stopped being RF-DETR-only the moment
+        # the service learned the two Ultralytics layouts — a hardcoded
+        # `rfdetr_dets_labels` would publish a score card whose declared layout
+        # is wrong for every YOLO artifact, which is the exact defect the
+        # declaration exists to prevent.
+        self.layout = self.detector.spec.output_layout
+        self.preprocessing = (
+            "session.preprocess (imagenet-normalised stretch)"
+            if self.layout == "rfdetr_dets_labels"
+            else "session.preprocess_letterbox (ultralytics letterbox, 114-grey, /255)"
+        )
         self.width = self.detector.spec.width
         self.height = self.detector.spec.height
         self.providers = list(self.detector.spec.providers)
@@ -575,7 +553,9 @@ class Frame:
         self._image = None
 
 
-def synthesise_vocabulary(name: str, class_names: "dict[int, str]") -> ModelVocabulary:
+def synthesise_vocabulary(
+    name: str, class_names: "dict[int, str]", output_layout: str
+) -> ModelVocabulary:
     """A ModelVocabulary over a model's OWN names, canonicalised the service's way.
 
     Synthesised rather than committed so the loadability check is a FAIR one: it
@@ -594,7 +574,35 @@ def synthesise_vocabulary(name: str, class_names: "dict[int, str]") -> ModelVoca
             continue
         slots[idx] = canonical
     return ModelVocabulary(
-        model_name=name, slot_names=slots, logit_slots=width, approx_ram_bytes=1
+        model_name=name,
+        slot_names=slots,
+        logit_slots=width,
+        approx_ram_bytes=1,
+        # THE LAYOUT IS DERIVED FROM THE GRAPH, and it is the only half
+        # `_read_graph_spec` reads — so the probe declares what it just measured
+        # rather than a guess, and an artifact whose layout this build cannot name
+        # fails `VariantContract` here and is recorded as not loadable, with the
+        # reason, which is the right answer to "would the service load this?".
+        #
+        # THE UNITS ARE THE IDENTITY, AND THAT IS THE MEASUREMENT, not a claim
+        # about the artifact. They cannot be known here — `classify_box_units`
+        # needs a forward pass, which happens later and lands in the run file's
+        # `variant` block — and one caller of this function (`ShippedRunner`)
+        # DOES decode with them. `pixels`/1.0 is the only pair that scales
+        # nothing, so the graph's own box magnitudes reach `box_abs_max`
+        # untouched and the classifier sees what the model actually emitted.
+        # Declaring the other pair would multiply by imgsz the very quantity
+        # being measured, and the verdict would then be a function of the guess.
+        #
+        # This is right for ONNX specifically: measured across all 17 committed
+        # twins, every ONNX side is pixels and only the LiteRT sides are
+        # normalised — and `.tflite` never reaches here, it routes to
+        # `LitertRunner`. A normalised ONNX artifact would still be REPORTED
+        # correctly by `classify_box_units`, because the identity leaves its
+        # 0..1 boxes as 0..1; what it would lose is per-class detections on
+        # padded frames, which is the signal to read the units verdict before
+        # trusting that candidate's score.
+        onnx_variant=VariantContract(output_layout, BOX_UNITS_PIXELS, 1.0),
     )
 
 
@@ -613,6 +621,7 @@ def probe_loadability(model_path: Path, class_names: "dict[int, str]") -> dict:
                     f"the detector service builds an onnxruntime InferenceSession and this is a "
                     f"{model_path.suffix} artifact: {str(exc).splitlines()[0][:200]}"
                 ),
+                "output_layout": None,
             }
 
     opts = ort.SessionOptions()
@@ -620,15 +629,28 @@ def probe_loadability(model_path: Path, class_names: "dict[int, str]") -> dict:
     session = ort.InferenceSession(
         str(model_path), sess_options=opts, providers=["CPUExecutionProvider"]
     )
+    # DERIVED ONCE AND RETURNED, because the only other way for a caller to learn
+    # it is to open a second session — and the caller that needs it sits between
+    # the RSS baseline and `rss_after_session`, where a transient session is
+    # exactly the contamination `introspect_in_subprocess` exists to avoid.
+    layout = classify_output_layout([list(o.shape) for o in session.get_outputs()])
     try:
-        vocabulary = synthesise_vocabulary("probe-only", class_names)
+        vocabulary = synthesise_vocabulary("probe-only", class_names, layout)
     except ValueError as exc:
-        return {"shipped_service_loadable": False, "refusal": f"vocabulary: {exc}"}
+        return {
+            "shipped_service_loadable": False,
+            "refusal": f"vocabulary: {exc}",
+            "output_layout": layout,
+        }
     try:
         _read_graph_spec(session, vocabulary)
     except ModelContractError as exc:
-        return {"shipped_service_loadable": False, "refusal": str(exc)}
-    return {"shipped_service_loadable": True, "refusal": None}
+        return {
+            "shipped_service_loadable": False,
+            "refusal": str(exc),
+            "output_layout": layout,
+        }
+    return {"shipped_service_loadable": True, "refusal": None, "output_layout": layout}
 
 
 def introspect_in_subprocess(model_path: Path) -> dict:
@@ -759,7 +781,10 @@ def main() -> int:
     elif model_path.suffix.lower() == ".tflite":
         runner = LitertRunner(model_path, declared_names)
     elif loadability["shipped_service_loadable"]:
-        runner = ShippedRunner(model_path, synthesise_vocabulary(args.candidate, declared_names))
+        runner = ShippedRunner(
+            model_path,
+            synthesise_vocabulary(args.candidate, declared_names, loadability["output_layout"]),
+        )
     else:
         runner = HarnessYoloRunner(model_path, declared_names)