screengraft 0.13.1 → 0.20.4

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -1,10 +1,8 @@
1
1
  # screengraft
2
2
 
3
- **Put a UI screenshot onto a photographed screen so the perspective is exactly right.**
3
+ **Put a UI screenshot — or a screen recording — onto a photographed screen so the perspective is exactly right.**
4
4
 
5
- ![A Figma screen composited onto a photographed iPhone](docs/hero.png)
6
-
7
- ![The fitting workbench: photo with the screen quad on the left, live composite on the right, and a magnified strip across the edge below](docs/social-preview.png)
5
+ ![The fitting workbench: photo with the screen quad on the left, live composite on the right, and a magnified strip across the edge below](docs/workbench.png)
8
6
 
9
7
  Every device mockup is a compromise. Templates give you three angles and someone
10
8
  else's lighting. Generative tools give you a screen that looks *like* your design
@@ -15,6 +13,9 @@ transform between them. The screenshot lands on the glass because the geometry
15
13
  says it must, not because a model thought it looked about right. Same inputs,
16
14
  same output, every time.
17
15
 
16
+ Point it at a **video** instead and the same fit renders every frame: record a
17
+ prototype, then put the recording inside a real photograph.
18
+
18
19
  ---
19
20
 
20
21
  ## What it does
@@ -28,6 +29,10 @@ same output, every time.
28
29
  - **Realism pass** *(optional)* — matches the screen's white balance and grain to
29
30
  the light in the room, and can lift the device's real reflections from a
30
31
  screen-off frame of the same shot.
32
+ - **Video, not just stills.** The screen source can be an `mp4`/`mov`/`webm`.
33
+ You match the edges on one frame and every frame gets that same geometry — the
34
+ photograph is still, so there is nothing to track and nothing to drift. Output
35
+ is H.264 at CRF 16 or ProRes 422 HQ.
31
36
  - **You confirm every fit.** Detection is advisory and says so; you drag the four
32
37
  edges onto the glass with a magnified loupe. A silent misdetection producing a
33
38
  confident, wrong result is the one failure this tool refuses to have.
@@ -38,6 +43,10 @@ same output, every time.
38
43
  without it. The installer provisions an isolated venv at `~/.screengraft/venv`
39
44
  and never touches your system Python.
40
45
 
46
+ Video rendering also uses **ffmpeg**, which arrives as a wheel (`imageio-ffmpeg`)
47
+ into that same venv — nothing is installed system-wide. It is optional: without
48
+ it, stills work exactly as before.
49
+
41
50
  ## Install as a Claude Code / Cowork plugin
42
51
 
43
52
  ```
@@ -106,15 +115,31 @@ outside the screen mask. It never touches the pixels you designed.
106
115
  3. **Realism pass** *(optional)* — white balance and exposure toward the
107
116
  surrounding light, grain matched to the photo's own noise floor, real
108
117
  speculars lifted from a screen-off reference.
118
+ 4. **Video**, when the source is a clip — everything a fixed photo and a fixed
119
+ quad make constant is computed once, and only the frame changes. Three
120
+ consequences worth naming, because each is a way video normally goes wrong:
121
+ the light match is measured **once** from the frame you fitted on, so the
122
+ screen cannot pulse as your UI scrolls from dark to light; the grain stays
123
+ frozen, because the photograph's own noise does not move; and the screen's
124
+ antialiased **edge is pixel-identical in every frame**, so there is no edge
125
+ crawl. Frame 0 of a render is byte-identical to the still composite — the
126
+ test suite asserts it, because that is what stops the two paths drifting.
109
127
 
110
128
  ## Roadmap
111
129
 
112
- Done: manual warp, advisory detectors, the fitting workbench, the realism pass.
130
+ Done: manual warp, advisory detectors, the fitting workbench, the realism pass,
131
+ video into a still photo.
132
+
133
+ Open: **camera-motion tracking** — the photograph itself must currently be a
134
+ still, so a clip of a moving phone is out of scope; **SAM 2 auto-detect** (built
135
+ and measured in a separate repo; it segments the phone body rather than the
136
+ glass, so it is not shipped); **occluder matte**, so a finger in front of the
137
+ screen stays in front.
113
138
 
114
- Open: video tracking (M3), SAM 2 auto-detect (M4 built and measured in a
115
- separate repo; it currently segments the phone body rather than the glass, so it
116
- is not shipped), occluder matte (M5), so a finger in front of the screen stays in
117
- front.
139
+ Known limits worth stating plainly: detection abstains rather than guessing when
140
+ the background is itself neutral (a pale tiled floor, a plain wall) you place
141
+ the edges by hand there. And a prototype recording has no motion blur, so a very
142
+ fast scroll will strobe; that is a property of the source, not of the composite.
118
143
 
119
144
  ## Contributing
120
145
 
package/package.json CHANGED
@@ -1,7 +1,7 @@
1
1
  {
2
2
  "name": "screengraft",
3
- "version": "0.13.1",
4
- "description": "Put a UI screenshot onto a photographed device screen with the perspective exactly right \u2014 a homography you confirm by hand, not a generative guess.",
3
+ "version": "0.20.4",
4
+ "description": "Put a UI screenshot or screen recording onto a photographed device screen with the perspective exactly right a homography you confirm by hand, not a generative guess.",
5
5
  "keywords": [
6
6
  "mockup",
7
7
  "device-frame",
@@ -9,6 +9,9 @@
9
9
  "perspective",
10
10
  "compositing",
11
11
  "screenshot",
12
+ "prototype",
13
+ "screen-recording",
14
+ "video",
12
15
  "design",
13
16
  "portfolio",
14
17
  "figma"
package/scripts/detect.py CHANGED
@@ -54,6 +54,62 @@ MIN_AREA_FRAC = 0.01
54
54
  MAX_AREA_FRAC = 0.60
55
55
  MIN_FILL = 0.60 # the source blob must fill this much of the final quad
56
56
  MIN_SIDE_RATIO = 0.08 # reject slivers: shortest side vs longest, after perspective
57
+ # Size stops being a virtue past this fraction of the frame. score_contour used
58
+ # to reward area linearly, so on a real photo (7 Sep 2026) the sunlit TABLE at
59
+ # 34% of the frame beat the phone screen at 9% by 3.6x on that term alone, and
60
+ # won outright. A screen is never the biggest thing in a photograph of a room;
61
+ # it is only ever big enough. Past this plateau extra area buys nothing, so
62
+ # candidates separate on fill and band width instead — which is what actually
63
+ # distinguishes glass from furniture.
64
+ AREA_PLATEAU = 0.15
65
+ # How many connected components each tone band offers up. See blobs_for_band:
66
+ # taking only the largest is what made a real photo undetectable, because the
67
+ # phone was the second-largest dark region in its band. Six is measured, not
68
+ # guessed — the winning component on that photo is rank 2, and nothing useful
69
+ # was found past rank 4 on any fixture; the extra two are headroom.
70
+ COMPONENTS_PER_BAND = 6
71
+ # Saturation percentile below which a region counts as "neutral". Devices are
72
+ # grey, black and white; furniture, skin, fabric and foliage are not. Measured
73
+ # 7 Sep 2026 on a photo where tone banding failed completely: the screen sits at
74
+ # saturation 1.9, the sunlit table it was being confused with at 78.6. Taking
75
+ # the image's own 25th percentile adapts to the photograph instead of fixing a
76
+ # level — a studio shot on white and a warm interior need different numbers.
77
+ # Swept, not fixed — the same reasoning the tone sweep is built on. A single
78
+ # percentile breaks whenever the neutral thing is smaller than the percentile
79
+ # (a phone occupying 15% of a colourful frame pulls p25 up into the colour and
80
+ # the mask swallows the picture). Trying several and letting the scoring decide
81
+ # costs one pass each and removes the guess.
82
+ NEUTRAL_PERCENTILES = (5, 10, 15, 25, 35)
83
+ NEUTRAL_FLOOR = 20 # never threshold below this: an all-grey photo
84
+ # A screen has rounded corners. A patch of table cut out by a threshold has
85
+ # perfectly sharp ones, and measure_corner_radius returns 0.0px for it — which
86
+ # turns out to be the cleanest way to tell a real screen from a lookalike, and
87
+ # it is a SHAPE property, not photometry (three photometric arbiters have been
88
+ # measured and rejected here; see the note above score_edge_contour).
89
+ MIN_ROUNDING_PX = 2.0
90
+ # measure_corner_radius makes FOUR independent estimates of one number. If they
91
+ # disagree by more than this, the outline is not a rounded rectangle with a
92
+ # consistent radius and the median is meaningless — so "it has rounded corners"
93
+ # is not evidence and must not be treated as any. Measured 7 Sep 2026 across
94
+ # nine real mockup photos: the seven correct detections spread 31-173%, the two
95
+ # confidently-wrong ones 247% and 464%.
96
+ #
97
+ # **This is the weakest number in the file.** 173 against 247 is a 1.4x margin
98
+ # on nine samples, where every other threshold here was set with a 4x margin or
99
+ # better. If a correct detection is ever rejected as "shape is not consistent",
100
+ # this is the line to raise, and it should be re-measured on a bigger set before
101
+ # it is trusted further.
102
+ MAX_RADIUS_SPREAD = 2.0
103
+ # A quad with a corner outside the frame is not a screen this tool can fit, and
104
+ # its handles cannot be grabbed — the user is left with a wrong quad and no way
105
+ # back (same photo: two corners at y=-50 and y=1837 on a 1792-tall image).
106
+ # Rejected at validate_quad, the one choke point every result passes through.
107
+ OOB_MARGIN = 2.0
108
+ # Gross disagreement between the two detectors, as a fraction of the image
109
+ # diagonal. Measured 7 Sep 2026: the reference photo, where detection genuinely
110
+ # works, sits at 0.033; the photo where both detectors missed the screen
111
+ # entirely sits at 0.555 — seventeen times worse. 0.15 is well clear of both.
112
+ ABSTAIN_GAP = 0.15
57
113
  # A screen fills MOST of the body it sits in; content drawn on a screen is a
58
114
  # small part of it. That one ratio separates "step inward to the screen" from
59
115
  # "don't step into a panel", and it arbitrates between the two detectors too.
@@ -65,8 +121,24 @@ def _odd(n: int) -> int:
65
121
  return n if n % 2 else n + 1
66
122
 
67
123
 
68
- def blob_for_band(gray: np.ndarray, lo: int, hi: int, close_k: int, open_k: int):
69
- """Mask -> morphology -> largest connected component. Returns its contour."""
124
+ def blobs_for_band(gray: np.ndarray, lo: int, hi: int, close_k: int, open_k: int,
125
+ keep: int = COMPONENTS_PER_BAND):
126
+ """Mask -> morphology -> the `keep` largest connected components.
127
+
128
+ Returns a list of contours, biggest first.
129
+
130
+ This used to return only the single largest component, and that one line
131
+ was why a real photo could not be detected at all. On an iPhone lying on a
132
+ sunlit table (7 Sep 2026) the phone was the SECOND-largest dark region in
133
+ its band — the table's shadow was bigger — so the screen was discarded
134
+ before scoring ever saw it. No amount of re-scoring can rank a candidate
135
+ that was never generated: measured, the best quad the old sweep could
136
+ produce sat 693px from the true screen; keeping the runners-up brings that
137
+ to 136px, which is a startable position.
138
+
139
+ "Largest" is a guess about the answer dressed up as an optimisation. The
140
+ scoring function is what decides; this function's job is only to offer.
141
+ """
70
142
  mask = cv2.inRange(gray, lo, hi)
71
143
  mask = cv2.morphologyEx(
72
144
  mask, cv2.MORPH_CLOSE,
@@ -81,8 +153,19 @@ def blob_for_band(gray: np.ndarray, lo: int, hi: int, close_k: int, open_k: int)
81
153
  # leaves fitLine with nothing to fit.
82
154
  contours, _ = cv2.findContours(mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE)
83
155
  if not contours:
84
- return None
85
- return max(contours, key=cv2.contourArea)
156
+ return []
157
+ return sorted(contours, key=cv2.contourArea, reverse=True)[:keep]
158
+
159
+
160
+ def size_term(area: float, img_area: float) -> float:
161
+ """Area reward that saturates at AREA_PLATEAU.
162
+
163
+ Below the plateau, bigger is better — it separates a real region from
164
+ speckle. At or above it, the term is 1.0 and stops discriminating, because
165
+ beyond a plausible screen size "bigger" stops being evidence of screen-ness
166
+ and starts being evidence of furniture.
167
+ """
168
+ return min(area / img_area, AREA_PLATEAU) / AREA_PLATEAU
86
169
 
87
170
 
88
171
  def score_contour(contour, img_area: float):
@@ -99,7 +182,7 @@ def score_contour(contour, img_area: float):
99
182
  # Fill ratio: a real screen fills its own quad almost completely. A shadow
100
183
  # or a wall patch is ragged and fills far less.
101
184
  fill = min(area / quad_area, 1.0)
102
- return (fill ** 3) * (area / img_area), quad
185
+ return (fill ** 3) * size_term(area, img_area), quad
103
186
 
104
187
 
105
188
  # Two "edge support" arbiters — score each candidate quad by how much of its
@@ -140,7 +223,7 @@ def score_edge_contour(contour, img_area: float):
140
223
  if quad_area < MIN_AREA_FRAC * img_area or quad_area > MAX_AREA_FRAC * img_area:
141
224
  return 0.0, None
142
225
  fill = min(float(cv2.contourArea(contour)) / max(quad_area, 1e-6), 1.0)
143
- return fill * (quad_area / img_area), quad
226
+ return fill * size_term(quad_area, img_area), quad
144
227
 
145
228
 
146
229
  def approx_quad(contour):
@@ -365,7 +448,7 @@ def measure_corner_radius(contour, corners: np.ndarray):
365
448
  # segmented contour, so it isn't picking between competing nearby edges.
366
449
 
367
450
 
368
- def validate_quad(corners: np.ndarray, contour, img_area: float):
451
+ def validate_quad(corners: np.ndarray, contour, img_area: float, img_shape=None):
369
452
  """Is this *final* quad plausibly a screen? Returns (ok, reason).
370
453
 
371
454
  The area gate in score_contour() runs against the CONTOUR, before
@@ -375,6 +458,15 @@ def validate_quad(corners: np.ndarray, contour, img_area: float):
375
458
  confident detection (3 Sep 2026, an earlier finding). Everything returned to a caller
376
459
  goes through here.
377
460
  """
461
+ if img_shape is not None:
462
+ h, w = float(img_shape[0]), float(img_shape[1])
463
+ for (x, y), label in zip(corners, ["TL", "TR", "BR", "BL"], strict=True):
464
+ if not (-OOB_MARGIN <= x <= w + OOB_MARGIN
465
+ and -OOB_MARGIN <= y <= h + OOB_MARGIN):
466
+ return False, ("%s lands outside the photo at (%.0f, %.0f) on a "
467
+ "%.0fx%.0f image — a screen this tool can fit is "
468
+ "inside the frame, and an off-canvas handle can't "
469
+ "be dragged back" % (label, x, y, w, h))
378
470
  area = float(cv2.contourArea(corners.astype(np.float32)))
379
471
  frac = area / img_area
380
472
  if frac < MIN_AREA_FRAC:
@@ -398,7 +490,7 @@ def validate_quad(corners: np.ndarray, contour, img_area: float):
398
490
  return True, ""
399
491
 
400
492
 
401
- def _finalize(candidates, img_area: float, refine: bool = True):
493
+ def _finalize(candidates, img_area: float, refine: bool = True, img_shape=None):
402
494
  """Best candidate that survives refinement AND validation.
403
495
 
404
496
  Walks candidates best-score-first rather than trusting the top one: a
@@ -422,7 +514,7 @@ def _finalize(candidates, img_area: float, refine: bool = True):
422
514
  else:
423
515
  refined, did_refine = quad, False
424
516
  corners = order_quad(refined)
425
- ok, why = validate_quad(corners, contour, img_area)
517
+ ok, why = validate_quad(corners, contour, img_area, img_shape)
426
518
  if ok:
427
519
  return {
428
520
  "corners": [[round(float(x), 1), round(float(y), 1)] for x, y in corners],
@@ -456,11 +548,10 @@ def detect_tone(gray: np.ndarray, tone=None):
456
548
 
457
549
  candidates = []
458
550
  for lo, hi in bands:
459
- contour = blob_for_band(gray, lo, hi, close_k, open_k)
460
- if contour is None:
461
- continue
462
- score, quad = score_contour(contour, img_area)
463
- if score > 0 and quad is not None:
551
+ for contour in blobs_for_band(gray, lo, hi, close_k, open_k):
552
+ score, quad = score_contour(contour, img_area)
553
+ if score <= 0 or quad is None:
554
+ continue
464
555
  # Prefer a narrow band. A screen is tonally uniform; a band wide
465
556
  # enough to also swallow the phone's shadowed body scores well on
466
557
  # area but produces edges that follow the body, not the glass.
@@ -469,7 +560,7 @@ def detect_tone(gray: np.ndarray, tone=None):
469
560
  score *= (16.0 / (hi - lo + 1)) ** 0.25
470
561
  candidates.append((score, order_quad(quad), contour, (lo, hi)))
471
562
 
472
- res = _finalize(candidates, img_area)
563
+ res = _finalize(candidates, img_area, img_shape=gray.shape[:2])
473
564
  if res is None:
474
565
  return None
475
566
  band = res.pop("_tag")
@@ -517,7 +608,7 @@ def detect_edges(gray: np.ndarray):
517
608
  if score > 0 and quad is not None:
518
609
  candidates.append((score, order_quad(quad), c, (lo, hi)))
519
610
 
520
- res = _finalize(candidates, img_area, refine=False)
611
+ res = _finalize(candidates, img_area, refine=False, img_shape=gray.shape[:2])
521
612
  if res is None:
522
613
  return None
523
614
  thr = res.pop("_tag")
@@ -526,7 +617,71 @@ def detect_edges(gray: np.ndarray):
526
617
  return res
527
618
 
528
619
 
529
- def detect(gray: np.ndarray, tone=None, method="auto"):
620
+ def detect_saturation(bgr: np.ndarray):
621
+ """Neutral-region segmentation — the third detector, and the only one that
622
+ looks at colour.
623
+
624
+ tone and edge both run on grayscale, which throws away the single most
625
+ useful cue a photograph of a device offers: **devices are neutral.** A
626
+ phone is grey, black or white; the table, sofa, hand or plant it is lying
627
+ on almost never is. On the 7 Sep 2026 photo the screen measured saturation
628
+ 1.9 against the sunlit table's 78.6, and this detector lands 29px from the
629
+ true quad where tone lands 1030px away.
630
+
631
+ The threshold is the image's own 25th saturation percentile rather than a
632
+ fixed level, so a studio shot on white and a warm interior both work.
633
+ """
634
+ hsv = cv2.cvtColor(bgr, cv2.COLOR_BGR2HSV)
635
+ sat = hsv[:, :, 1]
636
+ h, w = sat.shape[:2]
637
+ img_area = float(h * w)
638
+ short = min(h, w)
639
+ close_k = _odd(CLOSE_FRAC * short)
640
+ open_k = _odd(OPEN_FRAC * short)
641
+
642
+ thresholds = {int(max(NEUTRAL_FLOOR, np.percentile(sat, p)))
643
+ for p in NEUTRAL_PERCENTILES}
644
+ otsu, _ = cv2.threshold(sat, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)
645
+ thresholds.add(int(max(NEUTRAL_FLOOR, otsu)))
646
+ candidates = []
647
+ for thr in sorted(thresholds):
648
+ for contour in blobs_for_band(sat, 0, thr, close_k, open_k):
649
+ score, quad = score_contour(contour, img_area)
650
+ if score > 0 and quad is not None:
651
+ # Prefer a tighter neutral threshold, for the same reason the
652
+ # tone sweep prefers a narrow band: a loose one swallows the
653
+ # device body and the wall behind it along with the screen.
654
+ candidates.append((score * (32.0 / max(thr, 1)) ** 0.25,
655
+ order_quad(quad), contour, thr))
656
+
657
+ res = _finalize(candidates, img_area, img_shape=sat.shape[:2])
658
+ if res is None:
659
+ return None
660
+ res.pop("_tag")
661
+ res["method"] = "saturation"
662
+ res["neutral_threshold"] = thr
663
+ return res
664
+
665
+
666
+ def has_rounded_corners(result) -> bool:
667
+ """Did this quad's own outline actually curve at the corners?
668
+
669
+ A device screen is a rounded rectangle. A patch of table cut out of a
670
+ threshold mask is a polygon with sharp corners, and measure_corner_radius
671
+ reports 0.0px for it. That single number separated the right answer from
672
+ three wrong ones on the photo this was built for, and unlike edge strength
673
+ or ring contrast it is a property of the shape rather than of the light.
674
+ """
675
+ cr = result["corner_radius"]
676
+ if float(cr["photo_px"]) <= MIN_ROUNDING_PX:
677
+ return False
678
+ per = [float(v) for v in cr["per_corner_px"]]
679
+ r = float(cr["photo_px"])
680
+ spread = (max(per) - min(per)) / max(r, 1e-6)
681
+ return spread <= MAX_RADIUS_SPREAD
682
+
683
+
684
+ def detect(gray: np.ndarray, tone=None, method="auto", color=None):
530
685
  """Run both detectors; arbitrate on how the two quads nest.
531
686
 
532
687
  The two fail on opposite things. Tone banding needs a tonally uniform
@@ -559,13 +714,40 @@ def detect(gray: np.ndarray, tone=None, method="auto"):
559
714
  r = detect_edges(gray)
560
715
  if r:
561
716
  results.append(r)
717
+ if method in ("auto", "saturation") and tone is None and color is not None:
718
+ r = detect_saturation(color)
719
+ if r:
720
+ results.append(r)
562
721
 
563
722
  if not results:
564
723
  return None
565
724
 
725
+ # tone and saturation are the same algorithm on different channels, so they
726
+ # are compared to each other before anything else, on whether the region
727
+ # each found actually has rounded corners. A patch of table cut out of a
728
+ # threshold mask measures 0.0px; a screen measures a real radius. On the
729
+ # 7 Sep photo that is the whole ball game — tone 0.0px against saturation's
730
+ # 62.4px, so the channel that found the phone is the one that goes forward.
731
+ #
732
+ # **Only these two.** The edge detector is deliberately exempt: its contour
733
+ # is a Canny ring tracing both sides of a boundary, not a filled region's
734
+ # silhouette, so measure_corner_radius reports an artifact for it — 0.0px
735
+ # even when its quad is the correct one to 1.4px (measured on the
736
+ # gradient-screen fixture, where an earlier version of this filter threw
737
+ # away the right answer). _finalize already skips corner refinement on that
738
+ # path for the same reason.
739
+ region = [r for r in results if r["method"] in ("tone", "saturation")]
740
+ if len(region) == 2:
741
+ rounded = [r for r in region if has_rounded_corners(r)]
742
+ if len(rounded) == 1:
743
+ loser = next(r for r in region if r is not rounded[0])
744
+ results = [r for r in results if r is not loser]
745
+
566
746
  t = next((r for r in results if r["method"] == "tone"), None)
567
747
  e = next((r for r in results if r["method"] == "edge"), None)
568
- why = ""
748
+ sat = next((r for r in results if r["method"] == "saturation"), None)
749
+ why = "it was the only detector left after the rounded-corner filter" \
750
+ if len(results) == 1 else "it is the detector that refines corners"
569
751
  if t and e:
570
752
  tq, eq = t["_corners_np"], e["_corners_np"]
571
753
  ta = float(cv2.contourArea(tq.astype(np.float32)))
@@ -581,28 +763,46 @@ def detect(gray: np.ndarray, tone=None, method="auto"):
581
763
  "its area — a screen inside a device body" % (ratio * 100))
582
764
  else:
583
765
  best, why = t, "the two quads aren't nested; the tone detector refines corners"
766
+ elif t is None and sat is not None:
767
+ # tone was dropped for having sharp corners, or never fired. The
768
+ # surviving region detector refines corners; edge does not.
769
+ best, why = sat, ("the saturation detector found a region with rounded "
770
+ "corners where tone did not")
584
771
  else:
585
- best = t or e
586
-
587
- if len(results) == 2:
588
- a, b = (r["_corners_np"] for r in results)
772
+ best = t or e or sat
773
+ if best is None:
774
+ best = results[0]
775
+
776
+ if len(results) > 1:
777
+ # Agreement is measured against the CLOSEST other detector, not against
778
+ # "the other one" — there are three now, and a third opinion that lands
779
+ # somewhere else entirely should not erase the fact that two of them
780
+ # landed together. Corroboration by any one independent method is the
781
+ # evidence worth reporting.
589
782
  diag = float(np.hypot(*gray.shape[:2]))
590
- spread = float(np.max(np.linalg.norm(a - b, axis=1))) / diag
783
+ others = [r for r in results if r is not best]
784
+ gaps = [(float(np.max(np.linalg.norm(best["_corners_np"]
785
+ - r["_corners_np"], axis=1))) / diag, r)
786
+ for r in others]
787
+ spread, nearest = min(gaps, key=lambda g: g[0])
591
788
  agree = bool(spread < 0.02)
592
789
  best["agreement"] = {
593
790
  "both_found": True,
594
791
  "max_corner_gap_frac_of_diagonal": round(spread, 4),
595
792
  "agree": agree,
596
- "note": ("Both detectors independently landed on the same quad — "
597
- "that is real evidence, not one algorithm's opinion."
793
+ "agrees_with": nearest["method"] if agree else None,
794
+ "note": ("The %s and %s detectors independently landed on the same "
795
+ "quad — that is real evidence, not one algorithm's opinion."
796
+ % (best["method"], nearest["method"])
598
797
  if agree else
599
- "The two detectors disagree by %.0f%% of the image diagonal. "
600
- "Showing the %s one because %s — but check all four corners."
601
- % (spread * 100, best["method"], why)),
798
+ "No two detectors agree; the closest other (%s) is %.0f%% of "
799
+ "the image diagonal away. Showing the %s one because %s — "
800
+ "but check all four corners."
801
+ % (nearest["method"], spread * 100, best["method"], why)),
602
802
  }
603
803
  best["agreement"]["chosen_because"] = why
604
- other = next(r for r in results if r is not best)
605
- best["other"] = {"method": other["method"], "corners": other["corners"]}
804
+ best["other"] = [{"method": r["method"], "corners": r["corners"]}
805
+ for r in others]
606
806
  else:
607
807
  best["agreement"] = {
608
808
  "both_found": False,
@@ -610,6 +810,57 @@ def detect(gray: np.ndarray, tone=None, method="auto"):
610
810
  "note": ("Only the %s detector found anything — a first guess to "
611
811
  "correct, not a measurement." % best["method"]),
612
812
  }
813
+
814
+ # Abstention. detect.py has always PROMISED to fail honestly rather than
815
+ # emit a confident wrong quad, but nothing enforced it: on 7 Sep 2026 a real
816
+ # photo produced a quad on the table, with `agree` false, the two detectors
817
+ # 55% of the diagonal apart, the radius spread at 104%, and two corners off
818
+ # the image — every indicator of failure present, and a result returned
819
+ # anyway, exit 0. The evidence was already being computed; it just wasn't
820
+ # gating. Two conditions, either of which means "we do not know":
821
+ ag = best["agreement"]
822
+ # Disagreement only counts against a CREDIBLE peer. Measured 7 Sep 2026 on
823
+ # eight real mockup photos: where saturation correctly found a phone that
824
+ # tone and edge had both missed, the winner was of course miles from the two
825
+ # that failed — and this gate then threw the right answer away as
826
+ # "disagreement". A detector that found a sharp-cornered patch of floor does
827
+ # not get a vote on whether the rounded thing is a screen. So the gap is
828
+ # re-measured against peers that also found something screen-shaped; when
829
+ # there are none, being alone is not evidence of being wrong.
830
+ peers = [r for r in results if r is not best and has_rounded_corners(r)]
831
+ if peers:
832
+ diag = float(np.hypot(*gray.shape[:2]))
833
+ peer_gap = min(float(np.max(np.linalg.norm(best["_corners_np"]
834
+ - r["_corners_np"], axis=1))) / diag
835
+ for r in peers)
836
+ else:
837
+ peer_gap = None
838
+ gross = bool(peer_gap is not None and peer_gap > ABSTAIN_GAP)
839
+ if peer_gap is not None:
840
+ ag["credible_peer_gap"] = round(peer_gap, 4)
841
+ # A measurable corner radius counts as evidence in its own right: it says
842
+ # the thing found is shaped like a screen, which is what abstention exists
843
+ # to doubt. Without this a good saturation result on a hard photo would be
844
+ # thrown away for want of a second opinion.
845
+ uncorroborated = bool(not ag.get("agree")
846
+ and not best["corner_radius"]["confident"]
847
+ and not has_rounded_corners(best))
848
+ if gross or uncorroborated:
849
+ why = []
850
+ if gross:
851
+ why.append("two detectors that each found something screen-shaped are "
852
+ "%.0f%% of the image diagonal apart" % (peer_gap * 100))
853
+ if uncorroborated:
854
+ why.append("nothing corroborates the quad (the detectors don't agree "
855
+ "and the corner radius isn't measurable)")
856
+ best["abstained"] = True
857
+ best["abstain_reason"] = (
858
+ "Detection abstained: " + " and ".join(why) + ". The quad below is "
859
+ "kept for inspection but is not offered as a starting position — "
860
+ "place the four edges by hand."
861
+ )
862
+ else:
863
+ best["abstained"] = False
613
864
  return best
614
865
 
615
866
 
@@ -655,6 +906,8 @@ def main() -> None:
655
906
  if args.out_zooms:
656
907
  written["zooms"] = write_zooms(photo, corners, args.out_zooms)
657
908
 
909
+ if result.get("abstained"):
910
+ print(result["abstain_reason"], file=sys.stderr)
658
911
  print(json.dumps({
659
912
  **result,
660
913
  "photo_size": [photo.shape[1], photo.shape[0]],
@@ -672,6 +925,12 @@ def main() -> None:
672
925
  "this script, so an unconfirmed guess cannot reach a composite.",
673
926
  }, indent=1))
674
927
 
928
+ # The docstring's promise, finally enforced: an uncorroborated guess exits
929
+ # non-zero. The JSON is still printed above so a caller can inspect what was
930
+ # rejected and why.
931
+ if result.get("abstained"):
932
+ sys.exit(2)
933
+
675
934
 
676
935
  if __name__ == "__main__":
677
936
  main()
package/scripts/grade.py CHANGED
@@ -61,48 +61,75 @@ def _stats(lab: np.ndarray, sel: np.ndarray) -> tuple[np.ndarray, np.ndarray]:
61
61
  return px.mean(axis=0), px.std(axis=0) + 1e-6
62
62
 
63
63
 
64
- def match_light(photo: np.ndarray, warped: np.ndarray, mask: np.ndarray,
65
- strength: float = DEFAULT_STRENGTH) -> np.ndarray:
66
- """Move the injected screen's cast and exposure toward the surrounding light.
67
-
68
- Chroma (a,b) is matched on mean AND spread a cast is exactly a chroma mean
69
- offset, and a room with weak colour should not receive a saturated screen.
70
- Luminance is matched on MEAN ONLY, and bounded: a screen is emissive and is
71
- allowed to be brighter than its surroundings, so rescaling its L spread to
72
- the bezel's would crush the UI's own contrast. That asymmetry is the whole
73
- design of this function.
64
+ def light_params(photo: np.ndarray, warped: np.ndarray, mask: np.ndarray,
65
+ strength: float = DEFAULT_STRENGTH):
66
+ """Measure the correction ONCE, so it can be applied to many frames.
67
+
68
+ Split out of match_light for video. The correction depends on the
69
+ screen's own content through `m_in`/`s_in`, so measuring it per frame makes
70
+ it drift as the UI scrolls from a dark screen to a light one — the injected
71
+ screen would visibly pulse. Deriving the parameters from one frame and
72
+ applying the identical transform to all of them is the whole fix.
73
+
74
+ Returns None when there is too little context to measure honestly, which
75
+ the caller must treat as "leave the frame alone".
74
76
  """
75
77
  if strength <= 0:
76
- return warped
78
+ return None
77
79
  ring = surround_ring(mask)
78
- if int(ring.sum()) < 500: # too little context to measure honestly
79
- return warped
80
-
81
- lab_photo = cv2.cvtColor(photo, cv2.COLOR_BGR2LAB).astype(np.float64)
82
- lab_warp = cv2.cvtColor(warped, cv2.COLOR_BGR2LAB).astype(np.float64)
80
+ if int(ring.sum()) < 500:
81
+ return None
83
82
  inside = (mask > 200).astype(np.uint8)
84
83
  if int(inside.sum()) < 500:
85
- return warped
86
-
84
+ return None
85
+ lab_photo = cv2.cvtColor(photo, cv2.COLOR_BGR2LAB).astype(np.float64)
86
+ lab_warp = cv2.cvtColor(warped, cv2.COLOR_BGR2LAB).astype(np.float64)
87
87
  m_out, s_out = _stats(lab_photo, ring)
88
88
  m_in, s_in = _stats(lab_warp, inside)
89
-
89
+ return {
90
+ "m_in": m_in, "s_in": s_in, "m_out": m_out, "s_out": s_out,
91
+ "strength": float(strength),
92
+ # Same clamp as match_light: a screen is emissive and may be brighter
93
+ # than the room, so L moves by a bounded mean shift only.
94
+ "dL": float(np.clip(m_out[0] - m_in[0], -12.0, 12.0)) * float(strength),
95
+ }
96
+
97
+
98
+ def apply_light(warped: np.ndarray, params) -> np.ndarray:
99
+ """Apply parameters from light_params() to one frame."""
100
+ if params is None:
101
+ return warped
102
+ lab_warp = cv2.cvtColor(warped, cv2.COLOR_BGR2LAB).astype(np.float64)
103
+ m_in, s_in = params["m_in"], params["s_in"]
104
+ m_out, s_out = params["m_out"], params["s_out"]
105
+ strength = params["strength"]
90
106
  out = lab_warp.copy()
91
- # a,b: full Reinhard transfer, scaled by strength.
92
107
  for c in (1, 2):
93
108
  moved = (lab_warp[:, :, c] - m_in[c]) * float(s_out[c] / s_in[c]) + m_out[c]
94
109
  out[:, :, c] = lab_warp[:, :, c] + (moved - lab_warp[:, :, c]) * strength
95
- # L: mean shift only, and capped at +-12 L* so a dark room cannot switch
96
- # the screen off. 12 is about a stop; beyond that it stops reading as the
97
- # same screenshot.
98
- dL = float(np.clip(m_out[0] - m_in[0], -12.0, 12.0)) * strength
99
- out[:, :, 0] = lab_warp[:, :, 0] + dL
100
-
110
+ out[:, :, 0] = lab_warp[:, :, 0] + params["dL"]
101
111
  out[:, :, 0] = np.clip(out[:, :, 0], 0, 255)
102
112
  out[:, :, 1:] = np.clip(out[:, :, 1:], 0, 255)
103
113
  return cv2.cvtColor(out.astype(np.uint8), cv2.COLOR_LAB2BGR)
104
114
 
105
115
 
116
+ def match_light(photo: np.ndarray, warped: np.ndarray, mask: np.ndarray,
117
+ strength: float = DEFAULT_STRENGTH) -> np.ndarray:
118
+ """Move the injected screen's cast and exposure toward the surrounding light.
119
+
120
+ Chroma (a,b) is matched on mean AND spread — a cast is exactly a chroma mean
121
+ offset, and a room with weak colour should not receive a saturated screen.
122
+ Luminance is matched on MEAN ONLY, and bounded: a screen is emissive and is
123
+ allowed to be brighter than its surroundings, so rescaling its L spread to
124
+ the bezel's would crush the UI's own contrast. That asymmetry is the whole
125
+ design of this function.
126
+ """
127
+ # One implementation, two entry points: measuring and applying are the same
128
+ # arithmetic whether it runs on a still or on frame 900 of a clip. Keeping a
129
+ # second copy here is how the two paths would drift.
130
+ return apply_light(warped, light_params(photo, warped, mask, strength))
131
+
132
+
106
133
  def measure_grain(photo: np.ndarray, ring: np.ndarray) -> float:
107
134
  """The photo's noise floor, in grey levels, measured where the screen isn't.
108
135