screengraft 0.13.1 → 0.20.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +34 -9
- package/package.json +5 -2
- package/scripts/detect.py +289 -30
- package/scripts/grade.py +53 -26
- package/scripts/preflight.py +48 -0
- package/scripts/requirements.txt +1 -0
- package/scripts/ui.py +159 -6
- package/scripts/warp.py +288 -71
- package/skills/inject-screenshot/SKILL.md +27 -10
- package/ui/index.html +163 -81
package/README.md
CHANGED
|
@@ -1,10 +1,8 @@
|
|
|
1
1
|
# screengraft
|
|
2
2
|
|
|
3
|
-
**Put a UI screenshot onto a photographed screen so the perspective is exactly right.**
|
|
3
|
+
**Put a UI screenshot — or a screen recording — onto a photographed screen so the perspective is exactly right.**
|
|
4
4
|
|
|
5
|
-

|
|
5
|
+

|
|
8
6
|
|
|
9
7
|
Every device mockup is a compromise. Templates give you three angles and someone
|
|
10
8
|
else's lighting. Generative tools give you a screen that looks *like* your design
|
|
@@ -15,6 +13,9 @@ transform between them. The screenshot lands on the glass because the geometry
|
|
|
15
13
|
says it must, not because a model thought it looked about right. Same inputs,
|
|
16
14
|
same output, every time.
|
|
17
15
|
|
|
16
|
+
Point it at a **video** instead and the same fit renders every frame: record a
|
|
17
|
+
prototype, then put the recording inside a real photograph.
|
|
18
|
+
|
|
18
19
|
---
|
|
19
20
|
|
|
20
21
|
## What it does
|
|
@@ -28,6 +29,10 @@ same output, every time.
|
|
|
28
29
|
- **Realism pass** *(optional)* — matches the screen's white balance and grain to
|
|
29
30
|
the light in the room, and can lift the device's real reflections from a
|
|
30
31
|
screen-off frame of the same shot.
|
|
32
|
+
- **Video, not just stills.** The screen source can be an `mp4`/`mov`/`webm`.
|
|
33
|
+
You match the edges on one frame and every frame gets that same geometry — the
|
|
34
|
+
photograph is still, so there is nothing to track and nothing to drift. Output
|
|
35
|
+
is H.264 at CRF 16 or ProRes 422 HQ.
|
|
31
36
|
- **You confirm every fit.** Detection is advisory and says so; you drag the four
|
|
32
37
|
edges onto the glass with a magnified loupe. A silent misdetection producing a
|
|
33
38
|
confident, wrong result is the one failure this tool refuses to have.
|
|
@@ -38,6 +43,10 @@ same output, every time.
|
|
|
38
43
|
without it. The installer provisions an isolated venv at `~/.screengraft/venv`
|
|
39
44
|
and never touches your system Python.
|
|
40
45
|
|
|
46
|
+
Video rendering also uses **ffmpeg**, which arrives as a wheel (`imageio-ffmpeg`)
|
|
47
|
+
into that same venv — nothing is installed system-wide. It is optional: without
|
|
48
|
+
it, stills work exactly as before.
|
|
49
|
+
|
|
41
50
|
## Install as a Claude Code / Cowork plugin
|
|
42
51
|
|
|
43
52
|
```
|
|
@@ -106,15 +115,31 @@ outside the screen mask. It never touches the pixels you designed.
|
|
|
106
115
|
3. **Realism pass** *(optional)* — white balance and exposure toward the
|
|
107
116
|
surrounding light, grain matched to the photo's own noise floor, real
|
|
108
117
|
speculars lifted from a screen-off reference.
|
|
118
|
+
4. **Video**, when the source is a clip — everything a fixed photo and a fixed
|
|
119
|
+
quad make constant is computed once, and only the frame changes. Three
|
|
120
|
+
consequences worth naming, because each is a way video normally goes wrong:
|
|
121
|
+
the light match is measured **once** from the frame you fitted on, so the
|
|
122
|
+
screen cannot pulse as your UI scrolls from dark to light; the grain stays
|
|
123
|
+
frozen, because the photograph's own noise does not move; and the screen's
|
|
124
|
+
antialiased **edge is pixel-identical in every frame**, so there is no edge
|
|
125
|
+
crawl. Frame 0 of a render is byte-identical to the still composite — the
|
|
126
|
+
test suite asserts it, because that is what stops the two paths drifting.
|
|
109
127
|
|
|
110
128
|
## Roadmap
|
|
111
129
|
|
|
112
|
-
Done: manual warp, advisory detectors, the fitting workbench, the realism pass
|
|
130
|
+
Done: manual warp, advisory detectors, the fitting workbench, the realism pass,
|
|
131
|
+
video into a still photo.
|
|
132
|
+
|
|
133
|
+
Open: **camera-motion tracking** — the photograph itself must currently be a
|
|
134
|
+
still, so a clip of a moving phone is out of scope; **SAM 2 auto-detect** (built
|
|
135
|
+
and measured in a separate repo; it segments the phone body rather than the
|
|
136
|
+
glass, so it is not shipped); **occluder matte**, so a finger in front of the
|
|
137
|
+
screen stays in front.
|
|
113
138
|
|
|
114
|
-
|
|
115
|
-
|
|
116
|
-
|
|
117
|
-
|
|
139
|
+
Known limits worth stating plainly: detection abstains rather than guessing when
|
|
140
|
+
the background is itself neutral (a pale tiled floor, a plain wall) — you place
|
|
141
|
+
the edges by hand there. And a prototype recording has no motion blur, so a very
|
|
142
|
+
fast scroll will strobe; that is a property of the source, not of the composite.
|
|
118
143
|
|
|
119
144
|
## Contributing
|
|
120
145
|
|
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "screengraft",
|
|
3
|
-
"version": "0.
|
|
4
|
-
"description": "Put a UI screenshot onto a photographed device screen with the perspective exactly right
|
|
3
|
+
"version": "0.20.4",
|
|
4
|
+
"description": "Put a UI screenshot or screen recording onto a photographed device screen with the perspective exactly right — a homography you confirm by hand, not a generative guess.",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"mockup",
|
|
7
7
|
"device-frame",
|
|
@@ -9,6 +9,9 @@
|
|
|
9
9
|
"perspective",
|
|
10
10
|
"compositing",
|
|
11
11
|
"screenshot",
|
|
12
|
+
"prototype",
|
|
13
|
+
"screen-recording",
|
|
14
|
+
"video",
|
|
12
15
|
"design",
|
|
13
16
|
"portfolio",
|
|
14
17
|
"figma"
|
package/scripts/detect.py
CHANGED
|
@@ -54,6 +54,62 @@ MIN_AREA_FRAC = 0.01
|
|
|
54
54
|
MAX_AREA_FRAC = 0.60
|
|
55
55
|
MIN_FILL = 0.60 # the source blob must fill this much of the final quad
|
|
56
56
|
MIN_SIDE_RATIO = 0.08 # reject slivers: shortest side vs longest, after perspective
|
|
57
|
+
# Size stops being a virtue past this fraction of the frame. score_contour used
|
|
58
|
+
# to reward area linearly, so on a real photo (7 Sep 2026) the sunlit TABLE at
|
|
59
|
+
# 34% of the frame beat the phone screen at 9% by 3.6x on that term alone, and
|
|
60
|
+
# won outright. A screen is never the biggest thing in a photograph of a room;
|
|
61
|
+
# it is only ever big enough. Past this plateau extra area buys nothing, so
|
|
62
|
+
# candidates separate on fill and band width instead — which is what actually
|
|
63
|
+
# distinguishes glass from furniture.
|
|
64
|
+
AREA_PLATEAU = 0.15
|
|
65
|
+
# How many connected components each tone band offers up. See blobs_for_band:
|
|
66
|
+
# taking only the largest is what made a real photo undetectable, because the
|
|
67
|
+
# phone was the second-largest dark region in its band. Six is measured, not
|
|
68
|
+
# guessed — the winning component on that photo is rank 2, and nothing useful
|
|
69
|
+
# was found past rank 4 on any fixture; the extra two are headroom.
|
|
70
|
+
COMPONENTS_PER_BAND = 6
|
|
71
|
+
# Saturation percentile below which a region counts as "neutral". Devices are
|
|
72
|
+
# grey, black and white; furniture, skin, fabric and foliage are not. Measured
|
|
73
|
+
# 7 Sep 2026 on a photo where tone banding failed completely: the screen sits at
|
|
74
|
+
# saturation 1.9, the sunlit table it was being confused with at 78.6. Taking
|
|
75
|
+
# the image's own 25th percentile adapts to the photograph instead of fixing a
|
|
76
|
+
# level — a studio shot on white and a warm interior need different numbers.
|
|
77
|
+
# Swept, not fixed — the same reasoning the tone sweep is built on. A single
|
|
78
|
+
# percentile breaks whenever the neutral thing is smaller than the percentile
|
|
79
|
+
# (a phone occupying 15% of a colourful frame pulls p25 up into the colour and
|
|
80
|
+
# the mask swallows the picture). Trying several and letting the scoring decide
|
|
81
|
+
# costs one pass each and removes the guess.
|
|
82
|
+
NEUTRAL_PERCENTILES = (5, 10, 15, 25, 35)
|
|
83
|
+
NEUTRAL_FLOOR = 20 # never threshold below this: an all-grey photo
|
|
84
|
+
# A screen has rounded corners. A patch of table cut out by a threshold has
|
|
85
|
+
# perfectly sharp ones, and measure_corner_radius returns 0.0px for it — which
|
|
86
|
+
# turns out to be the cleanest way to tell a real screen from a lookalike, and
|
|
87
|
+
# it is a SHAPE property, not photometry (three photometric arbiters have been
|
|
88
|
+
# measured and rejected here; see the note above score_edge_contour).
|
|
89
|
+
MIN_ROUNDING_PX = 2.0
|
|
90
|
+
# measure_corner_radius makes FOUR independent estimates of one number. If they
|
|
91
|
+
# disagree by more than this, the outline is not a rounded rectangle with a
|
|
92
|
+
# consistent radius and the median is meaningless — so "it has rounded corners"
|
|
93
|
+
# is not evidence and must not be treated as any. Measured 7 Sep 2026 across
|
|
94
|
+
# nine real mockup photos: the seven correct detections spread 31-173%, the two
|
|
95
|
+
# confidently-wrong ones 247% and 464%.
|
|
96
|
+
#
|
|
97
|
+
# **This is the weakest number in the file.** 173 against 247 is a 1.4x margin
|
|
98
|
+
# on nine samples, where every other threshold here was set with a 4x margin or
|
|
99
|
+
# better. If a correct detection is ever rejected as "shape is not consistent",
|
|
100
|
+
# this is the line to raise, and it should be re-measured on a bigger set before
|
|
101
|
+
# it is trusted further.
|
|
102
|
+
MAX_RADIUS_SPREAD = 2.0
|
|
103
|
+
# A quad with a corner outside the frame is not a screen this tool can fit, and
|
|
104
|
+
# its handles cannot be grabbed — the user is left with a wrong quad and no way
|
|
105
|
+
# back (same photo: two corners at y=-50 and y=1837 on a 1792-tall image).
|
|
106
|
+
# Rejected at validate_quad, the one choke point every result passes through.
|
|
107
|
+
OOB_MARGIN = 2.0
|
|
108
|
+
# Gross disagreement between the two detectors, as a fraction of the image
|
|
109
|
+
# diagonal. Measured 7 Sep 2026: the reference photo, where detection genuinely
|
|
110
|
+
# works, sits at 0.033; the photo where both detectors missed the screen
|
|
111
|
+
# entirely sits at 0.555 — seventeen times worse. 0.15 is well clear of both.
|
|
112
|
+
ABSTAIN_GAP = 0.15
|
|
57
113
|
# A screen fills MOST of the body it sits in; content drawn on a screen is a
|
|
58
114
|
# small part of it. That one ratio separates "step inward to the screen" from
|
|
59
115
|
# "don't step into a panel", and it arbitrates between the two detectors too.
|
|
@@ -65,8 +121,24 @@ def _odd(n: int) -> int:
|
|
|
65
121
|
return n if n % 2 else n + 1
|
|
66
122
|
|
|
67
123
|
|
|
68
|
-
def
|
|
69
|
-
|
|
124
|
+
def blobs_for_band(gray: np.ndarray, lo: int, hi: int, close_k: int, open_k: int,
|
|
125
|
+
keep: int = COMPONENTS_PER_BAND):
|
|
126
|
+
"""Mask -> morphology -> the `keep` largest connected components.
|
|
127
|
+
|
|
128
|
+
Returns a list of contours, biggest first.
|
|
129
|
+
|
|
130
|
+
This used to return only the single largest component, and that one line
|
|
131
|
+
was why a real photo could not be detected at all. On an iPhone lying on a
|
|
132
|
+
sunlit table (7 Sep 2026) the phone was the SECOND-largest dark region in
|
|
133
|
+
its band — the table's shadow was bigger — so the screen was discarded
|
|
134
|
+
before scoring ever saw it. No amount of re-scoring can rank a candidate
|
|
135
|
+
that was never generated: measured, the best quad the old sweep could
|
|
136
|
+
produce sat 693px from the true screen; keeping the runners-up brings that
|
|
137
|
+
to 136px, which is a startable position.
|
|
138
|
+
|
|
139
|
+
"Largest" is a guess about the answer dressed up as an optimisation. The
|
|
140
|
+
scoring function is what decides; this function's job is only to offer.
|
|
141
|
+
"""
|
|
70
142
|
mask = cv2.inRange(gray, lo, hi)
|
|
71
143
|
mask = cv2.morphologyEx(
|
|
72
144
|
mask, cv2.MORPH_CLOSE,
|
|
@@ -81,8 +153,19 @@ def blob_for_band(gray: np.ndarray, lo: int, hi: int, close_k: int, open_k: int)
|
|
|
81
153
|
# leaves fitLine with nothing to fit.
|
|
82
154
|
contours, _ = cv2.findContours(mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE)
|
|
83
155
|
if not contours:
|
|
84
|
-
return
|
|
85
|
-
return
|
|
156
|
+
return []
|
|
157
|
+
return sorted(contours, key=cv2.contourArea, reverse=True)[:keep]
|
|
158
|
+
|
|
159
|
+
|
|
160
|
+
def size_term(area: float, img_area: float) -> float:
|
|
161
|
+
"""Area reward that saturates at AREA_PLATEAU.
|
|
162
|
+
|
|
163
|
+
Below the plateau, bigger is better — it separates a real region from
|
|
164
|
+
speckle. At or above it, the term is 1.0 and stops discriminating, because
|
|
165
|
+
beyond a plausible screen size "bigger" stops being evidence of screen-ness
|
|
166
|
+
and starts being evidence of furniture.
|
|
167
|
+
"""
|
|
168
|
+
return min(area / img_area, AREA_PLATEAU) / AREA_PLATEAU
|
|
86
169
|
|
|
87
170
|
|
|
88
171
|
def score_contour(contour, img_area: float):
|
|
@@ -99,7 +182,7 @@ def score_contour(contour, img_area: float):
|
|
|
99
182
|
# Fill ratio: a real screen fills its own quad almost completely. A shadow
|
|
100
183
|
# or a wall patch is ragged and fills far less.
|
|
101
184
|
fill = min(area / quad_area, 1.0)
|
|
102
|
-
return (fill ** 3) * (area
|
|
185
|
+
return (fill ** 3) * size_term(area, img_area), quad
|
|
103
186
|
|
|
104
187
|
|
|
105
188
|
# Two "edge support" arbiters — score each candidate quad by how much of its
|
|
@@ -140,7 +223,7 @@ def score_edge_contour(contour, img_area: float):
|
|
|
140
223
|
if quad_area < MIN_AREA_FRAC * img_area or quad_area > MAX_AREA_FRAC * img_area:
|
|
141
224
|
return 0.0, None
|
|
142
225
|
fill = min(float(cv2.contourArea(contour)) / max(quad_area, 1e-6), 1.0)
|
|
143
|
-
return fill * (quad_area
|
|
226
|
+
return fill * size_term(quad_area, img_area), quad
|
|
144
227
|
|
|
145
228
|
|
|
146
229
|
def approx_quad(contour):
|
|
@@ -365,7 +448,7 @@ def measure_corner_radius(contour, corners: np.ndarray):
|
|
|
365
448
|
# segmented contour, so it isn't picking between competing nearby edges.
|
|
366
449
|
|
|
367
450
|
|
|
368
|
-
def validate_quad(corners: np.ndarray, contour, img_area: float):
|
|
451
|
+
def validate_quad(corners: np.ndarray, contour, img_area: float, img_shape=None):
|
|
369
452
|
"""Is this *final* quad plausibly a screen? Returns (ok, reason).
|
|
370
453
|
|
|
371
454
|
The area gate in score_contour() runs against the CONTOUR, before
|
|
@@ -375,6 +458,15 @@ def validate_quad(corners: np.ndarray, contour, img_area: float):
|
|
|
375
458
|
confident detection (3 Sep 2026, an earlier finding). Everything returned to a caller
|
|
376
459
|
goes through here.
|
|
377
460
|
"""
|
|
461
|
+
if img_shape is not None:
|
|
462
|
+
h, w = float(img_shape[0]), float(img_shape[1])
|
|
463
|
+
for (x, y), label in zip(corners, ["TL", "TR", "BR", "BL"], strict=True):
|
|
464
|
+
if not (-OOB_MARGIN <= x <= w + OOB_MARGIN
|
|
465
|
+
and -OOB_MARGIN <= y <= h + OOB_MARGIN):
|
|
466
|
+
return False, ("%s lands outside the photo at (%.0f, %.0f) on a "
|
|
467
|
+
"%.0fx%.0f image — a screen this tool can fit is "
|
|
468
|
+
"inside the frame, and an off-canvas handle can't "
|
|
469
|
+
"be dragged back" % (label, x, y, w, h))
|
|
378
470
|
area = float(cv2.contourArea(corners.astype(np.float32)))
|
|
379
471
|
frac = area / img_area
|
|
380
472
|
if frac < MIN_AREA_FRAC:
|
|
@@ -398,7 +490,7 @@ def validate_quad(corners: np.ndarray, contour, img_area: float):
|
|
|
398
490
|
return True, ""
|
|
399
491
|
|
|
400
492
|
|
|
401
|
-
def _finalize(candidates, img_area: float, refine: bool = True):
|
|
493
|
+
def _finalize(candidates, img_area: float, refine: bool = True, img_shape=None):
|
|
402
494
|
"""Best candidate that survives refinement AND validation.
|
|
403
495
|
|
|
404
496
|
Walks candidates best-score-first rather than trusting the top one: a
|
|
@@ -422,7 +514,7 @@ def _finalize(candidates, img_area: float, refine: bool = True):
|
|
|
422
514
|
else:
|
|
423
515
|
refined, did_refine = quad, False
|
|
424
516
|
corners = order_quad(refined)
|
|
425
|
-
ok, why = validate_quad(corners, contour, img_area)
|
|
517
|
+
ok, why = validate_quad(corners, contour, img_area, img_shape)
|
|
426
518
|
if ok:
|
|
427
519
|
return {
|
|
428
520
|
"corners": [[round(float(x), 1), round(float(y), 1)] for x, y in corners],
|
|
@@ -456,11 +548,10 @@ def detect_tone(gray: np.ndarray, tone=None):
|
|
|
456
548
|
|
|
457
549
|
candidates = []
|
|
458
550
|
for lo, hi in bands:
|
|
459
|
-
contour
|
|
460
|
-
|
|
461
|
-
|
|
462
|
-
|
|
463
|
-
if score > 0 and quad is not None:
|
|
551
|
+
for contour in blobs_for_band(gray, lo, hi, close_k, open_k):
|
|
552
|
+
score, quad = score_contour(contour, img_area)
|
|
553
|
+
if score <= 0 or quad is None:
|
|
554
|
+
continue
|
|
464
555
|
# Prefer a narrow band. A screen is tonally uniform; a band wide
|
|
465
556
|
# enough to also swallow the phone's shadowed body scores well on
|
|
466
557
|
# area but produces edges that follow the body, not the glass.
|
|
@@ -469,7 +560,7 @@ def detect_tone(gray: np.ndarray, tone=None):
|
|
|
469
560
|
score *= (16.0 / (hi - lo + 1)) ** 0.25
|
|
470
561
|
candidates.append((score, order_quad(quad), contour, (lo, hi)))
|
|
471
562
|
|
|
472
|
-
res = _finalize(candidates, img_area)
|
|
563
|
+
res = _finalize(candidates, img_area, img_shape=gray.shape[:2])
|
|
473
564
|
if res is None:
|
|
474
565
|
return None
|
|
475
566
|
band = res.pop("_tag")
|
|
@@ -517,7 +608,7 @@ def detect_edges(gray: np.ndarray):
|
|
|
517
608
|
if score > 0 and quad is not None:
|
|
518
609
|
candidates.append((score, order_quad(quad), c, (lo, hi)))
|
|
519
610
|
|
|
520
|
-
res = _finalize(candidates, img_area, refine=False)
|
|
611
|
+
res = _finalize(candidates, img_area, refine=False, img_shape=gray.shape[:2])
|
|
521
612
|
if res is None:
|
|
522
613
|
return None
|
|
523
614
|
thr = res.pop("_tag")
|
|
@@ -526,7 +617,71 @@ def detect_edges(gray: np.ndarray):
|
|
|
526
617
|
return res
|
|
527
618
|
|
|
528
619
|
|
|
529
|
-
def
|
|
620
|
+
def detect_saturation(bgr: np.ndarray):
|
|
621
|
+
"""Neutral-region segmentation — the third detector, and the only one that
|
|
622
|
+
looks at colour.
|
|
623
|
+
|
|
624
|
+
tone and edge both run on grayscale, which throws away the single most
|
|
625
|
+
useful cue a photograph of a device offers: **devices are neutral.** A
|
|
626
|
+
phone is grey, black or white; the table, sofa, hand or plant it is lying
|
|
627
|
+
on almost never is. On the 7 Sep 2026 photo the screen measured saturation
|
|
628
|
+
1.9 against the sunlit table's 78.6, and this detector lands 29px from the
|
|
629
|
+
true quad where tone lands 1030px away.
|
|
630
|
+
|
|
631
|
+
The threshold is the image's own 25th saturation percentile rather than a
|
|
632
|
+
fixed level, so a studio shot on white and a warm interior both work.
|
|
633
|
+
"""
|
|
634
|
+
hsv = cv2.cvtColor(bgr, cv2.COLOR_BGR2HSV)
|
|
635
|
+
sat = hsv[:, :, 1]
|
|
636
|
+
h, w = sat.shape[:2]
|
|
637
|
+
img_area = float(h * w)
|
|
638
|
+
short = min(h, w)
|
|
639
|
+
close_k = _odd(CLOSE_FRAC * short)
|
|
640
|
+
open_k = _odd(OPEN_FRAC * short)
|
|
641
|
+
|
|
642
|
+
thresholds = {int(max(NEUTRAL_FLOOR, np.percentile(sat, p)))
|
|
643
|
+
for p in NEUTRAL_PERCENTILES}
|
|
644
|
+
otsu, _ = cv2.threshold(sat, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)
|
|
645
|
+
thresholds.add(int(max(NEUTRAL_FLOOR, otsu)))
|
|
646
|
+
candidates = []
|
|
647
|
+
for thr in sorted(thresholds):
|
|
648
|
+
for contour in blobs_for_band(sat, 0, thr, close_k, open_k):
|
|
649
|
+
score, quad = score_contour(contour, img_area)
|
|
650
|
+
if score > 0 and quad is not None:
|
|
651
|
+
# Prefer a tighter neutral threshold, for the same reason the
|
|
652
|
+
# tone sweep prefers a narrow band: a loose one swallows the
|
|
653
|
+
# device body and the wall behind it along with the screen.
|
|
654
|
+
candidates.append((score * (32.0 / max(thr, 1)) ** 0.25,
|
|
655
|
+
order_quad(quad), contour, thr))
|
|
656
|
+
|
|
657
|
+
res = _finalize(candidates, img_area, img_shape=sat.shape[:2])
|
|
658
|
+
if res is None:
|
|
659
|
+
return None
|
|
660
|
+
res.pop("_tag")
|
|
661
|
+
res["method"] = "saturation"
|
|
662
|
+
res["neutral_threshold"] = thr
|
|
663
|
+
return res
|
|
664
|
+
|
|
665
|
+
|
|
666
|
+
def has_rounded_corners(result) -> bool:
|
|
667
|
+
"""Did this quad's own outline actually curve at the corners?
|
|
668
|
+
|
|
669
|
+
A device screen is a rounded rectangle. A patch of table cut out of a
|
|
670
|
+
threshold mask is a polygon with sharp corners, and measure_corner_radius
|
|
671
|
+
reports 0.0px for it. That single number separated the right answer from
|
|
672
|
+
three wrong ones on the photo this was built for, and unlike edge strength
|
|
673
|
+
or ring contrast it is a property of the shape rather than of the light.
|
|
674
|
+
"""
|
|
675
|
+
cr = result["corner_radius"]
|
|
676
|
+
if float(cr["photo_px"]) <= MIN_ROUNDING_PX:
|
|
677
|
+
return False
|
|
678
|
+
per = [float(v) for v in cr["per_corner_px"]]
|
|
679
|
+
r = float(cr["photo_px"])
|
|
680
|
+
spread = (max(per) - min(per)) / max(r, 1e-6)
|
|
681
|
+
return spread <= MAX_RADIUS_SPREAD
|
|
682
|
+
|
|
683
|
+
|
|
684
|
+
def detect(gray: np.ndarray, tone=None, method="auto", color=None):
|
|
530
685
|
"""Run both detectors; arbitrate on how the two quads nest.
|
|
531
686
|
|
|
532
687
|
The two fail on opposite things. Tone banding needs a tonally uniform
|
|
@@ -559,13 +714,40 @@ def detect(gray: np.ndarray, tone=None, method="auto"):
|
|
|
559
714
|
r = detect_edges(gray)
|
|
560
715
|
if r:
|
|
561
716
|
results.append(r)
|
|
717
|
+
if method in ("auto", "saturation") and tone is None and color is not None:
|
|
718
|
+
r = detect_saturation(color)
|
|
719
|
+
if r:
|
|
720
|
+
results.append(r)
|
|
562
721
|
|
|
563
722
|
if not results:
|
|
564
723
|
return None
|
|
565
724
|
|
|
725
|
+
# tone and saturation are the same algorithm on different channels, so they
|
|
726
|
+
# are compared to each other before anything else, on whether the region
|
|
727
|
+
# each found actually has rounded corners. A patch of table cut out of a
|
|
728
|
+
# threshold mask measures 0.0px; a screen measures a real radius. On the
|
|
729
|
+
# 7 Sep photo that is the whole ball game — tone 0.0px against saturation's
|
|
730
|
+
# 62.4px, so the channel that found the phone is the one that goes forward.
|
|
731
|
+
#
|
|
732
|
+
# **Only these two.** The edge detector is deliberately exempt: its contour
|
|
733
|
+
# is a Canny ring tracing both sides of a boundary, not a filled region's
|
|
734
|
+
# silhouette, so measure_corner_radius reports an artifact for it — 0.0px
|
|
735
|
+
# even when its quad is the correct one to 1.4px (measured on the
|
|
736
|
+
# gradient-screen fixture, where an earlier version of this filter threw
|
|
737
|
+
# away the right answer). _finalize already skips corner refinement on that
|
|
738
|
+
# path for the same reason.
|
|
739
|
+
region = [r for r in results if r["method"] in ("tone", "saturation")]
|
|
740
|
+
if len(region) == 2:
|
|
741
|
+
rounded = [r for r in region if has_rounded_corners(r)]
|
|
742
|
+
if len(rounded) == 1:
|
|
743
|
+
loser = next(r for r in region if r is not rounded[0])
|
|
744
|
+
results = [r for r in results if r is not loser]
|
|
745
|
+
|
|
566
746
|
t = next((r for r in results if r["method"] == "tone"), None)
|
|
567
747
|
e = next((r for r in results if r["method"] == "edge"), None)
|
|
568
|
-
|
|
748
|
+
sat = next((r for r in results if r["method"] == "saturation"), None)
|
|
749
|
+
why = "it was the only detector left after the rounded-corner filter" \
|
|
750
|
+
if len(results) == 1 else "it is the detector that refines corners"
|
|
569
751
|
if t and e:
|
|
570
752
|
tq, eq = t["_corners_np"], e["_corners_np"]
|
|
571
753
|
ta = float(cv2.contourArea(tq.astype(np.float32)))
|
|
@@ -581,28 +763,46 @@ def detect(gray: np.ndarray, tone=None, method="auto"):
|
|
|
581
763
|
"its area — a screen inside a device body" % (ratio * 100))
|
|
582
764
|
else:
|
|
583
765
|
best, why = t, "the two quads aren't nested; the tone detector refines corners"
|
|
766
|
+
elif t is None and sat is not None:
|
|
767
|
+
# tone was dropped for having sharp corners, or never fired. The
|
|
768
|
+
# surviving region detector refines corners; edge does not.
|
|
769
|
+
best, why = sat, ("the saturation detector found a region with rounded "
|
|
770
|
+
"corners where tone did not")
|
|
584
771
|
else:
|
|
585
|
-
best = t or e
|
|
586
|
-
|
|
587
|
-
|
|
588
|
-
|
|
772
|
+
best = t or e or sat
|
|
773
|
+
if best is None:
|
|
774
|
+
best = results[0]
|
|
775
|
+
|
|
776
|
+
if len(results) > 1:
|
|
777
|
+
# Agreement is measured against the CLOSEST other detector, not against
|
|
778
|
+
# "the other one" — there are three now, and a third opinion that lands
|
|
779
|
+
# somewhere else entirely should not erase the fact that two of them
|
|
780
|
+
# landed together. Corroboration by any one independent method is the
|
|
781
|
+
# evidence worth reporting.
|
|
589
782
|
diag = float(np.hypot(*gray.shape[:2]))
|
|
590
|
-
|
|
783
|
+
others = [r for r in results if r is not best]
|
|
784
|
+
gaps = [(float(np.max(np.linalg.norm(best["_corners_np"]
|
|
785
|
+
- r["_corners_np"], axis=1))) / diag, r)
|
|
786
|
+
for r in others]
|
|
787
|
+
spread, nearest = min(gaps, key=lambda g: g[0])
|
|
591
788
|
agree = bool(spread < 0.02)
|
|
592
789
|
best["agreement"] = {
|
|
593
790
|
"both_found": True,
|
|
594
791
|
"max_corner_gap_frac_of_diagonal": round(spread, 4),
|
|
595
792
|
"agree": agree,
|
|
596
|
-
"
|
|
597
|
-
|
|
793
|
+
"agrees_with": nearest["method"] if agree else None,
|
|
794
|
+
"note": ("The %s and %s detectors independently landed on the same "
|
|
795
|
+
"quad — that is real evidence, not one algorithm's opinion."
|
|
796
|
+
% (best["method"], nearest["method"])
|
|
598
797
|
if agree else
|
|
599
|
-
"
|
|
600
|
-
"Showing the %s one because %s —
|
|
601
|
-
|
|
798
|
+
"No two detectors agree; the closest other (%s) is %.0f%% of "
|
|
799
|
+
"the image diagonal away. Showing the %s one because %s — "
|
|
800
|
+
"but check all four corners."
|
|
801
|
+
% (nearest["method"], spread * 100, best["method"], why)),
|
|
602
802
|
}
|
|
603
803
|
best["agreement"]["chosen_because"] = why
|
|
604
|
-
other =
|
|
605
|
-
|
|
804
|
+
best["other"] = [{"method": r["method"], "corners": r["corners"]}
|
|
805
|
+
for r in others]
|
|
606
806
|
else:
|
|
607
807
|
best["agreement"] = {
|
|
608
808
|
"both_found": False,
|
|
@@ -610,6 +810,57 @@ def detect(gray: np.ndarray, tone=None, method="auto"):
|
|
|
610
810
|
"note": ("Only the %s detector found anything — a first guess to "
|
|
611
811
|
"correct, not a measurement." % best["method"]),
|
|
612
812
|
}
|
|
813
|
+
|
|
814
|
+
# Abstention. detect.py has always PROMISED to fail honestly rather than
|
|
815
|
+
# emit a confident wrong quad, but nothing enforced it: on 7 Sep 2026 a real
|
|
816
|
+
# photo produced a quad on the table, with `agree` false, the two detectors
|
|
817
|
+
# 55% of the diagonal apart, the radius spread at 104%, and two corners off
|
|
818
|
+
# the image — every indicator of failure present, and a result returned
|
|
819
|
+
# anyway, exit 0. The evidence was already being computed; it just wasn't
|
|
820
|
+
# gating. Two conditions, either of which means "we do not know":
|
|
821
|
+
ag = best["agreement"]
|
|
822
|
+
# Disagreement only counts against a CREDIBLE peer. Measured 7 Sep 2026 on
|
|
823
|
+
# eight real mockup photos: where saturation correctly found a phone that
|
|
824
|
+
# tone and edge had both missed, the winner was of course miles from the two
|
|
825
|
+
# that failed — and this gate then threw the right answer away as
|
|
826
|
+
# "disagreement". A detector that found a sharp-cornered patch of floor does
|
|
827
|
+
# not get a vote on whether the rounded thing is a screen. So the gap is
|
|
828
|
+
# re-measured against peers that also found something screen-shaped; when
|
|
829
|
+
# there are none, being alone is not evidence of being wrong.
|
|
830
|
+
peers = [r for r in results if r is not best and has_rounded_corners(r)]
|
|
831
|
+
if peers:
|
|
832
|
+
diag = float(np.hypot(*gray.shape[:2]))
|
|
833
|
+
peer_gap = min(float(np.max(np.linalg.norm(best["_corners_np"]
|
|
834
|
+
- r["_corners_np"], axis=1))) / diag
|
|
835
|
+
for r in peers)
|
|
836
|
+
else:
|
|
837
|
+
peer_gap = None
|
|
838
|
+
gross = bool(peer_gap is not None and peer_gap > ABSTAIN_GAP)
|
|
839
|
+
if peer_gap is not None:
|
|
840
|
+
ag["credible_peer_gap"] = round(peer_gap, 4)
|
|
841
|
+
# A measurable corner radius counts as evidence in its own right: it says
|
|
842
|
+
# the thing found is shaped like a screen, which is what abstention exists
|
|
843
|
+
# to doubt. Without this a good saturation result on a hard photo would be
|
|
844
|
+
# thrown away for want of a second opinion.
|
|
845
|
+
uncorroborated = bool(not ag.get("agree")
|
|
846
|
+
and not best["corner_radius"]["confident"]
|
|
847
|
+
and not has_rounded_corners(best))
|
|
848
|
+
if gross or uncorroborated:
|
|
849
|
+
why = []
|
|
850
|
+
if gross:
|
|
851
|
+
why.append("two detectors that each found something screen-shaped are "
|
|
852
|
+
"%.0f%% of the image diagonal apart" % (peer_gap * 100))
|
|
853
|
+
if uncorroborated:
|
|
854
|
+
why.append("nothing corroborates the quad (the detectors don't agree "
|
|
855
|
+
"and the corner radius isn't measurable)")
|
|
856
|
+
best["abstained"] = True
|
|
857
|
+
best["abstain_reason"] = (
|
|
858
|
+
"Detection abstained: " + " and ".join(why) + ". The quad below is "
|
|
859
|
+
"kept for inspection but is not offered as a starting position — "
|
|
860
|
+
"place the four edges by hand."
|
|
861
|
+
)
|
|
862
|
+
else:
|
|
863
|
+
best["abstained"] = False
|
|
613
864
|
return best
|
|
614
865
|
|
|
615
866
|
|
|
@@ -655,6 +906,8 @@ def main() -> None:
|
|
|
655
906
|
if args.out_zooms:
|
|
656
907
|
written["zooms"] = write_zooms(photo, corners, args.out_zooms)
|
|
657
908
|
|
|
909
|
+
if result.get("abstained"):
|
|
910
|
+
print(result["abstain_reason"], file=sys.stderr)
|
|
658
911
|
print(json.dumps({
|
|
659
912
|
**result,
|
|
660
913
|
"photo_size": [photo.shape[1], photo.shape[0]],
|
|
@@ -672,6 +925,12 @@ def main() -> None:
|
|
|
672
925
|
"this script, so an unconfirmed guess cannot reach a composite.",
|
|
673
926
|
}, indent=1))
|
|
674
927
|
|
|
928
|
+
# The docstring's promise, finally enforced: an uncorroborated guess exits
|
|
929
|
+
# non-zero. The JSON is still printed above so a caller can inspect what was
|
|
930
|
+
# rejected and why.
|
|
931
|
+
if result.get("abstained"):
|
|
932
|
+
sys.exit(2)
|
|
933
|
+
|
|
675
934
|
|
|
676
935
|
if __name__ == "__main__":
|
|
677
936
|
main()
|
package/scripts/grade.py
CHANGED
|
@@ -61,48 +61,75 @@ def _stats(lab: np.ndarray, sel: np.ndarray) -> tuple[np.ndarray, np.ndarray]:
|
|
|
61
61
|
return px.mean(axis=0), px.std(axis=0) + 1e-6
|
|
62
62
|
|
|
63
63
|
|
|
64
|
-
def
|
|
65
|
-
|
|
66
|
-
"""
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
the
|
|
73
|
-
|
|
64
|
+
def light_params(photo: np.ndarray, warped: np.ndarray, mask: np.ndarray,
|
|
65
|
+
strength: float = DEFAULT_STRENGTH):
|
|
66
|
+
"""Measure the correction ONCE, so it can be applied to many frames.
|
|
67
|
+
|
|
68
|
+
Split out of match_light for video. The correction depends on the
|
|
69
|
+
screen's own content through `m_in`/`s_in`, so measuring it per frame makes
|
|
70
|
+
it drift as the UI scrolls from a dark screen to a light one — the injected
|
|
71
|
+
screen would visibly pulse. Deriving the parameters from one frame and
|
|
72
|
+
applying the identical transform to all of them is the whole fix.
|
|
73
|
+
|
|
74
|
+
Returns None when there is too little context to measure honestly, which
|
|
75
|
+
the caller must treat as "leave the frame alone".
|
|
74
76
|
"""
|
|
75
77
|
if strength <= 0:
|
|
76
|
-
return
|
|
78
|
+
return None
|
|
77
79
|
ring = surround_ring(mask)
|
|
78
|
-
if int(ring.sum()) < 500:
|
|
79
|
-
return
|
|
80
|
-
|
|
81
|
-
lab_photo = cv2.cvtColor(photo, cv2.COLOR_BGR2LAB).astype(np.float64)
|
|
82
|
-
lab_warp = cv2.cvtColor(warped, cv2.COLOR_BGR2LAB).astype(np.float64)
|
|
80
|
+
if int(ring.sum()) < 500:
|
|
81
|
+
return None
|
|
83
82
|
inside = (mask > 200).astype(np.uint8)
|
|
84
83
|
if int(inside.sum()) < 500:
|
|
85
|
-
return
|
|
86
|
-
|
|
84
|
+
return None
|
|
85
|
+
lab_photo = cv2.cvtColor(photo, cv2.COLOR_BGR2LAB).astype(np.float64)
|
|
86
|
+
lab_warp = cv2.cvtColor(warped, cv2.COLOR_BGR2LAB).astype(np.float64)
|
|
87
87
|
m_out, s_out = _stats(lab_photo, ring)
|
|
88
88
|
m_in, s_in = _stats(lab_warp, inside)
|
|
89
|
-
|
|
89
|
+
return {
|
|
90
|
+
"m_in": m_in, "s_in": s_in, "m_out": m_out, "s_out": s_out,
|
|
91
|
+
"strength": float(strength),
|
|
92
|
+
# Same clamp as match_light: a screen is emissive and may be brighter
|
|
93
|
+
# than the room, so L moves by a bounded mean shift only.
|
|
94
|
+
"dL": float(np.clip(m_out[0] - m_in[0], -12.0, 12.0)) * float(strength),
|
|
95
|
+
}
|
|
96
|
+
|
|
97
|
+
|
|
98
|
+
def apply_light(warped: np.ndarray, params) -> np.ndarray:
|
|
99
|
+
"""Apply parameters from light_params() to one frame."""
|
|
100
|
+
if params is None:
|
|
101
|
+
return warped
|
|
102
|
+
lab_warp = cv2.cvtColor(warped, cv2.COLOR_BGR2LAB).astype(np.float64)
|
|
103
|
+
m_in, s_in = params["m_in"], params["s_in"]
|
|
104
|
+
m_out, s_out = params["m_out"], params["s_out"]
|
|
105
|
+
strength = params["strength"]
|
|
90
106
|
out = lab_warp.copy()
|
|
91
|
-
# a,b: full Reinhard transfer, scaled by strength.
|
|
92
107
|
for c in (1, 2):
|
|
93
108
|
moved = (lab_warp[:, :, c] - m_in[c]) * float(s_out[c] / s_in[c]) + m_out[c]
|
|
94
109
|
out[:, :, c] = lab_warp[:, :, c] + (moved - lab_warp[:, :, c]) * strength
|
|
95
|
-
|
|
96
|
-
# the screen off. 12 is about a stop; beyond that it stops reading as the
|
|
97
|
-
# same screenshot.
|
|
98
|
-
dL = float(np.clip(m_out[0] - m_in[0], -12.0, 12.0)) * strength
|
|
99
|
-
out[:, :, 0] = lab_warp[:, :, 0] + dL
|
|
100
|
-
|
|
110
|
+
out[:, :, 0] = lab_warp[:, :, 0] + params["dL"]
|
|
101
111
|
out[:, :, 0] = np.clip(out[:, :, 0], 0, 255)
|
|
102
112
|
out[:, :, 1:] = np.clip(out[:, :, 1:], 0, 255)
|
|
103
113
|
return cv2.cvtColor(out.astype(np.uint8), cv2.COLOR_LAB2BGR)
|
|
104
114
|
|
|
105
115
|
|
|
116
|
+
def match_light(photo: np.ndarray, warped: np.ndarray, mask: np.ndarray,
|
|
117
|
+
strength: float = DEFAULT_STRENGTH) -> np.ndarray:
|
|
118
|
+
"""Move the injected screen's cast and exposure toward the surrounding light.
|
|
119
|
+
|
|
120
|
+
Chroma (a,b) is matched on mean AND spread — a cast is exactly a chroma mean
|
|
121
|
+
offset, and a room with weak colour should not receive a saturated screen.
|
|
122
|
+
Luminance is matched on MEAN ONLY, and bounded: a screen is emissive and is
|
|
123
|
+
allowed to be brighter than its surroundings, so rescaling its L spread to
|
|
124
|
+
the bezel's would crush the UI's own contrast. That asymmetry is the whole
|
|
125
|
+
design of this function.
|
|
126
|
+
"""
|
|
127
|
+
# One implementation, two entry points: measuring and applying are the same
|
|
128
|
+
# arithmetic whether it runs on a still or on frame 900 of a clip. Keeping a
|
|
129
|
+
# second copy here is how the two paths would drift.
|
|
130
|
+
return apply_light(warped, light_params(photo, warped, mask, strength))
|
|
131
|
+
|
|
132
|
+
|
|
106
133
|
def measure_grain(photo: np.ndarray, ring: np.ndarray) -> float:
|
|
107
134
|
"""The photo's noise floor, in grey levels, measured where the screen isn't.
|
|
108
135
|
|