screengraft 0.17.0 → 0.21.0

package/README.md

@@ -1,6 +1,6 @@
 # screengraft
 
-**Put a UI screenshot onto a photographed screen so the perspective is exactly right.**
+**Put a UI screenshot — or a screen recording — onto a photographed screen so the perspective is exactly right.**
 
 ![The fitting workbench: photo with the screen quad on the left, live composite on the right, and a magnified strip across the edge below](docs/workbench.png)
 
@@ -13,6 +13,9 @@ transform between them. The screenshot lands on the glass because the geometry
 says it must, not because a model thought it looked about right. Same inputs,
 same output, every time.
 
+Point it at a **video** instead and the same fit renders every frame: record a
+prototype, then put the recording inside a real photograph.
+
 ---
 
 ## What it does
@@ -26,6 +29,15 @@ same output, every time.
 - **Realism pass** *(optional)* — matches the screen's white balance and grain to
   the light in the room, and can lift the device's real reflections from a
   screen-off frame of the same shot.
+- **Emissive screens** *(optional)* — a display emits light *and* reflects the
+  room, which is why a switched-off phone looks dark grey rather than black.
+  Paint a true-black UI on flat and it reads as a hole cut in the photo. Turn
+  this on and the screenshot composites over the device's own glass, so the
+  photo's highlights carry across the screen.
+- **Video, not just stills.** The screen source can be an `mp4`/`mov`/`webm`.
+  You match the edges on one frame and every frame gets that same geometry — the
+  photograph is still, so there is nothing to track and nothing to drift. Output
+  is H.264 at CRF 16 or ProRes 422 HQ.
 - **You confirm every fit.** Detection is advisory and says so; you drag the four
   edges onto the glass with a magnified loupe. A silent misdetection producing a
   confident, wrong result is the one failure this tool refuses to have.
@@ -36,6 +48,10 @@ same output, every time.
 without it. The installer provisions an isolated venv at `~/.screengraft/venv`
 and never touches your system Python.
 
+Video rendering also uses **ffmpeg**, which arrives as a wheel (`imageio-ffmpeg`)
+into that same venv — nothing is installed system-wide. It is optional: without
+it, stills work exactly as before.
+
 ## Install as a Claude Code / Cowork plugin
 
 ```
@@ -104,15 +120,31 @@ outside the screen mask. It never touches the pixels you designed.
 3. **Realism pass** *(optional)* — white balance and exposure toward the
    surrounding light, grain matched to the photo's own noise floor, real
    speculars lifted from a screen-off reference.
+4. **Video**, when the source is a clip — everything a fixed photo and a fixed
+   quad make constant is computed once, and only the frame changes. Three
+   consequences worth naming, because each is a way video normally goes wrong:
+   the light match is measured **once** from the frame you fitted on, so the
+   screen cannot pulse as your UI scrolls from dark to light; the grain stays
+   frozen, because the photograph's own noise does not move; and the screen's
+   antialiased **edge is pixel-identical in every frame**, so there is no edge
+   crawl. Frame 0 of a render is byte-identical to the still composite — the
+   test suite asserts it, because that is what stops the two paths drifting.
 
 ## Roadmap
 
-Done: manual warp, advisory detectors, the fitting workbench, the realism pass.
+Done: manual warp, advisory detectors, the fitting workbench, the realism pass,
+video into a still photo.
+
+Open: **camera-motion tracking** — the photograph itself must currently be a
+still, so a clip of a moving phone is out of scope; **SAM 2 auto-detect** (built
+and measured in a separate repo; it segments the phone body rather than the
+glass, so it is not shipped); **occluder matte**, so a finger in front of the
+screen stays in front.
 
-Open: video tracking (M3), SAM 2 auto-detect (M4 — built and measured in a
-separate repo; it currently segments the phone body rather than the glass, so it
-is not shipped), occluder matte (M5), so a finger in front of the screen stays in
-front.
+Known limits worth stating plainly: detection abstains rather than guessing when
+the background is itself neutral (a pale tiled floor, a plain wall) — you place
+the edges by hand there. And a prototype recording has no motion blur, so a very
+fast scroll will strobe; that is a property of the source, not of the composite.
 
 ## Contributing
 

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "screengraft",
-  "version": "0.17.0",
-  "description": "Put a UI screenshot onto a photographed device screen with the perspective exactly right \u2014 a homography you confirm by hand, not a generative guess.",
+  "version": "0.21.0",
+  "description": "Put a UI screenshot or screen recording onto a photographed device screen with the perspective exactly right — a homography you confirm by hand, not a generative guess.",
   "keywords": [
     "mockup",
     "device-frame",
@@ -9,6 +9,9 @@
     "perspective",
     "compositing",
     "screenshot",
+    "prototype",
+    "screen-recording",
+    "video",
     "design",
     "portfolio",
     "figma"

package/scripts/grade.py

@@ -61,48 +61,75 @@ def _stats(lab: np.ndarray, sel: np.ndarray) -> tuple[np.ndarray, np.ndarray]:
     return px.mean(axis=0), px.std(axis=0) + 1e-6
 
 
-def match_light(photo: np.ndarray, warped: np.ndarray, mask: np.ndarray,
-                strength: float = DEFAULT_STRENGTH) -> np.ndarray:
-    """Move the injected screen's cast and exposure toward the surrounding light.
-
-    Chroma (a,b) is matched on mean AND spread — a cast is exactly a chroma mean
-    offset, and a room with weak colour should not receive a saturated screen.
-    Luminance is matched on MEAN ONLY, and bounded: a screen is emissive and is
-    allowed to be brighter than its surroundings, so rescaling its L spread to
-    the bezel's would crush the UI's own contrast. That asymmetry is the whole
-    design of this function.
+def light_params(photo: np.ndarray, warped: np.ndarray, mask: np.ndarray,
+                 strength: float = DEFAULT_STRENGTH):
+    """Measure the correction ONCE, so it can be applied to many frames.
+
+    Split out of match_light for video. The correction depends on the
+    screen's own content through `m_in`/`s_in`, so measuring it per frame makes
+    it drift as the UI scrolls from a dark screen to a light one — the injected
+    screen would visibly pulse. Deriving the parameters from one frame and
+    applying the identical transform to all of them is the whole fix.
+
+    Returns None when there is too little context to measure honestly, which
+    the caller must treat as "leave the frame alone".
     """
     if strength <= 0:
-        return warped
+        return None
     ring = surround_ring(mask)
-    if int(ring.sum()) < 500:          # too little context to measure honestly
-        return warped
-
-    lab_photo = cv2.cvtColor(photo, cv2.COLOR_BGR2LAB).astype(np.float64)
-    lab_warp = cv2.cvtColor(warped, cv2.COLOR_BGR2LAB).astype(np.float64)
+    if int(ring.sum()) < 500:
+        return None
     inside = (mask > 200).astype(np.uint8)
     if int(inside.sum()) < 500:
-        return warped
-
+        return None
+    lab_photo = cv2.cvtColor(photo, cv2.COLOR_BGR2LAB).astype(np.float64)
+    lab_warp = cv2.cvtColor(warped, cv2.COLOR_BGR2LAB).astype(np.float64)
     m_out, s_out = _stats(lab_photo, ring)
     m_in, s_in = _stats(lab_warp, inside)
-
+    return {
+        "m_in": m_in, "s_in": s_in, "m_out": m_out, "s_out": s_out,
+        "strength": float(strength),
+        # Same clamp as match_light: a screen is emissive and may be brighter
+        # than the room, so L moves by a bounded mean shift only.
+        "dL": float(np.clip(m_out[0] - m_in[0], -12.0, 12.0)) * float(strength),
+    }
+
+
+def apply_light(warped: np.ndarray, params) -> np.ndarray:
+    """Apply parameters from light_params() to one frame."""
+    if params is None:
+        return warped
+    lab_warp = cv2.cvtColor(warped, cv2.COLOR_BGR2LAB).astype(np.float64)
+    m_in, s_in = params["m_in"], params["s_in"]
+    m_out, s_out = params["m_out"], params["s_out"]
+    strength = params["strength"]
     out = lab_warp.copy()
-    # a,b: full Reinhard transfer, scaled by strength.
     for c in (1, 2):
         moved = (lab_warp[:, :, c] - m_in[c]) * float(s_out[c] / s_in[c]) + m_out[c]
         out[:, :, c] = lab_warp[:, :, c] + (moved - lab_warp[:, :, c]) * strength
-    # L: mean shift only, and capped at +-12 L* so a dark room cannot switch
-    # the screen off. 12 is about a stop; beyond that it stops reading as the
-    # same screenshot.
-    dL = float(np.clip(m_out[0] - m_in[0], -12.0, 12.0)) * strength
-    out[:, :, 0] = lab_warp[:, :, 0] + dL
-
+    out[:, :, 0] = lab_warp[:, :, 0] + params["dL"]
     out[:, :, 0] = np.clip(out[:, :, 0], 0, 255)
     out[:, :, 1:] = np.clip(out[:, :, 1:], 0, 255)
     return cv2.cvtColor(out.astype(np.uint8), cv2.COLOR_LAB2BGR)
 
 
+def match_light(photo: np.ndarray, warped: np.ndarray, mask: np.ndarray,
+                strength: float = DEFAULT_STRENGTH) -> np.ndarray:
+    """Move the injected screen's cast and exposure toward the surrounding light.
+
+    Chroma (a,b) is matched on mean AND spread — a cast is exactly a chroma mean
+    offset, and a room with weak colour should not receive a saturated screen.
+    Luminance is matched on MEAN ONLY, and bounded: a screen is emissive and is
+    allowed to be brighter than its surroundings, so rescaling its L spread to
+    the bezel's would crush the UI's own contrast. That asymmetry is the whole
+    design of this function.
+    """
+    # One implementation, two entry points: measuring and applying are the same
+    # arithmetic whether it runs on a still or on frame 900 of a clip. Keeping a
+    # second copy here is how the two paths would drift.
+    return apply_light(warped, light_params(photo, warped, mask, strength))
+
+
 def measure_grain(photo: np.ndarray, ring: np.ndarray) -> float:
     """The photo's noise floor, in grey levels, measured where the screen isn't.
 

package/scripts/preflight.py

@@ -87,7 +87,19 @@ def report():
         "install_command": f"{sys.executable} {os.path.abspath(__file__)} --install",
         "install_does": f"Creates a virtualenv at {VENV} (nothing touches system Python) and pip-installs "
                         f"{', '.join(pkg for _, pkg, _ in REQUIRED)} into it (~60 MB download).",
+        # Optional, and deliberately NOT part of `ready`. A missing ffmpeg
+        # stops video renders and nothing else; making it a hard requirement
+        # would fail preflight for every user who only ever injects a
+        # screenshot, which is most of them.
         "optional": [
+            {"name": "ffmpeg (video)", "package": "imageio-ffmpeg",
+             "status": "installed" if _have_ffmpeg() else "missing",
+             "needed_for": "rendering a video; stills do not use it",
+             "install_command": f"{sys.executable} {os.path.abspath(__file__)} --install-ffmpeg",
+             "install_size": "~25 MB, a wheel with a static binary; nothing system-wide",
+             "why": "Encodes video renders. Ships as a wheel with a static binary, "
+                    "so it lands in the same venv and nothing is installed "
+                    "system-wide. Stills do not need it."},
             {"name": "SAM 2 (M4)", "status": "not yet used by the plugin",
              "why": "Better screen detection on photos where tone-based detection fails. Optional; results are worse without it, not absent."}
         ],
@@ -95,6 +107,23 @@ def report():
     }
 
 
+def _have_ffmpeg() -> bool:
+    """Is the ffmpeg wheel importable IN THE VENV (not in whatever Python runs this)?"""
+    code = ("import json\n"
+            "try:\n"
+            "    import imageio_ffmpeg, os\n"
+            "    p = imageio_ffmpeg.get_ffmpeg_exe()\n"
+            "    print(json.dumps(bool(p and os.path.exists(p))))\n"
+            "except Exception:\n"
+            "    print('false')\n")
+    try:
+        r = subprocess.run([VENV_PY, "-c", code], capture_output=True, text=True,
+                           timeout=60, check=False)
+        return r.stdout.strip() == "true"
+    except Exception:
+        return False
+
+
 def install():
     os.makedirs(os.path.dirname(VENV), exist_ok=True)
     if not os.path.exists(VENV_PY):
@@ -103,12 +132,31 @@ def install():
     subprocess.check_call([VENV_PY, "-m", "pip", "install", "-q", "-r", REQ])
 
 
+def install_ffmpeg():
+    """Add just the video encoder to an existing venv.
+
+    A separate entry point because of who needs it: someone who installed
+    screengraft before video existed has a perfectly good venv with OpenCV in
+    it, `ready` is true, and nothing tells them anything is missing until a
+    render fails at the end of the job. This adds the one wheel, so the ask is
+    "~25 MB for video" rather than "reinstall everything".
+    """
+    if not os.path.exists(VENV_PY):
+        install()
+        return
+    subprocess.check_call([VENV_PY, "-m", "pip", "install", "-q", "imageio-ffmpeg"])
+
+
 def main():
     ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
     ap.add_argument("--install", action="store_true", help="Create the venv and install requirements (ask the user first)")
+    ap.add_argument("--install-ffmpeg", action="store_true",
+                    help="Add just the video encoder to an existing venv (ask the user first)")
     args = ap.parse_args()
     if args.install:
         install()
+    elif args.install_ffmpeg:
+        install_ffmpeg()
     rep = report()
     print(json.dumps(rep, indent=1))
     sys.exit(0 if rep["ready"] else 1)

package/scripts/requirements.txt

@@ -1,2 +1,3 @@
 opencv-python-headless>=4.9
 numpy>=1.26
+imageio-ffmpeg>=0.4

package/scripts/ui.py

@@ -168,6 +168,88 @@ def _read_image(path: str):
     return im, p
 
 
+VIDEO_EXT = (".mp4", ".mov", ".m4v", ".webm", ".avi", ".mkv")
+
+
+def _have_ffmpeg() -> bool:
+    try:
+        W.ffmpeg_exe()
+        return True
+    except RuntimeError:
+        return False
+
+
+def _is_video(path: str) -> bool:
+    return str(path).lower().endswith(VIDEO_EXT)
+
+
+def _read_source(path: str):
+    """Read the screen source, which may be a still OR a video.
+
+    Returns (frame, real_path, meta). For a video the frame is the poster —
+    the frame the designer fits on — and `meta` carries what the page needs to
+    show a scrubber. Everything downstream of this point treats that frame
+    exactly like a screenshot, which is the point: the fit, the loupe, the
+    compare view and the preview are all unchanged by video.
+    """
+    real = _safe_local_path(path)
+    if not _is_video(real):
+        im, rp = _read_image(real)
+        return im, rp, {"video": False}
+    n, fps, vw, vh = W.probe_video(real)
+    frame = W.read_frame_at(real, 0)
+    # Report the encoder's absence HERE, when the clip is chosen, rather than
+    # letting the render fail at the end of the job. Someone who installed
+    # screengraft before video existed has a working venv with no ffmpeg in it,
+    # and nothing else would tell them until they had done all the fitting.
+    return frame, real, {"video": True, "frames": n, "fps": fps, "size": [vw, vh],
+                         "ffmpeg": _have_ffmpeg()}
+
+
+# Render progress, read by /api/render_status. A ten-second clip is a few
+# hundred frames and a good few seconds of work, which is far too long to hold
+# an HTTP request open — so the render runs on its own thread and the page
+# polls. ThreadingHTTPServer is already the server class, so this needs no
+# other machinery.
+RENDER = {"state": "idle", "done": 0, "total": 0, "output": None, "message": None}
+RENDER_LOCK = threading.Lock()
+
+
+def _render_worker(photo, video_path, corners, dest, radius_px, gr, grain, preset, fit_frame,
+                   blend="replace", reflection=None):
+    def progress(done, total):
+        with RENDER_LOCK:
+            RENDER["done"], RENDER["total"] = done, total
+    try:
+        info = W.compose_video(photo, video_path, corners, dest,
+                               corner_radius=radius_px, grade=gr, grain=grain,
+                               preset=preset, fit_frame=fit_frame, progress=progress,
+                               blend=blend,
+                               reflection=(W.DEFAULT_REFLECTION if reflection is None
+                                           else reflection))
+        with RENDER_LOCK:
+            RENDER.update(state="done", output=dest, info=info,
+                          done=info["frames"], total=info["frames"], message=None)
+    except Exception as e:                     # noqa: BLE001 - surfaced to the page
+        with RENDER_LOCK:
+            RENDER.update(state="error", message=str(e))
+
+
+def _blend_args(b):
+    """(blend, reflection) from the page's single `reflection` field.
+
+    One field, not two: the page sends a number when the switch is on and null
+    when it is off, so there is no way to express the contradictory state
+    "emissive with no strength" — which is just `replace` under a different
+    name. compose() still takes both, because the engine should not have to
+    infer intent from a null.
+    """
+    r = b.get("reflection")
+    if r is None:
+        return "replace", W.DEFAULT_REFLECTION
+    return "emissive", float(max(0.0, min(1.0, float(r))))
+
+
 def _guess_type(corners):
     c = np.array(corners, dtype=float)
     w = (np.linalg.norm(c[1] - c[0]) + np.linalg.norm(c[2] - c[3])) / 2
@@ -259,6 +341,12 @@ class Handler(BaseHTTPRequestHandler):
                     if time.time() >= deadline:
                         return self._json({"status": "pending", "waited": True})
                     time.sleep(0.15)
+            if u.path == "/api/render_status":
+                # A poll, so a GET: no body, safe to repeat, and the page hits
+                # it once a second while a render runs.
+                with RENDER_LOCK:
+                    return self._json(dict(RENDER))
+
             return self._json({"error": "no such route"}, 404)
         except (PermissionError, FileNotFoundError, KeyError, ValueError) as e:
             return self._json({"error": str(e)}, 400)
@@ -274,17 +362,26 @@ class Handler(BaseHTTPRequestHandler):
                 dest = os.path.join(SESSION.dir, f"{role}-{int(time.time())}-{name}")
                 with open(dest, "wb") as f:
                     f.write(self._body())
-                im, real = _read_image(dest)
+                # A video is only ever a screen source; a photo must be a still.
+                if role == "screenshot":
+                    im, real, meta = _read_source(dest)
+                else:
+                    im, real = _read_image(dest)
+                    meta = {"video": False}
                 SESSION.update(**{role: real})
-                return self._json({"path": real, "size": [im.shape[1], im.shape[0]]})
+                return self._json({"path": real, "size": [im.shape[1], im.shape[0]], **meta})
 
             b = self._jbody()
 
             if u.path == "/api/use":
                 role = b["role"]
-                im, real = _read_image(b["path"])
+                if role == "screenshot":
+                    im, real, meta = _read_source(b["path"])
+                else:
+                    im, real = _read_image(b["path"])
+                    meta = {"video": False}
                 SESSION.update(**{role: real})
-                return self._json({"path": real, "size": [im.shape[1], im.shape[0]]})
+                return self._json({"path": real, "size": [im.shape[1], im.shape[0]], **meta})
 
             if u.path == "/api/figma":
                 return self._json(SESSION.enqueue({
@@ -350,9 +447,75 @@ class Handler(BaseHTTPRequestHandler):
                 res["type_guess"] = _guess_type(res["corners"])
                 return self._json(res)
 
+            if u.path == "/api/frame":
+                # One frame of the source video as a PNG the page can show — the
+                # poster, or whichever frame the scrubber is on. The fit, the
+                # loupe and the compare view all work on this exactly as they
+                # work on a screenshot, which is why none of them needed
+                # changing for video.
+                spath = _safe_local_path(SESSION.state["screenshot"])
+                if not _is_video(spath):
+                    return self._json({"error": "the screen source is not a video"}, 400)
+                idx = int(b.get("index") or 0)
+                frame = W.read_frame_at(spath, idx)
+                dest = os.path.join(SESSION.dir, f"frame-{idx:06d}.png")
+                cv2.imwrite(dest, frame, [cv2.IMWRITE_PNG_COMPRESSION, 1])
+                return self._json({"path": dest, "index": idx,
+                                   "size": [frame.shape[1], frame.shape[0]]})
+
+            if u.path == "/api/render":
+                # Video: same fit, same geometry, N frames instead of one.
+                photo, ppath = _read_image(SESSION.state["photo"])
+                spath = _safe_local_path(SESSION.state["screenshot"])
+                if not _is_video(spath):
+                    return self._json({"error": "the screen source is not a video"}, 400)
+                if not _have_ffmpeg():
+                    return self._json({"error": "ffmpeg is not installed",
+                                       "needs_ffmpeg": True}, 400)
+                with RENDER_LOCK:
+                    if RENDER["state"] == "running":
+                        return self._json({"error": "a render is already running"}, 409)
+                    RENDER.update(state="running", done=0, total=0,
+                                  output=None, message=None)
+                corners = b["corners"]
+                frac = float(b.get("radius_frac") or 0.0)
+                fit_frame = int(b.get("fit_frame") or 0)
+                first = W.read_frame_at(spath, fit_frame)
+                radius_px = frac * first.shape[1]
+                gr = float(b.get("grade") if b.get("grade") is not None else 0.0)
+                grain = bool(b.get("grain", gr > 0))
+                blend, reflection = _blend_args(b)
+                preset = "prores" if b.get("preset") == "prores" else "web"
+                ext = ".mov" if preset == "prores" else ".mp4"
+                os.makedirs(OUT_DIR, exist_ok=True)
+                stem = (f"{os.path.splitext(os.path.basename(ppath))[0]}__"
+                        f"{os.path.splitext(os.path.basename(spath))[0]}")
+                dest = os.path.join(OUT_DIR, stem + ext)
+                i = 2
+                while os.path.exists(dest):
+                    dest = os.path.join(OUT_DIR, f"{stem}-{i}{ext}"); i += 1
+                SESSION.update(corners=corners, radius_frac=frac,
+                               device=b.get("device"), grade=gr)
+                # Everything that changes the output goes in the sidecar, for the
+                # third time of asking (radius_px, then grade/grain, now the video
+                # fields). A render that cannot be reproduced from its own sidecar
+                # undercuts the determinism claim.
+                result = {"output": dest, "photo": ppath, "screenshot": spath,
+                          "corners": corners, "radius_frac": frac, "radius_px": radius_px,
+                          "device": b.get("device"), "grade": gr, "grain": grain,
+                          "video": True, "preset": preset, "fit_frame": fit_frame,
+                          "blend": blend, "reflection": reflection,
+                          "saved": time.time()}
+                _write_json_atomic(SESSION.result_path, result)
+                threading.Thread(target=_render_worker, daemon=True,
+                                 args=(photo, spath, corners, dest, radius_px,
+                                       gr, grain, preset, fit_frame,
+                                       blend, reflection)).start()
+                return self._json({"started": True, "output": dest, "preset": preset})
+
             if u.path in ("/api/preview", "/api/save"):
                 photo, ppath = _read_image(SESSION.state["photo"])
-                shot, spath = _read_image(SESSION.state["screenshot"])
+                shot, spath, _meta = _read_source(SESSION.state["screenshot"])
                 corners = b["corners"]
                 frac = float(b.get("radius_frac") or 0.0)
                 radius_px = frac * shot.shape[1]
@@ -361,8 +524,10 @@ class Handler(BaseHTTPRequestHandler):
                 # is the point (a brand review), and the grade is the right one
                 # when the photograph is (a portfolio shot).
                 gr = float(b.get("grade") if b.get("grade") is not None else 0.0)
+                blend, reflection = _blend_args(b)
                 out = W.compose(photo, shot, corners, radius_px,
-                                grade=gr, grain=bool(b.get("grain", gr > 0)))
+                                grade=gr, grain=bool(b.get("grain", gr > 0)),
+                                blend=blend, reflection=reflection)
                 SESSION.update(corners=corners, radius_frac=frac, device=b.get("device"),
                                grade=gr)
                 if u.path == "/api/preview":
@@ -396,6 +561,7 @@ class Handler(BaseHTTPRequestHandler):
                 result = {"output": dest, "photo": ppath, "screenshot": spath, "corners": corners,
                           "radius_frac": frac, "radius_px": radius_px, "device": b.get("device"),
                           "grade": gr, "grain": bool(b.get("grain", gr > 0)),
+                          "blend": blend, "reflection": reflection,
                           "saved": time.time()}
                 _write_json_atomic(SESSION.result_path, result)
                 SESSION.update(output=dest)