framelift 0.0.1__tar.gz

framelift-0.0.1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 rysk-tanaka
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

framelift-0.0.1/PKG-INFO

@@ -0,0 +1,134 @@
+Metadata-Version: 2.4
+Name: framelift
+Version: 0.0.1
+Summary: Extract structured data from screen recordings and screenshots with vision LLMs.
+Keywords: screenshot,screen-recording,vision,llm,structured-output,pydantic-ai
+Author: rysk-tanaka
+License-Expression: MIT
+License-File: LICENSE
+Classifier: Development Status :: 2 - Pre-Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Multimedia :: Graphics
+Classifier: Topic :: Scientific/Engineering :: Image Recognition
+Classifier: Typing :: Typed
+Requires-Dist: imagehash>=4.3
+Requires-Dist: pillow>=12
+Requires-Dist: pydantic>=2.14
+Requires-Dist: pydantic-ai-slim>=2.57
+Requires-Dist: typer>=0.27
+Requires-Dist: pydantic-ai-slim[anthropic]>=2.57 ; extra == 'anthropic'
+Requires-Dist: pydantic-evals>=2.57 ; extra == 'eval'
+Requires-Dist: pydantic-ai-slim[google]>=2.57 ; extra == 'google'
+Requires-Dist: pydantic-ai-slim[openai]>=2.57 ; extra == 'openai'
+Requires-Dist: av>=19 ; extra == 'video'
+Requires-Python: >=3.12
+Project-URL: Repository, https://github.com/rysk-tanaka/framelift
+Project-URL: Issues, https://github.com/rysk-tanaka/framelift/issues
+Project-URL: Changelog, https://github.com/rysk-tanaka/framelift/blob/main/CHANGELOG.md
+Provides-Extra: anthropic
+Provides-Extra: eval
+Provides-Extra: google
+Provides-Extra: openai
+Provides-Extra: video
+Description-Content-Type: text/markdown
+
+# framelift
+
+Extract structured data from screen recordings and screenshots with vision LLMs.
+
+You describe **what** to read as a Pydantic model and **how** to read it as a `Profile`;
+framelift turns images into validated, provenance-tagged JSON Lines.
+
+> **Status:** pre-alpha (0.x). The public API is limited to `Profile` and `Pipeline`
+> and may change between minor versions.
+
+## Features (0.1)
+
+- Image input (files, directories, glob patterns), each tagged with its SHA-256 and perceptual hash
+- Structured extraction with [Pydantic AI](https://ai.pydantic.dev/): your model is the output type,
+  your validators trigger automatic retries
+- Model-agnostic: pass any `provider:model` string supported by Pydantic AI
+- `observations.jsonl` output with source, usage and retry information per image
+- A synthetic sample generator (a fictional shop screen) for demos and tests
+
+Planned: video input with frame sampling and de-duplication (0.2), cross-frame aggregation (0.2),
+screen classification, cropping and redaction (0.3), and an evaluation harness built on
+`pydantic-evals`.
+
+## Install
+
+```bash
+pip install "framelift[anthropic]"   # or [openai], [google]
+```
+
+## Quick start
+
+```python
+from pydantic import BaseModel, Field
+from framelift import Pipeline, Profile
+
+
+class ShopItem(BaseModel):
+    name: str = Field(description="Item name as displayed")
+    price: int = Field(description="Price as an integer", gt=0)
+
+
+class ShopScreen(BaseModel):
+    shop_name: str
+    items: list[ShopItem]
+
+
+shop = Profile(
+    name="fantasy-shop",
+    version="1",
+    output_type=ShopScreen,
+    instructions="Read the shop's item list exactly as displayed.",
+)
+
+Pipeline(shop, model="anthropic:<model-name>", out_dir="out").run(["shots/*.png"])
+```
+
+Or from the command line:
+
+```bash
+framelift synth --out samples --n 3                       # generate fictional shop screenshots
+framelift run "samples/*.png" --profile examples.fantasy_shop:profile \
+  --model anthropic:<model-name> --out out
+```
+
+Each line of `out/observations.jsonl` looks like:
+
+```json
+{"schema": "framelift/observation/v1",
+ "profile": {"name": "fantasy-shop", "version": "1"},
+ "source": {"path": "samples/shop-000.png", "sha256": "…", "phash": "…"},
+ "data": {"shop_name": "…", "items": [{"name": "…", "price": 120}]},
+ "model": "anthropic:<model-name>",
+ "usage": {"input_tokens": 1534, "output_tokens": 96, "requests": 1, "latency_ms": 2104},
+ "retries": 0}
+```
+
+## Privacy
+
+framelift runs locally and only sends the images you pass to the model provider you choose.
+It has no telemetry. Cropping and redaction before sending are planned for 0.3.
+
+## Development
+
+```bash
+uv sync --all-extras
+uv run ruff check && uv run ruff format --check
+uv run ty check
+uv run pytest
+```
+
+Tests never call a real model (`ALLOW_MODEL_REQUESTS = False`); tests marked `live` are skipped by
+default.
+
+## License
+
+MIT

framelift-0.0.1/README.md

@@ -0,0 +1,96 @@
+# framelift
+
+Extract structured data from screen recordings and screenshots with vision LLMs.
+
+You describe **what** to read as a Pydantic model and **how** to read it as a `Profile`;
+framelift turns images into validated, provenance-tagged JSON Lines.
+
+> **Status:** pre-alpha (0.x). The public API is limited to `Profile` and `Pipeline`
+> and may change between minor versions.
+
+## Features (0.1)
+
+- Image input (files, directories, glob patterns), each tagged with its SHA-256 and perceptual hash
+- Structured extraction with [Pydantic AI](https://ai.pydantic.dev/): your model is the output type,
+  your validators trigger automatic retries
+- Model-agnostic: pass any `provider:model` string supported by Pydantic AI
+- `observations.jsonl` output with source, usage and retry information per image
+- A synthetic sample generator (a fictional shop screen) for demos and tests
+
+Planned: video input with frame sampling and de-duplication (0.2), cross-frame aggregation (0.2),
+screen classification, cropping and redaction (0.3), and an evaluation harness built on
+`pydantic-evals`.
+
+## Install
+
+```bash
+pip install "framelift[anthropic]"   # or [openai], [google]
+```
+
+## Quick start
+
+```python
+from pydantic import BaseModel, Field
+from framelift import Pipeline, Profile
+
+
+class ShopItem(BaseModel):
+    name: str = Field(description="Item name as displayed")
+    price: int = Field(description="Price as an integer", gt=0)
+
+
+class ShopScreen(BaseModel):
+    shop_name: str
+    items: list[ShopItem]
+
+
+shop = Profile(
+    name="fantasy-shop",
+    version="1",
+    output_type=ShopScreen,
+    instructions="Read the shop's item list exactly as displayed.",
+)
+
+Pipeline(shop, model="anthropic:<model-name>", out_dir="out").run(["shots/*.png"])
+```
+
+Or from the command line:
+
+```bash
+framelift synth --out samples --n 3                       # generate fictional shop screenshots
+framelift run "samples/*.png" --profile examples.fantasy_shop:profile \
+  --model anthropic:<model-name> --out out
+```
+
+Each line of `out/observations.jsonl` looks like:
+
+```json
+{"schema": "framelift/observation/v1",
+ "profile": {"name": "fantasy-shop", "version": "1"},
+ "source": {"path": "samples/shop-000.png", "sha256": "…", "phash": "…"},
+ "data": {"shop_name": "…", "items": [{"name": "…", "price": 120}]},
+ "model": "anthropic:<model-name>",
+ "usage": {"input_tokens": 1534, "output_tokens": 96, "requests": 1, "latency_ms": 2104},
+ "retries": 0}
+```
+
+## Privacy
+
+framelift runs locally and only sends the images you pass to the model provider you choose.
+It has no telemetry. Cropping and redaction before sending are planned for 0.3.
+
+## Development
+
+```bash
+uv sync --all-extras
+uv run ruff check && uv run ruff format --check
+uv run ty check
+uv run pytest
+```
+
+Tests never call a real model (`ALLOW_MODEL_REQUESTS = False`); tests marked `live` are skipped by
+default.
+
+## License
+
+MIT

framelift-0.0.1/pyproject.toml

@@ -0,0 +1,101 @@
+[project]
+name = "framelift"
+version = "0.0.1"
+description = "Extract structured data from screen recordings and screenshots with vision LLMs."
+readme = "README.md"
+license = "MIT"
+license-files = ["LICENSE"]
+requires-python = ">=3.12"
+keywords = [
+    "screenshot",
+    "screen-recording",
+    "vision",
+    "llm",
+    "structured-output",
+    "pydantic-ai",
+]
+classifiers = [
+    "Development Status :: 2 - Pre-Alpha",
+    "Intended Audience :: Developers",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Programming Language :: Python :: 3.14",
+    "Topic :: Multimedia :: Graphics",
+    "Topic :: Scientific/Engineering :: Image Recognition",
+    "Typing :: Typed",
+]
+dependencies = [
+    "imagehash>=4.3",
+    "pillow>=12",
+    "pydantic>=2.14",
+    "pydantic-ai-slim>=2.57",
+    "typer>=0.27",
+]
+
+[[project.authors]]
+name = "rysk-tanaka"
+
+[project.optional-dependencies]
+video = ["av>=19"]
+eval = ["pydantic-evals>=2.57"]
+anthropic = ["pydantic-ai-slim[anthropic]>=2.57"]
+openai = ["pydantic-ai-slim[openai]>=2.57"]
+google = ["pydantic-ai-slim[google]>=2.57"]
+
+[project.scripts]
+framelift = "framelift.cli:app"
+
+[project.urls]
+Repository = "https://github.com/rysk-tanaka/framelift"
+Issues = "https://github.com/rysk-tanaka/framelift/issues"
+Changelog = "https://github.com/rysk-tanaka/framelift/blob/main/CHANGELOG.md"
+
+[build-system]
+requires = ["uv_build>=0.11.32,<0.12.0"]
+build-backend = "uv_build"
+
+[dependency-groups]
+dev = [
+    "pytest>=9.1",
+    "ruff==0.17.1",
+    "ty==0.0.88",
+]
+
+[tool.ruff]
+line-length = 100
+target-version = "py312"
+
+[tool.ruff.lint]
+select = [
+    "E",
+    "W",
+    "F",
+    "I",
+    "B",
+    "UP",
+    "SIM",
+    "RUF",
+    "PTH",
+    "TC",
+]
+
+[tool.ruff.lint.per-file-ignores]
+"tests/**" = ["TC"]
+
+[tool.ty.environment]
+python-version = "3.12"
+
+[tool.ty.rules]
+unused-ignore-comment = "warn"
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+pythonpath = ["."]
+addopts = [
+    "-ra",
+    "--strict-markers",
+    "-m",
+    "not live",
+]
+markers = ["live: calls a real model provider (needs API keys; not run in CI)"]

framelift-0.0.1/pyproject.toml.orig

@@ -0,0 +1,88 @@
+[project]
+name = "framelift"
+version = "0.0.1"
+description = "Extract structured data from screen recordings and screenshots with vision LLMs."
+readme = "README.md"
+license = "MIT"
+license-files = ["LICENSE"]
+authors = [{ name = "rysk-tanaka" }]
+requires-python = ">=3.12"
+keywords = ["screenshot", "screen-recording", "vision", "llm", "structured-output", "pydantic-ai"]
+classifiers = [
+    "Development Status :: 2 - Pre-Alpha",
+    "Intended Audience :: Developers",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Programming Language :: Python :: 3.14",
+    "Topic :: Multimedia :: Graphics",
+    "Topic :: Scientific/Engineering :: Image Recognition",
+    "Typing :: Typed",
+]
+dependencies = [
+    "imagehash>=4.3",
+    "pillow>=12",
+    "pydantic>=2.14",
+    "pydantic-ai-slim>=2.57",
+    "typer>=0.27",
+]
+
+[project.optional-dependencies]
+video = ["av>=19"]
+eval = ["pydantic-evals>=2.57"]
+anthropic = ["pydantic-ai-slim[anthropic]>=2.57"]
+openai = ["pydantic-ai-slim[openai]>=2.57"]
+google = ["pydantic-ai-slim[google]>=2.57"]
+
+[project.scripts]
+framelift = "framelift.cli:app"
+
+[project.urls]
+Repository = "https://github.com/rysk-tanaka/framelift"
+Issues = "https://github.com/rysk-tanaka/framelift/issues"
+Changelog = "https://github.com/rysk-tanaka/framelift/blob/main/CHANGELOG.md"
+
+[build-system]
+requires = ["uv_build>=0.11.32,<0.12.0"]
+build-backend = "uv_build"
+
+[dependency-groups]
+dev = [
+    "pytest>=9.1",
+    "ruff==0.17.1",
+    "ty==0.0.88",
+]
+
+[tool.ruff]
+line-length = 100
+target-version = "py312"
+
+[tool.ruff.lint]
+select = [
+    "E", "W",   # pycodestyle
+    "F",        # pyflakes
+    "I",        # isort
+    "B",        # flake8-bugbear
+    "UP",       # pyupgrade
+    "SIM",      # flake8-simplify
+    "RUF",      # ruff-specific
+    "PTH",      # use pathlib
+    "TC",       # type-checking imports
+]
+
+[tool.ruff.lint.per-file-ignores]
+"tests/**" = ["TC"]
+
+[tool.ty.environment]
+python-version = "3.12"
+
+[tool.ty.rules]
+unused-ignore-comment = "warn"
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+pythonpath = ["."]
+addopts = ["-ra", "--strict-markers", "-m", "not live"]
+markers = [
+    "live: calls a real model provider (needs API keys; not run in CI)",
+]

framelift-0.0.1/src/framelift/__init__.py

@@ -0,0 +1,15 @@
+"""framelift: extract structured data from screen recordings and screenshots with vision LLMs.
+
+The public API is `Profile` and `Pipeline` (plus the `Observation` record they produce).
+Everything under ``framelift._internal`` may change in any release.
+"""
+
+from importlib.metadata import version as _version
+
+from framelift.observation import Observation
+from framelift.pipeline import Pipeline, RunResult
+from framelift.profile import Profile, Validator
+
+__all__ = ["Observation", "Pipeline", "Profile", "RunResult", "Validator", "__version__"]
+
+__version__ = _version("framelift")

framelift-0.0.1/src/framelift/_internal/__init__.py

@@ -0,0 +1 @@
+"""Internal building blocks. Not part of the public API; may change in any release."""

framelift-0.0.1/src/framelift/_internal/extract.py

@@ -0,0 +1,80 @@
+"""Structured extraction from one image with Pydantic AI."""
+
+from __future__ import annotations
+
+import time
+from dataclasses import dataclass
+from typing import TYPE_CHECKING
+
+from pydantic import BaseModel
+from pydantic_ai import Agent, AgentRetries, BinaryContent
+from pydantic_ai.messages import RetryPromptPart
+
+if TYPE_CHECKING:
+    from pydantic_ai.models import Model
+
+    from framelift._internal.sources import ImageSource
+    from framelift.profile import Profile, Validator
+
+USER_PROMPT = "Extract the data shown in this screenshot."
+
+
+@dataclass(frozen=True)
+class Extraction[OutputT: BaseModel]:
+    output: OutputT
+    input_tokens: int
+    output_tokens: int
+    requests: int
+    retries: int
+    latency_ms: int
+
+
+def _adapt[OutputT: BaseModel](validator: Validator[OutputT]):
+    def run(output: OutputT) -> OutputT:
+        result = validator(output)
+        return output if result is None else result
+
+    return run
+
+
+def model_label(model: Model | str) -> str:
+    if isinstance(model, str):
+        return model
+    return f"{model.system}:{model.model_name}"
+
+
+class Extractor[OutputT: BaseModel]:
+    """Reads one image into ``profile.output_type``, retrying when validation fails."""
+
+    def __init__(self, profile: Profile[OutputT], model: Model | str, *, max_retries: int) -> None:
+        self._model = model
+        self._agent: Agent[None, OutputT] = Agent(
+            output_type=profile.output_type,
+            instructions=profile.instructions,
+            retries=AgentRetries(output=max_retries),
+            name=f"framelift:{profile.name}",
+        )
+        for validator in profile.validators:
+            self._agent.output_validator(_adapt(validator))
+
+    def extract(self, image: ImageSource) -> Extraction[OutputT]:
+        started = time.perf_counter()
+        result = self._agent.run_sync(
+            [USER_PROMPT, BinaryContent(data=image.data, media_type=image.media_type)],
+            model=self._model,
+        )
+        latency_ms = round((time.perf_counter() - started) * 1000)
+        retries = sum(
+            isinstance(part, RetryPromptPart)
+            for message in result.all_messages()
+            for part in getattr(message, "parts", ())
+        )
+        usage = result.usage
+        return Extraction(
+            output=result.output,
+            input_tokens=usage.input_tokens,
+            output_tokens=usage.output_tokens,
+            requests=usage.requests,
+            retries=retries,
+            latency_ms=latency_ms,
+        )

framelift-0.0.1/src/framelift/_internal/sources.py

@@ -0,0 +1,93 @@
+"""Resolve user inputs into image files and load them."""
+
+from __future__ import annotations
+
+import hashlib
+import io
+from dataclasses import dataclass
+from pathlib import Path
+from typing import TYPE_CHECKING
+
+import imagehash
+from PIL import Image, UnidentifiedImageError
+
+if TYPE_CHECKING:
+    from collections.abc import Iterable
+
+IMAGE_SUFFIXES = frozenset({".png", ".jpg", ".jpeg", ".webp"})
+_MEDIA_TYPES = {"PNG": "image/png", "JPEG": "image/jpeg", "WEBP": "image/webp"}
+_GLOB_CHARS = frozenset("*?[")
+
+
+@dataclass(frozen=True)
+class ImageSource:
+    path: Path
+    data: bytes
+    media_type: str
+    sha256: str
+    phash: str
+
+
+def _is_image(path: Path) -> bool:
+    return path.is_file() and path.suffix.lower() in IMAGE_SUFFIXES
+
+
+def _glob(pattern: str) -> list[Path]:
+    p = Path(pattern)
+    if p.is_absolute():
+        root = Path(p.anchor)
+        relative = str(p.relative_to(root))
+    else:
+        root, relative = Path(), pattern
+    return sorted(root.glob(relative))
+
+
+def expand_inputs(inputs: Iterable[str | Path]) -> list[Path]:
+    """Expand files, directories and glob patterns into a de-duplicated list of image paths.
+
+    Directories contribute their image files (non-recursive, sorted). Raises `FileNotFoundError`
+    when an input matches nothing, so typos are not silently ignored.
+    """
+    seen: set[Path] = set()
+    result: list[Path] = []
+    for item in inputs:
+        text = str(item)
+        if any(c in text for c in _GLOB_CHARS):
+            candidates = [p for p in _glob(text) if _is_image(p)]
+        else:
+            path = Path(text)
+            if path.is_dir():
+                candidates = sorted(p for p in path.iterdir() if _is_image(p))
+            elif path.is_file():
+                candidates = [path]
+            else:
+                candidates = []
+        if not candidates:
+            raise FileNotFoundError(f"No image matched input: {text}")
+        for candidate in candidates:
+            key = candidate.resolve()
+            if key not in seen:
+                seen.add(key)
+                result.append(candidate)
+    return result
+
+
+def load_image(path: Path) -> ImageSource:
+    """Read an image, verify it decodes, and compute its SHA-256 and perceptual hash."""
+    data = path.read_bytes()
+    try:
+        with Image.open(io.BytesIO(data)) as image:
+            image.load()
+            media_type = _MEDIA_TYPES.get(image.format or "")
+            phash = str(imagehash.phash(image))
+    except UnidentifiedImageError as exc:
+        raise ValueError(f"Not a readable image: {path}") from exc
+    if media_type is None:
+        raise ValueError(f"Unsupported image format for {path}")
+    return ImageSource(
+        path=path,
+        data=data,
+        media_type=media_type,
+        sha256=hashlib.sha256(data).hexdigest(),
+        phash=phash,
+    )

framelift-0.0.1/src/framelift/_internal/synth.py

@@ -0,0 +1,97 @@
+"""Synthetic sample generator: screenshots of a fictional shop with known ground truth.
+
+The shop, item names and prices are invented, so the images can be shared, used in tests and
+used as a public benchmark without any real game or app content.
+"""
+
+from __future__ import annotations
+
+import json
+import random
+from dataclasses import asdict, dataclass
+from typing import TYPE_CHECKING, Literal
+
+from PIL import Image, ImageDraw, ImageFont
+
+if TYPE_CHECKING:
+    from pathlib import Path
+
+Orientation = Literal["portrait", "landscape"]
+
+_SHOP_PREFIXES = ["Moonlit", "Copper", "Wandering", "Silverleaf", "Lantern", "Hollow", "Amber"]
+_SHOP_SUFFIXES = ["Emporium", "Bazaar", "Trading Post", "Market", "Workshop", "Pantry"]
+_ITEM_ADJECTIVES = ["Moon", "Ember", "Frost", "Dew", "Iron", "Velvet", "Starlit", "Mossy", "Sun"]
+_ITEM_NOUNS = ["Herb", "Ingot", "Thread", "Lantern", "Tonic", "Feather", "Pebble", "Bread", "Rope"]
+
+_SIZES: dict[Orientation, tuple[int, int]] = {"portrait": (720, 1280), "landscape": (1280, 720)}
+_HEADER_HEIGHT, _LIST_TOP, _ROW_HEIGHT = 110, 150, 84
+_MIN_ITEMS, _MAX_ITEMS = 4, 7
+
+
+def _max_rows(orientation: Orientation) -> int:
+    height = _SIZES[orientation][1]
+    return min(_MAX_ITEMS, (height - _LIST_TOP) // _ROW_HEIGHT)
+
+
+@dataclass(frozen=True)
+class SampleItem:
+    name: str
+    price: int
+
+
+@dataclass(frozen=True)
+class SampleTruth:
+    file: str
+    orientation: Orientation
+    shop_name: str
+    items: list[SampleItem]
+
+
+def _random_truth(rng: random.Random, file: str, orientation: Orientation) -> SampleTruth:
+    shop = f"{rng.choice(_SHOP_PREFIXES)} {rng.choice(_SHOP_SUFFIXES)}"
+    names: set[str] = set()
+    count = rng.randint(_MIN_ITEMS, _max_rows(orientation))
+    while len(names) < count:
+        names.add(f"{rng.choice(_ITEM_ADJECTIVES)} {rng.choice(_ITEM_NOUNS)}")
+    items = [SampleItem(name=n, price=rng.randrange(10, 2000, 5)) for n in sorted(names)]
+    return SampleTruth(file=file, orientation=orientation, shop_name=shop, items=items)
+
+
+def render(truth: SampleTruth) -> Image.Image:
+    """Draw a simple shop screen for ``truth``."""
+    width, height = _SIZES[truth.orientation]
+    image = Image.new("RGB", (width, height), (34, 30, 46))
+    draw = ImageDraw.Draw(image)
+    title = ImageFont.load_default(size=40)
+    body = ImageFont.load_default(size=30)
+
+    draw.rectangle((0, 0, width, _HEADER_HEIGHT), fill=(70, 52, 96))
+    draw.text((32, 34), truth.shop_name, font=title, fill=(250, 236, 200))
+
+    for index, item in enumerate(truth.items):
+        y = _LIST_TOP + index * _ROW_HEIGHT
+        fill = (52, 46, 70) if index % 2 == 0 else (44, 39, 60)
+        draw.rounded_rectangle((24, y, width - 24, y + _ROW_HEIGHT - 12), radius=12, fill=fill)
+        draw.text((48, y + 20), item.name, font=body, fill=(235, 235, 245))
+        price = f"{item.price} G"
+        price_width = draw.textlength(price, font=body)
+        draw.text((width - 48 - price_width, y + 20), price, font=body, fill=(255, 214, 102))
+    return image
+
+
+def generate(out_dir: Path, *, n: int, seed: int = 0) -> list[SampleTruth]:
+    """Write ``n`` PNG screenshots and a ``truth.jsonl`` with their ground truth."""
+    if n < 1:
+        raise ValueError("n must be >= 1")
+    out_dir.mkdir(parents=True, exist_ok=True)
+    rng = random.Random(seed)
+    truths: list[SampleTruth] = []
+    for index in range(n):
+        orientation: Orientation = "portrait" if index % 2 == 0 else "landscape"
+        truth = _random_truth(rng, f"shop-{index:03d}.png", orientation)
+        render(truth).save(out_dir / truth.file, format="PNG")
+        truths.append(truth)
+    with (out_dir / "truth.jsonl").open("w", encoding="utf-8") as fh:
+        for truth in truths:
+            fh.write(json.dumps(asdict(truth), ensure_ascii=False) + "\n")
+    return truths

framelift-0.0.1/src/framelift/cli.py

@@ -0,0 +1,87 @@
+"""Command-line interface: ``framelift run`` and ``framelift synth``."""
+
+from __future__ import annotations
+
+import importlib
+import sys
+from pathlib import Path
+from typing import Annotated
+
+import typer
+
+from framelift._internal import synth as _synth
+from framelift.pipeline import Pipeline
+from framelift.profile import Profile
+
+app = typer.Typer(
+    help="Extract structured data from screen recordings and screenshots with vision LLMs.",
+    no_args_is_help=True,
+    add_completion=False,
+)
+
+
+def load_profile(spec: str) -> Profile:
+    """Load a `Profile` from ``"package.module:attribute"``.
+
+    The current directory is put on ``sys.path`` so that local profile modules can be used.
+    """
+    module_name, sep, attribute = spec.partition(":")
+    if not sep or not module_name or not attribute:
+        raise typer.BadParameter("expected 'package.module:attribute'", param_hint="--profile")
+    cwd = str(Path.cwd())
+    if cwd not in sys.path:
+        sys.path.insert(0, cwd)
+    try:
+        module = importlib.import_module(module_name)
+    except ImportError as exc:
+        raise typer.BadParameter(f"cannot import {module_name!r}: {exc}") from exc
+    profile = getattr(module, attribute, None)
+    if not isinstance(profile, Profile):
+        raise typer.BadParameter(f"{spec!r} is not a framelift Profile", param_hint="--profile")
+    return profile
+
+
+@app.command()
+def run(
+    inputs: Annotated[list[str], typer.Argument(help="Image files, directories or glob patterns")],
+    profile: Annotated[str, typer.Option(help="Profile to use, as 'package.module:attribute'")],
+    model: Annotated[str, typer.Option(help="Pydantic AI model, e.g. 'anthropic:<model-name>'")],
+    out: Annotated[Path, typer.Option(help="Output directory")] = Path("out"),
+    max_retries: Annotated[int, typer.Option(min=0, help="Retries when validation fails")] = 2,
+    anonymize_paths: Annotated[
+        bool, typer.Option(help="Write only hashes, not input paths, to the output")
+    ] = False,
+) -> None:
+    """Extract data from images and write observations.jsonl."""
+    pipeline = Pipeline(
+        load_profile(profile),
+        model=model,
+        out_dir=out,
+        max_retries=max_retries,
+        anonymize_paths=anonymize_paths,
+    )
+    try:
+        result = pipeline.run(inputs)
+    except FileNotFoundError as exc:
+        raise typer.BadParameter(str(exc), param_hint="INPUTS") from exc
+    typer.echo(
+        f"{len(result.succeeded)} ok, {len(result.failed)} failed -> {result.output_path}",
+        err=True,
+    )
+    if result.failed:
+        raise typer.Exit(code=1)
+
+
+@app.command()
+def synth(
+    out: Annotated[Path, typer.Option(help="Output directory")] = Path("samples"),
+    n: Annotated[int, typer.Option(min=1, help="Number of screenshots")] = 10,
+    seed: Annotated[int, typer.Option(help="Random seed")] = 0,
+) -> None:
+    """Generate screenshots of a fictional shop, with ground truth in truth.jsonl."""
+    truths = _synth.generate(out, n=n, seed=seed)
+    typer.echo(f"wrote {len(truths)} images and truth.jsonl to {out}", err=True)
+
+
+if __name__ == "__main__":
+    app()

framelift-0.0.1/src/framelift/observation.py

@@ -0,0 +1,50 @@
+"""The `framelift/observation/v1` record written to ``observations.jsonl``."""
+
+from __future__ import annotations
+
+from typing import Any, Literal
+
+from pydantic import BaseModel, ConfigDict, Field
+
+OBSERVATION_SCHEMA = "framelift/observation/v1"
+
+
+class _Record(BaseModel):
+    model_config = ConfigDict(frozen=True, extra="forbid", populate_by_name=True)
+
+
+class ProfileRef(_Record):
+    name: str
+    version: str
+
+
+class SourceRef(_Record):
+    path: str | None = Field(description="Input path; omitted when paths are anonymized")
+    sha256: str
+    phash: str = Field(description="Perceptual hash (hex) of the image")
+
+
+class Usage(_Record):
+    input_tokens: int = 0
+    output_tokens: int = 0
+    requests: int = 0
+    latency_ms: int = 0
+
+
+class Observation(_Record):
+    """What was read from one image."""
+
+    schema_id: Literal["framelift/observation/v1"] = Field(
+        default=OBSERVATION_SCHEMA, alias="schema"
+    )
+    status: Literal["ok", "error"]
+    profile: ProfileRef
+    source: SourceRef
+    data: dict[str, Any] | None = Field(description="The validated output; null on error")
+    error: str | None = None
+    model: str
+    usage: Usage
+    retries: int = Field(description="How many times validation asked the model to try again")
+
+    def to_json(self) -> str:
+        return self.model_dump_json(by_alias=True)

framelift-0.0.1/src/framelift/pipeline.py

@@ -0,0 +1,128 @@
+"""`Pipeline`: run a `Profile` over images and write ``observations.jsonl``."""
+
+from __future__ import annotations
+
+import time
+from dataclasses import dataclass, field
+from pathlib import Path
+from typing import TYPE_CHECKING
+
+from pydantic import BaseModel
+from pydantic_ai.exceptions import UnexpectedModelBehavior
+
+from framelift._internal.extract import Extractor, model_label
+from framelift._internal.sources import expand_inputs, load_image
+from framelift.observation import Observation, ProfileRef, SourceRef, Usage
+
+if TYPE_CHECKING:
+    from collections.abc import Iterable
+
+    from pydantic_ai.models import Model
+
+    from framelift._internal.sources import ImageSource
+    from framelift.profile import Profile
+
+OBSERVATIONS_FILE = "observations.jsonl"
+
+
+@dataclass
+class RunResult:
+    observations: list[Observation] = field(default_factory=list)
+    output_path: Path | None = None
+
+    @property
+    def succeeded(self) -> list[Observation]:
+        return [o for o in self.observations if o.status == "ok"]
+
+    @property
+    def failed(self) -> list[Observation]:
+        return [o for o in self.observations if o.status == "error"]
+
+
+class Pipeline[OutputT: BaseModel]:
+    """Extract ``profile.output_type`` from every input image.
+
+    Args:
+        profile: What to read.
+        model: A Pydantic AI model or a ``"provider:model"`` string.
+        out_dir: Where to write ``observations.jsonl``. ``None`` keeps results in memory only.
+        max_retries: How many times a failed validation may ask the model to try again.
+        anonymize_paths: Omit input paths from observations, keeping only hashes.
+    """
+
+    def __init__(
+        self,
+        profile: Profile[OutputT],
+        *,
+        model: Model | str,
+        out_dir: str | Path | None = None,
+        max_retries: int = 2,
+        anonymize_paths: bool = False,
+    ) -> None:
+        if max_retries < 0:
+            raise ValueError("max_retries must be >= 0")
+        self.profile = profile
+        self.model = model
+        self.out_dir = Path(out_dir) if out_dir is not None else None
+        self.anonymize_paths = anonymize_paths
+        self._extractor = Extractor(profile, model, max_retries=max_retries)
+
+    def run(self, inputs: str | Path | Iterable[str | Path]) -> RunResult:
+        """Process the inputs in order. A failure on one image does not stop the run."""
+        items = [inputs] if isinstance(inputs, str | Path) else list(inputs)
+        paths = expand_inputs(items)
+
+        result = RunResult()
+        writer = None
+        if self.out_dir is not None:
+            self.out_dir.mkdir(parents=True, exist_ok=True)
+            result.output_path = self.out_dir / OBSERVATIONS_FILE
+            writer = result.output_path.open("w", encoding="utf-8")
+        try:
+            for path in paths:
+                observation = self._observe(load_image(path))
+                result.observations.append(observation)
+                if writer is not None:
+                    writer.write(observation.to_json() + "\n")
+                    writer.flush()
+        finally:
+            if writer is not None:
+                writer.close()
+        return result
+
+    def _observe(self, image: ImageSource) -> Observation:
+        profile = ProfileRef(name=self.profile.name, version=self.profile.version)
+        source = SourceRef(
+            path=None if self.anonymize_paths else image.path.as_posix(),
+            sha256=image.sha256,
+            phash=image.phash,
+        )
+        model = model_label(self.model)
+        started = time.perf_counter()
+        try:
+            extraction = self._extractor.extract(image)
+        except UnexpectedModelBehavior as exc:
+            return Observation(
+                status="error",
+                profile=profile,
+                source=source,
+                data=None,
+                error=str(exc),
+                model=model,
+                usage=Usage(latency_ms=round((time.perf_counter() - started) * 1000)),
+                retries=0,
+            )
+        return Observation(
+            status="ok",
+            profile=profile,
+            source=source,
+            data=extraction.output.model_dump(mode="json"),
+            model=model,
+            usage=Usage(
+                input_tokens=extraction.input_tokens,
+                output_tokens=extraction.output_tokens,
+                requests=extraction.requests,
+                latency_ms=extraction.latency_ms,
+            ),
+            retries=extraction.retries,
+        )

framelift-0.0.1/src/framelift/profile.py

@@ -0,0 +1,61 @@
+"""`Profile`: what to read from a screen and how to read it."""
+
+from __future__ import annotations
+
+import hashlib
+import json
+from collections.abc import Callable, Sequence
+from dataclasses import dataclass, field
+
+from pydantic import BaseModel
+
+type Validator[OutputT: BaseModel] = Callable[[OutputT], OutputT | None]
+"""A check run on every extracted output.
+
+Raise `pydantic_ai.ModelRetry` with a message to make the model try again; return a (possibly
+modified) output, or `None` to keep it as is.
+"""
+
+
+@dataclass(frozen=True, kw_only=True)
+class Profile[OutputT: BaseModel]:
+    """Defines what to extract from a screen.
+
+    Args:
+        name: Identifier written to every observation, e.g. ``"fantasy-shop"``.
+        output_type: The Pydantic model to fill in. Field descriptions are passed to the model.
+        instructions: How to read the screen.
+        version: Bump this whenever the instructions or the output model change, so that
+            observations, caches and evaluation results can be told apart.
+        validators: Extra checks run on every output; see `Validator`.
+    """
+
+    name: str
+    output_type: type[OutputT]
+    instructions: str
+    version: str = "1"
+    validators: Sequence[Validator[OutputT]] = field(default_factory=tuple)
+
+    def __post_init__(self) -> None:
+        if not self.name.strip():
+            raise ValueError("Profile.name must not be empty")
+        if not self.version.strip():
+            raise ValueError("Profile.version must not be empty")
+        if not (isinstance(self.output_type, type) and issubclass(self.output_type, BaseModel)):
+            raise TypeError("Profile.output_type must be a pydantic BaseModel subclass")
+        object.__setattr__(self, "validators", tuple(self.validators))
+
+    @property
+    def fingerprint(self) -> str:
+        """A stable hash of everything that affects extraction results."""
+        payload = json.dumps(
+            {
+                "name": self.name,
+                "version": self.version,
+                "instructions": self.instructions,
+                "schema": self.output_type.model_json_schema(),
+            },
+            sort_keys=True,
+            ensure_ascii=False,
+        )
+        return hashlib.sha256(payload.encode()).hexdigest()