agentpdf 0.1.0__tar.gz

agentpdf-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Manuel Stocker
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

agentpdf-0.1.0/PKG-INFO

@@ -0,0 +1,129 @@
+Metadata-Version: 2.4
+Name: agentpdf
+Version: 0.1.0
+Summary: Convert large PDF documents into small, searchable files for AI coding agents
+Keywords: pdf,markdown,ai,agents,llm,documentation,search
+Author: Manuel Stocker
+Author-email: Manuel Stocker <mensi@mensi.ch>
+License-Expression: MIT
+License-File: LICENSE
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Text Processing :: Markup :: Markdown
+Classifier: Topic :: Utilities
+Requires-Dist: pdfplumber>=0.11
+Requires-Dist: pypdf>=5
+Requires-Dist: pypdfium2>=5
+Requires-Python: >=3.12
+Project-URL: Repository, https://github.com/mensi/agentpdf
+Project-URL: Issues, https://github.com/mensi/agentpdf/issues
+Description-Content-Type: text/markdown
+
+# agentpdf
+
+> [!WARNING]
+> This is a vibe-coded tool. The code was briefly reviewed by a human but not
+> directly changed / maintained manually.
+
+Make large PDFs — specifications, reference manuals, datasheets — usable by AI
+coding agents without reading them into context.
+
+agentpdf converts a PDF once into small Markdown section files, images of its
+tables and figures, a table of contents and a full-text search index. Agents
+then search and read only the parts they need, with their usual tools.
+
+```sh
+uvx agentpdf convert spec.pdf -o docs/reference
+uvx agentpdf search -d docs/reference "attach request timer"
+```
+
+```
+docs/reference/
+├── AGENTS.md          # how agents should use this directory
+├── search.sqlite      # full-text index over all documents
+└── spec/
+    ├── INDEX.md       # table of contents with page ranges and sizes
+    ├── PAGES.md       # page → section, table and figure images
+    ├── sections/      # Markdown, ≤ ~6k tokens per file, with page markers
+    └── images/        # one PNG per table and figure
+```
+
+Requires [uv](https://docs.astral.sh/uv/) (or `pip install agentpdf`). Runs
+locally on CPU, in seconds even for documents with hundreds of pages; no
+models or network access needed.
+
+## Using it with agents
+
+**Point the agent at the output.** Add one line to your project's `AGENTS.md`
+or `CLAUDE.md`:
+
+```markdown
+Reference documents (PDF specs) are in docs/reference/; see its AGENTS.md.
+```
+
+The generated `AGENTS.md` explains the layout and the search command in a few
+lines.
+
+**Or just ask.** Agents that can run commands can do the conversion
+themselves: *"Convert spec.pdf with `uvx agentpdf` (github.com/mensi/agentpdf)
+and use it to implement …"*
+
+**Or install the skill** (Claude Code), so agents convert and search PDFs on
+their own:
+
+```
+/plugin install agentpdf --marketplace mensi/agentpdf
+```
+
+On older versions: `/plugin marketplace add mensi/agentpdf`, then
+`/plugin install agentpdf@agentpdf`. The skill is a plain
+[Agent Skill](https://agentskills.io) in
+[`plugins/agentpdf/skills/agentpdf`](plugins/agentpdf/skills/agentpdf/SKILL.md),
+which you can also copy into `~/.claude/skills/` or another agent's skill
+directory.
+
+## How it works
+
+- Text is extracted with [PDFium](https://pdfium.googlesource.com/pdfium/) in
+  reading order; running headers, footers and printed tables of contents are
+  dropped; paragraphs are re-flowed so phrases are searchable.
+- Sections follow the PDF's bookmarks, or headings inferred from typography.
+  Large sections are split at subheadings to fit the size budget.
+- Tables and figures are not converted to Markdown tables, which is
+  unreliable. Each is rendered as an image — the source of truth — plus a
+  text transcription that keeps characters at their horizontal position.
+- Search uses SQLite FTS5 (BM25 ranking, stemming); hits point to file, line,
+  page and heading.
+
+Only PDFs with a text layer are supported; scanned pages are rendered as
+images but not OCRed.
+
+## Options
+
+```
+agentpdf convert PDF... -o DIR [--max-tokens 6000] [--dpi 150]
+                               [--page-images] [--split-pdf] [--no-index]
+                               [--force]
+agentpdf search QUERY [-d DIR] [-n 10] [--doc NAME] [--any]
+```
+
+`--page-images` also renders every page, `--split-pdf` writes one PDF per
+section, `--force` writes into an existing non-empty output directory.
+Converting again replaces a document's output.
+
+## Development
+
+```sh
+uv sync
+uv run pytest
+uv run ruff check . && uv run ruff format --check .
+```
+
+## License
+
+MIT

agentpdf-0.1.0/README.md

@@ -0,0 +1,103 @@
+# agentpdf
+
+> [!WARNING]
+> This is a vibe-coded tool. The code was briefly reviewed by a human but not
+> directly changed / maintained manually.
+
+Make large PDFs — specifications, reference manuals, datasheets — usable by AI
+coding agents without reading them into context.
+
+agentpdf converts a PDF once into small Markdown section files, images of its
+tables and figures, a table of contents and a full-text search index. Agents
+then search and read only the parts they need, with their usual tools.
+
+```sh
+uvx agentpdf convert spec.pdf -o docs/reference
+uvx agentpdf search -d docs/reference "attach request timer"
+```
+
+```
+docs/reference/
+├── AGENTS.md          # how agents should use this directory
+├── search.sqlite      # full-text index over all documents
+└── spec/
+    ├── INDEX.md       # table of contents with page ranges and sizes
+    ├── PAGES.md       # page → section, table and figure images
+    ├── sections/      # Markdown, ≤ ~6k tokens per file, with page markers
+    └── images/        # one PNG per table and figure
+```
+
+Requires [uv](https://docs.astral.sh/uv/) (or `pip install agentpdf`). Runs
+locally on CPU, in seconds even for documents with hundreds of pages; no
+models or network access needed.
+
+## Using it with agents
+
+**Point the agent at the output.** Add one line to your project's `AGENTS.md`
+or `CLAUDE.md`:
+
+```markdown
+Reference documents (PDF specs) are in docs/reference/; see its AGENTS.md.
+```
+
+The generated `AGENTS.md` explains the layout and the search command in a few
+lines.
+
+**Or just ask.** Agents that can run commands can do the conversion
+themselves: *"Convert spec.pdf with `uvx agentpdf` (github.com/mensi/agentpdf)
+and use it to implement …"*
+
+**Or install the skill** (Claude Code), so agents convert and search PDFs on
+their own:
+
+```
+/plugin install agentpdf --marketplace mensi/agentpdf
+```
+
+On older versions: `/plugin marketplace add mensi/agentpdf`, then
+`/plugin install agentpdf@agentpdf`. The skill is a plain
+[Agent Skill](https://agentskills.io) in
+[`plugins/agentpdf/skills/agentpdf`](plugins/agentpdf/skills/agentpdf/SKILL.md),
+which you can also copy into `~/.claude/skills/` or another agent's skill
+directory.
+
+## How it works
+
+- Text is extracted with [PDFium](https://pdfium.googlesource.com/pdfium/) in
+  reading order; running headers, footers and printed tables of contents are
+  dropped; paragraphs are re-flowed so phrases are searchable.
+- Sections follow the PDF's bookmarks, or headings inferred from typography.
+  Large sections are split at subheadings to fit the size budget.
+- Tables and figures are not converted to Markdown tables, which is
+  unreliable. Each is rendered as an image — the source of truth — plus a
+  text transcription that keeps characters at their horizontal position.
+- Search uses SQLite FTS5 (BM25 ranking, stemming); hits point to file, line,
+  page and heading.
+
+Only PDFs with a text layer are supported; scanned pages are rendered as
+images but not OCRed.
+
+## Options
+
+```
+agentpdf convert PDF... -o DIR [--max-tokens 6000] [--dpi 150]
+                               [--page-images] [--split-pdf] [--no-index]
+                               [--force]
+agentpdf search QUERY [-d DIR] [-n 10] [--doc NAME] [--any]
+```
+
+`--page-images` also renders every page, `--split-pdf` writes one PDF per
+section, `--force` writes into an existing non-empty output directory.
+Converting again replaces a document's output.
+
+## Development
+
+```sh
+uv sync
+uv run pytest
+uv run ruff check . && uv run ruff format --check .
+```
+
+## License
+
+MIT

agentpdf-0.1.0/pyproject.toml

@@ -0,0 +1,73 @@
+[project]
+name = "agentpdf"
+version = "0.1.0"
+description = "Convert large PDF documents into small, searchable files for AI coding agents"
+readme = "README.md"
+license = "MIT"
+license-files = ["LICENSE"]
+requires-python = ">=3.12"
+keywords = [
+    "pdf",
+    "markdown",
+    "ai",
+    "agents",
+    "llm",
+    "documentation",
+    "search",
+]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Environment :: Console",
+    "Intended Audience :: Developers",
+    "Operating System :: OS Independent",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Topic :: Text Processing :: Markup :: Markdown",
+    "Topic :: Utilities",
+]
+dependencies = [
+    "pdfplumber>=0.11",
+    "pypdf>=5",
+    "pypdfium2>=5",
+]
+
+[[project.authors]]
+name = "Manuel Stocker"
+email = "mensi@mensi.ch"
+
+[project.urls]
+Repository = "https://github.com/mensi/agentpdf"
+Issues = "https://github.com/mensi/agentpdf/issues"
+
+[project.scripts]
+agentpdf = "agentpdf.cli:main"
+
+[build-system]
+requires = ["uv_build>=0.13.1,<0.14.0"]
+build-backend = "uv_build"
+
+[dependency-groups]
+dev = [
+    "pytest>=9.1.1",
+    "ruff>=0.17.1",
+]
+
+[tool.ruff]
+line-length = 110
+
+[tool.ruff.lint]
+select = [
+    "E",
+    "F",
+    "W",
+    "I",
+    "B",
+    "UP",
+    "SIM",
+    "RUF",
+]
+ignore = ["RUF001"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]

agentpdf-0.1.0/pyproject.toml.orig

@@ -0,0 +1,56 @@
+[project]
+name = "agentpdf"
+version = "0.1.0"
+description = "Convert large PDF documents into small, searchable files for AI coding agents"
+readme = "README.md"
+license = "MIT"
+license-files = ["LICENSE"]
+authors = [
+    { name = "Manuel Stocker", email = "mensi@mensi.ch" }
+]
+requires-python = ">=3.12"
+keywords = ["pdf", "markdown", "ai", "agents", "llm", "documentation", "search"]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Environment :: Console",
+    "Intended Audience :: Developers",
+    "Operating System :: OS Independent",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Topic :: Text Processing :: Markup :: Markdown",
+    "Topic :: Utilities",
+]
+dependencies = [
+    "pdfplumber>=0.11",
+    "pypdf>=5",
+    "pypdfium2>=5",
+]
+
+[project.urls]
+Repository = "https://github.com/mensi/agentpdf"
+Issues = "https://github.com/mensi/agentpdf/issues"
+
+[build-system]
+requires = ["uv_build>=0.13.1,<0.14.0"]
+build-backend = "uv_build"
+
+[dependency-groups]
+dev = [
+    "pytest>=9.1.1",
+    "ruff>=0.17.1",
+]
+
+[project.scripts]
+agentpdf = "agentpdf.cli:main"
+
+[tool.ruff]
+line-length = 110
+
+[tool.ruff.lint]
+select = ["E", "F", "W", "I", "B", "UP", "SIM", "RUF"]
+# Typographic characters (dashes, bullets) are matched intentionally.
+ignore = ["RUF001"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]

agentpdf-0.1.0/src/agentpdf/__init__.py

@@ -0,0 +1,13 @@
+"""Convert large PDF documents into small, searchable files for AI coding agents."""
+
+from importlib.metadata import PackageNotFoundError, version
+
+try:
+    __version__ = version("agentpdf")
+except PackageNotFoundError:
+    __version__ = "0.1.0"
+
+from .convert import Options, convert
+from .search import Hit, build_index, search
+
+__all__ = ["Hit", "Options", "__version__", "build_index", "convert", "search"]

agentpdf-0.1.0/src/agentpdf/chars.py

@@ -0,0 +1,265 @@
+"""Characters from PDFium, grouped into lines, words and layout text.
+
+PDFium follows the content stream order (usually the intended reading order,
+including multi-column layouts), reconstructs word spacing and decodes glyphs
+reliably.
+"""
+
+import ctypes
+import itertools
+import math
+from collections.abc import Callable
+from dataclasses import dataclass
+
+import pypdfium2 as pdfium
+import pypdfium2.raw as pdfium_c
+
+from .model import BBox, Line, bbox_union
+
+# Characters PDFium uses to mark a hyphen at a line break.
+_SOFT_HYPHENS = "\ufffe\xad\u0002"
+
+
+@dataclass(slots=True)
+class Char:
+    text: str
+    index: int  # position in PDFium's text stream
+    bbox: BBox | None  # None for whitespace and line breaks
+    size: float = 0.0
+    bold: bool = False
+    upright: bool = True
+    soft_hyphen: bool = False
+
+
+Word = list[Char]
+
+
+def _display_transform(page: pdfium.PdfPage) -> Callable[[float, float, float, float], BBox]:
+    """Map PDF-space boxes (left, bottom, right, top) to displayed page
+    coordinates (top-left origin, after /Rotate)."""
+    mx0, my0, mx1, my1 = page.get_mediabox()
+    width, height = mx1 - mx0, my1 - my0
+    rotation = page.get_rotation() % 360
+
+    def point(x: float, y: float) -> tuple[float, float]:
+        x, y = x - mx0, y - my0
+        if rotation == 90:
+            return y, x
+        if rotation == 180:
+            return width - x, y
+        if rotation == 270:
+            return height - y, width - x
+        return x, height - y
+
+    def transform(left: float, bottom: float, right: float, top: float) -> BBox:
+        ax, ay = point(left, bottom)
+        bx, by = point(right, top)
+        return (min(ax, bx), min(ay, by), max(ax, bx), max(ay, by))
+
+    return transform
+
+
+def read_chars(page: pdfium.PdfPage) -> list[Char]:
+    """Read all characters of a page in stream order."""
+    textpage = page.get_textpage()
+    try:
+        return _read_chars(page, textpage)
+    finally:
+        textpage.close()
+
+
+def _read_chars(page: pdfium.PdfPage, textpage: pdfium.PdfTextPage) -> list[Char]:
+    to_display = _display_transform(page)
+    rotation = math.radians(page.get_rotation())
+    raw = textpage.raw
+    font_buf = ctypes.create_string_buffer(256)
+    font_flags = ctypes.c_int()
+    matrix = pdfium_c.FS_MATRIX()
+    chars: list[Char] = []
+    last_style = None
+    font = ""
+    # Characters are read by index (not by slicing get_text_range(), whose
+    # string offsets can drift from character indices).
+    for i in range(textpage.count_chars()):
+        code = pdfium_c.FPDFText_GetUnicode(raw, i)
+        ch = chr(code) if 0 < code < 0x110000 and not 0xD800 <= code < 0xE000 else "�"
+        if ch == "\r":
+            continue
+        if ch in _SOFT_HYPHENS:
+            chars.append(Char("-", i, None, soft_hyphen=True))
+            continue
+        if ch.isspace() or pdfium_c.FPDFText_IsGenerated(raw, i) == 1:
+            chars.append(Char("\n" if ch == "\n" else " ", i, None))
+            continue
+        # The nominal font size is often 1 with the actual size applied via
+        # the text matrix; the effective size accounts for both.
+        pdfium_c.FPDFText_GetMatrix(raw, i, ctypes.byref(matrix))
+        size = round(pdfium_c.FPDFText_GetFontSize(raw, i) * (math.hypot(matrix.c, matrix.d) or 1.0), 2)
+        weight = pdfium_c.FPDFText_GetFontWeight(raw, i)
+        if (size, weight) != last_style:
+            pdfium_c.FPDFText_GetFontInfo(raw, i, font_buf, len(font_buf), ctypes.byref(font_flags))
+            font = font_buf.value.decode("utf-8", "replace")
+            last_style = (size, weight)
+        # Upright as displayed, i.e. taking the page's /Rotate into account.
+        angle = pdfium_c.FPDFText_GetCharAngle(raw, i)
+        displayed = (angle + rotation) % (2 * math.pi)
+        chars.append(
+            Char(
+                text=ch,
+                index=i,
+                bbox=to_display(*textpage.get_charbox(i, loose=True)),
+                size=size,
+                bold=weight >= 600 or "bold" in font.lower(),
+                upright=angle < 0 or min(displayed, 2 * math.pi - displayed) < 0.05,
+            )
+        )
+    return chars
+
+
+def lines_from_chars(chars: list[Char]) -> list[Line]:
+    """Group characters into lines at PDFium's line breaks, and where the
+    geometry shows a break PDFium did not report (e.g. on rotated pages)."""
+    lines: list[Line] = []
+    current: list[Char] = []
+
+    def flush() -> None:
+        visible = [c for c in current if c.bbox is not None]
+        text = " ".join("".join(c.text for c in current).split())
+        if visible and text:
+            lines.append(
+                Line(
+                    text=text,
+                    bbox=_union(visible),
+                    size=max(c.size for c in visible),
+                    bold=sum(c.bold for c in visible) > len(visible) / 2,
+                    order=visible[0].index,
+                    soft_hyphen=current[-1].soft_hyphen,
+                    tabbed=any(
+                        b.bbox[0] - a.bbox[2] > 1.5 * max(a.size, b.size)
+                        for a, b in itertools.pairwise(visible)
+                    ),
+                )
+            )
+        current.clear()
+
+    prev: Char | None = None
+    for c in chars:
+        if c.text == "\n":
+            flush()
+            prev = None
+            continue
+        if c.bbox is not None:
+            if prev is not None and current and _starts_new_line(prev, c):
+                flush()
+            prev = c
+        current.append(c)
+        if c.soft_hyphen:
+            flush()
+    flush()
+    return lines
+
+
+def _starts_new_line(prev: Char, c: Char) -> bool:
+    if prev.upright != c.upright:
+        return True
+    if not c.upright:
+        return False
+    no_vertical_overlap = c.bbox[1] >= prev.bbox[3] or c.bbox[3] <= prev.bbox[1]
+    moved_back = c.bbox[0] < prev.bbox[0] - max(prev.size, c.size)
+    return no_vertical_overlap or moved_back
+
+
+def words(chars: list[Char]) -> list[Word]:
+    """Split visible characters into words at whitespace."""
+    return [
+        list(group) for visible, group in itertools.groupby(chars, lambda c: c.bbox is not None) if visible
+    ]
+
+
+def word_bbox(word: Word) -> BBox:
+    return _union(word)
+
+
+def _union(chars: list[Char]) -> BBox:
+    bbox = chars[0].bbox
+    for c in chars[1:]:
+        bbox = bbox_union(bbox, c.bbox)
+    return bbox
+
+
+def layout_text(chars: list[Char], bbox: BBox) -> str:
+    """Render characters as plain text that preserves horizontal positions.
+
+    Upright characters are placed on a character grid; rows are formed from
+    vertically overlapping characters so sub- and superscripts stay on their
+    base line. Rotated text (axis labels etc.) is listed separately.
+    """
+    upright = [c for c in chars if c.bbox is not None and c.upright]
+    rotated = [c for c in chars if c.bbox is not None and not c.upright]
+    out = _grid_rows(upright, bbox) if upright else []
+    if rotated:
+        out += ["", "[rotated text] " + " | ".join(_runs(rotated))]
+    indent = min((len(s) - len(s.lstrip()) for s in out if s.strip()), default=0)
+    return "\n".join(s[indent:] for s in out).strip("\n")
+
+
+def _grid_rows(chars: list[Char], bbox: BBox) -> list[str]:
+    sizes = sorted(c.size for c in chars)
+    body = sizes[len(sizes) // 2] or 8.0
+    unit = body * 0.5  # approximate width of one monospace column
+
+    # Tallest glyphs define the rows; smaller ones (subscripts) join a row
+    # they overlap substantially.
+    rows: list[tuple[float, float, list[Char]]] = []
+    for c in sorted(chars, key=lambda c: -c.size):
+        top, bottom = c.bbox[1], c.bbox[3]
+        for row_top, row_bottom, members in rows:
+            overlap = min(bottom, row_bottom) - max(top, row_top)
+            if overlap > 0.4 * min(bottom - top, row_bottom - row_top):
+                members.append(c)
+                break
+        else:
+            rows.append((top, bottom, [c]))
+    rows.sort(key=lambda r: r[0])
+
+    out: list[str] = []
+    prev_bottom = None
+    for top, bottom, members in rows:
+        if prev_bottom is not None and top - prev_bottom > body * 1.2:
+            out.append("")
+        prev_bottom = bottom
+        members.sort(key=lambda c: c.bbox[0])
+        line: list[str] = []
+        prev: Char | None = None
+        for c in members:
+            if prev is not None and c.index == prev.index + 1 and c.bbox[0] - prev.bbox[2] < c.size * 0.2:
+                # Continuation of a word: glyph widths vary, so append
+                # directly instead of snapping to the grid.
+                line.append(c.text)
+            else:
+                col = max(round((c.bbox[0] - bbox[0]) / unit), len(line) + (1 if line else 0))
+                line.extend(" " * (col - len(line)))
+                line.append(c.text)
+            prev = c
+        out.append("".join(line).rstrip())
+    return out
+
+
+def _runs(chars: list[Char]) -> list[str]:
+    """Join characters that are consecutive in the stream into labels."""
+    labels: list[str] = []
+    current: list[str] = []
+    last_index = 0
+    for c in sorted(chars, key=lambda c: c.index):
+        if current:
+            step = c.index - last_index
+            if step > 2:
+                labels.append("".join(current).strip())
+                current = []
+            elif step == 2:  # separated by a space
+                current.append(" ")
+        current.append(c.text)
+        last_index = c.index
+    if current:
+        labels.append("".join(current).strip())
+    return labels