@nexus-cortex/server 4.26.0

package/.cortex/skills/pdf-documents/sections/10-reading-pdfs.md

@@ -0,0 +1,499 @@
+# Reading and Deconstructing PDFs
+
+## The Problem
+
+Agents currently read PDFs via OCR or text extraction, which destroys:
+- Table structure (becomes ambiguous whitespace)
+- Diagram/chart data (becomes nothing or alt text)
+- Heading hierarchy (inferred from font size, unreliable)
+- Image-text association (captions detach from figures)
+- Links (become plain text)
+
+## Runtime Context — Choose Your Path
+
+| Runtime | Best Tool | How |
+|---|---|---|
+| **Claude Code** | Built-in Read tool | `Read({ file_path: "doc.pdf", pages: "1-10" })` — up to 20 pages/request |
+| **Agent with browser-MCP access** | nexus-browser-sandbox | `run_command` with `pdf2html` utility or `pdf2htmlEX` (both pre-installed) |
+| **Server-side / CLI** | PyMuPDF directly | `pip install pymupdf` then Python script |
+| **Browser-only (no sandbox)** | Vision model on screenshots | Browse PDF, screenshot pages, send to VLM |
+
+---
+
+## Nexus Browser Sandbox (for agents with browser-MCP access)
+
+The `nexus-browser-sandbox` container has **PyMuPDF**, **pdf2htmlEX**, and a pre-built extraction utility installed. Any agent with nexus-browser MCP access can use them via `run_command` or `run_code` — no setup, no pip install.
+
+### Container Additions (Dockerfile)
+
+```dockerfile
+# ── PDF Reading Toolkit ──────────────────────────────────────
+# PyMuPDF — structured HTML extraction, image extraction, text
+RUN pip install --no-cache-dir pymupdf beautifulsoup4
+
+# pdf2htmlEX — pixel-perfect visual fidelity conversion
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    pdf2htmlex \
+    && rm -rf /var/lib/apt/lists/*
+
+# Pre-built extraction utility
+COPY pdf2html /usr/local/bin/pdf2html
+RUN chmod +x /usr/local/bin/pdf2html
+```
+
+### Extraction Utility: `/usr/local/bin/pdf2html`
+
+A Python script pre-installed in the container. Takes a PDF path, outputs structured HTML + a JSON manifest to stdout. Agents call it via `run_command` and get agent-ready output with zero boilerplate.
+
+```python
+#!/usr/bin/env python3
+"""pdf2html — Extract structured HTML + manifest from a PDF.
+
+Usage:
+  pdf2html input.pdf                     # structured HTML (agent consumption)
+  pdf2html input.pdf --manifest          # JSON manifest only (outline, tables, images)
+  pdf2html input.pdf --images /out/dir   # also extract images as PNG files
+  pdf2html input.pdf --lossless          # delegates to pdf2htmlEX (pixel-perfect)
+  pdf2html input.pdf --text              # plain text only (cheapest)
+  pdf2html input.pdf --pages 1-5         # specific page range
+"""
+
+import sys
+import os
+import json
+import argparse
+
+def extract_structured(pdf_path, pages=None, extract_images_dir=None):
+    """PyMuPDF structured extraction — HTML + manifest."""
+    import fitz
+    doc = fitz.open(pdf_path)
+
+    page_range = range(len(doc))
+    if pages:
+        start, end = pages
+        page_range = range(max(0, start - 1), min(len(doc), end))
+
+    html_parts = []
+    manifest = {
+        "source": os.path.basename(pdf_path),
+        "pages": len(doc),
+        "extracted_pages": list(page_range),
+        "images": [],
+        "tables": [],
+        "headings": [],
+    }
+
+    for i in page_range:
+        page = doc[i]
+        page_num = i + 1
+
+        html_parts.append(f'<div class="page" data-page="{page_num}">')
+        html_parts.append(page.get_text('html'))
+        html_parts.append('</div>')
+
+        # Extract images
+        images = page.get_images(full=True)
+        for img_idx, img_info in enumerate(images):
+            img_id = f"img-p{page_num}-{img_idx}"
+            manifest["images"].append({
+                "id": img_id,
+                "page": page_num,
+                "xref": img_info[0],
+                "width": img_info[2],
+                "height": img_info[3],
+            })
+
+            if extract_images_dir:
+                xref = img_info[0]
+                pix = fitz.Pixmap(doc, xref)
+                if pix.n >= 5:  # CMYK
+                    pix = fitz.Pixmap(fitz.csRGB, pix)
+                out_path = os.path.join(extract_images_dir, f"{img_id}.png")
+                pix.save(out_path)
+                manifest["images"][-1]["file"] = out_path
+
+        # Extract text blocks for headings (heuristic: large font = heading)
+        blocks = page.get_text("dict")["blocks"]
+        for block in blocks:
+            if "lines" not in block:
+                continue
+            for line in block["lines"]:
+                for span in line["spans"]:
+                    size = span["size"]
+                    text = span["text"].strip()
+                    if text and size >= 14:
+                        level = 1 if size >= 22 else 2 if size >= 16 else 3
+                        manifest["headings"].append({
+                            "level": level,
+                            "text": text,
+                            "page": page_num,
+                            "font_size": round(size, 1),
+                        })
+
+        # Detect tables (PyMuPDF table detection)
+        try:
+            tables = page.find_tables()
+            for t_idx, table in enumerate(tables):
+                t_id = f"table-p{page_num}-{t_idx}"
+                extracted = table.extract()
+                headers = extracted[0] if extracted else []
+                manifest["tables"].append({
+                    "id": t_id,
+                    "page": page_num,
+                    "headers": [h for h in headers if h],
+                    "rows": len(extracted),
+                    "cols": len(headers),
+                })
+        except Exception:
+            pass  # table detection not available in older PyMuPDF
+
+    full_html = f"""<!DOCTYPE html>
+<html><head><meta charset="UTF-8"><title>{manifest['source']}</title>
+<style>
+  .page {{ border-bottom: 2px solid #ccc; padding: 20px 0; margin-bottom: 20px; }}
+  .page::before {{ content: "Page " attr(data-page); font-size: 10pt; color: #888;
+                   display: block; margin-bottom: 10px; }}
+  img {{ max-width: 100%; height: auto; }}
+  table {{ border-collapse: collapse; margin: 10px 0; }}
+  td, th {{ border: 1px solid #ddd; padding: 4px 8px; }}
+</style>
+</head><body>{''.join(html_parts)}</body></html>"""
+
+    return full_html, manifest
+
+
+def extract_text_only(pdf_path, pages=None):
+    """Cheapest extraction — plain text."""
+    import fitz
+    doc = fitz.open(pdf_path)
+    page_range = range(len(doc))
+    if pages:
+        start, end = pages
+        page_range = range(max(0, start - 1), min(len(doc), end))
+
+    parts = []
+    for i in page_range:
+        parts.append(f"\n--- PAGE {i+1} ---\n")
+        parts.append(doc[i].get_text("text"))
+    return ''.join(parts)
+
+
+def run_pdf2htmlex(pdf_path):
+    """Delegate to pdf2htmlEX for pixel-perfect conversion."""
+    import subprocess
+    out_dir = "/tmp/pdf2htmlex_out"
+    os.makedirs(out_dir, exist_ok=True)
+    basename = os.path.splitext(os.path.basename(pdf_path))[0]
+    result = subprocess.run(
+        ["pdf2htmlEX", "--zoom", "1.5", "--process-outline", "1",
+         "--dest-dir", out_dir, pdf_path],
+        capture_output=True, text=True
+    )
+    if result.returncode != 0:
+        print(f"pdf2htmlEX error: {result.stderr}", file=sys.stderr)
+        sys.exit(1)
+
+    out_file = os.path.join(out_dir, f"{basename}.html")
+    if os.path.exists(out_file):
+        with open(out_file) as f:
+            print(f.read())
+    else:
+        print(f"Output not found at {out_file}", file=sys.stderr)
+        sys.exit(1)
+
+
+def parse_pages(pages_str):
+    """Parse '1-5' or '3' into (start, end) tuple."""
+    if '-' in pages_str:
+        parts = pages_str.split('-')
+        return (int(parts[0]), int(parts[1]))
+    else:
+        n = int(pages_str)
+        return (n, n)
+
+
+def main():
+    parser = argparse.ArgumentParser(description="Extract structured HTML from PDF")
+    parser.add_argument("pdf", help="Path to PDF file")
+    parser.add_argument("--manifest", action="store_true", help="Output JSON manifest only")
+    parser.add_argument("--images", metavar="DIR", help="Extract images to directory")
+    parser.add_argument("--lossless", action="store_true", help="Use pdf2htmlEX (pixel-perfect)")
+    parser.add_argument("--text", action="store_true", help="Plain text only (cheapest)")
+    parser.add_argument("--pages", metavar="RANGE", help="Page range, e.g. 1-5 or 3")
+    args = parser.parse_args()
+
+    if not os.path.exists(args.pdf):
+        print(f"File not found: {args.pdf}", file=sys.stderr)
+        sys.exit(1)
+
+    pages = parse_pages(args.pages) if args.pages else None
+
+    if args.lossless:
+        run_pdf2htmlex(args.pdf)
+        return
+
+    if args.text:
+        print(extract_text_only(args.pdf, pages))
+        return
+
+    if args.images:
+        os.makedirs(args.images, exist_ok=True)
+
+    html, manifest = extract_structured(args.pdf, pages, args.images)
+
+    if args.manifest:
+        print(json.dumps(manifest, indent=2))
+    else:
+        # Print manifest as HTML comment at top, then full HTML
+        print(f"<!-- MANIFEST\n{json.dumps(manifest, indent=2)}\n-->")
+        print(html)
+
+
+if __name__ == "__main__":
+    main()
+```
+
+### Agent Workflow via nexus-browser MCP
+
+```bash
+# ── Structured extraction (default — best for agent consumption) ──
+run_command({ command: "pdf2html /tmp/document.pdf" })
+# Returns: HTML comment with JSON manifest + structured HTML body
+
+# ── Manifest only (cheapest — just the outline, tables, image refs) ──
+run_command({ command: "pdf2html /tmp/document.pdf --manifest" })
+# Returns: JSON with pages, headings, table summaries, image locations
+
+# ── Plain text only (cheapest tokens) ──
+run_command({ command: "pdf2html /tmp/document.pdf --text" })
+# Returns: raw text with page markers
+
+# ── Extract images to files ──
+run_command({ command: "pdf2html /tmp/document.pdf --images /tmp/pdf-images/" })
+# Returns: HTML + manifest with image file paths
+
+# ── Pixel-perfect lossless conversion (pdf2htmlEX) ──
+run_command({ command: "pdf2html /tmp/document.pdf --lossless" })
+# Returns: pixel-perfect HTML (absolute-positioned divs, visually exact)
+
+# ── Specific pages only ──
+run_command({ command: "pdf2html /tmp/document.pdf --pages 1-5 --manifest" })
+```
+
+**Getting the PDF into the sandbox:**
+```bash
+# Option 1: Download from URL
+run_command({ command: "curl -sL -o /tmp/document.pdf 'https://example.com/report.pdf'" })
+
+# Option 2: Browse to PDF URL (Chromium downloads it)
+browse({ url: "https://example.com/report.pdf" })
+# Then find it in the browser's download directory
+
+# Option 3: If PDF is already in VFS, copy via the agent's file tools
+# (depends on sandbox ↔ VFS bridge)
+```
+
+### When to Use --lossless (pdf2htmlEX)
+
+The `--lossless` flag delegates to pdf2htmlEX instead of PyMuPDF. Use it when:
+- User asks for **exact visual reproduction** of the PDF in a browser
+- The PDF has complex **multi-column layouts** that PyMuPDF misinterprets
+- The output will be **displayed to a human**, not parsed by an agent
+- You need to **embed the PDF in a web page** with exact fidelity
+
+Do NOT use `--lossless` for agent consumption — the output is CSS absolute-positioned divs that are impossible to parse semantically. Use the default PyMuPDF path for anything an agent needs to understand.
+
+---
+
+## Direct PyMuPDF Usage (CLI / Server-Side)
+
+When you have Python available (Claude Code bash, server-side scripts, CI), use PyMuPDF directly:
+
+```bash
+pip install pymupdf
+python3 -c "
+import fitz
+doc = fitz.open('input.pdf')
+html_parts = []
+for i, page in enumerate(doc):
+    html_parts.append(f'<div class=\"page\" data-page=\"{i+1}\">')
+    html_parts.append(page.get_text('html'))
+    html_parts.append('</div>')
+with open('output.html', 'w') as f:
+    f.write('<html><body>' + ''.join(html_parts) + '</body></html>')
+"
+```
+
+## Deconstructing PDF-to-HTML for Agent Consumption
+
+Once you have HTML from any extraction method, deconstruct it into an indexed structure:
+
+### Step 1: Build a Media Index
+
+```python
+from bs4 import BeautifulSoup
+import json
+
+with open('converted.html') as f:
+    soup = BeautifulSoup(f.read(), 'html.parser')
+
+media_index = {
+    "images": [],
+    "tables": [],
+    "headings": [],
+    "links": [],
+    "figures": [],
+}
+
+for i, img in enumerate(soup.find_all('img')):
+    media_index["images"].append({
+        "id": f"img-{i}",
+        "src": img.get('src', '')[:100],
+        "alt": img.get('alt', ''),
+        "page": img.find_parent(attrs={"data-page": True}).get("data-page", "?"),
+        "width": img.get('width'),
+        "height": img.get('height'),
+    })
+
+for i, table in enumerate(soup.find_all('table')):
+    rows = table.find_all('tr')
+    headers = [th.get_text(strip=True) for th in rows[0].find_all(['th', 'td'])] if rows else []
+    media_index["tables"].append({
+        "id": f"table-{i}",
+        "headers": headers,
+        "row_count": len(rows),
+        "page": table.find_parent(attrs={"data-page": True}).get("data-page", "?"),
+    })
+
+for h in soup.find_all(['h1','h2','h3','h4','h5','h6']):
+    media_index["headings"].append({
+        "level": int(h.name[1]),
+        "text": h.get_text(strip=True),
+    })
+
+print(json.dumps(media_index, indent=2))
+```
+
+### Step 2: Extract Text as Structured Markdown
+
+```python
+def html_to_agent_markdown(html_path):
+    with open(html_path) as f:
+        soup = BeautifulSoup(f.read(), 'html.parser')
+
+    sections = []
+    for page_div in soup.find_all('div', class_='page'):
+        page_num = page_div.get('data-page', '?')
+        sections.append(f"\n<!-- PAGE {page_num} -->\n")
+
+        for elem in page_div.children:
+            if elem.name in ('h1','h2','h3','h4','h5','h6'):
+                level = int(elem.name[1])
+                sections.append(f"{'#' * level} {elem.get_text(strip=True)}\n")
+            elif elem.name == 'table':
+                sections.append(table_to_markdown(elem))
+            elif elem.name == 'p':
+                text = elem.get_text(strip=True)
+                if text:
+                    sections.append(f"{text}\n")
+            elif elem.name == 'img':
+                alt = elem.get('alt', 'image')
+                sections.append(f"![{alt}](ref:img-{id})\n")
+            elif elem.name == 'figure':
+                caption = elem.find('figcaption')
+                sections.append(f"[FIGURE: {caption.get_text(strip=True) if caption else 'untitled'}]\n")
+
+    return '\n'.join(sections)
+```
+
+### Step 3: Agent Consumption Format
+
+The final output for agent ingestion should be:
+
+```
+DOCUMENT_MANIFEST:
+  source: input.pdf
+  pages: 13
+  images: 4 (indexed as img-0 through img-3)
+  tables: 3 (indexed as table-0 through table-2)
+  headings: 12 (see outline below)
+
+OUTLINE:
+  1. Introduction
+    1.1 Background
+    1.2 Methodology
+  2. Findings
+    2.1 Data Analysis (contains table-0, table-1)
+    2.2 Visual Results (contains img-1, img-2)
+  3. Conclusions
+
+TABLE_SUMMARIES:
+  table-0 (page 4): Revenue by Quarter | 4 cols | 12 rows
+  table-1 (page 5): Cost Breakdown | 6 cols | 8 rows
+  table-2 (page 9): Comparison Matrix | 5 cols | 5 rows
+
+IMAGE_DESCRIPTIONS:
+  img-0 (page 1): [cover photo] — requires vision model for description
+  img-1 (page 6): [bar chart] — requires vision model for data extraction
+  img-2 (page 7): [photograph] — requires vision model for description
+  img-3 (page 11): [diagram] — requires vision model for description
+
+TEXT_CONTENT:
+  [structured markdown follows...]
+```
+
+## Handling Charts and Diagrams in PDFs
+
+Charts in PDFs are stored as either:
+1. **Vector paths** (lines, curves, fills) — can be extracted as SVG but lose data labels
+2. **Rasterized images** — requires vision model to extract data
+
+**Recommended approach for charts:**
+```
+1. Extract chart region as image (PyMuPDF page.get_pixmap(clip=rect))
+2. Send image to vision model: "Extract the data from this chart as a markdown table"
+3. Include both: image reference (for visual context) + extracted data table (for agent reasoning)
+```
+
+```python
+import fitz
+
+doc = fitz.open('input.pdf')
+page = doc[5]  # page with the chart
+
+chart_rect = fitz.Rect(50, 200, 500, 450)
+pix = page.get_pixmap(clip=chart_rect, dpi=300)
+pix.save('chart-page6.png')
+```
+
+## Tool Selection Guide
+
+| Tool | Text | Tables | Images | Layout | Speed | Available In |
+|------|------|--------|--------|--------|-------|-------------|
+| **pdf2html utility** | Excellent | Good | Extract as PNG | Positioned HTML | Fast | nexus-browser-sandbox |
+| **pdf2htmlEX** | Excellent | Visual only | Embeds | **Pixel-perfect** | Medium | nexus-browser-sandbox |
+| **PyMuPDF (fitz)** | Excellent | Good (HTML mode) | Extract as PNG | Positioned HTML | Fast | sandbox, CLI, server |
+| **Claude Code Read** | Good | Basic | No | Text only | Instant | Claude Code |
+| **Tabula** | N/A | **Best** (structured) | N/A | N/A | Medium | CLI (Java) |
+| **Camelot** | N/A | Excellent | N/A | N/A | Medium | CLI (Python) |
+| **Vision model** | Good | Good | **Best** (understanding) | Visual | Expensive | Any agent |
+
+**Decision tree:**
+```
+Agent with browser-MCP access?
+  → run_command("pdf2html doc.pdf")              # structured extraction
+  → run_command("pdf2html doc.pdf --lossless")    # visual fidelity
+
+Agent in Claude Code?
+  → Read tool for text (up to 20 pages)
+  → Bash + PyMuPDF for structured HTML
+
+Need tables specifically?
+  → Tabula or Camelot (CLI only)
+
+Need chart/diagram data?
+  → Extract image region + vision model
+
+User wants exact visual reproduction?
+  → pdf2htmlEX via --lossless flag
+```