sophhub 0.2.1 → 0.2.2

package/skills/sophnet-pdf/src/SKILL.md

@@ -0,0 +1,413 @@
+---
+name: sophnet-pdf
+description: Comprehensive PDF processing for creating, editing, and analyzing PDF documents. Use when the user mentions PDF files, needs to create new PDFs (reports, invoices, certificates), edit existing PDFs (text replacement, image operations, watermarks), extract content, merge/split files, or perform OCR on scanned documents.
+---
+
+# PDF Processing Guide
+
+## MANDATORY: Working Directory
+
+**EVERY command in this skill MUST be executed from THIS skill's directory.** Before running ANY command — Python or bash script — you MUST `cd` into this skill's directory first. Determine the absolute path of this `SKILL.md` file and use its parent directory.
+
+```bash
+SKILL_DIR="<absolute-path-to-this-skills-sophnet-pdf-directory>"
+cd "$SKILL_DIR"
+```
+
+**NEVER run commands from the repository root or any other directory.** If you do, `uv run` won't find `pyproject.toml`, and `python` won't have access to required packages.
+
+## Overview
+
+This guide covers essential PDF processing operations using Python libraries and command-line tools. For advanced features, JavaScript libraries, and detailed examples, see REFERENCE.md. If you need to fill out a PDF form, read FORMS.md and follow its instructions.
+
+### ⚠️ CJK Text in PDFs — READ FIRST
+
+**If the PDF will contain ANY Chinese/Japanese/Korean text** (including titles, body text, table cells, or watermarks), you MUST handle CJK fonts correctly:
+
+- **reportlab**: Default fonts (`Helvetica`, `Times-Roman`) render CJK as **black boxes (█)**. You MUST register `STSong-Light` and call `addMapping()` before creating any content. See the "reportlab - Create PDFs" section for the exact setup code.
+- **PyMuPDF (fitz)**: Use `fontname="china-s"` for `insert_text()` / `insert_textbox()`. This is built-in and requires no extra setup.
+- **Rule of thumb**: When in doubt, always use CJK-safe fonts. `STSong-Light` (reportlab) and `china-s` (PyMuPDF) both handle mixed CJK+Latin text correctly.
+
+## Python Runtime (uv)
+
+**CRITICAL: All Python execution in this skill MUST use `uv run --project .` from the skill directory. NEVER use bare `python3`, `python`, or `pip install` directly — the required packages (pdfplumber, reportlab, pypdf, etc.) are ONLY available inside the uv virtual environment defined by this skill's `pyproject.toml`. Direct `python3` will fail with ModuleNotFoundError.**
+
+**IMPORTANT: Only use libraries available in this skill's pyproject.toml (reportlab, pypdf, pdfplumber, pillow, pdf2image, pymupdf). Do NOT import matplotlib, numpy, pandas, or other packages not listed — they will cause ModuleNotFoundError. For charts/graphs, use `reportlab.graphics.charts` (VerticalBarChart, HorizontalBarChart, Pie, etc.) instead of matplotlib.**
+
+First, ensure the environment is set up (run once per session):
+
+```bash
+cd "$SKILL_DIR"
+bash scripts/ensure_uv_env.sh
+```
+
+Then ALL Python commands must use this prefix:
+
+```bash
+cd "$SKILL_DIR" && uv run --project . python <script-or-module>
+```
+
+This applies to both the provided scripts AND any inline Python code you write. For inline code, use:
+
+```bash
+cd "$SKILL_DIR" && uv run --project . python -c "import pdfplumber; ..."
+```
+
+## Delivery
+
+Local PDF creation/editing/analysis does not require any Sophnet API key.
+
+**IMPORTANT: After creating or modifying a PDF, ALWAYS upload it and return the download URL to the user.** This is the default behavior — do not skip the upload step.
+
+```bash
+bash scripts/upload_file.sh --file <absolute-path-to-pdf>
+```
+
+Upload command output contract:
+
+- `FILE_PATH=<absolute-path>`
+- `UPLOAD_STATUS=uploaded|skipped`
+- `DOWNLOAD_URL=<https://...>` (present only when uploaded)
+
+Delivery rules:
+
+- **ALWAYS call `scripts/upload_file.sh` after producing a PDF file**, then return the `DOWNLOAD_URL` to the user.
+- If `UPLOAD_STATUS=uploaded`, return the exact `DOWNLOAD_URL` value to the user as a clickable link.
+- If `UPLOAD_STATUS=skipped` (missing API key), return `FILE_PATH` instead of failing the whole task.
+- Keep URL output logic independent inside `sophnet-pdf/scripts`. Do not call other skills' upload scripts.
+
+## Quick Start
+
+```python
+from pypdf import PdfReader, PdfWriter
+
+# Read a PDF
+reader = PdfReader("document.pdf")
+print(f"Pages: {len(reader.pages)}")
+
+# Extract text
+text = ""
+for page in reader.pages:
+    text += page.extract_text()
+```
+
+## Python Libraries
+
+### PyMuPDF (fitz) - Text Search & Replace
+
+**This is the ONLY method for replacing text in existing PDFs. Do NOT fall back to nano-pdf, pdftotext, tesseract, or any external CLI tool — use PyMuPDF exclusively.**
+
+**CRITICAL FONT RULES:**
+- NEVER use the original PDF's embedded font names (e.g. `Unnamed-T3`, `NotoSansCJKsc-Regular`, `LiberationSans`) with `insert_text()` — they are embedded fonts and will cause `Exception: need font file or buffer`.
+- Always use `page.search_for()` to locate text (it works across spans, unlike iterating `get_text("dict")` spans which may split multi-word terms).
+- Use the **fallback font picker** below to choose a suitable built-in font.
+
+**Built-in font aliases (Latin):** `helv` (Helvetica), `heit` (Helvetica-Bold), `tiro` (Times-Roman), `tibo` (Times-Bold), `cour` (Courier), `cobo` (Courier-Bold).
+
+**Built-in CJK font aliases:** `china-s` (Simplified Chinese), `china-t` (Traditional Chinese), `japan` (Japanese), `korea` (Korean). These also render Latin characters correctly.
+
+#### Find & Replace Text in PDF (Complete Example)
+
+```python
+import fitz  # PyMuPDF
+
+# --- Font fallback logic ---
+BUILTIN_FONTS = {"helv","heit","tiro","tibo","tibi","toit","cour","cobo","cobi","cout","symb","zadb"}
+CJK_FONTS = {"china-s","china-ss","china-t","china-ts","japan","japan-s","korea","korea-s"}
+
+def has_cjk(text):
+    """Check if text contains CJK characters."""
+    return any("\u4e00" <= c <= "\u9fff" or "\u3000" <= c <= "\u303f" for c in text)
+
+def pick_font(original_font, text):
+    """Pick a built-in font. Falls back from the original embedded font name."""
+    if original_font in BUILTIN_FONTS or original_font in CJK_FONTS:
+        return original_font
+    if has_cjk(text):
+        return "china-s"  # handles both CJK and Latin
+    lower = original_font.lower()
+    if "bold" in lower:
+        return "heit"
+    if "mono" in lower or "courier" in lower or "code" in lower:
+        return "cour"
+    return "helv"
+
+def get_span_info(page, rect):
+    """Get font size, font name, and color from the span overlapping rect."""
+    for block in page.get_text("dict")["blocks"]:
+        if "lines" not in block:
+            continue
+        for line in block["lines"]:
+            for span in line["spans"]:
+                if fitz.Rect(span["bbox"]).intersects(rect):
+                    return span["size"], span["font"], span.get("color", 0)
+    return 12, "helv", 0
+
+# --- Main replacement logic ---
+search_terms = ["OldText", "Old Text"]  # all variants to search for
+replacement  = "NewText"
+
+doc = fitz.open("input.pdf")
+
+for page in doc:
+    # Step 1: Find all instances using search_for (works across spans)
+    hits = []
+    for term in search_terms:
+        for rect in page.search_for(term):
+            size, font, color = get_span_info(page, rect)
+            hits.append({"rect": rect, "size": size, "font": font, "color": color})
+
+    # Step 2: Redact (erase) old text
+    for h in hits:
+        page.add_redact_annot(h["rect"], fill=(1, 1, 1))
+    page.apply_redactions()
+
+    # Step 3: Insert replacement text with fallback font
+    for h in hits:
+        font = pick_font(h["font"], replacement)
+        c = h["color"]
+        rgb = (((c>>16)&0xFF)/255, ((c>>8)&0xFF)/255, (c&0xFF)/255) if isinstance(c, int) else (0,0,0)
+        page.insert_text(
+            (h["rect"].x0, h["rect"].y0 + h["size"] * 0.85),
+            replacement,
+            fontname=font,
+            fontsize=h["size"],
+            color=rgb,
+        )
+
+doc.save("output.pdf", garbage=4, deflate=True)
+```
+
+**After replacement, always verify** by extracting text from the output PDF:
+
+```python
+doc2 = fitz.open("output.pdf")
+for i, p in enumerate(doc2):
+    text = p.get_text()
+    old_count = text.lower().count("oldtext")
+    new_count = text.lower().count("newtext")
+    print(f"Page {i+1}: OldText={old_count}, NewText={new_count}")
+```
+
+**Notes:**
+- PDF text replacement is inherently imperfect — PDFs are presentation-focused, not text-edit-focused.
+- The replacement font may look slightly different from the original embedded font. This is expected.
+- `china-s` is the best fallback for mixed CJK+Latin text; `helv` for Latin-only text.
+
+### PyMuPDF (fitz) - Image Operations
+
+**Use PyMuPDF for ALL image operations in existing PDFs.** Key APIs:
+- `page.insert_image(rect, filename=...)` — insert a new image
+- `page.replace_image(xref, filename=...)` — replace an existing image in-place (keeps original rect)
+- `page.delete_image(xref)` — remove an image (replaces with 1×1 transparent pixel)
+- `page.get_images(full=True)` — list images on a page (returns xref, etc.)
+- `page.get_image_info(xrefs=True)` — list images with bounding box info
+
+**CRITICAL: Always use `page.get_image_info(xrefs=True)` (not `get_images`) when you need the on-page bounding box of each image.**
+
+#### Insert Image into Existing PDF
+
+```python
+import fitz
+
+doc = fitz.open("input.pdf")
+page = doc[0]
+
+# Define target rectangle (x0, y0, x1, y1) — image will be scaled to fit
+# To center horizontally with 50% page width:
+pw = page.rect.width
+img_w = pw * 0.5
+# Load image to get aspect ratio
+img = fitz.Pixmap("image.png")
+aspect = img.width / img.height
+img_h = img_w / aspect
+x0 = (pw - img_w) / 2
+y0 = 700  # vertical position
+
+rect = fitz.Rect(x0, y0, x0 + img_w, y0 + img_h)
+page.insert_image(rect, filename="image.png")
+doc.save("output.pdf")
+```
+
+#### Replace an Existing Image
+
+```python
+import fitz
+
+doc = fitz.open("input.pdf")
+page = doc[0]
+
+# Find the image to replace
+images = page.get_images(full=True)
+target_xref = images[0][0]  # xref of the first image
+
+# Replace it — the new image fills the SAME rect as the original
+page.replace_image(target_xref, filename="new_image.png")
+doc.save("output.pdf")
+```
+
+**Note:** `replace_image` keeps the original bounding box. The new image is scaled to fit.
+
+#### Modify Image Layout (Reposition / Resize)
+
+```python
+import fitz
+
+doc = fitz.open("input.pdf")
+page = doc[0]
+
+# Get image placements with bounding boxes
+img_info = page.get_image_info(xrefs=True)
+target = img_info[0]  # first image
+xref = target["xref"]
+orig_rect = fitz.Rect(target["bbox"])
+
+# Extract original image bytes
+pix = fitz.Pixmap(doc, xref)
+img_bytes = pix.tobytes("png")
+
+# Delete the original (replaces with 1x1 transparent pixel)
+page.delete_image(xref)
+
+# Re-insert at new position/size (e.g. 70% size, centered)
+pw = page.rect.width
+new_w = orig_rect.width * 0.7
+new_h = orig_rect.height * 0.7
+new_x0 = (pw - new_w) / 2
+new_rect = fitz.Rect(new_x0, orig_rect.y0, new_x0 + new_w, orig_rect.y0 + new_h)
+page.insert_image(new_rect, stream=img_bytes)
+
+doc.save("output.pdf", garbage=4, deflate=True)
+```
+
+**Notes on image operations:**
+- `delete_image` replaces the image data with a 1×1 transparent pixel (the page reference remains but is invisible). This is normal PyMuPDF behavior.
+- For layout changes: extract → delete → re-insert at new rect. Do NOT try `doc.update_image()` (does not exist).
+- `insert_image` accepts `filename=` (file path) or `stream=` (bytes). Use `stream=` when re-inserting extracted images.
+- Use `garbage=4, deflate=True` in `doc.save()` to clean up unused objects and compress.
+
+### pypdf - Basic Operations
+
+#### Merge PDFs
+
+```python
+from pypdf import PdfWriter, PdfReader
+
+writer = PdfWriter()
+for pdf_file in ["doc1.pdf", "doc2.pdf", "doc3.pdf"]:
+    reader = PdfReader(pdf_file)
+    for page in reader.pages:
+        writer.add_page(page)
+
+with open("merged.pdf", "wb") as output:
+    writer.write(output)
+```
+
+#### Split PDF
+
+```python
+reader = PdfReader("input.pdf")
+for i, page in enumerate(reader.pages):
+    writer = PdfWriter()
+    writer.add_page(page)
+    with open(f"page_{i+1}.pdf", "wb") as output:
+        writer.write(output)
+```
+
+#### Extract Metadata
+
+```python
+reader = PdfReader("document.pdf")
+meta = reader.metadata
+print(f"Title: {meta.title}")
+print(f"Author: {meta.author}")
+print(f"Subject: {meta.subject}")
+print(f"Creator: {meta.creator}")
+```
+
+#### Rotate Pages
+
+```python
+reader = PdfReader("input.pdf")
+writer = PdfWriter()
+
+page = reader.pages[0]
+page.rotate(90)  # Rotate 90 degrees clockwise
+writer.add_page(page)
+
+with open("rotated.pdf", "wb") as output:
+    writer.write(output)
+```
+
+### pdfplumber - Text and Table Extraction
+
+#### Extract Text with Layout
+
+```python
+import pdfplumber
+
+with pdfplumber.open("document.pdf") as pdf:
+    for page in pdf.pages:
+        text = page.extract_text()
+        print(text)
+```
+
+#### Extract Tables
+
+```python
+with pdfplumber.open("document.pdf") as pdf:
+    for i, page in enumerate(pdf.pages):
+        tables = page.extract_tables()
+        for j, table in enumerate(tables):
+            print(f"Table {j+1} on page {i+1}:")
+            for row in table:
+                print(row)
+```
+
+#### Advanced Table Extraction
+
+```python
+import pandas as pd
+
+with pdfplumber.open("document.pdf") as pdf:
+    all_tables = []
+    for page in pdf.pages:
+        tables = page.extract_tables()
+        for table in tables:
+            if table:  # Check if table is not empty
+                df = pd.DataFrame(table[1:], columns=table[0])
+                all_tables.append(df)
+
+# Combine all tables
+if all_tables:
+    combined_df = pd.concat(all_tables, ignore_index=True)
+    combined_df.to_excel("extracted_tables.xlsx", index=False)
+```
+
+### reportlab - Create PDFs
+
+> ⚠️ **If the PDF contains ANY Chinese/Japanese/Korean text, you MUST follow the CJK rules below. Skipping them causes black boxes (█) or crash errors.**
+
+**FORBIDDEN patterns (these ALL break CJK rendering):**
+- ❌ Using `Helvetica`, `Times-Roman`, or `Courier` fonts for CJK text → renders as **black boxes (█)**
+- ❌ Registering `STSong-Light` without calling `addMapping()` → `Paragraph()` crashes with `ValueError: Can't map determine family/bold/italic for stsong-light`
+- ❌ Using `<font face="STSong-Light">` in Paragraph XML without `addMapping()` → same crash
+- ❌ Setting `style.fontName = "STSong-Light"` without `addMapping()` → same crash
+
+**CJK font setup (MUST be at the top of EVERY script with CJK text):**
+
+```python
+from reportlab.pdfbase import pdfmetrics
+from reportlab.pdfbase.cidfonts import UnicodeCIDFont
+from reportlab.lib.fonts import addMapping
+
+# Step 1: Register the CJK font
+pdfmetrics.registerFont(UnicodeCIDFont("STSong-Light"))
+# Step 2: MANDATORY — register font family mapping (without this Paragraph() crashes)
+addMapping("STSong-Light", 0, 0, "STSong-Light")  # normal
+addMapping("STSong-Light", 1, 0, "STSong-Light")  # bold
+addMapping("STSong-Light", 0, 1, "STSong-Light")  # italic
+addMapping("STSong-Light", 1, 1, "STSong-Light")  # bold+italic
+```
+
+**After setup, use `"STSong-L