calvyn-code 0.14.0

package/skills/productivity/ocr-and-documents/SKILL.md

@@ -0,0 +1,172 @@
+---
+name: ocr-and-documents
+description: "Extract text from PDFs/scans (pymupdf, marker-pdf)."
+version: 2.3.0
+author: Hermes Agent
+license: MIT
+platforms: [linux, macos, windows]
+metadata:
+  hermes:
+    tags: [PDF, Documents, Research, Arxiv, Text-Extraction, OCR]
+    related_skills: [powerpoint]
+---
+
+# PDF & Document Extraction
+
+For DOCX: use `python-docx` (parses actual document structure, far better than OCR).
+For PPTX: see the `powerpoint` skill (uses `python-pptx` with full slide/notes support).
+This skill covers **PDFs and scanned documents**.
+
+## Step 1: Remote URL Available?
+
+If the document has a URL, **always try `web_extract` first**:
+
+```
+web_extract(urls=["https://arxiv.org/pdf/2402.03300"])
+web_extract(urls=["https://example.com/report.pdf"])
+```
+
+This handles PDF-to-markdown conversion via Firecrawl with no local dependencies.
+
+Only use local extraction when: the file is local, web_extract fails, or you need batch processing.
+
+## Step 2: Choose Local Extractor
+
+| Feature | pymupdf (~25MB) | marker-pdf (~3-5GB) |
+|---------|-----------------|---------------------|
+| **Text-based PDF** | ✅ | ✅ |
+| **Scanned PDF (OCR)** | ❌ | ✅ (90+ languages) |
+| **Tables** | ✅ (basic) | ✅ (high accuracy) |
+| **Equations / LaTeX** | ❌ | ✅ |
+| **Code blocks** | ❌ | ✅ |
+| **Forms** | ❌ | ✅ |
+| **Headers/footers removal** | ❌ | ✅ |
+| **Reading order detection** | ❌ | ✅ |
+| **Images extraction** | ✅ (embedded) | ✅ (with context) |
+| **Images → text (OCR)** | ❌ | ✅ |
+| **EPUB** | ✅ | ✅ |
+| **Markdown output** | ✅ (via pymupdf4llm) | ✅ (native, higher quality) |
+| **Install size** | ~25MB | ~3-5GB (PyTorch + models) |
+| **Speed** | Instant | ~1-14s/page (CPU), ~0.2s/page (GPU) |
+
+**Decision**: Use pymupdf unless you need OCR, equations, forms, or complex layout analysis.
+
+If the user needs marker capabilities but the system lacks ~5GB free disk:
+> "This document needs OCR/advanced extraction (marker-pdf), which requires ~5GB for PyTorch and models. Your system has [X]GB free. Options: free up space, provide a URL so I can use web_extract, or I can try pymupdf which works for text-based PDFs but not scanned documents or equations."
+
+---
+
+## pymupdf (lightweight)
+
+```bash
+pip install pymupdf pymupdf4llm
+```
+
+**Via helper script**:
+```bash
+python scripts/extract_pymupdf.py document.pdf              # Plain text
+python scripts/extract_pymupdf.py document.pdf --markdown    # Markdown
+python scripts/extract_pymupdf.py document.pdf --tables      # Tables
+python scripts/extract_pymupdf.py document.pdf --images out/ # Extract images
+python scripts/extract_pymupdf.py document.pdf --metadata    # Title, author, pages
+python scripts/extract_pymupdf.py document.pdf --pages 0-4   # Specific pages
+```
+
+**Inline**:
+```bash
+python3 -c "
+import pymupdf
+doc = pymupdf.open('document.pdf')
+for page in doc:
+    print(page.get_text())
+"
+```
+
+---
+
+## marker-pdf (high-quality OCR)
+
+```bash
+# Check disk space first
+python scripts/extract_marker.py --check
+
+pip install marker-pdf
+```
+
+**Via helper script**:
+```bash
+python scripts/extract_marker.py document.pdf                # Markdown
+python scripts/extract_marker.py document.pdf --json         # JSON with metadata
+python scripts/extract_marker.py document.pdf --output_dir out/  # Save images
+python scripts/extract_marker.py scanned.pdf                 # Scanned PDF (OCR)
+python scripts/extract_marker.py document.pdf --use_llm      # LLM-boosted accuracy
+```
+
+**CLI** (installed with marker-pdf):
+```bash
+marker_single document.pdf --output_dir ./output
+marker /path/to/folder --workers 4    # Batch
+```
+
+---
+
+## Arxiv Papers
+
+```
+# Abstract only (fast)
+web_extract(urls=["https://arxiv.org/abs/2402.03300"])
+
+# Full paper
+web_extract(urls=["https://arxiv.org/pdf/2402.03300"])
+
+# Search
+web_search(query="arxiv GRPO reinforcement learning 2026")
+```
+
+## Split, Merge & Search
+
+pymupdf handles these natively — use `execute_code` or inline Python:
+
+```python
+# Split: extract pages 1-5 to a new PDF
+import pymupdf
+doc = pymupdf.open("report.pdf")
+new = pymupdf.open()
+for i in range(5):
+    new.insert_pdf(doc, from_page=i, to_page=i)
+new.save("pages_1-5.pdf")
+```
+
+```python
+# Merge multiple PDFs
+import pymupdf
+result = pymupdf.open()
+for path in ["a.pdf", "b.pdf", "c.pdf"]:
+    result.insert_pdf(pymupdf.open(path))
+result.save("merged.pdf")
+```
+
+```python
+# Search for text across all pages
+import pymupdf
+doc = pymupdf.open("report.pdf")
+for i, page in enumerate(doc):
+    results = page.search_for("revenue")
+    if results:
+        print(f"Page {i+1}: {len(results)} match(es)")
+        print(page.get_text("text"))
+```
+
+No extra dependencies needed — pymupdf covers split, merge, search, and text extraction in one package.
+
+---
+
+## Notes
+
+- `web_extract` is always first choice for URLs
+- pymupdf is the safe default — instant, no models, works everywhere
+- marker-pdf is for OCR, scanned docs, equations, complex layouts — install only when needed
+- Both helper scripts accept `--help` for full usage
+- marker-pdf downloads ~2.5GB of models to `~/.cache/huggingface/` on first use
+- For Word docs: `pip install python-docx` (better than OCR — parses actual structure)
+- For PowerPoint: see the `powerpoint` skill (uses python-pptx)

package/skills/productivity/ocr-and-documents/scripts/extract_marker.py

@@ -0,0 +1,87 @@
+#!/usr/bin/env python3
+"""Extract text from documents using marker-pdf. High-quality OCR + layout analysis.
+
+Requires ~3-5GB disk (PyTorch + models downloaded on first use).
+Supports: PDF, DOCX, PPTX, XLSX, HTML, EPUB, images.
+
+Usage:
+    python extract_marker.py document.pdf
+    python extract_marker.py document.pdf --output_dir ./output
+    python extract_marker.py presentation.pptx
+    python extract_marker.py spreadsheet.xlsx
+    python extract_marker.py scanned_doc.pdf           # OCR works here
+    python extract_marker.py document.pdf --json        # Structured output
+    python extract_marker.py document.pdf --use_llm     # LLM-boosted accuracy
+"""
+import sys
+import os
+
+def convert(path, output_dir=None, output_format="markdown", use_llm=False):
+    from marker.converters.pdf import PdfConverter
+    from marker.models import create_model_dict
+    from marker.config.parser import ConfigParser
+
+    config_dict = {}
+    if use_llm:
+        config_dict["use_llm"] = True
+
+    config_parser = ConfigParser(config_dict)
+    models = create_model_dict()
+    converter = PdfConverter(config=config_parser.generate_config_dict(), artifact_dict=models)
+    rendered = converter(path)
+
+    if output_format == "json":
+        import json
+        print(json.dumps({
+            "markdown": rendered.markdown,
+            "metadata": rendered.metadata if hasattr(rendered, "metadata") else {},
+        }, indent=2, ensure_ascii=False))
+    else:
+        print(rendered.markdown)
+
+    # Save images if output_dir specified
+    if output_dir and hasattr(rendered, "images") and rendered.images:
+        from pathlib import Path
+        Path(output_dir).mkdir(parents=True, exist_ok=True)
+        for name, img_data in rendered.images.items():
+            img_path = os.path.join(output_dir, name)
+            with open(img_path, "wb") as f:
+                f.write(img_data)
+        print(f"\nSaved {len(rendered.images)} image(s) to {output_dir}/", file=sys.stderr)
+
+
+def check_requirements():
+    """Check disk space before installing."""
+    import shutil
+    free_gb = shutil.disk_usage("/").free / (1024**3)
+    if free_gb < 5:
+        print(f"⚠️  Only {free_gb:.1f}GB free. marker-pdf needs ~5GB for PyTorch + models.")
+        print("Use pymupdf instead (scripts/extract_pymupdf.py) or free up disk space.")
+        sys.exit(1)
+    print(f"✓ {free_gb:.1f}GB free — sufficient for marker-pdf")
+
+
+if __name__ == "__main__":
+    args = sys.argv[1:]
+    if not args or args[0] in ("-h", "--help"):
+        print(__doc__)
+        sys.exit(0)
+
+    if args[0] == "--check":
+        check_requirements()
+        sys.exit(0)
+
+    path = args[0]
+    output_dir = None
+    output_format = "markdown"
+    use_llm = False
+
+    if "--output_dir" in args:
+        idx = args.index("--output_dir")
+        output_dir = args[idx + 1]
+    if "--json" in args:
+        output_format = "json"
+    if "--use_llm" in args:
+        use_llm = True
+
+    convert(path, output_dir=output_dir, output_format=output_format, use_llm=use_llm)

package/skills/productivity/ocr-and-documents/scripts/extract_pymupdf.py

@@ -0,0 +1,98 @@
+#!/usr/bin/env python3
+"""Extract text from documents using pymupdf. Lightweight (~25MB), no models.
+
+Usage:
+    python extract_pymupdf.py document.pdf
+    python extract_pymupdf.py document.pdf --markdown
+    python extract_pymupdf.py document.pdf --pages 0-4
+    python extract_pymupdf.py document.pdf --images output_dir/
+    python extract_pymupdf.py document.pdf --tables
+    python extract_pymupdf.py document.pdf --metadata
+"""
+import sys
+import json
+
+def extract_text(path, pages=None):
+    import pymupdf
+    doc = pymupdf.open(path)
+    page_range = range(len(doc)) if pages is None else pages
+    for i in page_range:
+        if i < len(doc):
+            print(f"\n--- Page {i+1}/{len(doc)} ---\n")
+            print(doc[i].get_text())
+
+def extract_markdown(path, pages=None):
+    import pymupdf4llm
+    md = pymupdf4llm.to_markdown(path, pages=pages)
+    print(md)
+
+def extract_tables(path):
+    import pymupdf
+    doc = pymupdf.open(path)
+    for i, page in enumerate(doc):
+        tables = page.find_tables()
+        for j, table in enumerate(tables.tables):
+            print(f"\n--- Page {i+1}, Table {j+1} ---\n")
+            df = table.to_pandas()
+            print(df.to_markdown(index=False))
+
+def extract_images(path, output_dir):
+    import pymupdf
+    from pathlib import Path
+    Path(output_dir).mkdir(parents=True, exist_ok=True)
+    doc = pymupdf.open(path)
+    count = 0
+    for i, page in enumerate(doc):
+        for img_idx, img in enumerate(page.get_images(full=True)):
+            xref = img[0]
+            pix = pymupdf.Pixmap(doc, xref)
+            if pix.n >= 5:
+                pix = pymupdf.Pixmap(pymupdf.csRGB, pix)
+            out_path = f"{output_dir}/page{i+1}_img{img_idx+1}.png"
+            pix.save(out_path)
+            count += 1
+    print(f"Extracted {count} images to {output_dir}/")
+
+def show_metadata(path):
+    import pymupdf
+    doc = pymupdf.open(path)
+    print(json.dumps({
+        "pages": len(doc),
+        "title": doc.metadata.get("title", ""),
+        "author": doc.metadata.get("author", ""),
+        "subject": doc.metadata.get("subject", ""),
+        "creator": doc.metadata.get("creator", ""),
+        "producer": doc.metadata.get("producer", ""),
+        "format": doc.metadata.get("format", ""),
+    }, indent=2))
+
+if __name__ == "__main__":
+    args = sys.argv[1:]
+    if not args or args[0] in ("-h", "--help"):
+        print(__doc__)
+        sys.exit(0)
+
+    path = args[0]
+    pages = None
+
+    if "--pages" in args:
+        idx = args.index("--pages")
+        p = args[idx + 1]
+        if "-" in p:
+            start, end = p.split("-")
+            pages = list(range(int(start), int(end) + 1))
+        else:
+            pages = [int(p)]
+
+    if "--metadata" in args:
+        show_metadata(path)
+    elif "--tables" in args:
+        extract_tables(path)
+    elif "--images" in args:
+        idx = args.index("--images")
+        output_dir = args[idx + 1] if idx + 1 < len(args) else "./images"
+        extract_images(path, output_dir)
+    elif "--markdown" in args:
+        extract_markdown(path, pages=pages)
+    else:
+        extract_text(path, pages=pages)

package/skills/productivity/powerpoint/LICENSE.txt

@@ -0,0 +1,30 @@
+© 2025 Anthropic, PBC. All rights reserved.
+
+LICENSE: Use of these materials (including all code, prompts, assets, files,
+and other components of this Skill) is governed by your agreement with
+Anthropic regarding use of Anthropic's services. If no separate agreement
+exists, use is governed by Anthropic's Consumer Terms of Service or
+Commercial Terms of Service, as applicable:
+https://www.anthropic.com/legal/consumer-terms
+https://www.anthropic.com/legal/commercial-terms
+Your applicable agreement is referred to as the "Agreement." "Services" are
+as defined in the Agreement.
+
+ADDITIONAL RESTRICTIONS: Notwithstanding anything in the Agreement to the
+contrary, users may not:
+
+- Extract these materials from the Services or retain copies of these
+  materials outside the Services
+- Reproduce or copy these materials, except for temporary copies created
+  automatically during authorized use of the Services
+- Create derivative works based on these materials
+- Distribute, sublicense, or transfer these materials to any third party
+- Make, offer to sell, sell, or import any inventions embodied in these
+  materials
+- Reverse engineer, decompile, or disassemble these materials
+
+The receipt, viewing, or possession of these materials does not convey or
+imply any license or right beyond those expressly granted above.
+
+Anthropic retains all right, title, and interest in these materials,
+including all copyrights, patents, and other intellectual property rights.

package/skills/productivity/powerpoint/SKILL.md

@@ -0,0 +1,237 @@
+---
+name: powerpoint
+description: "Create, read, edit .pptx decks, slides, notes, templates."
+license: Proprietary. LICENSE.txt has complete terms
+platforms: [linux, macos, windows]
+---
+
+# Powerpoint Skill
+
+## When to use
+
+Use this skill any time a .pptx file is involved in any way — as input, output, or both. This includes: creating slide decks, pitch decks, or presentations; reading, parsing, or extracting text from any .pptx file (even if the extracted content will be used elsewhere, like in an email or summary); editing, modifying, or updating existing presentations; combining or splitting slide files; working with templates, layouts, speaker notes, or comments. Trigger whenever the user mentions "deck," "slides," "presentation," or references a .pptx filename, regardless of what they plan to do with the content afterward. If a .pptx file needs to be opened, created, or touched, use this skill.
+
+## Quick Reference
+
+| Task | Guide |
+|------|-------|
+| Read/analyze content | `python -m markitdown presentation.pptx` |
+| Edit or create from template | Read [editing.md](editing.md) |
+| Create from scratch | Read [pptxgenjs.md](pptxgenjs.md) |
+
+---
+
+## Reading Content
+
+```bash
+# Text extraction
+python -m markitdown presentation.pptx
+
+# Visual overview
+python scripts/thumbnail.py presentation.pptx
+
+# Raw XML
+python scripts/office/unpack.py presentation.pptx unpacked/
+```
+
+---
+
+## Editing Workflow
+
+**Read [editing.md](editing.md) for full details.**
+
+1. Analyze template with `thumbnail.py`
+2. Unpack → manipulate slides → edit content → clean → pack
+
+---
+
+## Creating from Scratch
+
+**Read [pptxgenjs.md](pptxgenjs.md) for full details.**
+
+Use when no template or reference presentation is available.
+
+---
+
+## Design Ideas
+
+**Don't create boring slides.** Plain bullets on a white background won't impress anyone. Consider ideas from this list for each slide.
+
+### Before Starting
+
+- **Pick a bold, content-informed color palette**: The palette should feel designed for THIS topic. If swapping your colors into a completely different presentation would still "work," you haven't made specific enough choices.
+- **Dominance over equality**: One color should dominate (60-70% visual weight), with 1-2 supporting tones and one sharp accent. Never give all colors equal weight.
+- **Dark/light contrast**: Dark backgrounds for title + conclusion slides, light for content ("sandwich" structure). Or commit to dark throughout for a premium feel.
+- **Commit to a visual motif**: Pick ONE distinctive element and repeat it — rounded image frames, icons in colored circles, thick single-side borders. Carry it across every slide.
+
+### Color Palettes
+
+Choose colors that match your topic — don't default to generic blue. Use these palettes as inspiration:
+
+| Theme | Primary | Secondary | Accent |
+|-------|---------|-----------|--------|
+| **Midnight Executive** | `1E2761` (navy) | `CADCFC` (ice blue) | `FFFFFF` (white) |
+| **Forest & Moss** | `2C5F2D` (forest) | `97BC62` (moss) | `F5F5F5` (cream) |
+| **Coral Energy** | `F96167` (coral) | `F9E795` (gold) | `2F3C7E` (navy) |
+| **Warm Terracotta** | `B85042` (terracotta) | `E7E8D1` (sand) | `A7BEAE` (sage) |
+| **Ocean Gradient** | `065A82` (deep blue) | `1C7293` (teal) | `21295C` (midnight) |
+| **Charcoal Minimal** | `36454F` (charcoal) | `F2F2F2` (off-white) | `212121` (black) |
+| **Teal Trust** | `028090` (teal) | `00A896` (seafoam) | `02C39A` (mint) |
+| **Berry & Cream** | `6D2E46` (berry) | `A26769` (dusty rose) | `ECE2D0` (cream) |
+| **Sage Calm** | `84B59F` (sage) | `69A297` (eucalyptus) | `50808E` (slate) |
+| **Cherry Bold** | `990011` (cherry) | `FCF6F5` (off-white) | `2F3C7E` (navy) |
+
+### For Each Slide
+
+**Every slide needs a visual element** — image, chart, icon, or shape. Text-only slides are forgettable.
+
+**Layout options:**
+- Two-column (text left, illustration on right)
+- Icon + text rows (icon in colored circle, bold header, description below)
+- 2x2 or 2x3 grid (image on one side, grid of content blocks on other)
+- Half-bleed image (full left or right side) with content overlay
+
+**Data display:**
+- Large stat callouts (big numbers 60-72pt with small labels below)
+- Comparison columns (before/after, pros/cons, side-by-side options)
+- Timeline or process flow (numbered steps, arrows)
+
+**Visual polish:**
+- Icons in small colored circles next to section headers
+- Italic accent text for key stats or taglines
+
+### Typography
+
+**Choose an interesting font pairing** — don't default to Arial. Pick a header font with personality and pair it with a clean body font.
+
+| Header Font | Body Font |
+|-------------|-----------|
+| Georgia | Calibri |
+| Arial Black | Arial |
+| Calibri | Calibri Light |
+| Cambria | Calibri |
+| Trebuchet MS | Calibri |
+| Impact | Arial |
+| Palatino | Garamond |
+| Consolas | Calibri |
+
+| Element | Size |
+|---------|------|
+| Slide title | 36-44pt bold |
+| Section header | 20-24pt bold |
+| Body text | 14-16pt |
+| Captions | 10-12pt muted |
+
+### Spacing
+
+- 0.5" minimum margins
+- 0.3-0.5" between content blocks
+- Leave breathing room—don't fill every inch
+
+### Avoid (Common Mistakes)
+
+- **Don't repeat the same layout** — vary columns, cards, and callouts across slides
+- **Don't center body text** — left-align paragraphs and lists; center only titles
+- **Don't skimp on size contrast** — titles need 36pt+ to stand out from 14-16pt body
+- **Don't default to blue** — pick colors that reflect the specific topic
+- **Don't mix spacing randomly** — choose 0.3" or 0.5" gaps and use consistently
+- **Don't style one slide and leave the rest plain** — commit fully or keep it simple throughout
+- **Don't create text-only slides** — add images, icons, charts, or visual elements; avoid plain title + bullets
+- **Don't forget text box padding** — when aligning lines or shapes with text edges, set `margin: 0` on the text box or offset the shape to account for padding
+- **Don't use low-contrast elements** — icons AND text need strong contrast against the background; avoid light text on light backgrounds or dark text on dark backgrounds
+- **NEVER use accent lines under titles** — these are a hallmark of AI-generated slides; use whitespace or background color instead
+
+---
+
+## QA (Required)
+
+**Assume there are problems. Your job is to find them.**
+
+Your first render is almost never correct. Approach QA as a bug hunt, not a confirmation step. If you found zero issues on first inspection, you weren't looking hard enough.
+
+### Content QA
+
+```bash
+python -m markitdown output.pptx
+```
+
+Check for missing content, typos, wrong order.
+
+**When using templates, check for leftover placeholder text:**
+
+```bash
+python -m markitdown output.pptx | grep -iE "xxxx|lorem|ipsum|this.*(page|slide).*layout"
+```
+
+If grep returns results, fix them before declaring success.
+
+### Visual QA
+
+**⚠️ USE SUBAGENTS** — even for 2-3 slides. You've been staring at the code and will see what you expect, not what's there. Subagents have fresh eyes.
+
+Convert slides to images (see [Converting to Images](#converting-to-images)), then use this prompt:
+
+```
+Visually inspect these slides. Assume there are issues — find them.
+
+Look for:
+- Overlapping elements (text through shapes, lines through words, stacked elements)
+- Text overflow or cut off at edges/box boundaries
+- Decorative lines positioned for single-line text but title wrapped to two lines
+- Source citations or footers colliding with content above
+- Elements too close (< 0.3" gaps) or cards/sections nearly touching
+- Uneven gaps (large empty area in one place, cramped in another)
+- Insufficient margin from slide edges (< 0.5")
+- Columns or similar elements not aligned consistently
+- Low-contrast text (e.g., light gray text on cream-colored background)
+- Low-contrast icons (e.g., dark icons on dark backgrounds without a contrasting circle)
+- Text boxes too narrow causing excessive wrapping
+- Leftover placeholder content
+
+For each slide, list issues or areas of concern, even if minor.
+
+Read and analyze these images:
+1. /path/to/slide-01.jpg (Expected: [brief description])
+2. /path/to/slide-02.jpg (Expected: [brief description])
+
+Report ALL issues found, including minor ones.
+```
+
+### Verification Loop
+
+1. Generate slides → Convert to images → Inspect
+2. **List issues found** (if none found, look again more critically)
+3. Fix issues
+4. **Re-verify affected slides** — one fix often creates another problem
+5. Repeat until a full pass reveals no new issues
+
+**Do not declare success until you've completed at least one fix-and-verify cycle.**
+
+---
+
+## Converting to Images
+
+Convert presentations to individual slide images for visual inspection:
+
+```bash
+python scripts/office/soffice.py --headless --convert-to pdf output.pptx
+pdftoppm -jpeg -r 150 output.pdf slide
+```
+
+This creates `slide-01.jpg`, `slide-02.jpg`, etc.
+
+To re-render specific slides after fixes:
+
+```bash
+pdftoppm -jpeg -r 150 -f N -l N output.pdf slide-fixed
+```
+
+---
+
+## Dependencies
+
+- `pip install "markitdown[pptx]"` - text extraction
+- `pip install Pillow` - thumbnail grids
+- `npm install -g pptxgenjs` - creating from scratch
+- LibreOffice (`soffice`) - PDF conversion (auto-configured for sandboxed environments via `scripts/office/soffice.py`)
+- Poppler (`pdftoppm`) - PDF to images