@wentorai/research-plugins 1.0.0 → 1.2.0

package/skills/tools/document/large-document-reader/SKILL.md

@@ -0,0 +1,202 @@
+---
+name: large-document-reader
+description: "Split and read long documents chapter-by-chapter for structured analysis"
+metadata:
+  openclaw:
+    emoji: "book"
+    category: "tools"
+    subcategory: "document"
+    keywords: ["document reading", "chunking", "long document", "chapter splitting", "structured reading"]
+    source: "https://github.com/wentor-ai/research-plugins"
+---
+
+# Large Document Reader
+
+Split long documents (books, reports, theses, legal filings, technical manuals) into structured chapters or sections for systematic, chapter-by-chapter reading and analysis within LLM context windows.
+
+## Overview
+
+Large Language Models have finite context windows, and even models with 100K+ token limits can lose accuracy on information buried in the middle of very long inputs. Academic researchers frequently work with documents that exceed practical context limits: doctoral theses (200+ pages), government reports, book-length monographs, legal case compilations, and multi-volume technical standards.
+
+This skill provides a systematic approach to splitting large documents into semantically meaningful chapters or sections, maintaining cross-references between parts, and reading each section with full comprehension. Rather than naive fixed-size chunking that breaks mid-sentence or mid-argument, this approach respects document structure -- headings, chapter breaks, section markers, and logical boundaries.
+
+The result is a structured reading experience where each chapter is analyzed in full context, summaries are maintained across sessions, and the reader can navigate directly to any section of interest. This is especially valuable for literature reviews, systematic reviews, and comprehensive document analysis tasks.
+
+## Document Splitting Strategy
+
+### Hierarchy of Split Points
+
+Documents should be split at the highest-level structural boundary that keeps each chunk within the target size:
+
+| Priority | Boundary Type | Markers |
+|----------|--------------|---------|
+| 1 | Part/Volume | `PART I`, `Volume 2`, page breaks with Roman numerals |
+| 2 | Chapter | `Chapter 1`, `CHAPTER`, numbered headings level 1 |
+| 3 | Section | `1.1`, `Section`, headings level 2 |
+| 4 | Subsection | `1.1.1`, headings level 3 |
+| 5 | Paragraph break | Double newline, indentation change |
+| 6 | Sentence boundary | Period + space + capital letter |
+
+### Splitting Algorithm
+
+```python
+def split_document(text, max_tokens=8000, overlap_tokens=200):
+    """Split document respecting structural boundaries."""
+    # Step 1: Detect document structure
+    chapters = detect_chapters(text)
+
+    if not chapters:
+        # Fallback: split by sections
+        chapters = detect_sections(text)
+
+    if not chapters:
+        # Fallback: split by paragraphs with size limit
+        chapters = split_by_paragraphs(text, max_tokens)
+
+    # Step 2: Merge small adjacent sections
+    merged = merge_small_sections(chapters, min_tokens=500)
+
+    # Step 3: Split oversized sections
+    final = []
+    for chapter in merged:
+        if count_tokens(chapter.text) > max_tokens:
+            sub_parts = split_by_paragraphs(chapter.text, max_tokens)
+            for i, part in enumerate(sub_parts):
+                final.append(Section(
+                    title=f"{chapter.title} (Part {i+1})",
+                    text=part,
+                    index=len(final)
+                ))
+        else:
+            chapter.index = len(final)
+            final.append(chapter)
+
+    # Step 4: Add overlap for continuity
+    for i in range(1, len(final)):
+        final[i].context_prefix = get_last_n_tokens(
+            final[i-1].text, overlap_tokens
+        )
+
+    return final
+```
+
+### Structure Detection Patterns
+
+```python
+import re
+
+CHAPTER_PATTERNS = [
+    r'^#{1,2}\s+.+',                          # Markdown H1/H2
+    r'^Chapter\s+\d+',                         # "Chapter 1"
+    r'^\d+\.\s+[A-Z]',                        # "1. Introduction"
+    r'^PART\s+[IVX]+',                         # "PART III"
+    r'^\\(chapter|section)\{',                 # LaTeX commands
+    r'^\f',                                    # Form feed (page break)
+]
+
+def detect_chapters(text):
+    sections = []
+    current_title = "Preamble"
+    current_start = 0
+
+    for match in re.finditer('|'.join(CHAPTER_PATTERNS), text, re.MULTILINE):
+        if match.start() > current_start:
+            sections.append(Section(
+                title=current_title,
+                text=text[current_start:match.start()].strip()
+            ))
+        current_title = match.group().strip()
+        current_start = match.start()
+
+    sections.append(Section(title=current_title, text=text[current_start:].strip()))
+    return sections
+```
+
+## Structured Reading Workflow
+
+### Phase 1: Survey
+
+Read the table of contents, introduction, and conclusion first to build a mental model of the document's argument structure:
+
+```
+1. Extract and display Table of Contents
+2. Read Introduction (typically Chapter 1)
+3. Read Conclusion (typically last chapter)
+4. Generate a document map: chapter titles + estimated page counts
+5. Identify key themes and arguments
+```
+
+### Phase 2: Sequential Deep Reading
+
+Process each chapter with a standardized analysis template:
+
+```
+For each chapter:
+  - Chapter title and position in document
+  - Key arguments or findings (3-5 bullet points)
+  - Methodology described (if applicable)
+  - Data or evidence presented
+  - Connections to previous chapters
+  - Open questions or points for follow-up
+  - Notable quotes or passages (with page/section references)
+```
+
+### Phase 3: Synthesis
+
+After all chapters are read, generate cross-cutting analyses:
+
+```
+- Thematic summary across all chapters
+- Argument progression map
+- Methodology comparison (if multiple studies)
+- Contradiction or tension identification
+- Gap analysis relative to research questions
+```
+
+## Cross-Session Persistence
+
+For documents that take multiple sessions to read, maintain a reading state file:
+
+```json
+{
+  "document": "thesis_smith_2024.pdf",
+  "total_sections": 24,
+  "completed": [0, 1, 2, 3, 4, 5],
+  "current": 6,
+  "summaries": {
+    "0": "Preamble: Defines scope of study on...",
+    "1": "Chapter 1: Introduction to the problem of...",
+    "2": "Chapter 2: Literature review covering..."
+  },
+  "themes": ["data governance", "algorithmic fairness", "institutional trust"],
+  "open_questions": [
+    "How does the author reconcile findings in Ch3 with Ch5?"
+  ]
+}
+```
+
+## Format-Specific Handling
+
+| Format | Tool | Notes |
+|--------|------|-------|
+| PDF | `pdfplumber`, `PyMuPDF` | Extract text with layout awareness |
+| EPUB | `ebooklib` | Chapters are HTML files in the spine |
+| DOCX | `python-docx` | Headings define structure |
+| LaTeX | Regex on `\chapter`, `\section` | Native structure markers |
+| HTML | `BeautifulSoup` | Split on `<h1>`, `<h2>` tags |
+| Plain text | Heuristic detection | Use blank lines, indentation, page breaks |
+
+## Best Practices
+
+1. **Preserve cross-references**: When a chapter references "as discussed in Section 3.2," maintain a reference index so the reader can retrieve that section.
+2. **Maintain running context**: Each chunk should include a brief summary of preceding material (the overlap window) to maintain narrative continuity.
+3. **Respect tables and figures**: Never split in the middle of a table, code block, or figure caption. These should be kept as atomic units.
+4. **Index creation**: Build a searchable index of key terms, names, and concepts with section references for rapid lookup.
+5. **Citation extraction**: Pull out all references cited in each chapter to build a cumulative bibliography.
+
+## References
+
+- pdfplumber: https://github.com/jsvine/pdfplumber
+- python-docx: https://python-docx.readthedocs.io
+- ebooklib: https://github.com/aerkalov/ebooklib
+- PyMuPDF (fitz): https://pymupdf.readthedocs.io

package/skills/tools/document/md2pdf-xelatex/SKILL.md

@@ -0,0 +1,212 @@
+---
+name: md2pdf-xelatex
+description: "Convert Markdown to publication-ready PDF with LaTeX math and CJK support"
+metadata:
+  openclaw:
+    emoji: "📄"
+    category: "tools"
+    subcategory: "document"
+    keywords: ["markdown to pdf", "xelatex", "document conversion", "CJK support", "math rendering", "pandoc"]
+    source: "https://clawhub.ai/huaruoji/md2pdf-xelatex"
+---
+
+# Markdown to PDF Conversion with XeLaTeX
+
+## Overview
+
+Converting Markdown documents to publication-quality PDF is a common research workflow need. This guide covers using Pandoc with XeLaTeX as the PDF engine, which provides full Unicode support (including CJK characters), LaTeX math rendering, custom fonts, and professional typography. This is the recommended approach when your document contains mathematical formulas, non-Latin scripts, or requires precise typographic control.
+
+## Prerequisites
+
+### Installation
+
+```bash
+# macOS
+brew install pandoc
+brew install --cask mactex  # or: brew install --cask basictex (smaller)
+
+# Ubuntu/Debian
+sudo apt install pandoc texlive-xetex texlive-fonts-recommended texlive-lang-cjk
+
+# Windows (via Chocolatey)
+choco install pandoc miktex
+
+# Verify installation
+pandoc --version
+xelatex --version
+```
+
+### CJK Font Setup
+
+For Chinese/Japanese/Korean documents, install appropriate fonts:
+
+```bash
+# macOS — system fonts usually sufficient (PingFang, Hiragino)
+# Ubuntu — install Noto CJK
+sudo apt install fonts-noto-cjk fonts-noto-cjk-extra
+
+# Verify available CJK fonts
+fc-list :lang=zh family | sort | uniq
+```
+
+## Basic Conversion
+
+### Simple Markdown to PDF
+
+```bash
+# Default conversion (pdflatex engine)
+pandoc input.md -o output.pdf
+
+# With XeLaTeX engine (required for Unicode/CJK)
+pandoc input.md -o output.pdf --pdf-engine=xelatex
+
+# With custom font
+pandoc input.md -o output.pdf --pdf-engine=xelatex \
+  -V mainfont="Times New Roman" \
+  -V monofont="Fira Code"
+```
+
+### CJK Document
+
+```bash
+# Chinese document with proper font
+pandoc input.md -o output.pdf --pdf-engine=xelatex \
+  -V mainfont="Noto Serif CJK SC" \
+  -V CJKmainfont="Noto Serif CJK SC" \
+  -V monofont="Noto Sans Mono CJK SC" \
+  -V geometry:margin=2.5cm
+```
+
+### Document with Math
+
+Markdown with LaTeX math renders natively:
+
+```markdown
+The quadratic formula is:
+
+$$x = \frac{-b \pm \sqrt{b^2 - 4ac}}{2a}$$
+
+Inline math works too: $E = mc^2$.
+```
+
+```bash
+pandoc math_doc.md -o output.pdf --pdf-engine=xelatex
+```
+
+## Advanced Configuration
+
+### YAML Front Matter
+
+Add a YAML header to your Markdown for fine-grained control:
+
+```yaml
+---
+title: "Research Report Title"
+author: "Author Name"
+date: "2026-03-10"
+documentclass: article
+fontsize: 11pt
+geometry: margin=2.5cm
+mainfont: "Libertinus Serif"
+CJKmainfont: "Noto Serif CJK SC"
+monofont: "Fira Code"
+linestretch: 1.25
+header-includes:
+  - \usepackage{amsmath}
+  - \usepackage{booktabs}
+  - \usepackage{hyperref}
+  - \usepackage{fancyhdr}
+  - \pagestyle{fancy}
+---
+```
+
+### Custom LaTeX Template
+
+For full control, create a custom Pandoc template:
+
+```bash
+# Export default template as starting point
+pandoc -D latex > custom-template.tex
+
+# Use custom template
+pandoc input.md -o output.pdf --pdf-engine=xelatex \
+  --template=custom-template.tex
+```
+
+### Table of Contents and Numbering
+
+```bash
+# Add table of contents
+pandoc input.md -o output.pdf --pdf-engine=xelatex --toc
+
+# Number sections
+pandoc input.md -o output.pdf --pdf-engine=xelatex --toc -N
+
+# TOC depth (default 3)
+pandoc input.md -o output.pdf --pdf-engine=xelatex --toc --toc-depth=2
+```
+
+### Syntax-Highlighted Code Blocks
+
+```bash
+# List available highlight styles
+pandoc --list-highlight-styles
+
+# Use a specific style
+pandoc input.md -o output.pdf --pdf-engine=xelatex \
+  --highlight-style=tango
+```
+
+### Bibliography and Citations
+
+```bash
+# With BibTeX bibliography
+pandoc input.md -o output.pdf --pdf-engine=xelatex \
+  --citeproc --bibliography=references.bib \
+  --csl=ieee.csl
+```
+
+In the Markdown file, cite as `[@smith2024]` or `@smith2024`.
+
+## Batch Conversion Script
+
+```bash
+#!/bin/bash
+# Convert all .md files in current directory to PDF
+
+FONT_MAIN="Libertinus Serif"
+FONT_CJK="Noto Serif CJK SC"
+FONT_MONO="Fira Code"
+
+for f in *.md; do
+    echo "Converting: $f"
+    pandoc "$f" -o "${f%.md}.pdf" \
+        --pdf-engine=xelatex \
+        -V mainfont="$FONT_MAIN" \
+        -V CJKmainfont="$FONT_CJK" \
+        -V monofont="$FONT_MONO" \
+        -V geometry:margin=2.5cm \
+        -V fontsize=11pt \
+        --toc -N \
+        --highlight-style=tango
+done
+echo "Done. Converted $(ls *.pdf | wc -l) files."
+```
+
+## Troubleshooting
+
+| Problem | Cause | Solution |
+|---------|-------|---------|
+| CJK characters show as boxes | Missing CJK font | Install `fonts-noto-cjk` and set `CJKmainfont` |
+| `! LaTeX Error: File 'xxx.sty' not found` | Missing LaTeX package | `tlmgr install xxx` or install full texlive |
+| Math formulas not rendering | Using pdflatex engine | Switch to `--pdf-engine=xelatex` |
+| Emoji not displaying | Font missing emoji glyphs | Add `-V mainfont="Noto Color Emoji"` as fallback |
+| PDF too large | Embedded bitmap images | Convert images to vector (SVG→PDF) before inclusion |
+| Line breaks in CJK text | Pandoc treats newlines as spaces | Use `\newline` or blank lines between paragraphs |
+
+## References
+
+- [Pandoc User's Guide](https://pandoc.org/MANUAL.html)
+- [XeLaTeX on Overleaf](https://www.overleaf.com/learn/latex/XeLaTeX)
+- [Noto CJK Fonts](https://github.com/googlefonts/noto-cjk)
+- [Pandoc Templates](https://pandoc.org/MANUAL.html#templates)

package/skills/tools/document/openpaper-guide/SKILL.md

@@ -0,0 +1,232 @@
+---
+name: openpaper-guide
+description: "Open-source tool for organizing and annotating research papers"
+metadata:
+  openclaw:
+    emoji: "📄"
+    category: "tools"
+    subcategory: "document"
+    keywords: ["paper management", "PDF annotation", "research organizer", "paper reader", "document viewer", "open source"]
+    source: "https://github.com/nicehash/openpaper"
+---
+
+# OpenPaper Guide
+
+## Overview
+
+OpenPaper is an open-source research paper management and annotation tool. It provides PDF viewing with inline annotations, paper organization with tags and collections, metadata extraction, full-text search across your library, and export capabilities. Designed as a lightweight, privacy-focused alternative to commercial reference managers, running entirely locally.
+
+## Installation
+
+```bash
+# Install via pip
+pip install openpaper
+
+# Or from source
+git clone https://github.com/nicehash/openpaper.git
+cd openpaper && pip install -e .
+
+# Launch
+openpaper
+```
+
+## Library Management
+
+```python
+from openpaper import Library
+
+library = Library("./my_research_library")
+
+# Add papers
+paper = library.add("path/to/paper.pdf")
+print(f"Added: {paper.title}")
+print(f"Authors: {paper.authors}")
+print(f"Year: {paper.year}")
+
+# Bulk import
+added = library.import_directory(
+    "downloads/papers/",
+    recursive=True,
+    extract_metadata=True,   # Auto-extract from PDF
+    deduplicate=True,         # Skip duplicates by DOI/title
+)
+print(f"Imported {len(added)} papers, {added.duplicates} skipped")
+```
+
+## Organization
+
+```python
+# Tags
+paper.add_tag("transformer")
+paper.add_tag("attention")
+paper.add_tag("priority:high")
+
+# Collections
+library.create_collection("thesis-chapter-2")
+library.add_to_collection("thesis-chapter-2", paper)
+
+# Smart collections (auto-updating filters)
+library.create_smart_collection(
+    name="Recent NLP",
+    filters={
+        "tags": ["nlp"],
+        "year": {"gte": 2023},
+        "read_status": "unread",
+    },
+)
+
+# List and browse
+for p in library.search(tags=["transformer"], year=2024):
+    print(f"{p.title} ({p.year}) - {p.read_status}")
+```
+
+## Annotations
+
+```python
+# Add annotations to papers
+paper.annotate(
+    page=3,
+    type="highlight",
+    text="The attention mechanism allows the model to focus...",
+    color="yellow",
+    note="Key definition of attention",
+)
+
+paper.annotate(
+    page=5,
+    type="comment",
+    position=(100, 250),  # x, y coordinates
+    note="This contradicts the claim in Smith et al. 2023",
+)
+
+# Export annotations
+annotations = paper.get_annotations()
+for ann in annotations:
+    print(f"[p.{ann.page}] {ann.type}: {ann.text[:60]}...")
+    if ann.note:
+        print(f"  Note: {ann.note}")
+
+# Export to markdown
+paper.export_annotations("annotations.md")
+```
+
+## Search
+
+```python
+# Full-text search across library
+results = library.search_fulltext("attention mechanism")
+for r in results:
+    print(f"{r.title} (relevance: {r.score:.2f})")
+    for match in r.matches[:3]:
+        print(f"  p.{match.page}: ...{match.context}...")
+
+# Metadata search
+results = library.search(
+    query="transformer",        # Title/abstract search
+    authors="Vaswani",
+    year_range=(2020, 2025),
+    tags=["nlp"],
+)
+
+# Semantic search (if embeddings enabled)
+results = library.semantic_search(
+    "methods for reducing quadratic complexity of attention",
+    top_k=10,
+)
+```
+
+## Metadata Extraction
+
+```python
+# Auto-extract metadata from PDFs
+metadata = library.extract_metadata("paper.pdf")
+print(f"Title: {metadata.title}")
+print(f"Authors: {metadata.authors}")
+print(f"Abstract: {metadata.abstract[:200]}...")
+print(f"DOI: {metadata.doi}")
+print(f"Year: {metadata.year}")
+print(f"References: {len(metadata.references)}")
+
+# Enrich with external databases
+enriched = library.enrich_metadata(
+    paper,
+    sources=["crossref", "semantic_scholar"],
+)
+print(f"Citations: {enriched.citation_count}")
+print(f"Venue: {enriched.venue}")
+```
+
+## Export
+
+```python
+# Export bibliography
+library.export_bibtex("references.bib", collection="thesis-chapter-2")
+
+# Export reading list
+library.export_reading_list("reading_list.md", format="markdown")
+
+# Export annotations from all papers
+library.export_all_annotations("all_annotations.md")
+
+# Sync with reference manager
+library.export_ris("export.ris")          # RIS format
+library.export_csv("export.csv")          # CSV with metadata
+```
+
+## Configuration
+
+```json
+{
+  "library_path": "./research_library",
+  "pdf_viewer": "builtin",
+  "metadata": {
+    "auto_extract": true,
+    "enrich_sources": ["crossref"],
+    "language": "en"
+  },
+  "search": {
+    "fulltext_index": true,
+    "semantic_search": false,
+    "embedding_model": "all-MiniLM-L6-v2"
+  },
+  "storage": {
+    "copy_pdfs": true,
+    "organize_by": "year",
+    "max_library_size_gb": 10
+  }
+}
+```
+
+## CLI Usage
+
+```bash
+# Add paper
+openpaper add paper.pdf --tags "nlp,transformer"
+
+# Search
+openpaper search "attention mechanism" --limit 10
+
+# List library
+openpaper list --sort year --tags "priority:high"
+
+# Export
+openpaper export bibtex --collection thesis --output refs.bib
+
+# Stats
+openpaper stats
+# Papers: 342, Tagged: 289, Annotated: 156, Collections: 12
+```
+
+## Use Cases
+
+1. **Paper library**: Organize and search your PDF collection
+2. **Reading workflow**: Track read status, annotate, take notes
+3. **Reference management**: Export BibTeX for LaTeX papers
+4. **Literature review**: Tag and categorize papers by topic
+5. **Team sharing**: Export reading lists and annotations
+
+## References
+
+- [OpenPaper GitHub](https://github.com/nicehash/openpaper)
+- [Zotero](https://www.zotero.org/) — Popular open-source alternative
+- [Semantic Scholar API](https://api.semanticscholar.org/) — Metadata enrichment