@wentorai/research-plugins 1.0.0 → 1.2.0

package/skills/literature/search/arxiv-batch-reporting/SKILL.md

@@ -0,0 +1,133 @@
+---
+name: arxiv-batch-reporting
+description: "Batch search and report generation from arXiv preprint repository"
+metadata:
+  openclaw:
+    emoji: "📊"
+    category: "literature"
+    subcategory: "search"
+    keywords: ["arxiv", "batch search", "preprint", "report generation", "literature monitoring", "research trends"]
+    source: "https://github.com/sspaeti/arxiv-batch-search"
+---
+
+# arXiv Batch Reporting
+
+## Overview
+
+Keeping up with the flood of new preprints on arXiv is one of the most persistent challenges in fast-moving fields like machine learning, physics, mathematics, and computer science. The arXiv Batch Reporting skill provides a systematic approach to searching, filtering, and generating structured reports from arXiv at scale.
+
+Unlike ad-hoc manual searches, this skill enables researchers to define persistent query profiles, run batch searches across date ranges, and produce formatted reports that highlight the most relevant papers. It is particularly useful for weekly or monthly literature surveillance, lab meeting preparation, and trend analysis across subfields.
+
+The skill leverages the arXiv API and supports advanced query syntax, category filtering, and result ranking by relevance or recency. Reports can be generated in Markdown, HTML, or CSV formats for integration into existing workflows.
+
+## Setting Up Batch Queries
+
+### Query Profile Definition
+
+Define your search profiles as structured configurations. Each profile specifies the search terms, category filters, date range, and output preferences:
+
+```yaml
+profile_name: "transformer-architectures-weekly"
+queries:
+  - "ti:transformer AND abs:attention mechanism"
+  - "ti:vision transformer"
+  - "abs:efficient transformer AND cat:cs.LG"
+categories:
+  - cs.LG
+  - cs.CL
+  - cs.CV
+date_range: "last_7_days"
+max_results: 100
+sort_by: "submittedDate"
+sort_order: "descending"
+```
+
+### arXiv API Query Syntax
+
+The arXiv API supports field-specific searches:
+
+- `ti:` — Search in title
+- `abs:` — Search in abstract
+- `au:` — Search by author
+- `cat:` — Filter by category (e.g., `cs.AI`, `math.PR`, `physics.comp-ph`)
+- Boolean operators: `AND`, `OR`, `ANDNOT`
+- Group with parentheses for complex queries
+
+**Example queries:**
+- Find recent GAN papers in computer vision: `abs:generative adversarial AND cat:cs.CV`
+- Find a specific author's work: `au:bengio AND ti:deep learning`
+- Exclude survey papers: `abs:reinforcement learning ANDNOT ti:survey`
+
+### Rate Limiting and Pagination
+
+The arXiv API enforces rate limits. Follow these guidelines:
+
+- Wait at least 3 seconds between API requests
+- Use pagination with `start` and `max_results` parameters (max 2000 per request)
+- For large batch jobs, implement exponential backoff on HTTP 503 responses
+- Cache results locally to avoid redundant API calls
+
+## Report Generation
+
+### Standard Report Template
+
+After collecting batch results, generate a report with the following structure:
+
+```markdown
+# arXiv Batch Report: [Profile Name]
+**Date range:** [start] to [end]
+**Total results:** [N] papers
+**Generated:** [timestamp]
+
+## Highlights (Top 10 by Relevance)
+| # | Title | Authors | Category | Date |
+|---|-------|---------|----------|------|
+| 1 | [Title](arxiv-link) | First Author et al. | cs.LG | 2026-03-08 |
+
+## Category Breakdown
+- cs.LG: 45 papers
+- cs.CL: 23 papers
+- cs.CV: 18 papers
+
+## Keyword Frequency
+- "transformer": 38 mentions
+- "attention": 29 mentions
+- "efficient": 15 mentions
+
+## Full Results
+[Expandable table with all papers]
+```
+
+### Filtering and Ranking
+
+After retrieving raw results, apply post-processing filters to surface the most relevant papers:
+
+1. **Relevance scoring**: Score each paper based on keyword density in the title and abstract relative to your query terms.
+2. **Author filtering**: Boost papers from authors on your watch list (key researchers in your field).
+3. **Citation proxy**: Papers that appear in multiple query results likely sit at the intersection of your interests—rank them higher.
+4. **Novelty detection**: Flag papers whose abstracts contain terms not seen in your previous reports, indicating potentially new directions.
+
+## Automation and Scheduling
+
+For ongoing literature surveillance, automate your batch reports:
+
+- **Cron scheduling**: Run batch queries weekly (e.g., every Monday at 8 AM) using a scheduled task or CI pipeline.
+- **Diff reports**: Compare the current week's results against the previous week to highlight only new papers.
+- **Alert thresholds**: Set alerts when a report contains more than N papers matching a high-priority query, indicating a burst of activity in that area.
+- **Email or Slack delivery**: Route generated reports to your inbox or lab Slack channel for team-wide awareness.
+
+Store all generated reports in a versioned directory structure for longitudinal trend analysis:
+
+```
+reports/
+  transformer-architectures-weekly/
+    2026-03-03.md
+    2026-03-10.md
+    ...
+```
+
+## References
+
+- arXiv API documentation: https://info.arxiv.org/help/api/index.html
+- arXiv category taxonomy: https://arxiv.org/category_taxonomy
+- arXiv Batch Search: https://github.com/sspaeti/arxiv-batch-search

package/skills/literature/search/arxiv-cli-tools/SKILL.md

@@ -0,0 +1,172 @@
+---
+name: arxiv-cli-tools
+description: "Command-line tools for searching and batch-downloading arXiv papers"
+metadata:
+  openclaw:
+    emoji: "🔍"
+    category: "literature"
+    subcategory: "search"
+    keywords: ["arxiv", "command line", "paper download", "preprint search", "batch download", "literature retrieval"]
+    source: "https://pypi.org/project/arxiv-cli-tools/"
+---
+
+# arXiv CLI Tools
+
+## Overview
+
+`arxiv-cli-tools` is a Python command-line interface for searching and downloading papers from arXiv.org. It wraps the `arxiv` Python client library into convenient CLI commands, enabling researchers to search by keyword, author, or category, view abstracts, and batch-download PDFs directly from the terminal. No API key is required.
+
+## Installation
+
+```bash
+# Recommended: isolated install with pipx
+pipx install arxiv-cli-tools
+
+# Alternative: pip
+pip install arxiv-cli-tools
+
+# Verify installation
+arxiv-cli --help
+```
+
+## Searching Papers
+
+### Basic Search
+
+```bash
+# Search by keyword (default: 10 results)
+arxiv-cli search "transformer attention mechanism"
+
+# Limit results
+arxiv-cli search "quantum computing" -n 5
+
+# Show abstracts in results
+arxiv-cli search "prompt engineering" -n 5 --summary
+```
+
+### Filtered Search
+
+```bash
+# Filter by author
+arxiv-cli search "attention mechanism" --authors "Vaswani"
+
+# Filter by arXiv category
+arxiv-cli search "neural networks" --categories "cs.LG,cs.AI"
+
+# Combine filters
+arxiv-cli search "protein folding" --categories "q-bio" -n 20 --summary
+```
+
+### Common arXiv Categories
+
+| Prefix | Field | Popular Subcategories |
+|--------|-------|----------------------|
+| `cs` | Computer Science | cs.AI, cs.CL, cs.CV, cs.LG, cs.SE |
+| `math` | Mathematics | math.ST, math.OC, math.PR |
+| `physics` | Physics | physics.comp-ph, hep-th, cond-mat |
+| `stat` | Statistics | stat.ML, stat.ME, stat.TH |
+| `q-bio` | Quantitative Biology | q-bio.BM, q-bio.GN |
+| `q-fin` | Quantitative Finance | q-fin.ST, q-fin.PM |
+| `econ` | Economics | econ.EM, econ.GN |
+| `eess` | Electrical Engineering | eess.SP, eess.AS |
+
+## Downloading Papers
+
+### Single Paper
+
+```bash
+# Download by arXiv ID
+arxiv-cli download --id 1706.03762 --dest ~/papers
+
+# Download PDF format explicitly
+arxiv-cli download --id 2301.13688 --dest ~/papers --pdf
+```
+
+### Batch Download
+
+```bash
+# Download multiple papers
+arxiv-cli download --id 1706.03762 --id 2301.13688 --id 2303.08774 \
+  --dest ~/papers/transformers
+
+# Skip already downloaded files
+arxiv-cli download --id 1706.03762 --id 2301.13688 \
+  --dest ~/papers --skip-existing
+```
+
+### Download from Search Results
+
+A common workflow is to search first, then download selected papers:
+
+```bash
+# 1. Search and note IDs
+arxiv-cli search "diffusion models survey" -n 10 --summary
+
+# 2. Download the relevant ones
+arxiv-cli download --id 2209.00796 --id 2206.00364 --dest ~/papers/diffusion
+```
+
+## Python API Alternative
+
+For programmatic use, the underlying `arxiv` library provides a Python API:
+
+```python
+import arxiv
+
+# Search
+search = arxiv.Search(
+    query="large language models",
+    max_results=10,
+    sort_by=arxiv.SortCriterion.SubmittedDate
+)
+
+for result in arxiv.Client().results(search):
+    print(f"{result.entry_id}: {result.title}")
+    print(f"  Authors: {', '.join(a.name for a in result.authors)}")
+    print(f"  Published: {result.published.date()}")
+    print(f"  PDF: {result.pdf_url}")
+    print()
+
+# Download
+result.download_pdf(dirpath="./papers", filename="paper.pdf")
+```
+
+## Workflow Integration
+
+### Daily Paper Check Script
+
+```bash
+#!/bin/bash
+# Check for new papers in your research area
+DATE=$(date +%Y-%m-%d)
+LOG="$HOME/papers/daily_${DATE}.txt"
+
+echo "=== arXiv Papers for $DATE ===" > "$LOG"
+arxiv-cli search "retrieval augmented generation" \
+  --categories "cs.CL,cs.AI" -n 20 --summary >> "$LOG"
+
+echo "Paper digest saved to $LOG"
+```
+
+### Export to BibTeX
+
+After finding relevant papers, retrieve BibTeX entries via the arXiv API:
+
+```bash
+# Get BibTeX for a specific paper
+curl -s "https://arxiv.org/bibtex/1706.03762"
+```
+
+## Rate Limits and Etiquette
+
+- arXiv API allows **1 request per 3 seconds** for programmatic access
+- For bulk downloads, add delays between requests
+- The CLI tool respects rate limits by default
+- See [arXiv API Terms of Use](https://info.arxiv.org/help/api/tou.html)
+
+## References
+
+- [arxiv-cli-tools on PyPI](https://pypi.org/project/arxiv-cli-tools/)
+- [arxiv Python Client](https://github.com/lukasschwab/arxiv.py)
+- [arXiv API Documentation](https://info.arxiv.org/help/api/)
+- [arXiv Category Taxonomy](https://arxiv.org/category_taxonomy)

package/skills/literature/search/arxiv-osiris/SKILL.md

@@ -0,0 +1,199 @@
+---
+name: arxiv-osiris
+description: "Search and download arXiv papers via Python and PowerShell scripts"
+metadata:
+  openclaw:
+    emoji: "🔍"
+    category: "literature"
+    subcategory: "search"
+    keywords: ["arxiv", "paper download", "preprint search", "python script", "powershell", "literature retrieval"]
+    source: "https://clawhub.com/kostaskyq/arxiv-osiris"
+---
+
+# arXiv Osiris — Paper Search and Download Tool
+
+## Overview
+
+arXiv Osiris provides cross-platform scripts (Python and PowerShell) for searching and downloading scientific papers from arXiv.org. It supports keyword search, category filtering, metadata retrieval, and direct PDF download. Useful for researchers who prefer scripted automation over browser-based arXiv access, particularly for building local paper collections.
+
+## Installation
+
+```bash
+# Install the arxiv Python client (required dependency)
+pip install arxiv
+
+# Clone the tool (if using from source)
+git clone https://github.com/kostaskyq/arxiv-osiris.git
+```
+
+## Usage — Python API
+
+### Search for Papers
+
+```python
+import arxiv
+
+# Basic keyword search
+search = arxiv.Search(
+    query="quantum computing error correction",
+    max_results=10,
+    sort_by=arxiv.SortCriterion.Relevance
+)
+
+client = arxiv.Client()
+for result in client.results(search):
+    print(f"ID:       {result.entry_id}")
+    print(f"Title:    {result.title}")
+    print(f"Authors:  {', '.join(a.name for a in result.authors)}")
+    print(f"Published:{result.published.strftime('%Y-%m-%d')}")
+    print(f"PDF:      {result.pdf_url}")
+    print(f"Abstract: {result.summary[:200]}...")
+    print()
+```
+
+### Category-Filtered Search
+
+```python
+# Search within specific categories
+search = arxiv.Search(
+    query="cat:cs.CL AND transformer",
+    max_results=20,
+    sort_by=arxiv.SortCriterion.SubmittedDate
+)
+
+# Multiple categories
+search = arxiv.Search(
+    query="(cat:cs.AI OR cat:cs.LG) AND reinforcement learning",
+    max_results=15
+)
+```
+
+### Download Papers
+
+```python
+import os
+
+search = arxiv.Search(query="attention mechanism", max_results=5)
+client = arxiv.Client()
+download_dir = os.path.expanduser("~/papers/attention")
+os.makedirs(download_dir, exist_ok=True)
+
+for result in client.results(search):
+    # Download PDF
+    result.download_pdf(dirpath=download_dir)
+    print(f"Downloaded: {result.title}")
+
+    # Download source (LaTeX) if available
+    result.download_source(dirpath=download_dir)
+```
+
+## Usage — PowerShell Script
+
+### Search
+
+```powershell
+# Basic search
+.\arxiv.ps1 -Action search -Query "machine learning"
+
+# With max results
+.\arxiv.ps1 -Action search -Query "neural networks" -MaxResults 10
+
+# Filter by category
+.\arxiv.ps1 -Action search -Query "deep learning" -Categories "cs,stat"
+```
+
+### Download
+
+```powershell
+# Download by arXiv ID
+.\arxiv.ps1 -Action download -ArxivId "1706.03762"
+
+# Download to specific directory
+.\arxiv.ps1 -Action download -ArxivId "2301.13688" -OutputDir "C:\Papers"
+```
+
+## Advanced Queries
+
+The arXiv API supports a rich query syntax:
+
+| Operator | Meaning | Example |
+|----------|---------|---------|
+| `AND` | Both terms | `"deep learning" AND "drug discovery"` |
+| `OR` | Either term | `"GAN" OR "diffusion model"` |
+| `ANDNOT` | Exclude term | `"NLP" ANDNOT "translation"` |
+| `au:` | Author | `au:"Hinton"` |
+| `ti:` | Title contains | `ti:"attention"` |
+| `abs:` | Abstract contains | `abs:"protein folding"` |
+| `cat:` | Category | `cat:cs.CV` |
+
+### Complex Query Examples
+
+```python
+# Papers by a specific author on a specific topic
+search = arxiv.Search(query='au:"Yann LeCun" AND ti:"self-supervised"')
+
+# Recent papers in two categories excluding surveys
+search = arxiv.Search(
+    query='(cat:cs.CL OR cat:cs.AI) AND "large language model" ANDNOT ti:"survey"',
+    sort_by=arxiv.SortCriterion.SubmittedDate,
+    max_results=50
+)
+```
+
+## Building a Local Paper Library
+
+```python
+import arxiv
+import json
+import os
+from datetime import datetime
+
+def build_library(queries: dict, base_dir: str = "~/papers"):
+    """Build organized paper library from multiple search queries."""
+    base = os.path.expanduser(base_dir)
+    catalog = []
+    client = arxiv.Client()
+
+    for topic, query in queries.items():
+        topic_dir = os.path.join(base, topic)
+        os.makedirs(topic_dir, exist_ok=True)
+
+        search = arxiv.Search(query=query, max_results=20,
+                              sort_by=arxiv.SortCriterion.SubmittedDate)
+
+        for paper in client.results(search):
+            paper.download_pdf(dirpath=topic_dir)
+            catalog.append({
+                "id": paper.entry_id,
+                "title": paper.title,
+                "authors": [a.name for a in paper.authors],
+                "published": paper.published.isoformat(),
+                "topic": topic,
+                "pdf_path": os.path.join(topic_dir, f"{paper.get_short_id()}.pdf")
+            })
+
+    # Save catalog
+    with open(os.path.join(base, "catalog.json"), "w") as f:
+        json.dump(catalog, f, indent=2)
+    print(f"Library built: {len(catalog)} papers in {len(queries)} topics")
+
+# Usage
+build_library({
+    "rag": "cat:cs.CL AND retrieval augmented generation",
+    "agents": "cat:cs.AI AND (LLM agent OR tool use)",
+    "evaluation": "cat:cs.CL AND (benchmark OR evaluation) AND language model"
+})
+```
+
+## Rate Limits
+
+- arXiv API: **1 request per 3 seconds** for automated access
+- The `arxiv` Python client handles rate limiting automatically
+- For large-scale downloads, add explicit delays: `time.sleep(3)`
+- Respect [arXiv API Terms of Use](https://info.arxiv.org/help/api/tou.html)
+
+## References
+
+- [arxiv Python Client](https://github.com/lukasschwab/arxiv.py)
+- [arXiv API User Manual](https://info.arxiv.org/help/api/user-manual.html)
+- [arXiv Category Taxonomy](https://arxiv.org/category_taxonomy)

package/skills/literature/search/arxiv-paper-processor/SKILL.md

@@ -0,0 +1,141 @@
+---
+name: arxiv-paper-processor
+description: "Process and analyze arXiv papers systematically for research workflows"
+metadata:
+  openclaw:
+    emoji: "⚙️"
+    category: "literature"
+    subcategory: "search"
+    keywords: ["arxiv", "paper processing", "PDF parsing", "metadata extraction", "preprint analysis", "research pipeline"]
+    source: "https://github.com/tatsu-lab/gpt_paper_assistant"
+---
+
+# arXiv Paper Processor
+
+## Overview
+
+The arXiv Paper Processor skill provides a complete pipeline for downloading, parsing, and analyzing arXiv papers programmatically. While the arXiv API provides metadata, researchers often need to work with the full text—extracting sections, equations, figures, and references for deeper analysis.
+
+This skill covers the entire processing chain: retrieving papers by ID or search query, downloading PDF and LaTeX source files, extracting structured content, and producing analysis-ready outputs. It is particularly valuable for researchers conducting large-scale literature analysis, building training datasets from academic text, or automating evidence extraction for systematic reviews.
+
+The pipeline handles common challenges in academic PDF processing including multi-column layouts, mathematical notation, table extraction, and reference parsing. It integrates with tools like GROBID for PDF parsing and can work directly with arXiv LaTeX sources for higher-fidelity extraction.
+
+## Paper Retrieval and Download
+
+### Fetching by arXiv ID
+
+The most reliable method is to fetch papers by their arXiv identifier:
+
+```python
+import urllib.request
+import feedparser
+
+# Fetch metadata via Atom feed
+arxiv_id = "2301.07041"
+url = f"http://export.arxiv.org/api/query?id_list={arxiv_id}"
+response = urllib.request.urlopen(url)
+feed = feedparser.parse(response.read())
+
+entry = feed.entries[0]
+title = entry.title
+abstract = entry.summary
+authors = [a.name for a in entry.authors]
+pdf_url = entry.links[1].href  # PDF link
+```
+
+### Downloading Source Files
+
+arXiv stores LaTeX source files for most papers. These provide much richer structure than PDFs:
+
+```bash
+# Download LaTeX source (typically a .tar.gz)
+wget https://arxiv.org/e-print/2301.07041 -O paper_source.tar.gz
+tar -xzf paper_source.tar.gz -C paper_source/
+```
+
+Source files contain the original `.tex` files, figures, bibliography files, and any custom style files. Parsing LaTeX directly gives you access to section structure, equations in their original notation, citation keys, and figure captions without the ambiguity of PDF extraction.
+
+### Batch Download Guidelines
+
+When downloading multiple papers, respect arXiv's usage policies:
+
+- Limit requests to 1 per 3 seconds for API calls
+- Use the arXiv bulk data access (S3 or GCS) for large-scale processing (1000+ papers)
+- Cache all downloaded files locally and check before re-downloading
+- Include a descriptive User-Agent header in your HTTP requests
+
+## Content Extraction Pipeline
+
+### PDF Extraction with GROBID
+
+For papers where only PDF is available, use GROBID (GeneRation Of BIbliographic Data) for structured extraction:
+
+```bash
+# Run GROBID as a local service
+docker run --rm -p 8070:8070 grobid/grobid:0.8.0
+
+# Process a PDF
+curl -X POST "http://localhost:8070/api/processFulltextDocument" \
+  -F "input=@paper.pdf" \
+  -F "consolidateHeader=1" \
+  -F "consolidateCitations=1" \
+  > paper_tei.xml
+```
+
+GROBID outputs TEI-XML with structured sections including:
+- Header metadata (title, authors, affiliations, abstract)
+- Body text with section hierarchy
+- Equations (as MathML or raw text)
+- Figure and table references
+- Parsed bibliography entries with DOIs where available
+
+### LaTeX Source Parsing
+
+When LaTeX source is available, parse it directly for higher fidelity:
+
+1. Identify the main `.tex` file (look for `\documentclass` or `\begin{document}`)
+2. Resolve `\input{}` and `\include{}` directives to build the complete document
+3. Extract sections using `\section{}`, `\subsection{}` markers
+4. Extract equations from `equation`, `align`, `gather` environments
+5. Parse `\cite{}` commands and cross-reference with the `.bib` file
+6. Extract figure captions from `\caption{}` commands
+
+### Structured Output Schema
+
+Produce a standardized JSON output for each processed paper:
+
+```json
+{
+  "arxiv_id": "2301.07041",
+  "title": "Paper Title",
+  "authors": ["Author One", "Author Two"],
+  "abstract": "...",
+  "sections": [
+    {"heading": "Introduction", "level": 1, "text": "..."},
+    {"heading": "Related Work", "level": 1, "text": "..."}
+  ],
+  "equations": ["E = mc^2", "..."],
+  "figures": [{"id": "fig1", "caption": "..."}],
+  "references": [{"key": "smith2020", "title": "...", "doi": "..."}],
+  "processed_date": "2026-03-10"
+}
+```
+
+## Analysis and Integration
+
+Once papers are processed into structured format, several downstream analyses become possible:
+
+- **Section-level search**: Search across the methods sections of hundreds of papers to find specific techniques.
+- **Equation extraction**: Build a database of mathematical formulations used in your subfield.
+- **Citation graph construction**: Map which papers cite which, using extracted reference lists.
+- **Terminology tracking**: Monitor how specific terms evolve in usage frequency over time.
+- **Dataset identification**: Extract mentions of datasets and benchmarks from experimental sections.
+
+Integrate processed outputs with your reference manager by generating BibTeX entries enriched with extracted metadata, or feed structured JSON into a local search index for full-text retrieval across your paper collection.
+
+## References
+
+- arXiv API: https://info.arxiv.org/help/api/index.html
+- GROBID: https://github.com/kermitt2/grobid
+- GPT Paper Assistant: https://github.com/tatsu-lab/gpt_paper_assistant
+- arXiv bulk data access: https://info.arxiv.org/help/bulk_data/index.html