@wentorai/research-plugins 1.0.0 → 1.2.0

package/skills/tools/scraping/repository-harvesting-guide/SKILL.md

@@ -0,0 +1,310 @@
+---
+name: repository-harvesting-guide
+description: "Harvest metadata from open repositories using OAI-PMH protocol"
+metadata:
+  openclaw:
+    emoji: "tractor"
+    category: "tools"
+    subcategory: "scraping"
+    keywords: ["OAI-PMH", "metadata harvesting", "open repositories", "Dublin Core", "institutional repositories", "data providers"]
+    source: "wentor-research-plugins"
+---
+
+# Repository Harvesting Guide
+
+A skill for harvesting metadata from open access repositories using the OAI-PMH (Open Archives Initiative Protocol for Metadata Harvesting) protocol. Covers protocol fundamentals, building harvesters in Python, handling resumption tokens for large collections, metadata format parsing (Dublin Core, MARC, METS), selective harvesting by date and set, and integrating harvested data into research workflows.
+
+## OAI-PMH Protocol Fundamentals
+
+### What Is OAI-PMH
+
+OAI-PMH is a standardized protocol that allows metadata to be harvested from repository systems. It is the backbone of library interoperability and is supported by virtually every institutional repository, preprint server, and digital library worldwide.
+
+```
+OAI-PMH Architecture:
+
+Data Providers (repositories):
+  - Expose metadata through a standardized HTTP interface
+  - Must support Dublin Core as minimum metadata format
+  - May support additional formats (MARC, MODS, DataCite, etc.)
+  - Examples: arXiv, PubMed Central, DSpace repositories,
+    EPrints, institutional repositories
+
+Service Providers (harvesters):
+  - Send HTTP requests to data providers
+  - Collect, aggregate, and index metadata
+  - Build search services, union catalogs, analytics
+  - Examples: BASE (Bielefeld), CORE, OpenDOAR
+
+Protocol Version: 2.0 (current, since 2002)
+Transport: HTTP GET or POST
+Response format: XML
+Base URL example: https://arxiv.org/oai2
+```
+
+### Six OAI-PMH Verbs
+
+```
+OAI-PMH defines exactly six request types (verbs):
+
+1. Identify
+   Purpose: Describe the repository
+   URL: baseURL?verb=Identify
+   Returns: repository name, admin email, earliest datestamp,
+            granularity, compression support
+
+2. ListMetadataFormats
+   Purpose: List available metadata formats
+   URL: baseURL?verb=ListMetadataFormats
+   Returns: format prefixes (oai_dc, marc21, datacite, etc.)
+   Optional: identifier parameter to check formats for one record
+
+3. ListSets
+   Purpose: List available sets (collections/categories)
+   URL: baseURL?verb=ListSets
+   Returns: set names and specs for selective harvesting
+   Example sets: physics:hep-th, cs:AI, math:AG
+
+4. ListIdentifiers
+   Purpose: List record identifiers (headers only, no metadata)
+   URL: baseURL?verb=ListIdentifiers&metadataPrefix=oai_dc
+   Optional: from, until, set parameters
+   Returns: identifiers, datestamps, set memberships
+
+5. ListRecords
+   Purpose: Harvest full metadata records
+   URL: baseURL?verb=ListRecords&metadataPrefix=oai_dc
+   Optional: from, until, set parameters
+   Returns: complete metadata records in requested format
+
+6. GetRecord
+   Purpose: Retrieve a single record by identifier
+   URL: baseURL?verb=GetRecord&identifier=oai:arxiv:2301.00001
+         &metadataPrefix=oai_dc
+   Returns: one complete metadata record
+```
+
+## Building a Harvester in Python
+
+### Basic Harvester
+
+```python
+import requests
+import xml.etree.ElementTree as ET
+import time
+
+OAI_NS = "http://www.openarchives.org/OAI/2.0/"
+DC_NS = "http://purl.org/dc/elements/1.1/"
+
+def harvest_records(base_url, metadata_prefix="oai_dc",
+                    from_date=None, until_date=None,
+                    set_spec=None):
+    """
+    Harvest all records from an OAI-PMH endpoint.
+    Handles resumption tokens for paginated results.
+
+    Args:
+        base_url: OAI-PMH base URL
+        metadata_prefix: metadata format (default: oai_dc)
+        from_date: selective harvest start (YYYY-MM-DD)
+        until_date: selective harvest end (YYYY-MM-DD)
+        set_spec: restrict to a specific set
+    """
+    params = {
+        "verb": "ListRecords",
+        "metadataPrefix": metadata_prefix,
+    }
+
+    if from_date:
+        params["from"] = from_date
+    if until_date:
+        params["until"] = until_date
+    if set_spec:
+        params["set"] = set_spec
+
+    all_records = []
+    request_count = 0
+
+    while True:
+        response = requests.get(base_url, params=params, timeout=30)
+        response.raise_for_status()
+        request_count += 1
+
+        root = ET.fromstring(response.content)
+
+        # Parse records from this page
+        records = root.findall(
+            f".//{{{OAI_NS}}}record"
+        )
+
+        for record in records:
+            parsed = parse_dublin_core(record)
+            if parsed:
+                all_records.append(parsed)
+
+        # Check for resumption token
+        token_elem = root.find(
+            f".//{{{OAI_NS}}}resumptionToken"
+        )
+
+        if token_elem is not None and token_elem.text:
+            params = {
+                "verb": "ListRecords",
+                "resumptionToken": token_elem.text,
+            }
+            # Polite delay between requests
+            time.sleep(2)
+        else:
+            break
+
+    print(f"Harvested {len(all_records)} records "
+          f"in {request_count} requests")
+    return all_records
+
+
+def parse_dublin_core(record_element):
+    """
+    Parse a Dublin Core metadata record into a dictionary.
+    """
+    header = record_element.find(f"{{{OAI_NS}}}header")
+    metadata = record_element.find(f"{{{OAI_NS}}}metadata")
+
+    if header is None or metadata is None:
+        return None
+
+    # Check if record is deleted
+    status = header.get("status", "")
+    if status == "deleted":
+        return None
+
+    identifier = header.findtext(f"{{{OAI_NS}}}identifier", "")
+    datestamp = header.findtext(f"{{{OAI_NS}}}datestamp", "")
+
+    dc = metadata.find(f".//{{{DC_NS}}}../")
+
+    result = {
+        "oai_identifier": identifier,
+        "datestamp": datestamp,
+        "title": find_dc_text(metadata, "title"),
+        "creator": find_dc_all(metadata, "creator"),
+        "subject": find_dc_all(metadata, "subject"),
+        "description": find_dc_text(metadata, "description"),
+        "date": find_dc_text(metadata, "date"),
+        "type": find_dc_text(metadata, "type"),
+        "identifier": find_dc_all(metadata, "identifier"),
+        "language": find_dc_text(metadata, "language"),
+        "rights": find_dc_text(metadata, "rights"),
+    }
+
+    return result
+
+
+def find_dc_text(metadata, element_name):
+    """Find first Dublin Core element text."""
+    elem = metadata.find(f".//{{{DC_NS}}}{element_name}")
+    return elem.text if elem is not None else ""
+
+
+def find_dc_all(metadata, element_name):
+    """Find all values of a Dublin Core element."""
+    elems = metadata.findall(f".//{{{DC_NS}}}{element_name}")
+    return [e.text for e in elems if e.text]
+```
+
+## Selective Harvesting
+
+### By Date Range
+
+```
+Incremental harvesting strategy:
+
+First harvest: Get everything
+  from_date = None (or repository's earliestDatestamp)
+  until_date = today
+
+Subsequent harvests: Get only new/modified records
+  from_date = last_harvest_date
+  until_date = today
+
+Date granularity:
+  - Day-level: YYYY-MM-DD (most common)
+  - Second-level: YYYY-MM-DDThh:mm:ssZ (some repositories)
+  - Check the Identify response for supported granularity
+
+Important: OAI-PMH datestamps reflect the date the METADATA
+was last modified, not the publication date. A record edited
+yesterday to fix a typo will appear in a harvest with
+from=yesterday, even if the paper was published in 2015.
+```
+
+### By Set (Collection)
+
+```
+Common set structures by repository type:
+
+arXiv:
+  physics, physics:hep-th, cs, cs:AI, math, math:AG, etc.
+
+DSpace repositories:
+  com_12345_1 (community), col_12345_2 (collection)
+  Hierarchical: department -> collection
+
+PubMed Central:
+  By journal: pmc-journal-name
+  By funder: pmc-funder-name
+
+Strategy:
+  1. Call ListSets to see available sets
+  2. Identify sets relevant to your research topic
+  3. Harvest only those sets to reduce data volume
+  4. Store the set membership for each record
+```
+
+## Data Quality and Deduplication
+
+### Common Quality Issues
+
+```
+Quality problems in harvested metadata:
+
+1. Duplicate records:
+   - Same paper in multiple repositories
+   - Same paper in multiple sets within one repository
+   - Solution: Deduplicate by DOI, then by title similarity
+
+2. Incomplete metadata:
+   - Missing abstracts (very common)
+   - Missing author identifiers
+   - Missing dates or using inconsistent date formats
+   - Solution: Enrich with Crossref or OpenAlex lookups
+
+3. Encoding issues:
+   - Non-UTF-8 characters in older repositories
+   - HTML entities in text fields
+   - Solution: Normalize encoding, strip HTML tags
+
+4. Inconsistent formats:
+   - Dates as "2023", "2023-01", "2023-01-15", "January 2023"
+   - Author names as "Smith, John" vs "John Smith" vs "J. Smith"
+   - Solution: Parse and normalize to canonical formats
+```
+
+## Notable OAI-PMH Endpoints
+
+```
+Major repositories with OAI-PMH support:
+
+arXiv:          https://export.arxiv.org/oai2
+PubMed Central: https://www.ncbi.nlm.nih.gov/pmc/oai/oai.cgi
+Europeana:      https://oai.europeana.eu/oai
+HAL (France):   https://api.archives-ouvertes.fr/oai/hal
+DBLP:           https://dblp.org/oai
+CiteSeerX:      https://citeseerx.ist.psu.edu/oai2
+
+To find more endpoints:
+  - OpenDOAR directory: https://v2.sherpa.ac.uk/opendoar/
+  - ROAR (Registry of Open Access Repositories)
+  - BASE (Bielefeld Academic Search Engine) source list
+```
+
+OAI-PMH harvesting remains the most reliable method for building comprehensive metadata collections from open repositories. While newer APIs like ResourceSync and Signposting offer richer functionality, OAI-PMH's universal adoption and simplicity make it the practical choice for most academic metadata collection tasks.

package/skills/writing/citation/academic-citation-manager/SKILL.md

@@ -0,0 +1,314 @@
+---
+name: academic-citation-manager
+description: "Manage academic citations across BibTeX, APA, MLA, and Chicago formats"
+metadata:
+  openclaw:
+    emoji: "bookmark"
+    category: "writing"
+    subcategory: "citation"
+    keywords: ["citation", "BibTeX", "APA", "reference management", "bibliography", "formatting"]
+    source: "https://github.com/wentor-ai/research-plugins"
+---
+
+# Academic Citation Manager
+
+Manage academic citations across multiple formats (BibTeX, APA 7th, MLA 9th, Chicago, Vancouver, IEEE) with automated retrieval from DOIs, conversion between formats, deduplication, and validation. This skill handles the complete citation lifecycle from initial capture through final manuscript formatting.
+
+## Overview
+
+Citation management is a persistent friction point in academic writing. Researchers collect references from multiple sources (databases, PDFs, colleagues, web pages), store them in different formats, and must output them in the specific style required by each target journal. Errors in citations -- misspelled author names, incorrect years, broken DOIs, inconsistent formatting -- are among the most common reasons for desk rejection and reviewer criticism.
+
+This skill provides a comprehensive citation management workflow that goes beyond what GUI reference managers offer. It can retrieve complete metadata from a DOI in seconds, convert between any citation format, detect and merge duplicate entries, validate entries against CrossRef and Semantic Scholar databases, and generate properly formatted bibliographies for any major citation style.
+
+The approach is text-based and scriptable, making it ideal for integration with LaTeX workflows, Markdown writing pipelines, and automated document generation. All citation data is stored in standard BibTeX format as the canonical source, with on-demand conversion to other formats for specific manuscript requirements.
+
+## Citation Retrieval
+
+### From DOI
+
+```python
+import requests
+
+def get_bibtex_from_doi(doi):
+    """Retrieve BibTeX entry from a DOI via CrossRef."""
+    url = f"https://doi.org/{doi}"
+    headers = {"Accept": "application/x-bibtex"}
+    response = requests.get(url, headers=headers, allow_redirects=True)
+    if response.status_code == 200:
+        return response.text
+    return None
+
+# Example
+bibtex = get_bibtex_from_doi("10.1038/s41586-021-03819-2")
+print(bibtex)
+# @article{Jumper_2021,
+#   title={Highly accurate protein structure prediction with AlphaFold},
+#   author={Jumper, John and Evans, Richard and ...},
+#   journal={Nature},
+#   volume={596},
+#   pages={583--589},
+#   year={2021},
+#   publisher={Springer}
+# }
+```
+
+### From Semantic Scholar
+
+```python
+def get_citation_from_s2(paper_id):
+    """Retrieve citation data from Semantic Scholar API."""
+    url = f"https://api.semanticscholar.org/graph/v1/paper/{paper_id}"
+    params = {"fields": "title,authors,year,venue,doi,citationCount,externalIds"}
+    response = requests.get(url, params=params)
+    if response.status_code == 200:
+        data = response.json()
+        return format_as_bibtex(data)
+    return None
+
+def format_as_bibtex(s2_data):
+    """Convert Semantic Scholar data to BibTeX."""
+    authors = s2_data.get("authors", [])
+    author_str = " and ".join(a["name"] for a in authors)
+    first_author = authors[0]["name"].split()[-1] if authors else "Unknown"
+    year = s2_data.get("year", "")
+    key = f"{first_author}_{year}"
+
+    return f"""@article{{{key},
+  title={{{s2_data.get('title', '')}}},
+  author={{{author_str}}},
+  year={{{year}}},
+  journal={{{s2_data.get('venue', '')}}},
+  doi={{{s2_data.get('doi', '')}}}
+}}"""
+```
+
+### From arXiv ID
+
+```python
+def get_bibtex_from_arxiv(arxiv_id):
+    """Retrieve BibTeX from arXiv."""
+    import feedparser
+    url = f"http://export.arxiv.org/api/query?id_list={arxiv_id}"
+    feed = feedparser.parse(url)
+    if feed.entries:
+        entry = feed.entries[0]
+        authors = " and ".join(a["name"] for a in entry.authors)
+        first_author = entry.authors[0]["name"].split()[-1]
+        year = entry.published[:4]
+        return f"""@article{{{first_author}_{year},
+  title={{{entry.title.replace(chr(10), ' ')}}},
+  author={{{authors}}},
+  year={{{year}}},
+  journal={{arXiv preprint arXiv:{arxiv_id}}},
+  url={{https://arxiv.org/abs/{arxiv_id}}}
+}}"""
+    return None
+```
+
+## Format Conversion
+
+### BibTeX to APA 7th
+
+```python
+def bibtex_to_apa7(entry):
+    """Convert a parsed BibTeX entry to APA 7th edition format."""
+    authors = format_apa_authors(entry["author"])
+    year = entry.get("year", "n.d.")
+    title = entry["title"]
+    journal = entry.get("journal", "")
+    volume = entry.get("volume", "")
+    issue = entry.get("number", "")
+    pages = entry.get("pages", "")
+    doi = entry.get("doi", "")
+
+    # Article format
+    citation = f"{authors} ({year}). {title}. "
+    if journal:
+        citation += f"*{journal}*"
+        if volume:
+            citation += f", *{volume}*"
+        if issue:
+            citation += f"({issue})"
+        if pages:
+            citation += f", {pages}"
+        citation += "."
+    if doi:
+        citation += f" https://doi.org/{doi}"
+
+    return citation
+
+def format_apa_authors(author_string):
+    """Format author names in APA style: Last, F. M."""
+    authors = [a.strip() for a in author_string.split(" and ")]
+    formatted = []
+    for author in authors:
+        parts = author.split(", ") if ", " in author else author.rsplit(" ", 1)[::-1]
+        if len(parts) >= 2:
+            last = parts[0]
+            firsts = parts[1].split()
+            initials = " ".join(f"{f[0]}." for f in firsts)
+            formatted.append(f"{last}, {initials}")
+        else:
+            formatted.append(parts[0])
+
+    if len(formatted) == 1:
+        return formatted[0]
+    elif len(formatted) == 2:
+        return f"{formatted[0]}, & {formatted[1]}"
+    elif len(formatted) <= 20:
+        return ", ".join(formatted[:-1]) + f", & {formatted[-1]}"
+    else:
+        return ", ".join(formatted[:19]) + f", ... {formatted[-1]}"
+```
+
+### Format Examples
+
+The same reference in different styles:
+
+**BibTeX:**
+```bibtex
+@article{Jumper_2021,
+  title={Highly accurate protein structure prediction with AlphaFold},
+  author={Jumper, John and Evans, Richard and Pritzel, Alexander},
+  journal={Nature},
+  volume={596},
+  pages={583--589},
+  year={2021},
+  doi={10.1038/s41586-021-03819-2}
+}
+```
+
+**APA 7th:**
+Jumper, J., Evans, R., & Pritzel, A. (2021). Highly accurate protein structure prediction with AlphaFold. *Nature*, *596*, 583-589. https://doi.org/10.1038/s41586-021-03819-2
+
+**MLA 9th:**
+Jumper, John, Richard Evans, and Alexander Pritzel. "Highly Accurate Protein Structure Prediction with AlphaFold." *Nature*, vol. 596, 2021, pp. 583-89.
+
+**Chicago (Author-Date):**
+Jumper, John, Richard Evans, and Alexander Pritzel. 2021. "Highly Accurate Protein Structure Prediction with AlphaFold." *Nature* 596: 583-89.
+
+**Vancouver:**
+Jumper J, Evans R, Pritzel A. Highly accurate protein structure prediction with AlphaFold. Nature. 2021;596:583-9.
+
+**IEEE:**
+J. Jumper, R. Evans, and A. Pritzel, "Highly accurate protein structure prediction with AlphaFold," *Nature*, vol. 596, pp. 583-589, 2021.
+
+## Deduplication
+
+### Detecting Duplicate Entries
+
+```python
+from difflib import SequenceMatcher
+
+def find_duplicates(bib_entries, threshold=0.85):
+    """Find duplicate BibTeX entries by title similarity."""
+    duplicates = []
+    titles = [(key, normalize_title(entry["title"]))
+              for key, entry in bib_entries.items()]
+
+    for i in range(len(titles)):
+        for j in range(i + 1, len(titles)):
+            similarity = SequenceMatcher(
+                None, titles[i][1], titles[j][1]
+            ).ratio()
+            if similarity >= threshold:
+                duplicates.append({
+                    "entry_a": titles[i][0],
+                    "entry_b": titles[j][0],
+                    "similarity": similarity
+                })
+    return duplicates
+
+def normalize_title(title):
+    """Normalize title for comparison."""
+    import re
+    title = title.lower()
+    title = re.sub(r'[{}\\]', '', title)  # Remove LaTeX formatting
+    title = re.sub(r'[^a-z0-9\s]', '', title)  # Remove punctuation
+    title = ' '.join(title.split())  # Normalize whitespace
+    return title
+
+def merge_duplicates(entry_a, entry_b):
+    """Merge two duplicate entries, preferring the more complete one."""
+    merged = {}
+    all_fields = set(list(entry_a.keys()) + list(entry_b.keys()))
+    for field in all_fields:
+        val_a = entry_a.get(field, "")
+        val_b = entry_b.get(field, "")
+        # Prefer the longer (more complete) value
+        merged[field] = val_a if len(str(val_a)) >= len(str(val_b)) else val_b
+    return merged
+```
+
+## Validation
+
+### CrossRef Validation
+
+```python
+def validate_citation(doi):
+    """Validate a citation against CrossRef metadata."""
+    url = f"https://api.crossref.org/works/{doi}"
+    response = requests.get(url)
+    if response.status_code != 200:
+        return {"valid": False, "error": "DOI not found in CrossRef"}
+
+    data = response.json()["message"]
+    return {
+        "valid": True,
+        "title": data.get("title", [None])[0],
+        "authors": [f"{a.get('family', '')}, {a.get('given', '')}"
+                    for a in data.get("author", [])],
+        "year": data.get("published-print", {}).get("date-parts", [[None]])[0][0],
+        "journal": data.get("container-title", [None])[0],
+        "type": data.get("type", "unknown")
+    }
+```
+
+### Common Citation Errors
+
+| Error | Detection | Fix |
+|-------|-----------|-----|
+| Missing DOI | Check `doi` field is empty | Query CrossRef by title |
+| Wrong year | Compare against CrossRef | Use CrossRef year |
+| Author name variants | Fuzzy match against ORCID | Standardize to ORCID name |
+| Duplicate entries | Title similarity > 85% | Merge into single entry |
+| Broken URL | HTTP HEAD request returns 4xx/5xx | Update or remove URL |
+| Incomplete entry | Missing required fields for style | Retrieve from DOI |
+
+## Integration with Writing Tools
+
+### LaTeX
+
+```latex
+% In document preamble
+\usepackage[backend=biber,style=apa]{biblatex}
+\addbibresource{references.bib}
+
+% In text
+\textcite{Jumper_2021} showed that...
+As demonstrated by previous work \parencite{Jumper_2021}...
+
+% At end of document
+\printbibliography
+```
+
+### Pandoc Markdown
+
+```markdown
+Previous work [@Jumper_2021] showed that...
+
+## References
+```
+
+```bash
+pandoc paper.md --citeproc --bibliography=references.bib \
+  --csl=apa.csl -o paper.pdf
+```
+
+## References
+
+- CrossRef API: https://api.crossref.org
+- Semantic Scholar API: https://api.semanticscholar.org
+- APA 7th Edition Manual: https://apastyle.apa.org/products/publication-manual-7th-edition
+- BibTeX documentation: http://www.bibtex.org
+- CSL styles repository: https://github.com/citation-style-language/styles