@wentorai/research-plugins 1.0.0 → 1.2.0

package/skills/literature/fulltext/core-api-guide/SKILL.md

@@ -0,0 +1,144 @@
+---
+name: core-api-guide
+description: "Search and retrieve open access research papers via CORE aggregator"
+metadata:
+  openclaw:
+    emoji: "🔬"
+    category: "literature"
+    subcategory: "fulltext"
+    keywords: ["open-access", "fulltext", "research-papers", "aggregator", "CORE"]
+    source: "https://core.ac.uk/documentation/api"
+---
+
+# CORE API Guide
+
+## Overview
+
+CORE (COnnecting REpositories) is the world's largest aggregator of open access research papers, providing access to over 130 million articles harvested from thousands of data providers worldwide. The CORE API enables programmatic search, retrieval, and analysis of scholarly full-text content across repositories, journals, and preprint servers.
+
+The API is particularly valuable for researchers conducting systematic reviews, bibliometric analyses, and literature mining tasks. Unlike many scholarly APIs that only provide metadata, CORE specializes in delivering full-text content, making it essential for text mining and natural language processing workflows in academic research.
+
+CORE's v3 API provides a RESTful interface with JSON responses, supporting complex search queries with Boolean operators, field-specific filtering, and batch operations. It is free for non-commercial academic use, though an API key is required to access the service.
+
+## Authentication
+
+CORE requires a free API key for all requests. Register at https://core.ac.uk/services/api to obtain one.
+
+Always store your API key in an environment variable and reference it in requests:
+
+```bash
+export CORE_API_KEY=$CORE_API_KEY
+```
+
+Pass the key via the `Authorization` header:
+
+```bash
+curl -H "Authorization: Bearer $CORE_API_KEY" \
+  "https://api.core.ac.uk/v3/search/works?q=machine+learning"
+```
+
+## Core Endpoints
+
+### Search Works
+
+Search across the entire CORE corpus with full-text and metadata queries.
+
+```
+GET https://api.core.ac.uk/v3/search/works?q={query}&limit={n}&offset={n}
+```
+
+**Parameters:**
+- `q` (required): Search query string, supports Boolean operators (AND, OR, NOT)
+- `limit`: Number of results (default 10, max 100)
+- `offset`: Pagination offset
+- `entity_type`: Filter by type (e.g., `journal-article`, `preprint`)
+
+**Example: Search for climate change papers with full text:**
+
+```bash
+curl -s -H "Authorization: Bearer $CORE_API_KEY" \
+  "https://api.core.ac.uk/v3/search/works?q=climate+change+adaptation&limit=5" \
+  | python3 -m json.tool
+```
+
+**Python example:**
+
+```python
+import requests
+import os
+
+headers = {"Authorization": f"Bearer {os.environ['CORE_API_KEY']}"}
+params = {
+    "q": "deep learning AND medical imaging",
+    "limit": 20,
+    "offset": 0
+}
+resp = requests.get("https://api.core.ac.uk/v3/search/works", headers=headers, params=params)
+data = resp.json()
+
+for result in data.get("results", []):
+    print(f"Title: {result.get('title')}")
+    print(f"DOI: {result.get('doi')}")
+    print(f"Year: {result.get('yearPublished')}")
+    print(f"Full text length: {len(result.get('fullText', ''))}")
+    print("---")
+```
+
+### Get Work by ID
+
+Retrieve a specific paper by its CORE ID or DOI.
+
+```
+GET https://api.core.ac.uk/v3/works/{core_id}
+```
+
+```bash
+curl -s -H "Authorization: Bearer $CORE_API_KEY" \
+  "https://api.core.ac.uk/v3/works/doi:10.1234/example.doi" \
+  | python3 -m json.tool
+```
+
+### Batch Retrieval
+
+Retrieve multiple works in a single request using POST with a list of IDs.
+
+```bash
+curl -s -X POST -H "Authorization: Bearer $CORE_API_KEY" \
+  -H "Content-Type: application/json" \
+  -d '[12345, 67890, 11111]' \
+  "https://api.core.ac.uk/v3/works"
+```
+
+### Search Data Providers
+
+List or search CORE's data providers (repositories, journals).
+
+```
+GET https://api.core.ac.uk/v3/data-providers?q={query}
+```
+
+## Common Research Patterns
+
+**Systematic Literature Review:** Use Boolean queries to replicate a search strategy across the full-text corpus. Combine with date filters to identify papers within a specific time window, then export results for screening in tools like Rayyan or Covidence.
+
+**Full-Text Mining:** Retrieve full-text content programmatically for NLP pipelines. Extract named entities, key phrases, or citation contexts at scale across thousands of papers.
+
+**Repository Coverage Analysis:** Query data providers to understand which institutional repositories contribute to a specific field, useful for bibliometric and open-access policy research.
+
+**Trend Detection:** Run time-series queries for specific terms and track publication volume over years to identify emerging research fronts.
+
+## Rate Limits and Best Practices
+
+- **Free tier:** 150 requests per 15-minute window (10 req/min effective)
+- **Batch endpoints:** Use batch retrieval for multiple IDs to minimize request count
+- **Pagination:** Always use `offset` and `limit` for large result sets; do not fetch all results in one call
+- **Caching:** Cache responses locally for repeat queries, especially for static metadata
+- **Respect robots.txt:** When downloading full texts, add delays between requests
+- **Error handling:** The API returns standard HTTP status codes; implement exponential backoff for 429 (rate limit) responses
+
+## References
+
+- CORE API v3 Documentation: https://core.ac.uk/documentation/api
+- CORE Dashboard and Key Registration: https://core.ac.uk/services/api
+- CORE Data Dumps (for bulk access): https://core.ac.uk/documentation/dataset
+- CORE GitHub: https://github.com/oacore

package/skills/literature/fulltext/dataverse-api/SKILL.md

@@ -0,0 +1,215 @@
+---
+name: dataverse-api
+description: "Deposit and discover research datasets via Harvard Dataverse API"
+metadata:
+  openclaw:
+    emoji: "🗄️"
+    category: "literature"
+    subcategory: "fulltext"
+    keywords: ["Dataverse", "research data", "data repository", "Harvard", "dataset deposit", "data sharing"]
+    source: "https://dataverse.org/"
+---
+
+# Harvard Dataverse API
+
+## Overview
+
+Dataverse is an open-source research data repository platform developed by Harvard IQSS, hosting 150K+ datasets across 80+ installations worldwide. The Harvard Dataverse alone has 130K+ datasets covering social science, natural science, and humanities. The API supports search, metadata retrieval, file download, and dataset deposit. Free, no authentication for read access.
+
+## API Endpoints
+
+### Base URL
+
+```
+https://dataverse.harvard.edu/api
+```
+
+### Search
+
+```bash
+# Search datasets
+curl "https://dataverse.harvard.edu/api/search?q=climate+change&type=dataset&per_page=20"
+
+# Search files within datasets
+curl "https://dataverse.harvard.edu/api/search?q=temperature+data&type=file&per_page=20"
+
+# Filter by subject
+curl "https://dataverse.harvard.edu/api/search?q=survey+data&type=dataset&\
+fq=subject_ss:\"Social Sciences\""
+
+# Filter by publication date
+curl "https://dataverse.harvard.edu/api/search?q=genomics&type=dataset&\
+fq=dateSort:[2024-01-01T00:00:00Z TO *]"
+
+# Sort by relevance or date
+curl "https://dataverse.harvard.edu/api/search?q=machine+learning&type=dataset&\
+sort=date&order=desc"
+```
+
+### Get Dataset Metadata
+
+```bash
+# By persistent ID (DOI)
+curl "https://dataverse.harvard.edu/api/datasets/:persistentId/?persistentId=doi:10.7910/DVN/EXAMPLE"
+
+# By dataset ID
+curl "https://dataverse.harvard.edu/api/datasets/12345"
+
+# Get dataset versions
+curl "https://dataverse.harvard.edu/api/datasets/:persistentId/versions?persistentId=doi:10.7910/DVN/EXAMPLE"
+```
+
+### Download Files
+
+```bash
+# Download a specific file by ID
+curl -O "https://dataverse.harvard.edu/api/access/datafile/67890"
+
+# Download with original format
+curl -O "https://dataverse.harvard.edu/api/access/datafile/67890?format=original"
+
+# Download all files in a dataset (as zip)
+curl -O "https://dataverse.harvard.edu/api/access/dataset/:persistentId/?persistentId=doi:10.7910/DVN/EXAMPLE"
+```
+
+### Query Parameters (Search)
+
+| Parameter | Description | Example |
+|-----------|-------------|---------|
+| `q` | Search query | `q=voter+turnout` |
+| `type` | Item type | `dataset`, `file`, `dataverse` |
+| `per_page` | Results per page (max 1000) | `per_page=50` |
+| `start` | Pagination offset | `start=50` |
+| `sort` | Sort field | `name`, `date` |
+| `order` | Sort order | `asc`, `desc` |
+| `fq` | Filter query (Solr) | `fq=subject_ss:"Medicine"` |
+
+## Response Structure
+
+```json
+{
+  "status": "OK",
+  "data": {
+    "q": "climate change",
+    "total_count": 2450,
+    "items": [
+      {
+        "name": "Global Temperature Dataset 2024",
+        "type": "dataset",
+        "url": "https://doi.org/10.7910/DVN/EXAMPLE",
+        "global_id": "doi:10.7910/DVN/EXAMPLE",
+        "description": "Monthly global temperature anomalies...",
+        "published_at": "2024-03-15",
+        "publisher": "Harvard Dataverse",
+        "subjects": ["Earth and Environmental Sciences"],
+        "fileCount": 12,
+        "citation": "Smith, J. (2024). Global Temperature Dataset..."
+      }
+    ]
+  }
+}
+```
+
+## Python Usage
+
+```python
+import requests
+
+BASE_URL = "https://dataverse.harvard.edu/api"
+
+
+def search_datasets(query: str, per_page: int = 20,
+                    subject: str = None) -> list:
+    """Search Harvard Dataverse for datasets."""
+    params = {
+        "q": query,
+        "type": "dataset",
+        "per_page": per_page,
+        "sort": "date",
+        "order": "desc",
+    }
+    if subject:
+        params["fq"] = f'subject_ss:"{subject}"'
+
+    resp = requests.get(f"{BASE_URL}/search", params=params)
+    resp.raise_for_status()
+    data = resp.json()
+
+    results = []
+    for item in data.get("data", {}).get("items", []):
+        results.append({
+            "name": item.get("name"),
+            "doi": item.get("global_id"),
+            "description": item.get("description", "")[:300],
+            "published": item.get("published_at"),
+            "subjects": item.get("subjects", []),
+            "files": item.get("fileCount", 0),
+            "url": item.get("url"),
+        })
+    return results
+
+
+def get_dataset_files(doi: str) -> list:
+    """List files in a dataset."""
+    resp = requests.get(
+        f"{BASE_URL}/datasets/:persistentId/",
+        params={"persistentId": doi},
+    )
+    resp.raise_for_status()
+    data = resp.json().get("data", {})
+
+    files = []
+    version = data.get("latestVersion", {})
+    for f in version.get("files", []):
+        df = f.get("dataFile", {})
+        files.append({
+            "id": df.get("id"),
+            "filename": df.get("filename"),
+            "size": df.get("filesize"),
+            "content_type": df.get("contentType"),
+            "md5": df.get("md5"),
+        })
+    return files
+
+
+def download_file(file_id: int, output_path: str):
+    """Download a file from Dataverse."""
+    resp = requests.get(
+        f"{BASE_URL}/access/datafile/{file_id}",
+        stream=True,
+    )
+    resp.raise_for_status()
+    with open(output_path, "wb") as f:
+        for chunk in resp.iter_content(chunk_size=8192):
+            f.write(chunk)
+
+
+# Example: find social science datasets
+datasets = search_datasets("income inequality",
+                           subject="Social Sciences")
+for ds in datasets:
+    print(f"[{ds['published']}] {ds['name']} ({ds['files']} files)")
+    print(f"  DOI: {ds['doi']}")
+
+# Example: list files in a dataset
+# files = get_dataset_files("doi:10.7910/DVN/EXAMPLE")
+# for f in files:
+#     print(f"  {f['filename']} ({f['size']} bytes)")
+```
+
+## Other Dataverse Installations
+
+| Installation | URL | Focus |
+|-------------|-----|-------|
+| Harvard Dataverse | dataverse.harvard.edu | Multi-discipline |
+| UNC Dataverse | dataverse.unc.edu | Social science |
+| AUSSDA | data.aussda.at | Austrian social science |
+| Borealis (Canada) | borealisdata.ca | Canadian research |
+| DataverseNL | dataverse.nl | Dutch research |
+
+## References
+
+- [Harvard Dataverse](https://dataverse.harvard.edu/)
+- [Dataverse API Guide](https://guides.dataverse.org/en/latest/api/)
+- [Dataverse Project](https://dataverse.org/)
+- King, G. (2007). "An Introduction to the Dataverse Network." *Sociological Methods & Research* 36(2).

package/skills/literature/fulltext/hal-archive-api/SKILL.md

@@ -0,0 +1,218 @@
+---
+name: hal-archive-api
+description: "Access French and European research via the HAL open archive API"
+metadata:
+  openclaw:
+    emoji: "🇫🇷"
+    category: "literature"
+    subcategory: "fulltext"
+    keywords: ["HAL", "French research", "open archive", "CNRS", "European research", "institutional repository"]
+    source: "https://api.archives-ouvertes.fr/"
+---
+
+# HAL Open Archive API
+
+## Overview
+
+HAL (Hyper Articles en Ligne) is France's national open archive for scholarly deposits. Managed by CNRS, it hosts 4M+ full-text documents from French research institutions and international collaborators. The API provides Solr-based search with full metadata, PDF links, and OAI-PMH harvesting. Free, no authentication required.
+
+## API Endpoints
+
+### Search API
+
+```bash
+# Keyword search
+curl "https://api.archives-ouvertes.fr/search/?q=machine+learning&rows=20&wt=json"
+
+# Search specific fields
+curl "https://api.archives-ouvertes.fr/search/?q=title_s:\"deep learning\"&wt=json"
+
+# Filter by document type
+curl "https://api.archives-ouvertes.fr/search/?q=neural+networks&\
+fq=docType_s:ART&rows=20&wt=json"
+
+# Filter by year and language
+curl "https://api.archives-ouvertes.fr/search/?q=climate+change&\
+fq=producedDateY_i:[2023 TO 2026]&fq=language_s:en&wt=json"
+
+# Filter by institution
+curl "https://api.archives-ouvertes.fr/search/?q=robotics&\
+fq=structId_i:441569&wt=json"
+
+# Return specific fields
+curl "https://api.archives-ouvertes.fr/search/?q=CRISPR&\
+fl=halId_s,title_s,authFullName_s,producedDateY_i,uri_s,files_s&wt=json"
+```
+
+### Search Fields
+
+| Field | Description | Example |
+|-------|-------------|---------|
+| `title_s` | Title | `title_s:"attention mechanism"` |
+| `authFullName_s` | Author name | `authFullName_s:"Yann LeCun"` |
+| `abstract_s` | Abstract | `abstract_s:transformer` |
+| `keyword_s` | Keywords | `keyword_s:"natural language"` |
+| `producedDateY_i` | Year | `producedDateY_i:2024` |
+| `docType_s` | Document type | `docType_s:ART` |
+| `language_s` | Language | `language_s:en` |
+| `domain_s` | Domain/subject | `domain_s:info.info-ai` |
+| `journalTitle_s` | Journal name | `journalTitle_s:"Nature"` |
+| `structId_i` | Institution ID | Lab/university ID |
+
+### Document Types
+
+| Code | Type |
+|------|------|
+| `ART` | Journal article |
+| `COMM` | Conference paper |
+| `THESE` | PhD thesis |
+| `HDR` | Habilitation thesis |
+| `REPORT` | Report |
+| `COUV` | Book chapter |
+| `OUV` | Book |
+| `POSTER` | Poster |
+| `UNDEFINED` | Preprint/other |
+
+### Query Parameters
+
+| Parameter | Description |
+|-----------|-------------|
+| `q` | Solr query |
+| `fq` | Filter query |
+| `fl` | Fields to return |
+| `rows` | Results per page (max 10000) |
+| `start` | Pagination offset |
+| `sort` | Sort order (e.g., `producedDateY_i desc`) |
+| `wt` | Format: `json`, `xml`, `csv` |
+
+## Response Structure
+
+```json
+{
+  "response": {
+    "numFound": 12500,
+    "start": 0,
+    "docs": [
+      {
+        "halId_s": "hal-01234567",
+        "title_s": ["Deep Learning for Climate Modeling"],
+        "authFullName_s": ["Marie Dupont", "Jean Martin"],
+        "producedDateY_i": 2024,
+        "docType_s": "ART",
+        "journalTitle_s": "Environmental Modelling",
+        "uri_s": "https://hal.science/hal-01234567",
+        "files_s": ["https://hal.science/hal-01234567/document"],
+        "domain_s": ["sde.es", "info.info-ai"],
+        "abstract_s": ["We propose a novel deep learning approach..."],
+        "language_s": ["en"]
+      }
+    ]
+  }
+}
+```
+
+## Python Usage
+
+```python
+import requests
+
+BASE_URL = "https://api.archives-ouvertes.fr/search/"
+
+
+def search_hal(query: str, rows: int = 20,
+               doc_type: str = None, from_year: int = None,
+               language: str = None) -> list:
+    """Search HAL open archive."""
+    params = {
+        "q": query,
+        "wt": "json",
+        "rows": rows,
+        "fl": "halId_s,title_s,authFullName_s,producedDateY_i,"
+              "uri_s,files_s,docType_s,journalTitle_s,abstract_s",
+        "sort": "producedDateY_i desc",
+    }
+
+    fq = []
+    if doc_type:
+        fq.append(f"docType_s:{doc_type}")
+    if from_year:
+        fq.append(f"producedDateY_i:[{from_year} TO 2030]")
+    if language:
+        fq.append(f"language_s:{language}")
+    if fq:
+        params["fq"] = fq
+
+    resp = requests.get(BASE_URL, params=params)
+    resp.raise_for_status()
+    data = resp.json()
+
+    results = []
+    for doc in data.get("response", {}).get("docs", []):
+        title = doc.get("title_s", [""])[0] if isinstance(
+            doc.get("title_s"), list) else doc.get("title_s", "")
+        results.append({
+            "hal_id": doc.get("halId_s"),
+            "title": title,
+            "authors": doc.get("authFullName_s", []),
+            "year": doc.get("producedDateY_i"),
+            "type": doc.get("docType_s"),
+            "journal": doc.get("journalTitle_s"),
+            "url": doc.get("uri_s"),
+            "pdf": doc.get("files_s", [None])[0],
+        })
+    return results
+
+
+def search_theses(topic: str, from_year: int = 2020) -> list:
+    """Find French PhD theses on a topic."""
+    return search_hal(topic, rows=50, doc_type="THESE",
+                      from_year=from_year)
+
+
+def get_institution_publications(struct_id: int,
+                                 from_year: int = 2023) -> list:
+    """Get publications from a specific institution."""
+    params = {
+        "q": "*:*",
+        "fq": [f"structId_i:{struct_id}",
+               f"producedDateY_i:[{from_year} TO 2030]"],
+        "wt": "json",
+        "rows": 100,
+        "fl": "halId_s,title_s,authFullName_s,producedDateY_i,docType_s",
+        "sort": "producedDateY_i desc",
+    }
+    resp = requests.get(BASE_URL, params=params)
+    resp.raise_for_status()
+    return resp.json().get("response", {}).get("docs", [])
+
+
+# Example: find recent French AI research
+papers = search_hal("intelligence artificielle", from_year=2024)
+for p in papers:
+    pdf = " [PDF]" if p["pdf"] else ""
+    print(f"[{p['year']}] {p['title']}{pdf}")
+
+# Example: find PhD theses on NLP
+theses = search_theses("natural language processing")
+for t in theses:
+    print(f"{t['title']} — {', '.join(t['authors'][:2])}")
+```
+
+## HAL Domains
+
+| Code | Domain |
+|------|--------|
+| `info` | Computer Science |
+| `math` | Mathematics |
+| `phys` | Physics |
+| `sde` | Environmental Sciences |
+| `sdv` | Life Sciences |
+| `shs` | Social Sciences & Humanities |
+| `chim` | Chemistry |
+| `spi` | Engineering Sciences |
+
+## References
+
+- [HAL Open Archive](https://hal.science/)
+- [HAL API Documentation](https://api.archives-ouvertes.fr/docs)
+- [HAL Search Guide](https://doc.archives-ouvertes.fr/en/)