@wentorai/research-plugins 1.0.0 → 1.1.0

package/skills/literature/discovery/semantic-paper-radar/SKILL.md

@@ -0,0 +1,144 @@
+---
+name: semantic-paper-radar
+description: "Semantic literature discovery and synthesis using embeddings"
+metadata:
+  openclaw:
+    emoji: "📡"
+    category: "literature"
+    subcategory: "discovery"
+    keywords: ["semantic search", "embeddings", "literature synthesis", "paper discovery", "vector search", "knowledge mapping"]
+    source: "https://github.com/mukulpatnaik/researchgpt"
+---
+
+# Semantic Paper Radar
+
+## Overview
+
+Traditional literature search relies on keyword matching—you find papers that contain the exact terms you search for. Semantic paper discovery goes further by understanding the meaning of research content and finding papers that are conceptually related, even when they use different terminology. This is especially powerful for interdisciplinary research, where the same idea may be expressed in completely different vocabularies across fields.
+
+The Semantic Paper Radar skill provides methods for using embedding-based semantic search, vector databases, and AI-powered synthesis to build a comprehensive, continuously updated view of the literature relevant to your research. It enables you to discover papers you would never find through keyword search alone and to synthesize findings across large bodies of work.
+
+This skill covers setting up a personal semantic search index over your paper collection, querying public semantic search APIs, and using LLM-powered analysis to extract themes and connections from clusters of related papers.
+
+## Semantic Search Fundamentals
+
+### How Embedding-Based Search Works
+
+Semantic search represents both your query and each paper as dense numerical vectors (embeddings) in a high-dimensional space. Papers whose embeddings are close to your query's embedding are semantically similar, regardless of the specific words used.
+
+Key components:
+- **Embedding model**: Converts text to vectors. Models like SPECTER2, SciBERT, or general-purpose models like `text-embedding-3-small` work well for academic text.
+- **Vector database**: Stores and indexes embeddings for fast similarity search. Options include ChromaDB (local), Qdrant, Pinecone, or Weaviate.
+- **Similarity metric**: Cosine similarity is standard for comparing text embeddings.
+
+### Using Semantic Scholar's Embedding Search
+
+Semantic Scholar provides pre-computed SPECTER embeddings for millions of papers. You can use their search API for semantic queries:
+
+```bash
+# Semantic search via the Semantic Scholar API
+curl "https://api.semanticscholar.org/graph/v1/paper/search?query=attention+mechanisms+for+graph+neural+networks&fields=title,abstract,year,citationCount&limit=20"
+```
+
+The search endpoint uses semantic matching, not just keyword matching. A query like "methods for handling missing values in longitudinal studies" will find papers about imputation techniques, dropout analysis, and panel data methods even if they do not use the phrase "missing values."
+
+### Building a Personal Semantic Index
+
+For deeper control, build a local semantic search index over your own paper collection:
+
+```python
+import chromadb
+from sentence_transformers import SentenceTransformer
+
+# Initialize
+model = SentenceTransformer("allenai/specter2")
+client = chromadb.PersistentClient(path="./paper_index")
+collection = client.get_or_create_collection(
+    name="my_papers",
+    metadata={"hnsw:space": "cosine"}
+)
+
+# Index a paper
+abstract = "We propose a novel attention mechanism for graph neural networks..."
+embedding = model.encode(abstract).tolist()
+collection.add(
+    documents=[abstract],
+    embeddings=[embedding],
+    metadatas=[{"title": "Graph Attention v2", "year": 2025, "arxiv_id": "2501.xxxxx"}],
+    ids=["paper_001"]
+)
+
+# Query
+results = collection.query(
+    query_embeddings=[model.encode("message passing in GNNs").tolist()],
+    n_results=10
+)
+```
+
+This local index lets you search across all papers you have collected using natural language queries. As you add more papers, the index becomes a personalized discovery tool tuned to your specific research interests.
+
+## Discovery Workflows
+
+### Concept Expansion Radar
+
+Use semantic search to expand your awareness beyond your current reading:
+
+1. **Seed**: Take the abstract of your current paper (or a paragraph describing your research question).
+2. **Search**: Run it as a semantic query against a large corpus (Semantic Scholar, OpenAlex, or your local index).
+3. **Filter**: Remove papers you have already read. Sort by a combination of semantic similarity and recency.
+4. **Cluster**: Group the top 50 results into thematic clusters using k-means or HDBSCAN on their embeddings.
+5. **Explore clusters**: Each cluster represents a related subtopic. Read the most-cited paper in each cluster to understand the connection to your work.
+
+### Cross-Disciplinary Bridge Detection
+
+Semantic search excels at finding papers from other fields that address similar problems:
+
+1. Describe your research problem in plain, non-technical language.
+2. Run this as a semantic query without restricting to your field's journals or categories.
+3. Review results from unexpected fields—these are potential interdisciplinary connections.
+4. For each bridge paper, check its reference list for more domain-specific work in that field.
+
+### Novelty Radar
+
+Set up periodic semantic searches to detect new papers in your area:
+
+1. Define 3-5 "concept vectors" by encoding descriptions of your core research interests.
+2. Weekly, search against newly published papers (last 7 days) from arXiv or Semantic Scholar.
+3. Rank new papers by maximum similarity to any of your concept vectors.
+4. Papers above your similarity threshold enter your reading queue automatically.
+
+## Semantic Synthesis
+
+Once you have discovered a cluster of related papers, use AI-assisted synthesis to extract insights across the collection:
+
+### Theme Extraction
+
+Feed the abstracts of a cluster of papers to an LLM and ask for:
+- Common themes and findings across the papers
+- Points of disagreement or contradiction
+- Methodological trends (what approaches are gaining vs. losing popularity)
+- Open questions that none of the papers fully address
+
+### Evidence Mapping
+
+Create a structured evidence map from your semantic cluster:
+
+| Theme | Supporting Papers | Contradicting Papers | Strength of Evidence |
+|-------|-------------------|----------------------|---------------------|
+| Theme A | [1], [3], [7] | [5] | Strong |
+| Theme B | [2], [4] | None | Moderate |
+| Theme C | [6] | [1], [8] | Contested |
+
+This provides a bird's-eye view of where consensus exists and where debates remain open.
+
+### Gap Identification
+
+Compare your research question against the semantic landscape of existing work. Regions of embedding space where your query falls but few papers exist represent potential research gaps—areas where your contribution would be most novel.
+
+## References
+
+- Semantic Scholar API: https://api.semanticscholar.org
+- SPECTER2 model: https://huggingface.co/allenai/specter2
+- ChromaDB: https://www.trychroma.com
+- ResearchGPT: https://github.com/mukulpatnaik/researchgpt
+- OpenAlex: https://openalex.org

package/skills/literature/discovery/zotero-arxiv-daily-guide/SKILL.md

@@ -0,0 +1,94 @@
+---
+name: zotero-arxiv-daily-guide
+description: "Guide to Zotero arXiv Daily for personalized daily paper recommendations"
+metadata:
+  openclaw:
+    emoji: "📰"
+    category: literature
+    subcategory: discovery
+    keywords: ["zotero", "arxiv", "daily-papers", "recommendations", "preprint", "discovery"]
+    source: "https://github.com/TechPenguineer/zotero-arxiv-daily"
+---
+
+# Zotero arXiv Daily Guide
+
+## Overview
+
+Zotero arXiv Daily is a popular Zotero plugin with over 5,000 GitHub stars that delivers personalized daily paper recommendations from arXiv directly into your Zotero library. By analyzing the papers you already have in your collections, the plugin identifies your research interests and surfaces new preprints that are most relevant to your work.
+
+The challenge of staying current with preprint literature is well known to researchers. ArXiv publishes thousands of new papers daily across dozens of categories, and manually scanning listings or relying solely on keyword alerts often results in information overload or missed relevant work. Zotero arXiv Daily addresses this by using your existing library as a profile of your interests, producing recommendations that improve as your library grows.
+
+The plugin integrates naturally into the Zotero workflow. Recommended papers appear in a dedicated collection where you can review titles and abstracts, save promising papers to your working collections, and dismiss irrelevant suggestions. Over time the recommendation engine learns from your accept and dismiss decisions, refining its model of your interests.
+
+## Installation and Setup
+
+Install Zotero arXiv Daily through the standard Zotero plugin process:
+
+1. Download the latest `.xpi` release from https://github.com/TechPenguineer/zotero-arxiv-daily/releases
+2. In Zotero, go to Tools > Add-ons > gear icon > Install Add-on From File
+3. Select the `.xpi` file and restart Zotero
+
+Configure the plugin after installation:
+
+- Open Zotero Preferences > arXiv Daily
+- Select the arXiv categories relevant to your research (e.g., cs.AI, cs.CL, stat.ML, physics.comp-ph)
+- Choose which Zotero collections to use as the basis for recommendations (your core research collections work best)
+- Set the number of daily recommendations (10-30 is typical)
+- Configure the schedule for fetching new recommendations (daily at a specific time or on-demand)
+- Set up a dedicated Zotero collection where recommendations will appear
+
+For enhanced recommendation quality, ensure your library collections are well-organized. The algorithm performs better when it can distinguish between your core research interests and peripheral references. Consider creating a dedicated collection of your most representative papers to serve as the recommendation seed.
+
+## Core Features
+
+**Personalized Recommendations**: The plugin analyzes titles, abstracts, authors, and citation patterns in your Zotero library to build a profile of your research interests. New arXiv submissions are scored against this profile and the top matches are presented as daily recommendations.
+
+**Category Filtering**: Select specific arXiv categories to narrow the recommendation scope. This prevents the system from suggesting papers in completely unrelated fields while still allowing cross-disciplinary discoveries within your selected categories.
+
+**Daily Digest View**: Recommendations appear in a dedicated Zotero collection organized by date. Each entry includes the paper title, authors, abstract, arXiv identifier, and a relevance score indicating how closely it matches your library profile.
+
+**Quick Actions**: For each recommended paper, you can:
+- Save to a working collection with one click
+- Open the full paper on arXiv
+- Download the PDF directly to Zotero
+- Dismiss the recommendation (improves future suggestions)
+- Add tags for later organization
+
+**Trend Detection**: The plugin can highlight papers that are receiving unusual attention in your field based on early citation velocity and social media mentions. This helps you identify potentially important work before it becomes widely known.
+
+**Author Tracking**: When the plugin detects papers by authors who are frequently cited in your library, it flags these with higher priority. This ensures you never miss new work from the researchers most relevant to your field.
+
+## Research Workflow Integration
+
+**Morning Review Routine**: Start your research day by spending 10-15 minutes reviewing the daily arXiv recommendations. Scan titles and abstracts, save promising papers to a "To Read" collection, and dismiss irrelevant ones. This disciplined approach keeps you current without consuming excessive time.
+
+**Literature Review Enhancement**: During active literature review phases, increase the number of daily recommendations and expand the arXiv categories. The plugin helps identify relevant preprints that may not yet appear in traditional databases, giving your review a more comprehensive and timely scope.
+
+**Collaborative Discovery**: Share your recommended papers collection with lab members through a Zotero group library. This creates a collective discovery mechanism where the entire group benefits from each member's library-driven recommendations.
+
+**Research Trend Monitoring**: Track which topics appear frequently in your recommendations over weeks and months. Shifts in the recommendation patterns can signal emerging trends in your field, helping you anticipate where the research community is heading.
+
+**Optimizing Recommendation Quality**:
+- Maintain a well-curated "seed" collection of your most important papers
+- Regularly dismiss irrelevant recommendations to refine the algorithm
+- Update your arXiv category selections as your interests evolve
+- Add newly published papers from your own group to keep the profile current
+- Review recommendations from adjacent categories periodically for cross-disciplinary insights
+
+## Configuring Notification Preferences
+
+Control how and when you receive recommendation alerts:
+
+- **Desktop Notifications**: Enable system notifications when new recommendations arrive
+- **Batch Mode**: Accumulate recommendations and review them at a scheduled time
+- **Threshold Filtering**: Only show recommendations above a configurable relevance score
+- **Keyword Highlighting**: Specify key terms to highlight in recommended paper titles and abstracts
+
+For researchers who find the default recommendation volume too high, set a higher relevance threshold to receive only the most closely matched papers. Conversely, those in rapidly moving fields may want to lower the threshold and increase the daily count to ensure broad coverage.
+
+## References
+
+- GitHub Repository: https://github.com/TechPenguineer/zotero-arxiv-daily
+- arXiv API Documentation: https://info.arxiv.org/help/api
+- Zotero Plugin Directory: https://www.zotero.org/support/plugins
+- arXiv Category Taxonomy: https://arxiv.org/category_taxonomy

package/skills/literature/fulltext/core-api-guide/SKILL.md

@@ -0,0 +1,144 @@
+---
+name: core-api-guide
+description: "Search and retrieve open access research papers via CORE aggregator"
+metadata:
+  openclaw:
+    emoji: "🔬"
+    category: "literature"
+    subcategory: "fulltext"
+    keywords: ["open-access", "fulltext", "research-papers", "aggregator", "CORE"]
+    source: "https://core.ac.uk/documentation/api"
+---
+
+# CORE API Guide
+
+## Overview
+
+CORE (COnnecting REpositories) is the world's largest aggregator of open access research papers, providing access to over 130 million articles harvested from thousands of data providers worldwide. The CORE API enables programmatic search, retrieval, and analysis of scholarly full-text content across repositories, journals, and preprint servers.
+
+The API is particularly valuable for researchers conducting systematic reviews, bibliometric analyses, and literature mining tasks. Unlike many scholarly APIs that only provide metadata, CORE specializes in delivering full-text content, making it essential for text mining and natural language processing workflows in academic research.
+
+CORE's v3 API provides a RESTful interface with JSON responses, supporting complex search queries with Boolean operators, field-specific filtering, and batch operations. It is free for non-commercial academic use, though an API key is required to access the service.
+
+## Authentication
+
+CORE requires a free API key for all requests. Register at https://core.ac.uk/services/api to obtain one.
+
+Always store your API key in an environment variable and reference it in requests:
+
+```bash
+export CORE_API_KEY=$CORE_API_KEY
+```
+
+Pass the key via the `Authorization` header:
+
+```bash
+curl -H "Authorization: Bearer $CORE_API_KEY" \
+  "https://api.core.ac.uk/v3/search/works?q=machine+learning"
+```
+
+## Core Endpoints
+
+### Search Works
+
+Search across the entire CORE corpus with full-text and metadata queries.
+
+```
+GET https://api.core.ac.uk/v3/search/works?q={query}&limit={n}&offset={n}
+```
+
+**Parameters:**
+- `q` (required): Search query string, supports Boolean operators (AND, OR, NOT)
+- `limit`: Number of results (default 10, max 100)
+- `offset`: Pagination offset
+- `entity_type`: Filter by type (e.g., `journal-article`, `preprint`)
+
+**Example: Search for climate change papers with full text:**
+
+```bash
+curl -s -H "Authorization: Bearer $CORE_API_KEY" \
+  "https://api.core.ac.uk/v3/search/works?q=climate+change+adaptation&limit=5" \
+  | python3 -m json.tool
+```
+
+**Python example:**
+
+```python
+import requests
+import os
+
+headers = {"Authorization": f"Bearer {os.environ['CORE_API_KEY']}"}
+params = {
+    "q": "deep learning AND medical imaging",
+    "limit": 20,
+    "offset": 0
+}
+resp = requests.get("https://api.core.ac.uk/v3/search/works", headers=headers, params=params)
+data = resp.json()
+
+for result in data.get("results", []):
+    print(f"Title: {result.get('title')}")
+    print(f"DOI: {result.get('doi')}")
+    print(f"Year: {result.get('yearPublished')}")
+    print(f"Full text length: {len(result.get('fullText', ''))}")
+    print("---")
+```
+
+### Get Work by ID
+
+Retrieve a specific paper by its CORE ID or DOI.
+
+```
+GET https://api.core.ac.uk/v3/works/{core_id}
+```
+
+```bash
+curl -s -H "Authorization: Bearer $CORE_API_KEY" \
+  "https://api.core.ac.uk/v3/works/doi:10.1234/example.doi" \
+  | python3 -m json.tool
+```
+
+### Batch Retrieval
+
+Retrieve multiple works in a single request using POST with a list of IDs.
+
+```bash
+curl -s -X POST -H "Authorization: Bearer $CORE_API_KEY" \
+  -H "Content-Type: application/json" \
+  -d '[12345, 67890, 11111]' \
+  "https://api.core.ac.uk/v3/works"
+```
+
+### Search Data Providers
+
+List or search CORE's data providers (repositories, journals).
+
+```
+GET https://api.core.ac.uk/v3/data-providers?q={query}
+```
+
+## Common Research Patterns
+
+**Systematic Literature Review:** Use Boolean queries to replicate a search strategy across the full-text corpus. Combine with date filters to identify papers within a specific time window, then export results for screening in tools like Rayyan or Covidence.
+
+**Full-Text Mining:** Retrieve full-text content programmatically for NLP pipelines. Extract named entities, key phrases, or citation contexts at scale across thousands of papers.
+
+**Repository Coverage Analysis:** Query data providers to understand which institutional repositories contribute to a specific field, useful for bibliometric and open-access policy research.
+
+**Trend Detection:** Run time-series queries for specific terms and track publication volume over years to identify emerging research fronts.
+
+## Rate Limits and Best Practices
+
+- **Free tier:** 150 requests per 15-minute window (10 req/min effective)
+- **Batch endpoints:** Use batch retrieval for multiple IDs to minimize request count
+- **Pagination:** Always use `offset` and `limit` for large result sets; do not fetch all results in one call
+- **Caching:** Cache responses locally for repeat queries, especially for static metadata
+- **Respect robots.txt:** When downloading full texts, add delays between requests
+- **Error handling:** The API returns standard HTTP status codes; implement exponential backoff for 429 (rate limit) responses
+
+## References
+
+- CORE API v3 Documentation: https://core.ac.uk/documentation/api
+- CORE Dashboard and Key Registration: https://core.ac.uk/services/api
+- CORE Data Dumps (for bulk access): https://core.ac.uk/documentation/dataset
+- CORE GitHub: https://github.com/oacore

package/skills/literature/fulltext/institutional-repository-guide/SKILL.md

@@ -0,0 +1,212 @@
+---
+name: institutional-repository-guide
+description: "Access papers from institutional and subject repositories at scale"
+metadata:
+  openclaw:
+    emoji: "🏛️"
+    category: "literature"
+    subcategory: "fulltext"
+    keywords: ["institutional repository", "DSpace", "EPrints", "open access archive", "subject repository", "OpenDOAR"]
+    source: "wentor-research-plugins"
+---
+
+# Institutional Repository Guide
+
+Institutional repositories (IRs) are university-run digital archives that store and provide open access to their researchers' scholarly output — dissertations, journal articles, conference papers, datasets, and technical reports. Subject repositories like arXiv, bioRxiv, SSRN, and RePEc serve similar functions for specific disciplines. Together, they form a distributed network of open scholarship that complements commercial databases.
+
+This guide covers how to discover, access, and systematically harvest content from institutional and subject repositories for literature reviews, meta-analyses, and research data collection.
+
+## Repository Landscape
+
+### Types of Repositories
+
+```
+Institutional Repositories (IR):
+  - Run by universities to archive their researchers' output
+  - Examples: DSpace, EPrints, Fedora-based systems
+  - Discovery: OpenDOAR directory (v2.sherpa.ac.uk/opendoar)
+
+Subject Repositories:
+  - Discipline-specific archives
+  - arXiv (physics, CS, math), bioRxiv, SSRN, RePEc, EarthArXiv
+
+Aggregators:
+  - Harvest from many repositories into a single search interface
+  - BASE (Bielefeld Academic Search Engine)
+  - CORE (core.ac.uk, 200M+ open access articles)
+  - OpenAIRE (European research output)
+```
+
+### Discovering Repositories
+
+OpenDOAR (Directory of Open Access Repositories) is the primary registry for finding institutional repositories:
+
+```python
+import urllib.request
+import json
+
+def search_opendoar(subject: str = None, country: str = None) -> list:
+    """
+    Search the OpenDOAR registry for institutional repositories.
+
+    Args:
+        subject: Filter by subject area (e.g., "Biology", "Computer Science")
+        country: ISO country code (e.g., "US", "GB", "CN")
+    """
+    base_url = "https://v2.sherpa.ac.uk/cgi/retrieve"
+    params = "?item-type=repository&format=Json"
+    if subject:
+        params += f"&filter=[[\"{subject}\",\"subject\"]]"
+    if country:
+        params += f"&filter=[[\"{country}\",\"country\"]]"
+
+    req = urllib.request.Request(base_url + params)
+    response = urllib.request.urlopen(req)
+    data = json.loads(response.read())
+
+    repositories = []
+    for item in data.get("items", []):
+        repo_info = {
+            "name": item.get("repository_metadata", {}).get("name", [{}])[0].get("name", ""),
+            "url": item.get("repository_metadata", {}).get("url", ""),
+            "oai_url": item.get("repository_metadata", {}).get("oai_url", ""),
+            "software": item.get("repository_metadata", {}).get("software", {}).get("name", ""),
+            "type": item.get("repository_metadata", {}).get("type", "")
+        }
+        repositories.append(repo_info)
+
+    return repositories
+```
+
+## OAI-PMH Harvesting from Repositories
+
+Most institutional repositories support OAI-PMH (Open Archives Initiative Protocol for Metadata Harvesting), the standard protocol for metadata exchange:
+
+```python
+import xml.etree.ElementTree as ET
+import urllib.request
+
+def harvest_repository(base_url: str, metadata_prefix: str = "oai_dc",
+                       set_spec: str = None, from_date: str = None) -> list:
+    """
+    Harvest metadata records from a repository's OAI-PMH endpoint.
+
+    Args:
+        base_url: The OAI-PMH base URL
+        metadata_prefix: Metadata format (oai_dc, datacite, mets)
+        set_spec: Optional set/collection to restrict harvesting
+        from_date: Harvest only records added after this date (YYYY-MM-DD)
+    """
+    params = f"?verb=ListRecords&metadataPrefix={metadata_prefix}"
+    if set_spec:
+        params += f"&set={set_spec}"
+    if from_date:
+        params += f"&from={from_date}"
+
+    url = base_url + params
+    records = []
+
+    while url:
+        response = urllib.request.urlopen(url)
+        tree = ET.parse(response)
+        root = tree.getroot()
+        ns = {"oai": "http://www.openarchives.org/OAI/2.0/"}
+
+        for record in root.findall(".//oai:record", ns):
+            header = record.find("oai:header", ns)
+            identifier = header.find("oai:identifier", ns).text
+            datestamp = header.find("oai:datestamp", ns).text
+            records.append({"identifier": identifier, "datestamp": datestamp})
+
+        token_elem = root.find(".//oai:resumptionToken", ns)
+        if token_elem is not None and token_elem.text:
+            url = f"{base_url}?verb=ListRecords&resumptionToken={token_elem.text}"
+        else:
+            url = None
+
+    return records
+```
+
+### Key OAI-PMH Verbs
+
+| Verb | Purpose |
+|------|---------|
+| `Identify` | Get repository name, admin email, policies |
+| `ListSets` | List available collections/sets |
+| `ListMetadataFormats` | List supported metadata schemas |
+| `ListIdentifiers` | Lightweight listing of record headers |
+| `ListRecords` | Full metadata records with pagination |
+| `GetRecord` | Retrieve a single record by identifier |
+
+## Major Repository Platforms
+
+### DSpace
+
+The most widely deployed open-source repository platform (used by ~40% of repositories worldwide):
+
+- OAI-PMH endpoint: `{base-url}/oai/request`
+- REST API: `{base-url}/server/api`
+- Supports Dublin Core, METS, and custom metadata schemas
+- Examples: MIT DSpace, University of Cambridge Repository
+
+### EPrints
+
+Popular in the UK and Europe:
+
+- OAI-PMH endpoint: `{base-url}/cgi/oai2`
+- REST API: `{base-url}/cgi/export/{id}/{format}`
+- Strong support for research output types (articles, theses, conference items)
+- Examples: University of Southampton EPrints
+
+### Fedora / Islandora
+
+Used by larger institutions with complex digital collections:
+
+- Typically paired with a discovery layer (Solr/Blacklight)
+- Strong support for digital preservation workflows
+- Examples: University of Toronto, Smithsonian Institution
+
+## Building a Harvesting Pipeline
+
+### Systematic Collection Workflow
+
+```
+1. Identify target repositories
+   - Use OpenDOAR to find IRs by subject or country
+   - List subject repositories relevant to your discipline
+
+2. Test endpoints
+   - Send Identify request to verify the endpoint is active
+   - Check ListMetadataFormats for available schemas
+
+3. Harvest incrementally
+   - Use "from" parameter to harvest only new records
+   - Store last harvest date for each repository
+   - Respect rate limits (typically 1 request per second)
+
+4. Deduplicate
+   - Match records by DOI when available
+   - Use title + author fuzzy matching for records without DOIs
+   - Flag duplicates rather than deleting (keep provenance)
+
+5. Store and index
+   - Save metadata in structured format (JSON, SQLite, CSV)
+   - Build a local search index for efficient retrieval
+```
+
+## Ethical Considerations
+
+- Always respect `robots.txt` and repository rate limits
+- Metadata harvesting is generally permitted; bulk full-text download may require permission
+- Check each repository's terms of use before harvesting
+- Use harvested data for research purposes, not commercial redistribution
+- Attribute the source repository in publications using harvested data
+- Consider reaching out to repository administrators for large-scale harvesting projects
+
+## References
+
+- OpenDOAR: https://v2.sherpa.ac.uk/opendoar/
+- OAI-PMH specification: http://www.openarchives.org/OAI/openarchivesprotocol.html
+- CORE: https://core.ac.uk
+- BASE: https://www.base-search.net
+- DSpace documentation: https://wiki.lyrasis.org/display/DSPACE