@wentorai/research-plugins 1.0.0 → 1.2.0

package/skills/tools/scraping/easy-spider-guide/SKILL.md

@@ -0,0 +1,250 @@
+---
+name: easy-spider-guide
+description: "Guide to EasySpider for visual no-code web data collection"
+metadata:
+  openclaw:
+    emoji: "🕷️"
+    category: "tools"
+    subcategory: "scraping"
+    keywords: ["web scraping", "visual crawler", "no-code scraping", "data collection", "research data", "web automation"]
+    source: "https://github.com/NaiboWang/EasySpider"
+---
+
+# EasySpider Guide
+
+## Overview
+
+EasySpider is a visual, no-code web crawler tool with over 44K stars on GitHub. It provides a graphical interface where users design web scraping tasks by interacting directly with target web pages, clicking on elements to extract, and defining navigation flows visually. No programming knowledge is required to build functional scrapers, making it accessible to researchers across all disciplines.
+
+For academic researchers, data collection from web sources is a frequent need but often a technical barrier. Whether gathering publication metadata from journal websites, collecting survey responses from public forums, extracting pricing data for economic research, or archiving web content for digital humanities projects, EasySpider enables researchers to build custom scrapers without writing Python or JavaScript code. The visual approach also makes scrapers easier to maintain and modify when target websites change their structure.
+
+EasySpider runs as a desktop application on Windows, macOS, and Linux. It uses a built-in Chromium browser for rendering, which means it can handle JavaScript-heavy websites, single-page applications, and sites that require user interaction such as clicking buttons, scrolling, or filling forms. Scraped data can be exported as CSV, JSON, or directly to databases.
+
+## Installation
+
+### Download and Setup
+
+```bash
+# Download the latest release for your platform from GitHub releases
+# https://github.com/NaiboWang/EasySpider/releases
+
+# macOS - download the .dmg file and drag to Applications
+
+# Linux - download the AppImage
+chmod +x EasySpider-linux-x86_64.AppImage
+./EasySpider-linux-x86_64.AppImage
+
+# Or run from source
+git clone https://github.com/NaiboWang/EasySpider.git
+cd EasySpider
+npm install
+npm start
+```
+
+### System Requirements
+
+- Operating system: Windows 10+, macOS 10.15+, or Linux (Ubuntu 18.04+)
+- RAM: 4 GB minimum, 8 GB recommended for complex scraping tasks
+- Disk space: 500 MB for the application plus storage for scraped data
+- Network: stable internet connection for web scraping
+
+## Core Concepts
+
+### Task Design Workflow
+
+EasySpider follows a visual task design approach with these steps:
+
+1. **Open target page** - Enter the URL in EasySpider's built-in browser
+2. **Select elements** - Click on the data elements you want to extract
+3. **Define fields** - Name each extracted element (title, author, date, etc.)
+4. **Configure pagination** - Click the "next page" button to set up pagination
+5. **Set extraction rules** - Define how to handle lists, tables, and nested pages
+6. **Test and run** - Preview results, then execute the full scraping task
+
+### Element Selection Modes
+
+- **Single element** - Click one element to extract that specific item
+- **Similar elements** - Click two similar items and EasySpider detects the pattern for all matching elements on the page
+- **Table mode** - Select a table header row to extract entire structured tables
+- **Input mode** - Define form fields to fill before extracting (useful for search-based data collection)
+
+## Research Use Cases
+
+### Collecting Publication Metadata
+
+Researchers can use EasySpider to gather publication information from journal websites, conference proceedings pages, or institutional repositories.
+
+**Example workflow for scraping a conference proceedings page:**
+
+1. Navigate to the proceedings listing page
+2. Click on the first paper title to mark it as a "title" field
+3. Click on the second paper title; EasySpider recognizes the pattern and selects all titles
+4. Similarly select author names, abstract snippets, and publication dates
+5. If papers span multiple pages, click the "Next" pagination button
+6. Configure "click into each paper" to follow links and extract full abstracts
+7. Run the task and export as CSV
+
+### Monitoring Research Funding Opportunities
+
+```
+Task: Daily scan of funding agency websites for new opportunities
+
+Steps configured in EasySpider:
+1. Navigate to funding agency announcement page
+2. Extract: opportunity title, deadline, funding amount, eligibility
+3. Filter: only new announcements (since last check)
+4. Schedule: run daily at 8:00 AM
+5. Export: append to CSV file, send notification email
+```
+
+### Gathering Economic Data from Public Sources
+
+For economics and social science research, EasySpider can collect publicly available data from government statistics portals, price comparison websites, and public registries.
+
+```
+Task: Collect commodity prices from public market websites
+
+Fields to extract:
+- commodity_name: product identifier
+- price: current listed price
+- unit: measurement unit
+- date: listing date
+- source_url: page URL for reference
+
+Pagination: navigate through category pages
+Schedule: weekly collection
+Output: CSV with timestamp for time-series analysis
+```
+
+### Digital Humanities Web Archiving
+
+```
+Task: Archive public blog posts for discourse analysis
+
+Configuration:
+- Start URL: blog archive page
+- Follow: links matching pattern /posts/*
+- Extract per page:
+  - post_title
+  - post_date
+  - author_name
+  - post_content (full text)
+  - comment_count
+  - tags/categories
+- Pagination: follow archive navigation links
+- Output: JSON with full text content
+```
+
+## Advanced Features
+
+### Conditional Logic
+
+EasySpider supports conditional branches in task flows:
+
+- **If element exists** - Check for specific elements before attempting extraction
+- **If text contains** - Filter items based on content matching
+- **Loop control** - Set maximum iterations for pagination or nested page visits
+
+### Data Cleaning Options
+
+Built-in text processing options can be applied during extraction:
+
+- Remove HTML tags from extracted text
+- Trim whitespace and normalize spacing
+- Extract numbers from mixed text fields
+- Apply regex patterns to clean specific formats
+- Convert date strings to standardized formats
+
+### Handling Dynamic Content
+
+For JavaScript-rendered pages, EasySpider provides options to:
+
+- Wait for specific elements to appear before extracting
+- Scroll to load lazy-loaded content
+- Click "Load More" buttons automatically
+- Handle infinite scroll pages with configurable scroll limits
+
+### Anti-Detection Configuration
+
+For responsible scraping, EasySpider includes options to:
+
+```
+Request configuration:
+- Delay between requests: 2-5 seconds (randomized)
+- User-Agent rotation: enabled
+- Concurrent requests: 1 (sequential for politeness)
+- Respect robots.txt: check before scraping
+- Rate limiting: max 30 requests per minute
+```
+
+## Exporting Research Data
+
+### CSV Export
+
+The most common format for researchers. Data is exported with headers matching the field names defined during task design.
+
+```csv
+title,authors,year,journal,doi,abstract
+"Machine Learning in Materials Science","Smith J, Lee K",2025,"Nature Materials","10.1038/xxx","Abstract text here..."
+```
+
+### JSON Export
+
+Preserves nested structure for complex extractions:
+
+```json
+{
+  "task_name": "proceedings_scrape",
+  "extracted_at": "2026-03-10T14:30:00Z",
+  "records": [
+    {
+      "title": "Machine Learning in Materials Science",
+      "authors": ["Smith J", "Lee K"],
+      "year": 2025,
+      "metadata": {
+        "journal": "Nature Materials",
+        "doi": "10.1038/xxx"
+      }
+    }
+  ]
+}
+```
+
+### Database Export
+
+EasySpider can write directly to SQLite databases, which is convenient for subsequent analysis with Python pandas or R.
+
+```python
+import sqlite3
+import pandas as pd
+
+# Read EasySpider output database
+conn = sqlite3.connect("easyspider_results.db")
+df = pd.read_sql("SELECT * FROM scraped_data", conn)
+
+# Process and analyze
+print(f"Total records: {len(df)}")
+print(df.describe())
+conn.close()
+```
+
+## Ethical Web Scraping Guidelines for Researchers
+
+When using EasySpider for research data collection, follow these ethical guidelines:
+
+- **Check robots.txt** before scraping any website
+- **Respect rate limits** and add appropriate delays between requests
+- **Review terms of service** for target websites
+- **Use APIs when available** rather than scraping HTML (many services offer research APIs)
+- **Minimize data collection** to only what is needed for the research question
+- **Store data securely** especially when collecting personal information
+- **Cite data sources** in publications and include data collection methodology
+- **Obtain IRB approval** if scraping involves human subjects data
+- **Consider GDPR/privacy regulations** when scraping data from EU sources
+
+## References
+
+- EasySpider GitHub repository: https://github.com/NaiboWang/EasySpider
+- EasySpider documentation: https://github.com/NaiboWang/EasySpider/wiki
+- EasySpider video tutorials: https://github.com/NaiboWang/EasySpider#tutorials
+- Web scraping ethics in research: https://doi.org/10.1177/2053951720943006

package/skills/tools/scraping/google-scholar-scraper/SKILL.md

@@ -0,0 +1,255 @@
+---
+name: google-scholar-scraper
+description: "Ethical Google Scholar data collection techniques and best practices"
+metadata:
+  openclaw:
+    emoji: "mag"
+    category: "tools"
+    subcategory: "scraping"
+    keywords: ["Google Scholar", "web scraping", "bibliometrics", "citation data", "scholarly search", "rate limiting"]
+    source: "wentor-research-plugins"
+---
+
+# Google Scholar Scraper
+
+A skill for ethically collecting bibliometric data from Google Scholar, including search results, citation counts, author profiles, and related articles. Covers rate limiting, CAPTCHA avoidance, alternative APIs, legal considerations, data parsing, and practical workflows that balance data needs with responsible access.
+
+## Legal and Ethical Considerations
+
+### Before You Scrape
+
+Google Scholar does not offer an official API, and its Terms of Service restrict automated access. Researchers must weigh their data needs against legal and ethical constraints.
+
+```
+Legal landscape:
+
+Terms of Service:
+  - Google's ToS prohibit automated queries
+  - Violation can result in IP blocking (temporary or permanent)
+  - Institutional IPs can be blocked, affecting all campus users
+  - In some jurisdictions, ToS violations are not legally binding
+    for non-commercial academic research, but this is debated
+
+Ethical guidelines:
+  - Minimize load: respect the server, use delays between requests
+  - Cache aggressively: never request the same page twice
+  - Use official alternatives first (see below)
+  - Do not redistribute raw scraped data
+  - Cite Google Scholar as your data source in publications
+  - Consider whether your research question truly requires
+    Google Scholar data, or if Web of Science, Scopus, or
+    OpenAlex could answer it instead
+
+Official and semi-official alternatives:
+  - Semantic Scholar API: free, 100 requests/sec, excellent coverage
+  - OpenAlex API: free, comprehensive, well-documented
+  - Crossref API: free, DOI-based metadata and citation counts
+  - CORE API: free, full-text open access content
+  - Google Scholar Alerts: manual but ToS-compliant monitoring
+  - Publish or Perish (software): uses Google Scholar with built-in
+    rate limiting, commonly used in bibliometric research
+```
+
+## Data Collection Approaches
+
+### Using Scholarly (Python Library)
+
+The `scholarly` Python library wraps Google Scholar access with built-in rate limiting and proxy support. It is the most commonly used tool for academic Google Scholar scraping.
+
+```python
+from scholarly import scholarly, ProxyGenerator
+
+def setup_scholarly_with_proxy():
+    """
+    Configure scholarly with a free proxy to reduce blocking risk.
+    For heavy usage, consider ScraperAPI or similar paid services.
+    """
+    pg = ProxyGenerator()
+    # Free proxy (less reliable, suitable for small jobs)
+    pg.FreeProxies()
+    scholarly.use_proxy(pg)
+
+
+def search_scholar(query, max_results=20):
+    """
+    Search Google Scholar and collect structured results.
+
+    IMPORTANT: Add delays between queries to avoid blocking.
+    Recommended: 10-30 seconds between searches.
+    """
+    import time
+
+    results = []
+    search_query = scholarly.search_pubs(query)
+
+    for i in range(max_results):
+        try:
+            result = next(search_query)
+            parsed = {
+                "title": result["bib"].get("title", ""),
+                "author": result["bib"].get("author", []),
+                "year": result["bib"].get("pub_year", ""),
+                "venue": result["bib"].get("venue", ""),
+                "abstract": result["bib"].get("abstract", ""),
+                "citations": result.get("num_citations", 0),
+                "url": result.get("pub_url", ""),
+            }
+            results.append(parsed)
+
+            # Rate limiting: wait between result fetches
+            time.sleep(2)
+
+        except StopIteration:
+            break
+
+    return results
+
+
+def get_author_profile(author_name):
+    """
+    Retrieve an author's Google Scholar profile.
+    Includes h-index, i10-index, and publication list.
+    """
+    search_query = scholarly.search_author(author_name)
+    author = next(search_query)
+    author = scholarly.fill(author)
+
+    profile = {
+        "name": author.get("name", ""),
+        "affiliation": author.get("affiliation", ""),
+        "h_index": author.get("hindex", 0),
+        "i10_index": author.get("i10index", 0),
+        "cited_by": author.get("citedby", 0),
+        "interests": author.get("interests", []),
+        "publications": len(author.get("publications", [])),
+    }
+
+    return profile
+```
+
+## Rate Limiting and Anti-Blocking
+
+### Best Practices
+
+```
+Rate limiting strategy:
+
+1. Request delays:
+   - Between search queries: 15-30 seconds minimum
+   - Between profile lookups: 10-20 seconds
+   - Between citation fetches: 5-10 seconds
+   - Add random jitter: delay + random(0, 5) seconds
+
+2. Session management:
+   - Rotate user agents (maintain a list of 10+ real browser UAs)
+   - Clear cookies periodically
+   - Use residential proxies for large jobs (paid)
+   - Limit sessions to 100-200 requests before rotating proxy
+
+3. Caching:
+   - Cache every response to disk (shelve, sqlite, or JSON)
+   - Check cache before making any request
+   - Set cache expiry (7-30 days for citation counts)
+
+4. Batch scheduling:
+   - Spread collection over days, not hours
+   - Run during off-peak hours (late night UTC)
+   - Process in batches of 50-100 queries per session
+```
+
+### Handling CAPTCHAs and Blocks
+
+```python
+import time
+import random
+
+def resilient_search(query, max_retries=3):
+    """
+    Search with exponential backoff on failures.
+    When blocked, wait and retry with increasing delays.
+    """
+    for attempt in range(max_retries):
+        try:
+            results = search_scholar(query, max_results=10)
+            return results
+        except Exception as e:
+            if "CAPTCHA" in str(e) or "429" in str(e):
+                wait_time = (2 ** attempt) * 60 + random.randint(0, 30)
+                print(f"Blocked. Waiting {wait_time}s before retry "
+                      f"(attempt {attempt + 1}/{max_retries})")
+                time.sleep(wait_time)
+            else:
+                raise e
+
+    print("Max retries exceeded. Consider using a different proxy "
+          "or waiting 24 hours before resuming.")
+    return []
+```
+
+## Data Processing and Storage
+
+### Structuring Collected Data
+
+```python
+import pandas as pd
+import json
+from datetime import datetime
+
+def save_results(results, output_dir, query_name):
+    """
+    Save scraped results in multiple formats with metadata.
+    """
+    # Add collection metadata
+    metadata = {
+        "query": query_name,
+        "collected_at": datetime.now().isoformat(),
+        "n_results": len(results),
+        "source": "google_scholar",
+    }
+
+    # Save as JSON (preserves all structure)
+    with open(f"{output_dir}/{query_name}_results.json", "w") as f:
+        json.dump({"metadata": metadata, "results": results}, f, indent=2)
+
+    # Save as CSV (for spreadsheet analysis)
+    df = pd.DataFrame(results)
+    df.to_csv(f"{output_dir}/{query_name}_results.csv", index=False)
+
+    return f"Saved {len(results)} results for query: {query_name}"
+```
+
+## Recommended Alternatives to Scraping
+
+### When Not to Scrape Google Scholar
+
+```
+Use these free APIs instead when possible:
+
+OpenAlex (openalex.org):
+  - Coverage: 250M+ works
+  - API: REST, no key needed (polite pool with email)
+  - Rate limit: 10 requests/sec (polite pool), 100K/day
+  - Data: titles, abstracts, citations, authors, institutions
+  - Best for: large-scale bibliometric analysis
+
+Semantic Scholar (semanticscholar.org):
+  - Coverage: 200M+ papers
+  - API: REST, free key available
+  - Rate limit: 100 requests/sec with API key
+  - Data: titles, abstracts, citations, citation contexts, TLDR
+  - Best for: citation analysis, NLP on papers
+
+Crossref (crossref.org):
+  - Coverage: 130M+ DOIs
+  - API: REST, no key needed (polite pool with email)
+  - Data: metadata, reference lists, citation counts
+  - Best for: DOI resolution, reference matching
+
+Use Google Scholar scraping ONLY when:
+  - You need Google Scholar-specific metrics (h-index by GS)
+  - Your target papers are not indexed elsewhere
+  - You need Google Scholar's ranking/relevance ordering
+  - Small-scale collection (< 500 results)
+```
+
+Responsible data collection from Google Scholar requires balancing research needs with ethical obligations to shared infrastructure. When possible, prefer official APIs that are designed for programmatic access. When scraping is necessary, implement aggressive rate limiting, cache results, and keep total request volumes as low as your research question permits.