all-in-mcp 0.2.2__tar.gz → 0.2.5__tar.gz

{all_in_mcp-0.2.2 → all_in_mcp-0.2.5}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: all-in-mcp
-Version: 0.2.2
+Version: 0.2.5
 Summary: An MCP (Model Context Protocol) server providing daily-use utility functions and academic paper search capabilities
 Project-URL: Homepage, https://github.com/jiahaoxiang2000/all-in-mcp
 Project-URL: Repository, https://github.com/jiahaoxiang2000/all-in-mcp
@@ -53,6 +53,8 @@ An MCP (Model Context Protocol) server that provides daily-use utility functions
 
 - **Academic Research**: IACR ePrint Archive paper search, download, and reading
 - **Bibliography Search**: CryptoBib database search for cryptography papers
+- **Google Scholar**: Search academic papers across disciplines with citation data
+- **PDF Reading**: Read and extract text from local and online PDF files
 
 ### Paper Search Capabilities
 
@@ -70,6 +72,13 @@ An MCP (Model Context Protocol) server that provides daily-use utility functions
 - Retrieve structured paper metadata or raw BibTeX entries
 - Support for all major cryptography venues and conferences
 
+#### Google Scholar
+
+- Search academic papers across multiple disciplines
+- Access to citation counts and publication metadata
+- Broad coverage of academic literature from various sources
+- Year-based filtering for targeted searches
+
 ## Quick Start
 
 ### Prerequisites

{all_in_mcp-0.2.2 → all_in_mcp-0.2.5}/README.md

@@ -8,6 +8,8 @@ An MCP (Model Context Protocol) server that provides daily-use utility functions
 
 - **Academic Research**: IACR ePrint Archive paper search, download, and reading
 - **Bibliography Search**: CryptoBib database search for cryptography papers
+- **Google Scholar**: Search academic papers across disciplines with citation data
+- **PDF Reading**: Read and extract text from local and online PDF files
 
 ### Paper Search Capabilities
 
@@ -25,6 +27,13 @@ An MCP (Model Context Protocol) server that provides daily-use utility functions
 - Retrieve structured paper metadata or raw BibTeX entries
 - Support for all major cryptography venues and conferences
 
+#### Google Scholar
+
+- Search academic papers across multiple disciplines
+- Access to citation counts and publication metadata
+- Broad coverage of academic literature from various sources
+- Year-based filtering for targeted searches
+
 ## Quick Start
 
 ### Prerequisites

{all_in_mcp-0.2.2 → all_in_mcp-0.2.5}/docs/api.md

@@ -108,6 +108,56 @@ URL: https://eprint.iacr.org/2023/1234
 [Full extracted text content]
 ```
 
+## Google Scholar Search
+
+### search-google-scholar-papers
+
+Search academic papers from Google Scholar. This provides broad coverage across multiple academic disciplines and includes citation information.
+
+**Parameters:**
+
+- `query` (string, required): Search query string (e.g., 'machine learning', 'neural networks')
+- `max_results` (integer, optional): Maximum number of results to return (default: 10)
+- `year_low` (integer, optional): Minimum publication year for filtering
+- `year_high` (integer, optional): Maximum publication year for filtering
+
+**Returns:**
+
+- List of papers with metadata (title, authors, citations, year, URL, abstract)
+
+**Example:**
+
+```json
+{
+  "name": "search-google-scholar-papers",
+  "arguments": {
+    "query": "deep learning transformers",
+    "max_results": 5,
+    "year_low": 2020,
+    "year_high": 2024
+  }
+}
+```
+
+**Response:**
+
+```
+Found 3 Google Scholar papers for query 'deep learning transformers' in year range (2020-2024):
+
+1. **Attention Is All You Need**
+   - Authors: Ashish Vaswani, Noam Shazeer, Niki Parmar
+   - Citations: 85234
+   - Year: 2017
+   - URL: https://papers.nips.cc/paper/7181-attention-is-all-you-need
+   - Abstract: The dominant sequence transduction models are based on complex recurrent or convolutional neural networks...
+```
+
+**Limitations:**
+
+- No direct PDF downloads (redirects to publisher websites)
+- Rate limiting may apply for frequent requests
+- Results may vary based on geographic location
+
 ## CryptoBib Bibliography Search
 
 ### search-cryptobib-papers

all_in_mcp-0.2.5/docs/google_scholar.md

@@ -0,0 +1,215 @@
+# Google Scholar Integration
+
+This document describes the Google Scholar integration in the all-in-mcp server.
+
+## Overview
+
+The Google Scholar integration provides academic paper searching capabilities through Google Scholar's web interface. Unlike other academic platforms, Google Scholar aggregates papers from multiple sources and provides citation counts and broader coverage of academic literature.
+
+## Features
+
+- **Paper Search**: Search for academic papers across multiple disciplines
+- **Citation Information**: Get citation counts for papers
+- **Multiple Source Coverage**: Access papers from various publishers and repositories
+- **Year-based Filtering**: Filter search results by publication year range
+
+## Available Tools
+
+### search-google-scholar-papers
+
+Search for academic papers using Google Scholar.
+
+**Parameters:**
+
+- `query` (required): Search query string (e.g., "machine learning", "neural networks")
+- `max_results` (optional): Maximum number of papers to return (default: 10, max: 100)
+- `year_low` (optional): Minimum publication year for filtering results
+- `year_high` (optional): Maximum publication year for filtering results
+
+**Example Usage:**
+
+```json
+{
+  "query": "deep learning transformers",
+  "max_results": 5,
+  "year_low": 2020,
+  "year_high": 2024
+}
+```
+
+**Returns:**
+
+- Paper title
+- Authors list
+- Citation count (when available)
+- Publication year (when available)
+- Paper URL
+- Abstract (truncated to 300 characters)
+
+## Limitations
+
+### No Direct PDF Access
+
+Google Scholar doesn't provide direct PDF downloads. Users need to:
+
+1. Use the provided paper URL to access the publisher's website
+2. Check if the paper is available through institutional access
+3. Look for open access versions on author websites or repositories
+
+### Rate Limiting
+
+Google Scholar implements rate limiting to prevent automated scraping:
+
+- The implementation includes random delays between requests (1-3 seconds)
+- Multiple rapid requests may result in temporary blocks
+- Consider using other sources (IACR, arXiv) for bulk operations
+
+### Search Result Variability
+
+- Results may vary based on geographic location and Google's algorithms
+- Some papers may not be accessible due to publisher restrictions
+- Citation counts may not be real-time accurate
+
+## Implementation Details
+
+### Web Scraping Approach
+
+The implementation uses web scraping with:
+
+- Random user agent rotation to avoid detection
+- BeautifulSoup for HTML parsing
+- Request session management for cookie handling
+- Error handling for network issues and parsing failures
+
+### Paper Data Extraction
+
+The parser extracts:
+
+- **Title**: From `h3.gs_rt` elements, cleaned of PDF/HTML markers
+- **Authors**: From `div.gs_a` elements, parsed from publication info
+- **Abstract**: From `div.gs_rs` elements when available
+- **Citations**: From citation links in `div.gs_fl` elements
+- **Year**: Extracted from publication information using regex patterns
+- **URL**: From title links to source papers
+
+### Error Handling
+
+- Network timeouts (30 seconds)
+- HTTP error responses (rate limiting, server errors)
+- Parsing failures for malformed HTML
+- Missing required paper elements
+
+## Best Practices
+
+### Responsible Usage
+
+1. **Respect Rate Limits**: Don't make too many requests in quick succession
+2. **Cache Results**: Store search results locally to avoid repeated queries
+3. **Use Appropriate Delays**: The implementation includes built-in delays
+4. **Monitor for Blocks**: Be prepared to handle temporary access restrictions
+
+### Query Optimization
+
+1. **Specific Terms**: Use specific academic terms for better results
+2. **Author Names**: Include author names when searching for specific papers
+3. **Publication Venues**: Include conference or journal names for focused searches
+4. **Year Ranges**: Use year filters to narrow down results
+
+### Integration with Other Sources
+
+Google Scholar works best when combined with other academic sources:
+
+- Use IACR for cryptography papers with PDF access
+- Use arXiv for preprints with full-text access
+- Use institutional repositories for open access papers
+
+## Example Responses
+
+### Successful Search
+
+```
+Found 3 Google Scholar papers for query 'machine learning healthcare':
+
+1. **Machine Learning in Healthcare: A Review**
+   - Authors: John Smith, Jane Doe, Bob Johnson
+   - Citations: 245
+   - Year: 2023
+   - URL: https://example.com/paper1
+   - Abstract: This comprehensive review examines the applications of machine learning techniques in healthcare settings, covering diagnostic imaging, predictive analytics, and treatment optimization...
+
+2. **Deep Learning for Medical Diagnosis**
+   - Authors: Alice Brown, Charlie Wilson
+   - Citations: 156
+   - Year: 2022
+   - URL: https://example.com/paper2
+   - Abstract: We present a novel deep learning framework for automated medical diagnosis using convolutional neural networks...
+```
+
+### Empty Results
+
+```
+No papers found for query: obscure search term in year range (2025-2030)
+```
+
+### Error Response
+
+```
+Error searching Google Scholar: HTTP 429 - Rate limit exceeded. Please try again later.
+```
+
+## Troubleshooting
+
+### Common Issues
+
+**No Results Found**
+
+- Check query spelling and syntax
+- Try broader search terms
+- Remove year filters if too restrictive
+- Verify network connectivity
+
+**Rate Limiting Errors**
+
+- Wait before making additional requests
+- Reduce the frequency of searches
+- Consider using other academic sources
+- Check if IP address is temporarily blocked
+
+**Parsing Errors**
+
+- Usually indicates changes in Google Scholar's HTML structure
+- Check logs for specific parsing failures
+- May require updates to the parsing logic
+
+### Development and Testing
+
+**Testing Considerations**
+
+- Use mock responses for unit tests to avoid rate limiting
+- Test with various query types and edge cases
+- Include tests for error conditions and edge cases
+- Verify handling of malformed or incomplete results
+
+**Debugging Tips**
+
+- Enable detailed logging to see request/response details
+- Check network connectivity and DNS resolution
+- Verify user agent and headers are being sent correctly
+- Test individual parsing functions with real HTML samples
+
+## Future Enhancements
+
+### Potential Improvements
+
+1. **Enhanced Parsing**: Better extraction of publication venues and DOIs
+2. **Citation Tracking**: Track citation networks and related papers
+3. **Advanced Filtering**: Filter by publication type, author affiliation
+4. **Result Caching**: Implement intelligent caching to reduce API calls
+5. **Proxy Support**: Add proxy rotation for high-volume usage
+
+### Integration Opportunities
+
+1. **Cross-Reference Validation**: Verify results against other academic databases
+2. **Full-Text Integration**: Combine with repository APIs for full-text access
+3. **Citation Analysis**: Integrate with citation analysis tools
+4. **Recommendation System**: Suggest related papers based on search history

{all_in_mcp-0.2.2 → all_in_mcp-0.2.5}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "all-in-mcp"
-version = "0.2.2"
+version = "0.2.5"
 description = "An MCP (Model Context Protocol) server providing daily-use utility functions and academic paper search capabilities"
 readme = "README.md"
 license = { text = "MIT" }

{all_in_mcp-0.2.2 → all_in_mcp-0.2.5}/src/all_in_mcp/academic_platforms/__init__.py

@@ -1,6 +1,7 @@
 # all_in_mcp/academic_platforms/__init__.py
 from .base import PaperSource
 from .cryptobib import CryptoBibSearcher
+from .google_scholar import GoogleScholarSearcher
 from .iacr import IACRSearcher
 
-__all__ = ["PaperSource", "CryptoBibSearcher", "IACRSearcher"]
+__all__ = ["CryptoBibSearcher", "GoogleScholarSearcher", "IACRSearcher", "PaperSource"]

all_in_mcp-0.2.5/src/all_in_mcp/academic_platforms/crossref.py

@@ -0,0 +1,328 @@
+# all_in_mcp/academic_platforms/crossref.py
+import logging
+from datetime import datetime
+from typing import Optional
+from urllib.parse import quote_plus
+
+import httpx
+
+from ..paper import Paper
+from .base import PaperSource
+
+logger = logging.getLogger(__name__)
+
+
+class CrossrefSearcher(PaperSource):
+    """Crossref API paper search implementation"""
+
+    BASE_URL = "https://api.crossref.org"
+    WORKS_ENDPOINT = f"{BASE_URL}/works"
+
+    def __init__(self, email: Optional[str] = None):
+        """
+        Initialize Crossref searcher
+
+        Args:
+            email: Optional email for polite API usage (recommended by Crossref)
+        """
+        self.email = email
+        self.client = httpx.Client(timeout=30.0)
+
+    def _get_headers(self) -> dict:
+        """Get headers for API requests"""
+        headers = {
+            "User-Agent": "all-in-mcp/0.1.0 (https://github.com/user/all-in-mcp)"
+        }
+        if self.email:
+            headers["User-Agent"] += f" (mailto:{self.email})"
+        return headers
+
+    def _parse_date(self, date_parts: list) -> Optional[datetime]:
+        """Parse Crossref date parts into datetime"""
+        if not date_parts or not isinstance(date_parts, list):
+            return None
+
+        try:
+            # Crossref provides date as [[year, month, day]] or [[year, month]] or [[year]]
+            if len(date_parts) > 0 and isinstance(date_parts[0], list):
+                parts = date_parts[0]
+                year = parts[0] if len(parts) > 0 else 1
+                month = parts[1] if len(parts) > 1 else 1
+                day = parts[2] if len(parts) > 2 else 1
+                return datetime(year, month, day)
+        except (ValueError, IndexError, TypeError):
+            pass
+        return None
+
+    def _extract_authors(self, authors_data: list) -> list[str]:
+        """Extract author names from Crossref author data"""
+        authors = []
+        for author in authors_data or []:
+            if isinstance(author, dict):
+                given = author.get("given", "")
+                family = author.get("family", "")
+                if given and family:
+                    authors.append(f"{given} {family}")
+                elif family:
+                    authors.append(family)
+                elif given:
+                    authors.append(given)
+        return authors
+
+    def _parse_work(self, work: dict) -> Optional[Paper]:
+        """Parse a single work from Crossref API response"""
+        try:
+            # Extract basic information
+            title_list = work.get("title", [])
+            title = title_list[0] if title_list else ""
+
+            if not title:
+                return None
+
+            doi = work.get("DOI", "")
+            paper_id = doi or work.get("URL", "")
+
+            # Extract authors
+            authors = self._extract_authors(work.get("author", []))
+
+            # Extract abstract
+            abstract = work.get("abstract", "")
+            if abstract:
+                # Remove HTML tags if present
+                import re
+
+                abstract = re.sub(r"<[^>]+>", "", abstract)
+
+            # Extract publication date
+            published_date = (
+                self._parse_date(work.get("published-print", {}).get("date-parts"))
+                or self._parse_date(work.get("published-online", {}).get("date-parts"))
+                or self._parse_date(work.get("created", {}).get("date-parts"))
+            )
+
+            # Extract URLs
+            url = work.get("URL", "")
+            pdf_url = ""
+
+            # Look for PDF in links
+            links = work.get("link", [])
+            for link in links:
+                if link.get("content-type") == "application/pdf":
+                    pdf_url = link.get("URL", "")
+                    break
+
+            # Extract additional metadata
+            container_title = work.get("container-title", [])
+            journal = container_title[0] if container_title else ""
+
+            volume = work.get("volume", "")
+            issue = work.get("issue", "")
+            pages = work.get("page", "")
+
+            # Extract categories/subjects
+            categories = []
+            subjects = work.get("subject", [])
+            if subjects:
+                categories.extend(subjects)
+
+            # Citation count (if available)
+            citations = work.get("is-referenced-by-count", 0)
+
+            # Build extra metadata
+            extra = {
+                "journal": journal,
+                "volume": volume,
+                "issue": issue,
+                "pages": pages,
+                "type": work.get("type", ""),
+                "publisher": work.get("publisher", ""),
+                "issn": work.get("ISSN", []),
+                "isbn": work.get("ISBN", []),
+            }
+
+            # Remove empty values from extra
+            extra = {k: v for k, v in extra.items() if v}
+
+            return Paper(
+                paper_id=paper_id,
+                title=title,
+                authors=authors,
+                abstract=abstract,
+                doi=doi,
+                published_date=published_date or datetime(1900, 1, 1),
+                pdf_url=pdf_url,
+                url=url,
+                source="crossref",
+                categories=categories,
+                citations=citations,
+                extra=extra,
+            )
+
+        except Exception as e:
+            logger.error(f"Error parsing Crossref work: {e}")
+            return None
+
+    def search(
+        self,
+        query: str,
+        max_results: int = 10,
+        year_min: Optional[int] = None,
+        year_max: Optional[int] = None,
+        sort_by: str = "relevance",
+        **kwargs,
+    ) -> list[Paper]:
+        """
+        Search for papers using Crossref API
+
+        Args:
+            query: Search query string
+            max_results: Maximum number of results to return
+            year_min: Minimum publication year
+            year_max: Maximum publication year
+            sort_by: Sort order (relevance, published, indexed, updated)
+        """
+        if not query.strip():
+            return []
+
+        try:
+            params = {
+                "query": query,
+                "rows": min(max_results, 1000),  # Crossref max is 1000
+                "sort": sort_by,
+                "select": "DOI,title,author,abstract,published-print,published-online,created,URL,container-title,volume,issue,page,subject,is-referenced-by-count,type,publisher,ISSN,ISBN,link",
+            }
+
+            # Add year filters if specified
+            filters = []
+            if year_min:
+                filters.append(f"from-pub-date:{year_min}")
+            if year_max:
+                filters.append(f"until-pub-date:{year_max}")
+
+            if filters:
+                params["filter"] = ",".join(filters)
+
+            response = self.client.get(
+                self.WORKS_ENDPOINT, params=params, headers=self._get_headers()
+            )
+            response.raise_for_status()
+
+            data = response.json()
+            works = data.get("message", {}).get("items", [])
+
+            papers = []
+            for work in works:
+                paper = self._parse_work(work)
+                if paper:
+                    papers.append(paper)
+
+            return papers[:max_results]
+
+        except Exception as e:
+            logger.error(f"Error searching Crossref: {e}")
+            return []
+
+    def download_pdf(self, paper_id: str, save_path: str) -> str:
+        """
+        Download PDF for a paper (limited functionality for Crossref)
+
+        Note: Crossref is primarily a metadata service. PDF downloads
+        depend on publisher policies and may not always be available.
+        """
+        if not paper_id:
+            return "Error: paper_id is required"
+
+        try:
+            # If paper_id is a DOI, try to get work details first
+            if not paper_id.startswith("http"):
+                work_url = f"{self.WORKS_ENDPOINT}/{quote_plus(paper_id)}"
+                response = self.client.get(work_url, headers=self._get_headers())
+                response.raise_for_status()
+
+                work_data = response.json()
+                work = work_data.get("message", {})
+
+                # Look for PDF link
+                links = work.get("link", [])
+                pdf_url = None
+                for link in links:
+                    if link.get("content-type") == "application/pdf":
+                        pdf_url = link.get("URL")
+                        break
+
+                if not pdf_url:
+                    return f"Error: No PDF link found for DOI {paper_id}. Crossref provides metadata; PDFs are hosted by publishers."
+            else:
+                pdf_url = paper_id
+
+            # Attempt to download PDF
+            from pathlib import Path
+
+            save_path_obj = Path(save_path)
+            save_path_obj.mkdir(parents=True, exist_ok=True)
+
+            # Create filename from DOI or URL
+            if paper_id.startswith("10."):
+                filename = (
+                    f"crossref_{paper_id.replace('/', '_').replace('.', '_')}.pdf"
+                )
+            else:
+                filename = f"crossref_paper_{hash(paper_id) % 10000}.pdf"
+
+            file_path = save_path_obj / filename
+
+            pdf_response = self.client.get(pdf_url, headers=self._get_headers())
+            pdf_response.raise_for_status()
+
+            with open(file_path, "wb") as f:
+                f.write(pdf_response.content)
+
+            return str(file_path)
+
+        except Exception as e:
+            return f"Error downloading PDF: {e}"
+
+    def read_paper(self, paper_id: str, save_path: str) -> str:
+        """
+        Read paper text (downloads PDF first if needed)
+
+        Note: Success depends on PDF availability from publishers
+        """
+        if not paper_id:
+            return "Error: paper_id is required"
+
+        try:
+            # First try to download the PDF
+            pdf_path = self.download_pdf(paper_id, save_path)
+
+            if pdf_path.startswith("Error"):
+                return pdf_path
+
+            # Read the PDF using the existing read_pdf function
+            from ..paper import read_pdf
+
+            return read_pdf(pdf_path)
+
+        except Exception as e:
+            return f"Error reading paper: {e}"
+
+    def search_by_doi(self, doi: str) -> Optional[Paper]:
+        """Search for a specific paper by DOI"""
+        try:
+            work_url = f"{self.WORKS_ENDPOINT}/{quote_plus(doi)}"
+            response = self.client.get(work_url, headers=self._get_headers())
+            response.raise_for_status()
+
+            data = response.json()
+            work = data.get("message", {})
+
+            return self._parse_work(work)
+
+        except Exception as e:
+            logger.error(f"Error searching by DOI {doi}: {e}")
+            return None
+
+    def __del__(self):
+        """Clean up HTTP client"""
+        if hasattr(self, "client"):
+            self.client.close()