PyPI - scitex - Versions diffs - 2.13.0__py3-none-any.whl → 2.15.1__py3-none-any.whl - Mend

scitex 2.13.0py3-none-any.whl → 2.15.1py3-none-any.whl

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (1429) hide show

scitex/scholar/.legacy/metadata/doi/README.md DELETED Viewed

@@ -1,368 +0,0 @@
-<!-- ---
-!-- Timestamp: 2025-08-14 11:04:39
-!-- Author: ywatanabe
-!-- File: /home/ywatanabe/proj/SciTeX-Code/src/scitex/scholar/metadata/doi/README.md
-!-- --- -->
-# Scholar DOI Resolution System
-A comprehensive, production-ready DOI resolution system with multiple academic sources, intelligent rate limiting, and resumable batch processing. Now featuring a **unified resolver API** that automatically handles all input types.
-## 🚀 Quick Start
-### Simple, Reliable Methods (Recommended)
-```python
-from scitex.scholar.metadata.doi import DOIResolver
-await DOIResolver().metadata2doi_async("Attention is All You Need")
-DOIResolver.resolve_async("/home/ywatanabe/win/downloads/papers.bib")
-from scitex.scholar.metadata.doi import BibTeXDOIResolver, SingleDOIResolver
-# Method 1: Resolve DOIs from BibTeX file (most common use case)
-resolver = BibTeXDOIResolver()
-results = await resolver.resolve_from_bibtex("/home/ywatanabe/win/downloads/", project="my_research")
-# Method 2: Find DOI for a single paper by title/authors
-single_resolver = SingleDOIResolver()
-doi = await single_resolver.resolve_by_title(
-    title="Deep Learning",
-    authors=["LeCun, Y.", "Bengio, Y.", "Hinton, G."],
-    year=2015
-)
-# Method 3: Enrich papers that already have DOIs
-from scitex.scholar.metadata.enrichment import MetadataEnricher
-enricher = MetadataEnricher()
-enriched = await enricher.enrich_papers_with_metadata(papers)
-```
-### ⚠️ What NOT to do
-```python
-# DON'T: Pass a DOI to "resolve" it - DOIs are already resolved!
-# This is redundant and will cause unnecessary API calls:
-resolver.resolve_async("10.1126/science.aao0702")  # ❌ Wrong
-# DO: Use DOIs directly or fetch their metadata
-doi = "10.1126/science.aao0702"  # ✅ DOI is already resolved
-metadata = await enricher.fetch_metadata_for_doi(doi)  # ✅ Fetch metadata if needed
-```
-### Command Line Interface
-```bash
-# Resolve DOIs from a BibTeX file
-python -m scitex.scholar.cli.resolve_dois --bibtex papers.bib --project my_research
-# Find DOI for a specific paper
-python -m scitex.scholar.cli.resolve_dois --title "Deep Learning" --year 2015
-# Resume interrupted batch processing
-python -m scitex.scholar.cli.resolve_dois --bibtex papers.bib --resume
-```
-## 🏗️ System Architecture
-### Core Components
-#### **DOI Sources** (Priority Order)
-1. **URL Extractor** - Immediate DOI extraction from URL fields
-2. **CrossRef** - Comprehensive academic database (90M+ records)
-3. **Semantic Scholar** - AI/CS focused with enhanced CorpusID support
-4. **PubMed** - Biomedical literature database
-5. **OpenAlex** - Open scholarly database
-6. **ArXiv** - Preprint server for physics, math, CS
-#### **Resolution Pipeline**
-```
-Title/Metadata Input
-    ↓
-URL DOI Extraction (instant)
-    ↓
-Multi-Source API Resolution (parallel)
-    ↓
-Enhanced Title Matching (Jaccard + LaTeX/Unicode)
-    ↓
-DOI Validation & Cleaning
-    ↓
-Scholar Library Integration
-```
-### **New Features (2025-08-04)**
-#### ✅ **Enhanced CorpusID Support**
-- **Semantic Scholar CorpusID resolution** via API
-- **Dynamic rate limiting** with API key detection
-- **URL pattern matching** for CorpusId: format papers
-#### ✅ **Utility-Based Architecture**
-- **TextNormalizer** - LaTeX/Unicode handling, fuzzy title matching
-- **URLDOISource** - DOI extraction from multiple URL patterns
-- **PubMedConverter** - PMID to DOI conversion via E-utilities
-- **Shared utilities** across all source classes (Phase 1 refactoring completed)
-#### ✅ **Production Rate Limiting**
-- **Source-aware delays** (0.5-2.0s based on API keys)
-- **Exponential backoff** with jitter for failures
-- **429 error handling** with automatic retry
-- **Request statistics** tracking per source
-## 📖 Usage Guide
-### Clear, Purpose-Built Methods
-Use the right tool for the right job:
-```python
-# 1. BibTeX file -> DOIs (batch resolution)
-from scitex.scholar.metadata.doi import BibTeXDOIResolver
-resolver = BibTeXDOIResolver(project="my_research")
-results = await resolver.resolve_from_bibtex(
-    "papers.bib",
-    max_workers=8,                   # Concurrent processing
-    sources=["crossref", "pubmed"],  # Specific sources
-    resume=True                      # Resume from interruption
-)
-# 2. Paper metadata -> DOI (single resolution)
-from scitex.scholar.metadata.doi import SingleDOIResolver
-resolver = SingleDOIResolver()
-doi = await resolver.resolve_by_title(
-    title="Machine Learning Fundamentals",
-    authors=["Smith, J."],
-    year=2023
-)
-# 3. DOI -> Enriched metadata
-from scitex.scholar.metadata.enrichment import MetadataEnricher
-enricher = MetadataEnricher()
-metadata = await enricher.fetch_metadata_for_doi("10.1038/nature12373")
-```
-### Advanced Usage (Legacy API)
-For fine-grained control, the internal resolvers are still available:
-```python
-# Advanced users can still access specific resolvers
-from scitex.scholar.doi.resolvers._SingleDOIResolver import SingleDOIResolver
-from scitex.scholar.doi.resolvers._BibTeXDOIResolver import BibTeXDOIResolver
-# But most users should use the unified DOIResolver instead
-```
-### Configuration
-The system uses the centralized config at `src/scitex/scholar/config/_ScholarConfig.py`:
-```python
-# Email configuration for API compliance
-SCITEX_SCHOLAR_CROSSREF_EMAIL=research@yourorg.edu
-SCITEX_SCHOLAR_PUBMED_EMAIL=research@yourorg.edu
-SCITEX_SCHOLAR_OPENALEX_EMAIL=research@yourorg.edu
-SCITEX_SCHOLAR_SEMANTIC_SCHOLAR_EMAIL=research@yourorg.edu
-# API keys for enhanced rate limits
-SEMANTIC_SCHOLAR_API_KEY=your_api_key_here
-```
-### Progress Tracking & Resume
-**Rsync-style Progress Display:**
-```
-Resolving DOIs: [=========>          ] 45/75 (60.0%) ✓42 ✗3  2.1 items/s  elapsed: 0:21  eta: 0:14
-```
-**Progress Files:**
-- Auto-saved as `doi_resolution_YYYYMMDD_HHMMSS.progress.json`
-- Contains processed papers, success/failure status, rate limit state
-- Enables seamless resume with `--resume` flag
-**Statistics Tracking:**
-```json
-{
-  "statistics": {
-    "total": 75,
-    "processed": 45,
-    "resolved": 42,
-    "failed": 3,
-    "success_rate": 93.3,
-    "avg_time_per_paper": 2.1
-  }
-}
-```
-## 🔧 Advanced Features
-### Enhanced Source Capabilities
-#### **Semantic Scholar Enhanced**
-- **CorpusID resolution**: Converts Semantic Scholar CorpusId to DOI
-- **API key support**: Faster rate limits with registered API key
-- **Comprehensive metadata**: Authors, abstracts, venue information
-- **Fallback strategies**: Multiple DOI extraction methods
-#### **URL DOI Extractor**
-- **Direct DOI patterns**: doi.org, dx.doi.org URLs
-- **Publisher-specific**: IEEE, PubMed, Semantic Scholar patterns
-- **ID conversion**: PMID → DOI, CorpusID → DOI
-- **Immediate results**: No API calls for URL-based papers
-#### **Enhanced Title Matching**
-- **Unicode normalization**: Handles accented characters
-- **LaTeX processing**: Converts LaTeX markup (e.g., `\{\"u\}` → `ü`)
-- **Jaccard similarity**: Word-overlap based fuzzy matching
-- **Stop word filtering**: Improved matching accuracy
-### Performance Optimizations
-- **Concurrent processing**: Configurable worker_async pools (1-16 worker_asyncs)
-- **Intelligent caching**: LRU cache with 1000 entry limit
-- **Source rotation**: Dynamic source prioritization based on success rates
-- **Duplicate detection**: Groups similar papers to avoid redundant API calls
-### Error Handling & Resilience
-- **Graceful degradation**: Continues with available sources if some fail
-- **Rate limit detection**: Automatic detection and adaptive delays
-- **Connection resilience**: Timeout handling with exponential backoff
-- **Partial progress**: Saves progress after each successful resolution
-## 🗂️ Scholar Library Integration
-### Directory Structure
-```
-~/.scitex/scholar/library/
-├── MASTER/                    # Master paper collection
-│   ├── 8DIGITS_PAPERID/
-│   │   ├── metadata.json     # Paper metadata with DOI
-│   │   ├── attachments/      # PDFs and supplementary files
-│   │   └── screenshots/      # Download process screenshots
-│   └── ...
-└── project_name/             # Project-specific symlinks
-    ├── AUTHOR-YEAR-JOURNAL -> ../MASTER/8DIGITS_PAPERID/
-    └── ...
-```
-### Database Integration
-- **Automatic entries**: DOI resolution creates Scholar library entries
-- **Metadata enrichment**: Title, authors, journal, year, abstract
-- **Source attribution**: Tracks which source provided each DOI
-- **Symlink management**: Project-specific organization with readable names
-## 📊 Current Performance
-**PAC Project Results (75 papers):**
-- **Resolved**: 53/75 papers (70.7% coverage)
-- **Enhanced resolver**: 45/49 unresolved papers (91.8% success rate)
-- **Processing speed**: ~2.1 papers/second with rate limiting
-- **Source breakdown**: URL extraction (14), CrossRef (18), Semantic Scholar (12), PubMed (6), OpenAlex (3)
-## 🛠️ Recent Refactoring (Phase 1 Completed)
-**Utility Consolidation:**
-- ✅ Eliminated 150+ lines of duplicate code across source files
-- ✅ All sources now use shared `TextNormalizer`, `URLDOISource`, `PubMedConverter`
-- ✅ Enhanced `BaseDOISource` with lazy-loaded utility access
-- ✅ Consistent text processing and DOI extraction across all sources
-**Pending Refactoring:**
-- 🔄 **Phase 2**: Unify three different rate limiting implementations
-- 🔄 **Phase 3**: Decompose 1000+ line resolver files into focused classes
-## 🤝 Multi-Agent Development
-This system was enhanced through multi-agent collaboration:
-- **Main Agent**: Core functionality and API integration
-- **Code Reviewer**: Best practices and design patterns
-- **Code Refactorer**: Utility consolidation and code cleanup
-See `project_management/BULLETIN-BOARD.md` for collaboration details.
-## 📚 Command Reference
-### Basic Commands
-**New Unified Interface:**
-```bash
-# Interactive demo
-python -m scitex.scholar.doi --demo
-# Resolve any input type automatically
-python -m scitex.scholar.doi "10.1038/nature12373"    # Single DOI
-python -m scitex.scholar.doi "research_papers.bib"    # BibTeX file
-```
-**Legacy Interface (still supported):**
-```bash
-# Single paper
-python -m scitex.scholar.cli.resolve_doi_asyncs --title "Paper Title"
-# BibTeX batch
-python -m scitex.scholar.cli.resolve_doi_asyncs --bibtex papers.bib
-# Enhanced mode
-python -m scitex.scholar.cli.resolve_doi_asyncs --bibtex papers.bib --enhanced --worker_asyncs 8
-# Resume interrupted
-python -m scitex.scholar.cli.resolve_doi_asyncs --bibtex papers.bib --resume
-```
-### Advanced Options
-```bash
-# Source selection
---sources crossref semantic_scholar pubmed
-# Output options
---output results.json --update-bibtex
-# Performance tuning
---worker_asyncs 8 --enhanced
-# Logging control
---verbose --quiet
-# Progress management
---resume --progress custom_progress.json
-```
-## 🔗 Integration Points
-- **SciTeX Scholar Module**: Core component of academic research automation
-- **Config System**: Uses centralized `_ScholarConfig.py` for email/API management
-- **Scholar Library**: Automatic paper database population
-- **PDF Download**: Provides DOIs for subsequent PDF acquisition workflows
-## ✨ API Design Philosophy
-**Clear intent over magic**: Each method has a single, clear purpose:
-- **`resolve_by_title()`**: Find DOI when you have paper metadata
-- **`resolve_from_bibtex()`**: Process BibTeX files to add missing DOIs
-- **`fetch_metadata_for_doi()`**: Get metadata when you already have a DOI
-**Why explicit methods are better:**
-- ✅ **No ambiguity**: You know exactly what will happen
-- ✅ **Better error messages**: Failures are easier to understand
-- ✅ **Predictable performance**: No surprise API calls
-- ✅ **Easier to test**: Each method has clear inputs/outputs
-```python
-# Bad: "Magic" API that guesses what you want
-resolver.resolve(something)  # What does this do? Hard to tell!
-# Good: Clear, purpose-built methods
-resolver.resolve_by_title(title="...")     # Obviously searches for DOI
-resolver.resolve_from_bibtex("file.bib")   # Obviously processes BibTeX
-enricher.fetch_metadata_for_doi(doi)       # Obviously fetches metadata
-```
----
-**Status**: ✅ Production Ready | 🆕 Unified API Available | 🔄 Phase 2-3 Refactoring Planned
-**Last Updated**: 2025-08-05 by Claude Code
-**Coverage**: 70.7% (53/75) in current PAC project with 91.8% success rate on difficult papers
-<!-- EOF -->

scitex/scholar/.legacy/metadata/doi/__init__.py DELETED Viewed

@@ -1,7 +0,0 @@
-from .resolvers._DOIResolver import DOIResolver
-__all__ = [
-    "DOIResolver",
-]
-# EOF

scitex/scholar/.legacy/metadata/doi/batch/_MetadataHandlerForBatchDOIResolution.py DELETED Viewed

@@ -1,304 +0,0 @@
-#!/usr/bin/env python3
-# -*- coding: utf-8 -*-
-# Timestamp: "2025-08-11 07:22:30 (ywatanabe)"
-# File: /home/ywatanabe/proj/scitex_repo/src/scitex/scholar/metadata/doi/batch/_MetadataHandlerForBatchDOIResolution.py
-# ----------------------------------------
-from __future__ import annotations
-import os
-__FILE__ = __file__
-__DIR__ = os.path.dirname(__FILE__)
-# ----------------------------------------
-"""Metadata processing and enhancement for batch DOI resolution."""
-import re
-from difflib import SequenceMatcher
-from typing import Any, Dict, List, Optional
-from scitex import logging
-logger = logging.getLogger(__name__)
-class MetadataHandlerForBatchDOIResolution:
-    """Handles paper metadata processing, validation, and enhancement.
-    Responsibilities:
-    - Title normalization for deduplication
-    - Paper similarity detection and grouping
-    - BibTeX author string parsing
-    - Year parsing and validation
-    - Paper key generation for tracking
-    """
-    def __init__(self, similarity_threshold: float = 0.85):
-        """Initialize metadata enhancer.
-        Args:
-            similarity_threshold: Similarity threshold for duplicate detection (0-1)
-        """
-        self.similarity_threshold = similarity_threshold
-    @staticmethod
-    def normalize_title(title: str) -> str:
-        """Normalize title for deduplication.
-        Args:
-            title: Raw title string
-        Returns:
-            Normalized title string
-        """
-        if not title:
-            return ""
-        # Remove common variations
-        normalized = title.lower().strip()
-        normalized = re.sub(r"[^\w\s]", "", normalized)  # Remove punctuation
-        normalized = re.sub(r"\s+", " ", normalized)  # Normalize whitespace
-        return normalized
-    def find_similar_papers(self, papers: List[Dict[str, Any]]) -> Dict[str, List[int]]:
-        """Find potentially duplicate papers based on title similarity.
-        Args:
-            papers: List of paper dictionaries with 'title' field
-        Returns:
-            Dictionary mapping group IDs to lists of paper indices
-        """
-        groups = {}
-        processed = set()
-        for i, paper1 in enumerate(papers):
-            if i in processed:
-                continue
-            title1 = self.normalize_title(paper1.get("title", ""))
-            if not title1:
-                continue
-            group = [i]
-            for j, paper2 in enumerate(papers[i + 1 :], start=i + 1):
-                if j in processed:
-                    continue
-                title2 = self.normalize_title(paper2.get("title", ""))
-                if not title2:
-                    continue
-                # Check similarity
-                similarity = SequenceMatcher(None, title1, title2).ratio()
-                if similarity > self.similarity_threshold:
-                    group.append(j)
-                    processed.add(j)
-            if len(group) > 1:
-                group_key = f"group_{len(groups) + 1}"
-                groups[group_key] = group
-                logger.warning(
-                    f"Found potential duplicates ({len(group)} papers): "
-                    f"{[papers[idx].get('title', '')[:50] + '...' for idx in group[:2]]}"
-                )
-            processed.add(i)
-        return groups
-    @staticmethod
-    def parse_authors(authors_str: str) -> List[str]:
-        """Parse BibTeX author string into individual author names.
-        Args:
-            authors_str: BibTeX format author string (e.g., "Smith, J. and Doe, A.")
-        Returns:
-            List of individual author names
-        """
-        if not authors_str:
-            return []
-        return [a.strip() for a in authors_str.split(" and ") if a.strip()]
-    @staticmethod
-    def parse_year(year_str: str) -> Optional[int]:
-        """Parse year from string, handling various formats.
-        Args:
-            year_str: Year string (e.g., "2023", "2023-01-01", etc.)
-        Returns:
-            Parsed year as integer or None if parsing fails
-        """
-        if not year_str:
-            return None
-        try:
-            # Handle simple year
-            if year_str.isdigit():
-                return int(year_str)
-            # Handle date formats like "2023-01-01"
-            year_match = re.search(r"(\d{4})", str(year_str))
-            if year_match:
-                return int(year_match.group(1))
-            return None
-        except (ValueError, TypeError):
-            return None
-    def get_paper_key(self, paper: Dict[str, Any]) -> str:
-        """Generate unique key for paper tracking.
-        Args:
-            paper: Paper dictionary with title and optionally year
-        Returns:
-            Unique paper key for tracking
-        """
-        title = self.normalize_title(paper.get("title", ""))
-        year = self.parse_year(str(paper.get("year", "")))
-        if year:
-            return f"{title}_{year}"
-        return title
-    def validate_paper_metadata(self, paper: Dict[str, Any]) -> Dict[str, Any]:
-        """Validate and enhance paper metadata.
-        Args:
-            paper: Paper dictionary to validate
-        Returns:
-            Dictionary with validation results and enhanced metadata
-        """
-        result = {"valid": True, "warnings": [], "enhanced": {}}
-        # Check required fields
-        title = paper.get("title", "")
-        if not title or not title.strip():
-            result["valid"] = False
-            result["warnings"].append("Missing or empty title")
-        else:
-            result["enhanced"]["normalized_title"] = self.normalize_title(title)
-        # Parse and validate year
-        year_str = paper.get("year", "")
-        if year_str:
-            parsed_year = self.parse_year(year_str)
-            if parsed_year:
-                result["enhanced"]["parsed_year"] = parsed_year
-                # Validate year range (reasonable publication years)
-                if parsed_year < 1900 or parsed_year > 2030:
-                    result["warnings"].append(
-                        f"Unusual publication year: {parsed_year}"
-                    )
-            else:
-                result["warnings"].append(f"Could not parse year: {year_str}")
-        # Parse authors
-        authors_str = paper.get("authors", "") or paper.get("author", "")
-        if authors_str:
-            parsed_authors = self.parse_authors(authors_str)
-            result["enhanced"]["parsed_authors"] = parsed_authors
-            result["enhanced"]["author_count"] = len(parsed_authors)
-        else:
-            result["warnings"].append("Missing author information")
-        # Generate paper key
-        result["enhanced"]["paper_key"] = self.get_paper_key(paper)
-        return result
-    def enhance_paper_batch(self, papers: List[Dict[str, Any]]) -> Dict[str, Any]:
-        """Enhance a batch of papers with metadata processing.
-        Args:
-            papers: List of paper dictionaries to enhance
-        Returns:
-            Dictionary with enhancement results and statistics
-        """
-        results = {
-            "enhanced_papers": [],
-            "duplicate_groups": {},
-            "statistics": {
-                "total": len(papers),
-                "valid": 0,
-                "with_warnings": 0,
-                "duplicates_found": 0,
-            },
-        }
-        # Enhance individual papers
-        for paper in papers:
-            validation = self.validate_paper_metadata(paper)
-            enhanced_paper = {**paper, **validation["enhanced"]}
-            enhanced_paper["_validation"] = validation
-            results["enhanced_papers"].append(enhanced_paper)
-            if validation["valid"]:
-                results["statistics"]["valid"] += 1
-            if validation["warnings"]:
-                results["statistics"]["with_warnings"] += 1
-        # Find duplicates
-        duplicate_groups = self.find_similar_papers(papers)
-        results["duplicate_groups"] = duplicate_groups
-        results["statistics"]["duplicates_found"] = sum(
-            len(group) for group in duplicate_groups.values()
-        )
-        return results
-if __name__ == "__main__":
-    # Example usage
-    enhancer = MetadataHandlerForBatchDOIResolution()
-    # Test papers
-    test_papers = [
-        {
-            "title": "Machine Learning for Natural Language Processing",
-            "year": "2023",
-            "authors": "Smith, J. and Doe, A. and Johnson, B.",
-        },
-        {
-            "title": "Machine learning for natural language processing",  # Similar to first
-            "year": "2023",
-            "authors": "Smith, John and Doe, Alice",
-        },
-        {
-            "title": "Deep Learning Approaches",
-            "year": "invalid_year",
-            "authors": "Brown, C.",
-        },
-        {"title": "", "year": "2022"},  # Invalid title
-    ]
-    # Enhance papers
-    results = enhancer.enhance_paper_batch(test_papers)
-    print("Enhancement Results:")
-    print(f"Total papers: {results['statistics']['total']}")
-    print(f"Valid papers: {results['statistics']['valid']}")
-    print(f"Papers with warnings: {results['statistics']['with_warnings']}")
-    print(f"Duplicates found: {results['statistics']['duplicates_found']}")
-    print("\nDuplicate groups:")
-    for group_id, indices in results["duplicate_groups"].items():
-        print(f"  {group_id}: {indices}")
-    print("\nIndividual paper results:")
-    for i, paper in enumerate(results["enhanced_papers"]):
-        validation = paper["_validation"]
-        print(
-            f"  Paper {i}: Valid={validation['valid']}, Warnings={len(validation['warnings'])}"
-        )
-        if validation["warnings"]:
-            print(f"    Warnings: {validation['warnings']}")
-# python -m scitex.scholar.metadata.doi.batch._MetadataHandlerForBatchDOIResolution
-# EOF

scitex 2.13.0__py3-none-any.whl → 2.15.1__py3-none-any.whl

scitex 2.13.0py3-none-any.whl → 2.15.1py3-none-any.whl