scitex 2.1.2__py3-none-any.whl → 2.4.2__py3-none-any.whl

scitex/scholar/utils/validation/README.md

@@ -0,0 +1,169 @@
+# DOI Validation - scitex.scholar
+
+Validates DOI accessibility by checking resolution at https://doi.org/<DOI>
+
+## Components
+
+### 1. DOIValidator (Core Class)
+
+**Location**: `scitex/scholar/utils/validation/DOIValidator.py`
+
+**Features**:
+- Validates DOI format (must start with "10.", contain "/", etc.)
+- Checks DOI accessibility via https://doi.org/<DOI>
+- Supports retries on timeout
+- Returns validation status + resolved URL
+- Batch validation with rate limiting
+
+**Usage**:
+```python
+from scitex.scholar.utils import DOIValidator
+
+validator = DOIValidator()
+
+# Validate single DOI
+is_valid, message, status_code, resolved_url = validator.validate_doi("10.1186/1751-0473-8-7")
+
+# Batch validation
+results = validator.validate_batch(
+    dois=["10.1186/1751-0473-8-7", "10.1038/invalid"],
+    delay=0.5,  # Be polite to DOI service
+    progress_callback=lambda i, total, doi, valid: print(f"[{i}/{total}] {doi}: {valid}")
+)
+```
+
+### 2. Pipeline Integration
+
+**Location**: `scitex/scholar/pipelines/ScholarPipelineMetadataSingle.py`
+
+DOI validation is **optional** during metadata enrichment (disabled by default to avoid extra requests):
+
+```python
+from scitex.scholar.pipelines import ScholarPipelineMetadataSingle
+
+# With validation enabled
+pipeline = ScholarPipelineMetadataSingle(validate_dois=True)
+enriched_paper = await pipeline.enrich_paper_async(paper)
+
+# Without validation (default - faster)
+pipeline = ScholarPipelineMetadataSingle(validate_dois=False)
+```
+
+When enabled, invalid DOIs are **rejected** and not saved to paper metadata.
+
+### 3. Library Validation Script
+
+**Location**: `scitex/scholar/utils/validation/validate_library_dois.py`
+
+Validates all DOIs in an existing library:
+
+```bash
+# Validate all DOIs in library
+python -m scitex.scholar.utils.validation.validate_library_dois
+
+# Validate with custom delay
+python -m scitex.scholar.utils.validation.validate_library_dois --delay 1.0
+
+# Remove invalid DOIs from metadata files
+python -m scitex.scholar.utils.validation.validate_library_dois --fix
+
+# Save report to JSON
+python -m scitex.scholar.utils.validation.validate_library_dois --output report.json
+```
+
+## Example Output
+
+```
+================================================================================
+DOI VALIDATION REPORT
+================================================================================
+
+[1/102] Git can facilitate greater reproducibility...
+  Paper ID: 714B8892
+  DOI: 10.1186/1751-0473-8-7
+  ✓ Valid (resolved)
+  Resolved to: https://scfbm.biomedcentral.com/articles/10.1186/1751-0473-8-7
+
+[2/102] A Novel Method for Neural Data Analysis...
+  Paper ID: 7C6C8325
+  DOI: 10.1038/s41598-023-12345-6
+  ✗ DOI Not Found (404)
+
+================================================================================
+VALIDATION SUMMARY
+================================================================================
+Total papers: 102
+Papers with DOI: 53
+  ✓ Valid DOIs: 51 (96.2%)
+  ✗ Invalid DOIs: 2 (3.8%)
+Papers without DOI: 49
+```
+
+## Validation Logic
+
+1. **Format Check**: Validates DOI format (10.XXXX/suffix)
+2. **HTTP Request**: HEAD request to https://doi.org/<DOI>
+3. **Response Analysis**:
+   - 200-399: Valid DOI (follows redirects to publisher)
+   - 404: Invalid DOI (not found in DOI system)
+   - 405: Retry with GET (some publishers don't support HEAD)
+   - Timeout: Retry if enabled
+4. **Result**: Returns (is_valid, message, status_code, resolved_url)
+
+## Design Decisions
+
+### Why Validation is Optional in Pipeline?
+
+DOI validation adds HTTP requests, which:
+- Increases enrichment time (0.5s delay per paper × 100 papers = 50 seconds)
+- May hit rate limits if validating large batches
+- Most DOIs from API sources (CrossRef, Semantic Scholar) are already valid
+
+**Recommendation**:
+- **Disable** during initial enrichment (faster)
+- **Enable** for final validation or critical use cases
+- **Run** standalone validation script periodically on existing library
+
+### Rate Limiting
+
+Default delay: **0.5 seconds** between requests
+- Respects DOI service (https://doi.org)
+- Avoids triggering rate limits
+- Configurable via `delay` parameter
+
+## Testing
+
+```bash
+# Test the validator directly
+python -m scitex.scholar.utils.validation.DOIValidator
+
+# Test on your library (dry run - no modifications)
+python -m scitex.scholar.utils.validation.validate_library_dois
+
+# Test with fix mode (removes invalid DOIs)
+python -m scitex.scholar.utils.validation.validate_library_dois --fix
+```
+
+## Integration Points
+
+1. **During Enrichment** (optional): ScholarPipelineMetadataSingle
+2. **After Import**: Standalone validation script
+3. **Periodic Maintenance**: Re-validate library to catch DOIs that become invalid
+
+## Error Handling
+
+| Error | Behavior |
+|-------|----------|
+| Empty DOI | Returns (False, "Empty DOI", 0, None) |
+| Invalid format | Returns (False, "Invalid DOI format", 0, None) |
+| 404 Not Found | Returns (False, "DOI Not Found (404)", 404, None) |
+| Timeout | Retries (configurable), then returns (False, "Timeout", 0, None) |
+| Connection error | Returns (False, "Connection Error", 0, None) |
+| Valid DOI | Returns (True, "Valid (resolved)", 200, resolved_url) |
+
+## Future Enhancements
+
+- [ ] Cache validation results (avoid re-validating same DOIs)
+- [ ] Bulk validation API (if DOI.org provides one)
+- [ ] DOI normalization (handle variations in format)
+- [ ] Integration with DOI metadata enrichment (get title, authors from DOI.org)

scitex/scholar/utils/validation/__init__.py

@@ -0,0 +1,13 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+"""
+Validation utilities for scitex.scholar.
+
+Provides validators for DOIs, metadata, and other scholarly resources.
+"""
+
+from .DOIValidator import DOIValidator
+
+__all__ = ["DOIValidator"]
+
+# EOF

scitex/scholar/utils/validation/validate_library_dois.py

@@ -0,0 +1,221 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+# Timestamp: "2025-11-12 14:00:00 (ywatanabe)"
+# File: /home/ywatanabe/proj/scitex_repo/src/scitex/scholar/utils/validation/validate_library_dois.py
+# ----------------------------------------
+"""
+Validate all DOIs in a scitex.scholar library.
+
+Usage:
+    python -m scitex.scholar.utils.validation.validate_library_dois
+    python -m scitex.scholar.utils.validation.validate_library_dois --fix
+    python -m scitex.scholar.utils.validation.validate_library_dois --project my-project
+"""
+
+import json
+import time
+from pathlib import Path
+from typing import Dict, Optional
+
+from scitex import logging
+from scitex.scholar.config import ScholarConfig
+from .DOIValidator import DOIValidator
+
+logger = logging.getLogger(__name__)
+
+
+def validate_library_dois(
+    library_path: Optional[Path] = None,
+    project: Optional[str] = None,
+    delay_between_requests: float = 0.5,
+    fix_invalid: bool = False,
+    output_file: Optional[Path] = None,
+) -> Dict:
+    """Validate all DOIs in the scholar library.
+
+    Args:
+        library_path: Path to MASTER directory (default: from ScholarConfig)
+        project: Specific project to validate (default: validate all in MASTER)
+        delay_between_requests: Delay in seconds between DOI checks (be polite)
+        fix_invalid: If True, remove invalid DOIs from metadata.json files
+        output_file: Optional path to save validation report as JSON
+
+    Returns:
+        Dictionary with validation results
+    """
+    # Get library path
+    if library_path is None:
+        config = ScholarConfig()
+        library_path = config.path_manager.get_library_master_dir()
+
+    logger.info(f"Validating DOIs in: {library_path}")
+    logger.info(f"Delay between requests: {delay_between_requests}s")
+    if fix_invalid:
+        logger.warning("Fix mode enabled - invalid DOIs will be removed!")
+
+    # Initialize validator
+    validator = DOIValidator()
+
+    results = {
+        'total_papers': 0,
+        'papers_with_doi': 0,
+        'valid_dois': 0,
+        'invalid_dois': 0,
+        'empty_dois': 0,
+        'invalid_details': []
+    }
+
+    # Find all metadata.json files
+    metadata_files = sorted(library_path.glob("*/metadata.json"))
+    results['total_papers'] = len(metadata_files)
+
+    logger.info(f"Found {len(metadata_files)} papers to validate")
+
+    print("\n" + "="*80)
+    print("DOI VALIDATION REPORT")
+    print("="*80)
+    print(f"Library: {library_path}")
+    print(f"Papers: {len(metadata_files)}")
+    print("="*80)
+
+    for i, metadata_file in enumerate(metadata_files, 1):
+        paper_id = metadata_file.parent.name
+
+        # Load metadata
+        with open(metadata_file, 'r') as f:
+            metadata = json.load(f)
+
+        doi = metadata.get('metadata', {}).get('id', {}).get('doi', '')
+        title = metadata.get('metadata', {}).get('basic', {}).get('title', 'No title')[:60]
+
+        if not doi:
+            results['empty_dois'] += 1
+            continue
+
+        results['papers_with_doi'] += 1
+
+        print(f"\n[{i}/{len(metadata_files)}] {title}...")
+        print(f"  Paper ID: {paper_id}")
+        print(f"  DOI: {doi}")
+
+        # Check DOI accessibility
+        is_valid, message, status_code, resolved_url = validator.validate_doi(doi)
+
+        if is_valid:
+            results['valid_dois'] += 1
+            logger.success(f"  ✓ {message}")
+            if resolved_url:
+                print(f"  Resolved to: {resolved_url[:70]}...")
+        else:
+            results['invalid_dois'] += 1
+            logger.error(f"  ✗ {message}")
+
+            results['invalid_details'].append({
+                'paper_id': paper_id,
+                'title': title,
+                'doi': doi,
+                'reason': message,
+                'status_code': status_code,
+                'metadata_file': str(metadata_file)
+            })
+
+            # Fix invalid DOI if requested
+            if fix_invalid:
+                logger.warning(f"  Removing invalid DOI from {metadata_file}")
+                metadata['metadata']['id']['doi'] = ''
+                metadata['metadata']['id']['doi_engines'] = []
+                metadata['metadata']['url']['doi'] = None
+                metadata['metadata']['url']['doi_engines'] = []
+
+                with open(metadata_file, 'w') as f:
+                    json.dump(metadata, f, indent=2, ensure_ascii=False)
+
+                logger.info(f"  ✓ Removed invalid DOI from metadata")
+
+        # Be polite to DOI service
+        if i < len(metadata_files):
+            time.sleep(delay_between_requests)
+
+    # Summary
+    print("\n" + "="*80)
+    print("VALIDATION SUMMARY")
+    print("="*80)
+    print(f"Total papers: {results['total_papers']}")
+    print(f"Papers with DOI: {results['papers_with_doi']}")
+    if results['papers_with_doi'] > 0:
+        print(f"  ✓ Valid DOIs: {results['valid_dois']} ({results['valid_dois']*100/results['papers_with_doi']:.1f}%)")
+        print(f"  ✗ Invalid DOIs: {results['invalid_dois']} ({results['invalid_dois']*100/results['papers_with_doi']:.1f}%)")
+    print(f"Papers without DOI: {results['empty_dois']}")
+
+    # Invalid DOI details
+    if results['invalid_details']:
+        print("\n" + "="*80)
+        print(f"INVALID DOI DETAILS ({len(results['invalid_details'])} papers)")
+        print("="*80)
+        for item in results['invalid_details']:
+            print(f"\nTitle: {item['title']}")
+            print(f"  Paper ID: {item['paper_id']}")
+            print(f"  DOI: {item['doi']}")
+            print(f"  Reason: {item['reason']}")
+            print(f"  File: {item['metadata_file']}")
+
+    # Save results to file if requested
+    if output_file:
+        with open(output_file, 'w') as f:
+            json.dump(results, f, indent=2, ensure_ascii=False)
+        logger.success(f"\n✓ Validation results saved to: {output_file}")
+
+    return results
+
+
+if __name__ == "__main__":
+    import argparse
+
+    parser = argparse.ArgumentParser(description='Validate DOIs in scitex.scholar library')
+    parser.add_argument(
+        '--library-path',
+        type=Path,
+        help='Path to MASTER library directory (default: from config)'
+    )
+    parser.add_argument(
+        '--project',
+        type=str,
+        help='Validate specific project only'
+    )
+    parser.add_argument(
+        '--delay',
+        type=float,
+        default=0.5,
+        help='Delay between DOI checks in seconds (default: 0.5)'
+    )
+    parser.add_argument(
+        '--fix',
+        action='store_true',
+        help='Remove invalid DOIs from metadata.json files'
+    )
+    parser.add_argument(
+        '--output',
+        type=Path,
+        help='Save validation report to JSON file'
+    )
+
+    args = parser.parse_args()
+
+    # Validate DOIs
+    results = validate_library_dois(
+        library_path=args.library_path,
+        project=args.project,
+        delay_between_requests=args.delay,
+        fix_invalid=args.fix,
+        output_file=args.output,
+    )
+
+    logger.success("\n✓ DOI validation complete!")
+
+    # Exit with error code if invalid DOIs found
+    if results['invalid_dois'] > 0:
+        exit(1)
+    else:
+        exit(0)
+
+# EOF

scitex/security/README.md

@@ -0,0 +1,255 @@
+# SciTeX Security Module
+
+Reusable security utilities for the SciTeX ecosystem. Works everywhere: local, cloud, CLI.
+
+## Features
+
+- Check GitHub security alerts (secrets, Dependabot, code scanning)
+- Save alerts to timestamped files
+- Python API and CLI interface
+- Reusable across all SciTeX projects
+
+## Installation
+
+The module is part of the `scitex` package (editable install):
+
+```bash
+cd ~/proj/scitex-code
+pip install -e .
+```
+
+## Prerequisites
+
+Install and authenticate GitHub CLI:
+
+```bash
+# Install
+sudo apt install gh  # or: brew install gh
+
+# Authenticate
+gh auth login
+```
+
+## Usage
+
+### Python API
+
+```python
+from scitex.security import check_github_alerts, save_alerts_to_file
+
+# Check alerts for current repo
+alerts = check_github_alerts()
+
+# Check specific repo
+alerts = check_github_alerts(repo="owner/repo")
+
+# Save to file
+file_path = save_alerts_to_file(alerts)
+print(f"Saved to: {file_path}")
+
+# Check if there are open alerts
+total = sum(
+    len([a for a in alerts[key] if a.get("state") == "open"])
+    for key in alerts
+)
+if total > 0:
+    print(f"Found {total} security issues!")
+```
+
+### Command Line
+
+```bash
+# Check current repo
+python -m scitex.security.cli check
+
+# Check specific repo
+python -m scitex.security.cli check --repo SciTeX-AI/scitex-cloud
+
+# Save to file
+python -m scitex.security.cli check --save
+
+# Save to custom directory
+python -m scitex.security.cli check --save --output-dir /path/to/logs
+
+# Show latest report
+python -m scitex.security.cli latest
+```
+
+### From Django (scitex-cloud)
+
+```python
+# In any Django app
+from scitex.security import check_github_alerts, save_alerts_to_file
+
+def check_security_view(request):
+    try:
+        alerts = check_github_alerts()
+        file_path = save_alerts_to_file(
+            alerts,
+            output_dir=settings.BASE_DIR / "logs" / "security"
+        )
+        # Process alerts...
+    except GitHubSecurityError as e:
+        # Handle error...
+```
+
+### From Anywhere
+
+```python
+# Any Python script, anywhere
+from scitex.security import check_github_alerts
+
+alerts = check_github_alerts(repo="SciTeX-AI/scitex-cloud")
+print(f"Secrets: {len(alerts['secrets'])}")
+print(f"Dependabot: {len(alerts['dependabot'])}")
+print(f"Code Scanning: {len(alerts['code_scanning'])}")
+```
+
+## Output Format
+
+Alerts are saved to `logs/security/security-<timestamp>.txt`:
+
+```
+==================================================
+GitHub Security Alerts Report
+Generated: 2025-11-03 12:00:00
+==================================================
+
+### SECRET SCANNING ALERTS ###
+
+- [open] Checkout.com Production Secret Key
+  Created: 2025-11-03T10:42:00Z
+  URL: https://github.com/SciTeX-AI/scitex-cloud/security/secret-scanning/1
+
+==================================================
+
+### DEPENDABOT VULNERABILITY ALERTS ###
+
+- [open] HIGH: Django SQL Injection vulnerability
+  Package: django
+  CVE: CVE-2024-12345
+  URL: https://github.com/SciTeX-AI/scitex-cloud/security/dependabot/1
+
+==================================================
+
+### CODE SCANNING ALERTS ###
+
+No open code scanning alerts
+
+==================================================
+
+### SUMMARY ###
+
+Total open alerts: 2
+  - Secrets: 1
+  - Dependabot: 1
+  - Code Scanning: 0
+
+⚠️  ACTION REQUIRED: Security issues found!
+```
+
+## Integration Examples
+
+### Cron Job
+
+```bash
+# Check daily at 9 AM
+0 9 * * * cd ~/proj/scitex-cloud && python -m scitex.security.cli check --save
+```
+
+### GitHub Actions
+
+```yaml
+- name: Check Security
+  run: |
+    pip install -e ~/proj/scitex-code
+    python -m scitex.security.cli check --save
+```
+
+### Django Management Command
+
+Create `apps/security_app/management/commands/check_security.py`:
+
+```python
+from django.core.management.base import BaseCommand
+from scitex.security import check_github_alerts, save_alerts_to_file
+from django.conf import settings
+
+class Command(BaseCommand):
+    help = 'Check GitHub security alerts'
+
+    def handle(self, *args, **options):
+        alerts = check_github_alerts()
+        file_path = save_alerts_to_file(
+            alerts,
+            output_dir=settings.BASE_DIR / "logs" / "security"
+        )
+        self.stdout.write(f"Report saved to: {file_path}")
+```
+
+Then run:
+```bash
+python manage.py check_security
+```
+
+## Error Handling
+
+```python
+from scitex.security import check_github_alerts, GitHubSecurityError
+
+try:
+    alerts = check_github_alerts()
+except GitHubSecurityError as e:
+    if "not found" in str(e):
+        print("Install GitHub CLI: https://cli.github.com/")
+    elif "Not authenticated" in str(e):
+        print("Run: gh auth login")
+    else:
+        print(f"Error: {e}")
+```
+
+## API Reference
+
+### `check_github_alerts(repo=None)`
+Fetch all security alerts.
+
+**Args:**
+- `repo` (str, optional): Repository in format 'owner/repo'
+
+**Returns:**
+- `dict`: Dictionary with keys 'secrets', 'dependabot', 'code_scanning'
+
+**Raises:**
+- `GitHubSecurityError`: If GitHub CLI not installed or not authenticated
+
+### `save_alerts_to_file(alerts, output_dir=None, create_symlink=True)`
+Save alerts to timestamped file.
+
+**Args:**
+- `alerts` (dict): Dictionary from `check_github_alerts()`
+- `output_dir` (Path, optional): Output directory
+- `create_symlink` (bool): Create 'security-latest.txt' symlink
+
+**Returns:**
+- `Path`: Path to saved file
+
+### `get_latest_alerts_file(security_dir=None)`
+Get path to latest alerts file.
+
+**Args:**
+- `security_dir` (Path, optional): Security directory
+
+**Returns:**
+- `Path` or `None`: Path to latest file
+
+## Testing
+
+```bash
+cd ~/proj/scitex-code
+pytest tests/test_security.py
+```
+
+## See Also
+
+- Main security documentation: `~/proj/scitex-cloud/security.md`
+- GitHub Security: https://docs.github.com/en/code-security

scitex/security/__init__.py

@@ -0,0 +1,35 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+# File: ~/proj/scitex-code/src/scitex/security/__init__.py
+
+"""
+SciTeX Security Module
+
+Reusable security utilities for the SciTeX ecosystem.
+Handles GitHub security alerts, secret scanning, and vulnerability management.
+
+Usage:
+    from scitex.security import check_github_alerts
+
+    alerts = check_github_alerts()
+    if alerts:
+        print(f"Found {len(alerts)} security alerts!")
+"""
+
+from .github import (
+    check_github_alerts,
+    save_alerts_to_file,
+    get_latest_alerts_file,
+    format_alerts_report,
+    GitHubSecurityError,
+)
+
+__all__ = [
+    "check_github_alerts",
+    "save_alerts_to_file",
+    "get_latest_alerts_file",
+    "format_alerts_report",
+    "GitHubSecurityError",
+]
+
+__version__ = "0.1.0"