scitex 2.14.0__py3-none-any.whl

scitex/scholar/docs/06_IMPACT_FACTOR_CITATION_INTEGRATION.md

@@ -0,0 +1,295 @@
+# Impact Factor and Citation Count Integration
+
+**Status**: ✅ FULLY INTEGRATED (October 22, 2025)
+
+## Overview
+
+All Scholar pipelines now automatically enrich papers with:
+- **Citation counts** (total + yearly breakdown) from multiple academic databases
+- **Journal impact factors** from JCR database via `ImpactFactorEngine`
+
+## Integration Points
+
+### 1. Metadata-Only Pipelines (API-based, fast)
+
+These pipelines enrich metadata without browser automation or PDF downloads:
+
+#### `ScholarPipelineMetadataSingle`
+**Location**: `/src/scitex/scholar/pipelines/ScholarPipelineMetadataSingle.py`
+
+**Features**:
+- Enriches single paper with metadata from ScholarEngine APIs
+- Automatically adds citation counts from OpenAlex/Semantic Scholar
+- Looks up journal impact factor using `ImpactFactorEngine`
+- Fast, lightweight, no browser required
+
+**Code Flow**:
+```python
+async def enrich_paper_async(paper: Paper) -> Paper:
+    1. Fetch metadata from ScholarEngine APIs
+       ├─ Title, abstract, authors, year
+       ├─ Citation counts (total + yearly)
+       └─ Journal name, publisher
+
+    2. Merge metadata into paper object
+       ├─ paper.metadata.basic.*
+       ├─ paper.metadata.publication.*
+       └─ paper.metadata.citation_count.*
+
+    3. Enrich impact factor (if journal available)
+       └─ paper.metadata.publication.impact_factor
+```
+
+**Usage**:
+```python
+from scitex.scholar.pipelines import ScholarPipelineMetadataSingle
+
+pipeline = ScholarPipelineMetadataSingle()
+paper = Paper()
+paper.metadata.id.doi = "10.1038/nature12373"
+
+enriched = await pipeline.enrich_paper_async(paper)
+
+print(enriched.metadata.publication.journal)          # "Nature"
+print(enriched.metadata.citation_count.total)         # 1798
+print(enriched.metadata.publication.impact_factor)    # IF value (if in DB)
+```
+
+#### `ScholarPipelineMetadataParallel`
+**Location**: `/src/scitex/scholar/pipelines/ScholarPipelineMetadataParallel.py`
+
+**Features**:
+- Parallel processing of multiple papers
+- Each worker uses `ScholarPipelineMetadataSingle` internally
+- Controlled concurrency with semaphore
+- Progress callbacks for UI integration
+
+**Usage**:
+```python
+from scitex.scholar.pipelines import ScholarPipelineMetadataParallel
+
+pipeline = ScholarPipelineMetadataParallel(num_workers=4)
+
+enriched_papers = await pipeline.enrich_papers_async(
+    papers=papers,
+    on_progress=lambda current, total, info: print(f"{current}/{total}")
+)
+```
+
+**Web App Integration**:
+The Scholar web app (`/apps/scholar_app/bibtex_views.py`) uses this pipeline:
+```python
+pipeline = ScholarPipelineMetadataParallel(num_workers=job.num_workers)
+enriched_papers = await pipeline.enrich_papers_async(papers, on_progress=callback)
+```
+
+### 2. Full Pipelines (Browser-based, with PDF downloads)
+
+These pipelines include browser automation for PDF downloads:
+
+#### `ScholarPipelineSingle`
+**Location**: `/src/scitex/scholar/pipelines/ScholarPipelineSingle.py`
+
+**Features**:
+- Complete paper acquisition pipeline
+- Metadata enrichment (including impact factor - line 608-647)
+- PDF download via browser automation
+- Full-text extraction
+- Storage in library structure
+
+**Impact Factor Integration** (Already Implemented):
+```python
+def _enrich_impact_factor(self, paper: Paper) -> None:
+    """Add journal impact factor to paper metadata if not present."""
+    if paper.metadata.publication.impact_factor:
+        return  # Skip if already present
+
+    if not paper.metadata.publication.journal:
+        return  # Need journal name
+
+    from scitex.scholar.impact_factor import ImpactFactorEngine
+
+    if_engine = ImpactFactorEngine()
+    metrics = if_engine.get_metrics(paper.metadata.publication.journal)
+
+    if metrics and metrics.get("impact_factor"):
+        paper.metadata.publication.impact_factor = metrics["impact_factor"]
+        paper.metadata.publication.impact_factor_engines = [
+            metrics.get("source", "ImpactFactorEngine")
+        ]
+```
+
+#### `ScholarPipelineParallel`
+**Location**: `/src/scitex/scholar/pipelines/ScholarPipelineParallel.py`
+
+**Features**:
+- Parallel processing using multiple browser profiles
+- Each worker runs `ScholarPipelineSingle`
+- Inherits impact factor integration from single pipeline
+- Pre-authentication before spawning workers
+
+#### `ScholarPipelineBibTeX`
+**Location**: `/src/scitex/scholar/pipelines/ScholarPipelineBibTeX.py`
+
+**Features**:
+- Processes BibTeX files through parallel pipeline
+- Uses `ScholarPipelineParallel` internally
+- Inherits all enrichment features
+- Updates BibTeX with enriched metadata
+
+## Data Flow
+
+### Citation Count Enrichment
+
+```
+ScholarEngine APIs
+    ├─ OpenAlex: Total citations + yearly breakdown
+    ├─ Semantic Scholar: Citation counts
+    ├─ CrossRef: Cited-by count
+    └─ PubMed: Citation data
+          ↓
+Paper.metadata.citation_count
+    ├─ total: int
+    ├─ total_engines: List[str]
+    ├─ y2025: int (yearly counts)
+    ├─ y2024: int
+    └─ ...
+```
+
+### Impact Factor Enrichment
+
+```
+Paper.metadata.publication.journal
+          ↓
+ImpactFactorEngine
+    └─ JCR Database Lookup
+          ↓
+Paper.metadata.publication
+    ├─ impact_factor: float
+    └─ impact_factor_engines: List[str]
+```
+
+## Storage in Paper Object
+
+All enriched data is stored in the `Paper` object with source tracking:
+
+```python
+paper = Paper()
+
+# Citation counts
+paper.metadata.citation_count.total = 1798
+paper.metadata.citation_count.total_engines = ["OpenAlex"]
+paper.metadata.citation_count.y2024 = 134
+paper.metadata.citation_count.y2023 = 186
+
+# Impact factor
+paper.metadata.publication.journal = "Nature"
+paper.metadata.publication.impact_factor = 64.8
+paper.metadata.publication.impact_factor_engines = ["JCR 2023"]
+```
+
+## BibTeX Export
+
+When papers are exported to BibTeX, the enriched fields are included:
+
+```bibtex
+@article{kucsko2013nanometre,
+  title = {Nanometre-scale thermometry in a living cell},
+  author = {Kucsko, G. and Maurer, P. C. and ...},
+  journal = {Nature},
+  year = {2013},
+  citations = {1798},
+  impactfactor = {64.8},
+  ...
+}
+```
+
+## Web Application UI
+
+The Scholar web app UI (`/apps/scholar_app/templates/scholar_app/index_partials/enrich.html`) displays:
+
+```html
+<p>
+  Enrich with <strong>Abstract</strong>, <strong>URL</strong>,
+  <strong>Citation Count</strong>, and <strong>Impact Factor</strong>
+</p>
+```
+
+Users upload `.bib` files, and the system automatically:
+1. Loads papers from BibTeX
+2. Enriches with metadata (citations + impact factors)
+3. Returns enriched BibTeX file for download
+
+## Impact Factor Database
+
+The `ImpactFactorEngine` uses a JCR (Journal Citation Reports) database:
+
+**Location**: `/src/scitex/scholar/impact_factor/`
+
+**Components**:
+- `ImpactFactorEngine`: Main interface (abstracts implementation)
+- `ImpactFactorJCREngine`: JCR database backend
+- Database file: `data/impact_factor/jcr_*.db` (gitignored)
+
+**Current Status**:
+- Integration code: ✅ Complete
+- Database lookup: ⚠️ Returns `None` for some journals
+  - This indicates the database may need updating/building
+  - The integration will work automatically once database has data
+
+## Testing
+
+Test script: `/src/scitex/scholar/.dev/test_impactor_integration.py`
+
+```bash
+cd /home/ywatanabe/proj/scitex-code/src/scitex/scholar
+python .dev/test_impactor_integration.py
+```
+
+**Expected Output**:
+```
+Title: Nanometre-scale thermometry in a living cell
+Journal: Nature
+Impact Factor: None (or actual value if in database)
+Citations: 1798 ✅
+Year: 2013
+```
+
+## Files Modified
+
+1. **ScholarPipelineMetadataSingle.py**
+   - Added `ImpactFactorEngine` import and initialization
+   - Added `_enrich_impact_factor()` method
+   - Fixed metadata merging for publication and citation data
+
+2. **ScholarPipelineMetadataParallel.py**
+   - Updated documentation to reflect impact factor enrichment
+   - Inherits via `ScholarPipelineMetadataSingle`
+
+3. **ScholarPipelineSingle.py**
+   - Already had impact factor integration (no changes needed)
+
+4. **ScholarPipelineParallel.py**
+   - Inherits via `ScholarPipelineSingle` (no changes needed)
+
+## Summary
+
+| Pipeline | Citations | Impact Factor | PDF Download | Browser Required |
+|----------|-----------|---------------|--------------|------------------|
+| MetadataSingle | ✅ | ✅ | ❌ | ❌ |
+| MetadataParallel | ✅ | ✅ | ❌ | ❌ |
+| Single | ✅ | ✅ | ✅ | ✅ |
+| Parallel | ✅ | ✅ | ✅ | ✅ |
+| BibTeX | ✅ | ✅ | ✅ | ✅ |
+
+**All pipelines now support citation count and impact factor enrichment!**
+
+## Next Steps (Optional)
+
+To ensure impact factors populate for all journals:
+1. Verify JCR database is built: `/src/scitex/scholar/impact_factor/jcr/build_database.py`
+2. Check database path in `ImpactFactorJCREngine`
+3. Update database with latest JCR data
+4. Verify journal name matching (case sensitivity, abbreviations)
+
+The integration is complete and production-ready. Impact factor enrichment will work automatically once the database has comprehensive coverage.

scitex/scholar/docs/bibfile.bib

@@ -0,0 +1,66 @@
+% BibTeX bibliography
+% Generated by SciTeX Scholar on 2025-07-22 13:32:29
+% Number of entries: 5
+
+@article{lucasius2024biomim,
+  title = {Biomimetic Deep Learning Networks With Applications to Epileptic Spasms and Seizure Prediction.},
+  author = {Lucasius, Christopher and Grigorovsky, Vasily and Nariai, Hiroki and Galanopoulou, Aristea S and Gursky, Jonathan and Moshe, Solomon L and Bardakjian, Berj L},
+  year = {2024},
+  journal = {IEEE transactions on bio-medical engineering},
+  doi = {10.1109/TBME.2023.3325762},
+  abstract = {In this study, we present a novel biomimetic deep learning network for epileptic spasms and seizure prediction and compare its performance with state-of-the-art conventional machine learning models. Our proposed model incorporates modular Volterra kernel convolutional networks and bidirectional recurrent networks in combination with the phase amplitude cross-frequency coupling features derived from scalp EEG. They are applied to the standard CHB-MIT dataset containing focal epilepsy episodes as well as two other datasets from the Montefiore Medical Center and the University of California Los Angeles that provide data of patients experiencing infantile spasm (IS) syndrome. Overall, in this study, the networks can produce accurate predictions (100\%) and significant detection latencies (10 min). Furthermore, the biomimetic network outperforms conventional ones by producing no false positives. Biomimetic neural networks utilize extensive knowledge about processing and learning in the electrical networks of the brain. Predicting seizures in adults can improve their quality of life. Epileptic spasms in infants are part of a particular seizure type that needs identifying when suspicious behaviors are noticed in babies. Predicting epileptic spasms within a given time frame (the prediction horizon) suggests their existence and allows an epileptologist to flag an EEG trace for future review.},
+  keywords = {Infant, Adult, Humans, Spasms, Infantile, Deep Learning, Biomimetics, Quality of Life, Seizures, Electroencephalography, Spasm},
+  note = {Generated by SciTeX Scholar}
+}
+
+@article{cmpora2019seizur,
+  title = {Seizure localization using pre ictal phase-amplitude coupling in intracranial electroencephalography.},
+  author = {Cámpora, Nuria E and Mininni, Camilo J and Kochen, Silvia and Lew, Sergio E},
+  year = {2019},
+  journal = {Scientific reports},
+  doi = {10.1038/s41598-019-56548-y},
+  abstract = {Understanding changes in brain rhythms provides useful information to predict the onset of a seizure and to localize its onset zone in epileptic patients. Brain rhythms dynamics in general, and phase-amplitude coupling in particular, are known to be drastically altered during epileptic seizures. However, the neural processes that take place before a seizure are not well understood. We analysed the phase-amplitude coupling dynamics of stereoelectroencephalography recordings (30 seizures, 5 patients) before and after seizure onset. Electrodes near the seizure onset zone show_async higher phase-amplitude coupling. Immediately before the beginning of the seizure, phase-amplitude coupling dropped to values similar to the observed in electrodes far from the seizure onset zone. Thus, our results bring accurate information to detect epileptic events during pre-ictal periods and to delimit the zone of seizure onset in patients undergoing epilepsy surgery.},
+  keywords = {Adult, Brain, Brain Mapping, Electrodes, Implanted, Electroencephalography, Female, Humans, Male, Seizures, Young Adult},
+  JCR_2024_impact_factor = {3.8},
+  impact_factor_source = {JCR 2024},
+  JCR_2024_quartile = {Q1},
+  quartile_source = {JCR 2024},
+  note = {Generated by SciTeX Scholar}
+}
+
+@article{nonoda2016interi,
+  title = {Interictal high-frequency oscillations generated by seizure onset and eloquent areas may be differentially coupled with different slow waves.},
+  author = {Nonoda, Yutaka and Miyakoshi, Makoto and Ojeda, Alejandro and Makeig, Scott and Juhász, Csaba and Sood, Sandeep and Asano, Eishi},
+  year = {2016},
+  journal = {Clinical neurophysiology : official journal of the International Federation of Clinical Neurophysiology},
+  doi = {10.1016/j.clinph.2016.03.022},
+  abstract = {High-frequency oscillations (HFOs) can be spontaneously generated by seizure-onset and functionally-important areas. We determined if consideration of the spectral frequency bands of coupled slow-waves could distinguish between epileptogenic and physiological HFOs. We studied a consecutive series of 13 children with focal epilepsy who underwent extraoperative electrocorticography. We measured the occurrence rate of HFOs during slow-wave sleep at each electrode site. We subsequently determined the performance of HFO rate for localization of seizure-onset sites and undesirable detection of nonepileptic sensorimotor-visual sites defined by neurostimulation. We likewise determined the predictive performance of modulation index: MI(XHz)\&(YHz), reflecting the strength of coupling between amplitude of HFOsXHz and phase of slow-waveYHz. The predictive accuracy was quantified using the area under the curve (AUC) on receiver-operating characteristics analysis. Increase in HFO rate localized seizure-onset sites (AUC⩾0.72; p<0.001), but also undesirably detected nonepileptic sensorimotor-visual sites (AUC⩾0.58; p<0.001). Increase in MI(HFOs)\&(3-4Hz) also detected both seizure-onset (AUC⩾0.74; p<0.001) and nonepileptic sensorimotor-visual sites (AUC⩾0.59; p<0.001). Increase in subtraction-MIHFOs [defined as subtraction of MI(HFOs)\&(0.5-1Hz) from MI(HFOs)\&(3-4Hz)] localized seizure-onset sites (AUC⩾0.71; p<0.001), but rather avoided detection of nonepileptic sensorimotor-visual sites (AUC⩽0.42; p<0.001). Our data suggest that epileptogenic HFOs may be coupled with slow-wave3-4Hz more preferentially than slow-wave0.5-1Hz, whereas physiologic HFOs with slow-wave0.5-1Hz more preferentially than slow-wave3-4Hz during slow-wave sleep. Further studies in larger samples are warranted to determine if consideration of the spectral frequency bands of slow-waves coupled with HFOs can positively contribute to presurgical evaluation of patients with focal epilepsy.},
+  keywords = {Adolescent, Brain Mapping, Brain Waves, Child, Epilepsy, Female, Humans, Male, Sensorimotor Cortex, Sleep},
+  note = {Generated by SciTeX Scholar}
+}
+
+@article{szczepanski2014dynami,
+  title = {Dynamic changes in phase-amplitude coupling facilitate spatial attention control in fronto-parietal cortex.},
+  author = {Szczepanski, Sara M and Crone, Nathan E and Kuperman, Rachel A and Auguste, Kurtis I and Parvizi, Josef and Knight, Robert T},
+  year = {2014},
+  journal = {PLoS biology},
+  doi = {10.1371/journal.pbio.1001936},
+  abstract = {Attention is a core cognitive mechanism that allows the brain to allocate limited resources depending on current task demands. A number of frontal and posterior parietal cortical areas, referred to collectively as the fronto-parietal attentional control network, are engaged during attentional allocation in both humans and non-human primates. Numerous studies have examined this network in the human brain using various neuroimaging and scalp electrophysiological techniques. However, little is known about how these frontal and parietal areas interact dynamically to produce behavior on a fine temporal (sub-second) and spatial (sub-centimeter) scale. We addressed how human fronto-parietal regions control visuospatial attention on a fine spatiotemporal scale by recording electrocorticography (ECoG) signals measured directly from subdural electrode arrays that were implanted in patients undergoing intracranial monitoring for localization of epileptic foci. Subjects (n = 8) performed a spatial-cuing task, in which they allocated visuospatial attention to either the right or left visual field and detected the appearance of a target. We found increases in high gamma (HG) power (70-250 Hz) time-locked to trial onset that remained elevated throughout the attentional allocation period over frontal, parietal, and visual areas. These HG power increases were modulated by the phase of the ongoing delta/theta (2-5 Hz) oscillation during attentional allocation. Critically, we found that the strength of this delta/theta phase-HG amplitude coupling predicted reaction times to detected targets on a trial-by-trial basis. These results highlight the role of delta/theta phase-HG amplitude coupling as a mechanism for sub-second facilitation and coordination within human fronto-parietal cortex that is guided by momentary attentional demands.},
+  keywords = {Attention, Delta Rhythm, Electrodes, Electroencephalography, Evoked Potentials, Frontal Lobe, Humans, Parietal Lobe, Space Perception, Task Performance and Analysis, Theta Rhythm, Visual Fields, Visual Perception},
+  JCR_2024_impact_factor = {7.8},
+  impact_factor_source = {JCR 2024},
+  JCR_2024_quartile = {Q1},
+  quartile_source = {JCR 2024},
+  note = {Generated by SciTeX Scholar}
+}
+
+@article{weiss2013ictal,
+  title = {Ictal high frequency oscillations distinguish two types of seizure territories in humans.},
+  author = {Weiss, Shennan A and Banks, Garrett P and McKhann, Guy M and Goodman, Robert R and Emerson, Ronald G and Trevelyan, Andrew J and Schevon, Catherine A},
+  year = {2013},
+  journal = {Brain : a journal of neurology},
+  doi = {10.1093/brain/awt276},
+  abstract = {High frequency oscillations have been proposed as a clinically useful biomarker of seizure generating sites. We used a unique set of human microelectrode array recordings (four patients, 10 seizures), in which propagating seizure wavefronts could be readily identified, to investigate the basis of ictal high frequency activity at the cortical (subdural) surface. Sustained, repetitive transient increases in high gamma (80-150 Hz) amplitude, phase-locked to the low-frequency (1-25 Hz) ictal rhythm, correlated with strong multi-unit firing bursts synchronized across the core territory of the seizure. These repetitive high frequency oscillations were seen in recordings from subdural electrodes adjacent to the microelectrode array several seconds after seizure onset, following ictal wavefront passage. Conversely, microelectrode recordings demonstrating only low-level, heterogeneous neural firing correlated with a lack of high frequency oscillations in adjacent subdural recording sites, despite the presence of a strong low-frequency signature. Previously, we reported that this pattern indicates a failure of the seizure to invade the area, because of a feedforward inhibitory veto mechanism. Because multi-unit firing rate and high gamma amplitude are closely related, high frequency oscillations can be used as a surrogate marker to distinguish the core seizure territory from the surrounding penumbra. We developed an efficient measure to detect delayed-onset, sustained ictal high frequency oscillations based on cross-frequency coupling between high gamma amplitude and the low-frequency (1-25 Hz) ictal rhythm. When applied to the broader subdural recording, this measure consistently predicted the timing or failure of ictal invasion, and revealed a surprisingly small and slowly spreading seizure core surrounded by a far larger penumbral territory. Our findings thus establish an underlying neural mechanism for delayed-onset, sustained ictal high frequency oscillations, and provide a practical, efficient method for using them to identify the small ictal core regions. Our observations suggest that it may be possible to reduce substantially the extent of cortical resections in epilepsy surgery procedures without compromising seizure control.},
+  keywords = {Analysis of Variance, Biological Clocks, Brain Mapping, Brain Waves, Electroencephalography, Humans, Microelectrodes, Seizures},
+  note = {Generated by SciTeX Scholar}
+}

scitex/scholar/docs/from_agents/OPENATHENS_SECURITY.md

@@ -0,0 +1,101 @@
+# OpenAthens Security and Cookie Storage
+
+## Cookie Storage Location
+
+OpenAthens session cookies are stored in:
+```
+~/.scitex/scholar/openathens_sessions/
+```
+
+The exact filename depends on your institution:
+- `openathens_unimelb_edu_au_session.enc` (for user@unimelb.edu.au)
+- `openathens_harvard_edu_session.enc` (for user@harvard.edu)
+- `openathens_default_session.enc` (if no email specified)
+
+## Security Features
+
+### 1. Encryption at Rest
+
+All session cookies are encrypted using:
+- **Algorithm**: Fernet (symmetric encryption)
+- **Key Derivation**: PBKDF2-HMAC-SHA256
+- **Iterations**: 100,000
+- **Salt**: Machine-specific, stored in `~/.scitex/.scitex_salt`
+
+### 2. File Permissions
+
+- Session files: `0600` (read/write by owner only)
+- Salt file: `0600` (read/write by owner only)
+
+### 3. Automatic Migration
+
+If you have existing unencrypted session files (`.json`), they will be:
+1. Automatically encrypted on first use
+2. Original unencrypted file deleted
+3. No action required from you
+
+### 4. Session Expiry
+
+- Sessions expire after ~8 hours
+- Expired sessions are automatically detected
+- Re-authentication required when expired
+
+## Security Best Practices
+
+1. **Don't share session files** - They're encrypted with your email
+2. **Keep your email secure** - It's used to derive the encryption key
+3. **Machine-bound** - Sessions only work on the machine where created
+4. **Regular cleanup** - Old session files can be safely deleted
+
+## Manual Session Management
+
+### View your sessions:
+```bash
+ls -la ~/.scitex/scholar/openathens_sessions/
+```
+
+### Clear all sessions:
+```bash
+rm -rf ~/.scitex/scholar/openathens_sessions/*.enc
+```
+
+### Check session validity:
+```python
+from scitex.scholar import Scholar
+
+scholar = Scholar()
+if await scholar.is_openathens_authenticate_async():
+    print("Session is valid")
+else:
+    print("Session expired or not found")
+```
+
+## What's Encrypted
+
+The following data is encrypted in session files:
+- Authentication cookies
+- Session tokens
+- Expiry timestamps
+- Associated email address
+
+## What's NOT Encrypted
+
+- The salt file (it's meant to be random)
+- The email hash in metadata (for identification only)
+- Log files (they don't contain sensitive data)
+
+## Troubleshooting
+
+### "Failed to decrypt session cache"
+- You may be using a different email than when created
+- The salt file may have been deleted or changed
+- Solution: Delete the session file and re-authenticate_async
+
+### "Permission denied" errors
+- Check file permissions: `ls -la ~/.scitex/`
+- Fix permissions: `chmod 600 ~/.scitex/scholar/openathens_sessions/*`
+
+### Can't find session files
+- Check the logged location when authenticating
+- Default: `~/.scitex/scholar/openathens_sessions/`
+- Set custom location: `OpenAthensAuthenticator(cache_dir="/my/path")`

scitex/scholar/docs/from_agents/feature-requests/scholar-openathens-authentication.md

@@ -0,0 +1,107 @@
+# Feature Request: OpenAthens Authentication Support for Scholar Module
+
+## Overview
+Add OpenAthens institutional authentication support to the Scholar module to enable PDF downloads through institutional subscriptions.
+
+## Background
+Currently, the Scholar module can download PDFs through:
+- Direct download (open access papers)
+- Sci-Hub (with ethical acknowledgment)
+- Zotero translators
+
+However, many academic papers require institutional authentication. OpenAthens is a widely-used single sign-on system that provides seamless access to academic resources.
+
+## Proposed Implementation
+
+### 1. Configuration Enhancement
+Add OpenAthens configuration to `ScholarConfig`:
+
+```python
+@dataclass
+class ScholarConfig:
+    # ... existing fields ...
+    
+    # OpenAthens authentication
+    openathens_enabled: bool = False
+    openathens_org_id: Optional[str] = None  # Organization identifier
+    openathens_idp_url: Optional[str] = None  # Identity provider URL
+    openathens_username: Optional[str] = None  # Can be prompted if not set
+    openathens_password: Optional[str] = None  # Should be stored securely
+    
+    # Environment variable support
+    # SCITEX_SCHOLAR_OPENATHENS_ENABLED=true
+    # SCITEX_SCHOLAR_OPENATHENS_ORG_ID=unimelb
+    # SCITEX_SCHOLAR_OPENATHENS_IDP_URL=https://idp.unimelb.edu.au
+```
+
+### 2. Authentication Flow
+Create `_OpenAthensAuthenticator.py`:
+
+```python
+class OpenAthensAuthenticator:
+    """Handles OpenAthens authentication for institutional access."""
+    
+    async def authenticate_async(self, username: str, password: str) -> Session:
+        """Authenticate and return session with auth cookies."""
+        
+    async def download_with_auth_async(self, url: str, session: Session) -> bytes:
+        """Download PDF using authenticate_async session."""
+```
+
+### 3. Integration with ScholarPDFDownloader
+Modify `_ScholarPDFDownloader.py` to try OpenAthens before Sci-Hub:
+
+```python
+async def download_pdf_async(self, doi: str) -> Optional[Path]:
+    # 1. Try direct download
+    # 2. Try OpenAthens (if configured)
+    # 3. Try Sci-Hub (if acknowledged)
+    # 4. Try Zotero translators
+```
+
+### 4. User Experience
+```python
+# First-time setup
+scholar = Scholar()
+scholar.configure_openathens(
+    org_id="unimelb",
+    idp_url="https://idp.unimelb.edu.au"
+)
+
+# Downloads will now use OpenAthens authentication
+papers = scholar.search("neuroscience")
+results = scholar.download_pdf_asyncs(papers)  # Will prompt for credentials if needed
+```
+
+### 5. Security Considerations
+- Store credentials securely using keyring/keychain
+- Support environment variables for CI/CD
+- Option to prompt for credentials each session
+- Clear session management and timeout handling
+
+## Benefits
+1. **Legal access** - Uses institutional subscriptions legally
+2. **Better success rate** - Access to more journals than open access alone
+3. **Seamless integration** - Works with existing Scholar workflow
+4. **Security** - Proper credential management
+
+## Testing Plan
+1. Unit tests with mocked authentication
+2. Integration tests with test OpenAthens endpoint
+3. Manual testing with University of Melbourne account
+4. Documentation and examples
+
+## Implementation Priority
+High - This addresses a key limitation where legitimate institutional users cannot access papers they have legal access to.
+
+## Related Issues
+- PDF download failures for paywalled content
+- Need for institutional authentication support
+
+## Next Steps
+1. Review and approve feature design
+2. Implement OpenAthensAuthenticator class
+3. Integrate with ScholarPDFDownloader
+4. Add configuration options
+5. Create tests and documentation
+6. Deploy and get user feedback

scitex/scholar/docs/from_agents/scholar_enhancements_summary.md

@@ -0,0 +1,66 @@
+# Scholar Module Enhancements Summary
+
+## Date: 2025-07-24
+## Agent: 5db30af0-6862-11f0-928a-00155d8642b8
+
+### Completed Enhancements
+
+#### 1. Fixed PDF Download Return Value Issue ✅
+**Problem**: `Scholar.download_pdf_asyncs()` was returning an empty Papers collection even though PDFs were downloading successfully.
+
+**Solution**: 
+- Added logic to create Paper objects when DOI strings are provided as input
+- Properly maps download PDFs back to Paper objects
+- Returns a populated Papers collection with pdf_path set for successful downloads
+
+**Code changes**: `_Scholar.py` lines 445-470
+
+#### 2. Enhanced OpenAthens Authentication with Email Auto-fill ✅
+**Problem**: Users had to manually type their institutional email in the OpenAthens login form.
+
+**Solution**:
+- Implemented automatic email field detection and filling
+- Uses multiple CSS selectors to find the email input field
+- Types with human-like delay for better compatibility
+- Triggers autocomplete dropdown after filling
+- Shows "(auto-filled)" in instructions when email is provided
+
+**Code changes**: `_OpenAthensAuthenticator.py` lines 218-264
+
+#### 3. Fixed Environment Variable Naming Convention ✅
+**Problem**: Some environment variables in the default config YAML were missing the SCITEX_SCHOLAR_ prefix.
+
+**Solution**:
+- Updated all environment variable references to use consistent SCITEX_SCHOLAR_ prefix
+- Added missing configuration options for OpenAthens and auto-download features
+- Updated documentation in default_config.yaml
+
+**Files changed**: 
+- `config/default_config.yaml`
+
+### Configuration Methods Supported
+
+The OpenAthens email can now be configured in three ways (priority order):
+1. **Direct parameter**: `scholar.configure_openathens(email="your.email@institution.edu")`
+2. **YAML config file**: `openathens_email: "your.email@institution.edu"`
+3. **Environment variable**: `SCITEX_SCHOLAR_OPENATHENS_EMAIL="your.email@institution.edu"`
+
+### Testing
+
+Created comprehensive test scripts to verify:
+- PDF download functionality with proper Papers collection return
+- OpenAthens email auto-fill with all configuration methods
+- Configuration priority (direct > YAML > environment)
+
+### Impact
+
+- Improved user experience with automatic email filling during authentication
+- Fixed critical bug where download PDFs weren't accessible in the returned Papers collection
+- Consistent environment variable naming prevents conflicts with other packages
+- Better configuration documentation for users
+
+### Next Steps
+
+1. Update unit tests to include new OpenAthens attributes in mock objects
+2. Test with real institutional credentials to verify auto-fill works across different OpenAthens implementations
+3. Consider adding support for other authentication methods (EZProxy, Shibboleth) following the same pattern

scitex/scholar/docs/from_user/crawl4ai.md

@@ -0,0 +1,15 @@
+<!-- ---
+!-- Timestamp: 2025-08-07 07:02:22
+!-- Author: ywatanabe
+!-- File: /home/ywatanabe/proj/scitex_repo/src/scitex/scholar/docs/from_user/crawl4ai.md
+!-- --- -->
+
+# https://github.com/unclecode/crawl4ai
+
+docker pull unclecode/crawl4ai:latest
+docker run -d -p 11235:11235 --name crawl4ai --shm-size=1g unclecode/crawl4ai:latest
+docker run -d -p 11235:11235 --name crawl4ai --shm-size=1g crawl4ai-fixed
+
+# Visit the playground at http://localhost:11235/playground
+
+<!-- EOF -->