scitex 2.4.1__tar.gz → 2.4.2__tar.gz

{scitex-2.4.1 → scitex-2.4.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: scitex
-Version: 2.4.1
+Version: 2.4.2
 Summary: A comprehensive Python library for scientific computing and data analysis
 Project-URL: Homepage, https://github.com/ywatanabe1989/scitex-code
 Project-URL: Documentation, https://scitex.readthedocs.io

scitex-2.4.2/docs/CROSSREF_API_CONFIGURATION.md

@@ -0,0 +1,189 @@
+# CrossRef API Configuration
+
+The `CrossRefLocalEngine` in scitex package supports both internal and external CrossRef APIs, automatically detecting the format and adjusting endpoints accordingly.
+
+## API Types
+
+### Internal API (Recommended for NAS/Docker)
+- **Fast**: Direct database access, no rate limiting
+- **Private**: Only accessible within Docker network or VPN
+- **Large**: Requires 1.2TB database storage
+
+### External API (Recommended for Development/Remote)
+- **Accessible**: Works from anywhere over internet
+- **Rate-limited**: 100 requests/hour per IP
+- **Cached**: 1-hour Redis cache for faster responses
+
+## Environment Variable Configuration
+
+### Option 1: Internal API (NAS Docker Network)
+
+```bash
+export SCITEX_SCHOLAR_CROSSREF_API_URL=http://crossref:3333
+```
+
+**Use when:**
+- Running inside NAS Docker network
+- Need maximum speed (no rate limits)
+- Have access to local CrossRef database
+
+**Endpoints generated:**
+- Search: `http://crossref:3333/api/search/`
+- Health: `http://crossref:3333/api/health/`
+- Stats: `http://crossref:3333/api/stats/`
+
+### Option 2: Internal API (NAS IP via VPN)
+
+```bash
+export SCITEX_SCHOLAR_CROSSREF_API_URL=http://169.254.11.50:3333
+```
+
+**Use when:**
+- Connected to NAS via VPN/local network
+- Running on laptop but want internal speed
+- Testing from development machine
+
+**Endpoints generated:**
+- Search: `http://169.254.11.50:3333/api/search/`
+- Health: `http://169.254.11.50:3333/api/health/`
+- Stats: `http://169.254.11.50:3333/api/stats/`
+
+### Option 3: External API (Public Internet)
+
+```bash
+export SCITEX_SCHOLAR_CROSSREF_API_URL=https://scitex.ai/scholar/api/crossref
+```
+
+**Use when:**
+- Working remotely without VPN
+- Testing from anywhere
+- Don't need high request rates
+
+**Endpoints generated:**
+- Search: `https://scitex.ai/scholar/api/crossref/search/`
+- Health: `https://scitex.ai/scholar/api/crossref/health/`
+- Stats: `https://scitex.ai/scholar/api/crossref/stats/`
+
+**Rate limits:**
+- 100 requests/hour per IP address
+- Health and stats endpoints not rate-limited
+
+## Configuration Files
+
+### For Development (.env.dev)
+
+```bash
+# Internal API when on VPN
+SCITEX_SCHOLAR_CROSSREF_API_URL_DEV=http://169.254.11.50:3333
+
+# External API when remote
+# SCITEX_SCHOLAR_CROSSREF_API_URL_DEV=https://scitex.ai/scholar/api/crossref
+
+# Active configuration
+SCITEX_SCHOLAR_CROSSREF_API_URL=${SCITEX_SCHOLAR_CROSSREF_API_URL_DEV}
+```
+
+### For Production (.env.nas)
+
+```bash
+# Internal API (Docker network)
+SCITEX_SCHOLAR_CROSSREF_API_URL_NAS=http://crossref:3333
+
+# External API fallback
+SCITEX_SCHOLAR_CROSSREF_API_URL_DEV=https://scitex.ai/scholar/api/crossref
+
+# Active configuration
+SCITEX_SCHOLAR_CROSSREF_API_URL=${SCITEX_SCHOLAR_CROSSREF_API_URL_NAS}
+```
+
+## Testing Configuration
+
+### Quick Test (Python)
+
+```python
+from scitex.scholar.metadata_engines.individual import CrossRefLocalEngine
+
+# Test with your configuration
+engine = CrossRefLocalEngine("your@email.com")
+print(f"API URL: {engine.api_url}")
+print(f"Is External: {engine._is_external_api}")
+print(f"Search endpoint: {engine._build_endpoint_url('search')}")
+
+# Test search
+result = engine.search(doi="10.1038/nature12373")
+if result:
+    print(f"Title: {result.get('basic', {}).get('title')}")
+    print(f"Success! API is working.")
+```
+
+### Quick Test (Shell)
+
+```bash
+# Internal API test
+curl "http://crossref:3333/api/health/"
+
+# External API test
+curl "https://scitex.ai/scholar/api/crossref/health/"
+```
+
+## Automatic Detection
+
+The engine automatically detects API type:
+
+```python
+# Detection logic
+is_external = "/api/crossref" in api_url or "scitex.ai" in api_url
+
+if is_external:
+    # External: https://scitex.ai/scholar/api/crossref/search/
+    url = f"{api_url}/{endpoint}/"
+else:
+    # Internal: http://crossref:3333/api/search/
+    url = f"{api_url}/api/{endpoint}/"
+```
+
+## Database Information
+
+- **Total papers**: 167,008,748
+- **Database size**: 1.2TB (1,197,367 MB)
+- **Format**: SQLite with indexed fields
+- **Source**: CrossRef official database
+
+## Best Practices
+
+1. **Use internal API when possible** for speed and no rate limits
+2. **Use external API for remote work** when VPN not available
+3. **Set environment variables** rather than hardcoding URLs
+4. **Test configuration** before running large batch jobs
+5. **Respect rate limits** on external API (100/hour)
+
+## Troubleshooting
+
+### Connection Refused (Internal API)
+
+```
+CrossRef Local server not available at http://crossref:3333 (connection refused)
+```
+
+**Solutions:**
+- Check if CrossRef service is running: `docker ps | grep crossref`
+- Verify you're inside Docker network or VPN
+- Try external API instead: `export SCITEX_SCHOLAR_CROSSREF_API_URL=https://scitex.ai/scholar/api/crossref`
+
+### Rate Limited (External API)
+
+```
+HTTP 429 Too Many Requests
+```
+
+**Solutions:**
+- Wait 1 hour for rate limit to reset
+- Use internal API if available
+- Reduce request frequency
+- Implement caching in your application
+
+## See Also
+
+- [CrossRef API Proxy Implementation](../apps/scholar_app/api/crossref_proxy.py)
+- [CrossRef Local Server](../deployment/docker/crossref_local/server.py)
+- [Environment Variables Guide](../SECRETS/README.md)

scitex-2.4.2/examples/demo_seaborn_wrappers.py

@@ -0,0 +1,35 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+# Timestamp: "2025-12-02 21:42:58 (ywatanabe)"
+# File: /home/ywatanabe/proj/scitex-code/examples/demo_seaborn_wrappers.py
+
+
+"""Top-level docstring here"""
+
+
+# Imports
+import scitex as stx
+
+# # Parameters
+# CONFIG = stx.io.load_configs() # For imported files using `./config/*.yaml`
+
+
+# Functions and Classes
+@stx.session
+def main(
+    # arg1,
+    # kwarg1="value1",
+    CONFIG=stx.INJECTED,
+    plt=stx.INJECTED,
+    COLORS=stx.INJECTED,
+    rng_manager=stx.INJECTED,
+    logger=stx.INJECTED,
+):
+    """Help message for `$ python __file__ --help`"""
+    return 0
+
+
+if __name__ == "__main__":
+    main()
+
+# EOF

{scitex-2.4.1 → scitex-2.4.2}/examples/plt/demo_matplotlib_basic/_01_plot.py

@@ -1,6 +1,6 @@
 #!/usr/bin/env python3
 # -*- coding: utf-8 -*-
-# Timestamp: "2025-12-01 20:06:03 (ywatanabe)"
+# Timestamp: "2025-12-02 21:42:53 (ywatanabe)"
 # File: /home/ywatanabe/proj/scitex-code/examples/plt/demo_matplotlib_basic/_01_plot.py
 
 

scitex-2.4.1/examples/plt/demo_scitex_wrappers.py → scitex-2.4.2/examples/plt/run_demo_scitex_wrappers.py

@@ -1,7 +1,7 @@
 #!/usr/bin/env python3
 # -*- coding: utf-8 -*-
-# Timestamp: "2025-12-02 05:54:57 (ywatanabe)"
-# File: /home/ywatanabe/proj/scitex-code/examples/plt/demo_scitex_wrappers.py
+# Timestamp: "2025-12-02 06:15:00 (ywatanabe)"
+# File: /home/ywatanabe/proj/scitex-code/examples/plt/run_demo_scitex_wrappers.py
 
 
 """Demo: SciTeX custom wrapper methods (ax.stx_xxx)."""

{scitex-2.4.1 → scitex-2.4.2}/pyproject.toml

@@ -11,7 +11,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "scitex"
-version = "2.4.1"
+version = "2.4.2"
 description = "A comprehensive Python library for scientific computing and data analysis"
 readme = "README.md"
 requires-python = ">=3.8"

{scitex-2.4.1 → scitex-2.4.2}/src/scitex/__version__.py

@@ -9,6 +9,6 @@ __FILE__ = "./src/scitex/__version__.py"
 __DIR__ = os.path.dirname(__FILE__)
 # ----------------------------------------
 
-__version__ = "2.3.0"
+__version__ = "2.4.2"
 
 # EOF

scitex-2.4.2/src/scitex/browser/auth/__init__.py

@@ -0,0 +1,35 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+# Timestamp: "2025-12-04 (ywatanabe)"
+# File: ./src/scitex/browser/auth/__init__.py
+# ----------------------------------------
+"""
+scitex.browser.auth - Authentication helpers for browser automation.
+
+Provides reusable authentication handlers for various OAuth providers
+and login flows used in browser automation tasks.
+
+Features:
+- Google OAuth (popup-based flow)
+- Django session auth
+- Generic credential management
+
+Example:
+    from scitex.browser.auth import GoogleAuthHelper, google_login
+
+    # Quick login
+    success = await google_login(page, "user@gmail.com", "password")
+
+    # Or with helper class
+    auth = GoogleAuthHelper(email="user@gmail.com", password="password")
+    success = await auth.login_via_google_button(page)
+"""
+
+from .google import GoogleAuthHelper, google_login
+
+__all__ = [
+    "GoogleAuthHelper",
+    "google_login",
+]
+
+# EOF

scitex-2.4.2/src/scitex/browser/auth/google.py

@@ -0,0 +1,381 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+# Timestamp: "2025-12-04 (ywatanabe)"
+# File: ./src/scitex/browser/auth/google.py
+# ----------------------------------------
+"""
+Google OAuth authentication helper for Playwright.
+
+Handles Google OAuth popup flow for services that use "Continue with Google".
+
+Example:
+    from scitex.browser.auth import GoogleAuthHelper
+
+    auth = GoogleAuthHelper(
+        email="user@gmail.com",
+        password="your_password",
+    )
+
+    # Login to a service that uses Google OAuth
+    success = await auth.login_via_google_button(page, 'button:has-text("Continue with Google")')
+"""
+
+import os
+import sys
+from typing import Optional
+from playwright.async_api import Page
+
+
+class GoogleAuthHelper:
+    """
+    Google OAuth authentication helper.
+
+    Handles the popup-based Google OAuth flow used by many services.
+
+    Environment Variables:
+        GOOGLE_EMAIL: Default email if not provided
+        GOOGLE_PASSWORD: Default password if not provided
+    """
+
+    def __init__(
+        self,
+        email: Optional[str] = None,
+        password: Optional[str] = None,
+        debug: bool = False,
+    ):
+        """
+        Initialize GoogleAuthHelper.
+
+        Args:
+            email: Google account email
+            password: Google account password
+            debug: Print debug messages to stderr
+        """
+        self.email = email or os.getenv("GOOGLE_EMAIL", "")
+        self.password = password or os.getenv("GOOGLE_PASSWORD", "")
+        self.debug = debug or bool(os.getenv("GOOGLE_AUTH_DEBUG"))
+
+    def _log(self, msg: str):
+        """Print debug message if debug mode is enabled."""
+        if self.debug:
+            print(f"[GoogleAuth] {msg}", file=sys.stderr)
+
+    async def login_via_google_button(
+        self,
+        page: Page,
+        google_button_selector: str = 'button:has-text("Continue with Google")',
+        timeout: int = 60000,
+    ) -> bool:
+        """
+        Perform Google OAuth login via a "Continue with Google" button.
+
+        This handles the popup-based OAuth flow:
+        1. Click the Google button on the main page
+        2. Handle the Google popup for email/password entry
+        3. Wait for redirect back to the original service
+
+        Args:
+            page: Playwright Page object (the main page with the Google button)
+            google_button_selector: CSS selector for the Google login button
+            timeout: Maximum time to wait for login (ms)
+
+        Returns:
+            True if login successful, False otherwise
+        """
+        try:
+            # Find the Google button
+            google_btn = await page.query_selector(google_button_selector)
+            if not google_btn:
+                # Try alternative selectors
+                alternatives = [
+                    'button:has-text("Google")',
+                    '[data-testid="google-login"]',
+                    'button >> text=Continue with Google',
+                    'button >> text=Sign in with Google',
+                ]
+                for selector in alternatives:
+                    try:
+                        google_btn = await page.query_selector(selector)
+                        if google_btn:
+                            break
+                    except:
+                        continue
+
+            if not google_btn:
+                self._log("Google button not found")
+                return False
+
+            self._log("Found Google button, clicking...")
+
+            # Google OAuth opens in a popup - listen for it
+            async with page.context.expect_page(timeout=timeout) as popup_info:
+                await google_btn.click()
+
+            popup = await popup_info.value
+            self._log(f"Popup opened: {popup.url[:100]}...")
+
+            # Handle Google OAuth in popup
+            success = await self._handle_google_popup(popup, timeout)
+
+            if success:
+                # Wait for main page to update after OAuth completes
+                await page.wait_for_timeout(3000)
+                self._log(f"Login complete, main page URL: {page.url}")
+
+            return success
+
+        except Exception as e:
+            self._log(f"Login error: {e}")
+            return False
+
+    async def _handle_google_popup(self, popup: Page, timeout: int = 60000) -> bool:
+        """
+        Handle the Google OAuth popup flow.
+
+        Args:
+            popup: The Google OAuth popup page
+            timeout: Maximum time to wait (ms)
+
+        Returns:
+            True if authentication successful, False otherwise
+        """
+        try:
+            # Wait for Google login page to load
+            await popup.wait_for_load_state("domcontentloaded")
+            await popup.wait_for_timeout(2000)
+
+            # Step 1: Enter email
+            email_filled = await self._fill_email(popup)
+            if not email_filled:
+                self._log("Failed to fill email")
+                return False
+
+            # Step 2: Wait for password page and enter password
+            password_filled = await self._fill_password(popup)
+            if not password_filled:
+                self._log("Failed to fill password")
+                return False
+
+            # Step 3: Wait for popup to close (indicates success)
+            try:
+                await popup.wait_for_event("close", timeout=20000)
+                self._log("Popup closed - login successful")
+                return True
+            except:
+                # Check if we're still on Google or redirected
+                current_url = popup.url
+                if "accounts.google.com" not in current_url:
+                    self._log("Redirected away from Google - login successful")
+                    return True
+                self._log("Popup didn't close - possible error")
+                return False
+
+        except Exception as e:
+            self._log(f"Popup handling error: {e}")
+            return False
+
+    async def _fill_email(self, popup: Page) -> bool:
+        """Fill email on Google login page."""
+        try:
+            # Wait for email input
+            await popup.wait_for_selector('input[type="email"]', state="visible", timeout=10000)
+
+            self._log(f"Filling email: {self.email}")
+            await popup.fill('input[type="email"]', self.email)
+            await popup.wait_for_timeout(500)
+
+            # Click Next button
+            next_btn = await popup.query_selector('#identifierNext')
+            if not next_btn:
+                next_btn = await popup.query_selector('button:has-text("Next")')
+
+            if next_btn:
+                self._log("Clicking Next after email")
+                await next_btn.click()
+                await popup.wait_for_timeout(3000)
+                return True
+            else:
+                self._log("Next button not found after email")
+                return False
+
+        except Exception as e:
+            self._log(f"Email fill error: {e}")
+            return False
+
+    async def _fill_password(self, popup: Page) -> bool:
+        """Fill password on Google login page."""
+        try:
+            # Wait for password page to load (Google transitions between pages)
+            self._log("Waiting for password page...")
+
+            # Wait for password input to become visible
+            await popup.wait_for_selector(
+                'input[type="password"]',
+                state="visible",
+                timeout=15000
+            )
+
+            self._log("Filling password")
+            await popup.fill('input[type="password"]', self.password)
+            await popup.wait_for_timeout(500)
+
+            # Click Next button
+            next_btn = await popup.query_selector('#passwordNext')
+            if not next_btn:
+                next_btn = await popup.query_selector('button:has-text("Next")')
+
+            if next_btn:
+                self._log("Clicking Next after password")
+                await next_btn.click()
+                await popup.wait_for_timeout(5000)
+
+                # Handle 2FA if present
+                twofa_ok = await self._wait_for_2fa(popup, timeout=60000)
+                if not twofa_ok:
+                    return False
+
+                # Handle potential consent/continue screens
+                await self._handle_consent_screens(popup)
+
+                return True
+            else:
+                self._log("Next button not found after password")
+                return False
+
+        except Exception as e:
+            self._log(f"Password fill error: {e}")
+            return False
+
+    async def _handle_consent_screens(self, popup: Page) -> None:
+        """Handle OAuth consent or 'Continue' screens that may appear."""
+        try:
+            # Check for Continue button (consent screen)
+            continue_selectors = [
+                'button:has-text("Continue")',
+                'button:has-text("Allow")',
+                '#submit_approve_access',
+                'button[data-idom-class*="continue"]',
+            ]
+
+            for selector in continue_selectors:
+                try:
+                    btn = await popup.query_selector(selector)
+                    if btn and await btn.is_visible():
+                        self._log(f"Found consent button: {selector}")
+                        await btn.click()
+                        await popup.wait_for_timeout(3000)
+                        break
+                except:
+                    continue
+
+        except Exception as e:
+            self._log(f"Consent handling: {e}")
+
+    async def _wait_for_2fa(self, popup: Page, timeout: int = 60000) -> bool:
+        """
+        Wait for 2FA verification to complete.
+
+        Detects 2FA screens and waits for user to approve on their device.
+
+        Args:
+            popup: The Google OAuth popup page
+            timeout: Maximum time to wait for 2FA (ms)
+
+        Returns:
+            True if 2FA completed, False if timed out
+        """
+        try:
+            # Check if we're on a 2FA page
+            page_text = await popup.inner_text("body")
+            twofa_indicators = [
+                "2-Step Verification",
+                "Verify it's you",
+                "confirm it's you",
+                "Open the Gmail app",
+                "Check your phone",
+            ]
+
+            is_2fa = any(indicator.lower() in page_text.lower() for indicator in twofa_indicators)
+
+            if is_2fa:
+                self._log("2FA detected - waiting for user approval...")
+                # Wait for popup to close or URL to change (indicating 2FA success)
+                start_url = popup.url
+                check_interval = 2000  # Check every 2 seconds
+                elapsed = 0
+
+                while elapsed < timeout:
+                    await popup.wait_for_timeout(check_interval)
+                    elapsed += check_interval
+
+                    # Check if popup closed
+                    try:
+                        current_url = popup.url
+                        if current_url != start_url and "accounts.google.com" not in current_url:
+                            self._log("2FA completed - redirected")
+                            return True
+                    except:
+                        # Popup closed
+                        self._log("2FA completed - popup closed")
+                        return True
+
+                self._log("2FA timeout")
+                return False
+
+            return True  # Not a 2FA page
+
+        except Exception as e:
+            self._log(f"2FA check error: {e}")
+            return False
+
+    async def is_logged_in(self, page: Page, login_indicators: list = None) -> bool:
+        """
+        Check if user appears to be logged in.
+
+        Args:
+            page: Page to check
+            login_indicators: List of URL substrings that indicate NOT logged in
+                             (default: ["login", "signin", "oauth"])
+
+        Returns:
+            True if appears logged in, False otherwise
+        """
+        if login_indicators is None:
+            login_indicators = ["login", "signin", "oauth", "accounts.google.com"]
+
+        current_url = page.url.lower()
+        for indicator in login_indicators:
+            if indicator.lower() in current_url:
+                return False
+        return True
+
+
+# Convenience function for quick usage
+async def google_login(
+    page: Page,
+    email: str,
+    password: str,
+    button_selector: str = 'button:has-text("Continue with Google")',
+    debug: bool = False,
+) -> bool:
+    """
+    Quick Google OAuth login.
+
+    Args:
+        page: Playwright Page with Google login button
+        email: Google account email
+        password: Google account password
+        button_selector: CSS selector for Google button
+        debug: Print debug messages
+
+    Returns:
+        True if login successful, False otherwise
+
+    Example:
+        success = await google_login(page, "user@gmail.com", "password")
+    """
+    auth = GoogleAuthHelper(email=email, password=password, debug=debug)
+    return await auth.login_via_google_button(page, button_selector)
+
+
+# EOF