scitex 2.14.0__py3-none-any.whl → 2.16.0__py3-none-any.whl

scitex/scholar/examples/SUGGESTIONS.md

@@ -1,865 +0,0 @@
-<!-- ---
-!-- Timestamp: 2025-08-22 05:28:16
-!-- Author: ywatanabe
-!-- File: /home/ywatanabe/proj/SciTeX-Code/src/scitex/scholar/examples/SUGGESTIONS.md
-!-- --- -->
-
-You've hit another excellent and distinct debugging case. The error has changed, which tells us we are making progress and uncovering the complexities of running these translators outside their native environment.
-
-The new key error is:
-ERROR: Translator error: em.doWeb is not a function
-
-This is different and more informative than the previous one. It's not just a data format issue; it's a structural problem with how the translator code is being executed.
-
-## What's Happening Here?
-Zotero translators have evolved, and not all of them follow the simple, global detectWeb() and doWeb() function pattern. The "Frontiers" translator uses a more modern, object-oriented approach.
-
-When the Frontiers.js code is executed, it doesn't just define global functions. Instead, it likely creates an instance of an object (let's call it em).
-
-The main logic, including the doWeb function, is a method of this em object.
-
-Your current JavaScript wrapper and executor (zotero_wrapper.js) are not aware of this. They blindly look for a global doWeb function, which doesn't exist. The function they need to call is em.doWeb().
-
-Because this first step fails, the script execution halts, and the subsequent SyntaxError: Unexpected token '<' is likely a red herring—a downstream consequence of the initial failure.
-
-## The Solution: A More Intelligent Executor
-To solve this, your JavaScript executor needs to be smarter. It must first run the translator's code and then inspect the global scope to find out how that specific translator needs to be called. It needs to check for a global doWeb function, and if that doesn't exist, it should look for a common object pattern (like an object named em with a doWeb method).
-
-Here is an updated, more robust version of the execution logic for your _ZoteroTranslatorRunner.py. This change centralizes the execution logic within the page.evaluate call, making it more adaptable.
-
-Replace the page.evaluate block in the extract_urls_pdf_async method of _ZoteroTranslatorRunner.py with this:
-
-Python
-
-# In _ZoteroTranslatorRunner.py -> extract_urls_pdf_async method
-
-# ... after injecting the Zotero environment JS ...
-
-result = await page.evaluate(
-    """
-    async ([translatorCode, translatorLabel]) => {
-        // This is the main execution function, now running entirely in the browser.
-        // It's a combination of your wrapper and the execution logic.
-        const urls = new Set();
-        const items = [];
-        let translatorError = null;
-
-        // --- Start: Mock Zotero Environment (condensed from your zotero_wrapper.js) ---
-        window.Zotero = {
-            Item: function(type) {
-                this.itemType = type;
-                this.attachments = [];
-                this.url = null;
-                this.DOI = null;
-                this.complete = function() {
-                    if (this.url) urls.add(this.url);
-                    if (this.DOI) urls.add('https://doi.org/' + this.DOI);
-                    this.attachments.forEach(att => {
-                        if (att.url && (att.mimeType === 'application/pdf' || att.url.includes('.pdf'))) {
-                            urls.add(att.url);
-                        }
-                    });
-                    items.push(this);
-                };
-            },
-            loadTranslator: function() { return { setTranslator: function() {}, setString: function(s) {}, setHandler: function(e,h){}, translate: function() {} }; },
-            Utilities: {
-                xpath: (doc, xpath) => {
-                    const result = document.evaluate(xpath, doc, null, XPathResult.ORDERED_NODE_ITERATOR_TYPE, null);
-                    const nodes = [];
-                    let node;
-                    while (node = result.iterateNext()) {
-                        nodes.push(node);
-                    }
-                    return nodes;
-                },
-                xpathText: (doc, xpath) => document.evaluate(xpath, doc, null, XPathResult.STRING_TYPE, null).stringValue,
-                trimInternal: str => str ? str.trim().replace(/\\s+/g, ' ') : '',
-            },
-            debug: (msg) => console.log(`[Zotero Debug] ${msg}`),
-            done: () => console.log("Zotero.done() called."),
-        };
-        window.Z = window.Zotero;
-        window.ZU = window.Zotero.Utilities;
-        window.requestDocument = async (url) => document; // Simplified for this context
-        window.requestText = async (url, options) => {
-            if (url.startsWith('/')) url = window.location.origin + url;
-            try {
-                const response = await fetch(url, { credentials: 'include', ...(options || {}) });
-                const contentType = response.headers.get("content-type");
-                const text = await response.text();
-                if (contentType && contentType.includes("text/html")) {
-                    console.error("requestText received HTML instead of data for URL:", url);
-                    return null;
-                }
-                return text;
-            } catch (e) {
-                console.error("requestText failed for URL:", url, e);
-                return null;
-            }
-        };
-        // --- End: Mock Zotero Environment ---
-
-        try {
-            // Step 1: Execute the translator code to define its functions/objects
-            eval(translatorCode);
-
-            // Step 2: INTELLIGENTLY FIND AND CALL THE CORRECT doWeb function
-            let detected = false;
-            let doWebFunction = null;
-            let contextObject = window; // Assume global context by default
-
-            // Pattern 1: Standard global functions
-            if (typeof detectWeb === 'function' && typeof doWeb === 'function') {
-                if (detectWeb(document, window.location.href)) {
-                    detected = true;
-                    doWebFunction = doWeb;
-                }
-            }
-            // Pattern 2: Object-oriented pattern (like 'em.doWeb')
-            else if (typeof em === 'object' && typeof em.detectWeb === 'function' && typeof em.doWeb === 'function') {
-                if (em.detectWeb(document, window.location.href)) {
-                    detected = true;
-                    doWebFunction = em.doWeb;
-                    contextObject = em; // The function must be called on the 'em' object
-                }
-            }
-            // Add other patterns here if you discover them
-
-            // Step 3: Execute if a valid pattern was found
-            if (detected && doWebFunction) {
-                console.log(`Executing translator with pattern on context: ${contextObject}`);
-                await doWebFunction.call(contextObject, document, window.location.href);
-                // Give async operations a moment to complete
-                await new Promise(resolve => setTimeout(resolve, 500));
-            } else {
-                console.log("Translator pattern not recognized or detectWeb returned false.");
-            }
-
-        } catch (e) {
-            translatorError = e.message;
-            console.error('Translator execution error:', e);
-        }
-
-        return {
-            success: !translatorError && urls.size > 0,
-            translator: translatorLabel,
-            urls: Array.from(urls),
-            itemCount: items.length,
-            error: translatorError
-        };
-    }
-    """,
-    [translator["code"], translator["label"]],
-)
-
-## Why This Fix Works
-Self-Contained: It moves the entire mock environment and execution logic into the page.evaluate block. This is cleaner and ensures the environment is perfectly set up right before the translator runs.
-
-Pattern Detection: The core of the new logic is the "INTELLIGENTLY FIND AND CALL" section. It no longer assumes a single structure.
-
-It first checks for the "classic" global detectWeb/doWeb functions.
-
-If that fails, it checks for a common object-oriented pattern: an object named em that contains the detectWeb and doWeb methods.
-
-Correct this Context: When it finds the object-oriented pattern, it uses doWebFunction.call(contextObject, ...) to ensure that when doWeb is executed, the this keyword inside it correctly refers to the em object. This is what solves the em.doWeb is not a function error.
-
-Extensible: You can easily add more else if blocks to this logic if you encounter other translator patterns in the future.
-
-By making your runner smarter, you can handle the diversity of Zotero translators and significantly improve the reliability of your PDF extraction process.
-
-
-
-
-
-
-
-is you approach versatile for any publisher pages?
-
-
-
-Show thinking
-That's an excellent question. The short answer is yes, this approach is incredibly versatile and is likely the most robust and scalable method you can build for this purpose.
-
-However, it is not a "silver bullet" that will work flawlessly on every single publisher's website without occasional adjustments. Think of it as a powerful framework that handles 90% of the problem, but you'll still need to debug the tricky 10%.
-
-Here’s a breakdown of its versatility and its limitations.
-
-## Why This Approach Is So Powerful 🚀
-Leverages Massive Community Effort: You are tapping into the Zotero open-source community, which has spent years writing and maintaining thousands of translators for individual publisher sites, journals, databases, and news outlets. You don't have to write a custom scraper for Elsevier, another for Springer, another for Nature, etc. The community has already done that hard work.
-
-Handles Complex Scraping Logic: Zotero translators are far more sophisticated than simple CSS selectors. They contain logic to:
-
-Try multiple methods to find metadata.
-
-Parse hidden JSON data embedded in the page.
-
-Automate clicking "Export" buttons and fetching citation files (like RIS or BibTeX) in the background.
-
-Clean up messy data (e.g., formatting author names).
-
-Standardized Interface: As we've discovered, the translators have a standardized structure (the JSON header with a target regex and the detectWeb/doWeb entry points). This allows your Python runner to treat them like plugins, dynamically loading and executing the correct one for any given URL.
-
-## The Inevitable Limitations and Challenges 🛠️
-While versatile, it's not universally plug-and-play. The errors you've already encountered are perfect examples of the types of issues you will run into.
-
-Incomplete Mock Environment: Your zotero_wrapper.js is a mock of the real Zotero environment. We have implemented the common functions (Zotero.Item, requestText, etc.), but a translator for a niche website might call a less common utility function (e.g., ZU.cleanISBN()) that doesn't exist in your mock, causing an error.
-
-Solution: This is an iterative process. When a translator fails because of a missing function, you add a simple implementation of it to your wrapper.
-
-Aggressive Anti-Scraping Measures: Publishers actively try to block automated access. You may encounter:
-
-CAPTCHAs: If your script makes too many requests too quickly, a site like Google Scholar or Scopus will throw up a CAPTCHA, which will halt the process.
-
-Dynamic JavaScript Challenges: Some sites may use JavaScript techniques to detect and block automated browsers like Playwright.
-
-Complex Authentication: This is the most common failure point. Even if your main page is logged in, a translator's background fetch request might fail because:
-
-It's blocked by Cross-Origin Resource Sharing (CORS) policies.
-
-It doesn't carry the necessary authentication tokens or cookies that the main page uses.
-
-This is exactly what happened with your ScienceDirect attempt, which returned a login page (HTML) instead of data.
-
-Outdated or Buggy Translators: Sometimes, a publisher will update their website, and the corresponding Zotero translator will be temporarily broken until a community member fixes it. You might be the first to discover the issue.
-
-## Conclusion
-Your approach of using a Python/Playwright runner to execute Zotero translators is the gold standard for this kind of task. It is far more versatile and maintainable than trying to write and manage hundreds of custom scrapers yourself.
-
-The key is to think of it not as a single script you write once, but as a robust framework that you will occasionally need to debug and enhance as you encounter new publisher websites and new translator patterns.
-
-
-#!/usr/bin/env python3
-# -*- coding: utf-8 -*-
-# File: test_zotero_runner.py
-# ----------------------------------------
-"""
-Test Suite for ZoteroTranslatorRunner.
-
-This script uses pytest to validate that the ZoteroTranslatorRunner can
-correctly execute translators from various publishers with different internal
-architectures (e.g., global functions vs. object-oriented methods).
-
-Usage:
-1. Make sure you have pytest and pytest-asyncio installed:
-   pip install pytest pytest-asyncio playwright
-   playwright install
-2. Place this file in a 'tests' directory alongside your project source.
-3. Make sure your _ZoteroTranslatorRunner.py is importable.
-4. Run the tests from your project's root directory:
-   pytest
-"""
-
-import asyncio
-import pytest
-from playwright.async_api import Page, Browser, async_playwright
-
-# Note: ZoteroTranslatorRunner is now deprecated. Use the Python-based zotero_translators instead.
-# Old location (deprecated): from scitex.scholar.url.helpers.finders._ZoteroTranslatorRunner import ZoteroTranslatorRunner
-# New location: Use scitex.scholar.url.strategies.zotero_translators for Python-based implementation
-
-# --- Test Cases ---
-# A curated list of URLs that represent different publishers and translator patterns.
-# Each tuple contains: (test_id, url)
-TEST_CASES = [
-    # Pattern: Object-oriented ('em.doWeb'). Test for the fix from the previous issue.
-    ("frontiers", "https://www.frontiersin.org/journals/neuroscience/articles/10.3389/fnins.2019.00573/full"),
-    
-    # Pattern: Standard global functions. A very common and reliable open-access source.
-    ("arxiv", "https://arxiv.org/abs/2103.14030"),
-    
-    # Pattern: Major publisher (Nature). Often has complex pages.
-    ("nature", "https://www.nature.com/articles/s41586-021-03372-6"),
-    
-    # Pattern: Another major publisher (Elsevier). This is an open-access article.
-    ("sciencedirect_openaccess", "https://www.sciencedirect.com/science/article/pii/S009286742030120X"),
-    
-    # Pattern: Major publisher (Wiley).
-    ("wiley", "https://onlinelibrary.wiley.com/doi/full/10.1111/j.1365-2966.2006.10766.x"),
-    
-    # Case: A URL that should NOT match any translator. Tests the graceful failure case.
-    ("no_match", "https://www.google.com"),
-]
-
-# --- Pytest Fixtures ---
-
-@pytest.fixture(scope="session")
-def event_loop():
-    """Create an instance of the default event loop for the session."""
-    loop = asyncio.get_event_loop_policy().new_event_loop()
-    yield loop
-    loop.close()
-
-@pytest.fixture(scope="session")
-def runner() -> ZoteroTranslatorRunner:
-    """Provides a single instance of the ZoteroTranslatorRunner for all tests."""
-    print("\n--- Initializing ZoteroTranslatorRunner ---")
-    return ZoteroTranslatorRunner()
-
-@pytest.fixture(scope="session")
-async def browser() -> Browser:
-    """Launches a single browser instance for the entire test session."""
-    async with async_playwright() as p:
-        print("\n--- Launching Browser ---")
-        browser_instance = await p.chromium.launch(headless=True)
-        yield browser_instance
-        print("\n--- Closing Browser ---")
-        await browser_instance.close()
-
-@pytest.fixture
-async def page(browser: Browser) -> Page:
-    """Creates a new page for each test case."""
-    page = await browser.new_page()
-    yield page
-    await page.close()
-
-
-# --- Test Functions ---
-
-@pytest.mark.parametrize("test_id, url", TEST_CASES)
-async def test_translator_execution(runner: ZoteroTranslatorRunner, page: Page, test_id: str, url: str):
-    """
-    Tests the Zotero runner against a specific URL.
-    
-    Args:
-        runner: The ZoteroTranslatorRunner instance.
-        page: The Playwright page object for the test.
-        test_id: A friendly name for the test case.
-        url: The URL to test the translator against.
-    """
-    print(f"\n[Testing: {test_id}] Navigating to: {url}")
-    
-    try:
-        await page.goto(url, wait_until="domcontentloaded", timeout=60000)
-        # Add a small delay for dynamically loaded content to appear
-        await page.wait_for_timeout(3000)
-    except Exception as e:
-        pytest.fail(f"Failed to navigate to {url}: {e}")
-
-    # Execute the translator runner
-    pdf_urls = await runner.extract_urls_pdf_async(page)
-
-    # Assert the expected outcome
-    if test_id == "no_match":
-        assert pdf_urls is not None, "Runner should not crash on no match"
-        assert len(pdf_urls) == 0, f"Expected no URLs for {test_id}, but found {len(pdf_urls)}"
-        print(f"✅ [SUCCESS: {test_id}] Correctly found no matching translator.")
-    else:
-        assert pdf_urls is not None, "Runner should return a list, not None"
-        assert len(pdf_urls) > 0, f"Expected at least one PDF URL for {test_id}, but found none."
-        
-        first_url = pdf_urls[0]
-        assert isinstance(first_url, str), "Result should be a list of strings"
-        assert "pdf" in first_url.lower(), f"Expected 'pdf' in the URL, but got: {first_url}"
-        
-        print(f"✅ [SUCCESS: {test_id}] Found {len(pdf_urls)} PDF URL(s). First URL: {first_url}")
-
-# To run this script directly for demonstration purposes
-if __name__ == "__main__":
-    print("This is a pytest script. To run the tests, execute 'pytest' in your terminal.")
-    print("Example Test Cases:")
-    for test_id, url in TEST_CASES:
-        print(f"- {test_id.capitalize()}: {url}")
-
-
-#!/usr/bin/env python3
-# -*- coding: utf-8 -*-
-# File: test_zotero_runner.py
-# ----------------------------------------
-"""
-Test Suite for ZoteroTranslatorRunner.
-
-This script uses pytest to validate that the ZoteroTranslatorRunner can
-correctly execute translators from various publishers with different internal
-architectures (e.g., global functions vs. object-oriented methods).
-
-Usage:
-1. Make sure you have pytest and pytest-asyncio installed:
-   pip install pytest pytest-asyncio playwright
-   playwright install
-2. Place this file in a 'tests' directory alongside your project source.
-3. Make sure your _ZoteroTranslatorRunner.py is importable.
-4. Run the tests from your project's root directory:
-   pytest
-"""
-
-import asyncio
-import pytest
-from playwright.async_api import Page, Browser, async_playwright
-
-# Note: ZoteroTranslatorRunner is now deprecated. Use the Python-based zotero_translators instead.
-# Old location (deprecated): from scitex.scholar.url.helpers.finders._ZoteroTranslatorRunner import ZoteroTranslatorRunner
-# New location: Use scitex.scholar.url.strategies.zotero_translators for Python-based implementation
-
-# --- Test Cases ---
-# A curated list of URLs that represent different publishers and translator patterns.
-# Each tuple contains: (test_id, url)
-TEST_CASES = [
-    # --- Tier 1: Core Academic Publishers & Platforms ---
-    ("frontiers", "https://www.frontiersin.org/journals/neuroscience/articles/10.3389/fnins.2019.00573/full"),
-    ("arxiv", "https://arxiv.org/abs/2103.14030"),
-    ("nature", "https://www.nature.com/articles/s41586-021-03372-6"),
-    ("sciencedirect_openaccess", "https://www.sciencedirect.com/science/article/pii/S009286742030120X"),
-    ("wiley", "https://onlinelibrary.wiley.com/doi/full/10.1111/j.1365-2966.2006.10766.x"),
-    ("taylor_and_francis", "https://www.tandfonline.com/doi/full/10.1080/09540261.2021.1897716"),
-    ("sage", "https://journals.sagepub.com/doi/full/10.1177/0002716220987979"),
-    ("springer", "https://link.springer.com/article/10.1007/s00221-021-06089-y"),
-    ("pubmed", "https://pubmed.ncbi.nlm.nih.gov/33758224/"),
-    ("jstor", "https://www.jstor.org/stable/10.5325/jafrireli.14.1.0001"),
-    ("acm_digital_library", "https://dl.acm.org/doi/10.1145/3411764.3445623"),
-    ("acs_publications", "https://pubs.acs.org/doi/10.1021/jacs.0c12533"),
-    ("ieee_xplore", "https://ieeexplore.ieee.org/document/9349473"),
-    ("biomed_central", "https://genomebiology.biomedcentral.com/articles/10.1186/s13059-021-02262-1"),
-    ("plos", "https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0248220"),
-    ("project_muse", "https://muse.jhu.edu/article/786236"),
-    ("biorxiv_highwire", "https://www.biorxiv.org/content/10.1101/2021.03.11.434931v1"),
-    
-    # --- Tier 2: Expanded Academic & Research Sources ---
-    ("oxford_university_press", "https://academic.oup.com/mnras/article/372/2/643/1067311"),
-    ("cambridge_core", "https://www.cambridge.org/core/journals/international-organization/article/abs/power-and-plenty-in-the-first-millennium-ad/A55A51C6A531918118A51A25A39D6582"),
-    ("google_scholar", "https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&q=deep+learning+for+natural+language+processing&btnG="),
-    ("semantic_scholar", "https://www.semanticscholar.org/paper/Attention-Is-All-You-Need-Vaswani-Shazeer/204e3073870fae3d05bcbc2f6a8e263d9b72e776"),
-    ("researchgate", "https://www.researchgate.net/publication/322459231_BERT_Pre-training_of_Deep_Bidirectional_Transformers_for_Language_Understanding"),
-    ("nasa_ads", "https://ui.adsabs.harvard.edu/abs/2017Natur.549...80K/abstract"),
-    ("elife", "https://elifesciences.org/articles/65774"),
-    ("mdpi", "https://www.mdpi.com/1422-0067/22/6/2913"),
-
-    # --- Tier 3: News, Media, and Archives ---
-    ("guardian", "https://www.theguardian.com/science/2021/mar/11/new-light-shed-on-mystery-of-great-unconformity-in-grand-canyon"),
-    ("nytimes", "https://www.nytimes.com/2021/03/10/science/ancient-dna-mammoths.html"),
-    ("internet_archive", "https://archive.org/details/TheAdventuresOfTomSawyer_201303"),
-    ("bbc_news", "https://www.bbc.com/news/science-environment-56328948"),
-    ("reuters", "https://www.reuters.com/lifestyle/science/scientists-find-oldest-known-dna-ancient-mammoth-teeth-2021-02-17/"),
-    ("wired", "https://www.wired.com/story/the-teeny-tiny-master-of-the-energy-universe/"),
-    ("youtube", "https://www.youtube.com/watch?v=1t_Co0g_t4Y"), # Test for video metadata
-    
-    # --- Tier 4: Government, Institutional, and Niche Repositories ---
-    ("world_bank_okr", "https://openknowledge.worldbank.org/handle/10986/35223"),
-    ("github", "https://github.com/zotero/zotero"), # Test for software citation
-    ("imdb", "https://www.imdb.com/title/tt0111161/"), # Test for film metadata
-    ("wikipedia", "https://en.wikipedia.org/wiki/Zotero"), # Test for encyclopedia articles
-    ("hathitrust", "https://babel.hathitrust.org/cgi/pt?id=uc1.b4000000&view=1up&seq=7"),
-    ("dpla", "https://dp.la/item/05001541315c147233a0f6b8b98a39e3"), # Digital Public Library of America
-
-    # --- Tier 5: International Sources ---
-    ("cnki", "https://en.cnki.com.cn/Article_en/CJFDTotal-ZGNY202102001.htm"), # China National Knowledge Infrastructure
-    ("ciNii", "https://ci.nii.ac.jp/naid/130007997939"), # Japanese Institutional Repositories
-    ("scielo", "https://www.scielo.br/j/rbz/a/F9S3gY8YxGqXyQ5Z8zW3vXg/?lang=en"), # SciELO Brazil
-    
-    # --- Final Validation: No Match Case ---
-    ("no_match", "https://www.google.com"),
-]
-
-# --- Pytest Fixtures ---
-
-@pytest.fixture(scope="session")
-def event_loop():
-    """Create an instance of the default event loop for the session."""
-    loop = asyncio.get_event_loop_policy().new_event_loop()
-    yield loop
-    loop.close()
-
-@pytest.fixture(scope="session")
-def runner() -> ZoteroTranslatorRunner:
-    """Provides a single instance of the ZoteroTranslatorRunner for all tests."""
-    print("\n--- Initializing ZoteroTranslatorRunner ---")
-    return ZoteroTranslatorRunner()
-
-@pytest.fixture(scope="session")
-async def browser() -> Browser:
-    """Launches a single browser instance for the entire test session."""
-    async with async_playwright() as p:
-        print("\n--- Launching Browser ---")
-        browser_instance = await p.chromium.launch(headless=True)
-        yield browser_instance
-        print("\n--- Closing Browser ---")
-        await browser_instance.close()
-
-@pytest.fixture
-async def page(browser: Browser) -> Page:
-    """Creates a new page for each test case."""
-    page = await browser.new_page()
-    yield page
-    await page.close()
-
-
-# --- Test Functions ---
-
-@pytest.mark.parametrize("test_id, url", TEST_CASES)
-async def test_translator_execution(runner: ZoteroTranslatorRunner, page: Page, test_id: str, url: str):
-    """
-    Tests the Zotero runner against a specific URL.
-    
-    Args:
-        runner: The ZoteroTranslatorRunner instance.
-        page: The Playwright page object for the test.
-        test_id: A friendly name for the test case.
-        url: The URL to test the translator against.
-    """
-    print(f"\n[Testing: {test_id}] Navigating to: {url}")
-    
-    try:
-        await page.goto(url, wait_until="domcontentloaded", timeout=90000)
-        # Add a longer delay for complex sites or those with anti-bot measures
-        await page.wait_for_timeout(5000)
-    except Exception as e:
-        pytest.fail(f"Failed to navigate to {url}: {e}")
-
-    # Execute the translator runner
-    extracted_urls = await runner.extract_urls_pdf_async(page)
-
-    # Assert the expected outcome
-    if test_id == "no_match":
-        assert extracted_urls is not None, "Runner should not crash on no match"
-        assert len(extracted_urls) == 0, f"Expected no URLs for {test_id}, but found {len(extracted_urls)}"
-        print(f"✅ [SUCCESS: {test_id}] Correctly found no matching translator.")
-    else:
-        assert extracted_urls is not None, "Runner should return a list, not None"
-        assert len(extracted_urls) > 0, f"Expected at least one URL for {test_id}, but found none."
-        
-        first_url = extracted_urls[0]
-        assert isinstance(first_url, str), "Result should be a list of strings"
-        
-        # For non-academic or non-document sources, a PDF link might not exist.
-        # We'll assert that a URL was found, but only check for 'pdf' in relevant academic sources.
-        is_academic_document = test_id not in ["guardian", "nytimes", "internet_archive", "bbc_news", "reuters", "wired", "youtube", "github", "imdb", "wikipedia", "dpla"]
-        if is_academic_document:
-            # Some academic pages link to the abstract/full-text HTML first.
-            # A successful run is finding *any* relevant link, not strictly a PDF.
-            # We will check if the URL is at least a plausible document link.
-            assert any(kw in first_url.lower() for kw in ["pdf", "doi", "abs", "full", "content"]), \
-                f"Expected a document-like URL for {test_id}, but got: {first_url}"
-        
-        print(f"✅ [SUCCESS: {test_id}] Found {len(extracted_urls)} URL(s). First URL: {first_url}")
-
-# To run this script directly for demonstration purposes
-if __name__ == "__main__":
-    print("This is a pytest script. To run the tests, execute 'pytest' in your terminal.")
-    print("Example Test Cases:")
-    for test_id, url in TEST_CASES:
-        print(f"- {test_id.capitalize()}: {url}")
-
-
-
-
-
-
-
-
-
-
-
-
-
-could you check test cases in the javascript contents? I think we can make use of them to automate test in python as well
-
-
-
-Show thinking
-Of course. You've pointed out a fantastic feature of the Zotero translator files that we can absolutely leverage. Many of the .js files contain their own built-in test cases, which are a goldmine for creating a truly comprehensive and automated test suite.
-
-Your idea to use these is excellent because it allows us to:
-
-Automate Test Creation: We don't have to manually find URLs for every single translator.
-
-Verify Exact Outcomes: The embedded tests often specify the exact metadata that should be extracted, allowing for much more precise assertions than just checking if a PDF URL was found.
-
-Ensure Long-Term Stability: As you update your runner or the translators themselves, this automated suite will immediately catch any regressions.
-
-## The Automated Approach
-Here is the strategy to integrate these embedded test cases into your Python test script:
-
-Discover Translators: The script will first scan your zotero_translators directory to find all .js files.
-
-Parse Embedded Tests: For each file, it will read the content and look for the /** BEGIN TEST CASES **/ block. It will then parse the JSON array found inside.
-
-Dynamically Generate Pytest Cases: The script will dynamically create parameterized test cases for pytest based on the URLs and expected outcomes from the parsed JSON.
-
-Execute and Assert: The existing pytest framework will then run these generated tests. The assertions will be modified to check not just for success, but for the correctness of the extracted data when available.
-
-This makes the entire process scalable and self-maintaining.
-
-## The Enhanced pytest Script
-I've modified the test_zotero_runner.py script to automatically discover and use these embedded test cases. This new version is much more powerful.
-
-
-``` python
-#!/usr/bin/env python3
-# -*- coding: utf-8 -*-
-# File: test_zotero_runner.py
-# ----------------------------------------
-"""
-Comprehensive Test Suite for ZoteroTranslatorRunner.
-
-This script uses pytest to validate that the ZoteroTranslatorRunner can
-correctly execute translators. It dynamically discovers translators and their
-embedded test cases, creating a robust and automated testing framework.
-
-Usage:
-1.  Make sure you have pytest and pytest-asyncio installed:
-    pip install pytest pytest-asyncio playwright
-    playwright install
-2.  Place this file in a 'tests' directory.
-3.  Ensure your _ZoteroTranslatorRunner.py and the 'zotero_translators'
-    directory are accessible from your project root.
-4.  Run the tests from your project's root directory:
-    pytest
-"""
-
-import asyncio
-import pytest
-import os
-import json
-import re
-from playwright.async_api import Page, Browser, async_playwright
-
-# Note: ZoteroTranslatorRunner is now deprecated. Use the Python-based zotero_translators instead.
-# Old location (deprecated): from scitex.scholar.url.helpers.finders._ZoteroTranslatorRunner import ZoteroTranslatorRunner
-# New location: Use scitex.scholar.url.strategies.zotero_translators for Python-based implementation
-
-# --- Constants ---
-TRANSLATORS_DIR = os.path.join(os.path.dirname(__file__), '..', 'src', 'scitex', 'scholar', 'url', 'strategies', 'zotero_translators')
-
-# --- Test Case Discovery ---
-
-def discover_translator_test_cases():
-    """
-    Scans the translators directory, parses embedded test cases from each .js file,
-    and yields them for pytest parametrization.
-    """
-    if not os.path.isdir(TRANSLATORS_DIR):
-        print(f"\nWarning: Translators directory not found at {TRANSLATORS_DIR}")
-        return
-
-    for filename in os.listdir(TRANSLATORS_DIR):
-        if filename.endswith(".js"):
-            filepath = os.path.join(TRANSLATORS_DIR, filename)
-            with open(filepath, 'r', encoding='utf-8') as f:
-                content = f.read()
-                # Use a non-greedy match to find the test case block
-                match = re.search(r'/\*\* BEGIN TEST CASES \*\*/\s*(.*?)\s*/\*\* END TEST CASES \*\*/', content, re.DOTALL)
-                if match:
-                    test_case_json_str = match.group(1)
-                    try:
-                        # Some test cases are not valid JSON arrays, but a series of objects.
-                        # We'll wrap them in brackets to handle this.
-                        if not test_case_json_str.strip().startswith('['):
-                            test_case_json_str = f"[{test_case_json_str}]"
-                        
-                        test_cases = json.loads(test_case_json_str)
-                        for i, test_case in enumerate(test_cases):
-                            if test_case.get("type") == "web" and test_case.get("url"):
-                                test_id = f"{filename.replace('.js', '')}-{i}"
-                                yield pytest.param(test_id, test_case['url'], test_case['items'], id=test_id)
-                    except json.JSONDecodeError as e:
-                        print(f"\nWarning: Could not parse test cases in {filename}: {e}")
-
-
-# --- Pytest Fixtures ---
-
-@pytest.fixture(scope="session")
-def event_loop():
-    loop = asyncio.get_event_loop_policy().new_event_loop()
-    yield loop
-    loop.close()
-
-@pytest.fixture(scope="session")
-def runner() -> ZoteroTranslatorRunner:
-    print("\n--- Initializing ZoteroTranslatorRunner ---")
-    return ZoteroTranslatorRunner()
-
-@pytest.fixture(scope="session")
-async def browser() -> Browser:
-    async with async_playwright() as p:
-        print("\n--- Launching Browser ---")
-        browser_instance = await p.chromium.launch(headless=True)
-        yield browser_instance
-        print("\n--- Closing Browser ---")
-        await browser_instance.close()
-
-@pytest.fixture
-async def page(browser: Browser) -> Page:
-    page = await browser.new_page()
-    yield page
-    await page.close()
-
-
-# --- Test Functions ---
-
-@pytest.mark.parametrize("test_id, url, expected_items", discover_translator_test_cases())
-async def test_translator_execution(runner: ZoteroTranslatorRunner, page: Page, test_id: str, url: str, expected_items):
-    """
-    Tests the Zotero runner against a specific URL from an embedded test case.
-    """
-    print(f"\n[Testing: {test_id}] Navigating to: {url}")
-    
-    try:
-        await page.goto(url, wait_until="domcontentloaded", timeout=60000)
-        await page.wait_for_timeout(3000)
-    except Exception as e:
-        pytest.fail(f"Failed to navigate to {url}: {e}")
-
-    # Execute the translator runner
-    extracted_urls = await runner.extract_urls_pdf_async(page)
-
-    # Assert the expected outcome based on the embedded test case
-    if expected_items == "multiple":
-        assert extracted_urls is not None, "Runner should not crash"
-        assert len(extracted_urls) > 0, f"Expected multiple items for {test_id}, but found none."
-        print(f"✅ [SUCCESS: {test_id}] Correctly identified as multiple.")
-    elif isinstance(expected_items, list) and len(expected_items) == 0:
-        assert len(extracted_urls) == 0, f"Expected no items for {test_id}, but found {len(extracted_urls)}."
-        print(f"✅ [SUCCESS: {test_id}] Correctly found no items.")
-    elif isinstance(expected_items, list) and len(expected_items) > 0:
-        assert extracted_urls is not None, "Runner should return a list, not None"
-        assert len(extracted_urls) > 0, f"Expected at least one URL for {test_id}, but found none."
-        
-        first_url = extracted_urls[0]
-        assert isinstance(first_url, str), "Result should be a list of strings"
-        
-        # A simple check for now, can be expanded to compare metadata
-        print(f"✅ [SUCCESS: {test_id}] Found {len(extracted_urls)} URL(s). First URL: {first_url}")
-    else:
-        # This covers cases where 'items' might be something else, like a boolean.
-        # We can add more specific assertions here as needed.
-        print(f"✅ [SKIPPED: {test_id}] Non-standard 'items' field in test case: {expected_items}")
-
-
-if __name__ == "__main__":
-    print("This is a pytest script. To run the tests, execute 'pytest' in your terminal.")
-
-
-Yes, the translatorType in the header of each JavaScript file is a crucial piece of metadata that tells Zotero what kind of translator it is and how it should be used. It's a numeric code where different numbers (and their combinations) represent different functionalities.
-
-Based on your search results, here is a breakdown of the patterns and what they mean.
-
-Translator Type Categories
-The translatorType is a bitwise flag. This means that a single number can represent multiple capabilities by adding up the values of the types it includes.
-
-"translatorType": 4 (Web Scraper)
-
-Meaning: This is by far the most common type in your collection. It signifies a standard web translator designed to run on a live webpage.
-
-Function: It uses the detectWeb() and doWeb() entry points to scrape metadata directly from the HTML of a webpage.
-
-Your Use Case: These are the primary translators your Python runner is designed to execute.
-
-"translatorType": 1 (Import)
-
-Meaning: This is an import translator. It's not designed to read a live webpage.
-
-Function: Its purpose is to parse a file or a block of text that is already in a structured format (like BibTeX, RIS, or MARC) and convert it into Zotero items. For example, BibTeX.js would be an import translator.
-
-Your Use Case: Your runner would generally not use these, as you are starting from a URL, not a text file.
-
-"translatorType": 2 (Export)
-
-Meaning: This is an export translator.
-
-Function: It does the reverse of an import translator. It takes Zotero items and converts them into a specific text-based format (like a BibTeX file or a formatted bibliography). BibLaTeX.js is a perfect example.
-
-Your Use Case: These are not relevant to your scraping task.
-
-"translatorType": 8 (Search)
-
-Meaning: This is a search translator.
-
-Function: It's designed to work with library database search protocols like SRU or Z39.50. It doesn't scrape a webpage but rather sends a structured query to a database and parses the structured response. Your ADS Bibcode.js and various ISBN translators fall into this category.
-
-Your Use Case: Your current runner doesn't support this, as it would require implementing the specific search protocols.
-
-"translatorType": 12 (Web Scraper + Search)
-
-Meaning: This is a hybrid translator. The number 12 is the sum of 4 (Web) + 8 (Search).
-
-Function: This translator can both scrape a live webpage and perform a direct search query. PubMed.js and arXiv.org.js are great examples. They can scrape a single article page but can also be used by Zotero to search the entire PubMed or arXiv database.
-
-Your Use Case: Your runner can successfully use the web scraping (4) part of these translators.
-
-Summary of Patterns
-For your project, the most important distinction is that your runner is built to handle translators where the type is 4 or includes 4 (like 12). The other types (1, 2, 3, 8) are for different functions within Zotero (importing, exporting, and specialized searching) and won't be executed by your current web-based automation script.
-
-
-
-
-
-
-
-so, all we need is concatelate on 4 and 12;
-
-
-
-Then, another question is that do you think how many and which translators are vital for us?
-
-
-
-
-Show thinking
-You've asked two excellent strategic questions. Here’s a breakdown.
-
-Focusing on Translator Types 4 and 12
-Yes, your understanding is spot on. For your specific goal of building a web scraper that starts from a URL, you should focus exclusively on translators with translatorType 4 (Web) and 12 (Web + Search).
-
-Why? Your runner is designed to operate on live webpages. Only translators marked as "Web" (type 4) are built for this purpose. Type 12 is just a "Web" translator that also has a "Search" capability, so it's perfectly compatible with your system.
-
-Action: You can modify your Python script that loads the translators to simply skip any file where the translatorType is not 4 or 12. This will make your system more efficient and prevent potential errors from trying to run incompatible translator types.
-
-Which Translators are Vital for Academic Use?
-You have over 700 translators, but you don't need to ensure all of them work perfectly to have a highly effective system. The Pareto Principle (80/20 rule) applies here: a small number of key translators will cover the vast majority of academic sources you'll encounter.
-
-Here is a prioritized list of the most vital translators to focus on for academic purposes:
-
-Tier 1: The Core Giants (Must-Haves)
-These platforms host a massive percentage of all modern scientific literature. Ensuring these work flawlessly is your top priority.
-
-ScienceDirect.js: For Elsevier journals.
-
-Springer Link.js: For Springer content.
-
-Wiley Online Library.js: For Wiley journals.
-
-Taylor and Francis+NEJM.js: For Taylor & Francis and the New England Journal of Medicine.
-
-SAGE Journals.js: For SAGE publications.
-
-Tier 2: Major Aggregators & Databases
-These are the central hubs where researchers find papers from many different publishers.
-
-PubMed.js: The essential database for all biomedical research.
-
-Google Scholar.js: The most widely used academic search engine.
-
-JSTOR.js: A critical archive for humanities and social sciences.
-
-arXiv.org.js: The main preprint server for physics, computer science, and math.
-
-Project MUSE.js: Another key aggregator for humanities and social sciences.
-
-Tier 3: High-Impact Publishers & Societies
-These are individual publishers that are extremely influential and common.
-
-Nature Publishing Group.js: For Nature and its associated journals.
-
-ACS Publications.js: American Chemical Society.
-
-ACM Digital Library.js: Association for Computing Machinery.
-
-IEEE Xplore.js: Institute of Electrical and Electronics Engineers.
-
-Oxford University Press.js
-
-Cambridge Core.js
-
-Tier 4: Key Platforms & Open Access Hubs
-These translators cover platforms that host many journals or are crucial for finding open-access versions.
-
-HighWire 2.0.js: A platform used by many societies (and for preprint servers like bioRxiv).
-
-Atypon Journals.js: Another very large platform provider.
-
-BioMed Central.js and PLoS Journals.js: Major open-access publishers.
-
-By focusing your testing and development efforts on this "vital list" of about 20-25 translators, you will likely cover over 80% of the academic articles your system will need to process.
-```
-
-<!-- EOF -->