scitex 2.14.0__py3-none-any.whl

scitex/scholar/docs/zenrows_official/final_url.md

@@ -0,0 +1,204 @@
+<!-- ---
+!-- Timestamp: 2025-07-31 02:15:23
+!-- Author: ywatanabe
+!-- File: /home/ywatanabe/proj/scitex_repo/src/scitex/scholar/docs/zenros_final_url.md
+!-- --- -->
+
+Introduction to the Universal Scraper API
+The ZenRows® Universal Scraper API is a versatile tool designed to simplify and enhance the process of extracting data from websites. Whether you’re dealing with static or dynamic content, our API provides a range of features to meet your scraping needs efficiently.
+With Premium Proxies, ZenRows gives you access to over 55 million residential IPs from 190+ countries, ensuring 99.9% uptime and highly reliable scraping sessions. Our system also handles advanced fingerprinting, header rotation, and IP management, enabling you to scrape_async even the most protected sites without needing to manually configure these elements.
+ZenRows makes it easy to bypass complex anti-bot measures, handle JavaScript-heavy sites, and interact with web elements dynamically — all with the right features enabled.
+​
+Key Features
+​
+JavaScript Rendering
+Render JavaScript on web pages using a headless browser to scrape_async dynamic content that traditional methods might miss.
+When to use: Use this feature when targeting modern websites built with JavaScript frameworks (React, Vue, Angular), single-page applications (SPAs), or any site that loads content dynamically after the initial page load.
+Real-world scenarios:
+E-commerce product listings that load items as you scroll
+Dashboards and analytics platforms that render charts/data with JavaScript
+Social media feeds that dynamically append content
+Sites that hide_async certain content until JavaScript is rendered
+Additional options:
+Wait times to ensure elements are fully loaded
+Interaction with the page to click buttons, fill forms, or scroll
+Screenshot capabilities for visual verification
+CSS-based extraction of specific elements
+​
+Premium Proxies
+Leverage a vast network of residential IP addresses across 190+ countries, ensuring a 99.9% uptime for uninterrupted scraping.
+When to use: Essential for accessing websites with sophisticated anti-bot systems, geo-restricted content, or when you consistently encounter blocks with standard datacenter IPs.
+Real-world scenarios:
+Scraping major e-commerce platforms (Amazon, Walmart)
+Accessing real estate listings (Zillow, Redfin)
+Gathering pricing data from travel sites (Expedia, Booking.com)
+Collecting data from financial or investment platforms
+Additional options:
+Geolocation selection to access region-specific content
+Automatic IP rotation to prevent detection
+​
+Custom Headers
+Add custom HTTP headers to your requests for more control over how your requests appear to target websites.
+When to use: When you need to mimic specific browser behavior, set cookies, or a referer.
+Real-world scenarios:
+Setting language preferences to get content in specific languages
+Adding a referer to avoid being blocked by bot detection systems
+​
+Session Management
+Use a session ID to maintain the same IP address across multiple requests for up to 10 minutes.
+When to use: When scraping multi-page flows or processes that require maintaining the same IP across multiple requests.
+Real-world scenarios:
+Multi-step forms processes
+Maintaining consistent session for search results and item visits
+​
+Advanced Data Extraction
+Extract only the data you need with CSS selectors or automatic parsing.
+When to use: When you need specific information from pages and want to reduce bandwidth usage or simplify post-processing.
+Real-world scenarios:
+Extracting pricing information from product pages
+Gathering contact details from business directories
+Collecting specific metrics from analytics pages
+​
+Language agnostic
+While Python examples are provided, the API works with any programming language that can make HTTP requests.
+​
+Parameter Overview
+Customize your scraping requests using the following parameters:
+PARAMETER	TYPE	DEFAULT	DESCRIPTION
+apikey required	string	Get Your Free API Key	Your unique API key for authentication
+url required	string		The URL of the page you want to scrape_async
+js_render	boolean	false	Enable JavaScript rendering with a headless browser. Essential for modern web apps, SPAs, and sites with dynamic content.
+js_instructions	string		Execute custom JavaScript on the page to interact with elements, scroll, click buttons, or manipulate content. Use when you need to perform actions before the content is returned.
+custom_headers	boolean	false	Enables you to add custom HTTP headers to your request, such as cookies or referer, to better simulate real browser traffic or provide site-specific information.
+premium_proxy	boolean	false	Use residential IPs to bypass anti-bot protection. Essential for accessing protected sites.
+proxy_country	string		Set the country of the IP used for the request (requires Premium Proxies). Use for accessing geo-restricted content or seeing region-specific content.
+session_id	integer		Maintain the same IP for multiple requests for up to 10 minutes. Essential for multi-step processes.
+original_status	boolean	false	Return the original HTTP status code from the target page. Useful for debugging in case of errors.
+allowed_status_codes	string		Returns the content even if the target page fails with specified status codes. Useful for debugging or when you need content from error pages.
+wait_for	string		Wait for a specific CSS Selector to appear in the DOM before returning content. Essential for elements that load asynchronously.
+wait	integer	0	Wait a fixed amount of milliseconds after page load. Use for sites that load content in stages or have delayed rendering.
+block_resources	string		Block specific resources (images, fonts, etc.) from loading to speed up scraping and reduce bandwidth usage. Enabled by default, carefull when changing it.
+json_response	string	false	Capture network requests in JSON format, including XHR or Fetch data. Ideal for intercepting API calls made by the web page.
+css_extractor	string (JSON)		Extract specific elements using CSS selectors. Perfect for targeting only the data you need from complex pages.
+autoparse	boolean	false	Automatically extract structured data from HTML. Great for quick extraction without specifying selectors.
+response_type	string		Convert HTML to other formats (Markdown, Plaintext, PDF). Useful for content readability, storage, or to train AI models.
+screenshot	boolean	false	Capture an above-the-fold screenshot of the page. Helpful for visual verification or debugging.
+screenshot_fullpage	boolean	false	Capture a full-page screenshot. Useful for content that extends below the fold.
+screenshot_selector	string		Capture a screenshot of a specific element using CSS Selector. Perfect for capturing specific components.
+screenshot_format	string		Choose between png (default) and jpeg formats for screenshots.
+screenshot_quality	integer		For JPEG format, set quality from 1 to 100. Lower values reduce file size but decrease quality.
+outputs	string		Specify which data types to extract from the scrape_async HTML.
+​
+Pricing
+ZenRows® provides flexible plans tailored to different web scraping needs, starting from $69 per month. This entry-level plan allows you to scrape_async up to 250,000 URLs using basic requests. For more demanding needs, our Enterprise plans scale up to 38 million URLs or more.
+For complex or highly protected websites, enabling advanced features like JavaScript rendering (js_render) and Premium Proxies unlocks ZenRows’ full potential, ensuring the best success rate possible.
+The pricing depends on the complexity of the request — you only pay for the scraping tech you need.
+Basic request: Standard rate per 1,000 requests
+JS rendering: 5x cost
+Premium proxies: 10x cost
+Both (JS & proxies): 25x cost
+For example, on the Business plan:
+Basic: $0.10 per 1,000 requests
+JS: $0.45 per 1,000
+Proxies: $0.90 per 1,000
+Both: $2.50 per 1,000
+For detailed information about different plan options and pricing, visit our pricing page and our pricing documentation page.
+​
+Concurrency and Response Size Limits
+Concurrency determines how many requests can run simultaneously:
+Plan	Concurrency Limit	Response Size Limit
+Developer	5	5 MB
+Startup	20	10 MB
+Business	50	10 MB
+Business 500	100	20 MB
+Business 1K	150	20 MB
+Business 2K	200	50 MB
+Business 3K	250	50 MB
+Enterprise	Custom	50 MB
+Important notes about concurrency:
+Canceling requests on the client side does NOT immediately free up concurrency slots
+The server continues processing canceled requests until completion
+If you exceed your concurrency limit, you’ll receive a 429 Too Many Requests error
+If response size is exceeded:
+You’ll receive a 413 Content Too Large error
+No partial data will be returned when a size limit is hit
+Strategies for handling large pages:
+Use CSS selectors: Target only the specific data you need with css_extractor parameter
+Use response_type: Convert to markdown or plaintext to reduce size
+Disable screenshots: If using screenshot features, these can significantly increase response size
+Segment your scraping: Break down large pages into smaller, more manageable sections
+​
+Response Headers
+ZenRows provides useful information through response headers:
+Header	Description	Example Value	Usage
+Concurrency-Limit	Maximum concurrent requests allowed by your plan	20	Monitor your plan’s capacity
+Concurrency-Remaining	Available concurrent request slots	17	Adjust request rate dynamically
+X-Request-Cost	Cost of this request	0.001	Track balance consumption
+X-Request-Id	Unique identifier for this request	67fa4e35647515d8ad61bb3ee041e1bb	Include when contacting support
+Zr-Final-Url	The final URL after any redirects occurred during the request	https://example.com/page?id=123	Track redirects
+Why these headers matter:
+Monitoring usage: Track your concurrent usage and stay within limits
+Support requests: When reporting issues, always include the X-Request-Id for faster troubleshooting
+Cost tracking: The X-Request-Cost helps you monitor your usage per request
+Redirection tracking: Zr-Final-Url show_asyncs where you ended up after any redirects
+​
+Additional Considerations
+Beyond the core features and limits, these additional aspects are important to consider when using the Universal Scraper API:
+​
+Cancelled Request Behavior
+When you cancel a request on the client side:
+The server continues processing the request until completion
+The concurrency slot remains occupied for up to 3 minutes
+This can result in unexpected 429 Too Many Requests errors
+Implement request timeouts carefully to avoid depleting concurrency slots
+​
+Security Best Practices
+To keep your ZenRows integration secure:
+Store API keys as environment variables, never hardcode them
+Monitor usage patterns to detect unauthorized use
+Rotate API keys periodically for critical applications
+​
+Regional Performance Optimization
+To optimize performance based on target website location:
+Consider the geographical distance between your servers and the target website
+For global applications, distribute scraping across multiple regions - the system does it by default
+Monitor response times by region to identify optimization opportunities
+For region-specific content, use the appropriate proxy_country parameter
+​
+Compression Support
+ZenRows API supports response compression to optimize bandwidth usage and improve performance. Enabling compression offers several benefits for your scraping operations:
+Reduced latency: Smaller response sizes mean faster data transfer times
+Lower bandwidth consumption: Minimize data transfer costs and usage
+Improved client performance: Less data to process means reduced memory usage
+ZenRows supports the following compression encodings: gzip, deflate, br.
+To use compression, include the appropriate Accept-Encoding header in your requests. Most HTTP clients already compress the request automatically. But you can also provide simple options to enable it:
+
+Python (Requests)
+
+Javascript (Axios)
+
+cURL
+
+Copy
+
+Ask AI
+import requests
+
+response = requests.get(
+    "https://api.zenrows.com/v1/",
+    params={
+        "apikey": "YOUR_API_KEY",
+        "url": "https://example.com"
+    },
+    # Python Requests uses compression by default
+    # headers={"Accept-Encoding": "gzip,  deflate"}
+)
+Most modern HTTP clients automatically handle decompression, so you’ll receive the uncompressed content in your response object without any additional configuration.
+Was this page helpful?
+
+
+Yes
+
+No
+
+<!-- EOF -->

scitex/scholar/docs/zenrows_official/with_playwright.md

@@ -0,0 +1,336 @@
+<!-- ---
+!-- Timestamp: 2025-07-31 01:42:22
+!-- Author: ywatanabe
+!-- File: /home/ywatanabe/proj/scitex_repo/src/scitex/scholar/docs/zenrows_with_playwright.md
+!-- --- -->
+
+Integrating ZenRows Scraping Browser with Playwright
+Learn to extract data from any website using ZenRows’ Scraping Browser with Playwright. This guide walks you through creating your first browser-based scraping request that can handle complex JavaScript-heavy sites with full browser automation.
+ZenRows’ Scraping Browser provides cloud-based Chrome instances you can control using Playwright. Whether dealing with dynamic content, complex user interactions, or sophisticated anti-bot protection, you can get started in minutes with Playwright’s powerful automation capabilities.
+​
+1. Set Up Your Project
+​
+Set Up Your Development Environment
+Before diving in, ensure you have the proper development environment and Playwright installed. The Scraping Browser works seamlessly with both Python and Node.js versions of Playwright.
+While previous versions may work, we recommend using the latest stable versions for optimal performance and security.
+Python
+Node.js
+Python 3+ installed (latest stable version recommended). Using an IDE like PyCharm or Visual Studio Code with the Python extension is recommended.
+
+Copy
+
+Ask AI
+# Install Python (if not already installed)
+# Visit https://www.python.org/downloads/ or use package managers:
+
+# macOS (using Homebrew)
+brew install python
+
+# Ubuntu/Debian
+sudo apt update && sudo apt install python3 python3-pip
+
+# Windows (using Chocolatey)
+choco install python
+
+# Install Playwright
+pip install playwright
+playwright install
+If you need help setting up your environment, check out our detailed Playwright web scraping guide
+​
+Get Your API Key and Connection URL
+Sign Up for a free ZenRows account and get your API key from the Scraping Browser dashboard. You’ll need this key to authenticate_async your WebSocket connection.
+​
+2. Make Your First Request
+Start with a simple request to understand how the Scraping Browser works with Playwright. We’ll use the E-commerce Challenge page to demonstrate how to connect to the browser and extract the page title.
+
+Python
+
+Node.js
+
+Copy
+
+Ask AI
+# pip install playwright
+import asyncio
+from playwright.async_api import async_playwright
+
+# scraping browser connection URL
+connection_url = "wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY"
+
+async def scrape_asyncr_async():
+    async with async_playwright() as p:
+        # connect to the scraping browser
+        browser = await p.chromium.connect_over_cdp(connection_url)
+        context = browser.contexts[0] if browser.contexts else await browser.new_context()
+        page = await context.new_page()
+        
+        await page.goto('https://www.scrapingcourse.com/ecommerce/')
+        print(await page.title())
+        
+        await browser.close()
+
+if __name__ == "__main__":
+    asyncio.run(scrape_asyncr_async())
+Replace YOUR_ZENROWS_API_KEY with your actual API key and run the script:
+
+Python
+
+Node.js
+
+Copy
+
+Ask AI
+python scrape_asyncr_async.py
+​
+Expected Output
+The script will print the page title:
+
+Copy
+
+Ask AI
+ScrapingCourse.com E-commerce Challenge
+Perfect! You’ve just made your first web scraping request with the ZenRows Scraping Browser using Playwright.
+​
+3. Build a Real-World Scraping Scenario
+Let’s scale up to a practical scraping scenario by extracting product information from the e-commerce site. Using Playwright’s powerful selectors and data extraction methods, we’ll modify our code to extract product names, prices, and URLs from the page.
+
+Python
+
+Node.js
+
+Copy
+
+Ask AI
+# pip install playwright
+import asyncio
+from playwright.async_api import async_playwright
+
+# scraping browser connection URL
+connection_url = "wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY"
+
+async def scrape_asyncr_async(url):
+    async with async_playwright() as p:
+        # connect to the scraping browser
+        browser = await p.chromium.connect_over_cdp(connection_url)
+        page = await browser.new_page()
+        
+        try:
+            await page.goto(url)
+            
+            # extract the desired data
+            await page.wait_for_selector(".product")
+            products = await page.query_selector_all(".product")
+            data = []
+            
+            for product in products:
+                name = await product.query_selector(".product-name")
+                price = await product.query_selector(".price")
+                product_url = await product.query_selector(".woocommerce-LoopProduct-link")
+                
+                data.append({
+                    "name": await name.text_content() or "",
+                    "price": await price.text_content() or "",
+                    "productURL": await product_url.get_attribute("href") or "",
+                })
+            
+            return data
+        except Exception as error:
+            return error
+        finally:
+            await page.close()
+            await browser.close()
+
+if __name__ == "__main__":
+    url = "https://www.scrapingcourse.com/ecommerce/"
+    products = asyncio.run(scrape_asyncr_async(url))
+    print(products)
+​
+Run Your Application
+Execute your script to test the scraping functionality:
+
+Python
+
+Node.js
+
+Copy
+
+Ask AI
+python scrape_asyncr_async.py
+Example Output
+The script will extract and display product information:
+
+Copy
+
+Ask AI
+[
+    {
+        "name": "Abominable Hoodie",
+        "price": "$69.00",
+        "productURL": "https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie/"
+    },
+    {
+        "name": "Artemis Running Short",
+        "price": "$45.00",
+        "productURL": "https://www.scrapingcourse.com/ecommerce/product/artemis-running-short/"
+    }
+    // ... more products
+]
+Congratulations! 🎉 You’ve successfully built a real-world scraping scenario with Playwright and the ZenRows Scraping Browser.
+​
+4. Alternative: Using the ZenRows Browser SDK
+For a more streamlined development experience, you can use the ZenRows Browser SDK instead of managing WebSocket URLs manually. The SDK simplifies connection management and provides additional utilities.
+The ZenRows Browser SDK is currently only available for JavaScript. For more details, see the GitHub Repository.
+​
+Install the SDK
+Node.js
+
+Copy
+
+Ask AI
+npm install @zenrows/browser-sdk
+​
+Quick Migration from WebSocket URL
+If you have existing Playwright code using the WebSocket connection, migrating to the SDK requires minimal changes:
+Before (WebSocket URL):
+Node.js
+
+Copy
+
+Ask AI
+const { chromium } = require('playwright');
+const connectionURL = 'wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY';
+
+const browser = await chromium.connectOverCDP(connectionURL);
+After (SDK):
+Node.js
+
+Copy
+
+Ask AI
+const { chromium } = require('playwright');
+const { ScrapingBrowser } = require('@zenrows/browser-sdk');
+
+const scrapingBrowser = new ScrapingBrowser({ apiKey: 'YOUR_ZENROWS_API_KEY' });
+const connectionURL = scrapingBrowser.getConnectURL();
+const browser = await chromium.connectOverCDP(connectionURL);
+​
+Complete Example with SDK
+Node.js
+
+Copy
+
+Ask AI
+// npm install @zenrows/browser-sdk playwright
+const { chromium } = require('playwright');
+const { ScrapingBrowser } = require('@zenrows/browser-sdk');
+
+const scrape_asyncr_async = async () => {
+    // Initialize SDK
+    const scrapingBrowser = new ScrapingBrowser({ apiKey: 'YOUR_ZENROWS_API_KEY' });
+    const connectionURL = scrapingBrowser.getConnectURL();
+    
+    const browser = await chromium.connectOverCDP(connectionURL);
+    const page = await browser.newPage();
+    
+    await page.goto('https://www.scrapingcourse.com/ecommerce/');
+    console.log(await page.title());
+    
+    await browser.close();
+};
+
+scrape_asyncr_async();
+​
+SDK Benefits
+Simplified configuration: No need to manually construct WebSocket URLs
+Better error handling: Built-in error messages and debugging information
+Future-proof: Automatic updates to connection protocols and endpoints
+Additional utilities: Access to helper methods and advanced configuration options
+The SDK is particularly useful for production environments where you want cleaner code organization and better error handling.
+​
+How Playwright with Scraping Browser Helps
+Combining Playwright with ZenRows’ Scraping Browser provides powerful advantages for web scraping:
+​
+Key Benefits
+Cloud-based browser instances: Run Playwright scripts on remote Chrome instances, freeing up local resources for other tasks.
+Seamless integration: Connect your existing Playwright code to ZenRows with just a WebSocket URL change - no complex setup required.
+Advanced automation: Use Playwright’s full feature set, which includes page interactions, form submissions, file uploads, and complex user workflows.
+Built-in anti-detection: Benefit from residential proxy rotation and genuine browser fingerprints automatically.
+Cross-browser support: While we use Chromium for optimal compatibility, Playwright’s API remains consistent across different browser engines.
+High concurrency: Scale your Playwright scripts with up to 150 concurrent browser instances, depending on your plan.
+Reliable execution: Cloud infrastructure ensures consistent performance without local browser management overhead.
+​
+Troubleshooting
+Below are common issues you might encounter when using Playwright with the Scraping Browser:
+1
+Connection Refused
+
+If you receive a Connection Refused error, it might be due to:
+API Key Issues: Verify that you’re using the correct API key.
+Network Issues: Check your internet connection and firewall settings.
+WebSocket Endpoint: Ensure that the WebSocket URL (wss://browser.zenrows.com) is correct.
+2
+Empty Data or Timeout Errors
+
+Use page.waitForSelector() to ensure elements load before extraction
+Increase timeout values for slow-loading pages
+scrape_asyncr_async.js
+
+Copy
+
+Ask AI
+await page.goto('https://example.com', { timeout: 60000 });  // 60 seconds
+Verify CSS selectors are correct using browser developer tools
+Add page.waitForLoadState('networkidle') for dynamic content
+3
+Browser Context Issues
+
+Use existing context when available: browser.contexts[0] if browser.contexts else await browser.new_context()
+Properly close pages and browsers to prevent resource leaks
+Handle exceptions properly to ensure cleanup occurs
+4
+Geolocation Blocks
+
+Although ZenRows rotates IPs, some websites may block them based on location. Try adjusting the region or country settings.
+For more information, check our Scraping Browser Region Documentation and Country Documentation.
+5
+Get Help From ZenRows Experts
+
+Our support team is available to assist you if issues persist despite following these solutions. Use the Scraping Browser dashboard or email us for personalized help from ZenRows experts.
+​
+Next Steps
+You now have a solid foundation for Playwright-based web scraping with ZenRows. Here are some recommended next steps:
+Practical Use Cases:
+Learn common scraping patterns, including screenshots, custom JavaScript execution, and form handling.
+Advanced Features:
+Master complex automation workflows and advanced Playwright capabilities.
+Complete Scraping Browser Documentation:
+Explore all available features and advanced configuration options for the Scraping Browser.
+Playwright Advanced Features:
+Learn about page interactions, form handling, file uploads, and complex automation workflows.
+Playwright Web Scraping Guide:
+Dive deeper into Playwright techniques for sophisticated scraping scenarios.
+Pricing and Plans:
+Understand how browser usage is calculated and choose the plan that fits your scraping volume.
+​
+Frequently Asked Questions (FAQ)
+Can I use ZenRows® Scraping Browser with Puppeteer?
+
+Do I need to configure proxies manually with ZenRows® Scraping Browser?
+
+Can the Scraping Browser solve CAPTCHAs?
+
+Can I use all Playwright features with the Scraping Browser?
+
+How do I handle multiple pages or tabs?
+
+Can I use Playwright's built-in retry mechanisms?
+
+How do I take screenshots with Playwright and Scraping Browser?
+
+Can I use Playwright's network interception features?
+
+What's the difference between using Playwright locally vs. with Scraping Browser?
+
+How do I handle file downloads with Playwright and Scraping Browser?
+
+<!-- EOF -->