ultimate-pi 0.1.0 → 0.1.3

package/.agents/skills/scrapling-official/references/fetching/stealthy.md

@@ -1,255 +0,0 @@
-# StealthyFetcher
-
-`StealthyFetcher` is a stealthy browser-based fetcher similar to [DynamicFetcher](dynamic.md), using [Playwright's API](https://playwright.dev/python/docs/intro). It adds advanced anti-bot protection bypass capabilities, most handled automatically. It shares the same browser automation model as `DynamicFetcher`, using [Playwright's Page API](https://playwright.dev/python/docs/api/class-page) for page interaction.
-
-## Basic Usage
-You have one primary way to import this Fetcher, which is the same for all fetchers.
-
-```python
->>> from scrapling.fetchers import StealthyFetcher
-```
-Check out how to configure the parsing options [here](choosing.md#parser-configuration-in-all-fetchers)
-
-**Note:** The async version of the `fetch` method is `async_fetch`.
-
-## What does it do?
-
-The `StealthyFetcher` class is a stealthy version of the [DynamicFetcher](dynamic.md) class, and here are some of the things it does:
-
-1. It easily bypasses all types of Cloudflare's Turnstile/Interstitial automatically. 
-2. It bypasses CDP runtime leaks and WebRTC leaks.
-3. It isolates JS execution, removes many Playwright fingerprints, and stops detection through some of the known behaviors that bots do.
-4. It generates canvas noise to prevent fingerprinting through canvas.
-5. It automatically patches known methods to detect running in headless mode and provides an option to defeat timezone mismatch attacks.
-6. and other anti-protection options...
-
-## Full list of arguments
-Scrapling provides many options with this fetcher and its session classes. Before jumping to the [examples](#examples), here's the full list of arguments
-
-
-|      Argument       | Description                                                                                                                                                                                                                         | Optional |
-|:-------------------:|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------:|
-|         url         | Target url                                                                                                                                                                                                                          |    ❌     |
-|      headless       | Pass `True` to run the browser in headless/hidden (**default**) or `False` for headful/visible mode.                                                                                                                                |    ✔️    |
-|  disable_resources  | Drop requests for unnecessary resources for a speed boost. Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.                         |    ✔️    |
-|       cookies       | Set cookies for the next request.                                                                                                                                                                                                   |    ✔️    |
-|      useragent      | Pass a useragent string to be used. **Otherwise, the fetcher will generate and use a real Useragent of the same browser and version.**                                                                                              |    ✔️    |
-|    network_idle     | Wait for the page until there are no network connections for at least 500 ms.                                                                                                                                                       |    ✔️    |
-|      load_dom       | Enabled by default, wait for all JavaScript on page(s) to fully load and execute (wait for the `domcontentloaded` state).                                                                                                           |    ✔️    |
-|       timeout       | The timeout (milliseconds) used in all operations and waits through the page. The default is 30,000 ms (30 seconds).                                                                                                                |    ✔️    |
-|        wait         | The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the `Response` object.                                                                                                |    ✔️    |
-|     page_action     | Added for automation. Pass a function that takes the `page` object, runs after navigation, and does the necessary automation.                                                                                                       |    ✔️    |
-|     page_setup      | A function that takes the `page` object, runs before navigation. Use it to register event listeners or routes that must be set up before the page loads.                                                                            |    ✔️    |
-|    wait_selector    | Wait for a specific css selector to be in a specific state.                                                                                                                                                                         |    ✔️    |
-|     init_script     | An absolute path to a JavaScript file to be executed on page creation for all pages in this session.                                                                                                                                |    ✔️    |
-| wait_selector_state | Scrapling will wait for the given state to be fulfilled for the selector given with `wait_selector`. _Default state is `attached`._                                                                                                 |    ✔️    |
-|    google_search    | Enabled by default, Scrapling will set a Google referer header.                                                                                               |    ✔️    |
-|    extra_headers    | A dictionary of extra headers to add to the request. _The referer set by `google_search` takes priority over the referer set here if used together._                                                                   |    ✔️    |
-|        proxy        | The proxy to be used with requests. It can be a string or a dictionary with only the keys 'server', 'username', and 'password'.                                                                                                     |    ✔️    |
-|     real_chrome     | If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch and use an instance of your browser.                                                                                                |    ✔️    |
-|       locale        | Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect `navigator.language` value, `Accept-Language` request header value, as well as number and date formatting rules. Defaults to the system default locale. |    ✔️    |
-|     timezone_id     | Changes the timezone of the browser. Defaults to the system timezone.                                                                                                                                                               |    ✔️    |
-|       cdp_url       | Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.                                                                                                                          |    ✔️    |
-|    user_data_dir    | Path to a User Data Directory, which stores browser session data like cookies and local storage. The default is to create a temporary directory. **Only Works with sessions**                                                       |    ✔️    |
-|     extra_flags     | A list of additional browser flags to pass to the browser on launch.                                                                                                                                                                |    ✔️    |
-|  solve_cloudflare   | When enabled, fetcher solves all types of Cloudflare's Turnstile/Interstitial challenges before returning the response to you.                                                                                                      |    ✔️    |
-|    block_webrtc     | Forces WebRTC to respect proxy settings to prevent local IP address leak.                                                                                                                                                           |    ✔️    |
-|     hide_canvas     | Add random noise to canvas operations to prevent fingerprinting.                                                                                                                                                                    |    ✔️    |
-|     allow_webgl     | Enabled by default. Disabling it disables WebGL and WebGL 2.0 support entirely. Disabling WebGL is not recommended, as many WAFs now check if WebGL is enabled.                                                                     |    ✔️    |
-|   additional_args   | Additional arguments to be passed to Playwright's context as additional settings, and they take higher priority than Scrapling's settings.                                                                                          |    ✔️    |
-|   selector_config   | A dictionary of custom parsing arguments to be used when creating the final `Selector`/`Response` class.                                                                                                                            |    ✔️    |
-|   blocked_domains   | A set of domain names to block requests to. Subdomains are also matched (e.g., `"example.com"` blocks `"sub.example.com"` too).                                                                                                     |    ✔️    |
-|     block_ads       | Block requests to ~3,500 known ad/tracking domains. Can be combined with `blocked_domains`.                                                                                                                                         |    ✔️    |
-|   dns_over_https    | Route DNS queries through Cloudflare's DNS-over-HTTPS to prevent DNS leaks when using proxies.                                                                                                                                      |    ✔️    |
-|    proxy_rotator    | A `ProxyRotator` instance for automatic proxy rotation. Cannot be combined with `proxy`.                                                                                                                                            |    ✔️    |
-|       retries       | Number of retry attempts for failed requests. Defaults to 3.                                                                                                                                                                        |    ✔️    |
-|     retry_delay     | Seconds to wait between retry attempts. Defaults to 1.                                                                                                                                                                              |    ✔️    |
-|     capture_xhr     | Pass a regex URL pattern string to capture XHR/fetch requests matching it during page load. Captured responses are available via `response.captured_xhr`. Defaults to `None` (disabled).                                             |    ✔️    |
-|   executable_path   | Absolute path to a custom browser executable to use instead of the bundled Chromium. Useful for non-standard installations or custom browser builds.                                                                                |    ✔️    |
-
-In session classes, all these arguments can be set globally for the session. Still, you can configure each request individually by passing some of the arguments here that can be configured on the browser tab level like: `google_search`, `timeout`, `wait`, `page_action`, `page_setup`, `extra_headers`, `disable_resources`, `wait_selector`, `wait_selector_state`, `network_idle`, `load_dom`, `solve_cloudflare`, `blocked_domains`, `proxy`, and `selector_config`.
-
-**Notes:**
-
-1. It's basically the same arguments as [DynamicFetcher](dynamic.md) class, but with these additional arguments: `solve_cloudflare`, `block_webrtc`, `hide_canvas`, and `allow_webgl`.
-2. The `disable_resources` option made requests ~25% faster in tests for some websites and can help save proxy usage, but be careful with it, as it can cause some websites to never finish loading.
-3. The `google_search` argument is enabled by default for all requests, setting the referer to `https://www.google.com/`. If used together with `extra_headers`, it takes priority over the referer set there.
-4. If you didn't set a user agent and enabled headless mode, the fetcher will generate a real user agent for the same browser version and use it. If you didn't set a user agent and didn't enable headless mode, the fetcher will use the browser's default user agent, which is the same as in standard browsers in the latest versions.
-
-## Examples
-
-### Cloudflare and stealth options
-
-```python
-# Automatic Cloudflare solver
-page = StealthyFetcher.fetch('https://nopecha.com/demo/cloudflare', solve_cloudflare=True)
-
-# Works with other stealth options
-page = StealthyFetcher.fetch(
-    'https://protected-site.com',
-    solve_cloudflare=True,
-    block_webrtc=True,
-    real_chrome=True,
-    hide_canvas=True,
-    google_search=True,
-    proxy='http://username:password@host:port',  # It can also be a dictionary with only the keys 'server', 'username', and 'password'.
-)
-```
-
-The `solve_cloudflare` parameter enables automatic detection and solving all types of Cloudflare's Turnstile/Interstitial challenges:
-
-- JavaScript challenges (managed)
-- Interactive challenges (clicking verification boxes)
-- Invisible challenges (automatic background verification)
-
-And even solves the custom pages with embedded captcha.
-
-**Important notes:**
-
-1. Sometimes, with websites that use custom implementations, you will need to use `wait_selector` to make sure Scrapling waits for the real website content to be loaded after solving the captcha. Some websites can be the real definition of an edge case while we are trying to make the solver as generic as possible.
-2. The timeout should be at least 60 seconds when using the Cloudflare solver for sufficient challenge-solving time.
-3. This feature works seamlessly with proxies and other stealth options.
-
-### Browser Automation
-This is where your knowledge about [Playwright's Page API](https://playwright.dev/python/docs/api/class-page) comes into play. The function you pass here takes the page object from Playwright's API, performs the desired action, and then the fetcher continues.
-
-This function is executed immediately after waiting for `network_idle` (if enabled) and before waiting for the `wait_selector` argument, allowing it to be used for purposes beyond automation. You can alter the page as you want.
-
-In the example below, I used the pages' [mouse events](https://playwright.dev/python/docs/api/class-mouse) to scroll the page with the mouse wheel, then move the mouse.
-```python
-from playwright.sync_api import Page
-
-def scroll_page(page: Page):
-    page.mouse.wheel(10, 0)
-    page.mouse.move(100, 400)
-    page.mouse.up()
-
-page = StealthyFetcher.fetch('https://example.com', page_action=scroll_page)
-```
-Of course, if you use the async fetch version, the function must also be async.
-```python
-from playwright.async_api import Page
-
-async def scroll_page(page: Page):
-   await page.mouse.wheel(10, 0)
-   await page.mouse.move(100, 400)
-   await page.mouse.up()
-
-page = await StealthyFetcher.async_fetch('https://example.com', page_action=scroll_page)
-```
-
-### Wait Conditions
-```python
-# Wait for the selector
-page = StealthyFetcher.fetch(
-    'https://example.com',
-    wait_selector='h1',
-    wait_selector_state='visible'
-)
-```
-This is the last wait the fetcher will do before returning the response (if enabled). You pass a CSS selector to the `wait_selector` argument, and the fetcher will wait for the state you passed in the `wait_selector_state` argument to be fulfilled. If you didn't pass a state, the default would be `attached`, which means it will wait for the element to be present in the DOM.
-
-After that, if `load_dom` is enabled (the default), the fetcher will check again to see if all JavaScript files are loaded and executed (in the `domcontentloaded` state) or continue waiting. If you have enabled `network_idle`, the fetcher will wait for `network_idle` to be fulfilled again, as explained above.
-
-The states the fetcher can wait for can be any of the following ([source](https://playwright.dev/python/docs/api/class-page#page-wait-for-selector)):
-
-- `attached`: Wait for an element to be present in the DOM.
-- `detached`: Wait for an element to not be present in the DOM.
-- `visible`: wait for an element to have a non-empty bounding box and no `visibility:hidden`. Note that an element without any content or with `display:none` has an empty bounding box and is not considered visible.
-- `hidden`: wait for an element to be either detached from the DOM, or have an empty bounding box, or `visibility:hidden`. This is opposite to the `'visible'` option.
-
-
-### Real-world example (Amazon)
-This is for educational purposes only; this example was generated by AI, which also shows how easy it is to work with Scrapling through AI
-```python
-def scrape_amazon_product(url):
-    # Use StealthyFetcher to bypass protection
-    page = StealthyFetcher.fetch(url)
-
-    # Extract product details
-    return {
-        'title': page.css('#productTitle::text').get().clean(),
-        'price': page.css('.a-price .a-offscreen::text').get(),
-        'rating': page.css('[data-feature-name="averageCustomerReviews"] .a-popover-trigger .a-color-base::text').get(),
-        'reviews_count': page.css('#acrCustomerReviewText::text').re_first(r'[\d,]+'),
-        'features': [
-            li.get().clean() for li in page.css('#feature-bullets li span::text')
-        ],
-        'availability': page.css('#availability')[0].get_all_text(strip=True),
-        'images': [
-            img.attrib['src'] for img in page.css('#altImages img')
-        ]
-    }
-```
-
-## Session Management
-
-To keep the browser open until you make multiple requests with the same configuration, use `StealthySession`/`AsyncStealthySession` classes. Those classes can accept all the arguments that the `fetch` function can take, which enables you to specify a config for the entire session.
-
-```python
-from scrapling.fetchers import StealthySession
-
-# Create a session with default configuration
-with StealthySession(
-    headless=True,
-    real_chrome=True,
-    block_webrtc=True,
-    solve_cloudflare=True
-) as session:
-    # Make multiple requests with the same browser instance
-    page1 = session.fetch('https://example1.com')
-    page2 = session.fetch('https://example2.com') 
-    page3 = session.fetch('https://nopecha.com/demo/cloudflare')
-    
-    # All requests reuse the same tab on the same browser instance
-```
-
-### Async Session Usage
-
-```python
-import asyncio
-from scrapling.fetchers import AsyncStealthySession
-
-async def scrape_multiple_sites():
-    async with AsyncStealthySession(
-        real_chrome=True,
-        block_webrtc=True,
-        solve_cloudflare=True,
-        timeout=60000,  # 60 seconds for Cloudflare challenges
-        max_pages=3
-    ) as session:
-        # Make async requests with shared browser configuration
-        pages = await asyncio.gather(
-            session.fetch('https://site1.com'),
-            session.fetch('https://site2.com'), 
-            session.fetch('https://protected-site.com')
-        )
-        return pages
-```
-
-You may have noticed the `max_pages` argument. This is a new argument that enables the fetcher to create a **rotating pool of Browser tabs**. Instead of using a single tab for all your requests, you set a limit on the maximum number of pages that can be displayed at once. With each request, the library will close all tabs that have finished their task and check if the number of the current tabs is lower than the maximum allowed number of pages/tabs, then:
-
-1. If you are within the allowed range, the fetcher will create a new tab for you, and then all is as normal.
-2. Otherwise, it will keep checking every subsecond if creating a new tab is allowed or not for 60 seconds, then raise `TimeoutError`. This can happen when the website you are fetching becomes unresponsive.
-
-This logic allows for multiple URLs to be fetched at the same time in the same browser, which saves a lot of resources, but most importantly, is so fast :)
-
-In versions 0.3 and 0.3.1, the pool was reusing finished tabs to save more resources/time. That logic proved flawed, as it's nearly impossible to protect pages/tabs from contamination by the previous configuration used in the request before this one.
-
-### Session Benefits
-
-- **Browser reuse**: Much faster subsequent requests by reusing the same browser instance.
-- **Cookie persistence**: Automatic cookie and session state handling as any browser does automatically.
-- **Consistent fingerprint**: Same browser fingerprint across all requests.
-- **Memory efficiency**: Better resource usage compared to launching new browsers with each fetch.
-
-## When to Use
-
-Use StealthyFetcher when:
-
-- Bypassing anti-bot protection
-- Need a reliable browser fingerprint
-- Full JavaScript support needed
-- Want automatic stealth features
-- Need browser automation
-- Dealing with Cloudflare protection

package/.agents/skills/scrapling-official/references/mcp-server.md

@@ -1,214 +0,0 @@
-# Scrapling MCP Server
-
-The Scrapling MCP server exposes ten tools over the MCP protocol. It supports CSS-selector-based content narrowing (reducing tokens by extracting only relevant elements before returning results), three levels of scraping capability (plain HTTP, browser-rendered, and stealth/anti-bot bypass), persistent browser session management, and page screenshots returned as real image content blocks.
-
-All scraping tools return a `ResponseModel` with fields: `status` (int), `content` (list of strings), `url` (str). The `screenshot` tool returns a list of MCP content blocks: an `ImageContent` (the screenshot bytes) followed by a `TextContent` (the post-redirect URL).
-
-## Tools
-
-### `get` -- HTTP request (single URL)
-
-Fast HTTP GET with browser fingerprint impersonation (TLS, headers). Suitable for static pages with no/low bot protection.
-
-**Key parameters:**
-
-| Parameter           | Type                               | Default      | Description                                                        |
-|---------------------|------------------------------------|--------------|--------------------------------------------------------------------|
-| `url`               | str                                | required     | URL to fetch                                                       |
-| `extraction_type`   | `"markdown"` / `"html"` / `"text"` | `"markdown"` | Output format                                                      |
-| `css_selector`      | str or null                        | null         | CSS selector to narrow content (applied after `main_content_only`) |
-| `main_content_only` | bool                               | true         | Restrict to `<body>` content                                       |
-| `impersonate`       | str                                | `"chrome"`   | Browser fingerprint to impersonate                                 |
-| `proxy`             | str or null                        | null         | Proxy URL, e.g. `"http://user:pass@host:port"`                     |
-| `proxy_auth`        | dict or null                       | null         | `{"username": "...", "password": "..."}`                           |
-| `auth`              | dict or null                       | null         | HTTP basic auth, same format as proxy_auth                         |
-| `timeout`           | number                             | 30           | Seconds before timeout                                             |
-| `retries`           | int                                | 3            | Retry attempts on failure                                          |
-| `retry_delay`       | int                                | 1            | Seconds between retries                                            |
-| `stealthy_headers`  | bool                               | true         | Generate realistic browser headers and Google referer       |
-| `http3`             | bool                               | false        | Use HTTP/3 (may conflict with `impersonate`)                       |
-| `follow_redirects`  | bool or "safe"                     | "safe"       | Follow redirects. "safe" rejects redirects to internal/private IPs |
-| `max_redirects`     | int                                | 30           | Max redirects (-1 for unlimited)                                   |
-| `headers`           | dict or null                       | null         | Custom request headers                                             |
-| `cookies`           | dict or null                       | null         | Request cookies                                                    |
-| `params`            | dict or null                       | null         | Query string parameters                                            |
-| `verify`            | bool                               | true         | Verify HTTPS certificates                                          |
-
-### `bulk_get` -- HTTP request (multiple URLs)
-
-Async concurrent version of `get`. Same parameters except `url` is replaced by `urls` (list of strings). All URLs are fetched in parallel. Returns a list of `ResponseModel`.
-
-### `fetch` -- Browser fetch (single URL)
-
-Opens a Chromium browser via Playwright to render JavaScript. Suitable for dynamic/SPA pages with no/low bot protection.
-
-**Key parameters (beyond shared ones):**
-
-| Parameter             | Type                | Default      | Description                                                                     |
-|-----------------------|---------------------|--------------|---------------------------------------------------------------------------------|
-| `url`                 | str                 | required     | URL to fetch                                                                    |
-| `extraction_type`     | str                 | `"markdown"` | `"markdown"` / `"html"` / `"text"`                                              |
-| `css_selector`        | str or null         | null         | Narrow content before extraction                                                |
-| `main_content_only`   | bool                | true         | Restrict to `<body>`                                                            |
-| `headless`            | bool                | true         | Run browser hidden (true) or visible (false)                                    |
-| `proxy`               | str or dict or null | null         | String URL or `{"server": "...", "username": "...", "password": "..."}`         |
-| `timeout`             | number              | 30000        | Timeout in **milliseconds**                                                     |
-| `wait`                | number              | 0            | Extra wait (ms) after page load before extraction                               |
-| `wait_selector`       | str or null         | null         | CSS selector to wait for before extraction                                      |
-| `wait_selector_state` | str                 | `"attached"` | State for wait_selector: `"attached"` / `"visible"` / `"hidden"` / `"detached"` |
-| `network_idle`        | bool                | false        | Wait until no network activity for 500ms                                        |
-| `disable_resources`   | bool                | false        | Block fonts, images, media, stylesheets, etc. for speed                         |
-| `google_search`       | bool                | true         | Set a Google referer header                                            |
-| `real_chrome`         | bool                | false        | Use locally installed Chrome instead of bundled Chromium                        |
-| `cdp_url`             | str or null         | null         | Connect to existing browser via CDP URL                                         |
-| `extra_headers`       | dict or null        | null         | Additional request headers                                                      |
-| `useragent`           | str or null         | null         | Custom user-agent (auto-generated if null)                                      |
-| `cookies`             | list or null        | null         | Playwright-format cookies                                                       |
-| `timezone_id`         | str or null         | null         | Browser timezone, e.g. `"America/New_York"`                                     |
-| `locale`              | str or null         | null         | Browser locale, e.g. `"en-GB"`                                                  |
-| `session_id`          | str or null         | null         | Reuse a persistent session from `open_session` instead of creating a new browser |
-
-### `bulk_fetch` -- Browser fetch (multiple URLs)
-
-Concurrent browser version of `fetch`. Same parameters (including `session_id`) except `url` is replaced by `urls` (list of strings). Each URL opens in a separate browser tab. Returns a list of `ResponseModel`.
-
-### `stealthy_fetch` -- Stealth browser fetch (single URL)
-
-Anti-bot bypass fetcher with fingerprint spoofing. Use this for sites with Cloudflare Turnstile/Interstitial or other strong protections.
-
-**Additional parameters (beyond those in `fetch`):**
-
-| Parameter          | Type         | Default | Description                                                      |
-|--------------------|--------------|---------|------------------------------------------------------------------|
-| `solve_cloudflare` | bool         | false   | Automatically solve Cloudflare Turnstile/Interstitial challenges |
-| `hide_canvas`      | bool         | false   | Add noise to canvas operations to prevent fingerprinting         |
-| `block_webrtc`     | bool         | false   | Force WebRTC to respect proxy settings (prevents IP leak)        |
-| `allow_webgl`      | bool         | true    | Keep WebGL enabled (disabling is detectable by WAFs)             |
-| `additional_args`  | dict or null | null    | Extra Playwright context args (overrides Scrapling defaults)     |
-| `session_id`       | str or null  | null    | Reuse a persistent stealthy session from `open_session`          |
-
-All parameters from `fetch` are also accepted.
-
-### `bulk_stealthy_fetch` -- Stealth browser fetch (multiple URLs)
-
-Concurrent stealth version. Same parameters (including `session_id`) as `stealthy_fetch` except `url` is replaced by `urls` (list of strings). Returns a list of `ResponseModel`.
-
-### `open_session` -- Create a persistent browser session
-
-Opens a browser session that stays alive across multiple fetch calls, avoiding the overhead of launching a new browser each time. Returns a `SessionCreatedModel` with `session_id`, `session_type`, `created_at`, `is_alive`, and `message`.
-
-**Key parameters:**
-
-| Parameter          | Type                        | Default      | Description                                                                                           |
-|--------------------|-----------------------------|--------------|-------------------------------------------------------------------------------------------------------|
-| `session_type`     | `"dynamic"` / `"stealthy"`  | required     | Type of browser session to create                                                                     |
-| `session_id`       | str or null                 | null         | Custom ID for the session. If omitted, a random 12-char hex ID is generated. Raises if already in use |
-| `headless`         | bool                        | true         | Run browser hidden or visible                                                                         |
-| `max_pages`        | int                         | 5            | Max concurrent browser tabs (1-50)                                                                    |
-| `proxy`            | str or dict or null         | null         | Proxy for all requests in this session                                                                |
-| `timeout`          | number                      | 30000        | Default timeout in ms                                                                                 |
-| `solve_cloudflare` | bool                        | false        | (Stealthy only) Auto-solve Cloudflare challenges                                                      |
-| `hide_canvas`      | bool                        | false        | (Stealthy only) Canvas fingerprint noise                                                              |
-| `block_webrtc`     | bool                        | false        | (Stealthy only) Block WebRTC IP leak                                                                  |
-| `allow_webgl`      | bool                        | true         | (Stealthy only) Keep WebGL enabled                                                                    |
-
-Plus all other browser session parameters (`google_search`, `real_chrome`, `cdp_url`, `locale`, `timezone_id`, `useragent`, `extra_headers`, `cookies`, `disable_resources`, `network_idle`, `wait_selector`, `wait_selector_state`).
-
-A dynamic session can only be used with `fetch`/`bulk_fetch`. A stealthy session can only be used with `stealthy_fetch`/`bulk_stealthy_fetch`.
-
-### `close_session` -- Close a persistent browser session
-
-Closes a session and frees its browser resources. Always close sessions when done.
-
-| Parameter    | Type | Default  | Description                      |
-|--------------|------|----------|----------------------------------|
-| `session_id` | str  | required | Session ID from `open_session`   |
-
-Returns a `SessionClosedModel` with `session_id` and `message`.
-
-### `list_sessions` -- List active sessions
-
-Returns a list of `SessionInfo` objects, each with `session_id`, `session_type`, `created_at`, and `is_alive`.
-
-No parameters.
-
-### `screenshot` -- Capture a page screenshot
-
-Navigates to a URL inside an existing browser session and returns the screenshot as an MCP `ImageContent` block (the bytes the model can see directly, not a base64 string in JSON) followed by a `TextContent` block carrying the post-redirect URL.
-
-Requires an open browser session. Call `open_session` first, then pass the `session_id` here. Both `dynamic` and `stealthy` sessions are accepted.
-
-| Parameter             | Type                  | Default      | Description                                                                          |
-|-----------------------|-----------------------|--------------|--------------------------------------------------------------------------------------|
-| `url`                 | str                   | required     | URL to navigate to and capture                                                       |
-| `session_id`          | str                   | required     | ID of an open browser session created with `open_session`                            |
-| `image_type`          | `"png"` / `"jpeg"`    | `"png"`      | Image format. Use `"jpeg"` for smaller payloads                                      |
-| `full_page`           | bool                  | false        | Capture the full scrollable page instead of just the viewport                        |
-| `quality`             | int or null           | null         | JPEG quality 0-100. Raises if passed with `image_type="png"`                         |
-| `wait`                | number                | 0            | Extra wait (ms) after page load before capture                                       |
-| `wait_selector`       | str or null           | null         | CSS selector to wait for before capture                                              |
-| `wait_selector_state` | str                   | `"attached"` | State for `wait_selector`: `"attached"` / `"visible"` / `"hidden"` / `"detached"`    |
-| `network_idle`        | bool                  | false        | Wait until no network activity for 500ms                                             |
-| `timeout`             | number                | 30000        | Timeout in milliseconds                                                              |
-
-## Tool selection guide
-
-| Scenario                                 | Tool                                                          |
-|------------------------------------------|---------------------------------------------------------------|
-| Static page, no bot protection           | `get`                                                         |
-| Multiple static pages                    | `bulk_get`                                                    |
-| JavaScript-rendered / SPA page           | `fetch`                                                       |
-| Multiple JS-rendered pages               | `bulk_fetch`                                                  |
-| Cloudflare or strong anti-bot protection | `stealthy_fetch` (with `solve_cloudflare=true` for Turnstile) |
-| Multiple protected pages                 | `bulk_stealthy_fetch`                                         |
-| Multiple pages from the same site        | `open_session` + `fetch`/`stealthy_fetch` with `session_id`  |
-| Need a screenshot of a page              | `open_session` + `screenshot` with `session_id`              |
-
-Start with `get` (fastest, lowest resource cost). Escalate to `fetch` if content requires JS rendering. Escalate to `stealthy_fetch` only if blocked. For multiple pages from the same site, use a persistent session to avoid browser launch overhead.
-
-## Content extraction tips
-
-- Use `css_selector` to narrow results before they reach the model -- this saves significant tokens.
-- `main_content_only=true` (default) strips nav/footer by restricting to `<body>`.
-- `extraction_type="markdown"` (default) is best for readability. Use `"text"` for minimal output, `"html"` when structure matters.
-- If a `css_selector` matches multiple elements, all are returned in the `content` list.
-
-## Prompt injection protection
-
-When `main_content_only=true` (the default), the server automatically sanitizes scraped content to prevent prompt injection from malicious websites. It strips:
-
-- CSS-hidden elements (`display:none`, `visibility:hidden`, `opacity:0`, `font-size:0`, `height:0`, `width:0`)
-- `aria-hidden="true"` elements
-- `<template>` tags
-- HTML comments
-- Zero-width unicode characters
-
-Keep `main_content_only=true` for maximum protection.
-
-## Ad blocking
-
-All browser-based tools (`fetch`, `bulk_fetch`, `stealthy_fetch`, `bulk_stealthy_fetch`) and persistent sessions (`open_session`) automatically block requests to ~3,500 known ad and tracker domains. This is always enabled in the MCP server to save tokens and speed up page loads. No configuration needed.
-
-## Setup
-
-Start the server (stdio transport, used by most MCP clients):
-
-```bash
-scrapling mcp
-```
-
-Or with Streamable HTTP transport:
-
-```bash
-scrapling mcp --http
-scrapling mcp --http --host 127.0.0.1 --port 8000
-```
-
-Docker alternative:
-
-```bash
-docker pull pyd4vinci/scrapling
-docker run -i --rm scrapling mcp
-```
-
-The MCP server name when registering with a client is `ScraplingServer`. The command is the path to the `scrapling` binary and the argument is `mcp`.

package/.agents/skills/scrapling-official/references/migrating_from_beautifulsoup.md

@@ -1,86 +0,0 @@
-# Migrating from BeautifulSoup to Scrapling
-
-API comparison between BeautifulSoup and Scrapling. Scrapling is faster, provides equivalent parsing capabilities, and adds features for fetching and handling modern web pages.
-
-Some BeautifulSoup shortcuts have no direct Scrapling equivalent. Scrapling avoids those shortcuts to preserve performance.
-
-
-| Task                                                            | BeautifulSoup Code                                                                                            | Scrapling Code                                                                    |
-|-----------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------|
-| Parser import                                                   | `from bs4 import BeautifulSoup`                                                                               | `from scrapling.parser import Selector`                                           |
-| Parsing HTML from string                                        | `soup = BeautifulSoup(html, 'html.parser')`                                                                   | `page = Selector(html)`                                                           |
-| Finding a single element                                        | `element = soup.find('div', class_='example')`                                                                | `element = page.find('div', class_='example')`                                    |
-| Finding multiple elements                                       | `elements = soup.find_all('div', class_='example')`                                                           | `elements = page.find_all('div', class_='example')`                               |
-| Finding a single element (Example 2)                            | `element = soup.find('div', attrs={"class": "example"})`                                                      | `element = page.find('div', {"class": "example"})`                                |
-| Finding a single element (Example 3)                            | `element = soup.find(re.compile("^b"))`                                                                       | `element = page.find(re.compile("^b"))`<br/>`element = page.find_by_regex(r"^b")` |
-| Finding a single element (Example 4)                            | `element = soup.find(lambda e: len(list(e.children)) > 0)`                                                    | `element = page.find(lambda e: len(e.children) > 0)`                              |
-| Finding a single element (Example 5)                            | `element = soup.find(["a", "b"])`                                                                             | `element = page.find(["a", "b"])`                                                 |
-| Find element by its text content                                | `element = soup.find(text="some text")`                                                                       | `element = page.find_by_text("some text", partial=False)`                         |
-| Using CSS selectors to find the first matching element          | `elements = soup.select_one('div.example')`                                                                   | `elements = page.css('div.example').first`                                        |
-| Using CSS selectors to find all matching element                | `elements = soup.select('div.example')`                                                                       | `elements = page.css('div.example')`                                              |
-| Get a prettified version of the page/element source             | `prettified = soup.prettify()`                                                                                | `prettified = page.prettify()`                                                    |
-| Get a Non-pretty version of the page/element source             | `source = str(soup)`                                                                                          | `source = page.html_content`                                                      |
-| Get tag name of an element                                      | `name = element.name`                                                                                         | `name = element.tag`                                                              |
-| Extracting text content of an element                           | `string = element.string`                                                                                     | `string = element.text`                                                           |
-| Extracting all the text in a document or beneath a tag          | `text = soup.get_text(strip=True)`                                                                            | `text = page.get_all_text(strip=True)`                                            |
-| Access the dictionary of attributes                             | `attrs = element.attrs`                                                                                       | `attrs = element.attrib`                                                          |
-| Extracting attributes                                           | `attr = element['href']`                                                                                      | `attr = element['href']`                                                          |
-| Navigating to parent                                            | `parent = element.parent`                                                                                     | `parent = element.parent`                                                         |
-| Get all parents of an element                                   | `parents = list(element.parents)`                                                                             | `parents = list(element.iterancestors())`                                         |
-| Searching for an element in the parents of an element           | `target_parent = element.find_parent("a")`                                                                    | `target_parent = element.find_ancestor(lambda p: p.tag == 'a')`                   |
-| Get all siblings of an element                                  | N/A                                                                                                           | `siblings = element.siblings`                                                     |
-| Get next sibling of an element                                  | `next_element = element.next_sibling`                                                                         | `next_element = element.next`                                                     |
-| Searching for an element in the siblings of an element          | `target_sibling = element.find_next_sibling("a")`<br/>`target_sibling = element.find_previous_sibling("a")`   | `target_sibling = element.siblings.search(lambda s: s.tag == 'a')`                |
-| Searching for elements in the siblings of an element            | `target_sibling = element.find_next_siblings("a")`<br/>`target_sibling = element.find_previous_siblings("a")` | `target_sibling = element.siblings.filter(lambda s: s.tag == 'a')`                |
-| Searching for an element in the next elements of an element     | `target_parent = element.find_next("a")`                                                                      | `target_parent = element.below_elements.search(lambda p: p.tag == 'a')`           |
-| Searching for elements in the next elements of an element       | `target_parent = element.find_all_next("a")`                                                                  | `target_parent = element.below_elements.filter(lambda p: p.tag == 'a')`           |
-| Searching for an element in the ancestors of an element         | `target_parent = element.find_previous("a")` ¹                                                                | `target_parent = element.path.search(lambda p: p.tag == 'a')`                     |
-| Searching for elements in the ancestors of an element           | `target_parent = element.find_all_previous("a")` ¹                                                            | `target_parent = element.path.filter(lambda p: p.tag == 'a')`                     |
-| Get previous sibling of an element                              | `prev_element = element.previous_sibling`                                                                     | `prev_element = element.previous`                                                 |
-| Navigating to children                                          | `children = list(element.children)`                                                                           | `children = element.children`                                                     |
-| Get all descendants of an element                               | `children = list(element.descendants)`                                                                        | `children = element.below_elements`                                               |
-| Filtering a group of elements that satisfies a condition        | `group = soup.find('p', 'story').css.filter('a')`                                                             | `group = page.find_all('p', 'story').filter(lambda p: p.tag == 'a')`              |
-
-
-¹ **Note:** BS4's `find_previous`/`find_all_previous` searches all preceding elements in document order, while Scrapling's `path` only returns ancestors (the parent chain). These are not exact equivalents, but ancestor search covers the most common use case.
-
-BeautifulSoup supports modifying/manipulating the parsed DOM. Scrapling does not - it is read-only and optimized for extraction.
-
-### Full Example: Extracting Links
-
-**With BeautifulSoup:**
-
-```python
-import requests
-from bs4 import BeautifulSoup
-
-url = 'https://example.com'
-response = requests.get(url)
-soup = BeautifulSoup(response.text, 'html.parser')
-
-links = soup.find_all('a')
-for link in links:
-    print(link['href'])
-```
-
-**With Scrapling:**
-
-```python
-from scrapling import Fetcher
-
-url = 'https://example.com'
-page = Fetcher.get(url)
-
-links = page.css('a::attr(href)')
-for link in links:
-    print(link)
-```
-
-Scrapling combines fetching and parsing into a single step.
-
-**Note:**
-
-- **Parsers**: BeautifulSoup supports multiple parser engines. Scrapling always uses `lxml` for performance.
-- **Element Types**: BeautifulSoup elements are `Tag` objects; Scrapling elements are `Selector` objects. Both provide similar navigation and extraction methods.
-- **Error Handling**: Both libraries return `None` when an element is not found (e.g., `soup.find()` or `page.find()`). `page.css()` returns an empty `Selectors` list when no elements match. Use `page.css('.foo').first` to safely get the first match or `None`.
-- **Text Extraction**: Scrapling's `TextHandler` provides additional text processing methods such as `clean()` for removing extra whitespace, consecutive spaces, or unwanted characters.