ultimate-pi 0.1.2 → 0.1.4

package/.agents/skills/scrapling-official/references/spiders/proxy-blocking.md

@@ -1,235 +0,0 @@
-# Proxy management and handling Blocks
-
-Scrapling's `ProxyRotator` manages proxy rotation across requests. It works with all session types and integrates with the spider's blocked request retry system.
-
-## ProxyRotator
-
-The `ProxyRotator` class manages a list of proxies and rotates through them automatically. Pass it to any session type via the `proxy_rotator` parameter:
-
-```python
-from scrapling.spiders import Spider, Response
-from scrapling.fetchers import FetcherSession, ProxyRotator
-
-class MySpider(Spider):
-    name = "my_spider"
-    start_urls = ["https://example.com"]
-
-    def configure_sessions(self, manager):
-        rotator = ProxyRotator([
-            "http://proxy1:8080",
-            "http://proxy2:8080",
-            "http://user:pass@proxy3:8080",
-        ])
-        manager.add("default", FetcherSession(proxy_rotator=rotator))
-
-    async def parse(self, response: Response):
-        # Check which proxy was used
-        print(f"Proxy used: {response.meta.get('proxy')}")
-        yield {"title": response.css("title::text").get("")}
-```
-
-Each request automatically gets the next proxy in the rotation. The proxy used is stored in `response.meta["proxy"]` so you can track which proxy fetched which page.
-
-
-Browser sessions support both string and dict proxy formats:
-
-```python
-from scrapling.fetchers import AsyncDynamicSession, AsyncStealthySession, ProxyRotator
-
-# String proxies work for all session types
-rotator = ProxyRotator([
-    "http://proxy1:8080",
-    "http://proxy2:8080",
-])
-
-# Dict proxies (Playwright format) work for browser sessions
-rotator = ProxyRotator([
-    {"server": "http://proxy1:8080", "username": "user", "password": "pass"},
-    {"server": "http://proxy2:8080"},
-])
-
-# Then inside the spider
-def configure_sessions(self, manager):
-    rotator = ProxyRotator(["http://proxy1:8080", "http://proxy2:8080"])
-    manager.add("browser", AsyncStealthySession(proxy_rotator=rotator))
-```
-
-**Important:**
-
-1. You cannot use the `proxy_rotator` argument together with the static `proxy` or `proxies` parameters on the same session. Pick one approach when configuring the session, and override it per request later if needed.
-2. By default, all browser-based sessions use a persistent browser context with a pool of tabs. However, since browsers can't set a proxy per tab, when you use a `ProxyRotator`, the fetcher will automatically open a separate context for each proxy, with one tab per context. Once the tab's job is done, both the tab and its context are closed.
-
-## Custom Rotation Strategies
-
-By default, `ProxyRotator` uses cyclic rotation - it iterates through proxies sequentially, wrapping around at the end.
-
-You can provide a custom strategy function to change this behavior, but it has to match the below signature:
-
-```python
-from scrapling.core._types import ProxyType
-
-def my_strategy(proxies: list, current_index: int) -> tuple[ProxyType, int]:
-    ...
-```
-
-It receives the list of proxies and the current index, and must return the chosen proxy and the next index.
-
-Below are some examples of custom rotation strategies you can use.
-
-### Random Rotation
-
-```python
-import random
-from scrapling.fetchers import ProxyRotator
-
-def random_strategy(proxies, current_index):
-    idx = random.randint(0, len(proxies) - 1)
-    return proxies[idx], idx
-
-rotator = ProxyRotator(
-    ["http://proxy1:8080", "http://proxy2:8080", "http://proxy3:8080"],
-    strategy=random_strategy,
-)
-```
-
-### Weighted Rotation
-
-```python
-import random
-
-def weighted_strategy(proxies, current_index):
-    # First proxy gets 60% of traffic, others split the rest
-    weights = [60] + [40 // (len(proxies) - 1)] * (len(proxies) - 1)
-    proxy = random.choices(proxies, weights=weights, k=1)[0]
-    return proxy, current_index  # Index doesn't matter for weighted
-
-rotator = ProxyRotator(proxies, strategy=weighted_strategy)
-```
-
-
-## Per-Request Proxy Override
-
-You can override the rotator for individual requests by passing `proxy=` as a keyword argument:
-
-```python
-async def parse(self, response: Response):
-    # This request uses the rotator's next proxy
-    yield response.follow("/page1", callback=self.parse_page)
-
-    # This request uses a specific proxy, bypassing the rotator
-    yield response.follow(
-        "/special-page",
-        callback=self.parse_page,
-        proxy="http://special-proxy:8080",
-    )
-```
-
-This is useful when certain pages require a specific proxy (e.g., a geo-located proxy for region-specific content).
-
-## Blocked Request Handling
-
-The spider has built-in blocked request detection and retry. By default, it considers the following HTTP status codes blocked: `401`, `403`, `407`, `429`, `444`, `500`, `502`, `503`, `504`.
-
-The retry system works like this:
-
-1. After a response comes back, the spider calls the `is_blocked(response)` method.
-2. If blocked, it copies the request and calls the `retry_blocked_request()` method so you can modify it before retrying.
-3. The retried request is re-queued with `dont_filter=True` (bypassing deduplication) and lower priority, so it's not retried right away.
-4. This repeats up to `max_blocked_retries` times (default: 3).
-
-**Tip:**
-
-1. On retry, the previous `proxy`/`proxies` kwargs are cleared from the request automatically, so the rotator assigns a fresh proxy.
-2. The `max_blocked_retries` attribute is different than the session retries and doesn't share the counter.
-
-### Custom Block Detection
-
-Override `is_blocked()` to add your own detection logic:
-
-```python
-class MySpider(Spider):
-    name = "my_spider"
-    start_urls = ["https://example.com"]
-
-    async def is_blocked(self, response: Response) -> bool:
-        # Check status codes (default behavior)
-        if response.status in {403, 429, 503}:
-            return True
-
-        # Check response content
-        body = response.body.decode("utf-8", errors="ignore")
-        if "access denied" in body.lower() or "rate limit" in body.lower():
-            return True
-
-        return False
-
-    async def parse(self, response: Response):
-        yield {"title": response.css("title::text").get("")}
-```
-
-### Customizing Retries
-
-Override `retry_blocked_request()` to modify the request before retrying. The `max_blocked_retries` attribute controls how many times a blocked request is retried (default: 3):
-
-```python
-from scrapling.spiders import Spider, SessionManager, Request, Response
-from scrapling.fetchers import FetcherSession, AsyncStealthySession
-
-
-class MySpider(Spider):
-    name = "my_spider"
-    start_urls = ["https://example.com"]
-    max_blocked_retries = 5
-
-    def configure_sessions(self, manager: SessionManager) -> None:
-        manager.add('requests', FetcherSession(impersonate=['chrome', 'firefox', 'safari']))
-        manager.add('stealth', AsyncStealthySession(block_webrtc=True), lazy=True)
-
-    async def retry_blocked_request(self, request: Request, response: Response) -> Request:
-        request.sid = "stealth"
-        self.logger.info(f"Retrying blocked request: {request.url}")
-        return request
-
-    async def parse(self, response: Response):
-        yield {"title": response.css("title::text").get("")}
-```
-
-What happened above is that I left the blocking detection logic unchanged and had the spider mainly use requests until it got blocked, then switch to the stealthy browser.
-
-
-Putting it all together:
-
-```python
-from scrapling.spiders import Spider, SessionManager, Request, Response
-from scrapling.fetchers import FetcherSession, AsyncStealthySession, ProxyRotator
-
-
-cheap_proxies = ProxyRotator([ "http://proxy1:8080", "http://proxy2:8080"])
-
-# A format acceptable by the browser
-expensive_proxies = ProxyRotator([
-    {"server": "http://residential_proxy1:8080", "username": "user", "password": "pass"},
-    {"server": "http://residential_proxy2:8080", "username": "user", "password": "pass"},
-    {"server": "http://mobile_proxy1:8080", "username": "user", "password": "pass"},
-    {"server": "http://mobile_proxy2:8080", "username": "user", "password": "pass"},
-])
-
-
-class MySpider(Spider):
-    name = "my_spider"
-    start_urls = ["https://example.com"]
-    max_blocked_retries = 5
-
-    def configure_sessions(self, manager: SessionManager) -> None:
-        manager.add('requests', FetcherSession(impersonate=['chrome', 'firefox', 'safari'], proxy_rotator=cheap_proxies))
-        manager.add('stealth', AsyncStealthySession(block_webrtc=True, proxy_rotator=expensive_proxies), lazy=True)
-
-    async def retry_blocked_request(self, request: Request, response: Response) -> Request:
-        request.sid = "stealth"
-        self.logger.info(f"Retrying blocked request: {request.url}")
-        return request
-
-    async def parse(self, response: Response):
-        yield {"title": response.css("title::text").get("")}
-```
-The above logic is: requests are made with cheap proxies, such as datacenter proxies, until they are blocked, then retried with higher-quality proxies, such as residential or mobile proxies.

package/.agents/skills/scrapling-official/references/spiders/requests-responses.md

@@ -1,196 +0,0 @@
-# Requests & Responses
-
-This page covers the `Request` object in detail: how to construct requests, pass data between callbacks, control priority and deduplication, and use `response.follow()` for link-following.
-
-## The Request Object
-
-A `Request` represents a URL to be fetched. You create requests either directly or via `response.follow()`:
-
-```python
-from scrapling.spiders import Request
-
-# Direct construction
-request = Request(
-    "https://example.com/page",
-    callback=self.parse_page,
-    priority=5,
-)
-
-# Via response.follow (preferred in callbacks)
-request = response.follow("/page", callback=self.parse_page)
-```
-
-Here are all the arguments you can pass to `Request`:
-
-| Argument      | Type       | Default    | Description                                                                                           |
-|---------------|------------|------------|-------------------------------------------------------------------------------------------------------|
-| `url`         | `str`      | *required* | The URL to fetch                                                                                      |
-| `sid`         | `str`      | `""`       | Session ID - routes the request to a specific session (see [Sessions](sessions.md))                   |
-| `callback`    | `callable` | `None`     | Async generator method to process the response. Defaults to `parse()`                                 |
-| `priority`    | `int`      | `0`        | Higher values are processed first                                                                     |
-| `dont_filter` | `bool`     | `False`    | If `True`, skip deduplication (allow duplicate requests)                                              |
-| `meta`        | `dict`     | `{}`       | Arbitrary metadata passed through to the response                                                     |
-| `**kwargs`    |            |            | Additional keyword arguments passed to the session's fetch method (e.g., `headers`, `method`, `data`) |
-
-Any extra keyword arguments are forwarded directly to the underlying session. For example, to make a POST request:
-
-```python
-yield Request(
-    "https://example.com/api",
-    method="POST",
-    data={"key": "value"},
-    callback=self.parse_result,
-)
-```
-
-## Response.follow()
-
-`response.follow()` is the recommended way to create follow-up requests inside callbacks. It offers several advantages over constructing `Request` objects directly:
-
-- **Relative URLs** are resolved automatically against the current page URL
-- **Referer header** is set to the current page URL by default
-- **Session kwargs** from the original request are inherited (headers, proxy settings, etc.)
-- **Callback, session ID, and priority** are inherited from the original request if not specified
-
-```python
-async def parse(self, response: Response):
-    # Minimal - inherits callback, sid, priority from current request
-    yield response.follow("/next-page")
-
-    # Override specific fields
-    yield response.follow(
-        "/product/123",
-        callback=self.parse_product,
-        priority=10,
-    )
-
-    # Pass additional metadata to
-    yield response.follow(
-        "/details",
-        callback=self.parse_details,
-        meta={"category": "electronics"},
-    )
-```
-
-| Argument           | Type       | Default    | Description                                                |
-|--------------------|------------|------------|------------------------------------------------------------|
-| `url`              | `str`      | *required* | URL to follow (absolute or relative)                       |
-| `sid`              | `str`      | `""`       | Session ID (inherits from original request if empty)       |
-| `callback`         | `callable` | `None`     | Callback method (inherits from original request if `None`) |
-| `priority`         | `int`      | `None`     | Priority (inherits from original request if `None`)        |
-| `dont_filter`      | `bool`     | `False`    | Skip deduplication                                         |
-| `meta`             | `dict`     | `None`     | Metadata (merged with existing response meta)              |
-| **`referer_flow`** | `bool`     | `True`     | Set current URL as Referer header                          |
-| `**kwargs`         |            |            | Merged with original request's session kwargs              |
-
-### Disabling Referer Flow
-
-By default, `response.follow()` sets the `Referer` header to the current page URL. To disable this:
-
-```python
-yield response.follow("/page", referer_flow=False)
-```
-
-## Callbacks
-
-Callbacks are async generator methods on your spider that process responses. They must `yield` one of three types:
-
-- **`dict`**: A scraped item, added to the results
-- **`Request`**: A follow-up request, added to the queue
-- **`None`**: Silently ignored
-
-```python
-class MySpider(Spider):
-    name = "my_spider"
-    start_urls = ["https://example.com"]
-
-    async def parse(self, response: Response):
-        # Yield items (dicts)
-        yield {"url": response.url, "title": response.css("title::text").get("")}
-
-        # Yield follow-up requests
-        for link in response.css("a::attr(href)").getall():
-            yield response.follow(link, callback=self.parse_page)
-
-    async def parse_page(self, response: Response):
-        yield {"content": response.css("article::text").get("")}
-```
-
-**Note:** All callback methods must be `async def` and use `yield` (not `return`). Even if a callback only yields items with no follow-up requests, it must still be an async generator.
-
-## Request Priority
-
-Requests with higher priority values are processed first. This is useful when some pages are more important to be processed first before others:
-
-```python
-async def parse(self, response: Response):
-    # High priority - process product pages first
-    for link in response.css("a.product::attr(href)").getall():
-        yield response.follow(link, callback=self.parse_product, priority=10)
-
-    # Low priority - pagination links processed after products
-    next_page = response.css("a.next::attr(href)").get()
-    if next_page:
-        yield response.follow(next_page, callback=self.parse, priority=0)
-```
-
-When using `response.follow()`, the priority is inherited from the original request unless you specify a new one.
-
-## Deduplication
-
-The spider automatically deduplicates requests based on a fingerprint computed from the URL, HTTP method, request body, and session ID. If two requests produce the same fingerprint, the second one is silently dropped.
-
-To allow duplicate requests (e.g., re-visiting a page after login), set `dont_filter=True`:
-
-```python
-yield Request("https://example.com/dashboard", dont_filter=True, callback=self.parse_dashboard)
-
-# Or with response.follow
-yield response.follow("/dashboard", dont_filter=True, callback=self.parse_dashboard)
-```
-
-You can fine-tune what goes into the fingerprint using class attributes on your spider:
-
-| Attribute            | Default | Effect                                                                                                          |
-|----------------------|---------|-----------------------------------------------------------------------------------------------------------------|
-| `fp_include_kwargs`  | `False` | Include extra request kwargs (arguments you passed to the session fetch, like headers, etc.) in the fingerprint |
-| `fp_keep_fragments`  | `False` | Keep URL fragments (`#section`) when computing fingerprints                                                     |
-| `fp_include_headers` | `False` | Include request headers in the fingerprint                                                                      |
-
-For example, if you need to treat `https://example.com/page#section1` and `https://example.com/page#section2` as different URLs:
-
-```python
-class MySpider(Spider):
-    name = "my_spider"
-    fp_keep_fragments = True
-    # ...
-```
-
-## Request Meta
-
-The `meta` dictionary lets you pass arbitrary data between callbacks. This is useful when you need context from one page to process another:
-
-```python
-async def parse(self, response: Response):
-    for product in response.css("div.product"):
-        category = product.css("span.category::text").get("")
-        link = product.css("a::attr(href)").get()
-        if link:
-            yield response.follow(
-                link,
-                callback=self.parse_product,
-                meta={"category": category},
-            )
-
-async def parse_product(self, response: Response):
-    yield {
-        "name": response.css("h1::text").get(""),
-        "price": response.css(".price::text").get(""),
-        # Access meta from the request
-        "category": response.meta.get("category", ""),
-    }
-```
-
-When using `response.follow()`, the meta from the current response is merged with the new meta you provide (new values take precedence).
-
-The spider system also automatically stores some metadata. For example, the proxy used for a request is available as `response.meta["proxy"]` when proxy rotation is enabled.

package/.agents/skills/scrapling-official/references/spiders/sessions.md

@@ -1,205 +0,0 @@
-# Spiders sessions
-
-A spider can use multiple fetcher sessions simultaneously. For example, a fast HTTP session for simple pages and a stealth browser session for protected pages.
-
-## What are Sessions?
-
-A session is a pre-configured fetcher instance that stays alive for the duration of the crawl. Instead of creating a new connection or browser for every request, the spider reuses sessions, which is faster and more resource-efficient.
-
-By default, every spider creates a single [FetcherSession](../fetching/static.md). You can add more sessions or swap the default by overriding the `configure_sessions()` method, but you have to use the async version of each session only, as the table shows below:
-
-
-| Session Type                                    | Use Case                                 |
-|-------------------------------------------------|------------------------------------------|
-| [FetcherSession](../fetching/static.md)         | Fast HTTP requests, no JavaScript        |
-| [AsyncDynamicSession](../fetching/dynamic.md)   | Browser automation, JavaScript rendering |
-| [AsyncStealthySession](../fetching/stealthy.md) | Anti-bot bypass, Cloudflare, etc.        |
-
-
-## Configuring Sessions
-
-Override `configure_sessions()` on your spider to set up sessions. The `manager` parameter is a `SessionManager` instance - use `manager.add()` to register sessions:
-
-```python
-from scrapling.spiders import Spider, Response
-from scrapling.fetchers import FetcherSession
-
-class MySpider(Spider):
-    name = "my_spider"
-    start_urls = ["https://example.com"]
-
-    def configure_sessions(self, manager):
-        manager.add("default", FetcherSession())
-
-    async def parse(self, response: Response):
-        yield {"title": response.css("title::text").get("")}
-```
-
-The `manager.add()` method takes:
-
-| Argument     | Type      | Default    | Description                                  |
-|--------------|-----------|------------|----------------------------------------------|
-| `session_id` | `str`     | *required* | A name to reference this session in requests |
-| `session`    | `Session` | *required* | The session instance                         |
-| `default`    | `bool`    | `False`    | Make this the default session                |
-| `lazy`       | `bool`    | `False`    | Start the session only when first used       |
-
-**Notes:**
-
-1. In all requests, if you don't specify which session to use, the default session is used. The default session is determined in one of two ways:
-    1. The first session you add to the manager becomes the default automatically.
-    2. The session that gets `default=True` while added to the manager.
-2. The instances you pass of each session don't have to be already started by you; the spider checks on all sessions if they are not already started and starts them.
-3. If you want a specific session to start when used only, then use the `lazy` argument while adding that session to the manager. Example: start the browser only when you need it, not with the spider start.
-
-## Multi-Session Spider
-
-Here's a practical example: use a fast HTTP session for listing pages and a stealth browser for detail pages that have bot protection:
-
-```python
-from scrapling.spiders import Spider, Response
-from scrapling.fetchers import FetcherSession, AsyncStealthySession
-
-class ProductSpider(Spider):
-    name = "products"
-    start_urls = ["https://shop.example.com/products"]
-
-    def configure_sessions(self, manager):
-        # Fast HTTP for listing pages (default)
-        manager.add("http", FetcherSession())
-
-        # Stealth browser for protected product pages
-        manager.add("stealth", AsyncStealthySession(
-            headless=True,
-            network_idle=True,
-        ))
-
-    async def parse(self, response: Response):
-        for link in response.css("a.product::attr(href)").getall():
-            # Route product pages through the stealth session
-            yield response.follow(link, sid="stealth", callback=self.parse_product)
-
-        next_page = response.css("a.next::attr(href)").get()
-        if next_page:
-            yield response.follow(next_page)
-
-    async def parse_product(self, response: Response):
-        yield {
-            "name": response.css("h1::text").get(""),
-            "price": response.css(".price::text").get(""),
-        }
-```
-
-The key is the `sid` parameter - it tells the spider which session to use for each request. When you call `response.follow()` without `sid`, the session ID from the original request is inherited.
-
-Sessions can also be different instances of the same class with different configurations:
-
-```python
-from scrapling.spiders import Spider, Response
-from scrapling.fetchers import FetcherSession
-
-class ProductSpider(Spider):
-    name = "products"
-    start_urls = ["https://shop.example.com/products"]
-
-    def configure_sessions(self, manager):
-        chrome_requests = FetcherSession(impersonate="chrome")
-        firefox_requests = FetcherSession(impersonate="firefox")
-
-        manager.add("chrome", chrome_requests)
-        manager.add("firefox", firefox_requests)
-
-    async def parse(self, response: Response):
-        for link in response.css("a.product::attr(href)").getall():
-            yield response.follow(link, callback=self.parse_product)
-
-        next_page = response.css("a.next::attr(href)").get()
-        if next_page:
-            yield response.follow(next_page, sid="firefox")
-
-    async def parse_product(self, response: Response):
-        yield {
-            "name": response.css("h1::text").get(""),
-            "price": response.css(".price::text").get(""),
-        }
-```
-
-## Session Arguments
-
-Extra keyword arguments passed to a `Request` (or through `response.follow(**kwargs)`) are forwarded to the session's fetch method. This lets you customize individual requests without changing the session configuration:
-
-```python
-async def parse(self, response: Response):
-    # Pass extra headers for this specific request
-    yield Request(
-        "https://api.example.com/data",
-        headers={"Authorization": "Bearer token123"},
-        callback=self.parse_api,
-    )
-
-    # Use a different HTTP method
-    yield Request(
-        "https://example.com/submit",
-        method="POST",
-        data={"field": "value"},
-        sid="firefox",
-        callback=self.parse_result,
-    )
-```
-
-**Warning:** When using `FetcherSession` in spiders, you cannot use `.get()` and `.post()` methods directly. By default, the request is an HTTP GET request; to use another HTTP method, pass it to the `method` argument as in the above example. This unifies the `Request` interface across all session types.
-
-For browser sessions (`AsyncDynamicSession`, `AsyncStealthySession`), you can pass browser-specific arguments like `wait_selector`, `page_action`, or `extra_headers`:
-
-```python
-async def parse(self, response: Response):
-    # Use Cloudflare solver with the `AsyncStealthySession` we configured above
-    yield Request(
-        "https://nopecha.com/demo/cloudflare",
-        sid="stealth",
-        callback=self.parse_result,
-        solve_cloudflare=True,
-        block_webrtc=True,
-        hide_canvas=True,
-        google_search=True,
-    )
-
-    yield response.follow(
-        "/dynamic-page",
-        sid="browser",
-        callback=self.parse_dynamic,
-        wait_selector="div.loaded",
-        network_idle=True,
-    )
-```
-
-**Warning:** Session arguments (**kwargs) passed from the original request are inherited by `response.follow()`. New kwargs take precedence over inherited ones.
-
-```python
-from scrapling.spiders import Spider, Response
-from scrapling.fetchers import FetcherSession
-
-class ProductSpider(Spider):
-    name = "products"
-    start_urls = ["https://shop.example.com/products"]
-
-    def configure_sessions(self, manager):
-        manager.add("http", FetcherSession(impersonate='chrome'))
-
-    async def parse(self, response: Response):
-        # I don't want the follow request to impersonate a desktop Chrome like the previous request, but a mobile one
-        # so I override it like this
-        for link in response.css("a.product::attr(href)").getall():
-            yield response.follow(link, impersonate="chrome131_android", callback=self.parse_product)
-
-        next_page = response.css("a.next::attr(href)").get()
-        if next_page:
-            yield Request(next_page)
-
-    async def parse_product(self, response: Response):
-        yield {
-            "name": response.css("h1::text").get(""),
-            "price": response.css(".price::text").get(""),
-        }
-```
-**Note:** Upon spider closure, the manager automatically checks whether any sessions are still running and closes them before closing the spider.

package/PLAN.md

@@ -1,11 +0,0 @@
-# Plan
-
-## Harness execution model improvements
-1. Make harness mimic enterprise software engineering team execution.
-2. Require project wiki creation at project start.
-3. Require every design decision to be documented in wiki with rationale.
-4. Before code changes, require referencing relevant wiki design decisions/guidelines to maintain continuity.
-5. Reference https://handbook.gitlab.com/handbook/engineering/ while building the harness architecture.
-
-## Tracking note
-- New execution-model requirements are now tracked here for implementation.