google-browser-scraper 0.0.3__tar.gz → 0.0.4__tar.gz

{google_browser_scraper-0.0.3/src/google_browser_scraper.egg-info → google_browser_scraper-0.0.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: google-browser-scraper
-Version: 0.0.3
+Version: 0.0.4
 Summary: Google Search results from a real browser through your own sticky proxy, with captchas and blocks reported explicitly.
 Author: NodeMaven
 License: MIT
@@ -29,6 +29,10 @@ Requires-Dist: pytest>=8; extra == "dev"
 Requires-Dist: ruff>=0.6; extra == "dev"
 Dynamic: license-file
 
+<p align="center">
+  <img src="https://raw.githubusercontent.com/nodemaven/google-browser-scraper/main/assets/banner.png" alt="Google Browser Scraper">
+</p>
+
 # google-browser-scraper
 
 Google Search results from a real browser, through your own sticky proxy.
@@ -36,9 +40,15 @@ Google Search results from a real browser, through your own sticky proxy.
 Plain HTTP is no longer a reliable way to scrape Google: it often expects a
 browser that runs JavaScript, and a fresh browser on a fresh proxy exit can be
 refused quickly. This tool takes a browser path instead. It warms a sticky proxy
-session with ordinary browsing, types the query into Google's search box, checks
-whether Google served results or refused, and writes one JSON record per results
-page. Captchas, blocks and failures are reported as such, never as "no results".
+session with ordinary browsing, opens Google's results page for each query in
+that browser, checks whether Google served results or refused, and writes one
+JSON record per results page. Captchas, blocks and failures are reported as such, never as "no results".
+
+**Run it on Windows** - a desktop or a Windows VPS. That is where it is tested
+and where Google serves it. On a Linux server without a GPU, Google refused
+almost every search in testing; Chrome has no WebGL there, which is the main
+suspect, so Linux and Docker are experimental for now. `google-browser-scraper
+doctor` shows what your machine looks like to a website.
 
 ## Install
 
@@ -74,8 +84,8 @@ Success is a `results.jsonl` with one JSON object per results page:
 
 The first query takes a few minutes, because a new exit is warmed up first, and
 the warm-up costs noticeably more traffic than a search. Later queries on the
-same exit take seconds. A summary of pages served and traffic used is printed at
-the end of every run.
+same exit take seconds. A summary of pages served and traffic used, with the
+hosts that took most of it, is printed at the end of every run.
 
 With NodeMaven, the session id is handled for you and the credentials come from
 the environment:
@@ -116,7 +126,8 @@ shopping, images and other special blocks are not parsed.
 
 Each record also has an `exit` block: which proxy session answered and how many
 bytes the page cost. `search_metadata.timings` says where the page's time went,
-in seconds per step (`type`, `load`, `ready`, `read`, `parse`, `links`). A query
+in seconds per step (`type`, `load`, `ready`, `read`, `parse`, and `links`, which
+overlaps the next query and is not part of `total_time_taken`). A query
 that failed is a record too, with an `error`.
 `--format csv` writes the organic results as a table, and `--price-per-gb` adds
 the cost per 1000 pages to the summary.
@@ -124,8 +135,12 @@ the cost per 1000 pages to the summary.
 ## How it works
 
 1. **New identity**: a sticky proxy session and a fresh browser profile.
-2. **Warm-up**: a few ordinary pages, then Google's front page.
-3. **Search**: the query is typed into the box and submitted.
+2. **Warm-up**: a few ordinary pages.
+3. **Search**: each query opens Google's results page directly, which is
+   faster than the search box and skips the autocomplete traffic of typing.
+   `--direct follow-ups` sends an exit's first query through the box and
+   `--direct none` every query; `--input` sets how a query goes into the box
+   (`type` key by key, `insert` all at once, `words` word by word).
 4. **Check before parsing**: the page is classified first, so a captcha or a
    block is reported as one and never parsed as an empty result.
 5. **Keep or drop**: an exit Google answers is reused for the next queries; an
@@ -134,20 +149,22 @@ the cost per 1000 pages to the summary.
 
 A served exit is kept for up to 50 queries (`--max-per-exit`). Every new exit
 pays for a warm-up, so a short cap mostly buys more warm-up traffic. A browser
-error on a search is retried once from Google's front page before the exit is
-dropped; a network error drops it at once.
+error on a search is retried once on the same exit (through Google's front page
+when queries go through the search box) before the exit is dropped; a network
+error drops it at once.
 
 `--workers N` searches on N exits at once, one browser each, sharing one list
-of queries. Each browser needs memory and CPU of its own; if exit after exit is
-refused on any worker, the whole run stops.
+of queries. Each browser needs memory and CPU of its own. The run stops when
+exit after exit is refused or unreachable with no worker served in between.
 
 The command line copies the cookies of the first exit Google serves and
 starts up to 10 new exits from that copy (`--clone N`, `--clone 0` turns it
-off), on new sticky sessions, instead of warming each one. Google's acceptance follows the cookies more than the
-address, so a copy usually starts out served, and costs a few MB instead of a
-full warm-up. A copy refused on its first query ends the copying; the copied
-cookies are used for an hour at most (`--clone-max-age`). A copy waits 40-60
-seconds after its browser starts before the first search.
+off), on new sticky sessions, instead of warming each one; all workers share
+the copy. Google's acceptance follows the cookies more than the address, so a
+copy usually starts out served, and costs a few MB instead of a full warm-up.
+A copy refused on its first query ends the copying; the copied cookies are
+used for three hours at most (`--clone-max-age`). A copy waits 40-60 seconds
+after its browser starts before the first search.
 
 `--spare 1` warms the next exit in the background once the current one is
 served, so a retired exit is replaced without waiting for a warm-up. A spare
@@ -155,8 +172,9 @@ costs a warm-up's traffic whether it is used or not.
 
 Google wraps result links in encrypted `/goto` redirects. Each one is resolved
 with a small request through a separate short-lived proxy session, a few at a
-time, so link resolution does not touch the warmed browser session.
-`--no-resolve-links` skips it.
+time, so link resolution does not touch the warmed browser session. It runs in
+the background while the browser goes on to the next query; records still come
+out in the order each browser searched them. `--no-resolve-links` skips it.
 
 Traffic goes through a small local relay that counts bytes and blocks a large
 model download Chrome otherwise makes on every fresh profile.
@@ -226,9 +244,10 @@ Google refuses, and tells you what each page cost.
 
 ## Status
 
-Early. Tested on Windows. **Linux and Docker are experimental**: on a machine
-without a GPU, Chrome has no WebGL, which a normal desktop always has; the
-Docker image turns on a software renderer for it. The `cloak` engine
+Early. Windows is the tested and recommended platform. **Linux and Docker are
+experimental**: on a machine without a GPU, Chrome has no WebGL, which a normal
+desktop always has; the Docker image turns on a software renderer for it, and
+whether Google serves that is not measured yet. The `cloak` engine
 (`--engine cloak`) is experimental too; `patchright` is the default.
 
 Google changes its pages often. If results come back empty or wrong, run with

{google_browser_scraper-0.0.3 → google_browser_scraper-0.0.4}/README.md

@@ -1,3 +1,7 @@
+<p align="center">
+  <img src="https://raw.githubusercontent.com/nodemaven/google-browser-scraper/main/assets/banner.png" alt="Google Browser Scraper">
+</p>
+
 # google-browser-scraper
 
 Google Search results from a real browser, through your own sticky proxy.
@@ -5,9 +9,15 @@ Google Search results from a real browser, through your own sticky proxy.
 Plain HTTP is no longer a reliable way to scrape Google: it often expects a
 browser that runs JavaScript, and a fresh browser on a fresh proxy exit can be
 refused quickly. This tool takes a browser path instead. It warms a sticky proxy
-session with ordinary browsing, types the query into Google's search box, checks
-whether Google served results or refused, and writes one JSON record per results
-page. Captchas, blocks and failures are reported as such, never as "no results".
+session with ordinary browsing, opens Google's results page for each query in
+that browser, checks whether Google served results or refused, and writes one
+JSON record per results page. Captchas, blocks and failures are reported as such, never as "no results".
+
+**Run it on Windows** - a desktop or a Windows VPS. That is where it is tested
+and where Google serves it. On a Linux server without a GPU, Google refused
+almost every search in testing; Chrome has no WebGL there, which is the main
+suspect, so Linux and Docker are experimental for now. `google-browser-scraper
+doctor` shows what your machine looks like to a website.
 
 ## Install
 
@@ -43,8 +53,8 @@ Success is a `results.jsonl` with one JSON object per results page:
 
 The first query takes a few minutes, because a new exit is warmed up first, and
 the warm-up costs noticeably more traffic than a search. Later queries on the
-same exit take seconds. A summary of pages served and traffic used is printed at
-the end of every run.
+same exit take seconds. A summary of pages served and traffic used, with the
+hosts that took most of it, is printed at the end of every run.
 
 With NodeMaven, the session id is handled for you and the credentials come from
 the environment:
@@ -85,7 +95,8 @@ shopping, images and other special blocks are not parsed.
 
 Each record also has an `exit` block: which proxy session answered and how many
 bytes the page cost. `search_metadata.timings` says where the page's time went,
-in seconds per step (`type`, `load`, `ready`, `read`, `parse`, `links`). A query
+in seconds per step (`type`, `load`, `ready`, `read`, `parse`, and `links`, which
+overlaps the next query and is not part of `total_time_taken`). A query
 that failed is a record too, with an `error`.
 `--format csv` writes the organic results as a table, and `--price-per-gb` adds
 the cost per 1000 pages to the summary.
@@ -93,8 +104,12 @@ the cost per 1000 pages to the summary.
 ## How it works
 
 1. **New identity**: a sticky proxy session and a fresh browser profile.
-2. **Warm-up**: a few ordinary pages, then Google's front page.
-3. **Search**: the query is typed into the box and submitted.
+2. **Warm-up**: a few ordinary pages.
+3. **Search**: each query opens Google's results page directly, which is
+   faster than the search box and skips the autocomplete traffic of typing.
+   `--direct follow-ups` sends an exit's first query through the box and
+   `--direct none` every query; `--input` sets how a query goes into the box
+   (`type` key by key, `insert` all at once, `words` word by word).
 4. **Check before parsing**: the page is classified first, so a captcha or a
    block is reported as one and never parsed as an empty result.
 5. **Keep or drop**: an exit Google answers is reused for the next queries; an
@@ -103,20 +118,22 @@ the cost per 1000 pages to the summary.
 
 A served exit is kept for up to 50 queries (`--max-per-exit`). Every new exit
 pays for a warm-up, so a short cap mostly buys more warm-up traffic. A browser
-error on a search is retried once from Google's front page before the exit is
-dropped; a network error drops it at once.
+error on a search is retried once on the same exit (through Google's front page
+when queries go through the search box) before the exit is dropped; a network
+error drops it at once.
 
 `--workers N` searches on N exits at once, one browser each, sharing one list
-of queries. Each browser needs memory and CPU of its own; if exit after exit is
-refused on any worker, the whole run stops.
+of queries. Each browser needs memory and CPU of its own. The run stops when
+exit after exit is refused or unreachable with no worker served in between.
 
 The command line copies the cookies of the first exit Google serves and
 starts up to 10 new exits from that copy (`--clone N`, `--clone 0` turns it
-off), on new sticky sessions, instead of warming each one. Google's acceptance follows the cookies more than the
-address, so a copy usually starts out served, and costs a few MB instead of a
-full warm-up. A copy refused on its first query ends the copying; the copied
-cookies are used for an hour at most (`--clone-max-age`). A copy waits 40-60
-seconds after its browser starts before the first search.
+off), on new sticky sessions, instead of warming each one; all workers share
+the copy. Google's acceptance follows the cookies more than the address, so a
+copy usually starts out served, and costs a few MB instead of a full warm-up.
+A copy refused on its first query ends the copying; the copied cookies are
+used for three hours at most (`--clone-max-age`). A copy waits 40-60 seconds
+after its browser starts before the first search.
 
 `--spare 1` warms the next exit in the background once the current one is
 served, so a retired exit is replaced without waiting for a warm-up. A spare
@@ -124,8 +141,9 @@ costs a warm-up's traffic whether it is used or not.
 
 Google wraps result links in encrypted `/goto` redirects. Each one is resolved
 with a small request through a separate short-lived proxy session, a few at a
-time, so link resolution does not touch the warmed browser session.
-`--no-resolve-links` skips it.
+time, so link resolution does not touch the warmed browser session. It runs in
+the background while the browser goes on to the next query; records still come
+out in the order each browser searched them. `--no-resolve-links` skips it.
 
 Traffic goes through a small local relay that counts bytes and blocks a large
 model download Chrome otherwise makes on every fresh profile.
@@ -195,9 +213,10 @@ Google refuses, and tells you what each page cost.
 
 ## Status
 
-Early. Tested on Windows. **Linux and Docker are experimental**: on a machine
-without a GPU, Chrome has no WebGL, which a normal desktop always has; the
-Docker image turns on a software renderer for it. The `cloak` engine
+Early. Windows is the tested and recommended platform. **Linux and Docker are
+experimental**: on a machine without a GPU, Chrome has no WebGL, which a normal
+desktop always has; the Docker image turns on a software renderer for it, and
+whether Google serves that is not measured yet. The `cloak` engine
 (`--engine cloak`) is experimental too; `patchright` is the default.
 
 Google changes its pages often. If results come back empty or wrong, run with

{google_browser_scraper-0.0.3 → google_browser_scraper-0.0.4}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "google-browser-scraper"
-version = "0.0.3"
+version = "0.0.4"
 description = "Google Search results from a real browser through your own sticky proxy, with captchas and blocks reported explicitly."
 readme = "README.md"
 license = { text = "MIT" }

{google_browser_scraper-0.0.3 → google_browser_scraper-0.0.4}/src/google_browser_scraper/__init__.py

@@ -7,7 +7,7 @@ for record in Scraper(proxy, Settings(pages=2)).run(["best running shoes"]):
     print(record["organic_results"])
 """
 
-__version__ = "0.0.3"
+__version__ = "0.0.4"
 
 from .classify import Verdict, classify  # noqa: E402
 from .parse import parse_serp  # noqa: E402

{google_browser_scraper-0.0.3 → google_browser_scraper-0.0.4}/src/google_browser_scraper/browser.py

@@ -6,7 +6,10 @@
 - `locale` is never set: setting it through the context makes the main thread
   and Web Workers disagree. `timezone_id` goes through CDP emulation and agrees
   in workers, and is off unless asked for.
-- The query is typed into the box, not sent as `/search?q=`.
+- The first query on an exit is typed into the box. Later ones on the same
+  exit can go straight to `/search?q=` (`direct=True`): once Google has served
+  the exit, it serves those as readily, faster and without the autocomplete
+  traffic each keystroke fires.
 """
 
 from __future__ import annotations
@@ -16,8 +19,12 @@ import shutil
 import tempfile
 import time
 from dataclasses import dataclass, field, replace
+from urllib.parse import urlencode, urlsplit, urlunsplit
 
 HOME_URL = "https://www.google.com/?hl=en"
+#: How a query gets into the search box: key by key, all at once, or one word
+#: at a time.
+INPUT_MODES = ("type", "insert", "words")
 SEARCH_BOX = "textarea[name='q'], input[name='q']"
 READY = "#rso, #search"
 # Reject first: it sets the smaller cookie. The middle selector is the
@@ -69,7 +76,11 @@ class BrowserSession:
         browser_args: list[str] | tuple[str, ...] = (),
         rng: random.Random | None = None,
         profile_dir: str | None = None,
+        input_mode: str = "type",
     ) -> None:
+        if input_mode not in INPUT_MODES:
+            raise ValueError(f"input mode must be one of {', '.join(INPUT_MODES)}")
+        self._input = input_mode
         if any(a.startswith(("--disable-features", "--enable-features")) for a in browser_args):
             raise ValueError(
                 "--disable-features/--enable-features would replace the browser's "
@@ -163,8 +174,11 @@ class BrowserSession:
 
     # -- searching -----------------------------------------------------------
 
-    def search(self, query: str) -> Fetched:
-        """Type `query` into Google's box and return the page it led to."""
+    def search(self, query: str, direct: bool = False) -> Fetched:
+        """Type `query` into Google's box and return the page it led to; with
+        `direct`, open the results page for it instead."""
+        if direct:
+            return self._search_direct(query)
         page = self.page
         timings: dict[str, float] = {}
         mark = time.monotonic()
@@ -200,6 +214,22 @@ class BrowserSession:
         fetched = self._snapshot(nav.value, dismissed)
         return replace(fetched, timings={**timings, **fetched.timings})
 
+    def _search_direct(self, query: str) -> Fetched:
+        home = urlsplit(HOME_URL)
+        url = urlunsplit(
+            (home.scheme, home.netloc, "/search", urlencode({"q": query, "hl": "en"}), "")
+        )
+        mark = time.monotonic()
+        response = self.page.goto(url, wait_until="domcontentloaded", timeout=NAV_TIMEOUT_MS)
+        # A fresh exit can be shown the consent wall instead of results; clear
+        # it and open the results again, as the typed path does on the front page.
+        dismissed = self._dismiss_consent()
+        if dismissed:
+            response = self.page.goto(url, wait_until="domcontentloaded", timeout=NAV_TIMEOUT_MS)
+        load = round(time.monotonic() - mark, 2)
+        fetched = self._snapshot(response, dismissed)
+        return replace(fetched, timings={"load": load, **fetched.timings})
+
     def _box_visible(self) -> bool:
         try:
             return any(h.is_visible() for h in self.page.query_selector_all(SEARCH_BOX))
@@ -220,9 +250,23 @@ class BrowserSession:
         # so the typing replaces it.
         if box.input_value():
             box.press("ControlOrMeta+a")
-        box.type(query, delay=self._rng.randint(45, 140))
+        if self._input == "insert":
+            page.keyboard.insert_text(query)
+        elif self._input == "words":
+            words = query.split(" ")
+            for i, word in enumerate(words):
+                last = i == len(words) - 1
+                page.keyboard.insert_text(word if last else word + " ")
+                if not last:
+                    time.sleep(self._rng.uniform(0.15, 0.5))
+        else:
+            box.type(query, delay=self._rng.randint(45, 140))
         return box
 
+    def blank(self) -> None:
+        """Leave the current page, stopping whatever it is still loading."""
+        self.page.goto("about:blank")
+
     def next_page(self) -> Fetched | None:
         """Click "Next" on the current results page, if there is one."""
         link = self.page.query_selector("a#pnnext")

{google_browser_scraper-0.0.3 → google_browser_scraper-0.0.4}/src/google_browser_scraper/cli.py

@@ -9,13 +9,13 @@ import shlex
 import sys
 from pathlib import Path
 
-from . import __version__, engines, warmup
+from . import __version__, browser, engines, warmup
 from .classify import classify
 from .output import WRITERS, open_writer, use_utf8
 from .parse import parse_serp
 from .pool import Pool
 from .proxy import NodeMavenSource, ProxyTemplate
-from .scraper import ExitsRefused, Scraper, Settings
+from .scraper import DIRECT_MODES, ExitsRefused, Scraper, Settings
 
 
 def main(argv: list[str] | None = None) -> int:
@@ -138,9 +138,29 @@ def _scraper_options(p: argparse.ArgumentParser) -> None:
     p.add_argument(
         "--clone-max-age",
         type=float,
-        default=60.0,
+        default=180.0,
         metavar="MIN",
-        help="minutes copied cookies stay in use (default 60)",
+        help="minutes copied cookies stay in use (default 180)",
+    )
+    p.add_argument(
+        "--input",
+        choices=browser.INPUT_MODES,
+        default="type",
+        help="how a query is entered into Google's search box: typed key by key "
+        "(default), inserted at once, or inserted word by word",
+    )
+    p.add_argument(
+        "--direct",
+        choices=DIRECT_MODES,
+        default="all",
+        help="which queries open the results page directly instead of going through "
+        "the search box: all (default), all but an exit's first, or none",
+    )
+    p.add_argument(
+        "--stop-after-read",
+        action="store_true",
+        help="leave each results page for about:blank once it is read (experimental: "
+        "saves traffic Google's late scripts would spend)",
     )
     p.add_argument(
         "--gap",
@@ -214,6 +234,9 @@ def _build(args, *, pages: int = 1, worker: int | None = None) -> tuple[Scraper,
         spare=args.spare,
         clones=args.clone,
         clone_max_age=args.clone_max_age * 60,
+        input=args.input,
+        direct=args.direct,
+        stop_after_read=args.stop_after_read,
         gap_seconds=args.gap,
         headless=args.headless,
         channel=args.channel,

{google_browser_scraper-0.0.3 → google_browser_scraper-0.0.4}/src/google_browser_scraper/links.py

@@ -93,6 +93,9 @@ class LinkResolver:
     def bytes(self) -> int:
         return self._relay.total if self._relay is not None else 0
 
+    def hosts(self) -> dict[str, int]:
+        return self._relay.hosts() if self._relay is not None else {}
+
     def resolve(self, goto_url: str) -> str | None:
         if not self.enabled:
             return None

{google_browser_scraper-0.0.3 → google_browser_scraper-0.0.4}/src/google_browser_scraper/pool.py

@@ -5,15 +5,21 @@ page and the gap between queries, not the machine. Running `workers` scrapers
 side by side multiplies throughput by roughly that number until the machine
 runs out of CPU or memory for the browsers.
 
-Each worker is an ordinary `Scraper` with its own identities, spares, breaker
-and traffic count; the workers share only the queue of queries. Playwright's
-sync API is bound to the thread that starts it, and every worker starts its
-browsers on its own thread, so nothing is shared across threads.
-
-When one worker's breaker trips, the whole pool stops: exit after exit being
-refused is a statement about the hour or the pool, not about one worker, and
-the other workers would only spend traffic finding the same thing. Queries
-nobody answered by then are listed in `not_run`.
+Each worker is an ordinary `Scraper` with its own identities, spares and
+traffic count. The workers share the queue of queries, one breaker and one
+cookie-jar template. Playwright's sync API is bound to the thread that starts
+it, and every worker starts its browsers on its own thread, so no browser is
+shared across threads.
+
+The breaker counts refused exits across the pool and any served page resets
+it: one worker drawing bad exits while another keeps being served is not a bad
+hour. When nobody is served for `breaker` exits in a row the whole pool stops,
+since the other workers would only spend traffic finding the same thing.
+Queries nobody answered by then are listed in `not_run`.
+
+The template is shared for the same reason: a served cookie jar is worth as
+much to every worker, and a worker whose copy ran out should start from the
+other worker's jar rather than warm fresh exits.
 """
 
 from __future__ import annotations
@@ -24,7 +30,7 @@ import time
 from collections.abc import Callable, Iterable, Iterator
 from typing import Any
 
-from .scraper import ExitsRefused, Scraper, Stats
+from .scraper import Breaker, ExitsRefused, Scraper, Stats, Templates
 
 MAX_WORKERS = 16
 _DONE = object()
@@ -75,6 +81,22 @@ class Pool:
         out: queue.Queue[Any] = queue.Queue()
         stopped: list[ExitsRefused] = []
         self._scrapers = [self._make(i) for i in range(self.workers)]
+        first = self._scrapers[0].settings
+        for scraper in self._scrapers[1:]:
+            s = scraper.settings
+            if (s.clones, s.clone_max_age, s.breaker) != (
+                first.clones,
+                first.clone_max_age,
+                first.breaker,
+            ):
+                raise ValueError(
+                    "workers share one cookie template and one breaker, so clones, "
+                    "clone_max_age and breaker must be the same for every worker"
+                )
+        templates = Templates(first.clones, first.clone_max_age)
+        breaker = Breaker(first.breaker)
+        for scraper in self._scrapers:
+            scraper.share(breaker, templates)
         threads = [
             threading.Thread(
                 target=self._work,
@@ -100,6 +122,7 @@ class Pool:
             self._stop.set()
             for t in threads:
                 t.join()
+            templates.drop("run finished", lambda event: None)
         while not todo.empty():
             self.not_run.append(todo.get_nowait())
         if stopped:
@@ -158,5 +181,7 @@ def merge_stats(parts: Iterable[Stats]) -> Stats:
         ):
             setattr(merged, name, getattr(merged, name) + getattr(s, name))
         merged.counted = merged.counted and s.counted
+        for host, n in s.bytes_by_host.items():
+            merged.bytes_by_host[host] = merged.bytes_by_host.get(host, 0) + n
         merged._open.extend(s._open)
     return merged

{google_browser_scraper-0.0.3 → google_browser_scraper-0.0.4}/src/google_browser_scraper/relay.py

@@ -60,6 +60,7 @@ class Relay:
         self.tunnels = 0
         self.failures = 0
         self.blocked = 0
+        self._hosts: dict[str, int] = {}
         #: Status line of the last CONNECT the upstream proxy refused, such as
         #: `HTTP/1.1 402 traffic exhausted`. The browser only reports a failed
         #: tunnel; this is the proxy's reason.
@@ -100,6 +101,15 @@ class Relay:
         with self._lock:
             return self.bytes_up + self.bytes_down
 
+    def hosts(self) -> dict[str, int]:
+        """Bytes both ways per target host, the CONNECT exchange included.
+
+        A plain-HTTP connection the browser keeps alive for requests to other
+        hosts is counted under its first host; that traffic is a few small
+        revocation checks and redirects."""
+        with self._lock:
+            return dict(self._hosts)
+
     # -- serving ---------------------------------------------------------------
 
     def _accept_loop(self) -> None:
@@ -162,9 +172,9 @@ class Relay:
                     f"{self._auth}Proxy-Connection: Keep-Alive\r\n\r\n"
                 ).encode()
                 upstream.sendall(request)
-                self._count(up=len(request))
+                self._count(up=len(request), host=host)
                 reply = _read_head(upstream)
-                self._count(down=len(reply))
+                self._count(down=len(reply), host=host)
                 if _status(reply) != 200:
                     # Pass the gateway's own answer through: its status is the
                     # evidence (a bad parameter is 406 or 407 on some gateways).
@@ -184,12 +194,13 @@ class Relay:
         except OSError:
             _close(upstream)
             return
-        self._pump(client, upstream)
+        self._pump(client, upstream, host)
 
     def _forward(self, client: socket.socket, head: str) -> None:
         """Plain HTTP in absolute-URI form (revocation checks, http:// redirects)."""
         target = urlsplit(head.split("\r\n", 1)[0].split(" ")[1])
-        if (target.hostname or "") in self.block_hosts:
+        host = (target.hostname or "").lower()
+        if host in self.block_hosts:
             self._refuse(client)
             return
         lines = [ln for ln in head.split("\r\n") if not ln.lower().startswith("proxy-")]
@@ -207,13 +218,13 @@ class Relay:
                 rebuilt = lines[0] + "\r\n" + self._auth + "\r\n".join(lines[1:])
             data = rebuilt.encode("latin-1")
             upstream.sendall(data)
-            self._count(up=len(data))
+            self._count(up=len(data), host=host)
         except OSError:
             self._fail()
             return
-        self._pump(client, upstream)
+        self._pump(client, upstream, host)
 
-    def _pump(self, client: socket.socket, upstream: socket.socket) -> None:
+    def _pump(self, client: socket.socket, upstream: socket.socket, host: str) -> None:
         """Copy both ways until either end closes, counting per chunk so a
         keep-alive tunnel's bytes land on the page that used them."""
         client.settimeout(None)
@@ -231,19 +242,21 @@ class Relay:
                         return
                     if source is client:
                         upstream.sendall(data)
-                        self._count(up=len(data))
+                        self._count(up=len(data), host=host)
                     else:
                         client.sendall(data)
-                        self._count(down=len(data))
+                        self._count(down=len(data), host=host)
         except OSError:
             pass
         finally:
             _close(upstream)
 
-    def _count(self, *, up: int = 0, down: int = 0) -> None:
+    def _count(self, *, up: int = 0, down: int = 0, host: str = "") -> None:
         with self._lock:
             self.bytes_up += up
             self.bytes_down += down
+            if host:
+                self._hosts[host] = self._hosts.get(host, 0) + up + down
 
 
 def _read_head(sock: socket.socket) -> str: