google-browser-scraper 0.0.2__tar.gz → 0.0.3__tar.gz

google_browser_scraper-0.0.3/MANIFEST.in

@@ -0,0 +1 @@
+recursive-include tests *.py

{google_browser_scraper-0.0.2/src/google_browser_scraper.egg-info → google_browser_scraper-0.0.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: google-browser-scraper
-Version: 0.0.2
+Version: 0.0.3
 Summary: Google Search results from a real browser through your own sticky proxy, with captchas and blocks reported explicitly.
 Author: NodeMaven
 License: MIT
@@ -115,7 +115,9 @@ simple consumers can often switch with minimal mapping. Knowledge panels,
 shopping, images and other special blocks are not parsed.
 
 Each record also has an `exit` block: which proxy session answered and how many
-bytes the page cost. A query that failed is a record too, with an `error`.
+bytes the page cost. `search_metadata.timings` says where the page's time went,
+in seconds per step (`type`, `load`, `ready`, `read`, `parse`, `links`). A query
+that failed is a record too, with an `error`.
 `--format csv` writes the organic results as a table, and `--price-per-gb` adds
 the cost per 1000 pages to the summary.
 
@@ -130,10 +132,31 @@ the cost per 1000 pages to the summary.
    exit it refuses is dropped and the query is retried on a new one. If exit
    after exit is refused, the run stops instead of spending more traffic.
 
+A served exit is kept for up to 50 queries (`--max-per-exit`). Every new exit
+pays for a warm-up, so a short cap mostly buys more warm-up traffic. A browser
+error on a search is retried once from Google's front page before the exit is
+dropped; a network error drops it at once.
+
+`--workers N` searches on N exits at once, one browser each, sharing one list
+of queries. Each browser needs memory and CPU of its own; if exit after exit is
+refused on any worker, the whole run stops.
+
+The command line copies the cookies of the first exit Google serves and
+starts up to 10 new exits from that copy (`--clone N`, `--clone 0` turns it
+off), on new sticky sessions, instead of warming each one. Google's acceptance follows the cookies more than the
+address, so a copy usually starts out served, and costs a few MB instead of a
+full warm-up. A copy refused on its first query ends the copying; the copied
+cookies are used for an hour at most (`--clone-max-age`). A copy waits 40-60
+seconds after its browser starts before the first search.
+
+`--spare 1` warms the next exit in the background once the current one is
+served, so a retired exit is replaced without waiting for a warm-up. A spare
+costs a warm-up's traffic whether it is used or not.
+
 Google wraps result links in encrypted `/goto` redirects. Each one is resolved
-with a small request through a separate short-lived proxy session, so link
-resolution does not touch the warmed browser session. `--no-resolve-links`
-skips it.
+with a small request through a separate short-lived proxy session, a few at a
+time, so link resolution does not touch the warmed browser session.
+`--no-resolve-links` skips it.
 
 Traffic goes through a small local relay that counts bytes and blocks a large
 model download Chrome otherwise makes on every fresh profile.

{google_browser_scraper-0.0.2 → google_browser_scraper-0.0.3}/README.md

@@ -84,7 +84,9 @@ simple consumers can often switch with minimal mapping. Knowledge panels,
 shopping, images and other special blocks are not parsed.
 
 Each record also has an `exit` block: which proxy session answered and how many
-bytes the page cost. A query that failed is a record too, with an `error`.
+bytes the page cost. `search_metadata.timings` says where the page's time went,
+in seconds per step (`type`, `load`, `ready`, `read`, `parse`, `links`). A query
+that failed is a record too, with an `error`.
 `--format csv` writes the organic results as a table, and `--price-per-gb` adds
 the cost per 1000 pages to the summary.
 
@@ -99,10 +101,31 @@ the cost per 1000 pages to the summary.
    exit it refuses is dropped and the query is retried on a new one. If exit
    after exit is refused, the run stops instead of spending more traffic.
 
+A served exit is kept for up to 50 queries (`--max-per-exit`). Every new exit
+pays for a warm-up, so a short cap mostly buys more warm-up traffic. A browser
+error on a search is retried once from Google's front page before the exit is
+dropped; a network error drops it at once.
+
+`--workers N` searches on N exits at once, one browser each, sharing one list
+of queries. Each browser needs memory and CPU of its own; if exit after exit is
+refused on any worker, the whole run stops.
+
+The command line copies the cookies of the first exit Google serves and
+starts up to 10 new exits from that copy (`--clone N`, `--clone 0` turns it
+off), on new sticky sessions, instead of warming each one. Google's acceptance follows the cookies more than the
+address, so a copy usually starts out served, and costs a few MB instead of a
+full warm-up. A copy refused on its first query ends the copying; the copied
+cookies are used for an hour at most (`--clone-max-age`). A copy waits 40-60
+seconds after its browser starts before the first search.
+
+`--spare 1` warms the next exit in the background once the current one is
+served, so a retired exit is replaced without waiting for a warm-up. A spare
+costs a warm-up's traffic whether it is used or not.
+
 Google wraps result links in encrypted `/goto` redirects. Each one is resolved
-with a small request through a separate short-lived proxy session, so link
-resolution does not touch the warmed browser session. `--no-resolve-links`
-skips it.
+with a small request through a separate short-lived proxy session, a few at a
+time, so link resolution does not touch the warmed browser session.
+`--no-resolve-links` skips it.
 
 Traffic goes through a small local relay that counts bytes and blocks a large
 model download Chrome otherwise makes on every fresh profile.

{google_browser_scraper-0.0.2 → google_browser_scraper-0.0.3}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "google-browser-scraper"
-version = "0.0.2"
+version = "0.0.3"
 description = "Google Search results from a real browser through your own sticky proxy, with captchas and blocks reported explicitly."
 readme = "README.md"
 license = { text = "MIT" }

{google_browser_scraper-0.0.2 → google_browser_scraper-0.0.3}/src/google_browser_scraper/__init__.py

@@ -7,7 +7,7 @@ for record in Scraper(proxy, Settings(pages=2)).run(["best running shoes"]):
     print(record["organic_results"])
 """
 
-__version__ = "0.0.2"
+__version__ = "0.0.3"
 
 from .classify import Verdict, classify  # noqa: E402
 from .parse import parse_serp  # noqa: E402

{google_browser_scraper-0.0.2 → google_browser_scraper-0.0.3}/src/google_browser_scraper/browser.py

@@ -15,7 +15,7 @@ import random
 import shutil
 import tempfile
 import time
-from dataclasses import dataclass
+from dataclasses import dataclass, field, replace
 
 HOME_URL = "https://www.google.com/?hl=en"
 SEARCH_BOX = "textarea[name='q'], input[name='q']"
@@ -30,6 +30,7 @@ CONSENT_BUTTONS = (
 )
 NAV_TIMEOUT_MS = 60_000
 WARM_TIMEOUT_MS = 30_000
+TYPE_ATTEMPTS = 2
 
 
 @dataclass(frozen=True)
@@ -40,6 +41,8 @@ class Fetched:
     status: int | None
     html: str
     consent_dismissed: bool = False
+    #: Seconds per step, e.g. {"type": 3.1, "load": 4.2}: where the time went.
+    timings: dict[str, float] = field(default_factory=dict)
 
 
 class BrowserSession:
@@ -65,6 +68,7 @@ class BrowserSession:
         timezone_id: str | None = None,
         browser_args: list[str] | tuple[str, ...] = (),
         rng: random.Random | None = None,
+        profile_dir: str | None = None,
     ) -> None:
         if any(a.startswith(("--disable-features", "--enable-features")) for a in browser_args):
             raise ValueError(
@@ -79,19 +83,38 @@ class BrowserSession:
         self._rng = rng or random.Random()
         self._pw = None
         self._context = None
-        self._profile = None
+        # A given directory (a copy of another identity's profile) is used as
+        # it is; either way the profile is removed on close.
+        self._profile = profile_dir
         self.page = None
 
+    @property
+    def profile_dir(self) -> str | None:
+        return self._profile
+
     def __enter__(self) -> BrowserSession:
-        self._profile = tempfile.mkdtemp(prefix="gbs-profile-")
+        if self._profile is None:
+            self._profile = tempfile.mkdtemp(prefix="gbs-profile-")
         try:
-            self._context = self._launch(self._profile)
+            self._open()
         except Exception:
             self.close()
             raise
+        return self
+
+    def _open(self) -> None:
+        self._context = self._launch(self._profile)
         pages = self._context.pages
         self.page = pages[0] if pages else self._context.new_page()
-        return self
+
+    def suspend(self) -> str:
+        """Close the browser and keep its profile on disk, so it can be
+        copied; `resume()` opens it again on the same proxy."""
+        self._shut()
+        return self._profile
+
+    def resume(self) -> None:
+        self._open()
 
     def _launch(self, profile: str):
         from patchright.sync_api import sync_playwright
@@ -111,6 +134,12 @@ class BrowserSession:
         self.close()
 
     def close(self) -> None:
+        self._shut()
+        if self._profile:
+            shutil.rmtree(self._profile, ignore_errors=True)
+        self._profile = None
+
+    def _shut(self) -> None:
         for step in (
             lambda: self._context and self._context.close(),
             lambda: self._pw and self._pw.stop(),
@@ -119,9 +148,7 @@ class BrowserSession:
                 step()
             except Exception:
                 pass
-        if self._profile:
-            shutil.rmtree(self._profile, ignore_errors=True)
-        self._context = self._pw = self._profile = self.page = None
+        self._context = self._pw = self.page = None
 
     # -- warming -------------------------------------------------------------
 
@@ -139,52 +166,95 @@ class BrowserSession:
     def search(self, query: str) -> Fetched:
         """Type `query` into Google's box and return the page it led to."""
         page = self.page
-        if not any(h.is_visible() for h in page.query_selector_all(SEARCH_BOX)):
+        timings: dict[str, float] = {}
+        mark = time.monotonic()
+        if not self._box_visible():
             page.goto(HOME_URL, wait_until="domcontentloaded", timeout=NAV_TIMEOUT_MS)
-        dismissed = self._dismiss_consent()
+            timings["home"] = round(time.monotonic() - mark, 2)
+            mark = time.monotonic()
+        dismissed, typed = False, None
+        # The page can take focus or swap the box while the query is being
+        # typed, which leaves only its first characters behind. The box is
+        # found again and the query retyped once before giving up.
+        for _ in range(TYPE_ATTEMPTS):
+            dismissed = self._dismiss_consent() or dismissed
+            box = self._type_query(query)
+            try:
+                typed = box.input_value()
+            except Exception:
+                typed = None  # the box was replaced and this handle is gone
+            if typed == query:
+                break
+        else:
+            raise RuntimeError(
+                f"search box holds {typed!r} after typing {query!r} "
+                f"{TYPE_ATTEMPTS} times; not submitting"
+            )
+        timings["type"] = round(time.monotonic() - mark, 2)
+        mark = time.monotonic()
+        # `no_wait_after=True` keeps `press` from waiting for the navigation a
+        # second time with Playwright's own 30 s default under ours.
+        with page.expect_navigation(wait_until="domcontentloaded", timeout=NAV_TIMEOUT_MS) as nav:
+            box.press("Enter", no_wait_after=True, timeout=NAV_TIMEOUT_MS)
+        timings["load"] = round(time.monotonic() - mark, 2)
+        fetched = self._snapshot(nav.value, dismissed)
+        return replace(fetched, timings={**timings, **fetched.timings})
+
+    def _box_visible(self) -> bool:
+        try:
+            return any(h.is_visible() for h in self.page.query_selector_all(SEARCH_BOX))
+        except Exception:
+            # The page navigated while it was being read, as a warm-up page
+            # left idle can; the front page is loaded instead.
+            return False
+
+    def _type_query(self, query: str):
+        page = self.page
         # Act on the handle that was found, never on the selector again: the
         # selector also matches a hidden field, and `page.click` would pick it.
         box = page.wait_for_selector(SEARCH_BOX, state="visible", timeout=NAV_TIMEOUT_MS)
         box.click()
-        # A results page keeps the previous query in the box. Typing after it
-        # appends, and the appended query returns a real page for the wrong
-        # question. Select it so the typing replaces it.
+        # A results page keeps the previous query in the box, and a failed
+        # attempt leaves part of this one. Typing after either appends, and the
+        # appended query returns a real page for the wrong question. Select it
+        # so the typing replaces it.
         if box.input_value():
             box.press("ControlOrMeta+a")
         box.type(query, delay=self._rng.randint(45, 140))
-        typed = box.input_value()
-        if typed != query:
-            raise RuntimeError(f"search box holds {typed!r} after typing {query!r}; not submitting")
-        # `no_wait_after=True` keeps `press` from waiting for the navigation a
-        # second time with Playwright's own 30 s default under ours.
-        with page.expect_navigation(wait_until="domcontentloaded", timeout=NAV_TIMEOUT_MS) as nav:
-            box.press("Enter", no_wait_after=True, timeout=NAV_TIMEOUT_MS)
-        return self._snapshot(nav.value, dismissed)
+        return box
 
     def next_page(self) -> Fetched | None:
         """Click "Next" on the current results page, if there is one."""
         link = self.page.query_selector("a#pnnext")
         if link is None or not link.is_visible():
             return None
+        mark = time.monotonic()
         with self.page.expect_navigation(
             wait_until="domcontentloaded", timeout=NAV_TIMEOUT_MS
         ) as nav:
             link.click()
-        return self._snapshot(nav.value, False)
+        load = round(time.monotonic() - mark, 2)
+        fetched = self._snapshot(nav.value, False)
+        return replace(fetched, timings={"load": load, **fetched.timings})
 
     def _snapshot(self, response, dismissed: bool) -> Fetched:
         # Google builds results in the browser; reading at domcontentloaded
         # catches the "enable JavaScript" scaffold. A refusal never grows the
         # container, so the short timeout is expected to expire on those.
+        mark = time.monotonic()
         try:
             self.page.wait_for_selector(READY, state="attached", timeout=8_000)
         except Exception:
             pass
+        ready = round(time.monotonic() - mark, 2)
+        mark = time.monotonic()
+        html = self.page.content()
         return Fetched(
             url=self.page.url,
             status=response.status if response is not None else None,
-            html=self.page.content(),
+            html=html,
             consent_dismissed=dismissed,
+            timings={"ready": ready, "read": round(time.monotonic() - mark, 2)},
         )
 
     def _dismiss_consent(self) -> bool:

{google_browser_scraper-0.0.2 → google_browser_scraper-0.0.3}/src/google_browser_scraper/cli.py

@@ -13,6 +13,7 @@ from . import __version__, engines, warmup
 from .classify import classify
 from .output import WRITERS, open_writer, use_utf8
 from .parse import parse_serp
+from .pool import Pool
 from .proxy import NodeMavenSource, ProxyTemplate
 from .scraper import ExitsRefused, Scraper, Settings
 
@@ -45,6 +46,13 @@ def _parser() -> argparse.ArgumentParser:
     search.add_argument("-f", "--file", type=Path, help="one query per line")
     _scraper_options(search)
     search.add_argument("--pages", type=int, default=1, help="result pages per query (default 1)")
+    search.add_argument(
+        "--workers",
+        type=int,
+        default=1,
+        metavar="N",
+        help="exits to search on at once, one browser each (default 1)",
+    )
     search.add_argument("--format", choices=list(WRITERS), default="jsonl")
     search.add_argument("-o", "--output", type=Path, help="output file (default stdout)")
     search.add_argument(
@@ -108,15 +116,38 @@ def _scraper_options(p: argparse.ArgumentParser) -> None:
     p.add_argument(
         "--max-per-exit",
         type=int,
+        default=50,
+        help="queries before an exit is retired even if still served (default 50)",
+    )
+    p.add_argument(
+        "--spare",
+        type=int,
+        default=0,
+        metavar="N",
+        help="exits to warm in the background while the current one serves, so a "
+        "retired exit is replaced without waiting (default 0; each costs a warm-up)",
+    )
+    p.add_argument(
+        "--clone",
+        type=int,
         default=10,
-        help="queries before an exit is retired even if still served",
+        metavar="N",
+        help="start up to N new exits from copies of the first served exit's cookies "
+        "instead of warming each one (default 10; 0 turns it off)",
+    )
+    p.add_argument(
+        "--clone-max-age",
+        type=float,
+        default=60.0,
+        metavar="MIN",
+        help="minutes copied cookies stay in use (default 60)",
     )
     p.add_argument(
         "--gap",
         type=_pair,
-        default=(8.0, 20.0),
+        default=(2.0, 5.0),
         metavar="LOW,HIGH",
-        help="seconds between queries on one exit (default 8,20)",
+        help="seconds between queries on one exit (default 2,5)",
     )
     p.add_argument(
         "--headless",
@@ -151,7 +182,7 @@ def _scraper_options(p: argparse.ArgumentParser) -> None:
     p.add_argument("-v", "--verbose", action="store_true", help="log identities to stderr")
 
 
-def _build(args, *, pages: int = 1) -> tuple[Scraper, str]:
+def _build(args, *, pages: int = 1, worker: int | None = None) -> tuple[Scraper, str]:
     if args.nodemaven:
         proxy = NodeMavenSource(country=args.country)
     elif args.no_proxy:
@@ -180,18 +211,29 @@ def _build(args, *, pages: int = 1) -> tuple[Scraper, str]:
         warmup=args.warmup,
         pages=pages,
         max_queries_per_exit=args.max_per_exit,
+        spare=args.spare,
+        clones=args.clone,
+        clone_max_age=args.clone_max_age * 60,
         gap_seconds=args.gap,
         headless=args.headless,
         channel=args.channel,
         timezone_id=args.timezone,
         browser_args=browser_args,
         use_relay=not args.no_relay,
-        save_html=str(args.save_html) if args.save_html else None,
+        save_html=_html_dir(args.save_html, worker),
         resolve_links=not args.no_resolve_links,
     )
-    log = (lambda e: print(json.dumps(e), file=sys.stderr)) if args.verbose else None
-    described = proxy.describe() if proxy else "none (this machine's own address)"
+    log = None
     if args.verbose:
+        tag = {} if worker is None else {"worker": worker}
+
+        def _log(event: dict) -> None:
+            print(json.dumps({**tag, **event}), file=sys.stderr, flush=True)
+
+        log = _log
+
+    described = proxy.describe() if proxy else "none (this machine's own address)"
+    if args.verbose and not worker:
         print(f"proxy: {described}; engine: {args.engine}", file=sys.stderr)
     return Scraper(proxy, settings, on_event=log), described
 
@@ -199,10 +241,13 @@ def _build(args, *, pages: int = 1) -> tuple[Scraper, str]:
 def _search(args) -> int:
     queries = list(args.queries)
     if args.file:
-        queries += args.file.read_text(encoding="utf-8").splitlines()
+        queries += args.file.read_text(encoding="utf-8-sig").splitlines()
     if not any(q.strip() for q in queries):
         raise ValueError("no queries given")
-    scraper, _ = _build(args, pages=args.pages)
+    if args.workers != 1:
+        scraper = Pool(lambda i: _build(args, pages=args.pages, worker=i)[0], args.workers)
+    else:
+        scraper, _ = _build(args, pages=args.pages)
     stream = args.output.open("w", encoding="utf-8", newline="") if args.output else sys.stdout
     writer = open_writer(args.format, stream)
     failed, code = 0, 0
@@ -215,6 +260,9 @@ def _search(args) -> int:
         code = 3
     finally:
         writer.close()
+        not_run = getattr(scraper, "not_run", [])
+        if not_run:
+            print(f"not run: {len(not_run)} queries", file=sys.stderr)
         if args.output:
             stream.close()
         print("summary: " + json.dumps(scraper.stats.summary(args.price_per_gb)), file=sys.stderr)
@@ -269,11 +317,18 @@ def _doctor(args) -> int:
     return 1 if any(c.status == FAIL for c in checks) else 0
 
 
+def _html_dir(folder: Path | None, worker: int | None) -> str | None:
+    """Each worker numbers its pages from 1, so each gets a folder of its own."""
+    if folder is None:
+        return None
+    return str(folder if worker is None else folder / f"w{worker}")
+
+
 def _pair(text: str) -> tuple[float, float]:
     try:
         low, high = (float(x) for x in text.split(","))
     except ValueError:
-        raise argparse.ArgumentTypeError("expected LOW,HIGH, e.g. 8,20") from None
+        raise argparse.ArgumentTypeError("expected LOW,HIGH, e.g. 2,5") from None
     return low, high
 
 

{google_browser_scraper-0.0.2 → google_browser_scraper-0.0.3}/src/google_browser_scraper/links.py

@@ -11,9 +11,11 @@ through a relay so it is counted like everything else.
 
 from __future__ import annotations
 
+import threading
 import urllib.error
 import urllib.request
 from collections.abc import Callable
+from concurrent.futures import ThreadPoolExecutor
 from typing import Any
 from urllib.parse import urlsplit
 
@@ -24,6 +26,8 @@ UA = (
     "(KHTML, like Gecko) Chrome/151.0.0.0 Safari/537.36"
 )
 TIMEOUT_S = 30
+#: /goto links resolved at once for one page.
+PARALLEL = 4
 
 
 class _AlwaysProxy(urllib.request.ProxyHandler):
@@ -61,6 +65,7 @@ class LinkResolver:
         timeout: float = TIMEOUT_S,
     ) -> None:
         self._relay = None
+        self._lock = threading.Lock()
         self.resolved = 0
         self.failed = 0
         proxy_url = None
@@ -103,22 +108,36 @@ class LinkResolver:
             location = err.headers.get("Location") if 300 <= err.code < 400 else None
             err.read()
         except Exception:
-            self.failed += 1
+            with self._lock:
+                self.failed += 1
             return None
         host = urlsplit(location or "").netloc.lower()
         if not location or "/sorry" in location or host.startswith(("sorry.", "consent.")):
-            self.failed += 1
-            self.enabled = False
+            with self._lock:
+                self.failed += 1
+                self.enabled = False
             return None
-        self.resolved += 1
+        with self._lock:
+            self.resolved += 1
         return location
 
     def fill(self, results: list[dict[str, Any]]) -> None:
-        """Resolve every result that has a /goto redirect and no link yet."""
-        for result in results:
-            redirect = result.get("redirect_link") or ""
-            if result.get("link") is None and "/goto?" in redirect:
-                result["link"] = self.resolve(redirect)
+        """Resolve every result that has a /goto redirect and no link yet.
+
+        Up to PARALLEL requests run at once. A refusal disables the resolver,
+        so the requests still queued behind it are not sent.
+        """
+        todo = [
+            r
+            for r in results
+            if r.get("link") is None and "/goto?" in (r.get("redirect_link") or "")
+        ]
+        if not todo:
+            return
+        with ThreadPoolExecutor(max_workers=min(PARALLEL, len(todo))) as pool:
+            links = list(pool.map(lambda r: self.resolve(r["redirect_link"]), todo))
+        for result, link in zip(todo, links, strict=True):
+            result["link"] = link
 
     def close(self) -> None:
         if self._relay is not None:

{google_browser_scraper-0.0.2 → google_browser_scraper-0.0.3}/src/google_browser_scraper/output.py

@@ -57,6 +57,7 @@ def build_record(
     parsed: dict[str, Any] | None = None,
     error: str | None = None,
     hl: str = "en",
+    timings: dict[str, float] | None = None,
 ) -> dict[str, Any]:
     ok = error is None and parsed is not None
     record: dict[str, Any] = {
@@ -77,6 +78,9 @@ def build_record(
             "device": "desktop",
         },
     }
+    if timings:
+        # Seconds per step of this page: home, type, load, ready, read, parse, links.
+        record["search_metadata"]["timings"] = timings
     if parsed:
         record.update(parsed)
     record["exit"] = exit_info