google-browser-scraper 0.0.2__tar.gz → 0.0.4__tar.gz

google_browser_scraper-0.0.4/MANIFEST.in

@@ -0,0 +1 @@
+recursive-include tests *.py

{google_browser_scraper-0.0.2/src/google_browser_scraper.egg-info → google_browser_scraper-0.0.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: google-browser-scraper
-Version: 0.0.2
+Version: 0.0.4
 Summary: Google Search results from a real browser through your own sticky proxy, with captchas and blocks reported explicitly.
 Author: NodeMaven
 License: MIT
@@ -29,6 +29,10 @@ Requires-Dist: pytest>=8; extra == "dev"
 Requires-Dist: ruff>=0.6; extra == "dev"
 Dynamic: license-file
 
+<p align="center">
+  <img src="https://raw.githubusercontent.com/nodemaven/google-browser-scraper/main/assets/banner.png" alt="Google Browser Scraper">
+</p>
+
 # google-browser-scraper
 
 Google Search results from a real browser, through your own sticky proxy.
@@ -36,9 +40,15 @@ Google Search results from a real browser, through your own sticky proxy.
 Plain HTTP is no longer a reliable way to scrape Google: it often expects a
 browser that runs JavaScript, and a fresh browser on a fresh proxy exit can be
 refused quickly. This tool takes a browser path instead. It warms a sticky proxy
-session with ordinary browsing, types the query into Google's search box, checks
-whether Google served results or refused, and writes one JSON record per results
-page. Captchas, blocks and failures are reported as such, never as "no results".
+session with ordinary browsing, opens Google's results page for each query in
+that browser, checks whether Google served results or refused, and writes one
+JSON record per results page. Captchas, blocks and failures are reported as such, never as "no results".
+
+**Run it on Windows** - a desktop or a Windows VPS. That is where it is tested
+and where Google serves it. On a Linux server without a GPU, Google refused
+almost every search in testing; Chrome has no WebGL there, which is the main
+suspect, so Linux and Docker are experimental for now. `google-browser-scraper
+doctor` shows what your machine looks like to a website.
 
 ## Install
 
@@ -74,8 +84,8 @@ Success is a `results.jsonl` with one JSON object per results page:
 
 The first query takes a few minutes, because a new exit is warmed up first, and
 the warm-up costs noticeably more traffic than a search. Later queries on the
-same exit take seconds. A summary of pages served and traffic used is printed at
-the end of every run.
+same exit take seconds. A summary of pages served and traffic used, with the
+hosts that took most of it, is printed at the end of every run.
 
 With NodeMaven, the session id is handled for you and the credentials come from
 the environment:
@@ -115,25 +125,56 @@ simple consumers can often switch with minimal mapping. Knowledge panels,
 shopping, images and other special blocks are not parsed.
 
 Each record also has an `exit` block: which proxy session answered and how many
-bytes the page cost. A query that failed is a record too, with an `error`.
+bytes the page cost. `search_metadata.timings` says where the page's time went,
+in seconds per step (`type`, `load`, `ready`, `read`, `parse`, and `links`, which
+overlaps the next query and is not part of `total_time_taken`). A query
+that failed is a record too, with an `error`.
 `--format csv` writes the organic results as a table, and `--price-per-gb` adds
 the cost per 1000 pages to the summary.
 
 ## How it works
 
 1. **New identity**: a sticky proxy session and a fresh browser profile.
-2. **Warm-up**: a few ordinary pages, then Google's front page.
-3. **Search**: the query is typed into the box and submitted.
+2. **Warm-up**: a few ordinary pages.
+3. **Search**: each query opens Google's results page directly, which is
+   faster than the search box and skips the autocomplete traffic of typing.
+   `--direct follow-ups` sends an exit's first query through the box and
+   `--direct none` every query; `--input` sets how a query goes into the box
+   (`type` key by key, `insert` all at once, `words` word by word).
 4. **Check before parsing**: the page is classified first, so a captcha or a
    block is reported as one and never parsed as an empty result.
 5. **Keep or drop**: an exit Google answers is reused for the next queries; an
    exit it refuses is dropped and the query is retried on a new one. If exit
    after exit is refused, the run stops instead of spending more traffic.
 
+A served exit is kept for up to 50 queries (`--max-per-exit`). Every new exit
+pays for a warm-up, so a short cap mostly buys more warm-up traffic. A browser
+error on a search is retried once on the same exit (through Google's front page
+when queries go through the search box) before the exit is dropped; a network
+error drops it at once.
+
+`--workers N` searches on N exits at once, one browser each, sharing one list
+of queries. Each browser needs memory and CPU of its own. The run stops when
+exit after exit is refused or unreachable with no worker served in between.
+
+The command line copies the cookies of the first exit Google serves and
+starts up to 10 new exits from that copy (`--clone N`, `--clone 0` turns it
+off), on new sticky sessions, instead of warming each one; all workers share
+the copy. Google's acceptance follows the cookies more than the address, so a
+copy usually starts out served, and costs a few MB instead of a full warm-up.
+A copy refused on its first query ends the copying; the copied cookies are
+used for three hours at most (`--clone-max-age`). A copy waits 40-60 seconds
+after its browser starts before the first search.
+
+`--spare 1` warms the next exit in the background once the current one is
+served, so a retired exit is replaced without waiting for a warm-up. A spare
+costs a warm-up's traffic whether it is used or not.
+
 Google wraps result links in encrypted `/goto` redirects. Each one is resolved
-with a small request through a separate short-lived proxy session, so link
-resolution does not touch the warmed browser session. `--no-resolve-links`
-skips it.
+with a small request through a separate short-lived proxy session, a few at a
+time, so link resolution does not touch the warmed browser session. It runs in
+the background while the browser goes on to the next query; records still come
+out in the order each browser searched them. `--no-resolve-links` skips it.
 
 Traffic goes through a small local relay that counts bytes and blocks a large
 model download Chrome otherwise makes on every fresh profile.
@@ -203,9 +244,10 @@ Google refuses, and tells you what each page cost.
 
 ## Status
 
-Early. Tested on Windows. **Linux and Docker are experimental**: on a machine
-without a GPU, Chrome has no WebGL, which a normal desktop always has; the
-Docker image turns on a software renderer for it. The `cloak` engine
+Early. Windows is the tested and recommended platform. **Linux and Docker are
+experimental**: on a machine without a GPU, Chrome has no WebGL, which a normal
+desktop always has; the Docker image turns on a software renderer for it, and
+whether Google serves that is not measured yet. The `cloak` engine
 (`--engine cloak`) is experimental too; `patchright` is the default.
 
 Google changes its pages often. If results come back empty or wrong, run with

{google_browser_scraper-0.0.2 → google_browser_scraper-0.0.4}/README.md

@@ -1,3 +1,7 @@
+<p align="center">
+  <img src="https://raw.githubusercontent.com/nodemaven/google-browser-scraper/main/assets/banner.png" alt="Google Browser Scraper">
+</p>
+
 # google-browser-scraper
 
 Google Search results from a real browser, through your own sticky proxy.
@@ -5,9 +9,15 @@ Google Search results from a real browser, through your own sticky proxy.
 Plain HTTP is no longer a reliable way to scrape Google: it often expects a
 browser that runs JavaScript, and a fresh browser on a fresh proxy exit can be
 refused quickly. This tool takes a browser path instead. It warms a sticky proxy
-session with ordinary browsing, types the query into Google's search box, checks
-whether Google served results or refused, and writes one JSON record per results
-page. Captchas, blocks and failures are reported as such, never as "no results".
+session with ordinary browsing, opens Google's results page for each query in
+that browser, checks whether Google served results or refused, and writes one
+JSON record per results page. Captchas, blocks and failures are reported as such, never as "no results".
+
+**Run it on Windows** - a desktop or a Windows VPS. That is where it is tested
+and where Google serves it. On a Linux server without a GPU, Google refused
+almost every search in testing; Chrome has no WebGL there, which is the main
+suspect, so Linux and Docker are experimental for now. `google-browser-scraper
+doctor` shows what your machine looks like to a website.
 
 ## Install
 
@@ -43,8 +53,8 @@ Success is a `results.jsonl` with one JSON object per results page:
 
 The first query takes a few minutes, because a new exit is warmed up first, and
 the warm-up costs noticeably more traffic than a search. Later queries on the
-same exit take seconds. A summary of pages served and traffic used is printed at
-the end of every run.
+same exit take seconds. A summary of pages served and traffic used, with the
+hosts that took most of it, is printed at the end of every run.
 
 With NodeMaven, the session id is handled for you and the credentials come from
 the environment:
@@ -84,25 +94,56 @@ simple consumers can often switch with minimal mapping. Knowledge panels,
 shopping, images and other special blocks are not parsed.
 
 Each record also has an `exit` block: which proxy session answered and how many
-bytes the page cost. A query that failed is a record too, with an `error`.
+bytes the page cost. `search_metadata.timings` says where the page's time went,
+in seconds per step (`type`, `load`, `ready`, `read`, `parse`, and `links`, which
+overlaps the next query and is not part of `total_time_taken`). A query
+that failed is a record too, with an `error`.
 `--format csv` writes the organic results as a table, and `--price-per-gb` adds
 the cost per 1000 pages to the summary.
 
 ## How it works
 
 1. **New identity**: a sticky proxy session and a fresh browser profile.
-2. **Warm-up**: a few ordinary pages, then Google's front page.
-3. **Search**: the query is typed into the box and submitted.
+2. **Warm-up**: a few ordinary pages.
+3. **Search**: each query opens Google's results page directly, which is
+   faster than the search box and skips the autocomplete traffic of typing.
+   `--direct follow-ups` sends an exit's first query through the box and
+   `--direct none` every query; `--input` sets how a query goes into the box
+   (`type` key by key, `insert` all at once, `words` word by word).
 4. **Check before parsing**: the page is classified first, so a captcha or a
    block is reported as one and never parsed as an empty result.
 5. **Keep or drop**: an exit Google answers is reused for the next queries; an
    exit it refuses is dropped and the query is retried on a new one. If exit
    after exit is refused, the run stops instead of spending more traffic.
 
+A served exit is kept for up to 50 queries (`--max-per-exit`). Every new exit
+pays for a warm-up, so a short cap mostly buys more warm-up traffic. A browser
+error on a search is retried once on the same exit (through Google's front page
+when queries go through the search box) before the exit is dropped; a network
+error drops it at once.
+
+`--workers N` searches on N exits at once, one browser each, sharing one list
+of queries. Each browser needs memory and CPU of its own. The run stops when
+exit after exit is refused or unreachable with no worker served in between.
+
+The command line copies the cookies of the first exit Google serves and
+starts up to 10 new exits from that copy (`--clone N`, `--clone 0` turns it
+off), on new sticky sessions, instead of warming each one; all workers share
+the copy. Google's acceptance follows the cookies more than the address, so a
+copy usually starts out served, and costs a few MB instead of a full warm-up.
+A copy refused on its first query ends the copying; the copied cookies are
+used for three hours at most (`--clone-max-age`). A copy waits 40-60 seconds
+after its browser starts before the first search.
+
+`--spare 1` warms the next exit in the background once the current one is
+served, so a retired exit is replaced without waiting for a warm-up. A spare
+costs a warm-up's traffic whether it is used or not.
+
 Google wraps result links in encrypted `/goto` redirects. Each one is resolved
-with a small request through a separate short-lived proxy session, so link
-resolution does not touch the warmed browser session. `--no-resolve-links`
-skips it.
+with a small request through a separate short-lived proxy session, a few at a
+time, so link resolution does not touch the warmed browser session. It runs in
+the background while the browser goes on to the next query; records still come
+out in the order each browser searched them. `--no-resolve-links` skips it.
 
 Traffic goes through a small local relay that counts bytes and blocks a large
 model download Chrome otherwise makes on every fresh profile.
@@ -172,9 +213,10 @@ Google refuses, and tells you what each page cost.
 
 ## Status
 
-Early. Tested on Windows. **Linux and Docker are experimental**: on a machine
-without a GPU, Chrome has no WebGL, which a normal desktop always has; the
-Docker image turns on a software renderer for it. The `cloak` engine
+Early. Windows is the tested and recommended platform. **Linux and Docker are
+experimental**: on a machine without a GPU, Chrome has no WebGL, which a normal
+desktop always has; the Docker image turns on a software renderer for it, and
+whether Google serves that is not measured yet. The `cloak` engine
 (`--engine cloak`) is experimental too; `patchright` is the default.
 
 Google changes its pages often. If results come back empty or wrong, run with

{google_browser_scraper-0.0.2 → google_browser_scraper-0.0.4}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "google-browser-scraper"
-version = "0.0.2"
+version = "0.0.4"
 description = "Google Search results from a real browser through your own sticky proxy, with captchas and blocks reported explicitly."
 readme = "README.md"
 license = { text = "MIT" }

{google_browser_scraper-0.0.2 → google_browser_scraper-0.0.4}/src/google_browser_scraper/__init__.py

@@ -7,7 +7,7 @@ for record in Scraper(proxy, Settings(pages=2)).run(["best running shoes"]):
     print(record["organic_results"])
 """
 
-__version__ = "0.0.2"
+__version__ = "0.0.4"
 
 from .classify import Verdict, classify  # noqa: E402
 from .parse import parse_serp  # noqa: E402

{google_browser_scraper-0.0.2 → google_browser_scraper-0.0.4}/src/google_browser_scraper/browser.py

@@ -6,7 +6,10 @@
 - `locale` is never set: setting it through the context makes the main thread
   and Web Workers disagree. `timezone_id` goes through CDP emulation and agrees
   in workers, and is off unless asked for.
-- The query is typed into the box, not sent as `/search?q=`.
+- The first query on an exit is typed into the box. Later ones on the same
+  exit can go straight to `/search?q=` (`direct=True`): once Google has served
+  the exit, it serves those as readily, faster and without the autocomplete
+  traffic each keystroke fires.
 """
 
 from __future__ import annotations
@@ -15,9 +18,13 @@ import random
 import shutil
 import tempfile
 import time
-from dataclasses import dataclass
+from dataclasses import dataclass, field, replace
+from urllib.parse import urlencode, urlsplit, urlunsplit
 
 HOME_URL = "https://www.google.com/?hl=en"
+#: How a query gets into the search box: key by key, all at once, or one word
+#: at a time.
+INPUT_MODES = ("type", "insert", "words")
 SEARCH_BOX = "textarea[name='q'], input[name='q']"
 READY = "#rso, #search"
 # Reject first: it sets the smaller cookie. The middle selector is the
@@ -30,6 +37,7 @@ CONSENT_BUTTONS = (
 )
 NAV_TIMEOUT_MS = 60_000
 WARM_TIMEOUT_MS = 30_000
+TYPE_ATTEMPTS = 2
 
 
 @dataclass(frozen=True)
@@ -40,6 +48,8 @@ class Fetched:
     status: int | None
     html: str
     consent_dismissed: bool = False
+    #: Seconds per step, e.g. {"type": 3.1, "load": 4.2}: where the time went.
+    timings: dict[str, float] = field(default_factory=dict)
 
 
 class BrowserSession:
@@ -65,7 +75,12 @@ class BrowserSession:
         timezone_id: str | None = None,
         browser_args: list[str] | tuple[str, ...] = (),
         rng: random.Random | None = None,
+        profile_dir: str | None = None,
+        input_mode: str = "type",
     ) -> None:
+        if input_mode not in INPUT_MODES:
+            raise ValueError(f"input mode must be one of {', '.join(INPUT_MODES)}")
+        self._input = input_mode
         if any(a.startswith(("--disable-features", "--enable-features")) for a in browser_args):
             raise ValueError(
                 "--disable-features/--enable-features would replace the browser's "
@@ -79,19 +94,38 @@ class BrowserSession:
         self._rng = rng or random.Random()
         self._pw = None
         self._context = None
-        self._profile = None
+        # A given directory (a copy of another identity's profile) is used as
+        # it is; either way the profile is removed on close.
+        self._profile = profile_dir
         self.page = None
 
+    @property
+    def profile_dir(self) -> str | None:
+        return self._profile
+
     def __enter__(self) -> BrowserSession:
-        self._profile = tempfile.mkdtemp(prefix="gbs-profile-")
+        if self._profile is None:
+            self._profile = tempfile.mkdtemp(prefix="gbs-profile-")
         try:
-            self._context = self._launch(self._profile)
+            self._open()
         except Exception:
             self.close()
             raise
+        return self
+
+    def _open(self) -> None:
+        self._context = self._launch(self._profile)
         pages = self._context.pages
         self.page = pages[0] if pages else self._context.new_page()
-        return self
+
+    def suspend(self) -> str:
+        """Close the browser and keep its profile on disk, so it can be
+        copied; `resume()` opens it again on the same proxy."""
+        self._shut()
+        return self._profile
+
+    def resume(self) -> None:
+        self._open()
 
     def _launch(self, profile: str):
         from patchright.sync_api import sync_playwright
@@ -111,6 +145,12 @@ class BrowserSession:
         self.close()
 
     def close(self) -> None:
+        self._shut()
+        if self._profile:
+            shutil.rmtree(self._profile, ignore_errors=True)
+        self._profile = None
+
+    def _shut(self) -> None:
         for step in (
             lambda: self._context and self._context.close(),
             lambda: self._pw and self._pw.stop(),
@@ -119,9 +159,7 @@ class BrowserSession:
                 step()
             except Exception:
                 pass
-        if self._profile:
-            shutil.rmtree(self._profile, ignore_errors=True)
-        self._context = self._pw = self._profile = self.page = None
+        self._context = self._pw = self.page = None
 
     # -- warming -------------------------------------------------------------
 
@@ -136,55 +174,131 @@ class BrowserSession:
 
     # -- searching -----------------------------------------------------------
 
-    def search(self, query: str) -> Fetched:
-        """Type `query` into Google's box and return the page it led to."""
+    def search(self, query: str, direct: bool = False) -> Fetched:
+        """Type `query` into Google's box and return the page it led to; with
+        `direct`, open the results page for it instead."""
+        if direct:
+            return self._search_direct(query)
         page = self.page
-        if not any(h.is_visible() for h in page.query_selector_all(SEARCH_BOX)):
+        timings: dict[str, float] = {}
+        mark = time.monotonic()
+        if not self._box_visible():
             page.goto(HOME_URL, wait_until="domcontentloaded", timeout=NAV_TIMEOUT_MS)
+            timings["home"] = round(time.monotonic() - mark, 2)
+            mark = time.monotonic()
+        dismissed, typed = False, None
+        # The page can take focus or swap the box while the query is being
+        # typed, which leaves only its first characters behind. The box is
+        # found again and the query retyped once before giving up.
+        for _ in range(TYPE_ATTEMPTS):
+            dismissed = self._dismiss_consent() or dismissed
+            box = self._type_query(query)
+            try:
+                typed = box.input_value()
+            except Exception:
+                typed = None  # the box was replaced and this handle is gone
+            if typed == query:
+                break
+        else:
+            raise RuntimeError(
+                f"search box holds {typed!r} after typing {query!r} "
+                f"{TYPE_ATTEMPTS} times; not submitting"
+            )
+        timings["type"] = round(time.monotonic() - mark, 2)
+        mark = time.monotonic()
+        # `no_wait_after=True` keeps `press` from waiting for the navigation a
+        # second time with Playwright's own 30 s default under ours.
+        with page.expect_navigation(wait_until="domcontentloaded", timeout=NAV_TIMEOUT_MS) as nav:
+            box.press("Enter", no_wait_after=True, timeout=NAV_TIMEOUT_MS)
+        timings["load"] = round(time.monotonic() - mark, 2)
+        fetched = self._snapshot(nav.value, dismissed)
+        return replace(fetched, timings={**timings, **fetched.timings})
+
+    def _search_direct(self, query: str) -> Fetched:
+        home = urlsplit(HOME_URL)
+        url = urlunsplit(
+            (home.scheme, home.netloc, "/search", urlencode({"q": query, "hl": "en"}), "")
+        )
+        mark = time.monotonic()
+        response = self.page.goto(url, wait_until="domcontentloaded", timeout=NAV_TIMEOUT_MS)
+        # A fresh exit can be shown the consent wall instead of results; clear
+        # it and open the results again, as the typed path does on the front page.
         dismissed = self._dismiss_consent()
+        if dismissed:
+            response = self.page.goto(url, wait_until="domcontentloaded", timeout=NAV_TIMEOUT_MS)
+        load = round(time.monotonic() - mark, 2)
+        fetched = self._snapshot(response, dismissed)
+        return replace(fetched, timings={"load": load, **fetched.timings})
+
+    def _box_visible(self) -> bool:
+        try:
+            return any(h.is_visible() for h in self.page.query_selector_all(SEARCH_BOX))
+        except Exception:
+            # The page navigated while it was being read, as a warm-up page
+            # left idle can; the front page is loaded instead.
+            return False
+
+    def _type_query(self, query: str):
+        page = self.page
         # Act on the handle that was found, never on the selector again: the
         # selector also matches a hidden field, and `page.click` would pick it.
         box = page.wait_for_selector(SEARCH_BOX, state="visible", timeout=NAV_TIMEOUT_MS)
         box.click()
-        # A results page keeps the previous query in the box. Typing after it
-        # appends, and the appended query returns a real page for the wrong
-        # question. Select it so the typing replaces it.
+        # A results page keeps the previous query in the box, and a failed
+        # attempt leaves part of this one. Typing after either appends, and the
+        # appended query returns a real page for the wrong question. Select it
+        # so the typing replaces it.
         if box.input_value():
             box.press("ControlOrMeta+a")
-        box.type(query, delay=self._rng.randint(45, 140))
-        typed = box.input_value()
-        if typed != query:
-            raise RuntimeError(f"search box holds {typed!r} after typing {query!r}; not submitting")
-        # `no_wait_after=True` keeps `press` from waiting for the navigation a
-        # second time with Playwright's own 30 s default under ours.
-        with page.expect_navigation(wait_until="domcontentloaded", timeout=NAV_TIMEOUT_MS) as nav:
-            box.press("Enter", no_wait_after=True, timeout=NAV_TIMEOUT_MS)
-        return self._snapshot(nav.value, dismissed)
+        if self._input == "insert":
+            page.keyboard.insert_text(query)
+        elif self._input == "words":
+            words = query.split(" ")
+            for i, word in enumerate(words):
+                last = i == len(words) - 1
+                page.keyboard.insert_text(word if last else word + " ")
+                if not last:
+                    time.sleep(self._rng.uniform(0.15, 0.5))
+        else:
+            box.type(query, delay=self._rng.randint(45, 140))
+        return box
+
+    def blank(self) -> None:
+        """Leave the current page, stopping whatever it is still loading."""
+        self.page.goto("about:blank")
 
     def next_page(self) -> Fetched | None:
         """Click "Next" on the current results page, if there is one."""
         link = self.page.query_selector("a#pnnext")
         if link is None or not link.is_visible():
             return None
+        mark = time.monotonic()
         with self.page.expect_navigation(
             wait_until="domcontentloaded", timeout=NAV_TIMEOUT_MS
         ) as nav:
             link.click()
-        return self._snapshot(nav.value, False)
+        load = round(time.monotonic() - mark, 2)
+        fetched = self._snapshot(nav.value, False)
+        return replace(fetched, timings={"load": load, **fetched.timings})
 
     def _snapshot(self, response, dismissed: bool) -> Fetched:
         # Google builds results in the browser; reading at domcontentloaded
         # catches the "enable JavaScript" scaffold. A refusal never grows the
         # container, so the short timeout is expected to expire on those.
+        mark = time.monotonic()
         try:
             self.page.wait_for_selector(READY, state="attached", timeout=8_000)
         except Exception:
             pass
+        ready = round(time.monotonic() - mark, 2)
+        mark = time.monotonic()
+        html = self.page.content()
         return Fetched(
             url=self.page.url,
             status=response.status if response is not None else None,
-            html=self.page.content(),
+            html=html,
             consent_dismissed=dismissed,
+            timings={"ready": ready, "read": round(time.monotonic() - mark, 2)},
         )
 
     def _dismiss_consent(self) -> bool: