obxcura 0.3.0 → 0.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: ecefba7c3040b8f7659fceb7de9f9b3c36903ee3b370dbc43193362047edecb5
-  data.tar.gz: 84de6d7107dc9d147ba3f24ba2dc2e5567cd507fb202919a672d1b1cf01a410a
+  metadata.gz: 7f74cd065bb29071f0bc6468f6b1809c6b900f9c563b11a72afa9ed905b8ae3d
+  data.tar.gz: 2a5894fd3d306f10171e7f955e7142375b8354efe528e06d72816b6e44062e82
 SHA512:
-  metadata.gz: 554a2a92cfc7e93dc9d3a15bd3e9bf5c04751a03e9c933c8c291811043dae050f63246c7aa27c5c319d6d536bc4f78a6ea9282bf11c8328c26674aa614817ba6
-  data.tar.gz: 897e5a2a9ce08f4c3dd4d2a9f8861022e8f333b940c89d30f5ccea9bd3a5610b8f3b601b3a3682f739a678b782886b21c413481dc293adf1e7f71dace295c449
+  metadata.gz: 5997307fc7e40606dfeb0f086b66ea2073dd5730be22b787fe0dff226c3a10f1873aae498544161502d37d45dcb50e374ef7cbb008ffe3e3807f9ccea5b192ca
+  data.tar.gz: b10406aa827442b519f41b7f130ad51464776edf3c31fb3bc2d7172e49f771b9043bc7309ae2135ec0ee671c0ce7c90dad5919ee902edc6a93ad41232341ddf8

data/CHANGELOG.md

@@ -1,5 +1,160 @@
 ## [Unreleased]
 
+## [0.4.0] - 2026-08-18
+
+Catches the client up with Obscura 0.2.0 — which grew a native render engine —
+and gives extra headers and cookies proper read/write collections instead of raw
+CDP calls. As always, every behaviour below was measured against the binary
+rather than taken from the protocol docs, and the places where Obscura departs
+from Chrome are documented instead of papered over: cookie reads that ignore the
+URL you ask about, deletes that compare the path exactly, expired cookies that
+linger in the jar, and a PDF engine that draws text rather than embedding it.
+
+One breaking change, in `Page#cookies` — see below.
+
+Also fixes a way to kill the browser outright from the public API.
+
+### Changed
+
+- **`Page#cookies` returns a collection, not an `Array`.** It used to hand back
+  the connection's whole jar as an array of hashes; it now returns an
+  `Obxcura::Cookies`, enumerable over the cookies the page's URL would actually
+  send. `#map`, `#select`, `#find` and friends keep working, and the old value is
+  `page.cookies.all`.
+
+  What breaks quietly is integer indexing: `page.cookies[0]` used to be the first
+  cookie hash and now looks up the cookie *named* `"0"`, so it returns `nil` —
+  and `page.cookies[0]["value"]` becomes a `NoMethodError` on nil. Reach for
+  `page.cookies.to_a[0]`, or better, `page.cookies["session"]`.
+
+- CI pins Obscura `v0.2.0` and now installs the **render** build, since the
+  screenshot specs would otherwise skip rather than fail. It also triggered on
+  pushes to `master` while the default branch is `main`, so the push trigger
+  could never fire; both are fixed.
+
+### Added
+
+- **`Page#cookies`** — the browser's cookie jar seen from one page: `Enumerable`
+  over the cookies that page's URL would actually send, and writable. `#all`,
+  `#[]`, `#for_url`, `#size`, `#empty?` for reading; `#set`, `#remove`, `#clear`
+  for writing. Cookies stay raw CDP hashes with string keys, so what comes out of
+  a read goes straight back into `#set` — `fresh.cookies.set(saved)` moves a
+  logged-in session into another browser. Includes the `HttpOnly` cookies
+  `document.cookie` cannot see, which is the whole reason to read them over CDP.
+
+  `#set` takes `url:`, `domain:`, `path:`, `secure:`, `http_only:`, `same_site:`
+  and `expires:` (a `Time` or epoch seconds), defaulting to the page's current
+  URL because the browser refuses a cookie that belongs nowhere. Unknown options
+  and a bad `same_site:` raise `ArgumentError` rather than reaching the browser,
+  which accepts both silently — a bogus `sameSite` is stored as `Lax` without
+  complaint.
+
+  Four measured browser behaviours shaped this, all documented in
+  [`docs/cookies.md`](docs/cookies.md):
+
+  - **Cookie reads ignore the URL you ask about.** `Network.getCookies` accepts a
+    `urls:` parameter and Obscura ignores it — that call, `Storage.getCookies`
+    and `Network.getAllCookies` all return the whole connection-wide jar, from
+    any page's session. So the scoping is done in Ruby, following RFC 6265
+    §5.1.3–5.1.4: domain, path, `Secure`, expiry. Ports are not part of cookie
+    scope.
+  - **An expired cookie stays in the jar** and is simply never sent, so `#all`
+    can show what will never go on the wire while the scoped view drops it.
+  - **`Network.deleteCookies` compares the path exactly** — a delete aimed at
+    `/cookies` leaves a cookie set on `/` alone, silently. `#remove` therefore
+    looks the cookie up first and deletes at the domain and path the jar reports,
+    and returns whether the jar actually changed.
+  - **Host-only cookies cannot be told apart.** Chrome stores `Domain=example.com`
+    as `.example.com`; Obscura stores it verbatim, leaving nothing to distinguish
+    it from a host-only cookie. Domain matching is applied to every entry — over
+    reporting beats losing the session cookie you came for.
+
+- **`Page#headers`** — a small mutable collection for extra HTTP headers, in the
+  shape Ferrum uses: `#get`/`#to_h`, `#set`, `#add`, `#clear`, plus `#[]`,
+  `#[]=`, `#delete`, `#key?`, `#empty?` and `#size`. Names match
+  case-insensitively per RFC 9110, so writing `x-token` after `X-Token` replaces
+  it instead of sending both. Names and values are stringified, as CDP requires.
+
+  `#get` reports what the object last sent, because that is the only readable
+  record there is: CDP has no getter, and Obscura answers
+  `Network.getExtraHTTPHeaders` with `Unknown Network method`.
+
+  Two measured behaviours the documentation calls out rather than papering over.
+  **The scope is the connection, not the page** — the API hangs off `Page`
+  because that is the only session the command works from (sent without one it is
+  accepted and silently does nothing), but Obscura keeps one table per
+  connection, so two pages on a browser cannot carry different headers and the
+  last write wins. And **they cover navigation only**: script-initiated requests
+  never carry them, so `#post` still needs its own `headers` argument.
+
+- **`Page#pdf`** — print the page, now that Obscura 0.2.0 can. Returns the raw
+  PDF bytes, or writes them and returns the path with `path:`. Supports
+  `landscape:`, `print_background:`, `scale:` (0.1..2.0), `page_ranges:`,
+  `margin:` (one number or per side, in inches), and paper as either a name
+  (`:letter`, `:legal`, `:tabloid`, `:a3`, `:a4`, `:a5`) or explicit
+  `paper_width:`/`paper_height:` in inches.
+
+  Verified against the binary by reading the generated PDFs, not by trusting
+  byte sizes — an A4 render of the test page weighs *exactly* as much as the
+  Letter one, so only the `/MediaBox` reveals that paper size applied at all
+  (612x792pt vs 595x842pt). Landscape swaps the box, `scale:` changes the page
+  count, and `page_ranges:` trims it.
+
+  **Output is raster-backed.** Obscura reports `print-media-raster`, and a
+  generated document has zero font objects and no extractable text — measured.
+  It cannot be searched, selected, or read by a screen reader; use `#html` when
+  you need the text. Header/footer rendering and CSS `@page` sizing are not
+  implemented upstream, so those options are deliberately not exposed.
+
+  Needs a build with the render feature, like `#screenshot`; the same refusal is
+  mapped to an `Obxcura::Error` naming the asset to install. An unknown paper
+  name, `paper:` combined with explicit dimensions, and a `scale:` outside
+  0.1..2.0 all raise `ArgumentError` before any CDP round trip.
+
+- **`Page#screenshot`** — real rasterisation, now that Obscura 0.2.0 ships a
+  native render engine. Returns the raw image bytes (base64 is a CDP transport
+  detail and is decoded here), or writes them and returns the path when given
+  `path:`. Supports `:png` / `:jpeg` / `:webp`, `quality:` for `:jpeg`,
+  `full_page:` for the whole document, and `clip:` for a region with optional
+  `scale:`. Format is inferred from the `path:` extension when not given.
+
+  Every option was verified against the 0.2.0 binary: `full_page:` really does
+  capture past the viewport (1280x720 → 1280x2400 on the test page), `clip:` with
+  `scale: 2` really doubles the raster, and `quality:` really changes the jpeg
+  encode. Obscura's webp encoder is lossless and rejects `quality:` at any value,
+  so that combination is refused client-side rather than round-tripped.
+
+  Requires a build carrying the render feature. The `-no-render` archives of the
+  same version refuse `Page.captureScreenshot`; that refusal surfaces as an
+  `Obxcura::Error` naming the asset to install, not a bare `ProtocolError`.
+  Unsupported formats and contradictory options (`full_page:` with `clip:`) raise
+  `ArgumentError` before any CDP round trip.
+
+### Fixed
+
+- **`Browser#create_page(url)` no longer kills the browser.** Handing a URL to
+  `Target.createTarget` works exactly once: the *second* such call on a
+  connection takes `obscura serve` down outright — every command after it fails
+  with `connection closed: end of file reached`, and the process is gone, not
+  just the socket.
+
+  ```ruby
+  browser.create_page("https://www.google.com.mx")   # fine
+  browser.create_page("https://example.com")         # browser dies here
+  ```
+
+  Measured on 0.2.0 and fully deterministic, whatever the URLs are — two local
+  ones crash it just the same. One URL-loaded target is safe, any number of
+  `about:blank` targets is safe, navigation is safe, and two URL-loaded targets
+  on *separate* connections are safe. It is specifically the second
+  `createTarget` carrying a URL.
+
+  `#create_page` now always creates the target blank and navigates, so the
+  parameter is honoured without handing callers a way to kill the browser. It
+  blocks until the load event fires, which makes `#go_to` simply its expressive
+  name. The workaround note on `#go_to` was also slightly wrong: the crash needs
+  no `evaluate`, it lands on the `create_page` call itself.
+
 ## [0.3.0] - 2026-07-29
 
 Realigns the client with Obscura 0.1.11, which lifted several of the browser

data/README.md

@@ -15,143 +15,122 @@ own attached session. One connection, many pages.
 bundle add obxcura
 ```
 
-You also need the `obscura` binary on your `PATH`
-([releases](https://github.com/h4ckf0r0day/obscura/releases)).
-
-## Usage
-
-Start the browser first (defaults to port 9222):
+You also need the **`obscura` browser binary**, which is a separate project. The
+repo ships an installer for it:
 
 ```bash
-obscura serve
+scripts/obscura/install.sh
 ```
 
-Then:
-
-```ruby
-require "obxcura"
+That pulls the latest release for your platform into `/usr/local/bin` (asking for
+`sudo` only if the prefix is not writable), then verifies the install and prints
+the checksums.
 
-browser = Obxcura::Browser.new     # or Obxcura.start
-page    = browser.go_to("https://example.com")
+```bash
+scripts/obscura/install.sh --version v0.2.0      # pin a release
+scripts/obscura/install.sh --prefix ~/.local/bin # no sudo
+scripts/obscura/install.sh --no-stealth          # plain build
+scripts/obscura/install.sh --no-render           # smaller, no screenshot/pdf
+scripts/obscura/install.sh --help
+```
 
-page.html                          # rendered DOM, post-JS
-page.title                         # "Example Domain"
-page.current_url                   # "https://example.com/"
-page.evaluate("1 + 2")             # => 3
+Prefer it to a manual download, because the packaging has sharp edges:
+
+- **Since 0.2.0 there are four builds per platform**, and the suffix decides what
+  works:
+
+  | Asset suffix | Render engine | Stealth |
+  |---|---|---|
+  | *(none)* | yes | no |
+  | `-stealth` | yes | yes |
+  | `-no-render` | no | no |
+  | `-no-render-stealth` | no | yes |
+
+  The script defaults to **`-stealth`**: it carries the render engine that
+  `#screenshot` and `#pdf` need, *and* it is the build that gets past commercial
+  bot detection. `--no-render` trades both away for a smaller download.
+- **`--version` cannot tell the builds apart** — all four of a release report the
+  same string. That is why the script prints checksums: they are the only way to
+  identify later which build is actually installed.
+- **The archive holds two binaries**, `obscura` and `obscura-worker`, and `serve`
+  looks for the worker as a *sibling*. The script always moves the pair, since
+  installing one alone silently mixes builds.
+- **On macOS it removes the target before copying.** Overwriting a binary in
+  place leaves a stale cached code signature on that inode, and the kernel then
+  `SIGKILL`s it on exec — even though the bytes are fine.
+
+Driving anything on localhost also needs `--allow-private-network`, which is off
+by default:
 
-browser.quit
+```bash
+obscura serve --allow-private-network
 ```
 
-### Querying the DOM
+## Usage
 
-`#at_css` / `#css` return live `Obxcura::Node` handles backed by the real DOM:
+Start the browser first (defaults to port 9222):
 
-```ruby
-page.at_css("h1").text             # => "Example Domain"
-page.at_css("a")["href"]           # attribute value, or nil
-page.at_css("h1").outer_html       # "<h1>Example Domain</h1>"
-page.css("p").map(&:text)          # every <p>'s text
-page.at_css("#missing")            # => nil
+```bash
+obscura serve
 ```
 
-### Running JavaScript
-
-Pass Ruby values as arguments — they cross into the page as real values
-(`arguments[0]`, `arguments[1]`, …), never string-interpolated into source:
+Then:
 
 ```ruby
-page.evaluate("arguments[0] * 2", 21)          # => 42
+require "obxcura"
 
-# `evaluate_func` calls a function declaration directly:
-page.evaluate_func(<<~JS, "h1")
-  function(sel) { return document.querySelector(sel).textContent; }
-JS
-```
+browser = Obxcura::Browser.new              # or Obxcura.start
+begin
+  page = browser.go_to("https://example.com")
 
-### POSTing from inside the page
+  page.title                                # => "Example Domain"
+  page.current_url                          # => "https://example.com/"
+  page.html                                 # rendered DOM, post-JS
+  page.evaluate("1 + 2")                    # => 3
 
-`#post` runs the POST from the page's context with `fetch`, reusing its cookies.
-Values cross as arguments:
+  page.at_css("h1").text                    # => "Example Domain"
+  page.css("a").map { |a| a["href"] }
 
-```ruby
-result = page.post(
-  "https://example.com/api/login",
-  URI.encode_www_form(user: "me", pass: "secret"),   # payload
-  "application/x-www-form-urlencoded",                # content type
-  { "X-Requested-With" => "XMLHttpRequest" }          # headers
-)
-
-result["status"]   # => 200
-result["ok"]       # => true  (2xx)
-result["body"]     # response text
+  page.at_css("#username").type("guillermo")
+  page.at_css("form").submit
+
+  page.screenshot(path: "shot.png")
+  page.pdf(path: "page.pdf", paper: :a4)
+ensure
+  browser.quit                              # always, so targets don't leak
+end
 ```
 
-Any HTTP reply (including 4xx/5xx) comes back as `{ "status", "ok", "body" }`.
-A transport failure — the request never reached the server (CORS, the
-private-network SSRF guard, mixed origin, a dead host) — raises
-`Obxcura::ConnectionError`. If the server accepts the connection but never
-answers, the wait ends with `Obxcura::TimeoutError`; pass `timeout:` (seconds)
-to bound it, and the request is dropped in the page at roughly that mark. Some
-anti-bot endpoints tarpit non-stealth clients — try `obscura serve --stealth`.
-
-> Before Obscura 0.1.11 this had to use `XMLHttpRequest`, because `fetch` wasn't
-> routed. The old name `#xhr_post` still works as a deprecated alias.
-
-## API
-
-`Obxcura.start(**opts)` is sugar for `Obxcura::Browser.new`.
-
-- **`Obxcura::Browser`** — `.new(host:, port:, timeout:)`, `#create_page(url)`,
-  `#go_to`/`#goto`, `#targets`, `#version`, `#clear_cookies`, `#command`,
-  `#close`/`#quit`. Readers: `#client`, `#pages`, `#host`, `#port`.
-- **`Obxcura::Page`** — `#goto`/`#go_to`, `#evaluate`, `#evaluate_func`,
-  `#html`/`#body`, `#title`, `#current_url`, `#at_css`, `#css`, `#post`,
-  `#cookies`, `#network_log`, `#refresh`/`#reload`, `#command`, `#close`,
-  `#close_connection`. Readers: `#frame`, `#target_id`, `#session_id`, `#client`.
-- **`Obxcura::Node`** (from `#at_css`/`#css`) — `#text`, `#value`, `#[]`
-  (attribute), `#at_css`, `#focus`, `#type`, `#submit`, `#outer_html`.
-- **`Obxcura::Frame`** — the main frame behind a Page; carries the DOM/Runtime
-  methods Page delegates (`#evaluate`, `#at_css`, `#call_on`, …).
-- **`Obxcura::Client`** — the CDP transport, if you need raw `#command`,
-  `#subscribe` / `#unsubscribe`, or `#close`.
-
-Errors all descend from `Obxcura::Error`: `TimeoutError`, `ProtocolError`,
-`ConnectionError`.
-
-## Obscura's quirks (and how Obxcura handles them)
-
-These are properties of the **Obscura browser**, not of this gem. They shape the
-API, so they're worth knowing:
-
-- **No paint engine.** There is no screenshot API, and there never will be one
-  here. `Page.captureScreenshot` is unusable.
-- **DOM nodes don't serialize.** A node returned by value comes back as an
-  internal stub, so `#at_css` / `#css` resolve it (via `DOM.resolveNode`) to a
-  live handle and read it with `Runtime.callFunctionOn`.
-- **In-page throws vanish.** A JS `throw` comes back as `undefined` rather than
-  an exception, so the gem returns `{ error: ... }` sentinels and raises from
-  Ruby. It's also why `#post` bounds itself by racing a timer instead of using
-  `AbortSignal.timeout` — the abort works, but its rejection doesn't survive the
-  trip, so the reason has to come back as a resolved value.
-- **The network log only sees navigation.** `#network_log` records the document
-  and its subresources. Script-initiated requests — including `#post` — emit no
-  CDP network events at all, so they never appear.
-- **No bulk text insertion.** `Input.insertText` isn't implemented, so `#type`
-  dispatches one key event per character via `Input.dispatchKeyEvent`. Real
-  `keydown`/`input`/`keyup` fire; `change` does not, matching browsers, which
-  only fire it on blur.
-- **Private networks are blocked by default.** To drive a local site, start the
-  browser with `obscura serve --allow-private-network`.
-
-Two long-standing limits were lifted in Obscura 0.1.11, so if you've read older
-notes: the **~500–700 KB message ceiling is gone** (64 MiB now, so `#html`
-returns even a huge document in one round trip), and **cookies no longer leak
-between connections** — each `Browser` gets its own browser context, and
-`#clear_cookies` is now only about resetting state within one connection.
-
-The transport is built directly on `websocket-driver` rather than
-`websocket-client-simple`, which reads a byte at a time and wedges on large
-frames.
+That is the whole shape of it: a `Browser` owns the connection, a `Page` is a
+target on it, and `#at_css` / `#css` hand back live `Node` handles you can read
+from and act on.
+
+## Documentation
+
+Full reference in [`docs/`](docs/README.md), one page per area:
+
+| | |
+|---|---|
+| [Connecting](docs/connecting.md) | `Obxcura.start`, `Browser.new`, ports, timeouts, shutdown |
+| [Pages](docs/pages.md) | `#goto`, `#refresh`, `#close`, raw `#command` |
+| [Reading content](docs/content.md) | `#html`, `#title`, `#current_url` |
+| [Querying the DOM](docs/dom.md) | `#at_css`, `#css`, and every `Node` method |
+| [Running JavaScript](docs/javascript.md) | `#evaluate`, `#evaluate_func`, `#call_on` |
+| [Forms and input](docs/forms.md) | `#focus`, `#type`, `#submit`, `#value` |
+| [HTTP from the page](docs/http.md) | `#post`, `#network_log` |
+| [Cookies](docs/cookies.md) | `#cookies` — read, set, remove, replay a session |
+| [Headers](docs/headers.md) | `#headers` — read and set extra HTTP headers |
+| [Screenshots](docs/screenshots.md) | `#screenshot` and every option |
+| [PDF](docs/pdf.md) | `#pdf` and every option |
+| [Errors](docs/errors.md) | The exception hierarchy, and what does *not* raise |
+| [Browser constraints](docs/constraints.md) | Obscura limits that shape this API |
+
+Runnable examples live in [`examples/`](examples/).
+
+Two constraints are worth knowing before you write anything, both covered in
+[Browser constraints](docs/constraints.md): Obscura stops running background
+JavaScript shortly after `load`, so `goto` + `sleep` does not do what you expect;
+and an in-page `throw` comes back as `nil` rather than raising.
 
 ## Development
 
@@ -170,6 +149,9 @@ OBSCURA_BIN=/path/to/obscura bundle exec rspec
 ```
 
 Without the binary those specs skip cleanly, so `bundle exec rspec` still passes.
+The `#screenshot` and `#pdf` specs skip the same way on a `-no-render` build, so
+run them against a build that has the render engine — either the default from
+`scripts/obscura/install.sh` or the unsuffixed asset — if you touch either.
 
 ## Contributing
 

data/lib/obxcura/browser.rb

@@ -18,6 +18,10 @@ module Obxcura
     DEFAULT_HOST = "127.0.0.1"
     # @return [Integer] default CDP port for `obscura serve`.
     DEFAULT_PORT = 9222
+    # The only URL a target may safely be created at; see {#create_page}.
+    #
+    # @return [String]
+    BLANK_PAGE = "about:blank"
 
     # @return [Obxcura::Client] the underlying CDP transport.
     # @return [Array<Obxcura::Page>] the pages currently open.
@@ -41,29 +45,38 @@ module Obxcura
       @client = Client.new(browser_ws_url, timeout: @timeout)
     end
 
-    # Open a fresh page (a CDP target) and attach to it.
+    # Open a fresh page (a CDP target) and attach to it. With a `url`, the page
+    # is navigated there and the call blocks until its load event fires.
+    #
+    # The target is always created blank and then navigated, never opened at
+    # `url` directly. Handing a URL to `Target.createTarget` works exactly once:
+    # the *second* such call on a connection kills `obscura serve` outright —
+    # every command after it fails with `connection closed: end of file
+    # reached`, and the process is gone, not just the socket. Measured on 0.2.0
+    # and deterministic, whatever the URLs are; two blank targets are fine, and
+    # so is any amount of navigation. Since the parameter cannot be honoured
+    # literally without handing callers a way to kill the browser, it is honoured
+    # by navigating.
     #
-    # @param url [String] URL to open the target at (defaults to a blank page).
+    # @param url [String] URL to open the page at (defaults to a blank page).
     # @return [Obxcura::Page] the new, tracked page.
-    def create_page(url = "about:blank")
-      target_id = command("Target.createTarget", { url: url })["targetId"]
+    def create_page(url = BLANK_PAGE)
+      target_id = command("Target.createTarget", { url: BLANK_PAGE })["targetId"]
       session_id = command("Target.attachToTarget", { targetId: target_id, flatten: true })["sessionId"]
 
       page = Page.new(self, target_id: target_id, session_id: session_id)
       @pages << page
+      page.goto(url) unless url == BLANK_PAGE
       page
     end
 
-    # Open a page and navigate to `url` in one call.
-    #
-    # Navigates from a blank target rather than passing the URL straight to
-    # Target.createTarget: creating two URL-loaded targets and then evaluating
-    # crashes `obscura serve` (connection closed: end of file reached).
+    # Open a page and navigate to `url` in one call — the expressive name for
+    # {#create_page} with a URL.
     #
     # @param url [String] URL to navigate to.
     # @return [Obxcura::Page] the navigated page (load event fired).
     def go_to(url)
-      create_page.goto(url)
+      create_page(url)
     end
     alias goto go_to