unmagic-browser 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 1f442f73a1d9529c04c844c02cae4d3baeef123ae4ed73abe666f2de077d555c
+  data.tar.gz: b3b0f32e366f5c361fff8f883ace120af199b30d0ac8c6354d42993fe5a1f9e8
+SHA512:
+  metadata.gz: 0bcd7c33a2504e7f91e565ae10099f040eb2f764f29cafa5d8cd7f890ca6d2241031a7ee820b0bc530766d94d4ba173ecbdc3afd660a55bef813dfb4f4663d6b
+  data.tar.gz: 916197bf8151333633b6a7a66682393dda30ccd01cc321ba330b9cf4ba6ad27b33b047821d6c20cbc8c1dbbaf2dc0ac1067242293040fa92aacab04f25f262eb

data/CHANGELOG.md

@@ -0,0 +1,51 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+## [0.1.0]
+
+### Added
+- **One-off page grabs** — `markdown`, `html`, `screenshot` and `pdf` straight on
+  the browser, each taking the cheapest path its driver offers: Cloudflare's REST
+  rendering API where there is one, a throwaway page on a shared browser otherwise.
+- **Sessions** — `browser.open(url)` for a live page driven step by step, with
+  Capybara-flavoured verbs (`fill_in`, `click_button`, `select`, `check`,
+  `within`, `wait_for`, …) that auto-wait for their element.
+- **Smart locators** — a plain string is resolved by label, then visible text,
+  then ARIA role, falling back to CSS, so a caller that has never seen the page's
+  markup can still drive it. `css:`, `text:`, `label:` and `role:` name a
+  strategy explicitly.
+- **Sessions that outlive one call** — `open` without a block hands the session
+  back to keep, so an agent can open a page in one turn and act on it in the next.
+  Operations are serialised, and `#lock` makes a whole sequence atomic.
+- **Emulation** — colour scheme, reduced motion, viewport, device, user agent,
+  locale, timezone, geolocation and media type in one `emulate:` config, settable
+  when a session opens, live with `session.emulate`, or scoped to a block. What
+  you leave out renders at a fixed default rather than at the host machine's
+  preference, so a page looks the same on a laptop and on CI.
+- **Three drivers** — `Local` (Chrome on this machine, `headless: false` to watch
+  it), `Remote` (any CDP websocket — browserless, your own pool) and `Cloudflare`
+  (Browser Rendering, so no browser runs on your servers).
+- **Typed errors** — `TimedOut`, `Unreachable`, `RenderFailed`, `RateLimited`,
+  `Misconfigured`, `NotFound` and `SessionClosed`, so failures are classified by
+  rescuing rather than by parsing a message.
+- **An escape hatch** — `browser.driver` and `session.driver` hand back the
+  underlying driver and the live `Puppeteer::Page` for anything the curated API
+  doesn't wrap.
+- **A console** — `require "unmagic/browser/console"` and
+  `Unmagic::Browser::Console.start` for a browser you can type at: the verbs as
+  bare words (`open "https://example.com"; click "Sign in"; screenshot`) driving
+  one current session. Results are rendered rather than inspected — source
+  syntax-highlighted with [rouge](https://github.com/rouge-ruby/rouge),
+  screenshots drawn in the terminal over the [kitty graphics
+  protocol](https://sw.kovidgoyal.net/kitty/graphics-protocol/) at half its
+  width (`Graphics.scale` to change that) and written full-size to /tmp,
+  sessions and elements summarised. `start(prompt: :page)` for a prompt that
+  says what's open — a green dot and the page's title — instead of `browser>`.
+  Not required by default, and neither rouge nor a graphics-capable terminal is
+  a dependency — without them it runs monochrome and prints paths.

data/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Keith Pitt
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,372 @@
+# Unmagic::Browser
+
+A real browser you can drive from Ruby — for one-off page grabs *and* multi-step
+automation. It runs locally in development and against [Cloudflare Browser
+Rendering](https://developers.cloudflare.com/browser-rendering/) (or any remote Puppeteer
+endpoint) in production, behind one API. Built so an LLM agent — or plain Ruby code — can
+browse the web the way a person does.
+
+```ruby
+browser = Unmagic::Browser.new
+
+# one-off: grab a JS-rendered page as Markdown
+browser.markdown("https://example.com")
+
+# multi-step: log in, search, read the result
+browser.open("https://news.ycombinator.com/login") do |session|
+  session.fill_in "acct", with: "pg"
+  session.fill_in "pw",   with: ENV["HN_PASSWORD"]
+  session.click_button "login"
+  session.wait_for text: "logout"
+  session.screenshot
+end
+```
+
+## Installation
+
+```ruby
+gem "unmagic-browser"
+```
+
+Bundler requires it for you. Outside Bundler, the gem's name maps to its path in the usual
+way:
+
+```ruby
+require "unmagic/browser"
+```
+
+## Drivers
+
+The same API runs against three drivers. Pick one when you construct the browser; nothing
+else in your code changes. A symbol builds the driver with its defaults — when a driver
+needs options, construct it yourself and pass the instance.
+
+```ruby
+# A symbol builds the driver with no arguments…
+Unmagic::Browser.new(driver: :local)      # same as driver: Unmagic::Browser::Driver::Local.new
+Unmagic::Browser.new(driver: :remote)
+Unmagic::Browser.new(driver: :cloudflare)
+
+# …pass an instance to hand a driver options.
+
+# Local Chrome — great for development. headless: false opens a window you can watch.
+Unmagic::Browser.new(driver: Unmagic::Browser::Driver::Local.new(headless: false))
+
+# Any remote Puppeteer / CDP websocket — browserless, your own pool, etc.
+Unmagic::Browser.new(driver: Unmagic::Browser::Driver::Remote.new(ws: "ws://browserless.internal:3000"))
+
+# Cloudflare Browser Rendering — no browser runs on your servers.
+Unmagic::Browser.new(driver: Unmagic::Browser::Driver::Cloudflare.new(
+  account_id: ENV["CF_ACCOUNT_ID"],
+  token:      ENV["CF_BROWSER_TOKEN"],
+))
+```
+
+With no arguments, `Unmagic::Browser.new` uses `:local` in development and the configured
+[`default_driver`](#configuration) everywhere else.
+
+## One-off page grabs
+
+For "just get me this page", call straight on the browser. Each call takes the cheapest path
+its driver offers: the REST rendering API on Cloudflare (no browser to hold open), a
+throwaway page on local and remote.
+
+```ruby
+browser.markdown("https://example.com")              # => String  (clean text)
+browser.html("https://example.com")                  # => String  (fully JS-rendered HTML)
+browser.screenshot("https://example.com")            # => PNG bytes
+browser.screenshot("https://example.com",
+                   full_page: true,                  # the whole scrollable page
+                   selector:  "#main")               # or a single element
+browser.pdf("https://example.com")                   # => PDF bytes
+```
+
+One-offs always render with default settings. To capture a page in dark mode, on a phone
+viewport, or in another locale, open a session and use
+[emulation](#emulation-dark-mode-viewport-devices-locale).
+
+On local and remote the throwaway pages share one browser, started on first use and shut
+down when the process exits. `browser.close` lets it go sooner; sessions are unaffected,
+because each holds its own.
+
+## Sessions
+
+A **session** is a live browser page you drive step by step. `browser.open` works like
+`File.open`: give it a URL and a block, and you get a session — already on that page —
+that's cleaned up when the block ends.
+
+```ruby
+browser.open("https://store.example.com") do |session|
+  session.fill_in "Search", with: "espresso"
+  session.click_button "Go"
+  session.wait_for text: "results"
+
+  session.all(".product").map { |el| el.text }
+end
+```
+
+`Unmagic::Browser.open(url)` is shorthand for `Unmagic::Browser.new.open(url)`.
+
+Call `open` **without** a block and it returns the session for you to keep — see
+[sessions that outlive one call](#sessions-that-outlive-one-call).
+
+### Verbs
+
+The API reads like [Capybara](https://github.com/teamcapybara/capybara), so it's instantly
+familiar.
+
+```ruby
+# navigate
+session.visit(url)
+session.reload
+session.go_back
+session.go_forward
+session.current_url
+session.title
+
+# act
+session.fill_in("Email", with: "me@example.com")
+session.click_button("Sign in")
+session.click_link("Forgot password?")
+session.click("Anything at all")
+session.select("Australia", from: "Country")
+session.check("Remember me")
+session.uncheck("Send me email")
+session.choose("Express shipping")
+session.attach_file("Résumé", "/path/to/cv.pdf")
+session.press(:enter)
+session.hover("Profile")
+
+# read
+session.text
+session.html
+session.markdown
+session.value("Email")
+session.screenshot
+session.pdf
+session.evaluate("document.title")
+
+# wait & ask
+session.wait_for(text: "Welcome")
+session.wait_for(selector: ".loaded")
+session.has_text?("Welcome")
+session.has_button?("Sign in")
+session.has_link?("Log out")
+session.has_field?("Email")
+session.has_selector?(".row")
+
+# scope
+session.find(".cart")
+session.all(".product")
+session.within(".checkout") { session.click_button("Pay") }
+```
+
+Verbs **auto-wait**: `click_button "Sign in"` keeps looking until the button appears and is
+actionable, up to a timeout — so you rarely need an explicit `wait_for`.
+
+Questions don't. `has_button?` answers about the page as it stands and returns straight
+away, because a caller asking whether a challenge appeared wants "no" now, not in ten
+seconds. When you *mean* to wait, say so with `wait_for`.
+
+### Smart locators
+
+You don't have to hand-write CSS. A plain string is resolved by **label**, then visible
+**text**, then ARIA **role**, falling back to CSS — whichever matches. That's what keeps the
+API working when the caller (an LLM, say) doesn't know the page's markup.
+
+```ruby
+session.fill_in "Email", with: "me@example.com"   # finds <label>Email</label> → its input
+session.click_button "Sign in"                    # finds the button reading "Sign in"
+```
+
+Be explicit when you want to be:
+
+```ruby
+session.fill_in css:   "#user_email", with: "me@example.com"
+session.click_button text: "Sign in"
+session.find label: "Date of birth"
+```
+
+### Emulation (dark mode, viewport, devices, locale…)
+
+Everything about *how the page renders* — colour scheme, viewport, device, locale, timezone —
+lives in one `emulate:` config, set in exactly two places: passed to `open`, or changed live
+with `session.emulate`.
+
+```ruby
+browser.open("https://example.com",
+             emulate: { color_scheme: :dark,
+                        viewport: { width: 1440, height: 900 } }) do |session|
+  session.screenshot                   # dark mode, at 1440×900
+end
+```
+
+The full set of knobs — valid as keys in any `emulate:` hash, or as a reusable object:
+
+```ruby
+emulation = Unmagic::Browser::Emulation.new(
+  color_scheme:   :dark,                           # :light | :dark | :no_preference
+  reduced_motion: :reduce,                         # :reduce | :no_preference
+  viewport:       { width: 390, height: 844,
+                    scale: 2, mobile: true, touch: true },   # scale = device pixel ratio
+  device:         "iPhone 13",                     # shortcut: sets viewport + UA + touch
+  user_agent:     "MyBot/1.0",
+  locale:         "en-AU",                         # Accept-Language
+  timezone:       "Australia/Sydney",
+  geolocation:    { latitude: -33.87, longitude: 151.21 },
+  media:          :print,                          # :screen | :print (handy before #pdf)
+)
+
+# build it once, reuse it across sessions
+browser.open(url, emulate: emulation) { |session| ... }
+
+# change it live — applies to everything after
+session.emulate(color_scheme: :light)
+session.emulate(viewport: { width: 1280, height: 800 })
+
+# or scope a change to a block — reverts when the block ends
+session.emulate(color_scheme: :dark) do
+  session.screenshot                   # dark…
+end
+session.screenshot                     # …light again
+```
+
+Whatever you leave out renders at a fixed default — desktop viewport, light, screen media —
+not at whatever the machine underneath prefers. Headless Chrome inherits the host's
+appearance if you let it, which would mean the same page rendering dark on a laptop after
+sunset and light on CI. Pass `color_scheme: :no_preference` if you actually want the host's.
+
+Niche emulation (CPU/network throttling, offline, vision-deficiency simulation) isn't
+wrapped — reach it through the [escape hatch](#escape-hatch).
+
+## Sessions that outlive one call
+
+A block session is perfect inside a single method. An LLM agent, though, works *turn by
+turn* — open a page in one call, act on it in the next, let a human step in, then continue.
+So `open` without a block hands you the session to keep. It's a plain object; hold it
+wherever your app keeps state.
+
+```ruby
+session = browser.open("https://example.com/login")
+session.fill_in "Email", with: "me@example.com"
+session.click_button "Continue"
+
+# ...later, same object, same live browser...
+session.fill_in "Password", with: secret
+session.click_button "Sign in"
+session.text
+
+session.close
+```
+
+A kept session holds a real browser — and on Cloudflare, one of a limited number of
+concurrent slots — until you `close` it.
+
+Operations on a session are serialized, so two concurrent callers can't interleave a
+keystroke. When a *sequence* must be atomic against other callers, wrap it:
+
+```ruby
+session.lock do
+  session.fill_in "Amount", with: "100"
+  session.click_button "Transfer"
+end
+```
+
+### Human in the loop (captcha / 2FA)
+
+Because a kept session stays live between calls, a human can step in mid-flow: submit the
+login, detect a challenge, show the person a screenshot, and resume once they've answered —
+all against the same session.
+
+```ruby
+session = browser.open("https://example.com/login")
+session.fill_in "Email", with: "me@example.com"
+session.fill_in "Password", with: secret
+session.click_button "Sign in"
+
+if session.has_text?("Enter the code we texted you")
+  # ...show session.screenshot to a person, collect the code...
+end
+
+# later, once the human has answered — same session, still live:
+session.fill_in "Code", with: human_supplied_code
+session.click_button "Verify"
+```
+
+For visual captchas, use a local driver with `headless: false` and let the person click the
+real window, then keep driving.
+
+## Errors
+
+Failures are typed, so you classify them by rescuing — never by parsing a message.
+
+```ruby
+begin
+  browser.markdown(url)
+rescue Unmagic::Browser::Error::TimedOut      # the page took too long
+rescue Unmagic::Browser::Error::Unreachable   # couldn't reach the page or the rendering service
+rescue Unmagic::Browser::Error::RenderFailed  # the service answered but couldn't render
+rescue Unmagic::Browser::Error::RateLimited   # hit a concurrency / rate limit
+rescue Unmagic::Browser::Error::Misconfigured # missing credentials / config
+rescue Unmagic::Browser::Error::NotFound      # nothing on the page matched a locator
+rescue Unmagic::Browser::Error::SessionClosed # the session has been closed
+rescue Unmagic::Browser::Error                # base class for all of the above
+end
+```
+
+`NotFound` is a `TimedOut`, because locators auto-wait: "not found" always means "still not
+there when we ran out of patience".
+
+## Configuration
+
+```ruby
+Unmagic::Browser.configure do |config|
+  # used when Unmagic::Browser.new is called without a driver — a symbol or an instance
+  config.default_driver = Unmagic::Browser::Driver::Cloudflare.new(
+    account_id: ENV["CF_ACCOUNT_ID"],
+    token:      ENV["CF_BROWSER_TOKEN"],
+  )
+
+  # how long a locator keeps looking, and a `wait_for` waits, before giving up
+  config.timeout = 10
+
+  # how long a single navigation is given — longer, because a cold page on a
+  # slow network legitimately takes a while and there's nothing to retry against
+  config.navigation_timeout = 30
+end
+```
+
+Either timeout can be overridden per browser (`Unmagic::Browser.new(timeout: 5)`), per
+session (`browser.open(url, timeout: 5)`), or per call (`session.find(".row", wait: 2)`).
+
+The Cloudflare driver falls back to `CLOUDFLARE_ACCOUNT_ID` and, for the token,
+`CLOUDFLARE_BROWSER_RENDERING_TOKEN` then `CLOUDFLARE_API_TOKEN` — so a machine already set
+up for Cloudflare needs nothing, and a token scoped to just Browser Rendering still wins
+over the account-wide one. The remote driver falls back to `BROWSER_WS_ENDPOINT`. In a
+container you can usually skip the block entirely.
+
+## Escape hatch
+
+The curated API stays small on purpose. When you need something it doesn't wrap (network
+interception, tracing, the full Puppeteer surface), drop down to the object doing the real
+work. It's the mirror of the `driver:` option: you pick a driver when you construct the
+browser, and `driver` hands the underlying instance back.
+
+```ruby
+browser.driver        # => the driver instance (e.g. Unmagic::Browser::Driver::Cloudflare)
+
+browser.open("https://example.com") do |session|
+  puppeteer_page = session.driver     # => the live Puppeteer::Page this session is driving
+  puppeteer_page.request_interception = true
+  # ...full puppeteer-ruby API...
+end
+```
+
+Every driver drives the browser over CDP through
+[puppeteer-ruby](https://github.com/YusukeIwaki/puppeteer-ruby), so `session.driver` is
+always a `Puppeteer::Page`. Session operations stay serialized as usual; wrap multi-step
+native work in `session.lock` when it must be atomic against other callers.
+
+## License
+
+MIT © Keith Pitt

data/lib/unmagic/browser/configuration.rb

@@ -0,0 +1,55 @@
+# frozen_string_literal: true
+
+require "logger"
+
+module Unmagic
+  class Browser
+    # Global defaults, set once with {Unmagic::Browser.configure}. Anything a
+    # caller passes to {Unmagic::Browser#initialize} or to a session wins over
+    # what's here.
+    class Configuration
+      # The driver used when {Unmagic::Browser.new} is called without one, in
+      # every environment except development. A symbol (`:local`, `:remote`,
+      # `:cloudflare`) or a driver instance.
+      attr_accessor :default_driver
+
+      # Seconds a locator keeps looking, and a {Session#wait_for} waits, before
+      # giving up. Deliberately short: a verb that can't find its element is
+      # usually wrong rather than early.
+      attr_accessor :timeout
+
+      # Seconds a single navigation is given. Longer than {#timeout} because a
+      # cold page on a slow network legitimately takes a while, and there's
+      # nothing to retry against in the meantime.
+      attr_accessor :navigation_timeout
+
+      attr_writer :logger
+
+      def initialize
+        @default_driver = :cloudflare
+        @timeout = 10
+        @navigation_timeout = 30
+        @logger = nil
+      end
+
+      # Where the gem narrates what it's doing. Defaults to `Rails.logger` in a
+      # Rails app, and to a silent logger everywhere else — a library shouldn't
+      # write to a host app's stdout unasked.
+      #
+      # @return [Logger]
+      def logger
+        @logger ||= default_logger
+      end
+
+      private
+
+      def default_logger
+        if defined?(Rails) && Rails.respond_to?(:logger) && Rails.logger
+          Rails.logger
+        else
+          Logger.new(IO::NULL)
+        end
+      end
+    end
+  end
+end

data/lib/unmagic/browser/console/banner.rb

@@ -0,0 +1,82 @@
+# frozen_string_literal: true
+
+module Unmagic
+  class Browser
+    module Console
+      # What the console says for itself on the way in, and when asked for `help`.
+      module Banner
+        # The verb table, by heading: each row is what to type and what it does.
+        SECTIONS = {
+          "open" => [
+            ['open "https://example.com"', "open a page — every other verb drives it"],
+            ["visit url", "go somewhere else in the same session"],
+            ["back / forward / reload", "the browser buttons"],
+            ["close", "let the browser go"],
+            ["use :cloudflare", "switch drivers (:local, :remote, :cloudflare)"],
+          ],
+          "act" => [
+            ['fill_in "Email", with: "me@example.com"', "by label, placeholder, name or id"],
+            ['click "Sign in"', "anything — button, link, div"],
+            ['click_button "Go" / click_link "More"', "when you want to be specific"],
+            ['select "Australia", from: "Country"', "by the label a reader sees"],
+            ["check / uncheck / choose", "boxes and radios"],
+            ["press :enter", "at whatever has focus"],
+          ],
+          "read" => [
+            ["text / title / url", "the page as prose"],
+            ["html / markdown", "the page as source, highlighted"],
+            ["screenshot", "drawn right here, and saved to /tmp"],
+            ["screenshot full_page: true", "the whole scrollable page"],
+            ["pdf", "written to /tmp"],
+            ['js "document.title"', "run JavaScript in the page"],
+          ],
+          "find" => [
+            ['find ".cart"', "one element — waits for it"],
+            ['all ".product"', "every match, right now"],
+            ['within(".checkout") { click "Pay" }', "scope the verbs inside"],
+            ['wait_for text: "Welcome"', "block until it shows up"],
+            ['has_text? "Welcome"', "ask, and get an answer now"],
+          ],
+          "emulate" => [
+            ["emulate color_scheme: :dark", "and screenshot it"],
+            ['emulate device: "iPhone 13"', "viewport, user agent and touch"],
+            ["emulate viewport: { width: 1440, height: 900 }", ""],
+          ],
+        }.freeze
+
+        # The four verbs that take a URL work without a session too, which is
+        # worth saying once rather than five times in the table.
+        ONE_OFFS = "markdown, html, screenshot and pdf take a URL for a one-off grab: " \
+          'markdown "https://example.com"'
+
+        module_function
+
+        # @return [String] the banner shown at startup
+        def render
+          [
+            "",
+            "  #{Terminal.paint(Terminal::BOLD, "unmagic-browser")} #{Terminal.dim(Unmagic::Browser::VERSION)}",
+            "  #{Terminal.dim("a browser you can type at · `help` for the verbs")}",
+            "",
+          ].join("\n")
+        end
+
+        # @return [String] the full list of verbs
+        def help
+          lines = [""]
+          SECTIONS.each do |heading, rows|
+            lines << "  #{Terminal.paint(Terminal::GREEN, heading)}"
+            width = rows.map { |verb, _| verb.length }.max
+            rows.each do |verb, note|
+              lines << "    #{Terminal.paint(Terminal::BLUE, verb.ljust(width))}  #{Terminal.dim(note)}".rstrip
+            end
+            lines << ""
+          end
+          lines << "  #{Terminal.dim(ONE_OFFS)}"
+          lines << ""
+          lines.join("\n")
+        end
+      end
+    end
+  end
+end