agent_ferrum 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: e0f2cc1c04153bbcfec18fe9d0e5c68299bf3bac2eed2cff4b7ab4cdc8cd03a5
+  data.tar.gz: e1150457b6023ebb1176a671dbab3e2058e5ec183adab75aee59f4c067acd422
+SHA512:
+  metadata.gz: 72d310191f52a7b7aa0a179058c771611b1387c8e87def6d6d2051b32256f8fc48c0f739e14c657f64fe1bd2fc6c8955768f7765a4a5841667e59aa9a55439de
+  data.tar.gz: 49e604f1dad52a548b1180e77186a54bf6b5a8306ca478cb2c5679157d8928df91595ff6087c1d4e8320e62b786120d4c060388ff9a9dd1f892bd9253066778a

data/CHANGELOG.md

@@ -0,0 +1,29 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [0.1.0] - 2026-02-09
+
+### Added
+
+- **Hybrid snapshots** combining accessibility tree (interactive elements with `@eN` refs) and markdown content into a single compact output
+- **Ref-based actions** -- `click`, `fill`, `select`, `hover` via `@e1`, `@e2`... refs from the snapshot
+- **Visibility filtering** -- JS-based filtering removes hidden elements (display:none, visibility:hidden, aria-hidden, etc.), Nokogiri post-processing strips scripts, styles, and noise attributes
+- **Markdown conversion** -- HTML to clean markdown via ReverseMarkdown with whitespace compaction
+- **Stealth mode** -- Three profiles (`:minimal`, `:moderate`, `:maximum`) ported from puppeteer-extra-plugin-stealth
+  - `:minimal` -- removes `navigator.webdriver` flag
+  - `:moderate` -- adds vendor/platform spoofing, Chrome runtime, user-agent cleanup
+  - `:maximum` -- adds navigator plugins, WebGL vendor masking, iframe fixes
+- **Download management** -- set download path via CDP, wait for completion with timeout and optional filename filter
+- **Smart waiting** -- poll for CSS/XPath/text/block conditions with configurable timeout and interval
+- **Auto-retry** -- node actions retry automatically on `Ferrum::NodeMovingError` and `Ferrum::CoordinatesNotFoundError` (up to 3 attempts)
+- **AI-friendly errors** -- `RefNotFoundError` and `ElementNotFoundError` include actionable messages guiding the agent
+- **Configuration DSL** -- `AgentFerrum.configure` block or per-instance keyword arguments
+- **Custom user-agent** via CDP `Network.setUserAgentOverride`
+- **Timezone override** via CDP `Emulation.setTimezoneOverride`
+- **Locale support** via Chrome browser options
+
+[0.1.0]: https://github.com/Alqemist-labs/agent_ferrum/releases/tag/v0.1.0

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2025 Florian
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,345 @@
+# AgentFerrum
+
+[![Gem Version](https://badge.fury.io/rb/agent_ferrum.svg)](https://badge.fury.io/rb/agent_ferrum) [![Ruby](https://img.shields.io/badge/ruby-%3E%3D%203.4-ruby.svg)](https://www.ruby-lang.org) [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+
+**Browser automation for AI agents, in Ruby.** Powered by [Ferrum](https://github.com/rubycdp/ferrum).
+
+AgentFerrum wraps Chrome headless (via CDP) with an AI-optimized extraction layer. Instead of dumping raw HTML into your agent's context, it produces a compact snapshot: a markdown rendering of the visible page content + an accessibility tree of interactive elements with clickable refs. Typical reduction: **50-80% fewer tokens** compared to raw HTML (the heavier the page, the bigger the savings).
+
+> Inspired by [agent-browser](https://github.com/vercel-labs/agent-browser) (Vercel) for the accessibility tree + refs concept, [Crucible](https://github.com/joshfng/crucible) for stealth profiles and download management, and [FerrumMCP](https://github.com/Eth3rnit3/FerrumMCP) for Ruby/Ferrum patterns.
+
+## Why AgentFerrum?
+
+Most browser automation tools return the full DOM or raw HTML. An LLM agent processing a typical web page receives **thousands of tokens** of noise (scripts, styles, hidden elements, data attributes). AgentFerrum solves this with a hybrid snapshot:
+
+```
+# Shopping Cart                          # What your agent sees
+URL: https://shop.example.com/cart
+
+## Interactive Elements                  # Clickable refs
+@e1: [link] "Home" href="/"
+@e2: [link] "Products" href="/products"
+@e3: [textbox] "Search" value=""
+@e4: [button] "Remove"
+@e5: [button] "Checkout"
+
+## Page Content                          # Clean markdown
+# Your Cart
+
+| Product | Qty | Price |
+|---------|-----|-------|
+| Widget  | 2   | $20   |
+
+Total: **$20.00**
+```
+
+Your agent reads a compact snapshot instead of the full DOM. It clicks `@e5` to checkout. Done.
+
+## Benchmark
+
+Real-world token reduction measured on live pages (February 2026):
+
+| Site | Raw HTML | Snapshot | Reduction |
+|------|----------|----------|-----------|
+| **Hacker News** | ~8,600 tokens | ~4,500 tokens | **47%** |
+| **Wikipedia** (Ruby article) | ~140,000 tokens | ~31,000 tokens | **78%** |
+| **GitHub** (repo page) | ~265,000 tokens | ~22,000 tokens | **92%** |
+
+The heavier the page (scripts, styles, data attributes, hidden elements), the bigger the savings. Simple content-focused pages like HN see ~50% reduction. Rich web apps like GitHub or StackOverflow see 90%+.
+
+## Features
+
+- **Hybrid snapshots** -- Accessibility tree (interactive elements with refs) + markdown (visible content), combined into a single compact output
+- **Ref-based actions** -- Click, fill, select, hover via `@e1`, `@e2`... refs from the snapshot. No CSS selectors needed
+- **Visibility filtering** -- JS-based filtering removes hidden elements, then Nokogiri strips scripts, styles, and noise attributes
+- **Markdown conversion** -- HTML to clean markdown via [ReverseMarkdown](https://github.com/xijo/reverse_markdown), with whitespace compaction
+- **Stealth mode** -- Three profiles (`:minimal`, `:moderate`, `:maximum`) ported from puppeteer-extra-plugin-stealth
+- **Download management** -- Set download path, wait for completion with timeout
+- **Smart waiting** -- Poll for CSS/XPath/text/block conditions with configurable timeout and interval
+- **Auto-retry** -- Node actions retry automatically on transient errors (element moving, coordinates not found)
+- **AI-friendly errors** -- Error messages tell the agent what to do next ("Call browser.snapshot to refresh refs")
+
+## Installation
+
+Add to your Gemfile:
+
+```ruby
+gem "agent_ferrum"
+```
+
+Then:
+
+```bash
+bundle install
+```
+
+**Requirements:** Ruby 3.4+, Chrome/Chromium installed.
+
+## Quick Start
+
+### 1. Navigate and snapshot
+
+```ruby
+require "agent_ferrum"
+
+browser = AgentFerrum::Browser.new
+browser.navigate("https://example.com")
+
+snap = browser.snapshot
+puts snap.to_s
+# => Compact snapshot with interactive elements + markdown content
+```
+
+### 2. Interact via refs
+
+```ruby
+snap = browser.snapshot
+
+# Click a button by ref
+browser.click("@e3")
+
+# Fill a text field by ref
+browser.fill("@e2", "search query")
+
+# Or use CSS/XPath selectors
+browser.click("button.submit")
+browser.click("//a[@href='/about']")
+browser.fill(css: "input[name='email']", "user@example.com")
+```
+
+### 3. Wait for content
+
+```ruby
+# Wait for an element
+browser.wait_for(css: ".results", timeout: 10)
+
+# Wait for text to appear
+browser.wait_for(text: "Search complete")
+
+# Wait for a custom condition
+browser.wait_for { |b| b.evaluate("document.readyState") == "complete" }
+```
+
+### 4. Extract content
+
+```ruby
+# Full snapshot (accessibility tree + markdown)
+snap = browser.snapshot
+puts snap.to_s              # Combined output for AI
+puts snap.markdown           # Just the markdown content
+puts snap.accessibility_tree # Just the interactive elements
+puts snap.estimated_tokens   # Approximate token count
+
+# Quick markdown only
+puts browser.page_markdown
+```
+
+### 5. Clean up
+
+```ruby
+browser.quit
+```
+
+## Configuration
+
+```ruby
+AgentFerrum.configure do |c|
+  c.headless = true          # Run headless (default: true)
+  c.timeout = 30             # Default timeout in seconds
+  c.poll_interval = 0.1      # Polling interval for wait_for
+  c.viewport = [1920, 1080]  # Browser viewport size
+  c.stealth = :off           # Stealth profile: :off, :minimal, :moderate, :maximum
+  c.download_path = nil      # Directory for downloads
+  c.browser_path = nil       # Custom Chrome/Chromium path
+  c.user_agent = nil         # Custom user agent
+  c.locale = nil             # Browser locale (e.g., "fr-FR")
+end
+```
+
+Or pass options directly:
+
+```ruby
+browser = AgentFerrum::Browser.new(headless: false, timeout: 60, stealth: :maximum)
+```
+
+## Stealth Mode
+
+Three profiles of increasing evasion, ported from [puppeteer-extra-plugin-stealth](https://github.com/berstend/puppeteer-extra/tree/master/packages/puppeteer-extra-plugin-stealth):
+
+| Profile     | Scripts | What it does                                                   |
+| ----------- | ------- | -------------------------------------------------------------- |
+| `:minimal`  | 1       | Removes `navigator.webdriver` flag                             |
+| `:moderate` | 4       | + Vendor/platform spoofing, Chrome runtime, user-agent cleanup |
+| `:maximum`  | 7       | + Navigator plugins, WebGL vendor masking, iframe fixes        |
+
+```ruby
+# Enable at initialization
+browser = AgentFerrum::Browser.new(stealth: :maximum)
+
+# Or switch dynamically
+browser.stealth(:moderate)
+browser.navigate("https://bot-detection-site.com")
+```
+
+## Downloads
+
+```ruby
+browser.download_path = "/tmp/downloads"
+browser.click("@e5")  # Click a download link
+
+filepath = browser.wait_for_download(timeout: 30)
+puts filepath  # => "/tmp/downloads/report.pdf"
+
+# Or wait for a specific filename
+filepath = browser.wait_for_download(filename: "report.pdf", timeout: 60)
+```
+
+## Snapshot Format
+
+The snapshot output is designed for AI consumption. Here's the structure:
+
+```
+# Page Title
+URL: https://example.com/page
+
+## Interactive Elements
+@e1: [button] "Submit"
+@e2: [textbox] "Email" value=""
+@e3: [link] "Home" href="/"
+@e4: [checkbox] "Remember me" checked=true
+@e5: [combobox] "Country"
+@e6: [link] "Sign up" href="/register"
+
+## Page Content
+# Welcome
+
+Please fill in your details below.
+
+| Field | Required |
+|-------|----------|
+| Email | Yes |
+| Name  | No |
+
+[Terms of Service](/tos)
+```
+
+**Supported interactive roles:** button, link, textbox, checkbox, radio, combobox, menuitem, tab, slider, spinbutton, searchbox, switch, option, listbox, menu, menubar.
+
+**Element properties** included when present: `value`, `disabled`, `required`, `checked`, `selected`, `readonly`.
+
+## API Reference
+
+### Navigation
+
+| Method          | Description      |
+| --------------- | ---------------- |
+| `navigate(url)` | Navigate to URL  |
+| `back`          | Go back          |
+| `forward`       | Go forward       |
+| `refresh`       | Reload page      |
+| `current_url`   | Current page URL |
+| `title`         | Page title       |
+
+### Content Extraction
+
+| Method               | Returns             | Description                                          |
+| -------------------- | ------------------- | ---------------------------------------------------- |
+| `snapshot`           | `Snapshot`          | Full hybrid snapshot (accessibility tree + markdown) |
+| `page_markdown`      | `String`            | Markdown of visible content only                     |
+| `accessibility_tree` | `AccessibilityTree` | Interactive elements with refs                       |
+
+### Actions
+
+| Method                  | Description                              |
+| ----------------------- | ---------------------------------------- |
+| `click(target)`         | Click element (ref, CSS, XPath, or Hash) |
+| `fill(target, value)`   | Fill text field                          |
+| `select(target, value)` | Select dropdown option                   |
+| `hover(target)`         | Hover over element                       |
+| `type_text(text)`       | Type text via keyboard                   |
+
+**Target resolution:** `"@e1"` (ref) > `{css: ".btn"}` / `{xpath: "//a"}` (Hash) > `"//*[@id='x']"` (XPath string starting with `/`) > `"button.submit"` (CSS string).
+
+### Waiting
+
+| Method                | Description                     |
+| --------------------- | ------------------------------- |
+| `wait_for(css:)`      | Wait for CSS selector           |
+| `wait_for(xpath:)`    | Wait for XPath                  |
+| `wait_for(text:)`     | Wait for text content           |
+| `wait_for { block }`  | Wait for block to return truthy |
+| `wait_for_navigation` | Wait for page idle              |
+
+### Utilities
+
+| Method                     | Description                       |
+| -------------------------- | --------------------------------- |
+| `evaluate(js)`             | Execute JavaScript, return result |
+| `screenshot(path:, full:)` | Take screenshot                   |
+| `quit`                     | Close browser                     |
+
+## AI Agent Integration Example
+
+Here's how an AI agent loop might use AgentFerrum:
+
+```ruby
+browser = AgentFerrum::Browser.new(stealth: :moderate)
+
+# Agent navigates
+browser.navigate("https://shop.example.com")
+
+# Agent gets compact page representation
+snap = browser.snapshot
+# => Send snap.to_s to LLM (compact snapshot)
+
+# LLM decides to search for a product
+browser.fill("@e3", "wireless headphones")
+browser.click("@e4")  # Search button
+
+# Agent refreshes snapshot after action
+browser.wait_for(css: ".results")
+snap = browser.snapshot
+# => Updated snapshot with search results
+
+# LLM picks a product
+browser.click("@e7")  # Product link
+
+# Continue the loop...
+browser.quit
+```
+
+## Development
+
+```bash
+git clone https://github.com/Alqemist-labs/agent_ferrum
+cd agent_ferrum
+bundle install
+
+# Run unit tests
+bundle exec rake test
+
+# Run integration tests (requires Chrome)
+bundle exec rake integration
+```
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub.
+
+1. Fork it
+2. Create your feature branch (`git checkout -b feature/my-feature`)
+3. Commit your changes
+4. Push to the branch
+5. Create a Pull Request
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](LICENSE.txt).
+
+## See Also
+
+- [Ferrum](https://github.com/rubycdp/ferrum) -- The Chrome headless Ruby library this gem is built on
+- [agent-browser](https://github.com/vercel-labs/agent-browser) -- Vercel's CLI for AI browser automation (accessibility tree + refs concept)
+- [Crucible](https://github.com/joshfng/crucible) -- Ruby MCP server for browser automation with stealth mode
+- [FerrumMCP](https://github.com/Eth3rnit3/FerrumMCP) -- MCP server for Ferrum with AI agent integration
+- [RubyLLM::Tribunal](https://github.com/Alqemist-labs/ruby_llm-tribunal) -- LLM evaluation framework for Ruby

data/lib/agent_ferrum/browser/target_resolution.rb

@@ -0,0 +1,64 @@
+module AgentFerrum
+  class Browser
+    module TargetResolution
+      private
+
+      def resolve_target(target)
+        case target
+        when /\A@e\d+\z/
+          resolve_ref(target)
+        when Hash
+          resolve_hash_target(target)
+        when %r{\A/}
+          find_by_xpath(target)
+        else
+          find_by_css(target)
+        end
+      end
+
+      def resolve_hash_target(target)
+        if target[:css]
+          find_by_css(target[:css])
+        elsif target[:xpath]
+          find_by_xpath(target[:xpath])
+        else
+          raise ArgumentError, "Hash target must have :css or :xpath key"
+        end
+      end
+
+      def resolve_ref(ref)
+        node_info = @ref_map[ref]
+        raise RefNotFoundError, ref unless node_info
+
+        backend_node_id = node_info[:backend_node_id]
+
+        result = @ferrum.page.command("DOM.resolveNode", backendNodeId: backend_node_id)
+        object_id = result.dig("object", "objectId")
+
+        desc = @ferrum.page.command("DOM.describeNode", backendNodeId: backend_node_id)
+        description = desc["node"]
+
+        push_result = @ferrum.page.command("DOM.requestNode", objectId: object_id)
+        node_id = push_result["nodeId"]
+
+        frame = @ferrum.page.main_frame
+        target_id = @ferrum.page.target_id
+        Ferrum::Node.new(frame, target_id, node_id, description)
+      end
+
+      def find_by_css(selector)
+        node = @ferrum.at_css(selector)
+        raise ElementNotFoundError, selector unless node
+
+        node
+      end
+
+      def find_by_xpath(xpath)
+        node = @ferrum.at_xpath(xpath)
+        raise ElementNotFoundError, xpath unless node
+
+        node
+      end
+    end
+  end
+end

data/lib/agent_ferrum/browser.rb

@@ -0,0 +1,159 @@
+# frozen_string_literal: true
+
+require "ferrum"
+require_relative "browser/target_resolution"
+
+module AgentFerrum
+  class Browser
+    include TargetResolution
+
+    attr_reader :ferrum, :config
+
+    def initialize(**opts)
+      @config = AgentFerrum.configuration.dup
+      opts.each { |k, v| @config.public_send(:"#{k}=", v) if @config.respond_to?(:"#{k}=") }
+
+      @ref_map = {}
+      @stealth_manager = Stealth::Manager.new
+      @ferrum = Ferrum::Browser.new(**@config.ferrum_options)
+      @waiter = Waiter.new(self, default_timeout: @config.timeout, default_interval: @config.poll_interval)
+      @downloads = Downloads.new(self)
+
+      apply_stealth if @config.stealth != :off
+      apply_user_agent if @config.user_agent
+      apply_timezone if @config.timezone
+      @downloads.download_path = @config.download_path if @config.download_path
+    end
+
+    # --- Navigation ---
+
+    def navigate(url)
+      @ferrum.goto(url)
+    end
+
+    def back
+      @ferrum.back
+    end
+
+    def forward
+      @ferrum.forward
+    end
+
+    def refresh
+      @ferrum.refresh
+    end
+
+    # --- Content extraction ---
+
+    def snapshot
+      snap = Content::Snapshot.new(self)
+      @ref_map = snap.refs
+      snap
+    end
+
+    def page_markdown
+      filtered_html = Content::VisibilityFilter.new(self).filtered_html
+      Content::MarkdownConverter.new(filtered_html).convert
+    end
+
+    def accessibility_tree
+      tree = Content::AccessibilityTree.new(self)
+      @ref_map = tree.refs
+      tree
+    end
+
+    # --- Actions ---
+
+    def click(target)
+      node = resolve_target(target)
+      Node.new(node).click
+    end
+
+    def fill(target, value)
+      node = resolve_target(target)
+      Node.new(node).fill(value)
+    end
+
+    def select(target, value)
+      node = resolve_target(target)
+      Node.new(node).select(value)
+    end
+
+    def hover(target)
+      node = resolve_target(target)
+      Node.new(node).hover
+    end
+
+    def type_text(text)
+      @ferrum.page.keyboard.type(text)
+    end
+
+    # --- Wait ---
+
+    def wait_for(css: nil, xpath: nil, text: nil, timeout: nil, interval: nil, &)
+      @waiter.call(css:, xpath:, text:, timeout:, interval:, &)
+    end
+
+    def wait_for_navigation(timeout: nil)
+      timeout ||= @config.timeout
+      @ferrum.page.wait_for_idle(timeout: timeout)
+    end
+
+    # --- Downloads ---
+
+    def download_path=(path)
+      @downloads.download_path = path
+    end
+
+    def wait_for_download(timeout: 30, filename: nil)
+      @downloads.wait(timeout:, filename:)
+    end
+
+    # --- Stealth ---
+
+    def stealth(profile)
+      @config.stealth = profile
+      apply_stealth
+    end
+
+    # --- Utils ---
+
+    def current_url
+      @ferrum.current_url
+    end
+
+    def title
+      @ferrum.page.title
+    end
+
+    def evaluate(expression)
+      @ferrum.evaluate(expression)
+    end
+
+    def screenshot(path: nil, selector: nil, full: false)
+      opts = {}
+      opts[:path] = path if path
+      opts[:selector] = selector if selector
+      opts[:full] = full
+      @ferrum.screenshot(**opts)
+    end
+
+    def quit
+      @ferrum.quit
+    end
+
+    private
+
+    def apply_stealth
+      @stealth_manager.apply(@ferrum.page, @config.stealth)
+    end
+
+    def apply_user_agent
+      @ferrum.page.command("Network.setUserAgentOverride", userAgent: @config.user_agent)
+    end
+
+    def apply_timezone
+      @ferrum.page.command("Emulation.setTimezoneOverride", timezoneId: @config.timezone)
+    end
+  end
+end

data/lib/agent_ferrum/configuration.rb

@@ -0,0 +1,45 @@
+# frozen_string_literal: true
+
+module AgentFerrum
+  class Configuration
+    attr_accessor :headless, :timeout, :process_timeout, :poll_interval,
+      :viewport, :stealth, :download_path,
+      :browser_path, :chrome_args, :user_agent,
+      :locale, :timezone
+
+    def initialize
+      @headless = true
+      @timeout = 30
+      @process_timeout = nil
+      @poll_interval = 0.1
+      @viewport = [1920, 1080]
+      @stealth = :off
+      @download_path = nil
+      @browser_path = nil
+      @chrome_args = []
+      @user_agent = nil
+      @locale = nil
+      @timezone = nil
+    end
+
+    def initialize_dup(original)
+      super
+      @viewport = original.viewport.dup
+      @chrome_args = original.chrome_args.dup
+    end
+
+    def ferrum_options
+      opts = {
+        headless:        @headless,
+        timeout:         @timeout,
+        window_size:     @viewport,
+        browser_options: {}
+      }
+      opts[:process_timeout] = @process_timeout if @process_timeout
+      opts[:browser_path] = @browser_path if @browser_path
+      opts[:browser_options]["lang"] = @locale if @locale
+      @chrome_args.each { |arg| opts[:browser_options][arg.delete_prefix("--")] = nil }
+      opts
+    end
+  end
+end