crucible 0.1.2

data/TESTING.md

@@ -0,0 +1,319 @@
+# Testing Guide
+
+This document covers the testing setup and practices for Crucible.
+
+## Running the Server
+
+```bash
+# Run directly (no bundle exec needed)
+./exe/crucible
+
+# With options
+./exe/crucible --no-headless --width 1920 --height 1080
+
+# Show all options
+./exe/crucible --help
+```
+
+## Running Tests
+
+```bash
+# Run all tests
+bundle exec rspec
+
+# Run with documentation format
+bundle exec rspec --format doc
+
+# Run specific test file
+bundle exec rspec spec/tools/navigation_spec.rb
+
+# Run specific test by line number
+bundle exec rspec spec/tools/navigation_spec.rb:44
+
+# Run tests matching a pattern
+bundle exec rspec --example "navigate tool"
+```
+
+## Test Structure
+
+```
+spec/
+├── spec_helper.rb           # Test configuration and helpers
+├── crucible_spec.rb         # Core module tests
+├── configuration_spec.rb    # Configuration validation tests
+├── session_manager_spec.rb  # Session lifecycle tests
+├── tools/
+│   ├── navigation_spec.rb   # navigate, wait_for, back, forward, refresh
+│   ├── interaction_spec.rb  # click, type, fill_form, select_option, scroll, hover
+│   ├── extraction_spec.rb   # screenshot, get_content, pdf, evaluate, get_url, get_title
+│   ├── cookies_spec.rb      # get_cookies, set_cookies, clear_cookies
+│   ├── sessions_spec.rb     # list_sessions, close_session
+│   └── downloads_spec.rb    # set_download_path, wait_for_download, list_downloads, clear_downloads
+└── e2e/
+    └── stealth_e2e_spec.rb  # End-to-end stealth mode tests
+```
+
+## Test Helper
+
+The `ToolTestHelper` module provides a convenient way to call MCP tools in tests:
+
+```ruby
+module ToolTestHelper
+  def call_tool(tool, args = {})
+    tool.call(args, nil)
+  end
+end
+```
+
+MCP tools expect two arguments: `(args, context)`. The helper passes `nil` for context since tests don't need server context.
+
+## Mocking Strategy
+
+Tests use RSpec's `instance_double` to mock Ferrum objects:
+
+```ruby
+let(:session_manager) { instance_double(Crucible::SessionManager) }
+let(:page) { instance_double("Ferrum::Page") }
+let(:element) { instance_double("Ferrum::Node") }
+
+before do
+  allow(session_manager).to receive(:page).and_return(page)
+end
+```
+
+### Why instance_double?
+
+- Verifies mocked methods exist on the real class
+- Catches API mismatches early (e.g., wrong method signatures)
+- Provides clear error messages when expectations fail
+
+### Important: Ferrum is loaded for real
+
+The spec_helper loads the real Ferrum gem:
+
+```ruby
+require "ferrum"
+```
+
+This ensures `instance_double` can verify method signatures against the actual Ferrum API.
+
+## Testing MCP Tool Schemas
+
+MCP tools have input schemas that define their parameters. Test schema properties using the `.properties` and `.required` methods:
+
+```ruby
+it "has correct schema" do
+  schema = tool.input_schema_value
+
+  # Properties returns a hash with symbol keys
+  expect(schema.properties).to have_key(:url)
+  expect(schema.properties).to have_key(:session)
+
+  # Required returns an array of symbols
+  expect(schema.required).to include(:url)
+end
+```
+
+**Note**: Schema methods return symbols, not strings:
+
+- `schema.properties` → `{ url: {...}, session: {...} }`
+- `schema.required` → `[:url]`
+
+## Testing MCP Tool Responses
+
+MCP tools return `MCP::Tool::Response` objects:
+
+```ruby
+# Successful response
+result = call_tool(tool, url: "https://example.com")
+expect(result.content.first[:text]).to include("Navigated to")
+expect(result.error?).to be(false)
+
+# Error response
+result = call_tool(tool, url: "invalid")
+expect(result.error?).to be(true)
+expect(result.content.first[:text]).to include("failed")
+```
+
+### Response Structure
+
+```ruby
+result.content      # Array of content blocks
+result.error?       # Boolean indicating error state
+result.to_h         # Hash representation for MCP protocol
+```
+
+### Content Types
+
+```ruby
+# Text content
+{ type: "text", text: "Success message" }
+
+# Image content (screenshots)
+{ type: "image", data: "base64...", mimeType: "image/png" }
+
+# Resource content (PDFs)
+{ type: "resource", resource: { uri: "...", mimeType: "application/pdf", blob: "..." } }
+```
+
+## Testing Error Handling
+
+Tools should handle errors gracefully and return error responses:
+
+```ruby
+it "returns error on failure" do
+  allow(page).to receive(:go_to).and_raise(Ferrum::Error.new("Connection refused"))
+
+  result = call_tool(tool, url: "https://example.com")
+
+  expect(result.error?).to be(true)
+  expect(result.content.first[:text]).to include("Navigation failed")
+end
+
+it "returns error when element not found" do
+  allow(page).to receive(:at_css).and_return(nil)
+
+  result = call_tool(tool, selector: "#missing")
+
+  expect(result.error?).to be(true)
+  expect(result.content.first[:text]).to include("Element not found")
+end
+```
+
+## Code Coverage
+
+SimpleCov is configured to track coverage:
+
+```ruby
+require "simplecov"
+SimpleCov.start do
+  add_filter "/spec/"
+  enable_coverage :branch
+  minimum_coverage 50
+end
+```
+
+View the coverage report at `coverage/index.html` after running tests.
+
+Current coverage:
+
+- Line Coverage: ~87%
+- Branch Coverage: ~75%
+
+## Common Patterns
+
+### Testing session parameter
+
+Most tools accept an optional `session` parameter:
+
+```ruby
+it "uses specified session" do
+  allow(page).to receive(:go_to)
+
+  call_tool(tool, session: "my-session", url: "https://example.com")
+
+  expect(session_manager).to have_received(:page).with("my-session")
+end
+```
+
+### Testing optional parameters
+
+```ruby
+it "uses default format" do
+  allow(page).to receive(:screenshot).with(hash_including(format: :png)).and_return("base64data")
+
+  call_tool(tool)
+
+  expect(page).to have_received(:screenshot).with(hash_including(format: :png))
+end
+
+it "respects custom format" do
+  allow(page).to receive(:screenshot).with(hash_including(format: :jpeg)).and_return("base64data")
+
+  call_tool(tool, format: "jpeg")
+
+  expect(page).to have_received(:screenshot).with(hash_including(format: :jpeg))
+end
+```
+
+## Ferrum API Reference
+
+Key Ferrum methods used and their signatures:
+
+```ruby
+# Navigation
+page.go_to(url)
+page.back
+page.forward
+page.refresh
+
+# Element finding
+page.at_css(selector)           # Returns single element or nil
+
+# Element interaction
+element.click(mode: :left)      # :left, :right, or :double
+element.hover
+element.focus
+element.type("text")
+element.type("text", :Enter)    # Type with key
+element.scroll_into_view
+
+# Content extraction
+page.body                       # Full HTML
+page.current_url
+page.current_title
+element.text
+element.property("outerHTML")
+
+# JavaScript
+page.evaluate("expression")
+page.execute("script")
+
+# Screenshots/PDF
+page.screenshot(format: :png, full: false, quality: 100, path: "/tmp/screenshot.png")
+page.pdf(landscape: false, format: :A4, scale: 1.0, path: "/tmp/page.pdf")
+
+# Cookies
+page.cookies.all               # Hash of all cookies
+page.cookies[name]             # Get specific cookie
+page.cookies.set(name:, value:, ...)
+page.cookies.remove(name:, url:)
+page.cookies.clear
+
+# Downloads
+browser.downloads.set_behavior(save_path: "/tmp/downloads")
+browser.downloads.wait(timeout)
+browser.downloads.files        # List of downloaded file paths
+```
+
+## Debugging Tests
+
+```bash
+# Run with full backtrace
+bundle exec rspec --backtrace
+
+# Run single test in isolation
+bundle exec rspec spec/tools/navigation_spec.rb:44 --format doc
+
+# Add binding.irb to pause execution
+it "debugs something" do
+  result = call_tool(tool, url: "https://example.com")
+  binding.irb  # Pause here
+  expect(result).to be_valid
+end
+```
+
+## CI/CD Considerations
+
+The test suite:
+
+- Runs in ~2 seconds
+- Requires no network access (all Ferrum calls mocked)
+- Requires no Chrome/Chromium installation for unit tests
+- Uses random test ordering (`config.order = :random`)
+
+For integration tests that actually drive a browser, you would need:
+
+- Chrome/Chromium installed
+- Xvfb or headless mode on CI
+- Longer timeouts for browser operations

data/config.sample.yml

@@ -0,0 +1,48 @@
+# Crucible Sample Configuration
+# Copy to ~/.config/crucible/config.yml or use with --config flag
+#
+# All settings are optional - defaults are shown below
+
+browser:
+  headless: true              # Run browser without visible window (true/false)
+  window_size: [1280, 720]    # Viewport dimensions [width, height] in pixels
+  timeout: 30                 # Default timeout for operations in seconds
+  # chrome_path: /usr/bin/chromium  # Custom path to Chrome/Chromium executable
+
+stealth:
+  enabled: true               # Enable stealth mode to evade bot detection (true/false)
+  profile: moderate           # Stealth profile: minimal, moderate, or maximum
+                              #   minimal  - Basic evasions (webdriver flag, window dimensions)
+                              #   moderate - Common evasions for most sites (default)
+                              #   maximum  - All evasions for strictest detection
+  locale: "en-US,en"          # Browser locale for Accept-Language header
+
+server:
+  log_level: warn             # Logging verbosity: debug, info, warn, or error
+  # logfile: /path/to/crucible.log  # Optional log file path
+
+# Operating modes - predefined configurations for different use cases
+# Switch modes at runtime with set_stealth_profile or via default mode
+modes:
+  default: ai_agent           # Mode to use on startup (optional)
+
+  ai_agent:                   # Optimized for AI agent browser control
+    stealth_profile: maximum  # Use maximum stealth for web automation
+    screenshot_format: png    # Screenshot format: png, jpeg, or base64
+    wait_timeout: 30000       # Default wait timeout in milliseconds
+
+  scraping:                   # Optimized for web scraping tasks
+    stealth_profile: maximum  # Maximum stealth to avoid detection
+    # rate_limit: 1000        # (future) Delay between requests in ms
+    # retry_attempts: 3       # (future) Number of retry attempts
+    # respect_robots_txt: true  # (future) Honor robots.txt rules
+
+  testing:                    # Optimized for automated testing
+    stealth_profile: minimal  # Minimal stealth for faster execution
+    # capture_network: true   # (future) Capture network requests
+    # performance_metrics: true  # (future) Collect performance data
+    # screenshot_on_failure: true  # (future) Auto-screenshot on errors
+
+  manual:                     # For interactive/debugging sessions
+    stealth_profile: moderate # Balanced stealth settings
+    # expose_cdp: true        # (future) Expose Chrome DevTools Protocol

data/crucible.gemspec

@@ -0,0 +1,48 @@
+# frozen_string_literal: true
+
+require_relative 'lib/crucible/version'
+
+Gem::Specification.new do |spec|
+  spec.name = 'crucible'
+  spec.version = Crucible::VERSION
+  spec.authors = ['Josh Frye']
+  spec.email = ['me@joshfrye.dev']
+
+  spec.summary = 'MCP server for browser automation using Ferrum/Chrome'
+  spec.description = <<~DESC
+    An MCP (Model Context Protocol) server that provides browser automation tools
+    for AI agents using Ferrum and headless Chrome. Features 25 tools covering
+    navigation, screenshots, form interaction, JavaScript evaluation, cookies,
+    file downloads, and multi-session management.
+  DESC
+  spec.homepage = 'https://github.com/joshfng/crucible'
+  spec.license = 'MIT'
+  spec.required_ruby_version = '>= 3.2.0'
+
+  spec.metadata['homepage_uri'] = spec.homepage
+  spec.metadata['source_code_uri'] = spec.homepage
+  spec.metadata['changelog_uri'] = "#{spec.homepage}/blob/main/CHANGELOG.md"
+  spec.metadata['rubygems_mfa_required'] = 'true'
+
+  spec.files = Dir.chdir(__dir__) do
+    `git ls-files -z`.split("\x0").reject do |f|
+      (File.expand_path(f) == __FILE__) ||
+        f.start_with?(*%w[bin/ test/ spec/ features/ .git .github])
+    end
+  end
+  spec.bindir = 'exe'
+  spec.executables = spec.files.grep(%r{\Aexe/}) { |f| File.basename(f) }
+  spec.require_paths = ['lib']
+
+  # Runtime dependencies
+  spec.add_dependency 'ferrum'
+  spec.add_dependency 'mcp'
+
+  # Development dependencies
+  spec.add_development_dependency 'rake'
+  spec.add_development_dependency 'rspec'
+  spec.add_development_dependency 'rubocop'
+  spec.add_development_dependency 'rubocop-rake'
+  spec.add_development_dependency 'rubocop-rspec'
+  spec.add_development_dependency 'simplecov'
+end

data/exe/crucible

@@ -0,0 +1,122 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+# Add lib to load path for development
+$LOAD_PATH.unshift File.expand_path('../lib', __dir__)
+
+require 'optparse'
+require 'crucible'
+
+options = {}
+config_file = nil
+
+parser = OptionParser.new do |opts|
+  opts.banner = 'Usage: crucible [options]'
+  opts.separator ''
+  opts.separator 'MCP server for browser automation using Ferrum/Chrome'
+  opts.separator ''
+  opts.separator 'Options:'
+
+  opts.on('-c', '--config FILE', 'Path to YAML configuration file') do |v|
+    config_file = v
+  end
+
+  opts.on('--[no-]headless', 'Run browser in headless mode (default: true)') do |v|
+    options[:headless] = v
+  end
+
+  opts.on('-w', '--width WIDTH', Integer, 'Viewport width in pixels (default: 1280)') do |v|
+    options[:viewport_width] = v
+  end
+
+  opts.on('-h', '--height HEIGHT', Integer, 'Viewport height in pixels (default: 720)') do |v|
+    options[:viewport_height] = v
+  end
+
+  opts.on('--chrome PATH', 'Path to Chrome/Chromium executable') do |v|
+    options[:chrome_path] = v
+  end
+
+  opts.on('-t', '--timeout SECONDS', Integer, 'Default timeout in seconds (default: 30)') do |v|
+    options[:timeout] = v
+  end
+
+  opts.on('--error-level LEVEL', %w[debug info warn error],
+          'Logging level: debug, info, warn, error (default: warn)') do |v|
+    options[:error_level] = v.to_sym
+  end
+
+  opts.on('--screenshot-format FORMAT', %w[png jpeg base64],
+          'Default screenshot format: png, jpeg, base64 (default: png)') do |v|
+    options[:screenshot_format] = v.to_sym
+  end
+
+  opts.on('--content-format FORMAT', %w[html text],
+          'Default content format: html, text (default: html)') do |v|
+    options[:content_format] = v.to_sym
+  end
+
+  opts.separator ''
+  opts.separator 'Stealth Options:'
+
+  opts.on('--[no-]stealth', 'Enable/disable stealth mode (default: enabled)') do |v|
+    options[:stealth_enabled] = v
+  end
+
+  opts.on('--stealth-profile PROFILE', %w[minimal moderate maximum],
+          'Stealth profile: minimal, moderate, maximum (default: moderate)') do |v|
+    options[:stealth_profile] = v.to_sym
+  end
+
+  opts.on('--stealth-locale LOCALE', 'Browser locale for stealth mode (default: en-US,en)') do |v|
+    options[:stealth_locale] = v
+  end
+
+  opts.separator ''
+
+  opts.on('-v', '--version', 'Show version') do
+    puts "crucible #{Crucible::VERSION}"
+    exit
+  end
+
+  opts.on('--help', 'Show this help message') do
+    puts opts
+    exit
+  end
+end
+
+begin
+  parser.parse!
+rescue OptionParser::InvalidOption, OptionParser::MissingArgument => e
+  warn "Error: #{e.message}"
+  warn "Run 'crucible --help' for usage information"
+  exit 1
+end
+
+# Handle extra arguments
+warn "Warning: Ignoring extra arguments: #{ARGV.join(' ')}" unless ARGV.empty?
+
+begin
+  # Load configuration
+  config = if config_file
+             Crucible::Configuration.from_file(config_file)
+           else
+             Crucible::Configuration.from_defaults
+           end
+
+  # Apply command-line overrides
+  options.each do |key, value|
+    config.public_send(:"#{key}=", value) if config.respond_to?(:"#{key}=")
+  end
+
+  # Apply default mode if configured
+  config.apply_mode(config.modes[:default]) if config.modes && config.modes[:default]
+
+  Crucible::Server.new(config).run
+rescue Crucible::Error => e
+  warn "Error: #{e.message}"
+  exit 1
+rescue Interrupt
+  # Clean exit on Ctrl+C
+  exit 0
+end