wayfarer 0.4.5 → 0.4.7

data/docs/guides/tutorial.md

@@ -0,0 +1,60 @@
+# Tutorial
+
+Wayfarer is a web crawling framework written in Ruby.
+It works with plain HTTP or by automating web browsers and is deployed with
+Redis and a message queue (which can be Redis-based itself).
+In development, it can execute fully in memory, without Redis.
+
+You need a compatible version of Ruby installed.
+
+To get started, in an empty directory, generate a new `Gemfile` and install
+ActiveJob and Wayfarer:
+
+```sh
+bundle init
+bundle add activejob wayfarer
+bundle install
+```
+
+## Jobs, tasks and batches
+
+Wayfarer builds on Active Job, the message queue abstraction of Rails.
+You can use Wayfarer without Rails of course, as we do here.
+
+A message queue supports two operations: appending messages to the end and consuming
+messages from the front. In the case of Wayfarer, messages are tasks, a string pair
+consisting of a URL and a batch. When a task is consumed, it is processed by a job,
+a Ruby class.
+
+Let's give ourselves a `dummy_job.rb` that routes arbitrary URLs to its
+`index` instance method, where we print the current `task`:
+
+```ruby
+require "activejob"
+require "wayfarer"
+
+class DummyJob < ActiveJob::Base
+  include Wayfarer::Base
+
+  route.to :index
+
+  def index
+    puts task
+  end
+end
+```
+
+We can perform our job from the command line with Wayfarer's CLI and find
+that in between ActiveJob's log output, our task was printed with a generated
+UUID for its batch:
+
+```hl_lines="1 3"
+bundle exec wayfarer perform -r dummy_job.rb DummyJob https://example.com
+[ActiveJob] [DummyJob] [68853491-...] Performing DummyJob (Job ID: 68853491-...) from Async(default) with arguments: #<Wayfarer::Task url="https://example.com", batch="63d14035-...">
+#<Wayfarer::Task url="https://example.com", batch="63d14035-...">
+[ActiveJob] [DummyJob] [68853491-...] Performed DummyJob (Job ID: 68853491-) from Async(default) in 507.65ms
+```
+
+Many commands accept a `--batch` flag for setting the batch. If you don't
+provide one, a UUID is generated.
+

data/docs/guides/user_agents.md

@@ -0,0 +1,113 @@
+# User agents
+
+User agents are used by [jobs](../jobs) to retrieve the contents behind a URL into a
+[page](../pages). They are kept in a connection pool and all user agents in the pool
+share the same type and configuration. You can add custom user agents by implementing
+the [user agent API](custom_user_agents.md).
+
+Wayfarer comes with the following built-in user agents:
+
+* [`#!ruby :http`](http.md) (default)
+* [`#!ruby :ferrum`](ferrum.md) to automate Google Chrome
+* [`#!ruby :selenium`](selenium.md) to automate a variety of browsers
+* [`#!ruby :capybara`](capybara.md) to use Capybara sessions
+
+Configure the user agent with the global configuration option:
+
+```ruby
+Wayfarer.config[:network][:agent] = :ferrum # or :selenium, :capybara, ...
+```
+
+You can access the user agent that was checked out from the pool with
+`#user_agent` in action methods:
+
+```ruby
+class DummyJob < ActiveJob::Base
+  include Wayfarer::Base
+
+  route.to :index
+
+  def index
+    user_agent # => #<Ferrum::Browser ...>
+  end
+end
+```
+
+You can also implement [custom user agents](custom_user_agents.md) to support
+your own HTTP client or browser automation service/protocol.
+
+### Ad-hoc HTTP requests
+
+Regardless the configured user agent, you can always make ad-hoc HTTP GET requests
+that return pages with `#fetch(url)`:
+
+```ruby
+class DummyJob < ActiveJob::Base
+  include Wayfarer::Base
+
+  route.to :index
+
+  def index
+    page = fetch("https://example.com") # => #<Wayfarer::Page ...>
+  end
+end
+```
+
+!!! info "`#fetch` uses the configured `Wayfarer.config.network.http_headers`."
+
+## HTTP request headers
+
+You can set HTTP request headers for all built-in user agents:
+
+```ruby
+Wayfarer.config[:network][:http_headers] = { "User-Agent" => "MyCrawler" }
+```
+
+!!! attention "Selenium does not support configuring HTTP request headers."
+
+## Connection pooling
+
+Since user agents are expensive to create, especially in the case of browser
+processes,  Wayfarer keeps user agents within a connection pool. When a job
+performs and needs to retrieve the [page](../pages) for its task URL, an agent
+is checked out from the pool, and checked back in when the routed action method
+returns.
+
+The pool size is constant and it should equal the number of threads the
+underlying message queue operates with. For example, if you use Sidekiq,
+you should set the pool size to the number of Sidekiq threads:
+
+```ruby
+Wayfarer.config[:network][:pool_size] = Sidekiq.options[:concurrency]
+```
+
+!!! attention "The connection pool size is 1 by default"
+
+    Since there is no reliable way to detect the number of threads that
+    the underlying message queue operates with, Wayfarer defaults to a pool
+    size of 1, which creates a bottleneck in a concurrent environment.
+
+!!! attention "Browser sessions are shared across jobs"
+
+    The same browser session is used across jobs. This means that the browser
+    is not closed between jobs, and that the browser's state carries over from
+    job to job. You may account for this by resetting the browser's state
+    according to your needs, for which you can use [callbacks](../callbacks).
+
+### `UserAgentTimeoutError`: avoiding pool contention
+
+If you encounter `UserAgentTimeoutError` exceptions, a job has waited for a
+user agent to become available for too long. By default, this timeout is 10
+seconds. This is a sign that the pool size is too small for the message queue's
+concurrency.
+
+```
+#<Wayfarer::UserAgentTimeoutError: Waited 10 sec, 0/1 available>
+```
+
+You can configure the timeout, although you will likely want to increase the
+pool size instead:
+
+```ruby
+Wayfarer.config[:network][:pool_timeout] = 10 # seconds
+```

data/docs/index.md

@@ -1,56 +1,33 @@
 ---
 hide:
   - navigation
+  - toc
 ---
 
 # Wayfarer
 
-![CI status](https://github.com/actions/starter-workflows/workflows/CI/badge.svg)
-[![RubyGem](https://badge.fury.io/rb/wayfarer.svg)](https://rubygems.org/gems/wayfarer)
+## Ruby web crawling framework built on [ActiveJob]() and [Redis]()
 
-## Versatile web crawling with Ruby
+<small>
+  [Read the tutorial](/guides/tutorial){ .md-button .md-button--primary }
+</small>
 
-* Web scraping
-* Data extraction
-* Browser automation
+=== "Command line"
 
-!!! attention "Unstable software"
+    ```sh
+    gem install wayfarer
+    ```
 
-    Wayfarer is under development and releases should be considered unstable.
+=== "Gemfile"
 
-    Wayfarer complies to
-    [Semantic Versioning 2.0.0](https://semver.org/spec/v2.0.0.html) in
-    which v0.x means that there could be backward-incompatible changes for every
-    release:
+    ```ruby
+    gem "wayfarer"
+    ```
 
-    >Major version zero (0.y.z) is for initial development. Anything MAY change
-    at any time. The public API SHOULD NOT be considered stable.
-
-### Installation
-
-Install the RubyGem:
-
-```
-gem install wayfarer
-```
-
-Or add it to Bundler's Gemfile:
-
-```ruby
-gem "wayfarer"
-```
-
-### Features
-
-* Breadth-first, acyclic, multi-threaded graph traversal
-* Executes atop a variety of message queues thanks to [ActiveJob](https://edgeguides.rubyonrails.org/active_job_basics.html)
-* Browser automation via [Ferrum](https://github.com/rubycdp/ferrum)
+* Breadth-first, acyclic page traversal
+* Plain HTTP and browser automation via [Ferrum](https://github.com/rubycdp/ferrum)
 (<abbr title="Chrome DevTools Protocol">CDP</abbr>),
-[Selenium](https://www.selenium.dev) or plain HTTP via `net/http`
+[Selenium](https://www.selenium.dev) and custom user agents
 * Declarative routing DSL
 * URI normalization and deduplication
-* XML, HTML, JSON parsing
-* HTTP redirect handling
-* Storage-agnostic
-* Small footprint: <500 LoC
-* Open Source (MIT)
+* HTML, XML, JSON and custom Content-Type body parsing

data/docs/reference/cli.md

@@ -1,46 +1,46 @@
-# Command Line Interface
+# wayfarer
+
+The command-line interface to Wayfarer.
 
 ## Usage
 
 ```
-wayfarer [OPTIONS] [generate|job|route|version]
+wayfarer [OPTIONS] [perform|enqueue|execute|route|tree]
 ```
 
-All [environment variables](../environment_variables) are respected.
-
-## `wayfarer generate`
-
-### `wayfarer generate project NAME`
+See [Configuration](../reference/cli) for the respected environment variables.
 
-:   Generates a new project directory `NAME`.
+---
 
-## `wayfarer job`
+## `wayfarer perform JOB URL`
 
-### `wayfarer job perform JOB URL`
-
-:   Performs `JOB` with `URL`. The job does not reach any Active Job backend.
-    Staged jobs will not be processed.
+:   Performs `JOB` with `URL` in memory. The task is not sent to the message queue.
+    Staged jobs are ignored.
 
     ##### Options
 
     * `--mock-redis`: Use an in-memory implementation of Redis instead of
       talking to an actual server.
-    * `--batch=BATCH`: Set the job's batch. By default, a UUID is generated.
+    * `--batch=BATCH`: The job's batch. By default, a UUID is generated.
+
+---
 
-### `wayfarer job enqueue JOB URL`
+## `wayfarer enqueue JOB URL`
 
-:   Enqueues `JOB` with `URL` to the configured Active Job backend.
+:   Enqueues a task for `JOB` with `URL` to the message queue.
 
     ##### Options
 
-    * `--batch=BATCH`: Set the job's batch. By default, a UUID is generated.
+    * `--batch=BATCH`: The job's batch. By default, a UUID is generated.
+
+---
 
-### `wayfarer job execute JOB URL`
+## `wayfarer execute JOB URL`
 
-:   Execute `JOB` with `URL` by using the
-    [Active Job Async adapter](https://api.rubyonrails.org/classes/ActiveJob/QueueAdapters/AsyncAdapter.html).
-    The job does not reach any other Active Job backend. Blocks until the batch
-    completes.
+:   Execute `JOB` with `URL` with the in-memory
+    [Active Job Async adapter](https://api.rubyonrails.org/classes/ActiveJob/QueueAdapters/AsyncAdapter.html)
+    instead of writing the taks to an actual message queue. Blocks until the
+    batch has completed.
 
     ##### Options
 
@@ -50,12 +50,22 @@ All [environment variables](../environment_variables) are respected.
     * `--min-threads`: Minimum number of threads to use. Default: 1
     * `--max-threads`: Maximum number of threads to use. Default: 1
 
-## `wayfarer route`
+    !!! attention "Why are my jobs not getting retried with `wayfarer job execute`?"
+
+        You need to set the `wait: 0` option on `retry_on` in order for
+        `wayfarer job execute` to execute retries:
 
-### `wayfarer route result JOB URL`
+        ```ruby
+        retry_on StandardError, attempts: 3, wait: 0
+        ```
+---
+
+## `wayfarer route JOB URL`
 
 :   Prints the result of invoking `JOB`'s router with `URL`.
 
-### `wayfarer route tree JOB URL`
+---
+
+## `wayfarer tree JOB URL`
 
 :   Visualises the routing tree result of invoking `JOB`'s router with `URL`.

data/docs/reference/configuration.md

@@ -0,0 +1,36 @@
+---
+hide:
+  - toc
+---
+
+# Configuration
+
+You can configure Wayfarer by assigning to the `Wayfarer.config` Hash
+which has the following defaults:
+
+```ruby
+{
+  redis: {
+    url: "redis://localhost:6379/0",
+    factory: ->(redis) { ::Redis.new(url: redis[:url]) }
+  },
+  network: {
+    agent: :http,
+    pool_size: 1,
+    pool_timeout: 10,
+    http_headers: {},
+    renew_on: []
+  },
+  capybara: {
+    driver: nil
+  },
+  ferrum: {
+    options: {}
+  },
+  selenium: {
+    driver: :chrome,
+    options: {},
+    client_timeout: 60
+  }
+}
+```

data/lib/wayfarer/base.rb

@@ -1,60 +1,138 @@
 # frozen_string_literal: true
 
 module Wayfarer
-  class Base < ActiveJob::Base
-    include Wayfarer::Middleware::Controller
-
-    use Wayfarer::Middleware::Stage
-    use Wayfarer::Middleware::Dedup
-    use Wayfarer::Middleware::Normalize
-    use Wayfarer::Middleware::Router
-    use Wayfarer::Middleware::Fetch
-    use Wayfarer::Middleware::Dispatch
-
-    ErrorHandler = lambda do |&block|
-      lambda do |job, error|
-        task = job.arguments.first
-        task.barrier.seen?(task.url)
-        task.gc.run
-        block.call(job, error)
-      end
-    end
+  # @!attribute [r] task
+  #   @return [Wayfarer::Task] the current task
+  # @!attribute [r] uri
+  #   @return [Addressable::URI] Parsed task URL
+  # @!attribute [r] user_agent
+  #   @return [Object] the user agent that retrieved the page
+  # @!attribute [r] action
+  #   @return [Symbol, Object] action that the task URL was routed to.
+  # @!attribute [r] params
+  #   @return [HashWithIndifferentAccess] path parameters collected from routes
+  module Base
+    extend ActiveSupport::Concern
+    # @!method stage(urls)
+    # Adds URLs to an internal staging set so that they get enqueued
+    # eventually, once the job executed successfully.
+    # @overload stage(urls)
+    #   @param urls [Array<String>] URLs to add to the staging set.
+    # @overload stage(url)
+    #   @param url [String] URL to add to the staging set.
 
-    after_enqueue do |job|
-      task = job.arguments.first
-      task.counter.increment
-    end
+    # @!method fetch(url, follow: 3)
+    # @param url [String] URL to fetch using plain HTTP(S).
+    # @param follow [Fixnum] Number of redirects to follow.
+    # Retrieves the given URL to a {Page}.
 
-    after_perform do |job|
-      task = job.arguments.first
-      task.gc.run
-    end
+    # @!method page(live: false)
+    # @param url [live] whether to retrieve a new {Page}.
+    # @return [Wayfarer::Page]
+    # Returns the most recently retrieved page or a new page
+    # for the current task URL if the `follow` keyword is passed.
 
-    rescue_from(StandardError) do
-      task = arguments.first
-      task.gc.run
-    end
+    # @!scope class
 
-    def self.retry_on(*argv, &block)
-      super(*argv, &ErrorHandler.call(&block))
-    end
+    # @!attribute [r] route
+    # @return [Wayfarer::Routing::DSL]
+    # The job's {Wayfarer::Routing::DSL} that maps URLs to instance methods
+    # or to a {Handler}.
+    # @example Append a host route
+    #   route.host "examplxe.com", to: :index
 
-    def self.discard_on(*argv, &block)
-      super(*argv, &ErrorHandler.call(&block))
-    end
+    # @!method content_types(*content_types)
+    # @param content_types [*Array<String, Regexp>] Content-Types to whitelist
+    # Whitelists Content-Types. Once at least one Content-Type is set, only
+    # those Content-Types will be processed.
 
-    def self.crawl(url, batch: SecureRandom.uuid)
-      Task.new(url, batch).tap do |task|
-        perform_later(task)
-      end
-    end
+    # @!group Callbacks
+
+    # @!method before_fetch
+    # @overload before_fetch(callback)
+    #   @param callback [Symbol] Instance method to call
+    # @overload before_fetch(&block)
+    #   @yield [Wayfarer::Task]
+    # Registers a callback that is called before the page is fetched.
+    # If a symbol is passed, an instance method with the same name will be
+    # called.
+    # @example Accessing the user agent in {#before_fetch}
+    #   before_fetch do |task|
+    #     user_agent # => the user agent that will fetch the page
+    #   end
 
-    def retry_job(...)
-      super(...) # increments the counter by re-enqueuing the job
-      task = arguments.first
-      task.counter.decrement
+    # @!method around_fetch
+    # @overload around_fetch(callback)
+    #   @param callback [Symbol] Instance method to call
+    # @overload around_fetch(&block)
+    #   @yield [Wayfarer::Task]
+    # Registers a callback that is called around the page getting fetched.
+    # If a symbol is passed, an instance method with the same name will be
+    # called.
+
+    # @!method after_fetch
+    # @overload after_fetch(callback)
+    #   @param callback [Symbol] Instance method to call
+    # @overload after_fetch(&block)
+    #   @yield [Wayfarer::Task]
+    # Registers a callback that is called after the page was fetched.
+    # If a symbol is passed, an instance method with the same name will be
+    # called.
+
+    # @!method before_perform
+    # @overload before_perform(callback)
+    #   @param callback [Symbol] Instance method to call
+    # @overload before_perform(&block)
+    #   @yield [Wayfarer::Task]
+    # Registers a callback that is called before the task is performed.
+    # If a symbol is passed, an instance method with the same name will be
+    # called.
+
+    # @!method around_perform
+    # @overload around_perform(callback)
+    #   @param callback [Symbol] Instance method to call
+    # @overload around_perform(&block)
+    #   @yield [Wayfarer::Task]
+    # Registers a callback that is called around the task getting performed.
+    # If a symbol is passed, an instance method with the same name will be
+    # called.
+
+    # @!method after_perform
+    # @overload after_perform(callback)
+    #   @param callback [Symbol] Instance method to call
+    # @overload after_perform(&block)
+    #   @yield [Wayfarer::Task]
+    # Registers a callback that is called after the task was performed.
+    # If a symbol is passed, an instance method with the same name will be
+    # called.
+
+    # @!endgroup
+
+    included do
+      include Wayfarer::Middleware::Controller
+
+      # Implement ActiveJob's #perform by calling into our own middleware chain
+      alias_method :perform, :call
+
+      # Middleware stack
+      use Wayfarer::Middleware::Redis
+      use Wayfarer::Middleware::BatchCompletion
+      use Wayfarer::Middleware::UriParser
+      use Wayfarer::Middleware::Normalize
+      use Wayfarer::Middleware::Dedup
+      use Wayfarer::Middleware::Stage
+      use Wayfarer::Middleware::Router
+      use Wayfarer::Middleware::UserAgent
+      use Wayfarer::Middleware::ContentType
+      use Wayfarer::Middleware::Dispatch
     end
 
-    alias perform call
+    class_methods do
+      def crawl(url, batch: SecureRandom.uuid)
+        Task.new(url, batch).tap do |task|
+          perform_later(task)
+        end
+      end
+    end
   end
 end

data/lib/wayfarer/batch_completion.rb

@@ -0,0 +1,56 @@
+# frozen_string_literal: true
+
+module Wayfarer
+  # BatchCompletion tracks the completion of a batch of jobs.
+  # It does so by incrementing and decrementing a counter in Redis.
+  #
+  # The counter is incremented when a job is first enqueued and decremented when
+  # a job is performed. If a job is retried, the counter is not incremented.
+  # When a job succeeds or fails and thereby exceeds its retry count, the counter
+  # is decremented.
+  #
+  # When the counter reaches zero, garbage collection deletes the Redis keys
+  # associated with the batch.
+  module BatchCompletion
+  module_function
+
+    def subscribe!
+      ActiveSupport::Notifications.subscribe("enqueue.active_job", self)
+      ActiveSupport::Notifications.subscribe("perform.active_job", self)
+      ActiveSupport::Notifications.subscribe("retry_stopped.active_job", self)
+    end
+
+    def call(name, _, _, _, data)
+      return unless (job = data[:job]).is_a?(Wayfarer::Base)
+
+      task = job.arguments.first
+
+      # In the case of `enqueue.active_job` middleware hasn't executed yet
+      task[:redis_pool] ||= Wayfarer::Redis::Pool.instance # TODO: Test
+
+      counter = Redis::Counter.new(task) do
+        job.run_callbacks(:batch)
+      ensure
+        Wayfarer::GC.run(task)
+      end
+
+      handle(name, job, task, counter)
+    end
+
+    def handle(name, job, task, counter)
+      case name
+      when "enqueue.active_job" then counter.increment unless retry?(job)
+      when "perform.active_job" then counter.decrement if succeeded?(job, task)
+      when "retry_stopped.active_job" then counter.decrement
+      end
+    end
+
+    def retry?(job)
+      job.executions > 0
+    end
+
+    def succeeded?(job, task)
+      job.exception_executions == task[:initial_exception_executions]
+    end
+  end
+end