wayfarer 0.4.7 → 0.4.9

data/Rakefile

@@ -5,7 +5,14 @@ require "open-uri"
 require "zip"
 
 require "bundler/gem_tasks"
+require "pry"
 
 Dir.glob("rake/*.rake").each { |file| load(file) }
 
 task default: :build
+
+task :console do
+  require_relative "lib/wayfarer"
+
+  Pry.start
+end

data/docker-compose.yml

@@ -1,15 +1,14 @@
-version: "3"
 services:
   wayfarer:
     build: .
     tty: true
     volumes:
-      - "./:/opt/app"
+      - ./:/opt/app
     ports:
-      - "9876:9876"
-    environment:
-      - CI=true
+      - "${WAYFARER_PORT}:${WAYFARER_PORT}"
     hostname: test
+    environment:
+      CI: "${CI}"
     depends_on:
       - redis
       - chrome
@@ -17,26 +16,26 @@ services:
       - docs
 
   redis:
-    image: redis
+    image: ${REDIS_IMAGE}:${REDIS_VERSION}
 
   chrome:
-    image: browserless/chrome
+    image: ${CHROME_IMAGE}:${CHROME_VERSION}
     ports:
-      - "3000:3000"
+      - "${CHROME_PORT}:${CHROME_PORT}"
 
   firefox:
-    image: selenium/standalone-firefox:4.0.0-rc-2-prerelease-20210923
+    image: ${FIREFOX_IMAGE}:${FIREFOX_VERSION}
     ports:
-      - "4444:4444"
+      - "${FIREFOX_PORT}:${FIREFOX_PORT}"
     volumes:
-      - "/dev/shm:/dev/shm"
+      - /dev/shm:/dev/shm
 
   docs:
-    image: squidfunk/mkdocs-material:9.5.9
+    image: ${DOCS_IMAGE}:${DOCS_VERSION}
     volumes:
-      - "./:/docs"
+      - ./:/docs
     ports:
-      - "8000:8000"
+      - "${DOCS_PORT}:${DOCS_PORT}"
 
 networks:
   default:

data/docs/guides/callbacks.md

@@ -1,7 +1,7 @@
 # Callbacks
 
 Wayfarer supports a number of callbacks in addition to
-[ActiveJob's](https://edgeguides.rubyonrails.org/active_job_basics.html#callbacks).
+[ActiveJob callbacks](https://edgeguides.rubyonrails.org/active_job_basics.html#callbacks).
 
 ## Available callbacks
 
@@ -20,6 +20,8 @@ to process in a batch. Wayfarer instruments job execution and in- or decrements
 an integer counter in Redis on certain events. When the counter reaches zero,
 the current job's `after_batch` callbacks run.
 
+!!! info "`after_batch` callbacks fire at most once per batch."
+
 ## Conditional callbacks
 
 You can make callbacks conditional with the `#!ruby :if` and `#!ruby :unless`

data/docs/guides/configuration.md

@@ -1,39 +1,14 @@
-# Configuration
-
-Wayfarer can be configured in two ways:
-
-1. Using [environment variables](/reference/environment_variables)
-2. Using runtime configuration
-
-## Runtime configuration
-
-Wayfarer parses environment variables into a runtime configuration
-`Wayfarer::Config`. The configuration can then be altered or replaced via
-`Wayfarer.config`:
-
-```ruby
-# Which user agent to use to process tasks
-Wayfarer.config[:network][:agent] = :http # or :ferrum, :selenium
+---
+hide:
+  - toc
+---
 
-# How many user agents to instantiate
-Wayfarer.config[:network][:pool_size] = 3
-
-# How long an agent may be used while processing a task
-Wayfarer.config[:network][:pool_timeout] = 5000
-
-# Ferrum options
-Wayfarer.config[:ferrum][:options] = {}
-
-# Selenium driver to use
-Wayfarer.config[:selenium][:driver] = :chrome
-
-# Selenium HTTP client read timeout
-Wayfarer.config[:selenium][:client_timeout] = 10 # seconds
+# Configuration
 
-# Selenium options
-Wayfarer.config[:selenium][:options] = { url: "http://chrome" }
+You can configure Wayfarer by assigning to `Wayfarer.config` which defaults to:
 
-# HTTP request headers (Selenium is unsupported)
-Wayfarer.config[:network][:http_headers] = { "Field" => "Value" }
+```rb
+module Wayfarer
+--8<-- "lib/wayfarer.rb:48:96"
+end
 ```
-

data/docs/guides/development.md

@@ -0,0 +1,67 @@
+# Development
+
+## Release Procedure
+
+1. Ensure `Wayfarer::VERSION` was bumped appropriately.
+2. Ensure the version in wayfarer.gemspec matches.
+3. Open a release Pull Request develop -> master branch
+4. Merge the Pull Request
+5. Publish RubyGem and git tag as follows:
+
+```
+git checkout master
+git pull origin master --rebase
+bundle exec rake build
+gem push build/wayfarer-*.gem
+bundle exec rake clean
+git tag <VERSION>
+git push origin <VERSION>
+```
+
+## Conventions and guidelines
+
+* In source code, `url` refers to strings and `uri` refers to `Addressable::URI`
+* Avoid writing bash at all costs. Use Ruby instead
+
+## Design decisions and architecture
+
+### Navigate the web along URL patterns
+
+URLs are less prone to change than served markup.
+One reason for this is that changes to a URL's path can have negative
+consequences for its page ranking in search engines. Websites naturally implement
+architectural URL patterns like REST or expose surrogate keys.
+
+### Follow URLs verbatim as they appear in responses
+
+Normalized URLs are useful for deduplication, but URLs should be followed
+as they appear in responses. Navigating to normalized versions of URLs makes
+crawlers stick out from other user agents.
+
+### Tasks are version-less and don't persist metadata
+
+Tasks serialize to their URL and batch. No other data gets written to
+the message queue. There is also no need for versioning persisted tasks, since
+there will be never more to a task than URL and batch. All task metadata
+is ephemeral.
+
+### Why depend on Redis
+
+There are two core features that depend on Redis. First, per-batch acylicity is
+achieved by maintaining the set of processed URLs per batch in Redis.
+There's no option to follow links in a cyclic manner. Second, batch completion
+requires updating an integer value in Redis, and batch completion is a very
+useful feature, since most crawls should end eventually, and often you want to
+know when.
+
+### No configuration files
+
+Wayfarer can be configured through `Wayfarer.config` only, because `Wayfarer.config`
+may contain Ruby objects that don't de/serialize well, such as `Proc`s or `Set`s.
+
+### Features out of scope
+
+Wayfarer won't provide:
+
+* persistence or any sort of DOM data mapping abstractions
+* URL generation helpers

data/docs/guides/handlers.md

@@ -1,16 +1,16 @@
 # Handlers
 
-[Jobs](/jobs) can route tasks to handlers to delegate processing without
-writes to the message queue. Unlike jobs, handlers don't inherit from
-`ActiveJob::Base` and therefore cannot be enqueued. Handlers have routes, too,
-but they don't retrieve pages and a handler's router can be bypassed.
+Handlers are like [jobs](/jobs) but they don't inherit from `ActiveJob::Base`
+which is why they can't affect the message queue directly themselves.
+Instead, jobs and handlers can route tasks to other handlers. Handlers
+themselves have routes, but they can be bypassed.
 
-## Supported features
+## Handler capabilities
 
-Handlers support a subset of features compared to `Wayfarer::Base`:
+Like jobs, handlers support:
 
 * URL routing
-* enqueueing tasks with `#!ruby stage(*urls)`
+* staging tasks with `#!ruby stage(*urls)`
 * jobs can access the `user_agent` that retrieved the `page`
 * ad-hoc HTTP requests with `#!ruby fetch(url)`
 * callbacks, but only a subset of job callbacks

data/docs/guides/jobs.md

@@ -1,13 +1,15 @@
 # Jobs
 
 Jobs are [Active Job](https://edgeguides.rubyonrails.org/active_job_basics.html)s
-that use a DSL included from the `Wayfarer::Base` module to process [tasks](/guides/tasks)
-that they read from a message queue.
-Instead of implementing Active Job's `#perform` method yourself, you declare routes
-to instance methods, similiar to how web applications route incoming requests.
-Only URLs that match a [route](../routing) are requested or navigated to.
-The action method has access to the retrieved [page](../pages),
-the [user agent](../user-agents) that retrieved the page and the current task:
+that use a DSL to process [tasks](/guides/tasks) that they read from a message
+queue.
+
+Instead of implementing Active Job's `#perform` method yourself, you declare
+[routes](../routing) to instance methods, like web applications route incoming
+requests. Only URLs that match a route are retrieved and processed. All other
+URLs are considered successfully processed. The action has access to the
+retrieved [page](../pages), the [user agent](../user-agents) that retrieved the
+page and the current task:
 
 ```ruby
 class DummyJob < ActiveJob::Base
@@ -24,7 +26,7 @@ end
 ```
 
 You can start a crawl by appending a task to the message queue for the URL with
-`::crawl`. By default, a UUID is generated as the batch:
+`::crawl`. If you don't provide a batch, Wayfarer generates a UUID:
 
 ```ruby
 task = DummyJob.crawl("https://example.com")
@@ -51,10 +53,10 @@ You can also use Wayfarer's [CLI](../cli) to enqueue a task:
 wayfarer enqueue --batch my-batch DummyJob "https://example.com"
 ```
 
-## Navigating crawls
+## Following URLs
 
-Jobs navigate crawls by staging URLs with `#!ruby stage(urls)`. When you stage a URL, a normalized
-version of it is appended to an internal set. Once the action returns, all URLs
+Jobs navigate crawls by staging URLs with `stage(urls)`. When you stage a URL,
+it is appended verbatim to an internal set. Once the action returns, all URLs
 in the set are appended as tasks to the message queue.
 
 ```ruby
@@ -167,3 +169,44 @@ end
 
     Content-Types are compared regardless of their parameters. For example,
     `text/html; charset=UTF-8` is considered the same as `text/html`.
+
+## Handling errors
+
+!!! danger "Only ActiveJob error handling is supported"
+
+    Wayfarer exclusively supports ActiveJob's error handling. You cannot use
+    message queue-specific error handling, for example error handling with
+    `sidekiq_options` is unsupported. Otherwise batches get garbage-collected
+    too early as Wayfarer instruments ActiveJob.
+
+Wayfarer relies on ActiveJob's [error handling methods](https://guides.rubyonrails.org/active_job_basics.html#exceptions):
+
+* `retry_on` to retry jobs a number of times on certain errors:
+
+    ```ruby
+    class DummyJob < Wayfarer::Base
+      retry_on MyError, attempts: 3 do |job, error|
+        # This block runs once all 3 attempts have failed
+        # (1 initial attempt + 2 retries)
+      end
+    end
+    ```
+
+* `discard_on` to throw away jobs on certain errors:
+
+    ```ruby
+    class DummyJob < Wayfarer::Base
+      discard_on MyError do |job, error|
+        # This block runs once and buries the job
+      end
+    end
+    ```
+
+## Recreating user agents on certain errors
+
+You can configure a list of exception classes upon which user agents
+get recreated (see [User agent API]()):
+
+```ruby
+Wayfarer.config[:network][:renew_on] = [MyIrrecoverableError]
+```

data/docs/guides/networking/custom_adapters.md

@@ -1,141 +1,169 @@
 # User agent API
 
-Wayfarer retrieves web pages with user agents. There are two types of user
-agents: __stateful__ browsers which carry state and follow redirects implicitly,
-and __stateless__ HTTP clients, which handle redirects explicitly.
+Wayfarer retrieves URL contents with user agents. It supports two types of user
+agents:
 
-Because spawning browser processes or instantiating HTTP clients is expensive,
-Wayfarer keeps user agents in a pool and reuses them across jobs. Only on certain
-irrecoverable errors are individual user agents destroyed and recreated. For example,
-when a browser process crashes, it is replaced with a new one and checked back
-into the pool. The next job that checks out the user agent gets a fresh
-browser process.
+* __stateless__ HTTP clients which handle redirects explicitly.
+* __stateful__ browsers which carry state and follow redirects implicitly
+as they navigate to a URL
 
-## Base interface for custom user agents
+|                   | Stateless adapters | Stateful adapters |
+|-------------------|--------------------|-------------------|
+| interactive       | no                 | yes               |
+| redirect handling | explicit           | implicit          |
 
-You can implement both stateful and stateless agents by including the `Wayfarer::Networking::Strategy`
-module and defining callback methods. The interfaces for stateful and stateless
-share the following instance methods:
+Because spawning browser processes or instantiating HTTP clients is expensive,
+Wayfarer keeps user agents in a pool and reuses them across jobs. This means
+that browser state carries over between jobs, as each job checks out a previous
+job's user agent. When exceptions are raised, you must [handle them](/guides/jobs/#handling-errors).
+
+Only on certain irrecoverable errors are individual user agents
+destroyed and recreated. For example when a browser process crashes, it is
+replaced with a fresh browser process.
+
+## Implementing the user agent interfaces
+
+You implement both stateful and stateless agents by including the
+`Wayfarer::Networking::Strategy` module and defining callback methods:
+
+```mermaid
+classDiagram
+class BaseAgent {
+    <<Interface>>
+    +#create()*
+    +#destroy(instance)*
+    +::renew_on()$
+}
+
+class StatefulAgent {
+    <<Interface>>
+    +#navigate(instance, url)*
+    +#live(instance)*
+}
+
+class StatelessAgent {
+    <<Interface>>
+    +#fetch(instance, url)*
+}
+
+BaseAgent |>.. StatefulAgent : implements
+BaseAgent |>.. StatelessAgent : implements
+```
 
-* `#create` (__required__): Called when a new instance (browser process or HTTP client) is
-  needed.
-* `#destroy(instance)` (optional): Called when an instance should be destroyed. Browser
-  processes should be quit, and HTTP clients should be freed.
-* `#renew_on` (optional): Returns a list of exception classes upon which the existing
-  instance gets destroyed and replaced with a newly created one.
+Every user agent implementation must provide the `#create` instance callback
+which returns an initialized user agent. Typically, the optional
+`#destroy(instance)` instance callback is also implemented to free resources
+of an existing user agent.
 
-## Stateless interface
+Also a class method `::renew_on` can be defined which returns an array of
+exception classes upon which an instance of the user agent should get recreated
+(destroy-and-create).
 
-The stateless interface indicate HTTP 3xx redirect responses explicitly. This is how
-Wayfarer provides redirect handling out of the box, as there is a configurable limit
-on the number of retries to follow.
+### Stateless interface
 
-In addition to the base interface, stateless user agents implement `#fetch`
+In addition to the base interface, stateless user agents implement `#fetch(instance, url)`
 which fetches [pages](../pages) or indicates redirects:
 
-* `#fetch(instance, url)` (__required__): Called to retrieve a URL. Responses with a
+* `#create()` __(required)__
+* `#fetch(instance, url)` __(required)__: Called to retrieve a URL. Responses with a
   3xx status code must indicate the redirect URL by returning `redirect(url)`, since Wayfarer
   deals with redirects on your behalf to avoid redirect loops. All other status
   codes, including 4xx and 5xx, are considered successful and are indicated by calling
   `success(url:, body:, status_code:, headers:)`.
+* `#destroy(instance)` (optional)
+* `::renew_on` (optional)
 
-## Stateful interface
+The stateless interface indicate HTTP 3xx redirect responses explicitly. This is how
+Wayfarer provides redirect handling out of the box, as there is a configurable limit
+on the number of retries to follow.
 
-In addition to the base interface, stateful user agents implement two additional
-methods:
+!!! info "`redirect(url)` enqueues a task for the URL and stops processing the current task."
 
-* `#navigate(instance, url)` (__required__): Navigates the user agent to the given URL.
-  Stateful user agents follow redirects implicitly.
-* `#live(instance) -> Wayfarer::Page` (__required__): Turns the current user agent state
-  into a [page](../pages).
+!!! info "Pages with HTTP error status codes get routed"
 
-## Recreating user agents on error with `#renew_on`
+    If a HTTP request to a URL results in an error status code (for example, 404),
+    page retrieval is considered successful. This allows job actions to record
+    such data.
 
-Agents can optionally implement `#renew_on` to get themselves rereated on
-certain errors.
+### Stateful interface
 
-If `#fetch` or `#navigate` raise an exception and the exception class is listed
-in `#renew_on`, the instance is destroyed and recreated.
+In addition to the base interface, stateful user agents implement two additional
+instance callbacks:
 
-* `#renew_on` (optional): A list of exception classes upon which the existing instance gets
-  destroyed and replaced with a newly created one.
+* `#create()` __(required)__
+* `#navigate(instance, url)` __(required)__: Navigates the user agent to the given URL.
+* `#live(instance)` __(required)__: Turns the current user agent state
+into a [page](../pages) by calling `success(url:, body:, status_code:, headers:)`.
+* `#destroy(instance)` (optional)
+* `::renew_on` (optional)
 
 ## Example implementations
 
-=== "Stateful"
+=== "Stateless"
 
     ```ruby
-    class StatefulAgent
+    class StatelessAgent
       include Wayfarer::Networking::Strategy
 
-      # Required methods
-
-      def create
-        MyBrowser.new
+      def self.renew_on # optional
+        [MyIrrecoverableError]
       end
 
-      def navigate(browser, url)
-        browser.goto(url)
+      def create # required
+        MyClient.new
       end
 
-      def live(browser)
-        success(url: browser.url,
-                body: browser.body,
-                status_code: browser.status_code,
-                headers: browser.headers)
-      end
+      def fetch(client, url) # required
+        response = client.get(url)
 
-      # Optional methods
+        return redirect(response.redirect_url) if response.redirect?
 
-      def destroy(browser)
-        browser.quit
+        success(url: url,
+                body: response.body,
+                status_code: response.status_code,
+                headers: response.headers)
       end
 
-      def renew_on
-        [MyBrowser::IrrecoverableError]
+      def destroy(client) # optional
+        client.close
       end
     end
     ```
 
-=== "Stateless"
+=== "Stateful"
 
     ```ruby
-    class StatelessAgent
+    class StatefulAgent
       include Wayfarer::Networking::Strategy
 
-      # Required methods
-
-      def create
-        MyClient.new
+      def self.renew_on # optional
+        [MyIrrecoverableError]
       end
 
-      def fetch(client, url)
-        response = client.get(url)
-
-        return redirect(response.redirect_url) if response.redirect?
-
-        success(url: url,
-                body: response.body,
-                status_code: response.status_code,
-                headers: response.headers)
+      def create # required
+        MyBrowser.new
       end
 
-      # Optional methods
+      def navigate(browser, url) # required
+        browser.goto(url)
+      end
 
-      def destroy(client)
-        client.close
+      def live(browser) # required
+        success(url: browser.url,
+                body: browser.body,
+                status_code: browser.status_code,
+                headers: browser.headers)
       end
 
-      def renew_on # optional
-        [MyClient::IrrecoverableError]
+      def destroy(browser) # optional
+        browser.quit
       end
     end
     ```
 
-
-Register and use the strategy:
+Register and use the agent:
 
 ```ruby
-Wayfarer::Networking::Pool.registry[:my_agent] = MyAgent.new
+Wayfarer.config[:network][:agents][:my_agent] = MyAgent.new
 Wayfarer.config[:network][:agent] = :my_agent
 ```

data/docs/guides/networking/http.md

@@ -6,7 +6,7 @@ also alongside automated browsers.
 ## Ad-hoc GET requests
 
 When automating browsers, it can be useful to additionally retrieve another page
-over plain HTTP. Jobs can fetch URLs to [pages](/pages) with `#http`:
+or even the current page over plain HTTP. Jobs can fetch URLs to [pages](/pages) with `#http`:
 
 ```ruby
 class DummyJob < Wayfarer::Base

data/docs/guides/pages.md

@@ -47,6 +47,30 @@ MIME types:
 * `text/xml` or `application/xml` to [`#!ruby Nokogiri::XML::Document`](https://www.rubydoc.info/github/sparklemotion/nokogiri/Nokogiri/XML/Document)
 * `application/json` to `Hash`
 
+### Implementing a custom response body parser
+
+You can register an object that implements a `#parse` method for any MIME type:
+
+```ruby
+class MyJPEGParser
+  def parse(body)
+    # Read EXIF metadata here.
+    # Return value is accessible as `page.doc`
+  end
+end
+
+Wayfarer::Parsing.registry["image/jpeg"] = MyJPEGParser.new
+```
+
+!!! warning "`#parse` must be thread-safe!"
+
+!!! info "Handling responses without a Content-Type"
+
+    If a response has no `Content-Type` header, Wayfarer falls back to
+    `application/octet-stream`. A parser registered for
+    `application/octet-stream` will hence also handle all responses without
+    a Content-Type.
+
 ## Live pages
 
 `#!ruby page` initially returns a snapshot of the browser state
@@ -83,28 +107,6 @@ end
     default `#!ruby :http` user agent. Instead, stateless user agents always
     return the same page object.
 
-### Implementing a custom response body parser
-
-You can register an object that implements a `#parse` method for any MIME type:
-
-```ruby
-class MyJPEGParser
-  def parse(body)
-    # Read EXIF metadata here.
-    # Return value is accessible as `page.doc`
-  end
-end
-
-Wayfarer::Parsing.registry["image/jpeg"] = MyJPEGParser.new
-```
-
-!!! info "Handling responses without a Content-Type"
-
-    If a response has no `Content-Type` header, Wayfarer falls back to
-    `application/octet-stream`. A parser registered for
-    `application/octet-stream` will hence also handle all responses without
-    a Content-Type.
-
 ## Accessing page metadata with MetaInspector
 
 You have access to a [MetaInspector](https://github.com/jaimeiniesta/metainspector)