wayfarer 0.4.6 → 0.4.7

data/docs/cookbook/navigation.md

@@ -4,12 +4,12 @@
 
     ```ruby
     class DummyJob < Wayfarer::Base
-      route { to :index }
+      route.to :index
 
       def index
-        agent.goto("https://example.com")
-        agent.back
-        agent.forward
+        user_agent.goto("https://example.com")
+        user_agent.back
+        user_agent.forward
       end
     end
     ```
@@ -18,12 +18,12 @@
 
     ```ruby
     class DummyJob < Wayfarer::Base
-      route { to :index }
+      route.to :index
 
       def index
-        agent.navigate.to("https://example.com")
-        agent.navigate.back
-        agent.navigate.forward
+        user_agent.navigate.to("https://example.com")
+        user_agent.navigate.back
+        user_agent.navigate.forward
       end
     end
     ```
@@ -32,12 +32,12 @@
 
     ```ruby
     class DummyJob < Wayfarer::Base
-      route { to :index }
+      route.to :index
 
       def index
-        agent.visit("https://example.com")
-        agent.go_back
-        agent.go_forward
+        user_agent.visit("https://example.com")
+        user_agent.go_back
+        user_agent.go_forward
       end
     end
     ```

data/docs/cookbook/querying_html.md

@@ -6,7 +6,7 @@ See: [Nokogiri: Searching an HTML / XML Document](https://nokogiri.org/tutorials
 
     ```ruby
     class DummyJob < Wayfarer::Base
-      route { to :index }
+      route.to :index
 
       def index
         page.doc.css("html")
@@ -19,7 +19,7 @@ See: [Nokogiri: Searching an HTML / XML Document](https://nokogiri.org/tutorials
 
     ```ruby
     class DummyJob < Wayfarer::Base
-      route { to :index }
+      route.to :index
 
       def index
         browser.at_css("html")
@@ -32,7 +32,7 @@ See: [Nokogiri: Searching an HTML / XML Document](https://nokogiri.org/tutorials
 
     ```ruby
     class DummyJob < Wayfarer::Base
-      route { to :index }
+      route.to :index
 
       def index
         browser.find_elements(css: "html")

data/docs/cookbook/screenshots.md

@@ -6,7 +6,7 @@ Taking screenshots requires automating a browser.
 
     ```ruby
     class DummyJob < Wayfarer::Base
-      route { to :index }
+      route.to :index
 
       def index
         browser.screenshot(path: "screenshot.png")
@@ -18,7 +18,7 @@ Taking screenshots requires automating a browser.
 
     ```ruby
     class DummyJob < Wayfarer::Base
-      route { to :index }
+      route.to :index
 
       def index
         browser.save_screenshot("screenshot.png")

data/docs/cookbook/user_agent.md

@@ -3,5 +3,5 @@
 See: [Guides: Networking: HTTP request headers](/guides/networking#http-request-headers)
 
 ```ruby
-Wayfarer.config.network.http_headers = { "User-Agent" => "MyCrawler ..." }
+Wayfarer.config[:network][:http_headers] = { "User-Agent" => "MyCrawler ..." }
 ```

data/docs/design.md

@@ -0,0 +1,36 @@
+# Design decisions
+
+## Navigate the web along URL patterns
+
+URLs are less prone to change than served markup.
+One reason for this is that changes to a URL's path can have a negative effect
+on its page ranking in search engines. Many websites also implement common
+architectural URL patterns, for example REST and its variations, that
+lend themselves to pattern matching.
+
+## Follow URLs verbatim
+
+Normalized URLs are useful for deduplication, but URLs should be followed
+as they appear in responses. Navigating to normalized versions of URLs makes
+crawlers stick out from other user agents, for example.
+
+## Tasks are version-less and don't persist metadata
+
+Tasks serialize to their URL and batch. No other data gets written to
+the message queue. Wayfarer aims to minimise job payloads.
+There is also no need for versioning persisted tasks, since there is only one
+version of a task: URL and batch.
+
+## Why depend on Redis
+
+There are two core features that depend on Redis. First, per-batch acylicity is
+achieved by maintaining the set of processed URLs per batch in Redis.
+There's no option to follow links in a cyclic manner. Second, batch completion
+requires updating an integer value in Redis, and batch completion is a very
+useful feature, since most crawls should end eventually, and often you want to
+know when.
+
+## Persistence and document mapping not included
+
+Like Active Job, Wayfarer is not concerned with persistence.
+Model <-> DOM mapping abstractions are also out of scope.

data/docs/guides/callbacks.md

@@ -1,145 +1,43 @@
 # Callbacks
 
-## Active Job callbacks
+Wayfarer supports a number of callbacks in addition to
+[ActiveJob's](https://edgeguides.rubyonrails.org/active_job_basics.html#callbacks).
 
-Wayfarer naturally supports all of [Active Job's life cycle callbacks](https://edgeguides.rubyonrails.org/active_job_basics.html#callbacks).
+## Available callbacks
 
-## `before_fetch`
-
-Runs before a job fetches a page, either by making an HTTP request, or by
-navigating a browser to its task URL.
-
-```ruby
-class DummyJob < Wayfarer::Base
-  before_fetch :do_something
-
-  private
-
-  def do_something
-    # before the task.url is fetched
-  end
-end
-```
-
-## `before_action`
-
-Runs after a page was fetched, before an action method is called.
-
-```ruby
-class DummyJob < Wayfarer::Base
-  before_action :do_something
-
-  private
-
-  def do_something
-    # page is available at this point
-  end
-end
-```
+* `before_fetch`
+* `around_fetch`
+* `after_fetch`
+* `before_action`
+* `around_action`
+* `after_action`
+* `after_batch`
 
 ## `after_batch`
 
-Runs once the last job in a batch performed:
-
-```ruby
-class DummyJob < Wayfarer::Base
-  after_batch do
-    # All jobs in batch done
-  end
-end
-```
-
-Internally, a batch counter is in-/decremented on certain events. Once the
-counter reaches zero, `after_batch` callbacks runs in declaration order.
-
-The counter is incremented when within the batch:
+You can register `after_batch` callbacks that run when there are no more tasks
+to process in a batch. Wayfarer instruments job execution and in- or decrements
+an integer counter in Redis on certain events. When the counter reaches zero,
+the current job's `after_batch` callbacks run.
 
-* A job is enqueued.
+## Conditional callbacks
 
-The counter is decremented when:
-
-* A job succeeds.
-* A job errors due to an unhandled exception.
-* A job is discarded due to an exception.
-* A job errors and thereyby exhausts its maximum attempts.
-
-!!! attention "Batch callbacks can fail jobs"
-
-    If the last job's `after_batch` callbacks raises an exception, this can lead
-    to the job getting retried. If the exception raised by the callback is
-    unhandled or discarded, the callback never fully runs.
-
-## Callback options
-
-### Definition styles
-
-Callbacks can be registered either by supplying a block or a symbol identifying
-a callback instance method:
+You can make callbacks conditional with the `#!ruby :if` and `#!ruby :unless`
+keywords, for example to run a callback for some route `action` only:
 
 ```ruby
-class DummyJob < Wayfarer::Base
-  before_action do
-    # ...
-  end
-
-  before_action :my_callback
+class DummyJob < ActiveJob::Base
+  include Wayfarer::Base
 
-  private
+  route.host "example.com", to: :example
+  route.to :fallback
 
-  def my_callback
+  before_action unless: -> { action == :fallback } do
     # ...
   end
-end
-```
-
-### Conditionals
-
-Callbacks can be registered conditionally with the `:if` and `:unless` keywords:
-
-```ruby
-class DummyJob < Wayfarer::Base
-  before_fetch :my_callback, if: :my_condition
-
-  private
-
-  def my_callback
-  end
 
-  def my_condition
-  end
+  # ...
 end
 ```
 
-Callbacks can be registered for certain action methods only with the `:only` and
-`:except` keywords:
-
-```ruby
-class DummyJob < Wayfarer::Base
-  before_fetch :do_something, only: :foo
-
-  before_fetch except: [:foo, :qux] do
-    # runs only before bar
-  end
-
-  def foo
-  end
-
-  def bar
-  end
-end
-
-```
-
-### Early termination
-
-Callbacks that return `false` halt the callback chain:
-
-```ruby
-class DummyJob < Wayfarer::Base
-  before_action { false }
-
-  before_action do
-    # never runs
-  end
-end
-```
+You can also pass a symbol instead of a block to call an instance method.

data/docs/guides/configuration.md

@@ -13,27 +13,27 @@ Wayfarer parses environment variables into a runtime configuration
 
 ```ruby
 # Which user agent to use to process tasks
-Wayfarer.config.network.agent = :http # or :ferrum, :selenium
+Wayfarer.config[:network][:agent] = :http # or :ferrum, :selenium
 
 # How many user agents to instantiate
-Wayfarer.config.network.pool_size = 3
+Wayfarer.config[:network][:pool_size] = 3
 
 # How long an agent may be used while processing a task
-Wayfarer.config.network.pool_timeout = 5000
+Wayfarer.config[:network][:pool_timeout] = 5000
 
 # Ferrum options
-Wayfarer.config.ferrum.options = {}
+Wayfarer.config[:ferrum][:options] = {}
 
 # Selenium driver to use
-Wayfarer.config.selenium.driver = :chrome
+Wayfarer.config[:selenium][:driver] = :chrome
 
 # Selenium HTTP client read timeout
-Wayfarer.config.selenium.client_timeout = 10 # seconds
+Wayfarer.config[:selenium][:client_timeout] = 10 # seconds
 
 # Selenium options
-Wayfarer.config.selenium.options = { url: "http://chrome" }
+Wayfarer.config[:selenium][:options] = { url: "http://chrome" }
 
 # HTTP request headers (Selenium is unsupported)
-Wayfarer.config.network.http_headers = { "Field" => "Value" }
+Wayfarer.config[:network][:http_headers] = { "Field" => "Value" }
 ```
 

data/docs/guides/handlers.md

@@ -0,0 +1,60 @@
+# Handlers
+
+[Jobs](/jobs) can route tasks to handlers to delegate processing without
+writes to the message queue. Unlike jobs, handlers don't inherit from
+`ActiveJob::Base` and therefore cannot be enqueued. Handlers have routes, too,
+but they don't retrieve pages and a handler's router can be bypassed.
+
+## Supported features
+
+Handlers support a subset of features compared to `Wayfarer::Base`:
+
+* URL routing
+* enqueueing tasks with `#!ruby stage(*urls)`
+* jobs can access the `user_agent` that retrieved the `page`
+* ad-hoc HTTP requests with `#!ruby fetch(url)`
+* callbacks, but only a subset of job callbacks
+* Content-Type filtering
+
+```ruby
+class ExampleHandler
+  include Wayfarer::Handler
+
+  route.to: :index
+
+  def index
+    task # => #<Wayfarer::Task>
+    page # => #<Wayfarer::Page>
+    user_agent # => Browser or HTTP client
+  end
+end
+
+class DummyJob < ActiveJob::Base
+  include Wayfarer::Base
+
+  route.host "example.com", to: ExampleHandler
+end
+```
+
+You can also bypass a handler's router and route directly to an instance
+method:
+
+```ruby
+class DummyJob < ActiveJob::Base
+  include Wayfarer::Base
+
+  route.host "example.com", to: [ExampleHandler, :index]
+end
+
+class ExampleHandler
+  include Wayfarer::Handler
+
+  def index
+    task # => #<Wayfarer::Task>
+    page # => #<Wayfarer::Page>
+    user_agent # => Browser or HTTP client
+  end
+end
+```
+
+!!! `before_action` callbacks

data/docs/guides/index.md

@@ -0,0 +1 @@
+hello

data/docs/guides/jobs/error_handling.md

@@ -0,0 +1,40 @@
+# Error handling
+
+!!! danger "Only ActiveJob error handling is supported"
+
+    Wayfarer exclusively supports ActiveJob's error handling. You cannot use
+    message queue-specific error handling, for example error handling with
+    `sidekiq_options` is unsupported. Otherwise batches get garbage-collected
+    too early as Wayfarer instruments ActiveJob.
+
+Wayfarer relies on ActiveJob's [error handling methods](https://guides.rubyonrails.org/active_job_basics.html#exceptions):
+
+* `retry_on` to retry jobs a number of times on certain errors:
+
+    ```ruby
+    class DummyJob < Wayfarer::Base
+      retry_on MyError, attempts: 3 do |job, error|
+        # This block runs once all 3 attempts have failed
+        # (1 initial attempt + 2 retries)
+      end
+    end
+    ```
+
+* `discard_on` to throw away jobs on certain errors:
+
+    ```ruby
+    class DummyJob < Wayfarer::Base
+      discard_on MyError do |job, error|
+        # This block runs once and buries the job
+      end
+    end
+    ```
+
+## Recreating user agents on certain errors
+
+You can configure a list of exception classes upon which user agents
+get recreated (see [User agent API]()):
+
+```ruby
+Wayfarer.config[:network][:renew_on] = [MyIrrecoverableError]
+```

data/docs/guides/jobs.md

@@ -1,78 +1,124 @@
 # Jobs
 
-Jobs are Ruby classes that process [tasks](/guides/tasks) and look as follows:
+Jobs are [Active Job](https://edgeguides.rubyonrails.org/active_job_basics.html)s
+that use a DSL included from the `Wayfarer::Base` module to process [tasks](/guides/tasks)
+that they read from a message queue.
+Instead of implementing Active Job's `#perform` method yourself, you declare routes
+to instance methods, similiar to how web applications route incoming requests.
+Only URLs that match a [route](../routing) are requested or navigated to.
+The action method has access to the retrieved [page](../pages),
+the [user agent](../user-agents) that retrieved the page and the current task:
 
 ```ruby
-class DummyJob < Wayfarer::Base
-  route { to :index }
+class DummyJob < ActiveJob::Base
+  include Wayfarer::Base
+
+  route.to :index
 
   def index
+    task # => #<Wayfarer::Task>
+    page # => #<Wayfarer::Page>
+    user_agent # => Browser or HTTP client
   end
 end
 ```
 
-Here is how to enqueue a task for a URL:
+You can start a crawl by appending a task to the message queue for the URL with
+`::crawl`. By default, a UUID is generated as the batch:
 
 ```ruby
-DummyJob.crawl("https://example.com")
+task = DummyJob.crawl("https://example.com")
+# => #<Wayfarer::Task url="https://example.com", batch="498a13e0-...">
 ```
 
-This is the same as calling the Active Job API directly and passing a task
-and a random batch:
+This is exactly the same as calling Active Job's `#perform_later` and passing a
+task directly:
 
 ```ruby
 task = Wayfarer::Task.new("https://example.com", SecureRandom.uuid)
 DummyJob.perform_later(task)
 ```
 
-A batch can be specified with `::crawl`, too:
+Instead of a generated UUID, you can also set your own batch:
 
 ```ruby
 DummyJob.crawl("https://example.com", batch: "my-batch")
 ```
 
-## Current task
+You can also use Wayfarer's [CLI](../cli) to enqueue a task:
+
+```sh
+wayfarer enqueue --batch my-batch DummyJob "https://example.com"
+```
+
+## Navigating crawls
+
+Jobs navigate crawls by staging URLs with `#!ruby stage(urls)`. When you stage a URL, a normalized
+version of it is appended to an internal set. Once the action returns, all URLs
+in the set are appended as tasks to the message queue.
+
+```ruby
+class DummyJob < ActiveJob::Base
+  include Wayfarer::Base
+
+  route.to :index
+
+  def index
+    # Follow all out-going links of the page
+    stage page.meta.links.external
+  end
+end
+```
+
+## Accessing the current task
 
-Jobs consume [tasks](../tasks) from a message queue. The currently processed
-task is accessible like so:
+If the task's URL matched a [route](../routing), the URL is retrieved over the network,
+and the method that was routed to is called. The task is available as `#task`:
 
 ```ruby
-class DummyJob < Wayfarer::Base
-  route { to :index }
+class DummyJob < ActiveJob::Base
+  include Wayfarer::Base
+
+  route.to :index
 
   def index
-    task.url   # => "https://example.com"
+    task.url # => "https://example.com"
     task.batch # => "my-batch"
   end
 end
 ```
 
-## Current page
+## Accessing the current page
 
-A task's URL contents get fetched into a [page](../pages) object if the task URL
-matched a route:
+You have access to the retrieved [page](../pages):
 
 ```ruby
-class DummyJob < Wayfarer::Base
-  route { to :index, host: "example.com" }
+class DummyJob < ActiveJob::Base
+  include Wayfarer::Base
+
+  route.to :index
 
   def index
     page.url         # => "https://example.com"
     page.body        # => "<html>..."
     page.status_code # => 200
     page.headers     # { "Content-Type" => ... }
+    page.doc         # Only present for certain Content-Types
   end
 end
 ```
 
-## URL parameters
+## Routing URLs to methods and extracting `params`
 
-Jobs can extract data from URLs with their router:
+Jobs have a routing DSL that allows you to map URLs to methods and extract
+URL data:
 
 ```ruby
-class DummyJob < Wayfarer::Base
+class DummyJob < ActiveJob::Base
+  include Wayfarer::Base
+
   route do
-    path "/users/:id/profile"
+    path "/users/:id/profile", to: :index
   end
 
   def index
@@ -80,22 +126,44 @@ class DummyJob < Wayfarer::Base
   end
 end
 
-DummyJob.crawl("https://example.com/users/42/profile")
+DummyJob.crawl("https://example.com/users/42/profile?foo=bar")
 ```
 
+## Controlling the user agent
 
-## User agent
-
-The HTTP client or automated browser that fetched the URL is available:
+You can control the browser or HTTP client that retrieved the page:
 
 ```ruby
-Wayfarer.config.network.agent = :ferrum # Chrome DevTools Protocol
+Wayfarer.config[:network][:agent] = :ferrum # Chrome DevTools Protocol
 
-class DummyJob < Wayfarer::Base
-  route { to :index }
+class DummyJob < ActiveJob::Base
+  include Wayfarer::Base
+
+  route.to :index
 
   def index
-    browser.save_screenshot("capture.png")
+    user_agent.save_screenshot("capture.png")
   end
 end
 ```
+
+## Restricting the processed Content-Types
+
+By default, jobs process pages regardless of their Content-Type response
+header. You can allow a list of Content-Types as strings and Regexps and
+opt out of the default behaviour. Once at least one Content-Type is allowed,
+other Content-Types don't get processed:
+
+```ruby
+class DummyJob < ActiveJob::Base
+  include Wayfarer::Base
+
+  content_type "text/html", "application/json"
+  content_type /xml/
+end
+```
+
+!!! info "HTTP parameters in Content-Types are ignored for comparison"
+
+    Content-Types are compared regardless of their parameters. For example,
+    `text/html; charset=UTF-8` is considered the same as `text/html`.

data/docs/guides/navigation.md

@@ -13,7 +13,7 @@ set first.
 
 ```ruby
 class DummyJob < Wayfarer::Base
-  route { to :index }
+  route.to :index
 
   def index
     stage page.meta.links.all