wayfarer 0.4.1 → 0.4.2

data/docs/guides/performance.md

@@ -2,6 +2,114 @@
 
 How to write performant crawlers with Wayfarer.
 
+## Use a sufficiently sized user agent pool
+
+Automated browser processes or HTTP clients are kept in a [connection pool]() of
+static size. This avoids having to re-establish browser processes and enables
+their reuse.
+
+If the size of the pool is too small, the pool is a
+bottleneck. For example, if your message queue adapter uses 8 threads, but the
+pool only contains 1 user agent, the remaining 7 threads block until the agent
+is checked back in to the pool for use by one of the blocked threads.
+
+There is no reliable way to detect the number of threads of the underlying
+message queue adapter. The pool size should equal the number of threads;
+
+```ruby
+Wayfarer.config.network.pool_size = 8 # defaults to 1
+```
+
+### Job shedding
+
+There is a maximum number of seconds that jobs wait when checking out a user
+agent from the pool. Once this time is exceeded,
+a `Wayfarer::UserAgentTimeoutError` is raised. By default, the timeout is 10
+seconds.
+
+This hints there are more threads in use than user agents in the pool.
+
+## Stage less URLs
+
+Staging less URLs saves space and time:
+
+* Less tasks written to the message queue
+* Less time spent consuming tasks
+* Less time spent filtering URLs with Redis
+
+Wayfarer maintains a set of processed URLs for a batch in Redis. Every staged
+URL is checked for inclusion in this set before it gets appended as a task to
+the message queue.
+
+A common pattern is to stage all links of a page, and rely on routing to fetch
+only the relevant ones:
+
+```ruby
+class DummyJob < Wayfarer::Base
+  route { to: index, host: "example.com" }
+
+  def index
+    stage page.meta.links.all
+  end
+end
+```
+
+Pages commonly contain a large number of URLs.
+
+Every staged URL is:
+
+1. Normalized to a canonical form, for example by sorting query parameters
+   alphabetically.
+2. Checked for inclusion in the batch Redis set or discarded.
+3. Written to the message queue.
+4. Consumed from the queue and matched against the router.
+5. Fetched, if a route matches.
+
+Narrowing down the links in the document to follow speeds up the process.
+For example using Nokogiri, interesting links can be identified with a CSS
+selector:
+
+```ruby
+class DummyJob < Wayfarer::Base
+  route { to: index, host: "example.com" }
+
+  def index
+    stage interesting_links
+  end
+
+  private
+
+  def interesting_links
+    page.doc.css("a.interesting").map { |elem| elem["href"] }
+  end
+end
+```
+
+Because the router only accepts the single hostname `example.com`, the job can
+also ensure it stages only internal URLs by intersecting them with the
+interesting ones:
+
+```ruby
+class DummyJob < Wayfarer::Base
+  route { to: index, host: "example.com" }
+
+  def index
+    stage interesting_internal_links
+  end
+
+  private
+
+  def interesting_internal_links
+    page.meta.links.internal & interesting_links
+  end
+
+  def interesting_links
+    page.doc.css("a.interesting").map { |elem| elem["href"] }
+  end
+end
+```
+
+
 ## Use Redis >= 6.2.0
 
 Redis 6.2.0 introduced the

data/docs/guides/reliability.md

@@ -0,0 +1,41 @@
+# Reliablity
+
+## Durability
+
+Wayfarer executes atop reliable messages queues such as Sidekiq, Resque,
+RabbitMQ, etc. Its configuration is independent of the underlying queue
+infrastructure it reads from and writes to.
+
+## Self-healing user agents
+
+Wayfarer handles the scenario where a remote browser process has crashed and
+must be replaced by a fresh browser process.
+
+This can be tested locally by automating a browser with headless mode turned
+off, and then closing the opened browser window: The current job fails, but the
+next job has access to a newly established browser session again.
+
+For example Ferrum might raise `Ferrum::DeadBrowserError`. Wayfarer's
+user agents are self-healing and react to these kinds of errors internally. When
+a browser window is closed, the Ferrum user agent attempts to establish a new
+browser process as a replacement, for the next job to use.
+
+[Wayfarer never swallows exceptions](/guides/error_handling). This means
+that even though the user agent might heal itself, jobs still need to explicitly
+retry browser errors:
+
+```ruby
+class Foobar < Wayfarer::Base
+  route { to: :index }
+
+  retry_on Ferrum::DeadBrowserError, attempts: 3, wait: :exponentially_longer
+
+  # ...
+end
+```
+
+This leads to log entries like:
+
+```
+Retrying DummyJob in 3 seconds, due to a Ferrum::DeadBrowserError.
+```

data/docs/guides/routing/steering.md

@@ -0,0 +1,30 @@
+# Steering
+
+A job's router can receive arguments computed dynamically by `::steer`.
+Steering enables [batch routing](/cookbook/batch_routing).
+
+For example, the following router has hostname and path hard-coded:
+
+```ruby
+class DummyJob < Wayfarer::Base
+  route do
+    host "example.com", path: "/contact", to: :index
+  end
+end
+```
+
+Instead, hostname and path could be provided by `::steer`, too:
+
+```ruby
+class DummyJob < Wayfarer::Base
+  route do |hostname, path|
+    host hostname, path: path, to: :index
+  end
+
+  steer do |_task|
+    ["example.com", "/contact"]
+  end
+end
+```
+
+Note that `steer` yields the current [task](/guides/tasks).

data/docs/guides/tasks.md

@@ -1,38 +1,14 @@
 # Tasks
 
-Tasks are the units of work processed by jobs. A task consists of:
+Tasks are the immutable units of work processed by [jobs](/guides/jobs). A task
+consists of:
 
-1. The URL to process
-2. The batch the task belongs to
-
-Like URLs, batches are strings. Within a batch, every URL gets processed at most
-once.
-
-Tasks get appended to the end of a message queue, and consumed gfrom their
-beginning by jobs.
-
-When jobs process tasks, they search their routing tree for a matching route.
-URLs that match no route are not retrieved, and their task considered
-successfully processed without further action.
-
-## Task Metadata
-
-At runtime, tasks take the shape of a `Wayfarer::Task` object. While only URL
-and batch are persisted to message queues, tasks carry an arbitrarily assignable
-`metadata` object:
-
-```ruby
-task          # => #<Task url="https://example.com" batch="547b761-d0ad-...">
-task.metadata # => #<OpenStruct>
-task.metadata.my_piece_of_information = "hello"
-```
-
-`task.metadata` is ephemeral and only accessible at runtime.
-
-Once a job consumes a task, the job instance becomes accessible on it:
-
-```
-task.job # => #<DummyJob ...>
-```
+1. The __URL__ to process
+    * Within a batch, every URL gets processed at most once.
 
+2. The __batch__ the task belongs to
+    * Like URLs, batches are strings.
 
+Tasks get appended to the end of a message queue, and consumed from the
+beginning. Because jobs can enqueue other tasks, jobs are both consumers
+and producers of tasks.

data/docs/reference/api/base.md

@@ -1,162 +1,48 @@
 ---
-title: Base
+title: Wayfarer::Base
 ---
 
 # `Wayfarer::Base`
 
-Base functionality every job is equipped with:
-
-* Router for connecting URLs with instance methods and collecting
-  data.
-* Access to a parsed document representation.
-* Access to the browser or HTTP connection that retrieved the document.
-* Ability to stage URLs for future processing.
-* Ability to inject middleware before or after the worker.
+Wayfarer's complete job API.
 
 ---
 
-### `::route -> Wayfarer::Routing::Route`
-:   The class route that ties URLs to action methods via URL rules.
-
-    ##### Example
-
-    !!! example "Defining routes"
+### `::route`
+:   Draw routes to instance methods.
 
-        ```ruby
-        class DummyJob < Wayfarer::Base
-          route.to :index
+---
 
-          def index
-          end
-        end
-        ```
+### `::steer { (Wayfarer::Task) -> [any] }`
+:   Provide router arguments.
 
 ---
 
 ### `#task -> Wayfarer::Task`
 :   The currently processing task.
 
-    !!! example "Inspecting the current task"
-
-        ```ruby
-        class DummyJob < Wayfarer::Base
-          route.to :index
-
-          def index
-            task       # => #<Wayfarer::Task ...>
-            task.url   # => "https://example.com"
-            task.batch # => "2287ae65-359e-4dc0-..."
-          end
-        end
-        ```
-
 ---
 
 ### `#params -> Hash`
 :   URL parameters collected from the matching route.
 
-    !!! example "Accessing URL parameters"
-
-        ```ruby
-        class DummyJob < Wayfarer::Base
-          route.path "/users/:user_id/images/:id", to: :index
-
-          def index
-            params # => { "user_id" => ..., "id" => ... }
-          end
-        end
-        ```
-
 ---
 
-### `#stage(*urls) -> void`
+### `#stage(String | [String]) -> void`
 :   Add URLs to a processing set. URLs already processed within the
     current batch get discarded are not enqueued. Every staged URL gets
     normalized.
 
-    !!! example "Staging a URL"
-
-        ```ruby
-        class DummyJob < Wayfarer::Base
-          route.to :index
-
-          def index
-            stage "https://example.com"
-          end
-        end
-        ```
-
-    !!! example "Staging all URLs contained in the current page"
-
-        ```ruby
-        class DummyJob < Wayfarer::Base
-          route.to :index
-
-          def index
-            stage page.meta.links.all
-          end
-        end
-        ```
-
 ---
 
-### `#browser -> Ferrum::Browser | Selenium::WebDriver | nil`
-:   The browser process used to retrieve the current response.
-    If the configured agent is the default `:http`, `nil` is returned.
-
-    Guides:
-
-    * [Ferrum (Chrome DevTools Protocol)]()
-    * [Selenium]()
-
-    !!! example "Accessing a Google Chrome process"
-
-        ```ruby
-        Wayfarer.config.network.agent = :ferrum
-
-        class DummyJob < Wayfarer::Base
-          route.to :index
-
-          def index
-            browser # => #<Ferrum::Browser ...>
-          end
-        end
-        ```
-
-    !!! example "Accessing a Selenium WebDriver"
-
-        ```ruby
-        Wayfarer.config.network.agent = :selenium
-
-        class DummyJob < Wayfarer::Base
-          route.to :index
-
-          def index
-            browser # => #<Selenium::WebDriver ...>
-          end
-        end
-        ```
+### `#browser -> Object`
+:   The user agent that retrieved the current page.
 
 ---
 
-### `#page(live: false) -> Page`
+### `#page(live: true | false) -> Page`
 :   The page representing the response retrieved from the currently
     processing URL.
 
-    With `page(live: true)` passed, the returned `Page` reflects the current
-    browser DOM. No-op when the `net/http` agent is in use. Calls to
-    `page()` without the keyword return the most recent page.
-
----
-
-### `#doc -> Nokogiri::HTML | Nokogiri::XML | Hash`
-:   The parsed HTTP response body depending on the Content-Type:
-    * When XML or HTML then a parsed Nokogiri document
-    * When JSON, a parsed Hash
-
----
-
-### `#middleware -> [Middleware]`
-:   Template method that allows workers to inject middleware before or
-    after themselves.
-
+    With `live: true` called, a fresh `Page` is returned that reflects the
+    current browser DOM. Calls to `#page` return the most recent page.

data/docs/reference/api/route.md

@@ -1,5 +1,5 @@
 ---
-title: Route
+title: Wayfarer::Routing::DSL
 ---
 
 # `Wayfarer::Route`

data/docs/reference/cli.md

@@ -14,14 +14,6 @@ All [environment variables](../environment_variables) are respected.
 
 :   Generates a new project directory `NAME`.
 
-    ##### Example
-
-    !!! example "Create a new project directory"
-
-        ```
-        wayfarer generate project foobar
-        ```
-
 ## `wayfarer job`
 
 ### `wayfarer job perform JOB URL`
@@ -35,20 +27,6 @@ All [environment variables](../environment_variables) are respected.
       talking to an actual server.
     * `--batch=BATCH`: Set the job's batch. By default, a UUID is generated.
 
-    ##### Examples
-
-    !!! example "Perform a job"
-
-        ```
-        wayfarer job perform DummyJob https://example.com
-        ```
-
-    !!! example "Specify a batch"
-
-        ```
-        wayfarer job perform --batch=my-batch DummyJob https://example.com
-        ```
-
 ### `wayfarer job enqueue JOB URL`
 
 :   Enqueues `JOB` with `URL` to the configured Active Job backend.
@@ -57,20 +35,6 @@ All [environment variables](../environment_variables) are respected.
 
     * `--batch=BATCH`: Set the job's batch. By default, a UUID is generated.
 
-    ##### Examples
-
-    !!! example "Enqueue a job"
-
-        ```
-        wayfarer job enqueue DummyJob https://example.com
-        ```
-
-    !!! example "Specify a batch"
-
-        ```
-        wayfarer job enqueue --batch=my-batch DummyJob https://example.com
-        ```
-
 ### `wayfarer job execute JOB URL`
 
 :   Execute `JOB` with `URL` by using the
@@ -86,54 +50,12 @@ All [environment variables](../environment_variables) are respected.
     * `--min-threads`: Minimum number of threads to use. Default: 1
     * `--max-threads`: Maximum number of threads to use. Default: 1
 
-    ##### Examples
-
-    !!! example "Enqueue a job"
-
-        ```
-        wayfarer job execute DummyJob https://example.com
-        ```
-
-    !!! example "Mock Redis"
-
-        ```
-        wayfarer job execute --mock-redis DummyJob https://example.com
-        ```
-
-    !!! example "Specify a batch"
-
-        ```
-        wayfarer job execute --batch=my-batch DummyJob https://example.com
-        ```
-
-    !!! example "Use up to 4 threads"
-
-        ```
-        wayfarer job execute --min-threads=1 --max-threads=4 DummyJob https://example.com
-        ```
-
 ## `wayfarer route`
 
 ### `wayfarer route result JOB URL`
 
 :   Prints the result of invoking `JOB`'s router with `URL`.
 
-    ##### Example
-
-    !!! example "Route a URL"
-
-        ```
-        wayfarer route result DummyJob https://example.com
-        ```
-
 ### `wayfarer route tree JOB URL`
 
 :   Visualises the routing tree result of invoking `JOB`'s router with `URL`.
-
-    ##### Example
-
-    !!! example "Visualise the routing tree"
-
-        ```
-        wayfarer route tree DummyJob https://example.com
-        ```

data/docs/reference/configuration_keys.md

@@ -10,7 +10,7 @@ hide:
 | Runtime config key     | Environment variable                 | Description                                 | Default                          | Supported values                    |
 | ---------------------- | ------------------------------------ | ------------------------------------------- | -------------------------------- | ----------------------------------- |
 | `network.agent`        | `WAYFARER_NETWORK_AGENT`             | The user agent to use.                      | `:http`                          | `:http`, `:ferrum`, `:selenium`     |
-| `network.pool_size`    | `WAYFARER_NETWORK_POOL_SIZE`         | How many user agents to spawn.              | 3                                | Integers                            |
+| `network.pool_size`    | `WAYFARER_NETWORK_POOL_SIZE`         | How many user agents to spawn.              | 1                                | Integers                            |
 | `network.pool_timeout` | `WAYFARER_NETWORK_POOL_TIMEOUT`      | How long jobs may use an agent in seconds.  | 10                               | Integers                            |
 | `network.http_headers` | `WAYFARER_NETWORK_HTTP_HEADERS`      | HTTP headers to append to requests.         | `{}`                             | Hashes                              |
 

data/lib/wayfarer/cli/job.rb

@@ -13,12 +13,10 @@ module Wayfarer
 
         url = Addressable::URI.parse(url)
         job = job.classify.constantize.new
-        task = Wayfarer::Task.new(url, "tmp")
+        task = Wayfarer::Task.new(url, options[:batch])
         job.arguments.push(task)
         job.perform(task)
         GC.new(job).run
-
-        free_agent_pool
       end
 
       desc "enqueue JOB URL",

data/lib/wayfarer/cli/route.rb

@@ -11,7 +11,8 @@ module Wayfarer
         load_environment
         url = Addressable::URI.parse(url)
         job = job.classify.constantize
-        puts Wayfarer::Routing::PathFinder.result(job.route, url)
+        job.router.invoke(url, job.new.steer)
+        say Wayfarer::Routing::PathFinder.result(job.router.root, url)
       end
 
       desc "tree JOB URL",
@@ -20,7 +21,8 @@ module Wayfarer
         load_environment
         url = Addressable::URI.parse(url)
         job = job.classify.constantize
-        Wayfarer::CLI::RoutePrinter.print(job.route, url)
+        job.router.invoke(url, job.new.steer)
+        Wayfarer::CLI::RoutePrinter.print(job.router.root, url)
       end
     end
   end

data/lib/wayfarer/cli/templates/job.rb.tt

@@ -1,7 +1,9 @@
 # frozen_string_literal: true
 
 class <%= @name.camelize %> < Wayfarer::Base
-  route.to :index
+  route { to :index }
+
+  retry_on ConnectionPool::TimeoutError, attempts: 3
 
   def index
   end

data/lib/wayfarer/config/networking.rb

@@ -10,7 +10,7 @@ module Wayfarer
                             pool_size: {
                               env_key: "WAYFARER_NETWORK_POOL_SIZE",
                               type: Integer,
-                              default: 3
+                              default: 1
                             },
                             pool_timeout: {
                               env_key: "WAYFARER_NETWORK_POOL_TIMEOUT",

data/lib/wayfarer/config/struct.rb

@@ -39,7 +39,7 @@ module Wayfarer
 
         def define_reader(key, env_key: nil, type: nil, default: nil)
           define_singleton_method(key.to_sym) do
-            get(key) || set(key, get(key) || env_val(env_key, type) || default)
+            get(key) || set(key, env_val(env_key, type) || default)
           end
         end
 

data/lib/wayfarer/middleware/fetch.rb

@@ -3,6 +3,18 @@
 module Wayfarer
   module Middleware
     class Fetch
+      module API
+        def agent
+          task.metadata.agent
+        end
+
+        def page(live: false)
+          return task.metadata.page unless live
+
+          task.metadata.page = agent.live&.page || task.metadata.page
+        end
+      end
+
       include Wayfarer::Middleware::Stage::API
 
       attr_reader :pool
@@ -15,17 +27,16 @@ module Wayfarer
       def call(task)
         self.task = task
 
-        pool.with do |agent|
-          task.metadata.agent = agent
-
+        pool.with do |context|
           result = task.job.run_callbacks :fetch do
-            agent.fetch(task.url)
+            context.fetch(task.url)
           end
 
           case result
           when Networking::Result::Redirect
             stage(result.redirect_url)
           when Networking::Result::Success
+            task.metadata.agent = context.instance
             task.metadata.page = result.page
             yield if block_given?
           end

data/lib/wayfarer/middleware/router.rb

@@ -3,10 +3,42 @@
 module Wayfarer
   module Middleware
     class Router
+      module API
+        def self.included(base)
+          base.include(InstanceMethods)
+          base.extend(ClassMethods)
+        end
+
+        module InstanceMethods
+          def steer
+            []
+          end
+
+          def params
+            task.metadata.params
+          end
+        end
+
+        module ClassMethods
+          def router
+            @router ||= Wayfarer::Routing::Router.new
+          end
+
+          def route(&block)
+            router.draw(&block) if block_given?
+          end
+
+          def steer(&block)
+            define_method(:steer) { block.call(task) }
+          end
+        end
+      end
+
       def call(task)
-        route = task.job.class.route
+        router = task.job.class.router
+        url = Addressable::URI.parse(task.url)
 
-        case result = route.invoke(Addressable::URI.parse(task.url))
+        case result = router.invoke(url, task.job.steer)
         when Routing::Result::Mismatch
           return
         when Routing::Result::Match