wayfarer 0.4.7 → 0.4.9

data/docs/guides/routing.md

@@ -1,18 +1,32 @@
 # Routing
 
-Wayfarer equips jobs with a routing DSL that routes URLs to actions. Actions are
-either instance methods denoted by symbols, or [handlers](/guides/handlers).
+Wayfarer equips jobs with a declarative routing DSL that maps URLs to actions.
+Actions are instance methods denoted by symbols, or [handlers](/guides/handlers).
+[Pages](/guides/pages) are only retrieved from URLs which map to an action.
+
+!!! info "Routed URLs are normalized"
+
+    By default, Wayfarer [applies some transformations to each URL](../tasks/#url-normalization) to bring it
+    into a canonical form. Routing happens based on this canonical form.
+
+    You can always access a task's raw string as it was enqueued with `task.url`.
+
 A job's route declarations equate to a predicate tree.
 When a URL is routed, the predicate tree is searched depth-first. If a
-matching leaf predicate is found, the found path's action is dispatched,
-along with `params` collected from path parameters.
+matching leaf predicate is found, the found path's action is dispatched.
+You can extract data from URL path segments and query parameters and
+access it through `params` in jobs or handlers.
 
 The following routes:
 
 ```ruby
 route.host "example.com", scheme: :https do
-  path "/contact", to: :contact
-  path "/users/:id", to: [UserHandler, :show]
+  path "contact", to: :contact
+  path "users/:id" do
+    to [UserHandler, :show]
+
+    path "gallery", to: [UserHandler, :photos]
+  end
 end
 ```
 
@@ -20,43 +34,111 @@ Equate to the following predicate tree:
 
 ```mermaid
 flowchart LR
-  RootRoute-->Host["Host <code>example.com</code>"]
+  Root-->Host["Host <code>example.com</code>"]
   Host-->Scheme["Scheme <code>:https</code>"]
-  Scheme-->Path1["Path <code>/contact</code>"]
-  Scheme-->Path2["Path <code>/users/:id<code>"]
-  Path1-->TargetRoute1["Target <code>:contact</code>"]
-  Path2-->TargetRoute2["Target <code>[UserHandler, :show]</code>"]
+
+  %% first-level paths
+  Scheme-->PathContact["Path <code>contact</code>"]
+  Scheme-->PathUsersId["Path <code>users/:id</code>"]
+
+  %% their targets
+  PathContact-->TargetRouteContact["Target <code>:contact</code>"]
+  PathUsersId-->TargetRouteUserHandler["Target <code>[UserHandler, :show]</code>"]
+
+  %% nested path under /users/:id
+  PathUsersId-->PathGallery["Path <code>'gallery'</code>"]
+  PathGallery-->TargetRouteUserHandlerPhotos["Target <code>[UserHandler, :photos]</code>"]
 ```
 
-An invocation for the URL `https://example.com/users/42` leads to `[UserHandler, :show]`:
+Traversing the tree depth-first for `https://example.com/users/42` stops at the
+route with the action `[UserHandler, :show]`:
 
 ```mermaid
 flowchart LR
-  RootRoute:::active-->Host["Host <code>example.com</code>"]:::active
-  Host:::active-->Scheme["Scheme <code>:https</code>"]:::active
-  Scheme:::active-->Path1["Path <code>/contact</code>"]:::inactive
-  Scheme:::active-->Path2["Path <code>/users/:id<code>"]:::active
-  Path1:::inactive-->TargetRoute1["Target <code>:contact</code>"]:::active
-  Path2:::active-->TargetRoute2["Target <code>[UserHandler, :show]</code>"]:::activePlus
-  classDef active fill:#7CB342,stroke:#7CB342,color:#fff
-  classDef activePlus fill:#F1F8E9,stroke:#8BC34A,color:#33691E,stroke-width:4px
-  classDef inactive fill:#FFCDD2,stroke:#F44336,color:#B71C1C
-```
+  Root:::matching-->Host["Host <code>example.com</code>"]:::matching
+  Host:::matching-->Scheme["Scheme <code>:https</code>"]:::matching
 
-You can also visualise an invocation of the predicate tree on the command line
-with `wayfarer tree`
+  %% sibling paths from the scheme node
+  Scheme:::matching-->PathContact["Path <code>/contact</code>"]:::mismatching
+  Scheme:::matching-->PathUsersId["Path <code>/users/:id</code>"]:::matching
 
+  %% successful match for /users/:id
+  PathUsersId:::matching-->TargetRouteUserHandler["Target <code>[UserHandler, :show]</code>"]:::matching
+
+  %% gallery branch is never visited for /users/42
+  PathContact-->TargetRouteContact["Target <code>:contact</code>"]:::unvisited
+  PathUsersId:::matching-->PathGallery["Path <code>/gallery</code>"]:::unvisited
+  PathGallery:::unvisited-->TargetRouteUserHandlerPhotos["Target <code>[UserHandler, :photos]</code>"]:::unvisited
+
+  classDef matching     fill:#7CB342,stroke:#7CB342,color:#fff
+  classDef mismatching  fill:#FFCDD2,stroke:#F44336,color:#B71C1C
+  classDef unvisited    fill:#BDBDBD,stroke:#BDBDBD,color:#616161
 ```
-wayfarer tree -r dummy_job.rb DummyJob https://example.com/users/42/foobar
-Match([UserHandler, :show], params: {:id=>"42", :foo=>"foobar"})
-└──Host("example.com", match: true)
-   └──Scheme(:https, match: true)
-      ├──Path("/contact", match: false)
-      │  └──Target(match: true)
-      └──Path("/users/:id", match: true)
-         └──Target(match: true)
-            └──Path("/users/:id/:foo", match: true, params: {:id=>"42", :foo=>"foobar"})
-```
+
+??? note "You can also visualise a job's routing tree with with the [`route` CLI subcommand](/guides/cli)"
+
+    ```sh
+    wayfarer route DummyJob -r dummy_job.rb http://localhost:9000/users/42/gallery
+    ```
+
+    ```yaml
+    ---
+    routed: true
+    params:
+      id: '42'
+    action:
+      handler: Class
+      action: :photos
+    root_route:
+      match: true
+      params: {}
+      children:
+      - route:
+          host:
+            name: example.com
+          match: true
+          params: {}
+          children:
+          - route:
+              scheme:
+                scheme: :https
+              match: true
+              params: {}
+              children:
+              - route:
+                  path:
+                    pattern: "/contact"
+                  match: false
+                  params: {}
+                  children:
+                  - target_route:
+                      action:
+                      children: []
+              - route:
+                  path:
+                    pattern: "/users/:id"
+                  match: true
+                  params:
+                    id: '42'
+                  children:
+                  - target_route:
+                      action:
+                        handler: Class
+                        action: :show
+                      children: []
+                  - route:
+                      path:
+                        pattern: "/gallery"
+                      match: true
+                      params:
+                        id: '42'
+                      children:
+                      - target_route:
+                          action:
+                            handler: Class
+                            action: :photos
+                          children: []
+    ```
 
 As you can see, `Target` nodes always match. This means that we could have also defined
 our routes as:

data/docs/guides/tasks.md

@@ -19,20 +19,40 @@ Wayfarer ensures that no URL gets processed twice within a batch. It achieves
 this by maintaining a [Redis hash](https://redis.io/docs/data-types/hashes)
 keyed by normalized URLs.
 
+Wayfarer computes a canonical URL representation that it uses for cache lookups.
+
 ### URL normalization
 
 Wayfarer parses URLs with [Addressable](https://github.com/sporkmonger/addressable)
-and normalizes HTTP(S) URLs with [`normalize_url`](https://github.com/rwz/normalize_url/).
+and applies further normalizations. By default, all normalizations are applied
+and can be individually disabled.
+
+URL normalization is used only for deduplication, and does not affect the immutable
+`task.url`, which always returns the verbatim URL as enqueued.
+This allows you to follow the URLs exactly as parsed from response bodies.
+
+You can configure the global normalization behaviour by setting the following
+values on `Wayfarer.config.normalization` do which all default to `true`:
+
+ * `remove_www`: Remove `www.` prefix from hostnames?
+ * `remove_trailing_slash`: Remove a trailing path slash?
+ * `remove_fragment`: Remove the URL fragment?
+ * `order_query_parameters:` Order query parameters alphabetically?
+ * `remove_tracking_parameters`: Remove tracking parameters from the URL?
+
+When a job gets deduplicated, it succeeds and causes no retries.
+
+### Setting a custom key function
+
+You can customize how deduplication keys are computed. As a derived example,
+to process only one job per hostname:
 
-URL normalization is used only for deduplication, and does not affect the URL
-returned by `task.url`. Instead, `task.url` returns the verbatim URL as it was
-enqueud. This allows you to follow the exact URLs you may have parsed from a
-response body.
+```ruby
+Wayfarer.config[:deduplication][:key] = ->(task) { task[:uri].hostname }
+```
 
 ## Invalid URLs
 
-Tasks with invalid URLs (for example`ht%0atp://localhost/`, a newline in the
-protocol) are discarded, since they can't get retrieved. No exception is raised,
-and the job is considered successfully processed, since there are no corrective
-actions an error handler could take as tasks are immutable, and retries would
-not change the outcome.
+Tasks with invalid URLs are discarded (for example`ht%0atp://localhost/` which has a
+newline in its protocol), since there is no corrective action possible.
+No exception is raised, and the job is considered successfully processed without retries.

data/docs/guides/tutorial.md

@@ -1,14 +1,13 @@
 # Tutorial
 
 Wayfarer is a web crawling framework written in Ruby.
-It works with plain HTTP or by automating web browsers and is deployed with
-Redis and a message queue (which can be Redis-based itself).
-In development, it can execute fully in memory, without Redis.
+It works with plain HTTP and by automating web browsers interchangeably
+and is deployed with Redis and a message queue.
+During development it can execute fully in memory, without Redis.
 
-You need a compatible version of Ruby installed.
+## Getting started
 
-To get started, in an empty directory, generate a new `Gemfile` and install
-ActiveJob and Wayfarer:
+In an empty directory, generate a new `Gemfile` and install Wayfarer:
 
 ```sh
 bundle init
@@ -22,14 +21,16 @@ Wayfarer builds on Active Job, the message queue abstraction of Rails.
 You can use Wayfarer without Rails of course, as we do here.
 
 A message queue supports two operations: appending messages to the end and consuming
-messages from the front. In the case of Wayfarer, messages are tasks, a string pair
-consisting of a URL and a batch. When a task is consumed, it is processed by a job,
-a Ruby class.
+messages from the front. This is how Wayfarer processes tasks, a string pair
+of URL and batch. Wayfarer enforces that URLs are not processed more than
+once within their batch (excluding retries).
 
-Let's give ourselves a `dummy_job.rb` that routes arbitrary URLs to its
+When a task is consumed, it is processed by a job, a Ruby class.
+
+Let's give ourselves a `dummy_job.rb` that routes all URLs to its
 `index` instance method, where we print the current `task`:
 
-```ruby
+```ruby title="dummy_job.rb"
 require "activejob"
 require "wayfarer"
 
@@ -44,17 +45,22 @@ class DummyJob < ActiveJob::Base
 end
 ```
 
-We can perform our job from the command line with Wayfarer's CLI and find
-that in between ActiveJob's log output, our task was printed with a generated
-UUID for its batch:
+We can perform our job from the command line with the `wayfarer perform`
+subcommand. In between ActiveJob's log output, we see that Wayfarer
+has generated a UUID for the batch since we did not pass it:
 
-```hl_lines="1 3"
+```sh
 bundle exec wayfarer perform -r dummy_job.rb DummyJob https://example.com
+```
+
+```hl_lines="2"
 [ActiveJob] [DummyJob] [68853491-...] Performing DummyJob (Job ID: 68853491-...) from Async(default) with arguments: #<Wayfarer::Task url="https://example.com", batch="63d14035-...">
 #<Wayfarer::Task url="https://example.com", batch="63d14035-...">
 [ActiveJob] [DummyJob] [68853491-...] Performed DummyJob (Job ID: 68853491-) from Async(default) in 507.65ms
 ```
 
-Many commands accept a `--batch` flag for setting the batch. If you don't
-provide one, a UUID is generated.
+If you don't provide a batch, Wayfarer uses a generated UUID instead.
+We could have also used `DummyJob.crawl
+
+
 

data/docs/guides/user_agents.md

@@ -1,16 +1,18 @@
 # User agents
 
 User agents are used by [jobs](../jobs) to retrieve the contents behind a URL into a
-[page](../pages). They are kept in a connection pool and all user agents in the pool
-share the same type and configuration. You can add custom user agents by implementing
+[page](../pages), for example a remotely controlled Firefox process or a Ruby HTTP client.
+
+User agents are kept in a connection pool and all user agents in the pool
+share the same type and configuration. You can add your own custom user agents by implementing
 the [user agent API](custom_user_agents.md).
 
 Wayfarer comes with the following built-in user agents:
 
-* [`#!ruby :http`](http.md) (default)
-* [`#!ruby :ferrum`](ferrum.md) to automate Google Chrome
-* [`#!ruby :selenium`](selenium.md) to automate a variety of browsers
-* [`#!ruby :capybara`](capybara.md) to use Capybara sessions
+* [`:http`](http.md) (default)
+* [`:ferrum`](ferrum.md) to automate Google Chrome
+* [`:selenium`](selenium.md) to automate a variety of browsers
+* [`:capybara`](capybara.md) to use Capybara sessions
 
 Configure the user agent with the global configuration option:
 
@@ -53,7 +55,7 @@ class DummyJob < ActiveJob::Base
 end
 ```
 
-!!! info "`#fetch` uses the configured `Wayfarer.config.network.http_headers`."
+!!! info "`#fetch` respects `Wayfarer.config.network.http_headers` for all provided user agents."
 
 ## HTTP request headers
 
@@ -78,7 +80,7 @@ underlying message queue operates with. For example, if you use Sidekiq,
 you should set the pool size to the number of Sidekiq threads:
 
 ```ruby
-Wayfarer.config[:network][:pool_size] = Sidekiq.options[:concurrency]
+Wayfarer.config[:network][:pool][:size] = Sidekiq.options[:concurrency]
 ```
 
 !!! attention "The connection pool size is 1 by default"
@@ -109,5 +111,5 @@ You can configure the timeout, although you will likely want to increase the
 pool size instead:
 
 ```ruby
-Wayfarer.config[:network][:pool_timeout] = 10 # seconds
+Wayfarer.config[:network][:pool][:timeout] = 10 # seconds
 ```

data/lib/wayfarer/base.rb

@@ -4,15 +4,16 @@ module Wayfarer
   # @!attribute [r] task
   #   @return [Wayfarer::Task] the current task
   # @!attribute [r] uri
-  #   @return [Addressable::URI] Parsed task URL
+  #   @return [Addressable::URI] parsed task URL
   # @!attribute [r] user_agent
-  #   @return [Object] the user agent that retrieved the page
+  #   @return [Object] the user agent used to retrieve the page
   # @!attribute [r] action
-  #   @return [Symbol, Object] action that the task URL was routed to.
+  #   @return [Symbol, Handler] action that the task URL was routed to
   # @!attribute [r] params
   #   @return [HashWithIndifferentAccess] path parameters collected from routes
   module Base
     extend ActiveSupport::Concern
+
     # @!method stage(urls)
     # Adds URLs to an internal staging set so that they get enqueued
     # eventually, once the job executed successfully.
@@ -29,8 +30,8 @@ module Wayfarer
     # @!method page(live: false)
     # @param url [live] whether to retrieve a new {Page}.
     # @return [Wayfarer::Page]
-    # Returns the most recently retrieved page or a new page
-    # for the current task URL if the `follow` keyword is passed.
+    # The most recently retrieved page or a new page for the current task URL if
+    # the `live` keyword is passed.
 
     # @!scope class
 
@@ -111,15 +112,15 @@ module Wayfarer
     included do
       include Wayfarer::Middleware::Controller
 
-      # Implement ActiveJob's #perform by calling into our own middleware chain
+      # Implement ActiveJob's #perform by calling into our own middleware
+      # chain included from {Controller}
       alias_method :perform, :call
 
-      # Middleware stack
       use Wayfarer::Middleware::Redis
-      use Wayfarer::Middleware::BatchCompletion
       use Wayfarer::Middleware::UriParser
       use Wayfarer::Middleware::Normalize
       use Wayfarer::Middleware::Dedup
+      use Wayfarer::Middleware::BatchCompletion
       use Wayfarer::Middleware::Stage
       use Wayfarer::Middleware::Router
       use Wayfarer::Middleware::UserAgent

data/lib/wayfarer/batch_completion.rb

@@ -14,10 +14,10 @@ module Wayfarer
   module BatchCompletion
   module_function
 
+    EVENTS = %w[enqueue.active_job perform.active_job retry_stopped.active_job].freeze
+
     def subscribe!
-      ActiveSupport::Notifications.subscribe("enqueue.active_job", self)
-      ActiveSupport::Notifications.subscribe("perform.active_job", self)
-      ActiveSupport::Notifications.subscribe("retry_stopped.active_job", self)
+      EVENTS.each { |event| ActiveSupport::Notifications.subscribe(event, self) }
     end
 
     def call(name, _, _, _, data)
@@ -26,25 +26,29 @@ module Wayfarer
       task = job.arguments.first
 
       # In the case of `enqueue.active_job` middleware hasn't executed yet
-      task[:redis_pool] ||= Wayfarer::Redis::Pool.instance # TODO: Test
-
-      counter = Redis::Counter.new(task) do
-        job.run_callbacks(:batch)
-      ensure
-        Wayfarer::GC.run(task)
-      end
+      task[:redis_pool] ||= Wayfarer::Redis::Pool.instance
 
-      handle(name, job, task, counter)
+      handle(name, job, task)
     end
 
-    def handle(name, job, task, counter)
+    def handle(name, job, task)
+      counter = Wayfarer::Redis::Counter.new(task)
+
       case name
       when "enqueue.active_job" then counter.increment unless retry?(job)
-      when "perform.active_job" then counter.decrement if succeeded?(job, task)
-      when "retry_stopped.active_job" then counter.decrement
+      when "perform.active_job" then succeed!(task, counter) if succeeded?(job, task)
+      when "retry_stopped.active_job" then fail!(counter)
       end
     end
 
+    def succeed!(task, counter)
+      Wayfarer::GC.run(task) if counter.decrement == 0
+    end
+
+    def fail!(counter)
+      counter.decrement
+    end
+
     def retry?(job)
       job.executions > 0
     end

data/lib/wayfarer/callbacks.rb

@@ -13,32 +13,32 @@ module Wayfarer
     end
 
     class_methods do
-      def before_fetch(*filters, &block)
-        set_callback(:fetch, :before, *filters, &block)
+      def before_fetch(...)
+        set_callback(:fetch, :before, ...)
       end
 
-      def around_fetch(*filters, &block)
-        set_callback(:fetch, :around, *filters, &block)
+      def around_fetch(...)
+        set_callback(:fetch, :around, ...)
       end
 
-      def after_fetch(*filters, &block)
-        set_callback(:fetch, :after, *filters, &block)
+      def after_fetch(...)
+        set_callback(:fetch, :after, ...)
       end
 
-      def before_action(*filters, &block)
-        set_callback(:action, :before, *filters, &block)
+      def before_action(...)
+        set_callback(:action, :before, ...)
       end
 
-      def around_action(*filters, &block)
-        set_callback(:action, :around, *filters, &block)
+      def around_action(...)
+        set_callback(:action, :around, ...)
       end
 
-      def after_action(*filters, &block)
-        set_callback(:action, :after, *filters, &block)
+      def after_action(...)
+        set_callback(:action, :after, ...)
       end
 
-      def after_batch(*filters, &block)
-        set_callback(:batch, :after, *filters, &block)
+      def after_batch(...)
+        set_callback(:batch, :after, ...)
       end
     end
   end