wayfarer 0.4.7 → 0.4.8

data/docs/guides/tutorial.md

@@ -1,14 +1,13 @@
 # Tutorial
 
 Wayfarer is a web crawling framework written in Ruby.
-It works with plain HTTP or by automating web browsers and is deployed with
-Redis and a message queue (which can be Redis-based itself).
-In development, it can execute fully in memory, without Redis.
+It works with plain HTTP and by automating web browsers interchangeably
+and is deployed with Redis and a message queue.
+During development it can execute fully in memory, without Redis.
 
-You need a compatible version of Ruby installed.
+## Getting started
 
-To get started, in an empty directory, generate a new `Gemfile` and install
-ActiveJob and Wayfarer:
+In an empty directory, generate a new `Gemfile` and install Wayfarer:
 
 ```sh
 bundle init
@@ -22,14 +21,16 @@ Wayfarer builds on Active Job, the message queue abstraction of Rails.
 You can use Wayfarer without Rails of course, as we do here.
 
 A message queue supports two operations: appending messages to the end and consuming
-messages from the front. In the case of Wayfarer, messages are tasks, a string pair
-consisting of a URL and a batch. When a task is consumed, it is processed by a job,
-a Ruby class.
+messages from the front. This is how Wayfarer processes tasks, a string pair
+of URL and batch. Wayfarer enforces that URLs are not processed more than
+once within their batch (excluding retries).
 
-Let's give ourselves a `dummy_job.rb` that routes arbitrary URLs to its
+When a task is consumed, it is processed by a job, a Ruby class.
+
+Let's give ourselves a `dummy_job.rb` that routes all URLs to its
 `index` instance method, where we print the current `task`:
 
-```ruby
+```ruby title="dummy_job.rb"
 require "activejob"
 require "wayfarer"
 
@@ -44,17 +45,22 @@ class DummyJob < ActiveJob::Base
 end
 ```
 
-We can perform our job from the command line with Wayfarer's CLI and find
-that in between ActiveJob's log output, our task was printed with a generated
-UUID for its batch:
+We can perform our job from the command line with the `wayfarer perform`
+subcommand. In between ActiveJob's log output, we see that Wayfarer
+has generated a UUID for the batch since we did not pass it:
 
-```hl_lines="1 3"
+```sh
 bundle exec wayfarer perform -r dummy_job.rb DummyJob https://example.com
+```
+
+```hl_lines="2"
 [ActiveJob] [DummyJob] [68853491-...] Performing DummyJob (Job ID: 68853491-...) from Async(default) with arguments: #<Wayfarer::Task url="https://example.com", batch="63d14035-...">
 #<Wayfarer::Task url="https://example.com", batch="63d14035-...">
 [ActiveJob] [DummyJob] [68853491-...] Performed DummyJob (Job ID: 68853491-) from Async(default) in 507.65ms
 ```
 
-Many commands accept a `--batch` flag for setting the batch. If you don't
-provide one, a UUID is generated.
+If you don't provide a batch, Wayfarer uses a generated UUID instead.
+We could have also used `DummyJob.crawl
+
+
 

data/docs/guides/user_agents.md

@@ -1,16 +1,18 @@
 # User agents
 
 User agents are used by [jobs](../jobs) to retrieve the contents behind a URL into a
-[page](../pages). They are kept in a connection pool and all user agents in the pool
-share the same type and configuration. You can add custom user agents by implementing
+[page](../pages), for example a remotely controlled Firefox process or a Ruby HTTP client.
+
+User agents are kept in a connection pool and all user agents in the pool
+share the same type and configuration. You can add your own custom user agents by implementing
 the [user agent API](custom_user_agents.md).
 
 Wayfarer comes with the following built-in user agents:
 
-* [`#!ruby :http`](http.md) (default)
-* [`#!ruby :ferrum`](ferrum.md) to automate Google Chrome
-* [`#!ruby :selenium`](selenium.md) to automate a variety of browsers
-* [`#!ruby :capybara`](capybara.md) to use Capybara sessions
+* [`:http`](http.md) (default)
+* [`:ferrum`](ferrum.md) to automate Google Chrome
+* [`:selenium`](selenium.md) to automate a variety of browsers
+* [`:capybara`](capybara.md) to use Capybara sessions
 
 Configure the user agent with the global configuration option:
 
@@ -53,7 +55,7 @@ class DummyJob < ActiveJob::Base
 end
 ```
 
-!!! info "`#fetch` uses the configured `Wayfarer.config.network.http_headers`."
+!!! info "`#fetch` respects `Wayfarer.config.network.http_headers` for all provided user agents."
 
 ## HTTP request headers
 
@@ -78,7 +80,7 @@ underlying message queue operates with. For example, if you use Sidekiq,
 you should set the pool size to the number of Sidekiq threads:
 
 ```ruby
-Wayfarer.config[:network][:pool_size] = Sidekiq.options[:concurrency]
+Wayfarer.config[:network][:pool][:size] = Sidekiq.options[:concurrency]
 ```
 
 !!! attention "The connection pool size is 1 by default"
@@ -109,5 +111,5 @@ You can configure the timeout, although you will likely want to increase the
 pool size instead:
 
 ```ruby
-Wayfarer.config[:network][:pool_timeout] = 10 # seconds
+Wayfarer.config[:network][:pool][:timeout] = 10 # seconds
 ```

data/lib/wayfarer/base.rb

@@ -4,15 +4,16 @@ module Wayfarer
   # @!attribute [r] task
   #   @return [Wayfarer::Task] the current task
   # @!attribute [r] uri
-  #   @return [Addressable::URI] Parsed task URL
+  #   @return [Addressable::URI] parsed task URL
   # @!attribute [r] user_agent
-  #   @return [Object] the user agent that retrieved the page
+  #   @return [Object] the user agent used to retrieve the page
   # @!attribute [r] action
-  #   @return [Symbol, Object] action that the task URL was routed to.
+  #   @return [Symbol, Handler] action that the task URL was routed to
   # @!attribute [r] params
   #   @return [HashWithIndifferentAccess] path parameters collected from routes
   module Base
     extend ActiveSupport::Concern
+
     # @!method stage(urls)
     # Adds URLs to an internal staging set so that they get enqueued
     # eventually, once the job executed successfully.
@@ -29,8 +30,8 @@ module Wayfarer
     # @!method page(live: false)
     # @param url [live] whether to retrieve a new {Page}.
     # @return [Wayfarer::Page]
-    # Returns the most recently retrieved page or a new page
-    # for the current task URL if the `follow` keyword is passed.
+    # The most recently retrieved page or a new page for the current task URL if
+    # the `live` keyword is passed.
 
     # @!scope class
 
@@ -111,15 +112,15 @@ module Wayfarer
     included do
       include Wayfarer::Middleware::Controller
 
-      # Implement ActiveJob's #perform by calling into our own middleware chain
+      # Implement ActiveJob's #perform by calling into our own middleware
+      # chain included from {Controller}
       alias_method :perform, :call
 
-      # Middleware stack
       use Wayfarer::Middleware::Redis
-      use Wayfarer::Middleware::BatchCompletion
       use Wayfarer::Middleware::UriParser
       use Wayfarer::Middleware::Normalize
       use Wayfarer::Middleware::Dedup
+      use Wayfarer::Middleware::BatchCompletion
       use Wayfarer::Middleware::Stage
       use Wayfarer::Middleware::Router
       use Wayfarer::Middleware::UserAgent

data/lib/wayfarer/batch_completion.rb

@@ -14,10 +14,10 @@ module Wayfarer
   module BatchCompletion
   module_function
 
+    EVENTS = %w[enqueue.active_job perform.active_job retry_stopped.active_job].freeze
+
     def subscribe!
-      ActiveSupport::Notifications.subscribe("enqueue.active_job", self)
-      ActiveSupport::Notifications.subscribe("perform.active_job", self)
-      ActiveSupport::Notifications.subscribe("retry_stopped.active_job", self)
+      EVENTS.each { |event| ActiveSupport::Notifications.subscribe(event, self) }
     end
 
     def call(name, _, _, _, data)
@@ -26,25 +26,29 @@ module Wayfarer
       task = job.arguments.first
 
       # In the case of `enqueue.active_job` middleware hasn't executed yet
-      task[:redis_pool] ||= Wayfarer::Redis::Pool.instance # TODO: Test
-
-      counter = Redis::Counter.new(task) do
-        job.run_callbacks(:batch)
-      ensure
-        Wayfarer::GC.run(task)
-      end
+      task[:redis_pool] ||= Wayfarer::Redis::Pool.instance
 
-      handle(name, job, task, counter)
+      handle(name, job, task)
     end
 
-    def handle(name, job, task, counter)
+    def handle(name, job, task)
+      counter = Wayfarer::Redis::Counter.new(task)
+
       case name
       when "enqueue.active_job" then counter.increment unless retry?(job)
-      when "perform.active_job" then counter.decrement if succeeded?(job, task)
-      when "retry_stopped.active_job" then counter.decrement
+      when "perform.active_job" then succeed!(task, counter) if succeeded?(job, task)
+      when "retry_stopped.active_job" then fail!(counter)
       end
     end
 
+    def succeed!(task, counter)
+      Wayfarer::GC.run(task) if counter.decrement == 0
+    end
+
+    def fail!(counter)
+      counter.decrement
+    end
+
     def retry?(job)
       job.executions > 0
     end

data/lib/wayfarer/callbacks.rb

@@ -13,32 +13,32 @@ module Wayfarer
     end
 
     class_methods do
-      def before_fetch(*filters, &block)
-        set_callback(:fetch, :before, *filters, &block)
+      def before_fetch(...)
+        set_callback(:fetch, :before, ...)
       end
 
-      def around_fetch(*filters, &block)
-        set_callback(:fetch, :around, *filters, &block)
+      def around_fetch(...)
+        set_callback(:fetch, :around, ...)
       end
 
-      def after_fetch(*filters, &block)
-        set_callback(:fetch, :after, *filters, &block)
+      def after_fetch(...)
+        set_callback(:fetch, :after, ...)
       end
 
-      def before_action(*filters, &block)
-        set_callback(:action, :before, *filters, &block)
+      def before_action(...)
+        set_callback(:action, :before, ...)
       end
 
-      def around_action(*filters, &block)
-        set_callback(:action, :around, *filters, &block)
+      def around_action(...)
+        set_callback(:action, :around, ...)
       end
 
-      def after_action(*filters, &block)
-        set_callback(:action, :after, *filters, &block)
+      def after_action(...)
+        set_callback(:action, :after, ...)
       end
 
-      def after_batch(*filters, &block)
-        set_callback(:batch, :after, *filters, &block)
+      def after_batch(...)
+        set_callback(:batch, :after, ...)
       end
     end
   end

data/lib/wayfarer/cli/route_printer.rb

@@ -2,128 +2,110 @@
 
 module Wayfarer
   class CLI
-    class RoutePrinter < Thor::Shell::Color
-      attr_reader :url, :path_finder, :output
+    # Turns a routing tree into a Hash and prints it.
+    # Used by the `route` CLI subcommand.
+    #
+    # @api private
+    class RoutePrinter
+      # @return [Hash<Symbol, Proc>]
+      class_attribute :serializers,
+                      default: { yaml: ->(hash) { YAML.dump(hash.deep_stringify_keys) },
+                                 json: ->(hash) { JSON.pretty_generate(hash) },
+                                 ruby: ->(hash) { pp(hash) } },
+                      instance_accessor: false,
+                      instance_predicate: false
 
-      INDENT = "   "
-      REGULAR_SEGMENT = "│  "
-      JUNCTION_SEGMENT = "├──"
-      CORNER_SEGMENT = "└──"
+      BATCH = "tmp"
 
-      def self.print(route, url)
-        route.accept(new(url))
+      # Prints a routing tree.
+      #
+      # @param route [Wayfarer::Routing::Route] route to print
+      # @param url [String] URL to match
+      # @param format [String, Symbol] `:json`, `:yaml` or `:ruby`
+      def self.print(route, url, format:)
+        new(route, url, serializers.fetch(format.to_sym)).print
       end
 
-      def initialize(url)
-        @url = url
-        @path_finder = Wayfarer::Routing::PathFinder.new(url)
-        super()
-      end
-
-      def visit(route)
-        route.accept(path_finder) unless route.parent
-        puts format_route_output(route)
-        true
-      end
-
-    private
+      # @param route [Wayfarer::Routing::Route] route to print
+      # @param url [String] URL to match
+      # @param serializer [Proc<Hash=>String>] output serializer
+      def initialize(route, url, serializer)
+        @route = route
+        @serializer = serializer
 
-      def format_route_output(route)
-        [segments(route), route_description(route)].join[3..]
-      end
+        @nodes = {}
+        @root_hash = nil
 
-      def segments(route)
-        [parents(route).map { |parent| parent_segment(parent) }, segment(route)].join
+        task = Wayfarer::Task.new(url, BATCH)
+        task[:uri] = Addressable::URI.parse(url)
+        @path_finder = Wayfarer::Routing::PathFinder.new(
+          task,
+          stop_when_found: false,
+          &method(:call)
+        )
       end
 
-      def parent_segment(parent)
-        trailer?(parent) ? INDENT : REGULAR_SEGMENT
-      end
+      # Processes the routing trees and prints the serialized output.
+      def print
+        route.accept(path_finder)
 
-      def segment(route)
-        trailer?(route) ? CORNER_SEGMENT : JUNCTION_SEGMENT
-      end
+        hash = routing_result(path_finder).merge(root_hash)
 
-      def route_description(route)
-        attrs = [route_arg(route), routing_result(route), route_action(route), route_params(route)].compact
-        text = attrs.any? ? "#{matcher_name(route)}(#{attrs.join(', ')})" : matcher_name(route)
-        set_color(text, *route_colors(route))
+        puts serializer.call(hash)
       end
 
-      def matcher_name(route)
-        case route
-        when Wayfarer::Routing::TargetRoute
-          "Target"
-        when Wayfarer::Routing::RootRoute
-          Wayfarer::Routing::PathFinder.result(route, url).class.name.demodulize
-        else
-          route.matcher.class.name.demodulize
-        end
-      end
+      # Callback method called by `path_finder` with the result of matching
+      # the route.
+      #
+      # @param route [Wayfarer::Routing::Route] the current route
+      # @param result [true, false] routing result
+      # @param path_finder [Wayfarer::Routing::PathFinder] the path finder
+      def call(route, result, path_finder)
+        node = (nodes[route] ||= attributes(route, result, path_finder))
+        parent = route.parent
 
-      def routing_result(route)
-        return if route.is_a?(Wayfarer::Routing::RootRoute)
+        return @root_hash ||= node unless parent
 
-        "match: #{route.matcher.match(url)}"
+        nodes.dig(parent, route_type(parent), :children).append(node)
       end
 
-      def route_action(route)
-        return unless route.is_a?(Wayfarer::Routing::RootRoute)
-
-        result = Wayfarer::Routing::PathFinder.result(route, url)
-        result.action.inspect if result.is_a?(Wayfarer::Routing::Result::Match)
-      end
+    private
 
-      def route_arg(route)
-        return if route.is_a?(Wayfarer::Routing::RootRoute) || route.is_a?(Wayfarer::Routing::TargetRoute)
-
-        matcher = route.matcher
-        matcher_opts = case matcher
-                       when Wayfarer::Routing::Matchers::Host then matcher.host
-                       when Wayfarer::Routing::Matchers::Path then matcher.path
-                       when Wayfarer::Routing::Matchers::Query then matcher.fields
-                       when Wayfarer::Routing::Matchers::Custom then route.action.to_s
-                       when Wayfarer::Routing::Matchers::Scheme then matcher.scheme
-                       when Wayfarer::Routing::Matchers::Suffix then matcher.suffix
-                       end
-        matcher_opts.inspect
-      end
+      attr_reader :route,
+                  :path_finder,
+                  :serializer,
+                  :nodes,
+                  :root_hash
 
-      def route_params(route)
-        params = if route.is_a?(Wayfarer::Routing::RootRoute)
-                   result = Wayfarer::Routing::PathFinder.result(route, url)
-                   result.params if result.is_a?(Wayfarer::Routing::Result::Match)
-                 else
-                   route.matcher.params(url)
-                 end
+      def routing_result(path_finder)
+        return { routed: false } unless path_finder.found?
 
-        "params: #{params.symbolize_keys}" if params&.any?
+        action = path_finder.action
+        { routed: true,
+          params: path_finder.params,
+          action: case action
+                  when Array
+                    { handler: action.first.class.name, action: action.second }
+                  else action
+                  end }
       end
 
-      def parents(route, current = [])
-        return current unless route.parent
-
-        parents(route.parent, [route.parent, *current])
+      def attributes(route, result, path_finder)
+        { route_type(route) => route.to_h.merge!(
+          route_result(route, result, path_finder),
+          children: []
+        ) }
       end
 
-      def trailer?(route)
-        !route.parent || route.parent.children.last == route
-      end
+      def route_result(route, result, path_finder)
+        return {} if route.target?
 
-      def route_colors(route)
-        if path_finder.path.include?(route)
-          %i[green bold]
-        elsif route.matcher.match(url)
-          %i[green]
-        else
-          %i[red]
-        end
+        { match: result,
+          params: path_finder.params_stack.to_h }
       end
 
-      def set_color(string, *colors)
-        return string if ENV.key?("NO_COLOR")
-
-        super(string, *colors)
+      def route_type(route)
+        route.class.name.demodulize.underscore
       end
     end
   end

data/lib/wayfarer/cli.rb

@@ -13,36 +13,16 @@ module Wayfarer
 
     class_option :require, aliases: :r, type: :string, default: nil
 
-    desc "route JOB URL", "Routing result of URL for JOB"
+    desc "route JOB URL", "Routing tree for URL for JOB"
+    option :format, type: :string, enum: %w[yaml json ruby], default: "yaml"
     def route(job, url)
       load_environment
 
       url = parsed_url(url)
-      job = job.classify.constantize
+      job = job.camelize.constantize
       route = job.route
-      route.invoke(url)
 
-      result = Wayfarer::Routing::PathFinder.result(route, url)
-      result_type = result.class.name.demodulize
-
-      say case result
-          when Wayfarer::Routing::Result::Match
-            "#{result_type} => #{result.action.inspect}"
-          else
-            result_type
-          end
-    end
-
-    desc "tree JOB URL", "Visualize JOB's routing tree for URL"
-    def tree(job, url)
-      load_environment
-
-      url = parsed_url(url)
-      job = job.classify.constantize
-      route = job.route
-      route.invoke(url)
-
-      Wayfarer::CLI::RoutePrinter.print(route, url)
+      Wayfarer::CLI::RoutePrinter.print(route, url, format: options.fetch("format"))
     end
 
     desc "perform JOB URL", "Perform JOB with URL"
@@ -52,8 +32,8 @@ module Wayfarer
       load_environment
       mock_redis
 
-      job = job.classify.constantize
-      task = Wayfarer::Task.new(url, options[:batch])
+      job = job.camelize.constantize
+      task = Wayfarer::Task.new(url, options.fetch(:batch))
       job.new(task).perform_now
     end
 
@@ -62,7 +42,7 @@ module Wayfarer
     def enqueue(job, url)
       load_environment
 
-      job = job.classify.constantize
+      job = job.camelize.constantize
       job.crawl(url, batch: options[:batch])
     end
 
@@ -71,21 +51,23 @@ module Wayfarer
     option :batch, type: :string, default: SecureRandom.uuid
     option :min_threads, type: :numeric, default: 1
     option :max_threads, type: :numeric, default: 1
+    option :retain_pool, type: :boolean, default: false
     def execute(job, url)
       load_environment
       mock_redis
 
-      job = job.classify.constantize
+      job = job.camelize.constantize
       job.queue_adapter = ActiveJob::QueueAdapters::AsyncAdapter.new(min_threads: options[:min_threads],
                                                                      max_threads: options[:max_threads])
       scheduler = job.queue_adapter.instance_variable_get(:@scheduler)
       executor = scheduler.instance_variable_get(:@async_executor)
 
-      job.crawl(url, batch: options[:batch])
+      job.crawl(url, batch: options.fetch(:batch))
 
       sleep(0.1) while executor.scheduled_task_count > executor.completed_task_count
 
-      Wayfarer::Networking::Pool.instance.free
+      # Used in test suite to avoid pool recreation
+      Wayfarer::Networking::Pool.instance.free unless options.fetch(:retain_pool)
     end
 
   private

data/lib/wayfarer/gc.rb

@@ -6,12 +6,17 @@ module Wayfarer
     RESETTABLES = [Wayfarer::Redis::Barrier, Wayfarer::Redis::Counter].freeze
 
     class << self
-      include Wayfarer::Logging.emit(gc: [:info, "Garbage collecting %<resettable>s"])
+      include Wayfarer::Logging.emit(
+        after_batch: [:debug, "Running `after_batch` callback"],
+        gc: [:debug, "Garbage collecting %<resettable>s"]
+      )
     end
 
   module_function
 
     def run(task)
+      task[:job].run_callbacks(:batch)
+
       RESETTABLES.each do |resettable|
         log(:gc, task, resettable: resettable)
         resettable.new(task).reset!

data/lib/wayfarer/kv.rb

@@ -0,0 +1,28 @@
+# frozen_string_literal: true
+
+module Wayfarer
+  # Provides a key-value store via `[]` and `[]=`.
+  #
+  # @api private
+  module KV
+    # @param key [Object] key to fetch
+    # @return [Object, nil] value associated with the key or `nil`
+    def [](key)
+      kv[key]
+    end
+
+    # @param key [Object] key to set
+    # @param value [Object] value to set
+    # @return [Object] value that was set
+    def []=(key, value)
+      kv[key] = value
+    end
+
+  private
+
+    # @return [Hash<Object, Object>]
+    def kv
+      @kv ||= {}
+    end
+  end
+end

data/lib/wayfarer/middleware/chain.rb

@@ -2,9 +2,15 @@
 
 module Wayfarer
   module Middleware
-    Chain = Struct.new(:middlewares) do
+    class Chain
       extend Forwardable
 
+      attr_reader :middlewares
+
+      def initialize(middlewares)
+        @middlewares = middlewares
+      end
+
       def self.empty
         new([])
       end