wayfarer 0.4.5 → 0.4.7

data/docs/reference/configuration_keys.md

@@ -1,42 +0,0 @@
----
-hide:
-  - toc
----
-
-# Configuration Keys
-
-## `Wayfarer.config.network`
-
-| Runtime config key     | Environment variable                 | Description                                 | Default                          | Supported values                    |
-| ---------------------- | ------------------------------------ | ------------------------------------------- | -------------------------------- | ----------------------------------- |
-| `network.agent`        | `WAYFARER_NETWORK_AGENT`             | The user agent to use.                      | `:http`                          | `:http`, `:ferrum`, `:selenium`     |
-| `network.pool_size`    | `WAYFARER_NETWORK_POOL_SIZE`         | How many user agents to spawn.              | 1                                | Integers                            |
-| `network.pool_timeout` | `WAYFARER_NETWORK_POOL_TIMEOUT`      | How long jobs may use an agent in seconds.  | 10                               | Integers                            |
-| `network.http_headers` | `WAYFARER_NETWORK_HTTP_HEADERS`      | HTTP headers to append to requests.         | `{}`                             | Hashes                              |
-
-## `Wayfarer.config.ferrum`
-
-| Runtime config key     | Environment variable                 | Description                                 | Default                          | Supported values                    |
-| ---------------------- | ------------------------------------ | ------------------------------------------- | -------------------------------- | ----------------------------------- |
-| `ferrum.options`       | `WAYFARER_FERRUM_OPTIONS`            | Ferrum options.                             | `{}`                             | Hashes                              |
-
-## `Wayfarer.config.selenium`
-
-| Runtime config key        | Environment variable                 | Description                                 | Default                          | Supported values                    |
-| ----------------------    | ------------------------------------ | ------------------------------------------- | -------------------------------- | ----------------------------------- |
-| `selenium.driver`         | `WAYFARER_SELENIUM_DRIVER`           | Selenium driver to use.                     | `:chrome`                        | Symbols                             |
-| `selenium.options`        | `WAYFARER_SELENIUM_OPTIONS`          | Selenium options.                           | `{}`                             | Hashes                              |
-| `selenium.client_timeout` | `WAYFARER_SELENIUM_CLIENT_TIMEOUT`   | Selenium client timeout in seconds.         | 60                               | Integers                            |
-
-## `Wayfarer.config.redis`
-
-| Runtime config key     | Environment variable                 | Description                                 | Default                                    | Supported values                    |
-| ---------------------- | ------------------------------------ | ------------------------------------------- | ------------------------------------------ | ----------------------------------- |
-| `redis.url`            | `WAYFARER_REDIS_URL`                 | Redis URL to connect to.                    | http://localhost:6379                      | Strings                             |
-| `redis.factory`        | n/a                                  | Redis factory lambda.                       | ` ->(redis) { ::Redis.new(url: redis.url)` | Lambdas                             |
-
-## `Wayfarer.config.capybara`
-
-| Runtime config key     | Environment variable                 | Description                                 | Default                          | Supported values                    |
-| ---------------------- | ------------------------------------ | ------------------------------------------- | -------------------------------- | ----------------------------------- |
-| `capybara.driver`      | `WAYFARER_CAPYBARA_DRIVER`           | The Capybara driver to use.                 | n/a                              | Symbols                             |

data/docs/reference/environment_variables.md

@@ -1,83 +0,0 @@
-# Environment Variables
-
-## String formats
-
-Environment variable values can be parsed to Hash or Array at runtime
-with the following syntaxes:
-
-* Hash: Variable string `a:1,b:2,c:3` parses to `{a:1, b:2, c:3}` at runtime
-* Array: Variable string `a,b,c` parses to `[:a, :b, :c]` at runtime
-
-## Variables
-
-### `WAYFARER_AGENT`
-:   Either `ferrum`, `selenium` or `http`.
-
-    * Type: String
-    * Key: `config.agent`
-    * Default value: `:http`
-
-### `WAYFARER_POOL_SIZE`
-:   Number of user agents to maintain.
-
-    * Type: Integer
-    * Key: `config.pool_size`
-    * Default value: `1`
-
-### `WAYFARER_POOL_TIMEOUT`
-:   How long a user agent may remain checked out until the owning job
-    fails.
-
-    * Type: Integer
-    * Key: `config.agent_pool_timeout`
-    * Default value: `1`
-
----
-
-### `WAYFARER_FERRUM_OPTIONS`
-:   Key/value options passed to `Ferrum::Browser.new`.
-
-    * Type: Hash
-    * Key: `config.ferrum_options`
-    * Default value: `{}`
-
----
-
-### `WAYFARER_SELENIUM_DRIVER`
-:   Driver passed to `Selenium::WebDriver.for`.
-
-    * Type: Symbol
-    * Key: `config.selenium_driver`
-    * Default value: `:chrome`
-
----
-
-### `WAYFARER_SELENIUM_OPTIONS`
-:   Options passed to `Selenium::WebDriver.for`.
-
-    * Type: Hash
-    * Key: `config.selenium_options`
-    * Default value: `{}`
-    
----
-
-### `WAYFARER_SELENIUM_CLIENT_TIMEOUT`
-:   Selenium HTTP client timeout (seconds).
-
-    * Type: Integer
-    * Key: `config.selenium_client_timeout`
-    * Default value: `60`
-
----
-
-### `WAYFARER_HTTP_HEADERS`
-:   HTTP request headers used when retrieving pages.
-
-    * Type: Hash
-    * Key: `config.http_headers`
-    * Default value: `{}`
-    
-    !!! attention "Partial support"
-
-        Selenium does not support configuring HTTP request headers.
-

data/lib/wayfarer/cli/base.rb

@@ -1,45 +0,0 @@
-# frozen_string_literal: true
-
-module Wayfarer
-  module CLI
-    class Base < Thor
-      include Thor::Actions
-
-      def self.source_root
-        File.expand_path("templates", __dir__)
-      end
-
-    private
-
-      def mock_redis
-        Wayfarer.config.redis.factory = ->(_) { MockRedis.new }
-      end
-
-      def load_environment
-        load_rails || load_plain
-      end
-
-      def free_agent_pool
-        Wayfarer::Networking::Pool.instance.free
-      end
-
-      def load_rails
-        begin
-          require "rails/app_loader"
-        rescue LoadError
-          return
-        end
-
-        return unless Rails::AppLoader.find_executable
-
-        require File.expand_path("config/application", Dir.pwd)
-        require File.expand_path("config/boot", Dir.pwd)
-        require File.expand_path("config/environment", Dir.pwd)
-      end
-
-      def load_plain
-        Wayfarer::CLI::Runner.loader.setup
-      end
-    end
-  end
-end

data/lib/wayfarer/cli/generate.rb

@@ -1,17 +0,0 @@
-# frozen_string_literal: true
-
-module Wayfarer
-  module CLI
-    class Generate < Base
-      desc "project NAME",
-           "Generate new project directory NAME"
-      def project(name)
-        @name = name
-
-        empty_directory name
-        template "Gemfile.tt", "#{name}/Gemfile"
-        template "job.rb.tt", "#{name}/app/jobs/#{name}.rb"
-      end
-    end
-  end
-end

data/lib/wayfarer/cli/job.rb

@@ -1,56 +0,0 @@
-# frozen_string_literal: true
-
-module Wayfarer
-  module CLI
-    class Job < Base
-      desc "perform JOB URL",
-           "Perform JOB with URL"
-      option :mock_redis, type: :boolean
-      option :batch, type: :string, default: SecureRandom.uuid
-      def perform(job, url)
-        load_environment
-        mock_redis if options[:mock_redis]
-
-        job = job.classify.constantize.new
-        task = Wayfarer::Task.new(url, options[:batch])
-        job.arguments.push(task)
-        job.perform(task)
-        task.gc.run
-      end
-
-      desc "enqueue JOB URL",
-           "Enqueue JOB with URL"
-      option :batch, type: :string, default: SecureRandom.uuid
-      def enqueue(job, url)
-        load_environment
-        mock_redis if options[:mock_redis] # TODO: Remove, does not belong here
-
-        job = job.classify.constantize
-        job.crawl(url, batch: options[:batch])
-      end
-
-      desc "execute JOB URL",
-           "Execute JOB with async adapter"
-      option :mock_redis, type: :boolean
-      option :batch, type: :string, default: SecureRandom.uuid
-      option :min_threads, type: :numeric, default: 1
-      option :max_threads, type: :numeric, default: 1
-      def execute(job, url)
-        load_environment
-        mock_redis if options[:mock_redis]
-
-        job = job.classify.constantize
-        job.queue_adapter = ActiveJob::QueueAdapters::AsyncAdapter.new(min_threads: options[:min_threads],
-                                                                       max_threads: options[:max_threads])
-        scheduler = job.queue_adapter.instance_variable_get(:@scheduler)
-        executor = scheduler.instance_variable_get(:@async_executor)
-
-        job.crawl(url, batch: options[:batch])
-
-        sleep(1) while executor.scheduled_task_count > executor.completed_task_count
-
-        free_agent_pool
-      end
-    end
-  end
-end

data/lib/wayfarer/cli/route.rb

@@ -1,29 +0,0 @@
-# frozen_string_literal: true
-
-# # frozen_string_literal: true
-
-module Wayfarer
-  module CLI
-    class Route < Base
-      desc "result JOB URL",
-           "Invoke JOB's router with URL"
-      def result(job, url)
-        load_environment
-        url = Addressable::URI.parse(url)
-        job = job.classify.constantize
-        job.router.invoke(url, job.new.steer)
-        say Wayfarer::Routing::PathFinder.result(job.router.root, url)
-      end
-
-      desc "tree JOB URL",
-           "Visualize JOB's routing tree for URL"
-      def tree(job, url)
-        load_environment
-        url = Addressable::URI.parse(url)
-        job = job.classify.constantize
-        job.router.invoke(url, job.new.steer)
-        Wayfarer::CLI::RoutePrinter.print(job.router.root, url)
-      end
-    end
-  end
-end

data/lib/wayfarer/cli/runner.rb

@@ -1,34 +0,0 @@
-# frozen_string_literal: true
-
-module Wayfarer
-  module CLI
-    class Runner < Thor
-      def self.exit_on_failure?
-        true
-      end
-
-      def self.loader
-        # TODO: Raises if $PWD/app/jobs does not exist
-
-        @loader ||= Zeitwerk::Loader.new.tap do |load|
-          load.inflector = Zeitwerk::GemInflector.new(__FILE__)
-          load.push_dir("#{Dir.pwd}/app/jobs")
-        end
-      end
-
-      desc "version", "Print version"
-      def version
-        say Wayfarer::VERSION::STRING
-      end
-
-      desc "route", "Routing"
-      subcommand "route", Wayfarer::CLI::Route
-
-      desc "generate", "Generators"
-      subcommand "generate", Wayfarer::CLI::Generate
-
-      desc "job", "Jobs"
-      subcommand "job", Wayfarer::CLI::Job
-    end
-  end
-end

data/lib/wayfarer/cli/templates/Gemfile.tt

@@ -1,5 +0,0 @@
-# frozen_string_literal: true
-
-source "https://rubygems.org"
-
-gem "wayfarer", <%= Wayfarer::VERSION::STRING.inspect %>

data/lib/wayfarer/cli/templates/job.rb.tt

@@ -1,10 +0,0 @@
-# frozen_string_literal: true
-
-class <%= @name.camelize %> < Wayfarer::Base
-  route { to :index }
-
-  retry_on ConnectionPool::TimeoutError, attempts: 3
-
-  def index
-  end
-end

data/lib/wayfarer/config/capybara.rb

@@ -1,10 +0,0 @@
-# frozen_string_literal: true
-
-module Wayfarer
-  module Config
-    Capybara = Struct.new(driver: {
-                            env_key: "WAYFARER_CAPYBARA_DRIVER",
-                            type: Symbol
-                          })
-  end
-end

data/lib/wayfarer/config/ferrum.rb

@@ -1,11 +0,0 @@
-# frozen_string_literal: true
-
-module Wayfarer
-  module Config
-    Ferrum = Struct.new(options: {
-                          env_key: "WAYFARER_FERRUM_OPTIONS",
-                          type: Hash,
-                          default: {}
-                        })
-  end
-end

data/lib/wayfarer/config/networking.rb

@@ -1,26 +0,0 @@
-# frozen_string_literal: true
-
-module Wayfarer
-  module Config
-    Networking = Struct.new(agent: {
-                              env_key: "WAYFARER_NETWORK_AGENT",
-                              type: Symbol,
-                              default: :http
-                            },
-                            pool_size: {
-                              env_key: "WAYFARER_NETWORK_POOL_SIZE",
-                              type: Integer,
-                              default: 1
-                            },
-                            pool_timeout: {
-                              env_key: "WAYFARER_NETWORK_POOL_TIMEOUT",
-                              type: Integer,
-                              default: 10
-                            },
-                            http_headers: {
-                              env_key: "WAYFARER_NETWORK_HTTP_HEADERS",
-                              type: Hash,
-                              default: {}
-                            })
-  end
-end

data/lib/wayfarer/config/redis.rb

@@ -1,14 +0,0 @@
-# frozen_string_literal: true
-
-module Wayfarer
-  module Config
-    Redis = Struct.new(url: {
-                         env_key: "WAYFARER_REDIS_URL",
-                         type: String,
-                         default: "redis://localhost:6379"
-                       },
-                       factory: {
-                         default: ->(redis) { ::Redis.new(url: redis.url) }
-                       })
-  end
-end

data/lib/wayfarer/config/root.rb

@@ -1,11 +0,0 @@
-# frozen_string_literal: true
-
-module Wayfarer
-  module Config
-    Root = Struct.new(ferrum: { default: Wayfarer::Config::Ferrum.new },
-                      network: { default: Wayfarer::Config::Networking.new },
-                      redis: { default: Wayfarer::Config::Redis.new },
-                      selenium: { default: Wayfarer::Config::Selenium.new },
-                      capybara: { default: Wayfarer::Config::Capybara.new })
-  end
-end

data/lib/wayfarer/config/selenium.rb

@@ -1,21 +0,0 @@
-# frozen_string_literal: true
-
-module Wayfarer
-  module Config
-    Selenium = Struct.new(driver: {
-                            env_key: "WAYFARER_SELENIUM_DRIVER",
-                            type: Symbol,
-                            default: :chrome
-                          },
-                          options: {
-                            env_key: "WAYFARER_SELENIUM_OPTIONS",
-                            type: Hash,
-                            default: {}
-                          },
-                          client_timeout: {
-                            env_key: "WAYFARER_SELENIUM_CLIENT_TIMEOUT",
-                            type: Integer,
-                            default: 60 # seconds
-                          })
-  end
-end

data/lib/wayfarer/config/strconv.rb

@@ -1,45 +0,0 @@
-# frozen_string_literal: true
-
-module Wayfarer
-  module Config
-    module Strconv
-    module_function
-
-      def parse(str, type = nil)
-        return primitive(str) unless type
-
-        case type.name
-        when "Hash" then hash(str)
-        when "Array" then array(str)
-        when "Symbol" then str.to_sym
-        when "Integer" then Integer(str)
-        else str
-        end
-      end
-
-      def hash(str)
-        array(str).reduce({}) do |acc, pair|
-          k, v = pair.split(":", 2)
-          next acc unless k && v
-
-          acc.merge({ parse(k, Symbol) => primitive(v) })
-        end
-      end
-
-      def array(str)
-        str.split(",").map(&:strip)
-      end
-
-      def primitive(str)
-        return true if str == "true"
-        return false if str == "false"
-
-        begin
-          parse(str, Integer)
-        rescue StandardError
-          str
-        end
-      end
-    end
-  end
-end

data/lib/wayfarer/config/struct.rb

@@ -1,72 +0,0 @@
-# frozen_string_literal: true
-
-module Wayfarer
-  module Config
-    class Struct
-      module ClassMethods
-        attr_accessor :members
-      end
-
-      module InstanceMethods
-        extend Forwardable
-
-        delegate members: "self.class"
-
-        attr_reader :env
-
-        def initialize(env = ENV)
-          @env = env
-
-          define_writers
-          define_readers
-        end
-
-      private
-
-        def define_writers
-          members.each { |key, _| define_writer(key) }
-        end
-
-        def define_writer(key)
-          define_singleton_method(:"#{key}=") do |val|
-            set(key, val)
-          end
-        end
-
-        def define_readers
-          members.each { |key, options| define_reader(key, **options) }
-        end
-
-        def define_reader(key, env_key: nil, type: nil, default: nil)
-          define_singleton_method(key.to_sym) do
-            get(key) || set(key, env_val(env_key, type) || default)
-          end
-        end
-
-        def env_val(env_key, type)
-          return nil unless env_key
-          return nil unless env.key?(env_key)
-
-          Strconv.parse(env[env_key], type)
-        end
-
-        def get(key)
-          instance_variable_get(:"@#{key}")
-        end
-
-        def set(key, val)
-          instance_variable_set(:"@#{key}", val)
-        end
-      end
-
-      def self.new(members)
-        Class.new do
-          include InstanceMethods
-          extend ClassMethods
-
-          self.members = members
-        end
-      end
-    end
-  end
-end

data/lib/wayfarer/middleware/fetch.rb

@@ -1,56 +0,0 @@
-# frozen_string_literal: true
-
-module Wayfarer
-  module Middleware
-    class Fetch
-      extend Base
-
-      module API
-        def agent
-          context.instance
-        end
-
-        def context
-          task.metadata.context
-        end
-
-        def page(live: false)
-          return task.metadata.page unless live
-
-          task.metadata.page = context.live&.page || task.metadata.page
-        end
-
-        def http
-          @http ||= Wayfarer::Networking::Follow.new(
-            Wayfarer::Networking::Context.new(
-              Wayfarer::Networking::HTTP.new
-            )
-          )
-        end
-      end
-
-      def call(task)
-        pool.with do |context|
-          result = task.metadata.controller.run_callbacks(:fetch) do
-            context.fetch(task.url)
-          end
-
-          case result
-          when Networking::Result::Redirect
-            task.metadata.controller.stage(result.redirect_url)
-          when Networking::Result::Success
-            task.metadata.context = context
-            task.metadata.page = result.page
-            yield if block_given?
-          end
-        end
-      end
-
-    private
-
-      def pool
-        Wayfarer::Networking::Pool.instance
-      end
-    end
-  end
-end

data/lib/wayfarer/redis/connection.rb

@@ -1,13 +0,0 @@
-# frozen_string_literal: true
-
-module Wayfarer
-  module Redis
-    module Connection
-      extend Forwardable
-
-      delegate with: "Wayfarer::Redis::Pool.instance"
-
-      alias redis with
-    end
-  end
-end

data/lib/wayfarer/redis/version.rb

@@ -1,19 +0,0 @@
-# frozen_string_literal: true
-
-module Wayfarer
-  module Redis
-    module Version
-      extend Connection
-
-    module_function
-
-      def determine
-        @determine ||= server_version.split(".").first(3).map(&:to_i)
-      end
-
-      def server_version
-        redis { |conn| conn.info["redis_version"] }
-      end
-    end
-  end
-end

data/lib/wayfarer/routing/router.rb

@@ -1,28 +0,0 @@
-# frozen_string_literal: true
-
-module Wayfarer
-  module Routing
-    class Router
-      ArgumentCountError = Class.new(StandardError) # TODO: Unused, remove
-
-      extend Forwardable
-
-      attr_reader :root,
-                  :blocks
-
-      def initialize
-        @blocks = []
-      end
-
-      def draw(&block)
-        @blocks.push(block)
-      end
-
-      def invoke(url, arguments)
-        @root = Wayfarer::Routing::RootRoute.new
-        @blocks.each { |block| Docile.dsl_eval(@root, *arguments, &block) }
-        root.invoke(url)
-      end
-    end
-  end
-end