wayfarer 0.0.3 → 0.4.0

data/docs/cookbook/querying_html.md

@@ -0,0 +1,42 @@
+# Querying HTML
+
+See: [Nokogiri: Searching an HTML / XML Document](https://nokogiri.org/tutorials/searching_a_xml_html_document.html#basic-searching)
+
+=== "Nokogiri"
+
+    ```ruby
+    class DummyJob < Wayfarer::Base
+      route.to :index
+
+      def index
+        page.doc.css("html")
+        page.doc.xpath("//html")
+      end
+    end
+    ```
+
+=== "Ferrum"
+
+    ```ruby
+    class DummyJob < Wayfarer::Base
+      route.to :index
+
+      def index
+        browser.at_css("html")
+        browser.at_xpath("//html")
+      end
+    end
+    ```
+
+=== "Selenium"
+
+    ```ruby
+    class DummyJob < Wayfarer::Base
+      route.to :index
+
+      def index
+        browser.find_elements(css: "html")
+        browser.find_elements(xpath: "//html")
+      end
+    end
+    ```

data/docs/cookbook/screenshots.md

@@ -0,0 +1,27 @@
+# Screenshots
+
+Taking screenshots requires automating a browser.
+
+=== "Ferrum"
+
+    ```ruby
+    class DummyJob < Wayfarer::Base
+      route.to :index
+
+      def index
+        browser.screenshot(path: "screenshot.png")
+      end
+    end
+    ```
+
+=== "Selenium"
+
+    ```ruby
+    class DummyJob < Wayfarer::Base
+      route.to :index
+
+      def index
+        browser.save_screenshot("screenshot.png")
+      end
+    end
+    ```

data/docs/cookbook/user_agent.md

@@ -0,0 +1,7 @@
+# User Agent
+
+See: [Guides: Networking: HTTP request headers](/guides/networking#http-request-headers)
+
+```ruby
+Wayfarer.config.http_headers = { "User-Agent" => "MyCrawler ..." }
+```

data/docs/guides/browser_automation/capybara.md

@@ -0,0 +1,3 @@
+# Capybara
+
+TODO

data/docs/guides/browser_automation/ferrum.md

@@ -0,0 +1,37 @@
+---
+title: Ferrum (CDP)
+---
+
+# Ferrum (Chrome DevTools Protocol)
+
+Google Chrome can be automated using
+[Ferrum](https://github.com/rubycdp/ferrum).
+
+When Ferrum is in use, a Google Chrome process is accessible within jobs like
+so:
+
+```ruby
+class DummyWorker < Wayfarer::Worker
+  route.to :index
+
+  def index
+    browser # => #<Ferrum::Browser ...>
+  end
+end
+```
+
+## Configuring Ferrum
+
+=== "Runtime"
+
+    ```ruby
+    Wayfarer.config.adapter = :ferrum
+    Wayfarer.config.ferrum_options = { headless: false, url: "http://chrome:3000" }
+    ```
+
+=== "Environment variables"
+
+    ```
+    WAYFARER_ADAPTER=ferrum
+    WAYFARER_FERRUM_OPTIONS=headless:false,url:http://chrome:3000
+    ```

data/docs/guides/browser_automation/selenium.md

@@ -0,0 +1,59 @@
+# Selenium
+
+A variety of browsers can be automated using
+[Selenium](https://www.selenium.dev).
+
+When Selenium is in use, a remote browser process is accessible within jobs like
+so:
+
+```ruby
+class DummyWorker < Wayfarer::Worker
+  route.to :index
+
+  def index
+    browser # => #<Selenium::WebDriver ...>
+  end
+end
+```
+
+Using Selenium requires running a [driver](https://www.selenium.dev/downloads)
+process.
+
+!!! attention "Fake response header and status code"
+    Selenium does not provide response headers or status code.
+
+    Pages retrieved with a Selenium WebDriver return fake values:
+
+    ```ruby
+    Wayfarer.config.adapter = :selenium
+
+    class DummyJob < Wayfarer::Base
+      route.to :index
+
+      def index
+        page.headers     # => always {}
+        page.status_code # => always 200
+      end
+    end
+    ```
+
+!!! note "Consider using [Ferrum](../ferrum) instead"
+    Ferrum provides superior stability and a richer feature set compared to
+    Selenium drivers. However Ferrum automates only Google Chrome. Unless a
+    different browser is required, consider using Ferrum instead of Selenium.
+
+## Configuring Selenium
+
+=== "Runtime"
+
+    ```ruby
+    Wayfarer.config.adapter = :selenium
+    Wayfarer.config.selenium_argv = [:firefox]
+    ```
+
+=== "Environment variables"
+
+    ```
+    WAYFARER_ADAPTER=selenium
+    WAYFARER_SELENIUM_ARGV=firefox
+    ```

data/docs/guides/callbacks.md

@@ -1,45 +1,38 @@
----
-layout: default
-title: Callbacks
----
-
 # Callbacks
 
-Besides all [ActiveJob callbacks](http://api.rubyonrails.org/classes/ActiveJob/Callbacks/ClassMethods.html), three other callbacks are available. You have access to [locals](/guides/locals) in all callbacks.
+## Life cycle callbacks
 
-## `before_crawl`
-Fires __once__ before any pages have been retrieved.
+Wayfarer supports all of Active Job's life cycle callbacks:
 
-{% highlight ruby %}
-class DummyJob < Wayfarer::Job
-  before_crawl { puts "Work is about to happen"  }
-end
-{% endhighlight %}
+* [Active Job Basics: Callbacks](https://edgeguides.rubyonrails.org/active_job_basics.html#callbacks)
+* [ActiveJob::Callbacks](https://api.rubyonrails.org/classes/ActiveJob/Callbacks/ClassMethods.html)
+
+## `after_batch` callbacks
 
-## `after_crawl`
-Fires __once__ after all pages have been retrieved and processing is done.
+Jobs can register callbacks to run once all jobs in their batch have concluded:
+
+```ruby
+class DummyJob < Wayfarer::Base
+  after_batch do
+    # All jobs in batch done
+  end
 
-{% highlight ruby %}
-class DummyJob < Wayfarer::Job
-  after_crawl { puts "Work did happen"  }
+  after_batch do
+    # Multiple callbacks can be registered
+  end
 end
-{% endhighlight %}
+```
 
-## `setup_adapter`
-Fires for every adapter immediately after its creation.
+Internally, a batch counter is in-/decremented on certain events. Once the
+counter reaches zero, `after_batch` callbacks execute in declaration order.
 
-The block gets yielded an adapter. When using the Selenium HTTP adapter, both a WebDriver and a wrapping Capybara driver get yielded.
+The counter is incremented when:
 
-{% highlight ruby %}
-class DummyJob < Wayfarer::Job
-  config.http_adapter = :selenium
-  config.connection_count = 4
+* A job is enqueued within the batch.
 
-  setup_adapter do |adapter, driver, browser|
-    # This block gets called 4 times with different adapters
-    adapter # => The HTTP adapter
-    driver  # => #<Selenium::WebDriver::Driver:...> or nil
-    browser # => #<Capybara::Selenium::Driver:...> or nil
-  end
-end
-{% endhighlight %}
+The counter is decremented when:
+
+* A job succeeds.
+* A job fails due to an unhandled exception.
+* A job fails due to a discarded exception.
+* A job fails and thereyby exhausts its maximum attempts.

data/docs/guides/configuration.md

@@ -1,184 +1,33 @@
----
-layout: default
-title: Configuration
----
-
 # Configuration
 
-All job classes base their configuration off the global one.
-
-{% highlight ruby %}
-# Setting a key globally applies to all jobs ...
-Wayfarer.config.key = :value
-
-class DummyJob < Wayfarer::Job
-  # ... unless a job overrides it
-  config.key = :other_value
-end
-
-class DummyJob < Wayfarer::Job
-  # Have it yielded
-  config { |c| c.key = :other_value }
-end
-{% endhighlight %}
-
-## Recognized keys and values
-
-### `print_stacktraces`
-  * Default: `true`
-  * Recognized values: Booleans
-
-Whether to print stacktraces when encounterting unhandled exceptions in job actions. See [Error handling]({{base}}/guides/error_handling.html).
-
----
-
-### `reraise_exceptions`
-
-* Default: `false`
-* Recognized values: Booleans
-
-Whether to crash when encountering unhandled exceptions in job actions. See [Error handling]({{base}}/guides/error_handling.html).
-
----
-
-### `allow_circulation`
-
-* Default: `false`
-* Recognized values: Booleans
-
-Whether URIs may be visited twice.
-
-<aside class="note">
-Allowing circulation might cause your jobs to not terminate.
-</aside>
-
----
-
-### `normalize_uris`
-
-* Default: `true`
-* Recognized values: Booleans
-
-Whether to strip fragments, reorder query keys, etc. when staging and caching URIs. Customizable with the `:normalize_uri_options` key. See [normalize_url](https://github.com/rwz/normalize_url).
-
----
-
-### `normalize_uri_options`
-
-* Default: `{}`
-* Recognized values: See [normalize_url](https://github.com/rwz/normalize_url).
-
----
-
-### `frontier`
-* Default: `:memory`
-* Recognized values: See [(Redis) frontiers](frontiers.html).
-
-Which frontier to use.
-
-<aside class="note">
-Bloom filters may yield false positives. See the <a href="https://en.wikipedia.org/wiki/Bloom_filter">Wikipedia article</a>.
-</aside>
-
----
+Wayfarer can be configured in two ways:
 
-### `connection_count`
+1. Using [environment variables](/reference/environment_variables)
+2. Using runtime configuration
 
-* Default: `4`
-* Recognized values: Integers
+## Runtime configuration
 
-How many threads and HTTP adapters to use (1:1 correspondence).
+Wayfarer parses environment variables into a runtime configuration
+`Wayfarer::Config`. The configuration can then be altered or replaced via
+`Wayfarer.config`:
 
----
+```ruby
+# Which network adapter to use to process tasks
+Wayfarer.config.adapter = :net_http # or :ferrum, :selenium
 
-### `http_adapter`
+# How many network adapters to instantiate
+Wayfarer.config.adapter_pool_size = 3
 
-* Default: `:net_http`
-* Recognized values: `:net_http`, `:selenium`
+# How long an adapter may be used while processing a task
+Wayfarer.config.adapter_pool_timeout = 5000
 
-Which HTTP adapter to use. See [Selenium & Capybara](selenium_capybara.html).
+# Ferrum options
+Wayfarer.config.ferrum_options = {}
 
----
+# Selenium arguments
+Wayfarer.config.selenium_argv = [:chrome]
 
-### `connection_timeout`
-
-* Default: `Float::INFINITY`
-* Recognized values: Floats
-
-Time in seconds that a job instance may hold an HTTP adapter. Instances that exceed this time limit raise an exception.
-
----
-
-### `max_http_redirects`
-
-* Default: `3`
-* Recognized values: Integers
-
-How many 3xx redirects to follow.
-
-<aside class="note">
-Has no effect when using the <code>:selenium</code> HTTP adapter.
-</aside>
-
----
-
-### `selenium_argv`
-
-* Default: `[:firefox]`
-* Recognized values: See [Selenium & Capybara](selenium_capybara.html)
-
-Argument vector passed to [`Selenium::WebDriver::Driver::for`](http://www.rubydoc.info/gems/selenium-webdriver/Selenium/WebDriver/Driver#for-class_method).
-
----
-
-### `redis_opts`
-
-* Default: `{ host: "localhost", port: 6379 }`
-* Recognized values: [See documentation](http://www.rubydoc.info/github/redis/redis-rb/Redis%3Ainitialize)
-
-Options passed to [`Redis#initialize`](http://www.rubydoc.info/github/redis/redis-rb/Redis%3Ainitialize).
-
----
-
-### `bloomfilter_opts`
-
-* Default:
+# HTTP request headers (Selenium is unsupported)
+Wayfarer.config.http_headers = { "Field" => "Value" }
 ```
-{
-  size: 100,
-  hashes: 2,
-  seed: 1,
-  bucket: 3,
-  raise: false
-}
-```
-* Recognized values:
-  * `size`: Integers; number of buckets in a bloom filter
-  * `hashes`: Integers; number of hash functions
-  * `seed`: Integers; seed of hash functions
-  * `bucket`: Integers; number of bits in a bloom filter bucket
-  * `raise`: Booleans; whether to raise on bucket overflow
-
-Options for [bloomfilter-rb](https://github.com/igrigorik/bloomfilter-rb).
-
----
-
-### `window_size`
-
-* Default: `[1024, 768]`
-* Recognized values: `[Integer, Integer]`
-
-Dimensions of browser windows.
-
-<aside class="note">
-Only has an effect when using the <code>:selenium</code> HTTP adapter.
-</aside>
-
----
-
-### `mustermann_type`
-
-* Default: `:sinatra`
-* Recognized values: [See documentation](https://github.com/sinatra/mustermann)
 
-Which [Mustermann](https://github.com/sinatra/mustermann) pattern type to use.

data/docs/guides/error_handling.md

@@ -1,46 +1,31 @@
----
-layout: default
-title: Error handling
----
-
 # Error handling
-By default, all exceptions raised within actions are swallowed and only their stacktraces printed to stderr. This behaviour can be changed with two configuration keys (see [Configuration]()):
 
-1. `print_stacktraces`: Whether to print stacktraces (default: `true`)
-2. `reraise_exceptions`: Whether to crash when encountering unhandled exceptions (default: `false`)
+Wayfarer relies on Active Job's error handling facilities, `retry_on` and
+`discard_on`:
 
-Here’s an example to illustrate the default behaviour:
+* [Active Job Basics: Exceptions](https://guides.rubyonrails.org/active_job_basics.html#exceptions)
+* [ActiveJob::Exceptions](https://edgeapi.rubyonrails.org/classes/ActiveJob/Exceptions/ClassMethods.html)
 
-{% highlight ruby %}
-class DummyJob < Wayfarer::Job
-  def example
-    # Makes this instance fail, but processing goes on
-    # Prints the stacktrace to stderr
-    fail "It's okay, life goes on"
-  end
-end
-{% endhighlight %}
+## Unhandled exceptions
 
-The following reraises all exceptions, stops processing and returns with a non-zero exit code:
+Jobs with unhandled exceptions fail and are not retried.
 
-{% highlight ruby %}
-class DummyJob < Wayfarer::Job
-  config.reraise_exceptions = true
+## Retrying
 
-  def example
-    fail "This makes the exception bubble up"
+```ruby
+class DummyJob < Wayfarer::Base
+  retry_on MyError, attempts: 3 do |job, error|
+    # All 3 retry attempts have failed
   end
 end
-{% endhighlight %}
-
-And if you don’t want to be bothered with exceptions at all:
+```
 
-{% highlight ruby %}
-class DummyJob < Wayfarer::Job
-  config.print_stacktraces = false
+## Discarding
 
-  def example
-    fail "No one will know about this ..."
+```ruby
+class DummyJob < Wayfarer::Base
+  discard_on MyError do |job, error|
+    # The job will not get retried
   end
 end
-{% endhighlight %}
+```

data/docs/guides/jobs.md

@@ -0,0 +1,75 @@
+# Jobs
+
+Jobs are Ruby classes that look as follows:
+
+```ruby
+class DummyJob < Wayfarer::Base
+  route.to :index
+
+  def index
+  end
+end
+```
+
+## Current task
+
+Jobs consume [tasks](../tasks) from a message queue. The currently processed
+task is accessible like so:
+
+```ruby
+class DummyJob < Wayfarer::Base
+  route.to :index
+
+  def index
+    task.url   # => "https://example.com"
+    task.batch # => "55fe80d4-97ce-..."
+  end
+end
+```
+
+## Current page
+
+Once control is handed over to jobs, their task's URL has been retrieved into a
+[page](../pages) object:
+
+```ruby
+class DummyJob < Wayfarer::Base
+  route.to :index
+
+  def index
+    page.url  # => "https://example.com"
+    page.body # => "<html>..."
+  end
+end
+```
+
+## URL parameters
+
+TODO
+
+```ruby
+class DummyJob < Wayfarer::Base
+  route.to :index
+
+  def index
+    page.url  # => "https://example.com"
+    page.body # => "<html>..."
+  end
+end
+```
+
+
+## Automated browser
+
+When automating browsers, the remote browser process that retrieved the URL is
+accessible like so:
+
+```ruby
+class DummyJob < Wayfarer::Base
+  route.to :index
+
+  def index
+    browser # => #<Ferrum::Browser ...> or #<Selenium::WebDriver ...>
+  end
+end
+```

data/docs/guides/networking.md

@@ -0,0 +1,20 @@
+# Networking
+
+Wayfarer retrieves pages in two ways:
+
+1. Via plain HTTP requests
+2. By automating browsers
+
+TODO
+
+## HTTP request headers
+
+Request headers can be configured via the global configuration:
+
+```ruby
+Wayfarer.config.http_headers = { "Field" => "Value" }
+```
+
+!!! attention "Partial support"
+
+    Selenium does not support configuring HTTP request headers.

data/docs/guides/pages.md

@@ -0,0 +1,52 @@
+# Pages
+
+Retrieved pages are represented by `Wayfarer::Page` objects and are available
+within jobs like so:
+
+```ruby
+class DummyJob < Wayfarer::Worker
+  route.to :index
+
+  def index
+    page # => #<Wayfarer::Page ...>
+
+    page.url         # => "https://example.com"
+    page.body        # => "<html>..."
+    page.status_code # => 200
+    page.headers     # => { "Content-Type" => ... }
+
+    # A MetaInspector object for accessing page meta data.
+    # See: https://github.com/metainspector/metainspector
+    # Examples:
+    page.meta.links.internal
+    page.meta.images.favicon
+    page.meta.description
+    page.meta.feeds
+  end
+end
+```
+
+## Live pages
+
+When automating browsers, it is possible the page changes significantly at
+runtime, for example due to JavaScript altering the DOM or URL.
+
+To access a page reflecting the current browser state, pass the `live` keyword:
+
+```ruby
+class DummyJob < Wayfarer::Worker
+  route.to :index
+
+  def index
+    page # => #<Wayfarer::Page ...>
+
+    # Replaces the current Page object with a newer one,
+    # taking into account the DOM as currently rendered by the browser.
+    # Effectful only when automating browsers, no-op when using plain
+    # HTTP.
+    page(live: true)
+
+    page # => The live page returned above
+  end
+end
+```

data/docs/guides/performance.md

@@ -0,0 +1,22 @@
+# Performance
+
+How to write performant crawlers with Wayfarer.
+
+## Use Redis >= 6.2.0
+
+Redis 6.2.0 introduced the
+[`SMISMEMBER`](https://redis.io/commands/smismember) command which enables
+Wayfarer to check whether multiple URLs have been processed in a batch with a
+single command. With earlier versions, one command per URL is required.
+
+Wayfarer detects the Redis server version and uses `SMISMEMBER` without user
+configuration when supported.
+
+## Use Oj for JSON parsing
+
+Wayfarer uses [Oj](https://github.com/ohler55/oj) for JSON parsing if the gem
+has been required at runtime:
+
+```ruby
+require "oj"
+```