wayfarer 0.0.3 → 0.4.3

data/docs/guides/configuration.md

@@ -1,184 +1,39 @@
----
-layout: default
-title: Configuration
----
-
 # Configuration
 
-All job classes base their configuration off the global one.
-
-{% highlight ruby %}
-# Setting a key globally applies to all jobs ...
-Wayfarer.config.key = :value
-
-class DummyJob < Wayfarer::Job
-  # ... unless a job overrides it
-  config.key = :other_value
-end
-
-class DummyJob < Wayfarer::Job
-  # Have it yielded
-  config { |c| c.key = :other_value }
-end
-{% endhighlight %}
-
-## Recognized keys and values
-
-### `print_stacktraces`
-  * Default: `true`
-  * Recognized values: Booleans
-
-Whether to print stacktraces when encounterting unhandled exceptions in job actions. See [Error handling]({{base}}/guides/error_handling.html).
-
----
-
-### `reraise_exceptions`
-
-* Default: `false`
-* Recognized values: Booleans
-
-Whether to crash when encountering unhandled exceptions in job actions. See [Error handling]({{base}}/guides/error_handling.html).
-
----
-
-### `allow_circulation`
-
-* Default: `false`
-* Recognized values: Booleans
-
-Whether URIs may be visited twice.
-
-<aside class="note">
-Allowing circulation might cause your jobs to not terminate.
-</aside>
-
----
-
-### `normalize_uris`
-
-* Default: `true`
-* Recognized values: Booleans
-
-Whether to strip fragments, reorder query keys, etc. when staging and caching URIs. Customizable with the `:normalize_uri_options` key. See [normalize_url](https://github.com/rwz/normalize_url).
-
----
-
-### `normalize_uri_options`
-
-* Default: `{}`
-* Recognized values: See [normalize_url](https://github.com/rwz/normalize_url).
-
----
-
-### `frontier`
-* Default: `:memory`
-* Recognized values: See [(Redis) frontiers](frontiers.html).
-
-Which frontier to use.
-
-<aside class="note">
-Bloom filters may yield false positives. See the <a href="https://en.wikipedia.org/wiki/Bloom_filter">Wikipedia article</a>.
-</aside>
+Wayfarer can be configured in two ways:
 
----
+1. Using [environment variables](/reference/environment_variables)
+2. Using runtime configuration
 
-### `connection_count`
+## Runtime configuration
 
-* Default: `4`
-* Recognized values: Integers
+Wayfarer parses environment variables into a runtime configuration
+`Wayfarer::Config`. The configuration can then be altered or replaced via
+`Wayfarer.config`:
 
-How many threads and HTTP adapters to use (1:1 correspondence).
+```ruby
+# Which user agent to use to process tasks
+Wayfarer.config.network.agent = :http # or :ferrum, :selenium
 
----
+# How many user agents to instantiate
+Wayfarer.config.network.pool_size = 3
 
-### `http_adapter`
+# How long an agent may be used while processing a task
+Wayfarer.config.network.pool_timeout = 5000
 
-* Default: `:net_http`
-* Recognized values: `:net_http`, `:selenium`
+# Ferrum options
+Wayfarer.config.ferrum.options = {}
 
-Which HTTP adapter to use. See [Selenium & Capybara](selenium_capybara.html).
+# Selenium driver to use
+Wayfarer.config.selenium.driver = :chrome
 
----
+# Selenium HTTP client read timeout
+Wayfarer.config.selenium.client_timeout = 10 # seconds
 
-### `connection_timeout`
+# Selenium options
+Wayfarer.config.selenium.options = { url: "http://chrome" }
 
-* Default: `Float::INFINITY`
-* Recognized values: Floats
-
-Time in seconds that a job instance may hold an HTTP adapter. Instances that exceed this time limit raise an exception.
-
----
-
-### `max_http_redirects`
-
-* Default: `3`
-* Recognized values: Integers
-
-How many 3xx redirects to follow.
-
-<aside class="note">
-Has no effect when using the <code>:selenium</code> HTTP adapter.
-</aside>
-
----
-
-### `selenium_argv`
-
-* Default: `[:firefox]`
-* Recognized values: See [Selenium & Capybara](selenium_capybara.html)
-
-Argument vector passed to [`Selenium::WebDriver::Driver::for`](http://www.rubydoc.info/gems/selenium-webdriver/Selenium/WebDriver/Driver#for-class_method).
-
----
-
-### `redis_opts`
-
-* Default: `{ host: "localhost", port: 6379 }`
-* Recognized values: [See documentation](http://www.rubydoc.info/github/redis/redis-rb/Redis%3Ainitialize)
-
-Options passed to [`Redis#initialize`](http://www.rubydoc.info/github/redis/redis-rb/Redis%3Ainitialize).
-
----
-
-### `bloomfilter_opts`
-
-* Default:
-```
-{
-  size: 100,
-  hashes: 2,
-  seed: 1,
-  bucket: 3,
-  raise: false
-}
+# HTTP request headers (Selenium is unsupported)
+Wayfarer.config.network.http_headers = { "Field" => "Value" }
 ```
-* Recognized values:
-  * `size`: Integers; number of buckets in a bloom filter
-  * `hashes`: Integers; number of hash functions
-  * `seed`: Integers; seed of hash functions
-  * `bucket`: Integers; number of bits in a bloom filter bucket
-  * `raise`: Booleans; whether to raise on bucket overflow
-
-Options for [bloomfilter-rb](https://github.com/igrigorik/bloomfilter-rb).
-
----
-
-### `window_size`
-
-* Default: `[1024, 768]`
-* Recognized values: `[Integer, Integer]`
-
-Dimensions of browser windows.
-
-<aside class="note">
-Only has an effect when using the <code>:selenium</code> HTTP adapter.
-</aside>
-
----
-
-### `mustermann_type`
-
-* Default: `:sinatra`
-* Recognized values: [See documentation](https://github.com/sinatra/mustermann)
 
-Which [Mustermann](https://github.com/sinatra/mustermann) pattern type to use.

data/docs/guides/debugging.md

@@ -0,0 +1,17 @@
+# Debugging
+
+[Wayfarer's CLI](/reference/cli/) has two sub-commands that come in handy when
+diagnosing problems in the development workflow.
+
+## Routing a URL from the shell
+
+## `wayfarer route`
+
+### `wayfarer route result JOB URL`
+
+:   Prints the result of invoking `JOB`'s router with `URL`.
+
+### `wayfarer route tree JOB URL`
+
+:   Visualises the routing tree result of invoking `JOB`'s router with `URL`.
+

data/docs/guides/error_handling.md

@@ -1,46 +1,31 @@
----
-layout: default
-title: Error handling
----
-
 # Error handling
-By default, all exceptions raised within actions are swallowed and only their stacktraces printed to stderr. This behaviour can be changed with two configuration keys (see [Configuration]()):
-
-1. `print_stacktraces`: Whether to print stacktraces (default: `true`)
-2. `reraise_exceptions`: Whether to crash when encountering unhandled exceptions (default: `false`)
-
-Here’s an example to illustrate the default behaviour:
-
-{% highlight ruby %}
-class DummyJob < Wayfarer::Job
-  def example
-    # Makes this instance fail, but processing goes on
-    # Prints the stacktrace to stderr
-    fail "It's okay, life goes on"
-  end
-end
-{% endhighlight %}
-
-The following reraises all exceptions, stops processing and returns with a non-zero exit code:
-
-{% highlight ruby %}
-class DummyJob < Wayfarer::Job
-  config.reraise_exceptions = true
-
-  def example
-    fail "This makes the exception bubble up"
-  end
-end
-{% endhighlight %}
-
-And if you don’t want to be bothered with exceptions at all:
-
-{% highlight ruby %}
-class DummyJob < Wayfarer::Job
-  config.print_stacktraces = false
-
-  def example
-    fail "No one will know about this ..."
-  end
-end
-{% endhighlight %}
+
+## Wayfarer never swallows exceptions
+
+* Wayfarer never swallows exceptions.
+* Jobs with unhandled exceptions are not retried.
+
+## Retrying and discarding
+
+Wayfarer relies on [Active Job's two error handling facilities](https://guides.rubyonrails.org/active_job_basics.html#exceptions).
+
+* `retry_on` to retry jobs a number of times on certain errors:
+
+    ```ruby
+    class DummyJob < Wayfarer::Base
+      retry_on MyError, attempts: 3 do |job, error|
+        # This block runs once all 3 attempts have failed
+        # (1 initial attempt + 2 retries)
+      end
+    end
+    ```
+
+* `discard_on` to throw away jobs on certain errors:
+
+    ```ruby
+    class DummyJob < Wayfarer::Base
+      discard_on MyError do |job, error|
+        # This block runs once and buries the job
+      end
+    end
+    ```

data/docs/guides/jobs.md

@@ -0,0 +1,101 @@
+# Jobs
+
+Jobs are Ruby classes that process [tasks](/guides/tasks) and look as follows:
+
+```ruby
+class DummyJob < Wayfarer::Base
+  route { to :index }
+
+  def index
+  end
+end
+```
+
+Here is how to enqueue a task for a URL:
+
+```ruby
+DummyJob.crawl_later("https://example.com")
+```
+
+This is the same as calling the Active Job API directly and passing a task
+and a random batch:
+
+```ruby
+task = Wayfarer::Task.new("https://example.com", SecureRandom.uuid)
+DummyJob.perform_later(task)
+```
+
+A batch can be specified with `::crawl_later`, too:
+
+```ruby
+DummyJob.crawl_later("https://example.com", batch: "my-batch")
+```
+
+## Current task
+
+Jobs consume [tasks](../tasks) from a message queue. The currently processed
+task is accessible like so:
+
+```ruby
+class DummyJob < Wayfarer::Base
+  route { to :index }
+
+  def index
+    task.url   # => "https://example.com"
+    task.batch # => "my-batch"
+  end
+end
+```
+
+## Current page
+
+A task's URL contents get fetched into a [page](../pages) object if the task URL
+matched a route:
+
+```ruby
+class DummyJob < Wayfarer::Base
+  route { to :index, host: "example.com" }
+
+  def index
+    page.url         # => "https://example.com"
+    page.body        # => "<html>..."
+    page.status_code # => 200
+    page.headers     # { "Content-Type" => ... }
+  end
+end
+```
+
+## URL parameters
+
+Jobs can extract data from URLs with their router:
+
+```ruby
+class DummyJob < Wayfarer::Base
+  route do
+    path "/users/:id/profile"
+  end
+
+  def index
+    params[:id] # => "42"
+  end
+end
+
+DummyJob.crawl_later("https://example.com/users/42/profile")
+```
+
+
+## User agent
+
+The HTTP client or automated browser that fetched the URL is available:
+
+```ruby
+Wayfarer.config.network.agent = :ferrum # Chrome DevTools Protocol
+
+class DummyJob < Wayfarer::Base
+  route { to :index }
+
+  def index
+    browser.save_screenshot("capture.png")
+  end
+end
+```

data/docs/guides/navigation.md

@@ -0,0 +1,73 @@
+# Navigation
+
+Wayfarer has two mechanisms for navigating crawls:
+
+* Jobs have a router that decides if a task's URL gets fetched and processed.
+* Jobs can add URLs to a processing set with `#stage`.
+
+## Staging URLs
+
+Jobs can turn URLs into tasks within their own batch with `#stage`. Staging a
+URL does not enqueue it immediately. Instead, the URL is added to a processing
+set first.
+
+```ruby
+class DummyJob < Wayfarer::Base
+  route { to :index }
+
+  def index
+    stage page.meta.links.all
+  end
+end
+```
+
+Once the `index` action method returns, all URLs in `page.meta.links.all`
+are (1) normalized to a canonical form and (2) checked for inclusion in
+the batch's processed URL Redis set. All unprocessed URLs are enqueued as
+tasks within the same batch.
+
+`#stage` can be called arbitrarily often, with invalid URLs too, as they are
+filtered out behind the scenes:
+
+```ruby
+def index
+  stage "_bro:ken@url/" # => ["_bro:ken@url/"]
+end
+```
+
+See also: [Performance: Stage less URLs](/guides/performance)
+
+!!! attention "Failing action methods do not enqueue tasks"
+
+    If an action method fails as in:
+
+    ```ruby
+    def index
+      stage page.meta.links.all
+      fail "Error occured"
+    end
+    ```
+
+    None of the staged URLs are enqueued as tasks. Jobs that raise an exception
+    should get retried, or the exception should be handled.
+
+
+## Routing URLs
+
+In the following example, the task is written to the message queue, but the
+job's routes do not match the URL. When the task gets consumed, the URL does not
+get fetched and the action method not called.
+
+```ruby
+class DummyJob < Wayfarer::Base
+  route do
+    host "example.com", path: "/users/:user_id", to: :user
+  end
+
+  # ...
+end
+
+DummyJob.crawl_later("https://mismatching.host/users/42")
+```
+
+

data/docs/guides/networking.md

@@ -0,0 +1,94 @@
+# Networking
+
+Wayfarer navigates the web in two ways:
+
+1. Via plain HTTP requests
+2. By automating browsers
+
+Both options are mutually exclusive per Ruby process.
+
+## User agents
+
+A user agent is an entity that knows how to retrieve the contents behind a URL.
+
+The user agent can be configured via the global configuration:
+
+```ruby
+Wayfarer.config.network.agent = :http # or :ferrum, :selenium
+```
+
+## Connection pooling
+
+Wayfarer keeps user agents within a connection pool. When a job executes
+and needs to retrieve the contents behind a URL, an agent is checked out from
+the pool.
+
+The pool has a constant size and it should equal the number of threads the
+underlying message queue operates with. The size can be configured via the
+global configuration:
+
+```ruby
+Wayfarer.config.network.pool_size = 8
+```
+
+### Timeouts
+
+user agents may stay checked out from the pool by jobs for a limited time
+only. Once this time limit is exceeded, a `ConnectionPool::TimeoutError`
+exception is raised. This places a hard time limit on every job.
+
+The timeout can be configured via the global configuration:
+
+```ruby
+Wayfarer.config.network.pool_timeout = 20 # seconds
+```
+
+Because jobs with unhandled exceptions fail, explicit error handling is required
+if retries are desired:
+
+```ruby
+class DummyJob < Wayfarer::Base
+  retry_on ConnectionPool::TimeoutError, attempts: 3
+end
+```
+
+## Agent-specific client timeouts
+
+The time in seconds it may take to communicate with remote browser processes can
+be configured globally per agent:
+
+```ruby
+Wayfarer.config.ferrum.options = { timeout: 5 }
+Wayfarer.config.selenium.client_timeout = 60
+```
+
+### Shared state
+
+As user agents get checked in and out continously between jobs, their state
+carries over from job to job, too.
+
+For browser automation, this means:
+
+* A job finds the browser at the last URL the previous job has left off.
+* The browser's cookies might have been set, or other client-side state might
+  exist that significantly affects a page's behaviour.
+
+## HTTP redirect handling
+
+Browsers follow redirects transparently when they are navigated to a URL.
+
+When using plain HTTP, redirect URLs are enqueued transparently within the same
+batch. URLs that result in 3xx responses will not be retrieved again within
+their batch.
+
+## HTTP request headers
+
+Request headers can be configured via the global configuration:
+
+```ruby
+Wayfarer.config.network.http_headers = { "Field" => "Value" }
+```
+
+!!! attention "Partial support"
+
+    Selenium does not support configuring HTTP request headers.

data/docs/guides/pages.md

@@ -0,0 +1,52 @@
+# Pages
+
+Retrieved pages take the shape of `Wayfarer::Page` objects and are available
+to jobs:
+
+```ruby
+class DummyJob < Wayfarer::Worker
+  route { to :index }
+
+  def index
+    page # => #<Wayfarer::Page ...>
+
+    page.url         # => "https://example.com"
+    page.body        # => "<html>..."
+    page.status_code # => 200
+    page.headers     # => { "Content-Type" => ... }
+
+    # A MetaInspector object for accessing page meta data.
+    # See: https://github.com/metainspector/metainspector
+    # Examples:
+    page.meta.links.internal
+    page.meta.images.favicon
+    page.meta.description
+    page.meta.feeds
+  end
+end
+```
+
+## Live pages
+
+When automating browsers, it is possible the page changes significantly at
+runtime, for example due to JavaScript altering the DOM or URL.
+
+To access a page reflecting the current browser state, pass the `live` keyword:
+
+```ruby
+class DummyJob < Wayfarer::Worker
+  route { to :index }
+
+  def index
+    page # => #<Wayfarer::Page ...>
+
+    # Replaces the current Page object with a newer one,
+    # taking into account the DOM as currently rendered by the browser.
+    # Effectful only when automating browsers, no-op when using plain
+    # HTTP.
+    page(live: true)
+
+    page # => The live page returned above
+  end
+end
+```