wayfarer 0.4.6 → 0.4.8

data/docs/cookbook/navigation.md

@@ -4,12 +4,12 @@
 
     ```ruby
     class DummyJob < Wayfarer::Base
-      route { to :index }
+      route.to :index
 
       def index
-        agent.goto("https://example.com")
-        agent.back
-        agent.forward
+        user_agent.goto("https://example.com")
+        user_agent.back
+        user_agent.forward
       end
     end
     ```
@@ -18,12 +18,12 @@
 
     ```ruby
     class DummyJob < Wayfarer::Base
-      route { to :index }
+      route.to :index
 
       def index
-        agent.navigate.to("https://example.com")
-        agent.navigate.back
-        agent.navigate.forward
+        user_agent.navigate.to("https://example.com")
+        user_agent.navigate.back
+        user_agent.navigate.forward
       end
     end
     ```
@@ -32,12 +32,12 @@
 
     ```ruby
     class DummyJob < Wayfarer::Base
-      route { to :index }
+      route.to :index
 
       def index
-        agent.visit("https://example.com")
-        agent.go_back
-        agent.go_forward
+        user_agent.visit("https://example.com")
+        user_agent.go_back
+        user_agent.go_forward
       end
     end
     ```

data/docs/cookbook/querying_html.md

@@ -6,7 +6,7 @@ See: [Nokogiri: Searching an HTML / XML Document](https://nokogiri.org/tutorials
 
     ```ruby
     class DummyJob < Wayfarer::Base
-      route { to :index }
+      route.to :index
 
       def index
         page.doc.css("html")
@@ -19,7 +19,7 @@ See: [Nokogiri: Searching an HTML / XML Document](https://nokogiri.org/tutorials
 
     ```ruby
     class DummyJob < Wayfarer::Base
-      route { to :index }
+      route.to :index
 
       def index
         browser.at_css("html")
@@ -32,7 +32,7 @@ See: [Nokogiri: Searching an HTML / XML Document](https://nokogiri.org/tutorials
 
     ```ruby
     class DummyJob < Wayfarer::Base
-      route { to :index }
+      route.to :index
 
       def index
         browser.find_elements(css: "html")

data/docs/cookbook/screenshots.md

@@ -6,7 +6,7 @@ Taking screenshots requires automating a browser.
 
     ```ruby
     class DummyJob < Wayfarer::Base
-      route { to :index }
+      route.to :index
 
       def index
         browser.screenshot(path: "screenshot.png")
@@ -18,7 +18,7 @@ Taking screenshots requires automating a browser.
 
     ```ruby
     class DummyJob < Wayfarer::Base
-      route { to :index }
+      route.to :index
 
       def index
         browser.save_screenshot("screenshot.png")

data/docs/guides/callbacks.md

@@ -1,145 +1,45 @@
 # Callbacks
 
-## Active Job callbacks
+Wayfarer supports a number of callbacks in addition to
+[ActiveJob callbacks](https://edgeguides.rubyonrails.org/active_job_basics.html#callbacks).
 
-Wayfarer naturally supports all of [Active Job's life cycle callbacks](https://edgeguides.rubyonrails.org/active_job_basics.html#callbacks).
+## Available callbacks
 
-## `before_fetch`
-
-Runs before a job fetches a page, either by making an HTTP request, or by
-navigating a browser to its task URL.
-
-```ruby
-class DummyJob < Wayfarer::Base
-  before_fetch :do_something
-
-  private
-
-  def do_something
-    # before the task.url is fetched
-  end
-end
-```
-
-## `before_action`
-
-Runs after a page was fetched, before an action method is called.
-
-```ruby
-class DummyJob < Wayfarer::Base
-  before_action :do_something
-
-  private
-
-  def do_something
-    # page is available at this point
-  end
-end
-```
+* `before_fetch`
+* `around_fetch`
+* `after_fetch`
+* `before_action`
+* `around_action`
+* `after_action`
+* `after_batch`
 
 ## `after_batch`
 
-Runs once the last job in a batch performed:
-
-```ruby
-class DummyJob < Wayfarer::Base
-  after_batch do
-    # All jobs in batch done
-  end
-end
-```
-
-Internally, a batch counter is in-/decremented on certain events. Once the
-counter reaches zero, `after_batch` callbacks runs in declaration order.
-
-The counter is incremented when within the batch:
+You can register `after_batch` callbacks that run when there are no more tasks
+to process in a batch. Wayfarer instruments job execution and in- or decrements
+an integer counter in Redis on certain events. When the counter reaches zero,
+the current job's `after_batch` callbacks run.
 
-* A job is enqueued.
+!!! info "`after_batch` callbacks fire at most once per batch."
 
-The counter is decremented when:
+## Conditional callbacks
 
-* A job succeeds.
-* A job errors due to an unhandled exception.
-* A job is discarded due to an exception.
-* A job errors and thereyby exhausts its maximum attempts.
-
-!!! attention "Batch callbacks can fail jobs"
-
-    If the last job's `after_batch` callbacks raises an exception, this can lead
-    to the job getting retried. If the exception raised by the callback is
-    unhandled or discarded, the callback never fully runs.
-
-## Callback options
-
-### Definition styles
-
-Callbacks can be registered either by supplying a block or a symbol identifying
-a callback instance method:
+You can make callbacks conditional with the `#!ruby :if` and `#!ruby :unless`
+keywords, for example to run a callback for some route `action` only:
 
 ```ruby
-class DummyJob < Wayfarer::Base
-  before_action do
-    # ...
-  end
+class DummyJob < ActiveJob::Base
+  include Wayfarer::Base
 
-  before_action :my_callback
+  route.host "example.com", to: :example
+  route.to :fallback
 
-  private
-
-  def my_callback
+  before_action unless: -> { action == :fallback } do
     # ...
   end
-end
-```
-
-### Conditionals
-
-Callbacks can be registered conditionally with the `:if` and `:unless` keywords:
-
-```ruby
-class DummyJob < Wayfarer::Base
-  before_fetch :my_callback, if: :my_condition
-
-  private
-
-  def my_callback
-  end
-
-  def my_condition
-  end
-end
-```
-
-Callbacks can be registered for certain action methods only with the `:only` and
-`:except` keywords:
-
-```ruby
-class DummyJob < Wayfarer::Base
-  before_fetch :do_something, only: :foo
-
-  before_fetch except: [:foo, :qux] do
-    # runs only before bar
-  end
 
-  def foo
-  end
-
-  def bar
-  end
+  # ...
 end
-
 ```
 
-### Early termination
-
-Callbacks that return `false` halt the callback chain:
-
-```ruby
-class DummyJob < Wayfarer::Base
-  before_action { false }
-
-  before_action do
-    # never runs
-  end
-end
-```
+You can also pass a symbol instead of a block to call an instance method.

data/docs/guides/cli.md

@@ -0,0 +1,71 @@
+# wayfarer
+
+The command-line interface to Wayfarer.
+
+## Usage
+
+```
+wayfarer [OPTIONS] [perform|enqueue|execute|route|tree]
+```
+
+See [Configuration](../reference/cli) for the respected environment variables.
+
+---
+
+## `wayfarer perform JOB URL`
+
+:   Performs `JOB` with `URL` in memory. The task is not sent to the message queue.
+    Staged jobs are ignored.
+
+    ##### Options
+
+    * `--mock-redis`: Use an in-memory implementation of Redis instead of
+      talking to an actual server.
+    * `--batch=BATCH`: The job's batch. By default, a UUID is generated.
+
+---
+
+## `wayfarer enqueue JOB URL`
+
+:   Enqueues a task for `JOB` with `URL` to the message queue.
+
+    ##### Options
+
+    * `--batch=BATCH`: The job's batch. By default, a UUID is generated.
+
+---
+
+## `wayfarer execute JOB URL`
+
+:   Execute `JOB` with `URL` with the in-memory
+    [Active Job Async adapter](https://api.rubyonrails.org/classes/ActiveJob/QueueAdapters/AsyncAdapter.html)
+    instead of writing the taks to an actual message queue. Blocks until the
+    batch has completed.
+
+    ##### Options
+
+    * `--mock-redis`: Use an in-memory implementation of Redis instead of
+      talking to an actual server.
+    * `--batch=BATCH`: Set the job's batch. By default, a UUID is generated.
+    * `--min-threads`: Minimum number of threads to use. Default: 1
+    * `--max-threads`: Maximum number of threads to use. Default: 1
+
+    !!! attention "Why are my jobs not getting retried with `wayfarer job execute`?"
+
+        You need to set the `wait: 0` option on `retry_on` in order for
+        `wayfarer job execute` to execute retries:
+
+        ```ruby
+        retry_on StandardError, attempts: 3, wait: 0
+        ```
+---
+
+## `wayfarer route JOB URL`
+
+:   Prints the result of invoking `JOB`'s router with `URL`.
+
+---
+
+## `wayfarer tree JOB URL`
+
+:   Visualises the routing tree result of invoking `JOB`'s router with `URL`.

data/docs/guides/configuration.md

@@ -1,39 +1,14 @@
-# Configuration
-
-Wayfarer can be configured in two ways:
-
-1. Using [environment variables](/reference/environment_variables)
-2. Using runtime configuration
-
-## Runtime configuration
-
-Wayfarer parses environment variables into a runtime configuration
-`Wayfarer::Config`. The configuration can then be altered or replaced via
-`Wayfarer.config`:
-
-```ruby
-# Which user agent to use to process tasks
-Wayfarer.config.network.agent = :http # or :ferrum, :selenium
+---
+hide:
+  - toc
+---
 
-# How many user agents to instantiate
-Wayfarer.config.network.pool_size = 3
-
-# How long an agent may be used while processing a task
-Wayfarer.config.network.pool_timeout = 5000
-
-# Ferrum options
-Wayfarer.config.ferrum.options = {}
-
-# Selenium driver to use
-Wayfarer.config.selenium.driver = :chrome
-
-# Selenium HTTP client read timeout
-Wayfarer.config.selenium.client_timeout = 10 # seconds
+# Configuration
 
-# Selenium options
-Wayfarer.config.selenium.options = { url: "http://chrome" }
+You can configure Wayfarer by assigning to `Wayfarer.config` which defaults to:
 
-# HTTP request headers (Selenium is unsupported)
-Wayfarer.config.network.http_headers = { "Field" => "Value" }
+```rb
+module Wayfarer
+--8<-- "lib/wayfarer.rb:48:96"
+end
 ```
-

data/docs/guides/development.md

@@ -0,0 +1,67 @@
+# Development
+
+## Release Procedure
+
+1. Ensure `Wayfarer::VERSION` was bumped appropriately.
+2. Ensure the version in wayfarer.gemspec matches.
+3. Open a release Pull Request develop -> master branch
+4. Merge the Pull Request
+5. Publish RubyGem and git tag as follows:
+
+```
+git checkout master
+git pull origin master --rebase
+bundle exec rake build
+gem push build/wayfarer-*.gem
+bundle exec rake clean
+git tag <VERSION>
+git push origin <VERSION>
+```
+
+## Conventions and guidelines
+
+* In source code, `url` refers to strings and `uri` refers to `Addressable::URI`
+* Avoid writing bash at all costs. Use Ruby instead
+
+## Design decisions and architecture
+
+### Navigate the web along URL patterns
+
+URLs are less prone to change than served markup.
+One reason for this is that changes to a URL's path can have negative
+consequences for its page ranking in search engines. Websites naturally implement
+architectural URL patterns like REST or expose surrogate keys.
+
+### Follow URLs verbatim as they appear in responses
+
+Normalized URLs are useful for deduplication, but URLs should be followed
+as they appear in responses. Navigating to normalized versions of URLs makes
+crawlers stick out from other user agents.
+
+### Tasks are version-less and don't persist metadata
+
+Tasks serialize to their URL and batch. No other data gets written to
+the message queue. There is also no need for versioning persisted tasks, since
+there will be never more to a task than URL and batch. All task metadata
+is ephemeral.
+
+### Why depend on Redis
+
+There are two core features that depend on Redis. First, per-batch acylicity is
+achieved by maintaining the set of processed URLs per batch in Redis.
+There's no option to follow links in a cyclic manner. Second, batch completion
+requires updating an integer value in Redis, and batch completion is a very
+useful feature, since most crawls should end eventually, and often you want to
+know when.
+
+### No configuration files
+
+Wayfarer can be configured through `Wayfarer.config` only, because `Wayfarer.config`
+may contain Ruby objects that don't de/serialize well, such as `Proc`s or `Set`s.
+
+### Features out of scope
+
+Wayfarer won't provide:
+
+* persistence or any sort of DOM data mapping abstractions
+* URL generation helpers

data/docs/guides/handlers.md

@@ -0,0 +1,60 @@
+# Handlers
+
+Handlers are like [jobs](/jobs) but they don't inherit from `ActiveJob::Base`
+which is why they can't affect the message queue directly themselves.
+Instead, jobs and handlers can route tasks to other handlers. Handlers
+themselves have routes, but they can be bypassed.
+
+## Handler capabilities
+
+Like jobs, handlers support:
+
+* URL routing
+* staging tasks with `#!ruby stage(*urls)`
+* jobs can access the `user_agent` that retrieved the `page`
+* ad-hoc HTTP requests with `#!ruby fetch(url)`
+* callbacks, but only a subset of job callbacks
+* Content-Type filtering
+
+```ruby
+class ExampleHandler
+  include Wayfarer::Handler
+
+  route.to: :index
+
+  def index
+    task # => #<Wayfarer::Task>
+    page # => #<Wayfarer::Page>
+    user_agent # => Browser or HTTP client
+  end
+end
+
+class DummyJob < ActiveJob::Base
+  include Wayfarer::Base
+
+  route.host "example.com", to: ExampleHandler
+end
+```
+
+You can also bypass a handler's router and route directly to an instance
+method:
+
+```ruby
+class DummyJob < ActiveJob::Base
+  include Wayfarer::Base
+
+  route.host "example.com", to: [ExampleHandler, :index]
+end
+
+class ExampleHandler
+  include Wayfarer::Handler
+
+  def index
+    task # => #<Wayfarer::Task>
+    page # => #<Wayfarer::Page>
+    user_agent # => Browser or HTTP client
+  end
+end
+```
+
+!!! `before_action` callbacks

data/docs/guides/index.md

@@ -0,0 +1 @@
+hello