wayfarer 0.0.3

data/docs/guides/error_handling.md

@@ -0,0 +1,46 @@
+---
+layout: default
+title: Error handling
+---
+
+# Error handling
+By default, all exceptions raised within actions are swallowed and only their stacktraces printed to stderr. This behaviour can be changed with two configuration keys (see [Configuration]()):
+
+1. `print_stacktraces`: Whether to print stacktraces (default: `true`)
+2. `reraise_exceptions`: Whether to crash when encountering unhandled exceptions (default: `false`)
+
+Here’s an example to illustrate the default behaviour:
+
+{% highlight ruby %}
+class DummyJob < Wayfarer::Job
+  def example
+    # Makes this instance fail, but processing goes on
+    # Prints the stacktrace to stderr
+    fail "It's okay, life goes on"
+  end
+end
+{% endhighlight %}
+
+The following reraises all exceptions, stops processing and returns with a non-zero exit code:
+
+{% highlight ruby %}
+class DummyJob < Wayfarer::Job
+  config.reraise_exceptions = true
+
+  def example
+    fail "This makes the exception bubble up"
+  end
+end
+{% endhighlight %}
+
+And if you don’t want to be bothered with exceptions at all:
+
+{% highlight ruby %}
+class DummyJob < Wayfarer::Job
+  config.print_stacktraces = false
+
+  def example
+    fail "No one will know about this ..."
+  end
+end
+{% endhighlight %}

data/docs/guides/frontiers.md

@@ -0,0 +1,93 @@
+---
+layout: default
+title: Frontiers
+---
+
+# Frontiers
+
+Frontiers keep track of three sets of URIs:
+
+* Current URIs that are being processed
+* Staged URIs that might be processed in the next cycle
+* Cached URIs that have been processed
+
+All frontiers expose the same behaviour.
+
+<pre class="illustration">
+┌──────────────────────────────────────────────────────────┐
+│                          STAGED                          │
+│          {https://alpha.com, https://beta.com}           │
+└──────────────────────────────────────────────────────────┘
+┌──────────────────────────────────────────────────────────┐
+│                         CURRENT                          │
+│                   {https://gamma.com}                    │
+└──────────────────────────────────────────────────────────┘
+┌──────────────────────────────────────────────────────────┐
+│                          CACHED                          │
+│                    {https://beta.com}                    │
+└──────────────────────────────────────────────────────────┘
+                             │
+                           Cycle
+                             │
+                             ▼
+┌──────────────────────────────────────────────────────────┐
+│                         STAGED'                          │
+│                          {...}                           │
+└──────────────────────────────────────────────────────────┘
+┌──────────────────────────────────────────────────────────┐
+│                CURRENT' = STAGED \ CACHED                │
+│                   {https://alpha.com}                    │
+└──────────────────────────────────────────────────────────┘
+┌──────────────────────────────────────────────────────────┐
+│                CACHED' = CACHED ∪ CURRENT                │
+│          {https://beta.com, https://gamma.com}           │
+└──────────────────────────────────────────────────────────┘
+</pre>
+
+## Available frontiers
+Currently, there are 5 frontiers available:
+
+2. `:memory` (default): Uses sets from the standard lib.
+4. `:redis`: Uses Redis sets.
+3. `:memory_bloom`: Uses a [Bloom filter](https://github.com/igrigorik/bloomfilter-rb).
+5. `:redis_bloom`: Uses a Redis-backed Bloom filter.
+1. `:memory_trie`: Uses a [trie](https://github.com/tyler/trie) and sets.
+
+| Frontier | MRI support | JRuby support |
+| --- | --- |
+| `:memory` | Yes | Yes
+| `:redis` | Yes | Yes
+| `:memory_bloom` | Yes | No
+| `:redis_bloom` | Yes | No
+| `:memory_trie` | Yes | No
+
+## Setting the frontier
+
+Set the `:frontier` configuration key:
+
+{% highlight ruby %}
+class DummyJob < Wayfarer::Job
+  config.frontier = :foobar
+end
+{% endhighlight %}
+
+### Using a Redis frontier
+
+Set the `:redis_opts` and `:frontier` configuration keys:
+
+{% highlight ruby %}
+class DummyJob < Wayfarer::Job
+  config.redis_opts = { port: 4242 }
+  config.frontier = :redis
+end
+{% endhighlight %}
+
+### Setting bloomfilter parameters
+
+Set the `:bloomfilter_opts` configuration key:
+
+{% highlight ruby %}
+class DummyJob < Wayfarer::Job
+  config.bloomfilter_opts = { ... }
+end
+{% endhighlight %}

data/docs/guides/halting.md

@@ -0,0 +1,23 @@
+---
+layout: default
+title: Halting
+---
+
+# Halting
+Processing can be stopped by calling `#halt` within actions.
+
+`#halt` does not return immediately. Instead, it sets a halting flag internally, and once the action returns, all threads will stop instead of processing further URIs.
+
+Job instances run in separate threads. When a job signals that it wants to halt, all other threads will finish their current work, but will not process any further URIs. All instances have the chance to get their current work done.
+
+{% highlight ruby %}
+class DummyJob < Wayfarer::Job
+  def example
+    halt
+    puts "This will be printed!"
+
+    return halt
+    puts "This will not be printed!"
+  end
+end
+{% endhighlight %}

data/docs/guides/job_queues.md

@@ -0,0 +1,26 @@
+---
+layout: default
+title: Locals
+---
+
+# Job queues
+
+Thanks to [ActiveJob](http://edgeguides.rubyonrails.org/active_job_basics.html), jobs can be enqueued with various backends, e.g. Sidekiq or Resque:
+
+{% highlight ruby %}
+class DummyJob < Wayfarer::Job
+  # Overrides ActiveJob's global setting
+  self.queue_adapter = :resque
+
+  # Identifier for enqueued jobs
+  queue_as :dummy_job
+
+  # Alternatively, pass a block
+  queue_as do
+    [:first, :second].sample
+  end
+end
+
+# Alternatively, set the queue explicitly on call:
+DummyJob.set(queue: :something_else).perform_later(*uris)
+{% endhighlight %}

data/docs/guides/locals.md

@@ -0,0 +1,36 @@
+---
+layout: default
+title: Locals
+---
+
+# Locals
+
+Locals are Wayfarer's replacement for job instance variables. Both `let` and `let!` declare variables that are accessible within [callbacks]({{base}}/callbacks.html) and actions.
+
+Even though you might recognise them from RSpec, they have differing semantics: Values in `let` blocks will be replaced with thread-safe counterparts once the job is run. `let!` skips this. Both evaluate their block immediately.
+
+| Standard lib | Counterpart |
+| --- | --- |
+| Booleans | [`Concurrent::AtomicBoolean`](http://ruby-concurrency.github.io/concurrent-ruby/Concurrent/AtomicBoolean.html) |
+| `Fixnum` | [`Concurrent::AtomicFixnum`](http://ruby-concurrency.github.io/concurrent-ruby/Concurrent/AtomicFixnum.html) |
+| `Hash` | [`Concurrent::Hash`](http://ruby-concurrency.github.io/concurrent-ruby/Concurrent/Hash.html) |
+| `Array` | [`Concurrent::Array`](http://ruby-concurrency.github.io/concurrent-ruby/Concurrent/Array.html) |
+| Everything else | Untouched |
+
+{% highlight ruby %}
+class DummyJob < Wayfarer::Job
+  let(:values) { [1, 2, 3] }
+
+  before_crawl do
+    values.reverse!
+  end
+
+  after_crawl do
+    values # => [3, 2, 1, 0]
+  end
+
+  def some_action
+    values << 0
+  end
+end
+{% endhighlight %}

data/docs/guides/logging.md

@@ -0,0 +1,22 @@
+---
+layout: default
+title: Logging
+---
+
+# Logging
+
+{% highlight ruby %}
+# Global configuration serves as the template
+Wayfarer.logger.level = :fatal
+
+class DummyJob < Wayfarer::Job
+  # Jobs can tweak their logger
+  config.logger.level = :warn
+  config.logger.progname = "dummy-job"
+
+  def example
+    logger.info "No"
+    logger.warn "Yes"
+  end
+end
+{% endhighlight %}

data/docs/guides/page_objects.md

@@ -0,0 +1,67 @@
+---
+layout: default
+title: Page objects
+---
+
+# `Page` objects
+
+Retrieved pages are represented by `Page` objects and made accessible by `#page` within actions. `Page`s support the same set of features regardless of the HTTP adapter in use.
+
+<aside class="note">
+HTTP response headers and status codes are not supported by Selenium WebDrivers. Wayfarer emulates both by having the WebDriver fire an AJAX request to the current page and extracting them from the response. Clearly this is a hack, but it might even work for you. See <a href="https://github.com/bauerd/selenium-emulated_features">selenium-emulated_features</a>.
+</aside>
+
+<aside class="note">
+Even after having followed redirects, <code>Page#uri</code> always returns the URI that originally initiated the redirects. This behaviour stems from redirects being opaque to WebDrivers.
+</aside>
+
+A `Page` brings to the table all you'd wish for when doing web scraping:
+
+* [Nokogiri](http://www.nokogiri.org) parses HTML/XML
+* [Oj](https://github.com/ohler55/oj) or the standard lib parses JSON
+* __When running on MRI__, [Pismo](https://github.com/peterc/pismo) lets you access metadata, e.g. keywords, author, a summary, … No overhead if you don't use it!
+
+Let's see it in action:
+
+{% highlight ruby %}
+class DummyJob < Wayfarer::Job
+  # ...
+
+  def example
+    page # => #<Wayfarer::Page:...>
+
+    page.uri # => #<URI::...>
+    page.status_code # => Fixnum
+    page.body # => String
+    page.headers # => Hash
+
+    page.doc # => #<Nokogiri::HTML::Document:...> (HTML/XML) or Hash (JSON)
+                  # Also accessible as just `doc`
+
+    page.links # => [URI]
+    page.stylesheets # => [URI]
+    page.javascripts # => [URI]
+    page.images # => [URI]
+
+    # All previous four methods accept arbitrary many CSS selectors
+    page.links ".my-target", ".my-other-target"
+
+    # THESE ARE NOT SUPPORTED ON JRUBY!
+    # On MRI, the following methods get forwarded to a Pismo::Document
+    # See https://github.com/peterc/pismo
+    page.title
+    page.titles
+    page.author
+    page.lede
+    page.keywords
+    page.sentences(qty)
+    page.body
+    page.html_body
+    page.feed
+    page.feeds
+    page.favicon
+    page.description
+    page.datetime
+  end
+end
+{% endhighlight %}

data/docs/guides/peeking.md

@@ -0,0 +1,46 @@
+---
+layout: default
+title: Peeking
+---
+
+# Peeking
+Peeking allows bypassing the [frontier](frontiers.html) in an ad-hoc manner. Use Ruby's `yield` keyword to immediately retrieve and dispatch a URI from within actions. Control gets handed off to the action matching the yielded URI, if any.
+
+A matching route for the yielded URI is still required. If the yielded URI matches no route or raises an exception, `yield` returns `nil`.
+
+<aside class="note">
+The action that gets the URI dispatched to <strong>will</strong> get assigned another HTTP adapter! HTTP adapters are never shared across actions, i.e. if you're using the Selenium HTTP adapter, the peeked URI gets retrieved by a different browser process.
+</aside>
+
+{% highlight ruby %}
+class DummyJob < Wayfarer::Job
+  route.uri "https://example.com", to: :foo
+  route.uri "https://w3c.org", to: :bar
+
+  def foo
+    w3c_page = yield "https://w3c.org"
+  end
+
+  def bar
+    page
+  end
+end
+{% endhighlight %}
+
+__Recursive peeking does not work__, or else peeking might result in an infinite loop. The following does terminate:
+
+{% highlight ruby %}
+class DummyJob < Wayfarer::Job
+  route.uri "https://example.com", to: :foo
+  route.uri "https://w3c.org", to: :bar
+
+  def foo
+    w3c_page = yield "https://w3c.org"
+  end
+
+  def bar
+    # Silently ignored, assigns nil
+    example_page = yield "https://example.com"
+  end
+end
+{% endhighlight %}

data/docs/guides/selenium_capybara.md

@@ -0,0 +1,100 @@
+---
+layout: default
+title: Selenium & Capybara
+---
+
+# Selenium & Capybara
+
+[Selenium](http://www.seleniumhq.org) is a browser automation framework. [Capybara](https://github.com/teamcapybara/capybara) is an acceptance testing framework that puts an expressive DSL on Selenium's WebDrivers. Both are first-class citizens in Wayfarer and the best tools for automating browsers.
+
+## Selenium WebDrivers
+
+WebDrivers let you remote-control browsers, e.g. Firefox, Chrome, Safari and PhantomJS.
+
+Depending on what browser you want to automate, go install and run the corresponding driver first. For installation instructions, see the project websites:
+
+* Firefox: [geckodriver](https://github.com/mozilla/geckodriver)
+* Chrome: [chromedriver](https://sites.google.com/a/chromium.org/chromedriver)
+* Safari: [SafariDriver](https://github.com/SeleniumHQ/selenium/wiki/SafariDriver)
+* PhantomJS ships with an embedded driver.
+
+Other browsers are supported, too. For an exhaustive list, see the "Third Party Drivers, Bindings, and Plugins" section on the [Selenium downloads page](http://www.seleniumhq.org/download).
+
+If you want to run browser processes on a central server, consider using [Selenium Grid](http://www.seleniumhq.org/projects/grid).
+
+Wayfarer hides the details of managing Ruby driver objects from you. In order to use Selenium, set the `http_adapter` configuration key to `:selenium`. Pass in the desired browser and arguments by setting the `selenium_argv` key. The number of browser processes can be controlled with the `connection_count` key.
+
+{% highlight ruby %}
+class DummyJob < Wayfarer::Job
+  config do |c|
+    # Use 4 Firefox processes
+    c.http_adapter = :selenium
+    c.selenium_argv = [:firefox]
+    c.connection_count = 4
+
+    # Chrome
+    # c.selenium_argv = [:chrome]
+
+    # Safari
+    # c.selenium_argv = [:safari]
+
+    # PhantomJS
+    # c.selenium_argv = [:phantomjs]
+
+    # Selenium Grid
+    # c.selenium_argv = [
+    #   :remote,
+    #   url: "http://localhost:4444/wd/hub",
+    #   desired_capabilities: :firefox
+    # ]
+  end
+end
+{% endhighlight %}
+
+<aside class="note">
+In order to avoid redirect loops, the <code>:net_http</code> adapter supports the <code>max_http_redirects</code> configuration key. Because redirects are opaque to WebDrivers, the configuration key does not apply to the Selenium adapter. See <a href="configuration.html">Configuration</a>.
+</aside>
+
+### Accessing the WebDriver
+
+Within actions, `#driver` returns a [`Selenium::WebDriver::Driver`](http://www.rubydoc.info/gems/selenium-webdriver/Selenium/WebDriver/Driver):
+
+{% highlight ruby %}
+class DummyJob < Wayfarer::Job
+  config do |c|
+    c.http_adapter = :selenium
+    c.selenium_argv = [:firefox]
+  end
+
+  draw uri: "https://example.com"
+  def example
+    driver # => #<Selenium::WebDriver::Driver:...>
+  end
+end
+{% endhighlight %}
+
+<aside class="note">
+What you do with a WebDriver is opaque to Wayfarer. If you handle navigation yourself with a WebDriver and bypass the <a href="/guides/frontiers.html">frontier</a>, Wayfarer cannot ensure you don't visit URIs twice.
+</aside>
+
+## Capybara
+
+When using the `:selenium` HTTP adapter, `#browser` returns a [`Capybara::Selenium::Driver`](http://www.rubydoc.info/github/jnicklas/capybara/Capybara/Selenium/Driver) within actions:
+
+{% highlight ruby %}
+class DummyJob < Wayfarer::Job
+  config do |c|
+    c.http_adapter = :selenium
+    c.selenium_argv = [:firefox]
+  end
+
+  draw uri: "https://example.com"
+  def example
+    browser # => #<Capybara::Selenium::Driver:...>
+  end
+end
+{% endhighlight %}
+
+<aside class="note">
+What you do with a WebDriver is opaque to Wayfarer. If you handle navigation yourself with a WebDriver and bypass the <a href="/guides/frontiers.html">frontier</a>, Wayfarer cannot ensure you don't visit URIs twice.
+</aside>