wayfarer-jruby 0.0.1

data/docs/guides/cli.md

@@ -0,0 +1,52 @@
+---
+layout: default
+title: CLI
+---
+
+# Command-line interface
+Wayfarer ships with a small executable, `wayfarer`.
+
+Job classes are loaded by naming convention, e.g. if you pass `./directory/foo_bar.rb` as the `FILE` parameter, that file is expected to define the class `FooBar`. You can leave off the `.rb` extension.
+
+## `% wayfarer route FILE URI`
+Loads the job defined in `FILE`, and prints the first matching route for `URI`.
+
+## `% wayfarer enqueue FILE URI`
+Loads and enqueues the job in `FILE`, starting from `URI`.
+
+* `--log_level LEVEL`
+  Option. Which log messages to print.
+
+  * Default: `info`
+  * Recognized values: `unknown`, `debug`, `error`, `fatal`, `info`, `warn`
+
+* `--queue_adapter ADAPTER`
+  Option. Which ActiveJob queue adapter to use (e.g. `sidekiq`, `resque`).
+  * Recognized values: strings, see [documentation](http://api.rubyonrails.org/)
+
+* `--wait VALUE`
+  Option. Point of time when the enqueued job should be run.
+
+    1. If the value can be converted to an integer, it represents the seconds from now.
+    2. If the value can be parsed by `Time::parse`, the job gets scheduled at that point in time.
+    3. If the value is a human-readable time string that [Chronic](https://github.com/mojombo/chronic) can make sense of, the job is scheduled at that point in time.
+
+__Examples:__
+
+60 seconds from now:
+
+```
+% wayfarer enqueue ./foo_bar http://google.com --wait 60
+```
+
+6pm, today:
+
+```
+% wayfarer enqueue ./foo_bar http://google.com --wait 18:00
+```
+
+Tomorrow:
+
+```
+% wayfarer enqueue ./foo_bar http://google.com --wait tomorrow
+```

data/docs/guides/configuration.md

@@ -0,0 +1,184 @@
+---
+layout: default
+title: Configuration
+---
+
+# Configuration
+
+All job classes base their configuration off the global one.
+
+{% highlight ruby %}
+# Setting a key globally applies to all jobs ...
+Wayfarer.config.key = :value
+
+class DummyJob < Wayfarer::Job
+  # ... unless a job overrides it
+  config.key = :other_value
+end
+
+class DummyJob < Wayfarer::Job
+  # Have it yielded
+  config { |c| c.key = :other_value }
+end
+{% endhighlight %}
+
+## Recognized keys and values
+
+### `print_stacktraces`
+  * Default: `true`
+  * Recognized values: Booleans
+
+Whether to print stacktraces when encounterting unhandled exceptions in job actions. See [Error handling]({{base}}/guides/error_handling.html).
+
+---
+
+### `reraise_exceptions`
+
+* Default: `false`
+* Recognized values: Booleans
+
+Whether to crash when encountering unhandled exceptions in job actions. See [Error handling]({{base}}/guides/error_handling.html).
+
+---
+
+### `allow_circulation`
+
+* Default: `false`
+* Recognized values: Booleans
+
+Whether URIs may be visited twice.
+
+<aside class="note">
+Allowing circulation might cause your jobs to not terminate.
+</aside>
+
+---
+
+### `normalize_uris`
+
+* Default: `true`
+* Recognized values: Booleans
+
+Whether to strip fragments, reorder query keys, etc. when staging and caching URIs. Customizable with the `:normalize_uri_options` key. See [normalize_url](https://github.com/rwz/normalize_url).
+
+---
+
+### `normalize_uri_options`
+
+* Default: `{}`
+* Recognized values: See [normalize_url](https://github.com/rwz/normalize_url).
+
+---
+
+### `frontier`
+* Default: `:memory`
+* Recognized values: See [(Redis) frontiers](frontiers.html).
+
+Which frontier to use.
+
+<aside class="note">
+Bloom filters may yield false positives. See the <a href="https://en.wikipedia.org/wiki/Bloom_filter">Wikipedia article</a>.
+</aside>
+
+---
+
+### `connection_count`
+
+* Default: `4`
+* Recognized values: Integers
+
+How many threads and HTTP adapters to use (1:1 correspondence).
+
+---
+
+### `http_adapter`
+
+* Default: `:net_http`
+* Recognized values: `:net_http`, `:selenium`
+
+Which HTTP adapter to use. See [Selenium & Capybara](selenium_capybara.html).
+
+---
+
+### `connection_timeout`
+
+* Default: `Float::INFINITY`
+* Recognized values: Floats
+
+Time in seconds that a job instance may hold an HTTP adapter. Instances that exceed this time limit raise an exception.
+
+---
+
+### `max_http_redirects`
+
+* Default: `3`
+* Recognized values: Integers
+
+How many 3xx redirects to follow.
+
+<aside class="note">
+Has no effect when using the <code>:selenium</code> HTTP adapter.
+</aside>
+
+---
+
+### `selenium_argv`
+
+* Default: `[:firefox]`
+* Recognized values: See [Selenium & Capybara](selenium_capybara.html)
+
+Argument vector passed to [`Selenium::WebDriver::Driver::for`](http://www.rubydoc.info/gems/selenium-webdriver/Selenium/WebDriver/Driver#for-class_method).
+
+---
+
+### `redis_opts`
+
+* Default: `{ host: "localhost", port: 6379 }`
+* Recognized values: [See documentation](http://www.rubydoc.info/github/redis/redis-rb/Redis%3Ainitialize)
+
+Options passed to [`Redis#initialize`](http://www.rubydoc.info/github/redis/redis-rb/Redis%3Ainitialize).
+
+---
+
+### `bloomfilter_opts`
+
+* Default:
+```
+{
+  size: 100,
+  hashes: 2,
+  seed: 1,
+  bucket: 3,
+  raise: false
+}
+```
+* Recognized values:
+  * `size`: Integers; number of buckets in a bloom filter
+  * `hashes`: Integers; number of hash functions
+  * `seed`: Integers; seed of hash functions
+  * `bucket`: Integers; number of bits in a bloom filter bucket
+  * `raise`: Booleans; whether to raise on bucket overflow
+
+Options for [bloomfilter-rb](https://github.com/igrigorik/bloomfilter-rb).
+
+---
+
+### `window_size`
+
+* Default: `[1024, 768]`
+* Recognized values: `[Integer, Integer]`
+
+Dimensions of browser windows.
+
+<aside class="note">
+Only has an effect when using the <code>:selenium</code> HTTP adapter.
+</aside>
+
+---
+
+### `mustermann_type`
+
+* Default: `:sinatra`
+* Recognized values: [See documentation](https://github.com/sinatra/mustermann)
+
+Which [Mustermann](https://github.com/sinatra/mustermann) pattern type to use.

data/docs/guides/error_handling.md

@@ -0,0 +1,46 @@
+---
+layout: default
+title: Error handling
+---
+
+# Error handling
+By default, all exceptions raised within actions are swallowed and only their stacktraces printed to stderr. This behaviour can be changed with two configuration keys (see [Configuration]()):
+
+1. `print_stacktraces`: Whether to print stacktraces (default: `true`)
+2. `reraise_exceptions`: Whether to crash when encountering unhandled exceptions (default: `false`)
+
+Here’s an example to illustrate the default behaviour:
+
+{% highlight ruby %}
+class DummyJob < Wayfarer::Job
+  def example
+    # Makes this instance fail, but processing goes on
+    # Prints the stacktrace to stderr
+    fail "It's okay, life goes on"
+  end
+end
+{% endhighlight %}
+
+The following reraises all exceptions, stops processing and returns with a non-zero exit code:
+
+{% highlight ruby %}
+class DummyJob < Wayfarer::Job
+  config.reraise_exceptions = true
+
+  def example
+    fail "This makes the exception bubble up"
+  end
+end
+{% endhighlight %}
+
+And if you don’t want to be bothered with exceptions at all:
+
+{% highlight ruby %}
+class DummyJob < Wayfarer::Job
+  config.print_stacktraces = false
+
+  def example
+    fail "No one will know about this ..."
+  end
+end
+{% endhighlight %}

data/docs/guides/frontiers.md

@@ -0,0 +1,93 @@
+---
+layout: default
+title: Frontiers
+---
+
+# Frontiers
+
+Frontiers keep track of three sets of URIs:
+
+* Current URIs that are being processed
+* Staged URIs that might be processed in the next cycle
+* Cached URIs that have been processed
+
+All frontiers expose the same behaviour.
+
+<pre class="illustration">
+┌──────────────────────────────────────────────────────────┐
+│                          STAGED                          │
+│          {https://alpha.com, https://beta.com}           │
+└──────────────────────────────────────────────────────────┘
+┌──────────────────────────────────────────────────────────┐
+│                         CURRENT                          │
+│                   {https://gamma.com}                    │
+└──────────────────────────────────────────────────────────┘
+┌──────────────────────────────────────────────────────────┐
+│                          CACHED                          │
+│                    {https://beta.com}                    │
+└──────────────────────────────────────────────────────────┘
+                             │
+                           Cycle
+                             │
+                             ▼
+┌──────────────────────────────────────────────────────────┐
+│                         STAGED'                          │
+│                          {...}                           │
+└──────────────────────────────────────────────────────────┘
+┌──────────────────────────────────────────────────────────┐
+│                CURRENT' = STAGED \ CACHED                │
+│                   {https://alpha.com}                    │
+└──────────────────────────────────────────────────────────┘
+┌──────────────────────────────────────────────────────────┐
+│                CACHED' = CACHED ∪ CURRENT                │
+│          {https://beta.com, https://gamma.com}           │
+└──────────────────────────────────────────────────────────┘
+</pre>
+
+## Available frontiers
+Currently, there are 5 frontiers available:
+
+2. `:memory` (default): Uses sets from the standard lib.
+4. `:redis`: Uses Redis sets.
+3. `:memory_bloom`: Uses a [Bloom filter](https://github.com/igrigorik/bloomfilter-rb).
+5. `:redis_bloom`: Uses a Redis-backed Bloom filter.
+1. `:memory_trie`: Uses a [trie](https://github.com/tyler/trie) and sets.
+
+| Frontier | MRI support | JRuby support |
+| --- | --- |
+| `:memory` | Yes | Yes
+| `:redis` | Yes | Yes
+| `:memory_bloom` | Yes | No
+| `:redis_bloom` | Yes | No
+| `:memory_trie` | Yes | No
+
+## Setting the frontier
+
+Set the `:frontier` configuration key:
+
+{% highlight ruby %}
+class DummyJob < Wayfarer::Job
+  config.frontier = :foobar
+end
+{% endhighlight %}
+
+### Using a Redis frontier
+
+Set the `:redis_opts` and `:frontier` configuration keys:
+
+{% highlight ruby %}
+class DummyJob < Wayfarer::Job
+  config.redis_opts = { port: 4242 }
+  config.frontier = :redis
+end
+{% endhighlight %}
+
+### Setting bloomfilter parameters
+
+Set the `:bloomfilter_opts` configuration key:
+
+{% highlight ruby %}
+class DummyJob < Wayfarer::Job
+  config.bloomfilter_opts = { ... }
+end
+{% endhighlight %}

data/docs/guides/halting.md

@@ -0,0 +1,23 @@
+---
+layout: default
+title: Halting
+---
+
+# Halting
+Processing can be stopped by calling `#halt` within actions.
+
+`#halt` does not return immediately. Instead, it sets a halting flag internally, and once the action returns, all threads will stop instead of processing further URIs.
+
+Job instances run in separate threads. When a job signals that it wants to halt, all other threads will finish their current work, but will not process any further URIs. All instances have the chance to get their current work done.
+
+{% highlight ruby %}
+class DummyJob < Wayfarer::Job
+  def example
+    halt
+    puts "This will be printed!"
+
+    return halt
+    puts "This will not be printed!"
+  end
+end
+{% endhighlight %}

data/docs/guides/job_queues.md

@@ -0,0 +1,26 @@
+---
+layout: default
+title: Locals
+---
+
+# Job queues
+
+Thanks to [ActiveJob](http://edgeguides.rubyonrails.org/active_job_basics.html), jobs can be enqueued with various backends, e.g. Sidekiq or Resque:
+
+{% highlight ruby %}
+class DummyJob < Wayfarer::Job
+  # Overrides ActiveJob's global setting
+  self.queue_adapter = :resque
+
+  # Identifier for enqueued jobs
+  queue_as :dummy_job
+
+  # Alternatively, pass a block
+  queue_as do
+    [:first, :second].sample
+  end
+end
+
+# Alternatively, set the queue explicitly on call:
+DummyJob.set(queue: :something_else).perform_later(*uris)
+{% endhighlight %}

data/docs/guides/locals.md

@@ -0,0 +1,36 @@
+---
+layout: default
+title: Locals
+---
+
+# Locals
+
+Locals are Wayfarer's replacement for job instance variables. Both `let` and `let!` declare variables that are accessible within [callbacks]({{base}}/callbacks.html) and actions.
+
+Even though you might recognise them from RSpec, they have differing semantics: Values in `let` blocks will be replaced with thread-safe counterparts once the job is run. `let!` skips this. Both evaluate their block immediately.
+
+| Standard lib | Counterpart |
+| --- | --- |
+| Booleans | [`Concurrent::AtomicBoolean`](http://ruby-concurrency.github.io/concurrent-ruby/Concurrent/AtomicBoolean.html) |
+| `Fixnum` | [`Concurrent::AtomicFixnum`](http://ruby-concurrency.github.io/concurrent-ruby/Concurrent/AtomicFixnum.html) |
+| `Hash` | [`Concurrent::Hash`](http://ruby-concurrency.github.io/concurrent-ruby/Concurrent/Hash.html) |
+| `Array` | [`Concurrent::Array`](http://ruby-concurrency.github.io/concurrent-ruby/Concurrent/Array.html) |
+| Everything else | Untouched |
+
+{% highlight ruby %}
+class DummyJob < Wayfarer::Job
+  let(:values) { [1, 2, 3] }
+
+  before_crawl do
+    values.reverse!
+  end
+
+  after_crawl do
+    values # => [3, 2, 1, 0]
+  end
+
+  def some_action
+    values << 0
+  end
+end
+{% endhighlight %}