wayfarer-jruby 0.0.1

data/docs/index.md

@@ -0,0 +1,82 @@
+---
+layout: default
+title: Versatile web crawling with (J)Ruby
+---
+
+# Versatile web crawling with (J)Ruby
+
+Wayfarer is the swiss army knife for web crawling.
+
+MRI:
+```
+% [sudo] gem install wayfarer
+```
+
+JRuby:
+```
+% [sudo] gem install wayfarer-jruby
+```
+
+If you …
+
+* __Need to crawl page graphs breadth-first__
+* __Need to extract whatever data__
+* Do it multi-threaded
+* Integrate with Rails seamlessly
+* Want to automate a web browser
+* Need to execute arbitrary JavaScript
+* Need URI normalization
+* Need to take screenshots
+* Want to use a job queue and make work happen later
+* Want in-memory and Redis-backed frontiers, tries and Bloom filters
+
+… then you might like Wayfarer!
+
+## What it looks like
+
+Say you want to …
+
+* Automate Google Chrome
+* Start off with a random Wikipedia article
+* Follow all links until you find a page with the word "Foobar"
+* Take a screenshot of the page containing "Foobar"
+* Extract keywords from every page you encounter
+* Use 4 threads and Chrome processes to do so
+
+This amounts to 16 lines of code with Wayfarer:
+
+{% highlight ruby %}
+require "wayfarer"
+
+class FindFoobarOnWikipedia < Wayfarer::Job
+  config.http_adapter = :selenium
+  config.selenium_argv = [:chrome]
+  config.connection_count = 4
+
+  let(:keywords) { [] }
+
+  route.host "en.wikipedia.org", to: :article
+
+  def article
+    if page.body =~ /Foobar/
+      driver.save_screenshot("/tmp/foobar.png")
+      return halt
+    end
+
+    keywords << page.keywords
+    stage page.links
+  end
+end
+
+FindFoobarOnWikipedia.perform_now("https://en.wikipedia.org/wiki/Special:Random")
+{% endhighlight %}
+
+Wayfarer integrates with [ActiveJob]() and supports your favorite job queue out of the box. Your job is ready to be enqueued:
+{% highlight ruby %}
+FindFoobarOnWikipedia.perform_later("https://en.wikipedia.org/wiki/Special:Random")
+{% endhighlight %}
+
+### Where to go from here
+
+* [The tutorial]() shows how to collect all open issues from a GitHub repository
+* Read the [API documentation]()

data/docs/js/navigation.js

@@ -0,0 +1,11 @@
+document.addEventListener("DOMContentLoaded", function() {
+  var links = document.querySelectorAll(".navigation__link");
+
+  for (i = 0; i < links.length; i++) {
+    var link = links[i];
+
+    if (link.pathname === window.location.pathname) {
+      link.classList.add("navigation__link--active");
+    }
+  }
+});

data/docs/misc/contributing.md

@@ -0,0 +1,20 @@
+---
+layout: default
+title: Contributing
+---
+
+# Contributing
+
+1. Fork the repository
+2. Ensure the development dependencies are installed:
+    `% bundle install --with development`
+2. Make changes
+3. Ensure your (new?) tests pass:
+  `% bundle exec rake test`
+4. Autocorrect RubuCop offenses:
+  `% bundle exec rake rubocop:auto_correct`
+5. Fix remaining offenses or have a good excuse not to:
+  `% bundle exec rake rubocop`
+6. Write commit messages at least not worse than mine
+7. Open a pull request on GitHub
+8. Thank you

data/docs/misc/testing.md

@@ -0,0 +1,11 @@
+---
+layout: default
+title: Testing
+---
+
+# Testing
+
+Tests run on MRI 2.3.1 and JRuby 9.1.6.0.
+
+* Run `rake -T` to list all available (test) tasks.
+* When running tests, a HTTP server binds to port 9876 for integration-level tests.

data/docs/recipes/authentication.md

@@ -0,0 +1,23 @@
+---
+layout: default
+title: Authentication
+---
+
+# Authentication
+
+Authentication is best handled with the `setup_adapter` callback See [Callbacks]({{base}}/guides/callbacks.html).
+
+{% highlight ruby %}
+class DummyJob < Wayfarer::Job
+  config.http_adapter = :selenium
+
+  setup_adapter do |_, _, browser|
+    browser.visit("https://foo.com/login")
+
+    browser.fill_in("E-mail", with: "foo@bar.com")
+    browser.fill_in("Password", with: "password")
+
+    browser.click_button("Log in")
+  end
+end
+{% endhighlight %}

data/docs/recipes/csv.md

@@ -0,0 +1,29 @@
+---
+layout: default
+title: CSV output
+---
+
+# CSV output
+
+See:
+
+* [Locals]({{base}}/guides/locals.html)
+* [Callbacks]({{base}}/guides/callbacks.html)
+
+{% highlight ruby %}
+require "csv" # from Ruby's standard lib
+
+class DummyJob < Wayfarer::Job
+  let(:records) { [] }
+
+  after_crawl do
+    CSV.open("output.csv", "w") do |csv|
+      records.each { |r| csv << [r[:id], r[:name]] }
+    end
+  end
+
+  def detail
+    records << { id: ..., name: ...  }
+  end
+end
+{% endhighlight %}

data/docs/recipes/javascript.md

@@ -0,0 +1,20 @@
+---
+layout: default
+title: Executing JavaScript
+---
+
+# Executing JavaScript
+
+In order to execute JavaScript in a page's DOM context, use the Selenium HTTP adapter and call `#execute_script` on the WebDriver object:
+
+{% highlight ruby %}
+class DummyJob < Wayfarer::Job
+  config.http_adapter = :selenium
+
+  # ...
+
+  def foo
+    pathname = driver.execute_script("return window.location.pathname")
+  end
+end
+{% endhighlight %}

data/docs/recipes/multiple_uris.md

@@ -0,0 +1,18 @@
+---
+layout: default
+title: Starting from multiple URIs
+---
+
+# Starting from multiple URIs
+
+You can pass in as many URIs as desired when performing jobs:
+
+{% highlight ruby %}
+class DummyJob < Wayfarer::Job
+  # ...
+end
+
+uris = [...]
+
+DummyJob.perform_now(*uris)
+{% endhighlight %}

data/docs/recipes/screenshots.md

@@ -0,0 +1,20 @@
+---
+layout: default
+title: Taking screenshots
+---
+
+# Taking screenshots
+
+In order to take screenshots, use the Selenium HTTP adapter and call `#save_screenshot` on the WebDriver object:
+
+{% highlight ruby %}
+class DummyJob < Wayfarer::Job
+  config.http_adapter = :selenium
+
+  # ...
+
+  def foo
+    driver.save_screenshot("my_screenshot.png")
+  end
+end
+{% endhighlight %}

data/docs/routing/host_rules.md

@@ -0,0 +1,24 @@
+---
+layout: default
+title: Host rules
+---
+
+# Host rules
+
+Host rules match against a host string or RegExp.
+
+{% highlight ruby %}
+class DummyJob < Wayfarer::Job
+  route.host "example.com"
+  route.host /example/
+end
+{% endhighlight %}
+
+Matches:
+
+* All URIs hosted on `"example.com"`.
+* All URIs that contain `"example"`.
+
+<aside class="note">
+<code>"www.host.net"</code> and <code>"host.net"</code> are not considered equal. You have to specify the exact host when using strings. Consider using <code>/host.net/</code> instead.
+</aside>

data/docs/routing/path_rules.md

@@ -0,0 +1,33 @@
+---
+layout: default
+title: Path rules
+---
+
+# Path rules
+
+Path rules match against the path of a URI. Both strings and RegExps are accepted, and path segment pattern matching and RegExp captures are supported.
+
+{% highlight ruby %}
+class DummyJob < Wayfarer::Job
+  route.path "/:alpha/:beta", to: :foo
+  route.path /^foobar\/(.+)/, to: :bar
+
+  def foo
+    params[:alpha]
+    params[:beta]
+  end
+
+  def foo
+    params["0"]
+  end
+end
+{% endhighlight %}
+
+Matches:
+
+* All URIs with path segments matching `/:alpha/:beta`, e.g. `https://example.com/foo/bar`
+* All URIs starting with `"/foobar/"`.
+
+<aside class="note">
+<code>/:alpha/:beta</code> and <code>:alpha/:beta</code> are not considered equal. Note the opening slash.
+</aside>

data/docs/routing/query_rules.md

@@ -0,0 +1,69 @@
+---
+layout: default
+title: Query rules
+---
+
+# Query rules
+
+Query rules impose constraints on key-value query parameters. Strings, integers, RegExps and ranges are supported.
+
+## String constraints
+
+{% highlight ruby %}
+class DummyJob < Wayfarer::Job
+  route.query arg: "foo"
+end
+{% endhighlight %}
+
+* Matches `https://example.com?arg=foo`.
+
+## Integer constraints
+
+{% highlight ruby %}
+class DummyJob < Wayfarer::Job
+  route.query arg: 42
+end
+{% endhighlight %}
+
+* Matches `https://example.com?arg=42`.
+
+---
+
+## RegExp constraints
+
+{% highlight ruby %}
+class DummyJob < Wayfarer::Job
+  route.query arg: /foo/
+end
+{% endhighlight %}
+
+* Matches `https://example.com?arg=foo`.
+* Matches `https://example.com?arg=foobar`.
+
+---
+
+## Range constraints
+
+{% highlight ruby %}
+class DummyJob < Wayfarer::Job
+  route.query arg: 1..10
+end
+{% endhighlight %}
+
+* Matches `https://example.com?arg=1`.
+* Matches […]
+* Matches `https://example.com?arg=10`.
+
+---
+
+## Compound constraints
+
+{% highlight ruby %}
+class DummyJob < Wayfarer::Job
+  route.query foo: 1..5, bar: /baz/, qux: "zot", toto: 2
+end
+{% endhighlight %}
+
+* Matches `https://example.com?foo=4&bar=bazqux&qux=zot&toto=2`.
+
+---

data/docs/routing/routes.md

@@ -0,0 +1,96 @@
+---
+layout: default
+title: Routes
+categories: [Routing]
+---
+
+# Routes
+
+* Routes are filters for interesting URIs.
+* Routes put constraints on URIs that should get processed.
+* Routes map URIs to instance methods (actions).
+* Routes are tree nodes and thus nestable.
+
+Currently, the following rules are available:
+
+* [URI rules](uri_rules.html) match URIs against a string.
+* [Host rules](/routing/host_rules.html) match hosts against strings and RegExps.
+* [Path rules](/routing/path_rules.html) match paths against pattern strings and RegExps. They support  path segment capturing.
+* [Query rules](/routing/query_rules.html) match key-value pairs of query parameters against strings, integers, RegExps and ranges.
+
+Routes can be fordidden. URIs that match forbidden rules are never processed.
+
+## Route declaration
+
+### Declaration order matching
+
+{% highlight ruby %}
+class DummyJob < Wayfarer::Job
+  route.host "example.com", to: :foo
+  route.path "/foo", to: :bar
+
+  # Is equivalent to:
+  #
+  # routes do
+  #   host "example.com", to: :foo
+  #   path "/foo", to: :bar
+  # end
+end
+{% endhighlight %}
+
+* Dispatches `https://example.com/foo` to `:foo`.
+* Dispatches `https://example.com` to `:foo`.
+* Dispatches `https://yahoo.com/foo` to `:bar`.
+
+---
+
+### Nesting routes (child rules)
+
+A route matches if it has a child rule that matches. This applies recursively.
+
+{% highlight ruby %}
+class DummyJob < Wayfarer::Job
+  route.host "example.com", to: :foo do
+    path "/foo"
+  end
+
+  # Is equivalent to:
+  #
+  # route.host "example.com", path: "/foo", to: :foo
+  # route.path "/foo", host: "example.com", to: :foo
+end
+{% endhighlight %}
+
+* Dispatches `https://example.com/foo` to `:foo`.
+* Does not dispatch `https://example.com`.
+* Does not dispatch `https://yahoo.com/foo`.
+
+---
+
+### Deepest routes override actions
+
+{% highlight ruby %}
+class DummyJob < Wayfarer::Job
+  route.host "example.com", to: :foo do
+    path "/foo", to: :bar
+  end
+end
+{% endhighlight %}
+
+* Dispatches `https://example.com/foo` to `:bar`.
+* Does not dispatch `https://example.com`.
+* Does not dispatch `https://yahoo.com/foo`.
+
+---
+
+### Forbidding rules
+
+{% highlight ruby %}
+class DummyJob < Wayfarer::Job
+  route.forbid.path "/foo"
+  route.host "example.com", to: :foo
+end
+{% endhighlight %}
+
+* Dispatches `https://example.com` to `:bar`.
+* Does not dispatch `https://example.com/foo`.