wayfarer 0.0.3

data/docs/guides/tutorial.md

@@ -0,0 +1,452 @@
+---
+layout: default
+title: Tutorial
+---
+
+# Tutorial
+This tutorial walks you through 66.333% of what's to know about Wayfarer, a web crawling framework for Ruby. Along the way, we'll write a reusable crawler that collects the titles of all open issues from an arbitrary GitHub repository.
+
+First, we get ourselves a subclass of `Wayfarer::Job`. If you've ever worked with a typical MVC web framework, think of a job as a self-contained controller with routes. If you haven't, don't worry!
+
+{% highlight ruby %}
+require "wayfarer" # This line omitted hereafter
+
+class CollectGithubIssues < Wayfarer::Job
+end
+{% endhighlight %}
+
+Suppose we’re interested in Rails' GitHub repository, which is located at `https://github.com/rails/rails`. We need two things:
+1. A route that matches that URI and …
+2. an instance method (action) which handles that page:
+
+{% highlight ruby %}
+class CollectGithubIssues < Wayfarer::Job
+  route.uri "https://github.com/rails/rails", to: :repository # (1)
+
+  def repository # (2)
+    puts "This looks like Rails to me!"
+  end
+end
+{% endhighlight %}
+
+We set up a single route which maps the repository URI (and only that URI) to `CollectGithubIssues#repository`. When we feed our job the URI, the `#repository` method gets called.
+
+To run a job, , call `::perform_now` on your job class and pass an arbitrary number of URIs to start with:
+
+{% highlight ruby %}
+class CollectGithubIssues < Wayfarer::Job
+  # Gives more detailed output
+  # I'll omit this from now on
+  config.logger.level = :debug
+
+  route.uri "https://github.com/rails/rails", to: :repository
+
+  def repository
+    puts "This looks like Rails to me!"
+  end
+end
+
+CollectGithubIssues.perform_now("https://github.com/rails/rails", "https://example.com")
+{% endhighlight %}
+
+Note that we pass a URI we have no matching route for, `https://example.com`.
+
+Save and run your file as you would with every other Ruby file:
+
+```
+% ruby collect_github_issues.rb
+```
+
+… and you'll end up with output similiar to this:
+
+```
+Performing CollectGithubIssues (Job ID: …) from Async(default) with arguments: "https://github.com/rails/rails", "https://example.com"
+I, […]  INFO -- wayfarer: First cycle
+I, […]  INFO -- wayfarer: Frontier: URI-normalizing #<Wayfarer::Frontiers::MemoryFrontier:0x007fa2a6ae9cf0>
+I, […]  INFO -- wayfarer: Current cycle contains 2 URI(s)
+I, […]  INFO -- wayfarer: Dispatched to #repository: https://github.com/rails/rails
+This looks like Rails to me!
+I, […]  INFO -- wayfarer: Staging 0 URI(s)
+D, […] DEBUG -- wayfarer: No matching route for: https://example.com/
+I, […]  INFO -- wayfarer: No URIs left in current cycle
+I, […]  INFO -- wayfarer: About to cycle. 0 staged URI(s)
+Performed CollectGithubIssues (Job ID: …) from Async(default) in 863.69ms
+```
+
+Here is what happened:
+
+1. Both URIs we passed in were matched against our routes.
+2. Our matching GitHub URI's page was retrieved, the mismatching one ignored.
+3. Our `#repository` action was invoked and has access to the retrieved page.
+
+
+
+Let’s exchange our static string for the actual page `<title>`. Inside our instance method, we call `#doc` to get ahold of a [`Nokogiri::HTML::Document`](http://www.rubydoc.info/github/sparklemotion/nokogiri/Nokogiri/HTML/Document). [Nokogiri]() is a HTML/XML library, and a parsed document allows us to access the title tag easily:
+
+{% highlight ruby %}
+class CollectGithubIssues < Wayfarer::Job
+  route.uri "https://github.com/rails/rails", to: :repository
+
+  def repository
+    # Outputs the <title> attribute value
+    puts doc.title
+  end
+end
+
+CollectGithubIssues.perform_now("https://github.com/rails/rails")
+{% endhighlight %}
+
+Wayfarer does not attempt to do black magic on top of Nokogiri. When it comes to extracting specific data from pages, you’re mostly on your own. There are helpers for finding links, CSS/JavaScript files and images (see [`Page` objects](page_objects.html)). But figuring out what the interesting parts of a HTTP response are is still up to you.
+
+Wayfarer parses JSON, too. You'll get a `Hash` returned by `#doc` instead of a  Nokogiri document.
+
+Rails’ issues are located at `https://github.com/rails/rails/issues`. We need a new route and a new instance method to handle this issue index. By calling `#stage` and passing in an arbitrary number of URIs, we can stage URIs for processing. Note that just because a URI gets staged does not mean it will be fetched—a matching route is required for every URI. Also, Wayfarer will by default ensure that no URI gets processed twice. This behaviour can be turned off, though (see [Configuration](configuration.html)).
+
+{% highlight ruby %}
+class CollectGithubIssues < Wayfarer::Job
+  routes do
+    uri "https://github.com/rails/rails",        to: :repository
+    uri "https://github.com/rails/rails/issues", to: :index
+  end
+
+  def repository
+    # This is where we want to head at
+    stage "https://github.com/rails/rails/issues"
+  end
+
+  def index
+    puts "Arrived at the issue listing"
+  end
+end
+
+CollectGithubIssues.perform_now("https://github.com/rails/rails")
+{% endhighlight %}
+
+What we have so far works fine for the Rails repository, but not for others, because the URIs are hardcoded. That's a real pity, because there are more than 10 million repositories on GitHub. We can do better by switching to a host and path rule.
+
+A host rule narrows down the host portion of a URI, and a path rule the path. Instead of hard-coding the path, pattern matching can be used to have interesting parts of the path extracted:
+
+{% highlight ruby %}
+class CollectGithubIssues < Wayfarer::Job
+  routes do
+    # Both routes match only if
+    # (1) The host is github.com and
+    # (2) The path segments match
+    host "github.com" do
+      path "/:user/:repo",        to: :repository
+      path "/:user/:repo/issues", to: :index
+    end
+  end
+
+  def repository
+    stage "https://github.com/rails/rails/issues"
+  end
+
+  def index
+    # Captured path segments: params # => { repo: ..., user: ... }
+    # Prints 'rails belongs to rails'.
+    puts "#{params['repo']} belongs to #{params['user']}"
+  end
+end
+
+CollectGithubIssues.perform_now("https://github.com/rails/rails")
+{% endhighlight %}
+
+Note that we still have a hard-coded URI in `#repository`. Usually, there are two approaches to identify URIs that one wants to follow:
+
+1. Constructing the successor URI from the current URI.
+2. Reading the URI from the HTTP response, e.g. extracting an `<a>` tag's `href` property.
+
+For the first case, say we're on `https://github.com/:user/:repo` and want to go to `https://github.com/:user/:repo/issues`. `#stage` takes relative paths and URIs too, and constructs absolute URIs by appending to the current page's URI:
+
+{% highlight ruby %}
+class CollectGithubIssues < Wayfarer::Job
+  # ...
+
+  def index
+    # Stages "#{page.uri}/issues"
+    stage "issues"
+  end
+
+  # ...
+end
+{% endhighlight %}
+
+`#page` returns a [`Page` object]({{base}}/guides/page_objects.html), the general representation of a retrieved page. It gives one access to the page's origin URI, the response headers, the status code and the raw response body and more.
+
+The second case is where Wayfarer's routing shines. We know that the path structure is `/:user/:repo/issues` and that there's a link somewhere on the repository's frontpage that links to there. We can stage __all__ links of the current page, and have our routes ensure that only interesting ones get processed:
+
+{% highlight ruby %}
+class CollectGithubIssues < Wayfarer::Job
+  # ...
+
+  def repository
+    # But only route-matching ones get processed
+    stage page.links
+  end
+
+  # ...
+end
+{% endhighlight %}
+
+`Page#links` returns all links of the current site. But staging all links brings overhead with it, and we'll want to narrow down the links to stage, especially when crawling large page structures. `Page#links` accepts an arbitrary number of CSS selectors to narrow down links. For clarity, let's give the navigation links their own private helper method:
+
+{% highlight ruby %}
+class CollectGithubIssues < Wayfarer::Job
+  routes do
+    host "github.com" do
+      path "/:user/:repo",        to: :repository
+      path "/:user/:repo/issues", to: :index
+    end
+  end
+
+  def repository
+    stage navigation_links
+  end
+
+  def index
+    puts "#{params['repo']} belongs to #{params['user']}"
+  end
+
+  private
+
+  def navigation_links
+    page.links ".reponav-item"
+  end
+end
+
+CollectGithubIssues.perform_now("https://github.com/rails/rails")
+{% endhighlight %}
+
+URIs never get dispatched to private instance methods.
+
+We're prepared to go after the individual issues now. We add the `#issue` action, and route to it with a host and path rule. Links to issue tickets are wrapped in `.issues-listing`, so we can apply the same technique as above:
+
+{% highlight ruby %}
+class CollectGithubIssues < Wayfarer::Job
+  routes do
+    host "github.com" do
+      path "/:user/:repo",            to: :repository
+      path "/:user/:repo/issues",     to: :index
+      path "/:user/:repo/issues/:id", to: :show
+    end
+  end
+
+  def repository
+    stage navigation_links
+  end
+
+  def index
+    stage issue_listing_links
+  end
+
+  def show
+    puts "Issue No. #{params[:id]} @ #{page.uri}"
+  end
+
+  private
+
+  def navigation_links
+    page.links ".reponav-item"
+  end
+
+  def issue_listing_links
+    page.links ".issues-listing"
+  end
+end
+
+CollectGithubIssues.perform_now("https://github.com/rails/rails")
+{% endhighlight %}
+
+Handling pagination boils down to staging one more link in `#index`. As mentioned before, `#stage` accepts an arbitrary number of URIs:
+
+{% highlight ruby %}
+class CollectGithubIssues < Wayfarer::Job
+  routes do
+    host "github.com" do
+      path "/:user/:repo",            to: :repository
+      path "/:user/:repo/issues",     to: :index
+      path "/:user/:repo/issues/:id", to: :show
+    end
+  end
+
+  def repository
+    stage navigation_links
+  end
+
+  def index
+    stage issue_listing_links, next_page
+  end
+
+  def show
+    puts "Issue No. #{params[:id]} @ #{page.uri}"
+  end
+
+  private
+
+  def navigation_links
+    page.links ".reponav-item"
+  end
+
+  def issue_listing_links
+    page.links ".issues-listing"
+  end
+
+  def next_page
+    page.links ".next_page"
+  end
+end
+
+CollectGithubIssues.perform_now("https://github.com/rails/rails")
+{% endhighlight %}
+
+By default, all work happens within a single thread. We can speed up crawling by increasing the thread count:
+
+{% highlight ruby %}
+class CollectGithubIssues < Wayfarer::Job
+  config.connection_count = 4 # Four threads
+
+  # ...
+end
+{% endhighlight %}
+
+Next, we want to extract the issue's title, its ID, and the GitHub user who opened it and store that data somewhere.
+
+For extracting the text from the HTML, we add two private helper methods that query the HTML for the text.
+
+For storing the data, we introduce a [local]({{base}}/guides/locals.html) named `:records` which stores an array. In job actions, locals can be accessed and manipulated. But now that we've bumped up the thread count, multiple instances of our job class will run concurrently. That's why locals declared with `::let` are replaced with thread-safe counterparts behind the scenes.
+
+We stop processing with `halt` once we have collected 30 issue records:
+
+{% highlight ruby %}
+class CollectGithubIssues < Wayfarer::Job
+  config.connection_count = 4
+
+  let(:records) { [] }
+
+  routes do
+    host "github.com" do
+      path "/:user/:repo",            to: :repository
+      path "/:user/:repo/issues",     to: :index
+      path "/:user/:repo/issues/:id", to: :show
+    end
+  end
+
+  after_crawl do
+    records.each do |issue|
+      # Save them somewhere?
+      puts issue
+    end
+  end
+
+  def repository
+    stage navigation_links
+  end
+
+  def index
+    stage issue_listing_links, next_page
+  end
+
+  def show
+    return halt if records.count > 30
+
+    records << {
+      id: params[:id],
+      title: issue_title,
+      author: issue_author
+    }
+  end
+
+  private
+
+  def issue_title
+    doc.css(".js-issue-title").text.strip
+  end
+
+  def issue_author
+    doc.css(".TableObject-item .author").text.strip
+  end
+
+  def navigation_links
+    page.links ".reponav-item"
+  end
+
+  def issue_listing_links
+    page.links ".issues-listing"
+  end
+
+  def next_page
+    page.links ".next_page"
+  end
+end
+
+CollectGithubIssues.perform_now("https://github.com/rails/rails")
+{% endhighlight %}
+
+For the last part, we turn off the debugging output (if you have still enabled it) and output each record. You'd probably want to store them somewhere at this point, e.g. by [writing them to a CSV file]({{base}}/recipes/csv.html), or putting them into a database, etc.
+
+{% highlight ruby %}
+class CollectGithubIssues < Wayfarer::Job
+  config.connection_count = 4
+  config.logger.level = :fatal
+
+  let(:records) { [] }
+
+  routes do
+    host "github.com" do
+      path "/:user/:repo",            to: :repository
+      path "/:user/:repo/issues",     to: :index
+      path "/:user/:repo/issues/:id", to: :show
+    end
+  end
+
+  after_crawl do
+    records.each do |issue|
+      # Save them somewhere?
+      puts issue
+    end
+  end
+
+  def repository
+    stage navigation_links
+  end
+
+  def index
+    stage issue_listing_links, next_page
+  end
+
+  def show
+    return halt if records.count > 30
+
+    records << {
+      id: params[:id],
+      title: issue_title,
+      author: issue_author
+    }
+  end
+
+  private
+
+  def issue_title
+    doc.css(".js-issue-title").text.strip
+  end
+
+  def issue_author
+    doc.css(".TableObject-item .author").text.strip
+  end
+
+  def navigation_links
+    page.links ".reponav-item"
+  end
+
+  def issue_listing_links
+    page.links ".issues-listing"
+  end
+
+  def next_page
+    page.links ".next_page"
+  end
+end
+
+CollectGithubIssues.perform_now("https://github.com/rails/rails")
+{% endhighlight %}

data/docs/index.md

@@ -0,0 +1,82 @@
+---
+layout: default
+title: Versatile web crawling with (J)Ruby
+---
+
+# Versatile web crawling with (J)Ruby
+
+Wayfarer is the swiss army knife for web crawling.
+
+MRI:
+```
+% [sudo] gem install wayfarer
+```
+
+JRuby:
+```
+% [sudo] gem install wayfarer-jruby
+```
+
+If you …
+
+* __Need to crawl page graphs breadth-first__
+* __Need to extract whatever data__
+* Do it multi-threaded
+* Integrate with Rails seamlessly
+* Want to automate a web browser
+* Need to execute arbitrary JavaScript
+* Need URI normalization
+* Need to take screenshots
+* Want to use a job queue and make work happen later
+* Want in-memory and Redis-backed frontiers, tries and Bloom filters
+
+… then you might like Wayfarer!
+
+## What it looks like
+
+Say you want to …
+
+* Automate Google Chrome
+* Start off with a random Wikipedia article
+* Follow all links until you find a page with the word "Foobar"
+* Take a screenshot of the page containing "Foobar"
+* Extract keywords from every page you encounter
+* Use 4 threads and Chrome processes to do so
+
+This amounts to 16 lines of code with Wayfarer:
+
+{% highlight ruby %}
+require "wayfarer"
+
+class FindFoobarOnWikipedia < Wayfarer::Job
+  config.http_adapter = :selenium
+  config.selenium_argv = [:chrome]
+  config.connection_count = 4
+
+  let(:keywords) { [] }
+
+  route.host "en.wikipedia.org", to: :article
+
+  def article
+    if page.body =~ /Foobar/
+      driver.save_screenshot("/tmp/foobar.png")
+      return halt
+    end
+
+    keywords << page.keywords
+    stage page.links
+  end
+end
+
+FindFoobarOnWikipedia.perform_now("https://en.wikipedia.org/wiki/Special:Random")
+{% endhighlight %}
+
+Wayfarer integrates with [ActiveJob]() and supports your favorite job queue out of the box. Your job is ready to be enqueued:
+{% highlight ruby %}
+FindFoobarOnWikipedia.perform_later("https://en.wikipedia.org/wiki/Special:Random")
+{% endhighlight %}
+
+### Where to go from here
+
+* [The tutorial]() shows how to collect all open issues from a GitHub repository
+* Read the [API documentation]()

data/docs/js/navigation.js

@@ -0,0 +1,11 @@
+document.addEventListener("DOMContentLoaded", function() {
+  var links = document.querySelectorAll(".navigation__link");
+
+  for (i = 0; i < links.length; i++) {
+    var link = links[i];
+
+    if (link.pathname === window.location.pathname) {
+      link.classList.add("navigation__link--active");
+    }
+  }
+});

data/docs/misc/contributing.md

@@ -0,0 +1,20 @@
+---
+layout: default
+title: Contributing
+---
+
+# Contributing
+
+1. Fork the repository
+2. Ensure the development dependencies are installed:
+    `% bundle install --with development`
+2. Make changes
+3. Ensure your (new?) tests pass:
+  `% bundle exec rake test`
+4. Autocorrect RubuCop offenses:
+  `% bundle exec rake rubocop:auto_correct`
+5. Fix remaining offenses or have a good excuse not to:
+  `% bundle exec rake rubocop`
+6. Write commit messages at least not worse than mine
+7. Open a pull request on GitHub
+8. Thank you

data/docs/misc/testing.md

@@ -0,0 +1,11 @@
+---
+layout: default
+title: Testing
+---
+
+# Testing
+
+Tests run on MRI 2.3.1 and JRuby 9.1.6.0.
+
+* Run `rake -T` to list all available (test) tasks.
+* When running tests, a HTTP server binds to port 9876 for integration-level tests.

data/docs/recipes/authentication.md

@@ -0,0 +1,23 @@
+---
+layout: default
+title: Authentication
+---
+
+# Authentication
+
+Authentication is best handled with the `setup_adapter` callback See [Callbacks]({{base}}/guides/callbacks.html).
+
+{% highlight ruby %}
+class DummyJob < Wayfarer::Job
+  config.http_adapter = :selenium
+
+  setup_adapter do |_, _, browser|
+    browser.visit("https://foo.com/login")
+
+    browser.fill_in("E-mail", with: "foo@bar.com")
+    browser.fill_in("Password", with: "password")
+
+    browser.click_button("Log in")
+  end
+end
+{% endhighlight %}

data/docs/recipes/csv.md

@@ -0,0 +1,29 @@
+---
+layout: default
+title: CSV output
+---
+
+# CSV output
+
+See:
+
+* [Locals]({{base}}/guides/locals.html)
+* [Callbacks]({{base}}/guides/callbacks.html)
+
+{% highlight ruby %}
+require "csv" # from Ruby's standard lib
+
+class DummyJob < Wayfarer::Job
+  let(:records) { [] }
+
+  after_crawl do
+    CSV.open("output.csv", "w") do |csv|
+      records.each { |r| csv << [r[:id], r[:name]] }
+    end
+  end
+
+  def detail
+    records << { id: ..., name: ...  }
+  end
+end
+{% endhighlight %}

data/docs/recipes/javascript.md

@@ -0,0 +1,20 @@
+---
+layout: default
+title: Executing JavaScript
+---
+
+# Executing JavaScript
+
+In order to execute JavaScript in a page's DOM context, use the Selenium HTTP adapter and call `#execute_script` on the WebDriver object:
+
+{% highlight ruby %}
+class DummyJob < Wayfarer::Job
+  config.http_adapter = :selenium
+
+  # ...
+
+  def foo
+    pathname = driver.execute_script("return window.location.pathname")
+  end
+end
+{% endhighlight %}

data/docs/recipes/multiple_uris.md

@@ -0,0 +1,18 @@
+---
+layout: default
+title: Starting from multiple URIs
+---
+
+# Starting from multiple URIs
+
+You can pass in as many URIs as desired when performing jobs:
+
+{% highlight ruby %}
+class DummyJob < Wayfarer::Job
+  # ...
+end
+
+uris = [...]
+
+DummyJob.perform_now(*uris)
+{% endhighlight %}

data/docs/recipes/screenshots.md

@@ -0,0 +1,20 @@
+---
+layout: default
+title: Taking screenshots
+---
+
+# Taking screenshots
+
+In order to take screenshots, use the Selenium HTTP adapter and call `#save_screenshot` on the WebDriver object:
+
+{% highlight ruby %}
+class DummyJob < Wayfarer::Job
+  config.http_adapter = :selenium
+
+  # ...
+
+  def foo
+    driver.save_screenshot("my_screenshot.png")
+  end
+end
+{% endhighlight %}