wayfarer 0.0.3 → 0.4.3

data/docs/guides/page_objects.md

@@ -1,67 +0,0 @@
----
-layout: default
-title: Page objects
----
-
-# `Page` objects
-
-Retrieved pages are represented by `Page` objects and made accessible by `#page` within actions. `Page`s support the same set of features regardless of the HTTP adapter in use.
-
-<aside class="note">
-HTTP response headers and status codes are not supported by Selenium WebDrivers. Wayfarer emulates both by having the WebDriver fire an AJAX request to the current page and extracting them from the response. Clearly this is a hack, but it might even work for you. See <a href="https://github.com/bauerd/selenium-emulated_features">selenium-emulated_features</a>.
-</aside>
-
-<aside class="note">
-Even after having followed redirects, <code>Page#uri</code> always returns the URI that originally initiated the redirects. This behaviour stems from redirects being opaque to WebDrivers.
-</aside>
-
-A `Page` brings to the table all you'd wish for when doing web scraping:
-
-* [Nokogiri](http://www.nokogiri.org) parses HTML/XML
-* [Oj](https://github.com/ohler55/oj) or the standard lib parses JSON
-* __When running on MRI__, [Pismo](https://github.com/peterc/pismo) lets you access metadata, e.g. keywords, author, a summary, … No overhead if you don't use it!
-
-Let's see it in action:
-
-{% highlight ruby %}
-class DummyJob < Wayfarer::Job
-  # ...
-
-  def example
-    page # => #<Wayfarer::Page:...>
-
-    page.uri # => #<URI::...>
-    page.status_code # => Fixnum
-    page.body # => String
-    page.headers # => Hash
-
-    page.doc # => #<Nokogiri::HTML::Document:...> (HTML/XML) or Hash (JSON)
-                  # Also accessible as just `doc`
-
-    page.links # => [URI]
-    page.stylesheets # => [URI]
-    page.javascripts # => [URI]
-    page.images # => [URI]
-
-    # All previous four methods accept arbitrary many CSS selectors
-    page.links ".my-target", ".my-other-target"
-
-    # THESE ARE NOT SUPPORTED ON JRUBY!
-    # On MRI, the following methods get forwarded to a Pismo::Document
-    # See https://github.com/peterc/pismo
-    page.title
-    page.titles
-    page.author
-    page.lede
-    page.keywords
-    page.sentences(qty)
-    page.body
-    page.html_body
-    page.feed
-    page.feeds
-    page.favicon
-    page.description
-    page.datetime
-  end
-end
-{% endhighlight %}

data/docs/guides/peeking.md

@@ -1,46 +0,0 @@
----
-layout: default
-title: Peeking
----
-
-# Peeking
-Peeking allows bypassing the [frontier](frontiers.html) in an ad-hoc manner. Use Ruby's `yield` keyword to immediately retrieve and dispatch a URI from within actions. Control gets handed off to the action matching the yielded URI, if any.
-
-A matching route for the yielded URI is still required. If the yielded URI matches no route or raises an exception, `yield` returns `nil`.
-
-<aside class="note">
-The action that gets the URI dispatched to <strong>will</strong> get assigned another HTTP adapter! HTTP adapters are never shared across actions, i.e. if you're using the Selenium HTTP adapter, the peeked URI gets retrieved by a different browser process.
-</aside>
-
-{% highlight ruby %}
-class DummyJob < Wayfarer::Job
-  route.uri "https://example.com", to: :foo
-  route.uri "https://w3c.org", to: :bar
-
-  def foo
-    w3c_page = yield "https://w3c.org"
-  end
-
-  def bar
-    page
-  end
-end
-{% endhighlight %}
-
-__Recursive peeking does not work__, or else peeking might result in an infinite loop. The following does terminate:
-
-{% highlight ruby %}
-class DummyJob < Wayfarer::Job
-  route.uri "https://example.com", to: :foo
-  route.uri "https://w3c.org", to: :bar
-
-  def foo
-    w3c_page = yield "https://w3c.org"
-  end
-
-  def bar
-    # Silently ignored, assigns nil
-    example_page = yield "https://example.com"
-  end
-end
-{% endhighlight %}

data/docs/guides/selenium_capybara.md

@@ -1,100 +0,0 @@
----
-layout: default
-title: Selenium & Capybara
----
-
-# Selenium & Capybara
-
-[Selenium](http://www.seleniumhq.org) is a browser automation framework. [Capybara](https://github.com/teamcapybara/capybara) is an acceptance testing framework that puts an expressive DSL on Selenium's WebDrivers. Both are first-class citizens in Wayfarer and the best tools for automating browsers.
-
-## Selenium WebDrivers
-
-WebDrivers let you remote-control browsers, e.g. Firefox, Chrome, Safari and PhantomJS.
-
-Depending on what browser you want to automate, go install and run the corresponding driver first. For installation instructions, see the project websites:
-
-* Firefox: [geckodriver](https://github.com/mozilla/geckodriver)
-* Chrome: [chromedriver](https://sites.google.com/a/chromium.org/chromedriver)
-* Safari: [SafariDriver](https://github.com/SeleniumHQ/selenium/wiki/SafariDriver)
-* PhantomJS ships with an embedded driver.
-
-Other browsers are supported, too. For an exhaustive list, see the "Third Party Drivers, Bindings, and Plugins" section on the [Selenium downloads page](http://www.seleniumhq.org/download).
-
-If you want to run browser processes on a central server, consider using [Selenium Grid](http://www.seleniumhq.org/projects/grid).
-
-Wayfarer hides the details of managing Ruby driver objects from you. In order to use Selenium, set the `http_adapter` configuration key to `:selenium`. Pass in the desired browser and arguments by setting the `selenium_argv` key. The number of browser processes can be controlled with the `connection_count` key.
-
-{% highlight ruby %}
-class DummyJob < Wayfarer::Job
-  config do |c|
-    # Use 4 Firefox processes
-    c.http_adapter = :selenium
-    c.selenium_argv = [:firefox]
-    c.connection_count = 4
-
-    # Chrome
-    # c.selenium_argv = [:chrome]
-
-    # Safari
-    # c.selenium_argv = [:safari]
-
-    # PhantomJS
-    # c.selenium_argv = [:phantomjs]
-
-    # Selenium Grid
-    # c.selenium_argv = [
-    #   :remote,
-    #   url: "http://localhost:4444/wd/hub",
-    #   desired_capabilities: :firefox
-    # ]
-  end
-end
-{% endhighlight %}
-
-<aside class="note">
-In order to avoid redirect loops, the <code>:net_http</code> adapter supports the <code>max_http_redirects</code> configuration key. Because redirects are opaque to WebDrivers, the configuration key does not apply to the Selenium adapter. See <a href="configuration.html">Configuration</a>.
-</aside>
-
-### Accessing the WebDriver
-
-Within actions, `#driver` returns a [`Selenium::WebDriver::Driver`](http://www.rubydoc.info/gems/selenium-webdriver/Selenium/WebDriver/Driver):
-
-{% highlight ruby %}
-class DummyJob < Wayfarer::Job
-  config do |c|
-    c.http_adapter = :selenium
-    c.selenium_argv = [:firefox]
-  end
-
-  draw uri: "https://example.com"
-  def example
-    driver # => #<Selenium::WebDriver::Driver:...>
-  end
-end
-{% endhighlight %}
-
-<aside class="note">
-What you do with a WebDriver is opaque to Wayfarer. If you handle navigation yourself with a WebDriver and bypass the <a href="/guides/frontiers.html">frontier</a>, Wayfarer cannot ensure you don't visit URIs twice.
-</aside>
-
-## Capybara
-
-When using the `:selenium` HTTP adapter, `#browser` returns a [`Capybara::Selenium::Driver`](http://www.rubydoc.info/github/jnicklas/capybara/Capybara/Selenium/Driver) within actions:
-
-{% highlight ruby %}
-class DummyJob < Wayfarer::Job
-  config do |c|
-    c.http_adapter = :selenium
-    c.selenium_argv = [:firefox]
-  end
-
-  draw uri: "https://example.com"
-  def example
-    browser # => #<Capybara::Selenium::Driver:...>
-  end
-end
-{% endhighlight %}
-
-<aside class="note">
-What you do with a WebDriver is opaque to Wayfarer. If you handle navigation yourself with a WebDriver and bypass the <a href="/guides/frontiers.html">frontier</a>, Wayfarer cannot ensure you don't visit URIs twice.
-</aside>

data/docs/guides/tutorial.md

@@ -1,452 +0,0 @@
----
-layout: default
-title: Tutorial
----
-
-# Tutorial
-This tutorial walks you through 66.333% of what's to know about Wayfarer, a web crawling framework for Ruby. Along the way, we'll write a reusable crawler that collects the titles of all open issues from an arbitrary GitHub repository.
-
-First, we get ourselves a subclass of `Wayfarer::Job`. If you've ever worked with a typical MVC web framework, think of a job as a self-contained controller with routes. If you haven't, don't worry!
-
-{% highlight ruby %}
-require "wayfarer" # This line omitted hereafter
-
-class CollectGithubIssues < Wayfarer::Job
-end
-{% endhighlight %}
-
-Suppose we’re interested in Rails' GitHub repository, which is located at `https://github.com/rails/rails`. We need two things:
-1. A route that matches that URI and …
-2. an instance method (action) which handles that page:
-
-{% highlight ruby %}
-class CollectGithubIssues < Wayfarer::Job
-  route.uri "https://github.com/rails/rails", to: :repository # (1)
-
-  def repository # (2)
-    puts "This looks like Rails to me!"
-  end
-end
-{% endhighlight %}
-
-We set up a single route which maps the repository URI (and only that URI) to `CollectGithubIssues#repository`. When we feed our job the URI, the `#repository` method gets called.
-
-To run a job, , call `::perform_now` on your job class and pass an arbitrary number of URIs to start with:
-
-{% highlight ruby %}
-class CollectGithubIssues < Wayfarer::Job
-  # Gives more detailed output
-  # I'll omit this from now on
-  config.logger.level = :debug
-
-  route.uri "https://github.com/rails/rails", to: :repository
-
-  def repository
-    puts "This looks like Rails to me!"
-  end
-end
-
-CollectGithubIssues.perform_now("https://github.com/rails/rails", "https://example.com")
-{% endhighlight %}
-
-Note that we pass a URI we have no matching route for, `https://example.com`.
-
-Save and run your file as you would with every other Ruby file:
-
-```
-% ruby collect_github_issues.rb
-```
-
-… and you'll end up with output similiar to this:
-
-```
-Performing CollectGithubIssues (Job ID: …) from Async(default) with arguments: "https://github.com/rails/rails", "https://example.com"
-I, […]  INFO -- wayfarer: First cycle
-I, […]  INFO -- wayfarer: Frontier: URI-normalizing #<Wayfarer::Frontiers::MemoryFrontier:0x007fa2a6ae9cf0>
-I, […]  INFO -- wayfarer: Current cycle contains 2 URI(s)
-I, […]  INFO -- wayfarer: Dispatched to #repository: https://github.com/rails/rails
-This looks like Rails to me!
-I, […]  INFO -- wayfarer: Staging 0 URI(s)
-D, […] DEBUG -- wayfarer: No matching route for: https://example.com/
-I, […]  INFO -- wayfarer: No URIs left in current cycle
-I, […]  INFO -- wayfarer: About to cycle. 0 staged URI(s)
-Performed CollectGithubIssues (Job ID: …) from Async(default) in 863.69ms
-```
-
-Here is what happened:
-
-1. Both URIs we passed in were matched against our routes.
-2. Our matching GitHub URI's page was retrieved, the mismatching one ignored.
-3. Our `#repository` action was invoked and has access to the retrieved page.
-
-
-
-Let’s exchange our static string for the actual page `<title>`. Inside our instance method, we call `#doc` to get ahold of a [`Nokogiri::HTML::Document`](http://www.rubydoc.info/github/sparklemotion/nokogiri/Nokogiri/HTML/Document). [Nokogiri]() is a HTML/XML library, and a parsed document allows us to access the title tag easily:
-
-{% highlight ruby %}
-class CollectGithubIssues < Wayfarer::Job
-  route.uri "https://github.com/rails/rails", to: :repository
-
-  def repository
-    # Outputs the <title> attribute value
-    puts doc.title
-  end
-end
-
-CollectGithubIssues.perform_now("https://github.com/rails/rails")
-{% endhighlight %}
-
-Wayfarer does not attempt to do black magic on top of Nokogiri. When it comes to extracting specific data from pages, you’re mostly on your own. There are helpers for finding links, CSS/JavaScript files and images (see [`Page` objects](page_objects.html)). But figuring out what the interesting parts of a HTTP response are is still up to you.
-
-Wayfarer parses JSON, too. You'll get a `Hash` returned by `#doc` instead of a  Nokogiri document.
-
-Rails’ issues are located at `https://github.com/rails/rails/issues`. We need a new route and a new instance method to handle this issue index. By calling `#stage` and passing in an arbitrary number of URIs, we can stage URIs for processing. Note that just because a URI gets staged does not mean it will be fetched—a matching route is required for every URI. Also, Wayfarer will by default ensure that no URI gets processed twice. This behaviour can be turned off, though (see [Configuration](configuration.html)).
-
-{% highlight ruby %}
-class CollectGithubIssues < Wayfarer::Job
-  routes do
-    uri "https://github.com/rails/rails",        to: :repository
-    uri "https://github.com/rails/rails/issues", to: :index
-  end
-
-  def repository
-    # This is where we want to head at
-    stage "https://github.com/rails/rails/issues"
-  end
-
-  def index
-    puts "Arrived at the issue listing"
-  end
-end
-
-CollectGithubIssues.perform_now("https://github.com/rails/rails")
-{% endhighlight %}
-
-What we have so far works fine for the Rails repository, but not for others, because the URIs are hardcoded. That's a real pity, because there are more than 10 million repositories on GitHub. We can do better by switching to a host and path rule.
-
-A host rule narrows down the host portion of a URI, and a path rule the path. Instead of hard-coding the path, pattern matching can be used to have interesting parts of the path extracted:
-
-{% highlight ruby %}
-class CollectGithubIssues < Wayfarer::Job
-  routes do
-    # Both routes match only if
-    # (1) The host is github.com and
-    # (2) The path segments match
-    host "github.com" do
-      path "/:user/:repo",        to: :repository
-      path "/:user/:repo/issues", to: :index
-    end
-  end
-
-  def repository
-    stage "https://github.com/rails/rails/issues"
-  end
-
-  def index
-    # Captured path segments: params # => { repo: ..., user: ... }
-    # Prints 'rails belongs to rails'.
-    puts "#{params['repo']} belongs to #{params['user']}"
-  end
-end
-
-CollectGithubIssues.perform_now("https://github.com/rails/rails")
-{% endhighlight %}
-
-Note that we still have a hard-coded URI in `#repository`. Usually, there are two approaches to identify URIs that one wants to follow:
-
-1. Constructing the successor URI from the current URI.
-2. Reading the URI from the HTTP response, e.g. extracting an `<a>` tag's `href` property.
-
-For the first case, say we're on `https://github.com/:user/:repo` and want to go to `https://github.com/:user/:repo/issues`. `#stage` takes relative paths and URIs too, and constructs absolute URIs by appending to the current page's URI:
-
-{% highlight ruby %}
-class CollectGithubIssues < Wayfarer::Job
-  # ...
-
-  def index
-    # Stages "#{page.uri}/issues"
-    stage "issues"
-  end
-
-  # ...
-end
-{% endhighlight %}
-
-`#page` returns a [`Page` object]({{base}}/guides/page_objects.html), the general representation of a retrieved page. It gives one access to the page's origin URI, the response headers, the status code and the raw response body and more.
-
-The second case is where Wayfarer's routing shines. We know that the path structure is `/:user/:repo/issues` and that there's a link somewhere on the repository's frontpage that links to there. We can stage __all__ links of the current page, and have our routes ensure that only interesting ones get processed:
-
-{% highlight ruby %}
-class CollectGithubIssues < Wayfarer::Job
-  # ...
-
-  def repository
-    # But only route-matching ones get processed
-    stage page.links
-  end
-
-  # ...
-end
-{% endhighlight %}
-
-`Page#links` returns all links of the current site. But staging all links brings overhead with it, and we'll want to narrow down the links to stage, especially when crawling large page structures. `Page#links` accepts an arbitrary number of CSS selectors to narrow down links. For clarity, let's give the navigation links their own private helper method:
-
-{% highlight ruby %}
-class CollectGithubIssues < Wayfarer::Job
-  routes do
-    host "github.com" do
-      path "/:user/:repo",        to: :repository
-      path "/:user/:repo/issues", to: :index
-    end
-  end
-
-  def repository
-    stage navigation_links
-  end
-
-  def index
-    puts "#{params['repo']} belongs to #{params['user']}"
-  end
-
-  private
-
-  def navigation_links
-    page.links ".reponav-item"
-  end
-end
-
-CollectGithubIssues.perform_now("https://github.com/rails/rails")
-{% endhighlight %}
-
-URIs never get dispatched to private instance methods.
-
-We're prepared to go after the individual issues now. We add the `#issue` action, and route to it with a host and path rule. Links to issue tickets are wrapped in `.issues-listing`, so we can apply the same technique as above:
-
-{% highlight ruby %}
-class CollectGithubIssues < Wayfarer::Job
-  routes do
-    host "github.com" do
-      path "/:user/:repo",            to: :repository
-      path "/:user/:repo/issues",     to: :index
-      path "/:user/:repo/issues/:id", to: :show
-    end
-  end
-
-  def repository
-    stage navigation_links
-  end
-
-  def index
-    stage issue_listing_links
-  end
-
-  def show
-    puts "Issue No. #{params[:id]} @ #{page.uri}"
-  end
-
-  private
-
-  def navigation_links
-    page.links ".reponav-item"
-  end
-
-  def issue_listing_links
-    page.links ".issues-listing"
-  end
-end
-
-CollectGithubIssues.perform_now("https://github.com/rails/rails")
-{% endhighlight %}
-
-Handling pagination boils down to staging one more link in `#index`. As mentioned before, `#stage` accepts an arbitrary number of URIs:
-
-{% highlight ruby %}
-class CollectGithubIssues < Wayfarer::Job
-  routes do
-    host "github.com" do
-      path "/:user/:repo",            to: :repository
-      path "/:user/:repo/issues",     to: :index
-      path "/:user/:repo/issues/:id", to: :show
-    end
-  end
-
-  def repository
-    stage navigation_links
-  end
-
-  def index
-    stage issue_listing_links, next_page
-  end
-
-  def show
-    puts "Issue No. #{params[:id]} @ #{page.uri}"
-  end
-
-  private
-
-  def navigation_links
-    page.links ".reponav-item"
-  end
-
-  def issue_listing_links
-    page.links ".issues-listing"
-  end
-
-  def next_page
-    page.links ".next_page"
-  end
-end
-
-CollectGithubIssues.perform_now("https://github.com/rails/rails")
-{% endhighlight %}
-
-By default, all work happens within a single thread. We can speed up crawling by increasing the thread count:
-
-{% highlight ruby %}
-class CollectGithubIssues < Wayfarer::Job
-  config.connection_count = 4 # Four threads
-
-  # ...
-end
-{% endhighlight %}
-
-Next, we want to extract the issue's title, its ID, and the GitHub user who opened it and store that data somewhere.
-
-For extracting the text from the HTML, we add two private helper methods that query the HTML for the text.
-
-For storing the data, we introduce a [local]({{base}}/guides/locals.html) named `:records` which stores an array. In job actions, locals can be accessed and manipulated. But now that we've bumped up the thread count, multiple instances of our job class will run concurrently. That's why locals declared with `::let` are replaced with thread-safe counterparts behind the scenes.
-
-We stop processing with `halt` once we have collected 30 issue records:
-
-{% highlight ruby %}
-class CollectGithubIssues < Wayfarer::Job
-  config.connection_count = 4
-
-  let(:records) { [] }
-
-  routes do
-    host "github.com" do
-      path "/:user/:repo",            to: :repository
-      path "/:user/:repo/issues",     to: :index
-      path "/:user/:repo/issues/:id", to: :show
-    end
-  end
-
-  after_crawl do
-    records.each do |issue|
-      # Save them somewhere?
-      puts issue
-    end
-  end
-
-  def repository
-    stage navigation_links
-  end
-
-  def index
-    stage issue_listing_links, next_page
-  end
-
-  def show
-    return halt if records.count > 30
-
-    records << {
-      id: params[:id],
-      title: issue_title,
-      author: issue_author
-    }
-  end
-
-  private
-
-  def issue_title
-    doc.css(".js-issue-title").text.strip
-  end
-
-  def issue_author
-    doc.css(".TableObject-item .author").text.strip
-  end
-
-  def navigation_links
-    page.links ".reponav-item"
-  end
-
-  def issue_listing_links
-    page.links ".issues-listing"
-  end
-
-  def next_page
-    page.links ".next_page"
-  end
-end
-
-CollectGithubIssues.perform_now("https://github.com/rails/rails")
-{% endhighlight %}
-
-For the last part, we turn off the debugging output (if you have still enabled it) and output each record. You'd probably want to store them somewhere at this point, e.g. by [writing them to a CSV file]({{base}}/recipes/csv.html), or putting them into a database, etc.
-
-{% highlight ruby %}
-class CollectGithubIssues < Wayfarer::Job
-  config.connection_count = 4
-  config.logger.level = :fatal
-
-  let(:records) { [] }
-
-  routes do
-    host "github.com" do
-      path "/:user/:repo",            to: :repository
-      path "/:user/:repo/issues",     to: :index
-      path "/:user/:repo/issues/:id", to: :show
-    end
-  end
-
-  after_crawl do
-    records.each do |issue|
-      # Save them somewhere?
-      puts issue
-    end
-  end
-
-  def repository
-    stage navigation_links
-  end
-
-  def index
-    stage issue_listing_links, next_page
-  end
-
-  def show
-    return halt if records.count > 30
-
-    records << {
-      id: params[:id],
-      title: issue_title,
-      author: issue_author
-    }
-  end
-
-  private
-
-  def issue_title
-    doc.css(".js-issue-title").text.strip
-  end
-
-  def issue_author
-    doc.css(".TableObject-item .author").text.strip
-  end
-
-  def navigation_links
-    page.links ".reponav-item"
-  end
-
-  def issue_listing_links
-    page.links ".issues-listing"
-  end
-
-  def next_page
-    page.links ".next_page"
-  end
-end
-
-CollectGithubIssues.perform_now("https://github.com/rails/rails")
-{% endhighlight %}

data/docs/js/navigation.js

@@ -1,11 +0,0 @@
-document.addEventListener("DOMContentLoaded", function() {
-  var links = document.querySelectorAll(".navigation__link");
-
-  for (i = 0; i < links.length; i++) {
-    var link = links[i];
-
-    if (link.pathname === window.location.pathname) {
-      link.classList.add("navigation__link--active");
-    }
-  }
-});

data/docs/misc/contributing.md

@@ -1,20 +0,0 @@
----
-layout: default
-title: Contributing
----
-
-# Contributing
-
-1. Fork the repository
-2. Ensure the development dependencies are installed:
-    `% bundle install --with development`
-2. Make changes
-3. Ensure your (new?) tests pass:
-  `% bundle exec rake test`
-4. Autocorrect RubuCop offenses:
-  `% bundle exec rake rubocop:auto_correct`
-5. Fix remaining offenses or have a good excuse not to:
-  `% bundle exec rake rubocop`
-6. Write commit messages at least not worse than mine
-7. Open a pull request on GitHub
-8. Thank you