wgit 0.8.0 → 0.10.2

data/README.md

@@ -8,382 +8,184 @@
 
 ---
 
-Wgit is a Ruby library primarily used for crawling, indexing and searching HTML webpages.
+Wgit is a HTML web crawler, written in Ruby, that allows you to programmatically extract the data you want from the web.
 
-Fundamentally, Wgit is a HTTP indexer/scraper which crawls URL's to retrieve and serialise their page contents for later use. You can use Wgit to scrape entire websites if required. Wgit also provides a means to search indexed documents stored in a database. Therefore, this library provides the main components of a WWW search engine. The Wgit API is easily extended allowing you to pull out the parts of a webpage that are important to you, the code snippets or tables for example. As Wgit is a library, it supports many different use cases including data mining, analytics, web indexing and URL parsing to name a few.
+Wgit was primarily designed to crawl static HTML websites to index and  search their content - providing the basis of any search engine; but Wgit is suitable for many application domains including:
 
-Check out this [demo application](https://search-engine-rb.herokuapp.com) - a search engine (see its [repository](https://github.com/michaeltelford/search_engine)) built using Wgit and Sinatra, deployed to Heroku. Heroku's free tier is used so the initial page load may be slow. Try searching for "Ruby" or something else that's Ruby related.
+- URL parsing
+- Document content extraction (data mining)
+- Crawling entire websites (statistical analysis)
 
-Continue reading the rest of this `README` for more information on Wgit. When you've finished, check out the [wiki](https://github.com/michaeltelford/wgit/wiki).
+Wgit provides a high level, easy-to-use API and DSL that you can use in your own applications and scripts.
 
-## Table Of Contents
-
-1. [Installation](#Installation)
-2. [Basic Usage](#Basic-Usage)
-3. [Documentation](#Documentation)
-4. [Practical Examples](#Practical-Examples)
-5. [Database Example](#Database-Example)
-6. [Extending The API](#Extending-The-API)
-7. [Caveats](#Caveats)
-8. [Executable](#Executable)
-9. [Change Log](#Change-Log)
-10. [License](#License)
-11. [Contributing](#Contributing)
-12. [Development](#Development)
+Check out this [demo search engine](https://search-engine-rb.herokuapp.com) - [built](https://github.com/michaeltelford/search_engine) using Wgit and Sinatra - deployed to [Heroku](https://www.heroku.com/). Heroku's free tier is used so the initial page load may be slow. Try searching for "Matz" or something else that's Ruby related.
 
-## Installation
+## Table Of Contents
 
-Currently, the required Ruby version is:
+1. [Usage](#Usage)
+2. [Why Wgit?](#Why-Wgit)
+3. [Why Not Wgit?](#Why-Not-Wgit)
+4. [Installation](#Installation)
+5. [Documentation](#Documentation)
+6. [Executable](#Executable)
+7. [License](#License)
+8. [Contributing](#Contributing)
+9. [Development](#Development)
 
-`~> 2.5` a.k.a. `>= 2.5 && < 3`
+## Usage
 
-Add this line to your application's `Gemfile`:
+Let's crawl a [quotes website](http://quotes.toscrape.com/) extracting its *quotes* and *authors* using the Wgit DSL:
 
 ```ruby
-gem 'wgit'
-```
+require 'wgit'
+require 'json'
 
-And then execute:
+include Wgit::DSL
 
-    $ bundle
+start  'http://quotes.toscrape.com/tag/humor/'
+follow "//li[@class='next']/a/@href"
 
-Or install it yourself as:
+extract :quotes,  "//div[@class='quote']/span[@class='text']", singleton: false
+extract :authors, "//div[@class='quote']/span/small",          singleton: false
 
-    $ gem install wgit
+quotes = []
 
-Verify the install by using the executable (to start a shell session):
+crawl_site do |doc|
+  doc.quotes.zip(doc.authors).each do |arr|
+    quotes << {
+      quote:  arr.first,
+      author: arr.last
+    }
+  end
+end
 
-    $ wgit
+puts JSON.generate(quotes)
+```
 
-## Basic Usage
+But what if we want to crawl and store the content in a database, so that it can be searched? Wgit makes it easy to index and search HTML using [MongoDB](https://www.mongodb.com/):
 
 ```ruby
 require 'wgit'
 
-crawler = Wgit::Crawler.new # Uses Typhoeus -> libcurl underneath. It's fast!
-url = Wgit::Url.new 'https://wikileaks.org/What-is-Wikileaks.html'
-
-doc = crawler.crawl url # Or use #crawl_site(url) { |doc| ... } etc.
-crawler.last_response.class # => Wgit::Response is a wrapper for Typhoeus::Response.
-
-doc.class # => Wgit::Document
-doc.class.public_instance_methods(false).sort # => [
-# :==, :[], :author, :base, :base_url, :content, :css, :description, :doc, :empty?,
-# :external_links, :external_urls, :html, :internal_absolute_links,
-# :internal_absolute_urls,:internal_links, :internal_urls, :keywords, :links, :score,
-# :search, :search!, :size, :statistics, :stats, :text, :title, :to_h, :to_json,
-# :url, :xpath
-# ]
-
-doc.url   # => "https://wikileaks.org/What-is-Wikileaks.html"
-doc.title # => "WikiLeaks - What is WikiLeaks"
-doc.stats # => {
-          #   :url=>44, :html=>28133, :title=>17, :keywords=>0,
-          #   :links=>35, :text=>67, :text_bytes=>13735
-          # }
-doc.links # => ["#submit_help_contact", "#submit_help_tor", "#submit_help_tips", ...]
-doc.text  # => ["The Courage Foundation is an international organisation that <snip>", ...]
-
-results = doc.search 'corruption' # Searches doc.text for the given query.
-results.first # => "ial materials involving war, spying and corruption.
-              #     It has so far published more"
-```
-
-## Documentation
-
-100% of Wgit's code is documented using [YARD](https://yardoc.org/), deployed to [rubydoc.info](https://www.rubydoc.info/github/michaeltelford/wgit/master). This greatly benefits developers in using Wgit in their own programs. Another good source of information (as to how the library behaves) are the [tests](https://github.com/michaeltelford/wgit/tree/master/test). Also, see the [Practical Examples](#Practical-Examples) section below for real working examples of Wgit in action.
-
-## Practical Examples
-
-Below are some practical examples of Wgit in use. You can copy and run the code for yourself (it's all been tested).
-
-In addition to the practical examples below, the [wiki](https://github.com/michaeltelford/wgit/wiki) contains a useful 'How To' section with more specific usage of Wgit. You should finish reading this `README` first however.
-
-### WWW HTML Indexer
-
-See the [`Wgit::Indexer#index_www`](https://www.rubydoc.info/github/michaeltelford/wgit/master/Wgit%2Eindex_www) documentation and source code for an already built example of a WWW HTML indexer. It will crawl any external URL's (in the database) and index their HTML for later use, be it searching or otherwise. It will literally crawl the WWW forever if you let it!
-
-See the [Database Example](#Database-Example) for information on how to configure a database for use with Wgit.
-
-### Website Downloader
+include Wgit::DSL
 
-Wgit uses itself to download and save fixture webpages to disk (used in tests). See the script [here](https://github.com/michaeltelford/wgit/blob/master/test/mock/save_site.rb) and edit it for your own purposes.
+Wgit.logger.level = Logger::WARN
 
-### Broken Link Finder
+connection_string 'mongodb://user:password@localhost/crawler'
 
-The `broken_link_finder` gem uses Wgit under the hood to find and report a website's broken links. Check out its [repository](https://github.com/michaeltelford/broken_link_finder) for more details.
+start  'http://quotes.toscrape.com/tag/humor/'
+follow "//li[@class='next']/a/@href"
 
-### CSS Indexer
+extract :quotes,  "//div[@class='quote']/span[@class='text']", singleton: false
+extract :authors, "//div[@class='quote']/span/small",          singleton: false
 
-The below script downloads the contents of the first css link found on Facebook's index page.
-
-```ruby
-require 'wgit'
-require 'wgit/core_ext' # Provides the String#to_url and Enumerable#to_urls methods.
-
-crawler = Wgit::Crawler.new
-url = 'https://www.facebook.com'.to_url
-
-doc = crawler.crawl url
-
-# Provide your own xpath (or css selector) to search the HTML using Nokogiri underneath.
-hrefs = doc.xpath "//link[@rel='stylesheet']/@href"
+index_site
+search 'prejudice'
+```
 
-hrefs.class # => Nokogiri::XML::NodeSet
-href = hrefs.first.value # => "https://static.xx.fbcdn.net/rsrc.php/v3/y1/l/0,cross/NvZ4mNTW3Fd.css"
+The `search` call (on the last line) will return and output the results:
 
-css = crawler.crawl href.to_url
-css[0..50] # => "._3_s0._3_s0{border:0;display:flex;height:44px;min-"
+```text
+Quotes to Scrape
+“I am free of all prejudice. I hate everyone equally. ”
+http://quotes.toscrape.com/tag/humor/page/2/
 ```
 
-### Keyword Indexer (SEO Helper)
+Using a MongoDB [client](https://robomongo.org/), we can see that the two web pages have been indexed, along with their extracted *quotes* and *authors*:
+
+![MongoDBClient](https://raw.githubusercontent.com/michaeltelford/wgit/assets/assets/wgit_mongo_index.png)
 
-The below script downloads the contents of several webpages and pulls out their keywords for comparison. Such a script might be used by marketeers for search engine optimisation (SEO) for example.
+The [DSL](https://github.com/michaeltelford/wgit/wiki/How-To-Use-The-DSL) makes it easy to write scripts for experimenting with. Wgit's DSL is simply a wrapper around the underlying classes however. For comparison, here is the above example written using the Wgit API *instead of* the DSL:
 
 ```ruby
 require 'wgit'
-require 'wgit/core_ext' # => Provides the String#to_url and Enumerable#to_urls methods.
-
-my_pages_keywords = ['Everest', 'mountaineering school', 'adventure']
-my_pages_missing_keywords = []
-
-competitor_urls = [
-  'http://altitudejunkies.com',
-  'http://www.mountainmadness.com',
-  'http://www.adventureconsultants.com'
-].to_urls
+require 'json'
 
 crawler = Wgit::Crawler.new
-
-crawler.crawl(*competitor_urls) do |doc|
-  # If there are keywords present in the web document.
-  if doc.keywords.respond_to? :-
-    puts "The keywords for #{doc.url} are: \n#{doc.keywords}\n\n"
-    my_pages_missing_keywords.concat(doc.keywords - my_pages_keywords)
+url     = Wgit::Url.new('http://quotes.toscrape.com/tag/humor/')
+quotes  = []
+
+Wgit::Document.define_extractor(:quotes,  "//div[@class='quote']/span[@class='text']", singleton: false)
+Wgit::Document.define_extractor(:authors, "//div[@class='quote']/span/small",          singleton: false)
+
+crawler.crawl_site(url, follow: "//li[@class='next']/a/@href") do |doc|
+  doc.quotes.zip(doc.authors).each do |arr|
+    quotes << {
+      quote:  arr.first,
+      author: arr.last
+    }
   end
 end
 
-if my_pages_missing_keywords.empty?
-  puts 'Your pages are missing no keywords, nice one!'
-else
-  puts 'Your pages compared to your competitors are missing the following keywords:'
-  puts my_pages_missing_keywords.uniq
-end
-```
-
-## Database Example
-
-The next example requires a configured database instance. The use of a database for Wgit is entirely optional however and isn't required for crawling or URL parsing etc. A database is only needed when indexing (inserting crawled data into the database).
-
-Currently the only supported DBMS is MongoDB. See [MongoDB Atlas](https://www.mongodb.com/cloud/atlas) for a (small) free account or provide your own MongoDB instance. Take a look at this [Docker Hub image](https://hub.docker.com/r/michaeltelford/mongo-wgit) for an already built example of a `mongo` image configured for use with Wgit; the source of which can be found in the [`./docker`](https://github.com/michaeltelford/wgit/tree/master/docker) directory of this repository.
-
-[`Wgit::Database`](https://www.rubydoc.info/github/michaeltelford/wgit/master/Wgit/Database) provides a light wrapper of logic around the `mongo` gem allowing for simple database interactivity and object serialisation. Using Wgit you can index webpages, store them in a database and then search through all that's been indexed; quickly and easily.
-
-### Versioning
-
-The following versions of MongoDB are currently supported:
-
-| Gem    | Database |
-| ------ | -------- |
-| ~> 2.9 | ~> 4.0   |
-
-### Data Model
-
-The data model for Wgit is deliberately simplistic. The MongoDB collections consist of:
-
-| Collection  | Purpose                                         |
-| ----------- | ----------------------------------------------- |
-| `urls`      | Stores URL's to be crawled at a later date      |
-| `documents` | Stores web documents after they've been crawled |
-
-Wgit provides respective Ruby classes for each collection object, allowing for serialisation.
-
-### Configuring MongoDB
-
-Follow the steps below to configure MongoDB for use with Wgit. This is only required if you want to read/write database records using your own (manually configured) instance of Mongo DB.
-
-1) Create collections for: `urls` and `documents`.
-2) Add a [*unique index*](https://docs.mongodb.com/manual/core/index-unique/) for the `url` field in **both** collections using:
-
-| Collection  | Fields              | Options             |
-| ----------- | ------------------- | ------------------- |
-| `urls`      | `{ "url" : 1 }`     | `{ unique : true }` |
-| `documents` | `{ "url.url" : 1 }` | `{ unique : true }` |
-
-3) Enable `textSearchEnabled` in MongoDB's configuration (if not already so - it's typically enabled by default).
-4) Create a [*text search index*](https://docs.mongodb.com/manual/core/index-text/#index-feature-text) for the `documents` collection using:
-```json
-{
-  "text": "text",
-  "author": "text",
-  "keywords": "text",
-  "title": "text"
-}
+puts JSON.generate(quotes)
 ```
 
-**Note**: The *text search index* lists all document fields to be searched by MongoDB when calling `Wgit::Database#search`. Therefore, you should append this list with any other fields that you want searched. For example, if you [extend the API](#Extending-The-API) then you might want to search your new fields in the database by adding them to the index above.
+## Why Wgit?
 
-### Code Example
+There are many [other HTML crawlers](https://awesome-ruby.com/#-web-crawling) out there so why use Wgit?
 
-The below script shows how to use Wgit's database functionality to index and then search HTML documents stored in the database. If you're running the code for yourself, remember to replace the database [connection string](https://docs.mongodb.com/manual/reference/connection-string/) with your own.
+- Wgit has excellent unit testing, 100% documentation coverage and follows [semantic versioning](https://semver.org/) rules.
+- Wgit excels at crawling an entire website's HTML out of the box. Many alternative crawlers require you to provide the `xpath` needed to *follow* the next URLs to crawl. Wgit by default, crawls the entire site by extracting its internal links pointing to the same host.
+- Wgit allows you to define content *extractors* that will fire on every subsequent crawl; be it a single URL or an entire website. This enables you to focus on the content you want.
+- Wgit can index (crawl and store) HTML to a database making it a breeze to build custom search engines. You can also specify which page content gets searched, making the search more meaningful. For example, here's a script that will index the Wgit [wiki](https://github.com/michaeltelford/wgit/wiki) articles:
 
 ```ruby
 require 'wgit'
 
-### CONNECT TO THE DATABASE ###
-
-# In the absence of a connection string parameter, ENV['WGIT_CONNECTION_STRING'] will be used.
-db = Wgit::Database.connect '<your_connection_string>'
+ENV['WGIT_CONNECTION_STRING'] = 'mongodb://user:password@localhost/crawler'
 
-### SEED SOME DATA ###
+wiki = Wgit::Url.new('https://github.com/michaeltelford/wgit/wiki')
 
-# Here we create our own document rather than crawling the web (which works in the same way).
-# We provide the web page's URL and HTML Strings.
-doc = Wgit::Document.new(
-  'http://test-url.com',
-  "<html><p>How now brown cow.</p><a href='http://www.google.co.uk'>Click me!</a></html>"
-)
-db.insert doc
-
-### SEARCH THE DATABASE ###
-
-# Searching the database returns Wgit::Document's which have fields containing the query.
-query = 'cow'
-results = db.search query
-
-# By default, the MongoDB ranking applies i.e. results.first has the most hits.
-# Because results is an Array of Wgit::Document's, we can custom sort/rank e.g.
-# `results.sort_by! { |doc| doc.url.crawl_duration }` ranks via page load times with
-# results.first being the fastest. Any Wgit::Document attribute can be used, including
-# those you define yourself by extending the API.
-
-top_result = results.first
-top_result.class           # => Wgit::Document
-doc.url == top_result.url  # => true
-
-### PULL OUT THE BITS THAT MATCHED OUR QUERY ###
-
-# Searching each result gives the matching text snippets from that Wgit::Document.
-top_result.search(query).first # => "How now brown cow."
-
-### SEED URLS TO BE CRAWLED LATER ###
+# Only index the most recent of each wiki article, ignoring the rest of Github.
+opts = {
+  allow_paths:    'michaeltelford/wgit/wiki/*',
+  disallow_paths: 'michaeltelford/wgit/wiki/*/_history'
+}
 
-db.insert top_result.external_links
-urls_to_crawl = db.uncrawled_urls # => Results will include top_result.external_links.
+indexer = Wgit::Indexer.new
+indexer.index_site(wiki, **opts)
 ```
 
-## Extending The API
-
-Document serialising in Wgit is the means of downloading a web page and serialising parts of its content into accessible `Wgit::Document` attributes/methods. For example, `Wgit::Document#author` will return you the webpage's xpath value of `meta[@name='author']`.
-
-There are two ways to extend the Document serialising behaviour of Wgit for your own means:
-
-1. Add additional **textual** content to `Wgit::Document#text`.
-2. Define `Wgit::Document` instance methods for specific HTML **elements**.
+## Why Not Wgit?
 
-Below describes these two methods in more detail.
+So why might you not use Wgit, I hear you ask?
 
-### 1. Extending The Default Text Elements
+- Wgit doesn't allow for webpage interaction e.g. signing in as a user. There are better gems out there for that.
+- Wgit can parse a crawled page's Javascript, but it doesn't do so by default. If your crawls are JS heavy then you might best consider a pure browser-based crawler instead.
+- Wgit while fast (using `libcurl` for HTTP etc.), isn't multi-threaded; so each URL gets crawled sequentially. You could hand each crawled document to a worker thread for processing - but if you need concurrent crawling then you should consider something else.
 
-Wgit contains a set of `Wgit::Document.text_elements` defining which HTML elements contain text on a page; which in turn are serialised. Once serialised you can process this text content via methods like `Wgit::Document#text` and `Wgit::Document#search` etc.
-
-The below code example shows how to extract additional text from a webpage:
-
-```ruby
-require 'wgit'
-
-# The default text_elements cover most visible page text but let's say we
-# have a <table> element with text content that we want.
-Wgit::Document.text_elements << :table
-
-doc = Wgit::Document.new(
-  'http://some_url.com',
-  <<~HTML
-  <html>
-    <p>Hello world!</p>
-    <table>My table</table>
-  </html>
-  HTML
-)
-
-# Now every crawled Document#text will include <table> text content.
-doc.text            # => ["Hello world!", "My table"]
-doc.search('table') # => ["My table"]
-```
+## Installation
 
-**Note**: This only works for *textual* page content. For more control over the serialised *elements* themselves, see below.
+Only MRI Ruby is tested and supported, but Wgit may work with other Ruby implementations.
 
-### 2. Serialising Specific HTML Elements (via Document Extensions)
+Currently, the required MRI Ruby version is:
 
-Wgit provides some [default extensions](https://github.com/michaeltelford/wgit/blob/master/lib/wgit/document_extensions.rb) to extract a page's text, links etc. This of course is often not enough given the nature of the WWW and the differences from one webpage to the next.
+`ruby '>= 2.6', '< 4'`
 
-Therefore, you can define a Document extension for each HTML element(s) that you want to extract and serialise into a `Wgit::Document` instance variable, equipped with a getter method. Once an extension is defined, all crawled Documents will contain your extracted content.
+### Using Bundler
 
-Here's how to add a Document extension to serialise a specific page element:
+    $ bundle add wgit
 
-```ruby
-require 'wgit'
-
-# Let's get all the page's <table> elements.
-Wgit::Document.define_extension(
-  :tables,                  # Wgit::Document#tables will return the page's tables.
-  '//table',                # The xpath to extract the tables.
-  singleton: false,         # True returns the first table found, false returns all.
-  text_content_only: false, # True returns the table text, false returns the Nokogiri object.
-) do |tables|
-  # Here we can inspect/manipulate the tables before they're set as Wgit::Document#tables.
-  tables
-end
+### Using RubyGems
 
-# Our Document has a table which we're interested in. Note it doesn't matter how the Document
-# is initialised e.g. manually (as below) or via Wgit::Crawler methods etc.
-doc = Wgit::Document.new(
-  'http://some_url.com',
-  <<~HTML
-  <html>
-    <p>Hello world! Welcome to my site.</p>
-    <table>
-      <tr><th>Name</th><th>Age</th></tr>
-      <tr><td>Socrates</td><td>101</td></tr>
-      <tr><td>Plato</td><td>106</td></tr>
-    </table>
-    <p>I hope you enjoyed your visit :-)</p>
-  </html>
-  HTML
-)
-
-# Call our newly defined method to obtain the table data we're interested in.
-tables = doc.tables
-
-# Both the collection and each table within the collection are plain Nokogiri objects.
-tables.class       # => Nokogiri::XML::NodeSet
-tables.first.class # => Nokogiri::XML::Element
-
-# Note, the Document's stats now include our 'tables' extension.
-doc.stats # => {
-#   :url=>19, :html=>242, :links=>0, :text=>8, :text_bytes=>91, :tables=>1
-# }
-```
-
-See the [Wgit::Document.define_extension](https://www.rubydoc.info/github/michaeltelford/wgit/master/Wgit%2FDocument.define_extension) docs for more information.
+    $ gem install wgit
 
-**Extension Notes**:
+### Verify
 
-- It's recommended that URL's be mapped into `Wgit::Url` objects. `Wgit::Url`'s are treated as Strings when being inserted into the database.
-- A `Wgit::Document` extension (once initialised) will become a Document instance variable, meaning that the value will be inserted into the Database if it's a primitive type e.g. `String`, `Array` etc. Complex types e.g. Ruby objects won't be inserted. It's up to you to ensure the data you want inserted, can be inserted.
-- Once inserted into the Database, you can search a `Wgit::Document`'s extension attributes by updating the Database's *text search index*. See the [Database Example](#Database-Example) for more information.
+    $ wgit
 
-## Caveats
+Calling the installed executable will start an REPL session.
 
-Below are some points to keep in mind when using Wgit:
+## Documentation
 
-- All absolute `Wgit::Url`'s must be prefixed with an appropiate protocol e.g. `https://` etc.
-- By default, up to 5 URL redirects will be followed; this is [configurable](https://www.rubydoc.info/github/michaeltelford/wgit/master/Wgit/Crawler#redirect_limit-instance_method) however.
-- IRI's (URL's containing non ASCII characters) **are** supported and will be normalised/escaped prior to being crawled.
+- [Getting Started](https://github.com/michaeltelford/wgit/wiki/Getting-Started)
+- [Wiki](https://github.com/michaeltelford/wgit/wiki)
+- [API Yardocs](https://www.rubydoc.info/gems/wgit)
+- [CHANGELOG](https://github.com/michaeltelford/wgit/blob/master/CHANGELOG.md)
 
 ## Executable
 
-Installing the Wgit gem also adds the `wgit` executable to your `$PATH`. The executable launches an interactive shell session with the Wgit gem already loaded; making it super easy to index and search from the command line without the need for scripts.
+Installing the Wgit gem adds a `wgit` executable to your `$PATH`. The executable launches an interactive REPL session with the Wgit gem already loaded; making it super easy to index and search from the command line without the need for scripts.
 
 The `wgit` executable does the following things (in order):
 
@@ -391,21 +193,7 @@ The `wgit` executable does the following things (in order):
 2. `eval`'s a `.wgit.rb` file (if one exists in either the local or home directory, which ever is found first)
 3. Starts an interactive shell (using `pry` if it's installed, or `irb` if not)
 
-The `.wgit.rb` file can be used to seed fixture data or define helper functions for the session. For example, you could define a function which indexes your website for quick and easy searching everytime you start a new session. **Note** that variables should either be instance variables (e.g. `@url`) or be accessed via a getter method (e.g. `def url; ...; end`).
-
-## Change Log
-
-See the [CHANGELOG.md](https://github.com/michaeltelford/wgit/blob/master/CHANGELOG.md) for differences (including any breaking changes) between releases of Wgit.
-
-### Gem Versioning
-
-The `wgit` gem follows these versioning rules:
-
-- The version format is `MAJOR.MINOR.PATCH` e.g. `0.1.0`.
-- Since the gem hasn't reached `v1.0.0` yet, slightly different semantic versioning rules apply.
-- The `PATCH` represents *non breaking changes* while the `MINOR` represents *breaking changes* e.g. updating from version `0.1.0` to `0.2.0` will likely introduce breaking changes necessitating updates to your codebase.
-- To determine what changes are needed, consult the `CHANGELOG.md`. If you need help, raise an issue.
-- Once `wgit v1.0.0` is released, *normal* [semantic versioning](https://semver.org/) rules will apply e.g. only a `MAJOR` version change should introduce breaking changes.
+The `.wgit.rb` file can be used to seed fixture data or define helper functions for the session. For example, you could define a function which indexes your website for quick and easy searching everytime you start a new session.
 
 ## License
 
@@ -431,14 +219,14 @@ And you're good to go!
 
 ### Tooling
 
-Wgit uses the [`toys`](https://github.com/dazuma/toys) gem (instead of Rake) for task invocation e.g. running the tests etc. For a full list of available tasks AKA tools, run `toys --tools`. You can search for a tool using `toys -s tool_name`. The most commonly used tools are listed below...
+Wgit uses the [`toys`](https://github.com/dazuma/toys) gem (instead of Rake) for task invocation. For a full list of available tasks a.k.a. tools, run `toys --tools`. You can search for a tool using `toys -s tool_name`. The most commonly used tools are listed below...
 
-Run `toys db` to see a list of database related tools, enabling you to run a Mongo DB instance locally using Docker. Run `toys test` to execute the tests (or `toys test smoke` for a faster running subset that doesn't require a database).
+Run `toys db` to see a list of database related tools, enabling you to run a Mongo DB instance locally using Docker. Run `toys test` to execute the tests.
 
-To generate code documentation run `toys yardoc`. To browse the generated documentation in a browser run `toys yardoc --serve`. You can also use the `yri` command line tool e.g. `yri Wgit::Crawler#crawl_site` etc.
+To generate code documentation locally, run `toys yardoc`. To browse the docs in a browser run `toys yardoc --serve`. You can also use the `yri` command line tool e.g. `yri Wgit::Crawler#crawl_site` etc.
 
-To install this gem onto your local machine, run `toys install`.
+To install this gem onto your local machine, run `toys install` and follow the prompt.
 
 ### Console
 
-You can run `toys console` for an interactive shell using the `./bin/wgit` executable. The `toys setup` task will have created a `.env` and `.wgit.rb` file which gets loaded by the executable. You can use the contents of this [gist](https://gist.github.com/michaeltelford/b90d5e062da383be503ca2c3a16e9164) to turn the executable into a development console. It defines some useful functions, fixtures and connects to the database etc. Don't forget to set the `WGIT_CONNECTION_STRING` in the `.env` file.
+You can run `toys console` for an interactive shell using the `./bin/wgit` executable. The `toys setup` task will have created an `.env` and `.wgit.rb` file which get loaded by the executable. You can use the contents of this [gist](https://gist.github.com/michaeltelford/b90d5e062da383be503ca2c3a16e9164) to turn the executable into a development console. It defines some useful functions, fixtures and connects to the database etc. Don't forget to set the `WGIT_CONNECTION_STRING` in the `.env` file.

data/bin/wgit

@@ -2,18 +2,22 @@
 
 require 'wgit'
 
-# Eval .wgit.rb file (if it exists).
-def eval_wgit
-  puts 'Searching for .wgit.rb in local and home directories...'
+# Eval .wgit.rb file (if it exists somewhere).
+def eval_wgit(filepath = nil)
+  puts 'Searching for .wgit.rb file in local and home directories...'
 
-  ['.', Dir.home].each do |dir|
+  [filepath, Dir.pwd, Dir.home].each do |dir|
     path = "#{dir}/.wgit.rb"
     next unless File.exist?(path)
 
-    puts "Eval'ing #{path} (call `eval_wgit` after changes)"
+    puts "Eval'ing #{path}"
+    puts 'Call `eval_wgit` after changes to re-eval the file'
     eval(File.read(path))
+
     break
   end
+
+  nil
 end
 
 eval_wgit

data/lib/wgit/assertable.rb

@@ -6,7 +6,7 @@ module Wgit
     # Default type fail message.
     DEFAULT_TYPE_FAIL_MSG = 'Expected: %s, Actual: %s'
     # Wrong method message.
-    WRONG_METHOD_MSG = 'arr must be Enumerable, use a different method'
+    NON_ENUMERABLE_MSG = 'Expected an Enumerable responding to #each, not: %s'
     # Default duck fail message.
     DEFAULT_DUCK_FAIL_MSG = "%s doesn't respond_to? %s"
     # Default required keys message.
@@ -42,7 +42,7 @@ present: %s"
     # @raise [StandardError] If the assertion fails.
     # @return [Object] The given arr on successful assertion.
     def assert_arr_types(arr, type_or_types, msg = nil)
-      raise WRONG_METHOD_MSG unless arr.respond_to?(:each)
+      raise format(NON_ENUMERABLE_MSG, arr.class) unless arr.respond_to?(:each)
 
       arr.each { |obj| assert_types(obj, type_or_types, msg) }
     end
@@ -56,7 +56,7 @@ present: %s"
     # @raise [StandardError] If the assertion fails.
     # @return [Object] The given obj_or_objs on successful assertion.
     def assert_respond_to(obj_or_objs, methods, msg = nil)
-      methods = [methods] unless methods.respond_to?(:all?)
+      methods = *methods
 
       if obj_or_objs.respond_to?(:each)
         obj_or_objs.each { |obj| _assert_respond_to(obj, methods, msg) }

data/lib/wgit/base.rb

@@ -0,0 +1,39 @@
+module Wgit
+  # Class to inherit from, as an alternative form of using the `Wgit::DSL`.
+  # All subclasses must define a `#parse(doc, &block)` method.
+  class Base
+    extend Wgit::DSL
+
+    # Runs once before the crawl/index is run. Override as needed.
+    def setup; end
+
+    # Runs once after the crawl/index is complete. Override as needed.
+    def teardown; end
+
+    # Runs the crawl/index passing each crawled `Wgit::Document` and the given
+    # block to the subclass's `#parse` method.
+    def self.run(&block)
+      crawl_method = @method || :crawl
+      obj = new
+
+      unless obj.respond_to?(:parse)
+        raise "#{obj.class} must respond_to? #parse(doc, &block)"
+      end
+
+      obj.setup
+      send(crawl_method) { |doc| obj.parse(doc, &block) }
+      obj.teardown
+
+      obj
+    end
+
+    # Sets the crawl/index method to call when `Base.run` is called.
+    # The mode method must match one defined in the `Wgit::Crawler` or
+    # `Wgit::Indexer` class.
+    #
+    # @param method [Symbol] The crawl/index method to call.
+    def self.mode(method)
+      @method = method
+    end
+  end
+end