wgit 0.7.0 → 0.10.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 29d37a4a0f013fec64625d8fe5798ae2d062ae6f213811c51d223de311e16707
-  data.tar.gz: 213f6c43ccbb1fcc5c487a2bd5f31493506ab2320168562f8e1b6887cccc07b8
+  metadata.gz: d4f448212c206f3384275cb2df9a0c81139323cef6efa17dcfc1d907c037c08d
+  data.tar.gz: 28a12eee9935860c7c25b7500a718a6aa066e610a83dbc71003be7f63bb8a3dc
 SHA512:
-  metadata.gz: acd321f3ba039e6f54dd8a36a3e4ebec1fb40f1cda5ee1c982df4be22ee6d463f829c72f011ff959a4a4d2651676dc2d31866a273b60d3e5e630ccf77b3d7cbe
-  data.tar.gz: d0908d28e6fdaec440209479f75945672807cf3e9359fb8bd8f6cc9de45568a341ac0204ba5f50a2f6569b4a29f4f7ac3088353a35f2c5091a567af469027aab
+  metadata.gz: 0dba9a82f73f3872f4c845b1c6c546efaef0afff1563b89b13a50efc429fe86bfe3590811d127df777569cc802904676d3e9a127705a7f10ce9b895e4b581c7d
+  data.tar.gz: d1c90814c1eab72a3cf6d0ea0f9f3bc239e71fc575b6ff89382135dad5d72bf770a05d75219cd78e16b5c686e8f15c7a5a6c3af89319f64c90e976049f60e7b5

data/.yardopts

@@ -1,5 +1,5 @@
 --readme README.md
---title 'Wgit API Documentation'
+--title 'Wgit Gem Documentation'
 --charset utf-8
 --markup markdown
 --output .doc

data/CHANGELOG.md

@@ -9,6 +9,78 @@
 - ...
 ---
 
+## v0.10.1
+### Added
+- Support for Ruby 3.
+### Changed/Removed
+- Removed support for Ruby 2.5 (as it's too old).
+### Fixed
+- ...
+---
+
+## v0.10.0
+### Added
+- `Wgit::Url#scheme_relative?` method.
+### Changed/Removed
+- Breaking change: Changed method signature of `Wgit::Url#prefix_scheme` by making the previously named parameter a defaulted positional parameter. Remove the `protocol` named parameter for the old behaviour.
+### Fixed
+- [Scheme-relative bug](https://github.com/michaeltelford/wgit/issues/10) by adding support for scheme-relative URL's.
+---
+
+## v0.9.0
+This release is a big one with the introduction of a `Wgit::DSL` and Javascript parse support. The `README` has been revamped as a result with new usage examples. And all of the wiki articles have been updated to reflect the latest code base.
+### Added
+- `Wgit::DSL` module providing a wrapper around the underlying classes and methods. Check out the `README` for example usage.
+- `Wgit::Crawler#parse_javascript` which when set to `true` uses Chrome to parse a page's Javascript before returning the fully rendered HTML. This feature is disabled by default.
+- `Wgit::Base` class to inherit from, acting as an alternative form of using the DSL.
+- `Wgit::Utils.sanitize` which calls `.sanitize_*` underneath.
+- `Wgit::Crawler#crawl_site` now has a `follow:` named param - if set, it's xpath value is used to retrieve the next urls to crawl. Otherwise the `:default` is used (as it was before). Use this to override how the site is crawled.
+- `Wgit::Database` methods: `#clear_urls`, `#clear_docs`, `#clear_db`, `#text_index`, `#text_index=`, `#create_collections`, `#create_unique_indexes`, `#docs`, `#get`, `#exists?`, `#delete`, `#upsert`.
+- `Wgit::Database#clear_db!` alias.
+- `Wgit::Document` methods: `#at_xpath`, `#at_css` - which call nokogiri underneath.
+- `Wgit::Document#extract` method to perform one off content extractions.
+- `Wgit::Indexer#index_urls` method which can index several urls in one call.
+- `Wgit::Url` methods: `#to_user`, `#to_password`, `#to_sub_domain`, `#to_port`, `#omit_origin`, `#index?`.
+### Changed/Removed
+- Breaking change: Moved all `Wgit.index*` convienence methods into `Wgit::DSL`.
+- Breaking change: Removed `Wgit::Url#normalise`, use `#normalize` instead.
+- Breaking change: Removed `Wgit::Database#num_documents`, use `#num_docs` instead.
+- Breaking change: Removed `Wgit::Database#length` and `#count`, use `#size` instead.
+- Breaking change: Removed `Wgit::Database#document?`, use `#doc?` instead.
+- Breaking change: Renamed `Wgit::Indexer#index_page` to `#index_url`.
+- Breaking change: Renamed `Wgit::Url.parse_or_nil` to be `.parse?`.
+- Breaking change: Renamed `Wgit::Utils.process_*` to be `.sanitize_*`.
+- Breaking change: Renamed `Wgit::Utils.remove_non_bson_types` to be `Wgit::Model.select_bson_types`.
+- Breaking change: Changed `Wgit::Indexer.index*` named param default from `insert_externals: true` to `false`. Explicitly set it to `true` for the old behaviour.
+- Breaking change: Renamed `Wgit::Document.define_extension` to `define_extractor`. Same goes for `remove_extension -> remove_extractor` and `extensions -> extractors`. See the docs for more information.
+- Breaking change: Renamed `Wgit::Document#doc` to `#parser`.
+- Breaking change: Renamed `Wgit::Crawler#time_out` to `#timeout`. Same goes for the named param passed to `Wgit::Crawler.initialize`.
+- Breaking change: Refactored `Wgit::Url#relative?` now takes `:origin` instead of `:base` which takes the port into account. This has a knock on effect for some other methods too - check the docs if you're getting parameter errors.
+- Breaking change: Renamed `Wgit::Url#prefix_base` to `#make_absolute`.
+- Updated `Utils.printf_search_results` to return the number of results.
+- Updated `Wgit::Indexer.new` which can now be called without parameters - the first param (for a database) now defaults to `Wgit::Database.new` which works if `ENV['WGIT_CONNECTION_STRING']` is set.
+- Updated `Wgit::Document.define_extractor` to define a setter method (as well as the usual getter method).
+- Updated `Wgit::Document#search` to support a `Regexp` query (in addition to a String).
+### Fixed
+- [Re-indexing bug](https://github.com/michaeltelford/wgit/issues/8) so that indexing content a 2nd time will update it in the database - before it simply disgarded the document.
+- `Wgit::Crawler#crawl_site` params `allow/disallow_paths` values can now start with a `/`.
+---
+
+## v0.8.0
+### Added
+- To the range of `Wgit::Document.text_elements`. Now (only and) all visible page text should be extracted into `Wgit::Document#text` successfully.
+- `Wgit::Document#description` default extension.
+- `Wgit::Url.parse_or_nil` method.
+### Changed/Removed
+- Breaking change: Renamed `Document#stats[:text_snippets]` to be `:text`.
+- Breaking change: `Wgit::Document.define_extension`'s block return value now becomes the `var` value, even when `nil` is returned. This allows `var` to be set to `nil`.
+- Potential breaking change: Renamed `Wgit::Response#crawl_time` (alias) to be `#crawl_duration`.
+- Updated `Wgit::Crawler::SUPPORTED_FILE_EXTENSIONS` to be `Wgit::Crawler.supported_file_extensions`, making it configurable. Now you can add your own URL extensions if needed.
+- Updated the Wgit core extension `String#to_url` to use `Wgit::Url.parse` allowing instances of `Wgit::Url` to returned as is. This also affects `Enumerable#to_urls` in the same way.
+### Fixed
+- An issue where too much `Wgit::Document#text` was being extracted from the HTML. This was fixed by reverting the recent commit: "Document.text_elements_xpath is now `//*/text()`".
+---
+
 ## v0.7.0
 ### Added
 - `Wgit::Indexer.new` optional `crawler:` named param.
@@ -58,7 +130,7 @@
 - `Wgit::Response` class containing adapter agnostic HTTP response logic.
 ### Changed/Removed
 - Breaking changes: Removed `Wgit::Document#date_crawled` and `#crawl_duration` because both of these methods exist on the `Wgit::Document#url`. Instead, use `doc.url.date_crawled` etc.
-- Breaking changes: Added to and moved `Document.define_extension` block params, it's now `|value, source, type|`. The `source` is not what it used to be; it's now `type` - of either `:document` or `:object`. Confused? See the [docs](https://www.rubydoc.info/github/michaeltelford/wgit/master).
+- Breaking changes: Added to and moved `Document.define_extension` block params, it's now `|value, source, type|`. The `source` is not what it used to be; it's now `type` - of either `:document` or `:object`. Confused? See the [docs](https://www.rubydoc.info/gems/wgit).
 - Breaking changes: Changed `Wgit::Url#prefix_protocol` so that it no longer modifies the receiver.
 - Breaking changes: Updated `Wgit::Url#to_anchor` and `#to_query` logic to align with that of `Addressable::URI` e.g. the anchor value no longer contains `#` prefix; and the query value no longer contains `?` prefix.
 - Breaking changes: Renamed `Wgit::Url` methods containing `anchor` to now be named `fragment` e.g. `to_anchor` is now called `to_fragment` and `without_anchor` is `without_fragment` etc.
@@ -106,7 +178,7 @@
 ---
 
 ## v0.2.0
-This version of Wgit see's a major refactor of the code base involving multiple changes to method names and their signatures (optional parameters turned into named parameters in most cases). A list of the breaking changes are below including how to fix any breakages; but if you're having issues with the upgrade see the documentation at: https://www.rubydoc.info/github/michaeltelford/wgit/master
+This version of Wgit see's a major refactor of the code base involving multiple changes to method names and their signatures (optional parameters turned into named parameters in most cases). A list of the breaking changes are below including how to fix any breakages; but if you're having issues with the upgrade see the documentation at: https://www.rubydoc.info/gems/wgit
 ### Added
 - `Wgit::Url#absolute?` method.
 - `Wgit::Url#relative? base: url` support.

data/LICENSE.txt

@@ -1,6 +1,6 @@
 The MIT License (MIT)
 
-Copyright (c) 2016 - 2019 Michael Telford
+Copyright (c) 2016 - 2020 Michael Telford
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

data/README.md

@@ -8,346 +8,184 @@
 
 ---
 
-Wgit is a Ruby library primarily used for crawling, indexing and searching HTML webpages.
+Wgit is a HTML web crawler, written in Ruby, that allows you to programmatically extract the data you want from the web.
 
-Fundamentally, Wgit is a HTTP indexer/scraper which crawls URL's to retrieve and serialise their page contents for later use. You can use Wgit to scrape entire websites if required. Wgit also provides a means to search indexed documents stored in a database. Therefore, this library provides the main components of a WWW search engine. The Wgit API is easily extended allowing you to pull out the parts of a webpage that are important to you, the code snippets or tables for example. As Wgit is a library, it supports many different use cases including data mining, analytics, web indexing and URL parsing to name a few.
+Wgit was primarily designed to crawl static HTML websites to index and  search their content - providing the basis of any search engine; but Wgit is suitable for many application domains including:
 
-Check out this [demo application](https://search-engine-rb.herokuapp.com) - a search engine (see its [repository](https://github.com/michaeltelford/search_engine)) built using Wgit and Sinatra, deployed to Heroku. Heroku's free tier is used so the initial page load may be slow. Try searching for "Ruby" or something else that's Ruby related.
+- URL parsing
+- Document content extraction (data mining)
+- Crawling entire websites (statistical analysis)
 
-Continue reading the rest of this `README` for more information on Wgit. When you've finished, check out the [wiki](https://github.com/michaeltelford/wgit/wiki).
+Wgit provides a high level, easy-to-use API and DSL that you can use in your own applications and scripts.
 
-## Table Of Contents
-
-1. [Installation](#Installation)
-2. [Basic Usage](#Basic-Usage)
-3. [Documentation](#Documentation)
-4. [Practical Examples](#Practical-Examples)
-5. [Database Example](#Database-Example)
-6. [Extending The API](#Extending-The-API)
-7. [Caveats](#Caveats)
-8. [Executable](#Executable)
-9. [Change Log](#Change-Log)
-10. [License](#License)
-11. [Contributing](#Contributing)
-12. [Development](#Development)
+Check out this [demo search engine](https://search-engine-rb.herokuapp.com) - [built](https://github.com/michaeltelford/search_engine) using Wgit and Sinatra - deployed to [Heroku](https://www.heroku.com/). Heroku's free tier is used so the initial page load may be slow. Try searching for "Matz" or something else that's Ruby related.
 
-## Installation
+## Table Of Contents
 
-Currently, the required Ruby version is:
+1. [Usage](#Usage)
+2. [Why Wgit?](#Why-Wgit)
+3. [Why Not Wgit?](#Why-Not-Wgit)
+4. [Installation](#Installation)
+5. [Documentation](#Documentation)
+6. [Executable](#Executable)
+7. [License](#License)
+8. [Contributing](#Contributing)
+9. [Development](#Development)
 
-`~> 2.5` a.k.a. `>= 2.5 && < 3`
+## Usage
 
-Add this line to your application's `Gemfile`:
+Let's crawl a [quotes website](http://quotes.toscrape.com/) extracting its *quotes* and *authors* using the Wgit DSL:
 
 ```ruby
-gem 'wgit'
-```
+require 'wgit'
+require 'json'
 
-And then execute:
+include Wgit::DSL
 
-    $ bundle
+start  'http://quotes.toscrape.com/tag/humor/'
+follow "//li[@class='next']/a/@href"
 
-Or install it yourself as:
+extract :quotes,  "//div[@class='quote']/span[@class='text']", singleton: false
+extract :authors, "//div[@class='quote']/span/small",          singleton: false
 
-    $ gem install wgit
+quotes = []
 
-Verify the install by using the executable (to start a shell session):
+crawl_site do |doc|
+  doc.quotes.zip(doc.authors).each do |arr|
+    quotes << {
+      quote:  arr.first,
+      author: arr.last
+    }
+  end
+end
 
-    $ wgit
+puts JSON.generate(quotes)
+```
 
-## Basic Usage
+But what if we want to crawl and store the content in a database, so that it can be searched? Wgit makes it easy to index and search HTML using [MongoDB](https://www.mongodb.com/):
 
 ```ruby
 require 'wgit'
 
-crawler = Wgit::Crawler.new # Uses Typhoeus -> libcurl underneath. It's fast!
-url = Wgit::Url.new 'https://wikileaks.org/What-is-Wikileaks.html'
-
-doc = crawler.crawl url # Or use #crawl_site(url) { |doc| ... } etc.
-crawler.last_response.class # => Wgit::Response is a wrapper for Typhoeus::Response.
-
-doc.class # => Wgit::Document
-doc.class.public_instance_methods(false).sort # => [
-# :==, :[], :author, :base, :base_url, :content, :css, :doc, :empty?, :external_links,
-# :external_urls, :html, :internal_absolute_links, :internal_absolute_urls,
-# :internal_links, :internal_urls, :keywords, :links, :score, :search, :search!,
-# :size, :statistics, :stats, :text, :title, :to_h, :to_json, :url, :xpath
-# ]
-
-doc.url   # => "https://wikileaks.org/What-is-Wikileaks.html"
-doc.title # => "WikiLeaks - What is WikiLeaks"
-doc.stats # => {
-          #   :url=>44, :html=>28133, :title=>17, :keywords=>0,
-          #   :links=>35, :text_snippets=>67, :text_bytes=>13735
-          # }
-doc.links # => ["#submit_help_contact", "#submit_help_tor", "#submit_help_tips", ...]
-doc.text  # => ["The Courage Foundation is an international organisation that <snip>", ...]
-
-results = doc.search 'corruption' # Searches doc.text for the given query.
-results.first # => "ial materials involving war, spying and corruption.
-              #     It has so far published more"
-```
-
-## Documentation
-
-100% of Wgit's code is documented using [YARD](https://yardoc.org/), deployed to [rubydoc.info](https://www.rubydoc.info/github/michaeltelford/wgit/master). This greatly benefits developers in using Wgit in their own programs. Another good source of information (as to how the library behaves) are the [tests](https://github.com/michaeltelford/wgit/tree/master/test). Also, see the [Practical Examples](#Practical-Examples) section below for real working examples of Wgit in action.
-
-## Practical Examples
-
-Below are some practical examples of Wgit in use. You can copy and run the code for yourself (it's all been tested).
-
-In addition to the practical examples below, the [wiki](https://github.com/michaeltelford/wgit/wiki) contains a useful 'How To' section with more specific usage of Wgit. You should finish reading this `README` first however.
-
-### WWW HTML Indexer
+include Wgit::DSL
 
-See the [`Wgit::Indexer#index_www`](https://www.rubydoc.info/github/michaeltelford/wgit/master/Wgit%2Eindex_www) documentation and source code for an already built example of a WWW HTML indexer. It will crawl any external URL's (in the database) and index their HTML for later use, be it searching or otherwise. It will literally crawl the WWW forever if you let it!
+Wgit.logger.level = Logger::WARN
 
-See the [Database Example](#Database-Example) for information on how to configure a database for use with Wgit.
+connection_string 'mongodb://user:password@localhost/crawler'
 
-### Website Downloader
+start  'http://quotes.toscrape.com/tag/humor/'
+follow "//li[@class='next']/a/@href"
 
-Wgit uses itself to download and save fixture webpages to disk (used in tests). See the script [here](https://github.com/michaeltelford/wgit/blob/master/test/mock/save_site.rb) and edit it for your own purposes.
+extract :quotes,  "//div[@class='quote']/span[@class='text']", singleton: false
+extract :authors, "//div[@class='quote']/span/small",          singleton: false
 
-### Broken Link Finder
-
-The `broken_link_finder` gem uses Wgit under the hood to find and report a website's broken links. Check out its [repository](https://github.com/michaeltelford/broken_link_finder) for more details.
-
-### CSS Indexer
-
-The below script downloads the contents of the first css link found on Facebook's index page.
-
-```ruby
-require 'wgit'
-require 'wgit/core_ext' # Provides the String#to_url and Enumerable#to_urls methods.
-
-crawler = Wgit::Crawler.new
-url = 'https://www.facebook.com'.to_url
-
-doc = crawler.crawl url
-
-# Provide your own xpath (or css selector) to search the HTML using Nokogiri underneath.
-hrefs = doc.xpath "//link[@rel='stylesheet']/@href"
+index_site
+search 'prejudice'
+```
 
-hrefs.class # => Nokogiri::XML::NodeSet
-href = hrefs.first.value # => "https://static.xx.fbcdn.net/rsrc.php/v3/y1/l/0,cross/NvZ4mNTW3Fd.css"
+The `search` call (on the last line) will return and output the results:
 
-css = crawler.crawl href.to_url
-css[0..50] # => "._3_s0._3_s0{border:0;display:flex;height:44px;min-"
+```text
+Quotes to Scrape
+“I am free of all prejudice. I hate everyone equally. ”
+http://quotes.toscrape.com/tag/humor/page/2/
 ```
 
-### Keyword Indexer (SEO Helper)
+Using a MongoDB [client](https://robomongo.org/), we can see that the two web pages have been indexed, along with their extracted *quotes* and *authors*:
+
+![MongoDBClient](https://raw.githubusercontent.com/michaeltelford/wgit/assets/assets/wgit_mongo_index.png)
 
-The below script downloads the contents of several webpages and pulls out their keywords for comparison. Such a script might be used by marketeers for search engine optimisation (SEO) for example.
+The [DSL](https://github.com/michaeltelford/wgit/wiki/How-To-Use-The-DSL) makes it easy to write scripts for experimenting with. Wgit's DSL is simply a wrapper around the underlying classes however. For comparison, here is the above example written using the Wgit API *instead of* the DSL:
 
 ```ruby
 require 'wgit'
-require 'wgit/core_ext' # => Provides the String#to_url and Enumerable#to_urls methods.
-
-my_pages_keywords = ['Everest', 'mountaineering school', 'adventure']
-my_pages_missing_keywords = []
-
-competitor_urls = [
-  'http://altitudejunkies.com',
-  'http://www.mountainmadness.com',
-  'http://www.adventureconsultants.com'
-].to_urls
+require 'json'
 
 crawler = Wgit::Crawler.new
-
-crawler.crawl(*competitor_urls) do |doc|
-  # If there are keywords present in the web document.
-  if doc.keywords.respond_to? :-
-    puts "The keywords for #{doc.url} are: \n#{doc.keywords}\n\n"
-    my_pages_missing_keywords.concat(doc.keywords - my_pages_keywords)
+url     = Wgit::Url.new('http://quotes.toscrape.com/tag/humor/')
+quotes  = []
+
+Wgit::Document.define_extractor(:quotes,  "//div[@class='quote']/span[@class='text']", singleton: false)
+Wgit::Document.define_extractor(:authors, "//div[@class='quote']/span/small",          singleton: false)
+
+crawler.crawl_site(url, follow: "//li[@class='next']/a/@href") do |doc|
+  doc.quotes.zip(doc.authors).each do |arr|
+    quotes << {
+      quote:  arr.first,
+      author: arr.last
+    }
   end
 end
 
-if my_pages_missing_keywords.empty?
-  puts 'Your pages are missing no keywords, nice one!'
-else
-  puts 'Your pages compared to your competitors are missing the following keywords:'
-  puts my_pages_missing_keywords.uniq
-end
+puts JSON.generate(quotes)
 ```
 
-## Database Example
-
-The next example requires a configured database instance. The use of a database for Wgit is entirely optional however and isn't required for crawling or URL parsing etc. A database is only needed when indexing (inserting crawled data into the database).
-
-Currently the only supported DBMS is MongoDB. See [MongoDB Atlas](https://www.mongodb.com/cloud/atlas) for a (small) free account or provide your own MongoDB instance. Take a look at this [Docker Hub image](https://hub.docker.com/r/michaeltelford/mongo-wgit) for an already built example of a `mongo` image configured for use with Wgit; the source of which can be found in the [`./docker`](https://github.com/michaeltelford/wgit/tree/master/docker) directory of this repository.
-
-[`Wgit::Database`](https://www.rubydoc.info/github/michaeltelford/wgit/master/Wgit/Database) provides a light wrapper of logic around the `mongo` gem allowing for simple database interactivity and object serialisation. Using Wgit you can index webpages, store them in a database and then search through all that's been indexed; quickly and easily.
+## Why Wgit?
 
-### Versioning
+There are many [other HTML crawlers](https://awesome-ruby.com/#-web-crawling) out there so why use Wgit?
 
-The following versions of MongoDB are currently supported:
-
-| Gem    | Database |
-| ------ | -------- |
-| ~> 2.9 | ~> 4.0   |
-
-### Data Model
-
-The data model for Wgit is deliberately simplistic. The MongoDB collections consist of:
-
-| Collection  | Purpose                                         |
-| ----------- | ----------------------------------------------- |
-| `urls`      | Stores URL's to be crawled at a later date      |
-| `documents` | Stores web documents after they've been crawled |
-
-Wgit provides respective Ruby classes for each collection object, allowing for serialisation.
-
-### Configuring MongoDB
-
-Follow the steps below to configure MongoDB for use with Wgit. This is only required if you want to read/write database records using your own (manually configured) instance of Mongo DB.
-
-1) Create collections for: `urls` and `documents`.
-2) Add a [*unique index*](https://docs.mongodb.com/manual/core/index-unique/) for the `url` field in **both** collections using:
-
-| Collection  | Fields              | Options             |
-| ----------- | ------------------- | ------------------- |
-| `urls`      | `{ "url" : 1 }`     | `{ unique : true }` |
-| `documents` | `{ "url.url" : 1 }` | `{ unique : true }` |
-
-3) Enable `textSearchEnabled` in MongoDB's configuration (if not already so - it's typically enabled by default).
-4) Create a [*text search index*](https://docs.mongodb.com/manual/core/index-text/#index-feature-text) for the `documents` collection using:
-```json
-{
-  "text": "text",
-  "author": "text",
-  "keywords": "text",
-  "title": "text"
-}
-```
-
-**Note**: The *text search index* lists all document fields to be searched by MongoDB when calling `Wgit::Database#search`. Therefore, you should append this list with any other fields that you want searched. For example, if you [extend the API](#Extending-The-API) then you might want to search your new fields in the database by adding them to the index above.
-
-### Code Example
-
-The below script shows how to use Wgit's database functionality to index and then search HTML documents stored in the database. If you're running the code for yourself, remember to replace the database [connection string](https://docs.mongodb.com/manual/reference/connection-string/) with your own.
+- Wgit has excellent unit testing, 100% documentation coverage and follows [semantic versioning](https://semver.org/) rules.
+- Wgit excels at crawling an entire website's HTML out of the box. Many alternative crawlers require you to provide the `xpath` needed to *follow* the next URLs to crawl. Wgit by default, crawls the entire site by extracting its internal links pointing to the same host.
+- Wgit allows you to define content *extractors* that will fire on every subsequent crawl; be it a single URL or an entire website. This enables you to focus on the content you want.
+- Wgit can index (crawl and store) HTML to a database making it a breeze to build custom search engines. You can also specify which page content gets searched, making the search more meaningful. For example, here's a script that will index the Wgit [wiki](https://github.com/michaeltelford/wgit/wiki) articles:
 
 ```ruby
 require 'wgit'
 
-### CONNECT TO THE DATABASE ###
-
-# In the absence of a connection string parameter, ENV['WGIT_CONNECTION_STRING'] will be used.
-db = Wgit::Database.connect '<your_connection_string>'
-
-### SEED SOME DATA ###
-
-# Here we create our own document rather than crawling the web (which works in the same way).
-# We provide the web page's URL and HTML Strings.
-doc = Wgit::Document.new(
-  'http://test-url.com',
-  "<html><p>How now brown cow.</p><a href='http://www.google.co.uk'>Click me!</a></html>"
-)
-db.insert doc
-
-### SEARCH THE DATABASE ###
+ENV['WGIT_CONNECTION_STRING'] = 'mongodb://user:password@localhost/crawler'
 
-# Searching the database returns Wgit::Document's which have fields containing the query.
-query = 'cow'
-results = db.search query
+wiki = Wgit::Url.new('https://github.com/michaeltelford/wgit/wiki')
 
-# By default, the MongoDB ranking applies i.e. results.first has the most hits.
-# Because results is an Array of Wgit::Document's, we can custom sort/rank e.g.
-# `results.sort_by! { |doc| doc.url.crawl_duration }` ranks via page load times with
-# results.first being the fastest. Any Wgit::Document attribute can be used, including
-# those you define yourself by extending the API.
-
-top_result = results.first
-top_result.class           # => Wgit::Document
-doc.url == top_result.url  # => true
-
-### PULL OUT THE BITS THAT MATCHED OUR QUERY ###
-
-# Searching each result gives the matching text snippets from that Wgit::Document.
-top_result.search(query).first # => "How now brown cow."
-
-### SEED URLS TO BE CRAWLED LATER ###
+# Only index the most recent of each wiki article, ignoring the rest of Github.
+opts = {
+  allow_paths:    'michaeltelford/wgit/wiki/*',
+  disallow_paths: 'michaeltelford/wgit/wiki/*/_history'
+}
 
-db.insert top_result.external_links
-urls_to_crawl = db.uncrawled_urls # => Results will include top_result.external_links.
+indexer = Wgit::Indexer.new
+indexer.index_site(wiki, **opts)
 ```
 
-## Extending The API
+## Why Not Wgit?
 
-Document serialising in Wgit is the means of downloading a web page and extracting parts of its content into accessible document attributes/methods. For example, `Wgit::Document#author` will return you the webpage's HTML element value of `meta[@name='author']`.
+So why might you not use Wgit, I hear you ask?
 
-Wgit provides some [default extensions](https://github.com/michaeltelford/wgit/blob/master/lib/wgit/document_extensions.rb) to extract a page's text, links etc. This of course is often not enough given the nature of the WWW and the differences from one webpage to the next. Therefore, there exists a way to extend the default serialising logic.
+- Wgit doesn't allow for webpage interaction e.g. signing in as a user. There are better gems out there for that.
+- Wgit can parse a crawled page's Javascript, but it doesn't do so by default. If your crawls are JS heavy then you might best consider a pure browser-based crawler instead.
+- Wgit while fast (using `libcurl` for HTTP etc.), isn't multi-threaded; so each URL gets crawled sequentially. You could hand each crawled document to a worker thread for processing - but if you need concurrent crawling then you should consider something else.
 
-### Serialising Additional Page Elements via Document Extensions
+## Installation
 
-You can define a Document extension for each HTML element(s) that you want to extract and serialise into a `Wgit::Document` instance variable, equipped with a getter method. Once an extension is defined, all crawled Documents will contain your extracted content.
+Only MRI Ruby is tested and supported, but Wgit may work with other Ruby implementations.
 
-Once the page element has been serialised, you can do with it as you wish e.g. obtain it's text value or manipulate the element etc. Since you can choose to return the element's text or the [Nokogiri](https://www.rubydoc.info/github/sparklemotion/nokogiri) object, you have the full power that the Nokogiri gem gives you.
+Currently, the required MRI Ruby version is:
 
-Here's how to add a Document extension to serialise a specific page element:
+`ruby '>= 2.6', '< 4'`
 
-```ruby
-require 'wgit'
+### Using Bundler
 
-# Let's get all the page's <table> elements.
-Wgit::Document.define_extension(
-  :tables,                  # Wgit::Document#tables will return the page's tables.
-  '//table',                # The xpath to extract the tables.
-  singleton: false,         # True returns the first table found, false returns all.
-  text_content_only: false, # True returns one or more Strings of the tables text,
-                            # false returns the tables as Nokogiri objects (see below).
-) do |tables|
-  # Here we can manipulate the object(s) before they're set as Wgit::Document#tables.
-end
+    $ bundle add wgit
 
-# Our Document has a table which we're interested in.
-# Note, it doesn't matter how the Document is initialised e.g. manually or crawled.
-doc = Wgit::Document.new(
-  'http://some_url.com',
-  <<~HTML
-  <html>
-    <p>Hello world! Welcome to my site.</p>
-    <table>
-      <tr><th>Name</th><th>Age</th></tr>
-      <tr><td>Socrates</td><td>101</td></tr>
-      <tr><td>Plato</td><td>106</td></tr>
-    </table>
-    <p>I hope you enjoyed your visit :-)</p>
-  </html>
-  HTML
-)
-
-# Call our newly defined method to obtain the table data we're interested in.
-tables = doc.tables
-
-# Both the collection and each table within the collection are plain Nokogiri objects.
-tables.class       # => Nokogiri::XML::NodeSet
-tables.first.class # => Nokogiri::XML::Element
-
-# Notice the Document's stats now include our 'tables' extension.
-doc.stats # => {
-#   :url=>19, :html=>242, :links=>0, :text_snippets=>2, :text_bytes=>65, :tables=>1
-# }
-```
+### Using RubyGems
 
-See the [Wgit::Document.define_extension](https://www.rubydoc.info/github/michaeltelford/wgit/master/Wgit%2FDocument.define_extension) docs for more information.
+    $ gem install wgit
 
-**Extension Notes**:
+### Verify
 
-- It's recommended that URL's be mapped into `Wgit::Url` objects. `Wgit::Url`'s are treated as Strings when being inserted into the database.
-- A `Wgit::Document` extension (once initialised) will become a Document instance variable, meaning that the value will be inserted into the Database if it's a primitive type e.g. `String`, `Array` etc. Complex types e.g. Ruby objects won't be inserted. It's up to you to ensure the data you want inserted, can be inserted.
-- Once inserted into the Database, you can search a `Wgit::Document`'s extension attributes by updating the Database's *text search index*. See the [Database Example](#Database-Example) for more information.
+    $ wgit
 
-## Caveats
+Calling the installed executable will start an REPL session.
 
-Below are some points to keep in mind when using Wgit:
+## Documentation
 
-- All absolute `Wgit::Url`'s must be prefixed with an appropiate protocol e.g. `https://` etc.
-- By default, up to 5 URL redirects will be followed; this is [configurable](https://www.rubydoc.info/github/michaeltelford/wgit/master/Wgit/Crawler#redirect_limit-instance_method) however.
-- IRI's (URL's containing non ASCII characters) **are** supported and will be normalised/escaped prior to being crawled.
+- [Getting Started](https://github.com/michaeltelford/wgit/wiki/Getting-Started)
+- [Wiki](https://github.com/michaeltelford/wgit/wiki)
+- [API Yardocs](https://www.rubydoc.info/gems/wgit)
+- [CHANGELOG](https://github.com/michaeltelford/wgit/blob/master/CHANGELOG.md)
 
 ## Executable
 
-Installing the Wgit gem also adds the `wgit` executable to your `$PATH`. The executable launches an interactive shell session with the Wgit gem already loaded; making it super easy to index and search from the command line without the need for scripts.
+Installing the Wgit gem adds a `wgit` executable to your `$PATH`. The executable launches an interactive REPL session with the Wgit gem already loaded; making it super easy to index and search from the command line without the need for scripts.
 
 The `wgit` executable does the following things (in order):
 
@@ -355,21 +193,7 @@ The `wgit` executable does the following things (in order):
 2. `eval`'s a `.wgit.rb` file (if one exists in either the local or home directory, which ever is found first)
 3. Starts an interactive shell (using `pry` if it's installed, or `irb` if not)
 
-The `.wgit.rb` file can be used to seed fixture data or define helper functions for the session. For example, you could define a function which indexes your website for quick and easy searching everytime you start a new session. **Note** that variables should either be instance variables (e.g. `@url`) or be accessed via a getter method (e.g. `def url; ...; end`).
-
-## Change Log
-
-See the [CHANGELOG.md](https://github.com/michaeltelford/wgit/blob/master/CHANGELOG.md) for differences (including any breaking changes) between releases of Wgit.
-
-### Gem Versioning
-
-The `wgit` gem follows these versioning rules:
-
-- The version format is `MAJOR.MINOR.PATCH` e.g. `0.1.0`.
-- Since the gem hasn't reached `v1.0.0` yet, slightly different semantic versioning rules apply.
-- The `PATCH` represents *non breaking changes* while the `MINOR` represents *breaking changes* e.g. updating from version `0.1.0` to `0.2.0` will likely introduce breaking changes necessitating updates to your codebase.
-- To determine what changes are needed, consult the `CHANGELOG.md`. If you need help, raise an issue.
-- Once `wgit v1.0.0` is released, *normal* [semantic versioning](https://semver.org/) rules will apply e.g. only a `MAJOR` version change should introduce breaking changes.
+The `.wgit.rb` file can be used to seed fixture data or define helper functions for the session. For example, you could define a function which indexes your website for quick and easy searching everytime you start a new session.
 
 ## License
 
@@ -395,14 +219,14 @@ And you're good to go!
 
 ### Tooling
 
-Wgit uses the [`toys`](https://github.com/dazuma/toys) gem (instead of Rake) for task invocation e.g. running the tests etc. For a full list of available tasks AKA tools, run `toys --tools`. You can search for a tool using `toys -s tool_name`. The most commonly used tools are listed below...
+Wgit uses the [`toys`](https://github.com/dazuma/toys) gem (instead of Rake) for task invocation. For a full list of available tasks a.k.a. tools, run `toys --tools`. You can search for a tool using `toys -s tool_name`. The most commonly used tools are listed below...
 
-Run `toys db` to see a list of database related tools, enabling you to run a Mongo DB instance locally using Docker. Run `toys test` to execute the tests (or `toys test smoke` for a faster running subset that doesn't require a database).
+Run `toys db` to see a list of database related tools, enabling you to run a Mongo DB instance locally using Docker. Run `toys test` to execute the tests.
 
-To generate code documentation run `toys yardoc`. To browse the generated documentation in a browser run `toys yardoc --serve`. You can also use the `yri` command line tool e.g. `yri Wgit::Crawler#crawl_site` etc.
+To generate code documentation locally, run `toys yardoc`. To browse the docs in a browser run `toys yardoc --serve`. You can also use the `yri` command line tool e.g. `yri Wgit::Crawler#crawl_site` etc.
 
-To install this gem onto your local machine, run `toys install`.
+To install this gem onto your local machine, run `toys install` and follow the prompt.
 
 ### Console
 
-You can run `toys console` for an interactive shell using the `./bin/wgit` executable. The `toys setup` task will have created a `.env` and `.wgit.rb` file which gets loaded by the executable. You can use the contents of this [gist](https://gist.github.com/michaeltelford/b90d5e062da383be503ca2c3a16e9164) to turn the executable into a development console. It defines some useful functions, fixtures and connects to the database etc. Don't forget to set the `WGIT_CONNECTION_STRING` in the `.env` file.
+You can run `toys console` for an interactive shell using the `./bin/wgit` executable. The `toys setup` task will have created an `.env` and `.wgit.rb` file which get loaded by the executable. You can use the contents of this [gist](https://gist.github.com/michaeltelford/b90d5e062da383be503ca2c3a16e9164) to turn the executable into a development console. It defines some useful functions, fixtures and connects to the database etc. Don't forget to set the `WGIT_CONNECTION_STRING` in the `.env` file.