wgit 0.5.0 → 0.9.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 3e5c6b85b0ac78d234674d6003f8624b266c09668b4cfd78945106a917f78078
-  data.tar.gz: 3fc90cf5c132804f12e54f2b5f446143591923fff0677accc2ab907295ba34c4
+  metadata.gz: 07e1146e7ddcbb35abb813ae1461520e581576181750d4b9dc654de3f3375d4c
+  data.tar.gz: 6f43949fcdf13c731362d242110348dd43c5183c10130605c2e022e15cbe8cdb
 SHA512:
-  metadata.gz: f39df81391a07b344678a2b8d443b945391728d215e142ed73a55ef80cfc9c9a8407db9e4faa60c3e43e5b8e65bf8e84c3a343ff962b3c0276eed920639f3870
-  data.tar.gz: 1690895b56def00cbed58e485b23f5158ada0adb89f1c0e87bff3c638332648761dbac81b8f08e6c9c6ee911f4cbf9df72f3bfbce5d8abc2207d434edfde61ee
+  metadata.gz: 7288c42fe7b8598572e8b4c8013f8614bd60caa048474a039d8c9a1f4ae231695148158293730998ac78b1f36a4ccd52c9664be1df0c49e218d740fd881d64c4
+  data.tar.gz: 0e36ea8f76aa41f5576044902cdc3e92c3affeb742c179a2fa5ba2b404ad057dede949b5e767bc09eb771b47bc153cf9462e56d9e5a393a63cb9e120bae870a9

data/.yardopts

@@ -0,0 +1,7 @@
+--readme README.md
+--title 'Wgit Gem Documentation'
+--charset utf-8
+--markup markdown
+--output .doc
+--protected
+- *.md LICENSE.txt

data/CHANGELOG.md

@@ -0,0 +1,240 @@
+# Wgit Change Log
+
+## v0.0.0 (TEMPLATE - DO NOT EDIT)
+### Added
+- ...
+### Changed/Removed
+- ...
+### Fixed
+- ...
+---
+
+## v0.9.0
+This release is a big one with the introduction of a `Wgit::DSL` and Javascript parse support. The `README` has been revamped as a result with new usage examples. And all of the wiki articles have been updated to reflect the latest code base.
+### Added
+- `Wgit::DSL` module providing a wrapper around the underlying classes and methods. Check out the `README` for example usage.
+- `Wgit::Crawler#parse_javascript` which when set to `true` uses Chrome to parse a page's Javascript before returning the fully rendered HTML. This feature is disabled by default.
+- `Wgit::Base` class to inherit from, acting as an alternative form of using the DSL.
+- `Wgit::Utils.sanitize` which calls `.sanitize_*` underneath.
+- `Wgit::Crawler#crawl_site` now has a `follow:` named param - if set, it's xpath value is used to retrieve the next urls to crawl. Otherwise the `:default` is used (as it was before). Use this to override how the site is crawled.
+- `Wgit::Database` methods: `#clear_urls`, `#clear_docs`, `#clear_db`, `#text_index`, `#text_index=`, `#create_collections`, `#create_unique_indexes`, `#docs`, `#get`, `#exists?`, `#delete`, `#upsert`.
+- `Wgit::Database#clear_db!` alias.
+- `Wgit::Document` methods: `#at_xpath`, `#at_css` - which call nokogiri underneath.
+- `Wgit::Document#extract` method to perform one off content extractions.
+- `Wgit::Indexer#index_urls` method which can index several urls in one call.
+- `Wgit::Url` methods: `#to_user`, `#to_password`, `#to_sub_domain`, `#to_port`, `#omit_origin`, `#index?`.
+### Changed/Removed
+- Breaking change: Moved all `Wgit.index*` convienence methods into `Wgit::DSL`.
+- Breaking change: Removed `Wgit::Url#normalise`, use `#normalize` instead.
+- Breaking change: Removed `Wgit::Database#num_documents`, use `#num_docs` instead.
+- Breaking change: Removed `Wgit::Database#length` and `#count`, use `#size` instead.
+- Breaking change: Removed `Wgit::Database#document?`, use `#doc?` instead.
+- Breaking change: Renamed `Wgit::Indexer#index_page` to `#index_url`.
+- Breaking change: Renamed `Wgit::Url.parse_or_nil` to be `.parse?`.
+- Breaking change: Renamed `Wgit::Utils.process_*` to be `.sanitize_*`.
+- Breaking change: Renamed `Wgit::Utils.remove_non_bson_types` to be `Wgit::Model.select_bson_types`.
+- Breaking change: Changed `Wgit::Indexer.index*` named param default from `insert_externals: true` to `false`. Explicitly set it to `true` for the old behaviour.
+- Breaking change: Renamed `Wgit::Document.define_extension` to `define_extractor`. Same goes for `remove_extension -> remove_extractor` and `extensions -> extractors`. See the docs for more information.
+- Breaking change: Renamed `Wgit::Document#doc` to `#parser`.
+- Breaking change: Renamed `Wgit::Crawler#time_out` to `#timeout`. Same goes for the named param passed to `Wgit::Crawler.initialize`.
+- Breaking change: Refactored `Wgit::Url#relative?` now takes `:origin` instead of `:base` which takes the port into account. This has a knock on effect for some other methods too - check the docs if you're getting parameter errors.
+- Breaking change: Renamed `Wgit::Url#prefix_base` to `#make_absolute`.
+- Updated `Utils.printf_search_results` to return the number of results.
+- Updated `Wgit::Indexer.new` which can now be called without parameters - the first param (for a database) now defaults to `Wgit::Database.new` which works if `ENV['WGIT_CONNECTION_STRING']` is set.
+- Updated `Wgit::Document.define_extractor` to define a setter method (as well as the usual getter method).
+- Updated `Wgit::Document#search` to support a `Regexp` query (in addition to a String).
+### Fixed
+- [Re-indexing bug](https://github.com/michaeltelford/wgit/issues/8) so that indexing content a 2nd time will update it in the database - before it simply disgarded the document.
+- `Wgit::Crawler#crawl_site` params `allow/disallow_paths` values can now start with a `/`.
+---
+
+## v0.8.0
+### Added
+- To the range of `Wgit::Document.text_elements`. Now (only and) all visible page text should be extracted into `Wgit::Document#text` successfully.
+- `Wgit::Document#description` default extension.
+- `Wgit::Url.parse_or_nil` method.
+### Changed/Removed
+- Breaking change: Renamed `Document#stats[:text_snippets]` to be `:text`.
+- Breaking change: `Wgit::Document.define_extension`'s block return value now becomes the `var` value, even when `nil` is returned. This allows `var` to be set to `nil`.
+- Potential breaking change: Renamed `Wgit::Response#crawl_time` (alias) to be `#crawl_duration`.
+- Updated `Wgit::Crawler::SUPPORTED_FILE_EXTENSIONS` to be `Wgit::Crawler.supported_file_extensions`, making it configurable. Now you can add your own URL extensions if needed.
+- Updated the Wgit core extension `String#to_url` to use `Wgit::Url.parse` allowing instances of `Wgit::Url` to returned as is. This also affects `Enumerable#to_urls` in the same way.
+### Fixed
+- An issue where too much `Wgit::Document#text` was being extracted from the HTML. This was fixed by reverting the recent commit: "Document.text_elements_xpath is now `//*/text()`".
+---
+
+## v0.7.0
+### Added
+- `Wgit::Indexer.new` optional `crawler:` named param.
+- `bin/wgit` executable; available after `gem install wgit`. Just type `wgit` at the command line for an interactive shell session with the Wgit gem already loaded.
+- `Document.extensions` returning a Set of all defined extensions.
+### Changed/Removed
+- Potential breaking changes: Updated the default search param from `whole_sentence: false` to `true` across all search methods e.g. `Wgit::Database#search`, `Wgit::Document#search` `Wgit.indexed_search` etc. This brings back more relevant search results by default.
+- Updated the Docker image to now include index names; making it easier to identify them.
+### Fixed
+- ...
+---
+
+## v0.6.0
+### Added
+- Added `Wgit::Utils.proces_arr encode:` param.
+### Changed/Removed
+- Breaking changes: Updated `Wgit::Response#success?` and `#failure?` logic.
+- Breaking changes: Updated `Wgit::Crawler` redirect logic. See the [docs](https://www.rubydoc.info/github/michaeltelford/wgit/Wgit/Crawler#crawl_url-instance_method) for more info.
+- Breaking changes: Updated `Wgit::Crawler#crawl_site` path params logic to support globs e.g. `allow_paths: 'wiki/*'`. See the [docs](https://www.rubydoc.info/github/michaeltelford/wgit/Wgit/Crawler#crawl_site-instance_method) for more info.
+- Breaking changes: Refactored references of `encode_html:` to `encode:` in the `Wgit::Document` and `Wgit::Crawler` classes.
+- Breaking changes: `Wgit::Document.text_elements_xpath` is now `//*/text()`. This means that more text is extracted from each page and you can no longer be selective of the text elements on a page.
+- Improved `Wgit::Url#valid?` and `#relative?`.
+### Fixed
+- Bug fix in `Wgit::Crawler#crawl_site` where `*.php` URLs weren't being crawled. The fix was to implement `Wgit::Crawler::SUPPORTED_FILE_EXTENSIONS`.
+- Bug fix in `Wgit::Document#search`.
+---
+
+## v0.5.1
+### Added
+- `Wgit.version_str` method.
+### Changed/Removed
+- Switched to optimistic dependency versioning.
+### Fixed
+- Bug in `Wgit::Url#concat`.
+---
+
+## v0.5.0
+### Added
+- A Wgit Wiki! [https://github.com/michaeltelford/wgit/wiki](https://github.com/michaeltelford/wgit/wiki)
+- `Wgit::Document#content` alias for `#html`.
+- `Wgit::Url#prefix_base` method.
+- `Wgit::Url#to_addressable_uri` method.
+- Support for partially crawling a site using `Wgit::Crawler#crawl_site(allow_paths: [])` or `disallow_paths:`.
+- `Wgit::Url#+` as alias for `#concat`.
+- `Wgit::Url#invalid?` method.
+- `Wgit.version` method.
+- `Wgit::Response` class containing adapter agnostic HTTP response logic.
+### Changed/Removed
+- Breaking changes: Removed `Wgit::Document#date_crawled` and `#crawl_duration` because both of these methods exist on the `Wgit::Document#url`. Instead, use `doc.url.date_crawled` etc.
+- Breaking changes: Added to and moved `Document.define_extension` block params, it's now `|value, source, type|`. The `source` is not what it used to be; it's now `type` - of either `:document` or `:object`. Confused? See the [docs](https://www.rubydoc.info/github/michaeltelford/wgit/master).
+- Breaking changes: Changed `Wgit::Url#prefix_protocol` so that it no longer modifies the receiver.
+- Breaking changes: Updated `Wgit::Url#to_anchor` and `#to_query` logic to align with that of `Addressable::URI` e.g. the anchor value no longer contains `#` prefix; and the query value no longer contains `?` prefix.
+- Breaking changes: Renamed `Wgit::Url` methods containing `anchor` to now be named `fragment` e.g. `to_anchor` is now called `to_fragment` and `without_anchor` is `without_fragment` etc.
+- Breaking changes: Renamed `Wgit::Url#prefix_protocol` to `#prefix_scheme`. The `protocol:` param name remains unchanged.
+- Breaking changes: Renamed all `Wgit::Url` methods starting with `without_*` to `omit_*`.
+- Breaking changes: `Wgit::Indexer` no longer inserts invalid external URL's (to be crawled at a later date).
+- Breaking changes: `Wgit::Crawler#last_response` is now of type `Wgit::Response`. You can access the underlying `Typhoeus::Response` object with `crawler.last_response.adapter_response`.
+### Fixed
+- Bug in `Wgit::Document#base_url` around the handling of invalid base URL scenarios.
+- Several bugs in `Wgit::Database` class caused by the recent changes to the data model (in version 0.3.0).
+---
+
+## v0.4.1
+### Added
+- ...
+### Changed/Removed
+- ...
+### Fixed
+- A crawl bug that resulted in some servers dropping requests due to the use of Typhoeus's default `User-Agent` header. This has now been changed.
+---
+
+## v0.4.0
+### Added
+- `Wgit::Document#stats` alias `#statistics`.
+- `Wgit::Crawler#time_out` logic for long crawls. Can also be set via `initialize`.
+- `Wgit::Crawler#last_response#redirect_count` method logic.
+- `Wgit::Crawler#last_response#total_time` method logic.
+- `Wgit::Utils.fetch(hash, key, default = nil)` method which tries multiple key formats before giving up e.g. `:foo, 'foo', 'FOO'` etc.
+### Changed/Removed
+- Breaking changes: Updated `Wgit::Crawler` crawl logic to use `typhoeus` instead of `Net:HTTP`. Users should see a significant improvement in crawl speed as a result. This means that `Wgit::Crawler#last_response` is now of type `Typhoeus::Response`. See https://rubydoc.info/gems/typhoeus/Typhoeus/Response for more info.
+### Fixed
+- ...
+---
+
+## v0.3.0
+### Added
+- `Url#crawl_duration` method.
+- `Document#crawl_duration` method.
+- `Benchmark.measure` to Crawler logic to set `Url#crawl_duration`.
+### Changed/Removed
+- Breaking changes: Updated data model to embed the full `url` object inside the documents object.
+- Breaking changes: Updated data model by removing documents `score` attribute.
+### Fixed
+- ...
+---
+
+## v0.2.0
+This version of Wgit see's a major refactor of the code base involving multiple changes to method names and their signatures (optional parameters turned into named parameters in most cases). A list of the breaking changes are below including how to fix any breakages; but if you're having issues with the upgrade see the documentation at: https://www.rubydoc.info/github/michaeltelford/wgit/master
+### Added
+- `Wgit::Url#absolute?` method.
+- `Wgit::Url#relative? base: url` support.
+- `Wgit::Database.connect` method (alias for `Wgit::Database.new`).
+- `Wgit::Database#search` and `Wgit::Document#search` methods now support `case_sensitive:` and `whole_sentence:` named parameters.
+### Changed/Removed
+- Breaking changes: Renamed the following `Wgit` and `Wgit::Indexer` methods: `Wgit.index_the_web` to `Wgit.index_www`, `Wgit::Indexer.index_the_web` to `Wgit::Indexer.index_www`, `Wgit.index_this_site` to `Wgit.index_site`, `Wgit::Indexer.index_this_site` to `Wgit::Indexer.index_site`, `Wgit.index_this_page` to `Wgit.index_page`, `Wgit::Indexer.index_this_page` to `Wgit::Indexer.index_page`.
+- Breaking changes: All `Wgit::Indexer` methods now take named parameters.
+- Breaking changes: The following `Wgit::Url` method signatures have changed: `initialize` aka `new`,
+- Breaking changes: The following `Wgit::Url` class methods have been removed: `.validate`, `.valid?`, `.prefix_protocol`, `.concat` in favour of instance methods by the same names.
+- Breaking changes: The following `Wgit::Url` instance methods/aliases have been changed/removed: `#to_protocol` (now `#to_scheme`), `#to_query_string` and `#query_string` (now `#to_query`), `#relative_link?` (now `#relative?`), `#without_query_string` (now `#without_query`), `#is_query_string?` (now `#query?`).
+- Breaking changes: The database connection string is now passed directly to `Wgit::Database.new`; or in its absence, obtained from `ENV['WGIT_CONNECTION_STRING']`. See the `README.md` section entitled: `Practical Database Example` for an example.
+- Breaking changes: The following `Wgit::Database` instance methods now take named parameters: `#urls`, `#crawled_urls`, `#uncrawled_urls`, `#search`.
+- Breaking changes: The following `Wgit::Document` instance methods now take named parameters: `#to_h`, `#to_json`, `#search`, `#search!`.
+- Breaking changes: The following `Wgit::Document` instance methods/aliases have been changed/removed: `#internal_full_links` (now `#internal_absolute_links`).
+- Breaking changes: Any `Wgit::Document` method alias for returning links containing the word `relative` has been removed for clarity. Use `#internal_links`, `#internal_absolute_links` or `#external_links` instead.
+- Breaking changes: `Wgit::Crawler` instance vars `@docs` and `@urls` have been removed causing the following instance methods to also be removed: `#urls=`, `#[]`, `#<<`. Also, `.new` aka `#initialize` now requires no params.
+- Breaking changes: `Wgit::Crawler.new` now takes an optional `redirect_limit:` parameter. This is now the only way of customising the redirect crawl behavior. `Wgit::Crawler.redirect_limit` no longer exists.
+- Breaking changes: The following `Wgit::Crawler` instance methods signatures have changed: `#crawl_site` and `#crawl_url` now require a `url` param (which no longer defaults), `#crawl_urls` now requires one or more `*urls` (which no longer defaults).
+- Breaking changes: The following `Wgit::Assertable` method aliases have been removed: `.type`, `.types` (use `.assert_types` instead) and `.arr_type`, `.arr_types` (use `.assert_arr_types` instead).
+- Breaking changes: The following `Wgit::Utils` methods now take named parameters: `.to_h` and `.printf_search_results`.
+- Breaking changes: `Wgit::Utils.printf_search_results`'s method signature has changed; the search parameters have been removed. Before calling this method you must call `doc.search!` on each of the `results`. See the docs for the full details.
+- `Wgit::Document` instances can now be instantiated with `String` Url's (previously only `Wgit::Url`'s).
+### Fixed
+- ...
+---
+
+## v0.0.18
+### Added
+- `Wgit::Url#to_brand` method and updated `Wgit::Url#is_relative?` to support it.
+### Changed/Removed
+- Updated certain classes by changing some `private` methods to `protected`.
+### Fixed
+- ...
+---
+
+## v0.0.17
+### Added
+- Support for `<base>` element in `Wgit::Document`'s.
+- New `Wgit::Url` methods: `without_query_string`, `is_query_string?`, `is_anchor?`, `replace` (override of `String#replace`).
+### Changed/Removed
+- Breaking changes: Removed `Wgit::Document#internal_links_without_anchors` method.
+- Breaking changes (potentially): `Wgit::Url`'s are now replaced with the redirected to Url during a crawl.
+- Updated `Wgit::Document#base_url` to support an optional `link:` named parameter.
+- Updated `Wgit::Crawler#crawl_site` to allow the initial url to redirect to another host.
+- Updated `Wgit::Url#is_relative?` to support an optional `domain:` named parameter.
+### Fixed
+- Bug in `Wgit::Document#internal_full_links` affecting anchor and query string links including those used during `Wgit::Crawler#crawl_site`.
+- Bug causing an 'Invalid URL' error for `Wgit::Crawler#crawl_site`.
+---
+
+## v0.0.16
+### Added
+- Added `Wgit::Url.parse` class method as alias for `Wgit::Url.new`.
+### Changed/Removed
+- Breaking changes: Removed `Wgit::Url.relative_link?` (class method). Use `Wgit::Url#is_relative?` (instance method) instead e.g. `Wgit::Url.new('/blah').is_relative?`.
+### Fixed
+- Several URI related bugs in `Wgit::Url` affecting crawls.
+---
+
+## v0.0.15
+### Added
+- Support for IRI's (non ASCII based URL's).
+### Changed/Removed
+- Breaking changes: Removed `Document` and `Url#to_hash` aliases. Call `to_h` instead.
+### Fixed
+- Bug in `Crawler#crawl_site` where an internal redirect to an external site's page was being followed.
+---
+
+## v0.0.14
+### Added
+- `Indexer#index_this_page` method.
+### Changed/Removed
+- Breaking Changes: `Wgit::CONNECTION_DETAILS` now only requires `DB_CONNECTION_STRING`.
+### Fixed
+- Found and fixed a bug in `Document#new`.
+---

data/CODE_OF_CONDUCT.md

@@ -0,0 +1,76 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+In the interest of fostering an open and welcoming environment, we as
+contributors and maintainers pledge to making participation in our project and
+our community a harassment-free experience for everyone, regardless of age, body
+size, disability, ethnicity, sex characteristics, gender identity and expression,
+level of experience, education, socio-economic status, nationality, personal
+appearance, race, religion, or sexual identity and orientation.
+
+## Our Standards
+
+Examples of behavior that contributes to creating a positive environment
+include:
+
+* Using welcoming and inclusive language
+* Being respectful of differing viewpoints and experiences
+* Gracefully accepting constructive criticism
+* Focusing on what is best for the community
+* Showing empathy towards other community members
+
+Examples of unacceptable behavior by participants include:
+
+* The use of sexualized language or imagery and unwelcome sexual attention or
+ advances
+* Trolling, insulting/derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or electronic
+ address, without explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+ professional setting
+
+## Our Responsibilities
+
+Project maintainers are responsible for clarifying the standards of acceptable
+behavior and are expected to take appropriate and fair corrective action in
+response to any instances of unacceptable behavior.
+
+Project maintainers have the right and responsibility to remove, edit, or
+reject comments, commits, code, wiki edits, issues, and other contributions
+that are not aligned to this Code of Conduct, or to ban temporarily or
+permanently any contributor for other behaviors that they deem inappropriate,
+threatening, offensive, or harmful.
+
+## Scope
+
+This Code of Conduct applies both within project spaces and in public spaces
+when an individual is representing the project or its community. Examples of
+representing a project or community include using an official project e-mail
+address, posting via an official social media account, or acting as an appointed
+representative at an online or offline event. Representation of a project may be
+further defined and clarified by project maintainers.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported by contacting the project team at michael.telford@live.com. All
+complaints will be reviewed and investigated and will result in a response that
+is deemed necessary and appropriate to the circumstances. The project team is
+obligated to maintain confidentiality with regard to the reporter of an incident.
+Further details of specific enforcement policies may be posted separately.
+
+Project maintainers who do not follow or enforce the Code of Conduct in good
+faith may face temporary or permanent repercussions as determined by other
+members of the project's leadership.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
+available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html
+
+[homepage]: https://www.contributor-covenant.org
+
+For answers to common questions about this code of conduct, see
+https://www.contributor-covenant.org/faq

data/CONTRIBUTING.md

@@ -0,0 +1,21 @@
+# Contributing
+
+## Consult
+
+Before you make a contribution, reach out to michael.telford@live.com about what changes need made. Otherwise, your time spent might be wasted. Once you're clear on what needs done follow the technical steps below.
+
+## Technical Steps
+
+- Fork the repository
+- Create a branch
+- Write some tests (which fail)
+- Write some code
+- Re-run the tests (which now hopefully pass)
+- Push your branch to your `origin` remote
+- Open a GitHub Pull Request (with the target branch being wgit's `origin/master`)
+- Apply any requested changes
+- Wait for your PR to be merged
+
+## Thanks
+
+Thanks in advance for your contribution.

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2016 - 2020 Michael Telford
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,239 @@
+# Wgit
+
+[![Inline gem version](https://badge.fury.io/rb/wgit.svg)](https://rubygems.org/gems/wgit)
+[![Inline downloads](https://img.shields.io/gem/dt/wgit)](https://rubygems.org/gems/wgit)
+[![Inline build](https://travis-ci.org/michaeltelford/wgit.svg?branch=master)](https://travis-ci.org/michaeltelford/wgit)
+[![Inline docs](http://inch-ci.org/github/michaeltelford/wgit.svg?branch=master)](http://inch-ci.org/github/michaeltelford/wgit)
+[![Inline code quality](https://api.codacy.com/project/badge/Grade/d5a0de62e78b460997cb8ce1127cea9e)](https://www.codacy.com/app/michaeltelford/wgit?utm_source=github.com&utm_medium=referral&utm_content=michaeltelford/wgit&utm_campaign=Badge_Grade)
+
+---
+
+Wgit is a HTML web crawler, written in Ruby, that allows you to programmatically extract the data you want from the web.
+
+Wgit was primarily designed to crawl static HTML websites to index and search their content - providing the basis of any search engine; but Wgit is suitable for many application domains including:
+
+- URL parsing
+- Document content extraction (data mining)
+- Crawling entire websites (statistical analysis)
+
+Wgit provides a high level, easy-to-use API and DSL that you can use in your own applications and scripts.
+
+Check out this [demo search engine](https://search-engine-rb.herokuapp.com) - [built](https://github.com/michaeltelford/search_engine) using Wgit and Sinatra - deployed to [Heroku](https://www.heroku.com/). Heroku's free tier is used so the initial page load may be slow. Try searching for "Matz" or something else that's Ruby related.
+
+## Table Of Contents
+
+1. [Usage](#Usage)
+2. [Why Wgit?](#Why-Wgit)
+3. [Why Not Wgit?](#Why-Not-Wgit)
+4. [Installation](#Installation)
+5. [Documentation](#Documentation)
+6. [Executable](#Executable)
+7. [License](#License)
+8. [Contributing](#Contributing)
+9. [Development](#Development)
+
+## Usage
+
+Let's crawl a [quotes website](http://quotes.toscrape.com/) extracting its *quotes* and *authors* using the Wgit DSL:
+
+```ruby
+require 'wgit'
+require 'json'
+
+include Wgit::DSL
+
+start  'http://quotes.toscrape.com/tag/humor/'
+follow "//li[@class='next']/a/@href"
+
+extract :quotes,  "//div[@class='quote']/span[@class='text']", singleton: false
+extract :authors, "//div[@class='quote']/span/small",          singleton: false
+
+quotes = []
+
+crawl_site do |doc|
+  doc.quotes.zip(doc.authors).each do |arr|
+    quotes << {
+      quote:  arr.first,
+      author: arr.last
+    }
+  end
+end
+
+puts JSON.generate(quotes)
+```
+
+The [DSL](https://github.com/michaeltelford/wgit/wiki/How-To-Use-The-DSL) makes it easy to write scripts for experimenting with. Wgit's DSL is simply a wrapper around the underlying classes however. For comparison, here is the above example written using the Wgit API *instead of* the DSL:
+
+```ruby
+require 'wgit'
+require 'json'
+
+crawler = Wgit::Crawler.new
+url     = Wgit::Url.new('http://quotes.toscrape.com/tag/humor/')
+quotes  = []
+
+Wgit::Document.define_extractor(:quotes,  "//div[@class='quote']/span[@class='text']", singleton: false)
+Wgit::Document.define_extractor(:authors, "//div[@class='quote']/span/small",          singleton: false)
+
+crawler.crawl_site(url, follow: "//li[@class='next']/a/@href") do |doc|
+  doc.quotes.zip(doc.authors).each do |arr|
+    quotes << {
+      quote:  arr.first,
+      author: arr.last
+    }
+  end
+end
+
+puts JSON.generate(quotes)
+```
+
+But what if we want to crawl and store the content in a database, so that it can be searched? Wgit makes it easy to index and search HTML using [MongoDB](https://www.mongodb.com/):
+
+```ruby
+require 'wgit'
+
+include Wgit::DSL
+
+Wgit.logger.level = Logger::WARN
+
+connection_string 'mongodb://user:password@localhost/crawler'
+clear_db!
+
+extract :quotes,  "//div[@class='quote']/span[@class='text']", singleton: false
+extract :authors, "//div[@class='quote']/span/small",          singleton: false
+
+start  'http://quotes.toscrape.com/tag/humor/'
+follow "//li[@class='next']/a/@href"
+
+index_site
+search 'prejudice'
+```
+
+The `search` call (on the last line) will return and output the results:
+
+```text
+Quotes to Scrape
+“I am free of all prejudice. I hate everyone equally. ”
+http://quotes.toscrape.com/tag/humor/page/2/
+```
+
+Using a Mongo DB [client](https://robomongo.org/), we can see that the two webpages have been indexed, along with their extracted *quotes* and *authors*:
+
+![MongoDBClient](https://raw.githubusercontent.com/michaeltelford/wgit/assets/assets/wgit_mongo_index.png)
+
+## Why Wgit?
+
+There are many [other HTML crawlers](https://awesome-ruby.com/#-web-crawling) out there so why use Wgit?
+
+- Wgit has excellent unit testing, 100% documentation coverage and follows [semantic versioning](https://semver.org/) rules.
+- Wgit excels at crawling an entire website's HTML out of the box. Many alternative crawlers require you to provide the `xpath` needed to *follow* the next URLs to crawl. Wgit by default, crawls the entire site by extracting its internal links pointing to the same host.
+- Wgit allows you to define content *extractors* that will fire on every subsequent crawl; be it a single URL or an entire website. This enables you to focus on the content you want.
+- Wgit can index (crawl and store) HTML to a database making it a breeze to build custom search engines. You can also specify which page content gets searched, making the search more meaningful. For example, here's a script that will index the Wgit [wiki](https://github.com/michaeltelford/wgit/wiki) articles:
+
+```ruby
+require 'wgit'
+
+ENV['WGIT_CONNECTION_STRING'] = 'mongodb://user:password@localhost/crawler'
+
+wiki = Wgit::Url.new('https://github.com/michaeltelford/wgit/wiki')
+
+# Only index the most recent of each wiki article, ignoring the rest of Github.
+opts = {
+  allow_paths:    'michaeltelford/wgit/wiki/*',
+  disallow_paths: 'michaeltelford/wgit/wiki/*/_history'
+}
+
+indexer = Wgit::Indexer.new
+indexer.index_site(wiki, **opts)
+```
+
+## Why Not Wgit?
+
+So why might you not use Wgit, I hear you ask?
+
+- Wgit doesn't allow for webpage interaction e.g. signing in as a user. There are better gems out there for that.
+- Wgit can parse a crawled page's Javascript, but it doesn't do so by default. If your crawls are JS heavy then you might best consider a pure browser-based crawler instead.
+- Wgit while fast (using `libcurl` for HTTP etc.), isn't multi-threaded; so each URL gets crawled sequentially. You could hand each crawled document to a worker thread for processing - but if you need concurrent crawling then you should consider something else.
+
+## Installation
+
+Only MRI Ruby is tested and supported, but Wgit may work with other Ruby implementations.
+
+Currently, the required MRI Ruby version is:
+
+`~> 2.5` a.k.a. `>= 2.5 && < 3`
+
+### Using Bundler
+
+Add this line to your application's `Gemfile`:
+
+```ruby
+gem 'wgit'
+```
+
+And then execute:
+
+    $ bundle
+
+### Using RubyGems
+
+    $ gem install wgit
+
+Verify the install by using the executable (to start an REPL session):
+
+    $ wgit
+
+## Documentation
+
+- [Getting Started](https://github.com/michaeltelford/wgit/wiki/Getting-Started)
+- [Wiki](https://github.com/michaeltelford/wgit/wiki)
+- [Yardocs](https://www.rubydoc.info/github/michaeltelford/wgit/master)
+- [CHANGELOG](https://github.com/michaeltelford/wgit/blob/master/CHANGELOG.md)
+
+## Executable
+
+Installing the Wgit gem adds a `wgit` executable to your `$PATH`. The executable launches an interactive REPL session with the Wgit gem already loaded; making it super easy to index and search from the command line without the need for scripts.
+
+The `wgit` executable does the following things (in order):
+
+1. `require wgit`
+2. `eval`'s a `.wgit.rb` file (if one exists in either the local or home directory, which ever is found first)
+3. Starts an interactive shell (using `pry` if it's installed, or `irb` if not)
+
+The `.wgit.rb` file can be used to seed fixture data or define helper functions for the session. For example, you could define a function which indexes your website for quick and easy searching everytime you start a new session.
+
+## License
+
+The gem is available as open source under the terms of the MIT License. See [LICENSE.txt](https://github.com/michaeltelford/wgit/blob/master/LICENSE.txt) for more details.
+
+## Contributing
+
+Bug reports and feature requests are welcome on [GitHub](https://github.com/michaeltelford/wgit/issues). Just raise an issue, checking it doesn't already exist.
+
+The current road map is rudimentally listed in the [Road Map](https://github.com/michaeltelford/wgit/wiki/Road-Map) wiki page. Maybe your feature request is already there?
+
+Before you consider making a contribution, check out [CONTRIBUTING.md](https://github.com/michaeltelford/wgit/blob/master/CONTRIBUTING.md).
+
+## Development
+
+After checking out the repo, run the following commands:
+
+1. `gem install bundler toys`
+2. `bundle install --jobs=3`
+3. `toys setup`
+
+And you're good to go!
+
+### Tooling
+
+Wgit uses the [`toys`](https://github.com/dazuma/toys) gem (instead of Rake) for task invocation. For a full list of available tasks a.k.a. tools, run `toys --tools`. You can search for a tool using `toys -s tool_name`. The most commonly used tools are listed below...
+
+Run `toys db` to see a list of database related tools, enabling you to run a Mongo DB instance locally using Docker. Run `toys test` to execute the tests.
+
+To generate code documentation locally, run `toys yardoc`. To browse the docs in a browser run `toys yardoc --serve`. You can also use the `yri` command line tool e.g. `yri Wgit::Crawler#crawl_site` etc.
+
+To install this gem onto your local machine, run `toys install` and follow the prompt.
+
+### Console
+
+You can run `toys console` for an interactive shell using the `./bin/wgit` executable. The `toys setup` task will have created an `.env` and `.wgit.rb` file which get loaded by the executable. You can use the contents of this [gist](https://gist.github.com/michaeltelford/b90d5e062da383be503ca2c3a16e9164) to turn the executable into a development console. It defines some useful functions, fixtures and connects to the database etc. Don't forget to set the `WGIT_CONNECTION_STRING` in the `.env` file.