wgit 0.0.14 → 0.0.15

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: b2f1d98c88dcd1bd2a12b463732b08373f6b5db4d1c75661530293b0ed129c47
-  data.tar.gz: ed5ff0f5aa5e2427909ced8f67c8189dd1f7aec259b76947cd4cd73a1f8783d3
+  metadata.gz: b83cdb3dd0deec7e706c77a8302eb9a5d52d9ec930fbfab43dc2790334521160
+  data.tar.gz: a1ac1ae6d151a64db95aaee386457fddfce2b76f3d0b824b360926e689ceb64a
 SHA512:
-  metadata.gz: 6270ceff57af1936e7b0ae1fb3748def307981b26dc604d2f190a9e751e0952f103b4a7c227e2cbbc2d00f26bc7c4b1f46f44fa423e13add7e8bd4eb9fe046f4
-  data.tar.gz: 96b0eecd2713cbf55f7433c145a541fdae34d56b9e3bedd1b91b6bb30b463d06b7f07de59457ad899fd32bd095bbe92ee01d94c04ef923924fbc4a110bbe98f2
+  metadata.gz: 7b3831ba9f3d60810507ebf49ec19e2ca533da188af84657552371823b1924c2357bdf2581a34e3526f4c160aff6dc876e2d5eba927ff793b5b4a0d6d9ff4221
+  data.tar.gz: 2e6b4fc6f36f97f6d400488f52df69323666186d03fab596a8f1895cdd76b1a9d7df573fdc1336c4df759259a1aa309f9e55e5488c2118fb9fb8c0ab307c5dc7

data/README.md

@@ -61,7 +61,7 @@ Wgit::Document.instance_methods(false).sort # => [
 # :external_urls, :html, :internal_full_links, :internal_links,
 # :internal_links_without_anchors, :keywords, :links, :relative_full_links,
 # :relative_full_urls, :relative_links, :relative_urls, :score, :search,
-# :search!, :size, :stats, :text, :title, :to_h, :to_hash, :to_json, :url,
+# :search!, :size, :stats, :text, :title, :to_h, :to_json, :url,
 # :xpath
 #]
 
@@ -315,8 +315,9 @@ tables.first.class  # => Nokogiri::XML::Element
 
 Below are some points to keep in mind when using Wgit:
 
-- All absolute `Wgit::Url`'s must be prefixed with an appropiate protocol e.g. `https://`
+- All absolute `Wgit::Url`'s must be prefixed with an appropiate protocol e.g. `https://` etc.
 - By default, up to 5 URL redirects will be followed; this is configurable however.
+- IRI's (URL's containing non ASCII characters) are supported and will be normalised/escaped prior to being crawled.
 
 ## Executable
 
@@ -328,24 +329,24 @@ This executable will be very similar in nature to `./bin/console` which is curre
 
 ## Change Log
 
-See the [CHANGELOG.md](https://github.com/michaeltelford/wgit/blob/master/CHANGELOG.md) for differences between versions of Wgit.
+See the [CHANGELOG.md](https://github.com/michaeltelford/wgit/blob/master/CHANGELOG.md) for differences (including any breaking changes) between releases of Wgit.
 
-## Development
+## License
 
-The current road map is rudimentally listed in the [TODO.txt](https://github.com/michaeltelford/wgit/blob/master/TODO.txt) file.
+The gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).
 
-For a full list of available Rake tasks, run `bundle exec rake help`. The most commonly used tasks are listed below...
+## Contributing
 
-After checking out the repo, run `./bin/setup` to install dependencies (requires `bundler`). Then, run `bundle exec rake test` to run the tests. You can also run `./bin/console` for an interactive REPL that will allow you to experiment with the code.
+Bug reports and feature requests are welcome on [GitHub](https://github.com/michaeltelford/wgit/issues). Just raise an issue, checking it doesn't already exist.
 
-To generate code documentation run `bundle exec yard doc`. To browse the generated documentation run `bundle exec yard server -r`.
+The current road map is rudimentally listed in the [TODO.txt](https://github.com/michaeltelford/wgit/blob/master/TODO.txt) file. Maybe your feature request is already there?
 
-To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, see the *Gem Publishing Checklist* section of the `TODO.txt` file.
+## Development
 
-## Contributing
+For a full list of available Rake tasks, run `bundle exec rake help`. The most commonly used tasks are listed below...
 
-Bug reports and pull requests are welcome on [GitHub](https://github.com/michaeltelford/wgit).
+After checking out the repo, run `./bin/setup` to install dependencies (requires `bundler`). Then, run `bundle exec rake test` to run the tests. You can also run `./bin/console` for an interactive (`pry`) REPL that will allow you to experiment with the code.
 
-## License
+To generate code documentation run `bundle exec yard doc`. To browse the generated documentation run `bundle exec yard server -r`.
 
-The gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).
+To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, see the *Gem Publishing Checklist* section of the `TODO.txt` file.

data/lib/wgit/crawler.rb

@@ -91,13 +91,15 @@ module Wgit
     # Crawl the url and return the response document or nil.
     #
     # @param url [Wgit::Document] The URL to crawl.
+    # @param follow_external_redirects [Boolean] Whether or not to follow
+    #   external redirects. False will return nil for such a crawl.
     # @yield [Wgit::Document] The crawled HTML Document regardless if the
     #   crawl was successful or not. Therefore, the Document#url can be used.
     # @return [Wgit::Document, nil] The crawled HTML Document or nil if the
     #   crawl was unsuccessful.
-    def crawl_url(url = @urls.first)
+    def crawl_url(url = @urls.first, follow_external_redirects: true)
       assert_type(url, Wgit::Url)
-      markup = fetch(url)
+      markup = fetch(url, follow_external_redirects: follow_external_redirects)
       url.crawled = true
       doc = Wgit::Document.new(url, markup)
       yield(doc) if block_given?
@@ -116,7 +118,7 @@ module Wgit
     def crawl_site(base_url = @urls.first, &block)
       assert_type(base_url, Wgit::Url)
 
-      doc = crawl_url(base_url, &block)
+      doc = crawl_url(base_url, follow_external_redirects: false, &block)
       return nil if doc.nil?
 
       path = base_url.path.nil? ? '/' : base_url.path
@@ -133,9 +135,15 @@ module Wgit
         break if links.empty?
 
         links.each do |link|
-          doc = crawl_url(Wgit::Url.concat(base_url.to_base, link), &block)
+          doc = crawl_url(
+            Wgit::Url.concat(base_url.to_base, link),
+            follow_external_redirects: false,
+            &block
+          )
+
           crawled_urls << link
           next if doc.nil?
+
           internal_urls.concat(get_internal_links(doc))
           external_urls.concat(doc.external_links)
         end
@@ -160,8 +168,8 @@ module Wgit
     # The fetch method performs a HTTP GET to obtain the HTML document.
     # Invalid urls or any HTTP response that doesn't return a HTML body will be
     # ignored and nil will be returned. Otherwise, the HTML is returned.
-    def fetch(url)
-      response = resolve(url)
+    def fetch(url, follow_external_redirects: true)
+      response = resolve(url, follow_external_redirects: follow_external_redirects)
       @last_response = response
       response.body.empty? ? nil : response.body
     rescue Exception => ex
@@ -176,18 +184,30 @@ module Wgit
     # A certain amount of redirects will be followed by default before raising
     # an exception. Redirects can be disabled by setting `redirect_limit: 0`.
     # The Net::HTTPResponse will be returned.
-    def resolve(url, redirect_limit: Wgit::Crawler.default_redirect_limit)
+    def resolve(
+        url,
+        redirect_limit: Wgit::Crawler.default_redirect_limit,
+        follow_external_redirects: true
+      )
+      raise 'url must respond to :normalise' unless url.respond_to?(:normalise)
       redirect_count = -1
+
       begin
-        raise "Too many redirects" if redirect_count >= redirect_limit
+        raise 'Too many redirects' if redirect_count >= redirect_limit
         redirect_count += 1
 
-        response = Net::HTTP.get_response(URI(url))
+        response = Net::HTTP.get_response(url.normalise.to_uri)
         location = Wgit::Url.new(response.fetch('location', ''))
+
         if not location.empty?
+          if !follow_external_redirects and !location.is_relative?
+            raise 'External redirect encountered but not allowed'
+          end
+
           url = location.is_relative? ? url.to_base.concat(location) : location
         end
       end while response.is_a?(Net::HTTPRedirection)
+
       response
     end
 

data/lib/wgit/document.rb

@@ -364,6 +364,7 @@ module Wgit
     #   xpath value to be derived on Document initialisation (instead of when
     #   the extension is defined). The call method must return a valid xpath
     #   String.
+    # @param options [Hash] The options to define an extension with.
     # @option options [Boolean] :singleton The singleton option determines
     #   whether or not the result(s) should be in an Array. If multiple
     #   results are found and singleton is true then the first result will be
@@ -501,7 +502,6 @@ module Wgit
       end
     end
 
-    alias :to_hash :to_h
     alias :relative_links :internal_links
     alias :relative_urls :internal_links
     alias :relative_full_links :internal_full_links

data/lib/wgit/url.rb

@@ -1,13 +1,14 @@
 require_relative 'utils'
 require_relative 'assertable'
 require 'uri'
+require 'addressable/uri'
 
 module Wgit
 
   # Class modeling a web based URL.
   # Can be an internal/relative link e.g. "about.html" or a full URL
-  # e.g. "http://www.google.co.uk". Is a subclass of String and uses 'uri'
-  # internally.
+  # e.g. "http://www.google.co.uk". Is a subclass of String and uses
+  # 'addressable/uri' internally.
   class Url < String
     include Assertable
 
@@ -43,7 +44,7 @@ module Wgit
         date_crawled = obj["date_crawled"]
       end
 
-      @uri = URI(url)
+      @uri = Addressable::URI.parse(url)
       @crawled = crawled
       @date_crawled = date_crawled
 
@@ -169,9 +170,16 @@ module Wgit
       @date_crawled = bool ? Wgit::Utils.time_stamp : nil
     end
 
+    # Normalises/encodes self and returns a new Wgit::Url.
+    #
+    # @return [Wgit::Url] An encoded version of self.
+    def normalise
+      Wgit::Url.new(@uri.normalize.to_s)
+    end
+
     # Returns the @uri instance var of this URL.
     #
-    # @return [URI::HTTP, URI::HTTPS] The URI object of self.
+    # @return [Addressable::URI] The URI object of self.
     def to_uri
       @uri
     end
@@ -337,7 +345,6 @@ module Wgit
       Hash[h.to_a.insert(0, ["url", self])] # Insert url at position 0.
     end
 
-    alias :to_hash :to_h
     alias :uri :to_uri
     alias :url :to_url
     alias :scheme :to_scheme
@@ -358,5 +365,6 @@ module Wgit
     alias :is_relative? :relative_link?
     alias :is_internal? :relative_link?
     alias :crawled? :crawled
+    alias :normalize :normalise
   end
 end

data/lib/wgit/version.rb

@@ -3,5 +3,5 @@
 # @author Michael Telford
 module Wgit
   # The current gem version of Wgit.
-  VERSION = "0.0.14".freeze
+  VERSION = "0.0.15".freeze
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: wgit
 version: !ruby/object:Gem::Version
-  version: 0.0.14
+  version: 0.0.15
 platform: ruby
 authors:
 - Michael Telford
@@ -142,6 +142,20 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: '2.0'
+- !ruby/object:Gem::Dependency
+  name: addressable
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 2.6.0
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 2.6.0
 - !ruby/object:Gem::Dependency
   name: nokogiri
   requirement: !ruby/object:Gem::Requirement
@@ -196,9 +210,7 @@ files:
 - "./lib/wgit/url.rb"
 - "./lib/wgit/utils.rb"
 - "./lib/wgit/version.rb"
-- LICENSE.txt
 - README.md
-- TODO.txt
 homepage: https://github.com/michaeltelford/wgit
 licenses:
 - MIT

data/LICENSE.txt

@@ -1,21 +0,0 @@
-The MIT License (MIT)
-
-Copyright (c) 2019 Michael Telford
-
-Permission is hereby granted, free of charge, to any person obtaining a copy
-of this software and associated documentation files (the "Software"), to deal
-in the Software without restriction, including without limitation the rights
-to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-copies of the Software, and to permit persons to whom the Software is
-furnished to do so, subject to the following conditions:
-
-The above copyright notice and this permission notice shall be included in
-all copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
-THE SOFTWARE.

data/TODO.txt

@@ -1,34 +0,0 @@
-
-Primary
--------
-- Add <base> support for link processing.
-- Update Database#search & Document#search to have optional case sensitivity.
-- Have the ability to crawl sub sections of a site only e.g. https://www.honda.co.uk/motorcycles.html as the base url and crawl any links containing this as a prefix. For example, https://www.honda.co.uk/cars.html would not be crawled but https://www.honda.co.uk/motorcycles/africa-twin.html would be.
-- Create an executable based on the ./bin/console shipped as `wpry` or `wgit`.
-
-Secondary
----------
-- Think about how we handle invalid url's on crawled documents. Setup tests and implement logic for this scenario.
-- Think about ignoring non html documents/urls e.g. http://server/image.jpg etc. by implementing MIME types (defaulting to only HTML).
-- Check if Document::TEXT_ELEMENTS is expansive enough.
-- Possibly use refine instead of core-ext?
-- Think about potentially using DB._update's update_many func.
-
-Refactoring
------------
-- Refactor the 3 main classes and their tests (where needed): Url, Document & Crawler.
-- Think about reducing the amount of method aliases, pick the best for the method def and remove the aliases?
-- Replace method params with named parameters where applicable.
-
-Gem Publishing Checklist
-------------------------
-- Ensure a clean branch of master and create a 'release' branch.
-- Update standalone files (if necessary): README.md, TODO.txt, wgit.gemspec etc.
-- Increment the version number (in version.rb) and update the CHANGELOG.md.
-- Run 'bundle install' to update deps.
-- Run 'bundle exec rake compile' and ensure acceptable warnings/errors.
-- Run 'bundle exec rake test' and ensure all tests are passing.
-- Run `bundle exec rake install` to build and install the gem locally, then test it manually from outside this repo.
-- Run `bundle exec yard doc` to update documentation - should be very high percentage.
-- Commit, merge to master & push any changes made from the above steps.
-- Run `bundle exec rake RELEASE[origin]` to tag, build and push everything to github.com and rubygems.org.