html-proofer 2.4.2 → 2.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 110942931188ba50ea9cfa54113e766fdb2e82ab
-  data.tar.gz: 302722012ef4ae3c3357312bf48cf13214b43696
+  metadata.gz: 1b2c47bd5d3ea56e4d6f2a5f4ee3ba01ed43b337
+  data.tar.gz: 381527e029581dcb0249372f345d838a2950f136
 SHA512:
-  metadata.gz: aca39deda0c1c82c566d9b1d4723c632343c5b68193aa0923e6f4ad9ba9042d8eb6ef12a980627c1b021432f01c7bdabbb64f4516dd472a2072750e94526c3e6
-  data.tar.gz: 64e204388cff5f32d1995fae38a85ea981bc52f9a420802ae21b499da859ed525a32d23afca8ae4d51c1ab215987439623007279fb468606f50f24e2092ad8e4
+  metadata.gz: 0a72bfe1b0702177843865d6b36008a1136a1e8366c37ffce4ea37a91d8dc876e394167328a77068ae1f2b70d2fa48f7fa0bcd82d0d8b7518d219949ae8cd056
+  data.tar.gz: 25a3d37a3dcf3ff4f8ea439ac1344bdac5f55b4d5621017f7b8b22d4e963fb5f7c8e49d774a223a251117e7407afad464aae471839bdb04f4174edaa3f170370

data/README.md

@@ -22,16 +22,10 @@ Or install it yourself as:
 
 **NOTE:** When installation speed matters, set `NOKOGIRI_USE_SYSTEM_LIBRARIES` to `true` in your environment. This is useful for increasing the speed of your Continuous Integration builds.
 
-### Real-life examples
-
-Project | Repository
-:--- | :---
-[Raspberry Pi documentation](http://www.raspberrypi.org/documentation/) | [raspberrypi/documentation]( https://github.com/raspberrypi/documentation)
-[Open Whisper Systems website](https://whispersystems.org/) | [WhisperSystems/whispersystems.org](https://github.com/WhisperSystems/whispersystems.org)
-[Jekyll website](http://jekyllrb.com/) | [jekyll/jekyll](https://github.com/jekyll/jekyll)
-
 ## What's Tested?
 
+You can enable or disable most of the following checks.
+
 ### Images
 
 `img` elements:
@@ -44,24 +38,24 @@ Project | Repository
 
 `a`, `link` elements:
 
-* Whether your internal links are not broken; this includes hash references (`#linkToMe`)
+* Whether your internal links are working
+* Whether your internal hash references (`#linkToMe`) are working
 * Whether external links are working
 
 ### Scripts
 
 `script` elements:
 
-* Whether your internal script references are not broken
+* Whether your internal script references are working
 * Whether external scripts are loading
 
 ### Favicon
 
-Checks if your favicons are valid. This is an optional feature, set the `check_favicon` option to turn it on.
+* Whether your favicons are valid.
 
 ### HTML
 
-Nokogiri looks at the markup and [provides errors](http://www.nokogiri.org/tutorials/ensuring_well_formed_markup.html) when parsing your document.
-This is an optional feature, set the `check_html` option to enable validation errors from Nokogiri.
+* Whether your HTML markup is valid. This is done via [Nokogiri, to ensure well-formed markup](http://www.nokogiri.org/tutorials/ensuring_well_formed_markup.html).
 
 ## Usage
 
@@ -125,12 +119,34 @@ task :test do
 end
 ```
 
-Don't have or want a `Rakefile`? You _could_ also do something like the following:
+Don't have or want a `Rakefile`? You can also do something like the following:
 
 ```bash
 htmlproof ./_site
 ```
 
+### Array of links
+
+Instead of a directory as the first argument, you can also pass in an array of links:
+
+``` ruby
+HTML::Proofer.new(["http://github.com", "http://jekyllrb.com"])
+```
+
+This configures Proofer to just test those links to ensure they are valid. Note that for the command-line, you'll need to pass a special `--as-links` argument:
+
+``` bash
+htmlproof www.google.com,www.github.com --as-links
+```
+
+## Ignoring content
+
+Add the `data-proofer-ignore` attribute to any tag to ignore it from every check.
+
+``` html
+<a href="http://notareallink" data-proofer-ignore>Not checked.</a>
+```
+
 ## Configuration
 
 The `HTML::Proofer` constructor takes an optional hash of additional options:
@@ -148,12 +164,13 @@ The `HTML::Proofer` constructor takes an optional hash of additional options:
 | `error_sort` | Defines the sort order for error output. Can be `:path`, `:desc`, or `:status`. | `:path`
 | `ext` | The extension of your HTML files including the dot. | `.html`
 | `file_ignore` | An array of Strings or RegExps containing file paths that are safe to ignore. | `[]` |
-| `href_ignore` | An array of Strings or RegExps containing `href`s that are safe to ignore. Note that non-HTTP(S) URIs are always ignored. | `[]` |
-| `href_swap` | A hash containing key-value pairs of `RegExp => String`. It transforms links that match `RegExp` into `String` via `gsub`. | `{}` |
+| `href_ignore` | An array of Strings or RegExps containing `href`s that are safe to ignore. Note that non-HTTP(S) URIs are always ignored. **Will be renamed in a future release.** | `[]` |
+| `href_swap` | A hash containing key-value pairs of `RegExp => String`. It transforms links that match `RegExp` into `String` via `gsub`. **Will be renamed in a future release.** | `{}` |
 | `ignore_script_embeds` | When `check_html` is enabled, `script` tags containing markup [are reported as errors](http://git.io/vOovv). Enabling this option ignores those errors. | `false`
 | `only_4xx` | Only reports errors for links that fall within the 4xx status code range. | `false` |
 | `url_ignore` | An array of Strings or RegExps containing URLs that are safe to ignore. It affects all HTML attributes. Note that non-HTTP(S) URIs are always ignored. | `[]` |
-| `verbose` | If `true`, outputs extra information as the checking happens. Useful for debugging. | `false` |
+| `verbose` | If `true`, outputs extra information as the checking happens. Useful for debugging. **Will be deprecated in a future release.**| `false` |
+| `verbosity` | Sets the logging level, as determined by [Yell](https://github.com/rudionrails/yell). | `:info`
 
 ### Configuring Typhoeus and Hydra
 
@@ -163,7 +180,7 @@ The `HTML::Proofer` constructor takes an optional hash of additional options:
 HTML::Proofer.new("out/", {:ext => ".htm", :typhoeus => { :verbose => true, :ssl_verifyhost => 2 } })
 ```
 
-This sets `HTML::Proofer`'s extensions to use _.htm_, and gives Typhoeus a configuration for it to be verbose, and use specific SSL settings. Check [the Typhoeus documentation](https://github.com/typhoeus/typhoeus#other-curl-options) for more information on what options it can receive.
+This sets `HTML::Proofer`'s extensions to use _.htm_, gives Typhoeus a configuration for it to be verbose, and use specific SSL settings. Check [the Typhoeus documentation](https://github.com/typhoeus/typhoeus#other-curl-options) for more information on what options it can receive.
 
 You can similarly pass in a `:hydra` option with a hash configuration for Hydra.
 
@@ -171,40 +188,26 @@ The default value is `typhoeus => { :followlocation => true }`.
 
 ### Configuring Parallel
 
-[Parallel](https://github.com/grosser/parallel) is being used to speed internal file checks. You can pass in any of its options with the options "namespace" `:parallel`. For example:
+[Parallel](https://github.com/grosser/parallel) is used to speed internal file checks. You can pass in any of its options with the options namespace `:parallel`. For example:
 
 ``` ruby
 HTML::Proofer.new("out/", {:ext => ".htm", :parallel => { :in_processes => 3} })
 ```
 
-`:in_processes => 3` will be passed into Parallel as a configuration option.
-
-### Array of links
-
-Instead of a directory as the first argument, you can also pass in an array of links:
-
-``` ruby
-HTML::Proofer.new(["http://github.com", "http://jekyllrb.com"])
-```
-
-This configures Proofer to just test those links to ensure they are valid. Note that for the command-line, you'll need to pass a special `--as-links` argument:
+In this example, `:in_processes => 3` is passed into Parallel as a configuration option.
 
-``` bash
-bin/htmlproof www.google.com,www.github.com --as-links
-```
+## Logging
 
-## Ignoring content
+HTML-Proofer can be as noisy or as quiet as you'd like. There are two ways to log information:
 
-Add the `data-proofer-ignore` attribute to any tag to ignore it from the checks.
+* If you set the `:verbose` option to `true`, HTML-Proofer will provide some debug information.
+* If you set the `:verbosity` option, you can better define the level of logging. See the configuration table above for more information.
 
-
-``` html
-<a href="http://notareallink" data-proofer-ignore>Not checked.</a>
-```
+`:verbosity` is newer and offers better configuration. `:verbose` will be deprecated in a future 3.x.x release.
 
 ## Custom tests
 
-Want to write your own test? Sure! Just create two classes--one that inherits from `HTML::Proofer::CheckRunner`, and another that inherits from `HTML::Proofer::Checkable`.
+Want to write your own test? Sure! Just create two classes--one that inherits from `HTML::Proofer::Checkable`, and another that inherits from `HTML::Proofer::CheckRunner`.
 
 The `CheckRunner` subclass must define one method called `run`. This is called on your content, and is responsible for performing the validation on whatever elements you like. When you catch a broken issue, call `add_issue(message)` to explain the error.
 
@@ -214,7 +217,6 @@ Here's an example custom test that protects against `mailto` links that point to
 
 ``` ruby
 class OctocatLinkCheck < ::HTML::Proofer::Checkable
-
   def mailto?
     return false if @data_ignore_proofer || @href.nil? || @href.empty?
     return @href.match /^mailto\:/
@@ -227,13 +229,13 @@ class OctocatLinkCheck < ::HTML::Proofer::Checkable
 end
 
 class MailToOctocat < ::HTML::Proofer::CheckRunner
-
   def run
-    @html.css('a').each do |l|
-      link = OctocatLinkCheck.new l, self
+    @html.css('a').each do |node|
+      link = OctocatLinkCheck.new(node, self)
+      line = node.line
 
       if link.mailto? && link.octocat?
-        return add_issue("Don't email the Octocat directly!")
+        return add_issue("Don't email the Octocat directly!", line)
       end
     end
   end
@@ -244,7 +246,7 @@ end
 
 ### Certificates
 
-To ignore certificates turn off the Typhoeus SSL verification:
+To ignore certificates, turn off Typhoeus' SSL verification:
 
 ``` ruby
 HTML::Proofer.new("out/", {
@@ -264,3 +266,11 @@ HTML::Proofer.new("out/", {
     :headers => { "User-Agent" => "Mozilla/5.0 (compatible; My New User-Agent)" }
 }}).run
 ```
+
+## Real-life examples
+
+Project | Repository
+:--- | :---
+[Raspberry Pi documentation](http://www.raspberrypi.org/documentation/) | [raspberrypi/documentation]( https://github.com/raspberrypi/documentation)
+[Open Whisper Systems website](https://whispersystems.org/) | [WhisperSystems/whispersystems.org](https://github.com/WhisperSystems/whispersystems.org)
+[Jekyll website](http://jekyllrb.com/) | [jekyll/jekyll](https://github.com/jekyll/jekyll)

data/bin/htmlproof

@@ -8,7 +8,7 @@ require 'mercenary'
 require 'rubygems'
 
 def to_regex?(item)
-  if item.start_with? '/' and item.end_with? '/'
+  if item.start_with?('/') && item.end_with?('/')
     Regexp.new item[1...-1]
   else
     item
@@ -40,6 +40,7 @@ Mercenary.program(:htmlproof) do |p|
   p.option 'only_4xx', '--only-4xx', 'Only reports errors for links that fall within the 4x status code range.'
   p.option 'url_ignore', '--url-ignore link1,[link2,...]', Array, 'Comma-separated list of Strings or RegExps containing URLs that are safe to ignore.'
   p.option 'verbose', '--verbose', 'Enables more verbose logging.'
+  p.option 'verbosity', '--verbosity', String, 'Sets the logging level, as determined by Yell'
 
   p.action do |args, opts|
     args = ['.'] if args.empty?
@@ -47,7 +48,7 @@ Mercenary.program(:htmlproof) do |p|
 
     options = {}
 
-    # prepare every to go to proofer
+    # prepare everything to go to proofer
     p.options.select { |o| !opts[o.config_key].nil? }.each do |option|
       if option.return_type.to_s == 'Array' # TODO: is_a? doesn't work here?
         opts[option.config_key] = opts[option.config_key].map { |i| to_regex?(i) }
@@ -65,6 +66,7 @@ Mercenary.program(:htmlproof) do |p|
     end
 
     options[:error_sort] = opts['error-sort'].to_sym unless opts['error-sort'].nil?
+    options[:verbosity] = opts['verbosity'].to_sym unless opts['verbosity'].nil?
 
     path = path.delete(' ').split(',') if opts['as_links']
 

data/lib/html/proofer.rb

@@ -20,20 +20,27 @@ rescue LoadError; end
 module HTML
 
   class Proofer
-    include Utils
+    include HTML::Proofer::Utils
 
-    attr_reader :options, :typhoeus_opts, :hydra_opts, :parallel_opts, :validation_opts
+    attr_reader :options, :typhoeus_opts, :hydra_opts, :parallel_opts, :validation_opts, :external_urls
 
     TYPHOEUS_DEFAULTS = {
       :followlocation => true,
       :headers => {
-        "User-Agent" => "Mozilla/5.0 (compatible; HTML Proofer/#{VERSION}; +https://github.com/gjtorikian/html-proofer)"
+        'User-Agent' => "Mozilla/5.0 (compatible; HTML Proofer/#{VERSION}; +https://github.com/gjtorikian/html-proofer)"
       }
     }
 
     def initialize(src, opts = {})
       @src = src
 
+      if opts[:verbose]
+        warn '`@options[:verbose]` will be removed in a future 3.x.x release: http://git.io/vGHHh'
+      end
+      if opts[:href_ignore]
+        warn '`@options[:href_ignore]` will be renamed in a future 3.x.x release: http://git.io/vGHHy'
+      end
+
       @proofer_opts = {
         :ext => '.html',
         :check_favicon => false,
@@ -72,7 +79,7 @@ module HTML
     end
 
     def logger
-      @logger ||= HTML::Proofer::Log.new(@options[:verbose])
+      @logger ||= HTML::Proofer::Log.new(@options[:verbose], @options[:verbosity])
     end
 
     def run
@@ -95,27 +102,27 @@ module HTML
 
     def check_list_of_links
       if @options[:href_swap]
-        @src = @src.map do |external_url|
-          swap(external_url, @options[:href_swap])
+        @src = @src.map do |url|
+          swap(url, @options[:href_swap])
         end
       end
-      external_urls = Hash[*@src.map { |s| [s, nil] }.flatten]
-      validate_urls(external_urls)
+      @external_urls = Hash[*@src.map { |s| [s, nil] }.flatten]
+      validate_urls
     end
 
     # Collects any external URLs found in a directory of files. Also collectes
     # every failed test from check_files_for_internal_woes.
     # Sends the external URLs to Typhoeus for batch processing.
     def check_directory_of_files
-      external_urls = {}
+      @external_urls = {}
       results = check_files_for_internal_woes
 
       results.each do |item|
-        external_urls.merge!(item[:external_urls])
+        @external_urls.merge!(item[:external_urls])
         @failed_tests.concat(item[:failed_tests])
       end
 
-      validate_urls(external_urls) unless @options[:disable_external]
+      validate_urls unless @options[:disable_external]
 
       logger.log :info, :blue, "Ran on #{files.length} files!\n\n"
     end
@@ -137,8 +144,8 @@ module HTML
       end
     end
 
-    def validate_urls(external_urls)
-      url_validator = HTML::Proofer::UrlValidator.new(logger, external_urls, @options, @typhoeus_opts, @hydra_opts)
+    def validate_urls
+      url_validator = HTML::Proofer::UrlValidator.new(logger, @external_urls, @options, @typhoeus_opts, @hydra_opts)
       @failed_tests.concat(url_validator.run)
     end
 

data/lib/html/proofer/cache.rb

@@ -0,0 +1,16 @@
+module HTML
+  class Proofer
+    module Cache
+      def create_nokogiri(path)
+        if File.exist? path
+          content = File.open(path).read
+        else
+          content = path
+        end
+
+        Nokogiri::HTML(content)
+      end
+      module_function :create_nokogiri
+    end
+  end
+end

data/lib/html/proofer/check_runner.rb

@@ -6,7 +6,8 @@ module HTML
     class CheckRunner
 
       attr_reader :issues, :src, :path, :options, :typhoeus_opts, :hydra_opts, :parallel_opts, \
-                  :validation_opts, :external_urls, :href_ignores, :url_ignores, :alt_ignores, :empty_alt_ignore
+                  :validation_opts, :external_urls, :href_ignores, :url_ignores, :alt_ignores, \
+                  :empty_alt_ignore
 
       def initialize(src, path, html, options, typhoeus_opts, hydra_opts, parallel_opts, validation_opts)
         @src    = src
@@ -23,6 +24,7 @@ module HTML
         @alt_ignores = @options[:alt_ignore]
         @empty_alt_ignore = @options[:empty_alt_ignore]
         @external_urls = {}
+        @external_domain_paths_with_queries = {}
       end
 
       def run
@@ -33,14 +35,45 @@ module HTML
         @issues << Issue.new(@path, desc, line_number, status)
       end
 
-      def add_to_external_urls(href)
-        if @external_urls[href]
-          @external_urls[href] << @path
+      def add_to_external_urls(url, line)
+        return if @external_urls[url]
+        uri = Addressable::URI.parse(url)
+
+        if uri.query.nil?
+          add_path_for_url(url)
+        else
+          new_url_query_values?(uri, url)
+        end
+      end
+
+      def add_path_for_url(url)
+        if @external_urls[url]
+          @external_urls[url] << @path
         else
-          @external_urls[href] = [@path]
+          @external_urls[url] = [@path]
         end
       end
 
+      def new_url_query_values?(uri, url)
+        queries = uri.query_values.keys.join('-')
+        domain_path = extract_domain_path(uri)
+        if @external_domain_paths_with_queries[domain_path].nil?
+          add_path_for_url(url)
+          # remember queries we've seen, ignore future ones
+          @external_domain_paths_with_queries[domain_path] = [queries]
+        else
+          # add queries we haven't seen
+          unless @external_domain_paths_with_queries[domain_path].include?(queries)
+            add_path_for_url(url)
+            @external_domain_paths_with_queries[domain_path] << queries
+          end
+        end
+      end
+
+      def extract_domain_path(uri)
+        uri.host + uri.path
+      end
+
       def self.checks
         classes = []
 
@@ -52,7 +85,7 @@ module HTML
         classes
       end
 
-    private
+      private
 
       def remove_ignored(html)
         html.css('code, pre').each(&:unlink)

data/lib/html/proofer/checkable.rb

@@ -5,7 +5,8 @@ module HTML
   class Proofer
     # Represents the superclass from which all checks derive.
     class Checkable
-      include HTML::Utils
+      include HTML::Proofer::Utils
+
       attr_reader :line
 
       def initialize(obj, check)
@@ -75,12 +76,12 @@ module HTML
         return true if ignores_pattern_check(@check.url_ignores)
 
         # ignore user defined hrefs
-        if 'LinkCheckable' === @type
+        if 'LinkCheckable' == @type
           return true if ignores_pattern_check(@check.href_ignores)
         end
 
         # ignore user defined alts
-        if 'ImageCheckable' === @type
+        if 'ImageCheckable' == @type
           return true if ignores_pattern_check(@check.alt_ignores)
         end
       end
@@ -102,7 +103,7 @@ module HTML
       def file_path
         return if path.nil?
 
-        if path =~ /^\// # path relative to root
+        if path =~ %r{^/} # path relative to root
           base = File.directory?(@check.src) ? @check.src : File.dirname(@check.src)
         elsif File.exist?(File.expand_path path, @check.src) # relative links, path is a file
           base = File.dirname @check.path
@@ -158,7 +159,6 @@ module HTML
       def real_attr(attr)
         attr.to_s unless attr.nil? || attr.empty?
       end
-
     end
   end
 end