w3clove 0.2.1 → 0.3.0

data/README.rdoc

@@ -2,21 +2,7 @@
 
 This is my {Ruby Mendicant University}[http://university.rubymendicant.com/] personal project, and is currently in alpha status.
 
-I want to build a site-wide markup validator, a Ruby gem that lets you validate a whole web site against the W3C Markup Validator, from the command line, and generate a comprehensive report of all errors found.
-
-Currently, the official {W3C Validator site}[http://validator.w3.org/] only lets you validate one URL at a time, so when you want to validate all the pages on a web site, it can be a tedious process. There is a {related tool}[http://www.htmlhelp.com/tools/validator/batch.html.en] that lets you use a batch mode for this and submit a list of URLs to be checked, but it is still a semi-manual process, and the output is not very useful.
-
-My plan then is building a command line utility that would accept as input a XML sitemap file, or its URL, expecting it to be on the {Google Sitemap format}[http://en.wikipedia.org/wiki/Google_Sitemaps]. This utility will then check the markup validation of each URL on this sitemap querying the W3C Validator, and store all detected errors and warnings. After checking all the URLs, it will generate as output an HTML file, with a style similar to what RCov[https://github.com/relevance/rcov] produces, showing all these errors on an easy to read format, grouping common errors together, sorting them by popularity, and linking to the URLs and to the explanations on how to correct them.
-
-Internally, it would use the {w3c_validators gem}[http://rubygems.org/gems/w3c_validators] to do the individual checks, so my gem would be concerned only with the XML sitemap parsing, building the queue, storing the errors, grouping and sorting them, and producing the HTML output.
-
-I've already done something similar to this, I sent {a little contribution to docrails}[https://github.com/lifo/docrails/blob/master/railties/guides/w3c_validator.rb] that checks the generated guides using this gem.
-
-= Bonus points:
-
-* in addition to an XML file, accept as input the URL of a site and crawl the site to find all internal links
-* validate the markup locally, without querying the W3C site, for more speed and to not saturate the W3C site
-* store the results on a local database, so on subsequent checks, only the pages that had errors are re-checked (unless a --checkall force flag is passed). This way developers can check the whole site, get the errors, deploy the corrections, and recheck the site.
+It's a site-wide markup validator, a Ruby gem that lets you validate a whole web site against the W3C Markup Validator, from the command line, and generate a comprehensive report of all errors found.
 
 = Installation:
 
@@ -28,7 +14,9 @@ w3clove is a Ruby gem that can be installed on the usual way
 
 Pass it the url of an XML sitemap to be checked, like:
 
-  w3clove http://www.ryanair.com/sitemap.xml
+  w3clove https://github.com/jaimeiniesta/w3clove/raw/master/spec/samples/sitemap.xml
+
+This will validate all the URLs of your sitemap.xml (so be sure to pass it a short one or you'll wait for ages), and generate an HTML file with the full report.
 
 = Notes:
 

data/lib/w3clove.rb

@@ -5,4 +5,5 @@ module W3Clove
   require_relative './w3clove/sitemap'
   require_relative './w3clove/page'
   require_relative './w3clove/message'
+  require_relative './w3clove/reporter'
 end

data/lib/w3clove/page.rb

@@ -1,5 +1,6 @@
 # -*- encoding: utf-8 -*-
 
+require 'timeout'
 require 'w3c_validators'
 include W3CValidators
 
@@ -16,15 +17,19 @@ module W3Clove
     end
 
     ##
-    # Checks for errors and returns true if none found, false otherwise
-    # warnings are not considered as validation errors so a page with
-    # warnings but without errors will return true
+    # Checks for errors and returns true if none found, false otherwise.
+    # Warnings are not considered as validation errors so a page with
+    # warnings but without errors will return true.
     # If the validation goes well, errors should be an array. Otherwise
-    # it will still be nil, which will not be considered validated
+    # it will still be nil, which will not be considered validated.
     def valid?
       !errors.nil? && errors.empty?
     end
 
+    ##
+    # Returns the collection of errors from the validations of this page.
+    # If it has no validation errors, it will be an empty array.
+    # It an exception occurs, it will be nil.
     def errors
       @errors ||= validations.errors.map {|e|
                                           W3Clove::Message.new(e.message_id,
@@ -36,6 +41,10 @@ module W3Clove
       nil
     end
 
+    ##
+    # Returns the collection of warnings from the validations of this page.
+    # If it has no validation warnings, it will be an empty array.
+    # It an exception occurs, it will be nil.
     def warnings
       @warnings ||= validations.warnings.map {|w|
                                               W3Clove::Message.new(w.message_id,
@@ -49,8 +58,10 @@ module W3Clove
 
     private
 
+    ##
+    # Gets the validations for this page, ensuring it times out soon
     def validations
-      @validations ||= MarkupValidator.new.validate_uri(url)
+      @validations ||= Timeout::timeout(10) { MarkupValidator.new.validate_uri(url) }
     end
   end
 end

data/lib/w3clove/reporter.rb

@@ -0,0 +1,21 @@
+# -*- encoding: utf-8 -*-
+
+require 'erb'
+
+module W3Clove
+  module Reporter
+    extend self
+
+    ##
+    # Create the html report for the sitemap
+    def generate_html(sitemap)
+      template = ERB.new(open(File.dirname(__FILE__)+'/templates/w3clove.html.erb').read)
+
+      File.open('w3clove.html', 'w') do |f|
+                                       f.write(template.result(sitemap.get_binding))
+                                     end
+    rescue Exception => e
+      puts "ERROR generating report: #{e}"
+    end
+  end
+end

data/lib/w3clove/sitemap.rb

@@ -19,11 +19,15 @@ module W3Clove
     end
 
     def errors
-      pages.map {|p| p.errors}.flatten
+      @errors ||= pages.map {|p| p.errors}.flatten.reject {|e| e.nil?}
     end
 
     def warnings
-      pages.map {|p| p.warnings}.flatten
+      @warnings ||= pages.map {|p| p.warnings}.flatten.reject {|e| e.nil?}
+    end
+
+    def get_binding
+      binding
     end
 
     private

data/lib/w3clove/templates/w3clove.html.erb

@@ -0,0 +1,93 @@
+<html>
+  <head>
+    <title>W3Clove report for <%= @url %></title>
+    <link href='http://fonts.googleapis.com/css?family=Cabin+Sketch:bold' rel='stylesheet' type='text/css'>
+    <style type="text/css">
+      #header, #main, #footer {
+        padding: 5px; margin: 5px;
+        background-color: #eee;
+        border: 1px solid black;
+      }
+      #header, h1, h2, h3 {
+        font-family: 'Cabin Sketch', arial, serif;
+      }
+      #header {
+        font-size: 2.5em;
+      }
+      #footer {
+        font-size: .85em;
+        text-align: center;
+      }
+      .page {
+        padding: 5px; margin: 5px;
+        border: 1px solid black;
+        background-color: white;
+      }
+    </style>
+  </head>
+  <body>
+    <div id="header">
+      W3Clove :: site-wide markup validation
+    </div>
+
+    <div id="main">
+      <h1>Report for <%= @url %></h1>
+
+      <h2>SITEMAP SUMMARY</h2>
+      <p>TOTAL: <%= errors.length %> errors, <%= warnings.length %> warnings.</p>
+
+      <% if errors.length > 0 %>
+        <h2>POPULAR ERRORS</h2>
+        <ul>
+        <% errors.group_by {|e| e.message_id}.sort_by {|m,e| e.length}.reverse.each do |message_id, errors| %>
+          <li>error <a href="http://validator.w3.org/docs/errors.html#ve-<%= message_id %>"><%= message_id %></a> happens <%= errors.length %> times</li>
+        <% end %>
+        </ul>
+      <% end %>
+
+      <% if warnings.length > 0 %>
+        <h2>POPULAR WARNINGS</h2>
+        <ul>
+        <% warnings.group_by {|w| w.message_id}.sort_by {|m,w| w.length}.reverse.each do |message_id, warnings| %>
+          <li>warning <a href="http://validator.w3.org/docs/errors.html#ve-<%= message_id %>"><%= message_id %></a> found <%= warnings.length %> times</li>
+        <% end %>
+        </ul>
+      <% end %>
+
+      <% processed_pages = pages.select{|p| !p.exception} %>
+      <% if processed_pages.size > 0 %>
+        <h2>DETAILS PER PAGE</h2>
+        <% processed_pages.each do |page| %>
+          <div class='page'>
+            <h3><%= page.url %></h3>
+            <p class='page_summary'>
+              <%= "#{page.errors.length} errors, #{page.warnings.length} warnings" %>
+            </p>
+
+            <ul class="page_errors">
+            <% page.errors.each do |error| %>
+              <li>
+                Error <a href="http://validator.w3.org/docs/errors.html#ve-<%= error.message_id %>"><%= error.message_id %></a> on line <%= error.line %>:
+                <%= error.text %>
+              </li>
+            <% end %>
+            </ul>
+
+            <ul class="page_warnings">
+            <% page.warnings.each do |warning| %>
+              <li>
+                Warning <a href="http://validator.w3.org/docs/errors.html#ve-<%= warning.message_id %>"><%= warning.message_id %></a> on line <%= warning.line %>:
+                <%= warning.text %>
+              </li>
+            <% end %>
+            </ul>
+          </div>
+        <% end %>
+      <% end %>
+    </div>
+
+    <div id="footer">
+      <strong>w3clove</strong>@2011 <a href='http://jaimeiniesta.com'>Jaime Iniesta</a>. Get w3clove from <a href="http://github.com/jaimeiniesta/w3clove">github</a>.
+    </div>
+  </body>
+</html>

data/lib/w3clove/validator.rb

@@ -14,71 +14,21 @@ module W3Clove
     # Parses a remote xml sitemap and checks markup validation for each url
     # Shows progress on dot-style (...F...FFE..). A dot is a valid page,
     # an F is a page with errors, and an E is an exception
-    # After the checking is done, a detailed summary is shown
+    # After the checking is done, a detailed summary is generated
     def check(url)
       sitemap = W3Clove::Sitemap.new(url)
-      say "Validating #{sitemap.pages.length} pages..."
+      say "Validating #{sitemap.pages.length} pages"
 
       sitemap.pages.each do |page|
         say_inline page.valid? ? "." : (page.errors.nil? ? 'E' : 'F')
       end
 
-      show_results(sitemap)
+      W3Clove::Reporter.generate_html(sitemap)
+      say "\nValidation finished, see the report at w3clove.html"
     end
 
     private
 
-    ##
-    # Outputs the results of the validation
-    def show_results(sitemap)
-      show_sitemap_summary(sitemap)
-      show_popular_errors(sitemap)
-      show_popular_warnings(sitemap)
-      say "\n\nDETAILS PER PAGE"
-      sitemap.pages.select {|page| !page.errors.empty?}.each do |p|
-        show_page_summary(p)
-      end
-    end
-
-    def show_sitemap_summary(sitemap)
-      <<HEREDOC
-SITEMAP SUMMARY
-TOTAL: #{sitemap.errors.length} errors, #{sitemap.warnings.length} warnings
-HEREDOC
-    end
-
-    def show_popular_errors(sitemap)
-      say "\n\nMOST POPULAR ERRORS\n"
-      sitemap.errors.group_by {|e| e.message_id}
-        .sort_by {|m,e| e.length}
-        .reverse.each do |message_id, errors|
-          say "error #{message_id} happens #{errors.length} times"
-        end
-    end
-
-    def show_popular_warnings(sitemap)
-      say "\n\nMOST POPULAR WARNINGS\n"
-      sitemap.warnings.group_by {|e| e.message_id}
-        .sort_by {|m,e| e.length}
-        .reverse.each do |message_id, warnings|
-          say "warning #{message_id} happens #{warnings.length} times"
-        end
-    end
-
-    def show_page_summary(page)
-      say "\n  ** #{page.url} **"
-      "    #{page.errors.length} errors, #{page.warnings.length} warnings"
-      page.errors.each do |error|
-        say "\n    Error #{error.message_id} on line #{error.line}:"
-        say "    #{error.text}"
-      end
-
-      page.warnings.each do |warning|
-        say "\n    Warning #{warning.message_id} on line #{warning.line}:"
-        say "    #{warning.text}"
-      end
-    end
-
     def printer
       @printer ||= STDOUT
     end

data/lib/w3clove/version.rb

@@ -1,5 +1,5 @@
 # -*- encoding: utf-8 -*-
 
 module W3Clove
-  VERSION = "0.2.1"
+  VERSION = "0.3.0"
 end

metadata

@@ -4,9 +4,9 @@ version: !ruby/object:Gem::Version
   prerelease: false
   segments: 
   - 0
-  - 2
-  - 1
-  version: 0.2.1
+  - 3
+  - 0
+  version: 0.3.0
 platform: ruby
 authors: 
 - Jaime Iniesta
@@ -14,7 +14,7 @@ autorequire:
 bindir: bin
 cert_chain: []
 
-date: 2011-03-22 00:00:00 +01:00
+date: 2011-03-23 00:00:00 +01:00
 default_executable: 
 dependencies: 
 - !ruby/object:Gem::Dependency 
@@ -90,7 +90,9 @@ files:
 - lib/w3clove.rb
 - lib/w3clove/message.rb
 - lib/w3clove/page.rb
+- lib/w3clove/reporter.rb
 - lib/w3clove/sitemap.rb
+- lib/w3clove/templates/w3clove.html.erb
 - lib/w3clove/validator.rb
 - lib/w3clove/version.rb
 - spec/message_spec.rb