site_checker 0.1.1 → 0.2.0.pre

data/.rbenv-version

@@ -0,0 +1 @@
+1.9.3-p125

data/.rspec

@@ -0,0 +1 @@
+--color

data/History.md

@@ -0,0 +1,9 @@
+## [v0.1.1](https://github.com/ZsoltFabok/site_checker/compare/v0.1.0...v0.1.1)
+
+### Fixes
+* better dependency description in the gemspec (Zsolt Fabok)
+
+## [v0.1.0](https://github.com/ZsoltFabok/site_checker/tree/v0.1.0)
+
+### Notes
+First version

data/LICENSE

@@ -0,0 +1,29 @@
+New BSD License (3-clause license)
+
+Copyright (c) 2012, Zsolt Fabok
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+    * Redistributions of source code must retain the above copyright
+      notice, this list of conditions and the following disclaimer.
+
+    * Redistributions in binary form must reproduce the above copyright
+      notice, this list of conditions and the following disclaimer in the
+      documentation and/or other materials provided with the distribution.
+
+    * Neither the name of Zsolt Fabok nor the
+      names of its contributors may be used to endorse or promote products
+      derived from this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
+ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER BE LIABLE FOR ANY
+DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
+(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

data/README.md

@@ -0,0 +1,102 @@
+###Site Checker [![Code Climate](https://codeclimate.com/badge.png)](https://codeclimate.com/github/ZsoltFabok/site_checker) [![Build Status](https://travis-ci.org/ZsoltFabok/site_checker.png)](https://travis-ci.org/ZsoltFabok/site_checker) [![Dependency Status](https://gemnasium.com/ZsoltFabok/site_checker.png)](https://gemnasium.com/ZsoltFabok/site_checker)
+
+
+Site Checker is a simple ruby gem, which helps you check the integrity of your website by recursively visiting the referenced pages and images. I use it in my test environments to make sure that my websites don't have any dead links.
+
+### Install
+
+    gem install site_checker
+
+### Usage
+
+#### Default
+
+First, you have to load the `site_checker` by adding this line to the file where you would like to use it:
+
+    require 'site_checker'
+
+If you want to use it for testing, the line should goto the `test_helper.rb`.
+
+The usage is quite simple:
+
+    check_site("http://localhost:3000/app", "http://localhost:3000")
+    puts collected_remote_pages.inspect
+    puts collected_local_pages.inspect
+    puts collected_remote_images.inspect
+    puts collected_local_images.inspect
+    puts collected_problems.inspect
+
+The snippet above will open the `http://localhost:3000/app` link and will look for links and images. If it finds a link to a local page, it will recursively checkout out that page, too. The second argument - `http://localhost:3000` - defines the starting reference of your website.
+
+In case you don't want to use a DSL like API you can still do the following:
+
+    SiteChecker.check("http://localhost:3000/app", "http://localhost:3000")
+    puts SiteChecker.remote_pages.inspect
+    puts SiteChecker.local_pages.inspect
+    puts SiteChecker.remote_images.inspect
+    puts SiteChecker.local_images.inspect
+    puts SiteChecker.problems.inspect
+
+### Using on Generated Content
+If you have a static website (e.g. generated by [octopress](https://github.com/imathis/octopress)) you can tell `site_checker` to use folders from the file system. With this approach, you don't need a webserver for verifying your website:
+
+    check_site("./public", "./public")
+    puts collected_problems.inspect
+
+### Configuration
+You can instruct `site_checker` to ignore certain links:
+
+    SiteChecker.configure do |config|
+      config.ignore_list = ["/", "/atom.xml"]
+    end
+
+By default it won't check the conditions of the remote links and images - e.g. 404 or 500 -, but you can change it like this:
+
+    SiteChecker.configure do |config|
+      config.visit_references = true
+    end
+
+Too deep recursive calls may be expensive, so you can configure the maximum depth of the recursion with the following attribute:
+
+    SiteChecker.configure do |config|
+      config.max_recursion_depth = 3
+    end
+
+### Examples
+Make sure that there are no local dead links on the website (I'm using [rspec](https://github.com/rspec/rspec) syntax):
+
+    before(:each) do
+      SiteChecker.configure do |config|
+        config.ignore_list = ["/atom.xml", "/rss"]
+      end
+    end
+
+    it "should not have dead local links" do
+      check_site("http://localhost:3000", "http://localhost:3000")
+      # this will print out the difference and I don't have to re-run with print
+      collected_problems.should be_empty
+    end
+
+Check that all the local pages can be reached with maximum two steps:
+
+    before(:each) do
+      SiteChecker.configure do |config|
+        config.ignore_list = ["/atom.xml", "/rss"]
+        config.max_recursion_depth = 2
+      end
+
+      @number_of_local_pages = 100
+    end
+
+    it "all the local pages have to be visited" do
+      check_site("http://localhost:3000", "http://localhost:3000")
+      collected_local_pages.size.should eq @number_of_local_pages
+    end
+
+### Troubleshooting
+#### undefined method 'new' for SiteChecker:Module
+This error occurs when the test code calls v0.1.1 methods, but a newer version of the gem has already been installed. Update your test code following the examples above.
+
+### Copyright
+
+Copyright (c) 2012 Zsolt Fabok and Contributors. See LICENSE for details.

data/gem_tasks/rspec.rake

@@ -0,0 +1,6 @@
+require 'rspec/core/rake_task'
+
+RSpec::Core::RakeTask.new(:spec)
+
+desc "By default run the test cases"
+task :default  => :spec

data/gem_tasks/yard.rake

@@ -0,0 +1,6 @@
+require 'yard'
+require 'yard/rake/yardoc_task'
+
+YARD::Rake::YardocTask.new do |t|
+  t.files   = ['lib/**/*.rb']
+end

data/lib/site_checker/dsl.rb

@@ -0,0 +1,17 @@
+module SiteChecker
+  module DSL
+    { :check_site              => :check,
+      :collected_local_pages   => :local_pages,
+      :collected_remote_pages  => :remote_pages,
+      :collected_local_images  => :local_images,
+      :collected_remote_images => :remote_images,
+      :collected_problems      => :problems
+     }.each do |dsl_method, method|
+      define_method dsl_method do |*args, &block|
+        SiteChecker.send method, *args, &block
+      end
+    end
+  end
+end
+
+include SiteChecker::DSL

data/lib/site_checker/io/content_from_file_system.rb

@@ -0,0 +1,43 @@
+module SiteChecker
+  module IO
+    class ContentFromFileSystem
+
+      def initialize(visit_references, root)
+        @visit_references = visit_references
+        @root = root
+      end
+
+      def get(link)
+        begin
+          location = create_absolute_reference(link.url)
+          if link.local_page?
+            content = File.open(add_index_html(location)).read
+          elsif link.local_image?
+            File.open(location)
+          elsif @visit_references
+            open(link.url)
+          end
+        rescue Errno::ENOENT => e
+          raise "(404 Not Found)"
+        rescue => e
+          raise "(#{e.message.strip})"
+        end
+        content
+      end
+
+      private
+      def add_index_html(path)
+        path = $1 if path.match(/(.+)#/)
+        path.end_with?(".html") ? path : File.join(path, "index.html")
+      end
+
+      def create_absolute_reference(link)
+        if !link.eql?(@root)
+          File.join(@root, link)
+        else
+          @root
+        end
+      end
+    end
+  end
+end

data/lib/site_checker/io/content_from_web.rb

@@ -0,0 +1,36 @@
+module SiteChecker
+  module IO
+    class ContentFromWeb
+
+    	def initialize(visit_references, root)
+    		@visit_references = visit_references
+    		@root = root
+    	end
+
+    	def get(link)
+        begin
+          uri = create_absolute_reference(link.url)
+          if link.local_page?
+            content = open(uri)
+          elsif link.local_image?
+            open(uri)
+          elsif @visit_references
+            open(uri)
+          end
+        rescue => e
+          raise "(#{e.message.strip})"
+        end
+        content
+      end
+
+      private
+      def create_absolute_reference(link)
+        if link.start_with?(@root)
+          URI(link)
+        else
+          URI(@root).merge(link)
+        end
+      end
+    end
+  end
+end

data/lib/site_checker/link.rb

@@ -0,0 +1,60 @@
+module SiteChecker
+  class Link
+  	attr_accessor :url
+  	attr_accessor :parent_url
+  	attr_accessor :kind
+  	attr_accessor :location
+  	attr_accessor :problem
+
+  	def eql?(other)
+      ignore_trailing_slash(@url).eql? ignore_trailing_slash(other.url)
+    end
+
+    def ==(other)
+      eql?(other)
+    end
+
+    def hash
+      ignore_trailing_slash(@url).hash
+    end
+
+    def self.create(attrs)
+    	link = Link.new
+    	attrs.each do |key, value|
+    		if self.instance_methods.include?("#{key}=".to_sym)
+    			eval("link.#{key}=value")
+    		end
+    	end
+    	link
+    end
+
+    def has_problem?
+    	@problem != nil
+    end
+
+    def local_page?
+    	@location == :local && @kind == :page
+    end
+
+    def local_image?
+    	@location == :local && @kind == :image
+    end
+
+    def anchor?
+    	@kind == :anchor
+    end
+
+    def anchor_ref?
+      @kind == :anchor_ref
+    end
+
+    def anchor_related?
+      anchor? || anchor_ref?
+    end
+
+    private
+    def ignore_trailing_slash(url)
+      url.gsub(/^\//,"")
+    end
+  end
+end

data/lib/site_checker/link_collector.rb

@@ -0,0 +1,153 @@
+module SiteChecker
+  class LinkCollector
+    attr_accessor :ignore_list, :visit_references, :max_recursion_depth
+
+    def initialize
+      yield self if block_given?
+      @ignore_list ||= []
+      @visit_references ||= false
+      @max_recursion_depth ||= -1
+    end
+
+    def check(url, root)
+      @links = []
+      @recursion_depth = 0
+      @root = root
+
+      @content_reader = get_content_reader
+
+      link = Link.create({:url => url, :kind => :page, :location => :local})
+      register_visit(link)
+      process_local_page(link)
+      evaluate_anchors
+    end
+
+    def local_pages
+      get_urls(:local, :page)
+    end
+
+    def remote_pages
+      get_urls(:remote, :page)
+    end
+
+    def local_images
+      get_urls(:local, :image)
+    end
+
+    def remote_images
+      get_urls(:remote, :image)
+    end
+
+    def problems
+      problems = {}
+      @links.each do |link|
+        if link.has_problem?
+          problems[link.parent_url] ||= []
+          problems[link.parent_url] << "#{link.url} #{link.problem}"
+        end
+      end
+      problems
+    end
+
+    private
+    def get_content_reader
+      if URI(@root).absolute?
+        SiteChecker::IO::ContentFromWeb.new(@visit_references, @root)
+      else
+        SiteChecker::IO::ContentFromFileSystem.new(@visit_references, @root)
+      end
+    end
+
+    def get_urls(location, kind)
+      @links.find_all do |link|
+        if link.location == location && link.kind == kind
+          link
+        end
+      end.map do |link|
+        link.url
+      end
+    end
+
+    def process_local_page(parent)
+      links = collect_links(parent)
+
+      links.each do |link|
+        link.parent_url = parent.url
+        unless link.anchor_related?
+          visit(link) unless visited?(link)
+        else
+          @links << link
+        end
+      end
+    end
+
+    def register_visit(link)
+      @links << link unless visited?(link)
+    end
+
+    def visited?(link)
+      @links.include?(link)
+    end
+
+    def visit(link)
+      register_visit(link)
+      unless link.local_page?
+        open_reference(link)
+      else
+        unless stop_recursion?
+          @recursion_depth += 1
+          process_local_page(link)
+          @recursion_depth -= 1
+        end
+      end
+    end
+
+    def open_reference(link)
+      content = nil
+      begin
+        content = @content_reader.get(link)
+      rescue => e
+        link.problem = "#{e.message.strip}"
+      end
+      content
+    end
+
+    def collect_links(link)
+      content = open_reference(link)
+      return SiteChecker::Parse::Page.parse(content, @ignore_list, @root)
+    end
+
+    def stop_recursion?
+      if @max_recursion_depth == -1
+        false
+      elsif @max_recursion_depth > @recursion_depth
+        false
+      else
+        true
+      end
+    end
+
+    def evaluate_anchors
+      anchors = @links.find_all {|link| link.anchor?}
+      anchor_references = @links.find_all {|link| link.anchor_ref?}
+      anchor_references.each do |anchor_ref|
+        if find_matching_anchor(anchors, anchor_ref).empty?
+          anchor_ref.problem = "(404 Not Found)"
+        end
+      end
+    end
+
+    def find_matching_anchor(anchors, anchor_ref)
+      result = []
+      anchors.each do |anchor|
+        if (anchor.parent_url == anchor_ref.parent_url &&
+              anchor_ref.url == "##{anchor.url}") ||
+            (anchor.parent_url != anchor_ref.parent_url &&
+              anchor_ref.url == "#{anchor.parent_url}##{anchor.url}")
+          result << anchor
+        end
+      end
+      result
+    end
+  end
+end

data/lib/site_checker/parse/page.rb

@@ -0,0 +1,82 @@
+module SiteChecker
+	module Parse
+		class Page
+			def self.parse(content, ignore_list, root)
+				links = []
+				page = Nokogiri(content)
+
+				links.concat(get_links(page, ignore_list, root))
+				links.concat(get_images(page, ignore_list, root))
+				links.concat(get_anchors(page))
+				links.concat(local_pages_which_has_anchor_references(links, root))
+
+				links.uniq
+			end
+
+			private
+			def self.get_links(page, ignore_list, root)
+				links = []
+				page.xpath("//a").reject {|a| ignored?(ignore_list, a['href'])}.each do |a|
+					if a['href'].match(/(.*)#.+/) && !URI($1).absolute?
+						kind = :anchor_ref
+					else
+						kind = :page
+					end
+					links << Link.create({:url => a['href'], :kind => kind})
+				end
+				set_location(links, root)
+			end
+
+			def self.get_images(page, ignore_list, root)
+				links = []
+				page.xpath("//img").reject {|img| ignored?(ignore_list, img['src'])}.each do |img|
+					links << Link.create({:url => img['src'], :kind => :image})
+				end
+				set_location(links, root)
+			end
+
+			def self.set_location(links, root)
+				links.each do |link|
+			    uri = URI(link.url)
+			    if uri.to_s.start_with?(root)
+			    	link.problem = "(absolute path)"
+			    	link.location = :local
+			    else
+			      if uri.absolute?
+			        link.location = :remote
+			      else
+			        link.location = :local
+			      end
+			    end
+			  end
+			end
+
+			def self.ignored?(ignore_list, link)
+		    if link
+		      ignore_list.include? link
+		    else
+		      true
+		    end
+		  end
+
+		  def self.get_anchors(page)
+		  	anchors = []
+		  	page.xpath("//a").reject {|a| !a['id']}.each do |a|
+		  		anchors << Link.create({:url => a['id'], :kind => :anchor})
+		  	end
+		  	anchors
+		  end
+
+		  def self.local_pages_which_has_anchor_references(links, root)
+		  	new_links = []
+		  	links.find_all {|link| link.anchor_ref?}.each do |link|
+		  		uri = URI(link.url)
+		  		if link.url.match(/(.+)#/)
+		  			new_links << Link.create({:url => $1, :kind => :page})
+		  		end
+		  	end
+		  	set_location(new_links, root)
+		  end
+		end
+	end
+end