grell 1.3.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 85e9d00d79051e54ba1c5361e7f1edfd8abf8af6
+  data.tar.gz: 25b1f2db2ef87e61294158843fe6e396ed15b046
+SHA512:
+  metadata.gz: b00755884c96ddf04e6954d5761903e28cbd46207f45ba5a97f1f1b727b70aad2361386a35e3c96ecb8b65089f1412ac4d7a1bdc8a18d70b87009198883205c3
+  data.tar.gz: 927714b4cd7e7b520d75755e993eff343cebd06a8c36993bcd81abb7453e1ace497e05766f0b8d1ae4bed2e7c00447a1ded267c20028d6d118f32d661c36685e

data/.rspec

@@ -0,0 +1,2 @@
+--color
+--require spec_helper

data/CHANGELOG.md

@@ -0,0 +1,32 @@
+* Version 1.3
+  The Crawler object allows you to provide an external logger object.
+  Clearer semantics when an error happens, special headers are returned so the user can inspect the error
+
+  Caveats:
+  - The 'debug' option in the crawler does not have any affect anymore. Provide an external logger with 'logger' instead
+  - The errors provided in the headers by grell has changed from 'grell_status' to 'grellStatus'.
+  - The 'visited' property in the page was never supposed to be accesible. Use 'visited?' instead.
+
+* Version 1.2.1
+  Solve bug: URLs are case insensitive
+
+* Version 1.2
+  Grell now will consider two links to point to the same page only when the whole URL is exactly the same.
+  Versions previously would only consider two links to be the same when they shared the path.
+
+* Version 1.1.2
+  Solve bug where we were adding links in heads as if there were normal links in the body
+
+* Version 1.1.1
+  Solve bug with the new data-href functionality
+
+* Version 1.1
+  Solve problem with randomly failing spec
+  Search for elements with 'href' or 'data-href' to find links
+
+* Version 1.0.1
+  Rescueing Javascript errors
+
+* Version 1.0
+  Initial implementation
+  Basic support to crawling pages.

data/Gemfile

@@ -0,0 +1,3 @@
+source 'https://rubygems.org'
+
+gemspec

data/LICENSE.txt

@@ -0,0 +1,22 @@
+Copyright (c) 2015 Medidata Solutions Worldwide
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,78 @@
+# Grell
+
+Grell is a generic crawler for the web written in Ruby.
+It can be used to gather data, test pages in a given domain, etc.
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'grell'
+```
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install grell
+
+Grell uses PhantomJS, you will need to download and install it in your
+system. Check for instructions in http://phantomjs.org/
+Grell has been tested with PhantomJS v1.9.x
+
+## Usage
+
+
+### Crawling an entire site
+
+The main entry point of the library is Grell#start_crawling.
+Grell will yield to your code with each page it finds:
+
+```ruby
+require 'grell'
+
+crawler = Grell::Crawler.new
+crawler.start_crawling('http://www.google.com') do |page|
+  #Grell will keep iterating this block which each unique page it finds
+  puts "yes we crawled #{page.url}"
+  puts "status: #{page.status}"
+  puts "headers: #{page.headers}"
+  puts "body: #{page.body}"
+  puts "We crawled it at #{page.timestamp}"
+  puts "We found #{page.links.size} links"
+  puts "page id and parent_id #{page.id}, #{page.parent_id}"
+end
+
+```
+
+Grell keeps a list of pages previously crawled and do not visit the same page twice.
+This list is indexed by the complete url, including query parameters.
+
+### Pages' id
+
+Each page has an unique id, accessed by the property 'id'. Also each page stores the id of the page from which we found this page, accessed by the property 'parent_id'.
+The page object generated by accessing the first URL passed to the start_crawling(the root) has a 'parent_id' equal to 'nil' and an 'id' equal to 0.
+Using this information it is possible to construct a directed graph.
+
+
+### Errors
+When there is an error in the page or an internal error in the crawler (Javascript crashed the browser, etc). Grell will return with status 404 and the headers will have the following keys:
+- grellStatus: 'Error'
+- errorClass: The class of the error which broke this page.
+- errorMessage: A descriptive message with the information Grell could gather about the error.
+
+
+## Tests
+
+Run the tests with
+```ruby
+bundle exec rake ci
+```
+
+## Contributors
+Grell is (c) Medidata Solutions Worldwide and owned by its major contributors:
+* [Teruhide Hoshikawa](https://github.com/thoshikawa-mdsol)
+* [Jordi Polo Carres](https://github.com/jcarres-mdsol)

data/Rakefile

@@ -0,0 +1,2 @@
+require 'kender/tasks'
+

data/grell.gemspec

@@ -0,0 +1,30 @@
+# coding: utf-8
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'grell/version'
+
+Gem::Specification.new do |spec|
+  spec.name          = "grell"
+  spec.version       = Grell::VERSION
+  spec.authors       = ["Jordi Polo Carres"]
+  spec.email         = ["jcarres@mdsol.com"]
+  spec.summary       = %q{Ruby web crawler}
+  spec.description   = %q{Ruby web crawler using PhantomJS}
+  spec.homepage      = "https://github.com/mdsol/grell"
+
+  spec.files         = `git ls-files -z`.split("\x0")
+  spec.executables   = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
+  spec.test_files    = spec.files.grep(%r{^(test|spec|features)/})
+  spec.require_paths = ["lib"]
+
+  spec.add_dependency 'capybara', '~> 2.2'
+  spec.add_dependency 'poltergeist', '~> 1.5'
+
+  spec.add_development_dependency "bundler", "~> 1.6"
+  spec.add_development_dependency "byebug", "~> 4.0"
+  spec.add_development_dependency "kender"
+  spec.add_development_dependency "rake"
+  spec.add_development_dependency "webmock", '~> 1.18'
+  spec.add_development_dependency 'rspec', '~> 3.0'
+  spec.add_development_dependency 'puffing-billy', '~> 0.5'
+end

data/lib/grell.rb

@@ -0,0 +1,10 @@
+require 'capybara/poltergeist'
+require 'capybara/dsl'
+
+require 'grell/grell_logger'
+require 'grell/capybara_driver'
+require 'grell/crawler'
+require 'grell/rawpage'
+require 'grell/page'
+require 'grell/page_collection'
+require 'grell/reader'

data/lib/grell/capybara_driver.rb

@@ -0,0 +1,34 @@
+
+module Grell
+
+  #The driver for Capybara. It uses Portelgeist to control PhantomJS
+  class CapybaraDriver
+    include Capybara::DSL
+
+    USER_AGENT = "Mozilla/5.0 (Grell Crawler)"
+
+    def self.setup(options)
+      new.setup_capybara unless options[:external_driver]
+    end
+
+    def setup_capybara
+      Capybara.register_driver :poltergeist_crawler do |app|
+        Capybara::Poltergeist::Driver.new(app, {
+          js_errors: false,
+          inspector: false,
+          phantomjs_logger: open('/dev/null'),
+          phantomjs_options: ['--debug=no', '--load-images=no', '--ignore-ssl-errors=yes', '--ssl-protocol=TLSv1']
+         })
+      end
+
+      Capybara.default_wait_time = 3
+      Capybara.run_server = false
+      Capybara.default_driver = :poltergeist_crawler
+      page.driver.headers = {
+        "DNT" => 1,
+        "User-Agent" => USER_AGENT
+      }
+    end
+  end
+
+end

data/lib/grell/crawler.rb

@@ -0,0 +1,44 @@
+
+module Grell
+
+  # This is the class that starts and controls the crawling
+  class Crawler
+    attr_reader :collection
+
+    def initialize(options = {})
+      CapybaraDriver.setup(options)
+
+      if options[:logger]
+        Grell.logger = options[:logger]
+      else
+        Grell.logger = Logger.new(STDOUT)
+      end
+
+      @collection = PageCollection.new
+    end
+
+
+    def start_crawling(url, &block)
+      Grell.logger.info "GRELL Started crawling"
+      @collection = PageCollection.new
+      @collection.create_page(url, nil)
+      while !@collection.discovered_pages.empty?
+        crawl(@collection.next_page, block)
+      end
+      Grell.logger.info "GRELL finished crawling"
+    end
+
+    def crawl(site, block)
+      Grell.logger.info "Visiting #{site.url}, visited_links: #{@collection.visited_pages.size}, discovered #{@collection.discovered_pages.size}"
+      site.navigate
+
+      block.call(site) if block
+
+      site.links.each do |url|
+        @collection.create_page(url, site.id)
+      end
+    end
+
+  end
+
+end

data/lib/grell/grell_logger.rb

@@ -0,0 +1,10 @@
+require 'logger'
+
+#Very simple global logger for our crawler.
+module Grell
+  class << self
+    attr_accessor :logger
+  end
+end
+
+Grell.logger = Logger.new(STDOUT)

data/lib/grell/page.rb

@@ -0,0 +1,231 @@
+require 'forwardable'
+
+module Grell
+ # This class contains the logic related to work with each page we crawl. It is also the interface we use
+ # To access the information of each page.
+ # This information comes from result private classes below.
+  class Page
+    extend Forwardable
+
+    WAIT_TIME = 10
+    WAIT_INTERVAL = 0.5
+
+    attr_reader :url, :timestamp, :id, :parent_id, :rawpage
+    #Most of the interesting information accessed through this class is accessed by the methods below
+    def_delegators :@result_page, :headers, :body, :status, :links, :has_selector?, :host, :visited?
+
+    def initialize( url, id, parent_id)
+      @rawpage = RawPage.new
+      @url = url
+      @id = id
+      @parent_id = parent_id
+      @timestamp = nil
+      @result_page = UnvisitedPage.new
+    end
+
+    def navigate
+      # We wait a maximum of WAIT_TIME seconds to get an HTML page. We try or best to workaround inconsistencies on poltergeist
+      Reader.wait_for(->{@rawpage.navigate(url)}, WAIT_TIME, WAIT_INTERVAL ) do
+        @rawpage.status && !@rawpage.headers.empty? &&
+          @rawpage.headers["Content-Type"] && @rawpage.headers["Content-Type"].include?('text/html').equal?(true)
+      end
+      @result_page = VisitedPage.new(@rawpage)
+      @timestamp = Time.now
+    rescue Capybara::Poltergeist::JavascriptError => e
+      unavailable_page(404, e)
+    rescue Capybara::Poltergeist::BrowserError => e #This may happen internally on Poltergeist, they claim is a bug.
+      unavailable_page(404, e)
+    rescue URI::InvalidURIError => e #No cool URL means we report error
+      unavailable_page(404, e)
+    rescue Capybara::Poltergeist::TimeoutError => e #Poltergeist has its own timeout which is similar to Chromes.
+      unavailable_page(404, e)
+    rescue Capybara::Poltergeist::StatusFailError => e
+      unavailable_page(404, e)
+    end
+
+    private
+    def unavailable_page(status, exception)
+      Grell.logger.warn "The page with the URL #{@url} was not available. Exception #{exception}"
+      @result_page = ErroredPage.new(status, exception)
+      @timestamp = Time.now
+    end
+
+    # Private class.
+    # This is a result page when it has not been visited yet. Essentially empty of information
+    #
+    class UnvisitedPage
+      def status
+        nil
+      end
+
+      def body
+        ''
+      end
+
+      def headers
+        {grellStatus: 'NotVisited' }
+      end
+
+      def links
+        []
+      end
+
+      def host
+        ''
+      end
+
+      def visited?
+        false
+      end
+
+      def has_selector?(selector)
+        false
+      end
+
+    end
+
+    # Private class.
+    # This is a result page when some error happened. It provides some information about the error.
+    #
+    class ErroredPage
+      def initialize(error_code, exception)
+        @error_code = error_code
+        @exception = exception
+      end
+
+      def status
+        @error_code
+      end
+
+      def body
+        ''
+      end
+
+      def headers
+        message = begin
+          @exception.message
+        rescue StandardError
+          "Error message can not be accessed" #Poltergeist may try to access a nil object when accessing message
+        end
+
+        {
+          grellStatus: 'Error',
+          errorClass: @exception.class.to_s,
+          errorMessage: message
+        }
+      end
+
+      def links
+        []
+      end
+
+      def host
+        ''
+      end
+
+      def visited?
+        true
+      end
+
+      def has_selector?(selector)
+        false
+      end
+
+    end
+
+
+    # Private class.
+    # This is a result page when we successfully got some information back after visiting the page.
+    # It delegates most of the information to the @rawpage capybara page. But any transformation or logic is here
+    #
+    class VisitedPage
+      def initialize(rawpage)
+        @rawpage = rawpage
+      end
+
+      def status
+        @rawpage.status
+      end
+
+      def body
+        @rawpage.body
+      end
+
+      def headers
+        @rawpage.headers
+      rescue Capybara::Poltergeist::BrowserError => e #This may happen internally on Poltergeist, they claim is a bug.
+        {
+          grellStatus: 'Error',
+          errorClass: e.class.to_s,
+          errorMessage: e.message
+        }
+      end
+
+      def links
+        @links ||= all_links
+      end
+
+      def host
+        @rawpage.host
+      end
+
+      def visited?
+        true
+      end
+
+      def has_selector?(selector)
+        @rawpage.has_selector?(selector)
+      end
+
+      private
+      def all_links
+        # <link> can only be used in the <head> as of: https://developer.mozilla.org/en/docs/Web/HTML/Element/link
+        anchors_in_body = @rawpage.all_anchors.reject{|anchor| anchor.tag_name == 'link' }
+
+        unique_links = anchors_in_body.map do |anchor|
+         anchor['href'] || anchor['data-href']
+        end.compact
+
+        unique_links.map{|link| link_to_url(link)}.uniq.compact
+
+      rescue Capybara::Poltergeist::ObsoleteNode
+        Grell.logger.warn "We found an obsolete node in #{@url}. Ignoring all links"
+        # Sometimes Javascript and timing may screw this, we lose these links.
+        # TODO: Can we do something more intelligent here?
+        []
+      end
+
+      # We only accept links in this same host that start with a path
+      # nil from this
+      def link_to_url(link)
+        uri = URI.parse(link)
+        if uri.absolute?
+          if uri.host != URI.parse(host).host
+            Grell.logger.debug "GRELL does not follow links to external hosts: #{link}"
+            nil
+          else
+            link # Absolute link to our own host
+          end
+        else
+          if uri.path.nil?
+            Grell.logger.debug "GRELL does not follow links without a path: #{uri}"
+            nil
+          end
+          if uri.path.start_with?('/')
+            host + link  #convert to full URL
+          else #links like href="google.com" the browser would go to http://google.com like "http://#{link}"
+            Grell.logger.debug "GRELL Bad formatted link: #{link}, assuming external"
+            nil
+          end
+        end
+
+      rescue URI::InvalidURIError #We will have invalid links propagating till we navigate to them
+        link
+      end
+    end
+
+
+
+  end
+
+end