skyscraper 0.0.1

data/.gitignore

@@ -0,0 +1,17 @@
+*.gem
+*.rbc
+.bundle
+.config
+.yardoc
+Gemfile.lock
+InstalledFiles
+_yardoc
+coverage
+doc/
+lib/bundler/man
+pkg
+rdoc
+spec/reports
+test/tmp
+test/version_tmp
+tmp

data/.rspec

@@ -0,0 +1 @@
+--color

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in ..gemspec
+gemspec

data/LICENSE

@@ -0,0 +1,22 @@
+Copyright (c) 2012 Adam Dratwinski
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,180 @@
+# Skyscraper
+
+## Installation
+
+**Skyscraper** installation is simple, just run:
+
+`gem install skyscraper`
+
+or add following entry to your gemfile:
+
+`gem "skyscraper"`
+
+if you want to use it in your rails project.
+
+## Finding nodes by CSS Selectors
+
+```ruby
+>> Skyscraper::fetch("http://rubyonrails.org").first("title").text
+# => "Ruby on Rails"
+
+>> Skyscraper::fetch("http://rubyonrails.org").first(".copyright p").text
+# => "\\"Rails\\", \\"Ruby on Rails\\", and the Rails logo are registered trademarks of David Heinemeier Hansson. All rights reserved."
+```
+
+You can use this thanks to **Nokogiri#css** method.
+
+## Reading HTML attributes
+```ruby
+>> Skyscraper::fetch("http://rubyonrails.org").first(".announce").class
+# => "announce"
+>> Skyscraper::fetch("http://rubyonrails.org").first("img").height
+# => "112"
+>> Skyscraper::fetch("http://rubyonrails.org").first(".copyright").style
+# => "margin-top: 20px"
+
+```
+
+> ## Notice! 
+>**Skyscraper::Node::Base#class** method is overriden, to access original **class** method, please call Skyscraper::Node::Base#original_class
+
+You can find list of all available methods in [Reading attributes Section](wiki/reading_attributes)
+
+## Using Skyscraper as included module
+
+Fetch content from multiple pages and store it in the active record database is a common problem. You can do this quick, using **Skyscraper** as included module.
+
+```ruby
+class Sample
+  include Skyscraper
+
+  settings limit: 10, delay: { after: 5, time: 1 }, encoding: "utf-8"
+
+  pages ["http://google.com", "https://github.com", "http://rubyonrails.org"]
+  # pages method also accepts blocks as argument, then you can use Skyscraper::fetch method inside to get list of pages from website more dynamically
+
+  field :html, "html", :html
+  field :title, "title" do |node|
+    "'#{node.text}'"
+  end
+  field :first_link, "body" do |node|
+    "'#{node.first("a").href}'"
+  end
+  field :first_image, "img", :download
+
+  # field method takes following arguments: 
+  # field_name => name that the record will have in the results table
+  # selector => css selector of fetching element, so it can even looks like "tag #id.some_class"
+  # optionally symbol with the node method or block, if nothing is provided, text method on the node is fired
+
+  after_each do |result|
+    page = Page.new 
+    page.title       = result[:title]
+    page.html        = result[:html]
+    page.first_link  = result[:first_link]
+    page.first_image_path = results[:first_image]
+    page.save
+  end
+
+  after_all do 
+    puts "Job done"
+  end
+end
+
+Sample.new.fetch #this will run above code applying provided callbacks and returns array with results
+```
+You will find more details in [Including section](wiki/Including).
+
+## Traversing
+
+Traversing through **Skyscraper** nodes is very similar to the way **jQuery** provides.
+
+```ruby
+>> Skyscraper::fetch("https://github.com").first(".top-nav").find("li").map(&:html)
+# => ["<a href="https://github.com/plans">Signup and Pricing</a>", "<a href="https://github.com/explore">Explore GitHub</a>", "<a href="https://github.com/features">Features</a>", "<a href="https://github.com/blog">Blog</a>", "<a href="https://github.com/login">Login</a>"]
+```
+
+Of course you can write the same code in the easier way:
+
+```ruby
+>> Skyscraper::fetch("https://github.com").find(".top-nav li").map(&:html)
+# => ["<a href=\\"https://github.com/plans\\">Signup and Pricing</a>", "<a href=\\"https://github.com/explore\\">Explore GitHub</a>", "<a href=\\"https://github.com/features\\">Features</a>", "<a href=\\"https://github.com/blog\\">Blog</a>", "<a href=\\"https://github.com/login\\">Login</a>"]
+```
+
+or even:
+
+```ruby
+>> Skyscraper::fetch("https://github.com").find(".top-nav li a").map(&:content)
+# => ["<a href=\\"https://github.com/plans\\">Signup and Pricing</a>", "<a href=\\"https://github.com/explore\\">Explore GitHub</a>", "<a href=\\"https://github.com/feature\s\">Features</a>", "<a href=\\"https://github.com/blog\\">Blog</a>", "<a href=\\"https://github.com/login\\">Login</a>"]
+```
+
+Read more about traversing in [Traversing section](wiki/traversing)
+
+## Following 
+
+You can quickly follow node element if it have **href** attribute:
+
+```ruby
+>> Skyscraper::fetch("https://github.com").first(".top-nav li a").follow.first("title").html
+# => "Plans &amp; Pricing · GitHub"
+```
+
+This example visits first menu item from github.com page, and then fetch title of it.
+
+## Downloading
+
+When node element have **src** or **href** attribute, you can easily download it:
+
+```ruby
+>> Skyscraper::fetch("http://rubyonrails.org").first(".message img").download
+# => "/tmp/skyscraper/1/rails.png"
+```
+
+You can either provide download path and new file name in arguments. Default path is also available to set in configuration.
+
+```ruby
+>> Skyscraper::fetch("http://rubyonrails.org").first(".message img").download(path: "/tmp/test/:sequence/:file_name")
+# => "/tmp/test/1/rails.png"
+>> Skyscraper::fetch("http://rubyonrails.org").first(".message img").download(path: "/tmp/test/:sequence/:file_name")
+# => "/tmp/test/2/rails.png"
+>> Skyscraper::fetch("http://rubyonrails.org").first(".message img").download(path: "/tmp/test/my_file.png")
+# => "/tmp/test/my_file.png"
+>> Skyscraper.config.download_path = "/tmp/test/my_path_from_config/:file_name"
+# => "/tmp/test/my_path_from_config/:file_name"
+>> Skyscraper::fetch("http://rubyonrails.org").first(".message img").download
+# => "/tmp/test/my_path_from_config/rails.png"
+```
+
+\#download method returns path to saved file. 
+
+## Configuration
+
+Please visit [Configuration section](wiki/configuration) to get all details of **Skyscraper** configuration.
+
+## Testing
+
+Please consider that you can fetch not only remote sites but also local files. This can be very helpful when you prefer TDD coding. 
+
+## Other topics
+
+* [Fetching from sites with large amount of pages](wiki/fetching_large_pages) - dealing with limits, delays and other stuff
+
+## Requirements
+
+**Skyscraper** requires ruby in > 1.9 version. It's also depending on Nokogiri, Open-Uri, Uri and Actionpack libraries.
+
+## What is consider to be added?
+
+* POST requests support
+* Reattempt fetching on errors
+* Redirects support
+* Testing mode - downloading only small amount of records, and showing how they would look in database
+* Ruby < 1.9 versions support
+* Redis, ActiveRecord cache and storage
+* Ruby on Rails generators
+
+Please don't hesitate to post me a comment about above or other functionality that might be added.
+
+## Contributors
+
+Here I will post list of contributors, which helps to created documentation and create bug fixes.

data/Rakefile

@@ -0,0 +1,5 @@
+require "bundler/gem_tasks"
+require 'rspec/core/rake_task'
+
+RSpec::Core::RakeTask.new(:spec)
+task default: :spec

data/lib/skyscraper.rb

@@ -0,0 +1,56 @@
+require "open-uri"
+require "uri"
+require "nokogiri"
+require "active_support/core_ext"
+
+include ActiveSupport
+
+module Skyscraper
+  extend ActiveSupport::Autoload
+  extend ActiveSupport::Concern
+
+  autoload :Base
+  autoload :Config
+  autoload :Document
+  autoload :Field
+  autoload :Node
+  autoload :Pages
+  autoload :Path
+  autoload :Results
+  
+  mattr_accessor :defaults
+  @@defaults = {
+    delay: { sleep: 0, after: 1 },
+    limit: nil,
+    encoding: "utf-8",
+    download_path: "/tmp/skyscraper/:sequence/:file_name", 
+#    reattempt_times: 1,
+    noise_errors: true,
+    skip_on_error: true
+  }
+
+  def self.config 
+    @config ||= Skyscraper::Config.new @@defaults
+  end
+
+  def self.fetch path, encoding = Skyscraper.config.encoding
+    document = Skyscraper::Document.load path, encoding
+    Node::Base.new document.css("html")
+  end
+
+  def fetch
+    self.class.send(:base).fetch 
+  end
+
+  module ClassMethods
+    def method_missing method, *args, &block
+      base.send method, *args, &block
+    end
+
+    private 
+
+    def base
+      @base ||= Skyscraper::Base.new Skyscraper.config
+    end
+  end
+end

data/lib/skyscraper/base.rb

@@ -0,0 +1,44 @@
+module Skyscraper
+  class Base
+    attr_accessor :fields, :pages_object, :config, :results
+
+    def initialize config = nil
+      @config        = (config || Skyscraper::Config.new(Skyscraper.defaults.dup))
+      @fields        = []
+      @fetch_options = {}
+      @pages_object  = Pages.new
+      @results       = Results.new self
+    end
+    
+    def pages options = {}, &block
+      @pages_object.set options, &block
+    end
+
+    def field name, selector, attribute = nil, &block
+      @fields.delete @fields.detect { |f| f.name == name }
+      @fields << Field.new(name: name, selector: selector, callback: block)
+    end
+
+    def after_each &block
+      @results.add_after_each &block
+    end
+
+    def after_all &block
+      @results.add_after_all &block
+    end
+
+    def settings options = {}
+      options.each_pair do |key, val|
+        @config.send "#{key}=", val
+      end
+    end
+
+    def fetch
+      @results.fetch
+    end
+
+    def continue
+      @results.continue
+    end
+  end
+end

data/lib/skyscraper/config.rb

@@ -0,0 +1,15 @@
+module Skyscraper
+  class Config
+    def initialize settings = {}
+      @settings = settings
+    end
+
+    def method_missing name, value = nil
+      if name.to_s.match /\=$/ 
+        @settings.merge! name.to_s.delete("=").to_sym => value
+      else
+        @settings[name]
+      end
+    end
+  end
+end

data/lib/skyscraper/document.rb

@@ -0,0 +1,11 @@
+module Skyscraper
+  class Document < Nokogiri::HTML::Document
+    attr_accessor :path
+
+    def self.load path, encoding = 'utf-8'
+      document = Skyscraper::Document.parse open(path), nil, encoding
+      document.path = Skyscraper::Path.factory(path)
+      document
+    end
+  end
+end

data/lib/skyscraper/field.rb

@@ -0,0 +1,24 @@
+module Skyscraper
+  class Field
+    attr_accessor :name, :selector, :callback, :attribute, :value
+
+    def initialize options = {}
+      @name      = options[:name]
+      @selector  = options[:selector]
+      @attribute = options[:attribute]
+      @callback  = options[:callback]
+    end
+
+    def find_in_document document
+      first_node = document.first(@selector)
+
+      if @callback
+        @value = @callback.call(first_node) 
+      elsif @attribute
+        @value = first_node.send @attribute
+      else
+        @value = first_node.text
+      end
+    end
+  end
+end

data/lib/skyscraper/node.rb

@@ -0,0 +1,8 @@
+module Skyscraper
+  module Node
+    extend ActiveSupport::Autoload
+
+    autoload :Base
+    autoload :Resource
+  end
+end

data/lib/skyscraper/node/base.rb

@@ -0,0 +1,103 @@
+module Skyscraper
+  module Node
+    class Base
+      alias :original_class :class
+
+      attr_accessor :element
+
+      def initialize element
+        @element = element
+      end
+
+      def first selector
+        self.find(selector).first
+      end
+
+      def find selector
+        @element.css(selector).map do |element|
+          Base.new(element)
+        end
+      end
+
+      def children selector = nil
+        if selector
+          children = @element.css(selector)
+        else
+          children = @element.children
+        end
+
+        children.select do |element|
+          element.parent == @element and element.is_a?(Nokogiri::XML::Element)
+        end.map do |child|
+          Base.new(child)
+        end 
+      end
+
+      def parent
+        if @element.parent.is_a? Nokogiri::XML::Element
+          Base.new @element.parent
+        end
+      end
+
+      def have_parent?
+        self.parent.present? 
+      end
+
+      def parents selector = nil
+        node = self
+        parents = []
+
+        while node.have_parent?
+          node = node.parent
+          parents << node
+        end
+
+        parents.select! do |item|
+          item.element.matches? selector
+        end if selector
+
+        parents
+      end
+
+      def siblings
+        self.parent.children.select do |node|
+          node.element != self.element
+        end
+      end
+
+      def follow 
+        if self.href
+          Skyscraper::fetch(self.uri)
+        end
+      end
+
+      def html
+        @element.children.to_html
+      end
+
+      def class
+        @element.attribute("class").to_s
+      end
+
+      def download options = {}
+        Resource.new(self).download(options)
+      end
+
+      def uri
+        @element.document.path.full_path_for(self.href)
+      end
+
+      def method_missing name
+        @element.attribute(name.to_s).to_s
+      end
+
+      def text
+        @element.content.to_s.strip
+      end
+
+      def tag
+	@element.name
+      end
+    end
+  end
+end