robostripper 0.0.1

data/.gitignore

@@ -0,0 +1,18 @@
+docs
+*.gem
+*.rbc
+.bundle
+.config
+.yardoc
+Gemfile.lock
+InstalledFiles
+_yardoc
+coverage
+doc/
+lib/bundler/man
+pkg
+rdoc
+spec/reports
+test/tmp
+test/version_tmp
+tmp

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in robostripper.gemspec
+gemspec

data/LICENSE.txt

@@ -0,0 +1,22 @@
+Copyright (c) 2013 Brian Muller
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.rdoc

@@ -0,0 +1,111 @@
+= Robostripper
+
+Robostripper takes the magic of Nokogiri and makes it easy to treat a website like a database.
+
+== Installation
+
+Add this line to your application's Gemfile:
+
+   gem 'robostripper'
+
+And then execute:
+
+   $ bundle
+
+Or install it yourself as:
+
+   $ gem install robostripper
+
+== Usage
+
+Let's say you want to pull down information about a business from the Yellow Pages website.
+
+=== Single Item
+
+For the first example, let's pretend you want a single business' information and you already have the URL.
+
+First, create a class and then describe one of the items on the page that you want to pull out (in this case, the business details):
+
+  class YellowPage < Robostripper::Resource
+  end
+
+  YellowPage.add_item :details do
+    phone "p.phone strong"
+    street "span.street-address"
+    city "span.locality"
+    state "span.region"
+    zip "span.postal-code"
+    address { [ street, city, state, zip ].join(" ") }
+    payments { scan("dd.def-payment").split(", ").map(&:strip) }
+  end
+
+In the add_item block, I've described the attributes that I want to pull out and the query necessary to pull that information out.  For each attribute, you can give either CSS selector code or XPath or a mix of the two (as an array).  Alternatively, you can provide a block that will be executed each time you ask for the attribute.  For instance, in the example above, the *address* attribute references all of the other address attributes ad joins them together.
+
+The arguments to attributes are used in a call to the method _scan_, which is used explicitly in the *payments* attribute.  See the section below for more details about document scanning.  The payments on the page I'll be pulling from are comma separated - but I want to produce an array - so I split by commas and strip the results.
+
+It is then trivial to pull information from the page:
+
+  url = "http://www.yellowpages.com/silver-spring-md/mip/quarry-house-tavern-3342829"
+  yp = YellowPage.new(url)
+  yp.details.payments
+    => ["MASTER CARD", "DISCOVER", "AMEX", "VISA", "CASH ONLY"]
+  yp.details.phone
+    => "(301) 587-8350"
+  yp.details.address
+    => "8401 Georgia Ave, Silver Spring MD 20910"
+
+
+=== Pages with Lists
+
+For the second example, let's say you want to be able to search the Yellow Pages website and then get the details for each matching business.
+
+First, let's define our listing class.
+
+  class YellowPagesList < Robostripper::Resource
+    def initialize(city, name)
+      # First, replace spaces with dashes
+      city = city.gsub(',', '').gsub(' ', '-')
+      name = name.gsub(' ', '-')
+      super("http://www.yellowpages.com/#{city}/#{name}")
+    end
+  end
+
+Because the Yellow Pages site performs searches based on URL structure, we've simply crafted the structure based on an input city and business name.  Now, we can explain how to get each result out of the page:
+
+  YellowPagesList.add_list :businesses, "div.result" do
+    name "h3.business-name a"
+    url "h3.business-name a", :attribute => 'href'
+    details { YellowPage.new(url).details }
+  end
+
+For this example, each business' information can be foudn within a _div.result_.  Each of those will have three attributes.  The first two are pulled out of the div in the same way as the single result described above.  The details attribute, however, is special in that it creates a new *YellowPage* (which we defined above).  When you ask for a business' details, then Robostripper will go get the URL and parse out the details.
+
+Putting it all together:
+
+  ypl = YellowPagesList.new("Washington, DC", "Quarry House Tavern")
+  # the first match is all we care about
+  business = ypl.businesses.first
+  # now, get the details like we did above
+  business.details.phone
+   => "(301) 587-8350"
+  business.details.zip
+   => "20910"
+
+We can now search the Yellow Pages website and get the details for any business.  Like a Robot.
+
+== Scanning Documents
+
+If a block isn't given to an attribute definition in _add_list_ or _add_item_ then the arguments are passed along to the method _scan_.  The method _scan_ takes two arguments:
+
+1. A CSS path selector, an XPath selector, or an array containing a any mix of the two.  If the argument is an array, then each selection is applied in order.  For instance:
+
+     [ "p.something", "a[first()]" ]
+
+   will first find paragraphs with a class of _something_ using a CSS selctor, and then will pull out the first _a_ element using the XPath selector.
+
+2. The second argument is an options hash.  By default, _scan_ will just return the _text_ content of all matching nodes from #1.  The following options can modify that:
+   * *:all* => true: This signifies that _scan_ should just return all matching elements, without sucking out the text.
+   * *:nontext* => ' ': If this option is specified, it will replace all non-text elements with a spacing string.  This is necessary, for instance, if a website uses 
+     <br /> elements to separate strings and you want to preserve the separation.  For instance, if you want "123 Fake St<br />A City, CA" to have a space between the
+     street and the city.
+   * *:attribute* => 'href': Use this to pull out an element's specific attribute rather than it's inner text.

data/Rakefile

@@ -0,0 +1,9 @@
+require "bundler/gem_tasks"
+require 'rdoc/task'
+
+RDoc::Task.new("doc") { |rdoc|
+  rdoc.title = "Robostripper - Strip HTML websites like a Robot"
+  rdoc.rdoc_dir = 'docs'
+  rdoc.rdoc_files.include('README.rdoc')
+  rdoc.rdoc_files.include('lib/**/*.rb')
+}

data/lib/robostripper.rb

@@ -0,0 +1,3 @@
+require "robostripper/version"
+require "robostripper/resource"
+require "robostripper/http"

data/lib/robostripper/http.rb

@@ -0,0 +1,21 @@
+require 'httparty'
+require 'nokogiri'
+
+module Robostripper
+  module HTTP
+    HEADERS = {
+      'Accept' => 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
+      'Accept-Charset' => 'ISO-8859-1,utf-8;q=0.7,*;q=0.3',
+      'Accept-Language' => 'en-US,en;q=0.8',
+      'User-Agent' => 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.101 Safari/537.11'
+    }
+
+    def self.get(url)
+      response = HTTParty.get(url, :headers => HEADERS)
+      if response.code != 200
+        raise "Got response code of #{response.code} for #{url}"
+      end
+      Nokogiri::HTML response.body
+    end
+  end
+end

data/lib/robostripper/resource.rb

@@ -0,0 +1,69 @@
+require 'uri'
+require 'set'
+
+module Robostripper
+  class Resource
+    def initialize(source)
+      @html = (source.is_a? String) ? HTTP.get(source) : source
+      @cached = {}
+      @attrs = Set.new
+    end
+    
+    def encode(text)
+      URI.encode(text)
+    end
+    
+    def self.add_list(name, path, &block)
+      define_method(name) {
+        @cached[name] ||= @html.css(path).map { |result| 
+          Resource.new(result).tap { |r| r.instance_eval &block }
+        }
+      }
+    end
+
+    def self.add_item(name, &block)
+      define_method(name) {
+        @cached[name] ||= Resource.new(@html).tap { |r| r.instance_eval &block }
+      }
+    end
+
+    def method_missing(method, *args, &block)
+      return super if method == :to_ary or (args.length == 0 and block.nil?)
+      @attrs << method
+
+      if block.nil? and args.length == 0
+        block = lambda { nil }
+      elsif block.nil?
+        options = (args.length == 1) ? {} : args[1]
+        block = lambda { scan(args.first, options) }
+      end
+
+      define_singleton_method(method, &block)
+    end
+
+    def scan(paths, options = {})
+      result = @html
+      paths = [ paths ] unless paths.is_a? Array
+      paths.each { |path|
+        result = result.search(path)
+      }      
+
+      if result.nil? or result.length == 0
+        nil
+      elsif options.fetch(:all, false)
+        result
+      elsif options.has_key? :nontext
+        result.children.select(&:text?).map(&:text).join(options[:nontext])
+      elsif options.has_key? :attribute
+        result.first[options[:attribute]]
+      else
+        result.text
+      end
+    end
+
+    def to_s
+      attrs = @attrs.to_a.inject({}) { |h, attr| h[attr] = send(attr); h }
+      "<#{self.class.name} #{attrs}>"
+    end
+  end
+end

data/lib/robostripper/version.rb

@@ -0,0 +1,3 @@
+module Robostripper
+  VERSION = "0.0.1"
+end

data/robostripper.gemspec

@@ -0,0 +1,22 @@
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'robostripper/version'
+
+Gem::Specification.new do |gem|
+  gem.name          = "robostripper"
+  gem.version       = Robostripper::VERSION
+  gem.authors       = ["Brian Muller"]
+  gem.email         = ["bamuller@gmail.com"]
+  gem.description   = %q{Strip HTML websites as though they provided a REST interface.  Like a Robot.}
+  gem.summary       = %q{Strip HTML websites as though they provided a REST interface.  Like a Robot.}
+  gem.homepage      = "https://github.com/bmuller/robostripper"
+
+  gem.files         = `git ls-files`.split($/)
+  gem.executables   = gem.files.grep(%r{^bin/}).map{ |f| File.basename(f) }
+  gem.test_files    = gem.files.grep(%r{^(test|spec|features)/})
+  gem.require_paths = ["lib"]
+  gem.add_dependency("httparty", ">= 0.9.0")
+  gem.add_dependency("nokogiri", ">= 1.5.6")
+  gem.add_development_dependency("rake")
+  gem.add_development_dependency("rdoc")
+end

metadata

@@ -0,0 +1,113 @@
+--- !ruby/object:Gem::Specification 
+name: robostripper
+version: !ruby/object:Gem::Version 
+  prerelease: 
+  version: 0.0.1
+platform: ruby
+authors: 
+- Brian Muller
+autorequire: 
+bindir: bin
+cert_chain: []
+
+date: 2012-12-29 00:00:00 Z
+dependencies: 
+- !ruby/object:Gem::Dependency 
+  name: httparty
+  requirement: &id001 !ruby/object:Gem::Requirement 
+    none: false
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        version: 0.9.0
+  type: :runtime
+  prerelease: false
+  version_requirements: *id001
+- !ruby/object:Gem::Dependency 
+  name: nokogiri
+  requirement: &id002 !ruby/object:Gem::Requirement 
+    none: false
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        version: 1.5.6
+  type: :runtime
+  prerelease: false
+  version_requirements: *id002
+- !ruby/object:Gem::Dependency 
+  name: rake
+  requirement: &id003 !ruby/object:Gem::Requirement 
+    none: false
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        version: "0"
+  type: :development
+  prerelease: false
+  version_requirements: *id003
+- !ruby/object:Gem::Dependency 
+  name: rdoc
+  requirement: &id004 !ruby/object:Gem::Requirement 
+    none: false
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        version: "0"
+  type: :development
+  prerelease: false
+  version_requirements: *id004
+description: Strip HTML websites as though they provided a REST interface.  Like a Robot.
+email: 
+- bamuller@gmail.com
+executables: []
+
+extensions: []
+
+extra_rdoc_files: []
+
+files: 
+- .gitignore
+- Gemfile
+- LICENSE.txt
+- README.rdoc
+- Rakefile
+- lib/robostripper.rb
+- lib/robostripper/http.rb
+- lib/robostripper/resource.rb
+- lib/robostripper/version.rb
+- robostripper.gemspec
+homepage: https://github.com/bmuller/robostripper
+licenses: []
+
+post_install_message: 
+rdoc_options: []
+
+require_paths: 
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement 
+  none: false
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      hash: 4100496691358447740
+      segments: 
+      - 0
+      version: "0"
+required_rubygems_version: !ruby/object:Gem::Requirement 
+  none: false
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      hash: 4100496691358447740
+      segments: 
+      - 0
+      version: "0"
+requirements: []
+
+rubyforge_project: 
+rubygems_version: 1.8.24
+signing_key: 
+specification_version: 3
+summary: Strip HTML websites as though they provided a REST interface.  Like a Robot.
+test_files: []
+