jaimeiniesta-metainspector 1.0.3 → 1.1.0

data/CHANGELOG.rdoc

@@ -0,0 +1,12 @@
+= 1.1.0
+=== 14th May, 2009
+* Rewritten to use instance methods instead of class methods.
+* Easier interface, provides #new(address), #scrape!, #scraped?, #title, #description, #keywords, and #links instance methods
+* Added #full_doc method to access the temporary file that containts the raw html fetched
+* Added #scraped_doc method to get the whole Hpricot scraped doc
+* Added tests
+* Added samples, including a basic scraping and a little spider
+
+= 1.0.3
+=== 27th June, 2008
+* Initial published version.

data/MIT-LICENSE

@@ -0,0 +1,20 @@
+Copyright (c) 2009 Jaime Iniesta
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.rdoc

@@ -0,0 +1,106 @@
+= MetaInspector
+
+MetaInspector is a gem for web scraping purposes. You give it an URL, and it returns you a hash with metadata from it.
+
+= Installation
+
+Run the following if you haven't already:
+
+  gem sources -a http://gems.github.com
+
+Then install the gem:
+
+  sudo gem install jaimeiniesta-metainspector
+
+= Usage
+
+Initialize a MetaInspector instance with an URL like this:
+
+  page = MetaInspector.new('http://pagerankalert.com')
+
+Then you can tell it to fetch and scrape the URL:
+
+  page.scrape!
+
+Once scraped, you can see the returned data like this:
+
+  page.address       # URL of the page
+  page.title         # title of the page, as string
+  page.description   # meta description, as string
+  page.keywords      # meta keywords, as string
+  page.links         # array of strings, with every link found on the page
+  
+You can see if the scraping process went ok checking what page.scrape! returns (true or false), or checking the page.scraped? method, which returns false if no successfull scraping has been finished since the last address change.
+You can also change the address of the page to be scraped using the address= setter, like this:
+
+  page.address="http://jaimeiniesta.com"
+
+Doing so resets the state of the MetaInspector instance to the initial state (not scraped yet, cleared stored meta data). You can re-scrape it again by calling the page.scrape! method.
+
+The full fetched document and the scraped doc are accessible from:
+
+  page.full_doc    # it points to the temp file where the fetched doc is stored
+  page.scraped_doc # Hpricot doc that you can use it to get any element from the page
+
+= Examples
+
+You can find some sample scripts on the samples folder, including a basic scraping and a spider that will follow external links using a queue. What follows is an example of use from irb:
+
+  $ irb
+  >> require 'metainspector'
+  => true
+
+  >> page = MetaInspector.new('http://pagerankalert.com')
+  => #<MetaInspector:0x5fc594 @full_doc=nil, @scraped=false, @description=nil, @links=nil, @address="http://pagerankalert.com", @keywords=nil, @scraped_doc=nil, @title=nil>
+
+  >> page.scrape!
+  => true
+
+  >> page.title
+  => "PageRankAlert.com :: Track your pagerank changes"
+
+  >> page.description
+  => "Track your PageRank(TM) changes and receive alert by email"
+
+  >> page.keywords
+  => "pagerank, seo, optimization, google"
+
+  >> page.links.size
+  => 31
+
+  >> page.links[30]
+  => "http://www.nuvio.cz/"
+
+  >> page.full_doc
+  => #<File:/var/folders/X8/X8TBsDiWGYuMKzrB3bhWTU+++TI/-Tmp-/open-uri.6656.0>
+
+  >> page.scraped_doc.class
+  => Hpricot::Doc
+
+  >> page.scraped?
+  => true
+
+  >> page.address="http://jaimeiniesta.com"
+  => "http://jaimeiniesta.com"
+
+  >> page.scraped?
+  => false
+
+  >> page.scrape!
+  => true
+
+  >> page.scraped?
+  => true
+
+  >> page.title
+  => "ruby on rails freelance developer &#8212; Jaime Iniesta"
+    
+= To Do
+
+* Mocks
+* Check content type, process only HTML pages (i.e., dont try to scrape http://ftp.ruby-lang.org/pub/ruby/ruby-1.9.1-p129.tar.bz2)
+* Return array of images in page
+* Return contents of meta robots tag
+* Consider using nokogiri instead of hpricot
+
+Copyright (c) 2009 Jaime Iniesta, released under the MIT license

data/lib/metainspector.rb

@@ -1,47 +1,71 @@
-class MetaInspector
-  require 'open-uri'
-  require 'rubygems'
-  require 'hpricot'
+require 'open-uri'
+require 'rubygems'
+require 'hpricot'
 
-  VERSION = '1.0.2'
+# MetaInspector provides an easy way to scrape web pages and get its elements
+class MetaInspector
+  VERSION = '1.1.0'
 
   Hpricot.buffer_size = 300000
-
-  def self.scrape(url)
-    doc = Hpricot(open(url))
+  
+  attr_reader :address, :title, :description, :keywords, :links, :full_doc, :scraped_doc
+  
+  # Initializes a new instance of MetaInspector, setting the URL address to the one given
+  # TODO: validate address as http URL, dont initialize it if wrong format 
+  def initialize(address)
+    @address = address
+    @scraped = false
+    
+    @title = @description = @keywords = @links = @full_doc = @scraped_doc = nil
+  end
+  
+  # Setter for address. Initializes the whole state as the address is being changed.
+  def address=(address)
+    initialize(address)
+  end
+  
+  # Visit web page, get its contents, and parse it
+  def scrape!
+    @full_doc = open(@address)
+    @scraped_doc = Hpricot(@full_doc)
     
     # Searching title...
-    if doc.at('title')
-      title = doc.at('title').inner_html
+    if @scraped_doc.at('title')
+      @title = @scraped_doc.at('title').inner_html.strip
     else
-      title = ""
+      @title = ""
     end
     
     # Searching meta description...
-    if doc.at("meta[@name='description']")
-      description = doc.at("meta[@name='description']")['content']
+    if @scraped_doc.at("meta[@name='description']")
+      @description = @scraped_doc.at("meta[@name='description']")['content'].strip
     else
-      description = ""
+      @description = ""
     end
     
     # Searching meta keywords...
-    if doc.at("meta[@name='keywords']")
-      keywords = doc.at("meta[@name='keywords']")['content']
+    if @scraped_doc.at("meta[@name='keywords']")
+      @keywords = @scraped_doc.at("meta[@name='keywords']")['content'].strip
     else
-      keywords = ""
+      @keywords = ""
     end
         
     # Searching links...
-    links = []
-    doc.search("//a").each do |link|
-      links << link.attributes["href"] if (!link.attributes["href"].nil?)
+    @links = []
+    @scraped_doc.search("//a").each do |link|
+      @links << link.attributes["href"].strip if (!link.attributes["href"].nil?)
     end
-  
-    # Returning all data...
-    {'ok' => true, 'title' => title, 'description' => description, 'keywords' => keywords, 'links' => links}  
+    
+    # Mark scraping as success
+    @scraped = true
 
     rescue SocketError
       puts 'MetaInspector exception: The url provided does not exist or is temporarily unavailable (socket error)'
-      {'ok' => false, 'title' => nil, 'description' => nil, 'keywords' => nil, 'links' => nil}  
+      @scraped = false
+  end
+  
+  # Syntactic sugar
+  def scraped?
+    @scraped
   end
-end
+end

data/metainspector.gemspec

@@ -1,14 +1,14 @@
 Gem::Specification.new do |s|
   s.name = "metainspector"
-  s.version = "1.0.3"
-  s.date = "2008-06-27"
+  s.version = "1.1.0"
+  s.date = "2009-05-14"
   s.summary = "Ruby gem for web scraping"
   s.email = "jaimeiniesta@gmail.com"
-  s.homepage = "http://code.jaimeiniesta.com/metainspector"
+  s.homepage = "http://github.com/jaimeiniesta/metainspector/tree/master"
   s.description = "MetaInspector is a ruby gem for web scraping purposes, that returns a hash with metadata from a given URL"
   s.has_rdoc = false
   s.authors = ["Jaime Iniesta"]
-  s.files = ["README", "metainspector.gemspec", "lib/metainspector.rb", "test/test_metainspector.rb"]
+  s.files = ["README.rdoc", "CHANGELOG.rdoc", "MIT-LICENSE", "metainspector.gemspec", "lib/metainspector.rb", "test/test_metainspector.rb", "samples/basic_scraping.rb", "samples/spider.rb"]
   s.test_files = []
   s.rdoc_options = []
   s.extra_rdoc_files = []

data/samples/basic_scraping.rb

@@ -0,0 +1,19 @@
+# Some basic MetaInspector samples
+
+require '../lib/metainspector.rb'
+
+puts "Enter a valid http address to scrape it"
+address = gets
+page = MetaInspector.new(address)
+puts "Scraping #{address}"
+puts "...please wait..."
+page.scrape!
+
+puts "Scraping #{page.address} returned these results:"
+puts "TITLE: #{page.title}"
+puts "DESCRIPTION: #{page.description}"
+puts "KEYWORDS: #{page.keywords}"
+puts "#{page.links.size} links found..."
+page.links.each do |link|
+  puts " ==> #{link}"
+end

data/samples/spider.rb

@@ -0,0 +1,24 @@
+# A basic spider that will follow links on an infinite loop
+require '../lib/metainspector.rb'
+
+q = Queue.new
+visited_links=[]
+
+puts "Enter a valid http address to spider it following external links"
+address = gets.strip
+page = MetaInspector.new(address)
+q.push(address)
+
+while q.size > 0
+  visited_links << address = q.pop
+  page.address=address
+  puts "Spidering #{page.address}"
+  page.scrape!
+  puts "TITLE: #{page.title}"
+  page.links.each do |link|
+    if link[0..6].downcase == 'http://' && !visited_links.include?(link)
+      q.push(link)
+    end
+  end
+  puts "#{visited_links.size} pages visited, #{q.size} pages on queue\n\n"
+end

data/test/test_metainspector.rb

@@ -1 +1,60 @@
+require 'test/unit'
+require '../lib/metainspector.rb'
 
+class TestMetaInspector < Test::Unit::TestCase
+  # TODO: mock tests
+  
+  # Test we can initialize a new instance, setting its address, and initial state
+  # is not scraped and every meta data value set to nil
+  # TODO: validate URL format, only http and https allowed
+  def test_initialize
+    m = MetaInspector.new('http://pagerankalert.com')
+    assert_equal m.address, 'http://pagerankalert.com'
+    assert_equal m.scraped?, false
+    assert_nil m.title
+    assert_nil m.description
+    assert_nil m.keywords
+    assert_nil m.links
+    assert_nil m.full_doc
+    assert_nil m.scraped_doc
+  end
+  
+  # Test scraping an URL, marking it as scraped and setting meta data values
+  # TODO: check timeouts
+  def test_scrape!
+    m = MetaInspector.new('http://pagerankalert.com')
+    assert m.scrape!
+    assert m.scraped?
+    assert_equal m.title, 'PageRankAlert.com :: Track your pagerank changes'
+    assert_equal m.description, 'Track your PageRank(TM) changes and receive alert by email'
+    assert_equal m.keywords, 'pagerank, seo, optimization, google'
+    assert_equal m.links.size, 31
+    assert_equal m.links[30], 'http://www.nuvio.cz/'
+    assert_equal m.full_doc.class, Tempfile
+    assert_equal m.scraped_doc.class, Hpricot::Doc
+  end
+  
+  # Test changing the address resets the state of the instance
+  def test_address_setter
+    m = MetaInspector.new('http://pagerankalert.com')
+    assert_equal m.address, 'http://pagerankalert.com'
+    m.scrape!
+    assert m.scraped?
+    assert_not_nil m.title
+    assert_not_nil m.description
+    assert_not_nil m.keywords
+    assert_not_nil m.links
+    assert_not_nil m.full_doc
+    assert_not_nil m.scraped_doc
+    
+    m.address = 'http://jaimeiniesta.com'
+    assert_equal m.address, 'http://jaimeiniesta.com'
+    assert !m.scraped?
+    assert_nil m.title
+    assert_nil m.description
+    assert_nil m.keywords
+    assert_nil m.links
+    assert_nil m.full_doc
+    assert_nil m.scraped_doc    
+  end
+end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification 
 name: jaimeiniesta-metainspector
 version: !ruby/object:Gem::Version 
-  version: 1.0.3
+  version: 1.1.0
 platform: ruby
 authors: 
 - Jaime Iniesta
@@ -9,11 +9,12 @@ autorequire:
 bindir: bin
 cert_chain: []
 
-date: 2008-06-27 00:00:00 -07:00
+date: 2009-05-14 00:00:00 -07:00
 default_executable: 
 dependencies: 
 - !ruby/object:Gem::Dependency 
   name: hpricot
+  type: :runtime
   version_requirement: 
   version_requirements: !ruby/object:Gem::Requirement 
     requirements: 
@@ -30,12 +31,16 @@ extensions: []
 extra_rdoc_files: []
 
 files: 
-- README
+- README.rdoc
+- CHANGELOG.rdoc
+- MIT-LICENSE
 - metainspector.gemspec
 - lib/metainspector.rb
 - test/test_metainspector.rb
+- samples/basic_scraping.rb
+- samples/spider.rb
 has_rdoc: false
-homepage: http://code.jaimeiniesta.com/metainspector
+homepage: http://github.com/jaimeiniesta/metainspector/tree/master
 post_install_message: 
 rdoc_options: []
 

data/README

@@ -1,8 +0,0 @@
---------------
-MetaInspector
---------------
-DESCRIPTION: Ruby gem for web scraping purposes. It scrapes a given URL, and returns you a hash with data from it like for example the title, meta description, meta keywords, an array with all the links, all the images in it, etc.
-AUTHOR: Jaime Iniesta (jaimeiniesta@gmail.com, http://www.jaimeiniesta.com)
-LICENSE: GPLv3
-
-(more to come soon...)