pismo 0.5.0 → 0.6.0

data/LICENSE

@@ -1,32 +1,23 @@
-All EXCEPT the lib/pismo/readability.rb file:
+Copyright 2009, 2010 Peter Cooper
 
-  Copyright 2009, 2010 Peter Cooper
-  
-   Licensed under the Apache License, Version 2.0 (the "License");
-   you may not use this file except in compliance with the License.
-   You may obtain a copy of the License at
-  
-       http://www.apache.org/licenses/LICENSE-2.0
-  
-   Unless required by applicable law or agreed to in writing, software
-   distributed under the License is distributed on an "AS IS" BASIS,
-   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-   See the License for the specific language governing permissions and
-   limitations under the License.
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
 
+     http://www.apache.org/licenses/LICENSE-2.0
 
-For lib/pismo/readability.rb:
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
 
-  Copyright 2009, 2010 Arc90 Inc, starrhorne, and iterationlabs
-  
-   Licensed under the Apache License, Version 2.0 (the "License");
-   you may not use this file except in compliance with the License.
-   You may obtain a copy of the License at
-  
-       http://www.apache.org/licenses/LICENSE-2.0
-  
-   Unless required by applicable law or agreed to in writing, software
-   distributed under the License is distributed on an "AS IS" BASIS,
-   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-   See the License for the specific language governing permissions and
-   limitations under the License.
+--
+
+In short, you can use Pismo for whatever you like, but please include
+a brief credit somewhere deep in your license file or similar, and, 
+if you're a nice kinda person, let me know if you're using it and/or
+share any significant changes or improvements you make.
+
+Peter Cooper
+http://twitter.com/peterc

data/NOTICE

@@ -0,0 +1,4 @@
+Pismo is Copyright (c) 2009, 2010 Peter Cooper
+Pismo is Apache 2.0 Licensed
+Peter Cooper can be found at and contacted via http://twitter.com/peterc
+The source can be found at http://github.com/peterc/pismo

data/README.markdown

@@ -1,67 +1,55 @@
 # pismo - Web page content analysis and metadata extraction
-http://github.com/peterc/pismo
 
 ## DESCRIPTION:
 
-Pismo extracts metadata and machine-usable data from mostly unstructured (or poorly structured)
-English-language HTML documents. These data include titles, feed URLs, ledes, body text, graphics, date, and keywords.
+Pismo extracts machine-usable metadata from unstructured (or poorly structured) English-language HTML documents.
+Data that Pismo can extract include titles, feed URLs, ledes, body text, image URLs, date, and keywords.
+Pismo is used heavily in production on http://coder.io/ to extract data from Web pages.
 
-For example, if you have a blog post HTML file, Pismo, in theory, should
-extract the title, the actual "content", and analyze for keywords, among other things.
+All tests pass on Ruby 1.8.7 (MRI) and Ruby 1.9.1-p378 (MRI).
 
-## EXAMPLES:
+## USAGE:
+
+A basic example of extracting basic metadata from a Web page:
 
     require 'pismo'
     
-    # Load a Web page (you can pass an IO object or a string with existing HTML data along too, if you prefer)
+    # Load a Web page (you could pass an IO object or a string with existing HTML data along, as you prefer)
     doc = Pismo::Document.new('http://www.rubyinside.com/cramp-asychronous-event-driven-ruby-web-app-framework-2928.html')
     
     doc.title     # => "Cramp: Asychronous Event-Driven Ruby Web App Framework"
     doc.author    # => "Peter Cooper"
-    doc.lede      # => "Cramp (GitHub repo)is a new, asynchronous evented Web app framework by Pratik Naik of 37signals (and the Rails core team). It's built around Ruby's EventMachine library and was designed to use event-driven I/O throughout - making it ideal for situations where you need to handle a large number of open connections (such as Comet systems or streaming APIs.)"
+    doc.lede      # => "Cramp (GitHub repo) is a new, asynchronous evented Web app framework by Pratik Naik of 37signals (and the Rails core team). It's built around Ruby's EventMachine library and was designed to use event-driven I/O throughout - making it ideal for situations where you need to handle a large number of open connections (such as Comet systems or streaming APIs.)"
     doc.keywords  # => [["cramp", 7], ["controllers", 3], ["app", 3], ["basic", 2], ..., ... ]
     
-## STATUS:
-
-Pismo is a work in progress and is being used heavily in the development of http://coder.io/. Pismo is used in production systems on both Ruby 1.8 and 1.9. I do not know how it fares on JRuby, Rubinius, or others yet.
-
-Planned/forthcoming features include the fetching of "external" data like tags from Delicious, content analysis through 3rd party services, and extraction of graphics from the main article text (for thumbnailing, say).
-
-## NEW IN 0.5.0:
+There's also a shorter "convenience" method which might be handy in IRB - it does the same as Pismo::Document.new:
 
-### Stopword access
-
-You can now access Pismo's stopword list directly:
-
-    Pismo.stopwords    # => [.., .., ..]  
+    Pismo['http://www.rubyflow.com/items/4082'].title   # => "Install Ruby as a non-root User"
     
-### Convenience access method for IRB/debugging use
-
-Now you can get playing with Pismo faster. This is primarily useful for debugging/playing in IRB as it just uses open-uri and the Pismo document is cached in the class against the URL:
-
-    url = "http://www.rubyinside.com/the-why-what-and-how-of-rubinius-1-0-s-release-3261.html"
-    Pismo[url].title   # => "The Why, What, and How of Rubinius 1.0's Release"
-    Pismo[url].author  # => "Peter Cooper"
+The current metadata methods are #title, #titles, #author, #authors, #lede, #keywords, #sentences(qty), #body, #feed, #feeds, #favicon, #description and #datetime. These are not fully documented here yet, you'll just need to try them out. The plural methods like #titles, #authors, and #feeds will return multiple matches in an array, if present. This is so you can use your own techniques to choose a "best" result in ambiguous cases.
+    
+## CAUTIONS / WARNINGS:
 
-### Arrays of all matches for titles, ledes, authors, and feeds
+There are some shortcomings or problems that I'm aware of and am going to pursue:
 
-Pismo is not perfect and you might like to instead see all of the potential titles/ledes/authors or feeds that Pismo can find. You can now do this and judge them by your metrics.
+* I do not know how Pismo fares on JRuby, Rubinius, or others yet.
+* The "Reader" content extraction algorithm is not perfect. It can sometimes return crap and can barf on certain types of characters for sentence extraction.
+* The author name extraction is quite poor.
+* The image extraction only handles images with absolute URLs.
+* The stopword list leaves a bit to be desired. It errs on the side of being too long rather than too short, though (1024 words long!)
 
-    doc.titles    # => [..., ..., ...]    
-    doc.ledes     # => [..., ..., ...]    
-    doc.authors   # => [..., ..., ...]    
-    doc.feeds     # => [..., ..., ...]
-    
-## COMMAND LINE TOOL:
+## OTHER GROOVY STUFF:
+        
+### Command Line Tool
 
 A command line tool called "pismo" is included so that you can get metadata about a page from the command line. This is
 great for testing, or perhaps calling it from a non Ruby script. The output is currently in YAML.
 
-### Usage: 
+#### Usage: 
 
     ./bin/pismo http://www.rubyinside.com/cramp-asychronous-event-driven-ruby-web-app-framework-2928.html title lede author datetime
     
-### Output:
+#### Output:
 
     --- 
     :url: http://www.rubyinside.com/cramp-asychronous-event-driven-ruby-web-app-framework-2928.html
@@ -69,6 +57,15 @@ great for testing, or perhaps calling it from a non Ruby script. The output is c
     :lede: Cramp (GitHub repo)is a new, asynchronous evented Web app framework by Pratik Naik of 37signals
     :author: Peter Cooper
     :datetime: 2010-01-07 12:00:00 +00:00
+    
+If you call pismo without any arguments (except a URL), it starts an IRB session so you can directly work in Ruby. The URL provided is loaded
+and assigned to both the constant 'P' and the variable @p.
+
+### Stopword access
+
+You can access Pismo's stopword list directly:
+
+    Pismo.stopwords    # => [.., .., ..]
 
 ## Note on Patches/Pull Requests
  
@@ -81,8 +78,8 @@ great for testing, or perhaps calling it from a non Ruby script. The output is c
 ## COPYRIGHT AND LICENSE
 
 Apache 2.0 License - See LICENSE for details.
+Copyright (c) 2009, 2010 Peter Cooper
 
-All except lib/pismo/readability.rb is Copyright (c) 2009, 2010 Peter Cooper
-lib/pismo/readability.rb is Copyright (c) 2009, 2010 Arc90 Inc, starrhorne, and iterationlabs
+In short, you can use Pismo for whatever you like commercial or not, but please include a brief credit (as in the NOTICE file - as per the Apache 2.0 License) somewhere deep in your license file or similar, and, if you're nice and have the time, let me know if you're using it and/or share any significant changes or improvements you make.
 
-The readability stuff was ganked from http://github.com/iterationlabs/ruby-readability - sorry! I have respected the license, however. I have promised to contribute back to them directly and, hopefully, use that library as a regular dependency. But.. this takes time.
+http://github.com/peterc/pismo

data/Rakefile

@@ -13,9 +13,10 @@ begin
     gem.executables = "pismo"
     gem.default_executable = "pismo"
     gem.add_development_dependency "shoulda", ">= 0"
+    gem.add_development_dependency "awesome_print"
+    gem.add_dependency "jeweler"
     gem.add_dependency "nokogiri"
-    gem.add_dependency "loofah"
-    gem.add_dependency "httparty"
+    gem.add_dependency "sanitize"
     gem.add_dependency "fast-stemmer"
     gem.add_dependency "chronic"
   end

data/VERSION

@@ -1 +1 @@
-0.5.0
+0.6.0

data/bin/pismo

@@ -18,6 +18,7 @@ require 'yaml'
 require 'rubygems'
 $:.unshift(File.dirname(__FILE__) + "/../lib")
 require 'pismo'
+require 'irb'
 
 url = ARGV.shift
 
@@ -27,10 +28,17 @@ end
 
 doc = Pismo.document(url)
 
-output = { :url => doc.url }
-
-(ARGV.empty? ? Pismo::Document::ATTRIBUTE_METHODS : ARGV).each do |cmd|
-  output[cmd.to_sym] = doc.send(cmd) rescue nil
-end
-
-puts output.to_yaml
+if ARGV.empty?
+  P = doc
+  @p = doc
+  puts "Pismo has loaded #{url} into @p and P"
+  IRB.start
+else
+  output = { :url => doc.url }
+  
+  ARGV.each do |cmd|
+    output[cmd.to_sym] = doc.send(cmd) rescue nil
+  end
+  
+  puts output.to_yaml
+end

data/lib/pismo/document.rb

@@ -23,9 +23,9 @@ module Pismo
     
     def load(handle, url = nil)
       @url = url if url
-      @url = handle if handle =~ /\Ahttp/
+      @url = handle if handle =~ /\Ahttp/i
       
-      @html = if handle =~ /\Ahttp/
+      @html = if handle =~ /\Ahttp/i
                 open(handle).read
               elsif handle.is_a?(StringIO) || handle.is_a?(IO) || handle.is_a?(Tempfile)
                 handle.read

data/lib/pismo/internal_attributes.rb

@@ -24,6 +24,7 @@ module Pismo
                             '.title h1',                          
                             '.post h2',
                             'h2.title',
+                            '.entry h2 a',
                             '.entry h2',                                                      # Common style
                             '.boite_titre a',
                             ['meta[@name="title"]', lambda { |el| el.attr('content') }],
@@ -66,8 +67,6 @@ module Pismo
       title = @doc.match('title')
       return unless title
       title
-      # Strip off any leading or trailing site names - a scrappy way to try it out..
-      #title = title.split(/\s+(\-|\||\:)\s+/).sort_by { |i| i.length }.last.to_s.strip
     end
     
     # Return an estimate of when the page/content was created
@@ -209,16 +208,16 @@ module Pismo
                   '.entry-content',
                   '.body p',
                   '.document_description_short p',    # Scribd
-                  '.single-post p',
-                  'p'
+                  '.single-post p'
                   ], all)
-       
+
+      # TODO: Improve sentence extraction - this is dire even if it "works for now"
       if lede && String === lede
-        return lede[/^(.*?\.\s){2}/m] || lede
+        return lede[/^(.*?[\.\!\?]\s){2}/m] || lede
       elsif lede && Array === lede
-        return lede.map { |l| l.to_s[/^(.*?\.\s){2}/m] || l }.uniq
+        return lede.map { |l| l.to_s[/^(.*?[\.\!\?]\s){2}/m] || l }.uniq
       else
-        return body ? body[/^(.*?\.\s){2}/m] : nil
+        return reader_doc && !reader_doc.sentences(2).empty? ? reader_doc.sentences(2).join(' ') : nil
       end
     end
     
@@ -226,6 +225,16 @@ module Pismo
       lede(true)
     end
     
+    # Returns a string containing the first [limit] sentences as determined by the Reader algorithm
+    def sentences(limit = 3)
+      reader_doc && !reader_doc.sentences.empty? ? reader_doc.sentences(limit).join(' ') : nil
+    end
+
+    # Returns any images with absolute URLs in the document
+    def images(limit = 3)
+      reader_doc && !reader_doc.images.empty? ? reader_doc.images(limit) : nil
+    end
+    
     # Returns the "keywords" in the document (not the meta keywords - they're next to useless now)
     def keywords(options = {})
       options = { :stem_at => 20, :word_length_limit => 15, :limit => 20 }.merge(options)
@@ -253,15 +262,13 @@ module Pismo
       return w
     end
     
-    # Returns body text as determined by Arc90's Readability algorithm
+    def reader_doc
+      @reader_doc ||= Reader::Document.new(@doc.to_s)
+    end
+    
+    # Returns body text as determined by Reader algorithm
     def body
-      @body ||= Readability::Document.new(@doc.to_s).content.strip
-      
-      # HACK: Remove annoying DIV that readability leaves around
-      @body.sub!(/\A\<div\>/, '')
-      @body.sub!(/\<\/div\>\Z/, '')
-      
-      return @body
+      @body ||= reader_doc.content.strip      
     end
     
     # Returns URL to the site's favicon