pismo 0.6.1 → 0.6.2

data/README.markdown

@@ -26,11 +26,27 @@ There's also a shorter "convenience" method which might be handy in IRB - it doe
 
     Pismo['http://www.rubyflow.com/items/4082'].title   # => "Install Ruby as a non-root User"
     
-The current metadata methods are #title, #titles, #author, #authors, #lede, #keywords, #sentences(qty), #body, #html_body, #feed, #feeds, #favicon, #description and #datetime. These are not fully documented here yet, you'll just need to try them out. The plural methods like #titles, #authors, and #feeds will return multiple matches in an array, if present. This is so you can use your own techniques to choose a "best" result in ambiguous cases.
+The current metadata methods are:
+* title
+* titles
+* author
+* authors
+* lede
+* keywords
+* sentences(qty)
+* body
+* html_body
+* feed
+* feeds
+* favicon
+* description
+* datetime
+
+These methods are not fully documented here yet - you'll just need to try them out. The plural methods like #titles, #authors, and #feeds will return multiple matches in an array, if present. This is so you can use your own techniques to choose a "best" result in ambiguous cases.
 
 The html_body and body methods will be of particular interest. They return the "body" of the page as determined by Pismo's "Reader" (like Arc90's Readability or Safari Reader) algorithm. #body returns it as plain-text, #html_body maintains some basic HTML styling.
     
-## CAUTIONS / WARNINGS:
+## CAVEATS AND SHORTCOMINGS:
 
 There are some shortcomings or problems that I'm aware of and am going to pursue:
 

data/VERSION

@@ -1 +1 @@
-0.6.1
+0.6.2

data/bin/pismo

@@ -37,7 +37,7 @@ else
   output = { :url => doc.url }
   
   ARGV.each do |cmd|
-    output[cmd.to_sym] = doc.send(cmd) rescue nil
+    output[cmd.to_sym] = doc.send(cmd)
   end
   
   puts output.to_yaml

data/lib/pismo/internal_attributes.rb

@@ -216,7 +216,7 @@ module Pismo
       elsif lede && Array === lede
         return lede.map { |l| l.to_s[/^(.*?[\.\!\?]\s){2}/m].strip || l }.uniq
       else
-        return reader_doc && !reader_doc.sentences(2).empty? ? reader_doc.sentences(2).join(' ') : nil
+        return reader_doc && !reader_doc.sentences(3).empty? ? reader_doc.sentences(3).join(' ') : nil
       end
     end
     
@@ -242,7 +242,7 @@ module Pismo
       
       # Convert doc to lowercase, scrub out most HTML tags, then keep track of words
       cached_title = title
-      content_to_use = body.to_s.downcase + description.to_s.downcase
+      content_to_use = body.to_s.downcase + " " + description.to_s.downcase
 
       # old regex for safe keeping -- \b[a-z][a-z\+\.\'\+\#\-]*\b
       content_to_use.downcase.gsub(/\<[^\>]{1,100}\>/, '').gsub(/\.+\s+/, ' ').gsub(/\&\w+\;/, '').scan(/(\b|\s|\A)([a-z0-9][a-z0-9\+\.\'\+\#\-\/\\]*)(\b|\s|\Z)/i).map{ |ta1| ta1[1] }.each do |word|

data/lib/pismo/reader.rb

@@ -1,5 +1,6 @@
 require 'nokogiri'
 require 'sanitize'
+begin; require 'ap'; rescue LoadError; end
 
 module Pismo
   module Reader
@@ -20,7 +21,7 @@ module Pismo
       GOOD_WORDS = %w{content post blogpost main story body entry text desc asset hentry single entrytext postcontent bodycontent}.uniq
   
       # Words that indicate crap in general
-      BAD_WORDS = %w{reply metadata options commenting comments comment about footer header outer credit sidebar widget subscribe clearfix date social bookmarks links share video watch excerpt related supplement accessibility offscreen meta title signup blq secondary feedback featured clearfix small job jobs listing listings navigation nav byline addcomment postcomment trackback neighbor snap nopreview ads commentform fbfans login similar thumb link blogroll grid twitter wrapper container nav sitesub printfooter editsection visualclear catlinks hidden toc contentsub caption disqus rss shoutbox sponsor}.uniq
+      BAD_WORDS = %w{reply metadata options commenting comments comment about footer header outer credit sidebar widget subscribe clearfix date social bookmarks links share video watch excerpt related supplement accessibility offscreen meta title signup blq secondary feedback featured clearfix small job jobs listing listings navigation nav byline addcomment postcomment trackback neighbor ads commentform fbfans login similar thumb link blogroll grid twitter wrapper container nav sitesub printfooter editsection visualclear catlinks hidden toc contentsub caption disqus rss shoutbox sponsor}.uniq
       
       # Words that kill a branch dead
       FATAL_WORDS = %w{comments comment bookmarks social links ads related similar footer digg totop metadata sitesub nav sidebar commenting options addcomment leaderboard offscreen job prevlink prevnext navigation reply-link hide hidden sidebox archives vcard}
@@ -69,6 +70,8 @@ module Pismo
               
         @doc = Nokogiri::HTML(@raw_content, nil, 'utf-8')
         
+        #ap @raw_content
+        #exit
         build_analysis_tree
       end
     
@@ -221,7 +224,7 @@ module Pismo
       # Return the content from best match number of index (default 0) and, optionally, clean it to plain-text
       def content(clean = false, index = 0)
         return @content[[clean, index]] if @content[[clean, index]]
-        return '' unless @content_candidates && !@content_candidates.empty?
+        return '' if !@content_candidates || @content_candidates.empty?
         
         content_branch = @doc.at(@content_candidates[index].first)
         orphans_to_remove = []
@@ -361,7 +364,8 @@ module Pismo
         fodder = content(true) if fodder.to_s.length < 50
         fodder.gsub!(/\b\w\W\s/, '')
         
-        sentences = fodder.scan(/([\&\w\s\-\'\,\+\.\/\\\:\#\(\)\=\"\?\!]+?[\.\?\!])(\s|\Z)/im).map { |s| s.first }
+        #sentences = fodder.scan(/([\&\w\s\-\'\,\+\.\/\\\:\#\(\)\=\"\?\!]+?[\.\?\!])(\s|\Z)/im).map { |s| s.first }
+        sentences = fodder.scan(/(.+?[\.\?\!])(\s|\Z)/im).map { |s| s.first.strip }
         
         sentences.compact!
         sentences.map! { |s| s.strip }

data/pismo.gemspec

@@ -5,7 +5,7 @@
 
 Gem::Specification.new do |s|
   s.name = %q{pismo}
-  s.version = "0.6.1"
+  s.version = "0.6.2"
 
   s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
   s.authors = ["Peter Cooper"]

data/test/corpus/metadata_expected.yaml

@@ -2,7 +2,7 @@
 :rww: 
   :title: "Cartoon: Apple Tablet: Now With Barometer and Bird Call Generator"
   :feed: http://www.readwriteweb.com/rss.xml
-  :lede: I'm just aching to know if the new Apple tablet (insert caveats, weasel words and qualifiers here) is a potential Cintiq competitor. I don't think it will be, but you never know.  
+  :lede: I'm just aching to know if the new Apple tablet (insert caveats, weasel words and qualifiers here) is a potential Cintiq competitor. I don't think it will be, but you never know. It may also have a built in barometer and bird call generator.
   :feeds:
     - http://www.readwriteweb.com/rss.xml
     - http://www.readwriteweb.com/archives/2010/01/cartoon_apple_tablet_now_with_barometer_and_bird_c.xml  
@@ -42,7 +42,7 @@
 :spolsky:
   :title: The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!) - Joel on Software
   :description: Haven't mastered the basics of Unicode and character sets? Please don't write another line of code until you've read this article.
-  :lede: I've been dismayed to discover just how many software developers aren't really completely up to speed on the mysterious world of character sets, encodings, Unicode, all that stuff. A couple of years ago, a beta tester for FogBUGZ was wondering whether it could handle incoming email in Japanese.
+  :lede: I've been dismayed to discover just how many software developers aren't really completely up to speed on the mysterious world of character sets, encodings, Unicode, all that stuff. A couple of years ago, a beta tester for FogBUGZ was wondering whether it could handle incoming email in Japanese. Japanese?
   :author: Joel Spolsky
   :favicon: /favicon.ico
   :feed: http://www.joelonsoftware.com/rss.xml
@@ -68,6 +68,6 @@
   :sentences: I am pleased to report that the GCC Steering Committee and the FSF have approved the use of C++ in GCC itself. Of course, there's no reason for us to use C++ features just because we can. The goal is a better compiler for users, not a C++ code base for its own sake.
 :queness:
   :title: 18 Incredible CSS3 Effects You Have Never Seen Before
-  :lede: "CSS3 is hot these days and will soon be available in most modern browser. Just recently, I started to become aware to the present of CSS3 around the web."
+  :lede: "CSS3 is hot these days and will soon be available in most modern browser. Just recently, I started to become aware to the present of CSS3 around the web. I can see some of the websites such as twitter and designer portfolios websites are using it."
   :sentences: CSS3 is hot these days and will soon be available in most modern browser. Just recently, I started to become aware to the present of CSS3 around the web. I can see some of the websites such as twitter and designer portfolios websites are using it.
   :datetime: 2010-06-02 12:00:00 +01:00

data/test/corpus/reader_expected.yaml

@@ -17,9 +17,6 @@
 :gmane:
 - "I am pleased to report that the GCC Steering Committee and the FSF have approved the use of C++ in GCC itself."
 - "Of course, there's no reason for us to use C++ features just because we can."
-:huffington:
-- "The man on the motorcycle was going the wrong way down a one-way street, gesturing indignantly for the phalanx of traffic-clogged cars in front of him to move."
-- "\"Brother, why are you angry with us?\" said a passenger leaning out of one of the vehicles blocking his path."
 :queness:
 - "CSS3 is hot these days and will soon be available in most modern browser."
 - "Just recently, I started to become aware to the present of CSS3 around the web."

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification 
 name: pismo
 version: !ruby/object:Gem::Version 
-  version: 0.6.1
+  version: 0.6.2
 platform: ruby
 authors: 
 - Peter Cooper