nddrylliog_pismo 0.7.3

data/.document

@@ -0,0 +1,5 @@
+README.rdoc
+lib/**/*.rb
+bin/*
+features/**/*.feature
+LICENSE

data/.gitignore

@@ -0,0 +1,29 @@
+pkg/*
+*.gem
+.bundle
+Gemfile.lock
+
+## MAC OS
+.DS_Store
+
+## TEXTMATE
+*.tmproj
+tmtags
+
+## NETBEANS
+nbproject
+
+## EMACS
+*~
+\#*
+.\#*
+
+## VIM
+*.swp
+
+## PROJECT::GENERAL
+coverage
+rdoc
+pkg
+
+## PROJECT::SPECIFIC

data/Gemfile

@@ -0,0 +1,4 @@
+source "http://rubygems.org"
+
+# Specify your gem's dependencies in pismo.gemspec
+gemspec

data/LICENSE

@@ -0,0 +1,23 @@
+Copyright 2009, 2010 Peter Cooper
+
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+     http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+
+--
+
+In short, you can use Pismo for whatever you like, but please include
+a brief credit somewhere deep in your license file or similar, and, 
+if you're a nice kinda person, let me know if you're using it and/or
+share any significant changes or improvements you make.
+
+Peter Cooper
+http://twitter.com/peterc

data/NOTICE

@@ -0,0 +1,4 @@
+Pismo is Copyright (c) 2009, 2010 Peter Cooper
+Pismo is Apache 2.0 Licensed
+Peter Cooper can be found at and contacted via http://twitter.com/peterc
+The source can be found at http://github.com/peterc/pismo

data/README.markdown

@@ -0,0 +1,131 @@
+# pismo - Web page content analysis and metadata extraction
+
+## DESCRIPTION:
+
+Pismo extracts machine-usable metadata from unstructured (or poorly structured) English-language HTML documents.
+Data that Pismo can extract include titles, feed URLs, ledes, body text, image URLs, date, and keywords.
+Pismo is used heavily in production on http://coder.io/ to extract data from Web pages.
+
+All tests pass on Ruby 1.8.7, Ruby 1.9.2 (both MRI) and JRuby 1.5.6.
+
+## NEWS:
+
+December 19, 2010: Version 1.7.2 has been released - it includes a patch from Darcy Laycock to fix keyword extraction problems on some pages, has switched from Jeweler to Bundler for management of the gem, and adds support for JRuby 1.5.6 by skipping stemming on that platform.
+
+## USAGE:
+
+A basic example of extracting basic metadata from a Web page:
+
+    require 'pismo'
+    
+    # Load a Web page (you could pass an IO object or a string with existing HTML data along, as you prefer)
+    doc = Pismo::Document.new('http://www.rubyinside.com/cramp-asychronous-event-driven-ruby-web-app-framework-2928.html')
+    
+    doc.title     # => "Cramp: Asychronous Event-Driven Ruby Web App Framework"
+    doc.author    # => "Peter Cooper"
+    doc.lede      # => "Cramp (GitHub repo) is a new, asynchronous evented Web app framework by Pratik Naik of 37signals (and the Rails core team). It's built around Ruby's EventMachine library and was designed to use event-driven I/O throughout - making it ideal for situations where you need to handle a large number of open connections (such as Comet systems or streaming APIs.)"
+    doc.keywords  # => [["cramp", 7], ["controllers", 3], ["app", 3], ["basic", 2], ..., ... ]
+    
+There's also a shorter "convenience" method which might be handy in IRB - it does the same as Pismo::Document.new:
+
+    Pismo['http://www.rubyflow.com/items/4082'].title   # => "Install Ruby as a non-root User"
+    
+The current metadata methods are:
+
+* title
+* titles
+* author
+* authors
+* lede
+* keywords
+* sentences(qty)
+* body
+* html_body
+* feed
+* feeds
+* favicon
+* description
+* datetime
+
+These methods are not fully documented here yet - you'll just need to try them out. The plural methods like #titles, #authors, and #feeds will return multiple matches in an array, if present. This is so you can use your own techniques to choose a "best" result in ambiguous cases.
+
+The html_body and body methods will be of particular interest. They return the "body" of the page as determined by Pismo's "Reader". #body returns it as plain-text, #html_body maintains some basic HTML styling.
+
+The default reader is the "tree" reader. This works in a similar fashion to Arc90's Readability or Safari Reader algorithm.
+
+New! The keywords method accepts optional arguments. These are the current defaults:
+
+    :stem_at => 20, :word_length_limit => 15, :limit => 20, :remove_stopwords => true, :minimum_score => 2
+    
+You can also pass an array to keywords with :hints => arr if you want only words of your choosing to be found.
+    
+## CAVEATS AND SHORTCOMINGS:
+
+There are some shortcomings or problems that I'm aware of and am going to pursue:
+
+* I do not know how Pismo fares on Rubinius
+* pismo requires Bundler - get it :-)
+* pismo does not install on JRuby due to a problem in the fast-stemmer dependency
+* Some users have had issues with using Pismo from irb. This appears to be related to Nokogiri use causing a segfault
+* The "Reader" content extraction algorithm is not perfect. It can sometimes return crap and can barf on certain types of characters for sentence extraction
+* The author name extraction isn't very strong and is best avoided for now
+* The image extraction only deals with images with absolute URLs
+* The stopword list is a little too long (~1000 words) and needs to be trimmed
+* The corpus in test/corpus needs significantly extending
+
+## OTHER GROOVY STUFF:
+        
+### Command Line Tool
+
+A command line tool called "pismo" is included so that you can get metadata about a page from the command line. This is
+great for testing, or perhaps calling it from a non Ruby script. The output is currently in YAML.
+
+#### Usage: 
+
+    ./bin/pismo http://www.rubyinside.com/cramp-asychronous-event-driven-ruby-web-app-framework-2928.html title lede author datetime
+    
+#### Output:
+
+    --- 
+    :url: http://www.rubyinside.com/cramp-asychronous-event-driven-ruby-web-app-framework-2928.html
+    :title: "Cramp: Asychronous Event-Driven Ruby Web App Framework"
+    :lede: Cramp (GitHub repo)is a new, asynchronous evented Web app framework by Pratik Naik of 37signals
+    :author: Peter Cooper
+    :datetime: 2010-01-07 12:00:00 +00:00
+    
+If you call pismo without any arguments (except a URL), it starts an IRB session so you can directly work in Ruby. The URL provided is loaded
+and assigned to both the constant 'P' and the variable @p.
+
+### Stopword access
+
+You can access Pismo's stopword list directly:
+
+    Pismo.stopwords    # => [.., .., ..]
+
+### Alternate readers
+
+Pismo supports different readers for extracting the #body and #html_body from the web page.
+
+The "cluster" reader uses an algorithm that tries to cluster contiguous content blocks together to identify the main document body.  This is based on the ExtractContent gem (http://rubyforge.org/projects/extractcontent/).
+
+The reader can be specified as part of #Document.new :
+
+    doc = Document.new(url, :reader => :cluster)
+
+
+## Note on Patches/Pull Requests
+ 
+* Fork the project.
+* Make your feature addition or bug fix.
+* Add tests for it. This is important so I don't break it in a future version unintentionally.
+* Commit, do not mess with Rakefile, version, or history as it's handled by Jeweler (which is awesome, btw).
+* Send me a pull request. I may or may not accept it (sorry, practicality rules.. but message me and we can talk!)
+
+## COPYRIGHT AND LICENSE
+
+Apache 2.0 License - See LICENSE for details.
+Copyright (c) 2009, 2010 Peter Cooper
+
+In short, you can use Pismo for whatever you like commercial or not, but please include a brief credit (as in the NOTICE file - as per the Apache 2.0 License) somewhere deep in your license file or similar, and, if you're nice and have the time, let me know if you're using it and/or share any significant changes or improvements you make.
+
+http://github.com/peterc/pismo

data/Rakefile

@@ -0,0 +1,72 @@
+require 'bundler'
+Bundler::GemHelper.install_tasks
+
+require 'rake/testtask'
+Rake::TestTask.new(:test) do |test|
+  test.libs << 'lib' << 'test'
+  test.pattern = 'test/**/test_*.rb'
+  test.verbose = true
+end
+
+begin
+  require 'rcov/rcovtask'
+  Rcov::RcovTask.new do |test|
+    test.libs << 'test'
+    test.pattern = 'test/**/test_*.rb'
+    test.verbose = true
+  end
+rescue LoadError
+  task :rcov do
+    abort "RCov is not available. In order to run rcov, you must: sudo gem install spicycode-rcov"
+  end
+end
+
+task :default => :test
+
+require 'rake/rdoctask'
+Rake::RDocTask.new do |rdoc|
+  version = File.exist?('VERSION') ? File.read('VERSION') : ""
+
+  rdoc.rdoc_dir = 'rdoc'
+  rdoc.title = "pismo #{version}"
+  rdoc.rdoc_files.include('README*')
+  rdoc.rdoc_files.include('lib/**/*.rb')
+end
+
+desc 'Automatically run something when code is changed'
+task :on_update do
+  require 'find'
+  files = {}
+ 
+  loop do
+    changed = false
+    Find.find(File.dirname(__FILE__)) do |file|
+      next unless file =~ /\.rb$/
+      ctime = File.ctime(file).to_i
+ 
+      if ctime != files[file]
+        files[file] = ctime
+        changed = true
+      end
+    end
+ 
+    if changed
+      system ARGV[1] || 'rake'
+      puts "\n" + Time.now.to_s
+    end
+ 
+    sleep 4
+  end
+end
+
+desc 'Console mode'
+task :console do
+  require 'irb'
+  require 'lib/pismo'
+  require 'open-uri'
+  @d = Pismo.document(ARGV[1] || open('./test/corpus/bbcnews.html'))
+  
+  # Get around IRB's issues with ARGV..
+  ARGV = []
+  IRB.start
+end

data/bin/pismo

@@ -0,0 +1,45 @@
+#!/usr/bin/env ruby
+
+# pismo
+#
+# get metadata about a page from the command line
+#
+# Usage: 
+#     ./bin/pismo http://www.rubyinside.com/cramp-asychronous-event-driven-ruby-web-app-framework-2928.html title description author
+# Output:
+#     --- 
+#     :url: http://www.rubyinside.com/cramp-asychronous-event-driven-ruby-web-app-framework-2928.html
+#     :title: "Cramp: Asychronous Event-Driven Ruby Web App Framework"
+#     :description: The ideal book for beginners or developers merely new to Ruby. Goes from installation to OOP, webapps, SQL, and GUI apps.
+#     :author: Peter Cooper
+
+
+require 'yaml'
+require 'rubygems'
+$:.unshift(File.dirname(__FILE__) + "/../lib")
+require 'pismo'
+require 'irb'
+
+url = ARGV.shift
+
+unless url =~ /^http/
+  url = File.read(url)
+end
+
+doc = Pismo.document(url)
+
+if ARGV.empty?
+  P = doc
+  @p = doc
+  puts "Pismo has loaded #{url} into @p and P"
+  #puts "Note: There have been several reports of Nokogiri segfaulting while using Pismo from irb. If this happens, try the same code as a standalone Ruby app."
+  IRB.start
+else
+  output = { :url => doc.url }
+  
+  ARGV.each do |cmd|
+    output[cmd.to_sym] = doc.send(cmd)
+  end
+  
+  puts output.to_yaml
+end

data/lib/pismo.rb

@@ -0,0 +1,82 @@
+# encoding: utf-8
+
+require 'open-uri'
+require 'nokogiri'
+require 'chronic'
+require 'sanitize'
+require 'tempfile'
+
+$: << File.dirname(__FILE__)
+require 'pismo/document'
+require 'pismo/reader'
+require 'pismo/reader/base'
+require 'pismo/reader/tree'
+require 'pismo/reader/cluster'
+
+if RUBY_PLATFORM == "java"
+  class String; def stem; self; end; end
+else
+  require 'fast_stemmer'
+end
+
+module Pismo
+  # Sugar methods to make creating document objects nicer
+  def self.document(handle, options = {})
+    Document.new(handle, options)
+  end
+  
+  # Load a URL, as with Pismo['http://www.rubyinside.com'], and caches the Pismo document
+  # (mostly useful for debugging use)
+  def self.[](url)
+    @docs ||= {}
+    @docs[url] ||= Pismo::Document.new(url)
+  end
+  
+  
+  # Return stopword list
+  def self.stopwords
+    @stopwords ||= File.read(File.dirname(__FILE__) + '/pismo/stopwords.txt').split rescue []
+  end
+  
+  class NFunctions
+    def self.match_href(list, expression)
+      list.find_all { |node| node['href'] =~ /#{expression}/ }
+    end
+  end
+end
+
+# Add some sugar to Nokogiri
+class Nokogiri::HTML::Document
+  def get_the(search)
+    self.search(search).first rescue nil
+  end
+  
+  def match(queries = [], all = false)
+    r = [] if all
+    [*queries].each do |query|
+      if query.is_a?(String)
+        if el = self.search(query).first
+          if el.name.downcase == "meta"
+            result = el['content'].strip rescue nil
+          else
+            result = el.inner_text.strip rescue nil
+          end
+        end
+      elsif query.is_a?(Array)
+        result = query[1].call(self.search(query.first).first).strip rescue nil
+      end
+
+      if result
+        # TODO: Sort out sanitization in a more centralized way
+        result.gsub!('’', '\'')
+        result.gsub!('—', '-')
+        if all
+          r << result
+        else
+          return result
+        end
+      end
+    end
+    all && !r.empty? ? r : nil
+  end
+end

data/lib/pismo/document.rb

@@ -0,0 +1,67 @@
+require 'pismo/internal_attributes'
+require 'pismo/external_attributes'
+
+module Pismo
+  
+  # Pismo::Document represents a single HTML document within Pismo
+  class Document
+    attr_reader :doc, :url, :options
+    
+    ATTRIBUTE_METHODS = InternalAttributes.instance_methods + ExternalAttributes.instance_methods
+    
+    include Pismo::InternalAttributes
+    include Pismo::ExternalAttributes
+    
+    def initialize(handle, options = {})
+      @options = options
+      url = @options.delete(:url)
+      load(handle, url)
+    end
+    
+    # An HTML representation of the document
+    def html
+      @doc.to_s
+    end
+    
+    def load(handle, url = nil)
+      @url = url if url
+      @url = handle if handle =~ /\Ahttp/i
+      
+      @html = if handle =~ /\Ahttp/i
+                open(handle).read
+              elsif handle.is_a?(StringIO) || handle.is_a?(IO) || handle.is_a?(Tempfile)
+                handle.read
+              else
+                handle
+              end
+              
+      @html = self.class.clean_html(@html)
+      
+      @doc = Nokogiri::HTML(@html)
+    end
+    
+    def match(args = [], all = false)
+      @doc.match([*args], all)
+    end
+    
+    def self.clean_html(html)
+      # Normalize stupid entities
+      # TODO: Optimize this so we don't need all these sequential gsubs
+      html.encode!('UTF-8', 'UTF-8', :invalid => :replace)
+      html.gsub!(" ", " ")
+      html.gsub!(" ", " ")
+      html.gsub!(" ", " ")
+      html.gsub!('–', '-')
+      html.gsub!("‘", "'")
+      html.gsub!('’', "'")
+      html.gsub!('“', '"')
+      html.gsub!('”', '"')
+      html.gsub!("…", '...')
+      html.gsub!(' ', ' ')
+      html.gsub!('&lt;', '<')
+      html.gsub!('&gt;', '>')
+      html.gsub!('&amp;', '&')
+      html
+    end
+  end
+end