jsl-feedzirra 0.0.12.1

data/README.rdoc

@@ -0,0 +1,194 @@
+== Feedzirra
+
+=== Description
+
+Feedzirra is a feed library that is designed to get and update many feeds as quickly as possible. This includes using libcurl-multi through the 
+taf2-curb[link:http://github.com/taf2/curb/tree/master] gem for faster http gets, and libxml through nokogiri[link:http://github.com/tenderlove/nokogiri/tree/master] 
+and sax-machine[link:http://github.com/pauldix/sax-machine/tree/master] for faster parsing.
+
+It allows for easy customization of feed parsing options through the definition of custom parsing classes, and allows you to take as little or as much control as you want in updating feeds.  Feedzirra
+makes it easy to figure out which content in feeds is new by providing simple 'backends' so that Feedzirra can track the last contents fetched from a  particular feed.  Out of the box, Feedzirra can 
+store this information in the filesystem, Memcached or Tokyo Cabinet.  If you want to keep track of new or updated feeds on your own, just use the default backend which will will let you set options 
+for conditional fetching of feeds without the help of Feedzirra.
+
+=== Installation
+
+For now Feedzirra exists only on github. It also has a few gem requirements that are only on github. Before you start you need to have libcurl[link:http://curl.haxx.se/] and 
+libxml[link:http://xmlsoft.org/] installed. If you're on Leopard you have both. Otherwise, you'll need to grab them. Once you've got those libraries, these are the gems that get 
+used: nokogiri, pauldix-sax-machine, taf2-curb (note that this is a fork that lives on github and not the Ruby Forge version of curb), and pauldix-feedzirra. The feedzirra gemspec has all 
+the dependencies so you should be able to get up and running with the standard github gem install routine:
+
+    gem sources -a http://gems.github.com # if you haven't already
+    gem install pauldix-feedzirra
+
+==== Troubleshooting Installation
+
+*NOTE:*Some people have been reporting a few issues related to installation. First, the Ruby Forge version of curb is not what you want. It will not work. Nor will the curl-multi gem that lives on 
+Ruby Forge. You have to get the taf2-curb[link:http://github.com/taf2/curb/tree/master] fork installed.
+
+If you see this error when doing a require: 
+
+    /Library/Ruby/Site/1.8/rubygems/custom_require.rb:31:in `gem_original_require': no such file to load -- curb_core (LoadError)
+
+It means that the taf2-curb gem didn't build correctly. To resolve this you can do a git clone git://github.com/taf2/curb.git then run rake gem in the curb directory, then sudo gem 
+install pkg/curb-0.2.4.0.gem. After that you should be good.
+
+If you see something like this when trying to run it:
+
+    NoMethodError: undefined method `on_success' for #<Curl::Easy:0x1182724>
+        from ./lib/feedzirra/feed.rb:88:in `add_url_to_multi'
+
+This means that you are requiring curl-multi or the Ruby Forge version of Curb somewhere. You can't use those and need to get the taf2 version up and running.
+
+If you're on Debian or Ubuntu and getting errors while trying to install the taf2-curb gem, it could be because you don't have the latest version of libcurl installed. Do this to fix:
+
+    sudo apt-get install libcurl4-gnutls-dev
+
+Another problem could be if you are running Mac Ports and you have libcurl installed through there. You need to uninstall it for curb to work! The version in Mac Ports is old and doesn't play nice with curb. If you're running Leopard, you can just uninstall and you should be golden. If you're on an older version of OS X, you'll then need to {download curl}[http://curl.haxx.se/download.html] and build from source. Then you'll have to install the taf2-curb gem again. You might have to perform the step above.
+
+If you're still having issues, please let me know on the mailing list. Also, {Todd Fisher (taf2)}[link:http://github.com/taf2] is working on fixing the gem install. Please send him a full error report.
+
+=== Usage
+
+This experimental branch offers a new interface to feed fetching with persistent back-end stores.  This allows you to 
+easily run a script retrieving the feeds once per hour or once per day, and it will remember which feeds have been seen
+before and which are new.  This features uses the Feedzirra::Reader interface.
+
+You can create a Feedzirra::Reader object after the Feedzirra library (with require 'feedzirra') is loaded as follows:
+
+	reader = Feedzirra::Reader.new('http://www.woostercollective.com/rss/index.xml')
+	feed = reader.fetch
+
+The Reader object can take a single URL or a list of URLs followed by a Hash of options.  The options hash
+allows configuration of the backend store, as well as fetching options for the list of urls.  Following is 
+an example of configuration with the Memcache store connected to Tokyo Tyrant (the front-end for Tokyo Cabinet):
+
+	reader = Feedzirra::Reader.new('http://www.pauldix.net/atom.xml', :backend => 
+		{ :class => Feedzirra::Backend::Memcache, :port => 1978, :server => 'localhost' })
+
+Other options that may be put in the options hash follow the original API described below.
+
+Running reader.fetch will first check the back-end store to see if this feed was fetched previously.  If it was previously fetched, 
+Feedzirra uses this information to avoid fetching the whole body if it has already been downloaded based on etag.  If the feed
+has been updated, the new_entries will be populated based on the results of the last query.  The back-end store will be updated with
+the results of every fetch, so Feedzirra will maintain state between executions.  Feedzirra currently supports filesystem, memcache 
+and a Ruby Hash structure-based back end that doesn't attempt to persist any information.
+
+Once you've retrieved a single feed, you can use the accessors below to query the results.
+
+    # feed and entries accessors
+    feed.title          # => "Paul Dix Explains Nothing"
+    feed.url            # => "http://www.pauldix.net"
+    feed.feed_url       # => "http://feeds.feedburner.com/PaulDixExplainsNothing"
+    feed.etag           # => "GunxqnEP4NeYhrqq9TyVKTuDnh0"
+    feed.last_modified  # => Sat Jan 31 17:58:16 -0500 2009 # it's a Time object
+
+    entry = feed.entries.first
+    entry.title      # => "Ruby Http Client Library Performance"
+    entry.url        # => "http://www.pauldix.net/2009/01/ruby-http-client-library-performance.html"
+    entry.author     # => "Paul Dix"
+    entry.summary    # => "..."
+    entry.content    # => "..."
+    entry.published  # => Thu Jan 29 17:00:19 UTC 2009 # it's a Time object
+    entry.categories # => ["...", "..."]
+
+    # sanitizing an entry's content
+    entry.title.sanitize   # => returns the title with harmful stuff escaped
+    entry.author.sanitize  # => returns the author with harmful stuff escaped
+    entry.content.sanitize # => returns the content with harmful stuff escaped
+    entry.content.sanitize! # => returns content with harmful stuff escaped and replaces original (also exists for author and title)
+    entry.sanitize!         # => sanitizes the entry's title, author, and content in place (as in, it changes the value to clean versions)
+    feed.sanitize_entries!  # => sanitizes all entries in place
+
+    # updating a single feed
+    updated_feed = Feedzirra::Feed.update(feed)
+
+    # an updated feed has the following extra accessors
+    updated_feed.updated?     # returns true if any of the feed attributes have been modified. will return false if only new entries
+    updated_feed.new_entries  # a collection of the entry objects that are newer than the latest in the feed before update
+
+    # fetching multiple feeds
+    feed_urls = ["http://feeds.feedburner.com/PaulDixExplainsNothing", "http://feeds.feedburner.com/trottercashion"]
+    feeds = Feedzirra::Reader.new(feed_urls).fetch
+
+    # feeds is now a hash with the feed_urls as keys and the parsed feed objects as values. If an error was thrown
+    # there will be a Fixnum of the http response code instead of a feed object
+
+    # updating multiple feeds.  if you're using a persistent back-end, Feedzirra uses that to determine which entries are ones that you haven't seen before
+    updated_feeds = Feedzirra::reader.new(urls).fetch
+
+    # defining custom behavior on failure or success. note that a return status of 304 (not updated) will call the on_success handler
+    feed = Feedzirra::Reader.new("http://feeds.feedburner.com/PaulDixExplainsNothing",
+        :on_success => lambda {|feed| puts feed.title },
+        :on_failure => lambda {|url, response_code, response_header, response_body| puts response_body }).fetch
+        
+    # if a collection was passed into the initializer, the handlers will be called for each one
+
+=== Extending
+
+Feedzirra is easily extended with custom parsing classes and persistent back-ends.  You'll have to read the source to find out how, though, because we 
+still haven't written the documentation. :(
+
+=== Benchmarks
+
+One of the goals of Feedzirra is speed. This includes not only parsing, but fetching multiple feeds as quickly as possible. I ran a benchmark getting 20 feeds 10 times using Feedzirra, rFeedParser, and FeedNormalizer. For more details the {benchmark code can be found in the project in spec/benchmarks/feedzirra_benchmarks.rb}[http://github.com/pauldix/feedzirra/blob/7fb5634c5c16e9c6ec971767b462c6518cd55f5d/spec/benchmarks/feedzirra_benchmarks.rb]
+
+    feedzirra          5.170000   1.290000   6.460000 ( 18.917796)
+    rfeedparser      104.260000  12.220000 116.480000 (244.799063)
+    feed-normalizer   66.250000   4.010000  70.260000 (191.589862)
+
+The result of that benchmark is a bit sketchy because of the network variability. Running 10 times against the same 20 feeds was meant to smooth some of that out. However, there is also a {benchmark comparing parsing speed in spec/benchmarks/parsing_benchmark.rb}[http://github.com/pauldix/feedzirra/blob/7fb5634c5c16e9c6ec971767b462c6518cd55f5d/spec/benchmarks/parsing_benchmark.rb] on an atom feed.
+
+    feedzirra        0.500000   0.030000   0.530000 (  0.658744)
+    rfeedparser      8.400000   1.110000   9.510000 ( 11.839827)
+    feed-normalizer  5.980000   0.160000   6.140000 (  7.576140)
+
+There's also a {benchmark that shows the results of using Feedzirra to perform updates on feeds}[http://github.com/pauldix/feedzirra/blob/45d64319544c61a4c9eb9f7f825c73b9f9030cb3/spec/benchmarks/updating_benchmarks.rb] you've already pulled in. I tested against 179 feeds. The first is the initial pull and the second is an update 65 seconds later. I'm not sure how many of them support etag and last-modified, so performance may be better or worse depending on what feeds you're requesting.
+
+    feedzirra fetch and parse  4.010000   0.710000   4.720000 ( 15.110101)
+    feedzirra update           0.660000   0.280000   0.940000 (  5.152709)
+
+=== Discussion
+
+I'd like feedback on the api and any bugs encountered on feeds in the wild. I've set up a {google group here}[http://groups.google.com/group/feedzirra].
+
+=== TODO
+
+This thing needs to hammer on many different feeds in the wild. I'm sure there will be bugs. I want to find them and crush them. I didn't bother using the test suite for feedparser. i wanted to start fresh.
+
+Here are some more specific TODOs.
+* Make a feedzirra-rails gem to integrate feedzirra seamlessly with Rails and ActiveRecord.
+* Add support for authenticated feeds.
+* Create a super sweet DSL for defining new parsers.
+* Test against Ruby 1.9.1 and fix any bugs.
+* I'm not keeping track of modified on entries. Should I add this?
+* Clean up the fetching code inside feed.rb so it doesn't suck so hard.
+* Make the feed_spec actually mock stuff out so it doesn't hit the net.
+* Readdress how feeds determine if they can parse a document. Maybe I should use namespaces instead?
+
+=== LICENSE
+
+(The MIT License)
+ 
+Copyright (c) 2009:
+ 
+{Paul Dix}[http://pauldix.net]
+ 
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+'Software'), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+ 
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+ 
+THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
+IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
+CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
+TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
+SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/Rakefile

@@ -0,0 +1,56 @@
+require "spec"
+require "spec/rake/spectask"
+require 'rake/rdoctask'
+require 'lib/feedzirra.rb'
+
+# Grab recently touched specs
+def recent_specs(touched_since)
+  recent_specs = FileList['app/**/*'].map do |path|
+
+    if File.mtime(path) > touched_since
+      spec = File.join('spec', File.dirname(path).split("/")[1..-1].join('/'),
+        "#{File.basename(path, ".*")}_spec.rb")
+      spec if File.exists?(spec)
+    end
+  end.compact
+
+  recent_specs += FileList['spec/**/*_spec.rb'].select do |path|
+    File.mtime(path) > touched_since
+  end
+  recent_specs.uniq
+end
+
+desc "Run all the tests"
+task :default => :spec
+
+# Tasks
+Spec::Rake::SpecTask.new do |t|
+  t.spec_opts = ['--options', "\"#{File.dirname(__FILE__)}/spec/spec.opts\""]
+  t.spec_files = FileList['spec/**/*_spec.rb']
+end
+
+desc 'Run recent specs'
+Spec::Rake::SpecTask.new("spec:recent") do |t|
+  t.spec_opts = ["--format","specdoc","--color"]
+  t.spec_files = recent_specs(Time.now - 600) # 10 min.
+end
+
+Spec::Rake::SpecTask.new('spec:rcov') do |t|
+  t.spec_opts = ['--options', "\"#{File.dirname(__FILE__)}/spec/spec.opts\""]
+  t.spec_files = FileList['spec/**/*_spec.rb']
+  t.rcov = true
+  t.rcov_opts = ['--exclude', 'spec,/usr/lib/ruby,/usr/local,/var/lib,/Library', '--text-report']
+end
+
+Rake::RDocTask.new do |rd|
+  rd.title    = 'Feedzirra'
+  rd.rdoc_dir = 'rdoc'
+  rd.rdoc_files.include('README.rdoc', 'lib/feedzirra.rb', 'lib/feedzirra/**/*.rb')
+  rd.options = ["--quiet", "--opname", "index.html", "--line-numbers", "--inline-source", '--main', 'README.rdoc']
+end
+
+task :install do
+  rm_rf "*.gem"
+  puts `gem build feedzirra.gemspec`
+  puts `sudo gem install feedzirra-#{Feedzirra::VERSION}.gem`
+end

data/lib/core_ext/array.rb

@@ -0,0 +1,8 @@
+class Array
+    
+  # From Rails' ActiveSupport library
+  def extract_options!
+    last.is_a?(::Hash) ? pop : {}
+  end
+
+end

data/lib/core_ext/date.rb

@@ -0,0 +1,21 @@
+# Date code pulled from:
+# Ruby Cookbook by Lucas Carlson and Leonard Richardson
+# Published by O'Reilly
+# ISBN: 0-596-52369-6
+class Date
+  def feed_utils_to_gm_time
+    feed_utils_to_time(new_offset, :gm)
+  end
+
+  def feed_utils_to_local_time
+    feed_utils_to_time(new_offset(DateTime.now.offset-offset), :local)
+  end
+
+  private
+  def feed_utils_to_time(dest, method)
+    #Convert a fraction of a day to a number of microseconds
+    usec = (dest.sec_fraction * 60 * 60 * 24 * (10**6)).to_i
+    Time.send(method, dest.year, dest.month, dest.day, dest.hour, dest.min,
+              dest.sec, usec)
+  end
+end

data/lib/core_ext/string.rb

@@ -0,0 +1,9 @@
+class String
+  def sanitize!
+    self.replace(sanitize)
+  end
+  
+  def sanitize
+    Dryopteris.sanitize(self)
+  end
+end

data/lib/feedzirra/backend/filesystem.rb

@@ -0,0 +1,32 @@
+require 'md5'
+require 'uri'
+
+module Feedzirra
+  module Backend
+    class Filesystem
+      
+      DEFAULTS = {
+        :path => File.join(%w[ ~ / .feedzirra ])
+      }
+      
+      def initialize(options = { })
+        @options = DEFAULTS.merge(options)
+      end
+      
+      def get(url)
+        f = filename_for(url)
+        Marshal.load(File.read(filename_for(url))) if File.exist?(f)
+      end
+    
+      def set(url, result)
+        File.open(filename_for(url), 'w') {|f| f.write(Marshal.dump(result)) }
+      end
+      
+      private
+      
+      def filename_for(url)
+        File.join(@options[:path], MD5.hexdigest(URI.parse(url).normalize.to_s))
+      end
+    end
+  end
+end

data/lib/feedzirra/backend/memcache.rb

@@ -0,0 +1,37 @@
+require 'memcache'
+
+module Feedzirra
+  module Backend
+    
+    # Can be used to set up Memcache, or clients able to speak the Memcache protocol such as 
+    # Tokyo Tyrant, as a Feedzirra::Backend.
+    class Memcache
+      DEFAULTS = {
+        :server => 'localhost',
+        :port   => '11211'
+      }
+      
+      def initialize(options = { })
+        @options  = DEFAULTS.merge(options)
+        @cache    = MemCache.new([ @options[:server], @options[:port] ].join(':'), :namespace => 'Feedzirra')
+      end
+      
+      def get(url)
+        res = @cache.get(key_for(url))
+        Marshal.load(res) unless res.nil?
+      end
+    
+      def set(url, result)
+        @cache.set(key_for(url), Marshal.dump(result))
+      end
+      
+      private
+      
+      def key_for(url)
+        MD5.hexdigest(URI.parse(url).normalize.to_s)
+      end
+
+    end
+    
+  end
+end

data/lib/feedzirra/backend/memory.rb

@@ -0,0 +1,22 @@
+module Feedzirra
+  module Backend
+    
+    # Memory store uses a ruby Hash to store the results of feed fetches.
+    # It won't persist after the application exits.
+    class Memory
+      
+      def initialize(options = { })
+        @store = { }
+      end
+      
+      def get(url)
+        @store[url]
+      end
+    
+      def set(url, result)
+        @store[url] = result
+      end
+      
+    end
+  end
+end

data/lib/feedzirra/feed.rb

@@ -0,0 +1,68 @@
+module Feedzirra
+  
+  # TODO - remove this class, and only use the Feedzirra::Reader class for accessing feed data.
+  class Feed
+        
+    # Fetches and returns the parsed XML for each URL provided.
+    #
+    # === Parameters
+    # [urls<String> or <Array>] A single feed URL, or an array of feed URLs.
+    # [options<Hash>] Valid keys for this argument as as followed:
+    # * :user_agent - String that overrides the default user agent.
+    # * :if_modified_since - Time object representing when the feed was last updated.
+    # * :if_none_match - String, an etag for the request that was stored previously.
+    # * :on_success - Block that gets executed after a successful request.
+    # * :on_failure - Block that gets executed after a failed request.
+    # === Returns
+    # A Feed object if a single URL is passed.
+    #
+    # A Hash if multiple URL's are passed. The key will be the URL, and the value the Feed object.
+    def self.fetch_and_parse(urls, options = {})
+      multi = Feedzirra::HttpMulti.new(urls, options)
+      multi.perform
+      urls.is_a?(String) ? multi.responses.values.first : multi.responses
+    end
+
+    # Updates each feed for each Feed object provided.
+    #
+    # === Parameters
+    # [feeds<Feed> or <Array>] A single feed object, or an array of feed objects.
+    # [options<Hash>] Valid keys for this argument as as followed:
+    #                 * :user_agent - String that overrides the default user agent.
+    #                 * :on_success - Block that gets executed after a successful request.
+    #                 * :on_failure - Block that gets executed after a failed request.
+    # === Returns
+    # A updated Feed object if a single URL is passed.
+    #
+    # A Hash if multiple Feeds are passed. The key will be the URL, and the value the updated Feed object.
+    def self.update(feeds, options = {})
+      multi = Feedzirra::HttpMulti.new(urls, options)
+
+      multi.perform
+      multi.responses.size == 1 ? multi.responses.values.first : multi.responses.values
+    end
+
+    # Fetches and returns the raw XML for each URL provided.
+    #
+    # === Parameters
+    # [urls<String> or <Array>] A single feed URL, or an array of feed URLs.
+    # [options<Hash>] Valid keys for this argument as as followed:
+    #                 :user_agent - String that overrides the default user agent.
+    #                 :if_modified_since - Time object representing when the feed was last updated.
+    #                 :if_none_match - String that's normally an etag for the request that was stored previously.
+    #                 :on_success - Block that gets executed after a successful request.
+    #                 :on_failure - Block that gets executed after a failed request.
+    # === Returns
+    # A String of XML if a single URL is passed.
+    # 
+    # A Hash if multiple URL's are passed. The key will be the URL, and the value the XML.
+    #
+    # FIXME - Raw mode is currently not supported!
+    def self.fetch_raw(urls, options = {})
+      multi = Feedzirra::HttpMulti.new(urls, options.merge(:raw => true))
+      multi.perform
+      urls.is_a?(String) ? multi.responses.values.first : multi.responses
+    end
+     
+  end
+end

data/lib/feedzirra/feed_parser.rb

@@ -0,0 +1,64 @@
+module Feedzirra
+  class NoParserAvailable < StandardError; end
+  
+  # Determines the correct parser for an XML feed.
+  class FeedParser
+    attr_reader :xml, :parser
+    
+    def initialize(xml)
+      @xml    = xml
+      @parser = detect_feed_processor_for_xml(@xml)
+    end
+    
+    def detect_feed_processor_for_xml(xml)
+      start_of_doc = xml.slice(0, 1000)
+
+      if_none_found = lambda do
+        raise NoParserAvailable.new("No valid parser for XML.")
+      end
+
+      feed_classes.detect(if_none_found) do |klass| 
+        klass.able_to_parse?(start_of_doc)
+      end    
+    end
+    
+    # Determines the correct Parser for the XML document with which
+    # we were initialized and returns results of this parser on the 
+    # document.  Raises NoParserAvailable if no valid parser is 
+    # found.
+    def run
+      @parser.parse(@xml)
+    end
+
+    # Adds a new feed parsing class that will be used for parsing.
+    #
+    # === Parameters
+    # [klass<Constant>] The class/constant that you want to register.
+    # === Returns
+    # A updated array of feed parser class names.
+    def add_feed_class(klass) 
+      feed_classes.unshift klass
+    end
+
+    # Provides a list of registered feed parsing classes.
+    #
+    # === Returns
+    # A array of class names.
+    def feed_classes
+      @@feed_classes ||= [Feedzirra::Parser::RSS, Feedzirra::Parser::AtomFeedBurner, Feedzirra::Parser::Atom]
+    end
+    
+    # Makes all entry types look for the passed in element to parse. This is actually just a call to 
+    # element (a SAXMachine call) in the class
+    #
+    # === Parameters
+    # [element_tag<String>]
+    # [options<Hash>] Valid keys are same as with SAXMachine
+    def add_common_feed_entry_element(element_tag, options = {})
+      feed_classes.map{|k| eval("#{k}Entry") }.each do |klass|
+        klass.send(:element, element_tag, options)
+      end
+    end
+    
+  end
+end