logophobia-feedzirra 0.0.18 → 0.0.20

data/README.rdoc

@@ -0,0 +1,169 @@
+== Feedzirra
+
+I'd like feedback on the api and any bugs encountered on feeds in the wild. I've set up a {google group here}[http://groups.google.com/group/feedzirra].
+
+=== Description
+
+Feedzirra is a feed library that is designed to get and update many feeds as quickly as possible. This includes using libcurl-multi through the taf2-curb[link:http://github.com/taf2/curb/tree/master] gem for faster http gets, and libxml through nokogiri[link:http://github.com/tenderlove/nokogiri/tree/master] and sax-machine[link:http://github.com/pauldix/sax-machine/tree/master] for faster parsing.
+
+Once you have fetched feeds using Feedzirra, they can be updated using the feed objects. Feedzirra automatically inserts etag and last-modified information from the http response headers to lower bandwidth usage, eliminate unnecessary parsing, and make things speedier in general.
+
+Another feature present in Feedzirra is the ability to create callback functions that get called "on success" and "on failure" when getting a feed. This makes it easy to do things like log errors or update data stores.
+
+The fetching and parsing logic have been decoupled so that either of them can be used in isolation if you'd prefer not to use everything that Feedzirra offers. However, the code examples below use helper methods in the Feed class that put everything together to make things as simple as possible.
+
+The final feature of Feedzirra is the ability to define custom parsing classes. In truth, Feedzirra could be used to parse much more than feeds. Microformats, page scraping, and almost anything else are fair game.
+
+=== Installation
+
+For now Feedzirra exists only on github. It also has a few gem requirements that are only on github. Before you start you need to have libcurl[link:http://curl.haxx.se/] and libxml[link:http://xmlsoft.org/] installed. If you're on Leopard you have both. Otherwise, you'll need to grab them. Once you've got those libraries, these are the gems that get used: nokogiri, pauldix-sax-machine, taf2-curb (note that this is a fork that lives on github and not the Ruby Forge version of curb), and pauldix-feedzirra. The feedzirra gemspec has all the dependencies so you should be able to get up and running with the standard github gem install routine:
+
+    gem sources -a http://gems.github.com # if you haven't already
+    gem install pauldix-feedzirra
+
+*NOTE:*Some people have been reporting a few issues related to installation. First, the Ruby Forge version of curb is not what you want. It will not work. Nor will the curl-multi gem that lives on Ruby Forge. You have to get the taf2-curb[link:http://github.com/taf2/curb/tree/master] fork installed.
+
+If you see this error when doing a require: 
+
+    /Library/Ruby/Site/1.8/rubygems/custom_require.rb:31:in `gem_original_require': no such file to load -- curb_core (LoadError)
+
+It means that the taf2-curb gem didn't build correctly. To resolve this you can do a git clone git://github.com/taf2/curb.git then run rake gem in the curb directory, then sudo gem install pkg/curb-0.2.4.0.gem. After that you should be good.
+
+If you see something like this when trying to run it:
+
+    NoMethodError: undefined method `on_success' for #<Curl::Easy:0x1182724>
+        from ./lib/feedzirra/feed.rb:88:in `add_url_to_multi'
+
+This means that you are requiring curl-multi or the Ruby Forge version of Curb somewhere. You can't use those and need to get the taf2 version up and running.
+
+If you're on Debian or Ubuntu and getting errors while trying to install the taf2-curb gem, it could be because you don't have the latest version of libcurl installed. Do this to fix:
+
+    sudo apt-get install libcurl4-gnutls-dev
+
+Another problem could be if you are running Mac Ports and you have libcurl installed through there. You need to uninstall it for curb to work! The version in Mac Ports is old and doesn't play nice with curb. If you're running Leopard, you can just uninstall and you should be golden. If you're on an older version of OS X, you'll then need to {download curl}[http://curl.haxx.se/download.html] and build from source. Then you'll have to install the taf2-curb gem again. You might have to perform the step above.
+
+If you're still having issues, please let me know on the mailing list. Also, {Todd Fisher (taf2)}[link:http://github.com/taf2] is working on fixing the gem install. Please send him a full error report.
+
+=== Usage
+
+{A gist of the following code}[link:http://gist.github.com/57285]
+
+    require 'feedzirra'
+
+    # fetching a single feed
+    feed = Feedzirra::Feed.fetch_and_parse("http://feeds.feedburner.com/PaulDixExplainsNothing")
+
+    # feed and entries accessors
+    feed.title          # => "Paul Dix Explains Nothing"
+    feed.url            # => "http://www.pauldix.net"
+    feed.feed_url       # => "http://feeds.feedburner.com/PaulDixExplainsNothing"
+    feed.etag           # => "GunxqnEP4NeYhrqq9TyVKTuDnh0"
+    feed.last_modified  # => Sat Jan 31 17:58:16 -0500 2009 # it's a Time object
+
+    entry = feed.entries.first
+    entry.title      # => "Ruby Http Client Library Performance"
+    entry.url        # => "http://www.pauldix.net/2009/01/ruby-http-client-library-performance.html"
+    entry.author     # => "Paul Dix"
+    entry.summary    # => "..."
+    entry.content    # => "..."
+    entry.published  # => Thu Jan 29 17:00:19 UTC 2009 # it's a Time object
+    entry.categories # => ["...", "..."]
+
+    # sanitizing an entry's content
+    entry.title.sanitize   # => returns the title with harmful stuff escaped
+    entry.author.sanitize  # => returns the author with harmful stuff escaped
+    entry.content.sanitize # => returns the content with harmful stuff escaped
+    entry.content.sanitize! # => returns content with harmful stuff escaped and replaces original (also exists for author and title)
+    entry.sanitize!         # => sanitizes the entry's title, author, and content in place (as in, it changes the value to clean versions)
+    feed.sanitize_entries!  # => sanitizes all entries in place
+
+    # updating a single feed
+    updated_feed = Feedzirra::Feed.update(feed)
+
+    # an updated feed has the following extra accessors
+    updated_feed.updated?     # returns true if any of the feed attributes have been modified. will return false if only new entries
+    updated_feed.new_entries  # a collection of the entry objects that are newer than the latest in the feed before update
+
+    # fetching multiple feeds
+    feed_urls = ["http://feeds.feedburner.com/PaulDixExplainsNothing", "http://feeds.feedburner.com/trottercashion"]
+    feeds = Feedzirra::Feed.fetch_and_parse(feed_urls)
+
+    # feeds is now a hash with the feed_urls as keys and the parsed feed objects as values. If an error was thrown
+    # there will be a Fixnum of the http response code instead of a feed object
+
+    # updating multiple feeds. it expects a collection of feed objects
+    updated_feeds = Feedzirra::Feed.update(feeds.values)
+
+    # defining custom behavior on failure or success. note that a return status of 304 (not updated) will call the on_success handler
+    feed = Feedzirra::Feed.fetch_and_parse("http://feeds.feedburner.com/PaulDixExplainsNothing",
+        :on_success => lambda {|feed| puts feed.title },
+        :on_failure => lambda {|url, response_code, response_header, response_body| puts response_body })
+    # if a collection was passed into fetch_and_parse, the handlers will be called for each one
+
+    # the behavior for the handlers when using Feedzirra::Feed.update is slightly different. The feed passed into on_success will be
+    # the updated feed with the standard updated accessors. on failure it will be the original feed object passed into update
+
+    # Defining custom parsers
+    # TODO: the functionality is here, just write some good examples that show how to do this
+
+=== Extending
+
+=== Benchmarks
+
+One of the goals of Feedzirra is speed. This includes not only parsing, but fetching multiple feeds as quickly as possible. I ran a benchmark getting 20 feeds 10 times using Feedzirra, rFeedParser, and FeedNormalizer. For more details the {benchmark code can be found in the project in spec/benchmarks/feedzirra_benchmarks.rb}[http://github.com/pauldix/feedzirra/blob/7fb5634c5c16e9c6ec971767b462c6518cd55f5d/spec/benchmarks/feedzirra_benchmarks.rb]
+
+    feedzirra          5.170000   1.290000   6.460000 ( 18.917796)
+    rfeedparser      104.260000  12.220000 116.480000 (244.799063)
+    feed-normalizer   66.250000   4.010000  70.260000 (191.589862)
+
+The result of that benchmark is a bit sketchy because of the network variability. Running 10 times against the same 20 feeds was meant to smooth some of that out. However, there is also a {benchmark comparing parsing speed in spec/benchmarks/parsing_benchmark.rb}[http://github.com/pauldix/feedzirra/blob/7fb5634c5c16e9c6ec971767b462c6518cd55f5d/spec/benchmarks/parsing_benchmark.rb] on an atom feed.
+
+    feedzirra        0.500000   0.030000   0.530000 (  0.658744)
+    rfeedparser      8.400000   1.110000   9.510000 ( 11.839827)
+    feed-normalizer  5.980000   0.160000   6.140000 (  7.576140)
+
+There's also a {benchmark that shows the results of using Feedzirra to perform updates on feeds}[http://github.com/pauldix/feedzirra/blob/45d64319544c61a4c9eb9f7f825c73b9f9030cb3/spec/benchmarks/updating_benchmarks.rb] you've already pulled in. I tested against 179 feeds. The first is the initial pull and the second is an update 65 seconds later. I'm not sure how many of them support etag and last-modified, so performance may be better or worse depending on what feeds you're requesting.
+
+    feedzirra fetch and parse  4.010000   0.710000   4.720000 ( 15.110101)
+    feedzirra update           0.660000   0.280000   0.940000 (  5.152709)
+
+=== TODO
+
+This thing needs to hammer on many different feeds in the wild. I'm sure there will be bugs. I want to find them and crush them. I didn't bother using the test suite for feedparser. i wanted to start fresh.
+
+Here are some more specific TODOs.
+* Make a feedzirra-rails gem to integrate feedzirra seamlessly with Rails and ActiveRecord.
+* Add support for authenticated feeds.
+* Create a super sweet DSL for defining new parsers.
+* Test against Ruby 1.9.1 and fix any bugs.
+* I'm not keeping track of modified on entries. Should I add this?
+* Clean up the fetching code inside feed.rb so it doesn't suck so hard.
+* Make the feed_spec actually mock stuff out so it doesn't hit the net.
+* Readdress how feeds determine if they can parse a document. Maybe I should use namespaces instead?
+
+=== LICENSE
+
+(The MIT License)
+ 
+Copyright (c) 2009:
+ 
+{Paul Dix}[http://pauldix.net]
+ 
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+'Software'), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+ 
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+ 
+THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
+IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
+CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
+TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
+SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/lib/feedzirra/parser/mrss_category.rb

@@ -0,0 +1,11 @@
+module Feedzirra
+  module Parser
+    class MRSSCategory
+      include SAXMachine
+      
+      element :'media:category', :as => :category
+      element :'media:category', :value => :scheme, :as => :scheme
+      element :'media:category', :value => :label, :as => :label
+    end
+  end
+end

data/lib/feedzirra/parser/mrss_content.rb

@@ -1,24 +1,48 @@
+require File.dirname(__FILE__) + '/mrss_credit'
+require File.dirname(__FILE__) + '/mrss_restriction'
+require File.dirname(__FILE__) + '/mrss_category'
+require File.dirname(__FILE__) + '/mrss_copyright'
+require File.dirname(__FILE__) + '/mrss_hash'
+require File.dirname(__FILE__) + '/mrss_player'
+require File.dirname(__FILE__) + '/mrss_rating'
+require File.dirname(__FILE__) + '/mrss_restriction'
+require File.dirname(__FILE__) + '/mrss_text'
+require File.dirname(__FILE__) + '/mrss_thumbnail'
+
 module Feedzirra
   module Parser
-    class RSSEntry
-      class MRSSContent
-        include SAXMachine
+    class MRSSContent
+      include SAXMachine
+
+      element :'media:content', :as => :url, :value => :url
+      element :'media:content', :as => :content_type, :value => :type
+      element :'media:content', :as => :medium, :value => :medium
+      element :'media:content', :as => :duration, :value => :duration
+      element :'media:content', :as => :isDefault, :value => :isDefault
+      element :'media:content', :as => :expression, :value => :expression
+      element :'media:content', :as => :bitrate, :value => :bitrate
+      element :'media:content', :as => :framerate, :value => :framerate
+      element :'media:content', :as => :samplingrate, :value => :sampling
+      element :'media:content', :as => :channels, :value => :duration
+      element :'media:content', :as => :height, :value => :height
+      element :'media:content', :as => :width, :value => :width
+      element :'media:content', :as => :lang, :value => :lang
+      element :'media:content', :as => :fileSize, :value => :fileSize
+
+      # optional elements
+      element :'media:title', :as => :media_title
+      element :'media:keywords', :as => :media_keywords
+      element :'media:description', :as => :media_description
 
-        element :'media:content', :as => :url, :value => :url
-        element :'media:content', :as => :content_type, :value => :type
-        element :'media:content', :as => :medium, :value => :medium
-        element :'media:content', :as => :duration, :value => :duration
-        element :'media:content', :as => :isDefault, :value => :isDefault
-        element :'media:content', :as => :expression, :value => :expression
-        element :'media:content', :as => :bitrate, :value => :bitrate
-        element :'media:content', :as => :framerate, :value => :framerate
-        element :'media:content', :as => :samplingrate, :value => :sampling
-        element :'media:content', :as => :channels, :value => :duration
-        element :'media:content', :as => :height, :value => :height
-        element :'media:content', :as => :width, :value => :width
-        element :'media:content', :as => :lang, :value => :lang
-        element :'media:content', :as => :fileSize, :value => :fileSize
-      end
+      element :'media:thumbnail', :as => :media_thumbnail, :class => MRSSThumbnail
+      element :'media:rating', :as => :rating, :class => MRSSRating
+      element :'media:category', :as => :media_category, :class => MRSSCategory
+      element :'media:hash', :as => :media_hash, :class => MRSSHash
+      element :'media:player', :as => :media_player, :class => MRSSPlayer
+      elements :'media:credit', :as => :credits, :class => MRSSCredit
+      element :'media:copyright', :as => :copyright, :class => MRSSCopyright
+      element :'media:restriction', :as => :media_restriction, :class => MRSSRestriction
+      element :'media:text', :as => :text, :class => MRSSText
     end
   end
 end

data/lib/feedzirra/parser/mrss_copyright.rb

@@ -0,0 +1,10 @@
+module Feedzirra
+  module Parser
+    class MRSSCopyright
+      include SAXMachine
+      
+      element :'media:copyright', :as => :copyright
+      element :'media:copyright', :as => :url, :value => :url
+    end
+  end
+end

data/lib/feedzirra/parser/mrss_credit.rb

@@ -1,13 +1,11 @@
 module Feedzirra
   module Parser
-    class RSSEntry
-      class MRSSCredit
-        include SAXMachine
+    class MRSSCredit
+      include SAXMachine
 
-        element :'media:credit', :as => :role, :value => :role
-        element :'media:credit', :as => :scheme, :value => :scheme
-        element :'media:credit', :as => :name
-      end
+      element :'media:credit', :as => :role, :value => :role
+      element :'media:credit', :as => :scheme, :value => :scheme
+      element :'media:credit', :as => :name
     end
   end
 end

data/lib/feedzirra/parser/mrss_group.rb

@@ -1,13 +1,37 @@
 require File.dirname(__FILE__) + '/mrss_content'
+require File.dirname(__FILE__) + '/mrss_credit'
+require File.dirname(__FILE__) + '/mrss_restriction'
+require File.dirname(__FILE__) + '/mrss_group'
+require File.dirname(__FILE__) + '/mrss_category'
+require File.dirname(__FILE__) + '/mrss_copyright'
+require File.dirname(__FILE__) + '/mrss_hash'
+require File.dirname(__FILE__) + '/mrss_player'
+require File.dirname(__FILE__) + '/mrss_rating'
+require File.dirname(__FILE__) + '/mrss_restriction'
+require File.dirname(__FILE__) + '/mrss_text'
+require File.dirname(__FILE__) + '/mrss_thumbnail'
 
 module Feedzirra
   module Parser
-    class RSSEntry
-      class MRSSGroup
-        include SAXMachine
+    class MRSSGroup
+      include SAXMachine
 
-        elements :'media:content', :as => :media_content, :class => MRSSContent
-      end
+      elements :'media:content', :as => :media_content, :class => MRSSContent
+
+      # optional elements
+      element :'media:title', :as => :media_title
+      element :'media:keywords', :as => :media_keywords
+      element :'media:description', :as => :media_description
+
+      element :'media:thumbnail', :as => :media_thumbnail, :class => MRSSThumbnail
+      element :'media:rating', :as => :rating, :class => MRSSRating
+      element :'media:category', :as => :media_category, :class => MRSSCategory
+      element :'media:hash', :as => :media_hash, :class => MRSSHash
+      element :'media:player', :as => :media_player, :class => MRSSPlayer
+      elements :'media:credit', :as => :credits, :class => MRSSCredit
+      element :'media:copyright', :as => :copyright, :class => MRSSCopyright
+      element :'media:restriction', :as => :media_restriction, :class => MRSSRestriction
+      element :'media:text', :as => :text, :class => MRSSText
     end
   end
 end

data/lib/feedzirra/parser/mrss_hash.rb

@@ -0,0 +1,10 @@
+module Feedzirra
+  module Parser
+    class MRSSHash
+      include SAXMachine
+      
+      element :'media:hash', :as => :hash
+      element :'media:hash', :value => :algo, :as => :algo
+    end
+  end
+end

data/lib/feedzirra/parser/mrss_player.rb

@@ -0,0 +1,11 @@
+module Feedzirra
+  module Parser
+    class MRSSPlayer
+      include SAXMachine
+      
+      element :'media:player', :value => :url, :as => :url
+      element :'media:player', :value => :width, :as => :width
+      element :'media:player', :value => :height, :as => :height
+    end
+  end
+end

data/lib/feedzirra/parser/mrss_rating.rb

@@ -0,0 +1,10 @@
+module Feedzirra
+  module Parser
+    class MRSSRating
+      include SAXMachine
+      
+      element :'media:rating', :as => :rating
+      element :'media:rating', :value => :scheme, :as => :scheme
+    end
+  end
+end

data/lib/feedzirra/parser/mrss_restriction.rb

@@ -1,13 +1,11 @@
 module Feedzirra
   module Parser
-    class RSSEntry
-      class MRSSRestriction
-        include SAXMachine
+    class MRSSRestriction
+      include SAXMachine
 
-        element :'media:restriction', :as => :value
-        element :'media:restriction', :as => :scope, :value => :type
-        element :'media:restriction', :as => :relationship, :value => :relationship
-      end
+      element :'media:restriction', :as => :value
+      element :'media:restriction', :as => :scope, :value => :type
+      element :'media:restriction', :as => :relationship, :value => :relationship
     end
   end
 end

data/lib/feedzirra/parser/mrss_text.rb

@@ -0,0 +1,13 @@
+module Feedzirra
+  module Parser
+    class MRSSText
+      include SAXMachine
+      
+      element :'media:text', :as => :type, :value => :type
+      element :'media:text', :as => :lang, :value => :lang
+      element :'media:text', :as => :start, :value => :start
+      element :'media:text', :as => :end, :value => :end
+      element :'media:text', :as => :text
+    end
+  end
+end

data/lib/feedzirra/parser/mrss_thumbnail.rb

@@ -0,0 +1,11 @@
+module Feedzirra
+  module Parser
+    class MRSSThumbnail
+      include SAXMachine
+      
+      element :'media:thumbnail', :as => :url, :value => :url
+      element :'media:thumbnail', :as => :with, :value => :width
+      element :'media:thumbnail', :as => :height, :value => :height
+    end
+  end
+end

data/lib/feedzirra/parser/rss.rb

@@ -1,3 +1,14 @@
+require File.dirname(__FILE__) + '/mrss_credit'
+require File.dirname(__FILE__) + '/mrss_restriction'
+require File.dirname(__FILE__) + '/mrss_category'
+require File.dirname(__FILE__) + '/mrss_copyright'
+require File.dirname(__FILE__) + '/mrss_hash'
+require File.dirname(__FILE__) + '/mrss_player'
+require File.dirname(__FILE__) + '/mrss_rating'
+require File.dirname(__FILE__) + '/mrss_restriction'
+require File.dirname(__FILE__) + '/mrss_text'
+require File.dirname(__FILE__) + '/mrss_thumbnail'
+
 module Feedzirra
   module Parser
     # == Summary
@@ -49,6 +60,21 @@ module Feedzirra
       # elements :'itunes:category', :as => :itunes_categories,
       #   :class => ITunesCategory
 
+      # MediaRSS support
+      element :'media:title', :as => :media_title
+      element :'media:keywords', :as => :media_keywords
+      element :'media:description', :as => :media_description
+
+      element :'media:thumbnail', :as => :media_thumbnail, :class => MRSSThumbnail
+      element :'media:rating', :as => :rating, :class => MRSSRating
+      element :'media:category', :as => :media_category, :class => MRSSCategory
+      element :'media:hash', :as => :media_hash, :class => MRSSHash
+      element :'media:player', :as => :media_player, :class => MRSSPlayer
+      elements :'media:credit', :as => :credits, :class => MRSSCredit
+      element :'media:copyright', :as => :copyright, :class => MRSSCopyright
+      element :'media:restriction', :as => :media_restriction, :class => MRSSRestriction
+      element :'media:text', :as => :text, :class => MRSSText
+
       def self.able_to_parse?(xml) #:nodoc:
         xml =~ /\<rss|rdf/
       end

data/lib/feedzirra/parser/rss_entry.rb

@@ -2,6 +2,14 @@ require File.dirname(__FILE__) + '/mrss_content'
 require File.dirname(__FILE__) + '/mrss_credit'
 require File.dirname(__FILE__) + '/mrss_restriction'
 require File.dirname(__FILE__) + '/mrss_group'
+require File.dirname(__FILE__) + '/mrss_category'
+require File.dirname(__FILE__) + '/mrss_copyright'
+require File.dirname(__FILE__) + '/mrss_hash'
+require File.dirname(__FILE__) + '/mrss_player'
+require File.dirname(__FILE__) + '/mrss_rating'
+require File.dirname(__FILE__) + '/mrss_restriction'
+require File.dirname(__FILE__) + '/mrss_text'
+require File.dirname(__FILE__) + '/mrss_thumbnail'
 
 module Feedzirra
   module Parser
@@ -44,44 +52,22 @@ module Feedzirra
       element :"dc:creator", :as => :author
       element :"dcterms:modified", :as => :updated
 
-      # MediaRSS support
-      element :'media:thumbnail', :as => :media_thumbnail, :value => :url
-      element :'media:thumbnail', :as => :media_thumbnail_width, :value => :width
-      element :'media:thumbnail', :as => :media_thumbnail_height, :value => :height
-      element :'media:description', :as => :media_description
-
-      element :'media:rating', :as => :rating
-      element :'media:rating', :value => :scheme, :as => :rating_scheme
-
+      # MediaRSS support, optional elements
       element :'media:title', :as => :media_title
       element :'media:keywords', :as => :media_keywords
+      element :'media:description', :as => :media_description
 
-      element :'media:category', :as => :media_category
-      element :'media:category', :value => :scheme, :as => :media_category_scheme
-      element :'media:category', :value => :label, :as => :media_category_label
-
-      element :'media:hash', :as => :media_hash
-      element :'media:hash', :value => :algo, :as => :media_hash_algo
-
-      element :'media:player', :value => :url, :as => :media_player_url
-      element :'media:player', :value => :width, :as => :media_player_width
-      element :'media:player', :value => :height, :as => :media_player_height
-
+      element :'media:thumbnail', :as => :media_thumbnail, :class => MRSSThumbnail
+      element :'media:rating', :as => :rating, :class => MRSSRating
+      element :'media:category', :as => :media_category, :class => MRSSCategory
+      element :'media:hash', :as => :media_hash, :class => MRSSHash
+      element :'media:player', :as => :media_player, :class => MRSSPlayer
       elements :'media:credit', :as => :credits, :class => MRSSCredit
-
-      element :'media:copyright', :as => :copyright
-      element :'media:copyright', :as => :copyright_url, :value => :url
-
+      element :'media:copyright', :as => :copyright, :class => MRSSCopyright
       element :'media:restriction', :as => :media_restriction, :class => MRSSRestriction
-
+      element :'media:text', :as => :text, :class => MRSSText
       elements :'media:content', :as => :media_content, :class => MRSSContent
-      element :'media:group', :as => :media_group, :class => MRSSGroup
-
-      element :'media:text', :as => :media_text_type, :value => :type
-      element :'media:text', :as => :media_text_lang, :value => :lang
-      element :'media:text', :as => :media_text_start, :value => :start
-      element :'media:text', :as => :media_text_end, :value => :end
-      element :'media:text', :as => :media_text
+      elements :'media:group', :as => :media_groups, :class => MRSSGroup
 
       # iTunes
       element :'itunes:author', :as => :author

data/lib/feedzirra.rb

@@ -19,11 +19,20 @@ require 'feedzirra/feed_utilities'
 require 'feedzirra/feed_entry_utilities'
 require 'feedzirra/feed'
 
-require 'feedzirra/parser/rss_entry'
-require 'feedzirra/parser/rss_image'
 require 'feedzirra/parser/mrss_content'
+require 'feedzirra/parser/mrss_credit'
 require 'feedzirra/parser/mrss_restriction'
 require 'feedzirra/parser/mrss_group'
+require 'feedzirra/parser/mrss_category'
+require 'feedzirra/parser/mrss_copyright'
+require 'feedzirra/parser/mrss_hash'
+require 'feedzirra/parser/mrss_player'
+require 'feedzirra/parser/mrss_rating'
+require 'feedzirra/parser/mrss_restriction'
+require 'feedzirra/parser/mrss_text'
+require 'feedzirra/parser/mrss_thumbnail'
+require 'feedzirra/parser/rss_entry'
+require 'feedzirra/parser/rss_image'
 require 'feedzirra/parser/itunes_category'
 require 'feedzirra/parser/atom_entry'
 require 'feedzirra/parser/atom_feed_burner_entry'
@@ -33,5 +42,5 @@ require 'feedzirra/parser/atom'
 require 'feedzirra/parser/atom_feed_burner'
 
 module Feedzirra
-  VERSION = "0.0.18"
+  VERSION = "0.0.20"
 end

data/spec/benchmarks/feed_benchmarks.rb

@@ -0,0 +1,98 @@
+# this is some spike code to compare the speed of different methods for performing
+# multiple feed fetches
+require 'rubygems'
+require 'curb'
+require 'activesupport'
+
+require 'net/http'
+require 'uri'
+
+require 'benchmark'
+include Benchmark
+
+GET_COUNT = 1
+urls = ["http://www.pauldix.net"] * GET_COUNT
+
+
+benchmark do |t|
+  t.report("taf2-curb") do
+    multi = Curl::Multi.new
+    urls.each do |url|
+      easy = Curl::Easy.new(url) do |curl|
+        curl.headers["User-Agent"] = "feedzirra"
+    #    curl.headers["If-Modified-Since"] = Time.now.httpdate
+    #    curl.headers["If-None-Match"] = "ziEyTl4q9GH04BR4jgkImd0GvSE"
+        curl.follow_location = true
+        curl.on_success do |c|
+    #      puts c.header_str.inspect
+#          puts c.response_code
+    #      puts c.body_str.slice(0, 500)
+        end
+        curl.on_failure do |c|
+          puts "**** #{c.response_code}"
+        end
+      end
+      multi.add(easy)
+    end
+
+    multi.perform
+  end
+
+  t.report("nethttp") do
+    urls.each do |url|
+      res = Net::HTTP.get(URI.parse(url))
+#      puts res.slice(0, 500)
+    end
+  end
+  
+  require 'rfuzz/session'
+  include RFuzz 
+  t.report("rfuzz") do
+    GET_COUNT.times do
+      http = HttpClient.new("www.pauldix.net", 80)
+      response = http.get("/")
+      if response.http_status != "200" 
+        puts "***** #{response.http_status}"
+      else
+#        puts response.http_status
+  #      puts response.http_body.slice(0, 500)
+      end
+    end
+  end
+  
+  require 'eventmachine'
+  t.report("eventmachine") do
+    counter = GET_COUNT
+    EM.run do
+      GET_COUNT.times do
+        http = EM::Protocols::HttpClient2.connect("www.pauldix.net", 80)
+        request = http.get("/")
+        request.callback do
+#          puts request.status
+#          puts request.content.slice(0, 500)
+          counter -= 1
+          EM.stop if counter == 0
+        end
+      end
+    end
+  end
+  
+  
+  require 'curl-multi'
+  t.report("curl multi") do
+    multi = Curl::Multi.new
+    urls.each do |url|
+      on_failure = lambda do |ex|
+        puts "****** Failed to retrieve #{url}"
+      end
+
+      on_success = lambda do |body|
+#        puts "got #{url}"
+#        puts body.slice(0, 500)
+      end
+      multi.get(url, on_success, on_failure)
+    end
+
+    multi.select([], []) while multi.size > 0
+  end
+end

data/spec/benchmarks/feedzirra_benchmarks.rb

@@ -0,0 +1,40 @@
+require File.dirname(__FILE__) + '/../../lib/feedzirra.rb'
+require 'rfeedparser'
+require 'feed-normalizer'
+require 'open-uri'
+
+require 'benchmark'
+include Benchmark
+
+iterations = 10
+urls = File.readlines(File.dirname(__FILE__) + "/../sample_feeds/successful_feed_urls.txt").slice(0, 20)
+puts "benchmarks on #{urls.size} feeds"
+puts "************************************"
+benchmark do |t|    
+  t.report("feedzirra") do
+    iterations.times do
+      Feedzirra::Feed.fetch_and_parse(urls, :on_success => lambda { |url, feed| $stdout.print '.'; $stdout.flush })
+    end
+  end
+
+  t.report("rfeedparser") do
+    iterations.times do
+      urls.each do |url|
+        feed = FeedParser.parse(url)
+        $stdout.print '.'
+        $stdout.flush
+      end
+    end
+  end
+
+  t.report("feed-normalizer") do
+    iterations.times do
+      urls.each do |url|
+        # have to use the :force option to make feed-normalizer parse an atom feed
+        feed = FeedNormalizer::FeedNormalizer.parse(open(url), :force_parser => FeedNormalizer::SimpleRssParser)
+        $stdout.print '.'
+        $stdout.flush
+      end
+    end
+  end
+end

data/spec/benchmarks/fetching_benchmarks.rb

@@ -0,0 +1,28 @@
+require 'rubygems'
+require File.dirname(__FILE__) + '/../../lib/feedzirra.rb'
+
+require 'open-uri'
+
+require 'benchmark'
+include Benchmark
+
+iterations = 10
+urls = File.readlines(File.dirname(__FILE__) + "/../sample_feeds/successful_feed_urls.txt").slice(0, 20)
+puts "benchmarks on #{urls.size} feeds"
+puts "************************************"
+benchmark do |t|    
+  t.report("feedzirra open uri") do
+    iterations.times do
+      urls.each do |url|
+        Feedzirra::Feed.parse(open(url, "User-Agent" => "feedzirra http://github.com/pauldix/feedzirra/tree/master").read)
+        $stdout.print '.'; $stdout.flush
+      end
+    end
+  end
+
+  t.report("feedzirra fetch and parse") do
+    iterations.times do
+      Feedzirra::Feed.fetch_and_parse(urls, :on_success => lambda { |url, feed| $stdout.print '.'; $stdout.flush })
+    end
+  end  
+end

data/spec/benchmarks/parsing_benchmark.rb

@@ -0,0 +1,30 @@
+require File.dirname(__FILE__) + '/../../lib/feedzirra.rb'
+require 'rfeedparser'
+require 'feed-normalizer'
+
+require 'benchmark'
+include Benchmark
+
+iterations = 50
+xml = File.read(File.dirname(__FILE__) + '/../sample_feeds/PaulDixExplainsNothing.xml')
+
+benchmark do |t|    
+  t.report("feedzirra") do
+    iterations.times do
+      Feedzirra::Feed.parse(xml)
+    end
+  end
+
+  t.report("rfeedparser") do
+    iterations.times do
+      FeedParser.parse(xml)
+    end
+  end
+
+  t.report("feed-normalizer") do
+    iterations.times do
+      # have to use the :force option to make feed-normalizer parse an atom feed
+      FeedNormalizer::FeedNormalizer.parse(xml, :force_parser => FeedNormalizer::SimpleRssParser)
+    end
+  end
+end

data/spec/benchmarks/updating_benchmarks.rb

@@ -0,0 +1,33 @@
+require 'rubygems'
+require File.dirname(__FILE__) + '/../../lib/feedzirra.rb'
+
+require 'benchmark'
+include Benchmark
+
+urls = File.readlines(File.dirname(__FILE__) + "/../sample_feeds/successful_feed_urls.txt")
+puts "benchmarks on #{urls.size} feeds"
+puts "************************************"
+benchmark do |t|
+  feeds = {}
+  t.report("feedzirra fetch and parse") do
+    feeds = Feedzirra::Feed.fetch_and_parse(urls, 
+      :on_success => lambda { |url, feed| $stdout.print '.'; $stdout.flush },
+      :on_failure => lambda {|url, response_code, header, body| puts "#{response_code} ERROR on #{url}"})
+  end
+  
+  # curb caches the dns lookups for 60 seconds. to make things fair we have to wait for the cache to expire
+  puts "sleeping to wait for dns cache to clear"
+  65.times {$stdout.print('.'); sleep(1)}
+  puts "done"
+  
+  updated_feeds = []
+  t.report("feedzirra update") do
+    updated_feeds = Feedzirra::Feed.update(feeds.values.reject {|f| f.class == Fixnum}, 
+      :on_success => lambda {|feed| $stdout.print '.'; $stdout.flush},
+      :on_failure => lambda {|feed, response_code, header, body| puts "#{response_code} ERROR on #{feed.feed_url}"})
+  end
+  
+  updated_feeds.each do |feed|
+    puts feed.feed_url if feed.updated?
+  end
+end

data/spec/feedzirra/parser/mrss_content_spec.rb

@@ -0,0 +1,32 @@
+require File.join(File.dirname(__FILE__), %w[.. .. spec_helper])
+
+describe Feedzirra::Parser::RSSEntry::MRSSContent do
+  before(:each) do
+    # I don't really like doing it this way because these unit test should only rely on RSSEntry,
+    # but this is actually how it should work. You would never just pass entry xml straight to the AtomEnry
+    @entries = Feedzirra::Parser::RSS.parse(sample_mrss_feed).entries
+  end
+
+  it "should parse the media" do
+    entry = @entries.first
+    entry.media_content.size.should == 1
+    entry.media_description.should == 'The story began with a July 23 article in a local newspaper, The Independent. Jenna Hewitt, 26, of Montauk, and three friends said they found the ...'
+    entry.media_thumbnail.should == 'http://3.gvt0.com/vi/Y3rNEu4A8WM/default.jpg'
+    entry.media_thumbnail_width.should == '320'
+    entry.media_thumbnail_height.should == '240'
+  end
+
+  it "should handle multiple pieces of content" do
+    media = @entries[1].media_content
+    media.size.should == 2
+    media[0].url.should == 'http://www.youtube.com/v/pvaM6sjLbuA&fs=1'
+    media[0].content_type.should == 'application/x-shockwave-flash'
+    media[0].medium.should == 'video'
+    media[0].duration.should == '575'
+
+    media[1].url.should == 'http://www.youtube.com/v/pvaM6sjLbuA&fs=2'
+    media[1].content_type.should == 'video/mp4'
+    media[1].medium.should == 'video'
+    media[1].duration.should == '576'
+  end
+end

data/spec/sample_feeds/run_against_sample.rb

@@ -0,0 +1,20 @@
+require 'rubygems'
+require File.dirname(__FILE__) + "/../../lib/feedzirra.rb"
+
+feed_urls = File.readlines(File.dirname(__FILE__) + "/top5kfeeds.dat").collect {|line| line.split.first}
+
+success = lambda do |url, feed|
+  puts "SUCCESS - #{feed.title} - #{url}"
+end
+
+failed_feeds = []
+failure = lambda do |url, response_code, header, body|
+  failed_feeds << url if response_code == 200
+  puts "*********** FAILED with #{response_code} on #{url}"
+end
+
+Feedzirra::Feed.fetch_and_parse(feed_urls, :on_success => success, :on_failure => failure)
+
+File.open("./failed_urls.txt", "w") do |f|
+  f.write failed_feeds.join("\n")
+end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification 
 name: logophobia-feedzirra
 version: !ruby/object:Gem::Version 
-  version: 0.0.18
+  version: 0.0.20
 platform: ruby
 authors: 
 - Paul Dix
@@ -81,37 +81,51 @@ extensions: []
 extra_rdoc_files: []
 
 files: 
-- lib/core_ext/date.rb
-- lib/core_ext/string.rb
 - lib/feedzirra.rb
 - lib/feedzirra/feed.rb
-- lib/feedzirra/parser/atom.rb
-- lib/feedzirra/parser/atom_entry.rb
+- lib/feedzirra/feed_utilities.rb
+- lib/feedzirra/parser/rss_entry.rb
 - lib/feedzirra/parser/atom_feed_burner.rb
-- lib/feedzirra/parser/atom_feed_burner_entry.rb
-- lib/feedzirra/parser/itunes_category.rb
+- lib/feedzirra/parser/mrss_hash.rb
+- lib/feedzirra/parser/atom_entry.rb
 - lib/feedzirra/parser/rss.rb
-- lib/feedzirra/parser/rss_entry.rb
+- lib/feedzirra/parser/mrss_rating.rb
+- lib/feedzirra/parser/mrss_text.rb
+- lib/feedzirra/parser/mrss_player.rb
+- lib/feedzirra/parser/mrss_group.rb
 - lib/feedzirra/parser/rss_image.rb
-- lib/feedzirra/parser/mrss_content.rb
+- lib/feedzirra/parser/atom.rb
+- lib/feedzirra/parser/atom_feed_burner_entry.rb
+- lib/feedzirra/parser/mrss_thumbnail.rb
 - lib/feedzirra/parser/mrss_credit.rb
+- lib/feedzirra/parser/itunes_category.rb
 - lib/feedzirra/parser/mrss_restriction.rb
-- lib/feedzirra/parser/mrss_group.rb
-- lib/feedzirra/feed_utilities.rb
+- lib/feedzirra/parser/mrss_category.rb
+- lib/feedzirra/parser/mrss_content.rb
+- lib/feedzirra/parser/mrss_copyright.rb
 - lib/feedzirra/feed_entry_utilities.rb
+- lib/core_ext/date.rb
+- lib/core_ext/string.rb
+- README.rdoc
 - README.textile
 - Rakefile
-- spec/spec.opts
-- spec/spec_helper.rb
-- spec/feedzirra/feed_spec.rb
+- spec/feedzirra/parser/mrss_content_spec.rb
+- spec/feedzirra/parser/rss_entry_spec.rb
+- spec/feedzirra/parser/atom_feed_burner_entry_spec.rb
+- spec/feedzirra/parser/rss_spec.rb
 - spec/feedzirra/parser/atom_spec.rb
 - spec/feedzirra/parser/atom_entry_spec.rb
 - spec/feedzirra/parser/atom_feed_burner_spec.rb
-- spec/feedzirra/parser/atom_feed_burner_entry_spec.rb
-- spec/feedzirra/parser/rss_spec.rb
-- spec/feedzirra/parser/rss_entry_spec.rb
-- spec/feedzirra/feed_utilities_spec.rb
 - spec/feedzirra/feed_entry_utilities_spec.rb
+- spec/feedzirra/feed_spec.rb
+- spec/feedzirra/feed_utilities_spec.rb
+- spec/benchmarks/feed_benchmarks.rb
+- spec/benchmarks/updating_benchmarks.rb
+- spec/benchmarks/feedzirra_benchmarks.rb
+- spec/benchmarks/fetching_benchmarks.rb
+- spec/benchmarks/parsing_benchmark.rb
+- spec/sample_feeds/run_against_sample.rb
+- spec/spec_helper.rb
 has_rdoc: true
 homepage: http://github.com/pauldix/feedzirra
 post_install_message: 

data/spec/spec.opts

@@ -1,2 +0,0 @@
---diff
---color