jsl-feedtosis 0.0.3 → 0.0.3.1

data/README.rdoc

@@ -9,17 +9,16 @@ well as by pointing out which entries are new in any given feed.
 Feedtosis is designed to help you with book-keeping about feed fetching
 details. This is usually something that is mundane and not fundamentally related
 to the business logic of applications that deal with the consumption of
-syndicated content on the web. Feedtosis keeps track of these mundane
-details so you can just keep grabbing new content without wasting bandwidth in
-making unnecessary requests and programmer time in implementing algorithms to
-figure out which feed entries are new.
+syndicated content on the web. Feedtosis keeps track of these details so you can 
+just keep grabbing new content without wasting bandwidth in making unnecessary requests 
+and programmer time in implementing algorithms to figure out which feed entries are new.
 
 Feedtosis fits into other frameworks to do the heavy lifting, including the
 Curb library which does HTTP requests through curl, and FeedNormalizer which
 abstracts the differences between syndication formats. In the sense that it fits
 into these existing, robust programs, Feedtosis is a modular middleware
 piece that efficiently glues together disparate parts to create a helpful feed
-reader with a minimal (< 200 LOC), test-covered codebase.
+reader with a minimal, spec-covered codebase.
 
 == Installation
 
@@ -73,13 +72,11 @@ from memory. To configure a different backend, pass an options hash to the
 Feedtosis client initialization:
 
   url = "http://newsrss.bbc.co.uk/rss/newsonline_world_edition/south_asia/rss.xml" 
-  mf = Feedtosis::Client.new(url, :backend => {:moneta_klass => 'Moneta::Memcache', :server => 'localhost:1978'}) 
-  res = mf.fetch
+  f = Feedtosis::Client.new(url, :backend => Moneta::Memcache.new(:server => 'localhost:1978')) 
+  res = f.fetch
 
 This example sets up a Memcache backend, which in this case points to Tokyo
-Tyrant on port 1978. Note that Moneta::Memcache can be given as a string, in
-which case you don't have to manually require Moneta::Memcache before
-initializing the client.
+Tyrant on port 1978.
 
 Generally, Feedtosis supports all systems supported by Moneta, and any one
 of the supported systems can be given to the +moneta_klass+ parameter. Other
@@ -109,13 +106,19 @@ computation and storage) which feed entries should be presented to the user as
   { 
     :etag => "4c8f-46ac09fbbe940", 
     :last_modified => "Mon, 25 May 2009 18:17:33 GMT",
-    :digests => ["f2993783ded928637ce5f2dc2d837f10", "da64efa6dd9ce34e5699b9efe73a37a7"] 
+    :digests => [["f2993783ded928637ce5f2dc2d837f10", "da64efa6dd9ce34e5699b9efe73a37a7"]]
   }
 
 The data stored by Feedtosis in the summary structure allows it to be
 helpful to the user without storing lots of data that are unnecessary for
 efficient functioning.
 
+The summary structure keeps an Array of Arrays containing digests of feeds.  The reason
+for this is that some feeds, such as the Google blog search feeds, contain slightly different
+but often-recurring results in the result set.  Feedtosis keeps complete sets of entry digests
+for previous feed retrievals.  The number of digest sets that will be kept is configurable by
+setting the option :retained_digest_size on Feedtosis client initialization.
+
 == HTML cleaning/sanitizing
 
 Feedtosis doesn't do anything about feed sanitizing, as other libraries have
@@ -132,10 +135,6 @@ development of Feedtosis.
 Please let me know if you have any problems with or questions about
 Feedtosis.
 
-= References
-
-  (1) http://en.wikipedia.org/wiki/List_of_vores
-
 = Author
 
 Justin S. Leitgeb, mailto:justin@phq.org

data/feedtosis.gemspec

@@ -1,6 +1,6 @@
 Gem::Specification.new do |s|
   s.name = %q{feedtosis}
-  s.version = "0.0.3"
+  s.version = "0.0.3.1"
 
   s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
   s.authors = ["Justin Leitgeb"]
@@ -38,7 +38,7 @@ Gem::Specification.new do |s|
     '--inline-source'
    ]
 
-  %w[ taf2-curb jsl-moneta jsl-http_headers namelessjon-feed_me ].each do |dep|
+  %w[ taf2-curb jsl-moneta jsl-http_headers feed-normalizer ].each do |dep|
     s.add_dependency(dep)
   end
 

data/lib/extensions/core/hash.rb

@@ -1,17 +1,7 @@
 class Hash
-  # Returns a Hash containing only input keys.
-  # Method from merb-core.
-  def except(*rejected)
-    reject { |k,v| rejected.include?(k) }
-  end
   
   def reverse_merge(other_hash)
     other_hash.merge(self)
   end
-
-  # Returns a new hash containing only the input keys.
-  # Method from merb-core.
-  def only(*allowed)
-    reject { |k,v| !allowed.include?(k) }
-  end
+  
 end

data/lib/feedtosis.rb

@@ -1,10 +1,10 @@
 require 'rubygems'
+
 require 'curb'
-require 'moneta'
-require 'moneta/memory'
 require 'http_headers'
 require 'feed-normalizer'
 require 'md5'
+require 'uri'
 
 lib_dirs =  [ 'extensions', 'feedtosis' ].map do |d|
   File.join(File.dirname(__FILE__), d)

data/lib/feedtosis/client.rb

@@ -7,14 +7,37 @@ module Feedtosis
   # either new or not new.  Entries retrieved are normalized using the 
   # feed-normalizer gem.
   class Client
-    attr_reader :options, :url
+    attr_reader :url, :options, :backend
+    
+    DEFAULTS = {
+      :backend => Hash.new,
+      
+      # The namespace will be prefixed to the key used for storage of the summary value.  Based on your
+      # application needs, it may be useful to provide a custom prefix with initialization options.
+      :namespace => 'feedtosis',
+      
+      # Some feed aggregators that we may be pulling from have entries that are present in one fetch and 
+      # then disappear (Google blog search does this).  For these cases, we can't rely on only the digests of 
+      # the last fetch to guarantee "newness" of a feed that we may have previously consumed.  We keep a 
+      # number of previous sets of digests in order to make sure that we mark correct feeds as "new".
+      :retained_digest_size => 10
+    } unless defined?(DEFAULTS)
 
-    # Initializes a new feedtosis library.  The backend can be a hash of options, in 
-    # which case we initialize a new HashBack::Backend.  Or, it may be a pre-initialized
-    # backend, in which case we set the backend to the given HashBack::Backend object.
-    def initialize(url, backend = Moneta::Memory.new)
-      @url      = url
-      @backend  = backend
+    # Initializes a new feedtosis library.  It must be initialized with a valid URL as the first argument.
+    # A following Hash, if given, may have the following keys:
+    #   * backend: a key-value store to be used for summary structures of feeds fetched.  Moneta backends work well, but any object acting like a Hash is valid.
+    #   * retained_digest_size: an Integer specifying the number of previous MD5 sets of entries to keep, used for new feed detection    
+    def initialize(*args)
+      @url      = args.first
+      
+      @options  = args.extract_options!
+      @options  = @options.reverse_merge(DEFAULTS)
+
+      @backend  = @options[:backend]
+      
+      unless @url.match(URI.regexp('http'))
+        raise ArgumentError, "Url #{@url} is not valid!"
+      end
       
       unless @backend.respond_to?(:[]) && @backend.respond_to?(:[]=)
         raise ArgumentError, "Backend needs to be a key-value store"
@@ -37,12 +60,8 @@ module Feedtosis
     # Marks entries as either seen or not seen based on the unique signature of 
     # the entry, which is calculated by taking the MD5 of common attributes.
     def mark_new_entries(response)
-      digests = if summary_for_feed.nil? || summary_for_feed[:digests].nil?
-        [ ]
-      else
-        summary_for_feed[:digests]
-      end
-            
+      digests = summary_digests
+
       # For each entry in the responses object, mark @_seen as false if the 
       # digest of this entry doesn't exist in the cached object.
       response.entries.each do |e|
@@ -53,6 +72,14 @@ module Feedtosis
       response
     end
 
+    # Returns an Array of summary digests for this feed.  Since we keep a number of sets
+    # of digests, inject across these sets to accumulate unique identifiers.
+    def summary_digests
+      summary_for_feed[:digests].inject([]) do |r, e|
+        r |= e
+      end.uniq
+    end
+
     # Processes the results by identifying which entries are new if the response
     # is a 200.  Otherwise, returns the Curl::Easy object for the user to inspect.
     def process_curl_response(curl)
@@ -82,7 +109,7 @@ module Feedtosis
 
     # Returns the summary hash for this feed from the backend store.
     def summary_for_feed
-      @backend[key_for_cached]
+      @backend[key_for_cached] || { :digests => [ ] }
     end
 
     # Sets the headers from the backend, if available
@@ -97,8 +124,9 @@ module Feedtosis
       curl
     end
 
+    # Returns the key for the storage of the summary structure in the key-value system.
     def key_for_cached
-      MD5.hexdigest(@url)
+      [ @options[:namespace], MD5.hexdigest(@url) ].join('_')
     end
     
     # Stores information about the retrieval, including ETag, Last-Modified, 
@@ -116,11 +144,14 @@ module Feedtosis
       
       # Store digest for each feed entry so we can detect new feeds on the next 
       # retrieval
-      digests = feed.entries.map do |e|
+      new_digest_set = feed.entries.map do |e|
         digest_for(e)
       end
       
-      summary.merge!(:digests => digests)
+      new_digest_set = summary_for_feed[:digests].unshift(new_digest_set)
+      new_digest_set = new_digest_set[0..@options[:retained_digest_size]]
+      
+      summary.merge!( :digests => new_digest_set )
       set_summary(summary)
     end
     
@@ -132,7 +163,7 @@ module Feedtosis
     # This signature will be the MD5 of enough fields to have a reasonable 
     # probability of determining if the entry is unique or not.
     def digest_for(entry)      
-      MD5.hexdigest( [ entry.title, entry.content ].join )
+      MD5.hexdigest( [ entry.title, entry.content, entry.date_published ].join )
     end
     
     def parser_for_xml(xml)

data/spec/feedtosis/client_spec.rb

@@ -3,8 +3,8 @@ require File.join(File.dirname(__FILE__), %w[.. spec_helper])
 describe Feedtosis::Client do
   before do
     @url      = "http://www.example.com/feed.rss"
-    @backend  = Moneta::Memory.new
-    @fr       = Feedtosis::Client.new(@url, @backend)
+    @backend  = Hash.new
+    @fr       = Feedtosis::Client.new(@url, :backend => @backend)
   end
 
   describe "initialization" do    
@@ -12,12 +12,26 @@ describe Feedtosis::Client do
       @fr.url.should == @url
     end
     
+    describe "validation of url in first argument" do
+      it "should not raise an error on initialization with a valid HTTP url" do
+        lambda {
+          Feedtosis::Client.new('http://www.example.com')
+        }.should_not raise_error
+      end
+      
+      it "should raise an error on initialization with an invalid url" do
+        lambda {
+          Feedtosis::Client.new('ftp://www.example.com')
+        }.should raise_error(ArgumentError)
+      end
+    end
+    
     it "should set the If-None-Match and If-Modified-Since headers to the value of the summary hash" do
       curl_headers = mock('headers')
       curl_headers.expects(:[]=).with('If-None-Match', '42ab')
       curl_headers.expects(:[]=).with('If-Modified-Since', 'Mon, 25 May 2009 16:38:49 GMT')
       
-      summary = { :etag => '42ab', :last_modified => 'Mon, 25 May 2009 16:38:49 GMT' }
+      summary = { :etag => '42ab', :last_modified => 'Mon, 25 May 2009 16:38:49 GMT', :digests => [ ] }
       
       @fr.__send__(:set_summary, summary)
       
@@ -32,10 +46,17 @@ describe Feedtosis::Client do
       @fr.fetch
     end
     
+    describe "#summary_for_feed" do
+      it "should return a hash with :digests set to an empty Array when summary is nil" do
+        @fr.__send__(:set_summary, nil)
+        @fr.__send__(:summary_for_feed).should == {:digests => [ ]}
+      end
+    end
+    
     describe "when given a pre-initialized backend" do
       it "should set the @backend to the pre-initialized structure" do
         h   = Moneta::Memory.new
-        fc  = Feedtosis::Client.new(@url, h)
+        fc  = Feedtosis::Client.new(@url, :backend => h)
         fc.__send__(:instance_variable_get, :@backend).should == h
       end
       
@@ -43,12 +64,24 @@ describe Feedtosis::Client do
         o = Object.new
         
         lambda {
-          Feedtosis::Client.new(@url, o)          
+          Feedtosis::Client.new(@url, :backend => o)          
         }.should raise_error(ArgumentError)
       end
     end
   end
     
+  describe "#key_for_cached" do
+    it "should default to the MD5 of the url after the namespace" do
+      c = Feedtosis::Client.new(@url)
+      c.__send__(:key_for_cached).should == [ 'feedtosis', MD5.hexdigest(@url) ].join('_')
+    end
+    
+    it "should respect a custom namespace if given" do
+      c = Feedtosis::Client.new(@url, :namespace => 'justin')
+      c.__send__(:key_for_cached).should == [ 'justin', MD5.hexdigest(@url) ].join('_')
+    end
+  end
+    
   describe "#fetch" do
     it "should call Curl::Easy.perform with the url, and #process_curl_response" do
       curl_easy = mock('curl', :perform => true)
@@ -86,7 +119,7 @@ describe Feedtosis::Client do
           curl = mock('curl', :perform => true, :response_code => 200,
             :body_str => xml_fixture('wooster'), :header_str => http_header('wooster'))
           @fr.expects(:build_curl_easy).returns(curl)
-          @fr.fetch.new_entries.should == []          
+          @fr.fetch.new_entries.should == []
         end
       end
       

data/spec/spec_helper.rb

@@ -2,6 +2,9 @@ require 'rubygems'
 require 'mocha'
 require 'spec'
 
+require 'moneta'
+require 'moneta/memory'
+
 require File.join(File.dirname(__FILE__), %w[.. lib feedtosis])
 
 Spec::Runner.configure do |config|

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification 
 name: jsl-feedtosis
 version: !ruby/object:Gem::Version 
-  version: 0.0.3
+  version: 0.0.3.1
 platform: ruby
 authors: 
 - Justin Leitgeb
@@ -43,7 +43,7 @@ dependencies:
         version: "0"
     version: 
 - !ruby/object:Gem::Dependency 
-  name: namelessjon-feed_me
+  name: feed-normalizer
   type: :runtime
   version_requirement: 
   version_requirements: !ruby/object:Gem::Requirement