feed-normalizer 1.5.1 → 1.5.2

data/History.txt

@@ -2,51 +2,51 @@
 
  * Fix a bug that was breaking the parsing process for certain feeds. [reported by: Patrick Minton]
 
-1.5.0
-
- * Add support for new fields:
-   * Atom 0.3: issued is now available through entry.date_published.
-   * RSS: feed.skip_hours, feed.skip_days, feed.ttl [joshpeek]
-   * All: entry.last_updated, this is an alias to entry.date_published for RSS.
- * Rewrite relative links in content [joshpeek]
- * Handle CDATA sections consistently across all formats. [sam.lown]
- * Prevent SimpleRSS from doing its own escaping. [reported by: paul.stadig, lionel.bouton]
- * Reparse Time classes [reported by: sam.lown]
-
-1.4.0
-
- * Support content:encoded. Accessible via Entry#content.
- * Support categories. Accessible via Entry#categories.
- * Introduces a new parsing feature 'loose parsing'. Use :loose => true
-   when parsing if the required output should retain extra data, rather
-   than drop it in the interests of 'lowest common denomiator' normalization.
-   Currently affects how categories works. See the documentation in
-   FeedNormalizer#parse for more details.
-
-1.3.2
-
- * Add support for applicable dublin core elements. (dc:date and dc:creator)
- * Feeds can now be dumped to YAML.
-
-1.3.1
-
- * Small changes to work with hpricot 0.6. This release depends on hpricot 0.6.
- * Reduced the greediness of a regexp that was removing html comments.
-
-1.3.0
-
- * Small changes to work with hpricot 0.5.
-
-1.2.0
-
- * Added HtmlCleaner - sanitizes HTML and removes 'bad' URIs to a level suitable
-   for 'safe' display inside a web browser. Can be used as a standalone library,
-   or as part of the Feed object. See Feed.clean! for details about cleaning a
-   Feed instance. Also see HtmlCleaner and its unit tests. Uses Hpricot.
- * Added Feed-diffing. Differences between two feeds can be displayed using
-   Feed.diff. Works nicely with YAML for a readable diff.
- * FeedNormalizer.parse now takes a hash for its arguments.
- * Removed FN::Content.
- * Now uses Hoe!
-
-
+1.5.0
+
+ * Add support for new fields:
+   * Atom 0.3: issued is now available through entry.date_published.
+   * RSS: feed.skip_hours, feed.skip_days, feed.ttl [joshpeek]
+   * All: entry.last_updated, this is an alias to entry.date_published for RSS.
+ * Rewrite relative links in content [joshpeek]
+ * Handle CDATA sections consistently across all formats. [sam.lown]
+ * Prevent SimpleRSS from doing its own escaping. [reported by: paul.stadig, lionel.bouton]
+ * Reparse Time classes [reported by: sam.lown]
+
+1.4.0
+
+ * Support content:encoded. Accessible via Entry#content.
+ * Support categories. Accessible via Entry#categories.
+ * Introduces a new parsing feature 'loose parsing'. Use :loose => true
+   when parsing if the required output should retain extra data, rather
+   than drop it in the interests of 'lowest common denomiator' normalization.
+   Currently affects how categories works. See the documentation in
+   FeedNormalizer#parse for more details.
+
+1.3.2
+
+ * Add support for applicable dublin core elements. (dc:date and dc:creator)
+ * Feeds can now be dumped to YAML.
+
+1.3.1
+
+ * Small changes to work with hpricot 0.6. This release depends on hpricot 0.6.
+ * Reduced the greediness of a regexp that was removing html comments.
+
+1.3.0
+
+ * Small changes to work with hpricot 0.5.
+
+1.2.0
+
+ * Added HtmlCleaner - sanitizes HTML and removes 'bad' URIs to a level suitable
+   for 'safe' display inside a web browser. Can be used as a standalone library,
+   or as part of the Feed object. See Feed.clean! for details about cleaning a
+   Feed instance. Also see HtmlCleaner and its unit tests. Uses Hpricot.
+ * Added Feed-diffing. Differences between two feeds can be displayed using
+   Feed.diff. Works nicely with YAML for a readable diff.
+ * FeedNormalizer.parse now takes a hash for its arguments.
+ * Removed FN::Content.
+ * Now uses Hoe!
+
+

data/License.txt

@@ -1,27 +1,27 @@
-Copyright (c) 2006-2007, Andrew A. Smith
-All rights reserved.
-
-Redistribution and use in source and binary forms, with or without modification,
-are permitted provided that the following conditions are met:
-
-    * Redistributions of source code must retain the above copyright notice,
-      this list of conditions and the following disclaimer.
-
-    * Redistributions in binary form must reproduce the above copyright notice,
-      this list of conditions and the following disclaimer in the documentation
-      and/or other materials provided with the distribution.
-
-    * Neither the name of the copyright owner nor the names of its contributors
-      may be used to endorse or promote products derived from this software
-      without specific prior written permission.
-
-THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
-ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
-WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
-DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR
-ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
-(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
-LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
-ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
-SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+Copyright (c) 2006-2007, Andrew A. Smith
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without modification,
+are permitted provided that the following conditions are met:
+
+    * Redistributions of source code must retain the above copyright notice,
+      this list of conditions and the following disclaimer.
+
+    * Redistributions in binary form must reproduce the above copyright notice,
+      this list of conditions and the following disclaimer in the documentation
+      and/or other materials provided with the distribution.
+
+    * Neither the name of the copyright owner nor the names of its contributors
+      may be used to endorse or promote products derived from this software
+      without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
+ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR
+ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
+(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
+ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

data/Manifest.txt

@@ -1,19 +1,18 @@
-History.txt
-License.txt
-Manifest.txt
-Rakefile
-README.txt
-lib/feed-normalizer.rb
-lib/html-cleaner.rb
-lib/parsers/rss.rb
-lib/parsers/simple-rss.rb
-lib/structures.rb
-test/data/atom03.xml
-test/data/atom10.xml
-test/data/rdf10.xml
-test/data/rss20.xml
-test/data/rss20diff.xml
-test/data/rss20diff_short.xml
-test/test_all.rb
-test/test_feednormalizer.rb
-test/test_htmlcleaner.rb
+History.txt
+License.txt
+Manifest.txt
+Rakefile
+README.txt
+lib/feed-normalizer.rb
+lib/html-cleaner.rb
+lib/parsers/rss.rb
+lib/parsers/simple-rss.rb
+lib/structures.rb
+test/data/atom03.xml
+test/data/atom10.xml
+test/data/rdf10.xml
+test/data/rss20.xml
+test/data/rss20diff.xml
+test/data/rss20diff_short.xml
+test/test_feednormalizer.rb
+test/test_htmlcleaner.rb

data/README.txt

@@ -1,63 +1,63 @@
-== Feed Normalizer
-
-An extensible Ruby wrapper for Atom and RSS parsers.
-
-Feed normalizer wraps various RSS and Atom parsers, and returns a single unified
-object graph, regardless of the underlying feed format.
-
-== Download
-
-* gem install feed-normalizer
-* http://rubyforge.org/projects/feed-normalizer
-* svn co http://feed-normalizer.googlecode.com/svn/trunk
-
-== Usage
-
-    require 'feed-normalizer'
-    require 'open-uri'
-
-    feed = FeedNormalizer::FeedNormalizer.parse open('http://www.iht.com/rss/frontpage.xml')
-
-    feed.title # => "International Herald Tribune"
-    feed.url # => "http://www.iht.com/pages/index.php"
-    feed.entries.first.url # => "http://www.iht.com/articles/2006/10/03/frontpage/web.1003UN.php"
-
-    feed.class # => FeedNormalizer::Feed
-    feed.parser # => "RSS::Parser"
-
-Now read an Atom feed, and the same class is returned, and the same terminology applies:
-
-    feed = FeedNormalizer::FeedNormalizer.parse open('http://www.atomenabled.org/atom.xml')
-
-    feed.title # => "AtomEnabled.org"
-    feed.url # => "http://www.atomenabled.org/atom.xml"
-    feed.entries.first.url # => "http://www.atomenabled.org/2006/09/moving-toward-atom.php"
-
-The feed representation stays the same, even though a different parser was used.
-
-    feed.class # => FeedNormalizer::Feed
-    feed.parser # => "SimpleRSS"
-
-== Cleaning / Sanitizing
-
-    feed.title # => "My Feed > Your Feed"
-    feed.entries.first.content # => "<p x='y'>Hello</p><object></object></html>"
-    feed.clean!
-
-All elements should now be either clean HTML, or HTML escaped strings.
-
-    feed.title # => "My Feed &gt; Your Feed"
-    feed.entries.first.content # => "<p>Hello</p>"
-
-== Extending
-
-Implement a parser wrapper by extending the FeedNormalizer::Parser class and overriding
-the public methods. Also note the helper methods in the root Parser object to make
-mapping of output from the particular parser to the Feed object easier.
-
-See FeedNormalizer::RubyRssParser and FeedNormalizer::SimpleRssParser for examples.
-
-== Authors
-* Andrew A. Smith (andy@tinnedfruit.org)
-
-This library is released under the terms of the BSD License (see the License.txt file for details).
+== Feed Normalizer
+
+An extensible Ruby wrapper for Atom and RSS parsers.
+
+Feed normalizer wraps various RSS and Atom parsers, and returns a single unified
+object graph, regardless of the underlying feed format.
+
+== Download
+
+* gem install feed-normalizer
+* http://rubyforge.org/projects/feed-normalizer
+* svn co http://feed-normalizer.googlecode.com/svn/trunk
+
+== Usage
+
+    require 'feed-normalizer'
+    require 'open-uri'
+
+    feed = FeedNormalizer::FeedNormalizer.parse open('http://www.iht.com/rss/frontpage.xml')
+
+    feed.title # => "International Herald Tribune"
+    feed.url # => "http://www.iht.com/pages/index.php"
+    feed.entries.first.url # => "http://www.iht.com/articles/2006/10/03/frontpage/web.1003UN.php"
+
+    feed.class # => FeedNormalizer::Feed
+    feed.parser # => "RSS::Parser"
+
+Now read an Atom feed, and the same class is returned, and the same terminology applies:
+
+    feed = FeedNormalizer::FeedNormalizer.parse open('http://www.atomenabled.org/atom.xml')
+
+    feed.title # => "AtomEnabled.org"
+    feed.url # => "http://www.atomenabled.org/atom.xml"
+    feed.entries.first.url # => "http://www.atomenabled.org/2006/09/moving-toward-atom.php"
+
+The feed representation stays the same, even though a different parser was used.
+
+    feed.class # => FeedNormalizer::Feed
+    feed.parser # => "SimpleRSS"
+
+== Cleaning / Sanitizing
+
+    feed.title # => "My Feed > Your Feed"
+    feed.entries.first.content # => "<p x='y'>Hello</p><object></object></html>"
+    feed.clean!
+
+All elements should now be either clean HTML, or HTML escaped strings.
+
+    feed.title # => "My Feed &gt; Your Feed"
+    feed.entries.first.content # => "<p>Hello</p>"
+
+== Extending
+
+Implement a parser wrapper by extending the FeedNormalizer::Parser class and overriding
+the public methods. Also note the helper methods in the root Parser object to make
+mapping of output from the particular parser to the Feed object easier.
+
+See FeedNormalizer::RubyRssParser and FeedNormalizer::SimpleRssParser for examples.
+
+== Authors
+* Andrew A. Smith (andy@tinnedfruit.org)
+
+This library is released under the terms of the BSD License (see the License.txt file for details).

data/Rakefile

@@ -1,25 +1,29 @@
-require 'hoe'
-
-Hoe.new("feed-normalizer", "1.5.1") do |s|
-  s.author = "Andrew A. Smith"
-  s.email = "andy@tinnedfruit.org"
-  s.url = "http://feed-normalizer.rubyforge.org/"
-  s.summary = "Extensible Ruby wrapper for Atom and RSS parsers"
-  s.description = s.paragraphs_of('README.txt', 1..2).join("\n\n")
-  s.changes = s.paragraphs_of('History.txt', 0..1).join("\n\n")
-  s.extra_deps << ["simple-rss", ">= 1.1"]
-  s.extra_deps << ["hpricot", ">= 0.6"]
-  s.need_zip = true
-  s.need_tar = false
-end
-
-
-begin
-  require 'rcov/rcovtask'
-  Rcov::RcovTask.new("rcov") do |t|
-    t.test_files = Dir['test/test_all.rb']
-  end
-rescue LoadError
-  nil
-end
-
+require 'hoe'
+
+$: << "lib"
+require 'feed-normalizer'
+
+Hoe.spec("feed-normalizer") do |s|
+  s.version = "1.5.2"
+  s.author = "Andrew A. Smith"
+  s.email = "andy@tinnedfruit.org"
+  s.url = "http://github.com/aasmith/feed-normalizer"
+  s.summary = "Extensible Ruby wrapper for Atom and RSS parsers"
+  s.description = s.paragraphs_of('README.txt', 1..2).join("\n\n")
+  s.changes = s.paragraphs_of('History.txt', 0..1).join("\n\n")
+  s.extra_deps << ["simple-rss", ">= 1.1"]
+  s.extra_deps << ["hpricot", ">= 0.6"]
+  s.need_zip = true
+  s.need_tar = false
+end
+
+
+begin
+  require 'rcov/rcovtask'
+  Rcov::RcovTask.new("rcov") do |t|
+    t.test_files = Dir['test/test_all.rb']
+  end
+rescue LoadError
+  nil
+end
+

data/lib/feed-normalizer.rb

@@ -1,149 +1,149 @@
-require 'structures'
-require 'html-cleaner'
-
-module FeedNormalizer
-
-  # The root parser object. Every parser must extend this object.
-  class Parser
-
-    # Parser being used.
-    def self.parser
-      nil
-    end
-
-    # Parses the given feed, and returns a normalized representation.
-    # Returns nil if the feed could not be parsed.
-    def self.parse(feed, loose)
-      nil
-    end
-
-    # Returns a number to indicate parser priority.
-    # The lower the number, the more likely the parser will be used first,
-    # and vice-versa.
-    def self.priority
-      0
-    end
-
-    protected
-
-    # Some utility methods that can be used by subclasses.
-
-    # sets value, or appends to an existing value
-    def self.map_functions!(mapping, src, dest)
-
-      mapping.each do |dest_function, src_functions|
-        src_functions = [src_functions].flatten # pack into array
-
-        src_functions.each do |src_function|
-          value = if src.respond_to?(src_function)
-            src.send(src_function)
-          elsif src.respond_to?(:has_key?)
-            src[src_function]
-          end
-
-          unless value.to_s.empty?
-            append_or_set!(value, dest, dest_function)
-            break
-          end
-        end
-
-      end
-    end
-
-    def self.append_or_set!(value, object, object_function)
-      if object.send(object_function).respond_to? :push
-        object.send(object_function).push(value)
-      else
-        object.send(:"#{object_function}=", value)
-      end
-    end
-
-    private
-
-    # Callback that ensures that every parser gets registered.
-    def self.inherited(subclass)
-      ParserRegistry.register(subclass)
-    end
-
-  end
-
-
-  # The parser registry keeps a list of current parsers that are available.
-  class ParserRegistry
-
-    @@parsers = []
-
-    def self.register(parser)
-      @@parsers << parser
-    end
-
-    # Returns a list of currently registered parsers, in order of priority.
-    def self.parsers
-      @@parsers.sort_by { |parser| parser.priority }
-    end
-
-  end
-
-
-  class FeedNormalizer
-
-    # Parses the given xml and attempts to return a normalized Feed object.
-    # Setting +force_parser+ to a suitable parser will mean that parser is
-    # used first, and if +try_others+ is false, it is the only parser used,
-    # otherwise all parsers in the ParserRegistry are attempted, in
-    # order of priority.
-    #
-    # ===Available options
-    #
-    # * <tt>:force_parser</tt> - instruct feed-normalizer to try the specified
-    #   parser first. Takes a class, such as RubyRssParser, or SimpleRssParser.
-    #
-    # * <tt>:try_others</tt> - +true+ or +false+, defaults to +true+.
-    #   If +true+, other parsers will be used as described above. The option
-    #   is useful if combined with +force_parser+ to only use a single parser.
-    #
-    # * <tt>:loose</tt> - +true+ or +false+, defaults to +false+.
-    #
-    #   Specifies parsing should be done loosely. This means that when
-    #   feed-normalizer would usually throw away data in order to meet
-    #   the requirement of keeping resulting feed outputs the same regardless
-    #   of the underlying parser, the data will instead be kept. This currently
-    #   affects the following items:
-    #   * <em>Categories:</em> RSS allows for multiple categories per feed item.
-    #     * <em>Limitation:</em> SimpleRSS can only return the first category
-    #       for an item.
-    #     * <em>Result:</em> When loose is true, the extra categories are kept,
-    #       of course, only if the parser is not SimpleRSS.
-    def self.parse(xml, opts = {})
-
-      # Get a string ASAP, as multiple read()'s will start returning nil..
-      xml = xml.respond_to?(:read) ? xml.read : xml.to_s
-
-      if opts[:force_parser]
-        result = opts[:force_parser].parse(xml, opts[:loose])
-
-        return result if result
-        return nil if opts[:try_others] == false
-      end
-
-      ParserRegistry.parsers.each do |parser|
-        result = parser.parse(xml, opts[:loose])
-        return result if result
-      end
-
-      # if we got here, no parsers worked.
-      return nil
-    end
-  end
-
-
-  parser_dir = File.dirname(__FILE__) + '/parsers'
-
-  # Load up the parsers
-  Dir.open(parser_dir).each do |fn|
-    next unless fn =~ /[.]rb$/
-    require "parsers/#{fn}"
-  end
-
-end
-
+require 'structures'
+require 'html-cleaner'
+
+module FeedNormalizer
+
+  # The root parser object. Every parser must extend this object.
+  class Parser
+
+    # Parser being used.
+    def self.parser
+      nil
+    end
+
+    # Parses the given feed, and returns a normalized representation.
+    # Returns nil if the feed could not be parsed.
+    def self.parse(feed, loose)
+      nil
+    end
+
+    # Returns a number to indicate parser priority.
+    # The lower the number, the more likely the parser will be used first,
+    # and vice-versa.
+    def self.priority
+      0
+    end
+
+    protected
+
+    # Some utility methods that can be used by subclasses.
+
+    # sets value, or appends to an existing value
+    def self.map_functions!(mapping, src, dest)
+
+      mapping.each do |dest_function, src_functions|
+        src_functions = [src_functions].flatten # pack into array
+
+        src_functions.each do |src_function|
+          value = if src.respond_to?(src_function)
+            src.send(src_function)
+          elsif src.respond_to?(:has_key?)
+            src[src_function]
+          end
+
+          unless value.to_s.empty?
+            append_or_set!(value, dest, dest_function)
+            break
+          end
+        end
+
+      end
+    end
+
+    def self.append_or_set!(value, object, object_function)
+      if object.send(object_function).respond_to? :push
+        object.send(object_function).push(value)
+      else
+        object.send(:"#{object_function}=", value)
+      end
+    end
+
+    private
+
+    # Callback that ensures that every parser gets registered.
+    def self.inherited(subclass)
+      ParserRegistry.register(subclass)
+    end
+
+  end
+
+
+  # The parser registry keeps a list of current parsers that are available.
+  class ParserRegistry
+
+    @@parsers = []
+
+    def self.register(parser)
+      @@parsers << parser
+    end
+
+    # Returns a list of currently registered parsers, in order of priority.
+    def self.parsers
+      @@parsers.sort_by { |parser| parser.priority }
+    end
+
+  end
+
+
+  class FeedNormalizer
+
+    # Parses the given xml and attempts to return a normalized Feed object.
+    # Setting +force_parser+ to a suitable parser will mean that parser is
+    # used first, and if +try_others+ is false, it is the only parser used,
+    # otherwise all parsers in the ParserRegistry are attempted, in
+    # order of priority.
+    #
+    # ===Available options
+    #
+    # * <tt>:force_parser</tt> - instruct feed-normalizer to try the specified
+    #   parser first. Takes a class, such as RubyRssParser, or SimpleRssParser.
+    #
+    # * <tt>:try_others</tt> - +true+ or +false+, defaults to +true+.
+    #   If +true+, other parsers will be used as described above. The option
+    #   is useful if combined with +force_parser+ to only use a single parser.
+    #
+    # * <tt>:loose</tt> - +true+ or +false+, defaults to +false+.
+    #
+    #   Specifies parsing should be done loosely. This means that when
+    #   feed-normalizer would usually throw away data in order to meet
+    #   the requirement of keeping resulting feed outputs the same regardless
+    #   of the underlying parser, the data will instead be kept. This currently
+    #   affects the following items:
+    #   * <em>Categories:</em> RSS allows for multiple categories per feed item.
+    #     * <em>Limitation:</em> SimpleRSS can only return the first category
+    #       for an item.
+    #     * <em>Result:</em> When loose is true, the extra categories are kept,
+    #       of course, only if the parser is not SimpleRSS.
+    def self.parse(xml, opts = {})
+
+      # Get a string ASAP, as multiple read()'s will start returning nil..
+      xml = xml.respond_to?(:read) ? xml.read : xml.to_s
+
+      if opts[:force_parser]
+        result = opts[:force_parser].parse(xml, opts[:loose])
+
+        return result if result
+        return nil if opts[:try_others] == false
+      end
+
+      ParserRegistry.parsers.each do |parser|
+        result = parser.parse(xml, opts[:loose])
+        return result if result
+      end
+
+      # if we got here, no parsers worked.
+      return nil
+    end
+  end
+
+
+  parser_dir = File.dirname(__FILE__) + '/parsers'
+
+  # Load up the parsers
+  Dir.open(parser_dir).each do |fn|
+    next unless fn =~ /[.]rb$/
+    require "parsers/#{fn}"
+  end
+
+end
+