penso-feedbag 0.5.100 → 0.6.1

data/.gitignore

@@ -0,0 +1,30 @@
+*.swp
+**/*.pid
+log/*.log
+log/*.pid
+tmp
+.DS_Store
+public/cache/**/*
+public/system/**/*
+doc/**/*
+db/*.sqlite3
+.project
+.loadpath
+nbproject/
+.idea
+testjour.log
+*.so
+*.o
+Makefile
+mkmf.log
+*.bundle
+conftest
+content/
+.idea
+*.sw?
+.DS_Store
+coverage
+rdoc
+pkg
+pkg/*
+log/*

data/ChangeLog

@@ -0,0 +1,20 @@
+* 0.5.99 - Tue May 12 12:52:22 EDT 2009
+  - Added rails/init.rb to load easily on a Rails app.
+
+* 0.5.13.1 - Wed Apr 22 11:16:19 EDT 2009
+  - Changed args on find() from nil to {}
+
+* 0.5.13 - Wed Apr 22 11:12:40 EDT 2009
+  - Added :narrow option so find() skips feed_validate and A links.
+
+* 0.5.12 - Fri Mar 20 12:34:48 EDT 2009
+	- Added support for "feed://" URLs
+
+* 0.5.11 - Sat Mar  7 17:22:30 EST 2009
+	- Benchmark against Rfeedfinder added.
+
+* 0.5.10 - Wed Mar  4 13:32:33 EST 2009
+  - Feeds whose URLs contained query string arguments were not being
+    auto-discovered -- fixed
+
+** For previous changes, see the git log

data/README.markdown

@@ -1,7 +1,6 @@
 Feedbag
 =======
-> Do you want me to drag my sack across your face?
->                                 - Glenn Quagmire
+Forked version of feedback that returns title and url.
 
 Feedbag is a feed auto-discovery Ruby library. You don't need to know more about it. It is said to be:
 

data/TODO

@@ -0,0 +1 @@
+- Document Feedbag.feed?

data/VERSION

@@ -0,0 +1 @@
+0.6.0

data/feedbag.gemspec

@@ -0,0 +1,20 @@
+# -*- encoding: utf-8 -*-
+ 
+Gem::Specification.new do |s|
+  s.name = %q{feedbag}
+  s.version = "0.5.103"
+  s.homepage = "http://axiombox.com/feedbag"
+  #s.rubyforge_project = "feedbag"
+ 
+  s.authors = ["Axiombox", "David Moreno"]
+  s.date = %q{2009-02-10}
+  s.description = %q{Ruby's favorite feed auto-discoverty tool}
+  s.email = %q{david@axiombox.com}
+  s.extra_rdoc_files = ["README.markdown", "COPYING"]
+  s.files = ["lib/feedbag.rb", "benchmark/rfeedfinder_benchmark.rb"]
+  s.has_rdoc = true
+  s.rdoc_options = ["--main", "README.markdown"]
+  s.summary = %q{Ruby's favorite feed auto-discovery tool}
+	s.add_dependency("hpricot", '>= 0.6') 
+end
+

data/index.html

@@ -0,0 +1,115 @@
+<h1>Feedbag</h1>
+
+<blockquote>
+  <p>Do you want me to drag my sack across your face?
+                                - Glenn Quagmire</p>
+</blockquote>
+
+<p>Feedbag is a feed auto-discovery Ruby library. You don't need to know more about it. It is said to be:</p>
+
+<blockquote>
+  <p>Ruby's favorite auto-discovery tool/library!</p>
+</blockquote>
+
+<h3>Quick synopsis</h3>
+
+<pre><code>&gt;&gt; require "rubygems"
+=&gt; true
+&gt;&gt; require "feedbag"
+=&gt; true
+&gt;&gt; Feedbag.find "log.damog.net"
+=&gt; ["http://feeds.feedburner.com/TeoremaDelCerdoInfinito", "http://log.damog.net/comments/feed/"]
+</code></pre>
+
+<h3>Installation</h3>
+
+<pre><code>$ sudo gem install damog-feedbag -s http://gems.github.com/
+</code></pre>
+
+<p>Or just grab feedbag.rb and use it on your own project:</p>
+
+<pre><code>$ wget http://github.com/damog/feedbag/raw/master/lib/feedbag.rb
+</code></pre>
+
+<h2>Tutorial</h2>
+
+<p>So you want to know more about it.</p>
+
+<p>OK, if the URL passed to the find method is a feed itself, that only feed URL will be returned.</p>
+
+<pre><code>&gt;&gt; Feedbag.find "github.com/damog.atom"
+=&gt; ["http://github.com/damog.atom"]
+&gt;&gt;
+</code></pre>
+
+<p>Otherwise, it will always return LINK feeds first, A (anchor tags) feeds later. Between A feeds, the ones hosted on the same URL's host, will have larger priority:</p>
+
+<pre><code>&gt;&gt; Feedbag.find "http://ve.planetalinux.org"
+=&gt; ["http://feedproxy.google.com/PlanetaLinuxVenezuela", "http://rendergraf.wordpress.com/feed/", "http://rootweiller.wordpress.com/feed/", "http://skatox.com/blog/feed/", "http://kodegeek.com/atom.xml", "http://blog.0x29.com.ve/?feed=rss2&amp;cat=8"]
+&gt;&gt;
+</code></pre>
+
+<p>On your application you should only take the very first element of the array, most of the times:</p>
+
+<pre><code>&gt;&gt; Feedbag.find("planet.debian.org").first(3)
+=&gt; ["http://planet.debian.org/rss10.xml", "http://planet.debian.org/rss20.xml", "http://planet.debian.org/atom.xml"]
+&gt;&gt;
+</code></pre>
+
+<p>(Try running that same example without the "first" method. That example's host is a blog aggregator, so it has hundreds of feed URLs:)</p>
+
+<pre><code>&gt;&gt; Feedbag.find("planet.debian.org").size
+=&gt; 104
+&gt;&gt;
+</code></pre>
+
+<p>Feedbag will find them all, but it will return the most important ones on the first elements on the array returned.</p>
+
+<pre><code>&gt;&gt; Feedbag.find("cnn.com")
+=&gt; ["http://rss.cnn.com/rss/cnn_topstories.rss", "http://rss.cnn.com/rss/cnn_latest.rss", "http://rss.cnn.com/services/podcasting/robinmeade/rss.xml"]
+&gt;&gt;
+</code></pre>
+
+<h3>Why should you use it?</h3>
+
+<ul>
+<li>Because it's cool.</li>
+<li>Because it only uses <a href="https://code.whytheluckystiff.net/hpricot/">Hpricot</a> as dependency.</li>
+<li>Because it follows modern feed filename conventions (like those ones used by WordPress blogs, or Blogger, etc).</li>
+<li>Because it's a single file you can embed easily in your application.</li>
+<li>Because it passes most of the Mark Pilgrim's <a href="http://diveintomark.org/tests/client/autodiscovery/">Atom auto-discovery test suite</a>. It doesn't pass them all because some of those tests are  broken (citation needed).</li>
+</ul>
+
+<h3>Why did I build it?</h3>
+
+<ul>
+<li>Because I liked Benjamin Trott's <a href="http://search.cpan.org/~btrott/Feed-Find-0.06/lib/Feed/Find.pm">Feed::Find</a>.</li>
+<li>Because I thought it would be good to have Feed::Find's functionality in Ruby.</li>
+<li>Because I thought it was going to be easy to maintain.</li>
+<li>Because I was going to use it on <a href="http://github.com/damog/rfeed">rFeed</a>.</li>
+<li>And finally, because I didn't know <a href="http://rfeedfinder.rubyforge.org/">rfeedfinder</a> existed :-)</li>
+</ul>
+
+<h3>Bugs</h3>
+
+<p>Please, report bugs to <a href="rt@support.axiombox.com">rt@support.axiombox.com</a> or directly to the author.</p>
+
+<h3>Contribute</h3>
+
+<blockquote>
+  <p>git clone git://github.com/damog/feedbag.git</p>
+</blockquote>
+
+<p>...patch, build, hack and make pull requests. I'll be glad.</p>
+
+<h3>Author</h3>
+
+<p><a href="http://damog.net/">David Moreno</a> &lt;<a href="mailto:david@axiombox.com">david@axiombox.com</a>>.</p>
+
+<h3>Copyright</h3>
+
+<p>This is free software. See <a href="http://github.com/damog/feedbag/master/COPYING">COPYING</a> for more information.</p>
+
+<h3>Thanks</h3>
+
+<p><a href="http://maggit.net">Raquel</a>, for making <a href="http://axiombox.com">Axiombox</a> and most of my dreams possible. Also, <a href="http://github.com">GitHub</a> for making a nice code sharing service that doesn't suck.</p>

data/lib/feedbag.rb

@@ -20,8 +20,11 @@ require "rubygems"
 require "hpricot"
 require "open-uri"
 require "net/http"
+require 'timeout'
+require 'iconv'
 
 module Feedbag
+  Feed = Struct.new(:url, :title)
 
 	@content_types = [
 		'application/x.atom+xml',
@@ -64,6 +67,8 @@ module Feedbag
 		end
 		#url = "#{url_uri.scheme or 'http'}://#{url_uri.host}#{url_uri.path}"
 
+    return self.add_feed(url, nil) if looks_like_feed? url
+
 		# check if feed_valid is avail
     unless args[:narrow]
       begin
@@ -83,48 +88,39 @@ module Feedbag
     end
 
 		begin
-			html = open(url) do |f|
-				content_type = f.content_type.downcase
-				if content_type == "application/octet-stream" # open failed
-				  content_type = f.meta["content-type"].gsub(/;.*$/, '')
-				end
-				if @content_types.include?(content_type)
-					return self.add_feed(url, nil)
-				end
-
-				doc = Hpricot(f.read)
-
-				if doc.at("base") and doc.at("base")["href"]
-					$base_uri = doc.at("base")["href"]
-				else
-					$base_uri = nil
-				end
-
-				# first with links
-				(doc/"link").each do |l|
-					next unless l["rel"]
-					if l["type"] and @content_types.include?(l["type"].downcase.strip) and (l["rel"].downcase =~ /alternate/i or l["rel"] == "service.feed")
-						self.add_feed(l["href"], url, $base_uri)
-					end
-				end
-
-        unless args[:narrow]
-  				(doc/"a").each do |a|
-	  				next unless a["href"]
-		  			if self.looks_like_feed?(a["href"]) and (a["href"] =~ /\// or a["href"] =~ /#{url_uri.host}/)
-			  			self.add_feed(a["href"], url, $base_uri)
-				  	end
-  				end
-
-	  			(doc/"a").each do |a|
-		  			next unless a["href"]
-			  		if self.looks_like_feed?(a["href"])
-				  		self.add_feed(a["href"], url, $base_uri)
-					  end
-  				end
+      Timeout::timeout(10) do
+        open(url) do |f|
+          if @content_types.include?(f.content_type.downcase)
+            return self.add_feed(url, nil)
+          end
+
+					ic = Iconv.new('UTF-8//IGNORE', f.charset)
+          doc = Hpricot(ic.iconv(f.read))
+
+          if doc.at("base") and doc.at("base")["href"]
+            $base_uri = doc.at("base")["href"]
+          else
+            $base_uri = nil
+          end
+
+          # first with links
+          (doc/"link").each do |l|
+            next unless l["rel"]
+            if l["type"] and @content_types.include?(l["type"].downcase.strip) and (l["rel"].downcase =~ /alternate/i or l["rel"] == "service.feed")
+              self.add_feed(l["href"], url, $base_uri, l["title"])
+            end
+          end
+
+          unless args[:narrow]
+            (doc/"a").each do |a|
+              next unless a["href"]
+              if self.looks_like_feed?(a["href"])
+                self.add_feed(a["href"], url, $base_uri, a["title"] || a.inner_html || a['alt']) # multiple fallbacks, first title, then the tag content, then the alt tag (in case of image)
+              end
+            end
+          end
         end
-
-			end
+      end
 		rescue Timeout::Error => err
 			$stderr.puts "Timeout error ocurred with `#{url}: #{err}'"
 		rescue OpenURI::HTTPError => the_error
@@ -136,18 +132,17 @@ module Feedbag
 		ensure
 			return $feeds
 		end
-		
 	end
 
 	def self.looks_like_feed?(url)
-		if url =~ /(\.(rdf|xml|rdf|rss)$|feed=(rss|atom)|(atom|feed)\/?$)/i
+		if url =~ /((\.|\/)(rdf|xml|rdf|rss)$|feed=(rss|atom)|(atom|feed)\/?$)/i
 			true
 		else
 			false
 		end
 	end
 
-	def self.add_feed(feed_url, orig_url, base_uri = nil)
+	def self.add_feed(feed_url, orig_url, base_uri = nil, title = "")
 		# puts "#{feed_url} - #{orig_url}"
 		url = feed_url.sub(/^feed:/, '').strip
 
@@ -168,7 +163,7 @@ module Feedbag
 		end
 
 		# verify url is really valid
-		$feeds.push(url) unless $feeds.include?(url)# if self._is_http_valid(URI.parse(url), orig_url)
+		$feeds.push(Feed.new(url, title)) unless $feeds.any? { |f| f.url == url }# if self._is_http_valid(URI.parse(url), orig_url)
 	end
 
 	# not used. yet.

data/rails/init.rb

@@ -0,0 +1 @@
+require File.join File.dirname(__FILE__), "..", "lib", "feedbag"

data/test/atom_autodiscovery_test.rb

@@ -0,0 +1,46 @@
+#!/usr/bin/ruby
+
+require "#{File.dirname(__FILE__)}/../feedbag"
+require "test/unit"
+require "open-uri"
+require "hpricot"
+require "pp"
+
+class AtomAutoDiscoveryTest < Test::Unit::TestCase
+	def test_autodisc
+		base_url = "http://diveintomark.org/tests/client/autodiscovery/"
+		url = base_url + "html4-001.html"
+
+		i = 1
+		puts "trying now with #{url}"
+		while(i)
+			puts
+			i = 0 # unless otherwise found
+
+			f = Feedbag.find url
+
+			assert_instance_of Array, f
+			assert f.size == 1, "Feedbag didn't find a feed on #{url} or found more than one"
+
+			puts " found #{f[0]}"
+			feed = Hpricot(open(f[0]))
+	
+			(feed/"link").each do |l|
+				next unless l["rel"] == "alternate"
+				assert_equal l["href"], url
+			end
+
+			# ahora me voy al siguiente
+			html = Hpricot(open(url))
+			(html/"link").each do |l|
+				next unless l["rel"] == "next"
+				url = URI.parse(base_url).merge(l["href"]).to_s
+				puts "trying now with #{url}"
+				i = 1
+			end
+		
+		end	
+	end
+
+
+end

metadata

@@ -1,51 +1,58 @@
 --- !ruby/object:Gem::Specification 
 name: penso-feedbag
 version: !ruby/object:Gem::Version 
-  version: 0.5.100
+  version: 0.6.1
 platform: ruby
 authors: 
-- Axiombox
-- David Moreno
+- Joel Duffin
+- Justin Ball
 - Fabien Penso
 autorequire: 
 bindir: bin
 cert_chain: []
 
-date: 2010-02-11 00:00:00 +01:00
+date: 2009-11-10 00:00:00 +01:00
 default_executable: 
 dependencies: 
 - !ruby/object:Gem::Dependency 
-  name: hpricot
-  type: :runtime
+  name: shoulda
+  type: :development
   version_requirement: 
   version_requirements: !ruby/object:Gem::Requirement 
     requirements: 
     - - ">="
       - !ruby/object:Gem::Version 
-        version: "0.6"
+        version: "0"
     version: 
-description: Ruby's favorite feed auto-discoverty tool
-email: david@axiombox.com
+description: This gem will return title and url for each feed discovered at a given url, handling proper charset
+email: fabienpenso@gmail.com
 executables: []
 
 extensions: []
 
 extra_rdoc_files: 
+- ChangeLog
 - README.markdown
-- COPYING
 files: 
-- lib/feedbag.rb
-- benchmark/rfeedfinder_benchmark.rb
-- README.markdown
+- .gitignore
 - COPYING
+- ChangeLog
+- README.markdown
+- TODO
+- VERSION
+- benchmark/rfeedfinder_benchmark.rb
+- feedbag.gemspec
+- index.html
+- lib/feedbag.rb
+- rails/init.rb
+- test/atom_autodiscovery_test.rb
 has_rdoc: true
-homepage: http://axiombox.com/feedbag
+homepage: http://github.com/penso/muck-feedbag
 licenses: []
 
 post_install_message: 
 rdoc_options: 
-- --main
-- README.markdown
+- --charset=UTF-8
 require_paths: 
 - lib
 required_ruby_version: !ruby/object:Gem::Requirement 
@@ -62,10 +69,10 @@ required_rubygems_version: !ruby/object:Gem::Requirement
   version: 
 requirements: []
 
-rubyforge_project: feedbag
+rubyforge_project: 
 rubygems_version: 1.3.5
 signing_key: 
 specification_version: 3
-summary: Ruby's favorite feed auto-discovery tool
-test_files: []
-
+summary: Fork of the feedbag gem.
+test_files: 
+- test/atom_autodiscovery_test.rb