DRMacIver-gourmand 0.0.2

data/LICENSE

@@ -0,0 +1,25 @@
+Copyright (c) 2009, David R. MacIver
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+    * Redistributions of source code must retain the above copyright
+      notice, this list of conditions and the following disclaimer.
+    * Redistributions in binary form must reproduce the above copyright
+      notice, this list of conditions and the following disclaimer in the
+      documentation and/or other materials provided with the distribution.
+    * Neither the name of the gourmand nor the
+      names of its contributors may be used to endorse or promote products
+      derived from this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY David R. MacIver 'AS IS'' AND ANY
+EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL David R. MacIver BE LIABLE FOR ANY
+DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
+(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+

data/README.markdown

@@ -0,0 +1,26 @@
+## Gourmand
+
+Gourmand automatically imports links you've up voted on reddit and other social bookmarking sites into delicious. You simply provide it with your account details on each and set up a cron job to run it regularly. It takes care of the rest.
+
+### Notes on behaviour:
+
+  * Despite the name, reddilicous actually imports from a bunch of different sites. Currently only twitter, reddit and stumbleupon, but that's purely a function of the fact that those are the ones I use at the moment.
+  * Links are tagged with via:source, plus any tags that can be obtained from there.
+  * The date on the link will be set to that at which the URL was initially posted on the source site, not the time you imported it (which plays badly with historical data) or the time you up voted it (which doesn't seem to be available information everywhere).
+  * It will import all your histories, so the initial run will take a while. 
+  * This is very slow. There's a mixture of reasons for this - it's written in Ruby, it generate a fair bit of HTTP traffic and it deliberately rate limits itself in a lot of cases. The single biggest reason though is that I don't particularly care and I haven't optimised it. It's intended to be run a couple times an hour by an automated task, and you have to hit pretty damn heavy traffic before it's too slow for that.
+  * Twitter support will pull in any URLs mentioned on your friends time-line, automatically tagging them based on delicious suggestions and tagging them with information about who it was to and from.
+
+
+### Some general comments:
+
+  * The code is currently a bit grim in places. Some of this is inevitable - site scraping is never going to look pretty - some of it will probably be cleaned up at various points. 
+  * Patches are *exceedingly* welcome. There are a pile of sites this could reasonably import from, and I don't use so much as a tenth of them. If you want this to handle your favourite social bookmarking or similar site, please feel free to submit a patch. 
+  * I'm currently changing the internal format on a semi regular basis as I figure things out. Once I've got an actual release out I'll have a proper versioning system for upgrades, etc. but I don't yet.
+
+### Dependencies 
+
+ * [json](http://json.rubyforge.org/)
+ * [nokogiri](http://github.com/tenderlove/nokogiri/tree/master)
+ * [httparty](http://github.com/jnunemaker/httparty/tree/master)
+ * [mechanize](http://mechanize.rubyforge.org/mechanize/) (for stumbleupon)

data/Rakefile

@@ -0,0 +1,13 @@
+require 'rubygems'
+require 'rake'
+
+require 'jeweler'
+Jeweler::Tasks.new do |gem|
+  gem.name = "gourmand"
+  gem.summary = "gourmand is a tool for automatically importing links into delicious" 
+  gem.email = "david.maciver@gmail.com"
+  gem.homepage = "http://github.com/DRMacIver/gourmand"
+  gem.authors = ["David R. MacIver"]
+  
+  # gem is a Gem::Specification... see http://www.rubygems.org/read/chapter/20 for additional settings
+end

data/VERSION

@@ -0,0 +1 @@
+0.0.2

data/bin/gourmand

@@ -0,0 +1,51 @@
+#!/usr/bin/env ruby
+$: << File.join(File.dirname(__FILE__), "../lib")
+
+require "gourmand.rb"
+
+def usage
+  STDERR.puts <<-USAGE 
+  Usage:
+  gourmand update           # Update a gourmand instance, posting the new bookmarks to delicious"
+  gourmand undo [site]      # deletes all imported posts (from specific site, or all if not specified)
+  USAGE
+  Reddilicious.site_names.each do  |site|
+    STDERR.puts "  gourmand #{site}     # set your #{site} user" 
+  end  
+
+  exit(1)
+end
+
+dir=ENV["REDDILICIOUS_HOME"] || File.join(ENV["HOME"], ".gourmand")
+gourmand = Reddilicious.new(dir)
+
+if !File.directory?(dir)
+  puts "no such directory #{dir}. Creating..."
+  Dir.mkdir(dir)
+
+  puts "Delicious user name:" 
+  delicious = STDIN.gets.strip
+  delicious = nil if delicious == ""
+  puts "Delicious password:"
+  delicious_password = STDIN.gets.strip
+  gourmand.create!(delicious, delicious_password)
+end
+
+case ARGV[0]
+  when "update"
+    File.open("#{dir}/gourmand.log", "a") do |log|
+      log.sync = true
+      $stdout = log
+      $stderr = log
+      gourmand.transfer_to_delicious 
+    end
+  when *Reddilicious.site_names:
+    gourmand.site_for(ARGV[0]).ask_for_credentials
+  when "undo"    
+    sites = ARGV[1..-1].empty? ? Reddilicious.site_names : ARGV[1..-1]
+    puts "undo import for sites #{sites.inspect}: are you sure? (y/n)"
+    if STDIN.gets.strip.downcase == 'y'           
+      sites.each { |s|  gourmand.site_for(s).undo_import! }
+    end
+  else usage    
+end

data/gourmand.gemspec

@@ -0,0 +1,48 @@
+# -*- encoding: utf-8 -*-
+
+Gem::Specification.new do |s|
+  s.name = %q{gourmand}
+  s.version = "0.0.2"
+
+  s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
+  s.authors = ["David R. MacIver"]
+  s.date = %q{2009-07-16}
+  s.default_executable = %q{gourmand}
+  s.email = %q{david.maciver@gmail.com}
+  s.executables = ["gourmand"]
+  s.extra_rdoc_files = [
+    "LICENSE",
+     "README.markdown"
+  ]
+  s.files = [
+    "LICENSE",
+     "README.markdown",
+     "Rakefile",
+     "VERSION",
+     "bin/gourmand",
+     "gourmand.gemspec",
+     "lib/blacklist.rb",
+     "lib/delicious.rb",
+     "lib/gourmand.rb",
+     "lib/post.rb",
+     "lib/reddit.rb",
+     "lib/site.rb",
+     "lib/stumbleupon.rb",
+     "lib/twitter.rb"
+  ]
+  s.homepage = %q{http://github.com/DRMacIver/gourmand}
+  s.rdoc_options = ["--charset=UTF-8"]
+  s.require_paths = ["lib"]
+  s.rubygems_version = %q{1.3.4}
+  s.summary = %q{gourmand is a tool for automatically importing links into delicious}
+
+  if s.respond_to? :specification_version then
+    current_version = Gem::Specification::CURRENT_SPECIFICATION_VERSION
+    s.specification_version = 3
+
+    if Gem::Version.new(Gem::RubyGemsVersion) >= Gem::Version.new('1.2.0') then
+    else
+    end
+  else
+  end
+end

data/lib/blacklist.rb

@@ -0,0 +1,44 @@
+class Blacklist
+  def initialize(blacklist)
+    @blacklist = Hash.new{|h, k| h[k] = [] }
+
+    blacklist.each { |list|
+      list.each { |tag|
+        @blacklist[tag] << list
+      }
+    }
+  end
+
+  def self.from_file(file)
+    return nil if !File.exists?(file) 
+    Blacklist.new(IO.read(file).split("\n").map{|l| l.split})
+  end
+
+  def blacklisted?(tags)
+    if tags.is_a? String
+      tags = tags.split
+    end
+
+    if !tags.is_a? Array
+      raise "unrecognised argument #{tags.inspect}"
+    end
+
+   shallow_flatten(tags.
+      map{|t| @blacklist[t]}.
+      compact).
+      any?{|set|
+        !set.empty? && set.all?{|t|
+          tags.include?(t)
+        }
+      } 
+  end
+
+  private 
+   
+  def shallow_flatten(enum)
+    it = []
+    enum.each{|x| it += x }
+    it
+  end
+  
+end

data/lib/delicious.rb

@@ -0,0 +1,15 @@
+require "rubygems"
+require "httparty"
+
+module Delicious
+  include HTTParty  
+  base_uri "https://api.del.icio.us/v1"
+  
+  #Please set your User-Agent to something identifiable.
+  #The default identifiers like "Java/1.4.3" or "lwp-perl" etc tend to get banned from time to time.  
+  headers 'User-Agent' => 'gourmand (0.1)' 
+    
+  format :xml  
+end
+
+

data/lib/gourmand.rb

@@ -0,0 +1,195 @@
+require "site"
+require "httparty"
+require "delicious"
+require "json"
+require 'net/http'
+require 'uri'
+require "blacklist"
+
+class Reddilicious
+  # lambdas to get lazy loading
+  SitesToClasses = {
+    "reddit" => lambda{
+      require "reddit";
+      Reddit::Liked
+    },
+
+    "stumbleupon" => lambda{
+      require "stumbleupon"
+      StumbleUpon::Favourites
+    },
+    "twitter" => lambda{
+      require "twitter"
+      Twitter::FriendsTimeline
+    }
+  }
+
+  attr_accessor :dir
+  def initialize(dir)
+    @dir = dir
+    @blacklist = Blacklist.from_file(File.join(dir, "blacklist"))
+    @untiny_cache = if File.exists?(untiny_cache_file)
+      JSON.parse(IO.read(untiny_cache_file))
+    else
+      {}
+    end
+    
+    if File.exists?(details_file)
+      Delicious.basic_auth(details["delicious_user"], details["delicious_password"])
+    end
+  end
+
+  def sites
+    @sites ||= Dir["#{@dir}/*"].map{|x| site_for(File.basename(x))}.compact
+  end
+
+  def site_for(x)
+   c = SitesToClasses[x]
+   c && c.call.new(self)
+  end
+
+  def create!(delicious, delicious_password)
+    File.open(details_file, "w"){|o|
+      o.puts({:delicious_user => delicious, :delicious_password => delicious_password}.to_json)
+    }
+  end 
+
+  def details
+    @details ||= JSON.parse(IO.read(details_file))
+  end
+
+  def update_time
+    puts "Checking the server for last update time"
+    sleep 1
+    time = Delicious.get("/posts/update")["update"]["time"]
+    puts "last updated at #{time}"
+    time
+  end
+  
+  def update_untiny_cache
+    File.open(untiny_cache_file, 'w') { |f| f << JSON.pretty_generate(@untiny_cache) }
+  end
+
+  def untiny_url(url, n=0)
+   @untiny_cache[url] ||= begin      
+      puts "untiny #{url}"          
+      resp = Net::HTTP.get_response(URI.parse(url)) 
+      url = resp['location'] || url
+      if [301, 302].include?(resp.code.to_i) && n<=3 && resp['location']
+        untiny_url(resp['location'], ++n) 
+      else
+        url
+      end
+    rescue Exception => e
+      puts "WARNING: #{e}"
+      url
+    end          
+  end
+
+  def bookmark_for(url, suggest_tags=true)    
+    url = untiny_url(url)
+    Post.new do |post|
+      post.url = url      
+    end
+  end
+
+  def delicious_posts
+    puts "Reading existing delicious bookmarks"
+    delicious_post_file = File.join(@dir, "bookmarks.json")
+
+    @existing_posts = nil
+
+    if File.exist? delicious_post_file
+      @existing_posts ||= JSON.parse(IO.read(delicious_post_file))
+    end
+
+    if !@existing_posts || (update_time > @existing_posts["updated"]) 
+      puts "Bookmarks out of date. Fetching from server"
+      posts = Delicious.get("/posts/all")["posts"]
+      raise "error fetching all posts" unless posts 
+      
+      @existing_posts = {"updated" => posts["update"], "posts" => posts["post"] || []}
+      File.open(delicious_post_file, "w") do |o|
+        o.puts JSON.pretty_generate(@existing_posts)
+      end
+    else
+      puts "Nothing to do here: Our existing bookmarks are up to date"    
+    end
+    
+    posts = (@existing_posts["posts"] || []).map{|x| Post.new(x){|p| p.url = x["href"]; p.tags = x["tag"]}}
+    puts "found #{posts.length} existing bookmarks"
+    posts
+  end
+
+  def transfer_to_delicious
+    puts "Beginning import at #{Time.now}"
+    puts "Found importers for #{sites.join(", ")}"
+    new_updates = sites.map{|x| x.update!}.flatten
+    update_untiny_cache
+    
+    puts "#{new_updates.length} urls to import"
+
+    return if new_updates.empty?
+
+    urls_to_posts = {}
+
+    new_updates.each do |post|
+      urls_to_posts[post.url] = post.merge(urls_to_posts[post.url])
+    end
+
+    puts "checking for existing bookmarks"
+    delicious_posts.each do |post|
+      update = urls_to_posts[post.url]
+      if update
+        puts "merging existing post for #{post.url}"
+        urls_to_posts[post.url] = update.merge post
+      end
+    end
+
+    new_updates = urls_to_posts.values.compact
+
+    puts "#{new_updates.length} urls after merging"
+
+    new_updates.each do |update|
+      
+      blacklisted = @blacklist && @blacklist.blacklisted?(update.tags)
+
+      if blacklisted
+        puts "ignoring #{update.description} due to (#{update.url}) is blacklisting (tags #{update.tags})"
+      else
+        puts "importing #{update.description} (#{update.url})"
+        
+        update.fetch_metadata!(false)
+
+        res = Delicious.post("/posts/add", :query => update.to_h)
+        if !res['result'] || res['result']['code'] != 'done'
+          puts "error importing post: #{res.inspect}"
+        end
+        
+        sleep(1)      
+      end
+    end
+    puts "Saving data to storage"
+    sites.each{|x| x.save!}
+        
+    puts "Import complete"
+
+    nil
+  end
+
+
+  def self.site_names
+    SitesToClasses.keys
+  end
+    
+
+  private
+  def details_file
+    File.join(dir, "details.json")
+  end
+  
+  def untiny_cache_file
+    File.join(dir, "untiny.json")
+  end
+    
+end

data/lib/post.rb

@@ -0,0 +1,90 @@
+require "ostruct"
+require "set"
+require "rubygems"
+require "nokogiri"
+require "open-uri"
+
+class Post
+  NEW_MARKER = "imported_by:gourmand"
+  
+  Data = :url, :dt, :description, :extended, :tags
+  
+  attr_accessor *Data
+  
+  def initialize(hash=nil)    
+    yield self if block_given?        
+    if hash
+      hash.each do |key, value|
+        instance_variable_set("@" + key, value)
+      end
+    end        
+    raise "all posts must have a URL" if !self.url  
+  end
+
+  def to_h
+    it = {}
+    Data.each{|d| it[d] = instance_variable_get("@" + d.to_s)}
+    it
+  end
+
+  def auto_imported?
+    tag_set.include?(NEW_MARKER)
+  end
+   
+  def tag_set
+    if !self.tags then Set.new else Set[*self.tags.split] end
+  end
+  
+  def fetch_metadata!(suggest_tags=true)        
+    self.description ||= begin      
+      Nokogiri::HTML(open(url)).xpath("//title").text.gsub("\n", " ").gsub(/ +/, " ").strip 
+    rescue Exception => e
+      puts "WARNING: #{e}"
+      url
+    end
+    
+    self.description = url if description.empty?
+    
+    if suggest_tags
+      suggest = Delicious.get("/posts/suggest", :query => {:url => url} )      
+      if suggest['suggest']               
+        suggested_tags = suggest["suggest"]["popular"] || []
+        self.tags = suggested_tags.is_a? Array ? suggested_tags.join(" ") : suggested_tags
+      end
+      sleep 1
+    end    
+  end
+  
+  def merge(that)
+    return self if !that
+    raise "cannot merge posts with different URLS: #{self.url} != #{that.url}" if self.url != that.url
+
+    result = Post.new{|p| 
+      p.url = self.url
+      p.description = that.description
+
+      if(self.extended && !that.extended)
+        p.extended = self.extended 
+      else
+        p.extended = that.extended
+      end
+
+      old_tags = that.tag_set 
+      new_tags = self.tag_set - old_tags
+      
+      # remove the new marker unless it was already present in old post
+      new_tags -= [NEW_MARKER] unless that.auto_imported?
+      
+      p.tags = if !new_tags.empty? 
+        new_tags.to_a.join(" ") + " " + that.tags
+      else
+        that.tags
+      end
+
+      p.dt = [self.dt, that.dt].compact.min  
+    }
+    
+    result = nil if result == that  # FIXME    
+    result
+  end
+end

data/lib/reddit.rb

@@ -0,0 +1,60 @@
+require "site"
+require "httparty"
+
+# Ruby script for fetching the "posts" history of a reddit user as json
+
+module Reddit
+  include HTTParty
+  base_uri "http://reddit.com"
+  format :json
+
+  class Liked < Site
+    def name
+      "reddit"
+    end
+  
+    def update!
+      puts "Updating Reddit"
+      balance
+      results = []
+      new_results = nil
+      after = nil
+      i = 0
+      while !(new_results = merge_results(Reddit.get("/user/#{credentials["user"].strip}/liked/.json", :query => {"after" => after})["data"]["children"].map{|x| x["data"]})).empty? 
+        puts "fetching reddit page #{i}"
+        results += new_results
+        after = new_results[-1]["name"]
+        i += 1
+      end
+
+      results.map{ |update| to_post(update) }
+    end
+
+    def to_post(data)
+      Post.new({
+        "url" => data["url"].gsub("&amp;", "&"), # TODO: Better unescaping
+        "description" => data["title"],
+        "tags" => ["via:reddit", data["subreddit"]].join(" "),
+        "replace" => "yes",
+        "dt" => Time.at(data["created_utc"]).strftime("%Y-%m-%dT%H:%M:%SZ")
+      })
+    end
+
+    def date(post)
+      post["created_utc"]
+    end
+
+    private
+
+    def merge_results(results)
+      return if !results
+      done = false
+      new_results = results.select do |x|
+        !@ids.include?(identifier(x))
+      end
+      @posts += new_results
+      @ids.merge(results.map{|x| x["name"]})
+      new_results
+    end
+  end
+end

data/lib/site.rb

@@ -0,0 +1,90 @@
+require "post"
+require "rubygems"
+require "json"
+require "set"
+
+class Site
+  attr_accessor :posts
+
+  def initialize(reddilicous)
+    @gourmand = reddilicous
+    @dir = File.join(reddilicous.dir, name)
+    @ids = Set.new
+    Dir.mkdir(@dir) if !File.exists? @dir
+    yield(self) if block_given?
+  end  
+
+  def credentials=(credentials)
+    @credentials = credentials
+    File.open(credentials_file, "w"){|o| o.puts JSON.pretty_generate(@credentials)}
+  end
+
+  def credentials
+    @credentials ||= if File.exists? credentials_file then JSON.parse(IO.read(credentials_file)) else {} end
+  end
+
+  def posts
+    @posts ||= if File.exists? posts_file then JSON.parse(IO.read(posts_file)) else [] end;
+  end
+
+  def balance 
+    @ids.merge(posts.map{|p| identifier(p)})
+    posts.sort!{|x, y| date(y) <=> date(x)}
+  end
+
+  def save!
+    balance
+    File.open(posts_file, "w"){|o| o.puts JSON.pretty_generate(@posts)} if @posts
+  end
+
+  def to_post(data)
+    Post.new(data)
+  end
+
+  def identifier(post)
+    post["url"]
+  end
+
+  # Not required to return an actual date, only something which compares
+  # in the correct order to be read as such
+  def date(post)
+    post["dt"]
+  end
+
+  def to_s
+    name
+  end
+
+  def imported_links
+    Delicious.get("/posts/all", :query=>{:tag=>["via:#{name}", Post::NEW_MARKER].join(' ')})['posts']['post'] || []
+  end
+  
+  def undo_import!
+    posts = imported_links
+    posts = [posts] if posts.is_a?(Hash) 
+    puts "undo imports for #{self}" unless posts.empty?
+    
+    posts.each do |p|
+      puts "deleting #{p['description']}"
+      res = Delicious.delete('/posts/delete', :query=>{:url=>p['href']})
+      if !res['result'] || res['result']['code'] != 'done'
+        puts "delete failed: #{res.inspect}"
+      end
+    end
+  end
+  
+  def ask_for_credentials
+    puts "#{name} user name:"
+    self.credentials = {"username" => STDIN.gets.strip}    
+  end
+  
+  private 
+
+  def posts_file
+    File.join(@dir, "posts.json")
+  end  
+
+  def credentials_file
+    File.join(@dir, "credentials.json")
+  end
+end

data/lib/stumbleupon.rb

@@ -0,0 +1,106 @@
+require "rubygems"
+require "mechanize"
+require "json"
+require "set"
+require "site"
+
+module StumbleUpon
+
+  class Favourites < Site
+ 
+    def name
+      "stumbleupon"
+    end
+ 
+    def update!
+      puts "Updating Stumbleupon"
+      balance
+      i = 0
+
+      results = []
+      new_results = nil
+
+      while !new_results || !new_results.empty?
+        puts "fetching stumbleupon page #{i}"
+        new_results = StumbleUpon.fetch_page(credentials["user"], i).select{|x| !@ids.include? identifier(x)}
+
+        results += new_results
+        i += 1
+      end
+
+
+      @posts += results
+      balance
+      results.map{|u| to_post(u)}
+    end
+  end  
+
+  def self.fetch_page(user, page_number=nil)
+    url = "http://www.stumbleupon.com/stumbler/#{user}/favorites/" 
+    if page_number && (page_number > 0)
+      url += (page_number * 10).to_s
+      url += "/"
+    end
+
+    mechanize = WWW::Mechanize.new{|agent| agent.user_agent_alias = "Linux Mozilla"}
+
+    list_view = mechanize.get(url).search("a").select{|x| x["href"] =~ /viewmode=list/}[0]
+
+    mechanize.click(list_view) if list_view
+
+    mechanize.page.search("dl.dlBlog").map{|post| parse_review(post)}.compact 
+  end
+   
+
+  private
+   
+  def self.parse_review(review)
+    title_elem = review.search("dt")[0]
+    title = title_elem.text
+    
+    href = nil
+    # stumbleupon seems to be doing some mangling which means
+    # I'm seeing different results here than in firefox
+    # For the moment the following is our "best guess" as to
+    # what the URL should be.
+    
+    urls = review.search("a").map{|x| x["href"]}
+
+    urls.reject!{|x| x !~ /^http:\/\//} # remove javacript only and relative links
+    urls.reject!{|x| x =~ /^http:\/\/www.stumbleupon.com/} # remove internal links
+
+    if urls.length == 1
+      href = urls[0]
+    else
+      STDERR.puts "I didn't know what to do with the post #{title}. Its URLs made no sense"
+      return
+    end
+   
+    tags = review.search("a").map{|x| x["href"]}.grep(/\/tag\//).map{|h| h.gsub("/tag/", "").gsub("/", "")}
+   
+    tags << "via:stumbleupon"
+
+    contents = review.search("dd").select{|x| x["id"] =~ /blog_contents/}
+    if !contents.empty?
+      contents = contents[0].children.select{|x| x.text?}.join("\n")
+    else
+      contents = nil
+    end
+   
+    datetime_string = review.search(".stats")[0].text.gsub(/(am|pm).*$/){$1}.strip.gsub(",", " ").gsub(/ +/, " ")
+
+    # Try and parse something useful out of the SU date string.
+    time_re = /([0-9:]+(?:am|pm))/
+    date_string = datetime_string.gsub(time_re, "").strip
+    date = (if date_string == "" then DateTime.now else DateTime.parse(date_string) end)
+    time = DateTime.parse(datetime_string.scan(time_re)[0][0])
+    dt = Time.utc(date.year, date.month, date.day, time.hour, time.min)
+   
+    it = {"url" => href, "description" => title, "tags" => tags.join(" "), "dt" => dt && dt.strftime("%Y-%m-%dT%H:%M:%SZ")}
+   
+    it["extended"] = contents if contents
+   
+    it
+  end
+end
+

data/lib/twitter.rb

@@ -0,0 +1,96 @@
+require "rubygems"
+require "httparty"
+require "gourmand"
+
+module Twitter
+  include HTTParty
+  base_uri "http://twitter.com"
+  format :json
+ 
+  class FriendsTimeline < Site
+    def name
+      "twitter"
+    end
+
+    def update!
+      puts "Updating twitter"
+      balance
+
+      last_post_id = posts[0] && posts[0]["id"]
+
+      results = []
+    
+      new_tweets = nil
+
+      page = 0 
+
+      query = {:count => 200 }
+
+      query[:since_id] = last_post_id if last_post_id
+    
+      while !(new_tweets = get_tweets(query)).empty?
+        results += new_tweets
+        puts "importing twitter page #{query[:page] || 0}"
+        query[:page] = (query[:page] || 0) + 1
+      end 
+      
+      @posts += results
+
+      results.map do |res|              
+        urls = res["text"].scan(/(http:\/\/[^,()" ]+)/).flatten
+        ats = res["text"].scan(/@([[:alnum:]]+)/).flatten
+        hashtags = res["text"].scan(/#([[:alnum:]]+)/).flatten
+        retweet = res["text"] =~ /RT[^a-zA-Z]/ || res["text"] =~ /\(via @[^)]+\)/
+
+        urls.map do |url|
+          post = @gourmand.bookmark_for(url)
+          post.tags = [
+            "via:twitter",
+            Post::NEW_MARKER,
+            ats.map{|a| "to:" + a}.sort,
+            "from:#{res["user"]["screen_name"]}",
+            hashtags,        
+            ("retweet" if retweet),
+            post.tags
+          ].compact.flatten.join(" ").strip
+
+          post.extended = "Imported from http://twitter.com/#{res["user"]["screen_name"]}/status/#{res["id"]}\n\n#{res["text"]}"
+          post.dt = date(res).strftime("%Y-%m-%dT%H:%M:%SZ")
+
+          post
+        end
+      end.flatten
+    end
+
+
+    def get_tweets(query)
+      begin
+        res = Twitter.get("/statuses/friends_timeline.json", :query => query, :basic_auth => {:username => credentials["username"], :password => credentials["password"]})         
+        raise "Error fetching timeline: '#{res['error']}'" if res.is_a?(Hash) && res['error']
+        res
+      rescue Crack::ParseError
+        raise # TODO
+      end
+    end
+    
+    
+    def identifier(post)
+      post["id"]
+    end
+
+    def date(post)
+      DateTime.parse(post["created_at"])
+    end
+    
+    
+    def ask_for_credentials
+      puts "#{name} user name:"
+      user = STDIN.gets.strip
+      puts "#{name} password:"
+      pass = STDIN.gets.strip
+      self.credentials = {"username" => user, "password" => pass}
+    end
+  end
+  
+  
+end

metadata

@@ -0,0 +1,67 @@
+--- !ruby/object:Gem::Specification 
+name: DRMacIver-gourmand
+version: !ruby/object:Gem::Version 
+  version: 0.0.2
+platform: ruby
+authors: 
+- David R. MacIver
+autorequire: 
+bindir: bin
+cert_chain: []
+
+date: 2009-07-16 00:00:00 -07:00
+default_executable: gourmand
+dependencies: []
+
+description: 
+email: david.maciver@gmail.com
+executables: 
+- gourmand
+extensions: []
+
+extra_rdoc_files: 
+- LICENSE
+- README.markdown
+files: 
+- LICENSE
+- README.markdown
+- Rakefile
+- VERSION
+- bin/gourmand
+- gourmand.gemspec
+- lib/blacklist.rb
+- lib/delicious.rb
+- lib/gourmand.rb
+- lib/post.rb
+- lib/reddit.rb
+- lib/site.rb
+- lib/stumbleupon.rb
+- lib/twitter.rb
+has_rdoc: false
+homepage: http://github.com/DRMacIver/gourmand
+post_install_message: 
+rdoc_options: 
+- --charset=UTF-8
+require_paths: 
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement 
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      version: "0"
+  version: 
+required_rubygems_version: !ruby/object:Gem::Requirement 
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      version: "0"
+  version: 
+requirements: []
+
+rubyforge_project: 
+rubygems_version: 1.2.0
+signing_key: 
+specification_version: 3
+summary: gourmand is a tool for automatically importing links into delicious
+test_files: []
+