monkeyshines 0.0.2

data/.document

@@ -0,0 +1,4 @@
+README.textile
+lib/**/*.rb
+examples/**/*.rb
+LICENSE.textile

data/.gitignore

@@ -0,0 +1,43 @@
+\#*
+.\#*
+*~
+.DS_Store
+Icon?
+REVISION
+TAGS*
+nohup.out
+.bzr
+.hg
+.svn
+
+a.out
+*.o
+*.pyc
+*.so
+*.stackdump
+*.sw?
+*.tmproj
+*_flymake.*
+.project
+.pydevproject
+.settings
+.tasks-cache
+.yardoc
+
+/**/*DONTVERSION*
+/**/*private*
+/**/cache/*
+/**/log/*
+/**/tmp/*
+/coverage
+/doc/*
+/pkg/*
+/rdoc/*
+
+/db/*.sqlite3
+/db/sphinx
+/config/*.sphinx.conf
+/config/database.yml
+/config/sphinx.yml
+/public/stylesheets/compiled/*
+/vendor/src/**/*

data/LICENSE

@@ -0,0 +1,20 @@
+Copyright (c) 2009 Philip (flip) Kromer
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/LICENSE.textile

@@ -0,0 +1,20 @@
+---
+layout: default
+title:  MIT License
+---
+
+h1(gemheader). {{ site.gemname }} %(small):: license%
+
+<div class="toggle">
+
+h2. MIT License
+
+__Copyright (c) 2009 Philip (flip) Kromer__
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+</div>

data/README.textile

@@ -0,0 +1,125 @@
+Monkeyshines is a tool for doing an algorithmic scrape.
+
+It's designed to handle large-scale scrapes that may exceed the capabilities of single-machine relational databases, so it plays nicely with Hadoop / Wukong, with distributed databases (MongoDB, tokyocabinet, etc.), and distributed job queue (eg "edamame/beanstalk":http://mrflip.github.com/edamame).
+
+---------------------------------------------------------------------------
+
+h2. Overview
+
+A monkeyshines scraper is simple in principle:
+
+
+
+h2. Dependencies
+
+This is best run standalone -- not as a gem; it's still in heavy development. I recommend cloning
+
+* http://github.com/mrflip/edamame
+* http://github.com/mrflip/wuclan
+* http://github.com/mrflip/wukong
+* http://github.com/mrflip/monkeyshines (this repo)
+
+into a common directory.
+
+Additionally, you'll need some of these gems:
+
+* addressable (2.1.0)
+* extlib (0.9.12)
+* htmlentities (4.2.0)
+
+To build the gem, you'll need
+
+* git (1.2.2)
+* jeweler (1.2.1)
+* rake (0.8.7)
+* rspec (1.2.6)
+* rubyforge (1.0.4)
+* sources (0.0.1)
+
+And if you spell ruby with a 'j', you'll want
+
+* jruby-openssl (0.5.2)
+* json-jruby (1.1.7)
+
+
+
+
+h2. Request Queue
+
+h3. Periodic requests
+
+Request stream can be metered using read-through, scheduled (eg cron), or test-and-sleep.
+
+* Scheduled
+* Test and sleep. A queue of resources is cyclically polled, sleeping whenever bored.
+
+
+
+h2. Requests
+
+* Base: simple fetch and store of URI. (URI specifies immutable unique resource)
+* : single resource, want to check for updates over time.
+* Timeline:
+** Message stream, eg. twitter search or user timeline. Want to do paginated requests back to last-seen 
+** Feed: Poll the resource and extract contents, store by GUID. Want to poll frequently enough that single-page request gives full coverage.
+
+---------------------------------------------------------------------------
+
+h2. Scraper
+
+* HttpScraper --
+** JSON
+** HTML
+*** \0 separates records, \t separates initial fields; 
+*** map \ to \\, then tab, cr and newline to \t, \r and \n resp.
+*** map tab, cr and newline to 	 
 and 
 resp.
+
+
+x9 xa xd x7f
+
+* HeadScraper -- records the HEAD parameters
+
+---------------------------------------------------------------------------
+
+h2. Store 
+
+
+* Flat file (chunked)
+* Key store
+* Read-through cache
+
+---------------------------------------------------------------------------
+
+h2. Periodic
+
+* Log only every N requests, or t minutes, or whatever.
+* Restart session every hour
+* Close file and start new chunk every 4 hours or so. (Mitigates data loss if a file is corrupted, makes for easy batch processing).
+
+---------------------------------------------------------------------------
+
+h2. Pagination
+
+h4. Session
+
+* *Twitter Search*: Each req brings in up to 100 results in strict reverse ID (pseudo time) order. If the last item ID in a request is less than the previous scrape session's max_id, or if fewer than 100 results are returned, the scrape session is complete.  We maintain two scrape_intervals: one spans from the earliest seen search hit to the highest one from the previous scrape; the other ranges backwards from the highest in _this_ scrape session (the first item in the first successful page request) to the lowest in this scrape session (the last item on the most recent successful page request).
+
+** Set no upper limit on the first request. 
+** Request by page, holding the max_id fixed
+** Use the lowest ID from the previous request as the new max_id
+** Use the supplied 'next page' parameter
+
+* *Twitter Followers*: Each request brings in 100 followers in reverse order of when the relationship formed. A separate call to the user can tell you how many _total_ followers there are, and you can record how many there were at end of last scrape, but there's some slop (if 100 people in the middle of the list /un/follow and 100 more people at the front /follow/ then the total will be the same).  High-degree accounts may have as many as 2M followers (20,000 calls).
+
+* *FriendFeed*: Up to four pages. Expiry given by result set of <100 results.
+
+
+* Paginated: one resource, but requires one or more requests to 
+** Paginated + limit (max_id/since_date): rather than request by increasing page, request one page with a limit parameter until the last-on-page overlaps the previous scrape.  For example, say you are scraping search results, and that when you last made the request the max ID was 120_000; the current max_id is 155_000. Request the first page (no limit). Using the last result on each page as the new limit_id until that last result is less than 120_000.
+** Paginated + stop_on_duplicate: request pages until the last one on the page matches an already-requested instance.
+** Paginated + velocity_estimate: . For example, say a user acquires on average 4.1 followers/day and it has been 80 days since last scrape. With 100 followers/req you will want to request ceil( 4.1 * 80 / 100 ) = 4 pages.
+
+h4. Rescheduling
+
+Want to perform next scrape to give a couple pages or a mostly-full page. Need to track a rate (num_items / timespan), clamped to a min_reschedule / max_reschedule bounds.
+

data/Rakefile

@@ -0,0 +1,105 @@
+require 'rubygems'
+require 'rake'
+
+begin
+  require 'jeweler'
+  Jeweler::Tasks.new do |gem|
+    # gem is a Gem::Specification... see http://www.rubygems.org/read/chapter/20 for additional settings
+    gem.name         = "monkeyshines"
+    gem.summary      = %Q{A simple scraper for directed scrapes of APIs, feed or structured HTML.}
+    gem.description  = %Q{A simple scraper for directed scrapes of APIs, feed or structured HTML. Plays nicely with wuclan and wukong.}
+    gem.email        = "flip@infochimps.org"
+    gem.homepage     = "http://github.com/mrflip/monkeyshines"
+    gem.authors      = ["Philip (flip) Kromer"]
+    gem.add_dependency 'addressable'
+    gem.add_dependency 'uuid'
+    gem.add_dependency 'wukong'
+  end
+  Jeweler::GemcutterTasks.new
+rescue LoadError
+  puts "Jeweler (or a dependency) not available. Install it with: sudo gem install jeweler"
+end
+
+require 'spec/rake/spectask'
+Spec::Rake::SpecTask.new(:spec) do |spec|
+  spec.libs << 'lib' << 'spec'
+  spec.spec_files = FileList['spec/**/*_spec.rb']
+end
+
+Spec::Rake::SpecTask.new(:rcov) do |spec|
+  spec.libs << 'lib' << 'spec'
+  spec.pattern = 'spec/**/*_spec.rb'
+  spec.rcov = true
+end
+task :spec => :check_dependencies
+task :default => :spec
+
+begin
+  require 'reek/rake_task'
+  Reek::RakeTask.new do |t|
+    t.fail_on_error = true
+    t.verbose = false
+    t.source_files = ['lib/**/*.rb', 'examples/**/*.rb']
+  end
+rescue LoadError
+  task :reek do
+    abort "Reek is not available. In order to run reek, you must: sudo gem install reek"
+  end
+end
+
+begin
+  require 'roodi'
+  require 'roodi_task'
+  RoodiTask.new do |t|
+    t.verbose = false
+  end
+rescue LoadError
+  task :roodi do
+    abort "Roodi is not available. In order to run roodi, you must: sudo gem install roodi"
+  end
+end
+
+begin
+  require 'yard'
+  YARD::Rake::YardocTask.new do |yard|
+  end
+rescue LoadError
+  task :yardoc do
+    abort "YARD is not available. In order to run yardoc, you must: sudo gem install yard"
+  end
+end
+
+require 'rake/rdoctask'
+Rake::RDocTask.new do |rdoc|
+  require 'rdoc'
+  if File.exist?('VERSION')
+    version = File.read('VERSION')
+  else
+    version = ""
+  end
+
+  rdoc.options += [
+    '-SHN',
+    '-f', 'darkfish',  # use darkfish rdoc styler
+  ]
+  rdoc.rdoc_dir = 'rdoc'
+  rdoc.title = "edamame #{version}"
+  #
+  File.open(File.dirname(__FILE__)+'/.document').each{|line| rdoc.rdoc_files.include(line.chomp) }
+end
+
+require 'rake/rdoctask'
+Rake::RDocTask.new do |rdoc|
+  if File.exist?('VERSION.yml')
+    config = YAML.load(File.read('VERSION.yml'))
+    version = "#{config[:major]}.#{config[:minor]}.#{config[:patch]}"
+  else
+    version = ""
+  end
+
+  rdoc.rdoc_dir = 'rdoc'
+  rdoc.title = "monkeyshines #{version}"
+  rdoc.rdoc_files.include('README*')
+  rdoc.rdoc_files.include('lib/**/*.rb')
+end
+

data/VERSION

@@ -0,0 +1 @@
+0.0.2

data/examples/.gitignore

@@ -0,0 +1,4 @@
+facebook_stream
+friendster_html
+friendster_api
+myspace_api

data/examples/bulk_urls/scrape_bulk_urls.rb

@@ -0,0 +1,64 @@
+#!/usr/bin/env ruby
+require 'rubygems'
+require 'monkeyshines'
+require 'monkeyshines/runner'
+require 'pathname'
+
+#
+#
+#
+require 'wuclan/twitter'
+# un-namespace request classes.
+include Wuclan::Twitter::Scrape
+
+Monkeyshines::WORK_DIR = '/tmp'
+WORK_DIR = Pathname.new(Monkeyshines::WORK_DIR).realpath.to_s
+
+# ===========================================================================
+#
+# scrape_shorturls.rb --
+#
+# To scrape from a list of shortened urls:
+#
+#    ./shorturl_random_scrape.rb --from-type=FlatFileStore --from=request_urls.tsv
+#
+# To do a random scrape:
+#
+#    ./shorturl_random_scrape.rb --from-type=RandomUrlStream --base-url=tinyurl.com
+#       --base-url="http://tinyurl.com" --min-limit= --max-limit= --encoding_radix=
+#
+#
+opts = Trollop::options do
+  opt :log,            "Log to file instead of STDERR"
+  # input from file
+  opt :from,           "URI for scrape store to load from",            :type => String
+  opt :skip,           "Initial lines to skip",                        :type => Integer
+  # output storage
+  opt :cache_uri,      "URI for cache server",                         :type => String, :default => ':1978'
+  opt :chunk_time,     "Frequency to rotate chunk files (in seconds)", :type => Integer, :default => 60*60*4
+  opt :dest_dir,       "Filename base to store output. default ./work/ripd", :default => WORK_DIR+'/ripd'
+  opt :dest_pattern,   "Pattern for dump file output",                 :default => ":dest_dir/:date/:handle+:timestamp-:pid.tsv"
+  opt :into,           "URI for scrape store into",            :type => String
+end
+opts[:handle] ||= 'com.twitter'
+scrape_config = YAML.load(File.open(ENV['HOME']+'/.monkeyshines'))
+opts.merge! scrape_config
+
+# ******************** Log ********************
+if (opts[:log])
+  opts[:log] = (WORK_DIR+'/log/'+File.basename(opts[:from],'.tsv'))
+  $stdout = $stderr = File.open(opts[:log]+"-console.log", "a")
+end
+
+#
+# Execute the scrape
+#
+scraper = Monkeyshines::Runner.new(
+  :dest_store     => { :type => :conditional_store,
+    :cache => { :type => :tyrant_rdb_key_store, :uri => opts[:cache_uri] },
+    :store => opts.merge({ :type => :chunked_flat_file_store }), },
+    # :store => { :type => :flat_file_store, :filename => opts[:into] }, },
+  :request_stream => { :type => :base, :klass => Monkeyshines::ScrapeRequest,
+    :store => { :type => :flat_file_store, :filemode => 'r', :filename => opts[:from] } }
+  )
+scraper.run

data/examples/rename_tree/rename_hdp_tree.rb

@@ -0,0 +1,151 @@
+#!/usr/bin/env ruby19
+$: << ENV['HOME']+'/ics/rubygems/trollop-1.14/lib'
+$: << ENV['HOME']+'/ics/rubygems/log4r-1.0.5/src'
+require "monkeyshines/utils/logger"
+require "monkeyshines/utils/filename_pattern.rb"; include Monkeyshines::Utils
+require 'wukong'
+require 'fileutils'
+require 'trollop'
+
+#
+# This script demonstrates the use of FilenamePattern.
+#
+# The details are meaningless (it's a throwaway script I used to move to a more
+# unified naming scheme for scraped files), but I think it nicely demonstrates
+# how useful the FilenamePattern class can be.
+#
+
+opts = Trollop::options do
+  opt :root,    "base dir to move (tw0227, etc)", :required => true, :type => String
+  opt :go,      "actually do rename (otherwise do a dry run)"
+end
+
+# The tree to walk
+RIPD_ROOT = '/user/flip/ripd'
+
+#
+# Old files to rename
+#
+old_filename_pats = {
+  # "/user/flip/#{opts[:root]}/bundled/bundled_bundled_*/*.scrape.tsv" =>
+  #   ':any_id/bundled/bundled_bundled_:date/bundle+:timestamp.scrape.:ext',
+  # "/user/flip/#{opts[:root]}/bundled/bundled_fff_*/*fff_*-0*" => {
+  #   :patt => ':any_id/bundled/bundled_fff_:date-:any_id/:{flavor}_:date-:segment',
+  #   :toks => { :ext => 'tsv' } }
+  # "/user/flip/#{opts[:root]}/bundled/bundled_bundled_*/bundled-_20*.tsv" => {
+  #   :patt => ':any_id/bundled/bundled_bundled_:date/bundled-_:date.:ext',
+  #   :toks => { :flavor => 'bundled', :time => '000000' } }
+  # "/user/flip/#{opts[:root]}/bundled/bundled_idok_*/*idok_*-0*" => {
+  #   :patt => ':any_id/bundled/bundled_idok_0126_pt_0215-:any_id/:{flavor}_0126_pt_0215-:segment',
+  #   :toks => { :ext => 'tsv', :date => '20090215', :flavor => 'bundled_idok' } }
+  # "/user/flip/#{opts[:root]}/bundled/bundled_bundled_*/bundled-_*.tsv" =>
+  #   ':any_id/bundled/bundled_bundled_:date/bundled-_:date.:ext',
+  # '/user/flip/ripd/com.twitter.stream/hosebird-*' =>
+  #   '/user/flip/ripd/:handle/hosebird-:date-:time.:ext',
+  # "/user/flip/#{opts[:root]}/bundled/bundled_public_timeline_*/bundled_public_timeline_*.tsv" => {
+  #   :patt => ':any_id/bundled/bundled_public_timeline_:date/bundled_public_timeline_:date.:ext',
+  #   :toks => { :hostname => 'old+timeline' } }
+  # "/user/flip/#{opts[:root]}/bundled/bundled_public_timeline_*/bundled_public_timeline_*[0-9]" => {
+  #   :patt => ':any_id/bundled/bundled_public_timeline_:date/bundled_public_timeline_:date',
+  #   :toks => { :hostname => 'old+timeline', :ext => 'tsv' } }
+  "/user/flip/#{opts[:root]}/bundled/bundled_public_timeline_*/part-*[0-9]" => {
+    :patt => ':any_id/bundled/bundled_public_timeline_:date/part-:segment',
+    :toks => { :flavor => 'timeline', :ext => 'tsv' } }
+}
+
+#
+# How to template new filename
+#
+new_token_defaults = {
+  :dest_dir => RIPD_ROOT,
+  :pid      => '0',
+  :hostname => 'old',
+  :handle   => 'com.twitter'
+}
+new_filename_pat = FilenamePattern.new(
+  ':dest_dir/:handle_prefix/:handle/:date/:handle+:timestamp-:pid-:hostname+:flavor.:ext', new_token_defaults)
+
+MADE_DIR = { }
+#
+# Rename with logging and without overwriting
+#
+def rename_carefully old_file, new_filename, do_it=false
+  Log.info "%s%-87s\t=> %s" % [do_it ? "" : "DRY RUN ", old_file.path, new_filename]
+  return unless do_it
+  dirname = File.dirname(new_filename)
+  if !MADE_DIR[dirname] then Wukong::Dfs::HFile.mkdir_p(dirname) ; MADE_DIR[dirname] = true ; end
+  old_file.mv new_filename
+end
+
+#
+# Do this thing
+#
+old_filename_pats.each do |files_to_rename, old_filename_rule|
+  Log.info "Renaming files matching #{files_to_rename}"
+  if old_filename_rule.is_a? Hash
+    old_filename_pat = FilenamePattern.new(old_filename_rule[:patt])
+    more_toks = old_filename_rule[:toks] || { }
+  else
+    old_filename_pat = FilenamePattern.new(old_filename_rule)
+    more_toks = { }
+  end
+  #
+  # List files and rename
+  #
+  Wukong::Dfs.list_files(files_to_rename).each do |hdfs_file|
+    filename_tokens = old_filename_pat.recognize(hdfs_file.path, { :segment => '\d+', :flavor => '\w+'}) or next
+    filename_tokens.merge!(more_toks)
+    if (filename_tokens[:timestamp].blank?) && (!filename_tokens[:date].blank?)
+      timepart = filename_tokens[:time].blank? ? ('0'+filename_tokens[:segment]) : filename_tokens[:time]
+      filename_tokens[:timestamp] = filename_tokens[:date] + (timepart || '000000')
+    end
+    new_filename = new_filename_pat.make(filename_tokens)
+    rename_carefully hdfs_file, new_filename, opts[:go]
+  end
+end
+
+
+  # '/user/flip/pkgd/user/flip/tw0227/bundled/bundled_bundled_*.bz2' =>
+  # { :pat_str => ':any_id/bundled/bundled_bundled_:date.:ext',
+  #   :toks    => { :ext => '.tsv.bz2' } },
+
+
+
+
+# -rw-r--r--   3 flip supergroup 2055552674 2009-02-18 13:18 /user/flip/tw0218/bundled/bundled_fff_20090126-00000/bundled_fff_20090126-00000
+# -rw-r--r--   3 flip supergroup 2328853732 2009-02-18 13:08 /user/flip/tw0218/bundled/bundled_fff_20090126-00001/bundled_fff_20090126-00001
+# -rw-r--r--   3 flip supergroup  630259166 2009-02-18 13:55 /user/flip/tw0218/bundled/bundled_idok_0126_pt_0215-00053/bundled_idok_0126_pt_0215-00053
+# -rw-r--r--   3 flip supergroup 1714844022 2009-02-17 12:17 /user/flip/tw0218/bundled/bundled_bundled_20090118/bundled-_20090118.tsv
+# -rw-r--r--   3 flip supergroup 4053904382 2009-02-17 12:18 /user/flip/tw0218/bundled/bundled_bundled_20090119/bundled-_20090119.tsv
+# -rw-r--r--   3 flip supergroup 3612882035 2009-02-17 12:36 /user/flip/tw0218/bundled/bundled_bundled_20090120/bundled-_20090120.tsv
+# -rw-r--r--   3 flip supergroup 4309364084 2009-02-17 12:42 /user/flip/tw0218/bundled/bundled_bundled_20090121/bundled-_20090121.tsv
+# -rw-r--r--   3 flip supergroup 4375598899 2009-02-17 12:49 /user/flip/tw0218/bundled/bundled_bundled_20090122/bundled-_20090122.tsv
+# -rw-r--r--   3 flip supergroup 2414994564 2009-02-17 12:56 /user/flip/tw0218/bundled/bundled_bundled_20090123/bundled-_20090123.tsv
+# -rw-r--r--   3 flip supergroup        612 2009-02-17 13:01 /user/flip/tw0218/bundled/bundled_bundled_20090125/bundled-_20090125.tsv
+# -rw-r--r--   3 flip supergroup 1120007814 2009-02-17 13:03 /user/flip/tw0218/bundled/bundled_bundled_20090204/bundled-_20090204.tsv
+# -rw-r--r--   3 flip supergroup  534874538 2009-02-17 13:06 /user/flip/tw0218/bundled/bundled_bundled_20090205/bundled-_20090205.tsv
+# -rw-r--r--   3 flip supergroup  404436617 2009-02-17 13:07 /user/flip/tw0218/bundled/bundled_bundled_20090206/bundled-_20090206.tsv
+# -rw-r--r--   3 flip supergroup  359037171 2009-02-17 13:08 /user/flip/tw0218/bundled/bundled_bundled_20090207/bundled-_20090207.tsv
+# -rw-r--r--   3 flip supergroup  332668257 2009-02-17 13:08 /user/flip/tw0218/bundled/bundled_bundled_20090208/bundled-_20090208.tsv
+# -rw-r--r--   3 flip supergroup  304904205 2009-02-17 13:09 /user/flip/tw0218/bundled/bundled_bundled_20090209/bundled-_20090209.tsv
+# -rw-r--r--   3 flip supergroup  295217809 2009-02-17 13:09 /user/flip/tw0218/bundled/bundled_bundled_20090210/bundled-_20090210.tsv
+# -rw-r--r--   3 flip supergroup  257376099 2009-02-17 13:10 /user/flip/tw0218/bundled/bundled_bundled_20090211/bundled-_20090211.tsv
+# -rw-r--r--   3 flip supergroup  180147925 2009-02-17 13:10 /user/flip/tw0218/bundled/bundled_bundled_20090212/bundled-_20090212.tsv
+# -rw-r--r--   3 flip supergroup  150611510 2009-02-17 13:11 /user/flip/tw0218/bundled/bundled_bundled_20090214/bundled-_20090214.tsv
+# -rw-r--r--   3 flip supergroup  154181256 2009-02-17 13:11 /user/flip/tw0218/bundled/bundled_bundled_20090215/bundled-_20090215.tsv
+# -rw-r--r--   3 flip supergroup   74288574 2009-02-17 13:12 /user/flip/tw0218/bundled/bundled_bundled_20090216/bundled-_20090216.tsv
+# -rw-r--r--   3 flip supergroup    2006507 2009-02-17 13:12 /user/flip/tw0218/bundled/bundled_bundled_20090217/bundled-_20090217.tsv
+# -rw-r--r--   3 flip supergroup  232422855 2009-02-17 13:11 /user/flip/tw0219/bundled/bundled_bundled_20090213/bundled-_20090213.tsv
+# -rw-r--r--   3 flip supergroup  558290288 2009-02-27 16:52 /user/flip/tw0227/bundled/bundled_public_timeline_20090227/part-00004
+# -rw-r--r--   3 flip supergroup 1130590440 2009-02-27 16:52 /user/flip/tw0227/bundled/bundled_public_timeline_20090227/part-00009
+# -rw-r--r--   3 flip supergroup  523600649 2009-02-27 16:52 /user/flip/tw0227/bundled/bundled_public_timeline_20090227/part-00025
+# -rw-r--r--   3 flip supergroup  565480025 2009-02-27 16:52 /user/flip/tw0227/bundled/bundled_public_timeline_20090227/part-00028
+# -rw-r--r--   3 flip supergroup  566689087 2009-02-27 16:52 /user/flip/tw0227/bundled/bundled_public_timeline_20090227/part-00033
+# -rw-r--r--   3 flip supergroup  545436522 2009-02-27 16:52 /user/flip/tw0227/bundled/bundled_public_timeline_20090227/part-00036
+# -rw-r--r--   3 flip supergroup  563565767 2009-02-27 16:52 /user/flip/tw0227/bundled/bundled_public_timeline_20090227/part-00039
+# -rw-r--r--   3 flip supergroup  544478849 2009-02-27 16:52 /user/flip/tw0227/bundled/bundled_public_timeline_20090227/part-00046
+# -rw-r--r--   3 flip supergroup  566687292 2009-02-27 16:52 /user/flip/tw0227/bundled/bundled_public_timeline_20090227/part-00055
+
+# -rw-r--r--   3 flip supergroup  561407978 2009-03-03 01:28 /user/flip/tw0227/bundled/bundled_public_timeline_20090227-0301/bundled_public_timeline_20090227-0301.tsv
+# -rw-r--r--   3 flip supergroup  559109582 2009-03-03 01:22 /user/flip/tw0227/bundled/bundled_public_timeline_20090227-0302/bundled_public_timeline_20090227-0302.tsv
+# -rw-r--r--   3 flip supergroup 1126272691 2009-03-01 04:53 /user/flip/tw0227/bundled/bundled_public_timeline_20090227-27_28/bundled_public_timeline_20090227-27_28.tsv