extraloop-redis-storage 0.0.6 → 0.0.7

data/History.txt

@@ -1,3 +1,7 @@
+== 0.0.7  / 2012-11-03
+  * Datasets can now be pushed to Google Fusion tables 
+  * Added support for YAML export 
+
 == 0.0.6  / 2012-26-02
 
   * Added CSV export

data/README.rdoc

@@ -4,8 +4,8 @@
 
 Persistence layer for the {ExtraLoop}[https://github.com/afiore/extraloop] data extraction toolkit.
 This module is implemented as a wrapper around {Ohm}[http://ohm.keyvalue.org], an object-hash mapping library which 
-makes easy storing structured data into Redis. It comes with a convinent command line tool, which allows to 
-list, filter, delete, and export harvested datasets.
+makes easy storing structured data into Redis. Includes a convinent command line tool that allows to 
+list, filter, and delete harvested datasets, as well as exporting them on local files or remote data stores (i.e Google Fusion tables).
 
 == Installation 
 
@@ -33,46 +33,49 @@ with the +set_storage+ method: a helper method that allows to specify how the sc
    .run()
 
 At each scraper run, the ExtraLoop storage module internally instantiates a 
-session (see <code>ExtraLoop::Storage::ScrapingSession</code>) and link the extracted records to it.
-The +AmazonReview+ instances extracted and stored in the example above, can in fact be fetched by calling
-Ohm's +find+ with the session id as argument.
+session (see <code>ExtraLoop::Storage::ScrapingSession</code>) and associates the extracted records to it.
+The `AmazonReview` records just created, can now be accessed by calling the `#records` metod on scraper session object.
 
- reviews = AmazonReview.find :session_id => scraper.session
+ reviews = scraper.session.records
 
-The same set of reviews can alternatively be retrieved by calling the +record+ method on the scraping 
-session instance:
+=== #set_storage 
 
- reviews = scraper.session.records AmazonReview
+The +set_storage+ method accepts the following arguments:
 
-
-=== The #set_storage method
-
-The +set_storage+ method can be called with the following arguments:
-
-* _model_ A Ruby constant specifying the model to be used for storing the extracted data .
+* _model_ A Ruby constant or a symbol specifying the model to be used for storing the extracted data. If a symbol is passed, it is assumed that a model does not exist and the storage module dynamically generates one by subclassing <code>ExtraLoop::Storage::Record</code>. 
 * _session_title_ A human readable title for the extracted dataset (optional).
 
 == Command line interface 
 
-Once installed, the gem will also add to your system path the +extraloop+ executable, a command line interface to the datasets harvested through extraloop.
+Once installed, the gem will also add to your system path the +extraloop+ executable: a command line interface to the datasets harvested through ExtraLoop.
 A list of datasets can be obtained by running:
 
- extraloop datastore list:
+ extraloop datastore list
 
 This will generate a table like the following one:
-<code>
+
   id | title                              | model           | records
  --------------------------------------------------------------------
   48 | 1330106699 GoogleNewsStory Dataset | GoogleNewsStory | 110    
   49 | 1330106948 AmazonReview Dataset    | AmazonReview    | 0      
   51 | 1330107087 GoogleNewsStory Dataset | GoogleNewsStory | 110    
   52 | 1330111630 AmazonReview Dataset    | AmazonReview    | 10     
-</code>
 
-Datasets can by removed using the +delete+ subcommand:
+
+Datasets can be removed using the +delete+ subcommand:
 
  extraloop datastore delete [id]
 
 Where +id+ is either a single scraping session id, or a session id range (e.g. 48..52).
-Finally, the +export+ subcommand allows to export one or several datasets into a JSON or CSV documents.
-Please refer to the executable inline help (<code>extraloop datastore help [command]</code>) for more usage information.
+
+From the Redis datastore, ExtraLoop datasets can be exported to disk as CSV, JSON, or YAML documents:
+
+  extraloop datastore export 51..52 -f csv
+
+Similarly, stored datasets can be uploaded to a remote datastore:
+
+  extraloop datastore push 51..48 fusion_tables -c google_username:password
+
+While Google's Fusion Tables is currently the only one implemented, support for other remote datastores (e.g.
+[couchDB](http://couchdb.apache.org/), , [cartoDB](http://cartodb.com) ), and [CKAN Webstore](http://wiki.ckan.org/Webstore) will be added soon.
+

data/bin/extraloop

@@ -2,7 +2,6 @@
 $: << File.realpath(File.dirname(File.dirname(__FILE__))) + "/lib/extraloop"
 
 require 'thor'
-require 'pry'
 require 'fileutils'
 require 'thor/group'
 require 'redis-storage'
@@ -11,6 +10,15 @@ class DataStoreCommand < Thor
 
   ExtraLoop::Storage::autoload_models
 
+  class << self
+    def parse_config
+      config_file = File.join(Etc.getpwuid.dir, '.extraloop.yml')
+      File.exist?(config_file) && YAML::load_file(config_file) or {}
+    end
+  end
+
+  @@config = parse_config
+
   @@sessions = ExtraLoop::Storage::ScrapingSession.all
   @@redis = Ohm.redis
 
@@ -18,16 +26,16 @@ class DataStoreCommand < Thor
       "d" => :delete,
       "e" => :export
 
-  desc "list [sessions]", "List harvested datasets filtering by session id range (e.g '25..50')"
+  desc "list [sessions]", "Lists harvested datasets filtering by session id range (e.g '25..50')"
   def list(sessions=nil)
     data = (filter sessions).map { |session| [ session.id, session.title, session.model && session.model.name, session.model && session.records.size ]}
     $stdout.puts tabularize(%w[id title model records], data)
   end
 
-  desc "delete [sessions]", "Remove datasets by session id or session id range"
+  desc "delete [sessions]", "Removes datasets by session id or session id range"
   def delete(sessions)
     deleted = 0
-    (filter sessions).each { |session| (session.delete && session.records.each(&:delete) ) && deleted += 1 }
+    (filter sessions).each { |session| session.delete && session.records.each(&:delete) && deleted += 1 }
     $stderr.puts "\n => #{deleted > 0 && deleted or 'No' } record#{'s' if deleted > 1} deleted \n\n" 
     list
   end
@@ -44,15 +52,33 @@ class DataStoreCommand < Thor
     format = options[:format]
     dir = options[:directory]
     exception = DataStoreCommand::Exceptions::FormatNotImplemented.new "Format not supported #{format}"
-    raise exception unless %w[json csv].include? format
+    raise exception unless %w[json csv yaml].include? format
     FileUtils.mkdir(dir) unless File.exists? dir
 
-    (filter sessions).each do |session|
+    filter(sessions).each do |session|
       filename, data = *[ "#{session.id}_#{session.title.gsub(/\s/,"_")}", session.send("to_#{format}")]
       File.open("#{dir}/#{filename}.#{format}", "w") { |f| f.write data }
     end
   end
 
+  desc "push [sessions] [remote_store]", "Uploads one or several datasets to a remote data store"
+  method_option :schema, :type => 'hash', :aliases => "-s"
+  method_option :credentials, :type => 'string', :aliases => "-a"
+
+  def push(sessions, store_type=:fusion_tables)
+
+    filter(sessions).each do |session|
+      store_type = store_type.to_sym
+      begin
+        credentials = options.fetch('credentials', @@config[:datastore] && @@config[:datastore][:credentials] && @@config[:datastore][store_type]).split(':')
+      rescue NoMethodError
+        abort "Cannot find credentials for remote datastore.\nPlease specify them using the --credential switch (e.g. 'andrea:mypassword')"
+      end
+      datastore = ExtraLoop::Storage::RemoteStore::get_transport(store_type, credentials)
+      datastore.push session
+    end
+  end
+
   # override default banner
   def self.banner(task, namespace = true, subcommand = false)
     "datastore#{task.formatted_usage(self, true, subcommand).gsub(/data_store_command/,'')}"
@@ -65,7 +91,6 @@ class DataStoreCommand < Thor
       exception = DataStoreCommand::Exceptions::FileNotFound.new "cannot find #{path}"
       raise exception unless File.exists?(path)
       (File.directory? path) && Dir["#{path}/*.rb"] or path
-
     end.flatten
 
     files.each { |file| require "./#{file}" }
@@ -79,7 +104,6 @@ class DataStoreCommand < Thor
     else
       @@sessions
     end
-
   end
 
   def tabularize(headers, data)
@@ -100,6 +124,7 @@ end
 
 class DataStoreCommand::Exceptions
   class FormatNotImplemented < StandardError; end
+  class UnknownDatastore < StandardError; end
   class FileNotFound < StandardError; end
 end
 

data/examples/amazon_reviews.rb

@@ -1,5 +1,6 @@
 require "rubygems"
 require "date"
+require "pry"
 require "extraloop"
 require "../lib/extraloop/redis-storage.rb"
 require "./lib/models/amazon_review.rb"

data/lib/extraloop/redis-storage/model.rb

@@ -10,4 +10,7 @@ class ExtraLoop::Storage::Model < Ohm::Model
       memo.merge(attribute => send(attribute)) 
     })
   end
+  def to_yaml
+    to_hash.to_yaml
+  end
 end

data/lib/extraloop/redis-storage/record.rb

@@ -22,6 +22,10 @@ class ExtraLoop::Storage::Record < Ohm::Model
     })
   end
 
+  def to_yaml
+    to_hash.to_yaml
+  end
+
   def validate
     assert_present :session
   end

data/lib/extraloop/redis-storage/remote_store/fusion_tables.rb

@@ -0,0 +1,41 @@
+class ExtraLoop::Storage::FusionTables
+  @@connection = nil
+
+  def initialize(credentials, options={})
+    @options = options
+    @credentials = credentials
+    @api = connect
+  end
+
+  def push(session)
+    dataset = session.to_hash
+    records = dataset[:records]
+    title   = dataset[:title].gsub(/\sDataset/,'')
+    schema = make_schema(records.first)
+
+    table = @api.create_table("Dataset #{title}", schema)
+    table.insert records
+  end
+
+  private
+  def make_schema(record)
+    defaults = {
+      'session_id' => 'number'
+    }
+
+    schema = defaults.merge(@options.fetch :schema, {})
+
+    record.keys.
+      reject { |key| schema.keys.include?(key) }.
+      map    { |key| {:name => key.to_s, :type => 'string'} }.
+      concat(schema.map { |field, type| {:name => field.to_s, :type => type }})
+  end
+
+  def connect
+    return @@connection if @@connection
+
+    @@connection = GData::Client::FusionTables.new
+    @@connection.clientlogin(*@credentials)
+    @@connection
+  end
+end

data/lib/extraloop/redis-storage/remote_store.rb

@@ -0,0 +1,13 @@
+# Base class for pushing Extraloop datasets from the local Redis
+# store to remote ones (e.g. Google Fusion tables, Buzzdata, Cartodb)
+
+$: << path = File.dirname(__FILE__) + '/remote_store'
+Dir["#{path}/*.rb"].each { |store_adapter| require store_adapter }
+
+
+class ExtraLoop::Storage::RemoteStore
+  def self.get_transport(datastore, credentials, options={})
+    classname = datastore.to_s.gsub(/^.|_./) { |chars| chars.split("").last.upcase }
+    ExtraLoop::Storage.const_get(classname).new(credentials, options) if ExtraLoop::Storage.const_defined?(classname)
+  end
+end

data/lib/extraloop/redis-storage/scraping_session.rb

@@ -1,7 +1,5 @@
 class ExtraLoop::Storage::ScrapingSession < Ohm::Model
 
-  BOM = "\377\376" #Byte Order Mark
-
   include Ohm::Boundaries
   include Ohm::Timestamping
   include Ohm::Callbacks
@@ -9,22 +7,20 @@ class ExtraLoop::Storage::ScrapingSession < Ohm::Model
   attribute :title
   reference :model, ExtraLoop::Storage::Model
   
-
-
   def records(params={})
     klass = if Object.const_defined?(model.name)
-        Object.const_get(model.name)
-      else
-        dynamic_class = Class.new(ExtraLoop::Storage::Record) do
-          # override default to_hash so that it will return the Redis hash
-          # internally stored by Ohm
-          def to_hash
-            Ohm.redis.hgetall self.key
-          end
+      Object.const_get(model.name)
+    else
+      dynamic_class = Class.new(ExtraLoop::Storage::Record) do
+        # override default to_hash so that it will return the Redis hash
+        # internally stored by Ohm
+        def to_hash
+          Ohm.redis.hgetall self.key
         end
+      end
 
-        Object.const_set(model.name, dynamic_class)
-        dynamic_class
+      Object.const_set(model.name, dynamic_class)
+      dynamic_class
     end
 
     # set a session index, so that Ohm finder will work
@@ -56,4 +52,8 @@ class ExtraLoop::Storage::ScrapingSession < Ohm::Model
     data = [header].concat _records.map(&:values)
     output = data.map { |cells| CSV.generate_line cells }.join
   end
+
+  def to_yaml
+    to_hash.to_yaml
+  end
 end

data/lib/extraloop/redis-storage.rb

@@ -1,10 +1,19 @@
+require "rubygems"
 require "json"
+require "yaml"
 require "rubygems"
 require "redis"
 require "ohm"
 require "ohm/contrib"
 require "extraloop"
 
+begin
+  gem "fusion_tables", "~> 0.3.1"
+  require "fusion_tables"
+rescue Gem::LoadError
+end
+
+
 base_path = File.realpath(File.dirname(__FILE__))
 $: << "#{base_path}"
 
@@ -12,7 +21,7 @@ require "scraper_base"
 
 module ExtraLoop
   module Storage
-    VERSION ||= "0.0.1"
+    VERSION ||= "0.0.7"
 
     def self.connect(*args)
       Ohm.connect(*args)
@@ -26,10 +35,13 @@ module ExtraLoop
 end
 
 autoload :CSV, 'csv'
-autoload :Iconv, 'iconv'
-ExtraLoop::Storage.autoload :Record, "#{base_path}/redis-storage/record.rb"
-ExtraLoop::Storage.autoload :ScrapingSession, "#{base_path}/redis-storage/scraping_session.rb"
-ExtraLoop::Storage.autoload :Model,  "#{base_path}/redis-storage/model.rb"
-ExtraLoop::Storage.autoload :DatasetFactory,  "#{base_path}/redis-storage/dataset_factory.rb"
+autoload :Etc, 'etc'
+
+base_path << "/redis-storage"
 
+ExtraLoop::Storage.autoload :Record, "#{base_path}/record.rb"
+ExtraLoop::Storage.autoload :ScrapingSession, "#{base_path}/scraping_session.rb"
+ExtraLoop::Storage.autoload :Model,  "#{base_path}/model.rb"
+ExtraLoop::Storage.autoload :DatasetFactory,  "#{base_path}/dataset_factory.rb"
+ExtraLoop::Storage.autoload :RemoteStore, "#{base_path}/remote_store.rb"
 

data/lib/extraloop/scraper_base.rb

@@ -1,11 +1,13 @@
 class ExtraLoop::ScraperBase
   attr_reader :session
 
-  def set_storage(model, title=nil)
+
+  def set_storage(model, title=nil, options={})
     collection_name = "#{Time.now.to_i} #{model.to_s} Dataset"
     title ||= collection_name
 
     @model = model_klass = model.respond_to?(:new) && model || ExtraLoop::Storage::DatasetFactory.new(model.to_sym, @extractor_args.map(&:first)).get_class
+
     log_session! title
 
     on :data do |results|

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: extraloop-redis-storage
 version: !ruby/object:Gem::Version
-  version: 0.0.6
+  version: 0.0.7
   prerelease: 
 platform: ruby
 authors:
@@ -9,11 +9,11 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2012-02-26 00:00:00.000000000Z
+date: 2012-03-11 00:00:00.000000000Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: extraloop
-  requirement: &10773840 !ruby/object:Gem::Requirement
+  requirement: &14542580 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ~>
@@ -21,10 +21,10 @@ dependencies:
         version: 0.0.3
   type: :runtime
   prerelease: false
-  version_requirements: *10773840
+  version_requirements: *14542580
 - !ruby/object:Gem::Dependency
   name: ohm
-  requirement: &10773380 !ruby/object:Gem::Requirement
+  requirement: &14542000 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ~>
@@ -32,10 +32,10 @@ dependencies:
         version: 0.1.3
   type: :runtime
   prerelease: false
-  version_requirements: *10773380
+  version_requirements: *14542000
 - !ruby/object:Gem::Dependency
   name: ohm-contrib
-  requirement: &10772860 !ruby/object:Gem::Requirement
+  requirement: &14507840 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ~>
@@ -43,10 +43,10 @@ dependencies:
         version: 0.1.2
   type: :runtime
   prerelease: false
-  version_requirements: *10772860
+  version_requirements: *14507840
 - !ruby/object:Gem::Dependency
   name: thor
-  requirement: &10772260 !ruby/object:Gem::Requirement
+  requirement: &14507360 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - =
@@ -54,10 +54,21 @@ dependencies:
         version: 0.14.6
   type: :runtime
   prerelease: false
-  version_requirements: *10772260
+  version_requirements: *14507360
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: &14506900 !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: *14506900
 - !ruby/object:Gem::Dependency
   name: rspec
-  requirement: &10771600 !ruby/object:Gem::Requirement
+  requirement: &14506220 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ~>
@@ -65,10 +76,10 @@ dependencies:
         version: 2.7.0
   type: :development
   prerelease: false
-  version_requirements: *10771600
+  version_requirements: *14506220
 - !ruby/object:Gem::Dependency
   name: rr
-  requirement: &10771000 !ruby/object:Gem::Requirement
+  requirement: &14505600 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ~>
@@ -76,10 +87,10 @@ dependencies:
         version: 1.0.4
   type: :development
   prerelease: false
-  version_requirements: *10771000
+  version_requirements: *14505600
 - !ruby/object:Gem::Dependency
   name: pry
-  requirement: &10770520 !ruby/object:Gem::Requirement
+  requirement: &14505020 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ~>
@@ -87,8 +98,10 @@ dependencies:
         version: 0.9.7.4
   type: :development
   prerelease: false
-  version_requirements: *10770520
-description: Redis+Ohm based storage for data sets extracted using the ExtraLoop toolkit.
+  version_requirements: *14505020
+description: Redis-based Persistence layer for the ExtraLoop data extraction toolkit.
+  Includes a convinent command line tool allowing to list, filter, delete, and export
+  harvested datasets
 email: andrea.giulio.fiore@googlemail.com
 executables:
 - extraloop
@@ -105,6 +118,8 @@ files:
 - lib/extraloop/redis-storage/dataset_factory.rb
 - lib/extraloop/redis-storage/model.rb
 - lib/extraloop/redis-storage/record.rb
+- lib/extraloop/redis-storage/remote_store.rb
+- lib/extraloop/redis-storage/remote_store/fusion_tables.rb
 - lib/extraloop/redis-storage/scraping_session.rb
 - lib/extraloop/scraper_base.rb
 - spec/dataset_factory_spec.rb
@@ -125,6 +140,9 @@ required_ruby_version: !ruby/object:Gem::Requirement
   - - ! '>='
     - !ruby/object:Gem::Version
       version: '0'
+      segments:
+      - 0
+      hash: 1448249409185434738
 required_rubygems_version: !ruby/object:Gem::Requirement
   none: false
   requirements:
@@ -138,4 +156,3 @@ signing_key:
 specification_version: 2
 summary: Redis storage for Extraloop.
 test_files: []
-has_rdoc: