taps2 0.5.5 → 0.6.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 33670f0428acea1fd1b319fa7709121d06e1ca44
-  data.tar.gz: 62f305882ef883472629bf9f72ed41f8efcf11ab
+  metadata.gz: bd7e00f8e9f2311fc7b0d8e990cb6d225978e39b
+  data.tar.gz: ab2788206704d5cb6ec72811192e2584439eca14
 SHA512:
-  metadata.gz: a6f3f8eeea90f1da98a93213866b1549464cb3058accade4528cb6130854b1634e25a2a4501d64ac3633c1eb9a8f857791db57c6bf7b59a404c03fe28f66248e
-  data.tar.gz: 36ae8502a3165c24d1bd85cc3e80eebe89a14f77cf6d75deba118e565727c79da28296ae23eb489c87b9765d8dd3f3879351880b88ccefe70c1548e228eb5b4a
+  metadata.gz: e1fe890ae878b3f52e33d9dbf88caaa32eb51b7672a8967da9cc2a5c408ef6e877f8e0c8f5698ad9a095ece0cdaa09252bc566021a5b1293633045faa595e467
+  data.tar.gz: 10c5c43cfe50ac13c0428d62c9c2e007d2f2481240040827c47a301ba3af14e008a3dd972e52bf4256c99149e179a3c77f162db466bfb1016a72c27f013e5b16

data/{README.rdoc → README.md}

@@ -1,25 +1,25 @@
-= Taps (2) -- simple database import/export app
+# Taps (2) -- simple database import/export app
 
 A simple database agnostic import/export app to transfer data to/from a remote database.
 
-*Forked and updated* with fixes and improvements. Integrates fixes and updates from {taps-taps}(https://github.com/wijet/taps) and {tapsicle}(https://github.com/jiffyondemand/tapsicle) forks.
+*Forked and updated* with fixes and improvements. Integrates fixes and updates from [taps-taps](https://github.com/wijet/taps) and [tapsicle](https://github.com/jiffyondemand/tapsicle) forks.
 
-== Installation
+## Installation
 
 Renamed gem
 
-  $ gem install taps2
+    $ gem install taps2
 
 By default, Taps will attempt to create a SQLite3 database for sessions. Unless you specify a different database type, you'll need to install SQLite 3. (See _Environment Variables_ for alternative session databases.)
 
-  $ gem install sqlite3
+    $ gem install sqlite3
 
 Install the gems to support databases you want to work with, such as MySQL or PostgreSQL.
 
-  $ gem install mysql2
-  $ gem install pg
+    $ gem install mysql2
+    $ gem install pg
 
-== Configuration: Environment Variables
+## Configuration: Environment Variables
 
 _All environment variables are optional._
 
@@ -37,40 +37,42 @@ The `NO_DUMP_MARSHAL_ERRORS` variable allows you to disable dumping of marshalle
 
 The `NO_DEFLATE` variable allows you to disable gzip compression (`Rack::Deflater`) on the server.
 
-== Usage: Server
+## Usage: Server
 
 Here's how you start a taps server
 
-  $ taps server postgres://localdbuser:localdbpass@localhost/dbname httpuser httppassword
+    $ taps2 server postgres://localdbuser:localdbpass@localhost/dbname httpuser httppassword
 
 You can also specify an encoding in the database url
 
-  $ taps server mysql://localdbuser:localdbpass@localhost/dbname?encoding=latin1 httpuser httppassword
+    $ taps2 server mysql://localdbuser:localdbpass@localhost/dbname?encoding=latin1 httpuser httppassword
 
-== Usage: Client
+## Usage: Client
 
 When you want to pull down a database from a taps server
 
-  $ taps pull postgres://dbuser:dbpassword@localhost/dbname http://httpuser:httppassword@example.com:5000
+    $ taps2 pull postgres://dbuser:dbpassword@localhost/dbname http://httpuser:httppassword@example.com:5000
 
 or when you want to push a local database to a taps server
 
-  $ taps push postgres://dbuser:dbpassword@localhost/dbname http://httpuser:httppassword@example.com:5000
+    $ taps2 push postgres://dbuser:dbpassword@localhost/dbname http://httpuser:httppassword@example.com:5000
 
 or when you want to transfer a list of tables
 
-  $ taps push postgres://dbuser:dbpassword@localhost/dbname http://httpuser:httppassword@example.com:5000 --tables logs,tags
+    $ taps2 push postgres://dbuser:dbpassword@localhost/dbname http://httpuser:httppassword@example.com:5000 --tables logs,tags
 
 or when you want to transfer tables that start with a word
 
-  $ taps push postgres://dbuser:dbpassword@localhost/dbname http://httpuser:httppassword@example.com:5000 --filter '^log_'
+    $ taps2 push postgres://dbuser:dbpassword@localhost/dbname http://httpuser:httppassword@example.com:5000 --filter '^log_'
 
-== Troubleshooting
+## Troubleshooting
 
 * "Error: invalid byte sequence for encoding" can be resolved by adding `encoding` to database URI (https://github.com/ricardochimal/taps/issues/110)
-  * *Example:* `taps server mysql://root@localhost/example_database?encoding=UTF8 httpuser httppassword`
+  * *Example:* `taps2 server mysql://root@localhost/example_database?encoding=UTF8 httpuser httppassword`
+* SQLite3 database URI may require three slashes (e.g. `sqlite3:///path/to/file.db`)
+  * Make sure to use an absolute/full path to the file on the server
 
-== Known Issues
+## Known Issues
 
 * Foreign key constraints get lost in the schema transfer
 * Indexes may drop the "order" (https://github.com/ricardochimal/taps/issues/111)
@@ -78,14 +80,19 @@ or when you want to transfer tables that start with a word
 * Tables without primary keys will be incredibly slow to transfer. This is due to it being inefficient having large offset values in queries.
 * Multiple schemas are currently not supported (https://github.com/ricardochimal/taps/issues/97)
 * Taps does not drop tables when overwriting database (https://github.com/ricardochimal/taps/issues/94)
+* Oracle database classes not fully supported (https://github.com/ricardochimal/taps/issues/89)
+* Some blank default values may be converted to NULL in MySQL table schemas (https://github.com/ricardochimal/taps/issues/88)
+* Conversion of column data types can cause side effects when going from one database type to another
+  * MySQL `bigint` converts to PostgreSQL `string` (https://github.com/ricardochimal/taps/issues/77)
+* Passwords in database URI can cause issues with special characters (https://github.com/ricardochimal/taps/issues/74)
 
-== Feature Requests
+## Feature Requests
 
 * Allow a single Taps server to serve data from different databases (https://github.com/ricardochimal/taps/issues/103)
 
-== Meta
+## Meta
 
-Maintained by {Joel Van Horn}(http://github.com/joelvh)
+Maintained by [Joel Van Horn](http://github.com/joelvh)
 
 Written by Ricardo Chimal, Jr. (ricardo at heroku dot com) and Adam Wiggins (adam at heroku dot com)
 

data/bin/{schema → schema2}

@@ -1,14 +1,15 @@
 #!/usr/bin/env ruby
 
 require 'rubygems'
-gem 'sequel', '~> 3.20.0'
 
-$:.unshift File.dirname(__FILE__) + '/../lib'
+gem 'sequel', '~> 4.0'
+
+$LOAD_PATH.unshift File.dirname(__FILE__) + '/../lib'
 
 require 'taps/schema'
 
-cmd = ARGV.shift.strip rescue ''
-database_url = ARGV.shift.strip rescue ''
+cmd = ARGV.shift.to_s.strip
+database_url = ARGV.shift.to_s.strip
 
 def show_usage_and_exit
   puts <<EOTXT
@@ -35,12 +36,20 @@ when 'indexes'
 when 'indexes_individual'
   puts Taps::Schema.indexes_individual(database_url)
 when 'load_indexes'
-  filename = ARGV.shift.strip rescue ''
-  indexes = File.read(filename) rescue show_usage_and_exit
+  filename = ARGV.shift.to_s.strip
+  indexes = begin
+              File.read(filename)
+            rescue StandardError
+              show_usage_and_exit
+            end
   Taps::Schema.load_indexes(database_url, indexes)
 when 'load'
-  filename = ARGV.shift.strip rescue ''
-  schema = File.read(filename) rescue show_usage_and_exit
+  filename = ARGV.shift.to_s.strip
+  schema = begin
+             File.read(filename)
+           rescue StandardError
+             show_usage_and_exit
+           end
   Taps::Schema.load(database_url, schema)
 when 'reset_db_sequences'
   Taps::Schema.reset_db_sequences(database_url)

data/bin/{taps → taps2}

@@ -1,6 +1,6 @@
 #!/usr/bin/env ruby
 
-$:.unshift File.dirname(__FILE__) + '/../lib'
+$LOAD_PATH.unshift File.dirname(__FILE__) + '/../lib'
 require 'taps/cli'
 
 Taps::Cli.new(ARGV.dup).run

data/lib/taps/chunksize.rb

@@ -15,7 +15,7 @@ class Taps::Chunksize
   end
 
   def reset_chunksize
-    @chunksize = (retries <= 1) ? 10 : 1
+    @chunksize = retries <= 1 ? 10 : 1
   end
 
   def diff
@@ -24,7 +24,11 @@ class Taps::Chunksize
 
   def time_in_db=(t)
     @time_in_db = t
-    @time_in_db = @time_in_db.to_f rescue 0.0
+    @time_in_db = begin
+                    @time_in_db.to_f
+                  rescue
+                    0.0
+                  end
   end
 
   def time_delta
@@ -36,16 +40,16 @@ class Taps::Chunksize
 
   def calc_new_chunksize
     new_chunksize = if retries > 0
-      chunksize
-    elsif diff > 3.0
-      (chunksize / 3).ceil
-    elsif diff > 1.1
-      chunksize - 100
-    elsif diff < 0.8
-      chunksize * 2
-    else
-      chunksize + 100
-    end
+                      chunksize
+                    elsif diff > 3.0
+                      (chunksize / 3).ceil
+                    elsif diff > 1.1
+                      chunksize - 100
+                    elsif diff < 0.8
+                      chunksize * 2
+                    else
+                      chunksize + 100
+                    end
     new_chunksize = 1 if new_chunksize < 1
     new_chunksize
   end

data/lib/taps/cli.rb

@@ -9,7 +9,7 @@ Taps::Config.taps_database_url = ENV['TAPS_DATABASE_URL'] || ENV['DATABASE_URL']
   # this is dirty but it solves a weird problem where the tempfile disappears mid-process
   require 'sqlite3'
   $__taps_database = Tempfile.new('taps.db')
-  $__taps_database.open()
+  $__taps_database.open
   "sqlite://#{$__taps_database.path}"
 end
 
@@ -23,7 +23,7 @@ module Taps
 
     def run
       method = (argv.shift || 'help').to_sym
-      if [:pull, :push, :server, :version].include? method
+      if %i[pull push server version].include? method
         send(method)
       else
         help
@@ -59,12 +59,10 @@ module Taps
 
       Taps::Config.verify_database_url
       require 'taps/server'
-      Taps::Server.run!({
-        :port => opts[:port],
-        :environment => :production,
-        :logging => true,
-        :dump_errors => true,
-      })
+      Taps::Server.run!(port: opts[:port],
+                        environment: :production,
+                        logging: true,
+                        dump_errors: true)
     end
 
     def version
@@ -85,13 +83,13 @@ EOHELP
     end
 
     def serveroptparse
-      opts={:port => 5000, :database_url => nil, :login => nil, :password => nil, :debug => false}
+      opts = { port: 5000, database_url: nil, login: nil, password: nil, debug: false }
       OptionParser.new do |o|
-        o.banner = "Usage: #{File.basename($0)} server [OPTIONS] <local_database_url> <login> <password>"
-        o.define_head "Start a taps database import/export server"
+        o.banner = "Usage: #{File.basename($PROGRAM_NAME)} server [OPTIONS] <local_database_url> <login> <password>"
+        o.define_head 'Start a taps database import/export server'
 
-        o.on("-p", "--port=N", "Server Port") { |v| opts[:port] = v.to_i if v.to_i > 0 }
-        o.on("-d", "--debug", "Enable Debug Messages") { |v| opts[:debug] = true }
+        o.on('-p', '--port=N', 'Server Port') { |v| opts[:port] = v.to_i if v.to_i > 0 }
+        o.on('-d', '--debug', 'Enable Debug Messages') { |_v| opts[:debug] = true }
         o.parse!(argv)
 
         opts[:database_url] = argv.shift
@@ -99,17 +97,17 @@ EOHELP
         opts[:password] = argv.shift || ENV['TAPS_PASSWORD']
 
         if opts[:database_url].nil?
-          $stderr.puts "Missing Database URL"
+          warn 'Missing Database URL'
           puts o
           exit 1
         end
         if opts[:login].nil?
-          $stderr.puts "Missing Login"
+          warn 'Missing Login'
           puts o
           exit 1
         end
         if opts[:password].nil?
-          $stderr.puts "Missing Password"
+          warn 'Missing Password'
           puts o
           exit 1
         end
@@ -118,41 +116,41 @@ EOHELP
     end
 
     def clientoptparse(cmd)
-      opts={:default_chunksize => 1000, :database_url => nil, :remote_url => nil, :debug => false, :resume_filename => nil, :disable_compresion => false, :indexes_first => false}
+      opts = { default_chunksize: 1000, database_url: nil, remote_url: nil, debug: false, resume_filename: nil, disable_compresion: false, indexes_first: false }
       OptionParser.new do |o|
-        o.banner = "Usage: #{File.basename($0)} #{cmd} [OPTIONS] <local_database_url> <remote_url>"
+        o.banner = "Usage: #{File.basename($PROGRAM_NAME)} #{cmd} [OPTIONS] <local_database_url> <remote_url>"
 
         case cmd
         when :pull
-          o.define_head "Pull a database from a taps server"
+          o.define_head 'Pull a database from a taps server'
         when :push
-          o.define_head "Push a database to a taps server"
+          o.define_head 'Push a database to a taps server'
         end
 
-        o.on("-s", "--skip-schema", "Don't transfer the schema, just data") { |v| opts[:skip_schema] = true }
-        o.on("-i", "--indexes-first", "Transfer indexes first before data") { |v| opts[:indexes_first] = true }
-        o.on("-r", "--resume=file", "Resume a Taps Session from a stored file") { |v| opts[:resume_filename] = v }
-        o.on("-c", "--chunksize=N", "Initial Chunksize") { |v| opts[:default_chunksize] = (v.to_i < 10 ? 10 : v.to_i) }
-        o.on("-g", "--disable-compression", "Disable Compression") { |v| opts[:disable_compression] = true }
-        o.on("-f", "--filter=regex", "Regex Filter for tables") { |v| opts[:table_filter] = v }
-        o.on("-t", "--tables=A,B,C", Array, "Shortcut to filter on a list of tables") do |v|
-          r_tables = v.collect { |t| "^#{t}$" }.join("|")
+        o.on('-s', '--skip-schema', "Don't transfer the schema, just data") { |_v| opts[:skip_schema] = true }
+        o.on('-i', '--indexes-first', 'Transfer indexes first before data') { |_v| opts[:indexes_first] = true }
+        o.on('-r', '--resume=file', 'Resume a Taps Session from a stored file') { |v| opts[:resume_filename] = v }
+        o.on('-c', '--chunksize=N', 'Initial Chunksize') { |v| opts[:default_chunksize] = (v.to_i < 10 ? 10 : v.to_i) }
+        o.on('-g', '--disable-compression', 'Disable Compression') { |_v| opts[:disable_compression] = true }
+        o.on('-f', '--filter=regex', 'Regex Filter for tables') { |v| opts[:table_filter] = v }
+        o.on('-t', '--tables=A,B,C', Array, 'Shortcut to filter on a list of tables') do |v|
+          r_tables = v.collect { |t| "^#{t}$" }.join('|')
           opts[:table_filter] = "(#{r_tables})"
         end
-        o.on("-e", "--exclude_tables=A,B,C", Array, "Shortcut to exclude a list of tables") { |v| opts[:exclude_tables] = v }
-        o.on("-d", "--debug", "Enable Debug Messages") { |v| opts[:debug] = true }
+        o.on('-e', '--exclude_tables=A,B,C', Array, 'Shortcut to exclude a list of tables') { |v| opts[:exclude_tables] = v }
+        o.on('-d', '--debug', 'Enable Debug Messages') { |_v| opts[:debug] = true }
         o.parse!(argv)
 
         opts[:database_url] = argv.shift
         opts[:remote_url] = argv.shift
 
         if opts[:database_url].nil?
-          $stderr.puts "Missing Database URL"
+          warn 'Missing Database URL'
           puts o
           exit 1
         end
         if opts[:remote_url].nil?
-          $stderr.puts "Missing Remote Taps URL"
+          warn 'Missing Remote Taps URL'
           puts o
           exit 1
         end
@@ -183,14 +181,11 @@ EOHELP
 
       require 'taps/operation'
 
-      newsession = session.merge({
-        :default_chunksize => opts[:default_chunksize],
-        :disable_compression => opts[:disable_compression],
-        :resume => true,
-      })
+      newsession = session.merge(default_chunksize: opts[:default_chunksize],
+                                 disable_compression: opts[:disable_compression],
+                                 resume: true)
 
       Taps::Operation.factory(method, database_url, remote_url, newsession).run
     end
-
   end
 end

data/lib/taps/config.rb

@@ -22,8 +22,8 @@ module Taps
       attr_accessor :login, :password, :database_url, :remote_url
       attr_accessor :chunksize
 
-      def verify_database_url(db_url=nil)
-        db_url ||= self.database_url
+      def verify_database_url(db_url = nil)
+        db_url ||= database_url
         db = Sequel.connect(db_url)
         db.tables
         db.disconnect

data/lib/taps/data_stream.rb

@@ -6,338 +6,332 @@ require 'taps/errors'
 require 'vendor/okjson'
 
 module Taps
+  class DataStream
+    DEFAULT_CHUNKSIZE = 1000
+
+    attr_reader :db, :state
+
+    def initialize(db, state)
+      @db = db
+      @state = {
+        offset: 0,
+        avg_chunksize: 0,
+        num_chunksize: 0,
+        total_chunksize: 0
+      }.merge(state)
+      @state[:chunksize] ||= DEFAULT_CHUNKSIZE
+      @complete = false
+    end
 
-class DataStream
-  DEFAULT_CHUNKSIZE = 1000
-
-  attr_reader :db, :state
-
-  def initialize(db, state)
-    @db = db
-    @state = {
-      :offset => 0,
-      :avg_chunksize => 0,
-      :num_chunksize => 0,
-      :total_chunksize => 0,
-    }.merge(state)
-    @state[:chunksize] ||= DEFAULT_CHUNKSIZE
-    @complete = false
-  end
-
-  def log
-    Taps.log
-  end
+    def log
+      Taps.log
+    end
 
-  def error=(val)
-    state[:error] = val
-  end
+    def error=(val)
+      state[:error] = val
+    end
 
-  def error
-    state[:error] || false
-  end
+    def error
+      state[:error] || false
+    end
 
-  def table_name
-    state[:table_name].to_sym
-  end
+    def table_name
+      state[:table_name].to_sym
+    end
 
-  def table_name_sql
-    table_name.identifier
-  end
+    def table_name_sql
+      table_name.identifier
+    end
 
-  def to_hash
-    state.merge(:klass => self.class.to_s)
-  end
+    def to_hash
+      state.merge(klass: self.class.to_s)
+    end
 
-  def to_json
-    ::OkJson.encode(to_hash)
-  end
+    def to_json
+      ::OkJson.encode(to_hash)
+    end
 
-  def string_columns
-    @string_columns ||= Taps::Utils.incorrect_blobs(db, table_name)
-  end
+    def string_columns
+      @string_columns ||= Taps::Utils.incorrect_blobs(db, table_name)
+    end
 
-  def table
-    @table ||= db[table_name_sql]
-  end
+    def table
+      @table ||= db[table_name_sql]
+    end
 
-  def order_by(name=nil)
-    @order_by ||= begin
-      name ||= table_name
-      Taps::Utils.order_by(db, name)
+    def order_by(name = nil)
+      @order_by ||= begin
+        name ||= table_name
+        Taps::Utils.order_by(db, name)
+      end
     end
-  end
 
-  def increment(row_count)
-    state[:offset] += row_count
-  end
+    def increment(row_count)
+      state[:offset] += row_count
+    end
 
-  # keep a record of the average chunksize within the first few hundred thousand records, after chunksize
-  # goes below 100 or maybe if offset is > 1000
-  def fetch_rows
-    state[:chunksize] = fetch_chunksize
-    ds = table.order(*order_by).limit(state[:chunksize], state[:offset])
-    log.debug "DataStream#fetch_rows SQL -> #{ds.sql}"
-    rows = Taps::Utils.format_data(ds.all,
-      :string_columns => string_columns,
-      :schema => db.schema(table_name),
-      :table  => table_name
-    )
-    update_chunksize_stats
-    rows
-  end
+    # keep a record of the average chunksize within the first few hundred thousand records, after chunksize
+    # goes below 100 or maybe if offset is > 1000
+    def fetch_rows
+      state[:chunksize] = fetch_chunksize
+      ds = table.order(*order_by).limit(state[:chunksize], state[:offset])
+      log.debug "DataStream#fetch_rows SQL -> #{ds.sql}"
+      rows = Taps::Utils.format_data(ds.all,
+                                     string_columns: string_columns,
+                                     schema: db.schema(table_name),
+                                     table: table_name)
+      update_chunksize_stats
+      rows
+    end
 
-  def max_chunksize_training
-    20
-  end
+    def max_chunksize_training
+      20
+    end
 
-  def fetch_chunksize
-    chunksize = state[:chunksize]
-    return chunksize if state[:num_chunksize] < max_chunksize_training
-    return chunksize if state[:avg_chunksize] == 0
-    return chunksize if state[:error]
-    state[:avg_chunksize] > chunksize ? state[:avg_chunksize] : chunksize
-  end
+    def fetch_chunksize
+      chunksize = state[:chunksize]
+      return chunksize if state[:num_chunksize] < max_chunksize_training
+      return chunksize if state[:avg_chunksize] == 0
+      return chunksize if state[:error]
+      state[:avg_chunksize] > chunksize ? state[:avg_chunksize] : chunksize
+    end
 
-  def update_chunksize_stats
-    return if state[:num_chunksize] >= max_chunksize_training
-    state[:total_chunksize] += state[:chunksize]
-    state[:num_chunksize] += 1
-    state[:avg_chunksize] = state[:total_chunksize] / state[:num_chunksize] rescue state[:chunksize]
-  end
+    def update_chunksize_stats
+      return if state[:num_chunksize] >= max_chunksize_training
+      state[:total_chunksize] += state[:chunksize]
+      state[:num_chunksize] += 1
+      state[:avg_chunksize] = begin
+                                state[:total_chunksize] / state[:num_chunksize]
+                              rescue
+                                state[:chunksize]
+                              end
+    end
 
-  def encode_rows(rows)
-    Taps::Utils.base64encode(Marshal.dump(rows))
-  end
+    def encode_rows(rows)
+      Taps::Utils.base64encode(Marshal.dump(rows))
+    end
 
-  def fetch
-    log.debug "DataStream#fetch state -> #{state.inspect}"
+    def fetch
+      log.debug "DataStream#fetch state -> #{state.inspect}"
 
-    t1 = Time.now
-    rows = fetch_rows
-    encoded_data = encode_rows(rows)
-    t2 = Time.now
-    elapsed_time = t2 - t1
+      t1 = Time.now
+      rows = fetch_rows
+      encoded_data = encode_rows(rows)
+      t2 = Time.now
+      elapsed_time = t2 - t1
 
-    @complete = rows == { }
+      @complete = rows == {}
 
-    [encoded_data, (@complete ? 0 : rows[:data].size), elapsed_time]
-  end
+      [encoded_data, (@complete ? 0 : rows[:data].size), elapsed_time]
+    end
 
-  def complete?
-    @complete
-  end
+    def complete?
+      @complete
+    end
 
-  def fetch_remote(resource, headers)
-    params = fetch_from_resource(resource, headers)
-    encoded_data = params[:encoded_data]
-    json = params[:json]
+    def fetch_remote(resource, headers)
+      params = fetch_from_resource(resource, headers)
+      encoded_data = params[:encoded_data]
+      json = params[:json]
 
-    rows = parse_encoded_data(encoded_data, json[:checksum])
-    @complete = rows == { }
+      rows = parse_encoded_data(encoded_data, json[:checksum])
+      @complete = rows == {}
 
-    # update local state
-    state.merge!(json[:state].merge(:chunksize => state[:chunksize]))
+      # update local state
+      state.merge!(json[:state].merge(chunksize: state[:chunksize]))
 
-    unless @complete
-      import_rows(rows)
-      rows[:data].size
-    else
-      0
+      if @complete
+        0
+      else
+        import_rows(rows)
+        rows[:data].size
+      end
     end
-  end
 
-  # this one is used inside the server process
-  def fetch_remote_in_server(params)
-    json = self.class.parse_json(params[:json])
-    encoded_data = params[:encoded_data]
+    # this one is used inside the server process
+    def fetch_remote_in_server(params)
+      json = self.class.parse_json(params[:json])
+      encoded_data = params[:encoded_data]
 
-    rows = parse_encoded_data(encoded_data, json[:checksum])
-    @complete = rows == { }
+      rows = parse_encoded_data(encoded_data, json[:checksum])
+      @complete = rows == {}
 
-    unless @complete
-      import_rows(rows)
-      rows[:data].size
-    else
-      0
+      if @complete
+        0
+      else
+        import_rows(rows)
+        rows[:data].size
+      end
     end
-  end
 
-  def fetch_from_resource(resource, headers)
-    res = nil
-    log.debug "DataStream#fetch_from_resource state -> #{state.inspect}"
-    state[:chunksize] = Taps::Utils.calculate_chunksize(state[:chunksize]) do |c|
-      state[:chunksize] = c.to_i
-      res = resource.post({:state => ::OkJson.encode(self.to_hash)}, headers)
-    end
+    def fetch_from_resource(resource, headers)
+      res = nil
+      log.debug "DataStream#fetch_from_resource state -> #{state.inspect}"
+      state[:chunksize] = Taps::Utils.calculate_chunksize(state[:chunksize]) do |c|
+        state[:chunksize] = c.to_i
+        res = resource.post({ state: ::OkJson.encode(to_hash) }, headers)
+      end
 
-    begin
-      params = Taps::Multipart.parse(res)
-      params[:json] = self.class.parse_json(params[:json]) if params.has_key?(:json)
-      return params
-    rescue ::OkJson::ParserError
-      raise Taps::CorruptedData.new("Invalid OkJson Received")
+      begin
+        params = Taps::Multipart.parse(res)
+        params[:json] = self.class.parse_json(params[:json]) if params.key?(:json)
+        return params
+      rescue ::OkJson::ParserError
+        raise Taps::CorruptedData, 'Invalid OkJson Received'
+      end
     end
-  end
 
-  def self.parse_json(json)
-    hash = ::OkJson.decode(json).symbolize_keys
-    hash[:state].symbolize_keys! if hash.has_key?(:state)
-    hash
-  end
-
-  def parse_encoded_data(encoded_data, checksum)
-    raise Taps::CorruptedData.new("Checksum Failed") unless Taps::Utils.valid_data?(encoded_data, checksum)
+    def self.parse_json(json)
+      hash = ::OkJson.decode(json).symbolize_keys
+      hash[:state].symbolize_keys! if hash.key?(:state)
+      hash
+    end
 
-    begin
-      return Marshal.load(Taps::Utils.base64decode(encoded_data))
-    rescue Object
-      unless ENV['NO_DUMP_MARSHAL_ERRORS']
-        puts "Error encountered loading data, wrote the data chunk to dump.#{Process.pid}.dat"
-        File.open("dump.#{Process.pid}.dat", "w") { |f| f.write(encoded_data) }
+    def parse_encoded_data(encoded_data, checksum)
+      raise Taps::CorruptedData, 'Checksum Failed' unless Taps::Utils.valid_data?(encoded_data, checksum)
+
+      begin
+        return Marshal.load(Taps::Utils.base64decode(encoded_data))
+      rescue Object
+        unless ENV['NO_DUMP_MARSHAL_ERRORS']
+          puts "Error encountered loading data, wrote the data chunk to dump.#{Process.pid}.dat"
+          File.open("dump.#{Process.pid}.dat", 'w') { |f| f.write(encoded_data) }
+        end
+        raise
       end
-      raise
     end
-  end
 
-  def import_rows(rows)
-    table.import(rows[:header], rows[:data])
-    state[:offset] += rows[:data].size
-  rescue Exception => ex
-    case ex.message
-    when /integer out of range/ then
-      raise Taps::InvalidData, <<-ERROR, []
+    def import_rows(rows)
+      table.import(rows[:header], rows[:data])
+      state[:offset] += rows[:data].size
+    rescue Exception => ex
+      case ex.message
+      when /integer out of range/ then
+        raise Taps::InvalidData, <<-ERROR, []
 \nDetected integer data that exceeds the maximum allowable size for an integer type.
 This generally occurs when importing from SQLite due to the fact that SQLite does
 not enforce maximum values on integer types.
 ERROR
-    else raise ex
+      else raise ex
+      end
     end
-  end
 
-  def verify_stream
-    state[:offset] = table.count
-  end
-
-  def verify_remote_stream(resource, headers)
-    json_raw = resource.post({:state => ::OkJson.encode(self)}, headers).to_s
-    json = self.class.parse_json(json_raw)
-
-    self.class.new(db, json[:state])
-  end
-
-  def self.factory(db, state)
-    if defined?(Sequel::MySQL) && Sequel::MySQL.respond_to?(:convert_invalid_date_time=)
-      Sequel::MySQL.convert_invalid_date_time = :nil
+    def verify_stream
+      state[:offset] = table.count
     end
 
-    if state.has_key?(:klass)
-      return eval(state[:klass]).new(db, state)
-    end
+    def verify_remote_stream(resource, headers)
+      json_raw = resource.post({ state: ::OkJson.encode(self) }, headers).to_s
+      json = self.class.parse_json(json_raw)
 
-    if Taps::Utils.single_integer_primary_key(db, state[:table_name].to_sym)
-      DataStreamKeyed.new(db, state)
-    else
-      DataStream.new(db, state)
+      self.class.new(db, json[:state])
     end
-  end
-end
 
+    def self.factory(db, state)
+      if defined?(Sequel::MySQL) && Sequel::MySQL.respond_to?(:convert_invalid_date_time=)
+        Sequel::MySQL.convert_invalid_date_time = :nil
+      end
 
-class DataStreamKeyed < DataStream
-  attr_accessor :buffer
+      return eval(state[:klass]).new(db, state) if state.key?(:klass)
 
-  def initialize(db, state)
-    super(db, state)
-    @state = { :primary_key => order_by(state[:table_name]).first, :filter => 0 }.merge(state)
-    @state[:chunksize] ||= DEFAULT_CHUNKSIZE
-    @buffer = []
+      if Taps::Utils.single_integer_primary_key(db, state[:table_name].to_sym)
+        DataStreamKeyed.new(db, state)
+      else
+        DataStream.new(db, state)
+      end
+    end
   end
 
-  def primary_key
-    state[:primary_key].to_sym
-  end
+  class DataStreamKeyed < DataStream
+    attr_accessor :buffer
 
-  def buffer_limit
-    if state[:last_fetched] and state[:last_fetched] < state[:filter] and self.buffer.size == 0
-      state[:last_fetched]
-    else
-      state[:filter]
+    def initialize(db, state)
+      super(db, state)
+      @state = { primary_key: order_by(state[:table_name]).first, filter: 0 }.merge(state)
+      @state[:chunksize] ||= DEFAULT_CHUNKSIZE
+      @buffer = []
     end
-  end
 
-  def calc_limit(chunksize)
-    # we want to not fetch more than is needed while we're
-    # inside sinatra but locally we can select more than
-    # is strictly needed
-    if defined?(Sinatra)
-      (chunksize * 1.1).ceil
-    else
-      (chunksize * 3).ceil
+    def primary_key
+      state[:primary_key].to_sym
     end
-  end
 
-  def load_buffer(chunksize)
-    # make sure BasicObject is not polluted by subsequent requires
-    Sequel::BasicObject.remove_methods!
+    def buffer_limit
+      if state[:last_fetched] && (state[:last_fetched] < state[:filter]) && buffer.empty?
+        state[:last_fetched]
+      else
+        state[:filter]
+      end
+    end
 
-    num = 0
-    loop do
-      limit = calc_limit(chunksize)
-      # we have to use local variables in order for the virtual row filter to work correctly
-      key = primary_key
-      buf_limit = buffer_limit
-      ds = table.order(*order_by).filter { key.sql_number > buf_limit }.limit(limit)
-      log.debug "DataStreamKeyed#load_buffer SQL -> #{ds.sql}"
-      data = ds.all
-      self.buffer += data
-      num += data.size
-      if data.size > 0
-        # keep a record of the last primary key value in the buffer
-        state[:filter] = self.buffer.last[ primary_key ]
+    def calc_limit(chunksize)
+      # we want to not fetch more than is needed while we're
+      # inside sinatra but locally we can select more than
+      # is strictly needed
+      if defined?(Sinatra)
+        (chunksize * 1.1).ceil
+      else
+        (chunksize * 3).ceil
       end
+    end
 
-      break if num >= chunksize or data.size == 0
+    def load_buffer(chunksize)
+      # make sure BasicObject is not polluted by subsequent requires
+      Sequel::BasicObject.remove_methods!
+
+      num = 0
+      loop do
+        limit = calc_limit(chunksize)
+        # we have to use local variables in order for the virtual row filter to work correctly
+        key = primary_key
+        buf_limit = buffer_limit
+        ds = table.order(*order_by).filter { key.sql_number > buf_limit }.limit(limit)
+        log.debug "DataStreamKeyed#load_buffer SQL -> #{ds.sql}"
+        data = ds.all
+        self.buffer += data
+        num += data.size
+        unless data.empty?
+          # keep a record of the last primary key value in the buffer
+          state[:filter] = self.buffer.last[primary_key]
+        end
+
+        break if (num >= chunksize) || data.empty?
+      end
     end
-  end
 
-  def fetch_buffered(chunksize)
-    load_buffer(chunksize) if self.buffer.size < chunksize
-    rows = buffer.slice(0, chunksize)
-    state[:last_fetched] = if rows.size > 0
-      rows.last[ primary_key ]
-    else
-      nil
+    def fetch_buffered(chunksize)
+      load_buffer(chunksize) if self.buffer.size < chunksize
+      rows = buffer.slice(0, chunksize)
+      state[:last_fetched] = (rows.last[primary_key] unless rows.empty?)
+      rows
     end
-    rows
-  end
 
-  def import_rows(rows)
-    table.import(rows[:header], rows[:data])
-  end
+    def import_rows(rows)
+      table.import(rows[:header], rows[:data])
+    end
 
-  def fetch_rows
-    chunksize = state[:chunksize]
-    Taps::Utils.format_data(fetch_buffered(chunksize) || [],
-      :string_columns => string_columns)
-  end
+    def fetch_rows
+      chunksize = state[:chunksize]
+      Taps::Utils.format_data(fetch_buffered(chunksize) || [],
+                              string_columns: string_columns)
+    end
 
-  def increment(row_count)
-    # pop the rows we just successfully sent off the buffer
-    @buffer.slice!(0, row_count)
-  end
+    def increment(row_count)
+      # pop the rows we just successfully sent off the buffer
+      @buffer.slice!(0, row_count)
+    end
 
-  def verify_stream
-    key = primary_key
-    ds = table.order(*order_by)
-    current_filter = ds.max(key.sql_number)
+    def verify_stream
+      key = primary_key
+      ds = table.order(*order_by)
+      current_filter = ds.max(key.sql_number)
 
-    # set the current filter to the max of the primary key
-    state[:filter] = current_filter
-    # clear out the last_fetched value so it can restart from scratch
-    state[:last_fetched] = nil
+      # set the current filter to the max of the primary key
+      state[:filter] = current_filter
+      # clear out the last_fetched value so it can restart from scratch
+      state[:last_fetched] = nil
 
-    log.debug "DataStreamKeyed#verify_stream -> state: #{state.inspect}"
+      log.debug "DataStreamKeyed#verify_stream -> state: #{state.inspect}"
+    end
   end
 end
-
-end