tapsoob 0.1.10

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 1140973a2ff5d3c3acb175c2c58b21ac93ac1425
+  data.tar.gz: 1e0a3b8443be5d4fcbf567d8635a6a754d3e7a29
+SHA512:
+  metadata.gz: 61d6f73a2f3ad4d4173e9cad5090af8e0ca94def18b925adb9716565d458f0addfe8030bc67fd2d7b16caf7706ac2e9b3653c35912010ec4fde60ac102c7eda9
+  data.tar.gz: 443ae1a295828478dd89ba039c6f842e973eaa1dd491465865c25c0165732c8f288b7325161a4c31ea74102be9066cbf8a12119ef749c035c0de0d54a2280075

data/.gitignore

@@ -0,0 +1,4 @@
+# Ignore dev stuffs
+db/
+*.dat
+*.gem

data/Gemfile

@@ -0,0 +1,4 @@
+source "http://rubygems.org"
+
+# load the gem's dependencies
+gemspec

data/Gemfile.lock

@@ -0,0 +1,24 @@
+PATH
+  remote: .
+  specs:
+    tapsoob (0.1.0)
+      sequel (~> 3.45.0)
+
+GEM
+  remote: http://rubygems.org/
+  specs:
+    mysql (2.9.1)
+    mysql2 (0.3.11)
+    pg (0.14.1)
+    sequel (3.45.0)
+    sqlite3 (1.3.7)
+
+PLATFORMS
+  ruby
+
+DEPENDENCIES
+  mysql (~> 2.9.1)
+  mysql2 (~> 0.3.11)
+  pg (~> 0.14.1)
+  sqlite3 (~> 1.3.7)
+  tapsoob!

data/README.md

@@ -0,0 +1,66 @@
+# Tapsoob
+
+Tapsoob is a simple tool to export and import databases. It is inspired by <https://github.com/ricardochimal/taps> but instead of having to rely on a server and a client, databases are exported to the filesystem or imported from a previous export, hence the OOB (out-of-band). This was done in order to avoid exposing a whole database using the HyperText Protocol for security reasons.
+
+Although the code is not quite perfect yet and some improvement will probably be made in the future it's usable. However if you notice an issue don't hesitate to report it.
+
+
+## Database support
+
+Tapsoob currently rely on the Sequel ORM (<http://sequel.rubyforge.org/>) so we currently support every database that Sequel supports. If additionnal support is required in the future (NoSQL databases) I'll make my best to figure out a way to bring that in my ToDo list.
+
+### Oracle
+
+If you're using either Oracle or Oracle XE you will need some extra requirements. If you're using Ruby you'll need to have your ORACLE_HOME environnement variable set properly and the `ruby-oci8` gem installed. However if you're using jRuby you'll need to have the official Oracle JDBC driver (see here for more informations: <http://www.oracle.com/technetwork/articles/dsl/jruby-oracle11g-330825.html>) and it should be loaded prior to using Tapsoob otherwise you won't be able to connect the database.
+
+
+## Exporting your data
+
+    tapsoob pull [OPTIONS] <dump_path> <database_url>
+
+The `dump_path` is the location on the filesystem where you want your database exported and the `database_url` is an URL looking like this `postgres://user:password@localhost/blog`, you can find out more about how to connect to your RDBMS by refering yourself to this page: <http://sequel.rubyforge.org/rdoc/files/doc/opening_databases_rdoc.html> on the Sequel documentation.
+
+Regarding options a complete list of options can be displayed using the following command:
+
+    tapsoob pull -h
+
+
+## Importing your data
+
+    tapsoob push [OPTIONS] <dump_path> <database_url>
+
+As for exporting the `dump_path` is the path you previously exported a database you now wish to import and the `database_url` should conform to the same format described before.
+
+You can list all available options using the command:
+
+    tapsoob push -h
+
+
+## Integration with Rails
+
+If you're using Rails, there's also two Rake tasks provided:
+
+* `tapsoob:pull` which dumps the database into a new folder under the `db` folder
+* `tapsoob:push` which reads the last dump you made from `tapsoob:pull` from the `db` folder
+
+
+## Notes
+
+Your exports can be moved from one machine to another for backups or replication, you can also use Tapsoob to switch your RDBMS from one of the supported system to another.
+
+
+## Feature(s) to come
+
+* Tests
+
+
+## License
+
+The MIT License (MIT)
+Copyright © 2013 Félix Bellanger <felix.bellanger@gmail.com>
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/bin/schema

@@ -0,0 +1,54 @@
+#!/usr/bin/env ruby
+
+require 'rubygems'
+require 'sequel'
+
+$:.unshift File.dirname(__FILE__) + '/../lib'
+
+require 'tapsoob/schema'
+
+cmd = ARGV.shift.strip rescue ''
+database_url = ARGV.shift.strip rescue ''
+
+def show_usage_and_exit
+  puts <<EOTXT
+  schema console <database_url>
+  schema dump <database_url>
+  schema dump_table <database_url> <table>
+  schema indexes <database_url>
+  schema indexes_individual <database_url>
+  schema reset_db_sequences <database_url>
+  schema load <database_url> <schema_file>
+  schema load_indexes <database_url> <indexes_file>
+EOTXT
+  exit(1)
+end
+
+case cmd
+when 'dump'
+  puts Tapsoob::Schema.dump(database_url)
+when 'dump_table'
+  table = ARGV.shift.strip
+  puts Tapsoob::Schema.dump_table(database_url, table)
+when 'indexes'
+  puts Tapsoob::Schema.indexes(database_url)
+when 'indexes_individual'
+  puts Tapsoob::Schema.indexes_individual(database_url)
+when 'load_indexes'
+  filename = ARGV.shift.strip rescue ''
+  indexes = File.read(filename) rescue show_usage_and_exit
+  Tapsoob::Schema.load_indexes(database_url, indexes)
+when 'load'
+  filename = ARGV.shift.strip rescue ''
+  schema = File.read(filename) rescue show_usage_and_exit
+  Tapsoob::Schema.load(database_url, schema)
+when 'reset_db_sequences'
+  Tapsoob::Schema.reset_db_sequences(database_url)
+when 'console'
+  $db = Sequel.connect(database_url)
+  require 'irb'
+  require 'irb/completion'
+  IRB.start
+else
+  show_usage_and_exit
+end

data/bin/tapsoob

@@ -0,0 +1,6 @@
+#!/usr/bin/env ruby
+
+$:.unshift File.dirname(__FILE__) + '/../lib'
+require 'tapsoob/cli'
+
+Tapsoob::Cli.new(ARGV.dup).run

data/lib/tapsoob.rb

@@ -0,0 +1,9 @@
+# -*- encoding : utf-8 -*-
+$:.unshift File.dirname(__FILE__)
+
+# internal requires
+require 'tapsoob/operation'
+
+module Tapsoob
+  require 'tapsoob/railtie' if defined?(Rails)
+end

data/lib/tapsoob/chunksize.rb

@@ -0,0 +1,53 @@
+# -*- encoding : utf-8 -*-
+require 'tapsoob/errors'
+
+class Tapsoob::Chunksize
+  attr_accessor :idle_secs, :time_in_db, :start_time, :end_time, :retries
+  attr_reader :chunksize
+
+  def initialize(chunksize)
+    @chunksize = chunksize
+    @idle_secs = 0.0
+    @retries = 0
+  end
+
+  def to_i
+    chunksize
+  end
+
+  def reset_chunksize
+    @chunksize = (retries <= 1) ? 10 : 1
+  end
+
+  def diff
+    end_time - start_time - time_in_db - idle_secs
+  end
+
+  def time_in_db=(t)
+    @time_in_db = t
+    @time_in_db = @time_in_db.to_f rescue 0.0
+  end
+
+  def time_delta
+    t1 = Time.now
+    yield if block_given?
+    t2 = Time.now
+    t2 - t1
+  end
+
+  def calc_new_chunksize
+    new_chunksize = if retries > 0
+      chunksize
+    elsif diff > 3.0
+      (chunksize / 3).ceil
+    elsif diff > 1.1
+      chunksize - 100
+    elsif diff < 0.8
+      chunksize * 2
+    else
+      chunksize + 100
+    end
+    new_chunksize = 1 if new_chunksize < 1
+    new_chunksize
+  end
+end

data/lib/tapsoob/cli.rb

@@ -0,0 +1,145 @@
+# -*- encoding : utf-8 -*-
+require 'fileutils'
+require 'optparse'
+require 'tempfile'
+require 'tapsoob/config'
+require 'tapsoob/log'
+
+Tapsoob::Config.tapsoob_database_url = ENV['TAPSOOB_DATABASE_URL'] || begin
+  # this is dirty but it solves a weird problem where the tempfile disappears mid-process
+  require 'sqlite3'
+  $__taps_database = Tempfile.new('tapsoob.db')
+  $__taps_database.open()
+  "sqlite://#{$__taps_database.path}"
+end
+
+module Tapsoob
+  class Cli
+    attr_accessor :argv
+
+    def initialize(argv)
+      @argv = argv
+    end
+
+    def run
+      method = (argv.shift || 'help').to_sym
+      if [:pull, :push, :version].include? method
+        send(method)
+      else
+        help
+      end
+    end
+
+    def pull
+      opts = clientoptparse(:pull)
+      Tapsoob.log.level = Logger::DEBUG if opts[:debug]
+      if opts[:resume_filename]
+        clientresumexfer(:pull, opts)
+      else
+        clientxfer(:pull, opts)
+      end
+    end
+
+    def push
+      opts = clientoptparse(:push)
+      Tapsoob.log.level = Logger::DEBUG if opts[:debug]
+      if opts[:resume_filename]
+        clientresumexfer(:push, opts)
+      else
+        clientxfer(:push, opts)
+      end
+    end
+
+    def version
+      puts Tapsoob.version
+    end
+
+    def help
+      puts <<EOHELP
+Options
+=======
+pull      Pull a database and export it into a directory
+push      Push a database from a directory
+version   Tapsoob version
+
+Add '-h' to any command to see their usage
+EOHELP
+    end
+
+    def clientoptparse(cmd)
+      opts={:default_chunksize => 1000, :database_url => nil, :dump_path => nil, :debug => false, :resume_filename => nil, :disable_compression => false, :indexes_first => false}
+      OptionParser.new do |o|
+        o.banner = "Usage: #{File.basename($0)} #{cmd} [OPTIONS] <dump_path> <database_url>"
+
+        case cmd
+        when :pull
+          o.define_head "Pull a database and export it into a directory"
+        when :push
+          o.define_head "Push a database from a directory"
+        end
+
+        o.on("-s", "--skip-schema", "Don't transfer the schema just data") { |v| opts[:skip_schema] = true }
+        o.on("-i", "--indexes-first", "Transfer indexes first before data") { |v| opts[:indexes_first] = true }
+        o.on("-r", "--resume=file", "Resume a Tapsoob Session from a stored file") { |v| opts[:resume_filename] = v }
+        o.on("-c", "--chunksize=N", "Initial Chunksize") { |v| opts[:default_chunksize] = (v.to_i < 10 ? 10 : v.to_i) }
+        o.on("-g", "--disable-compression", "Disable Compression") { |v| opts[:disable_compression] = true }
+        o.on("-f", "--filter=regex", "Regex Filter for tables") { |v| opts[:table_filter] = v }
+        o.on("-t", "--tables=A,B,C", Array, "Shortcut to filter on a list of tables") do |v|
+          r_tables = v.collect { |t| "^#{t}" }.join("|")
+          opts[:table_filter] = "#{r_tables}"
+        end
+        o.on("-e", "--exclude-tables=A,B,C", Array, "Shortcut to exclude a list of tables") { |v| opts[:exclude_tables] = v }
+        o.on("-d", "--debug", "Enable Debug Messages") { |v| opts[:debug] = true }
+
+        opts[:dump_path] = argv.shift
+        opts[:database_url] = argv.shift
+
+        if opts[:database_url].nil?
+          $stderr.puts "Missing Database URL"
+          puts o
+          exit 1
+        end
+        if opts[:dump_path].nil?
+          $stderr.puts "Missing Tapsoob Dump Path"
+          puts o
+          exist 1
+        end
+      end
+
+      opts
+    end
+
+    def clientxfer(method, opts)
+      database_url = opts.delete(:database_url)
+      dump_path = opts.delete(:dump_path)
+
+      Tapsoob::Config.verify_database_url(database_url)
+
+      FileUtils.mkpath "#{dump_path}/schemas"
+      FileUtils.mkpath "#{dump_path}/data"
+      FileUtils.mkpath "#{dump_path}/indexes"
+
+      require 'tapsoob/operation'
+
+      Tapsoob::Operation.factory(method, database_url, dump_path, opts).run
+    end
+
+    def clientresumexfer(method, opts)
+      session = JSON.parse(File.read(opts.delete(:resume_filename)))
+      session.symbolize_recursively!
+
+      database_url = opts.delete(:database_url)
+      dump_path = opts.delete(:dump_path) || session.delete(:dump_path)
+
+      require 'taps/operation'
+
+      newsession = session.merge({
+        :default_chunksize => opts[:default_chunksize],
+        :disable_compression => opts[:disable_compression],
+        :resume => true
+      })
+
+      Tapsoob::Operation.factory(method, database_url, dump_path, newsession).run
+    end
+  end
+end

data/lib/tapsoob/config.rb

@@ -0,0 +1,33 @@
+# -*- encoding : utf-8 -*-
+require 'sequel'
+require 'tapsoob/version'
+
+Sequel.datetime_class = DateTime
+
+module Tapsoob
+  def self.exiting=(val)
+    @@exiting = val
+  end
+
+  def exiting?
+    (@@exiting ||= false) == true
+  end
+
+  class Config
+    class << self
+      attr_accessor :tapsoob_database_url
+      attr_accessor :login, :password, :database_url, :dump_path
+      attr_accessor :chunksize
+
+      def verify_database_url(db_url=nil)
+        db_url ||= self.database_url
+        db = Sequel.connect(db_url)
+        db.tables
+        db.disconnect
+      rescue Object => e
+        puts "Failed to connect to database:\n  #{e.class} -> #{e}"
+        exit 1
+      end
+    end
+  end
+end

data/lib/tapsoob/data_stream.rb

@@ -0,0 +1,350 @@
+# -*- encoding : utf-8 -*-
+require 'tapsoob/log'
+require 'tapsoob/utils'
+
+module Tapsoob
+  class DataStream
+    DEFAULT_CHUNKSIZE = 1000
+
+    attr_reader :db, :state
+
+    def initialize(db, state)
+      @db = db
+      @state = {
+        :offset          => 0,
+        :avg_chunksize   => 0,
+        :num_chunksize   => 0,
+        :total_chunksize => 0
+      }.merge(state)
+      @state[:chunksize] ||= DEFAULT_CHUNKSIZE
+      @complete = false
+    end
+
+    def log
+      Tapsoob.log
+    end
+
+    def error=(val)
+      state[:error] = val
+    end
+
+    def error
+      state[:error] || false
+    end
+
+    def table_name
+      state[:table_name].to_sym
+    end
+
+    def table_name_sql
+      table_name.identifier
+    end
+
+    def to_hash
+      state.merge(:klass => self.class.to_s)
+    end
+
+    def to_json
+      JSON.generate(to_hash)
+    end
+
+    def string_columns
+      @string_columns ||= Tapsoob::Utils.incorrect_blobs(db, table_name)
+    end
+
+    def table
+      @table ||= db[table_name_sql]
+    end
+
+    def order_by(name=nil)
+      @order_by ||= begin
+        name ||= table_name
+        Tapsoob::Utils.order_by(db, name)
+      end
+    end
+
+    def increment(row_count)
+      state[:offset] += row_count
+    end
+
+    # keep a record of the average chunksize within the first few hundred thousand records, after chunksize
+    # goes below 100 or maybe if offset is > 1000
+    def fetch_rows
+      state[:chunksize] = fetch_chunksize
+      ds = table.order(*order_by).limit(state[:chunksize], state[:offset])
+      log.debug "DataStream#fetch_rows SQL -> #{ds.sql}"
+      rows = Tapsoob::Utils.format_data(ds.all,
+        :string_columns => string_columns,
+        :schema => db.schema(table_name),
+        :table => table_name
+      )
+      update_chunksize_stats
+      rows
+    end
+
+    def fetch_file(dump_path)
+      state[:chunksize] = fetch_chunksize
+      ds = JSON.parse(File.read(File.join(dump_path, "data", "#{table_name}.json")))
+      log.debug "DataStream#fetch_file"
+      rows = {
+        :header => ds["header"],
+        :data   => ds["data"][state[:offset], (state[:offset] + state[:chunksize])] || [ ]
+      }
+      update_chunksize_stats
+      rows
+    end
+
+    def max_chunksize_training
+      20
+    end
+
+    def fetch_chunksize
+      chunksize = state[:chunksize]
+      return chunksize if state[:num_chunksize] < max_chunksize_training
+      return chunksize if state[:avg_chunksize] == 0
+      return chunksize if state[:error]
+      state[:avg_chunksize] > chunksize ? state[:avg_chunksize] : chunksize
+    end
+
+    def update_chunksize_stats
+      return if state[:num_chunksize] >= max_chunksize_training
+      state[:total_chunksize] += state[:chunksize]
+      state[:num_chunksize] += 1
+      state[:avg_chunksize] = state[:total_chunksize] / state[:num_chunksize] rescue state[:chunksize]
+    end
+
+    def encode_rows(rows)
+      Tapsoob::Utils.base64encode(Marshal.dump(rows))
+    end
+
+    def fetch(opts = {})
+      opts = (opts.empty? ? { :type => "database", :source => db.uri } : opts)
+
+      log.debug "DataStream#fetch state -> #{state.inspect}"
+
+      t1 = Time.now
+      rows = (opts[:type] == "file" ? fetch_file(opts[:source]) : fetch_rows)
+      encoded_data = encode_rows(rows)
+      t2 = Time.now
+      elapsed_time = t2 - t1
+
+      if opts[:type] == "file"
+        @complete = rows[:data] == [ ]
+      else
+        @complete = rows == { }
+      end
+
+      [encoded_data, (@complete ? 0 : rows[:data].size), elapsed_time]
+    end
+
+    def complete?
+      @complete
+    end
+
+    def fetch_database(dump_path)
+      params = fetch_from_database
+      encoded_data = params[:encoded_data]
+      json = params[:json]
+
+      rows = parse_encoded_data(encoded_data, json[:checksum])
+
+      @complete = rows == { }
+
+      # update local state
+      state.merge!(json[:state].merge(:chunksize => state[:chunksize]))
+
+      unless @complete
+        Tapsoob::Utils.export_rows(dump_path, table_name, rows)
+        state[:offset] += rows[:data].size
+        rows[:data].size
+      else
+        0
+      end
+    end
+
+    def fetch_from_database
+      res = nil
+      log.debug "DataStream#fetch_from_database state -> #{state.inspect}"
+      state[:chunksize] = Tapsoob::Utils.calculate_chunksize(state[:chunksize]) do |c|
+        state[:chunksize] = c.to_i
+        encoded_data = fetch.first
+
+        checksum = Tapsoob::Utils.checksum(encoded_data).to_s
+
+        res = {
+          :json         => { :checksum => checksum, :state => to_hash },
+          :encoded_data => encoded_data
+        }
+      end
+
+      res
+    end
+
+    def fetch_data_in_database(params)
+      encoded_data = params[:encoded_data]
+
+      rows = parse_encoded_data(encoded_data, params[:checksum])
+
+      @complete = rows[:data] == [ ]
+
+      unless @complete
+        import_rows(rows)
+        rows[:data].size
+      else
+        0
+      end
+    end
+
+    def self.parse_json(json)
+      hash = JSON.parse(json).symbolize_keys
+      hash[:state].symbolize_keys! if hash.has_key?(:state)
+      hash
+    end
+
+    def parse_encoded_data(encoded_data, checksum)
+      raise Tapsoob::CorruptedData.new("Checksum Failed") unless Tapsoob::Utils.valid_data?(encoded_data, checksum)
+
+      begin
+        return Marshal.load(Tapsoob::Utils.base64decode(encoded_data))
+      rescue Object => e
+        unless ENV['NO_DUMP_MARSHAL_ERRORS']
+          puts "Error encountered loading data, wrote the data chunk to dump.#{Process.pid}.dat"
+          File.open("dump.#{Process.pid}.dat", "w") { |f| f.write(encoded_data) }
+        end
+        raise e
+      end
+    end
+
+    def import_rows(rows)
+      table.import(rows[:header], rows[:data], :commit_every => 100)
+      state[:offset] += rows[:data].size
+    rescue Exception => ex
+      case ex.message
+      when /integer out of range/ then
+        raise Taps::InvalidData, <<-ERROR, []
+  \nDetected integer data that exceeds the maximum allowable size for an integer type.
+  This generally occurs when importing from SQLite due to the fact that SQLite does
+  not enforce maximum values on integer types.
+        ERROR
+      else raise ex
+      end
+    end
+
+    def verify_stream
+      state[:offset] = table.count
+    end
+
+    def self.factory(db, state)
+      if defined?(Sequel::MySQL) && Sequel::MySQL.respond_to?(:convert_invalid_date_time=)
+        Sequel::MySQL.convert_invalid_date_time = :nil
+      end
+
+      if state.has_key?(:klass)
+        return eval(state[:klass]).new(db, state)
+      end
+
+      if Tapsoob::Utils.single_integer_primary_key(db, state[:table_name].to_sym)
+        DataStreamKeyed.new(db, state)
+      else
+        DataStream.new(db, state)
+      end
+    end
+  end
+
+  class DataStreamKeyed < DataStream
+    attr_accessor :buffer
+
+    def initialize(db, state)
+      super(db, state)
+      @state = { :primary_key => order_by(state[:table_name]).first, :filter => 0 }.merge(@state)
+      @state[:chunksize] ||= DEFAULT_CHUNKSIZE
+      @buffer = []
+    end
+
+    def primary_key
+      state[:primary_key].to_sym
+    end
+
+    def buffer_limit
+      if state[:last_fetched] and state[:last_fetched] < state[:filter] and self.buffer.size == 0
+        state[:last_fetched]
+      else
+        state[:filter]
+      end
+    end
+
+    def calc_limit(chunksize)
+      # we want to not fetch more than is needed while we're
+      # inside sinatra but locally we can select more than
+      # is strictly needed
+      if defined?(Sinatra)
+        (chunksize * 1.1).ceil
+      else
+        (chunksize * 3).ceil
+      end
+    end
+
+    def load_buffer(chunksize)
+      # make sure BasicObject is not polluted by subsequent requires
+      Sequel::BasicObject.remove_methods!
+
+      num = 0
+      loop do
+        limit = calc_limit(chunksize)
+        # we have to use local variables in order for the virtual row filter to work correctly
+        key = primary_key
+        buf_limit = buffer_limit
+        ds = table.order(*order_by).filter { key.sql_number > buf_limit }.limit(limit)
+        log.debug "DataStreamKeyed#load_buffer SQL -> #{ds.sql}"
+        data = ds.all
+        self.buffer += data
+        num += data.size
+        if data.size > 0
+          # keep a record of the last primary key value in the buffer
+          state[:filter] = self.buffer.last[ primary_key ]
+        end
+
+        break if num >= chunksize or data.size == 0
+      end
+    end
+
+    def fetch_buffered(chunksize)
+      load_buffer(chunksize) if self.buffer.size < chunksize
+      rows = buffer.slice(0, chunksize)
+      state[:last_fetched] = if rows.size > 0
+        rows.last[ primary_key ]
+      else
+        nil
+      end
+      rows
+    end
+
+    #def import_rows(rows)
+    #  table.import(rows[:header], rows[:data])
+    #end
+
+    #def fetch_rows
+    #  chunksize = state[:chunksize]
+    #  Tapsoob::Utils.format_data(fetch_buffered(chunksize) || [],
+    #    :string_columns => string_columns)
+    #end
+
+    def increment(row_count)
+      # pop the rows we just successfully sent off the buffer
+      @buffer.slice!(0, row_count)
+    end
+
+    def verify_stream
+      key = primary_key
+      ds = table.order(*order_by)
+      current_filter = ds.max(key.sql_number)
+
+      # set the current filter to the max of the primary key
+      state[:filter] = current_filter
+      # clear out the last_fetched value so it can restart from scratch
+      state[:last_fetched] = nil
+
+      log.debug "DataStreamKeyed#verify_stream -> state: #{state.inspect}"
+    end
+  end
+end