upsert 0.2.2 → 0.3.0

data/CHANGELOG

@@ -1,8 +1,15 @@
+0.3.0 / 2012-06-21
+
+* Enhancements
+
+  * Remove all the sampling - just keep a cumulative total of sql bytes as we build up an ON DUPLICATE KEY UPDATE query.
+  * Deprecate Upsert.stream in favor of Upsert.batch (but provide an alias for backwards compat)
+
 0.2.2 / 2012-06-21
 
 * Bug fixes
 
-  * Correct and simplify how sql length is calculated when batching (streaming) MySQL upserts.
+  * Correct and simplify how sql length is calculated when batching MySQL upserts.
 
 0.2.1 / 2012-06-21
 

data/README.md

@@ -10,24 +10,24 @@ The second argument is currently (mis)named a "document" because this was inspir
 
 ### One by one
 
-Faster than just doing `Pet.create`... 85% faster on PostgreSQL, for example. But no validations or anything.
+Faster than just doing `Pet.create`... 85% faster on PostgreSQL, for example, than all the different native ActiveRecord methods I've tried. But no validations or anything.
 
     upsert = Upsert.new Pet.connection, Pet.table_name
     upsert.row({:name => 'Jerry'}, :breed => 'beagle')
     upsert.row({:name => 'Pierre'}, :breed => 'tabby')
 
-### Streaming
+### Batch mode
 
 Rows are buffered in memory until it's efficient to send them to the database. Currently this only provides an advantage on MySQL because it uses `ON DUPLICATE KEY UPDATE`... but if a similar method appears in PostgreSQL, the same code will still work.
 
-    Upsert.stream(Pet.connection, Pet.table_name) do |upsert|
+    Upsert.batch(Pet.connection, Pet.table_name) do |upsert|
       upsert.row({:name => 'Jerry'}, :breed => 'beagle')
       upsert.row({:name => 'Pierre'}, :breed => 'tabby')
     end
 
 ### `ActiveRecord::Base.upsert` (optional)
 
-For bulk upserts, you probably still want to use `Upsert.stream`.
+For bulk upserts, you probably still want to use `Upsert.batch`.
 
     require 'upsert/active_record_upsert'
     Pet.upsert({:name => 'Jerry'}, :breed => 'beagle')
@@ -37,7 +37,7 @@ For bulk upserts, you probably still want to use `Upsert.stream`.
 
 Currently, the first row you pass in determines the columns that will be used. That's useful for mass importing of many rows with the same columns, but is surprising if you're trying to use a single `Upsert` object to add arbitrary data. For example, this won't work:
 
-    Upsert.stream(Pet.connection, Pet.table_name) do |upsert|
+    Upsert.batch(Pet.connection, Pet.table_name) do |upsert|
       upsert.row({:name => 'Jerry'}, :breed => 'beagle')
       upsert.row({:tag_number => 456}, :spiel => 'great cat') # won't work - doesn't use same columns
     end
@@ -51,10 +51,9 @@ You would need to use a new `Upsert` object. On the other hand, this is totally
 
 Pull requests for any of these would be greatly appreciated:
 
-1. Somebody who understands statistics should look at how I'm sampling rows in `Upsert::Mysql2_Client#estimate_variable_sql_bytesize`... I think we can assume that row sizes are random, so I don't think we actually have to select random elements.
-2. Fix SQLite tests.
-3. If you think there's a fix for the "fixed column set" gotcha...
-4. Naming suggestions: should "document" be called "setters" or "attributes"? Should "stream" be "batch" instead?
+1. Fix SQLite tests.
+2. If you think there's a fix for the "fixed column set" gotcha...
+3. Naming suggestions: should "document" be called "setters" or "attributes"?
 
 ## Real-world usage
 
@@ -219,7 +218,7 @@ You could also use [activerecord-import](https://github.com/zdennis/activerecord
 
     Pet.import columns, all_values, :timestamps => false, :on_duplicate_key_update => columns
 
-This, however, only works on MySQL and requires ActiveRecord—and if all you are doing is upserts, `upsert` is tested to be 40% faster. And you don't have to put all of the rows to be upserted into a single huge array - you can stream them using `Upsert.stream`.
+This, however, only works on MySQL and requires ActiveRecord—and if all you are doing is upserts, `upsert` is tested to be 40% faster. And you don't have to put all of the rows to be upserted into a single huge array - you can batch them using `Upsert.batch`.
 
 ## Copyright
 

data/lib/upsert.rb

@@ -16,7 +16,7 @@ class Upsert
       Binary.new v
     end
 
-    # @yield [Upsert] An +Upsert+ object in streaming mode. You can call #row on it multiple times and it will try to optimize on speed.
+    # @yield [Upsert] An +Upsert+ object in batch mode. You can call #row on it multiple times and it will try to optimize on speed.
     #
     # @note Buffered in memory until it's efficient to send to the server a packet.
     #
@@ -25,16 +25,19 @@ class Upsert
     # @return [nil]
     #
     # @example Many at once
-    #   Upsert.stream(Pet.connection, Pet.table_name) do |upsert|
+    #   Upsert.batch(Pet.connection, Pet.table_name) do |upsert|
     #     upsert.row({:name => 'Jerry'}, :breed => 'beagle')
     #     upsert.row({:name => 'Pierre'}, :breed => 'tabby')
     #   end
-    def stream(connection, table_name)
+    def batch(connection, table_name)
       upsert = new connection, table_name
       upsert.async!
       yield upsert
       upsert.sync!
     end
+
+    # @deprecated Use .batch instead.
+    alias :stream :batch
   end
 
   # Raised if a query would be too large to send in a single packet.
@@ -58,13 +61,13 @@ class Upsert
   attr_reader :table_name
 
   # @private
-  attr_reader :rows
+  attr_reader :buffer
 
   # @param [Mysql2::Client,Sqlite3::Database,PG::Connection,#raw_connection] connection A supported database connection.
   # @param [String,Symbol] table_name The name of the table into which you will be upserting.
   def initialize(connection, table_name)
     @table_name = table_name
-    @rows = []
+    @buffer = []
 
     @connection = if connection.respond_to?(:raw_connection)
       # deal with ActiveRecord::Base.connection or ActiveRecord::Base.connection_pool.checkout
@@ -92,7 +95,7 @@ class Upsert
   #   upsert.row({:name => 'Jerry'}, :breed => 'beagle')
   #   upsert.row({:name => 'Pierre'}, :breed => 'tabby')
   def row(selector, document)
-    rows << Row.new(self, selector, document)
+    buffer.push Row.new(self, selector, document)
     if sql = chunk
       execute sql
     end

data/lib/upsert/mysql2_client.rb

@@ -1,50 +1,34 @@
 class Upsert
   # @private
   module Mysql2_Client
-    SAMPLE = 0.1
-
     def chunk
-      return if rows.empty?
-      all = rows.length
-      take = all
-      while take > 1 and probably_oversize?(take)
-        take -= 1
-      end
-      if async? and take == all
-        return
-      end
-      while take > 2 and oversize?(take)
-        $stderr.puts "   Length prediction via sampling failed, shrinking" if ENV['UPSERT_DEBUG'] == 'true'
-        take -= 2
+      return if buffer.empty?
+      if not async?
+        retval = sql
+        buffer.clear
+        return retval
       end
-      chunk = sql take
-      while take > 1 and chunk.bytesize > max_sql_bytesize
-        $stderr.puts "   Supposedly exact bytesize guess failed, shrinking" if ENV['UPSERT_DEBUG'] == 'true'
-        take -= 1
-        chunk = sql take
-      end
-      if chunk.bytesize > max_sql_bytesize
-        raise TooBig
+      @cumulative_sql_bytesize ||= static_sql_bytesize
+      new_row = buffer.pop
+      d = new_row.values_sql_bytesize + 3 # ),(
+      if @cumulative_sql_bytesize + d > max_sql_bytesize
+        retval = sql
+        buffer.clear
+        @cumulative_sql_bytesize = static_sql_bytesize + d
+      else
+        retval = nil
+        @cumulative_sql_bytesize += d
       end
-      $stderr.puts "   Chunk (#{take}/#{chunk.bytesize}) was #{(chunk.bytesize / max_sql_bytesize.to_f * 100).round}% of the max" if ENV['UPSERT_DEBUG'] == 'true'
-      @rows = rows.drop(take)
-      chunk
+      buffer.push new_row
+      retval
     end
 
     def execute(sql)
       connection.query sql
     end
 
-    def probably_oversize?(take)
-      estimate_sql_bytesize(take) > max_sql_bytesize
-    end
-
-    def oversize?(take)
-      sql_bytesize(take) > max_sql_bytesize
-    end
-
     def columns
-      @columns ||= rows.first.columns
+      @columns ||= buffer.first.columns
     end
 
     def insert_part
@@ -65,48 +49,11 @@ class Upsert
       @static_sql_bytesize ||= insert_part.bytesize + update_part.bytesize + 2
     end
 
-    
-    def variable_sql_bytesize(take)
-      memo = rows.first(take).inject(0) { |sum, row| sum + row.values_sql_bytesize }
-      if take > 0
-        # parens and comma
-        memo += 3*(take-1)
-      end
-      memo
-    end
-
-    def estimate_variable_sql_bytesize(take)
-      n = (take * SAMPLE).ceil
-      sample = if RUBY_VERSION >= '1.9'
-        rows.first(take).sample(n)
-      else
-        # based on https://github.com/marcandre/backports/blob/master/lib/backports/1.8.7/array.rb
-        memo = rows.first(take)
-        n.times do |i|
-          r = i + Kernel.rand(take - i)
-          memo[i], memo[r] = memo[r], memo[i]
-        end
-        memo.first(n)
-      end
-      memo = sample.inject(0) { |sum, row| sum + row.values_sql_bytesize } / SAMPLE
-      if take > 0
-        # parens and comma
-        memo += 3*(take-1)
-      end
-      memo
-    end
-
-    def sql_bytesize(take)
-      static_sql_bytesize + variable_sql_bytesize(take)
-    end
-
-    def estimate_sql_bytesize(take)
-      static_sql_bytesize + estimate_variable_sql_bytesize(take)
-    end
-
-    def sql(take)
-      all_value_sql = rows.first(take).map { |row| row.values_sql }
-      [ insert_part, '(', all_value_sql.join('),('), ')', update_part ].join
+    def sql
+      all_value_sql = buffer.map { |row| row.values_sql }
+      retval = [ insert_part, '(', all_value_sql.join('),('), ')', update_part ].join
+      raise TooBig if retval.bytesize > max_sql_bytesize
+      retval
     end
 
     # since setting an option like :as => :hash actually persists that option to the client, don't pass any options

data/lib/upsert/pg_connection.rb

@@ -7,8 +7,8 @@ class Upsert
     attr_reader :merge_function
 
     def chunk
-      return if rows.empty?
-      row = rows.shift
+      return if buffer.empty?
+      row = buffer.shift
       unless merge_function
         create_merge_function row
       end

data/lib/upsert/sqlite3_database.rb

@@ -2,8 +2,8 @@ class Upsert
   # @private
   module SQLite3_Database
     def chunk
-      return if rows.empty?
-      row = rows.shift
+      return if buffer.empty?
+      row = buffer.shift
       %{INSERT OR IGNORE INTO "#{table_name}" (#{row.columns_sql}) VALUES (#{row.values_sql});UPDATE "#{table_name}" SET #{row.set_sql} WHERE #{row.where_sql}}
     end
 

data/lib/upsert/version.rb

@@ -1,3 +1,3 @@
 class Upsert
-  VERSION = "0.2.2"
+  VERSION = "0.3.0"
 end

data/test/helper.rb

@@ -77,7 +77,7 @@ MiniTest::Spec.class_eval do
     
     Pet.delete_all
 
-    Upsert.stream(connection, :pets) do |upsert|
+    Upsert.batch(connection, :pets) do |upsert|
       records.each do |selector, document|
         upsert.row(selector, document)
       end
@@ -109,7 +109,7 @@ MiniTest::Spec.class_eval do
     sleep 1
 
     upsert_time = Benchmark.realtime do
-      Upsert.stream(connection, :pets) do |upsert|
+      Upsert.batch(connection, :pets) do |upsert|
         records.each do |selector, document|
           upsert.row(selector, document)
         end

data/test/shared/database.rb

@@ -44,17 +44,17 @@ shared_examples_for 'is a database with an upsert trick' do
       end
     end
   end
-  describe :stream do
+  describe :batch do
     it "works for multiple rows (base case)" do
       assert_creates(Pet, [{:name => 'Jerry', :gender => 'male'}]) do
-        Upsert.stream(connection, :pets) do |upsert|
+        Upsert.batch(connection, :pets) do |upsert|
           upsert.row({:name => 'Jerry'}, :gender => 'male')
         end
       end
     end
     it "works for multiple rows (not changing anything)" do
       assert_creates(Pet, [{:name => 'Jerry', :gender => 'male'}]) do
-        Upsert.stream(connection, :pets) do |upsert|
+        Upsert.batch(connection, :pets) do |upsert|
           upsert.row({:name => 'Jerry'}, :gender => 'male')
           upsert.row({:name => 'Jerry'}, :gender => 'male')
         end
@@ -62,7 +62,7 @@ shared_examples_for 'is a database with an upsert trick' do
     end
     it "works for multiple rows (changing something)" do
       assert_creates(Pet, [{:name => 'Jerry', :gender => 'neutered'}]) do
-        Upsert.stream(connection, :pets) do |upsert|
+        Upsert.batch(connection, :pets) do |upsert|
           upsert.row({:name => 'Jerry'}, :gender => 'male')
           upsert.row({:name => 'Jerry'}, :gender => 'neutered')
         end

data/test/shared/multibyte.rb

@@ -13,9 +13,9 @@ shared_examples_for "supports multibyte" do
       upsert.row({:name => 'I♥NY'}, {:gender => 'jÚrgen'})
     end
   end
-  it "works streaming" do
+  it "works batch" do
     assert_creates(Pet, [{:name => 'I♥NY', :gender => 'jÚrgen'}]) do
-      Upsert.stream(connection, :pets) do |upsert|
+      Upsert.batch(connection, :pets) do |upsert|
         upsert.row({:name => 'I♥NY'}, {:gender => 'périferôl'})
         upsert.row({:name => 'I♥NY'}, {:gender => 'jÚrgen'})
       end

data/test/shared/threaded.rb

@@ -13,9 +13,9 @@ shared_examples_for 'is thread-safe' do
       end
     end
   end
-  it "is safe to use streaming" do
+  it "is safe to use batch" do
     assert_creates(Pet, [{:name => 'Jerry', :gender => 'neutered'}]) do
-      Upsert.stream(connection, :pets) do |upsert|
+      Upsert.batch(connection, :pets) do |upsert|
         ts = []
         10.times do
           ts << Thread.new do

data/test/test_mysql2.rb

@@ -38,69 +38,4 @@ describe Upsert::Mysql2_Client do
   it_also "doesn't mess with timezones"
 
   it_also "doesn't blow up on reserved words"
-
-  describe '#sql_bytesize' do
-    def assert_exact(selector_proc, document_proc, show = false)
-      upsert = Upsert.new connection, :pets
-      0.upto(256) do |i|
-        upsert.rows << Upsert::Row.new(upsert, selector_proc.call(i), document_proc.call(i))
-        i.upto(upsert.rows.length) do |take|
-          expected_sql = upsert.sql(take)
-          actual = upsert.sql_bytesize(take)
-          if show and actual != expected_sql.bytesize
-            $stderr.puts
-            $stderr.puts "Expected: #{expected_sql.bytesize}"
-            $stderr.puts "Actual: #{actual}"
-            $stderr.puts expected_sql
-          end
-          actual.must_equal expected_sql.bytesize
-        end
-      end
-    end
-    def rand_string(length)
-      # http://www.dzone.com/snippets/generate-random-string-letters
-      # Array.new(length) { (rand(122-97) + 97).chr }.join
-      if RUBY_VERSION >= '1.9'
-        Array.new(length) { rand(512).chr(Encoding::UTF_8) }.join
-      else
-        Array.new(length) { rand(512) }.pack('C*')
-      end
-    end
-    it "is exact as selector length changes" do
-      selector_proc = proc do |i|
-        { :name => rand_string(i) }
-      end
-      document_proc = proc do |i|
-        {}
-      end
-      assert_exact selector_proc, document_proc
-    end
-    it "is exact as value length changes" do
-      selector_proc = proc do |i|
-        { :name => 'Jerry' }
-      end
-      document_proc = proc do |i|
-        { :spiel => rand_string(i) }
-      end
-      assert_exact selector_proc, document_proc
-    end
-    it "is exact as both selector and value length change" do
-      selector_proc = proc do |i|
-        { :name => rand_string(i) }
-      end
-      document_proc = proc do |i|
-        { :spiel => rand_string(i) }
-      end
-      assert_exact selector_proc, document_proc
-    end
-    it "is exact with numbers too" do
-      selector_proc = proc do |i|
-        { :tag_number => rand(1e5) }
-      end
-      document_proc = proc do |i|
-        { :lovability => rand }
-      end
-      assert_exact selector_proc, document_proc
-    end
-  end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: upsert
 version: !ruby/object:Gem::Version
-  version: 0.2.2
+  version: 0.3.0
   prerelease: 
 platform: ruby
 authors: