upsert 0.5.0 → 1.0.2

data/CHANGELOG

@@ -1,3 +1,32 @@
+1.0.2 / 2012-11-12
+
+* Bug fixes
+
+  * Fix filenames - they were updated on an apparently case-insensitive setup. Thanks @ihough! (https://github.com/seamusabshere/upsert/pull/8)
+  * Deliberately drop MySQL procedures before creating them. Also thanks to @ihough!
+
+1.0.1 / 2012-11-07
+
+* Bug fixes
+
+  * Fix incorrect gem description
+
+1.0.0 / 2012-11-07
+
+* Breaking changes (well, not really)
+
+  * Not using INSERT ... ON DUPLICATE KEY UPDATE for MySQL!
+
+* Enhancements
+
+  * Replaced ON DUPLICATE KEY with a true merge function (procedure)
+  * Simplified code - buffering is no longer used anywhere
+  * Clarified documentation
+
+* Bug fixes
+
+  * MySQL upserts won't fail if you have a multi-key selector and no multi-column UNIQUE index to cover them (https://github.com/seamusabshere/upsert/issues/6)
+
 0.5.0 / 2012-09-21
 
 * Breaking changes (well, not really)

data/README.md

@@ -7,46 +7,75 @@ MySQL, PostgreSQL, and SQLite all have different SQL MERGE tricks that you can u
 You pass a selector that uniquely identifies a row, whether it exists or not. You pass a set of attributes that should be set on that row. Syntax inspired by [mongo-ruby-driver's update method](http://api.mongodb.org/ruby/1.6.4/Mongo/Collection.html#update-instance_method).
 
 ### Single record
-
-    # if you have required 'upsert/active_record_upsert'
-    Pet.upsert({:name => 'Jerry'}, :breed => 'beagle')
-
-    # if you're not using activerecord, that's ok
-    connection = Mysql2::Client.new([...])
-    upsert = Upsert.new connection, 'pets'
-    upsert.row({:name => 'Jerry'}, :breed => 'beagle')
+    
+```ruby
+connection = Mysql2::Client.new([...])
+table_name = :pets
+upsert = Upsert.new connection, table_name
+upsert.row({:name => 'Jerry'}, :breed => 'beagle')
+```
+
+If you want to use an `ActiveRecord` helper method, try:
+
+```ruby
+require 'upsert/active_record_upsert'
+Pet.upsert({:name => 'Jerry'}, :breed => 'beagle')
+```
+
+So just to reiterate you've got a `selector` and a `setter`:
+
+```ruby
+connection = Mysql2::Client.new([...])
+table_name = :pets
+upsert = Upsert.new connection, table_name
+selector = { :name => 'Jerry' }
+setter = { :breed => 'beagle' }
+upsert.row(selector, setter)
+```
 
 ### Multiple records (batch mode)
 
-Rows are buffered in memory until it's efficient to send them to the database.
+Slightly faster.
 
-    connection = Mysql2::Client.new([...])
-    Upsert.batch(connection, 'pets') do |upsert|
-      upsert.row({:name => 'Jerry'}, :breed => 'beagle')
-      upsert.row({:name => 'Pierre'}, :breed => 'tabby')
-    end
+```ruby
+connection = Mysql2::Client.new([...])
+Upsert.batch(connection, :pets) do |upsert|
+  upsert.row({:name => 'Jerry'}, :breed => 'beagle')
+  upsert.row({:name => 'Pierre'}, :breed => 'tabby')
+end
+```
 
-Tested to be much about 60% faster on PostgreSQL and 60–90% faster on MySQL and SQLite3 than comparable methods (see the tests, which fail if they are not faster).
+Tested to be much about 80% faster on PostgreSQL, MySQL, and SQLite3 than comparable methods (see the tests, which fail if they are not faster).
 
 ## Gotchas
 
-### Undefined behavior without real UNIQUE indexes
+### No automatic typecasting beyond what the adapter/driver provides
 
-Make sure you're upserting against either primary key columns or columns with UNIQUE indexes or both.
+We don't have any logic to convert integers into strings, strings into integers, etc. in order to satisfy PostgreSQL's strictness on this issue.
 
-### For MySQL, columns are set based on the first row you pass
+So if you try to upsert a blank string (`''`) into an integer field in PostgreSQL, you will get a `PG::Error`.
 
-Currently, on MySQL, the first row you pass in determines the columns that will be used for all future upserts using the same Upsert object. That's useful for mass importing of many rows with the same columns, but is surprising if you're trying to use a single `Upsert` object to add arbitrary data. For example, this won't work:
+### Within a batch, it's assumed that you're always passing the same columns
 
-    Upsert.batch(Pet.connection, Pet.table_name) do |upsert|
-      upsert.row({:name => 'Jerry'}, :breed => 'beagle')
-      upsert.row({:tag_number => 456}, :spiel => 'great cat') # won't work - doesn't use same columns
-    end
+Currently, on MySQL, the first row you pass in determines the columns that will be used for all future upserts using the same Upsert object. That's useful for mass importing of many rows with the same columns, but is surprising if you're trying to use a single `Upsert` object to add arbitrary data. For example:
+
+```ruby
+# won't work - doesn't use same columns
+Upsert.batch(Pet.connection, Pet.table_name) do |upsert|
+  upsert.row({:name => 'Jerry'}, :breed => 'beagle')
+  upsert.row({:tag_number => 456}, :spiel => 'great cat')
+end
+```
 
 You would need to use a new `Upsert` object. On the other hand, this is totally fine:
 
-    Pet.upsert({:name => 'Jerry'}, :breed => 'beagle')
-    Pet.upsert({:tag_number => 456}, :spiel => 'great cat')
+```ruby
+# totally fine
+Pet.upsert({:name => 'Jerry'}, :breed => 'beagle')
+Pet.upsert({:tag_number => 456}, :spiel => 'great cat')
+```
+
+Hopefully this surprising behavior won't exist in the future!
 
 ## Wishlist
 
@@ -75,106 +104,129 @@ Originally written to speed up the [`data_miner`](https://github.com/seamusabshe
 
 Using the [mysql2](https://rubygems.org/gems/mysql2) driver.
 
-    upsert = Upsert.new(Mysql2::Connection.new(:username => 'root', :password => 'password', :database => 'upsert_test'), :pets)
+```ruby
+connection = Mysql2::Connection.new(:username => 'root', :password => 'password', :database => 'upsert_test')
+table_name = :pets
+upsert = Upsert.new(connection, table_name)
+```
 
 #### Speed
 
-From the tests (updated 9/21/12):
+From the tests (updated 11/7/12):
 
-    Upsert was 88% faster than find + new/set/save
+    Upsert was 82% faster than find + new/set/save
+    Upsert was 85% faster than find_or_create + update_attributes
     Upsert was 90% faster than create + rescue/find/update
-    Upsert was 90% faster than find_or_create + update_attributes
-    Upsert was 60% faster than faking upserts with activerecord-import
+    Upsert was 46% faster than faking upserts with activerecord-import
 
 #### SQL MERGE trick
 
-"ON DUPLICATE KEY UPDATE" where we just set everything to the value of the insert.
-
-    # http://dev.mysql.com/doc/refman/5.0/en/insert-on-duplicate.html
-    INSERT INTO table (a,b,c) VALUES (1,2,3), (4,5,6)
-      ON DUPLICATE KEY UPDATE a=VALUES(a),b=VALUES(b),c=VALUES(c);
-
-If `a` only appeared in the selector, then we avoid updating it in case of a duplicate key:
+Thanks to [Dennis Hennen's StackOverflow response!](http://stackoverflow.com/questions/11371479/how-to-translate-postgresql-merge-db-aka-upsert-function-into-mysql/)!
 
-    ON DUPLICATE KEY UPDATE a=a,b=VALUES(b),c=VALUES(c);
-
-Since this is an upsert helper library, not a general-use ON DUPLICATE KEY UPDATE wrapper, you **can't** do things like `c=c+1`.
+```sql
+CREATE PROCEDURE upsert_pets_SEL_name_A_tag_number_SET_name_A_tag_number(`name_sel` varchar(255), `tag_number_sel` int(11), `name_set` varchar(255), `tag_number_set` int(11))
+BEGIN
+  DECLARE done BOOLEAN;
+  REPEAT
+    BEGIN
+      -- If there is a unique key constraint error then 
+      -- someone made a concurrent insert. Reset the sentinel
+      -- and try again.
+      DECLARE ER_DUP_UNIQUE CONDITION FOR 23000;
+      DECLARE ER_INTEG CONDITION FOR 1062;
+      DECLARE CONTINUE HANDLER FOR ER_DUP_UNIQUE BEGIN
+        SET done = FALSE;
+      END;
+      
+      DECLARE CONTINUE HANDLER FOR ER_INTEG BEGIN
+        SET done = TRUE;
+      END;
+
+      SET done = TRUE;
+      SELECT COUNT(*) INTO @count FROM `pets` WHERE `name` = `name_sel` AND `tag_number` = `tag_number_sel`;
+      -- Race condition here. If a concurrent INSERT is made after
+      -- the SELECT but before the INSERT below we'll get a duplicate
+      -- key error. But the handler above will take care of that.
+      IF @count > 0 THEN 
+        -- UPDATE table_name SET b = b_SET WHERE a = a_SEL;
+        UPDATE `pets` SET `name` = `name_set`, `tag_number` = `tag_number_set` WHERE `name` = `name_sel` AND `tag_number` = `tag_number_sel`;
+      ELSE
+        -- INSERT INTO table_name (a, b) VALUES (k, data);
+        INSERT INTO `pets` (`name`, `tag_number`) VALUES (`name_set`, `tag_number_set`);
+      END IF;
+    END;
+  UNTIL done END REPEAT;
+END
+```
 
 ### PostgreSQL
 
 Using the [pg](https://rubygems.org/gems/pg) driver.
 
-    upsert = Upsert.new(PG.connect(:dbname => 'upsert_test'), :pets)
+```ruby
+connection = PG.connect(:dbname => 'upsert_test')
+table_name = :pets
+upsert = Upsert.new(connection, table_name)
+```
 
 #### Speed
 
 From the tests (updated 9/21/12):
 
-    Upsert was 65% faster than find + new/set/save
+    Upsert was 72% faster than find + new/set/save
     Upsert was 79% faster than find_or_create + update_attributes
-    Upsert was 76% faster than create + rescue/find/update
+    Upsert was 83% faster than create + rescue/find/update
     # (can't compare to activerecord-import because you can't fake it on pg)
 
 #### SQL MERGE trick
 
-    # http://www.postgresql.org/docs/current/interactive/plpgsql-control-structures.html#PLPGSQL-ERROR-TRAPPING
-    CREATE TABLE db (a INT PRIMARY KEY, b TEXT);
-    CREATE FUNCTION merge_db(key INT, data TEXT) RETURNS VOID AS
-    $$
+Adapted from the [canonical PostgreSQL upsert example](http://www.postgresql.org/docs/current/interactive/plpgsql-control-structures.html#PLPGSQL-ERROR-TRAPPING):
+
+```sql
+CREATE OR REPLACE FUNCTION upsert_pets_SEL_name_A_tag_number_SET_name_A_tag_number("name_sel" character varying(255), "tag_number_sel" integer, "name_set" character varying(255), "tag_number_set" integer) RETURNS VOID AS
+$$
+DECLARE
+  first_try INTEGER := 1;
+BEGIN
+  LOOP
+    -- first try to update the key
+    UPDATE "pets" SET "name" = "name_set", "tag_number" = "tag_number_set"
+      WHERE "name" = "name_sel" AND "tag_number" = "tag_number_sel";
+    IF found THEN
+      RETURN;
+    END IF;
+    -- not there, so try to insert the key
+    -- if someone else inserts the same key concurrently,
+    -- we could get a unique-key failure
     BEGIN
-        LOOP
-            -- first try to update the key
-            UPDATE db SET b = data WHERE a = key;
-            IF found THEN
-                RETURN;
-            END IF;
-            -- not there, so try to insert the key
-            -- if someone else inserts the same key concurrently,
-            -- we could get a unique-key failure
-            BEGIN
-                INSERT INTO db(a,b) VALUES (key, data);
-                RETURN;
-            EXCEPTION WHEN unique_violation THEN
-                -- Do nothing, and loop to try the UPDATE again.
-            END;
-        END LOOP;
+      INSERT INTO "pets"("name", "tag_number") VALUES ("name_set", "tag_number_set");
+      RETURN;
+    EXCEPTION WHEN unique_violation THEN
+      -- seamusabshere 9/20/12 only retry once
+      IF (first_try = 1) THEN
+        first_try := 0;
+      ELSE
+        RETURN;
+      END IF;
+      -- Do nothing, and loop to try the UPDATE again.
     END;
-    $$
-    LANGUAGE plpgsql;
-    SELECT merge_db(1, 'david');
-    SELECT merge_db(1, 'dennis');
+  END LOOP;
+END;
+$$
+LANGUAGE plpgsql;
+```
 
 I slightly modified it so that it only retries once - don't want infinite loops.
 
-The decision was made **not** to use the following because it's not straight from the manual:
-
-    # http://stackoverflow.com/questions/1109061/insert-on-duplicate-update-postgresql
-    UPDATE table SET field='C', field2='Z' WHERE id=3;
-    INSERT INTO table (id, field, field2)
-      SELECT 3, 'C', 'Z'
-      WHERE NOT EXISTS (SELECT 1 FROM table WHERE id=3);
-
-This was also rejected because there's something we can use in the manual:
-
-    # http://stackoverflow.com/questions/5269590/why-doesnt-this-rule-prevent-duplicate-key-violations
-    BEGIN;
-    CREATE TEMP TABLE stage_data(key_column, data_columns...) ON COMMIT DROP;
-    \copy stage_data from data.csv with csv header
-    -- prevent any other updates while we are merging input (omit this if you don't need it)
-    LOCK target_data IN SHARE ROW EXCLUSIVE MODE;
-    -- insert into target table
-    INSERT INTO target_data(key_column, data_columns...)
-       SELECT key_column, data_columns...
-       FROM stage_data
-       WHERE NOT EXISTS (SELECT 1 FROM target_data
-                         WHERE target_data.key_column = stage_data.key_column)
-    END;
-
 ### Sqlite
 
 Using the [sqlite3](https://rubygems.org/gems/sqlite3) driver.
 
-    upsert = Upsert.new(SQLite3::Database.open(':memory:'), :pets)
+```ruby
+connection = SQLite3::Database.open(':memory:')
+table_name = :pets
+upsert = Upsert.new(connection, table_name)
+```
 
 #### Speed
 
@@ -187,16 +239,20 @@ From the tests (updated 9/21/12):
 
 #### SQL MERGE trick
 
-    # http://stackoverflow.com/questions/2717590/sqlite-upsert-on-duplicate-key-update
-    # bad example because we're not doing on-duplicate-key update
-    INSERT OR IGNORE INTO visits VALUES (127.0.0.1, 1);
-    UPDATE visits SET visits = 1 WHERE ip LIKE 127.0.0.1;
+Thanks to [@dan04's answer on StackOverflow](http://stackoverflow.com/questions/2717590/sqlite-upsert-on-duplicate-key-update):
+
+```sql
+INSERT OR IGNORE INTO visits VALUES (127.0.0.1, 1);
+UPDATE visits SET visits = 1 WHERE ip LIKE 127.0.0.1;
+```
 
 ### Rails / ActiveRecord
 
 (assuming that one of the other three supported drivers is being used under the covers)
 
-    Upsert.new Pet.connection, Pet.table_name
+```ruby
+Upsert.new Pet.connection, Pet.table_name
+```
 
 #### Speed
 
@@ -219,22 +275,26 @@ In addition to correctness, the library's tests check that it is
 
 As below, all you need is a raw database connection like a `Mysql2::Connection`, `PG::Connection` or a `SQLite3::Database`. These are equivalent:
 
-    # with activerecord
-    Upsert.new ActiveRecord::Base.connection, :pets
-    # with activerecord, prettier
-    Upsert.new Pet.connection, Pet.table_name
-    # without activerecord
-    Upsert.new Mysql2::Connection.new([...]), :pets
+```ruby
+# with activerecord
+Upsert.new ActiveRecord::Base.connection, :pets
+# with activerecord, prettier
+Upsert.new Pet.connection, Pet.table_name
+# without activerecord
+Upsert.new Mysql2::Connection.new([...]), :pets
+```
 
 ### For a specific use case, faster and more portable than `activerecord-import`
 
 You could also use [activerecord-import](https://github.com/zdennis/activerecord-import) to upsert:
 
-    Pet.import columns, all_values, :timestamps => false, :on_duplicate_key_update => columns
+```ruby
+Pet.import columns, all_values, :timestamps => false, :on_duplicate_key_update => columns
+```
 
 This, however, only works on MySQL and requires ActiveRecord—and if all you are doing is upserts, `upsert` is tested to be 40% faster. And you don't have to put all of the rows to be upserted into a single huge array - you can batch them using `Upsert.batch`.
 
 ## Copyright
 
-Copyright 2012 Brighter Planet, Inc.
+Copyright 2012 Seamus Abshere
 

data/lib/upsert.rb

@@ -4,8 +4,9 @@ require 'logger'
 
 require 'upsert/version'
 require 'upsert/binary'
-require 'upsert/buffer'
 require 'upsert/connection'
+require 'upsert/merge_function'
+require 'upsert/column_definition'
 require 'upsert/row'
 require 'upsert/cell'
 
@@ -25,13 +26,14 @@ class Upsert
           ar_logger
         else
           my_logger = Logger.new $stderr
-          my_logger.level = Logger::INFO
+          case ENV['UPSERT_DEBUG']
+          when 'true'
+            my_logger.level = Logger::DEBUG
+          when 'false'
+            my_logger.level = Logger::INFO
+          end
           my_logger
         end
-        if ENV['UPSERT_DEBUG'] == 'true'
-          @logger.level = Logger::DEBUG
-        end
-        @logger
       end
     end
 
@@ -42,7 +44,7 @@ class Upsert
     # Currently only applies to PostgreSQL.
     def clear_database_functions(connection)
       dummy = new(connection, :dummy)
-      dummy.buffer.clear_database_functions
+      dummy.clear_database_functions
     end
 
     # @param [String] v A string containing binary data that should be inserted/escaped as such.
@@ -52,9 +54,7 @@ class Upsert
       Binary.new v
     end
 
-    # Guarantee that the most efficient way of buffering rows is used.
-    #
-    # Currently mostly helps for MySQL, but you should use it whenever possible in case future buffering-based optimizations become possible.
+    # More efficient way of upserting multiple rows at once.
     #
     # @param [Mysql2::Client,Sqlite3::Database,PG::Connection,#raw_connection] connection A supported database connection.
     # @param [String,Symbol] table_name The name of the table into which you will be upserting.
@@ -70,9 +70,7 @@ class Upsert
     #   end
     def batch(connection, table_name)
       upsert = new connection, table_name
-      upsert.buffer.async!
       yield upsert
-      upsert.buffer.sync!
     end
 
     # @deprecated Use .batch instead.
@@ -100,25 +98,32 @@ class Upsert
   # @return [String]
   attr_reader :table_name
 
-  # @private
-  attr_reader :buffer
-
   # @private
   attr_reader :row_class
 
   # @private
   attr_reader :cell_class
 
+  # @private
+  attr_reader :column_definition_class
+
+  # @private
+  attr_reader :merge_function_class
+
   # @param [Mysql2::Client,Sqlite3::Database,PG::Connection,#raw_connection] connection A supported database connection.
   # @param [String,Symbol] table_name The name of the table into which you will be upserting.
   def initialize(connection, table_name)
     @table_name = table_name.to_s
     raw_connection = connection.respond_to?(:raw_connection) ? connection.raw_connection : connection
     connection_class_name = HANDLER[raw_connection.class.name]
+    Dir[File.expand_path("../upsert/**/#{connection_class_name}.rb", __FILE__)].each do |path|
+      require path
+    end
     @connection = Connection.const_get(connection_class_name).new self, raw_connection
-    @buffer = Buffer.const_get(connection_class_name).new self
     @row_class = Row.const_get connection_class_name
     @cell_class = Cell.const_get connection_class_name
+    @column_definition_class = ColumnDefinition.const_get connection_class_name
+    @merge_function_class = MergeFunction.const_get connection_class_name
   end
 
   # Upsert a row given a selector and a setter.
@@ -137,12 +142,22 @@ class Upsert
   #   upsert.row({:name => 'Jerry'}, :breed => 'beagle')
   #   upsert.row({:name => 'Pierre'}, :breed => 'tabby')
   def row(selector, setter = {})
-    buffer << row_class.new(self, selector, setter)
+    merge_function_class.execute self, row_class.new(self, selector, setter)
     nil
   end
 
+  # @private
+  def clear_database_functions
+    merge_function_class.clear connection
+  end
+
   # @private
   def quoted_table_name
     @quoted_table_name ||= connection.quote_ident table_name
   end
+
+  # @private
+  def column_definitions
+    @column_definitions ||= column_definition_class.all connection, table_name
+  end
 end