upsert 0.4.0 → 0.5.0

data/CHANGELOG

@@ -1,3 +1,25 @@
+0.5.0 / 2012-09-21
+
+* Breaking changes (well, not really)
+
+  * "document" (as in the second argument to #row) has been renamed to "setter"!
+
+* Bug fixes
+
+  * If you say upsert({:name => 'Jerry', :color => 'red'}), make sure that it only affects rows really meeting those conditions
+  * Always sort selector and setter keys - i.e., column names - before doing anything with them
+  * Support PostgreSQL 9.1+
+  * Support MRI 1.8
+
+* Enhancements
+
+  * Slightly faster benchmarks for SQlite3 and MySQL
+  * Slightly slower on PostgreSQL (probably because the merge function requires more arguments), but more accurate
+  * Slightly clearer code structure
+  * Use bind parameters instead of quoting for PostgreSQL and SQLite3.
+  * Provide Upsert.clear_database_functions(connection) (currently only for PostgreSQL)
+  * Don't subclass String for Upset::Binary... hopefully save some strcpy()s?
+
 0.4.0 / 2012-09-04
 
 * Bug fixes
@@ -18,7 +40,7 @@
 
 * Enhancements
 
-  * Make document an optional argument
+  * Make setter an optional argument
 
 0.3.3 / 2012-06-26
 

data/README.md

@@ -26,7 +26,7 @@ Rows are buffered in memory until it's efficient to send them to the database.
       upsert.row({:name => 'Pierre'}, :breed => 'tabby')
     end
 
-Tested to be much about 85% faster on PostgreSQL and 50% faster on MySQL than comparable methods (see the tests).
+Tested to be much about 60% faster on PostgreSQL and 60–90% faster on MySQL and SQLite3 than comparable methods (see the tests, which fail if they are not faster).
 
 ## Gotchas
 
@@ -34,9 +34,9 @@ Tested to be much about 85% faster on PostgreSQL and 50% faster on MySQL than co
 
 Make sure you're upserting against either primary key columns or columns with UNIQUE indexes or both.
 
-### Columns are set based on the first row you pass
+### For MySQL, columns are set based on the first row you pass
 
-Currently, the first row you pass in determines the columns that will be used. That's useful for mass importing of many rows with the same columns, but is surprising if you're trying to use a single `Upsert` object to add arbitrary data. For example, this won't work:
+Currently, on MySQL, the first row you pass in determines the columns that will be used for all future upserts using the same Upsert object. That's useful for mass importing of many rows with the same columns, but is surprising if you're trying to use a single `Upsert` object to add arbitrary data. For example, this won't work:
 
     Upsert.batch(Pet.connection, Pet.table_name) do |upsert|
       upsert.row({:name => 'Jerry'}, :breed => 'beagle')
@@ -52,9 +52,11 @@ You would need to use a new `Upsert` object. On the other hand, this is totally
 
 Pull requests for any of these would be greatly appreciated:
 
+1. More correctness tests! What is the dictionary definition of "upsert," anyway?
+1. Sanity check my three benchmarks (four if you include activerecord-import on MySQL). Do they accurately represent optimized alternatives?
 1. Provide `require 'upsert/debug'` that will make sure you are selecting on columns that have unique indexes
-1. Make `Upsert` instances accept arbitrary columns, which is what people probably expect. (this should work on PG already)
-1. Naming suggestions: should "document" be called "setters" or "attributes"?
+1. Make `Upsert` instances accept arbitrary columns, which is what people probably expect. (this should work on PostgreSQL and SQLite3 already)
+1. JRuby support
 
 ## Real-world usage
 
@@ -73,16 +75,16 @@ Originally written to speed up the [`data_miner`](https://github.com/seamusabshe
 
 Using the [mysql2](https://rubygems.org/gems/mysql2) driver.
 
-    Upsert.new Mysql2::Connection.new([...]), :pets
+    upsert = Upsert.new(Mysql2::Connection.new(:username => 'root', :password => 'password', :database => 'upsert_test'), :pets)
 
 #### Speed
 
-From the tests:
+From the tests (updated 9/21/12):
 
-    Upsert was 77% faster than find + new/set/save
-    Upsert was 58% faster than create + rescue/find/update
-    Upsert was 80% faster than find_or_create + update_attributes
-    Upsert was 39% faster than faking upserts with activerecord-import
+    Upsert was 88% faster than find + new/set/save
+    Upsert was 90% faster than create + rescue/find/update
+    Upsert was 90% faster than find_or_create + update_attributes
+    Upsert was 60% faster than faking upserts with activerecord-import
 
 #### SQL MERGE trick
 
@@ -92,21 +94,25 @@ From the tests:
     INSERT INTO table (a,b,c) VALUES (1,2,3), (4,5,6)
       ON DUPLICATE KEY UPDATE a=VALUES(a),b=VALUES(b),c=VALUES(c);
 
+If `a` only appeared in the selector, then we avoid updating it in case of a duplicate key:
+
+    ON DUPLICATE KEY UPDATE a=a,b=VALUES(b),c=VALUES(c);
+
 Since this is an upsert helper library, not a general-use ON DUPLICATE KEY UPDATE wrapper, you **can't** do things like `c=c+1`.
 
 ### PostgreSQL
 
 Using the [pg](https://rubygems.org/gems/pg) driver.
 
-    Upsert.new PG.connect([...]), :pets
+    upsert = Upsert.new(PG.connect(:dbname => 'upsert_test'), :pets)
 
 #### Speed
 
-From the tests:
+From the tests (updated 9/21/12):
 
-    Upsert was 73% faster than find + new/set/save
-    Upsert was 84% faster than find_or_create + update_attributes
-    Upsert was 87% faster than create + rescue/find/update
+    Upsert was 65% faster than find + new/set/save
+    Upsert was 79% faster than find_or_create + update_attributes
+    Upsert was 76% faster than create + rescue/find/update
     # (can't compare to activerecord-import because you can't fake it on pg)
 
 #### SQL MERGE trick
@@ -138,6 +144,8 @@ From the tests:
     SELECT merge_db(1, 'david');
     SELECT merge_db(1, 'dennis');
 
+I slightly modified it so that it only retries once - don't want infinite loops.
+
 The decision was made **not** to use the following because it's not straight from the manual:
 
     # http://stackoverflow.com/questions/1109061/insert-on-duplicate-update-postgresql
@@ -166,11 +174,16 @@ This was also rejected because there's something we can use in the manual:
 
 Using the [sqlite3](https://rubygems.org/gems/sqlite3) driver.
 
-    Upsert.new SQLite3::Database.open([...]), :pets
+    upsert = Upsert.new(SQLite3::Database.open(':memory:'), :pets)
 
 #### Speed
 
-FIXME tests are segfaulting. Pull request would be lovely.
+From the tests (updated 9/21/12):
+
+    Upsert was 77% faster than find + new/set/save
+    Upsert was 80% faster than find_or_create + update_attributes
+    Upsert was 85% faster than create + rescue/find/update
+    # (can't compare to activerecord-import because you can't fake it on sqlite3)
 
 #### SQL MERGE trick
 

data/Rakefile

@@ -11,6 +11,7 @@ task :rspec_all_databases do
     puts
     pid = POSIX::Spawn.spawn({'ADAPTER' => adapter}, 'rspec', '--format', 'documentation', File.expand_path('../spec', __FILE__))
     Process.waitpid pid
+    raise unless $?.success?
   end
 end
 

data/lib/upsert.rb

@@ -7,6 +7,7 @@ require 'upsert/binary'
 require 'upsert/buffer'
 require 'upsert/connection'
 require 'upsert/row'
+require 'upsert/cell'
 
 class Upsert
   class << self
@@ -34,6 +35,16 @@ class Upsert
       end
     end
 
+    # @param [Mysql2::Client,Sqlite3::Database,PG::Connection,#raw_connection] connection A supported database connection.
+    #
+    # Clear any database functions that may have been created.
+    #
+    # Currently only applies to PostgreSQL.
+    def clear_database_functions(connection)
+      dummy = new(connection, :dummy)
+      dummy.buffer.clear_database_functions
+    end
+
     # @param [String] v A string containing binary data that should be inserted/escaped as such.
     #
     # @return [Upsert::Binary]
@@ -41,9 +52,14 @@ class Upsert
       Binary.new v
     end
 
-    # @yield [Upsert] An +Upsert+ object in batch mode. You can call #row on it multiple times and it will try to optimize on speed.
+    # Guarantee that the most efficient way of buffering rows is used.
+    #
+    # Currently mostly helps for MySQL, but you should use it whenever possible in case future buffering-based optimizations become possible.
     #
-    # @note Buffered in memory until it's efficient to send to the server a packet.
+    # @param [Mysql2::Client,Sqlite3::Database,PG::Connection,#raw_connection] connection A supported database connection.
+    # @param [String,Symbol] table_name The name of the table into which you will be upserting.
+    #
+    # @yield [Upsert] An +Upsert+ object in batch mode. You can call #row on it multiple times and it will try to optimize on speed.
     #
     # @return [nil]
     #
@@ -66,12 +82,17 @@ class Upsert
   SINGLE_QUOTE = %{'}
   DOUBLE_QUOTE = %{"}
   BACKTICK = %{`}
-  E_AND_SINGLE_QUOTE = %{E'}
   X_AND_SINGLE_QUOTE = %{x'}
   USEC_SPRINTF = '%06d'
   ISO8601_DATETIME = '%Y-%m-%d %H:%M:%S'
   ISO8601_DATE = '%F'
   NULL_WORD = 'NULL'
+  HANDLER = {
+    'SQLite3::Database' => 'SQLite3_Database',
+    'PGConn'            => 'PG_Connection',
+    'PG::Connection'    => 'PG_Connection',
+    'Mysql2::Client'    => 'Mysql2_Client',
+  }
 
   # @return [Upsert::Connection]
   attr_reader :connection
@@ -82,22 +103,32 @@ class Upsert
   # @private
   attr_reader :buffer
 
+  # @private
+  attr_reader :row_class
+
+  # @private
+  attr_reader :cell_class
+
   # @param [Mysql2::Client,Sqlite3::Database,PG::Connection,#raw_connection] connection A supported database connection.
   # @param [String,Symbol] table_name The name of the table into which you will be upserting.
   def initialize(connection, table_name)
     @table_name = table_name.to_s
     raw_connection = connection.respond_to?(:raw_connection) ? connection.raw_connection : connection
-    n = raw_connection.class.name.gsub(/\W+/, '_')
-    @connection = Connection.const_get(n).new self, raw_connection
-    @buffer = Buffer.const_get(n).new self
+    connection_class_name = HANDLER[raw_connection.class.name]
+    @connection = Connection.const_get(connection_class_name).new self, raw_connection
+    @buffer = Buffer.const_get(connection_class_name).new self
+    @row_class = Row.const_get connection_class_name
+    @cell_class = Cell.const_get connection_class_name
   end
 
-  # Upsert a row given a selector and a document.
+  # Upsert a row given a selector and a setter.
+  #
+  # The selector values are used as setters if it's a new row. So if your selector is `name=Jerry` and your setter is `age=4`, and there is no Jerry yet, then a new row will be created with name Jerry and age 4.
   #
   # @see http://api.mongodb.org/ruby/1.6.4/Mongo/Collection.html#update-instance_method Loosely based on the upsert functionality of the mongo-ruby-driver #update method
   #
   # @param [Hash] selector Key-value pairs that will be used to find or create a row.
-  # @param [Hash] document Key-value pairs that will be set on the row, whether it previously existed or not.
+  # @param [Hash] setter Key-value pairs that will be set on the row, whether it previously existed or not.
   #
   # @return [nil]
   #
@@ -105,8 +136,8 @@ class Upsert
   #   upsert = Upsert.new Pet.connection, Pet.table_name
   #   upsert.row({:name => 'Jerry'}, :breed => 'beagle')
   #   upsert.row({:name => 'Pierre'}, :breed => 'tabby')
-  def row(selector, document = {})
-    buffer << Row.new(self, selector, document)
+  def row(selector, setter = {})
+    buffer << row_class.new(self, selector, setter)
     nil
   end
 

data/lib/upsert/active_record_upsert.rb

@@ -1,9 +1,9 @@
 class Upsert
   module ActiveRecordUpsert
-    def upsert(selector, document = {})
+    def upsert(selector, setter = {})
       ActiveRecord::Base.connection_pool.with_connection do |c|
         upsert = Upsert.new c, table_name
-        upsert.row selector, document
+        upsert.row selector, setter
       end
     end
   end

data/lib/upsert/binary.rb

@@ -2,6 +2,5 @@ class Upsert
   # A wrapper class for binary strings so that Upsert knows to escape them as such.
   #
   # Create them with +Upsert.binary(x)+
-  class Binary < ::String
-  end
+  Binary = Struct.new(:value)
 end

data/lib/upsert/buffer/mysql2_client.rb

@@ -4,9 +4,9 @@ class Upsert
     class Mysql2_Client < Buffer
       def ready
         return if rows.empty?
-        c = parent.connection
+        connection = parent.connection
         if not async?
-          c.execute sql
+          connection.execute sql
           rows.clear
           return
         end
@@ -14,7 +14,7 @@ class Upsert
         new_row = rows.pop
         d = new_row.values_sql_bytesize + 3 # ),(
         if @cumulative_sql_bytesize + d > max_sql_bytesize
-          c.execute sql
+          connection.execute sql
           rows.clear
           @cumulative_sql_bytesize = static_sql_bytesize + d
         else
@@ -24,24 +24,33 @@ class Upsert
         nil
       end
 
-      def columns
-        @columns ||= rows.first.columns
+      def setter
+        @setter ||= rows.first.setter.keys
+      end
+
+      def original_setter
+        @original_setter ||= rows.first.original_setter_keys
       end
 
       def insert_part
         @insert_part ||= begin
           connection = parent.connection
-          columns_sql = columns.map { |k| connection.quote_ident(k) }.join(',')
-          %{INSERT INTO #{parent.quoted_table_name} (#{columns_sql}) VALUES }
+          column_names = setter.map { |k| connection.quote_ident(k) }
+          %{INSERT INTO #{parent.quoted_table_name} (#{column_names.join(',')}) VALUES }
         end
       end
 
       def update_part
         @update_part ||= begin
           connection = parent.connection
-          updaters = columns.map do |k|
-            qk = connection.quote_ident(k)
-            [ qk, "VALUES(#{qk})" ].join('=')
+          updaters = setter.map do |k|
+            quoted_name = connection.quote_ident(k)
+            if original_setter.include?(k)
+              "#{quoted_name}=VALUES(#{quoted_name})"
+            else
+              # NOOP
+              "#{quoted_name}=#{quoted_name}"
+            end
           end.join(',')
           %{ ON DUPLICATE KEY UPDATE #{updaters}}
         end
@@ -53,9 +62,13 @@ class Upsert
       end
 
       def sql
-        all_value_sql = rows.map { |row| row.values_sql }
-        retval = [ insert_part, '(', all_value_sql.join('),('), ')', update_part ].join
-        retval
+        [
+          insert_part,
+          '(',
+          rows.map { |row| row.quoted_setter_values.join(',') }.join('),('),
+          ')',
+          update_part
+        ].join
       end
 
       # since setting an option like :as => :hash actually persists that option to the client, don't pass any options

data/lib/upsert/buffer/pg_connection.rb

@@ -1,3 +1,4 @@
+require 'upsert/buffer/pg_connection/column_definition'
 require 'upsert/buffer/pg_connection/merge_function'
 
 class Upsert
@@ -7,48 +8,12 @@ class Upsert
       def ready
         return if rows.empty?
         row = rows.shift
-        MergeFunction.execute(self, row)
+        MergeFunction.execute self, row
       end
 
       def clear_database_functions
-        connection = parent.connection
-        # http://stackoverflow.com/questions/7622908/postgresql-drop-function-without-knowing-the-number-type-of-parameters
-        connection.execute <<-EOS
-CREATE OR REPLACE FUNCTION pg_temp.upsert_delfunc(text)
-  RETURNS void AS
-$BODY$
-DECLARE
-   _sql text;
-BEGIN
-
-FOR _sql IN
-   SELECT 'DROP FUNCTION ' || quote_ident(n.nspname)
-                    || '.' || quote_ident(p.proname)
-           || '(' || pg_catalog.pg_get_function_identity_arguments(p.oid) || ');'
-   FROM   pg_catalog.pg_proc p
-   LEFT   JOIN pg_catalog.pg_namespace n ON n.oid = p.pronamespace
-   WHERE  p.proname = $1
-   AND    pg_catalog.pg_function_is_visible(p.oid) -- you may or may not want this
-LOOP
-   EXECUTE _sql;
-END LOOP;
-
-END;
-$BODY$
-  LANGUAGE plpgsql;
-EOS
-        res = connection.execute(%{SELECT proname FROM pg_proc WHERE proname LIKE 'upsert_%'})
-        res.each do |row|
-          k = row['proname']
-          next if k == 'upsert_delfunc'
-          Upsert.logger.info %{[upsert] Dropping function #{k.inspect}}
-          connection.execute %{SELECT pg_temp.upsert_delfunc('#{k}')}
-        end
+        MergeFunction.clear self
       end
     end
-
-    # @private
-    # backwards compatibility - https://github.com/seamusabshere/upsert/issues/2
-    PGconn = PG_Connection
   end
 end

data/lib/upsert/buffer/pg_connection/column_definition.rb

@@ -0,0 +1,59 @@
+class Upsert
+  class Buffer
+    class PG_Connection < Buffer
+      # @private
+      class ColumnDefinition
+        class << self
+          # activerecord-3.2.5/lib/active_record/connection_adapters/postgresql_adapter.rb#column_definitions
+          def all(buffer, table_name)
+            connection = buffer.parent.connection
+            res = connection.execute <<-EOS
+SELECT a.attname AS name, format_type(a.atttypid, a.atttypmod) AS sql_type, d.adsrc AS default
+FROM pg_attribute a LEFT JOIN pg_attrdef d
+  ON a.attrelid = d.adrelid AND a.attnum = d.adnum
+WHERE a.attrelid = '#{connection.quote_ident(table_name)}'::regclass
+AND a.attnum > 0 AND NOT a.attisdropped
+EOS
+            res.map do |row|
+              new connection, row['name'], row['sql_type'], row['default']
+            end.sort_by do |cd|
+              cd.name
+            end
+          end
+        end
+
+        attr_reader :name
+        attr_reader :sql_type
+        attr_reader :default
+        attr_reader :quoted_name
+        attr_reader :quoted_selector_name
+        attr_reader :quoted_setter_name
+
+        def initialize(connection, name, sql_type, default)
+          @name = name
+          @sql_type = sql_type
+          @default = default
+          @quoted_name = connection.quote_ident name
+          @quoted_selector_name = connection.quote_ident "#{name}_selector"
+          @quoted_setter_name = connection.quote_ident "#{name}_setter"
+        end
+
+        def to_selector_arg
+          "#{quoted_selector_name} #{sql_type}"
+        end
+
+        def to_setter_arg
+          "#{quoted_setter_name} #{sql_type}"
+        end
+
+        def to_setter
+          "#{quoted_name} = #{quoted_setter_name}"
+        end
+
+        def to_selector
+          "#{quoted_name} = #{quoted_selector_name}"
+        end
+      end
+    end
+  end
+end