upsert 1.0.2 → 1.1.0

data/CHANGELOG

@@ -1,3 +1,10 @@
+1.1.0 / 2012-11-26
+
+* Enhancements
+
+  * Works on JRuby using bare-metal JDBC!
+  * Simplified.
+
 1.0.2 / 2012-11-12
 
 * Bug fixes

data/Gemfile

@@ -1,4 +1,8 @@
 source 'https://rubygems.org'
 
 # Specify your gem's dependencies in upsert.gemspec
+
 gemspec
+if RUBY_PLATFORM == 'java'
+  gem 'activerecord-jdbc-adapter', github: 'jruby/activerecord-jdbc-adapter'
+end

data/README.md

@@ -1,91 +1,73 @@
 # Upsert
 
-MySQL, PostgreSQL, and SQLite all have different SQL MERGE tricks that you can use to simulate upsert. This library codifies them under a single syntax.
+Make it easy to upsert on traditional RDBMS like MySQL, PostgreSQL, and SQLite3—hey look NoSQL!. Transparently creates (and re-uses) stored procedures/functions when necessary.
+
+You pass it a bare-metal connection to the database like `Mysql2::Client` (from `mysql2` gem on MRI) or `Java::OrgPostgresqlJdbc4::Jdbc4Connection` (from `jdbc-postgres` on Jruby).
+
+As databases start to natively support SQL MERGE (which is basically upsert), this library will take advantage (but you won't have to change your code).
+
+Does **not** depend on ActiveRecord.
+
+70–90%+ faster than emulating upsert with ActiveRecord.
+
+Supports MRI and JRuby.
 
 ## Usage
 
-You pass a selector that uniquely identifies a row, whether it exists or not. You pass a set of attributes that should be set on that row. Syntax inspired by [mongo-ruby-driver's update method](http://api.mongodb.org/ruby/1.6.4/Mongo/Collection.html#update-instance_method).
+You pass a __selector__ that uniquely identifies a row, whether it exists or not. You also pass a __setter__, attributes that should be set on that row.
+
+Syntax inspired by [mongo-ruby-driver's update method](http://api.mongodb.org/ruby/1.6.4/Mongo/Collection.html#update-instance_method).
 
-### Single record
+### Basic
     
 ```ruby
 connection = Mysql2::Client.new([...])
 table_name = :pets
 upsert = Upsert.new connection, table_name
+# N times...
 upsert.row({:name => 'Jerry'}, :breed => 'beagle')
 ```
 
-If you want to use an `ActiveRecord` helper method, try:
+So just to reiterate you've got a __selector__ and a __setter__:
 
 ```ruby
-require 'upsert/active_record_upsert'
-Pet.upsert({:name => 'Jerry'}, :breed => 'beagle')
-```
-
-So just to reiterate you've got a `selector` and a `setter`:
-
-```ruby
-connection = Mysql2::Client.new([...])
-table_name = :pets
-upsert = Upsert.new connection, table_name
 selector = { :name => 'Jerry' }
 setter = { :breed => 'beagle' }
 upsert.row(selector, setter)
 ```
 
-### Multiple records (batch mode)
+### Batch mode
 
-Slightly faster.
+By organizing your upserts into a batch, we can do work behind the scenes to make them faster.
 
 ```ruby
 connection = Mysql2::Client.new([...])
 Upsert.batch(connection, :pets) do |upsert|
+  # N times...
   upsert.row({:name => 'Jerry'}, :breed => 'beagle')
   upsert.row({:name => 'Pierre'}, :breed => 'tabby')
 end
 ```
 
-Tested to be much about 80% faster on PostgreSQL, MySQL, and SQLite3 than comparable methods (see the tests, which fail if they are not faster).
-
-## Gotchas
-
-### No automatic typecasting beyond what the adapter/driver provides
-
-We don't have any logic to convert integers into strings, strings into integers, etc. in order to satisfy PostgreSQL's strictness on this issue.
-
-So if you try to upsert a blank string (`''`) into an integer field in PostgreSQL, you will get a `PG::Error`.
+Batch mode is tested to be about 80% faster on PostgreSQL, MySQL, and SQLite3 than other ways to emulate upsert (see the tests, which fail if they are not faster).
 
-### Within a batch, it's assumed that you're always passing the same columns
-
-Currently, on MySQL, the first row you pass in determines the columns that will be used for all future upserts using the same Upsert object. That's useful for mass importing of many rows with the same columns, but is surprising if you're trying to use a single `Upsert` object to add arbitrary data. For example:
+### ActiveRecord helper method
 
 ```ruby
-# won't work - doesn't use same columns
-Upsert.batch(Pet.connection, Pet.table_name) do |upsert|
-  upsert.row({:name => 'Jerry'}, :breed => 'beagle')
-  upsert.row({:tag_number => 456}, :spiel => 'great cat')
-end
-```
-
-You would need to use a new `Upsert` object. On the other hand, this is totally fine:
-
-```ruby
-# totally fine
+require 'upsert/active_record_upsert'
+# N times...
 Pet.upsert({:name => 'Jerry'}, :breed => 'beagle')
-Pet.upsert({:tag_number => 456}, :spiel => 'great cat')
 ```
 
-Hopefully this surprising behavior won't exist in the future!
-
 ## Wishlist
 
 Pull requests for any of these would be greatly appreciated:
 
-1. More correctness tests! What is the dictionary definition of "upsert," anyway?
+1. Cache JDBC PreparedStatement objects.
+1. Optional "assume-merge-function-exists" mode. Currently, the fact that a merge function has been created is memoized per-process.
 1. Sanity check my three benchmarks (four if you include activerecord-import on MySQL). Do they accurately represent optimized alternatives?
 1. Provide `require 'upsert/debug'` that will make sure you are selecting on columns that have unique indexes
-1. Make `Upsert` instances accept arbitrary columns, which is what people probably expect. (this should work on PostgreSQL and SQLite3 already)
-1. JRuby support
+1. Test that `Upsert` instances accept arbitrary columns, even within a batch, which is what people probably expect.
 
 ## Real-world usage
 
@@ -98,18 +80,66 @@ We use `upsert` for [big data processing at Brighter Planet](http://brighterplan
 
 Originally written to speed up the [`data_miner`](https://github.com/seamusabshere/data_miner) data mining library.
 
-## Supported databases
+## Supported databases/drivers
+
+<table>
+  <tr>
+    <th>*</th>
+    <th>MySQL</th>
+    <th>PostgreSQL</th>
+    <th>SQLite3</th>
+  </tr>
+  <tr>
+    <th>MRI</th>
+    <td><a href="https://rubygems.org/gems/mysql2">mysql2</a></td>
+    <td><a href="https://rubygems.org/gems/pg">pg</a></td>
+    <td><a href="https://rubygems.org/gems/sqlite3">sqlite3</a></td>
+  </tr>
+  <tr>
+    <th>JRuby</th>
+    <td><a href="https://rubygems.org/gems/jdbc-mysql">jdbc-mysql</a></td>
+    <td><a href="https://rubygems.org/gems/jdbc-postgres">jdbc-postgres</a></td>
+    <td><a href="https://rubygems.org/gems/jdbc-sqlite3">jdbc-sqlite3</a></td>
+  </tr>
+</table>
+
+See below for details about what SQL MERGE trick (emulation of upsert) is used, performance, code examples, etc.
+
+### Rails / ActiveRecord
+
+(assuming that one of the other three supported drivers is being used under the covers)
+
+```ruby
+Upsert.new Pet.connection, Pet.table_name
+```
+
+#### Speed
+
+Depends on the driver being used!
+
+#### SQL MERGE trick
+
+Depends on the driver being used!
 
 ### MySQL
 
-Using the [mysql2](https://rubygems.org/gems/mysql2) driver.
+On MRI, use the [mysql2](https://rubygems.org/gems/mysql2) driver.
 
 ```ruby
+require 'mysql2'
 connection = Mysql2::Connection.new(:username => 'root', :password => 'password', :database => 'upsert_test')
 table_name = :pets
 upsert = Upsert.new(connection, table_name)
 ```
 
+On JRuby, use the [jdbc-mysql](https://rubygems.org/gems/jdbc-mysql) driver.
+
+```ruby
+require 'jdbc/mysql'
+java.sql.DriverManager.register_driver com.mysql.jdbc.Driver.new
+connection = java.sql.DriverManager.get_connection "jdbc:mysql://127.0.0.1/mydatabase?user=root&password=password"
+```
+
 #### Speed
 
 From the tests (updated 11/7/12):
@@ -161,14 +191,23 @@ END
 
 ### PostgreSQL
 
-Using the [pg](https://rubygems.org/gems/pg) driver.
+On MRI, use the [pg](https://rubygems.org/gems/pg) driver.
 
 ```ruby
+require 'pg'
 connection = PG.connect(:dbname => 'upsert_test')
 table_name = :pets
 upsert = Upsert.new(connection, table_name)
 ```
 
+On JRuby, use the [jdbc-postgres](https://rubygems.org/gems/jdbc-postgres) driver.
+
+```ruby
+require 'jdbc/postgres'
+java.sql.DriverManager.register_driver org.postgresql.Driver.new
+connection = java.sql.DriverManager.get_connection "jdbc:postgresql://127.0.0.1/mydatabase?user=root&password=password"
+```
+
 #### Speed
 
 From the tests (updated 9/21/12):
@@ -220,14 +259,24 @@ I slightly modified it so that it only retries once - don't want infinite loops.
 
 ### Sqlite
 
-Using the [sqlite3](https://rubygems.org/gems/sqlite3) driver.
+On MRI, use the [sqlite3](https://rubygems.org/gems/sqlite3) driver.
 
 ```ruby
+require 'sqlite3'
 connection = SQLite3::Database.open(':memory:')
 table_name = :pets
 upsert = Upsert.new(connection, table_name)
 ```
 
+On JRuby, use the [jdbc-sqlite3](https://rubygems.org/gems/jdbc-sqlite3) driver.
+
+```ruby
+# TODO somebody please verify
+require 'jdbc/sqlite3'
+java.sql.DriverManager.register_driver org.sqlite.Driver.new
+connection = java.sql.DriverManager.get_connection "jdbc:sqlite://127.0.0.1/mydatabase?user=root&password=password"
+```
+
 #### Speed
 
 From the tests (updated 9/21/12):
@@ -246,22 +295,6 @@ INSERT OR IGNORE INTO visits VALUES (127.0.0.1, 1);
 UPDATE visits SET visits = 1 WHERE ip LIKE 127.0.0.1;
 ```
 
-### Rails / ActiveRecord
-
-(assuming that one of the other three supported drivers is being used under the covers)
-
-```ruby
-Upsert.new Pet.connection, Pet.table_name
-```
-
-#### Speed
-
-Depends on the driver being used!
-
-#### SQL MERGE trick
-
-Depends on the driver being used!
-
 ## Features
 
 ### Tested to be fast and portable
@@ -292,7 +325,23 @@ You could also use [activerecord-import](https://github.com/zdennis/activerecord
 Pet.import columns, all_values, :timestamps => false, :on_duplicate_key_update => columns
 ```
 
-This, however, only works on MySQL and requires ActiveRecord&mdash;and if all you are doing is upserts, `upsert` is tested to be 40% faster. And you don't have to put all of the rows to be upserted into a single huge array - you can batch them using `Upsert.batch`.
+`activerecord-import`, however, only works on MySQL and requires ActiveRecord&mdash;and if all you are doing is upserts, `upsert` is tested to be 40% faster. And you don't have to put all of the rows to be upserted into a single huge array - you can batch them using `Upsert.batch`.
+
+## Gotchas
+
+### No automatic typecasting beyond what the adapter/driver provides
+
+We don't have any logic to convert integers into strings, strings into integers, etc. in order to satisfy PostgreSQL/etc.'s strictness on this issue.
+
+So if you try to upsert a blank string (`''`) into an integer field in PostgreSQL, you will get an error.
+
+### Dates and times are converted to UTC
+
+Datetimes are immediately converted to UTC and sent to the database as ISO8601 strings.
+
+If you're using MySQL, make sure server/connection timezone is UTC. If you're using Rails and/or ActiveRecord, you might want to check `ActiveRecord::Base.default_timezone`... it should probably be `:utc`.
+
+In general, run some upserts and make sure datetimes get persisted like you expect.
 
 ## Copyright
 

data/Rakefile

@@ -2,20 +2,31 @@
 require "bundler/gem_tasks"
 
 task :rspec_all_databases do
-  require 'posix-spawn'
-  %w{ postgresql mysql2 sqlite3 }.each do |adapter|
+  results = {}
+  %w{ postgresql mysql sqlite3 }.each do |db|
     puts
     puts '#'*50
-    puts "# Running specs against #{adapter}"
+    puts "# Running specs against #{db}"
     puts '#'*50
     puts
-    pid = POSIX::Spawn.spawn({'ADAPTER' => adapter}, 'rspec', '--format', 'documentation', File.expand_path('../spec', __FILE__))
+    # won't work on 1.8.7...
+    pid = Kernel.spawn({'DB' => db}, 'rspec', '--format', 'documentation', File.expand_path('../spec', __FILE__))
     Process.waitpid pid
-    raise unless $?.success?
+    results[db] = $?.success? 
   end
+  puts results.inspect
 end
 
 task :default => :rspec_all_databases
 
+task :n, :from, :to do |t, args|
+  Dir[File.expand_path("../lib/upsert/**/#{args.from}.*", __FILE__)].each do |path|
+    dir = File.dirname(path)
+    File.open("#{dir}/#{args.to}.rb", 'w') do |f|
+      f.write File.read(path).gsub(args.from, args.to)
+    end
+  end
+end
+
 require 'yard'
 YARD::Rake::YardocTask.new

data/lib/upsert.rb

@@ -8,7 +8,6 @@ require 'upsert/connection'
 require 'upsert/merge_function'
 require 'upsert/column_definition'
 require 'upsert/row'
-require 'upsert/cell'
 
 class Upsert
   class << self
@@ -37,7 +36,7 @@ class Upsert
       end
     end
 
-    # @param [Mysql2::Client,Sqlite3::Database,PG::Connection,#raw_connection] connection A supported database connection.
+    # @param [Mysql2::Client,Sqlite3::Database,PG::Connection,#metal] connection A supported database connection.
     #
     # Clear any database functions that may have been created.
     #
@@ -56,7 +55,7 @@ class Upsert
 
     # More efficient way of upserting multiple rows at once.
     #
-    # @param [Mysql2::Client,Sqlite3::Database,PG::Connection,#raw_connection] connection A supported database connection.
+    # @param [Mysql2::Client,Sqlite3::Database,PG::Connection,#metal] connection A supported database connection.
     # @param [String,Symbol] table_name The name of the table into which you will be upserting.
     #
     # @yield [Upsert] An +Upsert+ object in batch mode. You can call #row on it multiple times and it will try to optimize on speed.
@@ -75,6 +74,66 @@ class Upsert
 
     # @deprecated Use .batch instead.
     alias :stream :batch
+
+    # @private
+    def class_name(metal)
+      if RUBY_PLATFORM == 'java'
+        metal.class.name || metal.get_class.name
+      else
+        metal.class.name
+      end
+    end
+
+    # @private
+    def flavor(metal)
+      case class_name(metal)
+      when /sqlite/i
+        'Sqlite3'
+      when /mysql/i
+        'Mysql'
+      when /pg/i, /postgres/i
+        'Postgresql'
+      else
+        raise "[upsert] #{metal} not supported"
+      end
+    end
+
+    # @private
+    def adapter(metal)
+      metal_class_name = class_name metal
+      METAL_CLASS_ALIAS.fetch(metal_class_name, metal_class_name).gsub /\W+/, '_'
+    end
+
+    # @private
+    def metal(connection)
+      metal = connection.respond_to?(:raw_connection) ? connection.raw_connection : connection
+      if metal.class.name.to_s.start_with?('ActiveRecord::ConnectionAdapters')
+        metal = metal.connection
+      end
+      metal
+    end
+
+    # @private
+    def utc(time)
+      if time.is_a? DateTime
+        usec = time.sec_fraction * SEC_FRACTION
+        if time.offset != 0
+          time = time.new_offset(0)
+        end
+        Time.utc time.year, time.month, time.day, time.hour, time.min, time.sec, usec
+      elsif time.utc?
+        time
+      else
+        time.utc
+      end
+    end
+
+    # @private
+    def utc_iso8601(time, tz = true)
+      t = utc time
+      s = t.strftime(ISO8601_DATETIME) + '.' + (USEC_SPRINTF % t.usec)
+      tz ? (s + UTC_TZ) : s
+    end
   end
 
   SINGLE_QUOTE = %{'}
@@ -82,14 +141,20 @@ class Upsert
   BACKTICK = %{`}
   X_AND_SINGLE_QUOTE = %{x'}
   USEC_SPRINTF = '%06d'
+  if RUBY_VERSION >= '1.9.0'
+    SEC_FRACTION = 1e6
+    NANO_FRACTION = 1e9
+  else
+    SEC_FRACTION = 8.64e10
+    NANO_FRACTION = 8.64e13
+  end
   ISO8601_DATETIME = '%Y-%m-%d %H:%M:%S'
   ISO8601_DATE = '%F'
+  UTC_TZ = '+00:00'
   NULL_WORD = 'NULL'
-  HANDLER = {
-    'SQLite3::Database' => 'SQLite3_Database',
-    'PGConn'            => 'PG_Connection',
-    'PG::Connection'    => 'PG_Connection',
-    'Mysql2::Client'    => 'Mysql2_Client',
+  METAL_CLASS_ALIAS = {
+    'PGConn'           => 'PG::Connection',
+    'org.sqlite.Conn'  => 'Java::OrgSqliteConn' # for some reason, org.sqlite.Conn doesn't have a ruby class name
   }
 
   # @return [Upsert::Connection]
@@ -99,31 +164,27 @@ class Upsert
   attr_reader :table_name
 
   # @private
-  attr_reader :row_class
-
-  # @private
-  attr_reader :cell_class
+  attr_reader :merge_function_class
 
   # @private
-  attr_reader :column_definition_class
+  attr_reader :flavor
 
   # @private
-  attr_reader :merge_function_class
+  attr_reader :adapter
 
-  # @param [Mysql2::Client,Sqlite3::Database,PG::Connection,#raw_connection] connection A supported database connection.
+  # @param [Mysql2::Client,Sqlite3::Database,PG::Connection,#metal] connection A supported database connection.
   # @param [String,Symbol] table_name The name of the table into which you will be upserting.
   def initialize(connection, table_name)
     @table_name = table_name.to_s
-    raw_connection = connection.respond_to?(:raw_connection) ? connection.raw_connection : connection
-    connection_class_name = HANDLER[raw_connection.class.name]
-    Dir[File.expand_path("../upsert/**/#{connection_class_name}.rb", __FILE__)].each do |path|
+    metal = Upsert.metal connection
+    @flavor = Upsert.flavor metal
+    @adapter = Upsert.adapter metal
+    # todo memoize
+    Dir[File.expand_path("../upsert/**/{#{flavor.downcase},#{adapter}}.rb", __FILE__)].each do |path|
       require path
     end
-    @connection = Connection.const_get(connection_class_name).new self, raw_connection
-    @row_class = Row.const_get connection_class_name
-    @cell_class = Cell.const_get connection_class_name
-    @column_definition_class = ColumnDefinition.const_get connection_class_name
-    @merge_function_class = MergeFunction.const_get connection_class_name
+    @connection = Connection.const_get(adapter).new self, metal
+    @merge_function_class = MergeFunction.const_get adapter
   end
 
   # Upsert a row given a selector and a setter.
@@ -142,7 +203,7 @@ class Upsert
   #   upsert.row({:name => 'Jerry'}, :breed => 'beagle')
   #   upsert.row({:name => 'Pierre'}, :breed => 'tabby')
   def row(selector, setter = {})
-    merge_function_class.execute self, row_class.new(self, selector, setter)
+    merge_function_class.execute self, Row.new(selector, setter)
     nil
   end
 
@@ -158,6 +219,6 @@ class Upsert
 
   # @private
   def column_definitions
-    @column_definitions ||= column_definition_class.all connection, table_name
+    @column_definitions ||= ColumnDefinition.const_get(flavor).all connection, table_name
   end
 end