multisert 0.0.1 → 0.0.2

data/LICENSE

@@ -1,22 +1,13 @@
-Copyright (c) 2013 Jeff Iacono
+Copyright 2013 Jeff Iacono
 
-MIT License
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
 
-Permission is hereby granted, free of charge, to any person obtaining
-a copy of this software and associated documentation files (the
-"Software"), to deal in the Software without restriction, including
-without limitation the rights to use, copy, modify, merge, publish,
-distribute, sublicense, and/or sell copies of the Software, and to
-permit persons to whom the Software is furnished to do so, subject to
-the following conditions:
+    http://www.apache.org/licenses/LICENSE-2.0
 
-The above copyright notice and this permission notice shall be
-included in all copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
-EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
-MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
-NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
-LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
-OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
-WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

data/README.md

@@ -1,6 +1,7 @@
 # Multisert
 
-TODO: Write a gem description
+Multisert is a buffer that handles bundling up INSERTs, which increases runtime
+performance.
 
 ## Installation
 
@@ -70,6 +71,52 @@ script. This ensures that any pending entries are written to the database table
 that were not automatically taken care of by the auto-flush that will kick in
 during the iteration.
 
+## Performance
+
+The gem has a quick performance test built in that can be run via:
+```bash
+$ ruby ./performance/multisert_performance_test
+```
+We ran the performance test (with some modification to iterate the test 5
+times) and receive the following output:
+
+```bash
+$ ruby ./performance/multisert_performance_test
+# test 1:
+#   insert w/o buffer took 53.37s to insert 100000 entries
+#   multisert w/ buffer of 10000 took 1.77s to insert 100000 entries
+#
+# test 2:
+#   insert w/o buffer took 53.22s to insert 100000 entries
+#   multisert w/ buffer of 10000 took 1.84s to insert 100000 entries
+#
+# test 3:
+#   insert w/o buffer took 54.42s to insert 100000 entries
+#   multisert w/ buffer of 10000 took 1.9s to insert 100000 entries
+#
+# test 4:
+#   insert w/o buffer took 53.38s to insert 100000 entries
+#   multisert w/ buffer of 10000 took 1.81s to insert 100000 entries
+#
+# test 5:
+#   insert w/o buffer took 53.52s to insert 100000 entries
+#   multisert w/ buffer of 10000 took 1.78s to insert 100000 entries
+```
+
+As we can see, ~30x performance increase.
+
+The performance test was run on a computer with the following specs:
+
+    Model Name:             MacBook Air
+    Model Identifier:       MacBookAir4,2
+    Processor Name:         Intel Core i5
+    Processor Speed:        1.7 GHz
+    Number of Processors:   1
+    Total Number of Cores:  2
+    L2 Cache (per Core):    256 KB
+    L3 Cache:               3 MB
+    Memory:                 4 GB
+
 ## FAQ
 
 ### Packet Too Large / Connection Lost Errors
@@ -83,6 +130,20 @@ To learn more, [read the documentation](http://dev.mysql.com/doc/refman/5.5/en//
 If you need to you can adjust the buffer size by setting `max_buffer_count`
 attribute. Generally, 10,000 to 100,000 is a pretty good starting range.
 
+### Does it work with Dates?
+
+Yes, just pass in a Date instance and it will be converted to a mysql
+friendly format under the hood ("%Y-%m-%d"). If you need a special format,
+convert the date to a string that is in the form you want before passing it into
+Multisert.
+
+### Does it work with Times?
+
+Yes, just pass in a Time instance and it will be converted to a mysql
+friendly format under the hood ("%Y-%m-%d %H:%M:%S"). If you need a special
+format, convert the time to a string that is in the form you want before passing
+it into Multisert.
+
 ## Contributing
 
 1. Fork it
@@ -90,3 +151,19 @@ attribute. Generally, 10,000 to 100,000 is a pretty good starting range.
 3. Commit your changes (`git commit -am 'Added some feature'`)
 4. Push to the branch (`git push origin my-new-feature`)
 5. Create new Pull Request
+
+## License
+
+Copyright 2013 Jeff Iacono
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

data/lib/multisert.rb

@@ -68,13 +68,11 @@ private
 
   def cast value
     case value
-    when String
-      # TODO: want to escape the string too, checking for " and ;
-      "'#{value}'"
-    when Date
-      "'#{value}'"
-    else
-      value
+    # TODO: want to escape the string too, checking for " and ;
+    when String then "'#{value}'"
+    when Date   then "'#{value.strftime("%Y-%m-%d")}'"
+    when Time   then "'#{value.strftime("%Y-%m-%d %H:%M:%S")}'"
+    else value
     end
   end
 end

data/lib/multisert/version.rb

@@ -1,3 +1,3 @@
 class Multisert
-  VERSION = "0.0.1"
+  VERSION = "0.0.2"
 end

data/performance/multisert_performance_test.rb

@@ -0,0 +1,85 @@
+require './performance/performance_helper'
+
+PERFORMANCE_DATABASE    = 'multisert_performance'
+PERFORMANCE_TABLE       = 'performance_data'
+PERFORMANCE_DESTINATION = "#{PERFORMANCE_DATABASE}.#{PERFORMANCE_TABLE}"
+NUM_OF_OPERATIONS       = 100_000
+CONNECTION              = Mysql2::Client.new(host: 'localhost', username: 'root')
+
+def puts_with_time content
+  puts "[#{Time.now}] #{content}"
+end
+
+def generate_records records_count = NUM_OF_OPERATIONS
+  puts_with_time "generating #{records_count} random entries"
+  sample_records = (0...records_count).reduce([]) do |memo, i|
+    memo << {'field_1' => i,
+             'field_2' => i + 1,
+             'field_3' => i + 2,
+             'field_4' => i + 3}
+    memo
+  end
+  puts_with_time "generated #{records_count} random entries"
+  sample_records
+end
+
+def ensure_data_completeness! connection, datastore, expected_count
+  unless (res = connection.query("SELECT COUNT(*) AS the_count FROM #{datastore}").to_a.first['the_count']) == expected_count
+    raise RuntimeError, "data not written completely. Got #{res}, expected #{expected_count}"
+  end
+end
+
+def insert_performance_test connection, cleaner, sample_records, destination
+  fields = sample_records.first.keys.join(', ')
+
+  cleaner.ensure_clean_database!
+
+  (timer = Timer.new).start!
+  sample_records.each do |record|
+    connection.query %[
+      INSERT INTO #{destination} (#{fields})
+      VALUES (#{record.map { |k,v| v }.join(', ')})]
+  end
+  runtime = timer.stop!
+  ensure_data_completeness! connection, destination, sample_records.count
+  puts "insert w/o buffer took #{runtime.round(2)}s to insert #{sample_records.count} entries"
+end
+
+def multinsert_performance_test connection, cleaner, sample_records, destination, max_buffer_count = nil
+  database, table = destination.split('.')
+
+  buffer = Multisert.new connection:       connection,
+                         database:         database,
+                         table:            table,
+                         fields:           sample_records.first.keys,
+                         max_buffer_count: max_buffer_count
+
+  cleaner.ensure_clean_database!
+
+  (timer = Timer.new).start!
+  sample_records.each do |record|
+    buffer << record.map { |k, v| v }
+  end
+  buffer.flush!
+  runtime = timer.stop!
+  ensure_data_completeness! connection, destination, sample_records.count
+  puts "multisert w/ buffer of #{buffer.max_buffer_count} took #{runtime.round(2)}s to insert #{sample_records.count} entries"
+end
+
+cleaner = MrClean.new(database: PERFORMANCE_DATABASE, connection: CONNECTION)
+cleaner.create_table_schemas << %[
+  CREATE TABLE IF NOT EXISTS #{PERFORMANCE_DATABASE}.#{PERFORMANCE_TABLE} (
+      field_1 int default null
+    , field_2 int default null
+    , field_3 int default null
+    , field_4 int default null
+  )]
+
+sample_records = generate_records
+
+puts_with_time "starting performance test: using #{sample_records.count} random entries, writing to #{PERFORMANCE_DESTINATION}"
+
+#insert_performance_test     CONNECTION, cleaner, sample_records, PERFORMANCE_DESTINATION
+(0..10_000).step(10) do |i|
+  multinsert_performance_test CONNECTION, cleaner, sample_records, PERFORMANCE_DESTINATION, i
+end

data/performance/performance_helper.rb

@@ -0,0 +1,62 @@
+require 'rubygems'
+require 'bundler'
+Bundler.require
+require 'mysql2'
+require './lib/multisert'
+
+class Timer
+  def start!
+    @start = Time.now
+  end
+
+  def stop!
+    Time.now - @start
+  end
+end
+
+# TODO: convert to gem
+class MrClean
+  attr_accessor :connection, :database, :create_table_schemas
+
+  def initialize attrs = {}
+    @connection           = attrs[:connection]
+    @database             = attrs[:database]
+    @create_table_schemas = attrs[:create_table_schemas] || []
+    yield self if block_given?
+  end
+
+  def ensure_clean_database! opts = {}
+    clean_database! !!opts[:teardown_tables]
+    ensure_tables!
+  end
+
+private
+
+  def database_exists?
+    @connection.query('show databases').to_a.map { |database|
+      database['Database']
+    }.include?(@database)
+  end
+
+  def ensure_database!
+    @connection.query "create database if not exists #{@database}"
+  end
+
+  def clean_database! teardown_tables
+    return unless database_exists?
+    @connection.query("show tables in #{@database}").to_a.each do |table|
+      if teardown_tables
+        @connection.query("drop table if exists #{@database}.#{table["Tables_in_#{@database}"]}")
+      else
+        @connection.query("truncate #{@database}.#{table["Tables_in_#{@database}"]}")
+      end
+    end
+  end
+
+  def ensure_tables!
+    ensure_database!
+    @create_table_schemas.each do |create_table_schema|
+      @connection.query create_table_schema
+    end
+  end
+end

data/spec/multisert_spec.rb

@@ -9,7 +9,7 @@ TEST_TABLE    = 'test_data'
 # TODO: make into yaml config
 $connection = Mysql2::Client.new(host: 'localhost', username: 'root')
 
-$cleaner = MrClean.new(database: TEST_DATABASE, connection: $connection) do |mgr|
+$cleaner = MultisertSpec::MrClean.new(database: TEST_DATABASE, connection: $connection) do |mgr|
   mgr.create_table_schemas << %[
     CREATE TABLE IF NOT EXISTS #{mgr.database}.#{TEST_TABLE} (
       test_field_int_1 int default null,
@@ -17,7 +17,8 @@ $cleaner = MrClean.new(database: TEST_DATABASE, connection: $connection) do |mgr
       test_field_int_3 int default null,
       test_field_int_4 int default null,
       test_field_varchar varchar(10) default null,
-      test_field_date DATE default null
+      test_field_date DATE default null,
+      test_field_datetime DATETIME default null
     )]
 end
 
@@ -147,5 +148,32 @@ describe Multisert do
 
       buffer.entries.should == []
     end
+
+    it "works with times" do
+      pre_flush_records = connection.query "SELECT * FROM #{TEST_DATABASE}.#{TEST_TABLE}"
+      pre_flush_records.to_a.should == []
+
+      buffer.connection = connection
+      buffer.database   = TEST_DATABASE
+      buffer.table      = TEST_TABLE
+      buffer.fields     = ['test_field_datetime']
+
+      buffer << [Time.new(2013, 1, 15, 1, 5, 11)]
+      buffer << [Time.new(2013, 1, 16, 2, 6, 22)]
+      buffer << [Time.new(2013, 1, 17, 3, 7, 33)]
+      buffer << [Time.new(2013, 1, 18, 4, 8, 44)]
+
+      buffer.flush!
+
+      post_flush_records = connection.query %[SELECT test_field_datetime FROM #{TEST_DATABASE}.#{TEST_TABLE}]
+
+      post_flush_records.to_a.should == [
+        {'test_field_datetime' => Time.new(2013, 1, 15, 1, 5, 11)},
+        {'test_field_datetime' => Time.new(2013, 1, 16, 2, 6, 22)},
+        {'test_field_datetime' => Time.new(2013, 1, 17, 3, 7, 33)},
+        {'test_field_datetime' => Time.new(2013, 1, 18, 4, 8, 44)}]
+
+      buffer.entries.should == []
+    end
   end
 end

data/spec/spec_helper.rb

@@ -1,46 +1,47 @@
-class MrClean
-  attr_accessor :connection, :database, :create_table_schemas
+module MultisertSpec
+  class MrClean
+    attr_accessor :connection, :database, :create_table_schemas
 
-  def initialize attrs = {}
-    @connection           = attrs[:connection]
-    @database             = attrs[:database]
-    @create_table_schemas = attrs[:create_table_schemas] || []
-    yield self if block_given?
-  end
+    def initialize attrs = {}
+      @connection           = attrs[:connection]
+      @database             = attrs[:database]
+      @create_table_schemas = attrs[:create_table_schemas] || []
+      yield self if block_given?
+    end
 
-  def ensure_clean_database! opts = {}
-    clean_database! !!opts[:teardown_tables]
-    ensure_tables!
-  end
+    def ensure_clean_database! opts = {}
+      clean_database! !!opts[:teardown_tables]
+      ensure_tables!
+    end
 
-private
+  private
 
-  def database_exists?
-    @connection.query('show databases').to_a.map { |database|
-      database['Database']
-    }.include?(@database)
-  end
+    def database_exists?
+      @connection.query('show databases').to_a.map { |database|
+        database['Database']
+      }.include?(@database)
+    end
 
-  def ensure_database!
-    @connection.query "create database if not exists #{@database}"
-  end
+    def ensure_database!
+      @connection.query "create database if not exists #{@database}"
+    end
 
-  def clean_database! teardown_tables
-    return unless database_exists?
-    @connection.query("show tables in #{@database}").to_a.each do |table|
-      if teardown_tables
-puts "TEARING DOWN"
-        @connection.query("drop table if exists #{@database}.#{table["Tables_in_#{@database}"]}")
-      else
-        @connection.query("truncate #{@database}.#{table["Tables_in_#{@database}"]}")
+    def clean_database! teardown_tables
+      return unless database_exists?
+      @connection.query("show tables in #{@database}").to_a.each do |table|
+        if teardown_tables
+          @connection.query("drop table if exists #{@database}.#{table["Tables_in_#{@database}"]}")
+        else
+          @connection.query("truncate #{@database}.#{table["Tables_in_#{@database}"]}")
+        end
       end
     end
-  end
 
-  def ensure_tables!
-    ensure_database!
-    @create_table_schemas.each do |create_table_schema|
-      @connection.query create_table_schema
+    def ensure_tables!
+      ensure_database!
+      @create_table_schemas.each do |create_table_schema|
+        @connection.query create_table_schema
+      end
     end
   end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: multisert
 version: !ruby/object:Gem::Version
-  version: 0.0.1
+  version: 0.0.2
   prerelease: 
 platform: ruby
 authors:
@@ -9,7 +9,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2013-03-07 00:00:00.000000000 Z
+date: 2013-03-09 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: mysql2
@@ -91,6 +91,8 @@ files:
 - lib/multisert.rb
 - lib/multisert/version.rb
 - multisert.gemspec
+- performance/multisert_performance_test.rb
+- performance/performance_helper.rb
 - spec/multisert_spec.rb
 - spec/spec_helper.rb
 homepage: https://github.com/jeffreyiacono/multisert