mini_sql 0.2.2 → 1.0

data/Gemfile

@@ -1,6 +1,8 @@
+# frozen_string_literal: true
+
 source "https://rubygems.org"
 
-git_source(:github) {|repo_name| "https://github.com/#{repo_name}" }
+git_source(:github) { |repo_name| "https://github.com/#{repo_name}" }
 
 # Specify your gem's dependencies in mini_sql.gemspec
 gemspec

data/Guardfile

@@ -1,3 +1,5 @@
+# frozen_string_literal: true
+
 guard :minitest do
   watch(%r{^test/(.*)_test\.rb$})
   watch(%r{^lib/(.*/)?([^/]+)\.rb$})     { |m| "test/#{m[1]}#{m[2]}_test.rb" }

data/README.md

@@ -32,11 +32,28 @@ conn.query("select 1 id, 'bob' name").each do |user|
   puts user.id # 1
 end
 
+# extend result objects with additional method
+module ProductDecorator
+  def amount_price
+    price * quantity
+  end
+end
+
+conn.query_decorator(ProductDecorator, "select 20 price, 3 quantity").each do |user|
+  puts user.amount_price # 60
+end
+
 p conn.query_single('select 1 union select 2')
 # [1,2]
 
 p conn.query_hash('select 1 as a, 2 as b union select 3, 4')
 # [{"a" => 1, "b"=> 1},{"a" => 3, "b" => 4}
+ 
+p conn.query_array("select 1 as a, '2' as b union select 3, 'e'")
+# [[1, '2'], [3, 'e']]
+ 
+p conn.query_array("select 1 as a, '2' as b union select 3, 'e'").to_h
+# {1 => '2', 3 => 'e'}
 ```
 
 ## The query builder
@@ -63,9 +80,18 @@ end
 The builder allows for `order_by`, `where`, `select`, `set`, `limit`, `join`, `left_join` and `offset`.
 
 ## Is it fast?
-
 Yes, it is very fast. See benchmarks in [the bench directory](https://github.com/discourse/mini_sql/tree/master/bench).
 
+**Comparison mini_sql methods**
+```
+query_array     1351.6 i/s
+      query      963.8 i/s - 1.40x  slower
+ query_hash      787.4 i/s - 1.72x  slower
+
+query_single('select id from topics limit 1000')             2368.9 i/s
+ query_array('select id from topics limit 1000').flatten     1350.1 i/s - 1.75x  slower
+```
+
 As a rule it will outperform similar naive PG code while remaining safe.
 
 ```ruby
@@ -98,6 +124,68 @@ MiniSql is careful to always clear results as soon as possible.
 
 MiniSql's default type mapper prefers treating `timestamp without time zone` columns as utc. This is done to ensure widest amount of compatability and is a departure from the default in the PG 1.0 gem. If you wish to amend behavior feel free to pass in a custom type_map.
 
+## Custom type maps
+
+When using Postgres, native type mapping implementation is used. This is roughly
+implemented as:
+
+```ruby
+type_map ||= PG::BasicTypeMapForResults.new(conn)
+# additional specific decoders
+```
+
+The type mapper instansitated once on-demand at boot and reused by all mini_sql connections.
+
+Initializing the basic type map for Postgres can be a costly operation. You may
+wish to amend the type mapper so for example you only return strings:
+
+```
+# maybe you do not want Integer
+p cnn.query("select a 1").first.a
+"1"
+```
+
+To specify a different type mapper for your results use:
+
+```
+MiniSql::Connections.get(pg_connection, type_map: custom_type_map)
+```
+
+In the case of Rails you can opt to use the type mapper Rails uses with:
+
+```
+pg_cnn = ActiveRecord::Base.connection.raw_connection
+mini_sql_cnn = MiniSql::Connection.get(pg_cnn, type_map: pg_cnn.type_map_for_results)
+```
+
+Note the type mapper for Rails may miss some of the mapping MiniSql ships with such as `IPAddr`, MiniSql is also careful to use the very efficient TimestampUtc decoders where available.
+
+## Streaming support
+
+In some exceptional cases you may want to stream results directly from the database. This enables selection of 100s of thousands of rows with limited memory impact.
+
+Two interfaces exists for this:
+
+`query_each` : which can be used to get materialized objects  
+`query_each_hash` : which can be used to iterate through Hash objects
+
+Usage:
+
+```ruby
+mini_sql_cnn.query_each("SELECT * FROM tons_of_cows limit :limit", limit: 1_000_000) do |row|
+  puts row.cow_name
+  puts row.cow_age
+end
+
+mini_sql_cnn.query_each_hash("SELECT * FROM one_million_cows") do |row|
+  puts row["cow_name"]
+  puts row["cow_age"]
+end
+```
+
+Note, in Postgres streaming is going to be slower than non-streaming options due to internal implementation in the pq gem, each row gets a full result object and additional bookkeeping is needed. Only use it if you need to optimize memory usage.
+
+Streaming support is only implemented in the postgres backend at the moment, PRs welcome to add to other backends.
 
 ## I want more features!
 
@@ -109,7 +197,7 @@ To install this gem onto your local machine, run `bundle exec rake install`. To
 
 ## Contributing
 
-Bug reports and pull requests are welcome on GitHub at https://github.com/SamSaffron/mini_sql. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [Contributor Covenant](http://contributor-covenant.org) code of conduct.
+Bug reports and pull requests are welcome on GitHub at https://github.com/discourse/mini_sql. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [Contributor Covenant](http://contributor-covenant.org) code of conduct.
 
 ## License
 

data/Rakefile

@@ -1,3 +1,5 @@
+# frozen_string_literal: true
+
 require "bundler/gem_tasks"
 require "rake/testtask"
 
@@ -13,4 +15,4 @@ Rake::TestTask.new(:test) do |t|
   t.test_files = FileList[test_glob]
 end
 
-task :default => :test
+task default: :test

data/bench/timestamp_perf.rb

@@ -1,3 +1,5 @@
+# frozen_string_literal: true
+
 require 'bundler/inline'
 
 gemfile do
@@ -21,8 +23,8 @@ require 'benchmark/ips'
 require 'mini_sql'
 
 ActiveRecord::Base.establish_connection(
-  :adapter => "postgresql",
-  :database => "test_db"
+  adapter: "postgresql",
+  database: "test_db"
 )
 
 Sequel.default_timezone = :utc
@@ -47,20 +49,20 @@ SQL
 class Timestamp < ActiveRecord::Base
 end
 
-class TimestampSequel< Sequel::Model(:timestamps)
+class TimestampSequel < Sequel::Model(:timestamps)
 end
 
-
 Timestamp.transaction do
   stamps = {
   }
   Timestamp.columns.each do |c|
-    stamps[c.name.to_sym] = case c.type
-                           when :integer then 1
-                           when :datetime then Time.now
-                           when :boolean then false
-                           else "HELLO WORLD" * 2
-                           end
+    stamps[c.name.to_sym] =
+      case c.type
+      when :integer then 1
+      when :datetime then Time.now
+      when :boolean then false
+      else "HELLO WORLD" * 2
+      end
   end
 
   1000.times do |id|
@@ -71,7 +73,7 @@ end
 
 $conn = ActiveRecord::Base.connection.raw_connection
 
-def ar_pluck_times(l=1000)
+def ar_pluck_times(l = 1000)
   s = +""
   Timestamp.limit(l).order(:id).pluck(:time1, :time2).each do |time1, time2|
     s << time1.to_f.to_s
@@ -80,7 +82,7 @@ def ar_pluck_times(l=1000)
   s
 end
 
-def ar_select_times(l=1000)
+def ar_select_times(l = 1000)
   s = +""
   Timestamp.limit(l).order(:id).select(:time1, :time2).each do |t|
     s << t.time1.to_f.to_s
@@ -91,7 +93,7 @@ end
 
 $mini_sql = MiniSql::Connection.new($conn)
 
-def pg_times_params(l=1000)
+def pg_times_params(l = 1000)
   s = +""
   # use the safe pattern here
   r = $conn.async_exec_params(-"select time1, time2 from timestamps order by id limit $1", [l])
@@ -110,7 +112,7 @@ def pg_times_params(l=1000)
   s
 end
 
-def pg_times(l=1000)
+def pg_times(l = 1000)
   s = +""
   # use the safe pattern here
   r = $conn.async_exec("select time1, time2 from timestamps order by id limit #{l}")
@@ -129,7 +131,7 @@ def pg_times(l=1000)
   s
 end
 
-def mini_sql_times(l=1000)
+def mini_sql_times(l = 1000)
   s = +""
   $mini_sql.query(-"select time1, time2 from timestamps order by id limit ?", l).each do |t|
     s << t.time1.to_f.to_s
@@ -138,7 +140,7 @@ def mini_sql_times(l=1000)
   s
 end
 
-def sequel_times(l=1000)
+def sequel_times(l = 1000)
   s = +""
   TimestampSequel.limit(l).order(:id).select(:time1, :time2).each do |t|
     s << t.time1.to_f.to_s
@@ -147,7 +149,7 @@ def sequel_times(l=1000)
   s
 end
 
-def sequel_pluck_times(l=1000)
+def sequel_pluck_times(l = 1000)
   s = +""
   TimestampSequel.limit(l).order(:id).select_map([:time1, :time2]).each do |t|
     s << t[0].to_f.to_s
@@ -156,7 +158,7 @@ def sequel_pluck_times(l=1000)
   s
 end
 
-def sequel_raw_times(l=1000)
+def sequel_raw_times(l = 1000)
   s = +""
   DB[-"select time1, time2 from timestamps order by id limit ?", l].map([:time1, :time2]).each do |time1, time2|
     s << time1.to_f.to_s
@@ -166,13 +168,13 @@ def sequel_raw_times(l=1000)
 end
 
 # usage is not really recommended but just to compare to pluck lets have it
-def mini_sql_times_single(l=1000)
+def mini_sql_times_single(l = 1000)
   s = +""
   i = 0
   r = $mini_sql.query_single(-"select time1, time2 from timestamps order by id limit ?", l)
   while i < r.length
     s << r[i].to_f.to_s
-    s << r[i+1].to_f.to_s
+    s << r[i + 1].to_f.to_s
     i += 2
   end
   s
@@ -190,7 +192,6 @@ end
 #   s
 # end
 
-
 results = [
   ar_select_times,
   ar_pluck_times,

data/bench/topic_mysql_perf.rb

@@ -0,0 +1,304 @@
+require 'bundler/inline'
+
+gemfile do
+  source 'https://rubygems.org'
+  gem 'mysql2'
+  gem 'mini_sql', path: '../'
+  gem 'activesupport'
+  gem 'activerecord'
+  gem 'activemodel'
+  gem 'memory_profiler'
+  gem 'benchmark-ips'
+  gem 'sequel', github: 'jeremyevans/sequel'
+end
+
+require 'mysql2'
+require 'sequel'
+require 'active_record'
+require 'memory_profiler'
+require 'benchmark/ips'
+require 'mini_sql'
+
+ActiveRecord::Base.establish_connection(
+  :adapter => "mysql2",
+  :database => "test_db",
+  :username => "root",
+  :password => ''
+)
+
+DB = Sequel.connect("mysql2://root:@localhost/test_db")
+
+mysql = ActiveRecord::Base.connection.raw_connection
+
+mysql.query <<SQL
+drop table if exists topics
+SQL
+
+mysql.query <<~SQL
+  CREATE TABLE `topics` (
+    `id` bigint(20) NOT NULL AUTO_INCREMENT,
+    `title` varchar(255) COLLATE utf8mb4_unicode_ci NOT NULL,
+    `last_posted_at` datetime DEFAULT NULL,
+    `created_at` datetime NOT NULL,
+    `updated_at` datetime NOT NULL,
+    `views` int(11) NOT NULL DEFAULT '0',
+    `posts_count` int(11) NOT NULL DEFAULT '0',
+    `user_id` int(11) DEFAULT NULL,
+    `last_post_user_id` int(11) NOT NULL,
+    `reply_count` int(11) NOT NULL DEFAULT '0',
+    `featured_user1_id` int(11) DEFAULT NULL,
+    `featured_user2_id` int(11) DEFAULT NULL,
+    `featured_user3_id` int(11) DEFAULT NULL,
+    `avg_time` int(11) DEFAULT NULL,
+    `deleted_at` datetime DEFAULT NULL,
+    `highest_post_number` int(11) NOT NULL DEFAULT '0',
+    `image_url` varchar(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
+    `like_count` int(11) NOT NULL DEFAULT '0',
+    `incoming_link_count` int(11) NOT NULL DEFAULT '0',
+    `category_id` int(11) DEFAULT NULL,
+    `visible` tinyint(1) NOT NULL DEFAULT '1',
+    `moderator_posts_count` int(11) NOT NULL DEFAULT '0',
+    `closed` tinyint(1) NOT NULL DEFAULT '0',
+    `archived` tinyint(1) NOT NULL DEFAULT '0',
+    `bumped_at` datetime NOT NULL,
+    `has_summary` tinyint(1) NOT NULL DEFAULT '0',
+    `archetype` varchar(255) COLLATE utf8mb4_unicode_ci NOT NULL DEFAULT 'regular',
+    `featured_user4_id` int(11) DEFAULT NULL,
+    `notify_moderators_count` int(11) NOT NULL DEFAULT '0',
+    `spam_count` int(11) NOT NULL DEFAULT '0',
+    `pinned_at` datetime DEFAULT NULL,
+    `score` float DEFAULT NULL,
+    `percent_rank` float NOT NULL DEFAULT '1',
+    `subtype` varchar(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
+    `slug` varchar(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
+    `deleted_by_id` int(11) DEFAULT NULL,
+    `participant_count` int(11) DEFAULT '1',
+    `word_count` int(11) DEFAULT NULL,
+    `excerpt` varchar(1000) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
+    `pinned_globally` tinyint(1) NOT NULL DEFAULT '0',
+    `pinned_until` datetime DEFAULT NULL,
+    `fancy_title` varchar(400) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
+    `highest_staff_post_number` int(11) NOT NULL DEFAULT '0',
+    `featured_link` varchar(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
+    `reviewable_score` float NOT NULL DEFAULT '0',
+    PRIMARY KEY (`id`)
+  )
+SQL
+
+class Topic < ActiveRecord::Base
+end
+
+class TopicSequel < Sequel::Model(:topics)
+end
+
+Topic.transaction do
+  topic = {
+  }
+  Topic.columns.each do |c|
+    topic[c.name.to_sym] = case c.type
+                           when :integer then 1
+                           when :datetime then Time.now
+                           when :boolean then false
+                           when :float then 1.0
+                           else "HELLO WORLD" * 2
+                           end
+  end
+
+  1000.times do |id|
+    topic[:id] = id
+    Topic.create!(topic)
+  end
+end
+
+$conn = ActiveRecord::Base.connection.raw_connection
+
+def ar_title_id_pluck
+  s = +""
+  Topic.limit(1000).order(:id).pluck(:id, :title).each do |id, title|
+    s << id.to_s
+    s << title
+  end
+  s
+end
+
+def ar_title_id
+  s = +""
+  Topic.limit(1000).order(:id).select(:id, :title).each do |t|
+    s << t.id.to_s
+    s << t.title
+  end
+  s
+end
+
+def mysql_title_id
+  s = +""
+  # use the safe pattern here
+  r = $conn.query(-"select id, title from topics order by id limit 1000", as: :array)
+
+  r.each do |row|
+    s << row[0].to_s
+    s << row[1]
+  end
+  s
+end
+
+$mini_sql = MiniSql::Connection.get($conn)
+
+def mini_sql_title_id
+  s = +""
+  $mini_sql.query(-"select id, title from topics order by id limit 1000").each do |t|
+    s << t.id.to_s
+    s << t.title
+  end
+  s
+end
+
+def sequel_select_title_id
+  s = +""
+  TopicSequel.limit(1000).order(:id).select(:id, :title).each do |t|
+    s << t.id.to_s
+    s << t.title
+  end
+  s
+end
+
+def sequel_pluck_title_id
+  s = +""
+  TopicSequel.limit(1000).order(:id).select_map([:id, :title]).each do |t|
+    s << t[0].to_s
+    s << t[1]
+  end
+  s
+end
+
+# usage is not really recommended but just to compare to pluck lets have it
+def mini_sql_title_id_query_single
+  s = +""
+  i = 0
+  r = $mini_sql.query_single(-"select id, title from topics order by id limit 1000")
+  while i < r.length
+    s << r[i].to_s
+    s << r[i + 1]
+    i += 2
+  end
+  s
+end
+
+results = [
+  ar_title_id,
+  ar_title_id_pluck,
+  mysql_title_id,
+  mini_sql_title_id,
+  sequel_pluck_title_id,
+  sequel_select_title_id,
+  mini_sql_title_id_query_single
+]
+
+exit(-1) unless results.uniq.length == 1
+
+Benchmark.ips do |r|
+  r.report("ar select title id") do |n|
+    while n > 0
+      ar_title_id
+      n -= 1
+    end
+  end
+  r.report("ar select title id pluck") do |n|
+    while n > 0
+      ar_title_id_pluck
+      n -= 1
+    end
+  end
+  r.report("sequel title id select") do |n|
+    while n > 0
+      sequel_select_title_id
+      n -= 1
+    end
+  end
+  r.report("mysql select title id") do |n|
+    while n > 0
+      mysql_title_id
+      n -= 1
+    end
+  end
+  r.report("mini_sql select title id") do |n|
+    while n > 0
+      mini_sql_title_id
+      n -= 1
+    end
+  end
+  r.report("sequel title id pluck") do |n|
+    while n > 0
+      sequel_pluck_title_id
+      n -= 1
+    end
+  end
+  r.report("mini_sql query_single title id") do |n|
+    while n > 0
+      mini_sql_title_id_query_single
+      n -= 1
+    end
+  end
+  r.compare!
+end
+
+def wide_topic_ar
+  Topic.first
+end
+
+def wide_topic_mysql
+  r = $conn.query("select * from topics limit 1", as: :hash)
+  row = r.first
+  row
+end
+
+def wide_topic_sequel
+  TopicSequel.first
+end
+
+def wide_topic_mini_sql
+  $conn.query("select * from topics limit 1").first
+end
+
+Benchmark.ips do |r|
+  r.report("wide topic ar") do |n|
+    while n > 0
+      wide_topic_ar
+      n -= 1
+    end
+  end
+  r.report("wide topic sequel") do |n|
+    while n > 0
+      wide_topic_sequel
+      n -= 1
+    end
+  end
+  r.report("wide topic mysql") do |n|
+    while n > 0
+      wide_topic_mysql
+      n -= 1
+    end
+  end
+  r.report("wide topic mini sql") do |n|
+    while n > 0
+      wide_topic_mini_sql
+      n -= 1
+    end
+  end
+  r.compare!
+end
+
+# Comparison:
+# mysql select title id:      485.0 i/s
+# mini_sql query_single title id:      447.2 i/s - same-ish: difference falls within error
+# mini_sql select title id:      417.4 i/s - 1.16x  slower
+# sequel title id pluck:      370.2 i/s - 1.31x  slower
+# sequel title id select:      351.0 i/s - 1.38x  slower
+# ar select title id pluck:      317.1 i/s - 1.53x  slower
+#   ar select title id:      102.3 i/s - 4.74x  slower
+
+# Comparison:
+#  wide topic mini sql:     6768.7 i/s
+#     wide topic mysql:     6063.9 i/s - same-ish: difference falls within error
+#    wide topic sequel:     4908.6 i/s - same-ish: difference falls within error
+#        wide topic ar:     2630.2 i/s - 2.57x  slower