mini_sql 1.0 → 1.1.3

Sign up to get free protection for your applications and to get access to all the features.
Files changed (37) hide show
  1. checksums.yaml +4 -4
  2. data/.github/workflows/ci.yml +1 -1
  3. data/CHANGELOG +20 -0
  4. data/README.md +36 -0
  5. data/bench/builder_perf.rb +138 -0
  6. data/bench/decorator_perf.rb +143 -0
  7. data/bench/mini_sql_methods_perf.rb +80 -0
  8. data/bench/prepared_perf.rb +59 -0
  9. data/bench/shared/generate_data.rb +133 -0
  10. data/bench/topic_perf.rb +21 -327
  11. data/bench/topic_wide_perf.rb +92 -0
  12. data/lib/mini_sql.rb +17 -8
  13. data/lib/mini_sql/abstract/prepared_binds.rb +74 -0
  14. data/lib/mini_sql/abstract/prepared_cache.rb +45 -0
  15. data/lib/mini_sql/builder.rb +63 -30
  16. data/lib/mini_sql/inline_param_encoder.rb +4 -5
  17. data/lib/mini_sql/mysql/connection.rb +13 -3
  18. data/lib/mini_sql/mysql/deserializer_cache.rb +3 -3
  19. data/lib/mini_sql/mysql/prepared_binds.rb +15 -0
  20. data/lib/mini_sql/mysql/prepared_cache.rb +21 -0
  21. data/lib/mini_sql/mysql/prepared_connection.rb +47 -0
  22. data/lib/mini_sql/postgres/connection.rb +21 -7
  23. data/lib/mini_sql/postgres/deserializer_cache.rb +5 -5
  24. data/lib/mini_sql/postgres/prepared_binds.rb +15 -0
  25. data/lib/mini_sql/postgres/prepared_cache.rb +25 -0
  26. data/lib/mini_sql/postgres/prepared_connection.rb +39 -0
  27. data/lib/mini_sql/postgres_jdbc/connection.rb +3 -1
  28. data/lib/mini_sql/postgres_jdbc/deserializer_cache.rb +3 -3
  29. data/lib/mini_sql/result.rb +10 -0
  30. data/lib/mini_sql/serializer.rb +29 -15
  31. data/lib/mini_sql/sqlite/connection.rb +14 -2
  32. data/lib/mini_sql/sqlite/deserializer_cache.rb +3 -3
  33. data/lib/mini_sql/sqlite/prepared_binds.rb +15 -0
  34. data/lib/mini_sql/sqlite/prepared_cache.rb +21 -0
  35. data/lib/mini_sql/sqlite/prepared_connection.rb +43 -0
  36. data/lib/mini_sql/version.rb +1 -1
  37. metadata +20 -3
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 3d668e0b456e27a572040825399ca9477d6c4404bfc09e6e87b1330a106491a0
4
- data.tar.gz: 3fb2027b34d3837e06f6c92b7075f8811296c0750c5c4c3ae48d6053189a4000
3
+ metadata.gz: 8d46350915a655a1692de9b775f879609f83ff068bf9b62be6498672b223ee03
4
+ data.tar.gz: 192596f2834a676f123afcfb969a32267b311fe20318527ae13b92dd95efc757
5
5
  SHA512:
6
- metadata.gz: 60449b0a92a13aa471dbc1dd6db2b8ba0571bca67f81bd46644bf8678ab794b5447e59cd3ea7a24d4732bcdb8eca44c9dfec748c57418d826444305395f62190
7
- data.tar.gz: a249f769da221d659d1b4dceaf3dd153578fbda3f0951bdac938d178923a54ad852106629d76be38b5c4ad83c4175592b9971223f8e0aab49e91a48747ad7148
6
+ metadata.gz: 9820695fc27ced3ce61b2a131201c6bb954eca9919cd49f29f2877ebccbbd33ef8e36b4a0284d0a04fde9ec92c1f33bbfe75780515cdec26bdf9493e676b631d
7
+ data.tar.gz: d2ba6adbdf07a1a3fb11192ad949c8f632035a357d792eb895350ee3af30f7977d6b8e64d5df59b1338e9349d213e4472099593c6296e39203fac7ee8147da7d
@@ -20,7 +20,7 @@ jobs:
20
20
  name: Ruby ${{ matrix.ruby }}
21
21
  services:
22
22
  postgres:
23
- image: postgres:9.6
23
+ image: postgres:10
24
24
  env:
25
25
  POSTGRES_PASSWORD: postgres
26
26
  ports:
data/CHANGELOG CHANGED
@@ -1,3 +1,23 @@
1
+ 2021-03-22 - 1.1.3
2
+
3
+ - DEV: reduce coupling of internal interfaces and allow or cleaner override of prepared connections
4
+
5
+ 2021-03-22 - 1.1.2
6
+
7
+ - FEATURE: improve compatability with clients overriding raw_connection
8
+
9
+ 2021-03-22 - 1.1.1
10
+
11
+ - FIX: compatability with ActiveRecord param encoder
12
+
13
+ 2021-03-22 - 1.1.0
14
+
15
+ - FEATURE: added new APIs to support prepared statements
16
+
17
+ 2020-12-31 - 1.0.1
18
+
19
+ - FIX: revert perf fix broke param_encoder interface, we were expecting never to be called if no encoding was to happen
20
+
1
21
  2020-12-30 - 1.0
2
22
 
3
23
  - Added serialization support using MiniSql::Serializer.to_json / .from_json
data/README.md CHANGED
@@ -187,6 +187,21 @@ Note, in Postgres streaming is going to be slower than non-streaming options due
187
187
 
188
188
  Streaming support is only implemented in the postgres backend at the moment, PRs welcome to add to other backends.
189
189
 
190
+ ## Prepared Statements
191
+ See [benchmark mini_sql](https://github.com/discourse/mini_sql/tree/master/bench/prepared_perf.rb)
192
+ [benchmark mini_sql vs rails](https://github.com/discourse/mini_sql/tree/master/bench/bilder_perf.rb).
193
+
194
+ By default prepared cache size is 500 queries. Use prepared queries only for frequent queries.
195
+
196
+ ```ruby
197
+ conn.prepared.query("select * from table where id = ?", id: 10)
198
+
199
+ ids = rand(100) < 90 ? [1] : [1, 2]
200
+ builder = conn.build("select * from table /*where*/")
201
+ builder.where("id IN (?)", ids)
202
+ builder.prepared(ids.size == 1).query # most frequent query
203
+ ```
204
+
190
205
  ## I want more features!
191
206
 
192
207
  MiniSql is designed to be very minimal. Even though the query builder and type materializer give you a lot of mileage, it is not intended to be a fully fledged ORM. If you are looking for an ORM I recommend investigating ActiveRecord or Sequel which provide significantly more features.
@@ -194,6 +209,27 @@ MiniSql is designed to be very minimal. Even though the query builder and type m
194
209
  ## Development
195
210
 
196
211
  To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
212
+ ### Local testing
213
+ ```bash
214
+ docker run --name mini-sql-mysql --rm -it -p 33306:3306 -e MYSQL_DATABASE=test_mini_sql -e MYSQL_ALLOW_EMPTY_PASSWORD=yes -d mysql:5.7
215
+ export MINI_SQL_MYSQL_HOST=127.0.0.1
216
+ export MINI_SQL_MYSQL_PORT=33306
217
+
218
+ docker run --name mini-sql-postgres --rm -it -p 55432:5432 -e POSTGRES_DB=test_mini_sql -e POSTGRES_HOST_AUTH_METHOD=trust -d postgres
219
+ export MINI_SQL_PG_USER=postgres
220
+ export MINI_SQL_PG_HOST=127.0.0.1
221
+ export MINI_SQL_PG_PORT=55432
222
+
223
+ sleep 10 # waiting for up databases
224
+
225
+ bundle exec rake
226
+
227
+ # end working on mini-sql
228
+ docker stop mini-sql-postgres mini-sql-mysql
229
+ ```
230
+
231
+ Sqlite tests rely on the SQLITE_STMT view existing. This is enabled by default on most systems, however some may
232
+ opt for a leaner install. See: https://bugs.archlinux.org/task/70072. You may have to recompile sqlite on such systems.
197
233
 
198
234
  ## Contributing
199
235
 
@@ -0,0 +1,138 @@
1
+ # frozen_string_literal: true
2
+
3
+ require 'bundler/inline'
4
+
5
+ gemfile do
6
+ source 'https://rubygems.org'
7
+ gem 'mini_sql', path: '../'
8
+ gem 'pg'
9
+ gem 'activerecord'
10
+ gem 'activemodel'
11
+ gem 'benchmark-ips'
12
+ end
13
+
14
+ require 'active_record'
15
+ require 'benchmark/ips'
16
+ require 'mini_sql'
17
+
18
+ require '../mini_sql/bench/shared/generate_data'
19
+
20
+ ar_connection, _ = GenerateData.new(count_records: 10_000).call
21
+ MINI_SQL = MiniSql::Connection.get(ar_connection.raw_connection)
22
+
23
+ def mini_sql(is_prepared, user_id)
24
+ MINI_SQL
25
+ .build(<<~SQL)
26
+ /*select*/ from topics /*join*/ /*where*/ /*group_by*/
27
+ SQL
28
+ .select('users.first_name, count(distinct topics.id) topics_count')
29
+ .join('users on user_id = users.id')
30
+ .join('categories on category_id = categories.id')
31
+ .where('users.id = ?', user_id)
32
+ .group_by('users.id')
33
+ .prepared(is_prepared)
34
+ .query
35
+ end
36
+
37
+ def mini_sql_optim(is_prepared, user_id)
38
+ @builder ||=
39
+ MINI_SQL
40
+ .build(<<~SQL)
41
+ /*select*/ from topics /*join*/ /*where*/ /*group_by*/
42
+ SQL
43
+ .select('users.first_name, count(distinct topics.id) topics_count')
44
+ .join('users on user_id = users.id')
45
+ .join('categories on category_id = categories.id')
46
+ .where('users.id = :user_id')
47
+ .group_by('users.id')
48
+
49
+ @builder
50
+ .prepared(is_prepared)
51
+ .query(user_id: user_id)
52
+ end
53
+
54
+ def ar_prepared(user_id)
55
+ Topic
56
+ .select(User.arel_table[:first_name] , Topic.arel_table[:id].count)
57
+ .joins(:user, :category)
58
+ .where(user_id: user_id)
59
+ .group(User.arel_table[:id])
60
+ .load
61
+ end
62
+
63
+ def ar_prepared_optim(user_id)
64
+ @rel ||= Topic
65
+ .select(User.arel_table[:first_name] , Topic.arel_table[:id].count)
66
+ .joins(:user, :category)
67
+ .group(User.arel_table[:id])
68
+
69
+ @rel
70
+ .where(user_id: user_id)
71
+ .load
72
+ end
73
+
74
+ def ar_unprepared(user_id)
75
+ Topic
76
+ .select('users.first_name, count(distinct topics.id) topics_count')
77
+ .joins(:user, :category)
78
+ .where(user_id: user_id)
79
+ .group('users.id')
80
+ .load
81
+ end
82
+
83
+ Benchmark.ips do |x|
84
+ x.report("mini_sql_prepared") do |n|
85
+ while n > 0
86
+ mini_sql(true, rand(100))
87
+ n -= 1
88
+ end
89
+ end
90
+ x.report("mini_sql") do |n|
91
+ while n > 0
92
+ mini_sql(false, rand(100))
93
+ n -= 1
94
+ end
95
+ end
96
+
97
+ x.report("mini_sql_prepared_optim") do |n|
98
+ while n > 0
99
+ mini_sql_optim(true, rand(100))
100
+ n -= 1
101
+ end
102
+ end
103
+ x.report("mini_sql_optim") do |n|
104
+ while n > 0
105
+ mini_sql_optim(false, rand(100))
106
+ n -= 1
107
+ end
108
+ end
109
+
110
+ x.report("ar_prepared") do |n|
111
+ while n > 0
112
+ ar_prepared(rand(100))
113
+ n -= 1
114
+ end
115
+ end
116
+
117
+ x.report("ar_prepared_optim") do |n|
118
+ while n > 0
119
+ ar_prepared_optim(rand(100))
120
+ n -= 1
121
+ end
122
+ end
123
+
124
+ x.report("ar_unprepared") do |n|
125
+ while n > 0
126
+ ar_unprepared(rand(100))
127
+ n -= 1
128
+ end
129
+ end
130
+
131
+ x.compare!
132
+ end
133
+
134
+ # Comparison:
135
+ # mini_sql_prepared: 8386.2 i/s
136
+ # mini_sql: 2742.3 i/s - 3.06x (± 0.00) slower
137
+ # ar_prepared: 1599.3 i/s - 5.24x (± 0.00) slower
138
+ # ar_unprepared: 868.9 i/s - 9.65x (± 0.00) slower
@@ -0,0 +1,143 @@
1
+ # frozen_string_literal: true
2
+
3
+ require 'bundler/inline'
4
+
5
+ gemfile do
6
+ source 'https://rubygems.org'
7
+ gem 'pg', github: 'ged/ruby-pg'
8
+ gem 'mini_sql', path: '../'
9
+ gem 'activerecord'
10
+ gem 'activemodel'
11
+ gem 'benchmark-ips'
12
+ gem 'sequel', github: 'jeremyevans/sequel'
13
+ gem 'sequel_pg', github: 'jeremyevans/sequel_pg', require: 'sequel'
14
+ gem 'draper'
15
+ gem 'pry'
16
+ end
17
+
18
+ require 'active_record'
19
+ require 'benchmark/ips'
20
+ require 'mini_sql'
21
+
22
+ require '../mini_sql/bench/shared/generate_data'
23
+
24
+ ar_connection, conn_config = GenerateData.new(count_records: 1_000).call
25
+ MINI_SQL = MiniSql::Connection.get(ar_connection.raw_connection)
26
+ DB = Sequel.connect(ar_connection.instance_variable_get(:@config).slice(:database, :user, :password, :host, :adapter))
27
+
28
+
29
+ # https://github.com/drapergem/draper
30
+ class TopicDraper < Draper::Decorator
31
+ delegate :id
32
+
33
+ def title_bang
34
+ object.title + '!!!'
35
+ end
36
+ end
37
+
38
+ # https://ruby-doc.org/stdlib-2.5.1/libdoc/delegate/rdoc/SimpleDelegator.html
39
+ class TopicSimpleDelegator < SimpleDelegator
40
+ def title_bang
41
+ title + '!!!'
42
+ end
43
+ end
44
+
45
+ class TopicSequel < Sequel::Model(DB[:topics]); end
46
+ class TopicDecoratorSequel < TopicSequel
47
+ def title_bang
48
+ title + '!!!'
49
+ end
50
+ end
51
+
52
+ class Topic < ActiveRecord::Base;end
53
+ class TopicArModel < Topic
54
+ def title_bang
55
+ title + '!!!'
56
+ end
57
+ end
58
+
59
+ module TopicDecorator
60
+ def title_bang
61
+ title + '!!!'
62
+ end
63
+ end
64
+
65
+ Benchmark.ips do |r|
66
+ r.report('query_decorator') do |n|
67
+ while n > 0
68
+ MINI_SQL.query_decorator(TopicDecorator, 'select id, title from topics order by id limit 1000').each do |obj|
69
+ obj.title_bang
70
+ obj.id
71
+ end
72
+ n -= 1
73
+ end
74
+ end
75
+ r.report('extend') do |n|
76
+ while n > 0
77
+ MINI_SQL.query('select id, title from topics order by id limit 1000').each do |obj|
78
+ d_obj = obj.extend(TopicDecorator)
79
+ d_obj.title_bang
80
+ d_obj.id
81
+ end
82
+ n -= 1
83
+ end
84
+ end
85
+ r.report('draper') do |n|
86
+ while n > 0
87
+ MINI_SQL.query('select id, title from topics order by id limit 1000').each do |obj|
88
+ d_obj = TopicDraper.new(obj)
89
+ d_obj.title_bang
90
+ d_obj.id
91
+ end
92
+ n -= 1
93
+ end
94
+ end
95
+ r.report('simple_delegator') do |n|
96
+ while n > 0
97
+ MINI_SQL.query('select id, title from topics order by id limit 1000').each do |obj|
98
+ d_obj = TopicSimpleDelegator.new(obj)
99
+ d_obj.title_bang
100
+ d_obj.id
101
+ end
102
+ n -= 1
103
+ end
104
+ end
105
+ r.report('query') do |n|
106
+ while n > 0
107
+ MINI_SQL.query('select id, title from topics order by id limit 1000').each do |obj|
108
+ obj.title + '!!!'
109
+ obj.id
110
+ end
111
+ n -= 1
112
+ end
113
+ end
114
+ r.report('ar model') do |n|
115
+ while n > 0
116
+ TopicArModel.limit(1000).order(:id).select(:id, :title).each do |obj|
117
+ obj.title_bang
118
+ obj.id
119
+ end
120
+ n -= 1
121
+ end
122
+ end
123
+ r.report('sequel model') do |n|
124
+ while n > 0
125
+ TopicDecoratorSequel.limit(1000).order(:id).select(:id, :title).each do |obj|
126
+ obj.title_bang
127
+ obj.id
128
+ end
129
+ n -= 1
130
+ end
131
+ end
132
+
133
+ r.compare!
134
+ end
135
+
136
+ # Comparison:
137
+ # query: 1102.9 i/s
138
+ # query_decorator: 1089.0 i/s - same-ish: difference falls within error
139
+ # sequel model: 860.2 i/s - 1.28x (± 0.00) slower
140
+ # simple_delegator: 679.8 i/s - 1.62x (± 0.00) slower
141
+ # extend: 678.1 i/s - 1.63x (± 0.00) slower
142
+ # draper: 587.2 i/s - 1.88x (± 0.00) slower
143
+ # ar model: 172.5 i/s - 6.39x (± 0.00) slower
@@ -0,0 +1,80 @@
1
+ # frozen_string_literal: true
2
+
3
+ require 'bundler/inline'
4
+
5
+ gemfile do
6
+ source 'https://rubygems.org'
7
+ gem 'pg', github: 'ged/ruby-pg'
8
+ gem 'mini_sql', path: '../'
9
+ gem 'activerecord'
10
+ gem 'activemodel'
11
+ gem 'benchmark-ips'
12
+ gem 'draper'
13
+ gem 'pry'
14
+ end
15
+
16
+ require 'active_record'
17
+ require 'benchmark/ips'
18
+ require 'mini_sql'
19
+
20
+ require '../mini_sql/bench/shared/generate_data'
21
+
22
+ ar_connection, conn_config = GenerateData.new(count_records: 1_000).call
23
+ MINI_SQL = MiniSql::Connection.get(ar_connection.raw_connection)
24
+
25
+
26
+ Benchmark.ips do |r|
27
+ r.report('query_hash') do |n|
28
+ while n > 0
29
+ MINI_SQL.query_hash('select id, title from topics order by id limit 1000').each do |hash|
30
+ [hash['id'], hash['title']]
31
+ end
32
+ n -= 1
33
+ end
34
+ end
35
+ r.report('query_array') do |n|
36
+ while n > 0
37
+ MINI_SQL.query_array('select id, title from topics order by id limit 1000').each do |id, title|
38
+ [id, title]
39
+ end
40
+ n -= 1
41
+ end
42
+ end
43
+ r.report('query') do |n|
44
+ while n > 0
45
+ MINI_SQL.query('select id, title from topics order by id limit 1000').each do |obj|
46
+ [obj.id, obj.title]
47
+ end
48
+ n -= 1
49
+ end
50
+ end
51
+
52
+ r.compare!
53
+ end
54
+
55
+ # Comparison:
56
+ # query_array: 1663.3 i/s
57
+ # query: 1254.5 i/s - 1.33x (± 0.00) slower
58
+ # query_hash: 1095.4 i/s - 1.52x (± 0.00) slower
59
+
60
+
61
+ Benchmark.ips do |r|
62
+ r.report('query_single') do |n|
63
+ while n > 0
64
+ MINI_SQL.query_single('select id from topics order by id limit 1000')
65
+ n -= 1
66
+ end
67
+ end
68
+ r.report('query_array') do |n|
69
+ while n > 0
70
+ MINI_SQL.query_array('select id from topics order by id limit 1000').flatten
71
+ n -= 1
72
+ end
73
+ end
74
+
75
+ r.compare!
76
+ end
77
+
78
+ # Comparison:
79
+ # query_single: 2445.1 i/s
80
+ # query_array: 1681.1 i/s - 1.45x (± 0.00) slower