mini_sql 0.2.5 → 0.3

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: c8f5bcd09064cc2353fb1c2f172845cb738c72f9da2ff44786fdf1f119f7c36c
4
- data.tar.gz: 1d7abcda16fa7d972aa0ae6b64bc1f23532feda03daf0098dcbff00f1e2e0d49
3
+ metadata.gz: 3ea4564f53848c314c76c45db0c2af652c1e128ebfc1ba52d6ae11b3799149f4
4
+ data.tar.gz: ecdfdc07fdf11e60338bc2ceb0415cbcc027720c5bba645281217a51befc2402
5
5
  SHA512:
6
- metadata.gz: a84dca1674849011cd77e796046d0ece93e7952603470a51b959332180a1533fb1d15f1d3e3b69cb6f5bcc2f43e4f80c6902d85c0709443bc9004a15bc721921
7
- data.tar.gz: 4433947c712bcf6c9346fa2b0df792bc95c7fdaa453e5ae19b649241c24ba7ab3b57397cb66114b68ba4e622dfcc6242e6631beed129f5a8cd08bc618e13381d
6
+ metadata.gz: 76a040187f4932dc9fbc80ae3323633a38462753caec47233dbfad3e276e90d8eeb0fbddcbfe974ade3b0b99c18a487d5ef865ebdb1effc67048352ecd35d99d
7
+ data.tar.gz: 73c3519bf51a7934af89827830e3e7c45366eec01b1d65dc0c61afb2b87c12b093e9f3db99b1c4db081e6a6a1e983fd24d88db3aa13f63b9e546af0dce36beae
data/CHANGELOG CHANGED
@@ -1,3 +1,7 @@
1
+ 2020-06-25 - 0.3
2
+
3
+ - Added support for query_each and query_each_hash, which lazily queries rows and enables selecting large result sets by streaming
4
+
1
5
  2020-04-07 - 0.2.5
2
6
 
3
7
  - Added support for custom type maps with Postgres connections
data/README.md CHANGED
@@ -130,10 +130,12 @@ When using Postgres, native type mapping implementation is used. This is roughly
130
130
  implemented as:
131
131
 
132
132
  ```ruby
133
- type_map = PG::BasicTypeMapForResults.new(conn)
133
+ type_map ||= PG::BasicTypeMapForResults.new(conn)
134
134
  # additional specific decoders
135
135
  ```
136
136
 
137
+ The type mapper instansitated once on-demand at boot and reused by all mini_sql connections.
138
+
137
139
  Initializing the basic type map for Postgres can be a costly operation. You may
138
140
  wish to amend the type mapper so for example you only return strings:
139
141
 
@@ -158,6 +160,33 @@ mini_sql_cnn = MiniSql::Connection.get(pg_cnn, type_map: pg_cnn.type_map_for_res
158
160
 
159
161
  Note the type mapper for Rails may miss some of the mapping MiniSql ships with such as `IPAddr`, MiniSql is also careful to use the very efficient TimestampUtc decoders where available.
160
162
 
163
+ ## Streaming support
164
+
165
+ In some exceptional cases you may want to stream results directly from the database. This enables selection of 100s of thousands of rows with limited memory impact.
166
+
167
+ Two interfaces exists for this:
168
+
169
+ `query_each` : which can be used to get materialized objects
170
+ `query_each_hash` : which can be used to iterate through Hash objects
171
+
172
+ Usage:
173
+
174
+ ```ruby
175
+ mini_sql_cnn.query_each("SELECT * FROM tons_of_cows limit :limit", limit: 1_000_000) do |row|
176
+ puts row.cow_name
177
+ puts row.cow_age
178
+ end
179
+
180
+ mini_sql_cnn.query_each_hash("SELECT * FROM one_million_cows") do |row|
181
+ puts row["cow_name"]
182
+ puts row["cow_age"]
183
+ end
184
+ ```
185
+
186
+ Note, in Postgres streaming is going to be slower than non-streaming options due to internal implementation in the pq gem, each row gets a full result object and additional bookkeeping is needed. Only use it if you need to optimize memory usage.
187
+
188
+ Streaming support is only implemented in the postgres backend at the moment, PRs welcome to add to other backends.
189
+
161
190
  ## I want more features!
162
191
 
163
192
  MiniSql is designed to be very minimal. Even though the query builder and type materializer give you a lot of mileage, it is not intended to be a fully fledged ORM. If you are looking for an ORM I recommend investigating ActiveRecord or Sequel which provide significantly more features.
@@ -31,11 +31,23 @@ module MiniSql
31
31
  raise NotImplementedError, "must be implemented by child connection"
32
32
  end
33
33
 
34
- def exec(sql, *params)
34
+ def query_hash(sql, *params)
35
35
  raise NotImplementedError, "must be implemented by child connection"
36
36
  end
37
37
 
38
- def query_hash(sql, *params)
38
+ def query_decorator(sql, *params)
39
+ raise NotImplementedError, "must be implemented by child connection"
40
+ end
41
+
42
+ def query_each(sql, *params)
43
+ raise NotImplementedError, "must be implemented by child connection"
44
+ end
45
+
46
+ def query_each_hash(sql, *params)
47
+ raise NotImplementedError, "must be implemented by child connection"
48
+ end
49
+
50
+ def exec(sql, *params)
39
51
  raise NotImplementedError, "must be implemented by child connection"
40
52
  end
41
53
 
@@ -83,7 +83,7 @@ module MiniSql
83
83
  result = run(sql, params)
84
84
  result.type_map = type_map
85
85
  result.values
86
- ensure
86
+ ensure
87
87
  result.clear if result
88
88
  end
89
89
 
@@ -95,6 +95,72 @@ module MiniSql
95
95
  result.clear if result
96
96
  end
97
97
 
98
+ def query_each(sql, *params)
99
+ raise StandardError, "Please supply a block when calling query_each" if !block_given?
100
+ if params && params.length > 0
101
+ sql = param_encoder.encode(sql, *params)
102
+ end
103
+
104
+ raw_connection.send_query(sql)
105
+ raw_connection.set_single_row_mode
106
+
107
+ loop do
108
+ result = raw_connection.get_result
109
+ break if !result
110
+
111
+ result.check
112
+
113
+ if result.ntuples == 0
114
+ # skip, this happens at the end when we get totals
115
+ else
116
+ materializer ||= @deserializer_cache.materializer(result)
117
+ result.type_map = type_map
118
+ i = 0
119
+ # technically we should only get 1 row here
120
+ # but protect against future batching changes
121
+ while i < result.ntuples
122
+ yield materializer.materialize(result, i)
123
+ i += 1
124
+ end
125
+ end
126
+
127
+ result.clear
128
+ end
129
+ end
130
+
131
+ def query_each_hash(sql, *params)
132
+ raise StandardError, "Please supply a block when calling query_each_hash" if !block_given?
133
+ if params && params.length > 0
134
+ sql = param_encoder.encode(sql, *params)
135
+ end
136
+
137
+ raw_connection.send_query(sql)
138
+ raw_connection.set_single_row_mode
139
+
140
+ loop do
141
+ result = raw_connection.get_result
142
+ break if !result
143
+
144
+ result.check
145
+
146
+ if result.ntuples == 0
147
+ # skip, this happens at the end when we get totals
148
+ else
149
+ result.type_map = type_map
150
+ i = 0
151
+
152
+ # technically we should only get 1 row here
153
+ # but protect against future batching changes
154
+ while i < result.ntuples
155
+ yield result[i]
156
+ i += 1
157
+ end
158
+ end
159
+
160
+ result.clear
161
+ end
162
+ end
163
+
98
164
  def query_decorator(decorator, sql, *params)
99
165
  result = run(sql, params)
100
166
  result.type_map = type_map
@@ -11,6 +11,20 @@ module MiniSql
11
11
  @max_size = max_size || DEFAULT_MAX_SIZE
12
12
  end
13
13
 
14
+ def materializer(result)
15
+ key = result.fields
16
+
17
+ materializer = @cache.delete(key)
18
+ if materializer
19
+ @cache[key] = materializer
20
+ else
21
+ materializer = @cache[key] = new_row_matrializer(result)
22
+ @cache.shift if @cache.length > @max_size
23
+ end
24
+
25
+ materializer
26
+ end
27
+
14
28
  def materialize(result, decorator_module = nil)
15
29
  return [] if result.ntuples == 0
16
30
 
@@ -42,6 +56,15 @@ module MiniSql
42
56
  def new_row_matrializer(result)
43
57
  fields = result.fields
44
58
 
59
+ i = 0
60
+ while i < fields.length
61
+ # special handling for unamed column
62
+ if fields[i] == "?column?"
63
+ fields[i] = "column#{i}"
64
+ end
65
+ i += 1
66
+ end
67
+
45
68
  Class.new do
46
69
  attr_accessor(*fields)
47
70
 
@@ -1,4 +1,4 @@
1
1
  # frozen_string_literal: true
2
2
  module MiniSql
3
- VERSION = "0.2.5"
3
+ VERSION = "0.3"
4
4
  end
@@ -33,7 +33,7 @@ Gem::Specification.new do |spec|
33
33
  spec.require_paths = ["lib"]
34
34
 
35
35
  spec.add_development_dependency "bundler", "> 1.16"
36
- spec.add_development_dependency "rake", "~> 10.0"
36
+ spec.add_development_dependency "rake", "> 10"
37
37
  spec.add_development_dependency "minitest", "~> 5.0"
38
38
  spec.add_development_dependency "guard", "~> 2.14"
39
39
  spec.add_development_dependency "guard-minitest", "~> 2.4"
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: mini_sql
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.2.5
4
+ version: '0.3'
5
5
  platform: ruby
6
6
  authors:
7
7
  - Sam Saffron
8
- autorequire:
8
+ autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2020-04-07 00:00:00.000000000 Z
11
+ date: 2020-06-25 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: bundler
@@ -28,16 +28,16 @@ dependencies:
28
28
  name: rake
29
29
  requirement: !ruby/object:Gem::Requirement
30
30
  requirements:
31
- - - "~>"
31
+ - - ">"
32
32
  - !ruby/object:Gem::Version
33
- version: '10.0'
33
+ version: '10'
34
34
  type: :development
35
35
  prerelease: false
36
36
  version_requirements: !ruby/object:Gem::Requirement
37
37
  requirements:
38
- - - "~>"
38
+ - - ">"
39
39
  - !ruby/object:Gem::Version
40
- version: '10.0'
40
+ version: '10'
41
41
  - !ruby/object:Gem::Dependency
42
42
  name: minitest
43
43
  requirement: !ruby/object:Gem::Requirement
@@ -210,7 +210,7 @@ metadata:
210
210
  bug_tracker_uri: https://github.com/discourse/mini_sql/issues
211
211
  source_code_uri: https://github.com/discourse/mini_sql
212
212
  changelog_uri: https://github.com/discourse/mini_sql/blob/master/CHANGELOG
213
- post_install_message:
213
+ post_install_message:
214
214
  rdoc_options: []
215
215
  require_paths:
216
216
  - lib
@@ -226,7 +226,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
226
226
  version: '0'
227
227
  requirements: []
228
228
  rubygems_version: 3.0.3
229
- signing_key:
229
+ signing_key:
230
230
  specification_version: 4
231
231
  summary: A fast, safe, simple direct SQL executor
232
232
  test_files: []