sequel 5.41.0 → 5.42.0

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: b0886ae2f4e4f4490b56ced6137e2d849cac2458119d6e67d9a756328336ece6
4
- data.tar.gz: 9c4946c8ca82b2b1c99ee9065bef0aa81175128564cd7ab0c0405bd4e682b339
3
+ metadata.gz: 41e987d42df8d1b3ab41dd6555b5e338b94ba0e2fcd11fa3db7f9b955920d533
4
+ data.tar.gz: fdcaed20caa20bbaffe5854cbcd6836787e92a2bbadb89373337602926d476ca
5
5
  SHA512:
6
- metadata.gz: aa61ea3a0cb3d0e3ab021d27589e10250ed9d9d5a0b4bf314729d55661624161cabffe9c87278761890a900949756e185ce1f263bf742d9b317f2621a1f75c09
7
- data.tar.gz: a8253ea1f6e77b592eb40de7b2544b7b34ff8cd02a79e4dfe5bb26b796dd80d49f052e307590af71fb67b292bc9b9c1def8f400eea0977fdb0d85928a6e9f181
6
+ metadata.gz: c9a3927fe546ee7f91497e5bd50e9fa9241dd46aa7d3b3c331d10f8ac369eedd204a488207c509e909d9eb5d7deeeb13ab20888568f7e91f53fba526323df566
7
+ data.tar.gz: a9108c8ce0d78d93c94f0a947171687022dd20ee1305633533340f1cb1a71d012248f9a9bb72eb7af28cc258240ec2e8f090683d1ad7a68c33f69d261c684ea0
data/CHANGELOG CHANGED
@@ -1,3 +1,19 @@
1
+ === 5.42.0 (2021-03-01)
2
+
3
+ * Make the ado timestamp conversion proc a normal conversion proc that can be overridden similar to other conversion procs (jeremyevans)
4
+
5
+ * Add :reject_nil option to the nested_attributes method, to ignore calls where nil is passed as the associated object data (jeremyevans)
6
+
7
+ * Add async_thread_pool plugin for easier async usage with model classes and support for async destroy, with_pk, and with_pk! methods (jeremyevans)
8
+
9
+ * Add async_thread_pool Database extension for executing queries asynchronously using a thread pool (jeremyevans)
10
+
11
+ * Fix possible thread safety issue in Database#extension that could allow Module#extended to be called twice with the same Database instance (jeremyevans)
12
+
13
+ * Support cases where validations make modifications beyond setting errors in Model#freeze (jeremyevans)
14
+
15
+ * Add Model#to_json_data to the json_serializer plugin, returning a JSON data structure (jeremyevans)
16
+
1
17
  === 5.41.0 (2021-02-01)
2
18
 
3
19
  * Have explicit :text option for a String column take priority over :size option on PostgreSQL (jeremyevans) (#1750)
@@ -0,0 +1,136 @@
1
+ = New Features
2
+
3
+ * An async_thread_pool Database extension has been added, which
4
+ executes queries and processes results using a separate thread
5
+ pool. This allows you do do things like:
6
+
7
+ foos = DB[:foos].async.all
8
+ bars = DB[:bars].async.select_map(:name)
9
+ foo_bars = DB[:foo_bars].async.each{|x| p x}
10
+
11
+ and have the three method calls (all, select_map, and each)
12
+ execute concurrently. On Ruby implementations without a global
13
+ VM lock, such as JRuby, it will allow for parallel execution of
14
+ the method calls. On CRuby, the main benefit will be for cases
15
+ where query execution takes a long time or there is significant
16
+ latency between the application and the database.
17
+
18
+ When you call a method on foos, bars, or foo_bars, if the thread
19
+ pool hasn't finished processing the method, the calling code will
20
+ block until the method call has finished.
21
+
22
+ By default, for consistency, calling code will not preempt the
23
+ async thread pool. For example, if you do:
24
+
25
+ DB[:foos].async.all.size
26
+
27
+ The calling code will always wait for the async thread pool to
28
+ run the all method, and then the calling code will call size on
29
+ the result. This ensures that async queries will not use the
30
+ same connection as the the calling thread, even if calling thread
31
+ has a connection checked out.
32
+
33
+ In some cases, such as when the async thread pool is very busy,
34
+ preemption is desired for performance reasons. If you set the
35
+ :preempt_async_thread Database option before loading the
36
+ async_thread_pool extension, preemption will be allowed. With
37
+ preemption allowed, if the async thread pool has not started the
38
+ processing of the method at the time the calling code needs the
39
+ results of the method, the calling code will preempt the async
40
+ thread pool, and run the method on the current thread.
41
+
42
+ By default, the async thread pool uses the same number of threads as
43
+ the Database objects :max_connections attribute (the default for
44
+ that is 4). You can modify the number of async threads by setting
45
+ the :num_async_threads Database option before loading the Database
46
+ async_thread_pool extension.
47
+
48
+ Most Dataset methods that execute queries on the database and return
49
+ results will operate asynchronously if the the dataset is set to be
50
+ asynchronous via the Dataset#async method. This includes most
51
+ methods available due to the inclusion in Enumerable, even if not
52
+ defined by Dataset itself.
53
+
54
+ There are multiple caveats when using the async_thread_pool
55
+ extension:
56
+
57
+ * Asynchronous behavior is harder to understand and harder to
58
+ debug. It would be wise to only use this support in cases where
59
+ it provides is significant performance benefit.
60
+
61
+ * Dataset methods executed asynchronously will use a separate
62
+ database connection than the calling thread, so they will not
63
+ respect transactions in the calling thread, or other cases where
64
+ the calling thread checks out a connection directly using
65
+ Database#synchronize. They will also not respect the use of
66
+ Database#with_server (from the server_block extension) in the
67
+ calling thread.
68
+
69
+ * Dataset methods executed asynchronously should never ignore their
70
+ return value. Code such as:
71
+
72
+ DB[:table].async.insert(1)
73
+
74
+ is probablematic because without storing the return value, you
75
+ have no way to block until the insert has been completed.
76
+
77
+ * The returned object for Dataset methods executed asynchronously is
78
+ a proxy object (promise). So you should never do:
79
+
80
+ row = DB[:table].async.first
81
+ # ...
82
+ if row
83
+ end
84
+
85
+ # or:
86
+
87
+ bool = DB[:table].async.get(:boolean_column)
88
+ # ...
89
+ if bool
90
+ end
91
+
92
+ because the if branches will always be taken as row and bool will
93
+ never be nil or false. If you want to get the underlying value,
94
+ call itself on the proxy object (or __value if using Ruby <2.2).
95
+
96
+ For the same reason, you should not use the proxy objects directly
97
+ in case expressions or as arguments to Class#===. Use itself or
98
+ __value in those cases.
99
+
100
+ * Dataset methods executed asynchronously that include blocks have the
101
+ block executed asynchronously as well, assuming that the method
102
+ calls the block. Because these blocks are executed in a separate
103
+ thread, you cannot use control flow modifiers such as break or
104
+ return in them.
105
+
106
+ * An async_thread_pool model plugin has been added. This requires the
107
+ async_thread_pool extension has been loaded into the model's Database
108
+ object, and allows you to call Model.async instead of
109
+ Model.dataset.async. It also adds async support to the destroy,
110
+ with_pk, and with_pk! model dataset methods.
111
+
112
+ * Model#to_json_data has been added to the json_serializer plugin, for
113
+ returning a hash of data that can be converted to JSON, instead of
114
+ a JSON string.
115
+
116
+ * A :reject_nil option has been added to the nested_attributes method
117
+ in the nested_attributes plugin. This will ignore calls to the
118
+ nested attributes setter method where nil is passed as the setter
119
+ method argument.
120
+
121
+ = Other Improvements
122
+
123
+ * Model#freeze now works in case where model validation modifies the
124
+ object beyond adding errors.
125
+
126
+ * Model#freeze in the composition, serialization, and
127
+ serialization_modification_detection plugins now works in cases
128
+ where validation would end up loading the composed or
129
+ serialized values.
130
+
131
+ * Database#extension now avoids a possible thread safety issue that
132
+ could result in the extension being loaded into the Database twice.
133
+
134
+ * The ado adapter now supports overriding the timestamp conversion
135
+ proc. Previously, unlike other conversion procs, the timestamp
136
+ conversion proc was hard coded and could not be overridden.
data/doc/testing.rdoc CHANGED
@@ -157,6 +157,8 @@ The SEQUEL_INTEGRATION_URL environment variable specifies the Database connectio
157
157
 
158
158
  === Other
159
159
 
160
+ SEQUEL_ASYNC_THREAD_POOL :: Use the async_thread_pool extension when running the specs
161
+ SEQUEL_ASYNC_THREAD_POOL_PREEMPT :: Use the async_thread_pool extension when running the specs, with the :preempt_async_thread option
160
162
  SEQUEL_COLUMNS_INTROSPECTION :: Use the columns_introspection extension when running the specs
161
163
  SEQUEL_CONNECTION_VALIDATOR :: Use the connection validator extension when running the specs
162
164
  SEQUEL_DUPLICATE_COLUMNS_HANDLER :: Use the duplicate columns handler extension with value given when running the specs
@@ -195,10 +195,25 @@ module Sequel
195
195
  end
196
196
 
197
197
  @conversion_procs = CONVERSION_PROCS.dup
198
+ @conversion_procs[AdDBTimeStamp] = method(:adb_timestamp_to_application_timestamp)
198
199
 
199
200
  super
200
201
  end
201
202
 
203
+ def adb_timestamp_to_application_timestamp(v)
204
+ # This hard codes a timestamp_precision of 6 when converting.
205
+ # That is the default timestamp_precision, but the ado/mssql adapter uses a timestamp_precision
206
+ # of 3. However, timestamps returned by ado/mssql have nsec values that end up rounding to a
207
+ # the same value as if a timestamp_precision of 3 was hard coded (either xxx999yzz, where y is
208
+ # 5-9 or xxx000yzz where y is 0-4).
209
+ #
210
+ # ADO subadapters should override this they would like a different timestamp precision and the
211
+ # this code does not work for them (for example, if they provide full nsec precision).
212
+ #
213
+ # Note that fractional second handling for WIN32OLE objects is not correct on ruby <2.2
214
+ to_application_timestamp([v.year, v.month, v.day, v.hour, v.min, v.sec, (v.nsec/1000.0).round * 1000])
215
+ end
216
+
202
217
  def dataset_class_default
203
218
  Dataset
204
219
  end
@@ -233,23 +248,8 @@ module Sequel
233
248
  cols = []
234
249
  conversion_procs = db.conversion_procs
235
250
 
236
- ts_cp = nil
237
251
  recordset.Fields.each do |field|
238
- type = field.Type
239
- cp = if type == AdDBTimeStamp
240
- ts_cp ||= begin
241
- nsec_div = 1000000000.0/(10**(timestamp_precision))
242
- nsec_mul = 10**(timestamp_precision+3)
243
- meth = db.method(:to_application_timestamp)
244
- lambda do |v|
245
- # Fractional second handling is not correct on ruby <2.2
246
- meth.call([v.year, v.month, v.day, v.hour, v.min, v.sec, (v.nsec/nsec_div).round * nsec_mul])
247
- end
248
- end
249
- else
250
- conversion_procs[type]
251
- end
252
- cols << [output_identifier(field.Name), cp]
252
+ cols << [output_identifier(field.Name), conversion_procs[field.Type]]
253
253
  end
254
254
 
255
255
  self.columns = cols.map(&:first)
@@ -213,8 +213,7 @@ module Sequel
213
213
  Sequel.extension(*exts)
214
214
  exts.each do |ext|
215
215
  if pr = Sequel.synchronize{EXTENSIONS[ext]}
216
- unless Sequel.synchronize{@loaded_extensions.include?(ext)}
217
- Sequel.synchronize{@loaded_extensions << ext}
216
+ if Sequel.synchronize{@loaded_extensions.include?(ext) ? false : (@loaded_extensions << ext)}
218
217
  pr.call(self)
219
218
  end
220
219
  else
@@ -159,7 +159,7 @@ module Sequel
159
159
  nil
160
160
  end
161
161
 
162
- # Adds a named constraint (or unnamed if name is nil),
162
+ # Adds a named CHECK constraint (or unnamed if name is nil),
163
163
  # with the given block or args. To provide options for the constraint, pass
164
164
  # a hash as the first argument.
165
165
  #
@@ -167,6 +167,15 @@ module Sequel
167
167
  # # CONSTRAINT blah CHECK num >= 1 AND num <= 5
168
168
  # constraint({name: :blah, deferrable: true}, num: 1..5)
169
169
  # # CONSTRAINT blah CHECK num >= 1 AND num <= 5 DEFERRABLE INITIALLY DEFERRED
170
+ #
171
+ # If the first argument is a hash, the following options are supported:
172
+ #
173
+ # Options:
174
+ # :name :: The name of the CHECK constraint
175
+ # :deferrable :: Whether the CHECK constraint should be marked DEFERRABLE.
176
+ #
177
+ # PostgreSQL specific options:
178
+ # :not_valid :: Whether the CHECK constraint should be marked NOT VALID.
170
179
  def constraint(name, *args, &block)
171
180
  opts = name.is_a?(Hash) ? name : {:name=>name}
172
181
  constraints << opts.merge(:type=>:check, :check=>block || args)
@@ -262,6 +262,10 @@ module Sequel
262
262
  # # SELECT * FROM items WHERE foo
263
263
  # # WITH CHECK OPTION
264
264
  #
265
+ # DB.create_view(:bar_items, DB[:items].select(:foo), columns: [:bar])
266
+ # # CREATE VIEW bar_items (bar) AS
267
+ # # SELECT foo FROM items
268
+ #
265
269
  # Options:
266
270
  # :columns :: The column names to use for the view. If not given,
267
271
  # automatically determined based on the input dataset.
@@ -0,0 +1,438 @@
1
+ # frozen-string-literal: true
2
+ #
3
+ # The async_thread_pool extension adds support for running database
4
+ # queries in a separate threads using a thread pool. With the following
5
+ # code
6
+ #
7
+ # DB.extension :async_thread_pool
8
+ # foos = DB[:foos].async.where{:name=>'A'..'M'}.all
9
+ # bar_names = DB[:bar].async.select_order_map(:name)
10
+ # baz_1 = DB[:bazes].async.first(:id=>1)
11
+ #
12
+ # All 3 queries will be run in separate threads. +foos+, +bar_names+
13
+ # and +baz_1+ will be proxy objects. Calling a method on the proxy
14
+ # object will wait for the query to be run, and will return the result
15
+ # of calling that method on the result of the query method. For example,
16
+ # if you run:
17
+ #
18
+ # foos = DB[:foos].async.where{:name=>'A'..'M'}.all
19
+ # bar_names = DB[:bars].async.select_order_map(:name)
20
+ # baz_1 = DB[:bazes].async.first(:id=>1)
21
+ # sleep(1)
22
+ # foos.size
23
+ # bar_names.first
24
+ # baz_1.name
25
+ #
26
+ # These three queries will generally be run concurrently in separate
27
+ # threads. If you instead run:
28
+ #
29
+ # DB[:foos].async.where{:name=>'A'..'M'}.all.size
30
+ # DB[:bars].async.select_order_map(:name).first
31
+ # DB[:bazes].async.first(:id=>1).name
32
+ #
33
+ # Then will run each query sequentially, since you need the result of
34
+ # one query before running the next query. The queries will still be
35
+ # run in separate threads (by default).
36
+ #
37
+ # What is run in the separate thread is the entire method call that
38
+ # returns results. So with the original example:
39
+ #
40
+ # foos = DB[:foos].async.where{:name=>'A'..'M'}.all
41
+ # bar_names = DB[:bars].async.select_order_map(:name)
42
+ # baz_1 = DB[:bazes].async.first(:id=>1)
43
+ #
44
+ # The +all+, <tt>select_order_map(:name)</tt>, and <tt>first(:id=>1)</tt>
45
+ # calls are run in separate threads. If a block is passed to a method
46
+ # such as +all+ or +each+, the block is also run in that thread. If you
47
+ # have code such as:
48
+ #
49
+ # h = {}
50
+ # DB[:foos].async.each{|row| h[row[:id]] = row}
51
+ # bar_names = DB[:bars].async.select_order_map(:name)
52
+ # p h
53
+ #
54
+ # You may end up with it printing an empty hash or partial hash, because the
55
+ # async +each+ call will not have run or finished running. Since the
56
+ # <tt>p h</tt> code relies on a side-effect of the +each+ block and not the
57
+ # return value of the +each+ call, it will not wait for the loading.
58
+ #
59
+ # You should avoid using +async+ for any queries where you are ignoring the
60
+ # return value, as otherwise you have no way to wait for the query to be run.
61
+ #
62
+ # Datasets that use async will use async threads to load data for the majority
63
+ # of methods that can return data. However, dataset methods that return
64
+ # enumerators will not use an async thread (e.g. calling # Dataset#map
65
+ # without a block or arguments does not use an async thread or return a
66
+ # proxy object).
67
+ #
68
+ # Because async methods (including their blocks) run in a separate thread, you
69
+ # should not use control flow modifiers such as +return+ or +break+ in async
70
+ # queries. Doing so will result in a error.
71
+ #
72
+ # Because async results are returned as proxy objects, it's a bad idea
73
+ # to use them in a boolean setting:
74
+ #
75
+ # result = DB[:foo].async.get(:boolean_column)
76
+ # # or:
77
+ # result = DB[:foo].async.first
78
+ #
79
+ # # ...
80
+ # if result
81
+ # # will always execute this banch, since result is a proxy object
82
+ # end
83
+ #
84
+ # In this case, you can call the +__value+ method to return the actual
85
+ # result:
86
+ #
87
+ # if result.__value
88
+ # # will not execute this branch if the dataset method returned nil or false
89
+ # end
90
+ #
91
+ # Similarly, because a proxy object is used, you should be careful using the
92
+ # result in a case statement or an argument to <tt>Class#===</tt>:
93
+ #
94
+ # # ...
95
+ # case result
96
+ # when Hash, true, false
97
+ # # will never take this branch, since result is a proxy object
98
+ # end
99
+ #
100
+ # Similar to usage in an +if+ statement, you should use +__value+:
101
+ #
102
+ # case result.__value
103
+ # when Hash, true, false
104
+ # # will never take this branch, since result is a proxy object
105
+ # end
106
+ #
107
+ # On Ruby 2.2+, you can use +itself+ instead of +__value+. It's preferable to
108
+ # use +itself+ if you can, as that will allow code to work with both proxy
109
+ # objects and regular objects.
110
+ #
111
+ # Because separate threads and connections are used for async queries,
112
+ # they do not use any state on the current connection/thread. So if
113
+ # you do:
114
+ #
115
+ # DB.transaction{DB[:table].async.all}
116
+ #
117
+ # Be aware that the transaction runs on one connection, and the SELECT
118
+ # query on a different connection. If you use currently using
119
+ # transactional testing (running each test inside a transaction/savepoint),
120
+ # and want to start using this extension, you should first switch to
121
+ # non-transactional testing of the code that will use the async thread
122
+ # pool before using this extension, as otherwise the use of
123
+ # <tt>Dataset#async</tt> will likely break your tests.
124
+ #
125
+ # If you are using Database#synchronize to checkout a connection, the
126
+ # same issue applies, where the async query runs on a different
127
+ # connection:
128
+ #
129
+ # DB.synchronize{DB[:table].async.all}
130
+ #
131
+ # Similarly, if you are using the server_block extension, any async
132
+ # queries inside with_server blocks will not use the server specified:
133
+ #
134
+ # DB.with_server(:shard1) do
135
+ # DB[:a].all # Uses shard1
136
+ # DB[:a].async.all # Uses default shard
137
+ # end
138
+ #
139
+ # You need to manually specify the shard for any dataset using an async
140
+ # query:
141
+ #
142
+ # DB.with_server(:shard1) do
143
+ # DB[:a].all # Uses shard1
144
+ # DB[:a].async.server(:shard1).all # Uses shard1
145
+ # end
146
+ #
147
+ # When the async_thread_pool extension, the size of the async thread pool
148
+ # can be set by using the +:num_async_threads+ Database option, which must
149
+ # be set before loading the async_thread_pool extension. This defaults
150
+ # to the size of the Database object's connection pool.
151
+ #
152
+ # By default, for consistent behavior, the async_thread_pool extension
153
+ # will always run the query in a separate thread. However, in some cases,
154
+ # such as when the async thread pool is busy and the results of a query
155
+ # are needed right away, it can improve performance to allow preemption,
156
+ # so that the query will run in the current thread instead of waiting
157
+ # for an async thread to become available. With the following code:
158
+ #
159
+ # foos = DB[:foos].async.where{:name=>'A'..'M'}.all
160
+ # bar_names = DB[:bar].async.select_order_map(:name)
161
+ # if foos.length > 4
162
+ # baz_1 = DB[:bazes].async.first(:id=>1)
163
+ # end
164
+ #
165
+ # Whether you need the +baz_1+ variable depends on the value of foos.
166
+ # If the async thread pool is busy, and by the time the +foos.length+
167
+ # call is made, the async thread pool has not started the processing
168
+ # to get the +foos+ value, it can improve performance to start that
169
+ # processing in the current thread, since it is needed immediately to
170
+ # determine whether to schedule query to get the +baz_1+ variable.
171
+ # The default is to not allow preemption, because if the current
172
+ # thread is used, it may have already checked out a connection that
173
+ # could be used, and that connection could be inside a transaction or
174
+ # have some other manner of connection-specific state applied to it.
175
+ # If you want to allow preemption, you can set the
176
+ # +:preempt_async_thread+ Database option before loading the
177
+ # async_thread_pool extension.
178
+ #
179
+ # Related module: Sequel::Database::AsyncThreadPool::DatasetMethods
180
+
181
+
182
+ #
183
+ module Sequel
184
+ module Database::AsyncThreadPool
185
+ # JobProcessor is a wrapper around a single thread, that will
186
+ # process a queue of jobs until it is shut down.
187
+ class JobProcessor # :nodoc:
188
+ def self.create_finalizer(queue, pool)
189
+ proc{run_finalizer(queue, pool)}
190
+ end
191
+
192
+ def self.run_finalizer(queue, pool)
193
+ # Push a nil for each thread using the queue, signalling
194
+ # that thread to close.
195
+ pool.each{queue.push(nil)}
196
+
197
+ # Join each of the closed threads.
198
+ pool.each(&:join)
199
+
200
+ # Clear the thread pool. Probably not necessary, but this allows
201
+ # for a simple way to check whether this finalizer has been run.
202
+ pool.clear
203
+
204
+ nil
205
+ end
206
+ private_class_method :run_finalizer
207
+
208
+ def initialize(queue)
209
+ @thread = ::Thread.new do
210
+ while proxy = queue.pop
211
+ proxy.__send__(:__run)
212
+ end
213
+ end
214
+ end
215
+
216
+ # Join the thread, should only be called by the related finalizer.
217
+ def join
218
+ @thread.join
219
+ end
220
+ end
221
+
222
+ # Wrapper for exception instances raised by async jobs. The
223
+ # wrapped exception will be raised by the code getting the value
224
+ # of the job.
225
+ WrappedException = Struct.new(:exception)
226
+
227
+ # Base proxy object class for jobs processed by async threads and
228
+ # the returned result.
229
+ class BaseProxy < BasicObject
230
+ # Store a block that returns the result when called.
231
+ def initialize(&block)
232
+ ::Kernel.raise Error, "must provide block for an async job" unless block
233
+ @block = block
234
+ end
235
+
236
+ # Pass all method calls to the returned result.
237
+ def method_missing(*args, &block)
238
+ __value.public_send(*args, &block)
239
+ end
240
+ # :nocov:
241
+ ruby2_keywords(:method_missing) if respond_to?(:ruby2_keywords, true)
242
+ # :nocov:
243
+
244
+ # Delegate respond_to? calls to the returned result.
245
+ def respond_to_missing?(*args)
246
+ __value.respond_to?(*args)
247
+ end
248
+
249
+ # Override some methods defined by default so they apply to the
250
+ # returned result and not the current object.
251
+ [:!, :==, :!=, :instance_eval, :instance_exec].each do |method|
252
+ define_method(method) do |*args, &block|
253
+ __value.public_send(method, *args, &block)
254
+ end
255
+ end
256
+
257
+ # Wait for the value to be loaded if it hasn't already been loaded.
258
+ # If the code to load the return value raised an exception that was
259
+ # wrapped, reraise the exception.
260
+ def __value
261
+ unless defined?(@value)
262
+ __get_value
263
+ end
264
+
265
+ if @value.is_a?(WrappedException)
266
+ ::Kernel.raise @value
267
+ end
268
+
269
+ @value
270
+ end
271
+
272
+ private
273
+
274
+ # Run the block and return the block value. If the block call raises
275
+ # an exception, wrap the exception.
276
+ def __run_block
277
+ # This may not catch concurrent calls (unless surrounded by a mutex), but
278
+ # it's not worth trying to protect against that. It's enough to just check for
279
+ # multiple non-concurrent calls.
280
+ ::Kernel.raise Error, "Cannot run async block multiple times" unless block = @block
281
+
282
+ @block = nil
283
+
284
+ begin
285
+ block.call
286
+ rescue ::Exception => e
287
+ WrappedException.new(e)
288
+ end
289
+ end
290
+ end
291
+
292
+ # Default object class for async job/proxy result. This uses a queue for
293
+ # synchronization. The JobProcessor will push a result until the queue,
294
+ # and the code to get the value will pop the result from that queue (and
295
+ # repush the result to handle thread safety).
296
+ class Proxy < BaseProxy
297
+ def initialize
298
+ super
299
+ @queue = ::Queue.new
300
+ end
301
+
302
+ private
303
+
304
+ def __run
305
+ @queue.push(__run_block)
306
+ end
307
+
308
+ def __get_value
309
+ @value = @queue.pop
310
+
311
+ # Handle thread-safety by repushing the popped value, so that
312
+ # concurrent calls will receive the same value
313
+ @queue.push(@value)
314
+ end
315
+ end
316
+
317
+ # Object class for async job/proxy result when the :preempt_async_thread
318
+ # Database option is used. Uses a mutex for synchronization, and either
319
+ # the JobProcessor or the calling thread can run code to get the value.
320
+ class PreemptableProxy < BaseProxy
321
+ def initialize
322
+ super
323
+ @mutex = ::Mutex.new
324
+ end
325
+
326
+ private
327
+
328
+ def __get_value
329
+ @mutex.synchronize do
330
+ unless defined?(@value)
331
+ @value = __run_block
332
+ end
333
+ end
334
+ end
335
+ alias __run __get_value
336
+ end
337
+
338
+ module DatabaseMethods
339
+ def self.extended(db)
340
+ db.instance_exec do
341
+ unless pool.pool_type == :threaded || pool.pool_type == :sharded_threaded
342
+ raise Error, "can only load async_thread_pool extension if using threaded or sharded_threaded connection pool"
343
+ end
344
+
345
+ num_async_threads = opts[:num_async_threads] ? typecast_value_integer(opts[:num_async_threads]) : (Integer(opts[:max_connections] || 4))
346
+ raise Error, "must have positive number for num_async_threads" if num_async_threads <= 0
347
+
348
+ proxy_klass = typecast_value_boolean(opts[:preempt_async_thread]) ? PreemptableProxy : Proxy
349
+ define_singleton_method(:async_job_class){proxy_klass}
350
+
351
+ queue = @async_thread_queue = Queue.new
352
+ pool = @async_thread_pool = num_async_threads.times.map{JobProcessor.new(queue)}
353
+ ObjectSpace.define_finalizer(db, JobProcessor.create_finalizer(queue, pool))
354
+
355
+ extend_datasets(DatasetMethods)
356
+ end
357
+ end
358
+
359
+ private
360
+
361
+ # Wrap the block in a job/proxy object and schedule it to run using the async thread pool.
362
+ def async_run(&block)
363
+ proxy = async_job_class.new(&block)
364
+ @async_thread_queue.push(proxy)
365
+ proxy
366
+ end
367
+ end
368
+
369
+ ASYNC_METHODS = ([:all?, :any?, :drop, :entries, :grep_v, :include?, :inject, :member?, :minmax, :none?, :one?, :reduce, :sort, :take, :tally, :to_a, :to_h, :uniq, :zip] & Enumerable.instance_methods) + (Dataset::ACTION_METHODS - [:map, :paged_each])
370
+ ASYNC_BLOCK_METHODS = ([:collect, :collect_concat, :detect, :drop_while, :each_cons, :each_entry, :each_slice, :each_with_index, :each_with_object, :filter_map, :find, :find_all, :find_index, :flat_map, :max_by, :min_by, :minmax_by, :partition, :reject, :reverse_each, :sort_by, :take_while] & Enumerable.instance_methods) + [:paged_each]
371
+ ASYNC_ARGS_OR_BLOCK_METHODS = [:map]
372
+
373
+ module DatasetMethods
374
+ # Define an method in the given module that will run the given method using an async thread
375
+ # if the current dataset is async.
376
+ def self.define_async_method(mod, method)
377
+ mod.send(:define_method, method) do |*args, &block|
378
+ if @opts[:async]
379
+ ds = sync
380
+ db.send(:async_run){ds.send(method, *args, &block)}
381
+ else
382
+ super(*args, &block)
383
+ end
384
+ end
385
+ end
386
+
387
+ # Define an method in the given module that will run the given method using an async thread
388
+ # if the current dataset is async and a block is provided.
389
+ def self.define_async_block_method(mod, method)
390
+ mod.send(:define_method, method) do |*args, &block|
391
+ if block && @opts[:async]
392
+ ds = sync
393
+ db.send(:async_run){ds.send(method, *args, &block)}
394
+ else
395
+ super(*args, &block)
396
+ end
397
+ end
398
+ end
399
+
400
+ # Define an method in the given module that will run the given method using an async thread
401
+ # if the current dataset is async and arguments or a block is provided.
402
+ def self.define_async_args_or_block_method(mod, method)
403
+ mod.send(:define_method, method) do |*args, &block|
404
+ if (block || !args.empty?) && @opts[:async]
405
+ ds = sync
406
+ db.send(:async_run){ds.send(method, *args, &block)}
407
+ else
408
+ super(*args, &block)
409
+ end
410
+ end
411
+ end
412
+
413
+ # Override all of the methods that return results to do the processing in an async thread
414
+ # if they have been marked to run async and should run async (i.e. they don't return an
415
+ # Enumerator).
416
+ ASYNC_METHODS.each{|m| define_async_method(self, m)}
417
+ ASYNC_BLOCK_METHODS.each{|m| define_async_block_method(self, m)}
418
+ ASYNC_ARGS_OR_BLOCK_METHODS.each{|m| define_async_args_or_block_method(self, m)}
419
+
420
+ # Return a cloned dataset that will load results using the async thread pool.
421
+ def async
422
+ cached_dataset(:_async) do
423
+ clone(:async=>true)
424
+ end
425
+ end
426
+
427
+ # Return a cloned dataset that will not load results using the async thread pool.
428
+ # Only used if the current dataset has been marked as using the async thread pool.
429
+ def sync
430
+ cached_dataset(:_sync) do
431
+ clone(:async=>false)
432
+ end
433
+ end
434
+ end
435
+ end
436
+
437
+ Database.register_extension(:async_thread_pool, Database::AsyncThreadPool::DatabaseMethods)
438
+ end
@@ -1260,12 +1260,12 @@ module Sequel
1260
1260
  # Once an object is frozen, you cannot modify it's values, changed_columns,
1261
1261
  # errors, or dataset.
1262
1262
  def freeze
1263
- values.freeze
1264
- _changed_columns.freeze
1265
1263
  unless errors.frozen?
1266
1264
  validate
1267
1265
  errors.freeze
1268
1266
  end
1267
+ values.freeze
1268
+ _changed_columns.freeze
1269
1269
  this if !new? && model.primary_key
1270
1270
  super
1271
1271
  end
@@ -0,0 +1,39 @@
1
+ # frozen-string-literal: true
2
+
3
+ module Sequel
4
+ extension 'async_thread_pool'
5
+
6
+ module Plugins
7
+ # The async_thread_pool plugin makes it slightly easier to use the async_thread_pool
8
+ # Dataset extension with models. It makes Model.async return an async dataset for the
9
+ # model, and support async behavior for #destroy, #with_pk, and #with_pk! for model
10
+ # datasets:
11
+ #
12
+ # # Will load the artist with primary key 1 asynchronously
13
+ # artist = Artist.async.with_pk(1)
14
+ #
15
+ # You must load the async_thread_pool Database extension into the Database object the
16
+ # model class uses in order for async behavior to work.
17
+ #
18
+ # Usage:
19
+ #
20
+ # # Make all model subclass datasets support support async class methods and additional
21
+ # # async dataset methods
22
+ # Sequel::Model.plugin :async_thread_pool
23
+ #
24
+ # # Make the Album class support async class method and additional async dataset methods
25
+ # Album.plugin :async_thread_pool
26
+ module AsyncThreadPool
27
+ module ClassMethods
28
+ Plugins.def_dataset_methods(self, :async)
29
+ end
30
+
31
+ module DatasetMethods
32
+ [:destroy, :with_pk, :with_pk!].each do |meth|
33
+ ::Sequel::Database::AsyncThreadPool::DatasetMethods.define_async_method(self, meth)
34
+ end
35
+ end
36
+ end
37
+ end
38
+ end
39
+
@@ -171,8 +171,9 @@ module Sequel
171
171
 
172
172
  # Freeze compositions hash when freezing model instance.
173
173
  def freeze
174
- compositions.freeze
174
+ compositions
175
175
  super
176
+ compositions.freeze
176
177
  end
177
178
 
178
179
  # For each composition, set the columns in the model class based
@@ -133,21 +133,39 @@ module Sequel
133
133
  end
134
134
  end
135
135
 
136
- # Helper class used for making sure that cascading options
137
- # for model associations works correctly. Cascaded options
138
- # work by creating instances of this class, which take a
139
- # literal JSON string and have +to_json+ return it.
136
+ # SEQUEL6: Remove
137
+ # :nocov:
140
138
  class Literal
141
- # Store the literal JSON to use
142
139
  def initialize(json)
143
140
  @json = json
144
141
  end
145
142
 
146
- # Return the literal JSON to use
147
143
  def to_json(*a)
148
144
  @json
149
145
  end
150
146
  end
147
+ # :nocov:
148
+ Sequel::Deprecation.deprecate_constant(self, :Literal)
149
+
150
+ # Convert the given object to a JSON data structure using the given arguments.
151
+ def self.object_to_json_data(obj, *args, &block)
152
+ if obj.is_a?(Array)
153
+ obj.map{|x| object_to_json_data(x, *args, &block)}
154
+ else
155
+ if obj.respond_to?(:to_json_data)
156
+ obj.to_json_data(*args, &block)
157
+ else
158
+ begin
159
+ Sequel.parse_json(Sequel.object_to_json(obj, *args, &block))
160
+ # :nocov:
161
+ rescue Sequel.json_parser_error_class
162
+ # Support for old Ruby code that only supports parsing JSON object/array
163
+ Sequel.parse_json(Sequel.object_to_json([obj], *args, &block))[0]
164
+ # :nocov:
165
+ end
166
+ end
167
+ end
168
+ end
151
169
 
152
170
  module ClassMethods
153
171
  # The default opts to use when serializing model objects to JSON.
@@ -324,20 +342,7 @@ module Sequel
324
342
  end
325
343
 
326
344
  v = v.empty? ? [] : [v]
327
-
328
- objs = public_send(k)
329
-
330
- is_array = if r = model.association_reflection(k)
331
- r.returns_array?
332
- else
333
- objs.is_a?(Array)
334
- end
335
-
336
- h[key_name] = if is_array
337
- objs.map{|obj| Literal.new(Sequel.object_to_json(obj, *v))}
338
- else
339
- Literal.new(Sequel.object_to_json(objs, *v))
340
- end
345
+ h[key_name] = JsonSerializer.object_to_json_data(public_send(k), *v)
341
346
  end
342
347
  else
343
348
  Array(inc).each do |c|
@@ -347,7 +352,8 @@ module Sequel
347
352
  else
348
353
  key_name = c.to_s
349
354
  end
350
- h[key_name] = public_send(c)
355
+
356
+ h[key_name] = JsonSerializer.object_to_json_data(public_send(c))
351
357
  end
352
358
  end
353
359
  end
@@ -362,6 +368,15 @@ module Sequel
362
368
  h = yield h if block_given?
363
369
  Sequel.object_to_json(h, *a)
364
370
  end
371
+
372
+ # Convert the receiver to a JSON data structure using the given arguments.
373
+ def to_json_data(*args, &block)
374
+ if block
375
+ to_json(*args){|x| return block.call(x)}
376
+ else
377
+ to_json(*args){|x| return x}
378
+ end
379
+ end
365
380
  end
366
381
 
367
382
  module DatasetMethods
@@ -420,7 +435,7 @@ module Sequel
420
435
  else
421
436
  all
422
437
  end
423
- array.map{|obj| Literal.new(Sequel.object_to_json(obj, opts, &opts[:instance_block]))}
438
+ JsonSerializer.object_to_json_data(array, opts, &opts[:instance_block])
424
439
  else
425
440
  all
426
441
  end
@@ -108,9 +108,10 @@ module Sequel
108
108
  # array of the allowable fields.
109
109
  # :limit :: For *_to_many associations, a limit on the number of records
110
110
  # that will be processed, to prevent denial of service attacks.
111
- # :reject_if :: A proc that is given each attribute hash before it is
111
+ # :reject_if :: A proc that is called with each attribute hash before it is
112
112
  # passed to its associated object. If the proc returns a truthy
113
113
  # value, the attribute hash is ignored.
114
+ # :reject_nil :: Ignore nil objects passed to nested attributes setter methods.
114
115
  # :remove :: Allow disassociation of nested records (can remove the associated
115
116
  # object from the parent object, but not destroy the associated object).
116
117
  # :require_modification :: Whether to require modification of nested objects when
@@ -146,8 +147,9 @@ module Sequel
146
147
  def def_nested_attribute_method(reflection)
147
148
  @nested_attributes_module.class_eval do
148
149
  meth = :"#{reflection[:name]}_attributes="
150
+ assoc = reflection[:name]
149
151
  define_method(meth) do |v|
150
- set_nested_attributes(reflection[:name], v)
152
+ set_nested_attributes(assoc, v)
151
153
  end
152
154
  alias_method meth, meth
153
155
  end
@@ -161,6 +163,7 @@ module Sequel
161
163
  def set_nested_attributes(assoc, obj, opts=OPTS)
162
164
  raise(Error, "no association named #{assoc} for #{model.inspect}") unless ref = model.association_reflection(assoc)
163
165
  raise(Error, "nested attributes are not enabled for association #{assoc} for #{model.inspect}") unless meta = ref[:nested_attributes]
166
+ return if obj.nil? && meta[:reject_nil]
164
167
  meta = meta.merge(opts)
165
168
  meta[:reflection] = ref
166
169
  if ref.returns_array?
@@ -177,8 +177,9 @@ module Sequel
177
177
 
178
178
  # Freeze the deserialized values
179
179
  def freeze
180
- deserialized_values.freeze
180
+ deserialized_values
181
181
  super
182
+ deserialized_values.freeze
182
183
  end
183
184
 
184
185
  # Serialize deserialized values before saving
@@ -50,8 +50,8 @@ module Sequel
50
50
  # Freeze the original deserialized values when freezing the instance.
51
51
  def freeze
52
52
  @original_deserialized_values ||= {}
53
- @original_deserialized_values.freeze
54
53
  super
54
+ @original_deserialized_values.freeze
55
55
  end
56
56
 
57
57
  private
@@ -6,7 +6,7 @@ module Sequel
6
6
 
7
7
  # The minor version of Sequel. Bumped for every non-patch level
8
8
  # release, generally around once a month.
9
- MINOR = 41
9
+ MINOR = 42
10
10
 
11
11
  # The tiny version of Sequel. Usually 0, only bumped for bugfix
12
12
  # releases that fix regressions from previous versions.
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: sequel
3
3
  version: !ruby/object:Gem::Version
4
- version: 5.41.0
4
+ version: 5.42.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Jeremy Evans
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2021-02-01 00:00:00.000000000 Z
11
+ date: 2021-03-01 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: minitest
@@ -185,6 +185,7 @@ extra_rdoc_files:
185
185
  - doc/release_notes/5.4.0.txt
186
186
  - doc/release_notes/5.40.0.txt
187
187
  - doc/release_notes/5.41.0.txt
188
+ - doc/release_notes/5.42.0.txt
188
189
  - doc/release_notes/5.5.0.txt
189
190
  - doc/release_notes/5.6.0.txt
190
191
  - doc/release_notes/5.7.0.txt
@@ -254,6 +255,7 @@ files:
254
255
  - doc/release_notes/5.4.0.txt
255
256
  - doc/release_notes/5.40.0.txt
256
257
  - doc/release_notes/5.41.0.txt
258
+ - doc/release_notes/5.42.0.txt
257
259
  - doc/release_notes/5.5.0.txt
258
260
  - doc/release_notes/5.6.0.txt
259
261
  - doc/release_notes/5.7.0.txt
@@ -352,6 +354,7 @@ files:
352
354
  - lib/sequel/extensions/_pretty_table.rb
353
355
  - lib/sequel/extensions/any_not_empty.rb
354
356
  - lib/sequel/extensions/arbitrary_servers.rb
357
+ - lib/sequel/extensions/async_thread_pool.rb
355
358
  - lib/sequel/extensions/auto_literal_strings.rb
356
359
  - lib/sequel/extensions/blank.rb
357
360
  - lib/sequel/extensions/caller_logging.rb
@@ -447,6 +450,7 @@ files:
447
450
  - lib/sequel/plugins/association_multi_add_remove.rb
448
451
  - lib/sequel/plugins/association_pks.rb
449
452
  - lib/sequel/plugins/association_proxies.rb
453
+ - lib/sequel/plugins/async_thread_pool.rb
450
454
  - lib/sequel/plugins/auto_validations.rb
451
455
  - lib/sequel/plugins/before_after_save.rb
452
456
  - lib/sequel/plugins/blacklist_security.rb