activemodel-datastore 0.9.0 → 0.11.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +22 -0
- data/README.md +65 -0
- data/lib/active_model/datastore/connection.rb +1 -2
- data/lib/active_model/datastore/version.rb +1 -1
- data/lib/active_model/datastore.rb +175 -66
- metadata +10 -4
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: ad3eda8122e638f1b739f22844f8f116154ce792563e757f944e1fd04c44145d
|
|
4
|
+
data.tar.gz: '09316b9ed555cc216f51692ca3d8ec944921149ac3a624d74c598a2a04dea2ef'
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: b83379af680f03b96c148997a59181a9a34403f2c537672fda824899c939dd23f755045399901be38518bbb1cbadfa7ea68c0e1e375c43b9f27b986cbdac29e7
|
|
7
|
+
data.tar.gz: 3a4145461ae6b6ba321a8b8136e86b665b6a73447ab6a6e42dbc7d2fad68cbe28001c873d6dd2547ffdfe15bfb0d77b75b4bed30612e10ee20def1a512afe5ed
|
data/CHANGELOG.md
CHANGED
|
@@ -1,3 +1,25 @@
|
|
|
1
|
+
### 0.11.0 / 2026-10-14
|
|
2
|
+
|
|
3
|
+
Upgrade notes:
|
|
4
|
+
* requires `google-cloud-datastore` 2.3.1 or later; older versions will no longer resolve
|
|
5
|
+
* `DATASTORE_NETWORK_TIMEOUT` is no longer read; it never took effect, so removing it from your environment changes nothing (see below for configuring per-RPC deadlines)
|
|
6
|
+
* slow-operation warnings are enabled by default, so successful Datastore calls taking over 5 seconds now log at WARN level; set `ActiveModel::Datastore.slow_operation_threshold = nil` to keep the previous behavior
|
|
7
|
+
|
|
8
|
+
Changes:
|
|
9
|
+
* fix `all(limit:)` returning a nil cursor for full pages split across Datastore response batches; use the final batch's end cursor
|
|
10
|
+
* retry `all(limit:)` as a whole page, so a failure in any later Datastore response batch gets the configured `read_retry_count` retries and logging, and `slow_operation_threshold` times every batch of the page; each attempt rebuilds the query from the original options and discards rows from the failed attempt
|
|
11
|
+
* retry each later batch of queries without a limit and each deferred lookup in `find` / `find_entities` with the configured `read_retry_count` and retry logging; one `slow_operation_threshold` warning covers the total time of the call's successful requests
|
|
12
|
+
* raise `ActiveModel::Datastore::Error` instead of `NoMethodError` or `TypeError` if Datastore reports more query results without an end cursor
|
|
13
|
+
* require Ruby 2.5 or newer and `google-cloud-datastore` 2.3.1 or later within 2.x for correct query limits across batches
|
|
14
|
+
* warn when a successful Datastore attempt exceeds `ActiveModel::Datastore.slow_operation_threshold` (enabled by default at 5 seconds; `nil` or a blank string disables slow-success warnings)
|
|
15
|
+
* add opt-in ID preallocation via `ActiveModel::Datastore.preallocate_ids` (disabled by default) to prevent duplicate entities when retrying a save; allocation is retried at most twice
|
|
16
|
+
* remove `DATASTORE_NETWORK_TIMEOUT`, which never took effect because the Datastore client's per-RPC 60-second deadlines override the client-wide timeout; set `Google::Cloud::Datastore::V1::Datastore::Client.configure { |c| c.rpcs.<rpc>.timeout = ... }` instead
|
|
17
|
+
|
|
18
|
+
### 0.10.0 / 2026-09-22
|
|
19
|
+
* stop retrying invalid argument errors
|
|
20
|
+
* make read retry counts configurable independently of writes
|
|
21
|
+
* log a warning when a retry chain gives up, including its attempt count
|
|
22
|
+
|
|
1
23
|
### 0.9.0 / 2026-08-24
|
|
2
24
|
* identify the Datastore operation, entity kind, and failed-attempt elapsed time in retry logs
|
|
3
25
|
* use a configurable retry logger that defaults to Rails.logger and falls back to standard output
|
data/README.md
CHANGED
|
@@ -89,6 +89,71 @@ configure it during application initialization:
|
|
|
89
89
|
ActiveModel::Datastore.logger = MyApplication.logger
|
|
90
90
|
```
|
|
91
91
|
|
|
92
|
+
Successful attempts taking more than five seconds also log a warning, such as
|
|
93
|
+
`Datastore save for Report took 6000 ms`. This measures each attempt separately, excluding retry
|
|
94
|
+
backoff. Both retry helpers use this logging when an `operation:` label is provided.
|
|
95
|
+
|
|
96
|
+
`all(limit:)` retries and times the complete page, including all response batches. Each retry starts
|
|
97
|
+
from the original query options and discards rows from the failed attempt. A retry reads batches
|
|
98
|
+
that already succeeded again, and because cursors don't preserve a snapshot, the retried page can
|
|
99
|
+
include changes made between attempts. Queries without a limit and multi-key lookups (`find` with
|
|
100
|
+
several IDs, `find_entities`) retry each request separately, so a retry reads only the batch that
|
|
101
|
+
failed. Their slow-operation warning covers the whole call: the total time of its successful
|
|
102
|
+
requests, excluding failed attempts and retry backoff.
|
|
103
|
+
|
|
104
|
+
Configure the threshold in seconds during initialization:
|
|
105
|
+
|
|
106
|
+
```ruby
|
|
107
|
+
ActiveModel::Datastore.slow_operation_threshold = 2
|
|
108
|
+
```
|
|
109
|
+
|
|
110
|
+
Numbers and numeric strings (such as `"2"` from an environment variable) are accepted. Set it to
|
|
111
|
+
`nil` or a blank string to disable slow-success warnings. Invalid values raise when the setting
|
|
112
|
+
is assigned. Failure logging and retry behavior are unchanged.
|
|
113
|
+
|
|
114
|
+
To change the gem's read retry count (default: `5`), configure it during initialization:
|
|
115
|
+
|
|
116
|
+
```ruby
|
|
117
|
+
ActiveModel::Datastore.read_retry_count = 2
|
|
118
|
+
```
|
|
119
|
+
|
|
120
|
+
Set it to `0` to disable read retries. Write retry counts are unchanged. Invalid argument errors
|
|
121
|
+
are never retried.
|
|
122
|
+
|
|
123
|
+
ID preallocation is disabled by default, preserving the existing save behavior. To enable it for
|
|
124
|
+
models saved through `save` or `save!`, configure it during application initialization:
|
|
125
|
+
|
|
126
|
+
```ruby
|
|
127
|
+
ActiveModel::Datastore.preallocate_ids = true
|
|
128
|
+
```
|
|
129
|
+
|
|
130
|
+
The setter uses `ActiveModel::Type::Boolean`, so environment-variable strings such as `"false"`,
|
|
131
|
+
`"0"`, and `"off"` disable preallocation. Unset (`nil`) and empty values also leave it disabled.
|
|
132
|
+
|
|
133
|
+
When enabled, a new entity without an ID gets a complete key before the save retry loop. Every
|
|
134
|
+
attempt in that loop uses the same key, preventing duplicate entities if a commit succeeds but its
|
|
135
|
+
response fails. This adds one allocation RPC per create under normal conditions. Existing numeric
|
|
136
|
+
IDs and named keys require no allocation, and parent paths are preserved.
|
|
137
|
+
|
|
138
|
+
Allocation writes nothing, so it is retried at most twice before giving up, leaving more of the
|
|
139
|
+
request's time for the save. If allocation still fails, `save` returns `false` and `save!` raises
|
|
140
|
+
`ActiveModel::Datastore::EntityNotSavedError` without attempting the write.
|
|
141
|
+
|
|
142
|
+
Set the option to `false` to disable preallocation. It does not cover `update` on a model without an
|
|
143
|
+
ID, direct calls to `CloudDatastore.dataset.save`, or deduplication across separate save calls or
|
|
144
|
+
requests.
|
|
145
|
+
|
|
146
|
+
Each Datastore RPC has its own 60-second default deadline in the underlying client, which takes
|
|
147
|
+
precedence over the client-wide `timeout` option. Applications with a request timeout (for example,
|
|
148
|
+
Heroku's 30-second router timeout or Rack::Timeout) should give allocation a short deadline so a
|
|
149
|
+
stalled allocation cannot use the time the save needs:
|
|
150
|
+
|
|
151
|
+
```ruby
|
|
152
|
+
Google::Cloud::Datastore::V1::Datastore::Client.configure do |config|
|
|
153
|
+
config.rpcs.allocate_ids.timeout = 3
|
|
154
|
+
end
|
|
155
|
+
```
|
|
156
|
+
|
|
92
157
|
There is an example Puma config file [here](https://github.com/Agrimatics/activemodel-datastore/blob/main/test/support/datastore_example_rails_app/config/puma.rb).
|
|
93
158
|
|
|
94
159
|
## <a name="model"></a>Model Example
|
|
@@ -119,6 +119,9 @@ module ActiveModel::Datastore
|
|
|
119
119
|
include ActiveModel::Datastore::PropertyValues
|
|
120
120
|
include ActiveModel::Datastore::TrackChanges
|
|
121
121
|
|
|
122
|
+
# Allocation writes nothing, so a failure can give up quickly and leave time for the save.
|
|
123
|
+
ALLOCATE_RETRY_COUNT = 2
|
|
124
|
+
|
|
122
125
|
def self.logger
|
|
123
126
|
@logger || (Rails.logger if defined?(Rails) == 'constant')
|
|
124
127
|
end
|
|
@@ -127,8 +130,35 @@ module ActiveModel::Datastore
|
|
|
127
130
|
@logger = logger
|
|
128
131
|
end
|
|
129
132
|
|
|
133
|
+
def self.slow_operation_threshold
|
|
134
|
+
defined?(@slow_operation_threshold) ? @slow_operation_threshold : 5
|
|
135
|
+
end
|
|
136
|
+
|
|
137
|
+
def self.slow_operation_threshold=(seconds)
|
|
138
|
+
seconds = nil if seconds.is_a?(String) && seconds.strip.empty?
|
|
139
|
+
@slow_operation_threshold = seconds.nil? ? nil : Float(seconds)
|
|
140
|
+
end
|
|
141
|
+
|
|
142
|
+
def self.read_retry_count
|
|
143
|
+
@read_retry_count || 5
|
|
144
|
+
end
|
|
145
|
+
|
|
146
|
+
def self.read_retry_count=(count)
|
|
147
|
+
@read_retry_count = count
|
|
148
|
+
end
|
|
149
|
+
|
|
150
|
+
def self.preallocate_ids
|
|
151
|
+
@preallocate_ids || false
|
|
152
|
+
end
|
|
153
|
+
|
|
154
|
+
def self.preallocate_ids=(enabled)
|
|
155
|
+
@preallocate_ids = ActiveModel::Type::Boolean.new.cast(enabled)
|
|
156
|
+
end
|
|
157
|
+
|
|
130
158
|
included do
|
|
131
|
-
private_class_method :query_options, :query_sort, :query_property_filter, :find_all_entities
|
|
159
|
+
private_class_method :query_options, :query_sort, :query_property_filter, :find_all_entities,
|
|
160
|
+
:retry_read, :log_retry_failure, :log_slow_operation, :all_with_retries,
|
|
161
|
+
:missing_cursor_error, :retry_with_backoff
|
|
132
162
|
define_model_callbacks :save, :update, :destroy
|
|
133
163
|
attr_accessor :id, :parent_key_id, :entity_property_values
|
|
134
164
|
end
|
|
@@ -210,7 +240,8 @@ module ActiveModel::Datastore
|
|
|
210
240
|
|
|
211
241
|
run_callbacks :save do
|
|
212
242
|
entity = build_entity(parent)
|
|
213
|
-
success =
|
|
243
|
+
success = preallocate_entity_id(entity)
|
|
244
|
+
success &&= self.class.retry_on_exception?(operation: 'save') do
|
|
214
245
|
CloudDatastore.dataset.save entity
|
|
215
246
|
end
|
|
216
247
|
self.id = entity.key.id if success
|
|
@@ -219,6 +250,19 @@ module ActiveModel::Datastore
|
|
|
219
250
|
end
|
|
220
251
|
end
|
|
221
252
|
|
|
253
|
+
def preallocate_entity_id(entity)
|
|
254
|
+
return true unless ActiveModel::Datastore.preallocate_ids && entity.key.incomplete?
|
|
255
|
+
|
|
256
|
+
# A commit can succeed even when its response fails. Reuse one key for every save attempt.
|
|
257
|
+
key = self.class.retry_on_exception?(ALLOCATE_RETRY_COUNT, operation: 'allocate id') do
|
|
258
|
+
CloudDatastore.dataset.allocate_ids(entity.key, 1).first ||
|
|
259
|
+
raise(Google::Cloud::Error, 'allocate_ids returned no key')
|
|
260
|
+
end
|
|
261
|
+
return false unless key
|
|
262
|
+
|
|
263
|
+
entity.key = key
|
|
264
|
+
end
|
|
265
|
+
|
|
222
266
|
# Methods defined here will be class methods when 'include ActiveModel::Datastore'.
|
|
223
267
|
module ClassMethods
|
|
224
268
|
##
|
|
@@ -239,7 +283,7 @@ module ActiveModel::Datastore
|
|
|
239
283
|
def find_entity(id_or_name, parent = nil)
|
|
240
284
|
key = CloudDatastore.dataset.key name, id_or_name
|
|
241
285
|
key.parent = parent if parent.present?
|
|
242
|
-
|
|
286
|
+
retry_read('find') { CloudDatastore.dataset.find key }
|
|
243
287
|
end
|
|
244
288
|
|
|
245
289
|
##
|
|
@@ -248,10 +292,9 @@ module ActiveModel::Datastore
|
|
|
248
292
|
# additional values. LookupResults are returned in batches, and the batch size is
|
|
249
293
|
# determined by the Datastore API. Batch size is not guaranteed. It will be affected
|
|
250
294
|
# by the size of the data being returned, and by other forces such as how distributed
|
|
251
|
-
# and/or consistent the data in Datastore is.
|
|
252
|
-
#
|
|
253
|
-
#
|
|
254
|
-
# the model entity objects.
|
|
295
|
+
# and/or consistent the data in Datastore is. Deferred keys are retrieved by repeatedly
|
|
296
|
+
# loading #next until #next? returns false, retrying each request. We iterate on the
|
|
297
|
+
# resulting enumerator to return the model entity objects.
|
|
255
298
|
#
|
|
256
299
|
# @param [Integer, String] ids_or_names One or more ids to retrieve.
|
|
257
300
|
# @param [Google::Cloud::Datastore::Key] parent The parent Key of the entity.
|
|
@@ -260,8 +303,7 @@ module ActiveModel::Datastore
|
|
|
260
303
|
#
|
|
261
304
|
def find_entities(*ids_or_names, parent: nil)
|
|
262
305
|
ids_or_names = ids_or_names.flatten.compact.uniq
|
|
263
|
-
|
|
264
|
-
lookup_results.all.collect { |x| x }
|
|
306
|
+
find_all_entities(ids_or_names, parent).to_a
|
|
265
307
|
end
|
|
266
308
|
|
|
267
309
|
##
|
|
@@ -273,8 +315,11 @@ module ActiveModel::Datastore
|
|
|
273
315
|
# QueryResults are returned in batches, and the batch size is determined by the Datastore API.
|
|
274
316
|
# Batch size is not guaranteed. It will be affected by the size of the data being returned,
|
|
275
317
|
# and by other forces such as how distributed and/or consistent the data in Datastore is.
|
|
276
|
-
#
|
|
277
|
-
#
|
|
318
|
+
# With a limit, batches are collected using #next? and #next, retaining the final batch's
|
|
319
|
+
# end cursor for a full page. Short pages return a nil cursor. The whole page is retried as one
|
|
320
|
+
# read, rebuilding the query from the original options so a retry never reuses the reduced
|
|
321
|
+
# limit or cursor of a partially read page. Without a limit, from_entities iterates on an
|
|
322
|
+
# enumerator that retrieves all batches, retrying each request.
|
|
278
323
|
#
|
|
279
324
|
# Be sure to use as narrow a search criteria as possible. Please use with caution.
|
|
280
325
|
#
|
|
@@ -298,16 +343,26 @@ module ActiveModel::Datastore
|
|
|
298
343
|
# can be used to query for additional results.
|
|
299
344
|
#
|
|
300
345
|
def all(options = {})
|
|
301
|
-
|
|
302
|
-
|
|
303
|
-
|
|
304
|
-
CloudDatastore.dataset.run query
|
|
346
|
+
unless options[:limit]
|
|
347
|
+
query = build_query(options)
|
|
348
|
+
return from_entities(all_with_retries('run query') { CloudDatastore.dataset.run query })
|
|
305
349
|
end
|
|
306
|
-
|
|
307
|
-
|
|
308
|
-
|
|
350
|
+
|
|
351
|
+
# QueryResults#next lowers the shared query limit before each request, so every attempt
|
|
352
|
+
# builds a fresh query and page from the original options.
|
|
353
|
+
entities, next_cursor = retry_read('run query') do
|
|
354
|
+
query_results = CloudDatastore.dataset.run build_query(options)
|
|
355
|
+
page = []
|
|
356
|
+
loop do
|
|
357
|
+
page.concat(query_results)
|
|
358
|
+
break unless query_results.next?
|
|
359
|
+
|
|
360
|
+
query_results = query_results.next
|
|
361
|
+
raise missing_cursor_error('run query') unless query_results
|
|
362
|
+
end
|
|
363
|
+
[page, (query_results.cursor if page.size == options[:limit])]
|
|
309
364
|
end
|
|
310
|
-
from_entities(
|
|
365
|
+
[from_entities(entities.each), next_cursor]
|
|
311
366
|
end
|
|
312
367
|
|
|
313
368
|
##
|
|
@@ -332,8 +387,7 @@ module ActiveModel::Datastore
|
|
|
332
387
|
model_entity = from_entity(entity)
|
|
333
388
|
expects_array ? [model_entity].compact : model_entity
|
|
334
389
|
else
|
|
335
|
-
|
|
336
|
-
from_entities(lookup_results.all)
|
|
390
|
+
from_entities(find_all_entities(ids, parent))
|
|
337
391
|
end
|
|
338
392
|
end
|
|
339
393
|
|
|
@@ -354,17 +408,17 @@ module ActiveModel::Datastore
|
|
|
354
408
|
query.ancestor(args[:ancestor]) if args[:ancestor]
|
|
355
409
|
query.limit(1)
|
|
356
410
|
query.where(args.keys[0].to_s, '=', args.values[0])
|
|
357
|
-
query_results =
|
|
358
|
-
CloudDatastore.dataset.run query
|
|
359
|
-
end
|
|
411
|
+
query_results = retry_read('run query') { CloudDatastore.dataset.run query }
|
|
360
412
|
from_entity(query_results.first)
|
|
361
413
|
end
|
|
362
414
|
|
|
363
415
|
##
|
|
364
416
|
# Translates an Enumerator of Datastore::Entity objects to ActiveModel::Model objects.
|
|
365
417
|
#
|
|
366
|
-
#
|
|
367
|
-
#
|
|
418
|
+
# Within the gem, entities come from all_with_retries, which fetches each later batch of a
|
|
419
|
+
# lookup or unlimited query with retries, or from a page that `all(limit:)` has already read.
|
|
420
|
+
# Calling `all` on Dataset::LookupResults or Dataset::QueryResults also returns an enumerator,
|
|
421
|
+
# but fetches later batches without the gem's retries and timing.
|
|
368
422
|
#
|
|
369
423
|
# @param [Enumerator] entities An enumerator representing the datastore entities.
|
|
370
424
|
#
|
|
@@ -414,23 +468,46 @@ module ActiveModel::Datastore
|
|
|
414
468
|
query_options(query, options)
|
|
415
469
|
end
|
|
416
470
|
|
|
417
|
-
def retry_on_exception?(max_retry_count = 5, operation: nil, kind: name)
|
|
471
|
+
def retry_on_exception?(max_retry_count = 5, operation: nil, kind: name, &block)
|
|
472
|
+
retry_with_backoff(max_retry_count, operation: operation, kind: kind, &block)
|
|
473
|
+
rescue Google::Cloud::Error
|
|
474
|
+
# Only the error that ends the retries reaches here; it has already been logged.
|
|
475
|
+
false
|
|
476
|
+
end
|
|
477
|
+
|
|
478
|
+
def retry_on_exception(max_retry_count = 5, operation: nil, kind: name, &block)
|
|
479
|
+
retry_with_backoff(max_retry_count, operation: operation, kind: kind, &block)
|
|
480
|
+
end
|
|
481
|
+
|
|
482
|
+
def log_google_cloud_error
|
|
483
|
+
yield
|
|
484
|
+
rescue Google::Cloud::Error => e
|
|
485
|
+
puts "\e[33m[#{e.message.inspect}]\e[0m"
|
|
486
|
+
raise e
|
|
487
|
+
end
|
|
488
|
+
|
|
489
|
+
# **************** private ****************
|
|
490
|
+
|
|
491
|
+
##
|
|
492
|
+
# Shared by retry_on_exception and retry_on_exception?. Reading every batch of a call passes
|
|
493
|
+
# warn_slow: false and logs one slow-operation warning for the call instead of one per request.
|
|
494
|
+
#
|
|
495
|
+
def retry_with_backoff(max_retry_count, operation:, kind:, warn_slow: true)
|
|
418
496
|
retries = 0
|
|
419
497
|
sleep_time = 0.25
|
|
420
498
|
begin
|
|
421
499
|
started_at = Process.clock_gettime(Process::CLOCK_MONOTONIC) if operation
|
|
422
|
-
yield
|
|
500
|
+
result = yield
|
|
501
|
+
log_slow_operation(operation: operation, kind: kind, started_at: started_at,
|
|
502
|
+
warn_slow: warn_slow)
|
|
503
|
+
result
|
|
423
504
|
rescue Google::Cloud::Error => e
|
|
424
|
-
|
|
505
|
+
giving_up = e.is_a?(Google::Cloud::InvalidArgumentError) || retries >= max_retry_count
|
|
506
|
+
ending = "retrying in #{sleep_time}#{' s' if operation}"
|
|
507
|
+
ending = "giving up after #{retries + 1} attempts" if giving_up
|
|
508
|
+
log_retry_failure(e, ending, operation: operation, kind: kind, started_at: started_at)
|
|
509
|
+
raise if giving_up
|
|
425
510
|
|
|
426
|
-
if operation
|
|
427
|
-
elapsed_ms = ((Process.clock_gettime(Process::CLOCK_MONOTONIC) - started_at) * 1000).round
|
|
428
|
-
message = "\e[33mDatastore #{operation} failed for #{kind} after #{elapsed_ms} ms: " \
|
|
429
|
-
"#{e.message.inspect}; retrying in #{sleep_time} s\e[0m"
|
|
430
|
-
else
|
|
431
|
-
message = "\e[33mRescued exception #{e.message.inspect}, retrying in #{sleep_time}\e[0m"
|
|
432
|
-
end
|
|
433
|
-
ActiveModel::Datastore.logger ? ActiveModel::Datastore.logger.warn(message) : puts(message)
|
|
434
511
|
# 0.25, 0.5, 1, 2, and 4 second between retries.
|
|
435
512
|
sleep sleep_time
|
|
436
513
|
retries += 1
|
|
@@ -439,39 +516,69 @@ module ActiveModel::Datastore
|
|
|
439
516
|
end
|
|
440
517
|
end
|
|
441
518
|
|
|
442
|
-
def
|
|
443
|
-
|
|
444
|
-
|
|
445
|
-
begin
|
|
446
|
-
started_at = Process.clock_gettime(Process::CLOCK_MONOTONIC) if operation
|
|
447
|
-
yield
|
|
448
|
-
rescue Google::Cloud::Error => e
|
|
449
|
-
raise e if retries >= max_retry_count
|
|
519
|
+
def log_slow_operation(operation:, kind:, started_at: nil, elapsed: nil, warn_slow: true)
|
|
520
|
+
threshold = ActiveModel::Datastore.slow_operation_threshold
|
|
521
|
+
return unless warn_slow && operation && threshold
|
|
450
522
|
|
|
451
|
-
|
|
452
|
-
|
|
453
|
-
|
|
454
|
-
|
|
455
|
-
|
|
456
|
-
|
|
457
|
-
|
|
458
|
-
|
|
459
|
-
|
|
460
|
-
|
|
461
|
-
|
|
462
|
-
|
|
463
|
-
|
|
523
|
+
elapsed ||= Process.clock_gettime(Process::CLOCK_MONOTONIC) - started_at
|
|
524
|
+
return unless elapsed > threshold
|
|
525
|
+
|
|
526
|
+
message = "\e[33mDatastore #{operation} for #{kind} took #{(elapsed * 1000).round} ms\e[0m"
|
|
527
|
+
ActiveModel::Datastore.logger ? ActiveModel::Datastore.logger.warn(message) : puts(message)
|
|
528
|
+
end
|
|
529
|
+
|
|
530
|
+
def log_retry_failure(error, ending, operation:, kind:, started_at:)
|
|
531
|
+
if operation
|
|
532
|
+
elapsed_ms = ((Process.clock_gettime(Process::CLOCK_MONOTONIC) - started_at) * 1000).round
|
|
533
|
+
message = "\e[33mDatastore #{operation} failed for #{kind} after #{elapsed_ms} ms: " \
|
|
534
|
+
"#{error.message.inspect}; #{ending}\e[0m"
|
|
535
|
+
else
|
|
536
|
+
message = "\e[33mRescued exception #{error.message.inspect}, #{ending}\e[0m"
|
|
464
537
|
end
|
|
538
|
+
ActiveModel::Datastore.logger ? ActiveModel::Datastore.logger.warn(message) : puts(message)
|
|
465
539
|
end
|
|
466
540
|
|
|
467
|
-
def
|
|
468
|
-
|
|
469
|
-
rescue Google::Cloud::Error => e
|
|
470
|
-
puts "\e[33m[#{e.message.inspect}]\e[0m"
|
|
471
|
-
raise e
|
|
541
|
+
def retry_read(operation, &block)
|
|
542
|
+
retry_on_exception(ActiveModel::Datastore.read_retry_count, operation: operation, &block)
|
|
472
543
|
end
|
|
473
544
|
|
|
474
|
-
|
|
545
|
+
##
|
|
546
|
+
# Like QueryResults#all and LookupResults#all for the results of the given first request, but
|
|
547
|
+
# retries each batch request separately. Only for results whose #next repeats the same request:
|
|
548
|
+
# lookups and queries without a limit. A limited query lowers its shared limit in #next, so
|
|
549
|
+
# #all retries the whole page instead.
|
|
550
|
+
#
|
|
551
|
+
# One slow-operation warning covers the call: the total time of its successful requests,
|
|
552
|
+
# excluding failed attempts and retry backoff, as for a page of #all with a limit.
|
|
553
|
+
#
|
|
554
|
+
def all_with_retries(operation, &first_request)
|
|
555
|
+
retries = ActiveModel::Datastore.read_retry_count
|
|
556
|
+
Enumerator.new do |entities|
|
|
557
|
+
elapsed = 0.0
|
|
558
|
+
batch = nil
|
|
559
|
+
loop do
|
|
560
|
+
batch = retry_with_backoff(retries, operation: operation, kind: name, warn_slow: false) do
|
|
561
|
+
started_at = Process.clock_gettime(Process::CLOCK_MONOTONIC)
|
|
562
|
+
result = batch ? batch.next : first_request.call
|
|
563
|
+
elapsed += Process.clock_gettime(Process::CLOCK_MONOTONIC) - started_at
|
|
564
|
+
result
|
|
565
|
+
end
|
|
566
|
+
raise missing_cursor_error(operation) unless batch
|
|
567
|
+
|
|
568
|
+
batch.each { |entity| entities << entity }
|
|
569
|
+
break unless batch.next?
|
|
570
|
+
end
|
|
571
|
+
log_slow_operation(operation: operation, kind: name, elapsed: elapsed)
|
|
572
|
+
end
|
|
573
|
+
end
|
|
574
|
+
|
|
575
|
+
##
|
|
576
|
+
# QueryResults#next returns nil when a batch reports more results without an end cursor.
|
|
577
|
+
# Datastore always sends one, so fail loudly rather than return incomplete results.
|
|
578
|
+
#
|
|
579
|
+
def missing_cursor_error(operation)
|
|
580
|
+
Error.new("Datastore #{operation} for #{name} reported more results without an end cursor")
|
|
581
|
+
end
|
|
475
582
|
|
|
476
583
|
def query_options(query, options)
|
|
477
584
|
query.ancestor(options[:ancestor]) if options[:ancestor]
|
|
@@ -513,15 +620,17 @@ module ActiveModel::Datastore
|
|
|
513
620
|
|
|
514
621
|
##
|
|
515
622
|
# Finds entities by keys using the provided array items. Results provided by the
|
|
516
|
-
# dataset `find_all` is a Dataset::LookupResults object
|
|
623
|
+
# dataset `find_all` is a Dataset::LookupResults object, which may defer some keys
|
|
624
|
+
# to later requests.
|
|
517
625
|
#
|
|
518
626
|
# @param [Array<Integer>, Array<String>] ids_or_names An array of ids or names.
|
|
519
627
|
#
|
|
628
|
+
# @return [Enumerator] the found entities, including deferred ones.
|
|
520
629
|
#
|
|
521
630
|
def find_all_entities(ids_or_names, parent)
|
|
522
631
|
keys = ids_or_names.map { |id| CloudDatastore.dataset.key name, id }
|
|
523
632
|
keys.map { |key| key.parent = parent } if parent.present?
|
|
524
|
-
|
|
633
|
+
all_with_retries('find all') { CloudDatastore.dataset.find_all keys }
|
|
525
634
|
end
|
|
526
635
|
|
|
527
636
|
def build_model(entity)
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: activemodel-datastore
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.11.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Bryce McLean
|
|
@@ -43,14 +43,20 @@ dependencies:
|
|
|
43
43
|
requirements:
|
|
44
44
|
- - "~>"
|
|
45
45
|
- !ruby/object:Gem::Version
|
|
46
|
-
version: '2.
|
|
46
|
+
version: '2.3'
|
|
47
|
+
- - ">="
|
|
48
|
+
- !ruby/object:Gem::Version
|
|
49
|
+
version: 2.3.1
|
|
47
50
|
type: :runtime
|
|
48
51
|
prerelease: false
|
|
49
52
|
version_requirements: !ruby/object:Gem::Requirement
|
|
50
53
|
requirements:
|
|
51
54
|
- - "~>"
|
|
52
55
|
- !ruby/object:Gem::Version
|
|
53
|
-
version: '2.
|
|
56
|
+
version: '2.3'
|
|
57
|
+
- - ">="
|
|
58
|
+
- !ruby/object:Gem::Version
|
|
59
|
+
version: 2.3.1
|
|
54
60
|
- !ruby/object:Gem::Dependency
|
|
55
61
|
name: rake
|
|
56
62
|
requirement: !ruby/object:Gem::Requirement
|
|
@@ -191,7 +197,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
|
|
|
191
197
|
requirements:
|
|
192
198
|
- - ">="
|
|
193
199
|
- !ruby/object:Gem::Version
|
|
194
|
-
version: 2.
|
|
200
|
+
version: '2.5'
|
|
195
201
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
|
196
202
|
requirements:
|
|
197
203
|
- - ">="
|