activemodel-datastore 0.10.0 → 0.11.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 62ccc081e2437c2c1ca324b12a78f888cb14bf5fb0d6cabe8ebee0c1089f3807
-  data.tar.gz: 5def43f01da73999c34f0f2a6bc66ead4249eda5df08281dd552da42856b0fea
+  metadata.gz: ad3eda8122e638f1b739f22844f8f116154ce792563e757f944e1fd04c44145d
+  data.tar.gz: '09316b9ed555cc216f51692ca3d8ec944921149ac3a624d74c598a2a04dea2ef'
 SHA512:
-  metadata.gz: 03f469254125d0dc7184395103cdea93648f0551e5a1289721ec5fe2dc382d0ccdf906087734875cfc8e79c815f51be766ab917d76cb2fdd9cde69e5858e1308
-  data.tar.gz: 71473fab63f1e02a4472db2a3c828492d1a23fddde05ca45aa275e9037a63cf252dc53140e93ac9bf5bc354a5fa38d549fec5d2c64565cb9f7fc0e7bfd05c896
+  metadata.gz: b83379af680f03b96c148997a59181a9a34403f2c537672fda824899c939dd23f755045399901be38518bbb1cbadfa7ea68c0e1e375c43b9f27b986cbdac29e7
+  data.tar.gz: 3a4145461ae6b6ba321a8b8136e86b665b6a73447ab6a6e42dbc7d2fad68cbe28001c873d6dd2547ffdfe15bfb0d77b75b4bed30612e10ee20def1a512afe5ed

data/CHANGELOG.md

@@ -1,3 +1,20 @@
+### 0.11.0 / 2026-10-14
+
+Upgrade notes:
+* requires `google-cloud-datastore` 2.3.1 or later; older versions will no longer resolve
+* `DATASTORE_NETWORK_TIMEOUT` is no longer read; it never took effect, so removing it from your environment changes nothing (see below for configuring per-RPC deadlines)
+* slow-operation warnings are enabled by default, so successful Datastore calls taking over 5 seconds now log at WARN level; set `ActiveModel::Datastore.slow_operation_threshold = nil` to keep the previous behavior
+
+Changes:
+* fix `all(limit:)` returning a nil cursor for full pages split across Datastore response batches; use the final batch's end cursor
+* retry `all(limit:)` as a whole page, so a failure in any later Datastore response batch gets the configured `read_retry_count` retries and logging, and `slow_operation_threshold` times every batch of the page; each attempt rebuilds the query from the original options and discards rows from the failed attempt
+* retry each later batch of queries without a limit and each deferred lookup in `find` / `find_entities` with the configured `read_retry_count` and retry logging; one `slow_operation_threshold` warning covers the total time of the call's successful requests
+* raise `ActiveModel::Datastore::Error` instead of `NoMethodError` or `TypeError` if Datastore reports more query results without an end cursor
+* require Ruby 2.5 or newer and `google-cloud-datastore` 2.3.1 or later within 2.x for correct query limits across batches
+* warn when a successful Datastore attempt exceeds `ActiveModel::Datastore.slow_operation_threshold` (enabled by default at 5 seconds; `nil` or a blank string disables slow-success warnings)
+* add opt-in ID preallocation via `ActiveModel::Datastore.preallocate_ids` (disabled by default) to prevent duplicate entities when retrying a save; allocation is retried at most twice
+* remove `DATASTORE_NETWORK_TIMEOUT`, which never took effect because the Datastore client's per-RPC 60-second deadlines override the client-wide timeout; set `Google::Cloud::Datastore::V1::Datastore::Client.configure { |c| c.rpcs.<rpc>.timeout = ... }` instead
+
 ### 0.10.0 / 2026-09-22
 * stop retrying invalid argument errors
 * make read retry counts configurable independently of writes

data/README.md

@@ -89,6 +89,28 @@ configure it during application initialization:
 ActiveModel::Datastore.logger = MyApplication.logger
 ```
 
+Successful attempts taking more than five seconds also log a warning, such as
+`Datastore save for Report took 6000 ms`. This measures each attempt separately, excluding retry
+backoff. Both retry helpers use this logging when an `operation:` label is provided.
+
+`all(limit:)` retries and times the complete page, including all response batches. Each retry starts
+from the original query options and discards rows from the failed attempt. A retry reads batches
+that already succeeded again, and because cursors don't preserve a snapshot, the retried page can
+include changes made between attempts. Queries without a limit and multi-key lookups (`find` with
+several IDs, `find_entities`) retry each request separately, so a retry reads only the batch that
+failed. Their slow-operation warning covers the whole call: the total time of its successful
+requests, excluding failed attempts and retry backoff.
+
+Configure the threshold in seconds during initialization:
+
+```ruby
+ActiveModel::Datastore.slow_operation_threshold = 2
+```
+
+Numbers and numeric strings (such as `"2"` from an environment variable) are accepted. Set it to
+`nil` or a blank string to disable slow-success warnings. Invalid values raise when the setting
+is assigned. Failure logging and retry behavior are unchanged.
+
 To change the gem's read retry count (default: `5`), configure it during initialization:
 
 ```ruby
@@ -98,6 +120,40 @@ ActiveModel::Datastore.read_retry_count = 2
 Set it to `0` to disable read retries. Write retry counts are unchanged. Invalid argument errors
 are never retried.
 
+ID preallocation is disabled by default, preserving the existing save behavior. To enable it for
+models saved through `save` or `save!`, configure it during application initialization:
+
+```ruby
+ActiveModel::Datastore.preallocate_ids = true
+```
+
+The setter uses `ActiveModel::Type::Boolean`, so environment-variable strings such as `"false"`,
+`"0"`, and `"off"` disable preallocation. Unset (`nil`) and empty values also leave it disabled.
+
+When enabled, a new entity without an ID gets a complete key before the save retry loop. Every
+attempt in that loop uses the same key, preventing duplicate entities if a commit succeeds but its
+response fails. This adds one allocation RPC per create under normal conditions. Existing numeric
+IDs and named keys require no allocation, and parent paths are preserved.
+
+Allocation writes nothing, so it is retried at most twice before giving up, leaving more of the
+request's time for the save. If allocation still fails, `save` returns `false` and `save!` raises
+`ActiveModel::Datastore::EntityNotSavedError` without attempting the write.
+
+Set the option to `false` to disable preallocation. It does not cover `update` on a model without an
+ID, direct calls to `CloudDatastore.dataset.save`, or deduplication across separate save calls or
+requests.
+
+Each Datastore RPC has its own 60-second default deadline in the underlying client, which takes
+precedence over the client-wide `timeout` option. Applications with a request timeout (for example,
+Heroku's 30-second router timeout or Rack::Timeout) should give allocation a short deadline so a
+stalled allocation cannot use the time the save needs:
+
+```ruby
+Google::Cloud::Datastore::V1::Datastore::Client.configure do |config|
+  config.rpcs.allocate_ids.timeout = 3
+end
+```
+
 There is an example Puma config file [here](https://github.com/Agrimatics/activemodel-datastore/blob/main/test/support/datastore_example_rails_app/config/puma.rb).
  
 ## <a name="model"></a>Model Example

data/lib/active_model/datastore/connection.rb

@@ -31,8 +31,7 @@ module CloudDatastore
   end
 
   def self.dataset
-    timeout = ENV.fetch('DATASTORE_NETWORK_TIMEOUT', 15).to_i
-    @dataset ||= Google::Cloud.datastore(timeout: timeout)
+    @dataset ||= Google::Cloud.datastore
   end
 
   def self.reset_dataset

data/lib/active_model/datastore/version.rb

@@ -1,5 +1,5 @@
 module ActiveModel
   module Datastore
-    VERSION = '0.10.0'
+    VERSION = '0.11.0'
   end
 end

data/lib/active_model/datastore.rb

@@ -119,6 +119,9 @@ module ActiveModel::Datastore
   include ActiveModel::Datastore::PropertyValues
   include ActiveModel::Datastore::TrackChanges
 
+  # Allocation writes nothing, so a failure can give up quickly and leave time for the save.
+  ALLOCATE_RETRY_COUNT = 2
+
   def self.logger
     @logger || (Rails.logger if defined?(Rails) == 'constant')
   end
@@ -127,6 +130,15 @@ module ActiveModel::Datastore
     @logger = logger
   end
 
+  def self.slow_operation_threshold
+    defined?(@slow_operation_threshold) ? @slow_operation_threshold : 5
+  end
+
+  def self.slow_operation_threshold=(seconds)
+    seconds = nil if seconds.is_a?(String) && seconds.strip.empty?
+    @slow_operation_threshold = seconds.nil? ? nil : Float(seconds)
+  end
+
   def self.read_retry_count
     @read_retry_count || 5
   end
@@ -135,9 +147,18 @@ module ActiveModel::Datastore
     @read_retry_count = count
   end
 
+  def self.preallocate_ids
+    @preallocate_ids || false
+  end
+
+  def self.preallocate_ids=(enabled)
+    @preallocate_ids = ActiveModel::Type::Boolean.new.cast(enabled)
+  end
+
   included do
     private_class_method :query_options, :query_sort, :query_property_filter, :find_all_entities,
-                         :retry_read, :log_retry_failure
+                         :retry_read, :log_retry_failure, :log_slow_operation, :all_with_retries,
+                         :missing_cursor_error, :retry_with_backoff
     define_model_callbacks :save, :update, :destroy
     attr_accessor :id, :parent_key_id, :entity_property_values
   end
@@ -219,7 +240,8 @@ module ActiveModel::Datastore
 
     run_callbacks :save do
       entity = build_entity(parent)
-      success = self.class.retry_on_exception?(operation: 'save') do
+      success = preallocate_entity_id(entity)
+      success &&= self.class.retry_on_exception?(operation: 'save') do
         CloudDatastore.dataset.save entity
       end
       self.id = entity.key.id if success
@@ -228,6 +250,19 @@ module ActiveModel::Datastore
     end
   end
 
+  def preallocate_entity_id(entity)
+    return true unless ActiveModel::Datastore.preallocate_ids && entity.key.incomplete?
+
+    # A commit can succeed even when its response fails. Reuse one key for every save attempt.
+    key = self.class.retry_on_exception?(ALLOCATE_RETRY_COUNT, operation: 'allocate id') do
+      CloudDatastore.dataset.allocate_ids(entity.key, 1).first ||
+        raise(Google::Cloud::Error, 'allocate_ids returned no key')
+    end
+    return false unless key
+
+    entity.key = key
+  end
+
   # Methods defined here will be class methods when 'include ActiveModel::Datastore'.
   module ClassMethods
     ##
@@ -257,10 +292,9 @@ module ActiveModel::Datastore
     # additional values. LookupResults are returned in batches, and the batch size is
     # determined by the Datastore API. Batch size is not guaranteed. It will be affected
     # by the size of the data being returned, and by other forces such as how distributed
-    # and/or consistent the data in Datastore is. Calling `all` on the LookupResults retrieves
-    # all results by repeatedly loading #next until #next? returns false. The `all` method
-    # returns an enumerator unless passed a block. We iterate on the enumerator to return
-    # the model entity objects.
+    # and/or consistent the data in Datastore is. Deferred keys are retrieved by repeatedly
+    # loading #next until #next? returns false, retrying each request. We iterate on the
+    # resulting enumerator to return the model entity objects.
     #
     # @param [Integer, String] ids_or_names One or more ids to retrieve.
     # @param [Google::Cloud::Datastore::Key] parent The parent Key of the entity.
@@ -269,8 +303,7 @@ module ActiveModel::Datastore
     #
     def find_entities(*ids_or_names, parent: nil)
       ids_or_names = ids_or_names.flatten.compact.uniq
-      lookup_results = find_all_entities(ids_or_names, parent)
-      lookup_results.all.collect { |x| x }
+      find_all_entities(ids_or_names, parent).to_a
     end
 
     ##
@@ -282,8 +315,11 @@ module ActiveModel::Datastore
     # QueryResults are returned in batches, and the batch size is determined by the Datastore API.
     # Batch size is not guaranteed. It will be affected by the size of the data being returned,
     # and by other forces such as how distributed and/or consistent the data in Datastore is.
-    # Calling `all` on the QueryResults retrieves all results by repeatedly loading #next until
-    # #next? returns false. The `all` method returns an enumerator which from_entities iterates on.
+    # With a limit, batches are collected using #next? and #next, retaining the final batch's
+    # end cursor for a full page. Short pages return a nil cursor. The whole page is retried as one
+    # read, rebuilding the query from the original options so a retry never reuses the reduced
+    # limit or cursor of a partially read page. Without a limit, from_entities iterates on an
+    # enumerator that retrieves all batches, retrying each request.
     #
     # Be sure to use as narrow a search criteria as possible. Please use with caution.
     #
@@ -307,14 +343,26 @@ module ActiveModel::Datastore
     #   can be used to query for additional results.
     #
     def all(options = {})
-      next_cursor = nil
-      query = build_query(options)
-      query_results = retry_read('run query') { CloudDatastore.dataset.run query }
-      if options[:limit]
-        next_cursor = query_results.cursor if query_results.size == options[:limit]
-        return from_entities(query_results.all), next_cursor
+      unless options[:limit]
+        query = build_query(options)
+        return from_entities(all_with_retries('run query') { CloudDatastore.dataset.run query })
       end
-      from_entities(query_results.all)
+
+      # QueryResults#next lowers the shared query limit before each request, so every attempt
+      # builds a fresh query and page from the original options.
+      entities, next_cursor = retry_read('run query') do
+        query_results = CloudDatastore.dataset.run build_query(options)
+        page = []
+        loop do
+          page.concat(query_results)
+          break unless query_results.next?
+
+          query_results = query_results.next
+          raise missing_cursor_error('run query') unless query_results
+        end
+        [page, (query_results.cursor if page.size == options[:limit])]
+      end
+      [from_entities(entities.each), next_cursor]
     end
 
     ##
@@ -339,8 +387,7 @@ module ActiveModel::Datastore
         model_entity = from_entity(entity)
         expects_array ? [model_entity].compact : model_entity
       else
-        lookup_results = find_all_entities(ids, parent)
-        from_entities(lookup_results.all)
+        from_entities(find_all_entities(ids, parent))
       end
     end
 
@@ -368,8 +415,10 @@ module ActiveModel::Datastore
     ##
     # Translates an Enumerator of Datastore::Entity objects to ActiveModel::Model objects.
     #
-    # Results provided by the dataset `find_all` or `run query` will be a Dataset::LookupResults or
-    # Dataset::QueryResults object. Invoking `all` on those objects returns an enumerator.
+    # Within the gem, entities come from all_with_retries, which fetches each later batch of a
+    # lookup or unlimited query with retries, or from a page that `all(limit:)` has already read.
+    # Calling `all` on Dataset::LookupResults or Dataset::QueryResults also returns an enumerator,
+    # but fetches later batches without the gem's retries and timing.
     #
     # @param [Enumerator] entities An enumerator representing the datastore entities.
     #
@@ -419,33 +468,39 @@ module ActiveModel::Datastore
       query_options(query, options)
     end
 
-    def retry_on_exception?(max_retry_count = 5, operation: nil, kind: name)
-      retries = 0
-      sleep_time = 0.25
-      begin
-        started_at = Process.clock_gettime(Process::CLOCK_MONOTONIC) if operation
-        yield
-      rescue Google::Cloud::Error => e
-        giving_up = e.is_a?(Google::Cloud::InvalidArgumentError) || retries >= max_retry_count
-        ending = "retrying in #{sleep_time}#{' s' if operation}"
-        ending = "giving up after #{retries + 1} attempts" if giving_up
-        log_retry_failure(e, ending, operation: operation, kind: kind, started_at: started_at)
-        return false if giving_up
+    def retry_on_exception?(max_retry_count = 5, operation: nil, kind: name, &block)
+      retry_with_backoff(max_retry_count, operation: operation, kind: kind, &block)
+    rescue Google::Cloud::Error
+      # Only the error that ends the retries reaches here; it has already been logged.
+      false
+    end
 
-        # 0.25, 0.5, 1, 2, and 4 second between retries.
-        sleep sleep_time
-        retries += 1
-        sleep_time *= 2
-        retry
-      end
+    def retry_on_exception(max_retry_count = 5, operation: nil, kind: name, &block)
+      retry_with_backoff(max_retry_count, operation: operation, kind: kind, &block)
     end
 
-    def retry_on_exception(max_retry_count = 5, operation: nil, kind: name)
+    def log_google_cloud_error
+      yield
+    rescue Google::Cloud::Error => e
+      puts "\e[33m[#{e.message.inspect}]\e[0m"
+      raise e
+    end
+
+    # **************** private ****************
+
+    ##
+    # Shared by retry_on_exception and retry_on_exception?. Reading every batch of a call passes
+    # warn_slow: false and logs one slow-operation warning for the call instead of one per request.
+    #
+    def retry_with_backoff(max_retry_count, operation:, kind:, warn_slow: true)
       retries = 0
       sleep_time = 0.25
       begin
         started_at = Process.clock_gettime(Process::CLOCK_MONOTONIC) if operation
-        yield
+        result = yield
+        log_slow_operation(operation: operation, kind: kind, started_at: started_at,
+                           warn_slow: warn_slow)
+        result
       rescue Google::Cloud::Error => e
         giving_up = e.is_a?(Google::Cloud::InvalidArgumentError) || retries >= max_retry_count
         ending = "retrying in #{sleep_time}#{' s' if operation}"
@@ -461,14 +516,16 @@ module ActiveModel::Datastore
       end
     end
 
-    def log_google_cloud_error
-      yield
-    rescue Google::Cloud::Error => e
-      puts "\e[33m[#{e.message.inspect}]\e[0m"
-      raise e
-    end
+    def log_slow_operation(operation:, kind:, started_at: nil, elapsed: nil, warn_slow: true)
+      threshold = ActiveModel::Datastore.slow_operation_threshold
+      return unless warn_slow && operation && threshold
 
-    # **************** private ****************
+      elapsed ||= Process.clock_gettime(Process::CLOCK_MONOTONIC) - started_at
+      return unless elapsed > threshold
+
+      message = "\e[33mDatastore #{operation} for #{kind} took #{(elapsed * 1000).round} ms\e[0m"
+      ActiveModel::Datastore.logger ? ActiveModel::Datastore.logger.warn(message) : puts(message)
+    end
 
     def log_retry_failure(error, ending, operation:, kind:, started_at:)
       if operation
@@ -485,6 +542,44 @@ module ActiveModel::Datastore
       retry_on_exception(ActiveModel::Datastore.read_retry_count, operation: operation, &block)
     end
 
+    ##
+    # Like QueryResults#all and LookupResults#all for the results of the given first request, but
+    # retries each batch request separately. Only for results whose #next repeats the same request:
+    # lookups and queries without a limit. A limited query lowers its shared limit in #next, so
+    # #all retries the whole page instead.
+    #
+    # One slow-operation warning covers the call: the total time of its successful requests,
+    # excluding failed attempts and retry backoff, as for a page of #all with a limit.
+    #
+    def all_with_retries(operation, &first_request)
+      retries = ActiveModel::Datastore.read_retry_count
+      Enumerator.new do |entities|
+        elapsed = 0.0
+        batch = nil
+        loop do
+          batch = retry_with_backoff(retries, operation: operation, kind: name, warn_slow: false) do
+            started_at = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+            result = batch ? batch.next : first_request.call
+            elapsed += Process.clock_gettime(Process::CLOCK_MONOTONIC) - started_at
+            result
+          end
+          raise missing_cursor_error(operation) unless batch
+
+          batch.each { |entity| entities << entity }
+          break unless batch.next?
+        end
+        log_slow_operation(operation: operation, kind: name, elapsed: elapsed)
+      end
+    end
+
+    ##
+    # QueryResults#next returns nil when a batch reports more results without an end cursor.
+    # Datastore always sends one, so fail loudly rather than return incomplete results.
+    #
+    def missing_cursor_error(operation)
+      Error.new("Datastore #{operation} for #{name} reported more results without an end cursor")
+    end
+
     def query_options(query, options)
       query.ancestor(options[:ancestor]) if options[:ancestor]
       query.cursor(options[:cursor]) if options[:cursor]
@@ -525,15 +620,17 @@ module ActiveModel::Datastore
 
     ##
     # Finds entities by keys using the provided array items. Results provided by the
-    # dataset `find_all` is a Dataset::LookupResults object.
+    # dataset `find_all` is a Dataset::LookupResults object, which may defer some keys
+    # to later requests.
     #
     # @param [Array<Integer>, Array<String>] ids_or_names An array of ids or names.
     #
+    # @return [Enumerator] the found entities, including deferred ones.
     #
     def find_all_entities(ids_or_names, parent)
       keys = ids_or_names.map { |id| CloudDatastore.dataset.key name, id }
       keys.map { |key| key.parent = parent } if parent.present?
-      retry_read('find all') { CloudDatastore.dataset.find_all keys }
+      all_with_retries('find all') { CloudDatastore.dataset.find_all keys }
     end
 
     def build_model(entity)

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: activemodel-datastore
 version: !ruby/object:Gem::Version
-  version: 0.10.0
+  version: 0.11.0
 platform: ruby
 authors:
 - Bryce McLean
@@ -43,14 +43,20 @@ dependencies:
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '2.0'
+        version: '2.3'
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 2.3.1
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '2.0'
+        version: '2.3'
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 2.3.1
 - !ruby/object:Gem::Dependency
   name: rake
   requirement: !ruby/object:Gem::Requirement
@@ -191,7 +197,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
-      version: 2.2.2
+      version: '2.5'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="