created_id 1.1.0 → 1.1.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: afdabbea09a01fe7c621831292070fdd0138f39ef7efce693745765c5f468fdd
-  data.tar.gz: 6f48cde24728a3f4b92b3c1a003e092f92d4f781e22a68350bb504a210e35f81
+  metadata.gz: 41989b3e85e6a81b89589bbc05a24f9e7bfcba4c47a974964ec5dfa96b6268f0
+  data.tar.gz: 1fb19ba275fbfff80f75bbc9be0242597f1997989922420eb525e1a93b84cf2c
 SHA512:
-  metadata.gz: 500a45fdf791907a43d856c4315988343f69ecd571b294cdab0b5f83c347949fafba13e706e0bcfb2951342a96529a2b2d2a62ffc02982269c25f4b88b9c7003
-  data.tar.gz: 5f6ab7875355ffb9f15495c0d0d7ecc83fd01f1a726c734274bc922511a9acdaadcf831b71cc3e0fa3612581a450e4f561d99c0e0b5a87b3b675b99f4087bc69
+  metadata.gz: 7ef459aad9deec426b20ae18867380b880126730351d677f8b7e15d8f07900566d5fc7fc779845e9b047131584dbc2b59f1d8854bdb8f52d7f38fe5697fdf11e
+  data.tar.gz: b8016654eeb70c0b750ebefcc497aedacad8fe49f994c716fcce278ab61041c6ff0f9f0b140ab437760cd85aad25c8927e48e24394a49529fd31bd6291346f9b

data/CHANGELOG.md

@@ -4,6 +4,24 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## 1.1.2
+
+### Fixed
+
+- Fixed data corruption when re-running `index_ids_for` on an hour that had already been indexed. The range calculation was using the hour's own stored range as a bound, which excluded the hour's true minimum id (and could exclude the maximum) and caused records to be silently dropped from `created_between`, `created_after`, and `created_before` query results. Recalculating an indexed hour is now safe.
+- The upper bound used when calculating an hour's id range now always skips the immediately following hour, since adjacent hours' id ranges can legitimately overlap when ids are slightly out of order near an hour boundary.
+- Fixed a race condition in `index_ids_for` where two processes indexing the same class and hour concurrently could raise `ActiveRecord::RecordNotUnique`. The save is now retried so the losing process updates the existing row.
+- Fixed misspelled `ArgumentError` constant that caused a `NameError` when including `CreatedId` in a non-ActiveRecord class.
+
+### Added
+
+- Creating a record with an explicitly backdated `created_at` in an hour whose id range has already been indexed now raises `CreatedId::CreatedAtChangedError` and rolls back the insert, since the record's id would fall outside the stored range and be invisible to range queries.
+
+## 1.1.1
+
+### Fixed
+
+- Fixed issue with finding id ranges that prevented the query from using the indexed ids.
 ## 1.1.0
 
 ### Changed

data/README.md

@@ -111,12 +111,14 @@ Task.where(user_id: 1000).created_before(7.days.ago)
 Task.created_between(25.hours.ago, 24.hours.ago)
 ```
 
-You'll then need to set up a periodic task to store the id ranges for your models. For each model that includes `CreatedId`, you need to run the `index_ids_for` once per hour. This task should be run shortly after the top of the hour.
+You'll then need to set up a periodic task to store the id ranges for your models. For each model that includes `CreatedId`, you need to run the `index_ids_for` once per hour. This task should be run shortly after the top of the hour, but leave a few minutes of buffer — longer than your longest-running write transaction. A transaction that starts at the end of an hour but commits after the indexer has scanned that hour would otherwise not be visible to the indexer, and if that row happens to be the hour's minimum or maximum id, it would fall outside the stored range and be missed by queries.
 
 ```ruby
 Task.index_ids_for(1.hour.ago)
 ```
 
+Re-running `index_ids_for` for an hour that has already been indexed is safe and recalculates the stored range from the current data. As a self-healing measure, you can periodically re-index the last few hours to pick up any rows that were committed after their hour was first indexed.
+
 Finally, you'll need to run a script to calculate the id ranges for all of your existing data.
 
 ```ruby
@@ -130,7 +132,9 @@ end
 
 If an ID range is missing for a specific hour, your queries will still function, but with a broader range of IDs. You can recalculate missing ranges at any time to improve efficiency.
 
-There is an additional requirement for using this gem that you do not change the `created_at` value after a row is inserted since this can mess up the assumption about the correlation between ids and `created_at` timestamps. An error will be thrown if you try to change a record's timestamp after the id range has been created. The query logic can handle small variations between id order and timestamp order (i.e. if id 1000 has a timestamp a few seconds after id 1001).
+There is an additional requirement for using this gem that you do not change the `created_at` value after a row is inserted since this can mess up the assumption about the correlation between ids and `created_at` timestamps. An error will be thrown if you try to change a record's timestamp after the id range has been created, or if you insert a new record with a backdated `created_at` in an hour whose id range has already been indexed. The query logic can handle small variations between id order and timestamp order (i.e. if id 1000 has a timestamp a few seconds after id 1001).
+
+Note that these guards can only check hours that have already been indexed. Changing `created_at` across hours (or inserting backdated rows) into an hour that has *not* been indexed can still leave the row outside the id bounds inferred from neighboring indexed hours. If you need to backfill or correct historical data, re-run `index_ids_for` for the affected hours afterwards.
 
 ## Installation
 

data/VERSION

@@ -1 +1 @@
-1.1.0
+1.1.2

data/lib/created_id/id_range.rb

@@ -68,18 +68,21 @@ module CreatedId
         klass = klass.base_class
         hour = CreatedId.coerce_hour(time)
         next_hour = hour + 3600
-        prev_hour = hour - 3600
 
         finder = klass.unscoped.where(created_at: (hour...next_hour))
 
-        prev_id = CreatedId::IdRange.min_id(self, prev_hour)
-        if prev_id
-          finder = finder.where(klass.arel_table[:id].gt(prev_id)) if prev_id > 0
-
-          next_id = CreatedId::IdRange.min_id(self, next_hour + 3600)
-          if next_id
-            finder = finder.where(klass.arel_table[:id].lt(next_id)) if next_id > prev_id
-          end
+        # Lower bound: the min id of the latest indexed hour strictly before this one.
+        # A record in this hour can never have an id at or below the previous indexed
+        # hour's minimum, even when ids are out of order near the boundary.
+        prev_id = for_class(klass).where(hour: nil...hour).order(hour: :desc).first&.min_id
+        finder = finder.where(klass.arel_table[:id].gt(prev_id)) if prev_id && prev_id > 0
+
+        # Upper bound: the min id of the earliest indexed hour at least two hours after
+        # this one. The immediately following hour is skipped because its id range may
+        # legitimately overlap this hour's when ids are out of order near the boundary.
+        next_id = for_class(klass).created_after(next_hour + 3600).order(hour: :asc).first&.min_id
+        if next_id && (prev_id.nil? || next_id > prev_id)
+          finder = finder.where(klass.arel_table[:id].lt(next_id))
         end
 
         [finder.minimum(:id), finder.maximum(:id)]
@@ -93,10 +96,18 @@ module CreatedId
       # @param max_id [Integer] The maximum id for the class created in the given hour.
       # @return [void]
       def save_created_id(klass, time, min_id, max_id)
-        record = find_or_initialize_by(class_name: klass.base_class.name, hour: CreatedId.coerce_hour(time))
-        record.min_id = min_id
-        record.max_id = max_id
-        record.save!
+        retried = false
+        begin
+          record = find_or_initialize_by(class_name: klass.base_class.name, hour: CreatedId.coerce_hour(time))
+          record.min_id = min_id
+          record.max_id = max_id
+          record.save!
+        rescue ActiveRecord::RecordNotUnique
+          # A concurrent process indexed the same hour first; retry to update its record.
+          raise if retried
+          retried = true
+          retry
+        end
       end
     end
 

data/lib/created_id.rb

@@ -21,13 +21,14 @@ module CreatedId
 
   included do
     unless defined?(ActiveRecord) && self < ActiveRecord::Base
-      raise ArgmentError, "CreatedId can only be included in ActiveRecord models"
+      raise ArgumentError, "CreatedId can only be included in ActiveRecord models"
     end
 
     scope :created_after, ->(time) { created_between(time, nil) }
     scope :created_before, ->(time) { created_between(nil, time) }
 
     before_save :verify_created_at_created_id!, if: :created_at_changed?
+    after_create :verify_created_id_on_backdated_create!
   end
 
   class_methods do
@@ -71,11 +72,30 @@ module CreatedId
     # This is the normal case where created at is set to the current time on insert.
     return if id.nil? && created_at_was.nil?
 
-    new_hour = CreatedId.coerce_hour(created_at || Time.now)
-    range = CreatedId::IdRange.for_class(self.class).find_by(hour: new_hour)
+    verify_id_in_indexed_range!("created_at cannot be changed outside of the range of the created_ids for that time period")
+  end
+
+  # Verify that a record created with an explicit created_at in an hour that has already
+  # been indexed received an id within the stored range. The id is not known until after
+  # the insert, so this check must run after create; raising here rolls back the insert.
+  # Records created in the current hour skip the check since that hour cannot have been
+  # indexed yet.
+  #
+  # @return [void]
+  # @raise [CreatedId::CreatedAtChangedError] If the record's id is outside the range of the created_ids for its hour.
+  def verify_created_id_on_backdated_create!
+    return if created_at.nil?
+    return if CreatedId.coerce_hour(created_at) == CreatedId.coerce_hour(Time.now)
+
+    verify_id_in_indexed_range!("created_at cannot be set to an hour whose id range has already been indexed")
+  end
+
+  def verify_id_in_indexed_range!(message)
+    hour = CreatedId.coerce_hour(created_at || Time.now)
+    range = CreatedId::IdRange.for_class(self.class).find_by(hour: hour)
 
     if range && (id < range.min_id || id > range.max_id)
-      raise CreatedAtChangedError, "created_at cannot be changed outside of the range of the created_ids for that time period"
+      raise CreatedAtChangedError, message
     end
   end
 end

metadata

@@ -1,14 +1,13 @@
 --- !ruby/object:Gem::Specification
 name: created_id
 version: !ruby/object:Gem::Version
-  version: 1.1.0
+  version: 1.1.2
 platform: ruby
 authors:
 - Brian Durand
-autorequire:
 bindir: bin
 cert_chain: []
-date: 2024-11-13 00:00:00.000000000 Z
+date: 1980-01-02 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activerecord
@@ -38,7 +37,6 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
-description:
 email:
 - bbdurand@gmail.com
 executables: []
@@ -62,7 +60,6 @@ metadata:
   homepage_uri: https://github.com/bdurand/created_id
   source_code_uri: https://github.com/bdurand/created_id
   changelog_uri: https://github.com/bdurand/created_id/blob/main/CHANGELOG.md
-post_install_message:
 rdoc_options: []
 require_paths:
 - lib
@@ -77,8 +74,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.4.10
-signing_key:
+rubygems_version: 4.0.3
 specification_version: 4
 summary: Optimize ActiveRecord queries for filtering large tables on the created_at
   column by pre-computing id ranges.