rails-pg-adapter 0.1.4 → 0.1.5

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 648d36ac2c7b9990f850dc54483824206aba63db49dc17d9379c49ce5ec88e1d
-  data.tar.gz: bd0dd4cfd41cc749eb797d95da5d1f8261c586f04da5608068220694cbfdbbc6
+  metadata.gz: 432740419d2edda4a33caf34ba5efa2af778456763db8864c632b3d813f1953e
+  data.tar.gz: 60a026760c20847ace00bf181878c0039eface04fa30ff51227e677be88f95c0
 SHA512:
-  metadata.gz: 41c0930afcf109ae0402e36ade421227f3b245358a06ccb79a46bf5da0c5749623b996d401fcf60fe6b00d1ad79a3ed72ffdfc21aaa30f9a5b994536884aef2a
-  data.tar.gz: c96c0592c584ce6c3555bf87a880dcea495830f2b3deb1fcf03bc87f036a5b3b46afe00a8395c8b39757b0f7db159975562cd586cbe60c8ec02635103d4e03c4
+  metadata.gz: 5fb1aac2ec7330a599cc9005d339fab13613b409cf37152a003826b760c02b8b4060b0134d50799be4db7d77e6db4165b6ef99d0b1df0399133749f4746c5a25
+  data.tar.gz: 0450d73b3e3330da51c01d64a922828da6b32fe1d500d2c719ee40bd5dbbc2ac2c40d87cc93bc3a8a53676c3476f61180684a6bf05cf301d876f393eab6b5bee

data/CHANGELOG.md

@@ -1,4 +1,8 @@
-## [0.1.3] - 2023-04-04
+## [0.1.5] - 2023-04-19
+
+- Retry queries when not in transaction
+
+## [0.1.4] - 2023-04-04
 
 - Rescue and recover from `ActiveRecord::ConnectionNotEstablished`
 

data/CODE_OF_CONDUCT.md

@@ -10,21 +10,21 @@ We pledge to act and interact in ways that contribute to an open, welcoming, div
 
 Examples of behavior that contributes to a positive environment for our community include:
 
-* Demonstrating empathy and kindness toward other people
-* Being respectful of differing opinions, viewpoints, and experiences
-* Giving and gracefully accepting constructive feedback
-* Accepting responsibility and apologizing to those affected by our mistakes, and learning from the experience
-* Focusing on what is best not just for us as individuals, but for the overall community
+- Demonstrating empathy and kindness toward other people
+- Being respectful of differing opinions, viewpoints, and experiences
+- Giving and gracefully accepting constructive feedback
+- Accepting responsibility and apologizing to those affected by our mistakes, and learning from the experience
+- Focusing on what is best not just for us as individuals, but for the overall community
 
 Examples of unacceptable behavior include:
 
-* The use of sexualized language or imagery, and sexual attention or
+- The use of sexualized language or imagery, and sexual attention or
   advances of any kind
-* Trolling, insulting or derogatory comments, and personal or political attacks
-* Public or private harassment
-* Publishing others' private information, such as a physical or email
+- Trolling, insulting or derogatory comments, and personal or political attacks
+- Public or private harassment
+- Publishing others' private information, such as a physical or email
   address, without their explicit permission
-* Other conduct which could reasonably be considered inappropriate in a
+- Other conduct which could reasonably be considered inappropriate in a
   professional setting
 
 ## Enforcement Responsibilities
@@ -67,7 +67,7 @@ Community leaders will follow these Community Impact Guidelines in determining t
 
 ### 4. Permanent Ban
 
-**Community Impact**: Demonstrating a pattern of violation of community standards, including sustained inappropriate behavior,  harassment of an individual, or aggression toward or disparagement of classes of individuals.
+**Community Impact**: Demonstrating a pattern of violation of community standards, including sustained inappropriate behavior, harassment of an individual, or aggression toward or disparagement of classes of individuals.
 
 **Consequence**: A permanent ban from any sort of public interaction within the community.
 

data/README.md

@@ -2,16 +2,6 @@
 
 This project allows you to monkey patch `ActiveRecord` (PostgreSQL) and auto-heal applications in production when PostgreSQL database fails over or when a cached column (in `ActiveRecord` schema cache) is removed from the database from a migration in another process.
 
-## How does it work
-
-During a database failover in production, the `ActiveRecord` connection pool can become exhausted as queries are made against the database during the failover process. This can leave the `ActiveRecord` connection pools with stale or bad connections, even after the database has successfully recovered. Recovering from this issue usually requires a rolling restart of the application processes or containers.
-
-`RailsPgAdapter` addresses this problem by resetting the connection pool and re-raises the original exception from an `ActiveRecord` monkey patch. This allows the application to auto-heal from stale connections on its own (after database recovery) when performing queries for a new request, without requiring manual intervention.
-
-Another issue with `ActiveRecord` queries is `PG::UndefinedColumn`, which occurs when an `ActiveRecord` model includes a `SELECT` query with the name of a column that has been dropped from a Rails migration. This can happen even if the column isn't being referenced anywhere in the code. It occurs when a model is using `ignored_columns`, which prompts `ActiveRecord` to perform a dedicated lookup of the allowed columns in a select, such as `SELECT "users".name, "users".template_id...."`, instead of `SELECT "users".*`. When a column like `template_id` is dropped, PostgreSQL throws an undefined column error, which is bubbled up by `ActiveRecord` into `PG::UndefinedColumn`. Recovering from this issue also usually requires a rolling restart of the application processes or containers.
-
-`RailsPgAdapter` solves this second issue by resetting the `ActiveRecord` schema cache and memoized model column information when it detects a `PG::UndefinedColumn` raised from a monkey patch. Resetting the column information forces `ActiveRecord` to refresh its schema cache by loading the table information from the database and no longer reference the dropped column for new queries, without requiring manual intervention.
-
 ## Installation
 
 Install the gem and add to the application's Gemfile by executing:
@@ -27,7 +17,7 @@ If bundler is not being used to manage dependencies, install the gem by executin
 ### Auto healing connections when PostgreSQL database fails over
 
 ```ruby
-# config/initializer/rails_pg_adapter.rb
+# config/initializers/rails_pg_adapter.rb
 
 RailsPgAdapter.configure do |c|
   c.add_failover_patch = true
@@ -36,10 +26,28 @@ end
 
 This will add the monkey patch which resets the `ActiveRecord` connections in the connection pool when the database fails over. The patch will reset the connection and re-raise the error each time it detects that an exception related to a database failover is detected.
 
+### Retrying queries
+
+When the database is failing you can retry queries that are not in a transaction. The gem will perform a back off retry in establishing the connection.
+Once the back off is reached and no connection is found, it will bubble up the exception. Otherwise, the query will be retried with a new connection.
+
+It is an opt-in functionality. You can supply your own back off figures for retries (in seconds) as following:
+
+```ruby
+# config/initializers/rails_pg_adapter.rb
+
+RailsPgAdapter.configure do |c|
+  c.add_failover_patch = true
+  c.reconnect_with_backoff = [0.5, 1, 2, 4, 8, 16] # seconds
+...
+end
+
+```
+
 ### Refresh model column information on the fly after an existing column is dropped
 
 ```ruby
-# config/initializer/rails_pg_adapter.rb
+# config/initializers/rails_pg_adapter.rb
 
 RailsPgAdapter.configure do |c|
   c.add_reset_column_information_patch = true
@@ -48,6 +56,16 @@ end
 
 This will clear the `ActiveRecord` schema cache and reset the `ActiveRecord` column information memoized on the model. The patch will reset the relevant information and re-raise the error each time it detects that an exception related to a dropped column is raised.
 
+## How does it work
+
+During a database failover in production, the `ActiveRecord` connection pool can become exhausted as queries are made against the database during the failover process. This can leave the `ActiveRecord` connection pools with stale or bad connections, even after the database has successfully recovered. Recovering from this issue usually requires a rolling restart of the application processes or containers.
+
+`RailsPgAdapter` addresses this problem by resetting the connection pool and re-raises the original exception from an `ActiveRecord` monkey patch. This allows the application to auto-heal from stale connections on its own (after database recovery) when performing queries for a new request, without requiring manual intervention.
+
+Another issue with `ActiveRecord` queries is `PG::UndefinedColumn`, which occurs when an `ActiveRecord` model includes a `SELECT` query with the name of a column that has been dropped from a Rails migration. This can happen even if the column isn't being referenced anywhere in the code. It occurs when a model is using `ignored_columns`, which prompts `ActiveRecord` to perform a dedicated lookup of the allowed columns in a select, such as `SELECT "users".name, "users".template_id...."`, instead of `SELECT "users".*`. When a column like `template_id` is dropped, PostgreSQL throws an undefined column error, which is bubbled up by `ActiveRecord` into `PG::UndefinedColumn`. Recovering from this issue also usually requires a rolling restart of the application processes or containers.
+
+`RailsPgAdapter` solves this second issue by resetting the `ActiveRecord` schema cache and memoized model column information when it detects a `PG::UndefinedColumn` raised from a monkey patch. Resetting the column information forces `ActiveRecord` to refresh its schema cache by loading the table information from the database and no longer reference the dropped column for new queries, without requiring manual intervention.
+
 ## Development
 
 - Install ruby 3.0

data/lib/rails_pg_adapter/configuration.rb

@@ -2,11 +2,12 @@
 
 module RailsPgAdapter
   class Configuration
-    attr_accessor :add_failover_patch, :add_reset_column_information_patch
+    attr_accessor :add_failover_patch, :add_reset_column_information_patch, :reconnect_with_backoff
 
     def initialize(attrs)
       self.add_failover_patch = attrs[:add_failover_patch]
       self.add_reset_column_information_patch = attrs[:add_reset_column_information_patch]
+      self.reconnect_with_backoff = attrs[:reconnect_with_backoff]
     end
   end
 
@@ -14,6 +15,7 @@ module RailsPgAdapter
     @configuration ||= Configuration.new({
       add_failover_patch: false,
       add_reset_column_information_patch: false,
+      reconnect_with_backoff: [],
     })
   end
 
@@ -25,6 +27,10 @@ module RailsPgAdapter
     RailsPgAdapter.configuration.add_failover_patch || false
   end
 
+  def self.reconnect_with_backoff?
+    !RailsPgAdapter.configuration.reconnect_with_backoff.empty?
+  end
+
   def self.reset_column_information_patch?
     RailsPgAdapter.configuration.add_reset_column_information_patch || false
   end
@@ -33,6 +39,7 @@ module RailsPgAdapter
     @configuration = Configuration.new({
       add_failover_patch: false,
       add_reset_column_information_patch: false,
+      reconnect_with_backoff: [],
     })
   end
 end

data/lib/rails_pg_adapter/patch.rb

@@ -12,6 +12,7 @@ module RailsPgAdapter
       "PG::ConnectionBad",
       "the database system is starting up",
       "connection is closed",
+      "could not connect",
     ].freeze
     CONNECTION_ERROR_RE = /#{CONNECTION_ERROR.map { |w| Regexp.escape(w) }.join("|")}/.freeze
 
@@ -23,20 +24,37 @@ module RailsPgAdapter
     def exec_cache(*args)
       super(*args)
     rescue ::ActiveRecord::StatementInvalid, ::ActiveRecord::ConnectionNotEstablished => e
-      handle_error(e) || raise
+      raise unless supported_errors?(e)
+
+      try_reconnect?(e) ? retry : handle_error(e)
     end
 
     def exec_no_cache(*args)
       super(*args)
     rescue ::ActiveRecord::StatementInvalid, ::ActiveRecord::ConnectionNotEstablished => e
-      handle_error(e) || raise
+      raise unless supported_errors?(e)
+
+      try_reconnect?(e) ? retry : handle_error(e)
+    end
+
+    def try_reconnect?(e)
+      return false if in_transaction?
+      return false unless failover_error?(e.message)
+      return false unless RailsPgAdapter.reconnect_with_backoff?
+
+      begin
+        reconnect!
+        true
+      rescue ::ActiveRecord::ConnectionNotEstablished
+        false
+      end
     end
 
     def handle_error(e)
       if failover_error?(e.message) && RailsPgAdapter.failover_patch?
         warn("clearing connections due to #{e} - #{e.message}")
         disconnect_and_remove_conn!
-        raise
+        raise(e)
       end
 
       return unless missing_column_error?(e.message) && RailsPgAdapter.reset_column_information_patch?
@@ -70,7 +88,41 @@ module RailsPgAdapter
       return if Rails.logger.nil?
       ::Rails.logger.warn("[RailsPgAdapter::Patch] #{msg}")
     end
+
+    def supported_errors?(e)
+      return true if failover_error?(e.message) && RailsPgAdapter.failover_patch?
+      if missing_column_error?(e.message) && RailsPgAdapter.reset_column_information_patch?
+        return true
+      end
+      false
+    end
   end
 end
 
 ActiveRecord::ConnectionAdapters::PostgreSQLAdapter.prepend(RailsPgAdapter::Patch)
+
+# Override new client connection to bake in retries
+module ActiveRecord
+  module ConnectionAdapters
+    class PostgreSQLAdapter
+      class << self
+        old_new_client_method = instance_method(:new_client)
+
+        define_method(:new_client) do |args|
+          sleep_times = RailsPgAdapter.configuration.reconnect_with_backoff.dup
+          begin
+            old_new_client_method.bind(self).call(args)
+          rescue ::ActiveRecord::ConnectionNotEstablished => e
+            raise(e) unless RailsPgAdapter.failover_patch? && RailsPgAdapter.reconnect_with_backoff?
+
+            sleep_time = sleep_times.shift
+            raise unless sleep_time
+            warn( "Could not establish a connection from new_client, retrying again in #{sleep_time} sec.")
+            sleep(sleep_time)
+            retry
+          end
+        end
+      end
+    end
+  end
+end

data/lib/rails_pg_adapter/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module RailsPgAdapter
-  VERSION = "0.1.4"
+  VERSION = "0.1.5"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: rails-pg-adapter
 version: !ruby/object:Gem::Version
-  version: 0.1.4
+  version: 0.1.5
 platform: ruby
 authors:
 - Tines Engineering
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2023-04-14 00:00:00.000000000 Z
+date: 2023-04-19 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rails