mongo_ha 1.12.4 → 2.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: c98aae542b09bfdcb6a8cf4af14cd8eacb3f29fe
-  data.tar.gz: 6213123e29784aba1116471a878ed9bcdb9572da
+  metadata.gz: c07613ccc0db614856311ea723d0bb980bd53113
+  data.tar.gz: f449fd924fd37daac5bfd82d3d13950422d895d9
 SHA512:
-  metadata.gz: c3f18e4efa572627a4efb3c8da4d7a67e18f21040049c1e68c4cdb89837148303a51dc255740f65f1352c577237609490a07e99a64d34de8c8630102f265465e
-  data.tar.gz: f3afdbe8d28adfcc6cfbf5a02da254eaf83cd89ec4b54bd6f2261fc1fcb5f692783514cba1e8f0ca351ac4727f71c1882f122ce5cb13a63353c2b038773c64b0
+  metadata.gz: 4e65723c5ba6bb1ac48c2095c353ee6ca048d49016e92db7329f8820a568b95a63949151f0831a3a3379f477b9673d1c9f999749f2e4d623e56a5a3f2651abbe
+  data.tar.gz: 83e9ccb94dc81194df0c87b8558def4f5d17efff66297071c43ab3ed4b48c52d019a4175921b87418fcd44e42d9543b282c765d3e88bcbb8e91c8b1e5b192e8a

data/LICENSE.txt

@@ -0,0 +1,202 @@
+Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "{}"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright {yyyy} {name of copyright owner}
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+

data/README.md

@@ -4,8 +4,7 @@ High availability for the mongo ruby driver. Automatic reconnects and recovery w
 
 ## Status
 
-Production Ready: Used every day in an enterprise environment across
-remote data centers.
+Production Ready: Used every day in an enterprise environment across remote data centers.
 
 ## Overview
 
@@ -23,49 +22,9 @@ In the event of a connection failure, only one thread will attempt to re-establi
 connectivity to the Mongo server(s). This is to prevent swamping the mongo
 servers with reconnect attempts.
 
-Retries are initially performed quickly in case it is a brief network issue
-and then backs off to give the replica-set time to elect a new master.
+Supports Ruby Mongo driver V1 and V2
 
-Supports Ruby Mongo driver V1.x.x
-
-`mongo_ha` transparently supports `MongoMapper` since it uses the mongo ruby driver
-that is patched by loading this gem. Earlier versions of Mongoid will also benefit
-from `mongo_ha`, the latest version of Mongoid uses Moped that should be avoided
-since it is due to be replaced by the Ruby Mongo driver V2 when it is released.
-
-Mongo Router processes will often return a connection failure on their side
-as an OperationFailure. This code will also retry automatically when the router
-has errors talking to a sharded cluster.
-
-## Mongo Cursors
-
-Any operations that return a cursor need to be handled in your own code
-since the retry cannot be handled transparently.
-For example: `find` returns a cursor, whereas `find_one` is handled because
-it returns the data returned rather than a cursor
-
-Example
-
-```ruby
-# Wrap existing cursor based calls with a retry on connection failure block
-results_collection.retry_on_connection_failure do
-  results_collection.find({}, sort: '_id', timeout: false) do |cursor|
-    cursor.each do |record|
-      puts "Record: #{record.inspect}"
-    end
-  end
-end
-```
-
-### Note
-
-In the above example the block will be repeated from the _beginning_ of the
-collection should a connection failure occur. Without appropriate handling it
-is possible to read the same records twice.
-
-If the collection cannot be processed twice, it may be better to just let the
-`Mongo::ConnectionFailure` flow up into the application for it to deal with at
-a higher level.
+`mongo_ha` transparently supports `MongoMapper` and `Mongoid` which use the mongo ruby driver.
 
 ## Installation
 
@@ -85,70 +44,6 @@ If you are also using SemanticLogger, place `mongo_ha` below `semantic_logger`
 and/or `rails_semantic_logger` in the `Gemfile`. This way it will create a logger
 just for `Mongo::MongoClient` to improve the log output during connection recovery.
 
-## Configuration
-
-mongo_ha adds several new configuration options to fine tune the reconnect behavior
-for any environment.
-
-Sample mongo.yml:
-
-```yaml
-default_options: &default_options
-  :w:                           1
-  :pool_size:                   5
-  :pool_timeout:                5
-  :connect_timeout:             5
-  :reconnect_attempts:          53
-  :reconnect_retry_seconds:     0.1
-  :reconnect_retry_multiplier:  2
-  :reconnect_max_retry_seconds: 5
-
-development: &development
-  uri: mongodb://localhost:27017/development
-  options:
-    <<: *default_options
-
-test:
-  uri: mongodb://localhost:27017/test
-  options:
-    <<: *default_options
-
-# Sample Production Settings
-production:
-  uri: mongodb://mongo1.site.com:27017,mongo2.site.com:27017/production
-  options:
-    <<: *default_options
-    :pool_size:    50
-    :pool_timeout: 5
-```
-
-The following options can be specified in the Mongo configuration options
-to tune the retry intervals during a connection failure
-
-### :reconnect_attempts
-
-* Number of times to attempt to reconnect.
-* Default: 53
-
-### :reconnect_retry_seconds
-
-* Initial delay before retrying
-* Default: 0.1
-
-### :reconnect_retry_multiplier
-
-* Multiply delay by this number with each retry to prevent overwhelming the server
-* Default: 2
-
-### :reconnect_max_retry_seconds
-
-* Maximum number of seconds to wait before retrying again
-* Default: 5
-
-Using the above default values, will result in retry connects at the following intervals
-
-   0.1 0.2 0.4 0.8 1.6 3.2 5 5 5 5  ....
-
 ## Testing
 
 There is really only one place to test something like `mongo_ha` and that is in
@@ -157,16 +52,3 @@ This gem was created and tested with MongoDB running in an
 enterprise production environment with hundreds of connections to Mongo servers
 in remote data centers across a WAN. It adds high availability to standalone
 MongoDB servers, replica-sets, and sharded clusters.
-
-## Issues
-
-If the following output appears after adding the above connection options:
-
-```shell
-reconnect_attempts is not a valid option for Mongo::MongoClient
-reconnect_retry_seconds is not a valid option for Mongo::MongoClient
-reconnect_retry_multiplier is not a valid option for Mongo::MongoClient
-reconnect_max_retry_seconds is not a valid option for Mongo::MongoClient
-```
-
-Then the `mongo_ha` gem was not loaded prior to connecting to Mongo

data/lib/mongo_ha/operation_failure.rb

@@ -0,0 +1,34 @@
+require 'mongo/error/operation_failure'
+
+module Mongo
+  class Error
+    class OperationFailure
+      WRITE_RETRY_MESSAGES = [
+        'no master',
+        'not master',
+        'could not contact primary',
+        'Not primary'
+      ]
+
+      remove_const :RETRY_MESSAGES
+      RETRY_MESSAGES = WRITE_RETRY_MESSAGES + [
+        'transport error',
+        'socket exception',
+        "can't connect",
+        'connect failed',
+        'error querying',
+        'could not get last error',
+        'connection attempt failed',
+        'interrupted at shutdown',
+        'unknown replica set',
+        'dbclient error communicating with server'
+      ]
+
+      def write_retryable?
+        WRITE_RETRY_MESSAGES.any? { |m| message.include?(m) }
+      end
+
+    end
+  end
+end
+

data/lib/mongo_ha/retryable.rb

@@ -0,0 +1,62 @@
+require 'mongo/retryable'
+
+module Mongo
+  module Retryable
+
+    def read_with_retry(attempt = 0, &block)
+      begin
+        block.call
+      rescue Error::SocketError, Error::SocketTimeoutError => e
+        retry_operation(e, &block)
+      rescue Error::OperationFailure => e
+        # TODO: Non sharded, retryable due to Replicaset primary change
+
+        if cluster.sharded? && e.retryable?
+          if attempt < cluster.max_read_retries
+            # We don't scan the cluster in this case as Mongos always returns
+            # ready after a ping no matter what the state behind it is.
+            sleep(cluster.read_retry_interval)
+            read_with_retry(attempt + 1, &block)
+          else
+            raise e
+          end
+        else
+          raise e
+        end
+      end
+    end
+
+    def read_with_one_retry(&block)
+      block.call
+    rescue Error::SocketError, Error::SocketTimeoutError => e
+      Logger.logger.warn "Single retry due to: #{e.class.name} #{e.message}"
+      block.call
+    end
+
+    def write_with_retry(&block)
+      begin
+        block.call
+      rescue Error::SocketError => e
+        # During a master move in a replica-set the master closes existing client connections.
+        # Note: Small possibility the write occurs twice.
+        retry_operation(e, &block)
+      rescue Error::OperationFailure => e
+        if e.write_retryable?
+          retry_operation(e, &block)
+        else
+          raise e
+        end
+      end
+    end
+
+    private
+
+    # Log a warning on retry to prevent appearance of "hanging" during a failover.
+    def retry_operation(e, &block)
+      Logger.logger.warn "Retry due to: #{e.class.name} #{e.message}"
+      cluster.scan!
+      block.call
+    end
+
+  end
+end

data/lib/mongo_ha/version.rb

@@ -1,3 +1,3 @@
 module MongoHA #:nodoc
-  VERSION = '1.12.4'
+  VERSION = '2.0.0'
 end

data/lib/mongo_ha.rb

@@ -1,28 +1,3 @@
 require 'mongo'
-require 'mongo_ha/version'
-require 'mongo_ha/mongo_client'
-
-# Add in retry methods
-Mongo::MongoClient.send(:include, MongoHA::MongoClient::InstanceMethods)
-
-# Wrap critical Mongo methods with retry_on_connection_failure
-{
-  # Most calls use a cursor under the covers to return the result
-  # If the primary is lost and it connects to a different server an expired cursor exception is raised
-  Mongo::Cursor     => [:refresh],
-
-  # These methods do not use a Cursor
-  Mongo::Collection => [:insert, :remove, :update]
-}.each_pair do |klass, methods|
-  methods.each do |method|
-    original_method = "#{method}_original".to_sym
-    klass.send(:alias_method, original_method, method)
-    klass.send(:define_method, method) do |*args|
-      @connection.retry_on_connection_failure { send(original_method, *args) }
-    end
-  end
-end
-
-# Drop the max ping time to a more respectable time. Assuming it is in ms.
-Mongo::Pool.send(:remove_const, :MAX_PING_TIME)
-Mongo::Pool::MAX_PING_TIME = 5_000
+require 'mongo_ha/operation_failure'
+require 'mongo_ha/retryable'

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: mongo_ha
 version: !ruby/object:Gem::Version
-  version: 1.12.4
+  version: 2.0.0
 platform: ruby
 authors:
 - Reid Morrison
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2016-11-07 00:00:00.000000000 Z
+date: 2016-11-30 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: mongo
@@ -16,14 +16,14 @@ dependencies:
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '1.10'
+        version: '2.0'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '1.10'
+        version: '2.0'
 description: Automatic reconnects and recovery when replica-set changes, or connections
   are lost, with transparent recovery
 email:
@@ -32,12 +32,12 @@ executables: []
 extensions: []
 extra_rdoc_files: []
 files:
+- LICENSE.txt
 - README.md
-- Rakefile
 - lib/mongo_ha.rb
-- lib/mongo_ha/mongo_client.rb
+- lib/mongo_ha/operation_failure.rb
+- lib/mongo_ha/retryable.rb
 - lib/mongo_ha/version.rb
-- test/readme.md
 homepage: https://github.com/reidmorrison/mongo_ha
 licenses:
 - Apache-2.0
@@ -58,9 +58,8 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
 requirements: []
 rubyforge_project: 
-rubygems_version: 2.5.1
+rubygems_version: 2.6.8
 signing_key: 
 specification_version: 4
 summary: High availability for the mongo ruby driver
-test_files:
-- test/readme.md
+test_files: []

data/Rakefile

@@ -1,28 +0,0 @@
-require 'rake/clean'
-require 'rake/testtask'
-
-$LOAD_PATH.unshift File.expand_path("../lib", __FILE__)
-require 'mongo_ha/version'
-
-task :gem do
-  system "gem build mongo_ha.gemspec"
-end
-
-task :publish => :gem do
-  system "git tag -a v#{MongoHA::VERSION} -m 'Tagging #{MongoHA::VERSION}'"
-  system "git push --tags"
-  system "gem push mongo_ha-#{MongoHA::VERSION}.gem"
-  system "rm mongo_ha-#{MongoHA::VERSION}.gem"
-end
-
-desc "Run Test Suite"
-task :test do
-  Rake::TestTask.new(:functional) do |t|
-    t.test_files = FileList['test/*_test.rb']
-    t.verbose    = true
-  end
-
-  Rake::Task['functional'].invoke
-end
-
-task :default => :test

data/lib/mongo_ha/mongo_client.rb

@@ -1,205 +0,0 @@
-require 'mongo'
-module MongoHA
-  module MongoClient
-    CONNECTION_RETRY_OPTS    = [:reconnect_attempts, :reconnect_retry_seconds, :reconnect_retry_multiplier, :reconnect_max_retry_seconds]
-
-    # The following errors occur when mongos cannot connect to the shard
-    # They require a retry to resolve them
-    # This list was created through painful experience. Add any new ones as they are discovered
-    #   9001: socket exception
-    #   Operation failed with the following exception: Unknown error - Connection reset by peer:Unknown error - Connection reset by peer
-    #   DBClientBase::findOne: transport error
-    #   : db assertion failure
-    #   8002: 8002 all servers down!
-    #   Operation failed with the following exception: stream closed
-    #   Operation failed with the following exception: Bad file descriptor - Bad file descriptor:Bad file descriptor - Bad file descriptor
-    #   Failed to connect to primary node.
-    #   10009: ReplicaSetMonitor no master found for set: mdbb
-    OPERATION_FAILURE_ERRORS = [
-      'socket exception',
-      'Connection reset by peer',
-      'transport error',
-      'db assertion failure',
-      '8002',
-      'stream closed',
-      'Bad file descriptor',
-      'Failed to connect',
-      '10009',
-      'no master found',
-      'not master',
-      'Timed out waiting on socket',
-      "didn't get writeback",
-      'interrupted at shutdown'
-    ]
-
-    module InstanceMethods
-      # Add retry logic to MongoClient
-      def self.included(base)
-        base.class_eval do
-          # Give MongoClient a class-specific logger if SemanticLogger V2.12 or above is available
-          # to give better logging information during a connection recovery scenario
-          if defined?(SemanticLogger::DebugAsTraceLogger)
-            # Map Debug level calls to trace to reduce log file clutter
-            @@logger = SemanticLogger::DebugAsTraceLogger.new(self)
-
-            def self.logger
-              @@logger
-            end
-
-            def logger
-              self.class.logger
-            end
-          end
-
-          alias_method :valid_opts_original, :valid_opts
-          alias_method :setup_original, :setup
-
-          attr_accessor *CONNECTION_RETRY_OPTS
-
-          # Prevent multiple threads from trying to reconnect at the same time during
-          # connection failures
-          @@failover_mutex = Mutex.new
-
-          private
-
-          def valid_opts(*args)
-            valid_opts_original(*args) + CONNECTION_RETRY_OPTS
-          end
-
-          def setup(opts)
-            self.reconnect_attempts          = (opts.delete(:reconnect_attempts) || 53).to_i
-            self.reconnect_retry_seconds     = (opts.delete(:reconnect_retry_seconds) || 0.1).to_f
-            self.reconnect_retry_multiplier  = (opts.delete(:reconnect_retry_multiplier) || 2).to_f
-            self.reconnect_max_retry_seconds = (opts.delete(:reconnect_max_retry_seconds) || 5).to_f
-            setup_original(opts)
-          end
-
-        end
-      end
-
-      # Retry the supplied block when a Mongo::ConnectionFailure occurs
-      #
-      # Note: Check for Duplicate Key on inserts
-      #
-      # Returns the result of the block
-      #
-      # Example:
-      #   connection.retry_on_connection_failure { |retried| connection.ping }
-      def retry_on_connection_failure(&block)
-        raise 'Missing mandatory block parameter on call to Mongo::Connection#retry_on_connection_failure' unless block
-        # No need to double retry calls
-        return block.call(false) if Thread.current[:mongo_ha_active?]
-        retried        = false
-        mongos_retries = 0
-        begin
-          Thread.current[:mongo_ha_active?] = true
-          result                      = block.call(retried)
-          retried                     = false
-          result
-        rescue Mongo::ConnectionFailure => exc
-          # Retry if reconnected, but only once to prevent an infinite loop
-          logger.warn "Connection Failure: '#{exc.message}' [#{exc.error_code}]"
-          if !retried && _reconnect
-            retried = true
-            retry
-          end
-          raise exc
-        rescue Mongo::AuthenticationError => exc
-          # Retry once due to rare failures during authentication against MongoDB V3 servers
-          logger.warn "Authentication Failure: '#{exc.message}' [#{exc.error_code}]"
-          if !retried && _reconnect
-            retried = true
-            retry
-          end
-          raise exc
-        rescue Mongo::OperationTimeout => exc
-          logger.warn "OperationTimeout: #{exc.message}"
-          if !retried && _reconnect
-            retried = true
-            retry
-          end
-          raise exc
-        rescue Mongo::OperationFailure => exc
-          # Workaround not master issue. Disconnect connection when we get a not master
-          # error message. Master checks for an exact match on "not master", whereas
-          # it sometimes gets: "not master and slaveok=false"
-          if exc.result
-            error = exc.result['err'] || exc.result['errmsg']
-            close if error && error.include?('not master')
-          end
-
-          # These get returned when connected to a local mongos router when it in turn
-          # has connection failures talking to the remote shards. All we do is retry the same operation
-          # since it's connections to multiple remote shards may have failed.
-          # Disconnecting the current connection will not help since it is just to the mongos router
-          # First make sure it is connected to the mongos router
-          raise exc unless (OPERATION_FAILURE_ERRORS.any? { |err| exc.message.include?(err) }) || (exc.message.strip == ':')
-
-          mongos_retries += 1
-          if mongos_retries <= 60
-            retried = true
-            Kernel.sleep(0.5)
-            logger.warn "[#{primary.inspect}] Router Connection Failure. Retry ##{mongos_retries}. Exc: '#{exc.message}' [#{exc.error_code}]"
-            retry
-          end
-          raise exc
-        ensure
-          Thread.current[:mongo_ha_active?] = false
-        end
-      end
-
-      private
-
-      # Call this method whenever a Mongo::ConnectionFailure Exception
-      # has been raised to re-establish the connection
-      #
-      # This method is thread-safe and ensure that only one thread at a time
-      # per connection will attempt to re-establish the connection
-      #
-      # Returns whether the connection is connected again
-      def _reconnect
-        logger.debug 'Going to reconnect'
-
-        # Prevent other threads from invoking reconnect logic at the same time
-        @@failover_mutex.synchronize do
-          # Another thread may have already failed over the connection by the
-          # time this threads gets in
-          begin
-            ping
-          rescue Mongo::ConnectionFailure
-            # Connection still not available, run code below
-          end
-
-          if active?
-            logger.info "Connected to: #{primary.inspect}"
-            return true
-          end
-
-          if reconnect_attempts > 0
-            # Wait for other threads to finish working on their sockets
-            retries       = 1
-            retry_seconds = reconnect_retry_seconds
-            begin
-              logger.warn "Connection unavailable. Waiting: #{retry_seconds} seconds before retrying"
-              sleep retry_seconds
-              ping
-            rescue Mongo::ConnectionFailure => exc
-              if retries < reconnect_attempts
-                retries       += 1
-                retry_seconds *= reconnect_retry_multiplier
-                retry_seconds = reconnect_max_retry_seconds if retry_seconds > reconnect_max_retry_seconds
-                retry
-              end
-
-              logger.error "Auto-reconnect giving up after #{retries} reconnect attempts"
-              raise exc
-            end
-            logger.info "Successfully reconnected to: #{primary.inspect}"
-          end
-          connected?
-        end
-      end
-
-    end
-  end
-end

data/test/readme.md

@@ -1,48 +0,0 @@
-## Testing
-
-Unfortunately the only way to properly test mongo_ha is to startup a console with connections
-active and to stop / restart the mongo servers in the replicaset as follows:
-
-
-#### Run the following code in a console
-
-```ruby
-collection = Cache::Identity.database['test']
-collection.drop
-threads = 5.times.collect do |i|
-  Thread.new do
-    100.times do |j|
-      1_000.times do |k|
-        collection.insert(_id: "#{i}-#{j}-#{k}")
-        collection.find_one(_id: "#{i}-#{j}-#{k}")
-        puts("#{i}-#{j}-#{k}") if k % 1000 == 0
-      end
-      puts "#{i}-#{j} pausing"
-      sleep 5
-    end
-    puts "#{i} Complete"
-  end
-end
-```
-
-#### Steps
-
-While running the above code in the console
-
-* Stop 1 slave server
-
-Nothing should appear in the logs and everything should process fine
-
-* Stop another slave
-
-The logs should show retries
-
-* Start up one of the 2 slaves that were stopped
-
-The processing should resume successfully
-
-#### To stop the test
-
-```ruby
-threads.each(&:kill)
-```