sidekiq-transaction_guard 1.0.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 294e37857447f017a08970a3d0430042c69e0afee645d25ce3f2350ce93fb9e6
+  data.tar.gz: ee1c16813ca499c9c146fa3db6dcc4f7d3f1c4b8417769ca1fa9635cdcd793c0
+SHA512:
+  metadata.gz: b533961d40cc75425032fbb39ecf51a216bca82cf50f3e5d45d2b6c0a628b7da38b03e4c1bc5387711d4a4ddb36a120626826156d42020fd768ff6adb0e16069
+  data.tar.gz: 520048075eb69133f17233a5d8ef037e8d8f6ce4c886be5c1f398b966154bb7c0e27ed1a6e4c4694e17d46abd1e69a0651b5f72cd44c13520a2b8d07bbb05907

data/CHANGELOG.md

@@ -0,0 +1,3 @@
+# 1.0.0
+
+* Initial release

data/MIT_LICENSE.txt

@@ -0,0 +1,20 @@
+Copyright (c) 2018 Brian Durand, Winston Durand
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,158 @@
+# Sidekiq::TransactionGuard
+
+[![Build Status](https://travis-ci.com/bdurand/sidekiq-transaction_guard.svg?branch=master)](https://travis-ci.com/bdurand/sidekiq-transaction_guard)
+[![Maintainability](https://api.codeclimate.com/v1/badges/17bbf5cb6eda022028fe/maintainability)](https://codeclimate.com/github/bdurand/sidekiq-transaction_guard/maintainability)
+
+You should never call a Sidekiq worker that relies on the state of the database from within a database transaction. You will end up with a race condition since the worker could kick off before the transaction is actually written to the database. This gem can be used to highlight where your code may be scheduling workers in an indeterminate state.
+
+## The Problem
+
+Consider this case:
+
+```ruby
+class Post < ActiveRecord::Base
+  # BAD: DO NOT DO THIS
+  after_create do
+    PostCreatedWorker.perform_async(id)
+  end
+end
+
+class PostCreatedWorker
+  include Sidekiq::Worker
+
+  def perform(post_id)
+    post = Post.find_by(id: post_id)
+    if post
+      do_something_with(post)
+    end
+  end
+end
+```
+
+In this case, the `PostCreatedWorker` job will be created for a new `Post` record in Sidekiq before the data is actually written to the database. If Sidekiq picks up that worker and tries to execute it before the transaction is committed, `Post.find_by(id: post_id)` won't find anything and the worker will exit without performing it's task. Even if the worker doesn't need to read from the database, there is still a chance for an error to rollback the transaction leaving a possibility of workers running that should not have been scheduled.
+
+To solve this, workers like this should be invoked in ActiveRecord from an `after_commit` callback. These callbacks are guaranteed to only execute after the data has been written to the database. However, as your application grows and gets more complicated, it can be difficult to ensure that workers are not being scheduled in the middle of transactions.
+
+Switching from callbacks to service objects won't help you either, because service objects can be wrapped in transactions as well. The will just give you a new problem to solve.
+
+```ruby
+class CreatePost
+  def initialize(attributes)
+    @attributes = attributes
+  end
+
+  def call
+    post = Post.create!(attributes)
+    PostCreatedWorker.perform_async(post.id)
+  end
+end
+
+# Still calling `perform_async` inside a transaction.
+Post.transaction do
+  CreatePost.new(post_1_attributes)
+  CreatePost.new(post_2_attributes)
+end
+```
+
+## The Solution
+
+You can use this gem to add Sidekiq client middleware that will either warn you or raise an error when workers are scheduled inside of a database transaction. You can do this by simply adding this to your application's initialization code:
+
+```ruby
+require 'sidekiq/transaction-guard'
+
+Sidekiq.configure_client do |config|
+  config.client_middleware do |chain|
+    chain.add(Sidekiq::TransactionGuard::Middleware)
+  end
+end
+```
+
+### Mode
+
+By default, the behavior is to log that a worker is being scheduled inside of a transaction to the `Sidekiq.logger`. If you are running a test suite, you may want to expose the problematic calls by either raising errors or logging the calls to standard error. The mode can be one of `[:warn, :stderr, :error, :disabled]`.
+
+```ruby
+# Raise errors
+Sidekiq::TransactionGuard.mode = :error
+
+# Log to STDERR
+Sidekiq::TransactionGuard.mode = :stderr
+
+# Log to Sidekiq.logger
+Sidekiq::TransactionGuard.mode = :warn
+
+# Disable entirely
+Sidekiq::TransactionGuard.mode = :disabled
+```
+
+You can also set the mode on individual worker classes with `sidekiq_options transaction_guard: mode`.
+
+```ruby
+class SomeWorker
+  include Sidekiq::Worker
+
+  sidekiq_options transaction_guard: :error
+end
+```
+
+
+You can use the `:disabled` mode to allow individual worker classes to be scheduled inside of transactions where the worker logic doesn't care about the state of the database. For instance, if you use a Sidekiq worker to report errors, you would want to all it inside of transactions. If you don't control the worker you want to change the mode on, you simply call this in an initializer:
+
+```ruby
+SomeWorker.sidekiq_options.merge(transaction_guard: :disabled)
+```
+
+You could
+
+### Notification Handlers
+
+You can also set a block to be called if a worker is scheduled inside of a transaction. This can be useful if you use an error logging service to notify you of problematic calls in production so you can fix them.
+
+```ruby
+# Define a global notify handler
+Sidekiq::TransactionGuard.notify do |job|
+  # Do what ever you need to. The job argument will be a Sidekiq job hash.
+end
+
+# Define on a per worker level
+class SomeWorker
+  include Sidekiq::Worker
+
+  sidekiq_options notify_in_transaction: -> (job) { # Do something }
+end
+
+# Disable the global notification handler on a worker
+class SomeOtherWorker
+  include Sidekiq::Worker
+
+  sidekiq_options notify_in_transaction: false
+end
+```
+
+## Multiple Databases
+
+Out of the box, this gem only deals with one database and monitors the connection pool returned by `ActiveRecord::Base.connection`. If you have multiple databases (or even multiple connections to the same database) that you want to track, you need to tell `Sidekiq::TransactionGuard` about them.
+
+```ruby
+class MyClass < ActiveRecord::Base
+  # This estabilishes a new connection pool.
+  establish_connection(configurations["otherdb"])
+end
+
+Sidekiq::TransactionGuard.add_connection_class(MyClass)
+```
+
+The class is used to get to the connection pool used for the class. You only need to add one class per connection pool, so you don't need to add any subclasses of `MyClass`.
+
+## Transaction Fixtures In Tests
+
+If you're using transaction fixtures in your tests, there will always be a database transaction open. If you're using [DatabaseCleaner](https://github.com/DatabaseCleaner/database_cleaner) in your tests, you just need to include this snippet in your test suite initializer:
+
+```ruby
+require 'sidekiq/transaction_guard/database_cleaner'
+```
+
+This will add the appropriate code so that the surrounding transaction in the test suite is ignored (i.e. workers will only warn/error if there is more than one open transaction).
+
+If you're using something else for your transactional fixtures or have some other weird setup, look in the `lib/sidekiq_transaction_guard/database_cleaner.rb` file for an example of what you need to do.

data/VERSION

@@ -0,0 +1 @@
+1.0.0

data/lib/sidekiq-transaction_guard.rb

@@ -0,0 +1 @@
+require_relative "sidekiq/transaction_guard"

data/lib/sidekiq/transaction_guard.rb

@@ -0,0 +1,117 @@
+# frozen_string_literal: true
+
+require 'sidekiq'
+require 'set'
+require 'thread'
+
+require_relative 'transaction_guard/middleware'
+
+module Sidekiq
+  module TransactionGuard
+    class InsideTransactionError < StandardError
+    end
+
+    @lock = Mutex.new
+    @connection_classes = Set.new
+    @notify = nil
+    @mode = :warn
+
+    class << self
+      VALID_MODES = [:warn, :stderr, :error, :disabled].freeze
+
+      # Set the global mode to one of `[:warn, :stderr, :error, :disabled]`. The
+      # default mode is `:warn`. This controls the behavior of workers enqueued
+      # inside of transactions.
+      # * :warn - Log to Sidekiq.logger
+      # * :stderr - Log to STDERR
+      # * :error - Throw a `Sidekiq::TransactionGuard::InsideTransactionError`
+      # * :disabled - Allow workers inside of transactions
+      def mode=(symbol)
+        if VALID_MODES.include?(symbol)
+          @mode = symbol
+        else
+          raise ArgumentError.new("mode must be one of #{VALID_MODES.inspect}")
+        end
+      end
+
+      # Return the current mode.
+      def mode
+        @mode
+      end
+
+      # Define the global notify block. This block will be called with a Sidekiq
+      # job hash for all jobs enqueued inside transactions if the mode is `:warn`
+      # or `:stderr`.
+      def notify(&block)
+        @notify = block
+      end
+
+      # Return the block set as the notify handler with a call to `notify`.
+      def notify_block
+        @notify
+      end
+
+      # Add a class that maintains it's own connection pool to the connections
+      # being monitored for open transactions. You don't need to add `ActiveRecord::Base`
+      # or subclasses. Only the base class that establishes a new connection pool
+      # with a call to `establish_connection` needs to be added.
+      def add_connection_class(connection_class)
+        @lock.synchronize{ @connection_classes << connection_class }
+      end
+
+      # Return true if any connection is currently inside of a transaction.
+      def in_transaction?
+        connection_classes = [ActiveRecord::Base]
+        unless @connection_classes.empty?
+          connection_classes.concat(@lock.synchronize{ @connection_classes.to_a })
+        end
+        connection_classes.any? do |connection_class|
+          connection_pool = connection_class.connection_pool
+          connection = connection_class.connection if connection_pool.active_connection?
+          if connection
+            connection.open_transactions > allowed_transaction_level(connection_class)
+          else
+            false
+          end
+        end
+      end
+
+      # This method call needs to be wrapped around tests that use transactional fixtures.
+      # It sets up data structures used to track the number of open transactions.
+      def testing(&block)
+        var = :sidekiq_rails_transaction_guard
+        save_val = Thread.current[var]
+        begin
+          Thread.current[var] = (save_val ? save_val.dup : {})
+          yield
+        ensure
+          Thread.current[var] = save_val
+        end
+      end
+
+      # This method needs to be called to set the allowed transaction level for a connection
+      # class (see `add_connection_class` for more info). The current transaction level
+      # for that class' connection will be set as the zero point. This method can only
+      # be called inside a block wrapped with the `testing` method.
+      def set_allowed_transaction_level(connection_class)
+        connection_counts = Thread.current[:sidekiq_rails_transaction_guard]
+        unless connection_counts
+          raise("set_allowed_transaction_level is only allowed inside a testing block")
+        end
+        connection_counts[connection_class.name] = connection_class.connection.open_transactions if connection_counts
+      end
+
+      private
+
+      def allowed_transaction_level(connection_class)
+        connection_counts = Thread.current[:sidekiq_rails_transaction_guard]
+        (connection_counts && connection_counts[connection_class.name]) || 0
+      end
+    end
+  end
+end
+
+# Configure the default transaction guard mode for known testing environments.
+if ENV["RAILS_ENV"] == "test" || ENV["RACK_ENV"] == "test"
+  Sidekiq::TransactionGuard.mode = :stderr
+end

data/lib/sidekiq/transaction_guard/database_cleaner.rb

@@ -0,0 +1,26 @@
+require 'sidekiq/transaction_guard'
+require 'database_cleaner'
+require 'database_cleaner/active_record/transaction'
+
+module Sidekiq
+  module TransactionGuard
+    module DatabaseCleaner
+      # Override the start method to set the base number of allowed transactions to
+      # the current level. Anything above this number will then be considered to be
+      # in a transaction.
+      def start
+        retval = super
+        Sidekiq::TransactionGuard.set_allowed_transaction_level(connection_class)
+        retval
+      end
+
+      # Wrap the `Sidekiq::TransactionGuard.testing` which sets up the data structures
+      # needed for custom counting of the transaction level within a test block.
+      def cleaning(&block)
+        Sidekiq::TransactionGuard.testing{ super(&block) }
+      end
+    end
+  end
+end
+
+::DatabaseCleaner::ActiveRecord::Transaction.send(:prepend, Sidekiq::TransactionGuard::DatabaseCleaner)

data/lib/sidekiq/transaction_guard/middleware.rb

@@ -0,0 +1,80 @@
+# frozen_string_literal: true
+
+module Sidekiq
+  module TransactionGuard
+    # Sidekiq client middleware that will warn/error when workers are called inside of
+    # a database transaction.
+    #
+    # This middleware can read `sidekiq_options` set on the worker for
+    # `:transaction_guard` and `:notify_in_transaction` which will override
+    # the default behavior set in `Sidekiq::TransactionGuard.mode` and
+    # `Sidekiq::TransactionGuard.notify` respectively.
+    class Middleware
+      def call(worker_class, job, queue, redis_pool)
+        # Check if we need to log this. Also, convert worker_class to its actual class
+        log_transaction(worker_class.constantize, job) if in_transaction?
+
+        yield
+      end
+
+      private
+
+      def worker_mode(worker_class)
+        read_sidekiq_option(worker_class, :transaction_guard) || Sidekiq::TransactionGuard.mode
+      end
+
+      def in_transaction?
+        Sidekiq::TransactionGuard.in_transaction?
+      end
+
+      def notify_block(worker_class)
+        handler = read_sidekiq_option(worker_class, :notify_in_transaction)
+        if handler
+          handler
+        elsif handler == false
+          nil
+        else
+          Sidekiq::TransactionGuard.notify_block
+        end
+      end
+
+      def read_sidekiq_option(worker_class, option_name)
+        options = worker_class.sidekiq_options_hash
+        options[option_name.to_s] if options
+      end
+
+      def notify!(worker_class, job)
+        notify_handler = notify_block(worker_class)
+        if notify_handler
+          begin
+            notify_handler.call(job)
+          rescue => e
+            if Sidekiq.logger
+              Sidekiq.logger.error(e)
+            else
+              STDERR.write("ERROR on Sidekiq::TransactionGuard notify block for #{worker_class}: #{e.inspect}\n")
+            end
+          end
+        end
+      end
+
+      def log_transaction(worker_class, job)
+        mode = worker_mode(worker_class)
+        if mode != :disabled
+          message = "#{worker_class.name} was called from inside a database transaction"
+          if mode == :error
+            raise Sidekiq::TransactionGuard::InsideTransactionError.new(message)
+          else
+            logger = Sidekiq.logger unless mode == :stderr
+            if logger
+              logger.warn(message)
+            else
+              STDERR.write("WARNING #{message}\n")
+            end
+            notify!(worker_class, job)
+          end
+        end
+      end
+    end
+  end
+end

data/lib/sidekiq/transaction_guard/version.rb

@@ -0,0 +1,7 @@
+# frozen_string_literal: true
+
+module Sidekiq
+  module TransactionGuard
+    VERSION = File.read(File.expand_path("../../../../VERSION", __FILE__)).chomp.freeze
+  end
+end

data/sidekiq-transaction_guard.gemspec

@@ -0,0 +1,45 @@
+# frozen_string_literal: true
+
+require_relative "lib/sidekiq/transaction_guard/version"
+
+Gem::Specification.new do |spec|
+  spec.name          = "sidekiq-transaction_guard"
+  spec.version       = Sidekiq::TransactionGuard::VERSION
+  spec.authors       = ["Brian Durand", "Winston Durand"]
+  spec.email         = ["bbdurand@gmail.com", "me@winstondurand.com"]
+
+  spec.summary       = "Protect from accidentally invoking Sidekiq jobs when there are open database transactions"
+  spec.homepage      = "https://github.com/bdurand/sidekiq-transaction_guard"
+  spec.license       = "MIT"
+
+  # Specify which files should be added to the gem when it is released.
+  # The `git ls-files -z` loads the files in the RubyGem that have been added into git.
+  ignore_files = %w(
+    .gitignore
+    .travis.yml
+    Appraisals
+    Gemfile
+    Gemfile.lock
+    Rakefile
+    gemfiles/
+    spec/
+  )
+  spec.files         = Dir.chdir(File.expand_path('..', __FILE__)) do
+    `git ls-files -z`.split("\x0").reject{ |f| ignore_files.any?{ |path| f.start_with?(path) } }
+  end
+  spec.bindir        = "exe"
+  spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
+  spec.require_paths = ["lib"]
+
+  spec.required_ruby_version = '>= 2.2.2'
+
+  spec.add_dependency "activerecord", ">= 4.0"
+  spec.add_dependency "sidekiq", ">= 3.0"
+
+  spec.add_development_dependency "bundler", "~> 1.10"
+  spec.add_development_dependency "rake"
+  spec.add_development_dependency "rspec", "~> 3.0"
+  spec.add_development_dependency "database_cleaner"
+  spec.add_development_dependency "sqlite3"
+  spec.add_development_dependency "appraisal"
+end

metadata

@@ -0,0 +1,169 @@
+--- !ruby/object:Gem::Specification
+name: sidekiq-transaction_guard
+version: !ruby/object:Gem::Version
+  version: 1.0.0
+platform: ruby
+authors:
+- Brian Durand
+- Winston Durand
+autorequire: 
+bindir: exe
+cert_chain: []
+date: 2018-10-08 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: activerecord
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '4.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '4.0'
+- !ruby/object:Gem::Dependency
+  name: sidekiq
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '3.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '3.0'
+- !ruby/object:Gem::Dependency
+  name: bundler
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.10'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.10'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: rspec
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.0'
+- !ruby/object:Gem::Dependency
+  name: database_cleaner
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: sqlite3
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: appraisal
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+description: 
+email:
+- bbdurand@gmail.com
+- me@winstondurand.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- CHANGELOG.md
+- MIT_LICENSE.txt
+- README.md
+- VERSION
+- lib/sidekiq-transaction_guard.rb
+- lib/sidekiq/transaction_guard.rb
+- lib/sidekiq/transaction_guard/database_cleaner.rb
+- lib/sidekiq/transaction_guard/middleware.rb
+- lib/sidekiq/transaction_guard/version.rb
+- sidekiq-transaction_guard.gemspec
+homepage: https://github.com/bdurand/sidekiq-transaction_guard
+licenses:
+- MIT
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: 2.2.2
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.7.6
+signing_key: 
+specification_version: 4
+summary: Protect from accidentally invoking Sidekiq jobs when there are open database
+  transactions
+test_files: []