rails-transactional-outbox 0.1.0

data/README.md

@@ -0,0 +1,285 @@
+# RailsTransactionalOutbox
+
+An implementation of transactional outbox pattern to be used with Rails.
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'rails-transactional-outbox'
+```
+
+And then execute:
+
+    $ bundle install
+
+Or install it yourself as:
+
+    $ gem install rails-transactional-outbox
+
+## Usage
+
+To get familiar with the pattern:
+- [Pattern: Transactional outbox](https://microservices.io/patterns/data/transactional-outbox.html)
+- [Transactional Outbox: What is it and why you need it?](https://morningcoffee.io/what-is-a-transaction-outbox-and-why-you-need-it.html)
+
+Create the initializer with the following content:
+
+``` rb
+Rails.application.config.to_prepare do
+  RailsTransactionalOutbox.configure do |config|
+    config.database_connection_provider = ActiveRecord::Base # required 
+    config.transaction_provider = ActiveRecord::Base # required
+    config.logger = Rails.logger # required
+    config.outbox_model = OutboxEntry # required
+    config.error_handler = Sentry # non-required, but highly recommended, defaults to RailsTransactionalOutbox::ErrorHandlers::NullErrorHandler
+    
+    config.transactional_outbox_worker_sleep_seconds = 1 # optional, defaults to 0.5
+    config.transactional_outbox_worker_idle_delay_multiplier = 5 # optional, defaults to 1, if there are no outbox entries to be processed, then the sleep time for the thread will be equal to transactional_outbox_worker_idle_delay_multiplier * transactional_outbox_worker_sleep_seconds  
+    config.outbox_batch_size = 100 # optional, defaults to 100
+    config.add_record_processor(MyCustomOperationProcerssor) # optional, by default it contains only one processor for ActiveRecord, but you could add more
+  end
+end
+```
+
+Create OutboxEntry model (or use a different name, just make sure to adjust config/migration) with the following content:
+
+``` rb
+class OutboxEntry < ApplicationRecord
+  include RailsTransactionalOutbox::OutboxModel
+  
+  # optional, if you want to use encryption
+  crypt_keeper :changeset, :arguments, encryptor: :postgres_pgp, key: ENV.fetch("CRYPT_KEEPER_KEY"), encoding: "UTF-8"
+  outbox_encrypt_json_for :changeset, :arguments
+end
+```
+
+And use the following migration:
+
+``` rb
+create_table(:outbox_entries) do |t|
+  t.string "resource_class"
+  t.string "resource_id"
+  t.string "event_name", null: false
+  t.string "context", null: false
+  t.datetime "processed_at"
+  t.text "arguments", null: false, default: "{}"
+  t.text "changeset", null: false, default: "{}"
+  t.datetime "failed_at"
+  t.datetime "retry_at"
+  t.string "error_class"
+  t.string "error_message"
+  t.integer "attempts", null: false, default: 0
+  t.datetime "created_at", precision: 6, null: false
+  t.datetime "updated_at", precision: 6, null: false
+
+  t.index %w[resource_class event_name], name: "idx_outbox_enc_entries_on_resource_class_and_event"
+  t.index %w[resource_class resource_id], name: "idx_outbox_enc_entries_on_resource_class_and_resource_id"
+  t.index ["context"], name: "idx_outbox_enc_entries_on_topic"
+  t.index ["created_at"], name: "idx_outbox_enc_entries_on_created_at"
+  t.index ["created_at"], name: "idx_outbox_enc_entries_on_created_at_not_processed", where: "processed_at IS NULL"
+  t.index %w[resource_class created_at], name: "idx_outbox_enc_entries_on_resource_class_and_created_at"
+  t.index %w[resource_class processed_at], name: "idx_outbox_enc_entries_on_resource_class_and_processed_at"
+end
+```
+
+Keep in mind that `arguments` and `changeset` are `text` columns here. If you don't want to use encryption, replace them with `jsonb` columns:
+
+```rb 
+t.jsonb "arguments", null: false, default: {}
+t.jsonb "changeset", null: false, default: {}
+```
+
+The following columns: `resource_class`, `resource_id` and `changeset` are dedicated to ActiveRecord integration. Do not try to modify these columns for custom processors.
+
+As the last step, include `RailsTransactionalOutbox::ReliableModel` module in the models that are supposed to have reliable `after_commit` callbacks, for example:
+
+``` ruby
+class User < ActiveRecord::Base
+  include RailsTransactionalOutbox::ReliableModel
+end 
+```
+
+Now, you can just replace `after_commit` callbacks with `reliable_after_commit`. The interface is going to be the same as for `after_commit`:
+
+- you can provide `:on` option to specific when the callback should be executed
+- you can use both blocks or symbols as the method names
+- you can pass `:if` and `:unless` options
+- you can also use `reliable_after_create_commit`, `reliable_after_update_commit`, `reliable_after_destroy_commit`, `reliable_after_save_commit`.
+
+When executing the callbacks, you can use `previous_changes` which will contain the changes that are persisted as changesets. One potential gotcha is that Time/Date types are stored as strings, so oyu might need to handle some conversion to be on the safe side.
+
+Inclusion of this module will result in OutboxEntry records being created after create/update/destroy. For these entries, the `context` column will be populated with `active_record` value.
+
+### Custom processors
+
+If you want to add some custom processor either for ActiveRecord or for custom service objects, create an object inheriting from `RailsTransactionalOutbox::RecordProcessors::BaseProcessor`, which has the following interface:
+
+
+``` ruby
+class RailsTransactionalOutbox
+  class RecordProcessors
+    class BaseProcessor
+      def applies?(_record)
+        raise "implement me"
+      end
+
+      def call(_record)
+        raise "implement me"
+      end
+    end
+  end
+end
+```
+
+For a reference, this is an example of `ActiveRecordProcessor`:
+
+``` ruby
+class RailsTransactionalOutbox
+  class RecordProcessors
+    class ActiveRecordProcessor < RailsTransactionalOutbox::RecordProcessors::BaseProcessor
+      ACTIVE_RECORD_CONTEXT = "active_record"
+      private_constant :ACTIVE_RECORD_CONTEXT
+
+      def self.context
+        ACTIVE_RECORD_CONTEXT
+      end
+
+      def applies?(record)
+        record.context == ACTIVE_RECORD_CONTEXT
+      end
+
+      def call(record)
+        model = record.infer_model or raise CouldNotFindModelError.new(record)
+        model.previous_changes = record.transformed_changeset.with_indifferent_access
+        model.reliable_after_commit_callbacks.for_event_type(record.event_type).each do |callback|
+          callback.call(model)
+        end
+      end
+
+      class CouldNotFindModelError < StandardError
+        attr_reader :record
+
+        def initialize(record)
+          super()
+          @record = record
+        end
+
+        def to_s
+          "could not find model for outbox record: #{record.id}"
+        end
+      end
+    end
+  end
+end
+```
+
+If you want to extent the behavior of `ActiveRecordProcessor`, you could actually create a new processor that handles exactly the same context as multiple processors can be used for the same context.
+
+When adding a custom processor for service objects/operations, you might want to use `arguments` column, to keep all the arguments there.
+
+If you use encryption and you want to deal with properly deserialized hash, you `transformed_changeset` and `transformed_arguments` methods (like `ActiveRecordProcessor` does.)
+
+When dealing with custom service objects, remember to create OutboxEntry records inside the same transaction:
+
+``` rb
+class MyServiceObject
+  def call(user_id)
+    transaction do
+      execute_some_logic
+      OutboxEntry.create!(context: "service_object", event_name: "my_service_object_called", arguments: { user_id: user_id })
+    end
+  end
+end
+```
+
+### Running outbox worker
+
+Use the following Rake task:
+
+```
+RAILS_TRANSACTIONAL_OUTBOX_THREADS_NUMBER=5 DB_POOL=10 bundle exec rake rails_transactional_outbox:worker
+```
+
+If you want to use just a single thread:
+
+```
+bundle exec rake bookingsync_prometheus:producer
+```
+
+### Archiving old outbox records
+
+You will probably want to periodically archive/delete processed outbox records. It's recommended to use [tartarus-rb](https://github.com/BookingSync/tartarus-rb) for that.
+
+Here is an example config:
+
+```
+tartarus.register do |item|
+  item.model = OutboxEntry
+  item.cron = "5 4 * * *"
+  item.queue = "default"
+  item.archive_items_older_than = -> { 3.days.ago }
+  item.timestamp_field = :processed_at
+  item.archive_with = :delete_all_using_limit_in_batches
+end
+```
+
+### Health Checks
+
+First, you need to set `REDIS_URL` ENV variable to provide the URL for Redis.
+
+Then, you need to explicitly enable the health check (e.g. in the initializer):
+
+``` rb
+RailsTransactionalOutbox.enable_outbox_worker_healthcheck
+```
+
+To perform the actual health check, use `bin/rails_transactional_outbox_health_check`. On success, the script exits with `0` status and on failure, it logs the error and exits with `1` status.
+
+```
+bundle exec rails_transactional_outbox_health_check
+```
+
+The logic is based on checking a special value in Redis that is set (and unset) for a given container when Outbox workers are initialized/stopped/processing messages.
+
+It works for both readiness and liveness checks.
+
+#### Events, hooks and monitors
+
+You can subscribe to certain events that are published by `RailsTransactionalOutbox.monitor`. The monitor is based on [`dry-monitor`](https://github.com/dry-rb/dry-monitor).
+
+Available events and arguments are:
+
+- "rails_transactional_outbox.started", no arguments
+- "rails_transactional_outbox.stopped", no arguments
+- "rails_transactional_outbox.shutting_down", no arguments
+- "rails_transactional_outbox.record_processing_failed", arguments: outbox_record
+- "rails_transactional_outbox.record_processed", no arguments: outbox_record
+- "rails_transactional_outbox.error", arguments: error, error_message
+- "rails_transactional_outbox.heartbeat", no arguments
+
+
+#### Testing the logic from reliable_after_commit callbacks
+
+The fastest way to handle it would be to add this to `rails_helper.rb`:
+
+``` ruby
+ApplicationRecord.after_commit do
+  RailsTransactionalOutbox::OutboxEntriesProcessor.new.call
+end
+```
+
+## Development
+
+After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+
+To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and the created tag, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/BookingSync/rails-transactional-outbox.
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).

data/Rakefile

@@ -0,0 +1,10 @@
+# frozen_string_literal: true
+
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+
+RSpec::Core::RakeTask.new(:spec)
+
+task default: :spec
+
+import("./lib/tasks/rails_transactional_outbox.rake")

data/bin/console

@@ -0,0 +1,15 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require "bundler/setup"
+require "rails/transactional/outbox"
+
+# You can add fixtures and/or initialization code here to make experimenting
+# with your gem easier. You can also use a different console, if you like.
+
+# (If you use this, don't forget to add pry to your Gemfile!)
+# require "pry"
+# Pry.start
+
+require "irb"
+IRB.start(__FILE__)

data/bin/rails_transactional_outbox_health_check

@@ -0,0 +1,13 @@
+#!/usr/bin/env ruby
+
+require "bundler/setup"
+require "logger"
+require_relative "../lib/rails_transactional_outbox"
+
+result = RailsTransactionalOutbox::HealthCheck.check
+if result.empty?
+  exit 0
+else
+  Logger.new($stdout).fatal("[Rails Transactional Outbox Worker] health check failed: #{result}")
+  exit 1
+end

data/bin/setup

@@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+set -euo pipefail
+IFS=$'\n\t'
+set -vx
+
+bundle install
+
+# Do any other automated setup that you need to do here

data/lib/rails-transactional-outbox.rb

@@ -0,0 +1,3 @@
+# frozen_string_literal: true
+
+require "rails_transactional_outbox"

data/lib/rails_transactional_outbox/configuration.rb

@@ -0,0 +1,33 @@
+# frozen_string_literal: true
+
+class RailsTransactionalOutbox
+  class Configuration
+    attr_accessor :database_connection_provider, :logger, :outbox_model, :transaction_provider
+    attr_writer :error_handler, :transactional_outbox_worker_sleep_seconds,
+      :transactional_outbox_worker_idle_delay_multiplier, :outbox_batch_size
+
+    def error_handler
+      @error_handler || RailsTransactionalOutbox::ErrorHandlers::NullErrorHandler
+    end
+
+    def transactional_outbox_worker_sleep_seconds
+      @transactional_outbox_worker_sleep_seconds || 0.5
+    end
+
+    def transactional_outbox_worker_idle_delay_multiplier
+      @transactional_outbox_worker_idle_delay_multiplier || 10
+    end
+
+    def outbox_batch_size
+      @outbox_batch_size || 100
+    end
+
+    def record_processors
+      @record_processors ||= [RailsTransactionalOutbox::RecordProcessors::ActiveRecordProcessor.new]
+    end
+
+    def add_record_processor(record_processor)
+      record_processors << record_processor
+    end
+  end
+end

data/lib/rails_transactional_outbox/error_handlers/null_error_handler.rb

@@ -0,0 +1,9 @@
+# frozen_string_literal: true
+
+class RailsTransactionalOutbox
+  class ErrorHandlers
+    class NullErrorHandler
+      def self.capture_exception(_error); end
+    end
+  end
+end

data/lib/rails_transactional_outbox/error_handlers.rb

@@ -0,0 +1,6 @@
+# frozen_string_literal: true
+
+class RailsTransactionalOutbox
+  class ErrorHandlers
+  end
+end

data/lib/rails_transactional_outbox/event_type.rb

@@ -0,0 +1,37 @@
+# frozen_string_literal: true
+
+class RailsTransactionalOutbox
+  class EventType
+    EVENT_TYPES = %i[create update destroy].freeze
+    EVENT_NAME_SUFFIXES = %w[created updated destroyed].freeze
+    private_constant :EVENT_TYPES, :EVENT_NAME_SUFFIXES
+
+    def self.resolve_from_event_name(event_name)
+      EVENT_TYPES.find(-> { raise "unknown event type: #{event_name}" }) do |event_type|
+        event_name.to_s.end_with?(new(event_type).event_name_suffix)
+      end
+    end
+
+    attr_reader :event_type
+    private     :event_type
+
+    def initialize(event_type)
+      @event_type = event_type.to_sym
+    end
+
+    def to_sym
+      event_type
+    end
+
+    def event_name_suffix
+      EVENT_TYPES
+        .zip(EVENT_NAME_SUFFIXES)
+        .to_h
+        .fetch(event_type) { raise "unknown event type: #{event_type}" }
+    end
+
+    def destroy?
+      event_type == :destroy
+    end
+  end
+end

data/lib/rails_transactional_outbox/exponential_backoff.rb

@@ -0,0 +1,9 @@
+# frozen_string_literal: true
+
+class RailsTransactionalOutbox
+  class ExponentialBackoff
+    def self.backoff_for(multiplier, count)
+      (multiplier * (2**count))
+    end
+  end
+end

data/lib/rails_transactional_outbox/health_check.rb

@@ -0,0 +1,48 @@
+# frozen_string_literal: true
+
+require "redis"
+
+class RailsTransactionalOutbox
+  class HealthCheck
+    KEY_PREFIX = "__rails_transactional__outbox_worker__running__"
+    VALUE = "OK"
+    private_constant :KEY_PREFIX, :VALUE
+
+    def self.check(redis_url: ENV.fetch("REDIS_URL", nil), hostname: ENV.fetch("HOSTNAME", nil),
+      expiry_time_in_seconds: 120)
+      new(redis_url: redis_url, hostname: hostname, expiry_time_in_seconds: expiry_time_in_seconds).check
+    end
+
+    attr_reader :redis_client, :hostname, :expiry_time_in_seconds
+
+    def initialize(redis_url: ENV.fetch("REDIS_URL", nil), hostname: ENV.fetch("HOSTNAME", nil),
+      expiry_time_in_seconds: 120)
+      @redis_client = Redis.new(url: redis_url)
+      @hostname = hostname
+      @expiry_time_in_seconds = expiry_time_in_seconds
+    end
+
+    def check
+      value = redis_client.get(key)
+      if value == VALUE
+        ""
+      else
+        "[Rails Transactional Outbox Worker - expected #{VALUE} under #{key}, found: #{value}] "
+      end
+    end
+
+    def register_heartbeat
+      redis_client.set(key, VALUE, ex: expiry_time_in_seconds)
+    end
+
+    def worker_stopped
+      redis_client.del(key)
+    end
+
+    private
+
+    def key
+      "#{KEY_PREFIX}#{hostname}"
+    end
+  end
+end

data/lib/rails_transactional_outbox/monitor.rb

@@ -0,0 +1,47 @@
+# frozen_string_literal: true
+
+class RailsTransactionalOutbox
+  class Monitor < Dry::Monitor::Notifications
+    EVENTS = %w[
+      rails_transactional_outbox.started
+      rails_transactional_outbox.stopped
+      rails_transactional_outbox.shutting_down
+      rails_transactional_outbox.record_processing_failed
+      rails_transactional_outbox.record_processed
+      rails_transactional_outbox.error
+      rails_transactional_outbox.heartbeat
+    ].freeze
+
+    private_constant :EVENTS
+
+    def initialize
+      super(:rails_transactional_outbox)
+      EVENTS.each { |event| register_event(event) }
+    end
+
+    def subscribe(event)
+      return super if events.include?(event.to_s)
+
+      raise UnknownEventError.new(events, event)
+    end
+
+    def events
+      EVENTS
+    end
+
+    class UnknownEventError < StandardError
+      attr_reader :available_events, :current_event
+      private     :available_events, :current_event
+
+      def initialize(available_events, current_event)
+        super()
+        @available_events = available_events
+        @current_event = current_event
+      end
+
+      def message
+        "unknown event: #{current_event}, the available events are: #{available_events.join(", ")}"
+      end
+    end
+  end
+end

data/lib/rails_transactional_outbox/outbox_entries_processor.rb

@@ -0,0 +1,56 @@
+# frozen_string_literal: true
+
+class RailsTransactionalOutbox
+  class OutboxEntriesProcessor
+    attr_reader :config
+    private     :config
+
+    def initialize(config: RailsTransactionalOutbox.configuration)
+      @config = config
+    end
+
+    def call
+      return [] unless outbox_model.any_records_to_process?
+
+      transaction do
+        outbox_model.fetch_processable(outbox_batch_size).to_a.tap do |records_to_process|
+          failed_records = Concurrent::Array.new
+          # TODO: considering adding internal threads for better efficiency, these are just IO operations
+          # that don't require ordering of any kind
+          records_to_process.each do |record|
+            begin
+              process(record)
+            rescue => e
+              record.handle_error(e)
+              record.save!
+              failed_records << record
+            end
+            yield record if block_given?
+          end
+          processed_records = records_to_process - failed_records
+          mark_as_processed(processed_records)
+        end
+      end
+    end
+
+    private
+
+    delegate :outbox_model, :outbox_batch_size, :transaction_provider, to: :config
+    delegate :transaction, to: :transaction_provider
+
+    def mark_as_processed(processed_records)
+      outbox_model
+        .where(id: processed_records)
+        .update_all(processed_at: Time.current, error_class: nil, error_message: nil,
+          failed_at: nil, retry_at: nil)
+    end
+
+    def process(record)
+      record_processor.call(record)
+    end
+
+    def record_processor
+      @record_processor ||= RailsTransactionalOutbox::RecordProcessor.new
+    end
+  end
+end

data/lib/rails_transactional_outbox/outbox_entry_factory.rb

@@ -0,0 +1,32 @@
+# frozen_string_literal: true
+
+class RailsTransactionalOutbox
+  class OutboxEntryFactory
+    attr_reader :config
+    private     :config
+
+    def initialize(config: RailsTransactionalOutbox.configuration)
+      @config = config
+    end
+
+    def build(model, event_type)
+      config.outbox_model.new(attributes_for_entry(model, event_type))
+    end
+
+    private
+
+    def attributes_for_entry(model, event_type)
+      {
+        resource_class: model.class.to_s,
+        resource_id: model.id,
+        changeset: model.saved_changes,
+        event_name: "#{model.model_name.singular}_#{event_name_suffix(event_type)}",
+        context: RailsTransactionalOutbox::RecordProcessors::ActiveRecordProcessor.context
+      }
+    end
+
+    def event_name_suffix(event_type)
+      RailsTransactionalOutbox::EventType.new(event_type).event_name_suffix
+    end
+  end
+end