outboxer 0.1.11 → 1.0.0.pre.beta

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: f881a119f2c2123b34c0a7153590c2eab9907b44045e64d51e16ce914e731bb6
-  data.tar.gz: d30a09f38f02fea3bce19895922d294e40dfb25555e1fcf4fd06b33560d1e2db
+  metadata.gz: 02b688da126c6e1e723b3f7df4f5b2bcf985c9d204051292e4b10f76ee9f1060
+  data.tar.gz: 512b45ceaa6f67f454c5fa9690684852624c76af9c804697ba79d3f5119a9d6e
 SHA512:
-  metadata.gz: df0a95d0f24b7ff85038711be1acd4aec4dc5760accb390fb40fdf07342d911b5b07373c4eacbc2e05e25a7a89137ef26c20e201b556c6e07973469806eddaff
-  data.tar.gz: 1d329faae4668703b51cd5becdce8215e4251e1217d9c8311795ab602d71e140511b79874908963a947c24939025bd2dbcd8af0497aa985d9478f7919d7602d7
+  metadata.gz: f1aaaf3225fc64d1371ceaf295a1184296c691d355d8fa082ef201fe3efc82ed6767cfb60592855012853e1e8f1529a99d6d7f199d80a2e41a0b3dca3bfd2a8d
+  data.tar.gz: 2087f71e099d99f80b053bc61d4fee3e591205aa499946ced71c19ea0ad74dc17a0684029e52880f95de470ab93324827e6f29ca92f179a2e9ef2bc80b0c1fae

data/README.md

@@ -5,96 +5,130 @@
 
 ## Background
 
-Typically in event driven Ruby on Rails applications:
+Outboxer is an ActiveRecord implementation of the [transactional outbox pattern](https://microservices.io/patterns/data/transactional-outbox.html).
 
-1. a domain event model is created in an SQL database table
-2. a sidekiq worker is queued to handle the domain event asynchronously
+## Installation
 
-## Problem
+### 1. add gem to your application's gemfile
 
-As these two operations span multiple database types (SQL and redis), they can not be combined into a single atomic operation using a transaction. If either step fails, inconsistencies can occur.
-
-## Solution
-
-Outboxer is a simple Ruby on Rails implementation of the [transactional outbox pattern](https://microservices.io/patterns/data/transactional-outbox.html): a well established solution to this problem. It ensures both operations succeed _eventually_, or both fail.
-
-### Getting started
-
-### Installation
-
-1. Add the Outboxer gem to your application's Gemfile:
-
-```ruby
+```
 gem 'outboxer'
 ```
 
-2. Install the Outboxer gem:
+### 2. install gem
 
-```bash
+```
 bundle install
 ```
 
-3. Generate the migration and publisher files
+## Usage
+
+### 1. generate schema
 
 ```bash
-bin/rails generate outboxer:install
+bin/rails g outboxer:schema
 ```
 
-### Usage
+### 2. migrate schema
+
+```bash
+bin/rake db:migrate
+```
 
-#### 1. Include `Outboxer::Outboxable` into your existing Message model
+###  3. after event created, backlog message
 
 ```ruby
-class Message < ApplicationRecord
-  include Outboxer::Outboxable
+class Event < ActiveRecord::Base
+  # ...
+
+  after_create do |event|
+    Outboxer::Message.backlog(
+      messageable_type: event.class.name,
+      messageable_id: event.id)
+  end
 end
 ```
 
-#### 2. Update the generated bin/publish block e.g.
+### 4. add event created job
 
 ```ruby
-Outboxer::Publisher.publish do |message:, logger:|
-  logger.info("[#{message.id}] publishing")
+class EventCreatedJob
+  include Sidekiq::Job
 
-  HardJob.perform_async({ "message_id" => message.id })
+  def perform(args)
+    event = Event.find(args['id'])
 
-  logger.info("[#{message.id}] published")
+    # ...
+  end
 end
 ```
 
-#### 3. Migrate the database
+### 5. generate message publisher
 
 ```bash
-bin/rake db:migrate
+bin/rails g outboxer:message_publisher
 ```
 
-#### 4. Run the publisher
+### 6. update publish block to perform event created job async
+
+```ruby
+Outboxer::Publisher.publish do |message|
+  case message[:messageable_type]
+  when 'Event'
+    EventCreatedJob.perform_async({ 'id' => message[:messageable_id] })
+  end
+end
+```
+
+### 6. run message publisher
 
 ```bash
-bin/publisher
+bin/outboxer_message_publisher
 ```
 
-## Implementation
+### 7. manage messages
 
-1. when an `ActiveRecord` model that includes `Outbox::Outboxable` is created, an `unpublished` `Outboxer::Message` is automatically created in the same transaction, with `Outboxer::Message#message` polymorphically assigned to the original model
+manage backlogged, queued, publishing and failed messages with the web ui
 
-2. When the publisher finds a new `unpublished` `Outboxer::Message`, it yields to a user-supplied block and then:
-    - removes it if the task completes successfully
-    - marks it as failed and records the error if there's a problem
+<img width="1257" alt="Screenshot 2024-05-20 at 8 47 57 pm" src="https://github.com/fast-programmer/outboxer/assets/394074/0446bc7e-9d5f-4fe1-b210-ff394bdacdd6">
 
-To see all the parts working together in a single place, check out the [publisher_spec.rb](https://github.com/fast-programmer/outboxer/blob/master/spec/outboxer/publisher_spec.rb)
+#### rails
 
+##### config/routes.rb
 
-## Motivation
+```ruby
+require 'outboxer/web'
+
+Rails.application.routes.draw do
+  mount Outboxer::Web, at: '/outboxer'
+end
+```
+
+#### rack
 
-Outboxer was created to help high growth SAAS companies transition to event driven architecture quickly.
+##### config.ru
 
-Specifically this means:
+```ruby
+require 'outboxer/web'
+
+map '/outboxer' do
+  run Outboxer::Web
+end
+```
+
+### 8. monitor message publisher
+
+understanding how much memory and cpu is required by the message publisher
+
+<img width="310" alt="Screenshot 2024-05-20 at 10 41 57 pm" src="https://github.com/fast-programmer/outboxer/assets/394074/1222ad47-15e3-44d1-bb45-6abc6b3e4325">
+
+```bash
+run bin/outboxer_message_publishermon
+```
+
+## Motivation
 
-1. fast integration into existing Ruby on Rails applications (< 1 hour)
-2. comprehensive documentation
-3. high reliability in production environments
-4. forever free to use in commerical applications (MIT licence)
+Outboxer was created to help Rails teams migrate to eventually consistent event driven architecture quickly, using existing tools and infrastructure.
 
 ## Contributing
 

data/db/migrate/create_outboxer_exceptions.rb

@@ -0,0 +1,20 @@
+class CreateOutboxerExceptions < ActiveRecord::Migration[6.1]
+  def up
+    ActiveRecord::Base.transaction do
+      create_table :outboxer_exceptions do |t|
+        t.references :message, null: false, foreign_key: { to_table: :outboxer_messages }
+
+        t.text :class_name, null: false
+        t.text :message_text, null: false
+
+        t.timestamps
+      end
+
+      remove_column :outboxer_exceptions, :updated_at
+    end
+  end
+
+  def down
+    drop_table :outboxer_exceptions if table_exists?(:outboxer_exceptions)
+  end
+end

data/db/migrate/create_outboxer_frames.rb

@@ -0,0 +1,16 @@
+class CreateOutboxerFrames < ActiveRecord::Migration[6.1]
+  def up
+    create_table :outboxer_frames do |t|
+      t.references :exception, null: false, foreign_key: { to_table: :outboxer_exceptions }
+
+      t.integer :index, null: false
+      t.text :text, null: false
+
+      t.index [:exception_id, :index], unique: true
+    end
+  end
+
+  def down
+    drop_table :outboxer_frames if table_exists?(:outboxer_frames)
+  end
+end

data/db/migrate/create_outboxer_messages.rb

@@ -0,0 +1,19 @@
+class CreateOutboxerMessages < ActiveRecord::Migration[6.1]
+  def up
+    create_table :outboxer_messages do |t|
+      t.string :status, null: false, limit: 255
+
+      t.text :messageable_id, null: false
+      t.text :messageable_type, null: false
+
+      t.timestamps
+    end
+
+    add_index :outboxer_messages, :status
+    add_index :outboxer_messages, [:status, :updated_at]
+  end
+
+  def down
+    drop_table :outboxer_messages if table_exists?(:outboxer_messages)
+  end
+end

data/generators/message_publisher_generator.rb

@@ -0,0 +1,11 @@
+module Outboxer
+  class MessagePublisherGenerator < Rails::Generators::Base
+    include Rails::Generators::Migration
+
+    source_root File.expand_path('../', __dir__)
+    def copy_bin_file
+      template "bin/outboxer_message_publisher", "bin/outboxer_message_publisher"
+      run "chmod +x bin/outboxer_message_publisher"
+    end
+  end
+end

data/generators/{outboxer/install_generator.rb → schema_generator.rb}

@@ -1,8 +1,8 @@
 module Outboxer
-  class InstallGenerator < Rails::Generators::Base
+  class SchemaGenerator < Rails::Generators::Base
     include Rails::Generators::Migration
 
-    source_root File.expand_path("templates", __dir__)
+    source_root File.expand_path('../', __dir__)
 
     def self.next_migration_number(dirname)
       next_number = Time.now.utc.strftime("%Y%m%d%H%M%S").to_i
@@ -12,18 +12,13 @@ module Outboxer
       next_number.to_s
     end
 
-    def copy_bin_file
-      template "bin/publisher.rb", "bin/publisher"
-      run "chmod +x bin/publisher"
-    end
-
     def copy_migrations
       migration_template(
-        "migrations/create_outboxer_messages.rb",
+        "db/migrate/create_outboxer_messages.rb",
         "db/migrate/create_outboxer_messages.rb")
 
       migration_template(
-        "migrations/create_outboxer_exceptions.rb",
+        "db/migrate/create_outboxer_exceptions.rb",
         "db/migrate/create_outboxer_exceptions.rb")
     end
   end

data/lib/outboxer/database.rb

@@ -0,0 +1,44 @@
+require 'erb'
+require 'yaml'
+
+module Outboxer
+  module Database
+    extend self
+
+    def config(environment: ENV['RAILS_ENV'] || 'development', path: 'config/database.yml')
+      db_config_content = File.read(path)
+      db_config_erb_result = ERB.new(db_config_content).result
+      YAML.safe_load(db_config_erb_result, aliases: true)[environment]
+    end
+
+    def connect(config:, logger: Logger.new($stdout, level: Logger::INFO))
+      logger&.info  "              _   _                        "
+      logger&.info  "             | | | |                       "
+      logger&.info  "   ___  _   _| |_| |__   _____  _____ _ __ "
+      logger&.info  "  / _ \\| | | | __| '_ \\ / _ \\ \\/ / _ \\ '__|"
+      logger&.info  " | (_) | |_| | |_| |_) | (_) >  <  __/ |   "
+      logger&.info  "  \\___/ \\__,_|\\__|_.__/ \\___/_/\\_\\___|_|   "
+      logger&.info  "                                           "
+      logger&.info  "                                           "
+
+      logger&.info "Running in ruby #{RUBY_VERSION} " \
+        "(#{RUBY_RELEASE_DATE} revision #{RUBY_REVISION[0, 10]}) [#{RUBY_PLATFORM}]"
+
+      logger&.info "Connecting to database"
+      ActiveRecord::Base.establish_connection(config)
+      ActiveRecord::Base.connection_pool.with_connection {}
+      logger&.info "Connected to database"
+    end
+
+    def connected?
+      ActiveRecord::Base.connected?
+    end
+
+    def disconnect(logger: Logger.new($stdout, level: Logger::INFO))
+      logger&.info "Disconnecting from database"
+      ActiveRecord::Base.connection_handler.clear_active_connections!
+      ActiveRecord::Base.connection_handler.connection_pool_list.each(&:disconnect!)
+      logger&.info "Disconnected from database"
+    end
+  end
+end

data/lib/outboxer/logger.rb

@@ -0,0 +1,17 @@
+require 'logger'
+
+module Outboxer
+  class Logger < ::Logger
+    def initialize(*args, **kwargs)
+      super(*args, **kwargs)
+
+      self.formatter = proc do |severity, datetime, progname, msg|
+        formatted_time = datetime.strftime('%Y-%m-%dT%H:%M:%S.%LZ')
+        pid = Process.pid
+        tid = Thread.current.object_id.to_s(36)
+        level = severity
+        "#{formatted_time} pid=#{pid} tid=#{tid} #{level}: #{msg}\n"
+      end
+    end
+  end
+end

data/lib/outboxer/message.rb

@@ -1,19 +1,268 @@
-require "active_record"
-
 module Outboxer
-  class Message < ::ActiveRecord::Base
-    self.table_name = :outboxer_messages
+  module Message
+    extend self
+
+    Status = Models::Message::Status
+
+    def backlog(messageable_type:, messageable_id:)
+      ActiveRecord::Base.connection_pool.with_connection do
+        ActiveRecord::Base.transaction do
+          current_time = Time.now.utc
+
+          message = Models::Message.create!(
+            messageable_id: messageable_id,
+            messageable_type: messageable_type,
+            status: Models::Message::Status::BACKLOGGED,
+            created_at: current_time,
+            updated_at: current_time)
+
+          { id: message.id }
+        end
+      end
+    end
+
+    def find_by_id(id:)
+      ActiveRecord::Base.connection_pool.with_connection do
+        ActiveRecord::Base.transaction do
+          message = Models::Message.includes(exceptions: :frames).find_by!(id: id)
+
+          {
+            id: message.id,
+            status: message.status,
+            messageable_type: message.messageable_type,
+            messageable_id: message.messageable_id,
+            created_at: message.created_at.utc.to_s,
+            updated_at: message.updated_at.utc.to_s,
+            exceptions: message.exceptions.map do |exception|
+              {
+                id: exception.id,
+                class_name: exception.class_name,
+                message_text: exception.message_text,
+                created_at: exception.created_at.utc.to_s,
+                frames: exception.frames.map do |frame|
+                  {
+                    id: frame.id,
+                    index: frame.index,
+                    text: frame.text
+                  }
+                end
+              }
+            end
+          }
+        end
+      end
+    end
+
+    def publishing(id:)
+      ActiveRecord::Base.connection_pool.with_connection do
+        ActiveRecord::Base.transaction do
+          message = Models::Message.lock.find_by!(id: id)
+
+          if message.status != Models::Message::Status::QUEUED
+            raise ArgumentError,
+              "cannot transition outboxer message #{message.id} " \
+              "from #{message.status} to #{Models::Message::Status::PUBLISHING}"
+          end
+
+          message.update!(
+            status: Models::Message::Status::PUBLISHING,
+            updated_at: Time.now.utc)
+
+          {
+            id: id,
+            status: message.status,
+            messageable_type: message.messageable_type,
+            messageable_id: message.messageable_id
+          }
+        end
+      end
+    end
+
+    def published(id:)
+      ActiveRecord::Base.connection_pool.with_connection do
+        ActiveRecord::Base.transaction do
+          message = Models::Message.lock.find_by!(id: id)
+
+          if message.status != Models::Message::Status::PUBLISHING
+            raise ArgumentError,
+              "cannot transition outboxer message #{message.id} " \
+              "from #{message.status} to (deleted)"
+          end
+
+          message.exceptions.each { |exception| exception.frames.each(&:delete) }
+          message.exceptions.delete_all
+          message.delete
+
+          { id: id }
+        end
+      end
+    end
+
+    def failed(id:, exception:)
+      ActiveRecord::Base.connection_pool.with_connection do
+        ActiveRecord::Base.transaction do
+          message = Models::Message.order(created_at: :asc).lock.find_by!(id: id)
+
+          if message.status != Models::Message::Status::PUBLISHING
+            raise ArgumentError,
+              "cannot transition outboxer message #{id} " \
+              "from #{message.status} to #{Models::Message::Status::FAILED}"
+          end
+
+          message.update!(
+            status: Models::Message::Status::FAILED,
+            updated_at: Time.now.utc)
+
+          outboxer_exception = message.exceptions.create!(
+            class_name: exception.class.name, message_text: exception.message)
+
+          exception.backtrace.each_with_index do |frame, index|
+            outboxer_exception.frames.create!(index: index, text: frame)
+          end
+
+          { id: id }
+        end
+      end
+    end
+
+    def delete(id:)
+      ActiveRecord::Base.connection_pool.with_connection do
+        ActiveRecord::Base.transaction do
+          message = Models::Message.includes(exceptions: :frames).lock.find_by!(id: id)
+
+          message.exceptions.each { |exception| exception.frames.each(&:delete) }
+          message.exceptions.delete_all
+          message.delete
+
+          { id: id }
+        end
+      end
+    end
+
+    def republish(id:)
+      ActiveRecord::Base.connection_pool.with_connection do
+        ActiveRecord::Base.transaction do
+          message = Models::Message.lock.find_by!(id: id)
+
+          message.update!(
+            status: Models::Message::Status::BACKLOGGED,
+            updated_at: Time.now.utc)
+
+          { id: id }
+        end
+      end
+    end
+
+    def stop_publishing
+      @publishing = false
+    end
+
+    def publish(threads: 5, queue: 10, poll: 1,
+                logger: Logger.new($stdout, level: Logger::INFO),
+                kernel: Kernel, &block)
+      ruby_queue = Queue.new
+
+      @publishing = true
+
+      logger.info "Initializing #{threads} worker threads"
+      ruby_threads = threads.times.map do
+        Thread.new do
+          loop do
+            begin
+              queued_message = ruby_queue.pop
+              break if queued_message.nil?
+
+              queued_at = queued_message[:queued_at]
+
+              message = Message.publishing(id: queued_message[:id])
+              logger.info "Publishing message #{message[:id]} for " \
+                "#{message[:messageable_type]}::#{message[:messageable_id]} " \
+                "in #{(Time.now.utc - queued_at).round(3)}s"
+
+              begin
+                block.call(message)
+              rescue Exception => exception
+                Message.failed(id: message[:id], exception: exception)
+                logger.error "Failed to publish message #{message[:id]} for " \
+                  "#{message[:messageable_type]}::#{message[:messageable_id]} " \
+                  "in #{(Time.now.utc - queued_at).round(3)}s"
+
+                raise
+              end
+
+              Message.published(id: message[:id])
+              logger.info "Published message #{message[:id]} for " \
+                "#{message[:messageable_type]}::#{message[:messageable_id]} " \
+                "in #{(Time.now.utc - queued_at).round(3)}s"
+            rescue StandardError => exception
+              logger.error "#{exception.class}: #{exception.message} " \
+                "in #{(Time.now.utc - queued_at).round(3)}s"
+
+              exception.backtrace.each { |frame| logger.error frame }
+            rescue Exception => exception
+              logger.fatal "#{exception.class}: #{exception.message} " \
+                "in #{(Time.now.utc - queued_at).round(3)}s"
+              exception.backtrace.each { |frame| logger.error frame }
+
+              @publishing = false
+
+              break
+            end
+          end
+        end
+      end
+      logger.info "Initialized #{threads} worker threads"
+
+      logger.info "Queuing up to #{queue} messages every #{poll}s"
+      while @publishing
+        begin
+          messages = []
+
+          queue_available = queue - ruby_queue.length
+          queue_stats = { total: queue, available: queue_available, current: ruby_queue.length }
+          logger.debug "Queue: #{queue_stats}"
+
+          logger.debug "Connection pool: #{ActiveRecord::Base.connection_pool.stat}"
+
+          messages = (queue_available > 0) ? Messages.queue(limit: queue_available) : []
+          logger.debug "Updated #{messages.length} messages from backlogged to queued"
+
+          messages.each do |message|
+            logger.info "Queuing message #{message[:id]} for " \
+              "#{message[:messageable_type]}::#{message[:messageable_id]}"
+
+            ruby_queue.push({ id: message[:id], queued_at: Time.now.utc })
+          end
+
+          logger.debug "Pushed #{messages.length} messages to queue."
+
+          if messages.empty?
+            logger.debug "Sleeping for #{poll} seconds because no messages were queued..."
+
+            kernel.sleep(poll)
+          elsif ruby_queue.length >= queue
+            logger.debug "Sleeping for #{poll} seconds because queue was full..."
+
+            kernel.sleep(poll)
+          end
+        rescue StandardError => exception
+          logger.error "#{exception.class}: #{exception.message}"
+          exception.backtrace.each { |frame| logger.error frame }
 
-    STATUS = {
-      unpublished: "unpublished",
-      publishing: "publishing",
-      failed: "failed"
-    }.freeze
+          kernel.sleep(poll)
+        rescue Exception => exception
+          logger.fatal "#{exception.class}: #{exception.message}"
+          exception.backtrace.each { |frame| logger.fatal frame }
 
-    belongs_to :message, polymorphic: true
+          @publishing = false
+        end
+      end
+      logger.info "Stopped queueing messages"
 
-    has_many :exceptions, -> { order(created_at: :asc) },
-             class_name: "::Outboxer::Exception",
-             dependent: :destroy
+      logger.info "Shutting down #{threads} worker threads"
+      ruby_threads.length.times { ruby_queue.push(nil) }
+      ruby_threads.each(&:join)
+      logger.info "Shut down #{threads} worker threads"
+    end
   end
 end