journaled 6.0.0 → 6.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: cdfecf071b843818742d6fedc649d57e793e9abbd18906b09245124327ac4705
-  data.tar.gz: d92147f78529cbdbeeed3216c4ab14d0cfbcad9d716f3bff4a3f8b6af99c99e7
+  metadata.gz: bb4df3fde5197a7e78aa3adff771292b722d866c0c8ac21bc3b0f1629c760a4a
+  data.tar.gz: 20132d03f33b880433969c3604ca8cf234a54e67797abe38dd78cf5511ee04bf
 SHA512:
-  metadata.gz: eed184d251e7068985f9283bd8e4d855d547eef9b8abe090127eb46306984354583d7051dab90c78d8e351575467ea5a1c9213ffa210438cd016ea766caeb427
-  data.tar.gz: a25aee3b51b832a793a1a1168c9d58660f208787e0ee21af190a95e6b0c6d7de76dd285a2bccd8abf05c48ba3dd726ed9bf71be0b96eeca602ac103feb0d3d0e
+  metadata.gz: 9d540540921c7da3fc35599cb1c8232d0b898dbd7d1816b4b8a1960eaf5afc0b23f6052a6ed9c22e15c92eaf9fc6abe1fe67df73a97751d8442a58c599b56baa
+  data.tar.gz: 1a905b5b5e657f58fa5d9ebdd62defc012033ccbba7d4a0fa9f771fad9b3fc9412a3049da2838a18cef07a8500089161860d6fc4ed15e63111cd03f5728770e6

data/README.md

@@ -22,9 +22,11 @@ durable, eventually consistent record that discrete events happened.
 
 ## Installation
 
-1. If you haven't already,
-[configure ActiveJob](https://guides.rubyonrails.org/active_job_basics.html)
-to use one of the following queue adapters:
+1. **Configure a queue adapter** (only required if using the default ActiveJob delivery adapter):
+
+    If you haven't already,
+    [configure ActiveJob](https://guides.rubyonrails.org/active_job_basics.html)
+    to use one of the following queue adapters:
 
     - `:delayed_job` (via `delayed_job_active_record`)
     - `:que`
@@ -52,6 +54,8 @@ to use one of the following queue adapters:
 
     This configuration isn't necessary for applications running Rails 8+.
 
+    **Note:** If you plan to use the [Outbox-style Event Processing](#outbox-style-event-processing-optional) (Outbox adapter), you can skip this step entirely, as the Outbox adapter does not use ActiveJob.
+
 2. To integrate Journaled into your application, simply include the gem in your
 app's Gemfile.
 
@@ -129,6 +133,37 @@ Journaling provides a number of different configuation options that can be set i
 
   The number of seconds before the :http_handler should timeout while waiting for a HTTP response.
 
+#### `Journaled.delivery_adapter` (default: `Journaled::DeliveryAdapters::ActiveJobAdapter`)
+
+  Determines how events are delivered to Kinesis. Two options are available:
+
+  - **`Journaled::DeliveryAdapters::ActiveJobAdapter`** (default) - Enqueues events to ActiveJob. Requires a DB-backed queue adapter (see Installation).
+
+  - **`Journaled::Outbox::Adapter`** - Stores events in a database table and processes them via separate worker daemons. See [Outbox-style Event Processing](#outbox-style-event-processing-optional) for setup instructions.
+
+  Example:
+  ```ruby
+  # Use the Outbox-style adapter
+  Journaled.delivery_adapter = Journaled::Outbox::Adapter
+  ```
+
+#### `Journaled.outbox_base_class_name` (default: `'ActiveRecord::Base'`)
+
+  **Only relevant when using `Journaled::Outbox::Adapter`.**
+
+  Specifies which ActiveRecord base class the Outbox event storage model (`Journaled::Outbox::Event`) should use for its database connection. This is useful for multi-database setups where you want to store events in a separate database.
+
+  Example:
+  ```ruby
+  # Store outbox events in a separate database
+  class EventsRecord < ActiveRecord::Base
+    self.abstract_class = true
+    connects_to database: { writing: :events, reading: :events }
+  end
+
+  Journaled.outbox_base_class_name = 'EventsRecord'
+  ```
+
 #### ActiveJob `set` options
 
 Both model-level directives accept additional options to be passed into ActiveJob's `set` method:
@@ -143,6 +178,102 @@ has_audit_log enqueue_with: { priority: 30 }
 # Or for custom journaling:
 journal_attributes :email, enqueue_with: { priority: 20, queue: 'journaled' }
 ```
+##### Outbox-style Event Processing (Optional)
+
+Journaled includes a built-in Outbox-style delivery adapter with horizontally scalable workers.
+
+**Setup:**
+
+This feature requires creating database tables and is completely optional. Existing users are unaffected.
+
+1. **Install migrations:**
+
+```bash
+rake journaled:install:migrations
+rails db:migrate
+```
+
+This creates a table for storing events:
+- `journaled_outbox_events` - Queue of events to be processed (includes `failed_at` column for tracking failures)
+
+2. **Configure to use the database adapter:**
+
+```ruby
+# config/initializers/journaled.rb
+
+# Use the Outbox-style adapter instead of ActiveJob
+Journaled.delivery_adapter = Journaled::Outbox::Adapter
+
+# Optional: Customize worker behavior (these are the defaults)
+Journaled.worker_batch_size = 500        # Max events per Kinesis batch (Kinesis API limit)
+Journaled.worker_poll_interval = 5       # Seconds between polls
+```
+
+**Note:** When using the Outbox adapter, you do **not** need to configure an ActiveJob queue adapter (skip step 1 of Installation). The Outbox adapter uses the `journaled_outbox_events` table for event storage and its own worker daemons for processing, making it independent of ActiveJob. Transactional batching still works seamlessly with the Outbox adapter.
+
+3. **Start worker daemon(s):**
+
+```bash
+bundle exec rake journaled_worker:work
+```
+
+4. **Monitoring:**
+
+The system emits `ActiveSupport::Notifications` events:
+
+```ruby
+# config/initializers/journaled.rb
+
+# Emitted for every batch processed (regardless of outcome)
+ActiveSupport::Notifications.subscribe('journaled.worker.batch_process') do |name, start, finish, id, payload|
+  Statsd.increment('journaled.worker.batches', tags: ["worker:#{payload[:worker_id]}"])
+end
+
+# Emitted for successfully sent events
+ActiveSupport::Notifications.subscribe('journaled.worker.batch_sent') do |name, start, finish, id, payload|
+  Statsd.increment('journaled.worker.events_sent', payload[:event_count], tags: ["worker:#{payload[:worker_id]}"])
+end
+
+# Emitted for permanently failed events (marked as failed in database)
+ActiveSupport::Notifications.subscribe('journaled.worker.batch_failed') do |name, start, finish, id, payload|
+  Statsd.increment('journaled.worker.events_failed', payload[:event_count], tags: ["worker:#{payload[:worker_id]}"])
+end
+
+# Emitted for transiently failed events (will be retried)
+ActiveSupport::Notifications.subscribe('journaled.worker.batch_errored') do |name, start, finish, id, payload|
+  Statsd.increment('journaled.worker.events_errored', payload[:event_count], tags: ["worker:#{payload[:worker_id]}"])
+end
+
+# Emitted once per minute with queue statistics
+ActiveSupport::Notifications.subscribe('journaled.worker.queue_metrics') do |name, start, finish, id, payload|
+  Statsd.gauge('journaled.worker.queue.total', payload[:total_count], tags: ["worker:#{payload[:worker_id]}"])
+  Statsd.gauge('journaled.worker.queue.workable', payload[:workable_count], tags: ["worker:#{payload[:worker_id]}"])
+  Statsd.gauge('journaled.worker.queue.erroring', payload[:erroring_count], tags: ["worker:#{payload[:worker_id]}"])
+  Statsd.gauge('journaled.worker.queue.oldest_age_seconds', payload[:oldest_age_seconds], tags: ["worker:#{payload[:worker_id]}"]) if payload[:oldest_age_seconds]
+end
+```
+
+Queue metrics payload includes:
+- `total_count` - Total number of events in the queue (including failed)
+- `workable_count` - Events ready to be processed (not failed)
+- `erroring_count` - Events with errors but not yet marked as permanently failed
+- `oldest_non_failed_timestamp` - Timestamp of the oldest non-failed event (extracted from UUID v7)
+- `oldest_age_seconds` - Age in seconds of the oldest non-failed event
+
+Note: Metrics are collected in a background thread to avoid blocking the main worker loop.
+
+5. **Failed Events:**
+
+Inspect and requeue failed events:
+
+```ruby
+# Find failed events
+Journaled::Outbox::Event.failed.where(stream_name: 'my_stream')
+
+# Requeue a failed event (clears failure info and resets attempts)
+failed_event = Journaled::Outbox::Event.failed.find(123)
+failed_event.requeue!
+```
 
 ### Attribution
 

data/app/jobs/journaled/delivery_job.rb

@@ -2,8 +2,6 @@
 
 module Journaled
   class DeliveryJob < ApplicationJob
-    DEFAULT_REGION = 'us-east-1'
-
     rescue_from(Aws::Kinesis::Errors::InternalFailure, Aws::Kinesis::Errors::ServiceUnavailable, Aws::Kinesis::Errors::Http503Error) do |e|
       Rails.logger.error "Kinesis Error - Server Error occurred - #{e.class}"
       raise KinesisTemporaryFailure
@@ -20,16 +18,6 @@ module Journaled
       journal! if Journaled.enabled?
     end
 
-    def kinesis_client_config
-      {
-        region: ENV.fetch('AWS_DEFAULT_REGION', DEFAULT_REGION),
-        retry_limit: 0,
-        http_idle_timeout: Journaled.http_idle_timeout,
-        http_open_timeout: Journaled.http_open_timeout,
-        http_read_timeout: Journaled.http_read_timeout,
-      }.merge(credentials)
-    end
-
     private
 
     KinesisRecord = Struct.new(:serialized_event, :partition_key, :stream_name, keyword_init: true) do
@@ -51,42 +39,7 @@ module Journaled
     end
 
     def kinesis_client
-      Aws::Kinesis::Client.new(kinesis_client_config)
-    end
-
-    def credentials
-      if ENV.key?('JOURNALED_IAM_ROLE_ARN')
-        {
-          credentials: iam_assume_role_credentials,
-        }
-      else
-        legacy_credentials_hash_if_present
-      end
-    end
-
-    def legacy_credentials_hash_if_present
-      if ENV.key?('RUBY_AWS_ACCESS_KEY_ID')
-        {
-          access_key_id: ENV.fetch('RUBY_AWS_ACCESS_KEY_ID'),
-          secret_access_key: ENV.fetch('RUBY_AWS_SECRET_ACCESS_KEY'),
-        }
-      else
-        {}
-      end
-    end
-
-    def sts_client
-      Aws::STS::Client.new({
-        region: ENV.fetch('AWS_DEFAULT_REGION', DEFAULT_REGION),
-      }.merge(legacy_credentials_hash_if_present))
-    end
-
-    def iam_assume_role_credentials
-      @iam_assume_role_credentials ||= Aws::AssumeRoleCredentials.new(
-        client: sts_client,
-        role_arn: ENV.fetch('JOURNALED_IAM_ROLE_ARN'),
-        role_session_name: "JournaledAssumeRoleAccess",
-      )
+      @kinesis_client ||= KinesisClientFactory.build
     end
 
     class KinesisTemporaryFailure < NotTrulyExceptionalError

data/app/models/journaled/outbox/event.rb

@@ -0,0 +1,60 @@
+# frozen_string_literal: true
+
+module Journaled
+  module Outbox
+    # ActiveRecord model for Outbox-style event processing
+    #
+    # This model is only used when the Outbox::Adapter is configured.
+    # Events are stored in the database and processed by worker daemons.
+    #
+    # Successfully delivered events are deleted immediately.
+    # Failed events are marked with failed_at and can be queried or requeued.
+    class Event < Journaled.outbox_base_class_name.constantize
+      self.table_name = 'journaled_outbox_events'
+
+      self.record_timestamps = false # use db default
+
+      skip_audit_log
+
+      attribute :event_data, :json
+
+      validates :event_type, :event_data, :partition_key, :stream_name, presence: true
+
+      scope :ready_to_process, -> {
+        where(failed_at: nil)
+          .order(:id)
+      }
+
+      scope :failed, -> { where.not(failed_at: nil) }
+
+      # Fetch a batch of events for processing using SELECT FOR UPDATE
+      #
+      # @return [Array<Journaled::Outbox::Event>] Events locked for processing
+      def self.fetch_batch_for_update
+        ready_to_process
+          .limit(Journaled.worker_batch_size)
+          .lock
+          .to_a
+      end
+
+      # Requeue a failed event for processing
+      #
+      # Clears failure information so the event can be retried.
+      #
+      # @return [Boolean] Whether the requeue was successful
+      def requeue!
+        update!(
+          failed_at: nil,
+          failure_reason: nil,
+        )
+      end
+
+      # Get the oldest non-failed event's timestamp
+      #
+      # @return [Time, nil] The timestamp of the oldest event, or nil if no events exist
+      def self.oldest_non_failed_timestamp
+        ready_to_process.order(:id).limit(1).pick(:created_at)
+      end
+    end
+  end
+end

data/app/models/journaled/writer.rb

@@ -42,23 +42,13 @@ class Journaled::Writer
     events.group_by(&:journaled_enqueue_opts).each do |enqueue_opts, batch|
       job_opts = enqueue_opts.reverse_merge(priority: Journaled.job_priority)
       ActiveSupport::Notifications.instrument('journaled.batch.enqueue', batch: batch, **job_opts) do
-        Journaled::DeliveryJob.set(job_opts).perform_later(*delivery_perform_args(batch))
+        Journaled.delivery_adapter.deliver(events: batch, enqueue_opts: job_opts)
 
         batch.each { |event| ActiveSupport::Notifications.instrument('journaled.event.enqueue', event: event, **job_opts) }
       end
     end
   end
 
-  def self.delivery_perform_args(events)
-    events.map do |event|
-      {
-        serialized_event: event.journaled_attributes.to_json,
-        partition_key: event.journaled_partition_key,
-        stream_name: event.journaled_stream_name,
-      }
-    end
-  end
-
   private
 
   attr_reader :journaled_event

data/db/migrate/1_install_uuid_generate_v7.rb

@@ -0,0 +1,27 @@
+# frozen_string_literal: true
+
+class InstallUuidGenerateV7 < ActiveRecord::Migration[7.2]
+  def up
+    # Enable pgcrypto extension for gen_random_bytes()
+    enable_extension 'pgcrypto'
+
+    # Install UUID v7 generation function
+    # Source: https://github.com/Betterment/postgresql-uuid-generate-v7
+    execute <<-SQL.squish
+      CREATE OR REPLACE FUNCTION uuid_generate_v7()
+      RETURNS uuid
+      LANGUAGE plpgsql
+      PARALLEL SAFE
+      AS $$
+      DECLARE
+        unix_time_ms CONSTANT bytea NOT NULL DEFAULT substring(int8send((extract(epoch FROM clock_timestamp()) * 1000)::bigint) from 3);
+        buffer bytea NOT NULL DEFAULT unix_time_ms || gen_random_bytes(10);
+      BEGIN
+        buffer = set_byte(buffer, 6, (b'0111' || get_byte(buffer, 6)::bit(4))::bit(8)::int);
+        buffer = set_byte(buffer, 8, (b'10' || get_byte(buffer, 8)::bit(6))::bit(8)::int);
+        RETURN encode(buffer, 'hex');
+      END
+      $$;
+    SQL
+  end
+end

data/db/migrate/2_create_journaled_events.rb

@@ -0,0 +1,22 @@
+# frozen_string_literal: true
+
+class CreateJournaledEvents < ActiveRecord::Migration[7.2]
+  def change
+    # UUID v7 primary key (auto-generated by database using uuid_generate_v7())
+    create_table :journaled_outbox_events, id: :uuid, default: -> { "uuid_generate_v7()" } do |t|
+      # Event identification and data
+      t.string :event_type, null: false
+      t.jsonb :event_data, null: false
+      t.string :partition_key, null: false
+      t.string :stream_name, null: false
+
+      t.text :failure_reason
+
+      t.datetime :failed_at
+      t.datetime :created_at, null: false, default: -> { "clock_timestamp()" }
+    end
+
+    # Index for querying failed events
+    add_index :journaled_outbox_events, :failed_at
+  end
+end

data/lib/journaled/connection.rb

@@ -22,17 +22,7 @@ module Journaled
       end
 
       def connection
-        if Journaled.queue_adapter.in? %w(delayed delayed_job)
-          Delayed::Job.connection
-        elsif Journaled.queue_adapter == 'good_job'
-          GoodJob::BaseRecord.connection
-        elsif Journaled.queue_adapter == 'que'
-          Que::ActiveRecord::Model.connection
-        elsif Journaled.queue_adapter == 'test' && Rails.env.test?
-          ActiveRecord::Base.connection
-        else
-          raise "Unsupported adapter: #{Journaled.queue_adapter}"
-        end
+        Journaled.delivery_adapter.transaction_connection
       end
     end
 

data/lib/journaled/delivery_adapter.rb

@@ -0,0 +1,41 @@
+# frozen_string_literal: true
+
+module Journaled
+  # Base class for delivery adapters
+  #
+  # Journaled ships with two delivery adapters:
+  #   - Journaled::DeliveryAdapters::ActiveJobAdapter (default) - delivers via ActiveJob
+  #   - Journaled::Outbox::Adapter - delivers via Outbox-style workers
+  #
+  class DeliveryAdapter
+    # Delivers a batch of events
+    #
+    # @param events [Array] Array of journaled events to deliver
+    # @param enqueue_opts [Hash] Options for delivery (priority, queue, wait, wait_until, etc.)
+    # @return [void]
+    def self.deliver(events:, enqueue_opts:) # rubocop:disable Lint/UnusedMethodArgument
+      raise NoMethodError, "#{name} must implement .deliver(events:, enqueue_opts:)"
+    end
+
+    # Returns the database connection to use for transactional batching
+    #
+    # This allows delivery adapters to specify which database connection should be used
+    # when staging events during a transaction. This is only needed if you want to support
+    # transactional batching with your adapter.
+    #
+    # @return [ActiveRecord::ConnectionAdapters::AbstractAdapter]
+    def self.transaction_connection
+      raise NoMethodError, "#{name} must implement .transaction_connection"
+    end
+
+    # Validates that the adapter is properly configured
+    #
+    # Called during Rails initialization in production mode. Raise an error if the adapter
+    # is not configured correctly (e.g., missing required dependencies, invalid configuration).
+    #
+    # @return [void]
+    def self.validate_configuration!
+      # Default: no validation required
+    end
+  end
+end

data/lib/journaled/delivery_adapters/active_job_adapter.rb

@@ -0,0 +1,75 @@
+# frozen_string_literal: true
+
+module Journaled
+  module DeliveryAdapters
+    # Default delivery adapter that uses ActiveJob
+    #
+    # This adapter enqueues events to Journaled::DeliveryJob which
+    # sends them to Kinesis. This is the default behavior and maintains
+    # backward compatibility with previous versions of the gem.
+    class ActiveJobAdapter < Journaled::DeliveryAdapter
+      # Delivers events by enqueueing them to Journaled::DeliveryJob
+      #
+      # @param events [Array] Array of journaled events to deliver
+      # @param enqueue_opts [Hash] Options for ActiveJob (priority, queue, wait, wait_until, etc.)
+      # @return [void]
+      def self.deliver(events:, enqueue_opts:)
+        Journaled::DeliveryJob.set(enqueue_opts).perform_later(*delivery_perform_args(events))
+      end
+
+      # Serializes events into the format expected by DeliveryJob
+      #
+      # @param events [Array] Array of journaled events
+      # @return [Array<Hash>] Array of serialized event hashes
+      def self.delivery_perform_args(events)
+        events.map do |event|
+          {
+            serialized_event: event.journaled_attributes.to_json,
+            partition_key: event.journaled_partition_key,
+            stream_name: event.journaled_stream_name,
+          }
+        end
+      end
+
+      # Returns the database connection to use for transactional batching
+      #
+      # This is determined by the configured queue adapter, since ActiveJob
+      # enqueues jobs to the same database that should be used for transactions.
+      #
+      # @return [ActiveRecord::ConnectionAdapters::AbstractAdapter] The connection to use
+      def self.transaction_connection
+        queue_adapter = Journaled.queue_adapter
+
+        if queue_adapter.in? %w(delayed delayed_job)
+          Delayed::Job.connection
+        elsif queue_adapter == 'good_job'
+          GoodJob::BaseRecord.connection
+        elsif queue_adapter == 'que'
+          Que::ActiveRecord::Model.connection
+        elsif queue_adapter == 'test' && Rails.env.test?
+          ActiveRecord::Base.connection
+        else
+          raise "Unsupported queue adapter: #{queue_adapter}"
+        end
+      end
+
+      # Validates that a supported queue adapter is configured
+      #
+      # @return [void]
+      def self.validate_configuration!
+        unless Journaled::SUPPORTED_QUEUE_ADAPTERS.include?(Journaled.queue_adapter)
+          raise <<~MSG
+            Journaled has detected an unsupported ActiveJob queue adapter: `:#{Journaled.queue_adapter}`
+
+            Journaled jobs must be enqueued transactionally to your primary database.
+
+            Please install the appropriate gems and set `queue_adapter` to one of the following:
+            #{Journaled::SUPPORTED_QUEUE_ADAPTERS.map { |a| "- `:#{a}`" }.join("\n")}
+
+            Read more at https://github.com/Betterment/journaled
+          MSG
+        end
+      end
+    end
+  end
+end

data/lib/journaled/engine.rb

@@ -2,9 +2,11 @@
 
 module Journaled
   class Engine < ::Rails::Engine
+    engine_name 'journaled'
+
     config.after_initialize do
       ActiveSupport.on_load(:active_job) do
-        Journaled.detect_queue_adapter! unless Journaled.development_or_test?
+        Journaled.delivery_adapter.validate_configuration! unless Journaled.development_or_test?
       end
 
       ActiveSupport.on_load(:active_record) do

data/lib/journaled/kinesis_batch_sender.rb

@@ -0,0 +1,98 @@
+# frozen_string_literal: true
+
+module Journaled
+  # Sends batches of events to Kinesis using the PutRecord single-event API
+  #
+  # This class handles:
+  # - Sending events individually to support guaranteed ordering
+  # - Handling failures on a per-event basis
+  # - Classifying errors as transient vs permanent
+  #
+  # Returns structured results for the caller to handle event state management.
+  class KinesisBatchSender
+    FailedEvent = Struct.new(:event, :error_code, :error_message, :transient, keyword_init: true) do
+      def transient?
+        transient
+      end
+
+      def permanent?
+        !transient
+      end
+    end
+
+    PERMANENT_ERROR_CLASSES = [
+      Aws::Kinesis::Errors::ValidationException,
+    ].freeze
+
+    # Send a batch of database events to Kinesis
+    #
+    # Sends events one at a time to guarantee ordering. Stops on first transient failure.
+    #
+    # @param events [Array<Journaled::Outbox::Event>] Events to send
+    # @return [Hash] Result with:
+    #   - succeeded: Array of successfully sent events
+    #   - failed: Array of FailedEvent structs (only permanent failures)
+    def send_batch(events)
+      result = { succeeded: [], failed: [] }
+
+      events.each do |event|
+        event_result = send_event(event)
+        if event_result.is_a?(FailedEvent)
+          if event_result.transient?
+            emit_transient_failure_metric
+            break
+          else
+            result[:failed] << event_result
+          end
+        else
+          result[:succeeded] << event_result
+        end
+      end
+
+      result
+    end
+
+    private
+
+    # Send a single event to Kinesis
+    #
+    # @param event [Journaled::Outbox::Event] Event to send
+    # @return [Journaled::Outbox::Event, FailedEvent] The event on success, or FailedEvent on failure
+    def send_event(event)
+      # Merge the DB-generated ID into the event data before sending to Kinesis
+      event_data_with_id = event.event_data.merge(id: event.id)
+
+      kinesis_client.put_record(
+        stream_name: event.stream_name,
+        data: event_data_with_id.to_json,
+        partition_key: event.partition_key,
+      )
+
+      event
+    rescue *PERMANENT_ERROR_CLASSES => e
+      Rails.logger.error("Kinesis event send failed (permanent): #{e.class} - #{e.message}")
+      FailedEvent.new(
+        event:,
+        error_code: e.class.to_s,
+        error_message: e.message,
+        transient: false,
+      )
+    rescue StandardError => e
+      Rails.logger.error("Kinesis event send failed (transient): #{e.class} - #{e.message}")
+      FailedEvent.new(
+        event:,
+        error_code: e.class.to_s,
+        error_message: e.message,
+        transient: true,
+      )
+    end
+
+    def kinesis_client
+      @kinesis_client ||= KinesisClientFactory.build
+    end
+
+    def emit_transient_failure_metric
+      ActiveSupport::Notifications.instrument('journaled.kinesis_batch_sender.transient_failure')
+    end
+  end
+end

data/lib/journaled/kinesis_client_factory.rb

@@ -0,0 +1,62 @@
+# frozen_string_literal: true
+
+module Journaled
+  class KinesisClientFactory
+    DEFAULT_REGION = 'us-east-1'
+
+    def self.build
+      new.client
+    end
+
+    def client
+      @client ||= Aws::Kinesis::Client.new(config)
+    end
+
+    private
+
+    def config
+      {
+        region: ENV.fetch('AWS_DEFAULT_REGION', DEFAULT_REGION),
+        retry_limit: 0,
+        http_idle_timeout: Journaled.http_idle_timeout,
+        http_open_timeout: Journaled.http_open_timeout,
+        http_read_timeout: Journaled.http_read_timeout,
+      }.merge(credentials)
+    end
+
+    def credentials
+      if ENV.key?('JOURNALED_IAM_ROLE_ARN')
+        {
+          credentials: iam_assume_role_credentials,
+        }
+      else
+        legacy_credentials_hash_if_present
+      end
+    end
+
+    def legacy_credentials_hash_if_present
+      if ENV.key?('RUBY_AWS_ACCESS_KEY_ID')
+        {
+          access_key_id: ENV.fetch('RUBY_AWS_ACCESS_KEY_ID'),
+          secret_access_key: ENV.fetch('RUBY_AWS_SECRET_ACCESS_KEY'),
+        }
+      else
+        {}
+      end
+    end
+
+    def sts_client
+      @sts_client ||= Aws::STS::Client.new({
+        region: ENV.fetch('AWS_DEFAULT_REGION', DEFAULT_REGION),
+      }.merge(legacy_credentials_hash_if_present))
+    end
+
+    def iam_assume_role_credentials
+      @iam_assume_role_credentials ||= Aws::AssumeRoleCredentials.new(
+        client: sts_client,
+        role_arn: ENV.fetch('JOURNALED_IAM_ROLE_ARN'),
+        role_session_name: "JournaledAssumeRoleAccess",
+      )
+    end
+  end
+end

data/lib/journaled/outbox/adapter.rb

@@ -0,0 +1,96 @@
+# frozen_string_literal: true
+
+module Journaled
+  module Outbox
+    # Outbox-style delivery adapter for custom event processing
+    #
+    # This adapter stores events in a database table instead of enqueuing to ActiveJob.
+    # Events are processed by separate worker daemons that poll the database.
+    #
+    # Setup:
+    # 1. Generate migrations: rails generate journaled:database_events
+    # 2. Run migrations: rails db:migrate
+    # 3. Configure: Journaled.delivery_adapter = Journaled::Outbox::Adapter
+    # 4. Start workers: bundle exec rake journaled_worker:work
+    class Adapter < Journaled::DeliveryAdapter
+      class TableNotFoundError < StandardError; end
+
+      # Delivers events by inserting them into the database
+      #
+      # @param events [Array] Array of journaled events to deliver
+      # @param ** [Hash] Additional options (ignored, for interface compatibility)
+      # @return [void]
+      def self.deliver(events:, **)
+        check_table_exists!
+
+        records = events.map do |event|
+          # Exclude the application-level id - the database will generate its own using uuid_generate_v7()
+          event_data = event.journaled_attributes.except(:id)
+
+          {
+            event_type: event.journaled_attributes[:event_type],
+            event_data:,
+            partition_key: event.journaled_partition_key,
+            stream_name: event.journaled_stream_name,
+          }
+        end
+
+        # rubocop:disable Rails/SkipsModelValidations
+        Event.insert_all(records) if records.any?
+        # rubocop:enable Rails/SkipsModelValidations
+      end
+
+      # Check if the required database table exists
+      #
+      # @raise [TableNotFoundError] if the table doesn't exist
+      def self.check_table_exists!
+        return if @table_exists
+
+        unless Event.table_exists?
+          raise TableNotFoundError, <<~ERROR
+            Journaled::Outbox::Adapter requires the 'journaled_outbox_events' table.
+
+            To create the required tables, run:
+
+              rake journaled:install:migrations
+              rails db:migrate
+
+            For more information, see the README:
+            https://github.com/Betterment/journaled#outbox-style-event-processing-optional
+          ERROR
+        end
+
+        @table_exists = true
+      end
+
+      # Returns the database connection to use for transactional batching
+      #
+      # The Outbox adapter uses the same database as the Outbox events table,
+      # since events are staged in memory and then written to journaled_events
+      # within the same transaction.
+      #
+      # @return [ActiveRecord::ConnectionAdapters::AbstractAdapter] The connection to use
+      def self.transaction_connection
+        Event.connection
+      end
+
+      # Validates that PostgreSQL is being used
+      #
+      # The Outbox adapter requires PostgreSQL for UUID v7 support and row-level locking
+      #
+      # @raise [StandardError] if the database adapter is not PostgreSQL
+      def self.validate_configuration!
+        return if Event.connection.adapter_name == 'PostgreSQL'
+
+        raise <<~ERROR
+          Journaled::Outbox::Adapter requires PostgreSQL database adapter.
+
+          Current adapter: #{Event.connection.adapter_name}
+
+          The Outbox pattern uses PostgreSQL-specific features like UUID v7 generation
+          and row-level locking for distributed worker coordination. Other databases are not supported.
+        ERROR
+      end
+    end
+  end
+end

data/lib/journaled/outbox/batch_processor.rb

@@ -0,0 +1,86 @@
+# frozen_string_literal: true
+
+module Journaled
+  module Outbox
+    # Processes batches of outbox events
+    #
+    # This class handles the core business logic of:
+    # - Fetching events from the database (with FOR UPDATE)
+    # - Sending them to Kinesis one at a time to guarantee ordering
+    # - Handling successful deliveries (deleting events)
+    # - Handling permanent failures (marking with failed_at)
+    # - Handling ephemeral failures (stopping processing and committing)
+    #
+    # Events are processed one at a time to guarantee ordering. If an event fails
+    # with an ephemeral error, processing stops and the transaction commits
+    # (deleting successes and marking permanent failures), then the loop re-enters.
+    #
+    # All operations happen within a single database transaction for consistency.
+    # The Worker class delegates to this for actual event processing.
+    class BatchProcessor
+      def initialize
+        @batch_sender = KinesisBatchSender.new
+      end
+
+      # Process a single batch of events
+      #
+      # Wraps the entire batch processing in a single transaction:
+      # 1. SELECT FOR UPDATE (claim events)
+      # 2. Send to Kinesis (batch sender handles one-at-a-time and short-circuiting)
+      # 3. Delete successful events
+      # 4. Mark failed events (batch sender only returns permanent failures)
+      #
+      # @return [Hash] Statistics with :succeeded, :failed_permanently counts
+      def process_batch
+        ActiveRecord::Base.transaction do
+          events = Event.fetch_batch_for_update
+          Rails.logger.info("[journaled] Processing batch of #{events.count} events")
+
+          result = batch_sender.send_batch(events)
+
+          # Delete successful events
+          Event.where(id: result[:succeeded].map(&:id)).delete_all if result[:succeeded].any?
+
+          # Mark failed events
+          mark_events_as_failed(result[:failed]) if result[:failed].any?
+
+          Rails.logger.info(
+            "[journaled] Batch complete: #{result[:succeeded].count} succeeded, " \
+            "#{result[:failed].count} marked as failed (batch size: #{events.count})",
+          )
+
+          {
+            succeeded: result[:succeeded].count,
+            failed_permanently: result[:failed].count,
+          }
+        end
+      end
+
+      private
+
+      attr_reader :batch_sender
+
+      # Mark events as permanently failed
+      # Sets: failed_at = NOW, failure_reason = per-event message
+      def mark_events_as_failed(failed_events)
+        now = Time.current
+
+        records = failed_events.map do |failed_event|
+          failed_event.event.attributes.except('created_at').merge(
+            failed_at: now,
+            failure_reason: "#{failed_event.error_code}: #{failed_event.error_message}",
+          )
+        end
+
+        # rubocop:disable Rails/SkipsModelValidations
+        Event.upsert_all(
+          records,
+          unique_by: :id,
+          on_duplicate: :update,
+          update_only: %i(failed_at failure_reason),
+        )
+        # rubocop:enable Rails/SkipsModelValidations
+      end
+    end
+  end
+end

data/lib/journaled/outbox/metric_emitter.rb

@@ -0,0 +1,84 @@
+# frozen_string_literal: true
+
+module Journaled
+  module Outbox
+    # Handles metric emission for the Worker
+    #
+    # This class is responsible for collecting and emitting metrics about the outbox queue.
+    class MetricEmitter
+      def initialize(worker_id:)
+        @worker_id = worker_id
+      end
+
+      # Emit batch processing metrics
+      #
+      # @param stats [Hash] Processing statistics with :succeeded, :failed_permanently
+      def emit_batch_metrics(stats)
+        total_events = stats[:succeeded] + stats[:failed_permanently]
+
+        emit_metric('journaled.worker.batch_process', value: total_events)
+        emit_metric('journaled.worker.batch_sent', value: stats[:succeeded])
+        emit_metric('journaled.worker.batch_failed', value: stats[:failed_permanently])
+      end
+
+      # Collect and emit queue metrics
+      #
+      # This calculates various queue statistics and emits individual metrics for each.
+      def emit_queue_metrics
+        metrics = calculate_queue_metrics
+
+        emit_metric('journaled.worker.queue_total_count', value: metrics[:total_count])
+        emit_metric('journaled.worker.queue_workable_count', value: metrics[:workable_count])
+        emit_metric('journaled.worker.queue_erroring_count', value: metrics[:erroring_count])
+        emit_metric('journaled.worker.queue_oldest_age_seconds', value: metrics[:oldest_age_seconds])
+
+        Rails.logger.info(
+          "Queue metrics: total=#{metrics[:total_count]}, " \
+          "workable=#{metrics[:workable_count]}, " \
+          "erroring=#{metrics[:erroring_count]}, " \
+          "oldest_age=#{metrics[:oldest_age_seconds].round(2)}s",
+        )
+      end
+
+      private
+
+      attr_reader :worker_id
+
+      # Emit a single metric notification
+      #
+      # @param event_name [String] The name of the metric event
+      # @param payload [Hash] Additional payload data (event_count, value, etc.)
+      def emit_metric(event_name, payload)
+        ActiveSupport::Notifications.instrument(
+          event_name,
+          payload.merge(worker_id:),
+        )
+      end
+
+      # Calculate queue metrics
+      #
+      # @return [Hash] Metrics including counts and oldest event timestamp
+      def calculate_queue_metrics
+        # Use a single query with COUNT(*) FILTER to calculate all counts in one table scan
+        result = Event.connection.select_one(
+          Event.select(
+            'COUNT(*) AS total_count',
+            'COUNT(*) FILTER (WHERE failed_at IS NULL) AS workable_count',
+            'COUNT(*) FILTER (WHERE failure_reason IS NOT NULL AND failed_at IS NULL) AS erroring_count',
+            'MIN(created_at) FILTER (WHERE failed_at IS NULL) AS oldest_non_failed_timestamp',
+          ).to_sql,
+        )
+
+        oldest_timestamp = result['oldest_non_failed_timestamp']
+        oldest_age_seconds = oldest_timestamp ? Time.current - oldest_timestamp : 0
+
+        {
+          total_count: result['total_count'],
+          workable_count: result['workable_count'],
+          erroring_count: result['erroring_count'],
+          oldest_age_seconds:,
+        }
+      end
+    end
+  end
+end

data/lib/journaled/outbox/worker.rb

@@ -0,0 +1,135 @@
+# frozen_string_literal: true
+
+module Journaled
+  module Outbox
+    # Worker daemon for processing Outbox-style events
+    #
+    # This worker polls the database for pending events and sends them to Kinesis in batches.
+    # Multiple workers can run concurrently and will coordinate using row-level locking.
+    #
+    # The Worker handles the daemon lifecycle (start/stop, signal handling, run loop) and
+    # delegates actual batch processing to BatchProcessor.
+    #
+    # Usage:
+    #   worker = Journaled::Outbox::Worker.new
+    #   worker.start  # Blocks until shutdown signal received
+    class Worker
+      def initialize
+        @worker_id = "#{Socket.gethostname}-#{Process.pid}"
+        self.running = false
+        @processor = BatchProcessor.new
+        @metric_emitter = MetricEmitter.new(worker_id: @worker_id)
+        self.shutdown_requested = false
+        @last_metrics_emission = Time.current
+      end
+
+      # Start the worker (blocks until shutdown)
+      def start
+        check_prerequisites!
+
+        self.running = true
+        Rails.logger.info("Journaled worker starting (id: #{worker_id})")
+
+        setup_signal_handlers
+
+        run_loop
+      ensure
+        self.running = false
+        Rails.logger.info("Journaled worker stopped (id: #{worker_id})")
+      end
+
+      # Request graceful shutdown
+      def shutdown
+        self.shutdown_requested = true
+      end
+
+      # Check if worker is still running
+      def running?
+        running
+      end
+
+      private
+
+      attr_reader :worker_id, :processor, :metric_emitter
+      attr_accessor :shutdown_requested, :running, :last_metrics_emission
+
+      def run_loop
+        loop do
+          if shutdown_requested
+            Rails.logger.info("Shutdown requested for worker #{worker_id}")
+            break
+          end
+
+          events_processed = 0
+          begin
+            events_processed = process_batch
+            emit_metrics_if_needed
+          rescue StandardError => e
+            Rails.logger.error("Worker error: #{e.class} - #{e.message}")
+            Rails.logger.error(e.backtrace.join("\n"))
+          end
+
+          break if shutdown_requested
+
+          # Only sleep if no events were processed to prevent excessive polling on empty table
+          sleep(Journaled.worker_poll_interval) if events_processed.zero?
+        end
+      end
+
+      def process_batch
+        stats = processor.process_batch
+
+        instrument_batch_results(stats)
+
+        stats[:succeeded] + stats[:failed_permanently]
+      end
+
+      def instrument_batch_results(stats)
+        metric_emitter.emit_batch_metrics(stats)
+      end
+
+      def check_prerequisites!
+        unless Event.table_exists?
+          raise <<~ERROR
+            The 'journaled_outbox_events' table does not exist.
+
+            To create the required table, run:
+
+              rails generate journaled:database_events
+              rails db:migrate
+          ERROR
+        end
+
+        Rails.logger.info("Prerequisites check passed")
+      end
+
+      def setup_signal_handlers
+        %w(INT TERM).each do |signal|
+          Signal.trap(signal) do
+            shutdown
+          end
+        end
+      end
+
+      # Emit metrics if the interval has elapsed
+      def emit_metrics_if_needed
+        return unless Time.current - last_metrics_emission >= 60
+
+        # Collect and emit metrics in a background thread to avoid blocking the main loop
+        Thread.new do
+          collect_and_emit_metrics
+        rescue StandardError => e
+          Rails.logger.error("Error collecting metrics: #{e.class} - #{e.message}")
+          Rails.logger.error(e.backtrace.join("\n"))
+        end
+
+        self.last_metrics_emission = Time.current
+      end
+
+      # Collect and emit queue metrics
+      def collect_and_emit_metrics
+        metric_emitter.emit_queue_metrics
+      end
+    end
+  end
+end

data/lib/journaled/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Journaled
-  VERSION = "6.0.0"
+  VERSION = "6.2.0"
 end

data/lib/journaled.rb

@@ -8,6 +8,14 @@ require "journaled/engine"
 require "journaled/current"
 require "journaled/errors"
 require 'journaled/connection'
+require 'journaled/delivery_adapter'
+require 'journaled/delivery_adapters/active_job_adapter'
+require 'journaled/outbox/adapter'
+require 'journaled/kinesis_client_factory'
+require 'journaled/kinesis_batch_sender'
+require 'journaled/outbox/batch_processor'
+require 'journaled/outbox/metric_emitter'
+require 'journaled/outbox/worker'
 
 module Journaled
   SUPPORTED_QUEUE_ADAPTERS = %w(delayed delayed_job good_job que).freeze
@@ -18,8 +26,14 @@ module Journaled
   mattr_accessor(:http_open_timeout) { 2 }
   mattr_accessor(:http_read_timeout) { 60 }
   mattr_accessor(:job_base_class_name) { 'ActiveJob::Base' }
+  mattr_accessor(:outbox_base_class_name) { 'ActiveRecord::Base' }
+  mattr_accessor(:delivery_adapter) { Journaled::DeliveryAdapters::ActiveJobAdapter }
   mattr_writer(:transactional_batching_enabled) { true }
 
+  # Worker configuration (for Outbox-style event processing)
+  mattr_accessor(:worker_batch_size) { 1000 }
+  mattr_accessor(:worker_poll_interval) { 1 } # seconds
+
   def self.transactional_batching_enabled?
     Thread.current[:journaled_transactional_batching_enabled] || @@transactional_batching_enabled
   end
@@ -56,21 +70,6 @@ module Journaled
     job_base_class_name.constantize.queue_adapter_name
   end
 
-  def self.detect_queue_adapter!
-    unless SUPPORTED_QUEUE_ADAPTERS.include?(queue_adapter)
-      raise <<~MSG
-        Journaled has detected an unsupported ActiveJob queue adapter: `:#{queue_adapter}`
-
-        Journaled jobs must be enqueued transactionally to your primary database.
-
-        Please install the appropriate gems and set `queue_adapter` to one of the following:
-        #{SUPPORTED_QUEUE_ADAPTERS.map { |a| "- `:#{a}`" }.join("\n")}
-
-        Read more at https://github.com/Betterment/journaled
-      MSG
-    end
-  end
-
   def self.tagged(**tags)
     existing_tags = Current.tags
     tag!(**tags)

data/lib/tasks/journaled_worker.rake

@@ -0,0 +1,8 @@
+# frozen_string_literal: true
+
+namespace :journaled_worker do
+  desc "Start a Journaled worker to process Outbox-style events"
+  task work: :environment do
+    Journaled::Outbox::Worker.new.start
+  end
+end

metadata

@@ -1,17 +1,16 @@
 --- !ruby/object:Gem::Specification
 name: journaled
 version: !ruby/object:Gem::Version
-  version: 6.0.0
+  version: 6.2.0
 platform: ruby
 authors:
 - Jake Lipson
 - Corey Alexander
 - Cyrus Eslami
 - John Mileham
-autorequire:
 bindir: bin
 cert_chain: []
-date: 2025-01-24 00:00:00.000000000 Z
+date: 1980-01-02 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activejob
@@ -89,7 +88,7 @@ dependencies:
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: '7.0'
+        version: '7.2'
     - - "<"
       - !ruby/object:Gem::Version
         version: '8.1'
@@ -99,7 +98,7 @@ dependencies:
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: '7.0'
+        version: '7.2'
     - - "<"
       - !ruby/object:Gem::Version
         version: '8.1'
@@ -255,9 +254,12 @@ files:
 - app/models/journaled/event.rb
 - app/models/journaled/json_schema_model/validator.rb
 - app/models/journaled/not_truly_exceptional_error.rb
+- app/models/journaled/outbox/event.rb
 - app/models/journaled/writer.rb
 - config/initializers/change_protection.rb
 - config/spring.rb
+- db/migrate/1_install_uuid_generate_v7.rb
+- db/migrate/2_create_journaled_events.rb
 - journaled_schemas/base_event.json
 - journaled_schemas/journaled/audit_log/event.json
 - journaled_schemas/journaled/change.json
@@ -266,12 +268,21 @@ files:
 - lib/journaled/audit_log.rb
 - lib/journaled/connection.rb
 - lib/journaled/current.rb
+- lib/journaled/delivery_adapter.rb
+- lib/journaled/delivery_adapters/active_job_adapter.rb
 - lib/journaled/engine.rb
 - lib/journaled/errors.rb
+- lib/journaled/kinesis_batch_sender.rb
+- lib/journaled/kinesis_client_factory.rb
+- lib/journaled/outbox/adapter.rb
+- lib/journaled/outbox/batch_processor.rb
+- lib/journaled/outbox/metric_emitter.rb
+- lib/journaled/outbox/worker.rb
 - lib/journaled/relation_change_protection.rb
 - lib/journaled/rspec.rb
 - lib/journaled/transaction_ext.rb
 - lib/journaled/version.rb
+- lib/tasks/journaled_worker.rake
 homepage: http://github.com/Betterment/journaled
 licenses:
 - MIT
@@ -311,8 +322,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.5.23
-signing_key:
+rubygems_version: 3.6.8
 specification_version: 4
 summary: Journaling for Betterment apps.
 test_files: []