tobox 0.4.4 → 0.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 6df543cb30f2b30d82096fe3a8660207df0db55fe09f5d357c2b12258973c02d
-  data.tar.gz: 6bc6da61d457aa1a152da546b58d5229ac842d4efc039ae612bae71eb76d34c4
+  metadata.gz: '0481835fffcdbaf85fb53ebf116a6b117ad267cfb4a4efbcbae05b2f87058951'
+  data.tar.gz: 39589407d92d28fc538aebb59a039a606d833aeb0eb26f82b7cf96228e73d5b3
 SHA512:
-  metadata.gz: afe16575fd95b63ed164188005073ccc3eb72e0e8dc2933a50da8553d952860235382d8d5b85d3ed2da522fb6a80510af1aaf69d05df79295bd65de7b8430c51
-  data.tar.gz: d4a408a481f1f3e556b79e7344f71fe359db9d7b604d03858148f148d1ad9a9b86381ed49cded366b0807b2614e11d7c7e02c430045440876d3e3c2d1f22a1ba
+  metadata.gz: 8a1137598cc58cb22573ccbec80f448aa55db3505a97b741cc146eaef6f16392ba3159b9db9f61498cea530dba33c431bba4f9c847325064a770a96553d9ec20
+  data.tar.gz: 274be354130a8ae5ab59188ad98c4de208adac78cd7c9720b569db98de75d2ec0514a2560db05c080b9118e6967b573b7ea1fd11b0297edf9c07f05478acbefe

data/CHANGELOG.md

@@ -1,5 +1,28 @@
 ## [Unreleased]
 
+## [0.5.0] - 2024-09-16
+
+### Features
+
+A new `:progress` plugin can be used in order to release database transactions before handling events (useful for when event handling times vary and may cause transaction bookkeeping overhead in the database).
+
+**Note**: This may become the default behaviour in a future release.
+
+### Improvements
+
+* The event grouping and inbox capabilities were converted into plugins (respectively, `:event_grouping` and `:inbox`).
+
+### Bugfixes
+
+* exponential backoff calculation was broken.
+* behaviour fixed for databases which do not support `ON CONFLICT` or `UPDATE ... RETURNING` (like MySQL).
+
+## [0.4.5] - 2024-02-28
+
+### Bugfixes
+
+Fixed latest barrage of `ddtrace` discontinued config APIs, which is still happening under minor releases.
+
 ## [0.4.4] - 2023-07-26
 
 ### Improvements

data/README.md

@@ -13,10 +13,10 @@ Simple, data-first events processing framework based on the [transactional outbo
 - [Usage](#usage)
 - [Configuration](#configuration)
 - [Event](#event)
-- [Features](#features)
-  - [Ordered event processing](#ordered-event-processing)
-  - [Inbox](#inbox)
 - [Plugins](#plugins)
+  - [Progress](#progress)
+  - [Event Grouping](#event-grouping)
+  - [Inbox](#inbox)
   - [Zeitwerk](#zeitwerk)
   - [Sentry](#sentry)
   - [Datadog](#datadog)
@@ -204,7 +204,17 @@ table :outbox
 Maximum number of times a failed attempt to process an event will be retried (`10` by default).
 
 ```ruby
-concurrency 4
+max_attempts 4
+```
+
+**Note**: the new attempt will be retried in `n ** 4`, where `n` is the number of past attempts for that event.
+
+### `exponential_retry_factor`
+
+Factor by which the number of seconds until an event can be retried will be exponentially calculated, i.e. 2 seconds on first attempt, then 4, then 8, then 16 (`2` by default).
+
+```ruby
+exponential_retry_factor 2
 ```
 
 **Note**: the new attempt will be retried in `n ** 4`, where `n` is the number of past attempts for that event.
@@ -320,17 +330,6 @@ Overrides the internal logger (an instance of `Logger`).
 
 Overrides the default log level ("info" when in "production" environment, "debug" otherwise).
 
-### group_column
-
-Defines the column to be used for event grouping, when [ordered processing of events is a requirement](#ordered-event-processing).
-
-### inbox table
-
-Defines the name of the table to be used for inbox, when [inbox usage is a requirement](#inbox).
-
-### inbox column
-
-Defines the column in the outbox table which references the inbox table, when one is set.
 
 <a id="markdown-event" name="event"></a>
 ## Event
@@ -345,36 +344,85 @@ The event is composed of the following properties:
 
 (*NOTE*: The event is also composed of other properties which are only relevant for `tobox`.)
 
-<a id="markdown-features" name="features"></a>
-## Features
 
-There are a few extra features you can run on top a "vanilla" transactional outbox implementation. This is how you can accomplish them using `tobox`.
+<a id="markdown-plugins" name="plugins"></a>
+## Plugins
+
+`tobox` ships with a very simple plugin system. (TODO: add docs).
+
+Plugins can be loaded in the config via `plugin`:
+
+```ruby
+# tobox.rb
+plugin(:plugin_name)
+```
+
+<a id="markdown-progress" name="progress"></a>
+### Progress
+
+By default, the database transaction used to consume the event is kept open while the event is handled. While this ensures atomic event consumption, it may also cause overhead related to transaction management given enough load, particularly in cases where event handling time varies (i.e. throttled HTTP requests).
+
+The `:progress` plugin fixes this by releasing the databaase transaction after fetching the event. It does so by making the fetched event "invisible" for a certain period, during which the event must be successfully handled.
+
+Here's how to use it:
+
+```ruby
+# in your tobox.rb
+plugin :progress
+
+visibility_timeout 90 # default: 30
+```
+
+3. insert related outbox events with the same group id
+
+```ruby
+order = Order.new(
+  item_id: item.id,
+  price: 20_20,
+  currency: "EUR"
+)
+DB.transaction do
+  order.save
+  DB[:outbox].insert(event_type: "order_created", group_id: order.id, data_after: order.to_hash)
+  DB[:outbox].insert(event_type: "billing_event_started", group_id: order.id, data_after: order.to_hash)
+end
+
+# "order_created" will be processed first
+# "billing_event_created" will only start processing once "order_created" finishes
+```
+
+#### Configuration
 
-<a id="markdown-ordered-event-processing" name="ordered-event-processing"></a>
-### Ordered event processing
+##### `visibility_timeout`
+
+Timeout (in seconds) after which a previously marked-for-consumption event can be retried (default: `30`)
+
+<a id="markdown-event-grouping" name="event-grouping"></a>
+### Event grouping
 
 By default, events are taken and processed from the "outbox" table concurrently by workers, which means that, while worker A may process the most recent event, and worker B takes the following, worker B may process it faster than worker A. This may be an issue if the consumer expects events from a certain context to arrive in a certain order.
 
-One solution is to have a single worker processing the "outbox" events. Another is to use the `group_column` configuration.
+One solution is to have a single worker processing the "outbox" events. Another is to use the `:event_grouping` plugin.
 
-What you have to do is:
+All you have to do is:
 
 1. add a "group id" column to the "outbox" table
 
 ```ruby
 create_table(:outbox) do
   primary_key :id
-  column :group_id, :integer
-  # The type is irrelevant, could also be :string, :uuid...
+  column :group_id, :integer # The type is irrelevant, could also be :string, :uuid...
   # ..
+  index :group_id
 ```
 
-2. set the "group_column" configuration
+2. Enable the plugin
 
 ```ruby
 # in your tobox.rb
-group_column :group_id
-index :group_id
+plugin :event_grouping
+
+group_column :group_id # by default already `:group_id`
 ```
 
 3. insert related outbox events with the same group id
@@ -394,25 +442,28 @@ end
 # "order_created" will be processed first
 # "billing_event_created" will only start processing once "order_created" finishes
 ```
+
+#### Configuration
+
+##### `group_column`
+
+Defines the database column to be used for event grouping (`:group_id` by default).
+
 <a id="inbox" name="inbox"></a>
 ### Inbox
 
-`tobox` also supports the [inbox pattern](https://event-driven.io/en/outbox_inbox_patterns_and_delivery_guarantees_explained/), to ensure "exactly-once" processing of events. This is achieved by "tagging" events with a unique identifier, and registering them in the inbox before processing (and if they're there, ignoring it altogether).
+Via the `:inbox` plugin, `tobox` also supports the [inbox pattern](https://event-driven.io/en/outbox_inbox_patterns_and_delivery_guarantees_explained/), to ensure "exactly-once" processing of events. This is achieved by "tagging" events with a unique identifier, and registering them in the inbox before processing (and if they're there, ignoring it altogether).
 
 In order to do so, you'll have to:
 
-1. add an "inbox" table in the database
+1. add an "inbox" table in the database and the unique id reference in the outbox table:
 
 ```ruby
 create_table(:inbox) do
   column :inbox_id, :varchar, null: true, primary_key: true # it can also be a uuid, you decide
   column :created_at, "timestamp without time zone", null: false, default: Sequel::CURRENT_TIMESTAMP
 end
-```
 
-2. add the unique id reference in the outbox table:
-
-```ruby
 create_table(:outbox) do
   primary_key :id
   column :type, :varchar, null: false
@@ -421,15 +472,17 @@ create_table(:outbox) do
   foreign_key :inbox_id, :inbox
 ```
 
-3. reference them in the configuration
+2. Load the plugin and reference them in the configuration
 
 ```ruby
 # tobox.rb
-inbox_table :inbox
-inbox_column :inbox_id
+plugin :inbox
+
+inbox_table :inbox # :inbox by default already
+inbox_column :inbox_id # :inbox_id by default already
 ```
 
-4. insert related outbox events with an inbox id
+3. insert related outbox events with an inbox id
 
 ```ruby
 order = Order.new(
@@ -446,19 +499,19 @@ end
 # assuming this bit above runs two times in two separate workers, each will be processed by tobox only once.
 ```
 
-**NOTE**: make sure you keep cleaning the inbox periodically from older messages, once there's no more danger of receiving them again.
+#### Configuration
 
-<a id="markdown-plugins" name="plugins"></a>
-## Plugins
 
-`tobox` ships with a very simple plugin system. (TODO: add docs).
+##### inbox table
 
-Plugins can be loaded in the config via `plugin`:
+Defines the name of the table to be used for inbox (`:inbox` by default).
 
-```ruby
-# tobox.rb
-plugin(:plugin_name)
-```
+##### inbox column
+
+Defines the column in the outbox table which references the inbox table (`:inbox_id` by default).
+
+
+**NOTE**: make sure you keep cleaning the inbox periodically from older messages, once there's no more danger of receiving them again.
 
 It ships with the following integrations.
 
@@ -576,7 +629,7 @@ end
 <a id="markdown-supported-rubies" name="supported-rubies"></a>
 ## Supported Rubies
 
-All Rubies greater or equal to 2.6, and always latest JRuby and Truffleruby.
+All Rubies greater or equal to 2.7, and always latest JRuby and Truffleruby.
 
 
 <a id="markdown-rails-support" name="rails-support"></a>

data/lib/tobox/configuration.rb

@@ -7,7 +7,8 @@ module Tobox
   class Configuration
     extend Forwardable
 
-    attr_reader :handlers, :lifecycle_events, :arguments_handler, :default_logger, :database
+    attr_reader :plugins, :handlers, :lifecycle_events, :arguments_handler, :default_logger, :database, :fetcher_class,
+                :config
 
     def_delegator :@config, :[]
 
@@ -18,11 +19,9 @@ module Tobox
       database_uri: nil,
       database_options: nil,
       table: :outbox,
-      group_column: nil,
-      inbox_table: nil,
-      inbox_column: nil,
+      created_at_column: nil,
       max_attempts: 10,
-      exponential_retry_factor: 4,
+      exponential_retry_factor: 2,
       wait_for_events_delay: 5,
       shutdown_timeout: 10,
       concurrency: 4, # TODO: CPU count
@@ -45,6 +44,7 @@ module Tobox
       @handlers = {}
       @message_to_arguments = nil
       @plugins = []
+      @fetcher_class = Class.new(Fetcher)
 
       if block
         case block.arity
@@ -141,6 +141,8 @@ module Tobox
 
       extend(plugin::ConfigurationMethods) if defined?(plugin::ConfigurationMethods)
 
+      @fetcher_class.__send__(:include, plugin::FetcherMethods) if defined?(plugin::FetcherMethods)
+
       plugin.configure(self, **options, &block) if plugin.respond_to?(:configure)
     end
 
@@ -157,7 +159,7 @@ module Tobox
     private
 
     def method_missing(meth, *args, &block)
-      if DEFAULT_CONFIGURATION.key?(meth) && args.size == 1
+      if @config.key?(meth) && args.size == 1
         @config[meth] = args.first
       elsif /\Aon_(.*)\z/.match(meth) && args.empty?
         on(Regexp.last_match(1).to_sym, &block)
@@ -167,8 +169,8 @@ module Tobox
     end
 
     def respond_to_missing?(meth, *args)
-      super(meth, *args) ||
-        DEFAULT_CONFIGURATION.key?(meth) ||
+      super ||
+        @config.key?(meth) ||
         /\Aon_(.*)\z/.match(meth)
     end
   end

data/lib/tobox/fetcher.rb

@@ -13,14 +13,10 @@ module Tobox
       @db = configuration.database
 
       @table = configuration[:table]
-      @group_column = configuration[:group_column]
       @exponential_retry_factor = configuration[:exponential_retry_factor]
 
       max_attempts = configuration[:max_attempts]
 
-      @inbox_table = configuration[:inbox_table]
-      @inbox_column = configuration[:inbox_column]
-
       @ds = @db[@table]
 
       run_at_conds = [
@@ -32,6 +28,8 @@ module Tobox
                           .where(run_at_conds)
                           .order(Sequel.desc(:run_at, nulls: :first), :id)
 
+      @mark_as_fetched_params = { attempts: Sequel[@table][:attempts] + 1, last_error: nil }
+
       @before_event_handlers = Array(@configuration.lifecycle_events[:before_event])
       @after_event_handlers = Array(@configuration.lifecycle_events[:after_event])
       @error_event_handlers = Array(@configuration.lifecycle_events[:error_event])
@@ -39,75 +37,28 @@ module Tobox
 
     def fetch_events(&blk)
       num_events = 0
-      @db.transaction(savepoint: false) do
-        if @group_column
-          group = @pick_next_sql.for_update
-                                .skip_locked
-                                .limit(1)
-                                .select(@group_column)
-
-          # get total from a group, to compare to the number of future locked rows.
-          total_from_group = @ds.where(@group_column => group).count
-
-          event_ids = @ds.where(@group_column => group)
-                         .order(Sequel.desc(:run_at, nulls: :first), :id)
-                         .for_update.skip_locked.select_map(:id)
-
-          if event_ids.size != total_from_group
-            # this happens if concurrent workers locked different rows from the same group,
-            # or when new rows from a given group have been inserted after the lock has been
-            # acquired
-            event_ids = []
-          end
-
-          # lock all, process 1
-          event_ids = event_ids[0, 1]
-        else
-          event_ids = @pick_next_sql.for_update
-                                    .skip_locked
-                                    .limit(1).select_map(:id) # lock starts here
-        end
+      events_tr do
+        event_ids = EMPTY
 
-        events = nil
-        error = nil
-        unless event_ids.empty?
-          @db.transaction(savepoint: true) do
-            events = @ds.where(id: event_ids).returning.delete
-
-            if blk
-              num_events = events.size
-
-              events.map! do |ev|
-                try_insert_inbox(ev) do
-                  ev[:metadata] = try_json_parse(ev[:metadata])
-                  handle_before_event(ev)
-                  yield(to_message(ev))
-                  ev
-                end
-              rescue StandardError => e
-                error = e
-                raise Sequel::Rollback
-              end.compact!
-            else
-              events.map!(&method(:to_message))
-            end
-          end
+        event_ids_tr do
+          event_ids = fetch_event_ids
+          mark_as_fetched(event_ids)
         end
 
-        return blk ? 0 : [] if events.nil?
+        evts = nil
 
-        return events unless blk
+        unless event_ids.empty?
+          with_events(event_ids) do |events|
+            evts = events
+            num_events = events.count
 
-        if events
-          events.each do |event|
-            if error
-              event.merge!(mark_as_error(event, error))
-              handle_error_event(event, error)
-            else
-              handle_after_event(event)
+            evts = events.filter_map do |ev|
+              prepare_event(ev, &blk)
             end
           end
         end
+
+        return 0 if evts.nil?
       end
 
       num_events
@@ -115,19 +66,86 @@ module Tobox
 
     private
 
+    def prepare_event(event)
+      event[:metadata] = try_json_parse(event[:metadata])
+      handle_before_event(event)
+      yield(to_message(event))
+      event
+    end
+
+    def fetch_event_ids
+      @pick_next_sql.for_update
+                    .skip_locked
+                    .limit(1).select_map(:id) # lock starts here
+    end
+
+    def mark_as_fetched(event_ids)
+      @ds.where(id: event_ids).update(@mark_as_fetched_params) unless event_ids.empty?
+    end
+
+    def events_tr(&block)
+      @db.transaction(savepoint: false, &block)
+    end
+
+    def event_ids_tr
+      yield
+    end
+
+    def with_events(event_ids, &blk)
+      events, error = yield_events(event_ids, &blk)
+
+      events.each do |event|
+        if error
+          event.merge!(mark_as_error(event, error))
+          handle_error_event(event, error)
+        else
+          handle_after_event(event)
+        end
+      end
+    end
+
+    def yield_events(event_ids)
+      events_ds = @ds.where(id: event_ids)
+      events = EMPTY
+      error = nil
+
+      begin
+        events = events_ds.all
+
+        yield events
+
+        events_ds.delete
+      rescue StandardError => e
+        error = e
+      end
+
+      [events, error]
+    end
+
     def log_message(msg)
       "(worker: #{@label}) -> #{msg}"
     end
 
     def mark_as_error(event, error)
-      @ds.where(id: event[:id]).returning.update(
-        attempts: Sequel[@table][:attempts] + 1,
+      update_params = {
         run_at: Sequel.date_add(Sequel::CURRENT_TIMESTAMP,
-                                seconds: event[:attempts] + (1**@exponential_retry_factor)),
+                                seconds: @exponential_retry_factor**(event[:attempts] - 1)),
         # run_at: Sequel.date_add(Sequel::CURRENT_TIMESTAMP,
         #                         seconds: Sequel.function(:POWER, Sequel[@table][:attempts] + 1,  4)),
         last_error: "#{error.message}\n#{error.backtrace.join("\n")}"
-      ).first
+      }
+
+      set_event_retry_attempts(event, update_params)
+    end
+
+    def set_event_retry_attempts(event, update_params)
+      ds = @ds.where(id: event[:id])
+      if @ds.supports_returning?(:update)
+        ds.returning.update(update_params).first
+      else
+        ds.update(update_params)
+        ds.first
+      end
     end
 
     def to_message(event)
@@ -148,16 +166,6 @@ module Tobox
       data
     end
 
-    def try_insert_inbox(event)
-      return yield unless @inbox_table && @inbox_column
-
-      ret = @db[@inbox_table].insert_conflict.insert(@inbox_column => event[@inbox_column])
-
-      return unless ret
-
-      yield
-    end
-
     def handle_before_event(event)
       @logger.debug do
         log_message("outbox event (type: \"#{event[:type]}\", attempts: #{event[:attempts]}) starting...")

data/lib/tobox/plugins/datadog/configuration.rb

@@ -10,23 +10,57 @@ module Datadog
       module Tobox
         module Configuration
           class Settings < Contrib::Configuration::Settings
-            option :enabled do |o|
-              o.default { env_to_bool("DD_TOBOX_SIDEKIQ_ENABLED", true) }
-              o.lazy
-            end
+            if Gem::Version.new(DDTrace::VERSION::STRING) >= Gem::Version.new("1.13.0")
+              option :enabled do |o|
+                o.type :bool
+                o.env "DD_TOBOX_SIDEKIQ_ENABLED"
+                o.default true
+              end
 
-            option :analytics_enabled do |o|
-              o.default { env_to_bool("DD_TOBOX_ANALYTICS_ENABLED", false) }
-              o.lazy
-            end
+              option :analytics_enabled do |o|
+                o.type :bool
+                o.env "DD_TOBOX_ANALYTICS_ENABLED"
+                o.default false
+              end
+
+              option :analytics_sample_rate do |o|
+                o.type :float
+                o.env "DD_TRACE_TOBOX_ANALYTICS_SAMPLE_RATE"
+                o.default 1.0
+              end
+            else
+              option :enabled do |o|
+                o.default { env_to_bool("DD_TOBOX_SIDEKIQ_ENABLED", true) }
+                o.lazy
+              end
+
+              option :analytics_enabled do |o|
+                o.default { env_to_bool("DD_TOBOX_ANALYTICS_ENABLED", false) }
+                o.lazy
+              end
 
-            option :analytics_sample_rate do |o|
-              o.default { env_to_float("DD_TRACE_TOBOX_ANALYTICS_SAMPLE_RATE", 1.0) }
-              o.lazy
+              option :analytics_sample_rate do |o|
+                o.default { env_to_float("DD_TRACE_TOBOX_ANALYTICS_SAMPLE_RATE", 1.0) }
+                o.lazy
+              end
             end
 
             option :service_name
-            option :error_handler, default: Tracing::SpanOperation::Events::DEFAULT_ON_ERROR
+
+            if DDTrace::VERSION::STRING >= "1.15.0"
+              option :error_handler do |o|
+                o.type :proc
+                o.default_proc(&Tracing::SpanOperation::Events::DEFAULT_ON_ERROR)
+              end
+            elsif DDTrace::VERSION::STRING >= "1.13.0"
+              option :error_handler do |o|
+                o.type :proc
+                o.experimental_default_proc(&Tracing::SpanOperation::Events::DEFAULT_ON_ERROR)
+              end
+            else
+              option :error_handler, default: Tracing::SpanOperation::Events::DEFAULT_ON_ERROR
+            end
+
             option :distributed_tracing, default: false
           end
         end

data/lib/tobox/plugins/event_grouping.rb

@@ -0,0 +1,47 @@
+# frozen_string_literal: true
+
+module Tobox
+  module Plugins
+    module EventGrouping
+      def self.configure(conf)
+        conf.config[:group_column] = :group_id
+      end
+
+      module FetcherMethods
+        def initialize(_, configuration)
+          super
+
+          @group_column = configuration[:group_column]
+        end
+
+        private
+
+        def fetch_event_ids
+          group = @pick_next_sql.for_update
+                                .skip_locked
+                                .limit(1)
+                                .select(@group_column)
+
+          # get total from a group, to compare to the number of future locked rows.
+          total_from_group = @ds.where(@group_column => group).count
+
+          event_ids = @ds.where(@group_column => group)
+                         .order(Sequel.desc(:run_at, nulls: :first), :id)
+                         .for_update.skip_locked.select_map(:id)
+
+          if event_ids.size != total_from_group
+            # this happens if concurrent workers locked different rows from the same group,
+            # or when new rows from a given group have been inserted after the lock has been
+            # acquired
+            event_ids = []
+          end
+
+          # lock all, process 1
+          event_ids[0, 1]
+        end
+      end
+    end
+
+    register_plugin :event_grouping, EventGrouping
+  end
+end

data/lib/tobox/plugins/inbox.rb

@@ -0,0 +1,46 @@
+# frozen_string_literal: true
+
+module Tobox
+  module Plugins
+    module Inbox
+      def self.configure(conf)
+        conf.config[:inbox_table] = :inbox
+        conf.config[:inbox_column] = :unique_id
+      end
+
+      module FetcherMethods
+        def initialize(_, configuration)
+          super
+
+          inbox_table = configuration[:inbox_table]
+
+          @inbox_ds = @db[inbox_table]
+          @inbox_column = configuration[:inbox_column]
+        end
+
+        private
+
+        def prepare_event(event, &blk)
+          try_insert_inbox(event) { super }
+        end
+
+        def try_insert_inbox(event)
+          if @inbox_ds.respond_to?(:supports_insert_conflict?) && @inbox_ds.supports_insert_conflict?
+            ret = @inbox_ds.insert_conflict.insert(@inbox_column => event[@inbox_column])
+
+            return event unless ret
+          else
+            begin
+              @inbox_ds.insert(@inbox_column => event[@inbox_column])
+            rescue Sequel::UniqueConstraintViolation
+              return event
+            end
+          end
+
+          yield
+        end
+      end
+    end
+    register_plugin :inbox, Inbox
+  end
+end

data/lib/tobox/plugins/progress.rb

@@ -0,0 +1,33 @@
+# frozen_string_literal: true
+
+module Tobox
+  module Plugins
+    module Progress
+      def self.configure(conf)
+        conf.config[:visibility_timeout] = 30
+      end
+
+      module FetcherMethods
+        private
+
+        def initialize(_, configuration)
+          super
+
+          @mark_as_fetched_params[:run_at] = Sequel.date_add(
+            Sequel::CURRENT_TIMESTAMP,
+            seconds: configuration[:visibility_timeout]
+          )
+        end
+
+        def events_tr
+          yield
+        end
+
+        def event_ids_tr(&block)
+          @db.transaction(savepoint: false, &block)
+        end
+      end
+    end
+    register_plugin :progress, Progress
+  end
+end

data/lib/tobox/plugins/stats.rb

@@ -25,6 +25,8 @@ module Tobox
 
           config = @config
 
+          plugins = config.plugins.map(&:name)
+
           interval = config.stats_interval_seconds
           @stats_handlers = Array(config.lifecycle_events[:stats])
 
@@ -34,14 +36,24 @@ module Tobox
 
           @max_attempts = config[:max_attempts]
 
+          @created_at_column = config[:created_at_column]
+
           @db = Sequel.connect(config.database.opts.merge(max_connections: 1))
+          @db.loggers = config.database.loggers
           Array(config.lifecycle_events[:database_connect]).each { |cb| cb.call(@db) }
 
-          @outbox_table = config[:table]
-          @outbox_ds = @db[@outbox_table]
+          outbox_table = config[:table]
+          @outbox_ds = @db[outbox_table]
+
+          if plugins.include?("Tobox::Plugins::Inbox")
+            inbox_table = config[:inbox_table]
+            @inbox_ds = @db[inbox_table]
+          end
 
-          inbox_table = config[:inbox_table]
-          @inbox_ds = @db[inbox_table] if inbox_table
+          if @created_at_column
+            # discard already handled events
+            @oldest_event_age_ds = @outbox_ds.where(last_error: nil, run_at: nil).order(Sequel.asc(:id))
+          end
 
           logger = config.default_logger
 
@@ -104,6 +116,13 @@ module Tobox
           stats[:failed_count] ||= 0
 
           stats[:inbox_count] = @inbox_ds.count if @inbox_ds
+
+          if @oldest_event_age_ds
+            created_at = @oldest_event_age_ds.get(@created_at_column)
+            age = created_at ? (Time.now - created_at).to_i : 0
+            stats[:oldest_event_age_in_seconds] = age
+          end
+
           stats
         end
       end

data/lib/tobox/pool/threaded_pool.rb

@@ -56,6 +56,8 @@ module Tobox
           if worker.finished? && @running
             idx = @workers.index(worker)
 
+            raise Error, "worker not found" unless idx
+
             subst_worker = Worker.new(worker.label, @configuration)
             @workers[idx] = subst_worker
             subst_thread = start_thread_worker(subst_worker)

data/lib/tobox/pool.rb

@@ -1,9 +1,9 @@
 # frozen_string_literal: true
 
 module Tobox
-  class Pool
-    class KillError < Interrupt; end
+  class KillError < Interrupt; end
 
+  class Pool
     def initialize(configuration)
       @configuration = configuration
       @logger = @configuration.default_logger

data/lib/tobox/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Tobox
-  VERSION = "0.4.4"
+  VERSION = "0.5.0"
 end

data/lib/tobox/worker.rb

@@ -8,7 +8,7 @@ module Tobox
       @label = label
       @wait_for_events_delay = configuration[:wait_for_events_delay]
       @handlers = configuration.handlers || {}
-      @fetcher = Fetcher.new(label, configuration)
+      @fetcher = configuration.fetcher_class.new(label, configuration)
       @finished = false
 
       return unless (message_to_arguments = configuration.arguments_handler)

data/lib/tobox.rb

@@ -9,6 +9,8 @@ require "mutex_m"
 module Tobox
   class Error < StandardError; end
 
+  EMPTY = [].freeze
+
   module Plugins
     @plugins = {}
     @plugins.extend(Mutex_m)
@@ -34,8 +36,8 @@ module Tobox
   end
 end
 
-require_relative "tobox/configuration"
 require_relative "tobox/fetcher"
 require_relative "tobox/worker"
 require_relative "tobox/pool"
 require_relative "tobox/application"
+require_relative "tobox/configuration"

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: tobox
 version: !ruby/object:Gem::Version
-  version: 0.4.4
+  version: 0.5.0
 platform: ruby
 authors:
 - HoneyryderChuck
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2023-07-26 00:00:00.000000000 Z
+date: 2024-09-16 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: sequel
@@ -45,6 +45,9 @@ files:
 - lib/tobox/plugins/datadog/configuration.rb
 - lib/tobox/plugins/datadog/integration.rb
 - lib/tobox/plugins/datadog/patcher.rb
+- lib/tobox/plugins/event_grouping.rb
+- lib/tobox/plugins/inbox.rb
+- lib/tobox/plugins/progress.rb
 - lib/tobox/plugins/sentry.rb
 - lib/tobox/plugins/stats.rb
 - lib/tobox/plugins/zeitwerk.rb
@@ -71,14 +74,14 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
-      version: 2.6.0
+      version: 2.7.0
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.4.10
+rubygems_version: 3.5.3
 signing_key:
 specification_version: 4
 summary: Transactional outbox pattern implementation in ruby