tobox 0.4.5 → 0.5.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 0be3e9919cb080bf7e57127356e00cc5a13e5d8af6943ee87f688e515771d9ec
-  data.tar.gz: 215095033e4763f2b5b4bd166f4dcc66fb741424244a4c4a13989974f5dfbb14
+  metadata.gz: 15d95687a102c98fcc33859b3a369dc9026f04fa58f7f785bcad1d9ccb40010f
+  data.tar.gz: 656b296f9d72d67877945317d4a6d5505de63044f74e08c506970b2c841599f4
 SHA512:
-  metadata.gz: 914972ea18b05b0542bc091cd3cdd602f420c3551f24c598b9d6711ff879076e6e0a03879f27419da6526a449b7e4364ce8e6e0aebd70a698dc250696a0c54f5
-  data.tar.gz: 419bfcd4c5604c72ace312acb8deebfa3081546b137c7f7c5ee267be5e0f49a8f21a745147e17b99ee002b2e95db7c48a03d84cd578dc3ca72bd452ad8581280
+  metadata.gz: 172b54fd340f865dbc26a30266f815fe556424cd150c8205c872da02d3725def786648710097b0880de41cc16940a39bd34c8592049d22a907d527ff84b26a3a
+  data.tar.gz: 1dafbf50e8d18c4eb7f4dfd31539322230bd1d11d403f250e4612a0579e878c366fe6b2d44eb24ae17f32476bd25994457563d3fd5ef1a46d7499dc92a93ee1d

data/CHANGELOG.md

@@ -1,5 +1,29 @@
 ## [Unreleased]
 
+## [0.5.1] - 2024-09-26
+
+### Improvements
+
+* Refactoring of management of event id which replaces `SELECT id IN (?)` resulting queries with `SELECT id = ?`.
+* Process shutdown is now more predictable, also in the grace period.
+  * `grace_shutdown_timeout` is a new configuration, by default 5 (seconds).
+
+## [0.5.0] - 2024-09-16
+
+### Features
+
+A new `:progress` plugin can be used in order to release database transactions before handling events (useful for when event handling times vary and may cause transaction bookkeeping overhead in the database).
+
+**Note**: This may become the default behaviour in a future release.
+
+### Improvements
+
+* The event grouping and inbox capabilities were converted into plugins (respectively, `:event_grouping` and `:inbox`).
+
+### Bugfixes
+
+* exponential backoff calculation was broken.
+* behaviour fixed for databases which do not support `ON CONFLICT` or `UPDATE ... RETURNING` (like MySQL).
 
 ## [0.4.5] - 2024-02-28
 

data/README.md

@@ -13,10 +13,10 @@ Simple, data-first events processing framework based on the [transactional outbo
 - [Usage](#usage)
 - [Configuration](#configuration)
 - [Event](#event)
-- [Features](#features)
-  - [Ordered event processing](#ordered-event-processing)
-  - [Inbox](#inbox)
 - [Plugins](#plugins)
+  - [Progress](#progress)
+  - [Event Grouping](#event-grouping)
+  - [Inbox](#inbox)
   - [Zeitwerk](#zeitwerk)
   - [Sentry](#sentry)
   - [Datadog](#datadog)
@@ -204,7 +204,17 @@ table :outbox
 Maximum number of times a failed attempt to process an event will be retried (`10` by default).
 
 ```ruby
-concurrency 4
+max_attempts 4
+```
+
+**Note**: the new attempt will be retried in `n ** 4`, where `n` is the number of past attempts for that event.
+
+### `exponential_retry_factor`
+
+Factor by which the number of seconds until an event can be retried will be exponentially calculated, i.e. 2 seconds on first attempt, then 4, then 8, then 16 (`2` by default).
+
+```ruby
+exponential_retry_factor 2
 ```
 
 **Note**: the new attempt will be retried in `n ** 4`, where `n` is the number of past attempts for that event.
@@ -231,6 +241,10 @@ Time (in seconds) to wait before checking again for events in the outbox.
 
 Time (in seconds) to wait for events to finishing processing, before hard-killing the process.
 
+### `grace_shutdown_timeout`
+
+Grace period (in seconds) to wait after, hard-killing the work in progress, and before exiting the process.
+
 ### `on(event_type) { |before, after| }`
 
 callback executed when processing an event of the given type. By default, it'll yield the state of data before and after the event (unless `message_to_arguments` is set).
@@ -320,17 +334,6 @@ Overrides the internal logger (an instance of `Logger`).
 
 Overrides the default log level ("info" when in "production" environment, "debug" otherwise).
 
-### group_column
-
-Defines the column to be used for event grouping, when [ordered processing of events is a requirement](#ordered-event-processing).
-
-### inbox table
-
-Defines the name of the table to be used for inbox, when [inbox usage is a requirement](#inbox).
-
-### inbox column
-
-Defines the column in the outbox table which references the inbox table, when one is set.
 
 <a id="markdown-event" name="event"></a>
 ## Event
@@ -345,36 +348,85 @@ The event is composed of the following properties:
 
 (*NOTE*: The event is also composed of other properties which are only relevant for `tobox`.)
 
-<a id="markdown-features" name="features"></a>
-## Features
 
-There are a few extra features you can run on top a "vanilla" transactional outbox implementation. This is how you can accomplish them using `tobox`.
+<a id="markdown-plugins" name="plugins"></a>
+## Plugins
+
+`tobox` ships with a very simple plugin system. (TODO: add docs).
+
+Plugins can be loaded in the config via `plugin`:
+
+```ruby
+# tobox.rb
+plugin(:plugin_name)
+```
+
+<a id="markdown-progress" name="progress"></a>
+### Progress
+
+By default, the database transaction used to consume the event is kept open while the event is handled. While this ensures atomic event consumption, it may also cause overhead related to transaction management given enough load, particularly in cases where event handling time varies (i.e. throttled HTTP requests).
+
+The `:progress` plugin fixes this by releasing the databaase transaction after fetching the event. It does so by making the fetched event "invisible" for a certain period, during which the event must be successfully handled.
+
+Here's how to use it:
+
+```ruby
+# in your tobox.rb
+plugin :progress
+
+visibility_timeout 90 # default: 30
+```
+
+3. insert related outbox events with the same group id
+
+```ruby
+order = Order.new(
+  item_id: item.id,
+  price: 20_20,
+  currency: "EUR"
+)
+DB.transaction do
+  order.save
+  DB[:outbox].insert(event_type: "order_created", group_id: order.id, data_after: order.to_hash)
+  DB[:outbox].insert(event_type: "billing_event_started", group_id: order.id, data_after: order.to_hash)
+end
+
+# "order_created" will be processed first
+# "billing_event_created" will only start processing once "order_created" finishes
+```
+
+#### Configuration
+
+##### `visibility_timeout`
+
+Timeout (in seconds) after which a previously marked-for-consumption event can be retried (default: `30`)
 
-<a id="markdown-ordered-event-processing" name="ordered-event-processing"></a>
-### Ordered event processing
+<a id="markdown-event-grouping" name="event-grouping"></a>
+### Event grouping
 
 By default, events are taken and processed from the "outbox" table concurrently by workers, which means that, while worker A may process the most recent event, and worker B takes the following, worker B may process it faster than worker A. This may be an issue if the consumer expects events from a certain context to arrive in a certain order.
 
-One solution is to have a single worker processing the "outbox" events. Another is to use the `group_column` configuration.
+One solution is to have a single worker processing the "outbox" events. Another is to use the `:event_grouping` plugin.
 
-What you have to do is:
+All you have to do is:
 
 1. add a "group id" column to the "outbox" table
 
 ```ruby
 create_table(:outbox) do
   primary_key :id
-  column :group_id, :integer
-  # The type is irrelevant, could also be :string, :uuid...
+  column :group_id, :integer # The type is irrelevant, could also be :string, :uuid...
   # ..
+  index :group_id
 ```
 
-2. set the "group_column" configuration
+2. Enable the plugin
 
 ```ruby
 # in your tobox.rb
-group_column :group_id
-index :group_id
+plugin :event_grouping
+
+group_column :group_id # by default already `:group_id`
 ```
 
 3. insert related outbox events with the same group id
@@ -394,25 +446,28 @@ end
 # "order_created" will be processed first
 # "billing_event_created" will only start processing once "order_created" finishes
 ```
+
+#### Configuration
+
+##### `group_column`
+
+Defines the database column to be used for event grouping (`:group_id` by default).
+
 <a id="inbox" name="inbox"></a>
 ### Inbox
 
-`tobox` also supports the [inbox pattern](https://event-driven.io/en/outbox_inbox_patterns_and_delivery_guarantees_explained/), to ensure "exactly-once" processing of events. This is achieved by "tagging" events with a unique identifier, and registering them in the inbox before processing (and if they're there, ignoring it altogether).
+Via the `:inbox` plugin, `tobox` also supports the [inbox pattern](https://event-driven.io/en/outbox_inbox_patterns_and_delivery_guarantees_explained/), to ensure "exactly-once" processing of events. This is achieved by "tagging" events with a unique identifier, and registering them in the inbox before processing (and if they're there, ignoring it altogether).
 
 In order to do so, you'll have to:
 
-1. add an "inbox" table in the database
+1. add an "inbox" table in the database and the unique id reference in the outbox table:
 
 ```ruby
 create_table(:inbox) do
   column :inbox_id, :varchar, null: true, primary_key: true # it can also be a uuid, you decide
   column :created_at, "timestamp without time zone", null: false, default: Sequel::CURRENT_TIMESTAMP
 end
-```
-
-2. add the unique id reference in the outbox table:
 
-```ruby
 create_table(:outbox) do
   primary_key :id
   column :type, :varchar, null: false
@@ -421,15 +476,17 @@ create_table(:outbox) do
   foreign_key :inbox_id, :inbox
 ```
 
-3. reference them in the configuration
+2. Load the plugin and reference them in the configuration
 
 ```ruby
 # tobox.rb
-inbox_table :inbox
-inbox_column :inbox_id
+plugin :inbox
+
+inbox_table :inbox # :inbox by default already
+inbox_column :inbox_id # :inbox_id by default already
 ```
 
-4. insert related outbox events with an inbox id
+3. insert related outbox events with an inbox id
 
 ```ruby
 order = Order.new(
@@ -446,19 +503,19 @@ end
 # assuming this bit above runs two times in two separate workers, each will be processed by tobox only once.
 ```
 
-**NOTE**: make sure you keep cleaning the inbox periodically from older messages, once there's no more danger of receiving them again.
+#### Configuration
 
-<a id="markdown-plugins" name="plugins"></a>
-## Plugins
 
-`tobox` ships with a very simple plugin system. (TODO: add docs).
+##### inbox table
 
-Plugins can be loaded in the config via `plugin`:
+Defines the name of the table to be used for inbox (`:inbox` by default).
 
-```ruby
-# tobox.rb
-plugin(:plugin_name)
-```
+##### inbox column
+
+Defines the column in the outbox table which references the inbox table (`:inbox_id` by default).
+
+
+**NOTE**: make sure you keep cleaning the inbox periodically from older messages, once there's no more danger of receiving them again.
 
 It ships with the following integrations.
 
@@ -530,6 +587,7 @@ on_stats(5) do |stats_collector| # every 5 seconds
   # now you can send them to your statsd collector
   #
   StatsD.gauge('outbox_pending_backlog', stats[:pending_count])
+  StatsD.gauge('outbox_oldest_message_age', stats[:oldest_event_age_in_seconds])
 end
 ```
 
@@ -576,7 +634,7 @@ end
 <a id="markdown-supported-rubies" name="supported-rubies"></a>
 ## Supported Rubies
 
-All Rubies greater or equal to 2.6, and always latest JRuby and Truffleruby.
+All Rubies greater or equal to 2.7, and always latest JRuby and Truffleruby.
 
 
 <a id="markdown-rails-support" name="rails-support"></a>

data/lib/tobox/cli.rb

@@ -108,12 +108,18 @@ module Tobox
           opts[:tag] = arg
         end
 
-        o.on "-t", "--shutdown-timeout NUM", Integer, "Shutdown timeout (in seconds)" do |arg|
+        o.on "-t", "--shutdown-timeout NUM", Float, "Shutdown timeout (in seconds)" do |arg|
           raise ArgumentError, "must be positive" unless arg.positive?
 
           opts[:shutdown_timeout] = arg
         end
 
+        o.on "-g", "--shutdown-grace-timeout NUM", Float, "Shutdown grace timeout (in seconds)" do |arg|
+          raise ArgumentError, "must be positive" unless arg.positive?
+
+          opts[:grace_shutdown_timeout] = arg
+        end
+
         o.on "--verbose", "Print more verbose output" do |arg|
           opts[:verbose] = arg
         end

data/lib/tobox/configuration.rb

@@ -7,7 +7,8 @@ module Tobox
   class Configuration
     extend Forwardable
 
-    attr_reader :handlers, :lifecycle_events, :arguments_handler, :default_logger, :database
+    attr_reader :plugins, :handlers, :lifecycle_events, :arguments_handler, :default_logger, :database, :fetcher_class,
+                :config
 
     def_delegator :@config, :[]
 
@@ -18,13 +19,12 @@ module Tobox
       database_uri: nil,
       database_options: nil,
       table: :outbox,
-      group_column: nil,
-      inbox_table: nil,
-      inbox_column: nil,
+      created_at_column: nil,
       max_attempts: 10,
-      exponential_retry_factor: 4,
+      exponential_retry_factor: 2,
       wait_for_events_delay: 5,
       shutdown_timeout: 10,
+      grace_shutdown_timeout: 5,
       concurrency: 4, # TODO: CPU count
       worker: :thread
     }.freeze
@@ -45,6 +45,7 @@ module Tobox
       @handlers = {}
       @message_to_arguments = nil
       @plugins = []
+      @fetcher_class = Class.new(Fetcher)
 
       if block
         case block.arity
@@ -141,6 +142,8 @@ module Tobox
 
       extend(plugin::ConfigurationMethods) if defined?(plugin::ConfigurationMethods)
 
+      @fetcher_class.__send__(:include, plugin::FetcherMethods) if defined?(plugin::FetcherMethods)
+
       plugin.configure(self, **options, &block) if plugin.respond_to?(:configure)
     end
 
@@ -157,7 +160,7 @@ module Tobox
     private
 
     def method_missing(meth, *args, &block)
-      if DEFAULT_CONFIGURATION.key?(meth) && args.size == 1
+      if @config.key?(meth) && args.size == 1
         @config[meth] = args.first
       elsif /\Aon_(.*)\z/.match(meth) && args.empty?
         on(Regexp.last_match(1).to_sym, &block)
@@ -167,8 +170,8 @@ module Tobox
     end
 
     def respond_to_missing?(meth, *args)
-      super(meth, *args) ||
-        DEFAULT_CONFIGURATION.key?(meth) ||
+      super ||
+        @config.key?(meth) ||
         /\Aon_(.*)\z/.match(meth)
     end
   end

data/lib/tobox/fetcher.rb

@@ -13,14 +13,10 @@ module Tobox
       @db = configuration.database
 
       @table = configuration[:table]
-      @group_column = configuration[:group_column]
       @exponential_retry_factor = configuration[:exponential_retry_factor]
 
       max_attempts = configuration[:max_attempts]
 
-      @inbox_table = configuration[:inbox_table]
-      @inbox_column = configuration[:inbox_column]
-
       @ds = @db[@table]
 
       run_at_conds = [
@@ -32,6 +28,8 @@ module Tobox
                           .where(run_at_conds)
                           .order(Sequel.desc(:run_at, nulls: :first), :id)
 
+      @mark_as_fetched_params = { attempts: Sequel[@table][:attempts] + 1, last_error: nil }
+
       @before_event_handlers = Array(@configuration.lifecycle_events[:before_event])
       @after_event_handlers = Array(@configuration.lifecycle_events[:after_event])
       @error_event_handlers = Array(@configuration.lifecycle_events[:error_event])
@@ -39,73 +37,19 @@ module Tobox
 
     def fetch_events(&blk)
       num_events = 0
-      @db.transaction(savepoint: false) do
-        if @group_column
-          group = @pick_next_sql.for_update
-                                .skip_locked
-                                .limit(1)
-                                .select(@group_column)
-
-          # get total from a group, to compare to the number of future locked rows.
-          total_from_group = @ds.where(@group_column => group).count
-
-          event_ids = @ds.where(@group_column => group)
-                         .order(Sequel.desc(:run_at, nulls: :first), :id)
-                         .for_update.skip_locked.select_map(:id)
-
-          if event_ids.size != total_from_group
-            # this happens if concurrent workers locked different rows from the same group,
-            # or when new rows from a given group have been inserted after the lock has been
-            # acquired
-            event_ids = []
-          end
-
-          # lock all, process 1
-          event_ids = event_ids[0, 1]
-        else
-          event_ids = @pick_next_sql.for_update
-                                    .skip_locked
-                                    .limit(1).select_map(:id) # lock starts here
-        end
+      events_tr do
+        event_id = nil
 
-        events = nil
-        error = nil
-        unless event_ids.empty?
-          @db.transaction(savepoint: true) do
-            events = @ds.where(id: event_ids).returning.delete
-
-            if blk
-              num_events = events.size
-
-              events.map! do |ev|
-                try_insert_inbox(ev) do
-                  ev[:metadata] = try_json_parse(ev[:metadata])
-                  handle_before_event(ev)
-                  yield(to_message(ev))
-                  ev
-                end
-              rescue StandardError => e
-                error = e
-                raise Sequel::Rollback
-              end.compact!
-            else
-              events.map!(&method(:to_message))
-            end
-          end
+        event_id_tr do
+          event_id = fetch_event_id
+          mark_as_fetched(event_id) if event_id
         end
 
-        return blk ? 0 : [] if events.nil?
-
-        return events unless blk
+        if event_id
+          with_event(event_id) do |event|
+            num_events = 1
 
-        if events
-          events.each do |event|
-            if error
-              event.merge!(mark_as_error(event, error))
-              handle_error_event(event, error)
-            else
-              handle_after_event(event)
-            end
+            prepare_event(event, &blk)
           end
         end
       end
@@ -115,19 +59,82 @@ module Tobox
 
     private
 
+    def prepare_event(event)
+      event[:metadata] = try_json_parse(event[:metadata])
+      handle_before_event(event)
+      yield(to_message(event))
+    end
+
+    def fetch_event_id
+      @pick_next_sql.for_update
+                    .skip_locked
+                    .limit(1).select_map(:id).first # lock starts here
+    end
+
+    def mark_as_fetched(event_id)
+      @ds.where(id: event_id).update(@mark_as_fetched_params)
+    end
+
+    def events_tr(&block)
+      @db.transaction(savepoint: false, &block)
+    end
+
+    def event_id_tr
+      yield
+    end
+
+    def with_event(event_id, &blk)
+      event, error = yield_event(event_id, &blk)
+
+      if error
+        event.merge!(mark_as_error(event, error))
+        handle_error_event(event, error)
+      else
+        handle_after_event(event)
+      end
+    end
+
+    def yield_event(event_id)
+      events_ds = @ds.where(id: event_id)
+      event = error = nil
+
+      begin
+        event = events_ds.first
+
+        yield event
+
+        events_ds.delete
+      rescue StandardError => e
+        error = e
+      end
+
+      [event, error]
+    end
+
     def log_message(msg)
       "(worker: #{@label}) -> #{msg}"
     end
 
     def mark_as_error(event, error)
-      @ds.where(id: event[:id]).returning.update(
-        attempts: Sequel[@table][:attempts] + 1,
+      update_params = {
         run_at: Sequel.date_add(Sequel::CURRENT_TIMESTAMP,
-                                seconds: event[:attempts] + (1**@exponential_retry_factor)),
+                                seconds: @exponential_retry_factor**(event[:attempts] - 1)),
         # run_at: Sequel.date_add(Sequel::CURRENT_TIMESTAMP,
         #                         seconds: Sequel.function(:POWER, Sequel[@table][:attempts] + 1,  4)),
         last_error: "#{error.message}\n#{error.backtrace.join("\n")}"
-      ).first
+      }
+
+      set_event_retry_attempts(event, update_params)
+    end
+
+    def set_event_retry_attempts(event, update_params)
+      ds = @ds.where(id: event[:id])
+      if @ds.supports_returning?(:update)
+        ds.returning.update(update_params).first
+      else
+        ds.update(update_params)
+        ds.first
+      end
     end
 
     def to_message(event)
@@ -148,16 +155,6 @@ module Tobox
       data
     end
 
-    def try_insert_inbox(event)
-      return yield unless @inbox_table && @inbox_column
-
-      ret = @db[@inbox_table].insert_conflict.insert(@inbox_column => event[@inbox_column])
-
-      return unless ret
-
-      yield
-    end
-
     def handle_before_event(event)
       @logger.debug do
         log_message("outbox event (type: \"#{event[:type]}\", attempts: #{event[:attempts]}) starting...")

data/lib/tobox/plugins/event_grouping.rb

@@ -0,0 +1,47 @@
+# frozen_string_literal: true
+
+module Tobox
+  module Plugins
+    module EventGrouping
+      def self.configure(conf)
+        conf.config[:group_column] = :group_id
+      end
+
+      module FetcherMethods
+        def initialize(_, configuration)
+          super
+
+          @group_column = configuration[:group_column]
+        end
+
+        private
+
+        def fetch_event_id
+          group = @pick_next_sql.for_update
+                                .skip_locked
+                                .limit(1)
+                                .select(@group_column)
+
+          # get total from a group, to compare to the number of future locked rows.
+          total_from_group = @ds.where(@group_column => group).count
+
+          event_ids = @ds.where(@group_column => group)
+                         .order(Sequel.desc(:run_at, nulls: :first), :id)
+                         .for_update.skip_locked.select_map(:id)
+
+          if event_ids.size != total_from_group
+            # this happens if concurrent workers locked different rows from the same group,
+            # or when new rows from a given group have been inserted after the lock has been
+            # acquired
+            event_ids = []
+          end
+
+          # lock all, process 1
+          event_ids.first
+        end
+      end
+    end
+
+    register_plugin :event_grouping, EventGrouping
+  end
+end

data/lib/tobox/plugins/inbox.rb

@@ -0,0 +1,46 @@
+# frozen_string_literal: true
+
+module Tobox
+  module Plugins
+    module Inbox
+      def self.configure(conf)
+        conf.config[:inbox_table] = :inbox
+        conf.config[:inbox_column] = :unique_id
+      end
+
+      module FetcherMethods
+        def initialize(_, configuration)
+          super
+
+          inbox_table = configuration[:inbox_table]
+
+          @inbox_ds = @db[inbox_table]
+          @inbox_column = configuration[:inbox_column]
+        end
+
+        private
+
+        def prepare_event(event, &blk)
+          try_insert_inbox(event) { super }
+        end
+
+        def try_insert_inbox(event)
+          if @inbox_ds.respond_to?(:supports_insert_conflict?) && @inbox_ds.supports_insert_conflict?
+            ret = @inbox_ds.insert_conflict.insert(@inbox_column => event[@inbox_column])
+
+            return event unless ret
+          else
+            begin
+              @inbox_ds.insert(@inbox_column => event[@inbox_column])
+            rescue Sequel::UniqueConstraintViolation
+              return event
+            end
+          end
+
+          yield
+        end
+      end
+    end
+    register_plugin :inbox, Inbox
+  end
+end

data/lib/tobox/plugins/progress.rb

@@ -0,0 +1,33 @@
+# frozen_string_literal: true
+
+module Tobox
+  module Plugins
+    module Progress
+      def self.configure(conf)
+        conf.config[:visibility_timeout] = 30
+      end
+
+      module FetcherMethods
+        private
+
+        def initialize(_, configuration)
+          super
+
+          @mark_as_fetched_params[:run_at] = Sequel.date_add(
+            Sequel::CURRENT_TIMESTAMP,
+            seconds: configuration[:visibility_timeout]
+          )
+        end
+
+        def events_tr
+          yield
+        end
+
+        def event_id_tr(&block)
+          @db.transaction(savepoint: false, &block)
+        end
+      end
+    end
+    register_plugin :progress, Progress
+  end
+end

data/lib/tobox/plugins/stats.rb

@@ -25,6 +25,8 @@ module Tobox
 
           config = @config
 
+          plugins = config.plugins.map(&:name)
+
           interval = config.stats_interval_seconds
           @stats_handlers = Array(config.lifecycle_events[:stats])
 
@@ -34,14 +36,24 @@ module Tobox
 
           @max_attempts = config[:max_attempts]
 
+          @created_at_column = config[:created_at_column]
+
           @db = Sequel.connect(config.database.opts.merge(max_connections: 1))
+          @db.loggers = config.database.loggers
           Array(config.lifecycle_events[:database_connect]).each { |cb| cb.call(@db) }
 
-          @outbox_table = config[:table]
-          @outbox_ds = @db[@outbox_table]
+          outbox_table = config[:table]
+          @outbox_ds = @db[outbox_table]
+
+          if plugins.include?("Tobox::Plugins::Inbox")
+            inbox_table = config[:inbox_table]
+            @inbox_ds = @db[inbox_table]
+          end
 
-          inbox_table = config[:inbox_table]
-          @inbox_ds = @db[inbox_table] if inbox_table
+          if @created_at_column
+            # discard already handled events
+            @oldest_event_age_ds = @outbox_ds.where(last_error: nil, run_at: nil).order(Sequel.asc(:id))
+          end
 
           logger = config.default_logger
 
@@ -104,6 +116,13 @@ module Tobox
           stats[:failed_count] ||= 0
 
           stats[:inbox_count] = @inbox_ds.count if @inbox_ds
+
+          if @oldest_event_age_ds
+            created_at = @oldest_event_age_ds.get(@created_at_column)
+            age = created_at ? (Time.now - created_at).to_i : 0
+            stats[:oldest_event_age_in_seconds] = age
+          end
+
           stats
         end
       end

data/lib/tobox/pool/fiber_pool.rb

@@ -5,34 +5,74 @@ require "fiber_scheduler"
 
 module Tobox
   class FiberPool < Pool
-    def initialize(_configuration)
+    def initialize(_)
       Sequel.extension(:fiber_concurrency)
       super
+      @fibers = []
+
+      @fiber_mtx = Mutex.new
+      @fiber_cond = ConditionVariable.new
+      @fiber_thread = nil
     end
 
     def start
       @fiber_thread = Thread.start do
         Thread.current.name = "tobox-fibers-thread"
 
-        FiberScheduler do
-          @workers.each_with_index do |wk, _idx|
-            Fiber.schedule { do_work(wk) }
+        begin
+          FiberScheduler do
+            @fiber_mtx.synchronize do
+              @workers.each do |worker|
+                @fibers << start_fiber_worker(worker)
+              end
+              @fiber_cond.signal
+            end
           end
+        rescue KillError
+          @fibers.each { |f| f.raise(KillError) }
         end
       end
+      @fiber_mtx.synchronize do
+        @fiber_cond.wait(@fiber_mtx)
+      end
     end
 
     def stop
       shutdown_timeout = @configuration[:shutdown_timeout]
+      grace_shutdown_timeout = @configuration[:grace_shutdown_timeout]
 
       super
 
-      begin
-        Timeout.timeout(shutdown_timeout) { @fiber_thread.value }
-      rescue Timeout::Error
-        # hard exit
-        @fiber_thread.raise(KillError)
-        @fiber_thread.value
+      @fiber_thread.join(shutdown_timeout)
+
+      return unless @fiber_thread.alive?
+
+      @fiber_thread.raise(KillError)
+      @fiber_thread.join(grace_shutdown_timeout)
+      @fiber_thread.kill
+      @fiber_thread.join(1)
+    end
+
+    private
+
+    def start_fiber_worker(worker)
+      Fiber.schedule do
+        do_work(worker)
+
+        @fiber_mtx.synchronize do
+          @fibers.delete(Fiber.current)
+
+          if worker.finished? && @running
+            idx = @workers.index(worker)
+
+            raise Error, "worker not found" unless idx
+
+            subst_worker = Worker.new(worker.label, @configuration)
+            @workers[idx] = subst_worker
+            subst_fiber = start_fiber_worker(subst_worker)
+            @fiber_mtx.synchronize { @fibers << subst_fiber }
+          end
+        end
       end
     end
   end

data/lib/tobox/pool/threaded_pool.rb

@@ -22,24 +22,35 @@ module Tobox
 
     def stop
       shutdown_timeout = @configuration[:shutdown_timeout]
-
-      deadline = Process.clock_gettime(::Process::CLOCK_MONOTONIC)
+      grace_shutdown_timeout = @configuration[:grace_shutdown_timeout]
 
       super
       Thread.pass # let workers finish
 
       # soft exit
-      while Process.clock_gettime(::Process::CLOCK_MONOTONIC) - deadline < shutdown_timeout
-        return if @threads.empty?
+      join = lambda do |timeout|
+        start = Process.clock_gettime(::Process::CLOCK_MONOTONIC)
+
+        loop do
+          terminating_th = @threads.synchronize { @threads.first }
+
+          return unless terminating_th
+
+          elapsed = Process.clock_gettime(Process::CLOCK_MONOTONIC) - start
+
+          break if elapsed > timeout
 
-        sleep 0.5
+          terminating_th.join(timeout - elapsed)
+        end
       end
 
+      join.call(shutdown_timeout)
+
       # hard exit
-      @threads.each { |th| th.raise(KillError) }
-      while (th = @threads.pop)
-        th.value # waits
-      end
+      @threads.synchronize { @threads.each { |th| th.raise(KillError) } }
+      join.call(grace_shutdown_timeout)
+      @threads.synchronize { @threads.each(&:kill) }
+      join.call(1)
     end
 
     private
@@ -56,13 +67,13 @@ module Tobox
           if worker.finished? && @running
             idx = @workers.index(worker)
 
+            raise Error, "worker not found" unless idx
+
             subst_worker = Worker.new(worker.label, @configuration)
             @workers[idx] = subst_worker
             subst_thread = start_thread_worker(subst_worker)
             @threads << subst_thread
           end
-          # all workers went down abruply, we need to kill the process.
-          # @parent_thread.raise(Interrupt) if wk.finished? && @threads.empty? && @running
         end
       end
     end

data/lib/tobox/pool.rb

@@ -1,9 +1,9 @@
 # frozen_string_literal: true
 
 module Tobox
-  class Pool
-    class KillError < Interrupt; end
+  class KillError < Interrupt; end
 
+  class Pool
     def initialize(configuration)
       @configuration = configuration
       @logger = @configuration.default_logger

data/lib/tobox/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Tobox
-  VERSION = "0.4.5"
+  VERSION = "0.5.1"
 end

data/lib/tobox/worker.rb

@@ -8,7 +8,7 @@ module Tobox
       @label = label
       @wait_for_events_delay = configuration[:wait_for_events_delay]
       @handlers = configuration.handlers || {}
-      @fetcher = Fetcher.new(label, configuration)
+      @fetcher = configuration.fetcher_class.new(label, configuration)
       @finished = false
 
       return unless (message_to_arguments = configuration.arguments_handler)

data/lib/tobox.rb

@@ -9,6 +9,8 @@ require "mutex_m"
 module Tobox
   class Error < StandardError; end
 
+  EMPTY = [].freeze
+
   module Plugins
     @plugins = {}
     @plugins.extend(Mutex_m)
@@ -34,8 +36,8 @@ module Tobox
   end
 end
 
-require_relative "tobox/configuration"
 require_relative "tobox/fetcher"
 require_relative "tobox/worker"
 require_relative "tobox/pool"
 require_relative "tobox/application"
+require_relative "tobox/configuration"

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: tobox
 version: !ruby/object:Gem::Version
-  version: 0.4.5
+  version: 0.5.1
 platform: ruby
 authors:
 - HoneyryderChuck
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2024-02-28 00:00:00.000000000 Z
+date: 2024-09-26 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: sequel
@@ -45,6 +45,9 @@ files:
 - lib/tobox/plugins/datadog/configuration.rb
 - lib/tobox/plugins/datadog/integration.rb
 - lib/tobox/plugins/datadog/patcher.rb
+- lib/tobox/plugins/event_grouping.rb
+- lib/tobox/plugins/inbox.rb
+- lib/tobox/plugins/progress.rb
 - lib/tobox/plugins/sentry.rb
 - lib/tobox/plugins/stats.rb
 - lib/tobox/plugins/zeitwerk.rb
@@ -71,14 +74,14 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
-      version: 2.6.0
+      version: 2.7.0
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.4.10
+rubygems_version: 3.5.3
 signing_key:
 specification_version: 4
 summary: Transactional outbox pattern implementation in ruby