tobox 0.6.1 → 0.7.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 7fcb3a283fa1bb64036319c64c537891a3c4640331ef96058c0a80f311ff6d9a
-  data.tar.gz: e6f6a8cd2b0ec8f38cc5e52added655c1922cc79be890fa79c72df3a56418990
+  metadata.gz: d1414826d4ba25bb3b9ac45856cfe3d064efd57c4d0dbac1f5ad6b0fa5887286
+  data.tar.gz: 194b0b90a514e1e4809170baf4389090a06cc62e4456412117ced3094dab89d5
 SHA512:
-  metadata.gz: 9b48f7d89bfecba3313b4807fb7cc3139aac28f43c43755657feedddfca7c7fb9048e1b6898becbc3fe05ee9ba28b8a1e57948661743173870ea7b04785a2ddc
-  data.tar.gz: 24f01a9794d77d90e406b71be0045c309e7280e46b98ad8aad2375935d47647963780c834833b76a0a4ed838ba5dd95aa184c17e3707059e51cd4ed2f334c932
+  metadata.gz: ec514b27d335a02c0eb2c3d3a6a01cdf3ddd289484b4aa80e51b4664a0f3eff3a179986004d1b6d3b0db338df0a15645a9b622dd1771b81b8d3670c554541367
+  data.tar.gz: 36eabe5466a4912d994893496e8cf76fbc445e7cf437003536cd5688531ec4786ff714f25c391cf969ec3d3fdfe77f6017ccacb0d5285d9b341c5fe5456c0978

data/CHANGELOG.md

@@ -1,5 +1,35 @@
 ## [Unreleased]
 
+## [0.7.0] - 2024-12-18
+
+### Features
+
+#### `:pg_notify` plugin
+
+The `:pg_notify` plugin is introduced, which leverages the PostgreSQL-only `LISTEN/NOTIFY` statements to asynchronously notify workers when there are new events to be processed (instead of regularly polling the database).
+
+#### `visibility_column` and `attempts_column` configuration
+
+The `visibility_column` configuration (default: `:run_at`) can be used not only to point to a differently named column, it can both point to either a timestamp column (used as a visibility timeout for events) or a boolean column (used to hide events from other transactions); using the latter will also disable exponential backoff in retries.
+
+The `attempts_column` configuration (default: `:attempts`) can be used to point to a differently named column, or, when set to `nil`, uncap retries.
+
+See when they can be used for in the readme recommendations section.
+
+### Improvements
+
+#### fiber pool on async
+
+Support was removed for the `fiber_scheduler` gem, which is under-maintained, so the fiber worker pool will use an [async](https://github.com/socketry/async) scheduler going forward. Functionality and APIs will be the same, but you'll have to add it to your Gemfile.
+
+### Bugfixes
+
+* `datadog` integration: environment variables were wrongly named.
+
+### Chore
+
+* `mutex_m` usage was removed.
+
 ## [0.6.1] - 2024-10-30
 
 ### Improvements

data/README.md

@@ -21,8 +21,10 @@ Simple, data-first events processing framework based on the [transactional outbo
   - [Sentry](#sentry)
   - [Datadog](#datadog)
   - [Stats](#stats)
+  - [PG Notify](#pg-notify)
 - [Advanced](#advanced)
   - [Batch Events Handling](#batch-events)
+- [Recommendations](#recommendations)
 - [Supported Rubies](#supported-rubies)
 - [Rails support](#rails-support)
 - [Why?](#why)
@@ -201,6 +203,28 @@ the name of the database table where outbox events are stored (`:outbox` by defa
 table :outbox
 ```
 
+### `visibility_column`
+
+the name of the database column used to mark an event as invisible while being handled (`:run_at` by default).
+
+The column type MUST be either a datetime (or timestamp, depending of your database) or a boolean (if your database supports it, MySQL doesn't for example).
+
+If it's a datetime/timestamp column, this value will be used, along with the `visibility timeout` option, to mark the event as invisible for the given duration; this will ensure that the event will be picked up again in case of a crash eventually, in case of non-transactional event handling (via the `:progress` plugin); if it's a boolean column, the event is marked as invisible indefinitely, so in case of a crash, you'll need to recover it manually.
+
+```ruby
+visibility_column :run_at
+```
+
+### `attempts_column`
+
+the name of the database column where the number of times an event was handled and failed (`:attempts` by default). If `nil`, events will be retried indefinitely.
+
+### `created_at_column`
+
+the name of the database column where the event creation timestamp is stored (`:created_at` by default).
+
+When creating the outbox table, you're **recommended** to set this column default to `CURRENT_TIMESTAMP` (or the equivalent in your database), instead of passing it manually in the corresponding `INSERT` statements.
+
 ### `max_attempts`
 
 Maximum number of times a failed attempt to process an event will be retried (`10` by default).
@@ -536,7 +560,6 @@ end
 
 #### Configuration
 
-
 ##### inbox table
 
 Defines the name of the table to be used for inbox (`:inbox` by default).
@@ -589,16 +612,23 @@ end
 Plugin for [datadog](https://github.com/DataDog/dd-trace-rb) ruby SDK. It'll generate traces for event handling.
 
 ```ruby
-# you can init the datadog config in another file to load:
-Datadog.configure do |c|
-  c.tracing.instrument :tobox
-end
-
 # tobox.rb
 plugin(:datadog)
+# or, if you want to pass options to tracing call:
+plugin(:datadog, enabled: false)
+# or, if you want to access the datadog configuration:
+plugin(:datadog) do |c|
+  c.tracing.instrument :smth_else
+end
 ```
 
-<a id="markdown-datadog" name="stats"></a>
+`datadog` tracing functionality can also be enabled/disabled via environment variables, namely the following:
+
+* `DD_TOBOX_ENABLED`: enables/disables tobox tracing (defaults to `true`)
+* `DD_TOBOX_ANALYTICS_ENABLED`: enables/disables tobox analytics (defaults to `true`)
+* `DD_TRACE_TOBOX_ANALYTICS_SAMPLE_RATE`: sets tobox tracing sample rate (defaults to `1.0`)
+
+<a id="markdown-stats" name="stats"></a>
 ### Stats
 
 The `stats` plugin collects statistics related with the outbox table periodically, and exposes them to app code (which can then relay them to a statsD collector, or similar tool).
@@ -662,6 +692,29 @@ c.on_stats(5) do |stats_collector, db|
 end
 ```
 
+
+<a id="markdown-pg-notify" name="pg-notify"></a>
+### PG Notify
+
+The `pg_notify` plugin is a **PostgreSLQ only** plugin, which uses the [LISTEN](https://www.postgresql.org/docs/current/sql-listen.html) statement to pause the workers when no work is available in the outbox table, until the producer says so, by using the [NOTIFY](https://www.postgresql.org/docs/current/sql-notify.html) statement to notify the channel the workers are listening to.
+
+It reduces the `SELECT ... FOR UPDATE SKIP LOCKED` statements to the bare minimum required; without this plugin, these may, given enough load, become the cause of overhead in the master replica, considering that they're handled as "write statements", i.e. resources must be allocated, high frequency affects applying changes on (and using) indexes on the outbox table, which may make subsequent queries fall back to table scan, which will hold dead tuples from used transaction xids for longer, which won't be vacuumed fast, which increases replication lag, which... you get the gist.
+
+```ruby
+plugin(:pg_notify)
+notifier_channel :outbox_notifications # default
+
+# that's it
+```
+
+**NOTE**: this plugin can't be used with `jruby`.
+
+#### Configuration
+
+##### `notifier_channel`
+
+Identifies the name of the channel the `LISTEN` and `NOTIFY` SQL statements will refer to (`:outbox_notifications` by default).
+
 <a id="markdown-advanced" name="advanced"></a>
 ## Advanced
 
@@ -711,6 +764,104 @@ on("user_created", "user_updated") do |*events| # 10 events at most
   end
 end
 ```
+<a id="markdown-recommendations" name="recommendations"></a>
+## Recommendations
+
+There is no free lunch. Having a transactional outbox has a cost. Throughput is sacrificed in order to guarantee the processing of the event. The cost has to be reasonable, however.
+
+### PostgreSQL
+
+PostgreSQL is the most popular database around, and for good reason: extensible, feature-rich, and quite performant for most workloads. It does have some known drawbacks though: its implementation of MVCC, with creation of tuples for UPDATEs and DELETEs, along with the requirement for indexes to point to the address of the most recent tuple, and WAL logs having to bookkeep all of that (which impacts, among other things, disk usage and replication), highly impacts the performance of transaction management. This phenomenon is known as "write amplification".
+
+Considering the additional overhead that a transactional outbox introduces to the same database your main application uses, certain issues may escalate badly, and it'll be up to you to apply strategies to mitigate them. Here are some recommendations.
+
+### Tweak `max_connections`
+
+By default, a `tobox` consumer process will have as many database connections as there are workers (each worker polls the outbox table individually). As the system scales out to cope with more traffic, you may see that, as more workers are added, so will query latency (and database CPU usage).
+
+One way to address that is to limit the number of database connections that can be used by the workers in a `tobox` consumer process, by setting the `max_connections` configuration option to a number lower than `concurrency`, i.e. 1/3 or 1/4. As a result, workers will wait for an available connection to fetch work from, when none is available.
+
+#### Caveats
+
+This is not the main source of query latency overhead, you may start seeing "pool timeout" errors as a result, so do monitor their performance and apply other mitigations accordingly.
+
+### Handling events in batches
+
+By default, each worker will fetch-and-handle-then-delete events one by one. As surges happen and volume increases, the database will spend way more time and resources managing the transaction, than doing the actual work you need, thereby affecting overall turnaround time. In the case of PostgreSQL, the constant DELETEs and UPDATEs may result in the query planner deciding not to use indexes to find an event, and instead fallback to table scan, if an index is assumed to be "behind" due to a large queue of pending updates from valid transactions.
+
+A way to mitigate this is to [handle more events at once](#batch-events). It's a strategy that makes sense if the event handler APIs support batching. For instance, if all your event handler is doing is relaying to AWS SNS, you can use the [PublishBatch](https://docs.aws.amazon.com/sns/latest/api/API_PublishBatch.html) API (and adjust the batching window to the max threshold you're able to handle at once).
+
+#### Caveats
+
+As per above, it makes sense to use this if events can be handled as a batch; if that's not the case, and the handling block iterates across the batch one by one, this will cause significant variance in single event TaT metrics, as a "slow to handle" event will delay subsequent events in the batch. Delays can also cause visibility timeouts to expire, and make events visible to other handlers earlier than expected.
+
+Recovering from errors in a batch is also more convoluted, (see `Tobox.raise_batch_errors`).
+
+### Disable retries and ordering
+
+The `tobox` default configuration expects the `visibility_column` to be a datetime column ( default is `:run_at`), which is therefore used as a "visibility timeout", and along the `attempts` column, used to retry failed events gracefully with an exponential backoff interval.
+
+As a consequence, and in order to ensure reliable performance of the worker polling query, a sorted index is recommended; in PostgreSQL, it's `CREATE INDEX ... (id, run_at DESC NULLS FIRST)`, which ensures that new events get handled before retries, which can append ` WHERE attempts < 10` to the index statement, in order to rule out events which have exhausted attempts.
+
+This comes at the cost of increased overhead per event: when producing it via `INSERT` statement, the sorted index will have to be rebalanced. When picking it up, setting the "visibility timeout" before handling it will rebalance it again; and after handling it, whether successfully or not, it'll rebalance it again. This will increase the backlog associated with index management, which may have other consequences (described somewhere else in this section).
+
+You may observe in your systems that your handler either never fails, or when it does, it's the type of transient error which can be retried immediately after, and at a marginal cost. In such situations, the default "planning for failure" exponential backoff strategy described above imposes too much weight for little gain.
+
+You can improve this by setting `visibility_column` to a boolean column, with default set to `false`:
+
+```ruby
+# in migration
+column :in_progress, :boolean, default: false
+
+# tobox
+visibility_column :in_progress
+# and, if you require unbounded retries
+attempts_column nil
+```
+
+this should improve the performance of the main polling query, by **not requiring a sorted index on the visibility column** (i.e. the primary key index is all you need), and rely on conditional boolean statements (instead of the more expensive datetime logical operators).
+
+#### Caveats
+
+While using a boolean column as the `visibility_column` may improve the performance of most queries and reduce the overhead of writes, event handling will not be protected against database crashes, so you'll have to monitor idle events and recover them manually (by resetting the `visibility_column` to `false`).
+
+### Do not poll, get notified
+
+The database must allocate resources and bookkeep some data on each transaction. In some cases (i.e. PostgreSQL), some of that bookkeeping does not happen **until** the first write statement is processed. However, due to the usage of locks via `SELECT ... FOR UPDATE`, most databases will consider the polling statement as a write statement, which means that, in a `tobox` process, transaction overhead is ever present. In a high availability configuration scenario, transactional resources will need to be maintained and replicated to read replica nodes, which given enough replication lag and inability to vacuum data, may snowball resource usage in the master replica, which may trigger autoscaling, causing more workers to poll the database for more work, and eventually bringing the whole system down.
+
+This can be mitigated by either adjusting polling intervals (via `wait_for_events_delay` option), or replacing polling by asynchronously notifying workers of when there's work to do. For PostgreSQL, you can use the [pg_notify](#pg-notify) plugin, which will use the PostgreSQL-only `LISTEN`/`NOTIFY` statements for that effect.
+
+#### Caveats
+
+Using `LISTEN` requires maintaining a long-lived most-idle separate database connection; this approach may not be compatible with your setup, such as if you're using a connection pooler with a particular configuration. For instance, if you're using the popular [pgbouncer](https://www.pgbouncer.org/features.html), this plugin will be incompatible with transaction pooling.
+
+There will be a slight race condition between the moment that a worker wasn't able to fetch an event, and the moment it starts listening to the notification channel; if an event arrives meanwhile, and the notification is broadcasted before the worker starts listening, the worker won't pick up this work immediately. Given enough entropy and workers, this should be a non-scenario, but a theoretical one still.
+
+### Unlogged tables
+
+By design (storing the event in the same transaction where the associated changes happen), a transactional outbox consumer requires access that the outbox table is stored in the same database the application uses, and accesses it via the master replica. As already mentioned, this means associated bookkeeping overhead in the master replica, including WAL logs and replication lag, which under extreme load, leads to all kind of issues to guarantee data consistency, despite the outbox table being unused and irrelevant in read nodes.
+
+In such cases, you may want to set the outbox table as [unlogged](https://www.postgresql.org/docs/current/sql-createtable.html#SQL-CREATETABLE-UNLOGGED), which ensures that associated write statements aren't part of WAL logs, and aren't replicated either. This will massively improve throughput of associated traffic, while preserving most of the desired transactional properties of using a transactional outbox solution, i.e. writing events along with associated data, making it visible to consumers only after transaction commits, and **in case of a clean shutdown**, ensure that data is flushed to disk.
+
+#### Caveats
+
+The last statement leads to the biggest shortcoming of this recommendation: by choosing to do unlog the outbox table, your database cannot ensure 100% consistency for its data in case of a database crash or unclean shutdown, which means you may lose events in that event. And while outbox data should not be business critical,having less than 100% event handling may be unacceptable to you.
+
+You may decide to do it temporarily though whenever you expect the level of traffic that justifies foregoing 100% consistency, but be aware that an `ALTER TABLE ... SET UNLOGGED` statement **rewrites the table**, so bear in mind of that, if you try to do this during an ongoing traffic surge / incident; the recommendation is to do this **before** the surge happens, such as a thursday before a black friday.
+
+### WAL outbox consumer (Debezium/Kafka)
+
+It takes a lot of write statements to both produce and consume from the outbox table, in the manner in which it is implemented in `tobox`.In PostgreSQL, considering each write statement on a given row will just generate a new tuple, per event, that amounts to at least 3 tuples. In the long run, and given enough volume, the health of the whole database will be limited by how quickly dead tuples are vacuumed from the outbox table.
+
+An alternative way to consume outbox events which does not require consuming events via SQL is by using a broker which is able to relay outbox events directly from the WAL logs. One such alternative is [Debezium](https://debezium.io/documentation/reference/stable/integrations/outbox.html), which relays them into Kafka streams.
+
+This solution means not using `tobox` anymore.
+
+#### Caveats
+
+This solution is, at least at the time of writing, limited to Kafka streams; if events are to be relayed to other alternatives (AWS SNS, RabbitMQ...), or there's more to your event handler than relaying, this solution will not work for you either.
+
+There are also several shortcomings to consider when using Kafka streams; for once, events are consumed one at a time, which will affect event handling turnaround time.
 
 <a id="markdown-supported-rubies" name="supported-rubies"></a>
 ## Supported Rubies

data/lib/tobox/configuration.rb

@@ -7,7 +7,8 @@ module Tobox
   class Configuration
     extend Forwardable
 
-    attr_reader :plugins, :handlers, :lifecycle_events, :arguments_handler, :default_logger, :database, :fetcher_class,
+    attr_reader :plugins, :handlers, :lifecycle_events, :arguments_handler, :default_logger, :database,
+                :fetcher_class, :worker_class,
                 :config
 
     def_delegator :@config, :[]
@@ -19,6 +20,8 @@ module Tobox
       database_uri: nil,
       database_options: nil,
       table: :outbox,
+      visibility_column: :run_at,
+      attempts_column: :attempts,
       created_at_column: nil,
       batch_size: 1,
       max_attempts: 10,
@@ -47,6 +50,7 @@ module Tobox
       @message_to_arguments = nil
       @plugins = []
       @fetcher_class = Class.new(Fetcher)
+      @worker_class = Class.new(Worker)
 
       if block
         case block.arity
@@ -117,6 +121,11 @@ module Tobox
       self
     end
 
+    def on_start_worker(&callback)
+      (@lifecycle_events[:start_worker] ||= []) << callback
+      self
+    end
+
     def on_error_worker(&callback)
       (@lifecycle_events[:error_worker] ||= []) << callback
       self
@@ -132,6 +141,16 @@ module Tobox
       self
     end
 
+    def visibility_type_bool?
+      _, visibility_info = @database.schema(@config[:table]).find do |column, _|
+        column == @config[:visibility_column]
+      end
+
+      raise Error, "a visibility column is required" unless visibility_info
+
+      visibility_info[:type] == :boolean
+    end
+
     def plugin(plugin, **options, &block)
       raise Error, "Cannot add a plugin to a frozen config" if frozen?
 
@@ -145,6 +164,7 @@ module Tobox
       extend(plugin::ConfigurationMethods) if defined?(plugin::ConfigurationMethods)
 
       @fetcher_class.__send__(:include, plugin::FetcherMethods) if defined?(plugin::FetcherMethods)
+      @worker_class.__send__(:include, plugin::WorkerMethods) if defined?(plugin::WorkerMethods)
 
       plugin.configure(self, **options, &block) if plugin.respond_to?(:configure)
     end

data/lib/tobox/fetcher.rb

@@ -19,14 +19,27 @@ module Tobox
 
       @ds = @db[@table]
 
-      run_at_conds = [
-        { Sequel[@table][:run_at] => nil },
-        (Sequel.expr(Sequel[@table][:run_at]) < Sequel::CURRENT_TIMESTAMP)
-      ].reduce { |agg, cond| Sequel.expr(agg) | Sequel.expr(cond) }
+      @visibility_column = configuration[:visibility_column]
+      @attempts_column = configuration[:attempts_column]
 
-      @pick_next_sql = @ds.where(Sequel[@table][:attempts] < max_attempts) # filter out exhausted attempts
-                          .where(run_at_conds)
-                          .order(Sequel.desc(:run_at, nulls: :first), :id)
+      @pick_next_sql = @ds
+
+      if @attempts_column
+        # filter out exhausted attempts
+        @pick_next_sql = @pick_next_sql.where(Sequel[@table][@attempts_column] < max_attempts)
+      end
+
+      if configuration.visibility_type_bool?
+        @pick_next_sql = @pick_next_sql.where(@visibility_column => false).order(:id)
+      else
+        visibility_conds = [
+          { Sequel[@table][@visibility_column] => nil },
+          (Sequel.expr(Sequel[@table][@visibility_column]) < Sequel::CURRENT_TIMESTAMP)
+        ].reduce { |agg, cond| Sequel.expr(agg) | Sequel.expr(cond) }
+
+        @pick_next_sql = @pick_next_sql.where(visibility_conds)
+                                       .order(Sequel.desc(@visibility_column, nulls: :first), :id)
+      end
 
       @batch_size = configuration[:batch_size]
 
@@ -129,17 +142,27 @@ module Tobox
       end
     end
 
-    def log_message(msg)
-      "(worker: #{@label}) -> #{msg}"
+    def log_message(msg, event)
+      tags = { type: event[:type], attempts: event[@attempts_column] }.compact
+
+      "(worker: #{@label}) -> outbox event " \
+        "(#{tags.map { |*pair| pair.join(": ") }.join(", ")}) #{msg}"
     end
 
     def mark_as_error(event, error)
+      # @type var update_params: Hash[Symbol, untyped]
       update_params = {
-        run_at: calculate_event_retry_interval(event[:attempts]),
-        attempts: Sequel[@table][:attempts] + 1,
         last_error: error.full_message(highlight: false)
       }
 
+      update_params[@attempts_column] = Sequel[@table][@attempts_column] + 1 if @attempts_column
+
+      update_params[@visibility_column] = if @configuration.visibility_type_bool?
+                                            false
+                                          else
+                                            calculate_event_retry_interval(event[@attempts_column])
+                                          end
+
       set_event_retry_attempts(event, update_params)
     end
 
@@ -177,7 +200,7 @@ module Tobox
 
     def handle_before_event(event)
       @logger.debug do
-        log_message("outbox event (type: \"#{event[:type]}\", attempts: #{event[:attempts]}) starting...")
+        log_message("starting...", event)
       end
       @before_event_handlers.each do |hd|
         hd.call(event)
@@ -185,7 +208,7 @@ module Tobox
     end
 
     def handle_after_event(event)
-      @logger.debug { log_message("outbox event (type: \"#{event[:type]}\", attempts: #{event[:attempts]}) completed") }
+      @logger.debug { log_message("completed", event) }
       @after_event_handlers.each do |hd|
         hd.call(event)
       end
@@ -193,9 +216,9 @@ module Tobox
 
     def handle_error_event(event, error)
       @logger.error do
-        log_message("outbox event (type: \"#{event[:type]}\", attempts: #{event[:attempts]}) failed with error\n" \
+        log_message("failed with error\n" \
                     "#{error.class}: #{error.message}\n" \
-                    "#{error.backtrace.join("\n")}")
+                    "#{error.backtrace.join("\n")}", event)
       end
       @error_event_handlers.each do |hd|
         hd.call(event, error)

data/lib/tobox/plugins/datadog/configuration.rb

@@ -13,7 +13,7 @@ module Datadog
             if Gem::Version.new(DDTrace::VERSION::STRING) >= Gem::Version.new("1.13.0")
               option :enabled do |o|
                 o.type :bool
-                o.env "DD_TOBOX_SIDEKIQ_ENABLED"
+                o.env "DD_TOBOX_ENABLED"
                 o.default true
               end
 
@@ -30,7 +30,7 @@ module Datadog
               end
             else
               option :enabled do |o|
-                o.default { env_to_bool("DD_TOBOX_SIDEKIQ_ENABLED", true) }
+                o.default { env_to_bool("DD_TOBOX_ENABLED", true) }
                 o.lazy
               end
 

data/lib/tobox/plugins/datadog.rb

@@ -58,7 +58,7 @@ module Tobox
 
           span.set_tag("tobox.event.id", event[:id])
           span.set_tag("tobox.event.type", event[:type])
-          span.set_tag("tobox.event.retry", event[:attempts])
+          span.set_tag("tobox.event.retry", event[@attempts_column]) if @attempts_column
           span.set_tag("tobox.event.table", @db_table)
           span.set_tag("tobox.event.delay", (Time.now.utc - event[:created_at]).to_f)
 

data/lib/tobox/plugins/event_grouping.rb

@@ -26,8 +26,14 @@ module Tobox
           total_from_group = @ds.where(@group_column => group).count
 
           event_ids = @ds.where(@group_column => group)
-                         .order(Sequel.desc(:run_at, nulls: :first), :id)
-                         .for_update.skip_locked.select_map(:id)
+
+          event_ids = if @configuration.visibility_type_bool?
+                        event_ids.order(:id)
+                      else
+                        event_ids.order(Sequel.desc(@visibility_column, nulls: :first), :id)
+                      end
+
+          event_ids = event_ids.for_update.skip_locked.select_map(:id)
 
           if event_ids.size != total_from_group
             # this happens if concurrent workers locked different rows from the same group,

data/lib/tobox/plugins/pg_notify.rb

@@ -0,0 +1,85 @@
+# frozen_string_literal: true
+
+module Tobox
+  module Plugins
+    module PgNotify
+      class Notifier
+        def initialize(config)
+          @config = config
+          @running = false
+          @notifier_mutex = Thread::Mutex.new
+          @notifier_semaphore = Thread::ConditionVariable.new
+        end
+
+        def start
+          return if @running
+
+          config = @config
+
+          @db = Sequel.connect(config.database.opts.merge(max_connections: 1))
+
+          raise Error, "this plugin only works with postgresql" unless @db.database_type == :postgres
+
+          @db.loggers = config.database.loggers
+          Array(config.lifecycle_events[:database_connect]).each { |cb| cb.call(@db) }
+
+          channel = config[:notifier_channel]
+
+          @th = Thread.start do
+            Thread.current.name = "outbox-notifier"
+
+            @db.listen(channel, loop: true) do
+              signal
+            end
+          end
+
+          @running = true
+        end
+
+        def stop
+          return unless @running
+
+          @th.terminate
+
+          @db.disconnect
+
+          @running = false
+        end
+
+        def wait
+          @notifier_mutex.synchronize do
+            @notifier_semaphore.wait(@notifier_mutex)
+          end
+        end
+
+        def signal
+          @notifier_mutex.synchronize do
+            @notifier_semaphore.signal
+          end
+        end
+      end
+
+      module WorkerMethods
+        attr_writer :notifier
+
+        def wait_for_work
+          @notifier.wait
+        end
+      end
+
+      class << self
+        def configure(config)
+          config.config[:notifier_channel] = :outbox_notifications
+
+          notifier = Notifier.new(config)
+
+          config.on_start_worker { |wk| wk.notifier = notifier }
+
+          config.on_start(&notifier.method(:start))
+          config.on_stop(&notifier.method(:stop))
+        end
+      end
+    end
+    register_plugin :pg_notify, PgNotify
+  end
+end

data/lib/tobox/plugins/progress.rb

@@ -11,15 +11,22 @@ module Tobox
         private
 
         def do_fetch_events
-          # mark events as invisible by using run_at as a visibility timeout
+          # mark events as invisible
+
+          # @type var mark_as_fetched_params: Hash[Symbol, untyped]
           mark_as_fetched_params = {
-            run_at: Sequel.date_add(
-              Sequel::CURRENT_TIMESTAMP,
-              seconds: @configuration[:visibility_timeout]
-            ),
-            attempts: Sequel[@table][:attempts] + 1,
             last_error: nil
           }
+          mark_as_fetched_params[@attempts_column] = Sequel[@table][@attempts_column] + 1 if @attempts_column
+
+          mark_as_fetched_params[@visibility_column] = if @configuration.visibility_type_bool?
+                                                         true
+                                                       else
+                                                         Sequel.date_add(
+                                                           Sequel::CURRENT_TIMESTAMP,
+                                                           seconds: @configuration[:visibility_timeout]
+                                                         )
+                                                       end
 
           if @ds.supports_returning?(:update)
             @ds.where(id: fetch_event_ids).returning.update(mark_as_fetched_params)
@@ -36,7 +43,7 @@ module Tobox
         end
 
         def set_event_retry_attempts(event, update_params)
-          update_params.delete(:attempts)
+          update_params.delete(@attempts_column)
           super
         end
 

data/lib/tobox/plugins/sentry.rb

@@ -38,15 +38,15 @@ module Tobox
           scope = ::Sentry.get_current_scope
 
           scope.set_contexts(tobox: {
-                               id: event[:id],
-                               type: event[:type],
-                               attempts: event[:attempts],
-                               created_at: event[:created_at],
-                               run_at: event[:run_at],
-                               last_error: event[:last_error]&.byteslice(0..1000),
-                               version: Tobox::VERSION,
-                               db_adapter: @db_scheme
-                             })
+            id: event[:id],
+            type: event[:type],
+            @attempts_column => event[@config[:attempts_column]],
+            created_at: event[:created_at],
+            @visibility_column => event[@config[:visibility_column]],
+            last_error: event[:last_error]&.byteslice(0..1000),
+            version: Tobox::VERSION,
+            db_adapter: @db_scheme
+          }.compact)
           scope.set_tags(
             outbox: @db_table,
             event_id: event[:id],
@@ -116,7 +116,9 @@ module Tobox
         end
 
         def capture_exception(event, error)
-          if ::Sentry.configuration.tobox.report_after_retries && event[:attempts] && event[:attempts] < @max_attempts
+          if ::Sentry.configuration.tobox.report_after_retries &&
+             event[@config[:attempts_column]] &&
+             event[@config[:attempts_column]] < @max_attempts
             return
           end
 

data/lib/tobox/plugins/stats.rb

@@ -52,7 +52,14 @@ module Tobox
 
           if @created_at_column
             # discard already handled events
-            @oldest_event_age_ds = @outbox_ds.where(last_error: nil, run_at: nil).order(Sequel.asc(:id))
+            #
+            @oldest_event_age_ds = @outbox_ds.where(last_error: nil)
+            @oldest_event_age_ds = if config.visibility_type_bool?
+                                     @oldest_event_age_ds.where(config[:visibility_column] => false)
+                                   else
+                                     @oldest_event_age_ds.where(config[:visibility_column] => nil)
+                                   end
+            @oldest_event_age_ds = @oldest_event_age_ds.order(Sequel.asc(:id))
           end
 
           logger = config.default_logger
@@ -104,7 +111,7 @@ module Tobox
           stats = @outbox_ds.group_and_count(
             Sequel.case([
                           [{ last_error: nil }, "pending_count"],
-                          [Sequel.expr([:attempts]) < @max_attempts, "failing_count"]
+                          [Sequel.expr(@config[:attempts_column]) < @max_attempts, "failing_count"]
                         ],
                         "failed_count").as(:status)
           )

data/lib/tobox/pool/fiber_pool.rb

@@ -1,7 +1,7 @@
 # frozen_string_literal: true
 
 require "timeout"
-require "fiber_scheduler"
+require "async/scheduler"
 
 module Tobox
   class FiberPool < Pool
@@ -20,18 +20,19 @@ module Tobox
         Thread.current.name = "tobox-fibers-thread"
 
         begin
-          FiberScheduler do
-            @fiber_mtx.synchronize do
-              @workers.each do |worker|
-                @fibers << start_fiber_worker(worker)
-              end
-              @fiber_cond.signal
+          Fiber.set_scheduler(Async::Scheduler.new)
+
+          @fiber_mtx.synchronize do
+            @workers.each do |worker|
+              @fibers << start_fiber_worker(worker)
             end
+            @fiber_cond.signal
           end
         rescue KillError
           @fibers.each { |f| f.raise(KillError) }
         end
       end
+
       @fiber_mtx.synchronize do
         @fiber_cond.wait(@fiber_mtx)
       end
@@ -43,18 +44,29 @@ module Tobox
 
       super
 
-      @fiber_thread.join(shutdown_timeout)
+      th = @fiber_thread
+
+      return unless th
+
+      th.join(shutdown_timeout)
 
-      return unless @fiber_thread.alive?
+      return unless th.alive?
 
-      @fiber_thread.raise(KillError)
-      @fiber_thread.join(grace_shutdown_timeout)
-      @fiber_thread.kill
-      @fiber_thread.join(1)
+      th.raise(KillError)
+      th.join(grace_shutdown_timeout)
+      th.kill
+      th.join(1)
     end
 
     private
 
+    def handle_exception(wrk, exc)
+      # noop
+      return if exc.is_a?(::Async::Stop)
+
+      super
+    end
+
     def start_fiber_worker(worker)
       Fiber.schedule do
         do_work(worker)
@@ -67,12 +79,13 @@ module Tobox
 
             raise Error, "worker not found" unless idx
 
-            subst_worker = Worker.new(worker.label, @configuration)
+            subst_worker = @configuration.worker_class.new(worker.label, @configuration)
             @workers[idx] = subst_worker
-            subst_fiber = start_fiber_worker(subst_worker)
-            @fiber_mtx.synchronize { @fibers << subst_fiber }
+            @fibers << start_fiber_worker(subst_worker)
           end
         end
+      rescue KillError
+        # noop
       end
     end
   end

data/lib/tobox/pool/threaded_pool.rb

@@ -1,20 +1,18 @@
 # frozen_string_literal: true
 
-require "monitor"
-
 module Tobox
   class ThreadedPool < Pool
     def initialize(_configuration)
       @parent_thread = Thread.main
       @threads = []
-      @threads.extend(MonitorMixin)
+      @threads_mutex = Thread::Mutex.new
       super
     end
 
     def start
       @workers.each do |wk|
         th = start_thread_worker(wk)
-        @threads.synchronize do
+        @threads_mutex.synchronize do
           @threads << th
         end
       end
@@ -32,7 +30,7 @@ module Tobox
         start = Process.clock_gettime(::Process::CLOCK_MONOTONIC)
 
         loop do
-          terminating_th = @threads.synchronize { @threads.first }
+          terminating_th = @threads_mutex.synchronize { @threads.first }
 
           return unless terminating_th
 
@@ -47,9 +45,9 @@ module Tobox
       join.call(shutdown_timeout)
 
       # hard exit
-      @threads.synchronize { @threads.each { |th| th.raise(KillError) } }
+      @threads_mutex.synchronize { @threads.each { |th| th.raise(KillError) } }
       join.call(grace_shutdown_timeout)
-      @threads.synchronize { @threads.each(&:kill) }
+      @threads_mutex.synchronize { @threads.each(&:kill) }
       join.call(1)
     end
 
@@ -61,7 +59,7 @@ module Tobox
 
         do_work(worker)
 
-        @threads.synchronize do
+        @threads_mutex.synchronize do
           @threads.delete(Thread.current)
 
           if worker.finished? && @running
@@ -69,7 +67,7 @@ module Tobox
 
             raise Error, "worker not found" unless idx
 
-            subst_worker = Worker.new(worker.label, @configuration)
+            subst_worker = @configuration.worker_class.new(worker.label, @configuration)
             @workers[idx] = subst_worker
             subst_thread = start_thread_worker(subst_worker)
             @threads << subst_thread

data/lib/tobox/pool.rb

@@ -9,7 +9,7 @@ module Tobox
       @logger = @configuration.default_logger
       @num_workers = configuration[:concurrency]
       @workers = Array.new(@num_workers) do |idx|
-        Worker.new("tobox-worker-#{idx}", configuration)
+        @configuration.worker_class.new("tobox-worker-#{idx}", configuration)
       end
       @worker_error_handlers = Array(@configuration.lifecycle_events[:error_worker])
       @running = true
@@ -22,19 +22,28 @@ module Tobox
       @running = false
     end
 
+    private
+
     def do_work(wrk)
       wrk.work
-    rescue KillError
-    # noop
     rescue Exception => e # rubocop:disable Lint/RescueException
-      wrk.finish!
-      @logger.error do
-        "(worker: #{wrk.label}) -> " \
-          "crashed with error\n" \
-          "#{e.class}: #{e.message}\n" \
-          "#{e.backtrace.join("\n")}"
+      handle_exception(wrk, e)
+    end
+
+    def handle_exception(wrk, exc)
+      case exc
+      when KillError
+      # noop
+      when Exception
+        wrk.finish!
+        @logger.error do
+          "(worker: #{wrk.label}) -> " \
+            "crashed with error\n" \
+            "#{exc.class}: #{exc.message}\n" \
+            "#{exc.backtrace.join("\n")}"
+        end
+        @worker_error_handlers.each { |hd| hd.call(exc) }
       end
-      @worker_error_handlers.each { |hd| hd.call(e) }
     end
   end
 

data/lib/tobox/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Tobox
-  VERSION = "0.6.1"
+  VERSION = "0.7.0"
 end

data/lib/tobox/worker.rb

@@ -11,9 +11,13 @@ module Tobox
       @fetcher = configuration.fetcher_class.new(label, configuration)
       @finished = false
 
-      return unless (message_to_arguments = configuration.arguments_handler)
+      if (message_to_arguments = configuration.arguments_handler)
+        define_singleton_method(:message_to_arguments, &message_to_arguments)
+      end
 
-      define_singleton_method(:message_to_arguments, &message_to_arguments)
+      Array(configuration.lifecycle_events[:start_worker]).each do |hd|
+        hd.call(self)
+      end
     end
 
     def finished?
@@ -47,7 +51,11 @@ module Tobox
 
       return if @finished
 
-      sleep(@wait_for_events_delay) if sum_fetched_events.zero?
+      wait_for_work if sum_fetched_events.zero?
+    end
+
+    def wait_for_work
+      sleep(@wait_for_events_delay)
     end
 
     def message_to_arguments(event)

data/lib/tobox.rb

@@ -4,25 +4,23 @@ require "sequel"
 
 require_relative "tobox/version"
 
-require "mutex_m"
-
 module Tobox
   class Error < StandardError; end
 
   EMPTY = [].freeze
 
   module Plugins
+    PLUGINS_MUTEX = Thread::Mutex.new
     @plugins = {}
-    @plugins.extend(Mutex_m)
 
     # Loads a plugin based on a name. If the plugin hasn't been loaded, tries to load
     # it from the load path under "httpx/plugins/" directory.
     #
     def self.load_plugin(name)
       h = @plugins
-      unless (plugin = h.synchronize { h[name] })
+      unless (plugin = PLUGINS_MUTEX.synchronize { h[name] })
         require "tobox/plugins/#{name}"
-        raise "Plugin #{name} hasn't been registered" unless (plugin = h.synchronize { h[name] })
+        raise "Plugin #{name} hasn't been registered" unless (plugin = PLUGINS_MUTEX.synchronize { h[name] })
       end
       plugin
     end
@@ -31,7 +29,7 @@ module Tobox
     #
     def self.register_plugin(name, mod)
       h = @plugins
-      h.synchronize { h[name] = mod }
+      PLUGINS_MUTEX.synchronize { h[name] = mod }
     end
   end
 
@@ -54,11 +52,12 @@ module Tobox
   #       Tobox.raise_batch_error(batch_errors)
   #     end
   def self.raise_batch_errors(batch_errors)
-    unless batch_errors.respond_to?(:to_hash) && batch_errors.all? { |k, v| k.is_a?(Integer) && v.is_a?(Exception) }
-      raise "batch errors must be an array of index-to-exception tuples"
+    batch_errors = Hash.try_convert(batch_errors)
+    unless batch_errors && batch_errors.all? { |k, v| k.is_a?(Integer) && v.is_a?(Exception) }
+      raise Error, "batch errors must be an array of index-to-exception tuples"
     end
 
-    throw(:tobox_batch_errors, batch_errors.to_h)
+    throw(:tobox_batch_errors, batch_errors)
   end
 end
 

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: tobox
 version: !ruby/object:Gem::Version
-  version: 0.6.1
+  version: 0.7.0
 platform: ruby
 authors:
 - HoneyryderChuck
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2024-10-30 00:00:00.000000000 Z
+date: 2024-12-18 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: logger
@@ -61,6 +61,7 @@ files:
 - lib/tobox/plugins/datadog/patcher.rb
 - lib/tobox/plugins/event_grouping.rb
 - lib/tobox/plugins/inbox.rb
+- lib/tobox/plugins/pg_notify.rb
 - lib/tobox/plugins/progress.rb
 - lib/tobox/plugins/sentry.rb
 - lib/tobox/plugins/stats.rb