tobox 0.3.2 → 0.4.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: cb81b3eab97621080f2cf55ba787943d26a11a38c0e807313a810adc8c0d8136
-  data.tar.gz: 0556f5ba59e7b2feb8a47b04d53e00869d041e747a6789c7f4fe4067f17ad795
+  metadata.gz: 4960d6151461ab6e051605687d99cc69089e498aaea66d97416b92158e48cc4e
+  data.tar.gz: e3f2e0a6408696ff1852a484101ca6fa95e5e5e713d1ccb0b829040bb0a748ad
 SHA512:
-  metadata.gz: d109a8b77dafed4066d60cff257541e3eb1a7a6950efa7e94ec75e6e68fb1cf60e5684edc46b9ec99f0f56573807f354b16b7ee5c188b94a848cc52423528c30
-  data.tar.gz: f8d87bd79921b774326e7db4afd532696814c2b38d5e45deae582eaacdd7e7f161c92b81d627ab2b1c17c51bad0a8a0a1be000839cd675da885a78107e0f4d16
+  metadata.gz: ac5d961bc737299d4cd1e3d353955f97b12d984e4b3378b56dbeaf84c5d969ed3b6c02902c70b7ed0fd14230696d44992012d35053565988349b13405f6144ef
+  data.tar.gz: 3a921e5ca3f42fa47f8485a0a9bf44c42c31f0f767cfafc81ef1af6d6e43ac83966baffcb20ff28df9c90abb8a4c16e6829c4e9aa9e86ca009ab4fde38a92bd1

data/CHANGELOG.md

@@ -1,13 +1,64 @@
 ## [Unreleased]
 
+## [0.4.1] - 2023-05-24
 
-## [0.3.2] - 2022-03-06
+### Features
+
+#### `on_database_connect`
+
+this adds an extension point for internal sequel database objects, in cases where some tweaks are required (such as in the case of, when using database SSL proxies, setting connection validators).
+
+```ruby
+# tobox.rb
+on_database_connect do |db|
+  db.extension(:connection_validator)
+end
+```
+
+## [0.4.0] - 2023-05-19
+
+### Features
+
+#### `:stats` plugin
+
+The `:stats` plugin collects statistics related with the outbox table periodically, and exposes them to app code (which can then relay them to a statsD collector, or similar tool).
+
+
+```ruby
+plugin(:stats)
+on_stats(5) do |stats_collector| # every 5 seconds
+  stats = stats_collector.collect
+  StatsD.gauge('outbox_pending_backlog', stats[:pending_count])
+end
+```
+
+Read more about it in [the project README](https://gitlab.com/os85/tobox#stats).
+
+#### on_start/on_stop callbacks
+
+The `on_start` and `on_stop` callbacks can now be defined in `tobox` configuration:
+
+```ruby
+# tobox.rb
+on_start do
+  puts "tobox is starting..."
+end
+on_stop do
+  puts "tobox is stopping..."
+end
+```
+
+### Bugfixes
+
+* tobox configuration file is now only loaded after everything else, so access to application code is guaranteed.
+
+## [0.3.2] - 2023-03-06
 
 ### Bugfixes
 
 * allow sentry error capture if `report_after_retries` option is turned off.
 
-## [0.3.1] - 2022-03-03
+## [0.3.1] - 2023-03-03
 
 ### Bugfixes
 

data/README.md

@@ -20,6 +20,7 @@ Simple, data-first events processing framework based on the [transactional outbo
   - [Zeitwerk](#zeitwerk)
   - [Sentry](#sentry)
   - [Datadog](#datadog)
+  - [Stats](#stats)
 - [Supported Rubies](#supported-rubies)
 - [Rails support](#rails-support)
 - [Why?](#why)
@@ -267,6 +268,20 @@ callback executed when an exception was raised in the worker, before processing
 on_error_worker { |exception| Sentry.capture_exception(exception) }
 ```
 
+### `on_database_connect { |db| }`
+
+Callback executed right after initializing the `sequel` database object. This can be used, for example, to load database-level extensions and plugins, and set parameters (such as connection pool tweaks). This callback will also be used by plugins which instantiate its own separate database objects (such as in the case of the [stats](#stats) plugin).
+
+This callback won't be executed if the database object is created outside of `tobox` configuration parameters.
+
+
+```ruby
+on_database_connect do |db|
+  db.extension(:connection_validator)
+  db.pool.connection_validation_timeout = -1
+end
+```
+
 ### `message_to_arguments { |event| }`
 
 if exposing raw data to the `on` handlers is not what you'd want, you can always override the behaviour by providing an alternative "before/after fetcher" implementation.
@@ -482,6 +497,69 @@ end
 plugin(:datadog)
 ```
 
+<a id="markdown-datadog" name="stats"></a>
+### Stats
+
+The `stats` plugin collects statistics related with the outbox table periodically, and exposes them to app code (which can then relay them to a statsD collector, or similar tool).
+
+```ruby
+plugin(:stats)
+on_stats(5) do |stats_collector| # every 5 seconds
+  stats = stats_collector.collect
+  #
+  # stats => {
+  #   pending_count: number of new events in the outbox table
+  #   failing_count: number of events which have failed processing but haven't reached the threshold
+  #   failed_count: number of events which have failed the max number of tries
+  #   inbox_count: (if used) number of events marked as received in the inbox table
+  # }
+  #
+  # now you can send them to your statsd collector
+  #
+  StatsD.gauge('outbox_pending_backlog', stats[:pending_count])
+end
+```
+
+#### Bring your own leader election
+
+The stats collection runs on every `tobox` initiated. If you're launching it in multiple servers / containers / pods, this means you'll be collecting statistics about the same database on all of these instances. This may not be desirable, and you may want to do this collection in a single instance. This is not a problem that `tobox` can solve by itself, so you'll have to take care of that yourself. Still, here are some cheap recommendations.
+
+##### Postgres advisory locks
+
+If your database is PostgreSQL, you can leverage session-level [advisory locks](https://www.postgresql.org/docs/current/explicit-locking.html#ADVISORY-LOCKS) to ensure single-instance access to this functionality. `tobox` also exposes the database instance to the `on_stats` callback:
+
+```ruby
+c.on_stats(5) do |stats_collector, db|
+  if db.get(Sequel.function(:pg_try_advisory_lock, 1))
+    stats = stats_collector.collect
+    StatsD.gauge('outbox_pending_backlog', stats[:pending_count])
+  end
+end
+```
+
+If a server goes down, one of the remaining ones will acquire the lock and ensure stats processing.
+
+##### Redis distributed locks
+
+If you're already using [redis](https://redis.io/), you can use its distributed lock feature to achieve the goal:
+
+```ruby
+# using redlock
+c.on_stats(5) do |stats_collector, db|
+  begin
+    lock_info = lock_manager.lock("outbox", 5000)
+
+    stats = stats_collector.collect
+    StatsD.gauge('outbox_pending_backlog', stats[:pending_count])
+
+    # extend to hold the lock for the next loop
+    lock_info = lock_manager.lock("outbox", 5000, extend: lock_info)
+  rescue Redlock::LockError
+    # some other server already has the lock, try later
+  end
+end
+```
+
 <a id="markdown-supported-rubies" name="supported-rubies"></a>
 ## Supported Rubies
 

data/lib/tobox/application.rb

@@ -6,18 +6,23 @@ module Tobox
       @configuration = configuration
       @running = false
 
-      worker = @configuration[:worker]
+      @on_start_handlers = Array(configuration.lifecycle_events[:on_start])
+      @on_stop_handlers = Array(configuration.lifecycle_events[:on_stop])
+
+      worker = configuration[:worker]
 
       @pool = case worker
               when :thread then ThreadedPool
               when :fiber then FiberPool
               else worker
-              end.new(@configuration)
+              end.new(configuration)
     end
 
     def start
       return if @running
 
+      @on_start_handlers.each(&:call)
+
       @pool.start
       @running = true
     end
@@ -25,6 +30,8 @@ module Tobox
     def stop
       return unless @running
 
+      @on_stop_handlers.each(&:call)
+
       @pool.stop
 
       @running = false

data/lib/tobox/cli.rb

@@ -18,15 +18,15 @@ module Tobox
     def run
       options = @options
 
+      # boot
+      options.fetch(:require).each(&method(:require))
+
       config = Configuration.new do |c|
         c.instance_eval(File.read(options.fetch(:config_file)), options.fetch(:config_file), 1)
       end
 
       logger = config.default_logger
 
-      # boot
-      options.fetch(:require).each(&method(:require))
-
       # signals
       pipe_read, pipe_write = IO.pipe
       %w[INT TERM].each do |sig|

data/lib/tobox/configuration.rb

@@ -7,7 +7,7 @@ module Tobox
   class Configuration
     extend Forwardable
 
-    attr_reader :handlers, :lifecycle_events, :arguments_handler, :default_logger
+    attr_reader :handlers, :lifecycle_events, :arguments_handler, :default_logger, :database
 
     def_delegator :@config, :[]
 
@@ -60,6 +60,24 @@ module Tobox
       @default_logger = @config[:logger] || Logger.new(STDERR, formatter: DEFAULT_LOG_FORMATTER) # rubocop:disable Style/GlobalStdStream
       @default_logger.level = @config[:log_level] || (env == "production" ? Logger::INFO : Logger::DEBUG)
 
+      @database = if @config[:database_uri]
+                    database_opts = {}
+                    database_opts[:max_connections] = @config[:concurrency] if @config[:worker] == :thread
+                    db = Sequel.connect(@config[:database_uri].to_s, database_opts)
+                    Array(@lifecycle_events[:database_connect]).each { |cb| cb.call(db) }
+                    db
+                  else
+                    Sequel::DATABASES.first
+                  end
+      raise Error, "no database found" unless @database
+
+      if @database.frozen?
+        raise "#{@database} must have the :date_arithmetic extension loaded" unless Sequel.respond_to?(:date_add)
+      else
+        @database.extension :date_arithmetic
+        @database.loggers << @default_logger unless @config[:environment] == "production"
+      end
+
       freeze
     end
 
@@ -70,6 +88,16 @@ module Tobox
       self
     end
 
+    def on_start(&callback)
+      (@lifecycle_events[:on_start] ||= []) << callback
+      self
+    end
+
+    def on_stop(&callback)
+      (@lifecycle_events[:on_stop] ||= []) << callback
+      self
+    end
+
     def on_before_event(&callback)
       (@lifecycle_events[:before_event] ||= []) << callback
       self
@@ -90,6 +118,11 @@ module Tobox
       self
     end
 
+    def on_database_connect(&callback)
+      (@lifecycle_events[:database_connect] ||= []) << callback
+      self
+    end
+
     def message_to_arguments(&callback)
       @arguments_handler = callback
       self
@@ -116,6 +149,7 @@ module Tobox
       @handlers.each_value(&:freeze).freeze
       @lifecycle_events.each_value(&:freeze).freeze
       @plugins.freeze
+      @database.freeze
       super
     end
 

data/lib/tobox/fetcher.rb

@@ -10,13 +10,7 @@ module Tobox
 
       @logger = @configuration.default_logger
 
-      database_uri = @configuration[:database_uri]
-      @db = database_uri ? Sequel.connect(database_uri.to_s) : Sequel::DATABASES.first
-      raise Error, "no database found" unless @db
-
-      @db.extension :date_arithmetic
-
-      @db.loggers << @logger unless @configuration[:environment] == "production"
+      @db = configuration.database
 
       @table = configuration[:table]
       @group_column = configuration[:group_column]

data/lib/tobox/plugins/sentry.rb

@@ -30,16 +30,16 @@ module Tobox
 
           scope = ::Sentry.get_current_scope
 
-          scope.set_contexts(
-            id: event[:id],
-            type: event[:type],
-            attempts: event[:attempts],
-            created_at: event[:created_at],
-            run_at: event[:run_at],
-            last_error: event[:last_error]&.byteslice(0..1000),
-            version: Tobox::VERSION,
-            db_adapter: @db_scheme
-          )
+          scope.set_contexts(tobox: {
+                               id: event[:id],
+                               type: event[:type],
+                               attempts: event[:attempts],
+                               created_at: event[:created_at],
+                               run_at: event[:run_at],
+                               last_error: event[:last_error]&.byteslice(0..1000),
+                               version: Tobox::VERSION,
+                               db_adapter: @db_scheme
+                             })
           scope.set_tags(
             outbox: @db_table,
             event_id: event[:id],

data/lib/tobox/plugins/stats.rb

@@ -0,0 +1,123 @@
+# frozen_string_literal: true
+
+module Tobox
+  module Plugins
+    module Stats
+      module ConfigurationMethods
+        attr_reader :stats_interval_seconds
+
+        def on_stats(stats_interval_seconds, &callback)
+          @stats_interval_seconds = stats_interval_seconds
+
+          (@lifecycle_events[:stats] ||= []) << callback
+          self
+        end
+      end
+
+      class StatsEmitter
+        def initialize(config)
+          @config = config
+          @running = false
+        end
+
+        def start
+          return if @running
+
+          config = @config
+
+          interval = config.stats_interval_seconds
+          @stats_handlers = Array(config.lifecycle_events[:stats])
+
+          return if @stats_handlers.empty?
+
+          @error_handlers = Array(config.lifecycle_events[:error_worker])
+
+          @max_attempts = config[:max_attempts]
+
+          @db = Sequel.connect(config.database.opts.merge(single_threaded: true))
+          Array(config.lifecycle_events[:database_connect]).each { |cb| cb.call(@db) }
+
+          @outbox_table = config[:table]
+          @outbox_ds = @db[@outbox_table]
+
+          inbox_table = config[:inbox_table]
+          @inbox_ds = @db[inbox_table] if inbox_table
+
+          logger = config.default_logger
+
+          stats = method(:collect_event_stats)
+          stats.instance_eval do
+            alias collect call
+          end
+
+          @th = Thread.start do
+            Thread.current.name = "outbox-stats"
+
+            loop do
+              logger.debug { "stats worker: sleep for #{interval}s..." }
+              sleep interval
+
+              begin
+                emit_event_stats(stats)
+              rescue RuntimeError => e
+                @error_handlers.each { |hd| hd.call(e) }
+              end
+
+              break unless @running
+            end
+          end
+
+          @running = true
+        end
+
+        def stop
+          return unless @running
+
+          @th.terminate
+
+          @db.disconnect
+
+          @running = false
+        end
+
+        private
+
+        def emit_event_stats(stats)
+          @stats_handlers.each do |hd|
+            hd.call(stats, @db)
+          end
+        end
+
+        def collect_event_stats
+          stats = @outbox_ds.group_and_count(
+            Sequel.case([
+                          [{ last_error: nil }, "pending_count"],
+                          [Sequel.expr([:attempts]) < @max_attempts, "failing_count"]
+                        ],
+                        "failed_count").as(:status)
+          )
+          stats = stats.as_hash(:status, :count).transform_keys(&:to_sym)
+
+          # fill it in
+          stats[:pending_count] ||= 0
+          stats[:failing_count] ||= 0
+          stats[:failed_count] ||= 0
+
+          stats[:inbox_count] = @inbox_ds.count if @inbox_ds
+          stats
+        end
+      end
+
+      class << self
+        def configure(config)
+          emitter = StatsEmitter.new(config)
+
+          config.on_start(&emitter.method(:start))
+          config.on_stop(&emitter.method(:stop))
+        end
+      end
+    end
+
+    register_plugin :stats, Stats
+  end
+end

data/lib/tobox/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Tobox
-  VERSION = "0.3.2"
+  VERSION = "0.4.1"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: tobox
 version: !ruby/object:Gem::Version
-  version: 0.3.2
+  version: 0.4.1
 platform: ruby
 authors:
 - HoneyryderChuck
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2023-03-06 00:00:00.000000000 Z
+date: 2023-05-24 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: sequel
@@ -46,6 +46,7 @@ files:
 - lib/tobox/plugins/datadog/integration.rb
 - lib/tobox/plugins/datadog/patcher.rb
 - lib/tobox/plugins/sentry.rb
+- lib/tobox/plugins/stats.rb
 - lib/tobox/plugins/zeitwerk.rb
 - lib/tobox/pool.rb
 - lib/tobox/pool/fiber_pool.rb
@@ -77,7 +78,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.3.7
+rubygems_version: 3.4.6
 signing_key:
 specification_version: 4
 summary: Transactional outbox pattern implementation in ruby