sbmt-outbox 6.2.0 → 6.3.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 7a8ef5d8d2289f5e84b0218df4d0ae6ebf4c3887891f4a694c53f5b3fb2a617c
-  data.tar.gz: 5df6474dd32187affba9667f86e3638dfb51349519626e247e0928d9b44bdb4b
+  metadata.gz: 952db6c0164054fe8a19297680900749dacf124e2987d2a49a02304f4a02f0db
+  data.tar.gz: b765f1c66e936431070f61e946ac9a50b7b2b2966af01f124923c8cb5f70cdab
 SHA512:
-  metadata.gz: 4635896765ebef22224ea23123cbcd6d47facd6c88793f5ecd419cab54fa7afc281527cd43a1fe7466a098c53008d6d692903a85e1d23cd53080713f770ae010
-  data.tar.gz: 0b632a652cf286f792fe57dfa373409804782ccc5a8bfd6f7859edde9a6067de8ebdd9f5b653e84e61840e1c252607c2fd943746bd7ef409a415707a8c5a0e98
+  metadata.gz: 2c6e1a536dc8ece25ceb807db39e1f4f52fa70f79d6eadfd834707da8d7f4593575cc0563e638133f973fa4205c8e9fe14f1e28ffbbe44bf1f5f35ec1748fb9a
+  data.tar.gz: f5b8196a367b1c38ed65fdaa95cecdd7dd245524c17e317088f73387ea114262d81ed1062b4f1486cf084f55e0a31b64b08999a0d42b30117e41c841bbb15012

data/README.md

@@ -3,12 +3,12 @@
 
 # Sbmt-Outbox
 
+<img src="https://raw.githubusercontent.com/SberMarket-Tech/sbmt-outbox/master/.github/outbox-logo.png" alt="sbmt-outbox logo" height="220" align="right" />
+
 Microservices often publish messages after a transaction has been committed. Writing to the database and publishing a message are two separate transactions, so they must be atomic. A failed publication of a message could lead to a critical failure of the business process.
 
 The Outbox pattern provides a reliable solution for message publishing. The idea behind this approach is to have an "outgoing message table" in the service's database. Before the main transaction completes, a new message row is added to this table. As a result, two actions take place as part of a single transaction. An asynchronous process retrieves new rows from the database table and, if they exist, publishes the messages to the broker.
 
-Read more about the Outbox pattern at https://microservices.io/patterns/data/transactional-outbox.html
-
 ## Installation
 
 Add this line to your application's Gemfile:
@@ -97,6 +97,91 @@ Example of a Grafana dashboard that you can import [from a file](./examples/graf
 
 ## Manual configuration
 
+### `Outboxfile`
+
+First of all you shoudl create an `Outboxfile` at the root of your application with the following code:
+
+```ruby
+# frozen_string_literal: true
+
+require_relative "config/environment"
+
+# Comment out this line if you don't want to use a metrics exporter
+Yabeda::Prometheus::Exporter.start_metrics_server!
+```
+
+### `config/initializers/outbox.rb`
+
+The `config/initializers/outbox.rb` file contains the overall general configuration.
+
+```ruby
+# config/initializers/outbox.rb
+
+Rails.application.config.outbox.tap do |config|
+  config.redis = {url: ENV.fetch("REDIS_URL")} # Redis is used as a coordinator service
+  config.paths << Rails.root.join("config/outbox.yml").to_s # optional; configuration file paths, deep merged at the application start, useful with Rails engines
+
+  # optional (worker v2: default)
+  c.poller = ActiveSupport::OrderedOptions.new.tap do |pc|
+    # max parallel threads (per box-item, globally)
+    pc.concurrency = 6
+    # max threads count (per worker process)
+    pc.threads_count = 1
+    # maximum processing time of the batch, after which the batch will be considered hung and processing will be aborted
+    pc.general_timeout = 60
+    # poll buffer consists of regular items (errors_count = 0, i.e. without any processing errors) and retryable items (errors_count > 0)
+    # max poll buffer size = regular_items_batch_size + retryable_items_batch_size
+    pc.regular_items_batch_size = 200
+    pc.retryable_items_batch_size = 100
+
+    # poll tactic: default is optimal for most cases: rate limit + redis job-queue size threshold
+    # poll tactic: aggressive is for high-intencity data: without rate limits + redis job-queue size threshold
+    # poll tactic: low-priority is for low-intencity data: rate limits + redis job-queue size threshold + + redis job-queue lag threshold
+    pc.tactic = "default"
+    # number of batches that one thread will process per rate interval
+    pc.rate_limit = 60
+    # rate interval in seconds
+    pc.rate_interval = 60
+    # mix / max redis job queue thresholds per box-item for default / aggressive / low-priority poll tactics
+    pc.min_queue_size = 10
+    pc.max_queue_size = 100
+    # min redis job queue time lag threshold per box-item for low-priority poll tactic (in seconds)
+    pc.min_queue_timelag = 5
+    # throttling delay for default / aggressive / low-priority poll tactics (in seconds)
+    pc.queue_delay = 0.1
+  end
+
+  # optional (worker v2: default)
+  c.processor = ActiveSupport::OrderedOptions.new.tap do |pc|
+    # max threads count (per worker process)
+    pc.threads_count = 4
+    # maximum processing time of the batch, after which the batch will be considered hung and processing will be aborted
+    pc.general_timeout = 120
+    # BRPOP delay (in seconds) for polling redis job queue per box-item
+    pc.brpop_delay = 2
+  end
+
+  # optional (worker v1: DEPRECATED)
+  config.process_items.tap do |x|
+    # maximum processing time of the batch, after which the batch will be considered hung and processing will be aborted
+    x.general_timeout = 180
+    # maximum batch processing time, after which the processing of the batch will be aborted in the current thread,
+    # and the next thread that picks up the batch will start processing from the same place
+    x.cutoff_timeout = 60
+    # batch size
+    x.batch_size = 200
+  end
+
+  # optional (worker v1: DEPRECATED)
+  config.worker.tap do |worker|
+    # number of batches that one thread will process per rate interval
+    worker.rate_limit = 10
+    # rate interval in seconds
+    worker.rate_interval = 60
+  end
+end
+```
+
 ### Outbox pattern
 
 You should create a database table in order for the process to view your outgoing messages.
@@ -199,76 +284,6 @@ outbox_items:
         topic: "orders_completed_topic"
 ```
 
-#### outbox.rb
-
-The `outbox.rb` file contains the overall general configuration.
-
-```ruby
-# config/initializers/outbox.rb
-
-Rails.application.config.outbox.tap do |config|
-  config.redis = {url: ENV.fetch("REDIS_URL")} # Redis is used as a coordinator service
-  config.paths << Rails.root.join("config/outbox.yml").to_s # optional; configuration file paths, deep merged at the application start, useful with Rails engines
-
-  # optional
-  config.process_items.tap do |x|
-    # maximum processing time of the batch, after which the batch will be considered hung and processing will be aborted
-    x[:general_timeout] = 180
-    # maximum batch processing time, after which the processing of the batch will be aborted in the current thread,
-    # and the next thread that picks up the batch will start processing from the same place
-    x[:cutoff_timeout] = 60
-    # batch size
-    x[:batch_size] = 200
-  end
-
-  # optional (worker v1: DEPRECATED)
-  config.worker.tap do |worker|
-    # number of batches that one thread will process per rate interval
-    worker[:rate_limit] = 10
-    # rate interval in seconds
-    worker[:rate_interval] = 60
-  end
-  
-  # optional (worker v2: default)
-  c.poller = ActiveSupport::OrderedOptions.new.tap do |pc|
-    # max parallel threads (per box-item, globally)
-    pc.concurrency = 6
-    # max threads count (per worker process)
-    pc.threads_count = 1
-    # maximum processing time of the batch, after which the batch will be considered hung and processing will be aborted
-    pc.general_timeout = 60
-    # poll buffer consists of regular items (errors_count = 0, i.e. without any processing errors) and retryable items (errors_count > 0)
-    # max poll buffer size = regular_items_batch_size + retryable_items_batch_size
-    pc.regular_items_batch_size = 200
-    pc.retryable_items_batch_size = 100
-
-    # poll tactic: default is optimal for most cases: rate limit + redis job-queue size threshold
-    # poll tactic: aggressive is for high-intencity data: without rate limits + redis job-queue size threshold
-    # poll tactic: low-priority is for low-intencity data: rate limits + redis job-queue size threshold + + redis job-queue lag threshold
-    pc.tactic = "default"
-    # number of batches that one thread will process per rate interval
-    pc.rate_limit = 60
-    # rate interval in seconds
-    pc.rate_interval = 60
-    # mix / max redis job queue thresholds per box-item for default / aggressive / low-priority poll tactics
-    pc.min_queue_size = 10
-    pc.max_queue_size = 100
-    # min redis job queue time lag threshold per box-item for low-priority poll tactic (in seconds)
-    pc.min_queue_timelag = 5
-    # throttling delay for default / aggressive / low-priority poll tactics (in seconds)
-    pc.queue_delay = 0.1
-  end
-  c.processor = ActiveSupport::OrderedOptions.new.tap do |pc|
-    # max threads count (per worker process)
-    pc.threads_count = 4
-    # maximum processing time of the batch, after which the batch will be considered hung and processing will be aborted
-    pc.general_timeout = 120
-    # BRPOP delay (in seconds) for polling redis job queue per box-item
-    pc.brpop_delay = 2
-  end
-end
-```
-
 ### Inbox pattern
 
 The database migration will be the same as described in the Outbox pattern.
@@ -376,7 +391,7 @@ outbox_items:
 
 The worker process consists of a poller and a processor, each of which has its own thread pool.
 The poller is responsible for fetching messages ready for processing from the database table.
-The processor, in turn, is used for their consistent processing (while preserving the order of messages and the partitioning key). 
+The processor, in turn, is used for their consistent processing (while preserving the order of messages and the partitioning key).
 Each bunch of buckets (i.e. buckets partition) is consistently fetched by poller one at a time. Each bucket is processed one at a time by a processor.
 A bucket is a number in a row in the `bucket` column generated by the partitioning strategy based on the `event_key` column when a message was committed to the database within the range of zero to `bucket_size`.
 The number of bucket partitions, which poller uses is 6 by default. The number of poller threads is 2 by default and is not intended for customization.
@@ -469,12 +484,41 @@ end
 
 The gem is optionally integrated with OpenTelemetry. If your main application has `opentelemetry-*` gems, the tracing will be configured automatically.
 
-## CLI Arguments (v1: DEPRECATED)
+## Web UI
 
-| Key                   | Description                                                               |
-|-----------------------|---------------------------------------------------------------------------|
-| `--boxes or -b`       | Outbox/Inbox processors to start`                                         |
-| `--concurrency or -c` | Number of threads. Default 10.                                            |
+Outbox comes with a [Ract web application](https://github.com/SberMarket-Tech/sbmt-outbox-ui) that can list existing outbox and inbox models.
+
+```ruby
+Rails.application.routes.draw do
+  mount Sbmt::Outbox::Engine => "/outbox-ui"
+end
+```
+
+**The path `/outbox-ui` cannot be changed for now**
+
+Under the hood it uses a React application provided as [npm package](https://www.npmjs.com/package/sbmt-outbox-ui).
+
+By default, the npm packages is served from `https://cdn.jsdelivr.net/npm/sbmt-outbox-ui@x.y.z/dist/assets/index.js`. It could be changed by the following config option:
+```ruby
+# config/initializers/outbox.rb
+Rails.application.config.outbox.tap do |config|
+  config.cdn_url = "https://some-cdn-url"
+end
+```
+
+### UI development
+
+If you want to implement some features for Outbox UI, you can serve javascript locally like the following:
+1. Start React application by `npm run dev`
+2. Configure Outbox to serve UI scripts locally:
+```ruby
+# config/initializers/outbox.rb
+Rails.application.config.outbox.tap do |config|
+  config.ui.serve_local = true
+end
+```
+
+We would like to see more features added to the web UI. If you have any suggestions, please feel free to submit a pull request 🤗.
 
 ## CLI Arguments (v2: default)
 
@@ -487,6 +531,13 @@ The gem is optionally integrated with OpenTelemetry. If your main application ha
 | `--poll-tactic or -t`      | Poll tactic. Default "default".                                      |
 | `--worker-version or -w`   | Worker version. Default 2.                                           |
 
+## CLI Arguments (v1: DEPRECATED)
+
+| Key                   | Description                                                               |
+|-----------------------|---------------------------------------------------------------------------|
+| `--boxes or -b`       | Outbox/Inbox processors to start`                                         |
+| `--concurrency or -c` | Number of threads. Default 10.                                            |
+
 ## Development & Test
 
 ### Installation

data/app/controllers/sbmt/outbox/api/base_controller.rb

@@ -0,0 +1,24 @@
+# frozen_string_literal: true
+
+module Sbmt
+  module Outbox
+    module Api
+      class BaseController < Sbmt::Outbox.action_controller_api_base_class
+        private
+
+        def render_ok
+          render json: "OK"
+        end
+
+        def render_one(record)
+          render json: record
+        end
+
+        def render_list(records)
+          response.headers["X-Total-Count"] = records.size
+          render json: records
+        end
+      end
+    end
+  end
+end

data/app/controllers/sbmt/outbox/api/inbox_classes_controller.rb

@@ -0,0 +1,41 @@
+# frozen_string_literal: true
+
+module Sbmt
+  module Outbox
+    module Api
+      class InboxClassesController < BaseController
+        def index
+          render_list(Sbmt::Outbox.inbox_item_classes.map do |item|
+            Sbmt::Outbox::Api::InboxClass.find_or_initialize(item.box_id)
+          end)
+        end
+
+        def show
+          render_one Sbmt::Outbox::Api::InboxClass.find_or_initialize(params.require(:id))
+        end
+
+        def update
+          record = Sbmt::Outbox::Api::InboxClass.find_or_initialize(params.require(:id))
+          record.assign_attributes(
+            params.require(:inbox_class).permit(:polling_enabled)
+          )
+          record.save
+
+          render_one record
+        end
+
+        def destroy
+          record = Sbmt::Outbox::Api::InboxClass.find(params.require(:id))
+          unless record
+            render_ok
+            return
+          end
+
+          record.destroy
+
+          render_one record
+        end
+      end
+    end
+  end
+end

data/app/controllers/sbmt/outbox/api/outbox_classes_controller.rb

@@ -0,0 +1,41 @@
+# frozen_string_literal: true
+
+module Sbmt
+  module Outbox
+    module Api
+      class OutboxClassesController < BaseController
+        def index
+          render_list(Sbmt::Outbox.outbox_item_classes.map do |item|
+            Api::OutboxClass.find_or_initialize(item.box_id)
+          end)
+        end
+
+        def show
+          render_one Api::OutboxClass.find_or_initialize(params.require(:id))
+        end
+
+        def update
+          record = Api::OutboxClass.find_or_initialize(params.require(:id))
+          record.assign_attributes(
+            params.require(:outbox_class).permit(:polling_enabled)
+          )
+          record.save
+
+          render_one record
+        end
+
+        def destroy
+          record = Api::OutboxClass.find(params.require(:id))
+          unless record
+            render_ok
+            return
+          end
+
+          record.destroy
+
+          render_one record
+        end
+      end
+    end
+  end
+end

data/app/controllers/sbmt/outbox/root_controller.rb

@@ -0,0 +1,12 @@
+# frozen_string_literal: true
+
+module Sbmt
+  module Outbox
+    class RootController < Sbmt::Outbox.action_controller_base_class
+      def index
+        @local_endpoint = Outbox.config.ui.local_endpoint
+        @cdn_url = Outbox.config.ui.cdn_url
+      end
+    end
+  end
+end

data/app/jobs/sbmt/outbox/base_delete_stale_items_job.rb

@@ -31,7 +31,7 @@ module Sbmt
         self.item_class = item_class_name.constantize
 
         client = if Gem::Version.new(Redlock::VERSION) >= Gem::Version.new("2.0.0")
-          RedisClientFactory.build(config.redis)
+          Sbmt::Outbox.redis
         else
           Redis.new(config.redis)
         end

data/app/models/sbmt/outbox/api/application_record.rb

@@ -0,0 +1,80 @@
+# frozen_string_literal: true
+
+module Sbmt
+  module Outbox
+    module Api
+      class ApplicationRecord
+        include ActiveModel::Model
+        include ActiveModel::Attributes
+
+        delegate :redis, to: "Sbmt::Outbox"
+
+        class << self
+          delegate :redis, to: "Sbmt::Outbox"
+
+          def find(id)
+            attributes = redis.call "HGETALL", redis_key(id)
+            return nil if attributes.empty?
+
+            new(attributes)
+          end
+
+          def find_or_initialize(id, params = {})
+            record = find(id)
+            record || new(params.merge(id: id))
+          end
+
+          def delete(id)
+            redis.call "DEL", redis_key(id)
+          end
+
+          def attributes(*attrs)
+            attrs.each do |name|
+              attribute name
+            end
+          end
+
+          def attribute(name, type = ActiveModel::Type::Value.new, **options)
+            super
+            # Add predicate methods for boolean types
+            alias_method :"#{name}?", name if type == :boolean || type.is_a?(ActiveModel::Type::Boolean)
+          end
+
+          def redis_key(id)
+            "#{name}:#{id}"
+          end
+        end
+
+        def initialize(params)
+          super
+          assign_attributes(params)
+        end
+
+        def save
+          redis.call "HMSET", redis_key, attributes.to_a.flatten.map(&:to_s)
+        end
+
+        def destroy
+          self.class.delete(id)
+        end
+
+        def as_json(*)
+          attributes
+        end
+
+        def eql?(other)
+          return false unless other.is_a?(self.class)
+          id == other.id
+        end
+
+        alias_method :==, :eql?
+
+        private
+
+        def redis_key
+          self.class.redis_key(id)
+        end
+      end
+    end
+  end
+end

data/app/models/sbmt/outbox/api/box_class.rb

@@ -0,0 +1,12 @@
+# frozen_string_literal: true
+
+module Sbmt
+  module Outbox
+    module Api
+      class BoxClass < Api::ApplicationRecord
+        attribute :id, :string
+        attribute :polling_enabled, :boolean, default: -> { !Outbox.yaml_config.fetch(:polling_auto_disabled, false) }
+      end
+    end
+  end
+end

data/app/models/sbmt/outbox/api/inbox_class.rb

@@ -0,0 +1,10 @@
+# frozen_string_literal: true
+
+module Sbmt
+  module Outbox
+    module Api
+      class InboxClass < Api::BoxClass
+      end
+    end
+  end
+end

data/app/models/sbmt/outbox/api/outbox_class.rb

@@ -0,0 +1,10 @@
+# frozen_string_literal: true
+
+module Sbmt
+  module Outbox
+    module Api
+      class OutboxClass < Api::BoxClass
+      end
+    end
+  end
+end

data/app/models/sbmt/outbox/base_item.rb

@@ -16,8 +16,12 @@ module Sbmt
           @box_name ||= name.underscore
         end
 
+        def box_id
+          @box_id ||= name.underscore.tr("/", "-").dasherize
+        end
+
         def config
-          @config ||= lookup_config.new(box_name)
+          @config ||= lookup_config.new(box_id: box_id, box_name: box_name)
         end
 
         def calc_bucket_partitions(count)

data/app/models/sbmt/outbox/base_item_config.rb

@@ -6,7 +6,10 @@ module Sbmt
       DEFAULT_BUCKET_SIZE = 16
       DEFAULT_PARTITION_STRATEGY = :number
 
-      def initialize(box_name)
+      delegate :yaml_config, :memory_store, to: "Sbmt::Outbox"
+
+      def initialize(box_id:, box_name:)
+        self.box_id = box_id
         self.box_name = box_name
 
         validate!
@@ -15,11 +18,11 @@ module Sbmt
       def owner
         return @owner if defined? @owner
 
-        @owner = options[:owner].presence || Outbox.yaml_config[:owner].presence
+        @owner = options[:owner].presence || yaml_config[:owner].presence
       end
 
       def bucket_size
-        @bucket_size ||= (options[:bucket_size] || Outbox.yaml_config.fetch(:bucket_size, DEFAULT_BUCKET_SIZE)).to_i
+        @bucket_size ||= (options[:bucket_size] || yaml_config.fetch(:bucket_size, DEFAULT_BUCKET_SIZE)).to_i
       end
 
       def partition_size
@@ -107,7 +110,7 @@ module Sbmt
 
       private
 
-      attr_accessor :box_name
+      attr_accessor :box_id, :box_name
 
       def options
         @options ||= lookup_config || {}
@@ -120,6 +123,23 @@ module Sbmt
       def validate!
         raise ConfigError, "Bucket size should be greater or equal to partition size" if partition_size > bucket_size
       end
+
+      def polling_auto_disabled?
+        return @polling_auto_disabled if defined?(@polling_auto_disabled)
+        @polling_auto_disabled = yaml_config.fetch(:polling_auto_disabled, false)
+      end
+
+      def polling_enabled_for?(api_model)
+        record = memory_store.fetch("sbmt/outbox/outbox_item_config/#{box_id}", expires_in: 10) do
+          api_model.find(box_id)
+        end
+
+        if record.nil?
+          !polling_auto_disabled?
+        else
+          record.polling_enabled?
+        end
+      end
     end
   end
 end

data/app/models/sbmt/outbox/inbox_item_config.rb

@@ -3,10 +3,14 @@
 module Sbmt
   module Outbox
     class InboxItemConfig < BaseItemConfig
+      def polling_enabled?
+        polling_enabled_for?(Sbmt::Outbox::Api::InboxClass)
+      end
+
       private
 
       def lookup_config
-        Outbox.yaml_config.dig(:inbox_items, box_name)
+        yaml_config.dig(:inbox_items, box_name)
       end
     end
   end

data/app/models/sbmt/outbox/outbox_item_config.rb

@@ -3,10 +3,14 @@
 module Sbmt
   module Outbox
     class OutboxItemConfig < BaseItemConfig
+      def polling_enabled?
+        polling_enabled_for?(Sbmt::Outbox::Api::OutboxClass)
+      end
+
       private
 
       def lookup_config
-        Outbox.yaml_config.dig(:outbox_items, box_name) || Outbox.yaml_config.dig(:items, box_name)
+        yaml_config.dig(:outbox_items, box_name)
       end
     end
   end

data/app/views/sbmt/outbox/root/index.erb

@@ -0,0 +1,140 @@
+<!DOCTYPE html>
+<html lang="en">
+    <head>
+        <% if Sbmt::Outbox.config.ui.serve_local %>
+        <script type="module">
+            import RefreshRuntime from "<%= @local_endpoint %>/@react-refresh"
+            RefreshRuntime.injectIntoGlobalHook(window)
+            window.$RefreshReg$ = () => {}
+            window.$RefreshSig$ = () => (type) => type
+            window.__vite_plugin_react_preamble_installed__ = true
+        </script>
+        <script type="module" src="<%= @local_endpoint %>/@vite/client"></script>
+        <% end %>
+        <meta charset="utf-8" />
+        <meta
+            name="viewport"
+            content="minimum-scale=1, initial-scale=1, width=device-width, shrink-to-fit=no"
+        />
+        <meta name="theme-color" content="#000000" />
+        <link rel="manifest" href="./manifest.json" />
+        <link rel="shortcut icon" href="./favicon.ico" />
+        <title>outbox-ui</title>
+        <style>
+            body {
+                margin: 0;
+                padding: 0;
+                font-family: sans-serif;
+            }
+
+            .loader-container {
+                display: flex;
+                align-items: center;
+                justify-content: center;
+                flex-direction: column;
+                position: absolute;
+                top: 0;
+                bottom: 0;
+                left: 0;
+                right: 0;
+                background-color: #fafafa;
+            }
+
+            /* CSS Spinner from https://projects.lukehaas.me/css-loaders/ */
+
+            .loader,
+            .loader:before,
+            .loader:after {
+                border-radius: 50%;
+            }
+
+            .loader {
+                color: #283593;
+                font-size: 11px;
+                text-indent: -99999em;
+                margin: 55px auto;
+                position: relative;
+                width: 10em;
+                height: 10em;
+                box-shadow: inset 0 0 0 1em;
+                -webkit-transform: translateZ(0);
+                -ms-transform: translateZ(0);
+                transform: translateZ(0);
+            }
+
+            .loader:before,
+            .loader:after {
+                position: absolute;
+                content: '';
+            }
+
+            .loader:before {
+                width: 5.2em;
+                height: 10.2em;
+                background: #fafafa;
+                border-radius: 10.2em 0 0 10.2em;
+                top: -0.1em;
+                left: -0.1em;
+                -webkit-transform-origin: 5.2em 5.1em;
+                transform-origin: 5.2em 5.1em;
+                -webkit-animation: load2 2s infinite ease 1.5s;
+                animation: load2 2s infinite ease 1.5s;
+            }
+
+            .loader:after {
+                width: 5.2em;
+                height: 10.2em;
+                background: #fafafa;
+                border-radius: 0 10.2em 10.2em 0;
+                top: -0.1em;
+                left: 5.1em;
+                -webkit-transform-origin: 0px 5.1em;
+                transform-origin: 0px 5.1em;
+                -webkit-animation: load2 2s infinite ease;
+                animation: load2 2s infinite ease;
+            }
+
+            @-webkit-keyframes load2 {
+                0% {
+                    -webkit-transform: rotate(0deg);
+                    transform: rotate(0deg);
+                }
+                100% {
+                    -webkit-transform: rotate(360deg);
+                    transform: rotate(360deg);
+                }
+            }
+
+            @keyframes load2 {
+                0% {
+                    -webkit-transform: rotate(0deg);
+                    transform: rotate(0deg);
+                }
+                100% {
+                    -webkit-transform: rotate(360deg);
+                    transform: rotate(360deg);
+                }
+            }
+        </style>
+        <link rel="preconnect" href="https://fonts.gstatic.com" />
+        <link
+            href="https://fonts.googleapis.com/css2?family=Roboto:wght@300;400;500;700&display=swap"
+            rel="stylesheet"
+        />
+    </head>
+
+    <body>
+        <noscript> You need to enable JavaScript to run this app. </noscript>
+        <div id="root">
+            <div class="loader-container">
+                <div class="loader">Loading...</div>
+            </div>
+        </div>
+        <% if Sbmt::Outbox.config.ui.serve_local %>
+        <script type="module" crossorigin src="<%= @local_endpoint %>/src/index.tsx"></script>
+        <% else %>
+        <script type="module" crossorigin src="<%= @cdn_url %>"></script>
+        <% end %>
+    </body>
+
+</html>

data/config/routes.rb

@@ -0,0 +1,10 @@
+# frozen_string_literal: true
+
+Sbmt::Outbox::Engine.routes.draw do
+  root to: "root#index"
+
+  namespace :api, defaults: {format: :json} do
+    resources :outbox_classes, only: [:index, :show, :update, :destroy]
+    resources :inbox_classes, only: [:index, :show, :update, :destroy]
+  end
+end

data/lib/generators/outbox/install/templates/outbox.rb

@@ -1,32 +1,5 @@
 # frozen_string_literal: true
 
 Rails.application.config.outbox.tap do |config|
-  # setup custom ErrorTracker
-  # config.error_tracker = "ErrorTracker"
-
-  # customize redis
-  # config.redis = {url: ENV.fetch("REDIS_URL", "redis://127.0.0.1:6379")}
-
-  # setup custom batch process middlewares
-  # config.batch_process_middlewares << "MyBatchProcessMiddleware"
-
-  # setup custom item process middlewares
-  # config.item_process_middlewares << "MyItemProcessMiddleware"
-
-  # config.process_items.tap do |x|
-  #   # maximum processing time of the batch, after which the batch will be considered hung and processing will be aborted
-  #   x[:general_timeout] = 180
-  #   # maximum patch processing time, after which the processing of the patch will be aborted in the current thread,
-  #   # and the next thread that picks up the batch will start processing from the same place
-  #   x[:cutoff_timeout] = 60
-  #   # batch size
-  #   x[:batch_size] = 200
-  # end
-
-  # config.worker.tap do |worker|
-  #   # number of batches that one thread will process per rate interval
-  #   worker[:rate_limit] = 10
-  #   # rate interval in seconds
-  #   worker[:rate_interval] = 60
-  # end
+  config.redis = {url: ENV.fetch("REDIS_URL", "redis://127.0.0.1:6379")}
 end

data/lib/sbmt/outbox/engine.rb

@@ -10,12 +10,20 @@ module Sbmt
       config.outbox = ActiveSupport::OrderedOptions.new.tap do |c|
         c.active_record_base_class = "ApplicationRecord"
         c.active_job_base_class = "ApplicationJob"
+        c.action_controller_api_base_class = "ActionController::API"
+        # We cannot use ApplicationController because often it could be inherited from ActionController::API
+        c.action_controller_base_class = "ActionController::Base"
         c.error_tracker = "Sbmt::Outbox::ErrorTracker"
         c.outbox_item_classes = []
         c.inbox_item_classes = []
         c.paths = []
         c.disposable_transports = false
         c.redis = {url: ENV.fetch("REDIS_URL", "redis://127.0.0.1:6379")}
+        c.ui = ActiveSupport::OrderedOptions.new.tap do |c|
+          c.serve_local = false
+          c.local_endpoint = "http://localhost:5173"
+          c.cdn_url = "https://cdn.jsdelivr.net/npm/sbmt-outbox-ui@0.0.8/dist/assets/index.js"
+        end
         c.process_items = ActiveSupport::OrderedOptions.new.tap do |c|
           c.general_timeout = 120
           c.cutoff_timeout = 60

data/lib/sbmt/outbox/v2/poll_throttler/paused_box.rb

@@ -0,0 +1,28 @@
+# frozen_string_literal: true
+
+require "sbmt/outbox/v2/poll_throttler/base"
+require "sbmt/outbox/v2/thread_pool"
+
+module Sbmt
+  module Outbox
+    module V2
+      module PollThrottler
+        class PausedBox < Base
+          def initialize(delay: 0.1)
+            super()
+
+            @delay = delay
+          end
+
+          def wait(worker_num, poll_task, task_result)
+            return Success(Sbmt::Outbox::V2::Throttler::NOOP_STATUS) if poll_task.item_class.config.polling_enabled?
+
+            sleep(@delay)
+
+            Success(Sbmt::Outbox::V2::Throttler::SKIP_STATUS)
+          end
+        end
+      end
+    end
+  end
+end

data/lib/sbmt/outbox/v2/poll_throttler.rb

@@ -7,6 +7,7 @@ require "sbmt/outbox/v2/poll_throttler/fixed_delay"
 require "sbmt/outbox/v2/poll_throttler/noop"
 require "sbmt/outbox/v2/poll_throttler/redis_queue_size"
 require "sbmt/outbox/v2/poll_throttler/redis_queue_time_lag"
+require "sbmt/outbox/v2/poll_throttler/paused_box"
 
 module Sbmt
   module Outbox
@@ -17,10 +18,14 @@ module Sbmt
         def self.build(tactic, redis, poller_config)
           raise "WARN: invalid poller poll tactic provided: #{tactic}, available options: #{POLL_TACTICS}" unless POLL_TACTICS.include?(tactic)
 
+          # no-op, for testing purposes
+          return Noop.new if tactic == "noop"
+
           if tactic == "default"
             # composite of RateLimited & RedisQueueSize (upper bound only)
             # optimal polling performance for most cases
             Composite.new(throttlers: [
+              PausedBox.new,
               RedisQueueSize.new(redis: redis, max_size: poller_config.max_queue_size, delay: poller_config.queue_delay),
               RateLimited.new(limit: poller_config.rate_limit, interval: poller_config.rate_interval)
             ])
@@ -30,6 +35,7 @@ module Sbmt
             # and also by min redis queue oldest item lag
             # optimal polling performance for low-intensity data flow
             Composite.new(throttlers: [
+              PausedBox.new,
               RedisQueueSize.new(redis: redis, min_size: poller_config.min_queue_size, max_size: poller_config.max_queue_size, delay: poller_config.queue_delay),
               RedisQueueTimeLag.new(redis: redis, min_lag: poller_config.min_queue_timelag, delay: poller_config.queue_delay),
               RateLimited.new(limit: poller_config.rate_limit, interval: poller_config.rate_interval)
@@ -37,10 +43,10 @@ module Sbmt
           elsif tactic == "aggressive"
             # throttles only by max job queue size, max polling performance
             # optimal polling performance for high-intensity data flow
-            RedisQueueSize.new(redis: redis, max_size: poller_config.max_queue_size, delay: poller_config.queue_delay)
-          elsif tactic == "noop"
-            # no-op, for testing purposes
-            Noop.new
+            Composite.new(throttlers: [
+              PausedBox.new,
+              RedisQueueSize.new(redis: redis, max_size: poller_config.max_queue_size, delay: poller_config.queue_delay)
+            ])
           end
         end
       end

data/lib/sbmt/outbox/version.rb

@@ -2,6 +2,6 @@
 
 module Sbmt
   module Outbox
-    VERSION = "6.2.0"
+    VERSION = "6.3.1"
   end
 end

data/lib/sbmt/outbox.rb

@@ -12,6 +12,7 @@ require "cutoff"
 require "http_health_check"
 require "redis-client"
 require "connection_pool"
+require "ostruct"
 
 begin
   require "sentry-rails"
@@ -75,6 +76,14 @@ module Sbmt
       @active_job_base_class ||= config.active_job_base_class.safe_constantize || ::ActiveJob::Base
     end
 
+    def action_controller_api_base_class
+      @action_controller_api_base_class ||= config.action_controller_api_base_class.safe_constantize || ::ActionController::API
+    end
+
+    def action_controller_base_class
+      @action_controller_base_class ||= config.action_controller_base_class.safe_constantize || ::ActionController::Base
+    end
+
     def error_tracker
       @error_tracker ||= config.error_tracker.constantize
     end
@@ -83,6 +92,16 @@ module Sbmt
       @database_switcher ||= config.database_switcher.constantize
     end
 
+    def redis
+      @redis ||= ConnectionPool::Wrapper.new(size: ENV.fetch("RAILS_MAX_THREADS", 5).to_i) do
+        RedisClientFactory.build(config.redis)
+      end
+    end
+
+    def memory_store
+      @memory_store ||= ActiveSupport::Cache::MemoryStore.new
+    end
+
     def outbox_item_classes
       @outbox_item_classes ||= if config.outbox_item_classes.empty?
         (yaml_config[:outbox_items] || {}).keys.map { |name| name.camelize.constantize }

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: sbmt-outbox
 version: !ruby/object:Gem::Version
-  version: 6.2.0
+  version: 6.3.1
 platform: ruby
 authors:
 - Sbermarket Ruby-Platform Team
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2024-04-18 00:00:00.000000000 Z
+date: 2024-05-02 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: connection_pool
@@ -513,6 +513,10 @@ extra_rdoc_files: []
 files:
 - README.md
 - Rakefile
+- app/controllers/sbmt/outbox/api/base_controller.rb
+- app/controllers/sbmt/outbox/api/inbox_classes_controller.rb
+- app/controllers/sbmt/outbox/api/outbox_classes_controller.rb
+- app/controllers/sbmt/outbox/root_controller.rb
 - app/interactors/sbmt/outbox/base_create_item.rb
 - app/interactors/sbmt/outbox/create_inbox_item.rb
 - app/interactors/sbmt/outbox/create_outbox_item.rb
@@ -528,14 +532,20 @@ files:
 - app/jobs/sbmt/outbox/base_delete_stale_items_job.rb
 - app/jobs/sbmt/outbox/delete_stale_inbox_items_job.rb
 - app/jobs/sbmt/outbox/delete_stale_outbox_items_job.rb
+- app/models/sbmt/outbox/api/application_record.rb
+- app/models/sbmt/outbox/api/box_class.rb
+- app/models/sbmt/outbox/api/inbox_class.rb
+- app/models/sbmt/outbox/api/outbox_class.rb
 - app/models/sbmt/outbox/base_item.rb
 - app/models/sbmt/outbox/base_item_config.rb
 - app/models/sbmt/outbox/inbox_item.rb
 - app/models/sbmt/outbox/inbox_item_config.rb
 - app/models/sbmt/outbox/outbox_item.rb
 - app/models/sbmt/outbox/outbox_item_config.rb
+- app/views/sbmt/outbox/root/index.erb
 - config/initializers/schked.rb
 - config/initializers/yabeda.rb
+- config/routes.rb
 - config/schedule.rb
 - exe/outbox
 - lib/generators/helpers.rb
@@ -589,6 +599,7 @@ files:
 - lib/sbmt/outbox/v2/poll_throttler/composite.rb
 - lib/sbmt/outbox/v2/poll_throttler/fixed_delay.rb
 - lib/sbmt/outbox/v2/poll_throttler/noop.rb
+- lib/sbmt/outbox/v2/poll_throttler/paused_box.rb
 - lib/sbmt/outbox/v2/poll_throttler/rate_limited.rb
 - lib/sbmt/outbox/v2/poll_throttler/redis_queue_size.rb
 - lib/sbmt/outbox/v2/poll_throttler/redis_queue_time_lag.rb