jetstream_bridge 1.4.0

data/README.md

@@ -0,0 +1,217 @@
+# Jetstream Bridge
+
+**Production-safe realtime data bridge** between systems using **NATS JetStream**.
+Includes durable consumers, backpressure, retries, **DLQ**, and optional **Inbox/Outbox** for end-to-end reliability.
+
+---
+
+## ✨ Features
+
+- 🔌 Simple **Publisher** and **Consumer** interfaces
+- 🛡 **Outbox** (reliable send) & **Inbox** (idempotent receive)
+- 🧨 **DLQ** for poison messages
+- ⚙️ Durable `pull_subscribe` with exponential backoff & `max_deliver`
+- 🎯 Configurable **source** and **destination** applications
+- 📊 Built-in observability
+
+---
+
+## 📦 Install
+
+```ruby
+# Gemfile
+gem "jetstream_bridge"
+```
+
+```bash
+bundle install
+```
+
+---
+
+## 🔧 Configure (Rails)
+
+```ruby
+# config/initializers/jetstream_bridge.rb
+JetstreamBridge.configure do |config|
+  # NATS Connection
+  config.nats_urls       = ENV.fetch("NATS_URLS", "nats://localhost:4222")
+  config.env             = ENV.fetch("NATS_ENV", "development")
+  config.app_name        = ENV.fetch("APP_NAME", "app")
+  config.destination_app = ENV["DESTINATION_APP"]
+
+  # Consumer Tuning
+  config.max_deliver = 5
+  config.ack_wait    = "30s"
+  config.backoff     = %w[1s 5s 15s 30s 60s]
+
+  # Reliability Features
+  config.use_outbox = true
+  config.use_inbox  = true
+  config.use_dlq    = true
+
+  # Models (override if custom)
+  config.outbox_model = "JetstreamBridge::OutboxEvent"
+  config.inbox_model  = "JetstreamBridge::InboxEvent"
+end
+```
+
+> **Note:**
+> - `stream_name` defaults to `{env}-stream-bridge`
+> - `dlq_subject` defaults to `data.sync.dlq`
+
+---
+
+## 📡 Subject Conventions
+
+| Direction     | Subject Pattern                |
+|---------------|--------------------------------|
+| **Publish**   | `{env}.data.sync.{app}.{dest}` |
+| **Subscribe** | `{env}.data.sync.{dest}.{app}` |
+| **DLQ**       | `{env}.data.sync.dlq`          |
+
+- `{app}`: Your `app_name`
+- `{dest}`: Your `destination_app`
+- `{env}`: Your `env``
+
+---
+
+## 🗃 Database Setup (Inbox/Outbox)
+
+Run the installer:
+
+```bash
+rails jetstream_bridge:install --all
+```
+
+This creates:
+
+1. **Initializer** (`config/initializers/jetstream_bridge.rb`)
+2. **Migrations** (if enabled):
+
+```ruby
+# Outbox
+create_table :jetstream_outbox_events do |t|
+  t.string  :resource_type, null: false
+  t.string  :resource_id,   null: false
+  t.string  :event_type,    null: false
+  t.jsonb   :payload,       null: false, default: {}
+  t.datetime :published_at
+  t.integer  :attempts,     default: 0
+  t.text     :last_error
+  t.timestamps
+end
+add_index :jetstream_outbox_events, [:resource_type, :resource_id]
+
+# Inbox
+create_table :jetstream_inbox_events do |t|
+  t.string   :event_id,  null: false
+  t.string   :subject,   null: false
+  t.datetime :processed_at
+  t.text     :error
+  t.timestamps
+end
+add_index :jetstream_inbox_events, :event_id, unique: true
+```
+
+---
+
+## 📤 Publish Events
+
+```ruby
+publisher = JetstreamBridge::Publisher.new
+publisher.publish(
+  resource_type: "user",
+  resource_id:   "01H1234567890ABCDEF",
+  event_type:    "created",
+  payload:       { id: "01H...", name: "Ada" }
+)
+```
+
+> **Ephemeral Mode** (for short-lived scripts):
+> ```ruby
+> JetstreamBridge::Publisher.new(persistent: false).publish(...)
+> ```
+
+---
+
+## 🔄 Outbox (If Enabled)
+
+Events are written to the Outbox table. Flush periodically:
+
+```ruby
+# app/jobs/outbox_flush_job.rb
+class OutboxFlushJob < ApplicationJob
+  def perform
+    JetstreamBridge::Publisher.new.flush_outbox
+  end
+end
+```
+
+Schedule this job to run every minute.
+
+---
+
+## 📥 Consume Events
+
+```ruby
+JetstreamBridge::Consumer.new(
+  durable_name: "#{Rails.env}-peerapp-events",
+  batch_size:   25
+) do |event, subject, deliveries|
+  # Your idempotent domain logic here
+  UserCreatedHandler.call(event.payload)
+end.run!
+```
+
+---
+
+## 📬 Envelope Format
+
+Published events include:
+
+```json
+{
+  "event_id":       "01H1234567890ABCDEF",
+  "schema_version": 1,
+  "producer":       "myapp",
+  "resource_type":  "user",
+  "resource_id":    "01H1234567890ABCDEF",
+  "event_type":     "created",
+  "occurred_at":    "2025-08-13T21:00:00Z",
+  "trace_id":       "abc123",
+  "payload":        { "id": "01H...", "name": "Ada" }
+}
+```
+
+---
+
+## 🛠 Operations Guide
+
+### Monitoring
+- **Consumer Lag**: `nats consumer info <stream> <durable>`
+- **Outbox Growth**: Alert if `jetstream_outbox_events` grows unexpectedly
+- **DLQ Messages**: Monitor `data.sync.dlq` subscription
+
+### Scaling
+- Run consumers in **separate processes/containers**
+- Scale independently of web workers
+
+### When to Use
+- **Inbox**: When replays or duplicates are possible
+- **Outbox**: When "DB commit ⇒ event published" guarantee is required
+
+---
+
+## 🚀 Getting Started
+
+1. Install the gem 
+2. Configure the initializer 
+3. Run migrations: `rails db:migrate`
+4. Start publishing/consuming!
+
+---
+
+## 📄 License
+
+[MIT License](LICENSE)

data/jetstream_bridge.gemspec

@@ -0,0 +1,61 @@
+# frozen_string_literal: true
+
+require_relative 'lib/jetstream_bridge/version'
+
+Gem::Specification.new do |spec|
+  spec.name                  = 'jetstream_bridge'
+  spec.version               = JetstreamBridge::VERSION
+  spec.authors               = ['Mike Attara']
+  spec.email                 = ['mpyebattara@gmail.com']
+
+  # Clear, value-focused copy
+  spec.summary     = 'Reliable realtime bridge over NATS JetStream for Rails/Ruby apps'
+  spec.description = <<~DESC.strip
+    Publisher/Consumer utilities for NATS JetStream with environment-scoped subjects,
+    overlap guards, DLQ routing, retries/backoff, and optional Inbox/Outbox patterns.
+    Includes topology setup helpers for production-safe operation.
+  DESC
+
+  spec.license     = 'MIT'
+  spec.homepage    = 'https://github.com/attaradev/jetstream_bridge'
+
+  # Ruby & RubyGems requirements
+  spec.required_ruby_version     = '>= 2.7.0'
+  spec.required_rubygems_version = '>= 3.3.0'
+
+  # Rich metadata for RubyGems.org
+  spec.metadata = {
+    'homepage_uri'         => 'https://github.com/attaradev/jetstream_bridge',
+    'source_code_uri'      => 'https://github.com/attaradev/jetstream_bridge',
+    'changelog_uri'        => 'https://github.com/attaradev/jetstream_bridge/blob/main/CHANGELOG.md',
+    'documentation_uri'    => 'https://github.com/attaradev/jetstream_bridge#readme',
+    'bug_tracker_uri'      => 'https://github.com/attaradev/jetstream_bridge/issues',
+    'github_repo'          => 'ssh://github.com/attaradev/jetstream_bridge',
+    'rubygems_mfa_required'=> 'true'
+  }
+
+  # Safer file list for published gem
+  # (falls back to Dir[] if not in a git repo — e.g., CI tarballs)
+  spec.files = if system('git rev-parse --is-inside-work-tree > /dev/null 2>&1')
+                 `git ls-files -z`.split("\x0").reject { |f| f.start_with?('spec/fixtures/') }
+               else
+                 Dir['lib/**/*', 'README*', 'CHANGELOG*', 'LICENSE*']
+               end
+
+  spec.require_paths = ['lib']
+
+  # Runtime dependencies
+  spec.add_dependency 'activerecord',  '>= 6.0'
+  spec.add_dependency 'activesupport', '>= 6.0'
+  spec.add_dependency 'nats-pure',     '~> 2.4'
+  spec.add_dependency 'rails',         '>= 6.0'
+
+  # Development / quality dependencies
+  spec.add_development_dependency 'rake',                '>= 13.0'
+  spec.add_development_dependency 'rspec',               '>= 3.12'
+  spec.add_development_dependency 'rubocop',             '~> 1.66'
+  spec.add_development_dependency 'rubocop-performance', '~> 1.21'
+  spec.add_development_dependency 'rubocop-rake',        '~> 0.6'
+  spec.add_development_dependency 'rubocop-packaging',   '~> 0.5'
+  spec.add_development_dependency 'bundler-audit',       '>= 0.9.1'
+end

data/lib/jetstream_bridge/config.rb

@@ -0,0 +1,48 @@
+# frozen_string_literal: true
+
+module JetstreamBridge
+  class Config
+    attr_accessor :destination_app, :nats_urls, :env, :app_name,
+                  :max_deliver, :ack_wait, :backoff,
+                  :use_outbox, :use_inbox, :inbox_model, :outbox_model,
+                  :use_dlq
+
+    def initialize
+      @nats_urls       = ENV['NATS_URLS'] || ENV['NATS_URL'] || 'nats://localhost:4222'
+      @env             = ENV['NATS_ENV']  || 'development'
+      @app_name        = ENV['APP_NAME']  || 'app'
+      @destination_app = ENV['DESTINATION_APP']
+
+      @max_deliver = 5
+      @ack_wait    = '30s'
+      @backoff     = %w[1s 5s 15s 30s 60s]
+
+      @use_outbox   = false
+      @use_inbox    = false
+      @use_dlq      = true
+      @outbox_model = 'JetstreamBridge::OutboxEvent'
+      @inbox_model  = 'JetstreamBridge::InboxEvent'
+    end
+
+    # Single stream name per env
+    def stream_name
+      "#{env}-jetstream-bridge-stream"
+    end
+
+    # Base subjects
+    # Producer publishes to:   {env}.data.sync.{app}.{dest}
+    # Consumer subscribes to:  {env}.data.sync.{dest}.{app}
+    def source_subject
+      "#{env}.data.sync.#{app_name}.#{destination_app}"
+    end
+
+    def destination_subject
+      "#{env}.data.sync.#{destination_app}.#{app_name}"
+    end
+
+    # DLQ
+    def dlq_subject
+      "#{env}.data.sync.dlq"
+    end
+  end
+end

data/lib/jetstream_bridge/connection.rb

@@ -0,0 +1,75 @@
+# frozen_string_literal: true
+
+require 'nats/io/client'
+require 'singleton'
+require 'json'
+require_relative 'duration'
+require_relative 'logging'
+require_relative 'topology'
+require_relative 'config'
+
+module JetstreamBridge
+  # Singleton connection to NATS.
+  class Connection
+    include Singleton
+
+    DEFAULT_CONN_OPTS = {
+      reconnect: true,
+      reconnect_time_wait: 2,
+      max_reconnect_attempts: 10,
+      connect_timeout: 5
+    }.freeze
+
+    class << self
+      # Thread-safe delegator to the singleton instance
+      def connect!
+        @__mutex ||= Mutex.new
+        @__mutex.synchronize { instance.connect! }
+      end
+    end
+
+    # Idempotent: returns an existing, healthy JetStream context or establishes one.
+    def connect!
+      return @jts if connected?
+
+      servers = nats_servers
+      raise 'No NATS URLs configured' if servers.empty?
+
+      establish_connection(servers)
+      Logging.info(
+        "Connected to NATS (#{servers.size} server#{servers.size == 1 ? '' : 's'}): #{sanitize_urls(servers).join(',')}",
+        tag: 'JetstreamBridge::Connection'
+      )
+
+      Topology.ensure!(@jts)
+      @jts
+    end
+
+    private
+
+    def connected?
+      @nc&.connected?
+    end
+
+    def nats_servers
+      JetstreamBridge.config.nats_urls
+                     .to_s
+                     .split(',')
+                     .map(&:strip)
+                     .reject(&:empty?)
+    end
+
+    def establish_connection(servers)
+      @nc = NATS::IO::Client.new
+      @nc.connect({ servers: servers }.merge(DEFAULT_CONN_OPTS))
+      @jts = @nc.jetstream
+    end
+
+    # Mask credentials in NATS URLs:
+    # - "nats://user:pass@host:4222" -> "nats://user:***@host:4222"
+    # - "nats://token@host:4222"     -> "nats://***@host:4222"
+    def sanitize_urls(urls)
+      urls.map { |u| Logging.sanitize_url(u) }
+    end
+  end
+end

data/lib/jetstream_bridge/consumer.rb

@@ -0,0 +1,106 @@
+# frozen_string_literal: true
+
+require 'json'
+require 'securerandom'
+require_relative 'connection'
+require_relative 'duration'
+require_relative 'logging'
+require_relative 'consumer_config'
+require_relative 'message_processor'
+require_relative 'config'
+
+module JetstreamBridge
+  # Subscribes to "{env}.data.sync.{dest}.{app}" and processes messages.
+  class Consumer
+    DEFAULT_BATCH_SIZE   = 25
+    FETCH_TIMEOUT_SECS   = 5
+    IDLE_SLEEP_SECS      = 0.05
+
+    # @param durable_name [String] Consumer name
+    # @param batch_size [Integer] Max messages per fetch
+    # @yield [event, subject, deliveries] Message handler
+    def initialize(durable_name:, batch_size: DEFAULT_BATCH_SIZE, &block)
+      @handler    = block
+      @batch_size = batch_size
+      @durable    = durable_name
+      @jts        = Connection.connect!
+
+      ensure_destination!
+      ensure_consumer!
+      subscribe!
+      @processor = MessageProcessor.new(@jts, @handler)
+    end
+
+    # Starts the consumer loop.
+    def run!
+      Logging.info("Consumer #{@durable} started…", tag: 'JetstreamBridge::Consumer')
+      loop do
+        processed = process_batch
+        sleep(IDLE_SLEEP_SECS) if processed.zero?
+      end
+    end
+
+    private
+
+    def ensure_destination!
+      return unless JetstreamBridge.config.destination_app.to_s.empty?
+      raise ArgumentError, 'destination_app must be configured'
+    end
+
+    def stream_name
+      JetstreamBridge.config.stream_name
+    end
+
+    def filter_subject
+      JetstreamBridge.config.destination_subject
+    end
+
+    def ensure_consumer!
+      @jts.consumer_info(stream_name, @durable)
+      Logging.info("Consumer #{@durable} exists.", tag: 'JetstreamBridge::Consumer')
+    rescue NATS::JetStream::Error
+      @jts.add_consumer(stream_name, **ConsumerConfig.consumer_config(@durable, filter_subject))
+      Logging.info("Created consumer #{@durable} (filter=#{filter_subject})",
+                   tag: 'JetstreamBridge::Consumer')
+    end
+
+    def subscribe!
+      @psub = @jts.pull_subscribe(
+        filter_subject,
+        @durable,
+        stream: stream_name,
+        config: ConsumerConfig.subscribe_config
+      )
+      Logging.info("Subscribed to #{filter_subject} (durable=#{@durable})",
+                   tag: 'JetstreamBridge::Consumer')
+    end
+
+    # Returns number of messages processed; 0 on timeout/idle or after recovery.
+    def process_batch
+      msgs = @psub.fetch(@batch_size, timeout: FETCH_TIMEOUT_SECS)
+      msgs.each { |m| @processor.handle_message(m) }
+      msgs.size
+    rescue NATS::Timeout, NATS::IO::Timeout
+      0
+    rescue NATS::JetStream::Error => e
+      # Handle common recoverable states by re-ensuring consumer & subscription.
+      if recoverable_consumer_error?(e)
+        Logging.warn("Recovering subscription after error: #{e.class} #{e.message}",
+                     tag: 'JetstreamBridge::Consumer')
+        ensure_consumer!
+        subscribe!
+        0
+      else
+        Logging.error("Fetch failed: #{e.class} #{e.message}",
+                      tag: 'JetstreamBridge::Consumer')
+        0
+      end
+    end
+
+    def recoverable_consumer_error?(error)
+      msg = error.message.to_s
+      msg =~ /consumer.*(not\s+found|deleted)/i ||
+        msg =~ /no\s+responders/i
+    end
+  end
+end

data/lib/jetstream_bridge/consumer_config.rb

@@ -0,0 +1,32 @@
+# frozen_string_literal: true
+
+require_relative 'duration'
+require_relative 'logging'
+require_relative 'config'
+
+module JetstreamBridge
+  # Consumer configuration helpers.
+  module ConsumerConfig
+    module_function
+
+    def consumer_config(durable, filter_subject)
+      {
+        durable_name: durable,
+        filter_subject: filter_subject,
+        ack_policy: 'explicit',
+        max_deliver: JetstreamBridge.config.max_deliver,
+        ack_wait: Duration.to_millis(JetstreamBridge.config.ack_wait),
+        backoff: Array(JetstreamBridge.config.backoff).map { |d| Duration.to_millis(d) }
+      }
+    end
+
+    def subscribe_config
+      {
+        ack_policy: 'explicit',
+        max_deliver: JetstreamBridge.config.max_deliver,
+        ack_wait: Duration.to_millis(JetstreamBridge.config.ack_wait),
+        backoff: Array(JetstreamBridge.config.backoff).map { |d| Duration.to_millis(d) }
+      }
+    end
+  end
+end

data/lib/jetstream_bridge/dlq.rb

@@ -0,0 +1,24 @@
+# frozen_string_literal: true
+
+require_relative 'overlap_guard'
+require_relative 'logging'
+require_relative 'subject_matcher'
+
+module JetstreamBridge
+  # Ensures the DLQ subject is added to the stream.
+  class DLQ
+    def self.ensure!(jts)
+      name  = JetstreamBridge.config.stream_name
+      info  = jts.stream_info(name)
+      subs  = Array(info.config.subjects || [])
+      dlq   = JetstreamBridge.config.dlq_subject
+      return if SubjectMatcher.covered?(subs, dlq)
+
+      desired = (subs + [dlq]).uniq
+      OverlapGuard.check!(jts, name, desired)
+
+      jts.update_stream(name: name, subjects: desired)
+      Logging.info("Added DLQ subject #{dlq} to stream #{name}", tag: 'JetstreamBridge::DLQ')
+    end
+  end
+end

data/lib/jetstream_bridge/duration.rb

@@ -0,0 +1,46 @@
+# frozen_string_literal: true
+
+# JetstreamBridge
+#
+module JetstreamBridge
+  # Utility for parsing human-friendly durations into milliseconds.
+  # Examples:
+  #   Duration.to_millis(30)       #=> 30000
+  #   Duration.to_millis("30s")    #=> 30000
+  #   Duration.to_millis("500ms")  #=> 500
+  #   Duration.to_millis(0.5)      #=> 500
+  module Duration
+    MULTIPLIER = { 'ms' => 1, 's' => 1_000, 'm' => 60_000, 'h' => 3_600_000 }.freeze
+    NUMBER_RE  = /\A\d+\z/.freeze
+    TOKEN_RE   = /\A(\d+(?:\.\d+)?)\s*(ms|s|m|h)\z/i.freeze
+
+    module_function
+
+    def to_millis(val)
+      return int_to_ms(val) if val.is_a?(Integer)
+      return float_to_ms(val) if val.is_a?(Float)
+      return string_to_ms(val) if val.is_a?(String)
+      return float_to_ms(val.to_f) if val.respond_to?(:to_f)
+
+      raise ArgumentError, "invalid duration type: #{val.class}"
+    end
+
+    def int_to_ms(i)
+      i >= 1_000 ? i : i * 1_000
+    end
+
+    def float_to_ms(f)
+      (f * 1_000).round
+    end
+
+    def string_to_ms(str)
+      s = str.strip
+      return int_to_ms(s.to_i) if NUMBER_RE.match?(s)
+
+      m = TOKEN_RE.match(s)
+      raise ArgumentError, "invalid duration: #{str.inspect}" unless m
+
+      (m[1].to_f * MULTIPLIER.fetch(m[2].downcase)).round
+    end
+  end
+end

data/lib/jetstream_bridge/inbox_event.rb

@@ -0,0 +1,46 @@
+# frozen_string_literal: true
+
+# If ActiveRecord is not available, a shim class is defined that raises
+# a helpful error when used.
+begin
+  require 'active_record'
+rescue LoadError
+  # Ignore; we handle the lack of AR below.
+end
+
+module JetstreamBridge
+  # InboxEvent is the default ActiveRecord model used by the gem when
+  # `use_inbox` is enabled.
+  # It records processed event IDs for idempotency.
+  if defined?(ActiveRecord::Base)
+    class InboxEvent < ActiveRecord::Base
+      self.table_name = 'jetstream_inbox_events'
+
+      validates :event_id, presence: true, uniqueness: true
+      validates :subject,  presence: true
+    end
+  else
+    # Shim that fails loudly if the app misconfigures the gem without AR.
+    class InboxEvent
+      class << self
+        def method_missing(method_name, *_args, &_block)
+          raise_missing_ar!('Inbox', method_name)
+        end
+
+        def respond_to_missing?(_method_name, _include_private = false)
+          false
+        end
+
+        private
+
+        def raise_missing_ar!(which, method_name)
+          raise(
+            "#{which} requires ActiveRecord (tried to call ##{method_name}). " \
+              'Enable `use_inbox` only in apps with ActiveRecord, or add ' \
+              '`gem "activerecord"` to your Gemfile.'
+          )
+        end
+      end
+    end
+  end
+end