jetstream_bridge 4.4.1 → 4.5.0

data/docs/PRODUCTION.md

@@ -32,22 +32,26 @@ production:
 ### Sizing Guidelines
 
 **Publishers (Web/API processes):**
+
 - 1-2 connections per process (uses existing AR pool)
 - Example: 4 Puma workers × 5 threads = 20 connections minimum
 
 **Consumers:**
+
 - Dedicated connections per consumer process
 - Recommended: 2-5 connections per consumer
 - Example: 3 consumer processes = 6-15 connections
 
 **Total Formula:**
-```
+
+```markdown
 Total Connections = (Web Workers × Threads) + (Consumers × 3) + 10 buffer
 ```
 
 ### Example Calculation
 
 For a typical production setup:
+
 - 4 Puma workers × 5 threads = 20 connections
 - 3 consumer processes × 3 connections = 9 connections
 - 10 connection buffer = 10 connections
@@ -74,9 +78,9 @@ JetstreamBridge.configure do |config|
   config.connect_retry_delay = 3     # Default: 2 seconds
 
   # Required configuration
-  config.env = ENV.fetch("RAILS_ENV", "production")
   config.app_name = ENV.fetch("APP_NAME", "myapp")
   config.destination_app = ENV.fetch("DESTINATION_APP")
+  config.stream_name = ENV.fetch("STREAM_NAME", "myapp-stream")
 
   # Enable reliability features
   config.use_outbox = true
@@ -96,6 +100,31 @@ JetstreamBridge.configure do |config|
 end
 ```
 
+### Permissions and Inbox Prefix
+
+If your NATS account restricts `_INBOX.>` subscriptions, set an allowed prefix:
+
+```ruby
+JetstreamBridge.configure do |config|
+  config.inbox_prefix = "$RPC"
+end
+```
+
+For pre-provisioned streams and consumers:
+
+```ruby
+JetstreamBridge.configure do |config|
+  config.stream_name = "my-stream"      # required
+  config.durable_name = "my-durable"    # optional
+  config.disable_js_api = true          # skip JetStream management APIs
+end
+```
+
+Minimum NATS permissions:
+
+- **Publish**: `$JS.API.>`, `$JS.ACK.>`, source/destination subjects, DLQ subject
+- **Subscribe**: `_INBOX.>` (or custom inbox_prefix), destination subject
+
 ---
 
 ## Consumer Tuning
@@ -135,9 +164,10 @@ end
 ### Memory Management
 
 Long-running consumers automatically:
+
 - Log health checks every 10 minutes (iterations, memory, uptime)
 - Warn when memory exceeds 1GB
-- Suggest garbage collection when heap grows large
+- Warn once when heap object counts grow large so you can profile/trigger GC in the host app
 
 Monitor these logs to detect memory leaks early.
 
@@ -148,7 +178,7 @@ Monitor these logs to detect memory leaks early.
 ### Key Metrics to Track
 
 | Metric | Description | Alert Threshold |
-|--------|-------------|-----------------|
+| -------- | ------------- | ----------------- |
 | Consumer Lag | Pending messages in stream | > 1000 messages |
 | DLQ Size | Messages in dead letter queue | > 100 messages |
 | Connection Status | Health check failures | 2 consecutive failures |
@@ -164,7 +194,7 @@ Use the built-in health check for monitoring:
 # config/routes.rb
 Rails.application.routes.draw do
   get '/health/jetstream', to: proc { |env|
-    health = JetstreamBridge.health_check
+    health = JetstreamBridge.health
     status = health[:healthy] ? 200 : 503
     [status, { 'Content-Type' => 'application/json' }, [health.to_json]]
   }
@@ -197,9 +227,10 @@ end
     "destination_app": "worker",
     "use_outbox": true,
     "use_inbox": true,
-    "use_dlq": true
+    "use_dlq": true,
+    "disable_js_api": true
   },
-  "version": "4.0.3"
+  "version": "4.4.0"
 }
 ```
 
@@ -253,6 +284,7 @@ end
 ### Subject Validation
 
 JetStream Bridge validates subject components to prevent injection attacks. The following are automatically rejected:
+
 - NATS wildcards (`.`, `*`, `>`)
 - Spaces and control characters
 - Components exceeding 255 characters
@@ -270,6 +302,7 @@ config.nats_urls = ENV.fetch("NATS_URLS")
 ```
 
 Credentials in logs are automatically sanitized:
+
 - `nats://user:pass@host:4222` → `nats://user:***@host:4222`
 - `nats://token@host:4222` → `nats://***@host:4222`
 
@@ -345,6 +378,7 @@ spec:
 ### Health Probes
 
 **Liveness Probe:** Checks if the consumer process is running
+
 ```yaml
 livenessProbe:
   exec:
@@ -354,6 +388,7 @@ livenessProbe:
 ```
 
 **Readiness Probe:** Checks if NATS connection is healthy
+
 ```yaml
 readinessProbe:
   httpGet:
@@ -413,7 +448,7 @@ CREATE INDEX idx_inbox_stream_seq ON jetstream_inbox_events(stream, stream_seq);
 CREATE INDEX idx_inbox_status ON jetstream_inbox_events(status);
 ```
 
-2. **Partition large tables** (for high-volume applications):
+1. **Partition large tables** (for high-volume applications):
 
 ```sql
 -- Partition outbox by month
@@ -426,7 +461,7 @@ CREATE TABLE jetstream_outbox_events_2025_11
   FOR VALUES FROM ('2025-11-01') TO ('2025-12-01');
 ```
 
-3. **Archive old records** to prevent table bloat:
+1. **Archive old records** to prevent table bloat:
 
 ```ruby
 # lib/tasks/jetstream_maintenance.rake
@@ -462,24 +497,28 @@ end
 ### Common Issues
 
 **High Consumer Lag:**
+
 - Scale up consumer instances
 - Increase batch size
 - Optimize handler processing time
 - Check database connection pool
 
 **Memory Leaks:**
+
 - Monitor consumer health logs
 - Enable memory profiling
 - Check for circular references in handlers
 - Restart consumers periodically (Kubernetes handles this)
 
 **Connection Issues:**
+
 - Verify NATS server is accessible
 - Check firewall rules
 - Validate TLS certificates
 - Review connection retry settings
 
 **DLQ Growing:**
+
 - Investigate failed message patterns
 - Fix bugs in message handlers
 - Increase max_deliver for transient errors
@@ -499,5 +538,6 @@ end
 ## Support
 
 For issues or questions:
-- GitHub Issues: https://github.com/attaradev/jetstream_bridge/issues
-- Documentation: https://github.com/attaradev/jetstream_bridge
+
+- GitHub Issues: <https://github.com/attaradev/jetstream_bridge/issues>
+- Documentation: <https://github.com/attaradev/jetstream_bridge>

data/docs/TESTING.md

@@ -28,31 +28,28 @@ RSpec.describe MyService do
   include JetstreamBridge::TestHelpers::Matchers
 
   before do
-    # Reset singleton to ensure clean state
-    JetstreamBridge::Connection.instance_variable_set(:@singleton__instance__, nil)
+    # Reset state
     JetstreamBridge.reset!
 
     # Enable test mode - automatically sets up mock NATS
     JetstreamBridge::TestHelpers.enable_test_mode!
 
     JetstreamBridge.configure do |config|
-      config.env = 'test'
       config.app_name = 'my_app'
       config.destination_app = 'worker'
+      config.stream_name = 'test-stream'
     end
 
-    # Setup mock stream and stub topology
+    # Setup mock stream
     mock_jts = JetstreamBridge::TestHelpers.mock_connection.jetstream
     mock_jts.add_stream(
-      name: 'test-jetstream-bridge-stream',
-      subjects: ['test.>']
+      name: 'test-stream',
+      subjects: ['my_app.sync.worker', 'worker.sync.my_app']
     )
-    allow(JetstreamBridge::Topology).to receive(:ensure!)
   end
 
   after do
     JetstreamBridge::TestHelpers.reset_test_mode!
-    JetstreamBridge::Connection.instance_variable_set(:@singleton__instance__, nil)
   end
 
   it 'publishes events through the full stack' do
@@ -266,27 +263,20 @@ end.to raise_error(NATS::JetStream::Error, 'consumer not found')
 ```ruby
 before do
   JetstreamBridge.reset!
-
-  mock_conn = JetstreamBridge::TestHelpers::MockNats.create_mock_connection
-  mock_jts = mock_conn.jetstream
-
-  allow(NATS::IO::Client).to receive(:new).and_return(mock_conn)
-  allow(JetstreamBridge::Connection).to receive(:connect!).and_call_original
-
-  # Setup stream
-  mock_jts.add_stream(
-    name: 'test-jetstream-bridge-stream',
-    subjects: ['test.>']
-  )
-
-  # Allow topology check to succeed
-  allow(JetstreamBridge::Topology).to receive(:ensure!)
+  JetstreamBridge::TestHelpers.enable_test_mode!
 
   JetstreamBridge.configure do |config|
-    config.env = 'test'
     config.app_name = 'api'
     config.destination_app = 'worker'
+    config.stream_name = 'test-stream'
   end
+
+  # Setup mock stream
+  mock_jts = JetstreamBridge::TestHelpers.mock_connection.jetstream
+  mock_jts.add_stream(
+    name: 'test-stream',
+    subjects: ['api.sync.worker', 'worker.sync.api']
+  )
 end
 
 it 'publishes through JetstreamBridge' do
@@ -296,9 +286,9 @@ it 'publishes through JetstreamBridge' do
     payload: { id: 1, name: 'Test' }
   )
 
-  expect(result).to be_publish_success
+  expect(result.success?).to be true
   expect(result.event_id).to be_present
-  expect(result.subject).to eq('test.api.sync.worker')
+  expect(result.subject).to eq('api.sync.worker')
 
   # Verify in storage
   storage = JetstreamBridge::TestHelpers.mock_storage
@@ -306,16 +296,15 @@ it 'publishes through JetstreamBridge' do
 end
 ```
 
-### Full Consuming Flow
+### Full Consumer Flow
 
 ```ruby
-it 'consumes through JetstreamBridge' do
-  mock_conn = JetstreamBridge::TestHelpers.mock_connection
-  mock_jts = mock_conn.jetstream
+it 'consumes messages through JetstreamBridge' do
+  mock_jts = JetstreamBridge::TestHelpers.mock_connection.jetstream
 
   # Publish message to destination subject
   mock_jts.publish(
-    'test.worker.sync.api',
+    'worker.sync.api',
     Oj.dump({
       'event_id' => 'event-1',
       'schema_version' => 1,
@@ -338,9 +327,9 @@ it 'consumes through JetstreamBridge' do
 
   # Mock subscription
   subscription = mock_jts.pull_subscribe(
-    'test.worker.sync.api',
-    'test-consumer',
-    stream: 'test-jetstream-bridge-stream'
+    'worker.sync.api',
+    'api-workers',
+    stream: 'test-stream'
   )
 
   allow_any_instance_of(JetstreamBridge::SubscriptionManager)
@@ -391,7 +380,7 @@ storage.reset!
 3. **Test both success and failure paths**: Use the mock to simulate errors
 4. **Verify message content**: Check that envelopes are correctly formatted
 5. **Test idempotency**: Verify duplicate detection and redelivery behavior
-6. **Mock topology setup**: Remember to stub `JetstreamBridge::Topology.ensure!`
+6. **Set stream_name**: Always configure `stream_name` in your tests
 
 ## Examples
 

data/lib/generators/jetstream_bridge/health_check/health_check_generator.rb

@@ -4,7 +4,7 @@ require 'rails/generators'
 
 module JetstreamBridge
   module Generators
-    class HealthCheckGenerator < Rails::Generators::Base
+    class HealthCheckGenerator < ::Rails::Generators::Base
       source_root File.expand_path('templates', __dir__)
       desc 'Creates a health check endpoint for JetStream Bridge monitoring'
 

data/lib/generators/jetstream_bridge/initializer/initializer_generator.rb

@@ -4,7 +4,7 @@ require 'rails/generators'
 
 module JetstreamBridge
   module Generators
-    class InitializerGenerator < Rails::Generators::Base
+    class InitializerGenerator < ::Rails::Generators::Base
       source_root File.expand_path('templates', __dir__)
       desc 'Creates config/initializers/jetstream_bridge.rb'
 

data/lib/generators/jetstream_bridge/initializer/templates/jetstream_bridge.rb

@@ -13,10 +13,6 @@ JetstreamBridge.configure do |config|
   # NATS server URLs (comma-separated for cluster)
   config.nats_urls = ENV.fetch('NATS_URLS', 'nats://localhost:4222')
 
-  # Environment identifier (e.g., 'development', 'production')
-  # Used in stream names and subject routing
-  config.env = ENV.fetch('NATS_ENV', Rails.env)
-
   # Application name (used in subject routing)
   config.app_name = ENV.fetch('APP_NAME', Rails.application.class.module_parent_name.underscore)
 

data/lib/generators/jetstream_bridge/install/install_generator.rb

@@ -5,16 +5,16 @@ require 'rails/generators'
 module JetstreamBridge
   module Generators
     # Install generator.
-    class InstallGenerator < Rails::Generators::Base
+    class InstallGenerator < ::Rails::Generators::Base
       desc 'Creates JetstreamBridge initializer and migrations'
       def create_initializer
-        Rails::Generators.invoke('jetstream_bridge:initializer', [], behavior: behavior,
-                                                                     destination_root: destination_root)
+        ::Rails::Generators.invoke('jetstream_bridge:initializer', [], behavior: behavior,
+                                                                       destination_root: destination_root)
       end
 
       def create_migrations
-        Rails::Generators.invoke('jetstream_bridge:migrations', [], behavior: behavior,
-                                                                    destination_root: destination_root)
+        ::Rails::Generators.invoke('jetstream_bridge:migrations', [], behavior: behavior,
+                                                                      destination_root: destination_root)
       end
     end
   end

data/lib/generators/jetstream_bridge/migrations/migrations_generator.rb

@@ -6,8 +6,8 @@ require 'rails/generators/active_record'
 module JetstreamBridge
   module Generators
     # Migrations generator.
-    class MigrationsGenerator < Rails::Generators::Base
-      include Rails::Generators::Migration
+    class MigrationsGenerator < ::Rails::Generators::Base
+      include ::Rails::Generators::Migration
 
       source_root File.expand_path('templates', __dir__)
       desc 'Creates Inbox/Outbox migrations for JetstreamBridge'

data/lib/jetstream_bridge/consumer/consumer.rb

@@ -2,7 +2,6 @@
 
 require 'oj'
 require 'securerandom'
-require_relative '../core/connection'
 require_relative '../core/duration'
 require_relative '../core/logging'
 require_relative '../core/config'
@@ -10,6 +9,7 @@ require_relative '../core/model_utils'
 require_relative 'message_processor'
 require_relative 'subscription_manager'
 require_relative 'inbox/inbox_processor'
+require_relative 'health_monitor'
 
 module JetstreamBridge
   # Subscribes to destination subject and processes messages via a pull durable consumer.
@@ -48,6 +48,14 @@ module JetstreamBridge
     IDLE_SLEEP_SECS       = 0.05
     # Maximum sleep duration during idle periods (seconds)
     MAX_IDLE_BACKOFF_SECS = 1.0
+    # Maximum number of batches to drain during shutdown
+    MAX_DRAIN_BATCHES = 5
+    # Drain timeout per batch in seconds
+    DRAIN_BATCH_TIMEOUT = 1
+    # Minimum reconnect backoff in seconds
+    MIN_RECONNECT_BACKOFF = 0.1
+    # Maximum reconnect backoff in seconds
+    MAX_RECONNECT_BACKOFF = 30.0
 
     # Alias middleware classes for easier access
     MiddlewareChain = ConsumerMiddleware::MiddlewareChain
@@ -64,60 +72,51 @@ module JetstreamBridge
     # @return [MiddlewareChain] Middleware chain for processing
     attr_reader :middleware_chain
 
-    # Initialize a new Consumer instance.
+    # Initialize a new Consumer instance with dependency injection.
     #
     # @param handler [Proc, #call, nil] Message handler that processes events.
     #   Must respond to #call(event) or #call(event, subject, deliveries).
+    # @param connection [NATS::JetStream::JS] JetStream connection
+    # @param config [Config] Configuration instance
     # @param durable_name [String, nil] Optional durable consumer name override.
-    #   Defaults to config.durable_name.
     # @param batch_size [Integer, nil] Number of messages to fetch per batch.
-    #   Defaults to DEFAULT_BATCH_SIZE (25).
     # @yield [event] Optional block as handler. Receives Models::Event object.
     #
-    # @raise [ArgumentError] If neither handler nor block provided
-    # @raise [ArgumentError] If destination_app not configured
-    # @raise [ConnectionError] If unable to connect to NATS
+    # @raise [ArgumentError] If required dependencies are missing
     #
     # @example With proc handler
     #   handler = ->(event) { puts "Received: #{event.type}" }
-    #   consumer = JetstreamBridge::Consumer.new(handler)
+    #   consumer = JetstreamBridge::Consumer.new(handler, connection: jts, config: config)
     #
     # @example With block
-    #   consumer = JetstreamBridge::Consumer.new do |event|
+    #   consumer = JetstreamBridge::Consumer.new(connection: jts, config: config) do |event|
     #     UserEventHandler.process(event)
     #   end
     #
-    # @example With custom configuration
-    #   consumer = JetstreamBridge::Consumer.new(
-    #     handler,
-    #     durable_name: "my-consumer",
-    #     batch_size: 10
-    #   )
-    #
-    def initialize(handler = nil, durable_name: nil, batch_size: nil, &block)
+    def initialize(handler = nil, connection:, config:, durable_name: nil, batch_size: nil, &block)
       @handler = handler || block
       raise ArgumentError, 'handler or block required' unless @handler
+      raise ArgumentError, 'connection is required' unless connection
+      raise ArgumentError, 'config is required' unless config
+
+      @jts = connection
+      @config = config
 
       @batch_size    = Integer(batch_size || DEFAULT_BATCH_SIZE)
-      @durable       = durable_name || JetstreamBridge.config.durable_name
+      @durable       = durable_name || @config.durable_name
       @idle_backoff  = IDLE_SLEEP_SECS
       @reconnect_attempts = 0
       @running = true
       @shutdown_requested = false
-      @start_time    = Time.now
-      @iterations    = 0
-      @last_health_check = Time.now
-      # Use existing connection (should already be established)
-      @jts = Connection.jetstream
-      raise ConnectionError, 'JetStream connection not available. Call JetstreamBridge.startup! first.' unless @jts
 
       @middleware_chain = MiddlewareChain.new
+      @health_monitor = ConsumerHealthMonitor.new(@durable)
 
       ensure_destination_app_configured!
 
-      @sub_mgr = SubscriptionManager.new(@jts, @durable, JetstreamBridge.config)
+      @sub_mgr = SubscriptionManager.new(@jts, @durable, @config)
       @processor  = MessageProcessor.new(@jts, @handler, middleware_chain: @middleware_chain)
-      @inbox_proc = InboxProcessor.new(@processor) if JetstreamBridge.config.use_inbox
+      @inbox_proc = InboxProcessor.new(@processor) if @config.use_inbox
 
       ensure_subscription!
       setup_signal_handlers
@@ -193,17 +192,15 @@ module JetstreamBridge
     #
     def run!
       Logging.info(
-        "Consumer #{@durable} started (batch=#{@batch_size}, dest=#{JetstreamBridge.config.destination_subject})…",
+        "Consumer #{@durable} started (batch=#{@batch_size}, dest=#{@config.destination_subject})…",
         tag: 'JetstreamBridge::Consumer'
       )
       while @running
         processed = process_batch
         idle_sleep(processed)
 
-        @iterations += 1
-
-        # Periodic health checks every 10 minutes (600 seconds)
-        perform_health_check_if_due
+        @health_monitor.increment_iterations
+        @health_monitor.check_health_if_due
       end
 
       # Drain in-flight messages before exiting
@@ -245,13 +242,13 @@ module JetstreamBridge
     private
 
     def ensure_destination_app_configured!
-      return unless JetstreamBridge.config.destination_app.to_s.empty?
+      return unless @config.destination_app.to_s.empty?
 
       raise ArgumentError, 'destination_app must be configured'
     end
 
     def ensure_subscription!
-      @sub_mgr.ensure_consumer!
+      @sub_mgr.ensure_consumer! unless @config.disable_js_api
       @psub = @sub_mgr.subscribe!
     end
 
@@ -314,10 +311,8 @@ module JetstreamBridge
     end
 
     def calculate_reconnect_backoff(attempt)
-      # Exponential backoff: 0.1s, 0.2s, 0.4s, 0.8s, 1.6s, ... up to 30s max
-      base_delay = 0.1
-      max_delay = 30.0
-      [base_delay * (2**(attempt - 1)), max_delay].min
+      # Exponential backoff: 0.1s, 0.2s, 0.4s, 0.8s, 1.6s, ... up to max
+      [MIN_RECONNECT_BACKOFF * (2**(attempt - 1)), MAX_RECONNECT_BACKOFF].min
     end
 
     def recoverable_consumer_error?(error)
@@ -357,70 +352,13 @@ module JetstreamBridge
       Logging.debug("Could not set up signal handlers: #{e.message}", tag: 'JetstreamBridge::Consumer')
     end
 
-    def perform_health_check_if_due
-      now = Time.now
-      time_since_check = now - @last_health_check
-
-      return unless time_since_check >= 600 # 10 minutes
-
-      @last_health_check = now
-      uptime = now - @start_time
-      memory_mb = memory_usage_mb
-
-      Logging.info(
-        "Consumer health: iterations=#{@iterations}, " \
-        "memory=#{memory_mb}MB, uptime=#{uptime.round}s",
-        tag: 'JetstreamBridge::Consumer'
-      )
-
-      # Warn if memory usage is high (over 1GB)
-      if memory_mb > 1000
-        Logging.warn(
-          "High memory usage detected: #{memory_mb}MB",
-          tag: 'JetstreamBridge::Consumer'
-        )
-      end
-
-      # Suggest GC if heap is growing significantly
-      suggest_gc_if_needed
-    rescue StandardError => e
-      Logging.debug(
-        "Health check failed: #{e.class} #{e.message}",
-        tag: 'JetstreamBridge::Consumer'
-      )
-    end
-
-    def memory_usage_mb
-      # Get memory usage from OS (works on Linux/macOS)
-      rss_kb = `ps -o rss= -p #{Process.pid}`.to_i
-      rss_kb / 1024.0
-    rescue StandardError
-      0.0
-    end
-
-    def suggest_gc_if_needed
-      # Suggest GC if heap has many live slots (Ruby-specific optimization)
-      return unless defined?(GC) && GC.respond_to?(:stat)
-
-      stats = GC.stat
-      heap_live_slots = stats[:heap_live_slots] || stats['heap_live_slots'] || 0
-
-      # Suggest GC if we have over 100k live objects
-      GC.start if heap_live_slots > 100_000
-    rescue StandardError => e
-      Logging.debug(
-        "GC check failed: #{e.class} #{e.message}",
-        tag: 'JetstreamBridge::Consumer'
-      )
-    end
-
     def drain_inflight_messages
       return unless @psub
 
       Logging.info('Draining in-flight messages...', tag: 'JetstreamBridge::Consumer')
       # Process any pending messages with a short timeout
-      5.times do
-        msgs = @psub.fetch(@batch_size, timeout: 1)
+      MAX_DRAIN_BATCHES.times do
+        msgs = @psub.fetch(@batch_size, timeout: DRAIN_BATCH_TIMEOUT)
         break if msgs.nil? || msgs.empty?
 
         msgs.each { |m| process_one(m) }