jetstream_bridge 7.0.6 → 7.1.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: d405fd265deca0547e5ce7384b99c5c229cff755859981171fbc0a24cb9bea96
-  data.tar.gz: 458d7ecc088dde7691be9e2b0f0c00b1a185539ec6456eadc66579639089728a
+  metadata.gz: 65a794cf3e9a17511c64f2ef29edc70d56214f4cfc1c63eed8148afb65e05505
+  data.tar.gz: 44dea5d3d1900999fc788aad9e9d676a02315c18678256f4ce14d5443cb0b816
 SHA512:
-  metadata.gz: 1da0a56666a6bb35c892dbd1f3fcacba77a1428a3ce09d9fa74de2fbe2ac309325b2f9b3a42360d500fa4f74ad3b180a3581f521fab3df5ba6fa4cbe3d076fcf
-  data.tar.gz: 82efe7aace08352be950b7b7389bde00d0eb603041445af90e810c86567884139db6ce716873aadecbd15251e3f5d033b58a1f9acfa451d579b5f8d13eb3d588
+  metadata.gz: 360c547ece5e7443487c3796d18139843ea86a2ce7f29a775fd1245d6e084d76bec11df72cdafef78ac9fa383ad99eae680f9220dc9eef451f62ba012fe20de9
+  data.tar.gz: 62e2100993ba1defc32df1ad744392148f4600527c5fab80db6cce2d211b80c79d6228c64497e61106ba13fdef9579300df1455d5caf3d4e75a83ccd896a5051

data/CHANGELOG.md

@@ -5,6 +5,32 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [7.1.1] - 2026-02-02
+
+### Changed
+
+- Fail-fast consumer provisioning: when running in push mode with `auto_provision=false`, the bridge now raises `ConsumerProvisioningError` instead of silently skipping consumer setup, preventing idle workers when the durable is missing.
+- Permission-aware creation: consumer creation now surfaces `ConsumerProvisioningError` on permissions violations (pull or push mode), guiding operators to grant the minimal `$JS.API` rights or pre-provision the durable.
+
+### Added
+
+- New `ConsumerProvisioningError` topology error for clearer operator feedback when consumer setup cannot proceed.
+
+## [7.1.0] - 2026-02-01
+
+### Added
+
+- **Auto-create consumer on subscription** - Consumers are now automatically created if they don't exist when subscribing, regardless of the `auto_provision` setting. This eliminates the need for manual consumer provisioning while `auto_provision` continues to control only stream topology creation.
+- `SubscriptionManager#stream_exists?` - Public method to check if a stream exists.
+- `SubscriptionManager#consumer_exists?` - Public method to check if a consumer exists in the stream.
+- `SubscriptionManager#create_consumer_if_missing!` - Public method to create a consumer only if it doesn't already exist (race-condition safe). Raises `StreamNotFoundError` if the stream doesn't exist.
+
+### Changed
+
+- `ensure_consumer!` now always auto-creates the consumer if it doesn't exist. The `auto_provision` setting only controls stream topology (stream creation), not consumer creation.
+- Consumer recovery automatically attempts to create the consumer when a "consumer not found" error is detected.
+- **Stream must exist** - Auto-create consumer fails fast with `StreamNotFoundError` if the stream doesn't exist. Streams must be provisioned separately via `auto_provision=true` or manual provisioning.
+
 ## [7.0.0] - 2026-01-30
 
 ### Added

data/README.md

@@ -26,16 +26,16 @@
 
 - Transactional outbox and idempotent inbox (optional) for exactly-once pipelines.
 - Durable pull (default) or push consumers with retries, backoff, and DLQ routing.
-- Auto stream/consumer provisioning with overlap protection.
+- Auto stream provisioning with overlap protection; consumers auto-created on subscription.
 - Rails-native: generators, migrations, health check, and eager-loading safety.
-- Least-privilege friendly: run with `auto_provision=false` plus pre-created consumers.
+- Least-privilege friendly: run with `auto_provision=false` (stream must exist; consumers are auto-created).
 - Mock NATS for fast, no-infra testing.
 
 ## Quick Start
 
 ```ruby
 # Gemfile
-gem "jetstream_bridge", "~> 7.0"
+gem "jetstream_bridge", "~> 7.1"
 ```
 
 ```bash

data/docs/API.md

@@ -40,7 +40,7 @@ JetstreamBridge.configure do |config|
   config.backoff      = ["1s", "5s", "15s", "30s", "60s"]
 
   # Provisioning
-  config.auto_provision = true  # Auto-create stream/consumer on startup
+  config.auto_provision = true  # Auto-create stream on startup (consumers are always auto-created)
 
   # Connection behavior
   config.lazy_connect = false  # Set true to skip autostart
@@ -224,6 +224,28 @@ provisioner.provision_stream!
 provisioner.provision_consumer!
 ```
 
+### `JetstreamBridge::SubscriptionManager`
+
+Manages consumer lifecycle during subscription. Used internally by `Consumer`, but also available for advanced use cases.
+
+```ruby
+# Create subscription manager
+sub_mgr = JetstreamBridge::SubscriptionManager.new(jetstream_context, "my-durable")
+
+# Check if stream exists
+sub_mgr.stream_exists?  # => true/false
+
+# Check if consumer exists
+sub_mgr.consumer_exists?  # => true/false
+
+# Create consumer if missing (raises StreamNotFoundError if stream doesn't exist)
+sub_mgr.create_consumer_if_missing!
+```
+
+**Note:** Consumers are automatically created when subscribing, regardless of the `auto_provision` setting. The `auto_provision` setting only controls stream topology creation.
+
+**Raises:** `JetstreamBridge::StreamNotFoundError` if the stream doesn't exist when calling `create_consumer_if_missing!`
+
 ## Health & Diagnostics
 
 ### `JetstreamBridge.health_check`
@@ -347,6 +369,20 @@ rescue JetstreamBridge::PublishError => e
 end
 ```
 
+### `JetstreamBridge::StreamNotFoundError`
+
+Raised when attempting to create a consumer on a stream that doesn't exist.
+
+```ruby
+begin
+  consumer = JetstreamBridge::Consumer.new { |event| process(event) }
+  consumer.run!
+rescue JetstreamBridge::StreamNotFoundError => e
+  logger.error("Stream not found: #{e.message}")
+  # Stream must be provisioned separately (use auto_provision=true or run provisioning with admin credentials)
+end
+```
+
 ### Custom Error Handler
 
 ```ruby

data/docs/GETTING_STARTED.md

@@ -6,7 +6,7 @@ This guide covers installation, Rails setup, configuration, and basic publish/co
 
 ```ruby
 # Gemfile
-gem "jetstream_bridge", "~> 7.0"
+gem "jetstream_bridge", "~> 7.1"
 ```
 
 ```bash
@@ -109,6 +109,12 @@ JetstreamBridge.configure do |config|
   config.max_deliver = 5
   config.ack_wait    = "30s"
   config.backoff     = %w[1s 5s 15s 30s 60s]
+
+  # Provisioning (default: true)
+  # - When true: auto-creates stream on startup
+  # - When false: stream must exist (provision separately with admin credentials)
+  # Note: Consumers are always auto-created on subscription regardless of this setting
+  config.auto_provision = true
 end
 
 # Note: `configure` only sets options; it does not connect. Rails will start
@@ -119,6 +125,8 @@ end
 
 Rails autostart runs after initialization (including in console). You can opt out for rake tasks or other tooling with `config.lazy_connect = true` or `JETSTREAM_BRIDGE_DISABLE_AUTOSTART=1`; it will then connect on first publish/subscribe.
 
+**Important:** When `auto_provision=false`, the stream must exist before subscribing. Consumers are automatically created when they don't exist, but attempting to create a consumer on a missing stream will raise `StreamNotFoundError`.
+
 ### Push consumer mode (restricted credentials)
 
 If your NATS user cannot publish to `$JS.API.*`, switch to push consumers and pre-create the durable consumer:

data/docs/RESTRICTED_PERMISSIONS.md

@@ -36,13 +36,15 @@ When you cannot modify NATS server permissions, you have **two options**:
 
 For both options, you need to:
 
-0. **Turn off runtime provisioning** so the app never calls `$JS.API.*`:
+0. **Turn off runtime provisioning** so the app never calls `$JS.API.*` for stream creation:
    - Set `config.auto_provision = false`
-   - Provision stream + consumer once with admin credentials (CLI below or `bundle exec rake jetstream_bridge:provision`)
-1. **Pre-create the consumer** using a privileged NATS account
-2. **Ensure the consumer configuration matches** what your app expects
+   - Provision the **stream** once with admin credentials (CLI below or `bundle exec rake jetstream_bridge:provision`)
+1. **Ensure the stream exists** - consumers are auto-created when subscribing, but the stream must be provisioned separately
+2. **Optionally pre-create the consumer** using a privileged NATS account (consumers are auto-created if missing)
 
-> Tip: When `auto_provision=false`, the app still connects/publishes/consumes but skips JetStream management APIs (account_info, stream_info, consumer_info). Health checks will report basic connectivity only.
+> **Note (v7.1.0+):** Consumers are now auto-created on subscription regardless of the `auto_provision` setting. The `auto_provision` setting only controls stream topology creation. If the stream doesn't exist, subscription will fail with `StreamNotFoundError`.
+
+> Tip: When `auto_provision=false`, the app still connects/publishes/consumes but skips JetStream management APIs (account_info, stream_info) for stream creation. Health checks will report basic connectivity only.
 
 ---
 
@@ -543,19 +545,21 @@ nats stream info pwas-heavyworth-sync
 
 Check if the issue is:
 
-1. **Consumer doesn't exist** - Pre-create it with NATS CLI
+1. **Stream doesn't exist** - Provision it with NATS CLI or admin credentials (consumers are auto-created)
 2. **Filter subject mismatch** - Verify with `nats consumer info`
 3. **Permissions still insufficient** - User needs subscribe permissions on the filtered subject
 4. **Wrong stream name** - Ensure stream name matches your configured `config.stream_name`
 
+**Note (v7.1.0+):** Consumers are auto-created if they don't exist, so "consumer doesn't exist" is no longer a common cause. However, the **stream must exist** or you'll see `StreamNotFoundError`.
+
 ---
 
 ## Security Considerations
 
-1. **Consumer pre-creation requires privileged access** - Keep admin credentials secure
+1. **Stream pre-creation requires privileged access** - Keep admin credentials secure
 2. **Configuration drift risk** - If app config doesn't match consumer config, message delivery may fail silently
-3. **No automatic recovery** - If consumer is deleted, it won't be recreated automatically
-4. **Consider automation** - Use infrastructure-as-code (Terraform, Ansible) to manage consumer creation
+3. **Automatic consumer recovery (v7.1.0+)** - If consumer is deleted, it will be auto-recreated on subscription. Stream must exist.
+4. **Consider automation** - Use infrastructure-as-code (Terraform, Ansible) to manage stream creation
 
 ---
 

data/lib/jetstream_bridge/consumer/consumer.rb

@@ -429,6 +429,9 @@ module JetstreamBridge
           tag: 'JetstreamBridge::Consumer'
         )
 
+        # If this looks like a consumer-not-found error, try to auto-create it
+        auto_create_consumer_on_error(error) if consumer_not_found_error?(error)
+
         sleep(backoff_secs)
         ensure_subscription!
 
@@ -440,6 +443,27 @@ module JetstreamBridge
       0
     end
 
+    def consumer_not_found_error?(error)
+      msg = error.message.to_s.downcase
+      (msg.include?('consumer') && (msg.include?('not found') || msg.include?('does not exist'))) ||
+        msg.include?('no responders')
+    end
+
+    def auto_create_consumer_on_error(_error)
+      Logging.info(
+        "Consumer not found error detected, attempting auto-creation for #{@durable}...",
+        tag: 'JetstreamBridge::Consumer'
+      )
+
+      @sub_mgr.create_consumer_if_missing!
+    rescue StandardError => e
+      Logging.warn(
+        "Auto-create consumer failed: #{e.class} #{e.message}",
+        tag: 'JetstreamBridge::Consumer'
+      )
+      # Don't re-raise - let the normal recovery flow continue
+    end
+
     def calculate_reconnect_backoff(attempt)
       # Exponential backoff: 0.1s, 0.2s, 0.4s, 0.8s, 1.6s, ... up to 30s max
       base_delay = 0.1

data/lib/jetstream_bridge/consumer/subscription_manager.rb

@@ -28,14 +28,107 @@ module JetstreamBridge
       @desired_cfg
     end
 
-    def ensure_consumer!(force: false)
-      # Runtime path: never hit JetStream management APIs to avoid admin permissions.
-      unless force || @cfg.auto_provision
-        log_runtime_skip
+    # Ensure consumer exists, auto-creating if missing.
+    #
+    # @param force [Boolean] Kept for backward compatibility but no longer used.
+    #   Consumers are always auto-created regardless of this parameter.
+    def ensure_consumer!(**_options)
+      # Always auto-create consumer if it doesn't exist, regardless of auto_provision setting.
+      # auto_provision only controls stream topology creation, not consumer creation.
+      create_consumer_if_missing!
+    end
+
+    # Check if stream exists.
+    #
+    # @return [Boolean] true if stream exists, false otherwise
+    def stream_exists?
+      @jts.stream_info(stream_name)
+      true
+    rescue StandardError => e
+      msg = e.message.to_s.downcase
+      return false if msg.include?('not found') || msg.include?('does not exist') || msg.include?('no responders')
+
+      # Re-raise unexpected errors
+      raise unless msg.include?('stream')
+
+      false
+    end
+
+    # Check if consumer exists in the stream.
+    #
+    # @return [Boolean] true if consumer exists, false otherwise
+    def consumer_exists?
+      @jts.consumer_info(stream_name, @durable)
+      true
+    rescue StandardError => e
+      msg = e.message.to_s.downcase
+      return false if msg.include?('not found') || msg.include?('does not exist') || msg.include?('no responders')
+
+      # Re-raise unexpected errors
+      raise unless msg.include?('consumer')
+
+      false
+    end
+
+    # Create consumer only if it doesn't already exist.
+    #
+    # Fails if the stream doesn't exist - streams must be provisioned separately.
+    #
+    # This is a safer alternative to create_consumer! that won't fail
+    # if the consumer was already created by another process.
+    #
+    # @raise [StreamNotFoundError] if the stream doesn't exist
+    def create_consumer_if_missing!
+      # In restricted environments (push + auto_provision=false), we still want fail-fast semantics.
+      # Attempt creation and raise a clear error so operators know the consumer must be pre-provisioned
+      # or the account must allow the minimal $JS.API consumer permissions.
+      if skip_consumer_management?
+        raise JetstreamBridge::ConsumerProvisioningError,
+              "Consumer '#{@durable}' not ensured because auto_provision=false and push mode is enabled. " \
+              'Provision the consumer with admin credentials or grant minimal consumer create/info permissions.'
+      end
+
+      # First, verify stream exists - fail fast with clear error if not
+      unless stream_exists?
+        raise StreamNotFoundError,
+              "Stream '#{stream_name}' does not exist. " \
+              'Streams must be provisioned separately ' \
+              '(use auto_provision=true or run provisioning with admin credentials).'
+      end
+
+      if consumer_exists?
+        Logging.info(
+          "Consumer #{@durable} already exists (stream=#{stream_name})",
+          tag: 'JetstreamBridge::Consumer'
+        )
         return
       end
 
+      Logging.info(
+        "Consumer #{@durable} not found, auto-creating on stream #{stream_name}...",
+        tag: 'JetstreamBridge::Consumer'
+      )
       create_consumer!
+    rescue StreamNotFoundError
+      raise
+    rescue StandardError => e
+      raise ConsumerProvisioningError, permission_error_message(e) if permission_denied?(e)
+
+      # If creation fails due to consumer already existing (race condition), that's OK
+      msg = e.message.to_s.downcase
+      if msg.include?('already') || msg.include?('exists')
+        Logging.info(
+          "Consumer #{@durable} was created by another process",
+          tag: 'JetstreamBridge::Consumer'
+        )
+        return
+      end
+
+      Logging.error(
+        "Failed to auto-create consumer #{@durable}: #{e.class} #{e.message}",
+        tag: 'JetstreamBridge::Consumer'
+      )
+      raise
     end
 
     # Bind a subscriber to the existing durable consumer.
@@ -128,6 +221,21 @@ module JetstreamBridge
       config
     end
 
+    def skip_consumer_management?
+      @cfg.push_consumer? && !@cfg.auto_provision
+    end
+
+    def permission_denied?(error)
+      msg = error.message.to_s.downcase
+      msg.include?('permission') || msg.include?('permissions violation')
+    end
+
+    def permission_error_message(error)
+      "Consumer '#{@durable}' could not be ensured due to permissions: #{error.message}. " \
+        'Grant $JS.API.STREAM.INFO/$JS.API.CONSUMER.INFO/$JS.API.CONSUMER.CREATE for the stream, ' \
+        'or pre-provision the consumer with an admin account.'
+    end
+
     def create_consumer!
       @jts.add_consumer(stream_name, **desired_consumer_cfg)
       Logging.info(
@@ -136,14 +244,6 @@ module JetstreamBridge
       )
     end
 
-    def log_runtime_skip
-      Logging.info(
-        "Skipping consumer provisioning/verification for #{@durable} at runtime to avoid JetStream API usage. " \
-        'Ensure it is pre-created via provisioning.',
-        tag: 'JetstreamBridge::Consumer'
-      )
-    end
-
     def resolve_nc
       return @jts.nc if @jts.respond_to?(:nc)
       return @jts.instance_variable_get(:@nc) if @jts.instance_variable_defined?(:@nc)

data/lib/jetstream_bridge/errors.rb

@@ -67,6 +67,7 @@ module JetstreamBridge
 
   # Topology errors
   class TopologyError < Error; end
+  class ConsumerProvisioningError < TopologyError; end
   class StreamNotFoundError < TopologyError; end
   class SubjectOverlapError < TopologyError; end
   class StreamCreationFailedError < TopologyError; end

data/lib/jetstream_bridge/version.rb

@@ -4,5 +4,5 @@
 #
 # Version constant for the gem.
 module JetstreamBridge
-  VERSION = '7.0.6'
+  VERSION = '7.1.1'
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: jetstream_bridge
 version: !ruby/object:Gem::Version
-  version: 7.0.6
+  version: 7.1.1
 platform: ruby
 authors:
 - Mike Attara
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2026-02-01 00:00:00.000000000 Z
+date: 2026-02-02 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activerecord