jetstream_bridge 5.1.0 → 7.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: '087fb2993c66be217aa3b544a333d0b166aec88f9a006051010e46309faac51f'
-  data.tar.gz: 2489d32b3047955699c11ae640014baec66855cad384543b3220fe778ff2b65a
+  metadata.gz: 471b7dd68739c9de058cbaeedb800dbb5c0e1eadaa2d54602eee34d8469f2668
+  data.tar.gz: 7d16630ce0e83b6d6fb1ab6f9e6c5151a6aa5df7467810f950fd984e035bb9f0
 SHA512:
-  metadata.gz: 88e316fb7cfc4f5048290ca37e3c065fc7cb6e059627f5900455c14adf59957c9f8db8e428b82f879511979db2e52ff1a15decb754c7156f9a51f81d766d30f3
-  data.tar.gz: 897a1695d32323c7d33dcc8a3811cb9a7952e45088d14b8c81713f08085869805d97a40de34c3c10f33ae8c39861972ff4f9c1cff1af57699c10aa5e8bf05a50
+  metadata.gz: d58a47de952678819cd5681b750907ca9d56be3bf9799d5e6f614acd828c9d86da80c7be1006280cbfa42c88ebe8fd6a804da829b5f73672dd59766955df1d16
+  data.tar.gz: f963140e8a639fc285b633f112df806ff3638e3d83f1b834702d91a47d4a2f28c704071c8ae2f3d763c6d750aa708e12babe33872dbb3778d53058965cd987e0

data/CHANGELOG.md

@@ -5,6 +5,43 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [7.0.0] - 2026-01-30
+
+### Added
+
+- Full Rails reference examples (non-restrictive and restrictive) with Docker Compose, provisioner service, and end-to-end test scripts.
+- Architecture and API reference docs covering topology, consumer modes, and public surface.
+
+### Changed
+
+- Major refactor separating provisioning from runtime to support least-privilege deployments; `auto_provision=false` now avoids JetStream management APIs at runtime.
+- Config helpers simplified bidirectional setup (`configure_bidirectional`, `setup_rails_lifecycle`) replacing verbose initializer boilerplate.
+- Consumer pipeline hardened: push-consumer mode, safer signal handling, improved drain behavior, and pull subscription shim reliability.
+
+### Fixed
+
+- Provisioner keyword argument handling, generator load edge cases, and connection/consumer recovery during JetStream context refresh.
+
+## [5.0.0] - 2026-01-30
+
+### Added
+
+- **Push consumer mode** for restricted NATS credentials — `consumer_mode: :push` with optional `delivery_subject`, no `$JS.API.*` permissions required, documented in the restricted permissions guide.
+- **Reference examples**: full Rails apps for non-restrictive (auto-provisioning) and restrictive (least-privilege + provisioner) deployments with Docker Compose and end-to-end test scripts.
+- **ConfigHelpers & docs**: helper to configure bidirectional sync in one call, Rails lifecycle helper, new Architecture/API docs describing topology and public surface.
+
+### Changed
+
+- **Provisioning flow** separated from runtime: when `auto_provision=false` runtime skips JetStream management APIs; provisioning handled via rake task/CLI with admin creds. Health/test_connection honor restricted mode.
+- **Consumer reliability**: refactored subscription builder, trap-safe signal handling, push-consumer drain via `next_msg`, sturdier pull subscription shim, JetStream context refresh retries after reconnects.
+- **Quality & coverage**: generator template tweaks, provisioner keyword fixes, and test suite expanded (~96% coverage) across provisioning, consumer, and connection lifecycle paths.
+
+### Fixed
+
+- Fixed TypeError/timeout handling in push consumer drain and process loop.
+- Fixed Rails generator load edge case and provisioner keyword argument bugs.
+- Guarded JetStream API calls when constants are private, preserving connection state during JetStream context refresh failures.
+
 ## [4.4.1] - 2026-01-16
 
 ### Fixed

data/README.md

@@ -20,21 +20,22 @@
   </a>
 </p>
 
-Production-ready NATS JetStream bridge for Ruby/Rails with outbox, inbox, DLQ, and overlap-safe stream provisioning.
+ Production-ready NATS JetStream bridge for Ruby/Rails with outbox, inbox, DLQ, and overlap-safe stream provisioning.
 
 ## Highlights
 
 - Transactional outbox and idempotent inbox (optional) for exactly-once pipelines.
-- Durable pull consumers with retries, backoff, and DLQ routing.
+- Durable pull (default) or push consumers with retries, backoff, and DLQ routing.
 - Auto stream/consumer provisioning with overlap protection.
 - Rails-native: generators, migrations, health check, and eager-loading safety.
+- Least-privilege friendly: run with `auto_provision=false` plus pre-created consumers.
 - Mock NATS for fast, no-infra testing.
 
 ## Quick Start
 
 ```ruby
 # Gemfile
-gem "jetstream_bridge", "~> 5.0"
+gem "jetstream_bridge", "~> 7.0"
 ```
 
 ```bash
@@ -63,10 +64,12 @@ consumer.run!
 ## Documentation
 
 - [Getting Started](docs/GETTING_STARTED.md) - Setup, configuration, and basic usage
+- [API Reference](docs/API.md) - Complete API documentation for all public methods
 - [Architecture & Topology](docs/ARCHITECTURE.md) - Internal architecture, message flow, and patterns
 - [Production Guide](docs/PRODUCTION.md) - Production deployment and monitoring
 - [Restricted Permissions & Provisioning](docs/RESTRICTED_PERMISSIONS.md) - Manual provisioning and security
 - [Testing with Mock NATS](docs/TESTING.md) - Fast, no-infra testing
+- [Reference Examples (Rails 7)](examples/README.md) - Non-restrictive and restrictive Dockerized examples with E2E tests
 
 ## License
 

data/docs/API.md

@@ -0,0 +1,395 @@
+# API Reference
+
+Complete reference for JetstreamBridge public API.
+
+## Table of Contents
+
+- [Configuration](#configuration)
+- [Lifecycle Methods](#lifecycle-methods)
+- [Publishing](#publishing)
+- [Consuming](#consuming)
+- [Provisioning](#provisioning)
+- [Health & Diagnostics](#health--diagnostics)
+- [Models](#models)
+
+## Configuration
+
+### `JetstreamBridge.configure`
+
+Configure the library. Must be called before connecting.
+
+```ruby
+JetstreamBridge.configure do |config|
+  # Required
+  config.app_name        = "my_app"
+  config.destination_app = "other_app"
+
+  # Connection
+  config.nats_urls = "nats://localhost:4222"  # or array of URLs
+  config.stream_name = "jetstream-bridge-stream"
+
+  # Features
+  config.use_outbox = true   # Transactional publish (requires ActiveRecord)
+  config.use_inbox  = true   # Idempotent consume (requires ActiveRecord)
+  config.use_dlq    = true   # Dead letter queue for poison messages
+
+  # Consumer settings
+  config.durable_name = "#{app_name}-workers"
+  config.max_deliver  = 5     # Max delivery attempts
+  config.ack_wait     = "30s" # Time to wait for ACK
+  config.backoff      = ["1s", "5s", "15s", "30s", "60s"]
+
+  # Provisioning
+  config.auto_provision = true  # Auto-create stream/consumer on startup
+
+  # Connection behavior
+  config.lazy_connect = false  # Set true to skip autostart
+  config.connect_retry_attempts = 3
+  config.connect_retry_delay = 1  # seconds
+end
+```
+
+### `JetstreamBridge.config`
+
+Returns the current configuration object (read-only).
+
+```ruby
+stream_name = JetstreamBridge.config.stream_name
+```
+
+## Lifecycle Methods
+
+### `JetstreamBridge.startup!`
+
+Explicitly start the connection and provision topology (if `auto_provision=true`).
+
+```ruby
+JetstreamBridge.startup!
+```
+
+**Note:** Rails applications auto-start after initialization. Non-Rails apps should call this manually or rely on auto-connect on first publish/subscribe.
+
+### `JetstreamBridge.shutdown!`
+
+Gracefully close the NATS connection.
+
+```ruby
+JetstreamBridge.shutdown!
+```
+
+### `JetstreamBridge.reset!`
+
+Reset all internal state (for testing).
+
+```ruby
+JetstreamBridge.reset!
+```
+
+## Publishing
+
+### `JetstreamBridge.publish`
+
+Publish an event to the destination app.
+
+```ruby
+JetstreamBridge.publish(
+  event_type: "user.created",     # Required
+  resource_type: "user",          # Required
+  resource_id: user.id,           # Optional
+  payload: { id: user.id, email: user.email },
+  headers: { correlation_id: "..." },  # Optional
+  event_id: "custom-uuid"         # Optional (auto-generated)
+)
+```
+
+**Returns:** `JetstreamBridge::PublishResult`
+
+**With Outbox:**
+
+```ruby
+# Transactional publish (commits with your DB transaction)
+User.transaction do
+  user.save!
+  JetstreamBridge.publish(event_type: "user.created", resource_type: "user", payload: user)
+end
+```
+
+### `JetstreamBridge.publish!`
+
+Like `publish` but raises on error.
+
+```ruby
+JetstreamBridge.publish!(event_type: "user.created", resource_type: "user", payload: data)
+```
+
+**Raises:** `JetstreamBridge::PublishError` on failure
+
+### `JetstreamBridge.publish_batch`
+
+Publish multiple events efficiently.
+
+```ruby
+results = JetstreamBridge.publish_batch do |batch|
+  users.each do |user|
+    batch.publish(event_type: "user.created", resource_type: "user", payload: user)
+  end
+end
+
+puts "Published: #{results.success_count}, Failed: #{results.failure_count}"
+```
+
+## Consuming
+
+### `JetstreamBridge::Consumer.new`
+
+Create a consumer to process incoming events.
+
+```ruby
+consumer = JetstreamBridge::Consumer.new do |event|
+  # Process event
+  User.upsert({ id: event.payload["id"], email: event.payload["email"] })
+end
+```
+
+**Options:**
+
+```ruby
+consumer = JetstreamBridge::Consumer.new(
+  batch_size: 10,           # Process up to 10 messages at once
+  error_handler: ->(error, event) { logger.error(error) }
+) do |event|
+  # ...
+end
+```
+
+### `Consumer#run!`
+
+Start consuming messages (blocks until interrupted).
+
+```ruby
+consumer.run!
+```
+
+### `Consumer#stop!`
+
+Gracefully stop the consumer.
+
+```ruby
+consumer.stop!
+```
+
+### Event Object
+
+The event object passed to your handler:
+
+```ruby
+event.event_id       # => "evt_123"
+event.event_type     # => "user.created"
+event.resource_type  # => "user"
+event.resource_id    # => "456"
+event.payload        # => { "id" => 456, "email" => "..." }
+event.headers        # => { "correlation_id" => "..." }
+event.subject        # => "source_app.sync.my_app"
+event.stream         # => "jetstream-bridge-stream"
+event.seq            # => 123
+event.deliveries     # => 1
+```
+
+## Provisioning
+
+### `JetstreamBridge.provision!`
+
+Manually provision stream and consumer.
+
+```ruby
+# Provision both stream and consumer
+JetstreamBridge.provision!
+
+# Provision stream only
+JetstreamBridge.provision!(provision_consumer: false)
+```
+
+### `JetstreamBridge::Provisioner`
+
+Dedicated provisioning class for advanced use cases.
+
+```ruby
+provisioner = JetstreamBridge::Provisioner.new
+
+# Provision everything
+provisioner.provision!
+
+# Or separately
+provisioner.provision_stream!
+provisioner.provision_consumer!
+```
+
+## Health & Diagnostics
+
+### `JetstreamBridge.health_check`
+
+Get comprehensive health status.
+
+```ruby
+health = JetstreamBridge.health_check
+
+health[:status]           # => "healthy" | "unhealthy"
+health[:connected]        # => true/false
+health[:stream_exists]    # => true/false
+health[:messages]         # => 123
+health[:consumers]        # => 2
+health[:nats_rtt_ms]      # => 1.2
+health[:version]          # => "7.0.0"
+```
+
+### `JetstreamBridge.stream_info`
+
+Get detailed stream information.
+
+```ruby
+info = JetstreamBridge.stream_info
+
+info[:name]               # => "jetstream-bridge-stream"
+info[:subjects]           # => ["app1.sync.app2", ...]
+info[:messages]           # => 1000
+info[:bytes]              # => 204800
+info[:first_seq]          # => 1
+info[:last_seq]           # => 1000
+info[:consumer_count]     # => 2
+```
+
+### `JetstreamBridge.connection_info`
+
+Get NATS connection details.
+
+```ruby
+info = JetstreamBridge.connection_info
+
+info[:connected]          # => true
+info[:servers]            # => ["nats://localhost:4222"]
+info[:connected_at]       # => 2024-01-29 12:00:00 UTC
+```
+
+## Models
+
+### `JetstreamBridge::OutboxEvent`
+
+ActiveRecord model for outbox events (when `use_outbox=true`).
+
+```ruby
+# Create outbox event
+event = JetstreamBridge::OutboxEvent.create!(
+  event_id: SecureRandom.uuid,
+  event_type: "user.created",
+  resource_type: "user",
+  resource_id: "123",
+  payload: { id: 123, email: "user@example.com" },
+  subject: "my_app.sync.other_app",
+  status: "pending"
+)
+
+# Query
+JetstreamBridge::OutboxEvent.pending.limit(100)
+JetstreamBridge::OutboxEvent.failed
+
+# Mark as published
+event.mark_published!
+
+# Cleanup old events
+JetstreamBridge::OutboxEvent.cleanup_published(older_than: 7.days)
+```
+
+### `JetstreamBridge::InboxEvent`
+
+ActiveRecord model for inbox events (when `use_inbox=true`).
+
+```ruby
+# Find by event_id
+event = JetstreamBridge::InboxEvent.find_by(event_id: "evt_123")
+
+# Query
+JetstreamBridge::InboxEvent.received
+JetstreamBridge::InboxEvent.processing
+JetstreamBridge::InboxEvent.processed
+JetstreamBridge::InboxEvent.failed
+JetstreamBridge::InboxEvent.recent(100)
+
+# Mark as processed
+event.mark_processed!
+
+# Mark as failed
+event.mark_failed!("Error message")
+
+# Cleanup old events
+JetstreamBridge::InboxEvent.cleanup_processed(older_than: 30.days)
+
+# Statistics
+stats = JetstreamBridge::InboxEvent.processing_stats
+stats[:total]       # => 1000
+stats[:processed]   # => 950
+stats[:failed]      # => 30
+stats[:pending]     # => 20
+```
+
+## Error Handling
+
+### `JetstreamBridge::PublishError`
+
+Raised by `publish!` when publishing fails.
+
+```ruby
+begin
+  JetstreamBridge.publish!(event_type: "test", resource_type: "test", payload: {})
+rescue JetstreamBridge::PublishError => e
+  logger.error("Publish failed: #{e.message}")
+  logger.error("Event ID: #{e.event_id}")
+  logger.error("Subject: #{e.subject}")
+end
+```
+
+### Custom Error Handler
+
+```ruby
+consumer = JetstreamBridge::Consumer.new(
+  error_handler: lambda { |error, event|
+    logger.error("Failed to process event #{event.event_id}: #{error.message}")
+    Sentry.capture_exception(error, extra: { event_id: event.event_id })
+  }
+) do |event|
+  # Process event
+end
+```
+
+## Testing
+
+### Test Mode
+
+Enable mock NATS for testing without infrastructure.
+
+```ruby
+# RSpec
+RSpec.configure do |config|
+  config.before(:each, :jetstream) do
+    JetstreamBridge::TestHelpers.enable_test_mode!
+  end
+
+  config.after(:each, :jetstream) do
+    JetstreamBridge::TestHelpers.reset_test_mode!
+  end
+end
+
+# Test
+it "publishes events", :jetstream do
+  result = JetstreamBridge.publish(event_type: "test", resource_type: "test", payload: {})
+  expect(result).to be_success
+end
+```
+
+See [TESTING.md](TESTING.md) for comprehensive testing documentation.
+
+## See Also
+
+- [Getting Started](GETTING_STARTED.md) - Setup and basic usage
+- [Architecture](ARCHITECTURE.md) - Internal architecture and patterns
+- [Production Guide](PRODUCTION.md) - Production deployment
+- [Testing Guide](TESTING.md) - Testing with Mock NATS

data/docs/ARCHITECTURE.md

@@ -331,7 +331,7 @@ Overlap detection ensures messages route to exactly one stream.
                        ▼
 ┌──────────────────────────────────────────────────────────────┐
 │ 3. [OPTIONAL] Outbox pattern                                 │
-│    - OutboxRepository.persist_pre()                          │
+│    - OutboxRepository.record_publish_attempt()               │
 │    - State: "publishing"                                     │
 │    - Database transaction commits                            │
 └──────────────────────┬───────────────────────────────────────┘
@@ -347,8 +347,8 @@ Overlap detection ensures messages route to exactly one stream.
                        ▼
 ┌──────────────────────────────────────────────────────────────┐
 │ 5. [OPTIONAL] Outbox update                                  │
-│    - Success: OutboxRepository.persist_success()             │
-│    - Failure: OutboxRepository.persist_failure()             │
+│    - Success: OutboxRepository.record_publish_success()      │
+│    - Failure: OutboxRepository.record_publish_failure()      │
 └──────────────────────┬───────────────────────────────────────┘
                        │
                        ▼
@@ -437,7 +437,7 @@ Overlap detection ensures messages route to exactly one stream.
 
 ### Outbox Pattern (Publisher Side)
 
-**Purpose:** Guarantee at-least-once delivery by persisting events to database before publishing.
+**Purpose:** Guarantee at-most-once delivery by persisting events to database before publishing.
 
 **Configuration:**
 
@@ -486,6 +486,50 @@ config.inbox_model = 'JetstreamBridge::InboxEvent'
 - `processed` - Successfully processed
 - `failed` - Failed processing
 
+**Schema Requirements:**
+
+Generate the inbox events table migration:
+
+```bash
+rails generate jetstream_bridge:migration
+rails db:migrate
+```
+
+**Required Fields:**
+
+| Field | Type | Nullable | Description |
+| ----- | ---- | -------- | ----------- |
+| `event_id` | string | NO | Unique event identifier for deduplication |
+| `event_type` | string | NO | Type of event (e.g., 'created', 'updated') |
+| `payload` | text | NO | Full event payload as JSON |
+| `status` | string | NO | Processing status (received/processing/processed/failed) |
+| `processing_attempts` | integer | NO | Number of processing attempts (default: 0) |
+| `created_at` | timestamp | NO | When the record was created |
+| `updated_at` | timestamp | NO | When the record was last updated |
+
+**Optional Fields (useful for debugging and querying):**
+
+| Field | Type | Nullable | Description |
+| ----- | ---- | -------- | ----------- |
+| `resource_type` | string | YES | Type of resource (e.g., 'organization', 'user') |
+| `resource_id` | string | YES | ID of the resource being synced |
+| `subject` | string | YES | NATS subject the message was received on |
+| `headers` | jsonb | YES | NATS message headers |
+| `stream` | string | YES | JetStream stream name |
+| `stream_seq` | bigint | YES | Stream sequence number (fallback deduplication key) |
+| `deliveries` | integer | YES | Number of delivery attempts from NATS |
+| `error_message` | text | YES | Error message if processing failed |
+| `received_at` | timestamp | YES | When the event was first received |
+| `processed_at` | timestamp | YES | When the event was successfully processed |
+| `failed_at` | timestamp | YES | When the event failed processing |
+
+**Indexes:**
+
+- `event_id` - Unique index for fast deduplication
+- `status` - Index for querying by processing status
+- `created_at` - Index for time-based queries
+- `(stream, stream_seq)` - Unique composite index for fallback deduplication
+
 **Deduplication:**
 
 - Uses `event_id` for primary deduplication
@@ -508,6 +552,24 @@ inbox.processed_at # => 2024-01-01 00:00:00
 # Skip processing, already done
 ```
 
+**Field Population:**
+
+The InboxRepository automatically extracts and sets fields from the message payload:
+
+```ruby
+# Extracted from message body
+event_type: msg.body['type'] || msg.body['event_type']
+resource_type: msg.body['resource_type']
+resource_id: msg.body['resource_id']
+
+# NATS metadata
+subject: msg.subject
+headers: msg.headers
+stream: msg.stream
+stream_seq: msg.seq
+deliveries: msg.deliveries
+```
+
 ### Dead Letter Queue (DLQ)
 
 **Purpose:** Route unrecoverable messages to separate subject for manual intervention.

data/docs/GETTING_STARTED.md

@@ -6,7 +6,7 @@ This guide covers installation, Rails setup, configuration, and basic publish/co
 
 ```ruby
 # Gemfile
-gem "jetstream_bridge", "~> 5.0"
+gem "jetstream_bridge", "~> 7.0"
 ```
 
 ```bash
@@ -33,6 +33,64 @@ Generators create:
 - `db/migrate/*_create_jetstream_inbox_events.rb`
 - `app/controllers/jetstream_health_controller.rb` (health check)
 
+### Database Migrations
+
+The generated migrations create tables for inbox and outbox patterns:
+
+**Outbox Events** (`jetstream_bridge_outbox_events`):
+
+```ruby
+create_table :jetstream_bridge_outbox_events do |t|
+  t.string :event_id, null: false, index: { unique: true }
+  t.string :event_type, null: false
+  t.string :resource_type, null: false
+  t.string :resource_id
+  t.jsonb :payload, default: {}, null: false
+  t.string :subject, null: false
+  t.jsonb :headers, default: {}
+  t.string :status, default: "pending", null: false, index: true
+  t.text :error_message
+  t.integer :publish_attempts, default: 0
+  t.datetime :published_at, index: true
+  t.datetime :failed_at
+  t.timestamps
+end
+```
+
+**Inbox Events** (`jetstream_bridge_inbox_events`):
+
+```ruby
+create_table :jetstream_bridge_inbox_events do |t|
+  t.string :event_id, null: false, index: { unique: true }
+  t.string :event_type
+  t.string :resource_type
+  t.string :resource_id
+  t.jsonb :payload, default: {}, null: false
+  t.string :subject, null: false
+  t.jsonb :headers, default: {}
+  t.string :stream
+  t.bigint :stream_seq, index: true
+  t.integer :deliveries, default: 0
+  t.string :status, default: "received", null: false, index: true
+  t.text :error_message
+  t.integer :processing_attempts, default: 0
+  t.datetime :received_at, index: true
+  t.datetime :processed_at, index: true
+  t.datetime :failed_at
+  t.timestamps
+end
+```
+
+**Key Fields:**
+
+- `event_id`: Unique identifier for deduplication
+- `event_type`: Type of event (e.g., "user.created")
+- `resource_type`/`resource_id`: Entity being synchronized
+- `payload`: Event data (JSON)
+- `status`: Event lifecycle state (pending/processing/processed/failed)
+- `stream_seq`: NATS JetStream sequence number (inbox only)
+- `deliveries`: Delivery attempt count (inbox only)
+
 ## Configuration
 
 ```ruby
@@ -61,6 +119,19 @@ end
 
 Rails autostart runs after initialization (including in console). You can opt out for rake tasks or other tooling with `config.lazy_connect = true` or `JETSTREAM_BRIDGE_DISABLE_AUTOSTART=1`; it will then connect on first publish/subscribe.
 
+### Push consumer mode (restricted credentials)
+
+If your NATS user cannot publish to `$JS.API.*`, switch to push consumers and pre-create the durable consumer:
+
+```ruby
+config.consumer_mode = :push
+# Optional: override delivery subject (defaults to "#{config.destination_subject}.worker")
+# config.delivery_subject = "worker.sync.my_app.worker"
+config.auto_provision = false # pre-create stream/consumer with admin creds
+```
+
+Provision the consumer with NATS CLI (`--deliver <subject>`) or `bundle exec rake jetstream_bridge:provision` using admin credentials. See [docs/RESTRICTED_PERMISSIONS.md](RESTRICTED_PERMISSIONS.md) for the full least-privilege guide.
+
 ## Publish
 
 ```ruby

data/docs/PRODUCTION.md

@@ -204,7 +204,7 @@ end
     "use_inbox": true,
     "use_dlq": true
   },
-  "version": "4.0.3"
+  "version": "7.0.0"
 }
 ```