jetstream_bridge 2.10.0 → 3.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 15fe547a615edbbd3bbeafb5bf5193bfdc4f1bd9911d7497803cf5d331ced36b
-  data.tar.gz: 5c0b98f2cbbff19762e98abcb39fa7c5f006cbfcd951295020adbcc08da80d49
+  metadata.gz: d47bc85ddad1450a71960670d94969ec5f6416dc98c07c8409e4d7243ec92f3e
+  data.tar.gz: b9130b48f7129d6c61c9f2cddc61a2e292c906c282925a3a550f79c24cc7eff4
 SHA512:
-  metadata.gz: 4135f2c9287c5ddc0716f9eb7f3bcfeab104072c0ed59bdf41ff7a3df07dd5254240147e6a176d80f3a93198240ee633ff24ed742caf6142c859d6f259110c56
-  data.tar.gz: d341432852e94c1983128f5d0b0094e812c867f7133c609f609fb4606a4945ee2e5129cc20a24ef166e9e36dbd73a06e6bcb3b3a6d8c19752906a26756883345
+  metadata.gz: 430fa83af6f8b1d9eb22fdf8edfbd2787c4a0609c6abf6626239a05132f1732bf9ecd8f0e6ad92cab7e53ffd234d0c3df5e626f06c332680a99b84849bdf7bc0
+  data.tar.gz: b17ccb8f7bf525aa5f21917c87932115a11ccb4058d79c81af741ee985203c97d326b3c7cd8960819c4a93f5d088a8d0b39e1f9bbef108e2619e7114e621f74a

data/CHANGELOG.md

@@ -0,0 +1,164 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [3.0.0] - 2025-11-23
+
+### Added
+
+#### Production-Ready Features
+
+- **Health checks** - Comprehensive health check API via `JetstreamBridge.health_check`
+  - NATS connection status monitoring
+  - Stream existence and subject verification
+  - Configuration validation
+  - Returns structured health data for monitoring systems
+- **Health check generator** - Rails generator for creating health check endpoints
+  - `rails g jetstream_bridge:health_check` creates controller and route
+  - Automatic route injection into `config/routes.rb`
+  - Returns appropriate HTTP status codes (200/503)
+- **Auto-reconnection** - Automatic recovery from connection failures
+  - Exponential backoff retry strategy
+  - Configurable retry limits and delays
+  - Connection state tracking with timestamps
+- **Connection factory** - Centralized connection management
+  - Singleton pattern for connection handling
+  - Thread-safe connection access
+  - Public `connected?` and `connected_at` accessors
+
+#### Error Handling
+
+- **Comprehensive error hierarchy** - Well-organized exception classes
+  - `ConfigurationError` - Base for configuration issues
+  - `ConnectionError` - Base for connection problems
+  - `PublishError` - Base for publishing failures
+  - `ConsumerError` - Base for consumption issues
+  - `TopologyError` - Base for stream/subject topology errors
+  - `DlqError` - Base for dead-letter queue operations
+  - All inherit from `JetstreamBridge::Error`
+
+#### Developer Experience
+
+- **Debug helper** - Comprehensive debugging utility via `JetstreamBridge::DebugHelper`
+  - Configuration dump
+  - Connection status
+  - Stream information
+  - Subject validation
+  - Model availability checks
+- **Rake tasks** - CLI tools for operations
+  - `rake jetstream_bridge:health` - Check health and connection status
+  - `rake jetstream_bridge:validate` - Validate configuration
+  - `rake jetstream_bridge:test_connection` - Test NATS connection
+  - `rake jetstream_bridge:debug` - Show comprehensive debug information
+- **Value objects** - Type-safe domain models
+  - `EventEnvelope` - Structured event representation
+  - `Subject` - Subject pattern validation and parsing
+
+#### Testing & Quality
+
+- **Comprehensive test suite** - 248 RSpec tests covering all core functionality
+  - Configuration validation specs
+  - Connection factory specs with edge cases (48 new tests added)
+    - Server URL normalization (single, comma-separated, arrays, whitespace)
+    - Authentication options (user, pass, token)
+    - NATS URL validation (nil, empty, whitespace-only)
+    - JetStream context creation and nc accessor
+  - Event envelope specs with full coverage (18 new tests added)
+    - Initialization edge cases (trace ID, Time objects, resource ID extraction)
+    - Hash serialization and deserialization
+    - Deep immutability validation
+    - Equality and hashing behavior
+    - Error handling for invalid timestamps
+  - Retry strategy specs
+  - Model specs
+  - Error hierarchy specs
+- **Subject validation** - Prevents NATS wildcards in configuration
+  - Validates `env`, `app_name`, and `destination_app` don't contain `.`, `*`, or `>`
+  - Clear error messages for invalid subjects
+- **Bug fixes in implementation** - Fixed `EventEnvelope.from_h` to properly handle deserialization
+  - Removed invalid `schema_version` parameter
+  - Added missing `resource_id` parameter
+
+### Changed
+
+#### Architecture Improvements
+
+- **Consolidated configuration** - Streamlined `Config` class
+  - Removed redundant configuration classes
+  - Centralized validation logic
+  - Better default values
+- **Enhanced generator templates** - Improved initializer template
+  - Better documentation and comments
+  - Organized sections (Connection, Consumer, Reliability, Models, Logging)
+  - Clear indication of required vs optional settings
+- **Simplified consumer initialization** - Cleaner API
+  - Sensible defaults for `durable_name` and `batch_size`
+  - Removed unnecessary configuration classes
+- **Better logging** - Configurable logger with sensible defaults
+  - Respects `config.logger` setting
+  - Falls back to `Rails.logger` in Rails apps
+  - Uses `STDOUT` logger otherwise
+
+#### Bug Fixes
+
+- **Fixed duration parsing** - Correct integer conversion for duration strings
+  - Handles edge cases properly
+  - Added comprehensive tests
+- **Fixed subject format** - Corrected DLQ subject pattern
+  - Changed from `{env}.data.sync.dlq` to `{env}.sync.dlq`
+  - Updated all documentation
+- **Repository improvements**
+  - Transaction safety for all database operations
+  - Pessimistic locking for outbox operations
+  - Race condition protection
+  - Better error handling and logging
+
+### Refactored
+
+- **Consumer architecture** - Consolidated consumer support classes
+  - Merged related functionality
+  - Reduced file count
+  - Clearer separation of concerns
+- **Model handling** - Streamlined inbox/outbox models
+  - Graceful column detection without connection boot
+  - Better validation guards
+  - Simplified attribute handling
+- **JSON handling** - Switched to Oj for performance
+  - Faster JSON parsing/serialization
+  - Consistent JSON handling across the gem
+- **Topology management** - Enhanced stream and subject handling
+  - Improved overlap detection
+  - Better error messages
+  - More robust subject matching
+
+### Documentation
+
+- **Updated README** - Comprehensive documentation updates
+  - Added section for Rails generators and rake tasks
+  - Documented all new health check features
+  - Fixed incorrect subject patterns
+  - Added operations guide
+  - Improved getting started section
+- **Enhanced code comments** - Better inline documentation
+  - YARD-style documentation for public APIs
+  - Clear explanations of complex logic
+  - Usage examples in comments
+- **Generator improvements** - Better user guidance
+  - Clear success messages
+  - Usage instructions after generation
+  - Example configurations
+
+### Internal
+
+- **Removed deprecated code**
+  - Cleaned up unused classes and methods
+  - Removed `BackoffStrategy` (consolidated into retry strategy)
+  - Removed `ConsumerConfig` (merged into main config)
+  - Removed `MessageContext` (functionality integrated)
+
+## [2.10.0] and earlier
+
+See git history for changes in earlier versions.

data/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Mike Attara
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,379 @@
+<p align="center">
+  <img src="logo.svg" alt="JetStream Bridge Logo" width="200"/>
+</p>
+
+<h1 align="center">JetStream Bridge</h1>
+
+<p align="center">
+  <strong>Production-safe realtime data bridge</strong> between systems using <strong>NATS JetStream</strong>
+</p>
+
+<p align="center">
+  Includes durable consumers, backpressure, retries, <strong>DLQ</strong>, optional <strong>Inbox/Outbox</strong>, and <strong>overlap-safe stream provisioning</strong>
+</p>
+
+<p align="center">
+  <a href="#-features">Features</a> •
+  <a href="#-install">Install</a> •
+  <a href="#-getting-started">Getting Started</a> •
+  <a href="#-operations-guide">Operations</a>
+</p>
+
+---
+
+## ✨ Features
+
+### Core Capabilities
+
+* 🔌 Simple **Publisher** and **Consumer** interfaces
+* 🛡 **Outbox** (reliable send) & **Inbox** (idempotent receive), opt-in
+* 🧨 **DLQ** for poison messages
+* ⚙️ Durable `pull_subscribe` with backoff & `max_deliver`
+* 🎯 Clear **source/destination** subject conventions
+* 🧱 **Overlap-safe stream ensure** (prevents "subjects overlap" BadRequest)
+* 🚂 **Rails generators** for initializer & migrations, plus an install **rake task**
+* ⚡️ **Eager-loaded models** via Railtie (production)
+* 📊 Configurable logging with sensible defaults
+
+### Production-Ready Features (v2.10+)
+
+* 🏥 **Health checks** - Monitor NATS connection and stream status
+* 🔄 **Auto-reconnection** - Automatic recovery from connection failures
+* 🔒 **Race condition protection** - Pessimistic locking for outbox operations
+* 🛡️ **Transaction safety** - All database operations wrapped in transactions
+* 🎯 **Subject validation** - Prevents NATS wildcards in configuration
+* 🚦 **Graceful shutdown** - Signal handlers and message draining
+* 📈 **Retry strategies** - Pluggable exponential/linear backoff algorithms
+* 🎨 **Value objects** - Type-safe domain models for events and subjects
+* 🏗️ **SOLID architecture** - Clean separation of concerns and dependency injection
+
+---
+
+## 📦 Install
+
+```ruby
+# Gemfile
+gem "jetstream_bridge", "~> 2.10"
+```
+
+```bash
+bundle install
+```
+
+---
+
+## 🧰 Rails Generators & Rake Tasks
+
+### Installation
+
+From your Rails app:
+
+```bash
+# Create initializer + migrations
+bin/rails g jetstream_bridge:install
+
+# Or run them separately:
+bin/rails g jetstream_bridge:initializer
+bin/rails g jetstream_bridge:migrations
+
+# Create health check endpoint
+bin/rails g jetstream_bridge:health_check
+```
+
+Then:
+
+```bash
+bin/rails db:migrate
+```
+
+> The generators create:
+>
+> * `config/initializers/jetstream_bridge.rb`
+> * `db/migrate/*_create_jetstream_outbox_events.rb`
+> * `db/migrate/*_create_jetstream_inbox_events.rb`
+> * `app/controllers/jetstream_health_controller.rb` (if health_check generator used)
+
+### Rake Tasks
+
+```bash
+# Check health and connection status
+bin/rake jetstream_bridge:health
+
+# Validate configuration
+bin/rake jetstream_bridge:validate
+
+# Test NATS connection
+bin/rake jetstream_bridge:test_connection
+
+# Show comprehensive debug information
+bin/rake jetstream_bridge:debug
+```
+
+---
+
+## 🔧 Configure (Rails)
+
+```ruby
+# config/initializers/jetstream_bridge.rb
+JetstreamBridge.configure do |config|
+  # NATS connection
+  config.nats_urls       = ENV.fetch("NATS_URLS", "nats://localhost:4222")
+  config.env             = ENV.fetch("NATS_ENV",  "development")
+  config.app_name        = ENV.fetch("APP_NAME",  "app")
+  config.destination_app = ENV["DESTINATION_APP"] # required
+
+  # Consumer tuning
+  config.max_deliver = 5
+  config.ack_wait    = "30s"
+  config.backoff     = %w[1s 5s 15s 30s 60s]
+
+  # Reliability features (opt-in)
+  config.use_outbox = true
+  config.use_inbox  = true
+  config.use_dlq    = true
+
+  # Models (override if you use custom AR classes/table names)
+  config.outbox_model = "JetstreamBridge::OutboxEvent"
+  config.inbox_model  = "JetstreamBridge::InboxEvent"
+
+  # Logging
+  # config.logger = Rails.logger
+end
+```
+
+> **Defaults:**
+>
+> * `stream_name` → `#{env}-jetstream-bridge-stream`
+> * `dlq_subject` → `#{env}.sync.dlq`
+
+### Logging
+
+JetstreamBridge logs through `config.logger` when set, falling back to `Rails.logger` or STDOUT. Provide any `Logger`-compatible instance in the initializer to integrate with your application's logging setup.
+
+---
+
+## 📡 Subject Conventions
+
+| Direction     | Subject Pattern           |
+|---------------|---------------------------|
+| **Publish**   | `{env}.{app}.sync.{dest}` |
+| **Subscribe** | `{env}.{dest}.sync.{app}` |
+| **DLQ**       | `{env}.sync.dlq`          |
+
+* `{app}`: `app_name`
+* `{dest}`: `destination_app`
+* `{env}`: `env`
+
+---
+
+## 🧱 Stream Topology (auto-ensure and overlap-safe)
+
+On first connection, Jetstream Bridge **ensures** a single stream exists for your `env` and that it covers:
+
+* `source_subject` (`{env}.{app}.sync.{dest}`)
+* `destination_subject` (`{env}.{dest}.sync.{app}`)
+* `dlq_subject` (if enabled)
+
+It’s **overlap-safe**:
+
+* Skips adding subjects already covered by existing wildcards
+* Pre-filters subjects owned by other streams to avoid `BadRequest: subjects overlap with an existing stream`
+* Retries once on concurrent races, then logs and continues safely
+
+---
+
+## 🗃 Database Setup (Inbox / Outbox)
+
+Inbox/Outbox are **optional**. The library detects columns at runtime and only sets what exists, so you can start minimal and evolve later.
+
+### Generator-created tables (recommended)
+
+```ruby
+# jetstream_outbox_events
+create_table :jetstream_outbox_events do |t|
+  t.string  :event_id, null: false
+  t.string  :subject,  null: false
+  t.jsonb   :payload,  null: false, default: {}
+  t.jsonb   :headers,  null: false, default: {}
+  t.string  :status,   null: false, default: "pending" # pending|publishing|sent|failed
+  t.integer :attempts, null: false, default: 0
+  t.text    :last_error
+  t.datetime :enqueued_at
+  t.datetime :sent_at
+  t.timestamps
+end
+add_index :jetstream_outbox_events, :event_id, unique: true
+add_index :jetstream_outbox_events, :status
+
+# jetstream_inbox_events
+create_table :jetstream_inbox_events do |t|
+  t.string   :event_id                              # preferred dedupe key
+  t.string   :subject,     null: false
+  t.jsonb    :payload,     null: false, default: {}
+  t.jsonb    :headers,     null: false, default: {}
+  t.string   :stream
+  t.bigint   :stream_seq
+  t.integer  :deliveries
+  t.string   :status,      null: false, default: "received" # received|processing|processed|failed
+  t.text     :last_error
+  t.datetime :received_at
+  t.datetime :processed_at
+  t.timestamps
+end
+add_index :jetstream_inbox_events, :event_id, unique: true, where: 'event_id IS NOT NULL'
+add_index :jetstream_inbox_events, [:stream, :stream_seq], unique: true, where: 'stream IS NOT NULL AND stream_seq IS NOT NULL'
+add_index :jetstream_inbox_events, :status
+```
+
+> Already have different table names? Point the config to your AR classes via `config.outbox_model` / `config.inbox_model`.
+
+---
+
+## 📤 Publish Events
+
+```ruby
+publisher = JetstreamBridge::Publisher.new
+publisher.publish(
+  resource_type: "user",
+  event_type:    "created",
+  payload:       { id: "01H...", name: "Ada" },  # resource_id inferred from payload[:id] / payload["id"]
+  # optional:
+  # event_id: "uuid-or-ulid",
+  # trace_id: "hex",
+  # occurred_at: Time.now.utc
+)
+```
+
+If **Outbox** is enabled, the publish call:
+
+* Upserts an outbox row by `event_id`
+* Publishes with `nats-msg-id` (idempotent)
+* Marks status `sent` or records `failed` with `last_error`
+
+---
+
+## 📥 Consume Events
+
+```ruby
+JetstreamBridge::Consumer.new do |event, subject, deliveries|
+  # Your idempotent domain logic here
+  # `event` is the parsed envelope hash
+  UserCreatedHandler.call(event["payload"])
+end.run!
+```
+
+`durable_name` and `batch_size` default to the configured values and can be
+overridden if needed:
+
+```ruby
+JetstreamBridge::Consumer.new(durable_name: 'my-durable', batch_size: 10) do |event, subject, deliveries|
+  # ...
+end.run!
+```
+
+If **Inbox** is enabled, the consumer:
+
+* Dedupes by `event_id` (falls back to stream sequence if needed)
+* Records processing state, errors, and timestamps
+* Skips already-processed messages (acks immediately)
+
+---
+
+## 📬 Envelope Format
+
+```json
+{
+  "event_id":       "01H1234567890ABCDEF",
+  "schema_version": 1,
+  "event_type":     "created",
+  "producer":       "myapp",
+  "resource_type":  "user",
+  "resource_id":    "01H1234567890ABCDEF",
+  "occurred_at":    "2025-08-13T21:00:00Z",
+  "trace_id":       "abc123",
+  "payload":        { "id": "01H...", "name": "Ada" }
+}
+```
+
+* `resource_id` is inferred from `payload.id` when publishing.
+
+---
+
+## 🧨 Dead-Letter Queue (DLQ)
+
+When enabled, the topology ensures the DLQ subject exists:
+**`{env}.sync.dlq`**
+
+You may run a separate process to subscribe and triage messages that exceed `max_deliver` or are NAK'ed to the DLQ.
+
+---
+
+## 🛠 Operations Guide
+
+### Monitoring
+
+* **Consumer lag**: `nats consumer info <stream> <durable>`
+* **DLQ volume**: subscribe/metrics on `{env}.sync.dlq`
+* **Outbox backlog**: alert on `jetstream_outbox_events` with `status != 'sent'` and growing count
+
+### Scaling
+
+* Run consumers in **separate processes/containers**
+* Scale consumers independently of web
+* Tune `batch_size`, `ack_wait`, `max_deliver`, and `backoff`
+
+### Health Checks
+
+The gem provides built-in health check functionality for monitoring:
+
+```ruby
+# Get comprehensive health status
+health = JetstreamBridge.health_check
+# => {
+#   healthy: true,
+#   nats_connected: true,
+#   connected_at: "2025-11-22T20:00:00Z",
+#   stream: { exists: true, name: "...", ... },
+#   config: { env: "production", ... },
+#   version: "2.10.0"
+# }
+
+# Force-connect & ensure topology at boot or in a check
+JetstreamBridge.ensure_topology!
+
+# Debug helper for troubleshooting
+JetstreamBridge::DebugHelper.debug_info
+```
+
+### When to Use
+
+* **Inbox**: you need idempotent processing and replay safety
+* **Outbox**: you want “DB commit ⇒ event published (or recorded for retry)” guarantees
+
+---
+
+## 🧩 Troubleshooting
+
+* **`subjects overlap with an existing stream`**
+  The library pre-filters overlapping subjects and retries once. If another team owns a broad wildcard (e.g., `env.data.sync.>`), coordinate subject boundaries.
+
+* **Consumer exists with mismatched filter**
+  The library detects and recreates the durable with the desired filter subject.
+
+* **Repeated redeliveries**
+  Increase `ack_wait`, review handler acks/NACKs, or move poison messages to DLQ.
+
+---
+
+## 🚀 Getting Started
+
+1. Add the gem & run `bundle install`
+2. `bin/rails g jetstream_bridge:install`
+3. `bin/rails db:migrate`
+4. Start publishing/consuming!
+
+---
+
+## 📄 License
+
+[MIT License](LICENSE)

data/lib/generators/jetstream_bridge/health_check/health_check_generator.rb

@@ -0,0 +1,65 @@
+# frozen_string_literal: true
+
+require 'rails/generators'
+
+module JetstreamBridge
+  module Generators
+    class HealthCheckGenerator < Rails::Generators::Base
+      source_root File.expand_path('templates', __dir__)
+      desc 'Creates a health check endpoint for JetStream Bridge monitoring'
+
+      def create_controller
+        template 'health_controller.rb', 'app/controllers/jetstream_health_controller.rb'
+      end
+
+      def add_route
+        route_content = "  # JetStream Bridge health check endpoint\n" \
+                       "  get '/health/jetstream', to: 'jetstream_health#show'"
+
+        if File.exist?('config/routes.rb')
+          inject_into_file 'config/routes.rb', after: /Rails\.application\.routes\.draw do\n/ do
+            "#{route_content}\n"
+          end
+          say 'Added health check route to config/routes.rb', :green
+        else
+          say 'Could not find config/routes.rb - please add route manually:', :yellow
+          say route_content, :yellow
+        end
+      end
+
+      def show_usage
+        say "\n" + '=' * 70, :green
+        say 'Health Check Endpoint Created!', :green
+        say '=' * 70, :green
+        say "\nThe health check endpoint is now available at:"
+        say "  GET /health/jetstream", :cyan
+        say "\nExample response:"
+        say <<~EXAMPLE, :white
+          {
+            "healthy": true,
+            "nats_connected": true,
+            "connected_at": "2025-11-22T20:00:00Z",
+            "stream": {
+              "exists": true,
+              "name": "development-jetstream-bridge-stream",
+              "subjects": ["dev.app1.sync.app2"],
+              "messages": 42
+            },
+            "config": {
+              "env": "development",
+              "app_name": "my_app",
+              "destination_app": "other_app"
+            },
+            "version": "2.10.0"
+          }
+        EXAMPLE
+        say "\nUse this endpoint for:"
+        say "  • Kubernetes liveness/readiness probes", :white
+        say "  • Docker health checks", :white
+        say "  • Monitoring and alerting", :white
+        say "  • Load balancer health checks", :white
+        say '=' * 70, :green
+      end
+    end
+  end
+end

data/lib/generators/jetstream_bridge/health_check/templates/health_controller.rb

@@ -0,0 +1,38 @@
+# frozen_string_literal: true
+
+# Health check controller for JetStream Bridge monitoring
+#
+# This controller provides a health check endpoint for monitoring the JetStream
+# connection status. Use it for Kubernetes liveness/readiness probes, Docker
+# health checks, or load balancer health checks.
+#
+# Example usage:
+#   GET /health/jetstream
+#
+# Returns:
+#   200 OK - when healthy
+#   503 Service Unavailable - when unhealthy
+class JetstreamHealthController < ActionController::API
+  # GET /health/jetstream
+  #
+  # Returns comprehensive health status including:
+  # - NATS connection status
+  # - JetStream stream information
+  # - Configuration details
+  # - Gem version
+  def show
+    health = JetstreamBridge.health_check
+
+    if health[:healthy]
+      render json: health, status: :ok
+    else
+      render json: health, status: :service_unavailable
+    end
+  rescue StandardError => e
+    # Ensure we always return a valid JSON response
+    render json: {
+      healthy: false,
+      error: "#{e.class}: #{e.message}"
+    }, status: :service_unavailable
+  end
+end