lepus 0.0.1.beta2 → 0.1.0

data/docs/middleware.md

@@ -0,0 +1,240 @@
+# Middleware
+
+Lepus middlewares form a chain around each message — both for producers (around each publish) and consumers (around each delivery). Every middleware gets the message and the next app in the chain:
+
+```ruby
+class MyMiddleware < Lepus::Middleware
+  def initialize(option: nil)
+    @option = option
+  end
+
+  def call(message, app)
+    # pre-processing
+    new_message = message.mutate(payload: transform(message.payload))
+    # pass down the chain (and let the consumer/publisher actually run)
+    result = app.call(new_message)
+    # post-processing
+    result
+  end
+
+  private
+
+  def transform(payload) = payload
+end
+```
+
+## Registering
+
+### Per consumer / per producer
+
+```ruby
+class OrdersConsumer < Lepus::Consumer
+  use :json, symbolize_keys: true        # built-in by symbol
+  use :max_retry, retries: 5
+  use MyMiddleware, option: 'value'      # your own class
+end
+```
+
+### Globally
+
+```ruby
+Lepus.configure do |config|
+  config.consumer_middlewares do |chain|
+    chain.use :json, symbolize_keys: true
+    chain.use :exception_logger
+  end
+
+  config.producer_middlewares do |chain|
+    chain.use :json
+    chain.use :correlation_id
+    chain.use :instrumentation
+  end
+end
+```
+
+Global middlewares run **before** per-class middlewares.
+
+## Built-in consumer middlewares
+
+### `:json`
+
+Parse the payload from JSON.
+
+```ruby
+use :json, symbolize_keys: true, on_error: proc { :reject }
+```
+
+Options:
+
+- `symbolize_keys: true` — hash keys become symbols.
+- `on_error:` — a callable that takes the exception; its return becomes the disposition (default: `:reject`).
+
+### `:max_retry`
+
+Track retry count via RabbitMQ `x-death` headers and route to an error queue after N attempts.
+
+```ruby
+use :max_retry, retries: 5, error_queue: 'orders.error'
+```
+
+Requires the consumer's `configure` to declare a `retry_queue:` so requeued messages loop through a delay queue.
+
+### `:exception_logger`
+
+Catch exceptions from downstream middlewares/perform, log with backtrace, then re-raise (so higher-level error handling still sees the exception).
+
+```ruby
+use :exception_logger
+```
+
+### `:honeybadger`
+
+Notify Honeybadger on exceptions. Requires the `honeybadger` gem.
+
+```ruby
+use :honeybadger
+```
+
+### `:unique`
+
+Dedupe messages by `correlation_id`. Requires a storage backend (your code wires Redis or similar).
+
+```ruby
+use :unique, store: MyRedisStore
+```
+
+## Built-in producer middlewares
+
+### `:json`
+
+Serialize a hash payload as JSON; set `content_type` to `application/json`.
+
+```ruby
+use :json
+```
+
+### `:correlation_id`
+
+Auto-generate a UUID correlation id if one isn't already set on the message.
+
+```ruby
+use :correlation_id
+```
+
+### `:header`
+
+Add headers to every publish.
+
+```ruby
+use :header, 'X-Service', 'orders-api'
+use :header, 'X-Request-Id', -> { Current.request_id }
+```
+
+The value can be a static value or a callable.
+
+### `:instrumentation`
+
+Emit `ActiveSupport::Notifications` events before and after each publish. Event name: `publish.lepus`.
+
+```ruby
+use :instrumentation
+```
+
+Subscribe:
+
+```ruby
+ActiveSupport::Notifications.subscribe('publish.lepus') do |name, start, finish, id, payload|
+  puts "published to #{payload[:exchange]} in #{(finish - start) * 1000}ms"
+end
+```
+
+### `:unique`
+
+Drop duplicate publishes. Requires a storage backend.
+
+```ruby
+use :unique, store: MyRedisStore
+```
+
+## Writing your own
+
+### Consumer middleware
+
+```ruby
+class LogLevelMiddleware < Lepus::Middleware
+  def initialize(level: :info)
+    @level = level
+  end
+
+  def call(message, app)
+    Lepus.logger.public_send(@level, "Processing: #{message.payload.inspect}")
+    app.call(message)
+  end
+end
+
+class OrdersConsumer < Lepus::Consumer
+  use LogLevelMiddleware, level: :debug
+end
+```
+
+### Producer middleware
+
+Same interface. Typical use: decorate the message with metadata before it's published.
+
+```ruby
+class TenantScopingMiddleware < Lepus::Middleware
+  def call(message, app)
+    with_tenant = message.mutate(metadata: message.metadata.merge(headers: message.metadata[:headers].merge('X-Tenant-Id' => Current.tenant_id)))
+    app.call(with_tenant)
+  end
+end
+```
+
+### Mutating a message
+
+`message.mutate(**kwargs)` returns a new message with updated fields — payload, metadata, delivery_info. Don't mutate in place; build a new one and pass it down.
+
+```ruby
+app.call(message.mutate(payload: transformed_payload))
+```
+
+## Order of execution
+
+For a consumer:
+
+```
+incoming delivery
+  → global consumer middlewares (in config order)
+  → per-consumer middlewares (in class-file order)
+  → perform(message)
+  → each middleware unwinds (post-processing)
+  → disposition returned
+```
+
+For a producer:
+
+```
+publish(payload)
+  → per-producer middlewares
+  → global producer middlewares
+  → Bunny publish
+  → unwind
+```
+
+## Conditional middleware
+
+Middleware can decide not to call the next app — effectively short-circuiting:
+
+```ruby
+class SkipDuringMaintenanceMiddleware < Lepus::Middleware
+  def call(message, app)
+    if Feature.maintenance_mode?
+      Lepus.logger.warn('skipping publish during maintenance')
+      return  # do NOT call app.call(message)
+    end
+    app.call(message)
+  end
+end
+```
+
+For consumers, returning early means the consumer's `perform` never runs. The return value of `call` becomes the disposition.

data/docs/producers.md

@@ -0,0 +1,173 @@
+# Producers
+
+A producer publishes messages to an exchange. It's a subclass of `Lepus::Producer` with a `configure` call and whatever convenience methods you want on top.
+
+## Minimal example
+
+```ruby
+class OrdersProducer < Lepus::Producer
+  configure(
+    exchange: { name: 'orders', type: :topic, durable: true },
+    publish:  { persistent: true }
+  )
+
+  use :json
+  use :correlation_id
+end
+```
+
+Publish from anywhere in your app:
+
+```ruby
+OrdersProducer.publish(
+  { order_id: 42, total: 99.99 },
+  routing_key: 'order.created'
+)
+```
+
+## `configure` DSL
+
+```ruby
+configure(
+  exchange: { name: 'orders', type: :topic, durable: true },
+  publish:  { persistent: true, mandatory: false }
+)
+```
+
+| Key | Purpose |
+|-----|---------|
+| `exchange` | String (name only) or hash (`name`, `type`, `durable`, `auto_delete`). Auto-declared on first publish. |
+| `publish` | Default publish options merged into every `publish` call: `persistent`, `mandatory`, `content_type`, `expiration`, etc. |
+
+## Publishing
+
+```ruby
+OrdersProducer.publish(payload, routing_key: 'order.created')
+OrdersProducer.publish(payload, routing_key: 'order.paid', headers: { tenant_id: 123 })
+OrdersProducer.publish(payload, routing_key: 'order.created', expiration: 60_000)
+```
+
+All Bunny publish options are supported. Frequently useful:
+
+| Option | Purpose |
+|--------|---------|
+| `routing_key` | Routing key. |
+| `persistent` | Persist the message across RabbitMQ restarts. |
+| `headers` | Custom headers. |
+| `correlation_id` | RPC pattern correlation. Auto-set by `:correlation_id` middleware. |
+| `content_type` | MIME type. Auto-set to `application/json` by `:json` middleware. |
+| `expiration` | Per-message TTL in milliseconds. |
+| `mandatory` | Return the message if it can't be routed. |
+
+## Payload types
+
+- A Hash — pair with `:json` middleware to serialize.
+- A String — sent as-is.
+- Anything else — pair with a middleware that turns it into bytes.
+
+## Middleware
+
+```ruby
+class OrdersProducer < Lepus::Producer
+  use :json
+  use :correlation_id
+  use :instrumentation
+  use :header, 'X-Service', 'orders-api'
+end
+```
+
+Built-in producer middlewares:
+
+| Name | Purpose |
+|------|---------|
+| `:json` | Serialize a hash into JSON; sets `content_type`. |
+| `:correlation_id` | Auto-generate a UUID `correlation_id` if absent. |
+| `:header` | Add a static or dynamic header: `use :header, 'X-Foo', 'bar'`. |
+| `:instrumentation` | Emit `ActiveSupport::Notifications` events around each publish. |
+| `:unique` | Reject duplicate publishes (by correlation id — needs external storage). |
+
+Write your own — see [middleware.md](middleware.md).
+
+## Convenience class methods
+
+Wrap `publish` with domain-specific signatures:
+
+```ruby
+class OrdersProducer < Lepus::Producer
+  configure(exchange: { name: 'orders', type: :topic, durable: true })
+  use :json
+  use :correlation_id
+
+  def self.order_created(order)
+    publish(
+      { order_id: order.id, total: order.total, created_at: order.created_at },
+      routing_key: 'order.created'
+    )
+  end
+
+  def self.order_shipped(order)
+    publish(
+      { order_id: order.id, shipped_at: order.shipped_at },
+      routing_key: 'order.shipped'
+    )
+  end
+end
+
+# In your code:
+OrdersProducer.order_created(order)
+```
+
+## Connection pooling
+
+Producers share a single connection pool across the process:
+
+```ruby
+Lepus.configure do |config|
+  config.producer do |p|
+    p.pool_size    = 5
+    p.pool_timeout = 5.0
+  end
+end
+```
+
+## Enable / disable publishing
+
+Useful when you want to disable side effects in tests or a specific environment.
+
+```ruby
+# Globally
+Lepus::Producers.disable!
+Lepus::Producers.enabled?(OrdersProducer)  # => false
+
+# By producer class
+Lepus::Producers.disable!(OrdersProducer)
+
+# By exchange name
+Lepus::Producers.disable!('orders')
+
+# Block-scoped
+Lepus::Producers.without_publishing do
+  OrdersProducer.publish(...)  # no-op
+end
+
+Lepus::Producers.with_publishing do
+  OrdersProducer.publish(...)  # forced through even if disabled
+end
+```
+
+## Error handling
+
+A publish can fail for many reasons — connection down, channel closed, etc. By default, Bunny retries with exponential backoff within `recovery_attempts`. For anything beyond that, you catch and handle it:
+
+```ruby
+begin
+  OrdersProducer.order_created(order)
+rescue Bunny::Exception => e
+  Rails.logger.error("publish failed: #{e.message}")
+  # queue for later retry, fall back to a DB outbox, etc.
+end
+```
+
+## Testing
+
+See [testing.md](testing.md) — there's a `Lepus::Testing.producer_messages(ProducerClass)` helper that captures publishes in-memory.

data/docs/prometheus.md

@@ -0,0 +1,112 @@
+# Prometheus metrics
+
+Lepus ships an optional integration with [`prometheus_exporter`](https://github.com/discourse/prometheus_exporter). Consumer and producer processes forward metric payloads over TCP to a collector server; Prometheus scrapes the collector's `/metrics` endpoint.
+
+## Requirements
+
+- `prometheus_exporter` in your Gemfile.
+- A collector server reachable from every process that calls `require "lepus/prometheus"`. Typically this is the Lepus supervisor process itself, listening on `localhost:9394` (so forked workers can reach it without networking changes).
+
+## Enabling instrumentation
+
+In each process that runs Lepus (supervisor + workers), load:
+
+```ruby
+require "lepus/prometheus"
+```
+
+This prepends instrumentation onto `Lepus::Consumers::Handler` and `Lepus::Consumers::Worker` and subscribes to the `publish.lepus` `ActiveSupport::Notifications` event. Requiring must happen before the supervisor forks workers so every child inherits the instrumentation.
+
+## Starting the collector server
+
+Run the collector inside the supervisor so forked workers can send metrics to `localhost:9394`:
+
+```ruby
+# config/initializers/lepus.rb
+require "lepus/web"
+
+if Rails.application.config.prometheus_enabled
+  require "lepus/prometheus"
+  require "lepus/prometheus/collector"
+end
+
+Lepus.configure do |config|
+  # …
+end
+
+if Rails.application.config.prometheus_enabled
+  Lepus::Supervisor.on_start do
+    require "prometheus_exporter/server"
+
+    collector = PrometheusExporter::Server::Collector.new
+    collector.register_collector(Lepus::Prometheus::Collector.new)
+
+    server = PrometheusExporter::Server::WebServer.new(
+      port: 9394,
+      bind: "0.0.0.0",
+      collector: collector
+    )
+    server.start
+  end
+end
+```
+
+Point Prometheus at `<lepus-host>:9394` in your scrape config.
+
+## Polling RabbitMQ queue stats
+
+Queue depth is not published by the workers — it's pulled from the RabbitMQ Management API by a poller thread. Start it from `on_start`:
+
+```ruby
+Lepus::Supervisor.on_start do
+  Lepus::Prometheus.watch_queues(interval: 30)
+end
+```
+
+The poller emits a `lepus_queue_poll_last_success_timestamp_seconds` gauge after each successful API round-trip and a `lepus_queue_poll_errors_total` counter on failure — alert on stale timestamps to catch a silently broken poller.
+
+## Metrics
+
+| Metric | Type | Labels |
+| --- | --- | --- |
+| `lepus_messages_processed_total` | counter | `consumer`, `queue`, `result` (`ack`/`reject`/`requeue`/`nack`/`error`), `error` (exception class, empty on success) |
+| `lepus_delivery_duration_seconds` | histogram | `consumer`, `queue` |
+| `lepus_messages_published_total` | counter | `exchange`, `routing_key` |
+| `lepus_publish_duration_seconds` | histogram | `exchange`, `routing_key` |
+| `lepus_process_rss_memory_bytes` | gauge | `kind`, `name` |
+| `lepus_process_info` | gauge (always `1`) | `kind`, `name`, `pid`, `hostname` |
+| `lepus_queue_messages` | gauge | `name` |
+| `lepus_queue_messages_ready` | gauge | `name` |
+| `lepus_queue_messages_unacknowledged` | gauge | `name` |
+| `lepus_queue_consumers` | gauge | `name` |
+| `lepus_queue_memory_bytes` | gauge | `name` |
+| `lepus_queue_poll_last_success_timestamp_seconds` | gauge | — |
+| `lepus_queue_poll_errors_total` | counter | `error` |
+
+`result="error"` is recorded for every delivery that raises out of the consumer, alongside the exception class — queryable as `rate(lepus_messages_processed_total{result="error"}[5m])`.
+
+## Configuration
+
+```ruby
+Lepus.configure do |config|
+  # Histogram buckets (in seconds) used for delivery and publish latency.
+  config.prometheus_buckets = [0.01, 0.05, 0.1, 0.5, 1, 5]
+end
+```
+
+To send metrics to a collector on a different host:
+
+```ruby
+require "prometheus_exporter/client"
+Lepus::Prometheus.client = PrometheusExporter::Client.new(host: "collector.internal", port: 9394)
+```
+
+## Standalone collector mode
+
+You can also run `prometheus_exporter` as a separate process and load just the collector:
+
+```sh
+prometheus_exporter -a lepus/prometheus/collector
+```
+
+`lib/lepus/prometheus/collector.rb` deliberately does not require the rest of the gem, so this works without a full Lepus boot.

data/docs/rails.md

@@ -0,0 +1,161 @@
+# Rails Integration
+
+When Rails is loaded, Lepus's Railtie wires up sensible defaults. No initializer is strictly required.
+
+## What the Railtie does
+
+- Sets `config.logger = Rails.logger` (unless you've already set one).
+- Sets `config.app_executor = Rails.application.executor` — every consumer `perform` runs inside the Rails executor, which means:
+  - Autoloading works (Zeitwerk is active).
+  - Query cache is cleared after each message.
+  - Connection cleanup runs after each message.
+  - Reloading works in development.
+- Subscribes the log subscriber for friendly production log lines.
+- Integrates with `Rails.error` — unhandled exceptions are reported there.
+
+## Overriding
+
+```ruby
+# config/initializers/lepus.rb
+Lepus.configure do |config|
+  config.rabbitmq_url    = ENV.fetch('RABBITMQ_URL')
+  config.connection_name = 'my-service'
+
+  # Override the defaults:
+  config.logger       = MyLogger.new
+  config.app_executor = nil   # disable executor wrapping entirely
+
+  config.consumers_directory = 'app/consumers'
+
+  config.worker(:default) do |w|
+    w.pool_size = 5
+    w.before_fork { ActiveRecord::Base.connection_handler.clear_all_connections! }
+    w.after_fork  { ActiveRecord::Base.establish_connection }
+  end
+end
+```
+
+## Defining consumers and producers
+
+Put them in `app/consumers/` and `app/producers/`. With `config.consumers_directory = 'app/consumers'`, `lepus start` with no arguments auto-loads all of them.
+
+```ruby
+# app/consumers/orders_consumer.rb
+class OrdersConsumer < Lepus::Consumer
+  configure(
+    queue:    'orders',
+    exchange: { name: 'orders', type: :topic, durable: true },
+    routing_key: 'order.*'
+  )
+  use :json, symbolize_keys: true
+
+  def perform(message)
+    Order.create!(message.payload)
+    :ack
+  end
+end
+```
+
+```ruby
+# app/producers/orders_producer.rb
+class OrdersProducer < Lepus::Producer
+  configure(exchange: { name: 'orders', type: :topic, durable: true })
+  use :json
+  use :correlation_id
+end
+```
+
+## Running alongside a Rails web app
+
+```bash
+# Terminal 1 — web
+bin/rails server
+
+# Terminal 2 — consumers
+bundle exec lepus start --require_file config/environment.rb
+```
+
+Or use Foreman:
+
+```
+# Procfile
+web:    bin/rails server
+worker: bundle exec lepus start --require_file config/environment.rb
+```
+
+## The Puma plugin
+
+For apps that want consumers running inside the same Puma process (development convenience, or small-scale production where you don't want another service):
+
+```ruby
+# config/puma.rb
+plugin :lepus
+```
+
+The plugin forks consumer workers as Puma's cluster workers would, tying their lifecycle to Puma's.
+
+**Caution:** running consumers inside Puma means process restarts on deploy take longer (waiting for in-flight messages) and you can't scale consumers independently of web requests. For non-trivial workloads, run them as a separate service.
+
+## Mounting the web dashboard
+
+```ruby
+# config/routes.rb
+require 'lepus/web'
+
+authenticate :user, ->(u) { u.admin? } do
+  mount Lepus::Web::App, at: '/lepus'
+end
+```
+
+See [web.md](web.md).
+
+## Testing
+
+```ruby
+# spec/rails_helper.rb (or spec/spec_helper.rb)
+require 'lepus/testing'
+
+RSpec.configure do |config|
+  config.before(:each) { Lepus::Testing.enable!; Lepus::Testing.reset! }
+end
+```
+
+See [testing.md](testing.md).
+
+## Active Record gotchas
+
+Since `app_executor` is set to `Rails.application.executor`, Active Record connection management is handled per message — no manual `clear_active_connections!` needed.
+
+For worker subprocesses, the `before_fork` / `after_fork` hooks close and reopen connections cleanly:
+
+```ruby
+Lepus.configure do |config|
+  config.worker(:default) do |w|
+    w.before_fork { ActiveRecord::Base.connection_handler.clear_all_connections! }
+    w.after_fork  { ActiveRecord::Base.establish_connection }
+  end
+end
+```
+
+Without these hooks, all children inherit the same database socket and things go badly fast.
+
+## Exception reporting
+
+Lepus's Rails integration reports unhandled exceptions via `Rails.error`:
+
+```ruby
+# config/application.rb or an initializer
+Rails.error.subscribe do |exception, handled:, severity:, context:, source:|
+  Honeybadger.notify(exception, context: context) if source == 'lepus'
+end
+```
+
+Or use the `:honeybadger` middleware directly on your consumers — see [middleware.md](middleware.md).
+
+## Zeitwerk
+
+Consumer and producer classes are autoloaded by Zeitwerk like any other Rails class. `app/consumers/orders_consumer.rb` → `OrdersConsumer`, namespaced with the usual folder-to-module rules.
+
+## Generators
+
+None at the moment. Create consumer and producer files by hand (or your editor's snippet of choice).