lepus 0.0.1.rc2 → 0.1.0

data/docs/consumers.md

@@ -0,0 +1,168 @@
+# Consumers
+
+Consumers subscribe to a queue, receive messages, and process them. A consumer is a subclass of `Lepus::Consumer` with a `configure` call and a `perform` method.
+
+## Minimal example
+
+```ruby
+class OrdersConsumer < Lepus::Consumer
+  configure(
+    queue:    'orders',
+    exchange: { name: 'orders', type: :topic, durable: true },
+    routing_key: ['order.*']
+  )
+
+  use :json, symbolize_keys: true
+
+  def perform(message)
+    Order.create!(message.payload)
+    :ack
+  end
+end
+```
+
+## `configure` DSL
+
+```ruby
+configure(
+  queue:       'orders',                                    # string or hash
+  exchange:    { name: 'orders', type: :topic, durable: true },
+  routing_key: ['order.*', 'invoice.created'],              # array or string
+  prefetch:    1,                                            # channel QoS
+  retry_queue: { delay: 5_000 },                            # or true, or false
+  error_queue: true,                                         # or queue name, or false
+  process:     { name: :default, threads: 5 },              # worker assignment
+  channel:     { pool_size: 1, shutdown_timeout: 60, abort_on_exception: false }
+)
+```
+
+| Key | Purpose |
+|-----|---------|
+| `queue` | The queue name — string, or a hash for advanced options (`name`, `durable`, `arguments`, …). |
+| `exchange` | Bind to this exchange. String, or hash (`name`, `type`, `durable`, `auto_delete`). |
+| `routing_key` | Binding routing key(s). |
+| `prefetch` | Channel QoS (`basic.qos`). Leave at `1` unless you know you need more. |
+| `retry_queue` | When truthy, sets up a retry queue with a delay exchange. Messages requeued via `:requeue` or `max_retry` go here. |
+| `error_queue` | Queue name (string) or `true` to auto-name. Messages that exhaust retries land here. |
+| `process.name` | The named worker pool this consumer runs in — see `Lepus.configure.worker(:name)`. |
+| `process.threads` | Threads inside the consumer's worker dedicated to this consumer. |
+
+## The `perform` method
+
+```ruby
+def perform(message)
+  # ...
+end
+```
+
+`message` is a `Lepus::Message` with:
+
+- `#payload` — the decoded body (raw bytes, or parsed data if a decoding middleware is in the chain).
+- `#delivery_info` — `#exchange`, `#routing_key`, `#redelivered?`, `#delivery_tag`, `#consumer_tag`.
+- `#metadata` — `#headers`, `#content_type`, `#content_encoding`, `#correlation_id`, `#message_id`, `#reply_to`, `#type`, `#timestamp`, `#user_id`, `#app_id`, `#priority`.
+
+Return a disposition symbol or call one of the helpers:
+
+| Return | Helper | Effect |
+|--------|--------|--------|
+| `:ack` | `ack!` | Acknowledge the message. |
+| `:reject` | `reject!` | Reject without requeue (drops, or goes to DLX if configured). |
+| `:requeue` | `requeue!` | Reject with requeue — back of the queue. |
+| `:nack` | `nack!` | Negative-ack. |
+
+A `perform` that returns nothing implicitly acks. A `perform` that raises is logged, `on_thread_error` fires, and the message is rejected.
+
+## Middleware
+
+Add middlewares to a consumer with `use`:
+
+```ruby
+class OrdersConsumer < Lepus::Consumer
+  use :json, symbolize_keys: true
+  use :max_retry, retries: 5, error_queue: 'orders.error'
+  use :exception_logger
+  use :honeybadger
+end
+```
+
+Built-in middlewares:
+
+| Name | Purpose |
+|------|---------|
+| `:json` | Parse JSON body into a hash. |
+| `:max_retry` | Track `x-death` headers; after N retries, route to error queue. |
+| `:exception_logger` | Log unhandled exceptions with backtrace. |
+| `:honeybadger` | Notify Honeybadger on exceptions. |
+| `:unique` | Idempotent dedupe by `correlation_id` (requires storage). |
+
+Write your own — see [middleware.md](middleware.md).
+
+## Error handling
+
+A `perform` that raises:
+
+1. The message is rejected by default.
+2. `config.on_thread_error` is called with the exception.
+3. If `:exception_logger` middleware is in the chain, it logs with backtrace.
+4. If `:max_retry` is in the chain, it tracks retry count via RabbitMQ's `x-death` header and redirects to an error queue after N rejections.
+
+## Retries
+
+The built-in retry pattern uses RabbitMQ's dead-letter machinery:
+
+```ruby
+configure(
+  queue:       'orders',
+  exchange:    { name: 'orders', type: :topic, durable: true },
+  routing_key: 'order.*',
+  retry_queue: { delay: 5_000 },  # 5-second delay before retry
+  error_queue: 'orders.error'
+)
+use :max_retry, retries: 5, error_queue: 'orders.error'
+```
+
+On rejection:
+
+1. `max_retry` inspects `x-death` headers.
+2. If retries are below the limit, it re-publishes to the retry queue.
+3. After TTL, the retry queue dead-letters back to the original queue for another attempt.
+4. After N failures, the middleware routes to the error queue.
+
+## Worker assignment
+
+Assign a consumer to a named worker pool:
+
+```ruby
+configure(
+  queue:    'orders',
+  exchange: { name: 'orders', type: :topic },
+  process:  { name: :high_priority, threads: 10 }
+)
+```
+
+Then in the global config:
+
+```ruby
+Lepus.configure do |config|
+  config.worker(:high_priority) do |w|
+    w.pool_size = 20
+  end
+end
+```
+
+Consumers assigned to the same worker share a single subprocess.
+
+## Logging
+
+Inside `perform`, `logger` is the Lepus logger (tagged with the consumer class name):
+
+```ruby
+def perform(message)
+  logger.info("processing #{message.delivery_info.routing_key}")
+  # ...
+end
+```
+
+## Testing
+
+See [testing.md](testing.md) for the test-mode helpers.

data/docs/getting-started.md

@@ -0,0 +1,136 @@
+# Getting Started
+
+## Install
+
+```ruby
+# Gemfile
+gem 'lepus'
+```
+
+```bash
+bundle install
+```
+
+You'll also need a running RabbitMQ instance. For local dev:
+
+```bash
+docker run -d --rm -p 5672:5672 -p 15672:15672 rabbitmq:3-management
+```
+
+## Configure
+
+```ruby
+# config/initializers/lepus.rb (Rails) or your bootstrap file
+Lepus.configure do |config|
+  config.rabbitmq_url    = ENV.fetch('RABBITMQ_URL', 'amqp://guest:guest@localhost:5672')
+  config.connection_name = 'my-service'
+end
+```
+
+See [configuration.md](configuration.md) for the full DSL.
+
+## Define a consumer
+
+```ruby
+# app/consumers/orders_consumer.rb
+class OrdersConsumer < Lepus::Consumer
+  configure(
+    queue:       'orders',
+    exchange:    { name: 'orders', type: :topic, durable: true },
+    routing_key: ['order.*']
+  )
+
+  use :json, symbolize_keys: true
+
+  def perform(message)
+    # message.payload       => the decoded JSON body (a hash, due to the :json middleware)
+    # message.delivery_info => exchange, routing_key, redelivered?
+    # message.metadata      => headers, content_type, correlation_id, etc.
+    Order.create!(message.payload)
+    :ack
+  end
+end
+```
+
+The `perform` method returns a symbol indicating disposition:
+
+- `:ack` — acknowledge; RabbitMQ removes the message.
+- `:reject` — reject; message is dropped (or routed to a dead-letter exchange if configured).
+- `:requeue` — reject and requeue for another delivery attempt.
+- `:nack` — negative-ack without requeue (similar to `:reject`).
+
+`ack!`, `reject!`, `requeue!`, `nack!` are also available as helper methods.
+
+## Define a producer
+
+```ruby
+# app/producers/orders_producer.rb
+class OrdersProducer < Lepus::Producer
+  configure(
+    exchange: { name: 'orders', type: :topic, durable: true },
+    publish:  { persistent: true }
+  )
+
+  use :json
+  use :correlation_id
+end
+```
+
+Publish:
+
+```ruby
+OrdersProducer.publish(
+  { order_id: 42, total: 99.99 },
+  routing_key: 'order.created'
+)
+```
+
+## Run
+
+### One-off (development)
+
+```bash
+bundle exec lepus start OrdersConsumer
+```
+
+Flags worth knowing:
+
+```bash
+bundle exec lepus start OrdersConsumer PaymentsConsumer \
+  --require_file config/environment.rb \
+  --debug
+```
+
+See [cli.md](cli.md).
+
+### As a long-running service
+
+In production, run `lepus start` with all your consumer classes (or auto-load them — see below) under your favorite process supervisor (systemd, Foreman, Kamal, Kubernetes).
+
+Auto-load consumers from a directory:
+
+```ruby
+Lepus.configure do |config|
+  config.consumers_directory = 'app/consumers'
+end
+```
+
+```bash
+bundle exec lepus start   # with no class args, starts all consumers from consumers_directory
+```
+
+## Monitor
+
+```bash
+bundle exec lepus web --port 9292
+```
+
+Visit http://localhost:9292 to see consumer status, throughput, and recent activity. See [web.md](web.md).
+
+## Next steps
+
+- [Consumers](consumers.md) — full consumer DSL, retries, error handling
+- [Producers](producers.md) — exchange config, publishing options, hooks
+- [Middleware](middleware.md) — built-in middlewares and writing your own
+- [Supervisor](supervisor.md) — process model, graceful shutdown, worker pools
+- [Rails integration](rails.md) — Railtie, executor wrapping

data/docs/middleware.md

@@ -0,0 +1,240 @@
+# Middleware
+
+Lepus middlewares form a chain around each message — both for producers (around each publish) and consumers (around each delivery). Every middleware gets the message and the next app in the chain:
+
+```ruby
+class MyMiddleware < Lepus::Middleware
+  def initialize(option: nil)
+    @option = option
+  end
+
+  def call(message, app)
+    # pre-processing
+    new_message = message.mutate(payload: transform(message.payload))
+    # pass down the chain (and let the consumer/publisher actually run)
+    result = app.call(new_message)
+    # post-processing
+    result
+  end
+
+  private
+
+  def transform(payload) = payload
+end
+```
+
+## Registering
+
+### Per consumer / per producer
+
+```ruby
+class OrdersConsumer < Lepus::Consumer
+  use :json, symbolize_keys: true        # built-in by symbol
+  use :max_retry, retries: 5
+  use MyMiddleware, option: 'value'      # your own class
+end
+```
+
+### Globally
+
+```ruby
+Lepus.configure do |config|
+  config.consumer_middlewares do |chain|
+    chain.use :json, symbolize_keys: true
+    chain.use :exception_logger
+  end
+
+  config.producer_middlewares do |chain|
+    chain.use :json
+    chain.use :correlation_id
+    chain.use :instrumentation
+  end
+end
+```
+
+Global middlewares run **before** per-class middlewares.
+
+## Built-in consumer middlewares
+
+### `:json`
+
+Parse the payload from JSON.
+
+```ruby
+use :json, symbolize_keys: true, on_error: proc { :reject }
+```
+
+Options:
+
+- `symbolize_keys: true` — hash keys become symbols.
+- `on_error:` — a callable that takes the exception; its return becomes the disposition (default: `:reject`).
+
+### `:max_retry`
+
+Track retry count via RabbitMQ `x-death` headers and route to an error queue after N attempts.
+
+```ruby
+use :max_retry, retries: 5, error_queue: 'orders.error'
+```
+
+Requires the consumer's `configure` to declare a `retry_queue:` so requeued messages loop through a delay queue.
+
+### `:exception_logger`
+
+Catch exceptions from downstream middlewares/perform, log with backtrace, then re-raise (so higher-level error handling still sees the exception).
+
+```ruby
+use :exception_logger
+```
+
+### `:honeybadger`
+
+Notify Honeybadger on exceptions. Requires the `honeybadger` gem.
+
+```ruby
+use :honeybadger
+```
+
+### `:unique`
+
+Dedupe messages by `correlation_id`. Requires a storage backend (your code wires Redis or similar).
+
+```ruby
+use :unique, store: MyRedisStore
+```
+
+## Built-in producer middlewares
+
+### `:json`
+
+Serialize a hash payload as JSON; set `content_type` to `application/json`.
+
+```ruby
+use :json
+```
+
+### `:correlation_id`
+
+Auto-generate a UUID correlation id if one isn't already set on the message.
+
+```ruby
+use :correlation_id
+```
+
+### `:header`
+
+Add headers to every publish.
+
+```ruby
+use :header, 'X-Service', 'orders-api'
+use :header, 'X-Request-Id', -> { Current.request_id }
+```
+
+The value can be a static value or a callable.
+
+### `:instrumentation`
+
+Emit `ActiveSupport::Notifications` events before and after each publish. Event name: `publish.lepus`.
+
+```ruby
+use :instrumentation
+```
+
+Subscribe:
+
+```ruby
+ActiveSupport::Notifications.subscribe('publish.lepus') do |name, start, finish, id, payload|
+  puts "published to #{payload[:exchange]} in #{(finish - start) * 1000}ms"
+end
+```
+
+### `:unique`
+
+Drop duplicate publishes. Requires a storage backend.
+
+```ruby
+use :unique, store: MyRedisStore
+```
+
+## Writing your own
+
+### Consumer middleware
+
+```ruby
+class LogLevelMiddleware < Lepus::Middleware
+  def initialize(level: :info)
+    @level = level
+  end
+
+  def call(message, app)
+    Lepus.logger.public_send(@level, "Processing: #{message.payload.inspect}")
+    app.call(message)
+  end
+end
+
+class OrdersConsumer < Lepus::Consumer
+  use LogLevelMiddleware, level: :debug
+end
+```
+
+### Producer middleware
+
+Same interface. Typical use: decorate the message with metadata before it's published.
+
+```ruby
+class TenantScopingMiddleware < Lepus::Middleware
+  def call(message, app)
+    with_tenant = message.mutate(metadata: message.metadata.merge(headers: message.metadata[:headers].merge('X-Tenant-Id' => Current.tenant_id)))
+    app.call(with_tenant)
+  end
+end
+```
+
+### Mutating a message
+
+`message.mutate(**kwargs)` returns a new message with updated fields — payload, metadata, delivery_info. Don't mutate in place; build a new one and pass it down.
+
+```ruby
+app.call(message.mutate(payload: transformed_payload))
+```
+
+## Order of execution
+
+For a consumer:
+
+```
+incoming delivery
+  → global consumer middlewares (in config order)
+  → per-consumer middlewares (in class-file order)
+  → perform(message)
+  → each middleware unwinds (post-processing)
+  → disposition returned
+```
+
+For a producer:
+
+```
+publish(payload)
+  → per-producer middlewares
+  → global producer middlewares
+  → Bunny publish
+  → unwind
+```
+
+## Conditional middleware
+
+Middleware can decide not to call the next app — effectively short-circuiting:
+
+```ruby
+class SkipDuringMaintenanceMiddleware < Lepus::Middleware
+  def call(message, app)
+    if Feature.maintenance_mode?
+      Lepus.logger.warn('skipping publish during maintenance')
+      return  # do NOT call app.call(message)
+    end
+    app.call(message)
+  end
+end
+```
+
+For consumers, returning early means the consumer's `perform` never runs. The return value of `call` becomes the disposition.

data/docs/producers.md

@@ -0,0 +1,173 @@
+# Producers
+
+A producer publishes messages to an exchange. It's a subclass of `Lepus::Producer` with a `configure` call and whatever convenience methods you want on top.
+
+## Minimal example
+
+```ruby
+class OrdersProducer < Lepus::Producer
+  configure(
+    exchange: { name: 'orders', type: :topic, durable: true },
+    publish:  { persistent: true }
+  )
+
+  use :json
+  use :correlation_id
+end
+```
+
+Publish from anywhere in your app:
+
+```ruby
+OrdersProducer.publish(
+  { order_id: 42, total: 99.99 },
+  routing_key: 'order.created'
+)
+```
+
+## `configure` DSL
+
+```ruby
+configure(
+  exchange: { name: 'orders', type: :topic, durable: true },
+  publish:  { persistent: true, mandatory: false }
+)
+```
+
+| Key | Purpose |
+|-----|---------|
+| `exchange` | String (name only) or hash (`name`, `type`, `durable`, `auto_delete`). Auto-declared on first publish. |
+| `publish` | Default publish options merged into every `publish` call: `persistent`, `mandatory`, `content_type`, `expiration`, etc. |
+
+## Publishing
+
+```ruby
+OrdersProducer.publish(payload, routing_key: 'order.created')
+OrdersProducer.publish(payload, routing_key: 'order.paid', headers: { tenant_id: 123 })
+OrdersProducer.publish(payload, routing_key: 'order.created', expiration: 60_000)
+```
+
+All Bunny publish options are supported. Frequently useful:
+
+| Option | Purpose |
+|--------|---------|
+| `routing_key` | Routing key. |
+| `persistent` | Persist the message across RabbitMQ restarts. |
+| `headers` | Custom headers. |
+| `correlation_id` | RPC pattern correlation. Auto-set by `:correlation_id` middleware. |
+| `content_type` | MIME type. Auto-set to `application/json` by `:json` middleware. |
+| `expiration` | Per-message TTL in milliseconds. |
+| `mandatory` | Return the message if it can't be routed. |
+
+## Payload types
+
+- A Hash — pair with `:json` middleware to serialize.
+- A String — sent as-is.
+- Anything else — pair with a middleware that turns it into bytes.
+
+## Middleware
+
+```ruby
+class OrdersProducer < Lepus::Producer
+  use :json
+  use :correlation_id
+  use :instrumentation
+  use :header, 'X-Service', 'orders-api'
+end
+```
+
+Built-in producer middlewares:
+
+| Name | Purpose |
+|------|---------|
+| `:json` | Serialize a hash into JSON; sets `content_type`. |
+| `:correlation_id` | Auto-generate a UUID `correlation_id` if absent. |
+| `:header` | Add a static or dynamic header: `use :header, 'X-Foo', 'bar'`. |
+| `:instrumentation` | Emit `ActiveSupport::Notifications` events around each publish. |
+| `:unique` | Reject duplicate publishes (by correlation id — needs external storage). |
+
+Write your own — see [middleware.md](middleware.md).
+
+## Convenience class methods
+
+Wrap `publish` with domain-specific signatures:
+
+```ruby
+class OrdersProducer < Lepus::Producer
+  configure(exchange: { name: 'orders', type: :topic, durable: true })
+  use :json
+  use :correlation_id
+
+  def self.order_created(order)
+    publish(
+      { order_id: order.id, total: order.total, created_at: order.created_at },
+      routing_key: 'order.created'
+    )
+  end
+
+  def self.order_shipped(order)
+    publish(
+      { order_id: order.id, shipped_at: order.shipped_at },
+      routing_key: 'order.shipped'
+    )
+  end
+end
+
+# In your code:
+OrdersProducer.order_created(order)
+```
+
+## Connection pooling
+
+Producers share a single connection pool across the process:
+
+```ruby
+Lepus.configure do |config|
+  config.producer do |p|
+    p.pool_size    = 5
+    p.pool_timeout = 5.0
+  end
+end
+```
+
+## Enable / disable publishing
+
+Useful when you want to disable side effects in tests or a specific environment.
+
+```ruby
+# Globally
+Lepus::Producers.disable!
+Lepus::Producers.enabled?(OrdersProducer)  # => false
+
+# By producer class
+Lepus::Producers.disable!(OrdersProducer)
+
+# By exchange name
+Lepus::Producers.disable!('orders')
+
+# Block-scoped
+Lepus::Producers.without_publishing do
+  OrdersProducer.publish(...)  # no-op
+end
+
+Lepus::Producers.with_publishing do
+  OrdersProducer.publish(...)  # forced through even if disabled
+end
+```
+
+## Error handling
+
+A publish can fail for many reasons — connection down, channel closed, etc. By default, Bunny retries with exponential backoff within `recovery_attempts`. For anything beyond that, you catch and handle it:
+
+```ruby
+begin
+  OrdersProducer.order_created(order)
+rescue Bunny::Exception => e
+  Rails.logger.error("publish failed: #{e.message}")
+  # queue for later retry, fall back to a DB outbox, etc.
+end
+```
+
+## Testing
+
+See [testing.md](testing.md) — there's a `Lepus::Testing.producer_messages(ProducerClass)` helper that captures publishes in-memory.