lepus 0.0.1.beta2 → 0.1.0

data/docs/README.md

@@ -0,0 +1,80 @@
+# lepus
+
+RabbitMQ-backed producer/consumer framework for Ruby — with a supervisor, middleware chains, a CLI, and a live web dashboard.
+
+Think Sidekiq or SolidQueue, but on top of RabbitMQ rather than Redis or a database. Lepus handles the operational concerns (process supervision, graceful shutdown, connection pooling, signal handling, per-worker pools) so your application code can stay focused on "what does this message do".
+
+## Contents
+
+- [Getting started](getting-started.md) — install, define your first consumer and producer, run them
+- [Configuration](configuration.md) — the full `Lepus.configure` DSL
+- [Consumers](consumers.md) — queue bindings, lifecycle, result codes, retries
+- [Producers](producers.md) — exchanges, publishing, enable/disable hooks
+- [Middleware](middleware.md) — built-in middlewares and how to write your own
+- [CLI](cli.md) — `lepus start`, `lepus web`
+- [Supervisor](supervisor.md) — process model, signals, graceful shutdown
+- [Web dashboard](web.md) — the monitoring UI
+- [Testing](testing.md) — testing consumers and producers
+- [Rails integration](rails.md) — Railtie, executor wrapping, Puma plugin
+
+## Install
+
+```ruby
+# Gemfile
+gem 'lepus'
+```
+
+## One-minute tour
+
+```ruby
+# config/initializers/lepus.rb
+Lepus.configure do |config|
+  config.rabbitmq_url   = ENV.fetch('RABBITMQ_URL', 'amqp://guest:guest@localhost:5672')
+  config.connection_name = 'my-service'
+end
+
+# app/consumers/orders_consumer.rb
+class OrdersConsumer < Lepus::Consumer
+  configure(
+    queue:       'orders',
+    exchange:    { name: 'orders', type: :topic, durable: true },
+    routing_key: ['order.*']
+  )
+
+  use :json, symbolize_keys: true
+  use :max_retry, retries: 5
+
+  def perform(message)
+    Order.create!(message.payload)
+    :ack
+  end
+end
+
+# app/producers/orders_producer.rb
+class OrdersProducer < Lepus::Producer
+  configure(exchange: { name: 'orders', type: :topic, durable: true })
+  use :json
+  use :correlation_id
+end
+```
+
+Run the consumer:
+
+```bash
+bundle exec lepus start OrdersConsumer
+```
+
+Publish from anywhere in your app:
+
+```ruby
+OrdersProducer.publish({ order_id: 42, total: 99.99 }, routing_key: 'order.created')
+```
+
+## Version & dependencies
+
+- Ruby: `>= 2.7.0`
+- Runtime: `bunny`, `thor`, `zeitwerk`, `concurrent-ruby`, `multi_json`
+
+## License
+
+MIT.

data/docs/cli.md

@@ -0,0 +1,108 @@
+# CLI
+
+The `lepus` executable is the main operational entry point. It's a Thor-based CLI with two subcommands.
+
+```bash
+bundle exec lepus <command> [args] [options]
+```
+
+## `lepus start`
+
+Boot the supervisor and run the specified consumers.
+
+```bash
+bundle exec lepus start [CONSUMER_CLASSES...] [options]
+```
+
+If no classes are specified **and** `config.consumers_directory` is set, every consumer class discovered in that directory is started.
+
+### Options
+
+| Flag | Default | Purpose |
+|------|---------|---------|
+| `--debug` | `false` | Set logger level to DEBUG. |
+| `--logfile PATH` | stdout | Write logs to a file. |
+| `--pidfile PATH` | none | Write the supervisor PID to a file. |
+| `--require_file PATH`, `-r PATH` | none | Require this file before starting (e.g. `config/environment.rb`). |
+
+### Examples
+
+```bash
+# One consumer, no framework auto-load
+bundle exec lepus start OrdersConsumer
+
+# Multiple consumers, Rails env loaded first
+bundle exec lepus start OrdersConsumer PaymentsConsumer \
+  --require_file config/environment.rb
+
+# Auto-load everything under config.consumers_directory
+bundle exec lepus start
+
+# Debug mode + pidfile (typical for a systemd unit)
+bundle exec lepus start OrdersConsumer \
+  --require_file config/environment.rb \
+  --pidfile /var/run/lepus.pid \
+  --logfile /var/log/lepus.log \
+  --debug
+```
+
+### What happens at start
+
+1. Load `--require_file` if given.
+2. Resolve consumer classes (CLI args or auto-discovered).
+3. Group consumers by their `process.name` (defaulting to `:default`).
+4. Fork one worker subprocess per group. Parent becomes the supervisor.
+5. Each worker opens a Bunny channel, declares queues/exchanges/bindings, and starts consuming with its configured thread pool.
+6. Supervisor monitors children via pipes and restarts any that crash.
+
+See [supervisor.md](supervisor.md) for the full lifecycle.
+
+## `lepus web`
+
+Run the web dashboard.
+
+```bash
+bundle exec lepus web [options]
+```
+
+### Options
+
+| Flag | Default | Purpose |
+|------|---------|---------|
+| `--port PORT`, `-p PORT` | `9292` | Port to bind. |
+| `--host HOST`, `-o HOST` | `0.0.0.0` | Host to bind. |
+
+### Example
+
+```bash
+bundle exec lepus web --port 9292 --host 0.0.0.0
+```
+
+Visit http://localhost:9292.
+
+The web UI reads from the process registry backend (`config.process_registry_backend`). For multi-host visibility, set the backend to `:rabbitmq`. See [web.md](web.md).
+
+### Mounting in Rails
+
+Instead of running the CLI, you can mount the web app directly:
+
+```ruby
+# config/routes.rb
+require 'lepus/web'
+mount Lepus::Web::App, at: '/lepus'
+```
+
+See [rails.md](rails.md) for authorization patterns.
+
+## Exit codes
+
+- `0` — graceful shutdown (SIGTERM / SIGINT).
+- Non-zero — unrecoverable error at boot (config invalid, RabbitMQ unreachable, etc.).
+
+Workers that crash after boot are restarted by the supervisor; the supervisor itself only exits on an unrecoverable event or a requested shutdown.
+
+## Environment variables
+
+- `RABBITMQ_URL` — fallback for `config.rabbitmq_url` if not explicitly set.
+
+Everything else is configured via `Lepus.configure`. See [configuration.md](configuration.md).

data/docs/configuration.md

@@ -0,0 +1,171 @@
+# Configuration
+
+Lepus is configured via a single block:
+
+```ruby
+Lepus.configure do |config|
+  # ... see below ...
+end
+```
+
+The configuration is mutable at boot and should be set once. Values are read lazily — most only matter at worker-start time.
+
+## Connection
+
+```ruby
+Lepus.configure do |config|
+  config.rabbitmq_url                    = ENV.fetch('RABBITMQ_URL', 'amqp://guest:guest@localhost:5672')
+  config.connection_name                 = 'my-service'
+  config.recovery_attempts               = 10          # nil = infinite
+  config.recover_from_connection_close   = true
+end
+```
+
+| Key | Default | Purpose |
+|-----|---------|---------|
+| `rabbitmq_url` | `ENV['RABBITMQ_URL']` or `amqp://guest:guest@localhost:5672` | Connection string |
+| `connection_name` | gem-generated | Shown in RabbitMQ management UI — set to your service name |
+| `recovery_attempts` | `10` | Max automatic reconnects; `nil` for infinite |
+| `recover_from_connection_close` | `true` | Auto-recover after a clean close |
+
+## Application metadata
+
+```ruby
+config.application_name       = 'orders-service'
+config.management_api_url     = 'http://rabbitmq:15672'   # only for :rabbitmq registry backend
+```
+
+Shown in the web dashboard. `management_api_url` is only required if `process_registry_backend = :rabbitmq`.
+
+## Consumer discovery
+
+```ruby
+config.consumers_directory = 'app/consumers'
+```
+
+When present, `lepus start` with no class arguments auto-loads every class under this directory and runs all configured consumers.
+
+## Worker pools
+
+A **worker** is a subprocess the supervisor forks. Consumers are grouped into workers by their `process.name`.
+
+```ruby
+config.worker(:default) do |w|
+  w.pool_size    = 5       # threads per worker
+  w.pool_timeout = 10.0    # seconds before yielding to another task
+
+  w.before_fork { ActiveRecord::Base.connection_handler.clear_all_connections! }
+  w.after_fork  { ActiveRecord::Base.establish_connection }
+end
+
+config.worker(:high_priority, pool_size: 10)
+```
+
+| Key | Default | Purpose |
+|-----|---------|---------|
+| `pool_size` | `2` | Max threads per worker process |
+| `pool_timeout` | `10.0` | Seconds threads wait on queue checkout |
+| `before_fork` | no-op | Block run in parent before fork — close sockets, drop DB connections, etc. |
+| `after_fork` | no-op | Block run in child after fork — reconnect, reseed RNG, etc. |
+
+Assign consumers to a named worker via their own `configure(process: { name: :high_priority })` (see [consumers.md](consumers.md)).
+
+## Producer pool
+
+Producers share a connection pool:
+
+```ruby
+config.producer do |p|
+  p.pool_size    = 5
+  p.pool_timeout = 5.0
+end
+```
+
+## Global middleware
+
+Middleware chains run around every message published or consumed, in addition to any per-producer or per-consumer `use` calls.
+
+```ruby
+config.producer_middlewares do |chain|
+  chain.use :instrumentation
+  chain.use :correlation_id
+end
+
+config.consumer_middlewares do |chain|
+  chain.use :json, symbolize_keys: true
+end
+```
+
+See [middleware.md](middleware.md).
+
+## Process registry
+
+The supervisor keeps a registry of running processes, visible to the web dashboard.
+
+```ruby
+config.process_registry_backend     = :file      # or :rabbitmq
+config.process_heartbeat_interval   = 60         # seconds
+config.process_alive_threshold      = 5 * 60     # seconds
+```
+
+- `:file` — metadata in local files under `/tmp/lepus/...`. Single-host.
+- `:rabbitmq` — metadata in RabbitMQ itself; multi-host visibility in the web dashboard.
+
+## Rails integration
+
+When Rails is loaded, a Railtie wires:
+
+- `config.app_executor = Rails.application.executor` automatically
+- `config.logger = Rails.logger`
+
+You can override either:
+
+```ruby
+config.app_executor = nil             # disable executor wrapping
+config.logger       = MyLogger.new
+```
+
+See [rails.md](rails.md).
+
+## Threading error handler
+
+```ruby
+config.on_thread_error = ->(exception) {
+  Rails.error.report(exception)
+  Honeybadger.notify(exception)
+}
+```
+
+Called when a worker thread raises. Does not stop the worker.
+
+## Logger
+
+```ruby
+config.logger = Logger.new($stdout)
+```
+
+Default: `Logger.new($stdout)` (or `Rails.logger` if Rails is present). The `--debug` CLI flag sets the level to DEBUG.
+
+## Full example
+
+```ruby
+Lepus.configure do |config|
+  config.rabbitmq_url    = ENV.fetch('RABBITMQ_URL')
+  config.connection_name = 'orders-service'
+  config.application_name = 'orders-service'
+  config.consumers_directory = 'app/consumers'
+
+  config.worker(:default) do |w|
+    w.pool_size = 5
+    w.before_fork { ActiveRecord::Base.connection_handler.clear_all_connections! }
+    w.after_fork  { ActiveRecord::Base.establish_connection }
+  end
+
+  config.producer_middlewares do |chain|
+    chain.use :instrumentation
+    chain.use :correlation_id
+  end
+
+  config.on_thread_error = ->(exc) { Honeybadger.notify(exc) }
+end
+```

data/docs/consumers.md

@@ -0,0 +1,168 @@
+# Consumers
+
+Consumers subscribe to a queue, receive messages, and process them. A consumer is a subclass of `Lepus::Consumer` with a `configure` call and a `perform` method.
+
+## Minimal example
+
+```ruby
+class OrdersConsumer < Lepus::Consumer
+  configure(
+    queue:    'orders',
+    exchange: { name: 'orders', type: :topic, durable: true },
+    routing_key: ['order.*']
+  )
+
+  use :json, symbolize_keys: true
+
+  def perform(message)
+    Order.create!(message.payload)
+    :ack
+  end
+end
+```
+
+## `configure` DSL
+
+```ruby
+configure(
+  queue:       'orders',                                    # string or hash
+  exchange:    { name: 'orders', type: :topic, durable: true },
+  routing_key: ['order.*', 'invoice.created'],              # array or string
+  prefetch:    1,                                            # channel QoS
+  retry_queue: { delay: 5_000 },                            # or true, or false
+  error_queue: true,                                         # or queue name, or false
+  process:     { name: :default, threads: 5 },              # worker assignment
+  channel:     { pool_size: 1, shutdown_timeout: 60, abort_on_exception: false }
+)
+```
+
+| Key | Purpose |
+|-----|---------|
+| `queue` | The queue name — string, or a hash for advanced options (`name`, `durable`, `arguments`, …). |
+| `exchange` | Bind to this exchange. String, or hash (`name`, `type`, `durable`, `auto_delete`). |
+| `routing_key` | Binding routing key(s). |
+| `prefetch` | Channel QoS (`basic.qos`). Leave at `1` unless you know you need more. |
+| `retry_queue` | When truthy, sets up a retry queue with a delay exchange. Messages requeued via `:requeue` or `max_retry` go here. |
+| `error_queue` | Queue name (string) or `true` to auto-name. Messages that exhaust retries land here. |
+| `process.name` | The named worker pool this consumer runs in — see `Lepus.configure.worker(:name)`. |
+| `process.threads` | Threads inside the consumer's worker dedicated to this consumer. |
+
+## The `perform` method
+
+```ruby
+def perform(message)
+  # ...
+end
+```
+
+`message` is a `Lepus::Message` with:
+
+- `#payload` — the decoded body (raw bytes, or parsed data if a decoding middleware is in the chain).
+- `#delivery_info` — `#exchange`, `#routing_key`, `#redelivered?`, `#delivery_tag`, `#consumer_tag`.
+- `#metadata` — `#headers`, `#content_type`, `#content_encoding`, `#correlation_id`, `#message_id`, `#reply_to`, `#type`, `#timestamp`, `#user_id`, `#app_id`, `#priority`.
+
+Return a disposition symbol or call one of the helpers:
+
+| Return | Helper | Effect |
+|--------|--------|--------|
+| `:ack` | `ack!` | Acknowledge the message. |
+| `:reject` | `reject!` | Reject without requeue (drops, or goes to DLX if configured). |
+| `:requeue` | `requeue!` | Reject with requeue — back of the queue. |
+| `:nack` | `nack!` | Negative-ack. |
+
+A `perform` that returns nothing implicitly acks. A `perform` that raises is logged, `on_thread_error` fires, and the message is rejected.
+
+## Middleware
+
+Add middlewares to a consumer with `use`:
+
+```ruby
+class OrdersConsumer < Lepus::Consumer
+  use :json, symbolize_keys: true
+  use :max_retry, retries: 5, error_queue: 'orders.error'
+  use :exception_logger
+  use :honeybadger
+end
+```
+
+Built-in middlewares:
+
+| Name | Purpose |
+|------|---------|
+| `:json` | Parse JSON body into a hash. |
+| `:max_retry` | Track `x-death` headers; after N retries, route to error queue. |
+| `:exception_logger` | Log unhandled exceptions with backtrace. |
+| `:honeybadger` | Notify Honeybadger on exceptions. |
+| `:unique` | Idempotent dedupe by `correlation_id` (requires storage). |
+
+Write your own — see [middleware.md](middleware.md).
+
+## Error handling
+
+A `perform` that raises:
+
+1. The message is rejected by default.
+2. `config.on_thread_error` is called with the exception.
+3. If `:exception_logger` middleware is in the chain, it logs with backtrace.
+4. If `:max_retry` is in the chain, it tracks retry count via RabbitMQ's `x-death` header and redirects to an error queue after N rejections.
+
+## Retries
+
+The built-in retry pattern uses RabbitMQ's dead-letter machinery:
+
+```ruby
+configure(
+  queue:       'orders',
+  exchange:    { name: 'orders', type: :topic, durable: true },
+  routing_key: 'order.*',
+  retry_queue: { delay: 5_000 },  # 5-second delay before retry
+  error_queue: 'orders.error'
+)
+use :max_retry, retries: 5, error_queue: 'orders.error'
+```
+
+On rejection:
+
+1. `max_retry` inspects `x-death` headers.
+2. If retries are below the limit, it re-publishes to the retry queue.
+3. After TTL, the retry queue dead-letters back to the original queue for another attempt.
+4. After N failures, the middleware routes to the error queue.
+
+## Worker assignment
+
+Assign a consumer to a named worker pool:
+
+```ruby
+configure(
+  queue:    'orders',
+  exchange: { name: 'orders', type: :topic },
+  process:  { name: :high_priority, threads: 10 }
+)
+```
+
+Then in the global config:
+
+```ruby
+Lepus.configure do |config|
+  config.worker(:high_priority) do |w|
+    w.pool_size = 20
+  end
+end
+```
+
+Consumers assigned to the same worker share a single subprocess.
+
+## Logging
+
+Inside `perform`, `logger` is the Lepus logger (tagged with the consumer class name):
+
+```ruby
+def perform(message)
+  logger.info("processing #{message.delivery_info.routing_key}")
+  # ...
+end
+```
+
+## Testing
+
+See [testing.md](testing.md) for the test-mode helpers.

data/docs/getting-started.md

@@ -0,0 +1,136 @@
+# Getting Started
+
+## Install
+
+```ruby
+# Gemfile
+gem 'lepus'
+```
+
+```bash
+bundle install
+```
+
+You'll also need a running RabbitMQ instance. For local dev:
+
+```bash
+docker run -d --rm -p 5672:5672 -p 15672:15672 rabbitmq:3-management
+```
+
+## Configure
+
+```ruby
+# config/initializers/lepus.rb (Rails) or your bootstrap file
+Lepus.configure do |config|
+  config.rabbitmq_url    = ENV.fetch('RABBITMQ_URL', 'amqp://guest:guest@localhost:5672')
+  config.connection_name = 'my-service'
+end
+```
+
+See [configuration.md](configuration.md) for the full DSL.
+
+## Define a consumer
+
+```ruby
+# app/consumers/orders_consumer.rb
+class OrdersConsumer < Lepus::Consumer
+  configure(
+    queue:       'orders',
+    exchange:    { name: 'orders', type: :topic, durable: true },
+    routing_key: ['order.*']
+  )
+
+  use :json, symbolize_keys: true
+
+  def perform(message)
+    # message.payload       => the decoded JSON body (a hash, due to the :json middleware)
+    # message.delivery_info => exchange, routing_key, redelivered?
+    # message.metadata      => headers, content_type, correlation_id, etc.
+    Order.create!(message.payload)
+    :ack
+  end
+end
+```
+
+The `perform` method returns a symbol indicating disposition:
+
+- `:ack` — acknowledge; RabbitMQ removes the message.
+- `:reject` — reject; message is dropped (or routed to a dead-letter exchange if configured).
+- `:requeue` — reject and requeue for another delivery attempt.
+- `:nack` — negative-ack without requeue (similar to `:reject`).
+
+`ack!`, `reject!`, `requeue!`, `nack!` are also available as helper methods.
+
+## Define a producer
+
+```ruby
+# app/producers/orders_producer.rb
+class OrdersProducer < Lepus::Producer
+  configure(
+    exchange: { name: 'orders', type: :topic, durable: true },
+    publish:  { persistent: true }
+  )
+
+  use :json
+  use :correlation_id
+end
+```
+
+Publish:
+
+```ruby
+OrdersProducer.publish(
+  { order_id: 42, total: 99.99 },
+  routing_key: 'order.created'
+)
+```
+
+## Run
+
+### One-off (development)
+
+```bash
+bundle exec lepus start OrdersConsumer
+```
+
+Flags worth knowing:
+
+```bash
+bundle exec lepus start OrdersConsumer PaymentsConsumer \
+  --require_file config/environment.rb \
+  --debug
+```
+
+See [cli.md](cli.md).
+
+### As a long-running service
+
+In production, run `lepus start` with all your consumer classes (or auto-load them — see below) under your favorite process supervisor (systemd, Foreman, Kamal, Kubernetes).
+
+Auto-load consumers from a directory:
+
+```ruby
+Lepus.configure do |config|
+  config.consumers_directory = 'app/consumers'
+end
+```
+
+```bash
+bundle exec lepus start   # with no class args, starts all consumers from consumers_directory
+```
+
+## Monitor
+
+```bash
+bundle exec lepus web --port 9292
+```
+
+Visit http://localhost:9292 to see consumer status, throughput, and recent activity. See [web.md](web.md).
+
+## Next steps
+
+- [Consumers](consumers.md) — full consumer DSL, retries, error handling
+- [Producers](producers.md) — exchange config, publishing options, hooks
+- [Middleware](middleware.md) — built-in middlewares and writing your own
+- [Supervisor](supervisor.md) — process model, graceful shutdown, worker pools
+- [Rails integration](rails.md) — Railtie, executor wrapping