lepus 0.0.1.rc2 → 0.1.0

data/docs/prometheus.md

@@ -0,0 +1,112 @@
+# Prometheus metrics
+
+Lepus ships an optional integration with [`prometheus_exporter`](https://github.com/discourse/prometheus_exporter). Consumer and producer processes forward metric payloads over TCP to a collector server; Prometheus scrapes the collector's `/metrics` endpoint.
+
+## Requirements
+
+- `prometheus_exporter` in your Gemfile.
+- A collector server reachable from every process that calls `require "lepus/prometheus"`. Typically this is the Lepus supervisor process itself, listening on `localhost:9394` (so forked workers can reach it without networking changes).
+
+## Enabling instrumentation
+
+In each process that runs Lepus (supervisor + workers), load:
+
+```ruby
+require "lepus/prometheus"
+```
+
+This prepends instrumentation onto `Lepus::Consumers::Handler` and `Lepus::Consumers::Worker` and subscribes to the `publish.lepus` `ActiveSupport::Notifications` event. Requiring must happen before the supervisor forks workers so every child inherits the instrumentation.
+
+## Starting the collector server
+
+Run the collector inside the supervisor so forked workers can send metrics to `localhost:9394`:
+
+```ruby
+# config/initializers/lepus.rb
+require "lepus/web"
+
+if Rails.application.config.prometheus_enabled
+  require "lepus/prometheus"
+  require "lepus/prometheus/collector"
+end
+
+Lepus.configure do |config|
+  # …
+end
+
+if Rails.application.config.prometheus_enabled
+  Lepus::Supervisor.on_start do
+    require "prometheus_exporter/server"
+
+    collector = PrometheusExporter::Server::Collector.new
+    collector.register_collector(Lepus::Prometheus::Collector.new)
+
+    server = PrometheusExporter::Server::WebServer.new(
+      port: 9394,
+      bind: "0.0.0.0",
+      collector: collector
+    )
+    server.start
+  end
+end
+```
+
+Point Prometheus at `<lepus-host>:9394` in your scrape config.
+
+## Polling RabbitMQ queue stats
+
+Queue depth is not published by the workers — it's pulled from the RabbitMQ Management API by a poller thread. Start it from `on_start`:
+
+```ruby
+Lepus::Supervisor.on_start do
+  Lepus::Prometheus.watch_queues(interval: 30)
+end
+```
+
+The poller emits a `lepus_queue_poll_last_success_timestamp_seconds` gauge after each successful API round-trip and a `lepus_queue_poll_errors_total` counter on failure — alert on stale timestamps to catch a silently broken poller.
+
+## Metrics
+
+| Metric | Type | Labels |
+| --- | --- | --- |
+| `lepus_messages_processed_total` | counter | `consumer`, `queue`, `result` (`ack`/`reject`/`requeue`/`nack`/`error`), `error` (exception class, empty on success) |
+| `lepus_delivery_duration_seconds` | histogram | `consumer`, `queue` |
+| `lepus_messages_published_total` | counter | `exchange`, `routing_key` |
+| `lepus_publish_duration_seconds` | histogram | `exchange`, `routing_key` |
+| `lepus_process_rss_memory_bytes` | gauge | `kind`, `name` |
+| `lepus_process_info` | gauge (always `1`) | `kind`, `name`, `pid`, `hostname` |
+| `lepus_queue_messages` | gauge | `name` |
+| `lepus_queue_messages_ready` | gauge | `name` |
+| `lepus_queue_messages_unacknowledged` | gauge | `name` |
+| `lepus_queue_consumers` | gauge | `name` |
+| `lepus_queue_memory_bytes` | gauge | `name` |
+| `lepus_queue_poll_last_success_timestamp_seconds` | gauge | — |
+| `lepus_queue_poll_errors_total` | counter | `error` |
+
+`result="error"` is recorded for every delivery that raises out of the consumer, alongside the exception class — queryable as `rate(lepus_messages_processed_total{result="error"}[5m])`.
+
+## Configuration
+
+```ruby
+Lepus.configure do |config|
+  # Histogram buckets (in seconds) used for delivery and publish latency.
+  config.prometheus_buckets = [0.01, 0.05, 0.1, 0.5, 1, 5]
+end
+```
+
+To send metrics to a collector on a different host:
+
+```ruby
+require "prometheus_exporter/client"
+Lepus::Prometheus.client = PrometheusExporter::Client.new(host: "collector.internal", port: 9394)
+```
+
+## Standalone collector mode
+
+You can also run `prometheus_exporter` as a separate process and load just the collector:
+
+```sh
+prometheus_exporter -a lepus/prometheus/collector
+```
+
+`lib/lepus/prometheus/collector.rb` deliberately does not require the rest of the gem, so this works without a full Lepus boot.

data/docs/rails.md

@@ -0,0 +1,161 @@
+# Rails Integration
+
+When Rails is loaded, Lepus's Railtie wires up sensible defaults. No initializer is strictly required.
+
+## What the Railtie does
+
+- Sets `config.logger = Rails.logger` (unless you've already set one).
+- Sets `config.app_executor = Rails.application.executor` — every consumer `perform` runs inside the Rails executor, which means:
+  - Autoloading works (Zeitwerk is active).
+  - Query cache is cleared after each message.
+  - Connection cleanup runs after each message.
+  - Reloading works in development.
+- Subscribes the log subscriber for friendly production log lines.
+- Integrates with `Rails.error` — unhandled exceptions are reported there.
+
+## Overriding
+
+```ruby
+# config/initializers/lepus.rb
+Lepus.configure do |config|
+  config.rabbitmq_url    = ENV.fetch('RABBITMQ_URL')
+  config.connection_name = 'my-service'
+
+  # Override the defaults:
+  config.logger       = MyLogger.new
+  config.app_executor = nil   # disable executor wrapping entirely
+
+  config.consumers_directory = 'app/consumers'
+
+  config.worker(:default) do |w|
+    w.pool_size = 5
+    w.before_fork { ActiveRecord::Base.connection_handler.clear_all_connections! }
+    w.after_fork  { ActiveRecord::Base.establish_connection }
+  end
+end
+```
+
+## Defining consumers and producers
+
+Put them in `app/consumers/` and `app/producers/`. With `config.consumers_directory = 'app/consumers'`, `lepus start` with no arguments auto-loads all of them.
+
+```ruby
+# app/consumers/orders_consumer.rb
+class OrdersConsumer < Lepus::Consumer
+  configure(
+    queue:    'orders',
+    exchange: { name: 'orders', type: :topic, durable: true },
+    routing_key: 'order.*'
+  )
+  use :json, symbolize_keys: true
+
+  def perform(message)
+    Order.create!(message.payload)
+    :ack
+  end
+end
+```
+
+```ruby
+# app/producers/orders_producer.rb
+class OrdersProducer < Lepus::Producer
+  configure(exchange: { name: 'orders', type: :topic, durable: true })
+  use :json
+  use :correlation_id
+end
+```
+
+## Running alongside a Rails web app
+
+```bash
+# Terminal 1 — web
+bin/rails server
+
+# Terminal 2 — consumers
+bundle exec lepus start --require_file config/environment.rb
+```
+
+Or use Foreman:
+
+```
+# Procfile
+web:    bin/rails server
+worker: bundle exec lepus start --require_file config/environment.rb
+```
+
+## The Puma plugin
+
+For apps that want consumers running inside the same Puma process (development convenience, or small-scale production where you don't want another service):
+
+```ruby
+# config/puma.rb
+plugin :lepus
+```
+
+The plugin forks consumer workers as Puma's cluster workers would, tying their lifecycle to Puma's.
+
+**Caution:** running consumers inside Puma means process restarts on deploy take longer (waiting for in-flight messages) and you can't scale consumers independently of web requests. For non-trivial workloads, run them as a separate service.
+
+## Mounting the web dashboard
+
+```ruby
+# config/routes.rb
+require 'lepus/web'
+
+authenticate :user, ->(u) { u.admin? } do
+  mount Lepus::Web::App, at: '/lepus'
+end
+```
+
+See [web.md](web.md).
+
+## Testing
+
+```ruby
+# spec/rails_helper.rb (or spec/spec_helper.rb)
+require 'lepus/testing'
+
+RSpec.configure do |config|
+  config.before(:each) { Lepus::Testing.enable!; Lepus::Testing.reset! }
+end
+```
+
+See [testing.md](testing.md).
+
+## Active Record gotchas
+
+Since `app_executor` is set to `Rails.application.executor`, Active Record connection management is handled per message — no manual `clear_active_connections!` needed.
+
+For worker subprocesses, the `before_fork` / `after_fork` hooks close and reopen connections cleanly:
+
+```ruby
+Lepus.configure do |config|
+  config.worker(:default) do |w|
+    w.before_fork { ActiveRecord::Base.connection_handler.clear_all_connections! }
+    w.after_fork  { ActiveRecord::Base.establish_connection }
+  end
+end
+```
+
+Without these hooks, all children inherit the same database socket and things go badly fast.
+
+## Exception reporting
+
+Lepus's Rails integration reports unhandled exceptions via `Rails.error`:
+
+```ruby
+# config/application.rb or an initializer
+Rails.error.subscribe do |exception, handled:, severity:, context:, source:|
+  Honeybadger.notify(exception, context: context) if source == 'lepus'
+end
+```
+
+Or use the `:honeybadger` middleware directly on your consumers — see [middleware.md](middleware.md).
+
+## Zeitwerk
+
+Consumer and producer classes are autoloaded by Zeitwerk like any other Rails class. `app/consumers/orders_consumer.rb` → `OrdersConsumer`, namespaced with the usual folder-to-module rules.
+
+## Generators
+
+None at the moment. Create consumer and producer files by hand (or your editor's snippet of choice).

data/docs/supervisor.md

@@ -0,0 +1,112 @@
+# Supervisor
+
+The supervisor is the parent process when you run `lepus start`. It forks workers, monitors them, and handles signals.
+
+## Process model
+
+```
+Supervisor (pid = $$)
+├── Worker[:default]        pid = 1001
+│   ├── Thread OrdersConsumer
+│   └── Thread OrdersConsumer
+└── Worker[:high_priority]  pid = 1002
+    └── Thread PaymentsConsumer
+```
+
+- **One worker per named pool.** Consumers with `process.name = :default` share a worker; consumers with `process.name = :high_priority` share a different one. Unnamed consumers default to `:default`.
+- **Multiple threads per worker.** Each worker runs a thread pool sized by `config.worker(:name).pool_size`.
+- **One channel per consumer.** Each consumer gets its own Bunny channel to avoid cross-consumer interference.
+
+Fork boundaries matter because child processes inherit open file descriptors, sockets, and DB connections. See the `before_fork` / `after_fork` hooks below.
+
+## Fork hooks
+
+```ruby
+Lepus.configure do |config|
+  config.worker(:default) do |w|
+    # Runs in the SUPERVISOR just before fork()
+    w.before_fork do
+      ActiveRecord::Base.connection_handler.clear_all_connections!
+      Redis.current.disconnect!
+    end
+
+    # Runs in the CHILD after fork()
+    w.after_fork do
+      ActiveRecord::Base.establish_connection
+      Redis.current = Redis.new
+      SecureRandom.hex(4)  # implicit reseed
+    end
+  end
+end
+```
+
+The classic pattern: close long-lived connections in `before_fork`, reopen them in `after_fork`. Otherwise all children end up sharing the same socket.
+
+## Signals
+
+| Signal | Supervisor response |
+|--------|---------------------|
+| `SIGTERM` | Graceful shutdown — see below. |
+| `SIGINT` | Graceful shutdown (same as SIGTERM). |
+| `SIGQUIT` | Graceful shutdown, slightly more aggressive. |
+| `SIGTTIN` | Dump thread backtraces of every worker (debugging). |
+| `SIGUSR1` | Reopen log files (useful for logrotate). |
+
+Child workers respond to the same signals, but normally the supervisor handles signals and forwards the shutdown to children.
+
+## Graceful shutdown
+
+1. Supervisor writes a shutdown message to each worker's pipe.
+2. Each worker stops accepting new messages (`basic.cancel` on its channels).
+3. In-flight `perform` calls are allowed to finish.
+4. Workers close connections and exit with code 0.
+5. If a worker takes longer than `shutdown_timeout` (default 5 seconds per message, tuned via consumer `channel.shutdown_timeout`), it's killed with `SIGKILL`.
+6. Supervisor exits.
+
+Rule of thumb for deploy scripts: send SIGTERM, wait at least `pool_size × average_message_duration + 5 seconds`, then the supervisor should have exited cleanly.
+
+## Worker crash recovery
+
+If a worker exits unexpectedly (raised exception outside a `perform` call, OOM, killed by OOM killer):
+
+1. Supervisor detects the pipe close.
+2. Logs the exit status.
+3. Forks a new worker using the same `before_fork` / `after_fork` hooks.
+4. Worker re-declares its queues and resumes consuming.
+
+Within a `perform`, exceptions are caught and routed to `on_thread_error`; the worker is not restarted for those. Only crashes at the worker level (outside message handling) trigger a restart.
+
+## Heartbeats and the process registry
+
+Each worker (and the supervisor itself) emits heartbeats to the process registry:
+
+```ruby
+config.process_heartbeat_interval = 60     # seconds
+config.process_alive_threshold    = 5 * 60 # seconds
+```
+
+The web dashboard reads from this registry to display process status. Processes whose last heartbeat is older than `process_alive_threshold` are considered dead and hidden from the UI (but kept briefly so transient restarts don't flash).
+
+## Pidfile
+
+```bash
+bundle exec lepus start --pidfile /var/run/lepus.pid
+```
+
+Written by the supervisor at boot, removed at graceful shutdown. If the file exists when lepus starts, the supervisor checks if the PID is alive; if so, it exits (don't run two supervisors). If not, it overwrites.
+
+## Running under a process manager
+
+- **systemd:** `ExecStart=/path/to/bundle exec lepus start --pidfile /run/lepus.pid`, `KillSignal=SIGTERM`, `TimeoutStopSec=60`.
+- **Foreman / Procfile:** `worker: bundle exec lepus start`. No pidfile needed.
+- **Kubernetes:** set a `preStop` lifecycle hook to send SIGTERM, and tune `terminationGracePeriodSeconds` to be larger than your longest `perform`.
+
+## Debugging a stuck worker
+
+Send SIGTTIN:
+
+```bash
+kill -TTIN <worker-pid>
+```
+
+The worker prints thread backtraces to the log. Useful when you suspect a deadlock or a `perform` stuck on a slow network call.

data/docs/testing.md

@@ -0,0 +1,141 @@
+# Testing
+
+Lepus ships a test-mode module that captures publishes and runs consumer `perform` methods synchronously — no RabbitMQ connection required.
+
+## Enabling
+
+```ruby
+# spec/spec_helper.rb (RSpec)
+require 'lepus/testing'
+
+RSpec.configure do |config|
+  config.before(:each) { Lepus::Testing.enable! }
+  config.after(:each)  { Lepus::Testing.reset! }
+end
+```
+
+Once enabled:
+
+- Publishes don't hit RabbitMQ. They're captured in an in-memory buffer keyed by producer class.
+- Consumer handling is synchronous when invoked through `Lepus::Testing.consumer_perform`.
+
+## Testing a consumer
+
+```ruby
+describe OrdersConsumer do
+  it 'creates an order' do
+    result = Lepus::Testing.consumer_perform(
+      OrdersConsumer,
+      { order_id: 42, total: 99.99 }
+    )
+
+    expect(result).to eq(:ack)
+    expect(Order.find(42).total).to eq(99.99)
+  end
+
+  it 'rejects invalid payloads' do
+    result = Lepus::Testing.consumer_perform(OrdersConsumer, { bad: 'data' })
+    expect(result).to eq(:reject)
+  end
+
+  it 'sets delivery info and metadata' do
+    result = Lepus::Testing.consumer_perform(
+      OrdersConsumer,
+      { order_id: 7 },
+      delivery_info: { routing_key: 'order.created' },
+      metadata:      { correlation_id: 'abc-123' }
+    )
+    expect(result).to eq(:ack)
+  end
+end
+```
+
+`consumer_perform` signature:
+
+```ruby
+Lepus::Testing.consumer_perform(
+  ConsumerClass,
+  payload,
+  delivery_info: {},
+  metadata: {}
+)
+```
+
+It builds a `Lepus::Message`, runs the full middleware chain (including global middlewares), and returns the disposition symbol.
+
+## Testing a producer
+
+```ruby
+describe OrdersProducer do
+  it 'publishes the order' do
+    order = Order.create!(id: 42, total: 99.99)
+    OrdersProducer.order_created(order)
+
+    messages = Lepus::Testing.producer_messages(OrdersProducer)
+    expect(messages.size).to eq(1)
+    expect(messages[0][:payload]).to include(order_id: 42)
+    expect(messages[0][:routing_key]).to eq('order.created')
+  end
+
+  it 'runs through middleware' do
+    OrdersProducer.publish({ foo: 'bar' }, routing_key: 'x')
+
+    msg = Lepus::Testing.producer_messages(OrdersProducer).last
+    expect(msg[:metadata][:correlation_id]).to be_present  # set by :correlation_id middleware
+    expect(msg[:metadata][:content_type]).to eq('application/json')
+  end
+end
+```
+
+`producer_messages(ProducerClass)` returns an array of hashes with `:payload`, `:routing_key`, `:delivery_info`, `:metadata`.
+
+## RSpec matchers
+
+```ruby
+# spec/spec_helper.rb
+require 'lepus/testing/rspec_matchers'
+```
+
+Then:
+
+```ruby
+expect { OrdersProducer.order_created(order) }
+  .to have_published_message(OrdersProducer)
+  .with_payload(include(order_id: order.id))
+  .to_routing_key('order.created')
+```
+
+## Testing middleware in isolation
+
+Middlewares are plain Ruby objects with a `call(message, app)` method. Unit-test them directly:
+
+```ruby
+describe LogLevelMiddleware do
+  it 'logs before calling down the chain' do
+    middleware = LogLevelMiddleware.new(level: :debug)
+    message   = Lepus::Testing::MessageBuilder.build(payload: { x: 1 })
+    captured  = nil
+
+    allow(Lepus.logger).to receive(:debug) { |msg| captured = msg }
+    middleware.call(message, ->(m) { :ack })
+
+    expect(captured).to include('Processing:')
+  end
+end
+```
+
+`Lepus::Testing::MessageBuilder.build(**kwargs)` builds a realistic `Lepus::Message` for unit tests.
+
+## Resetting between tests
+
+`Lepus::Testing.reset!` clears captured publishes. In shared setup:
+
+```ruby
+RSpec.configure do |config|
+  config.before(:each) { Lepus::Testing.enable!; Lepus::Testing.reset! }
+end
+```
+
+## When you do need a real RabbitMQ
+
+Integration tests that exercise the full round-trip (publish → RabbitMQ → consume) benefit from a real broker. Use `docker run rabbitmq:3-management` or your test infra's existing one, and skip `Lepus::Testing.enable!` for those specs.

data/docs/web.md

@@ -0,0 +1,85 @@
+# Web Dashboard
+
+![Lepus web dashboard](images/lepus-web.png)
+
+Lepus ships a Rack-based monitoring UI showing consumer status, throughput, and recent activity.
+
+## Running it standalone
+
+```bash
+bundle exec lepus web --port 9292 --host 0.0.0.0
+```
+
+Visit http://localhost:9292.
+
+## Mounting in Rails
+
+```ruby
+# config/routes.rb
+require 'lepus/web'
+
+authenticate :user, ->(u) { u.admin? } do
+  mount Lepus::Web::App, at: '/lepus'
+end
+```
+
+Or with Devise:
+
+```ruby
+authenticate :admin_user do
+  mount Lepus::Web::App, at: '/admin/lepus'
+end
+```
+
+**Important:** the dashboard has no built-in auth. Wrap it with whatever your app already uses.
+
+## What it shows
+
+- **Supervisors.** Every running `lepus start` process.
+- **Workers.** Subprocesses per supervisor, with their named pool and PID.
+- **Consumers.** Per-consumer message counts (processed, rejected, errored), queue names, routing keys.
+- **Exchanges & queues.** RabbitMQ topology as seen by the gem.
+- **Recent activity.** Last N messages — timestamps, routing keys, dispositions.
+
+## Registry backend
+
+The dashboard reads from the process registry, configured at `Lepus.configure`:
+
+```ruby
+config.process_registry_backend = :file      # single-host
+# or
+config.process_registry_backend = :rabbitmq  # multi-host
+```
+
+- `:file` — metadata stored under `/tmp/lepus/...`. Works out of the box but only for a single host.
+- `:rabbitmq` — metadata stored in RabbitMQ itself. Multiple `lepus start` processes across multiple hosts show up in one dashboard.
+
+For `:rabbitmq`, also set:
+
+```ruby
+config.management_api_url = 'http://rabbitmq:15672'
+```
+
+## Heartbeats
+
+Each worker heartbeats into the registry every `config.process_heartbeat_interval` (default 60 seconds). Processes are considered "alive" if their last heartbeat is within `config.process_alive_threshold` (default 5 minutes).
+
+## API endpoints
+
+The dashboard exposes a minimal read-only JSON API at `/api/...` — the UI is a single-page app that consumes it. You can also consume the API directly for custom dashboards or alerting.
+
+Endpoints include (subject to change):
+
+- `GET /api/processes` — all tracked processes
+- `GET /api/consumers` — all registered consumers with counts
+- `GET /api/queues` — queue metadata from the RabbitMQ management API (if `management_api_url` is set)
+
+## Prometheus metrics
+
+See [prometheus.md](prometheus.md) for the metric list, label cardinality notes, and how to wire the collector server inside a Lepus supervisor process.
+
+## Operating in production
+
+- Put the dashboard behind your existing auth layer (OAuth proxy, Rails authentication, Basic auth).
+- Use `:rabbitmq` registry backend for multi-node visibility.
+- Retain logs separately — the dashboard is for live state, not audit trails.