lepus 0.0.1.rc2 → 0.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: f2928402610a7a4f5027f33e31ea7ea790f645ebb9029e1aeba9c735681d803c
-  data.tar.gz: fab7a599c6e24140e523d215b54bb6aed06e1c40b4ee2263ffd8c26b6de726d8
+  metadata.gz: 79f0f6758c114b63fdc54298ffe89a0d314fde163f38603632fd249d4136c5e3
+  data.tar.gz: a644ff3638a8be640d79c625fa8ce338cc15ab838cd12c731833716e6324775e
 SHA512:
-  metadata.gz: 68c80bbe807bb05bd6ac3acafe8a534b70ce965190ffa19589ac2dc49e6bfae14399706c0516bb0eeb4f850b27e73dbb52ea2e16165f82d9b00615746f99dd7d
-  data.tar.gz: 8074660176b3f287a531ad9f8c73feb81b434da8fe7411d6e9b6a309e79474a6d0598b40d0439524b106733293b00b515ae7c74b4ffaf4c93e9ba3ad81000565
+  metadata.gz: e607ed235f468663e71dddedd6aaed1953ef7aaa4a265cbd92454354fa1a2cd60a3495b243c750d89055db2dced9eda7eb97068aa0785c0f2d6347ed55b8908f
+  data.tar.gz: c54bbaf33e0dda1cbe46a81fb2756623c49ef1c8bdeefdc5ebae9ba988c3b8c7dfc7d00da5ec14cbbede7e8ff47c3d2e241b4244ef8db0bc67b4c7240f639bf3

data/.gitignore

@@ -5,7 +5,6 @@
 /_yardoc/
 /coverage/
 /doc/
-/examples/
 /pkg/
 /spec/reports/
 /tmp/

data/Gemfile

@@ -6,3 +6,8 @@ source "https://rubygems.org"
 gemspec
 
 gem "connection_pool", "< 3"
+
+# prometheus_exporter 2.1.1+ requires Ruby 3.0+; 2.3+ requires Ruby 3.2+.
+# The CI matrix runs this Gemfile on Ruby 2.7 and 3.0, so pin to a version
+# compatible across the matrix.
+gem "prometheus_exporter", "= 2.1.0"

data/Gemfile.lock

@@ -1,7 +1,8 @@
 PATH
   remote: .
   specs:
-    lepus (0.0.1.rc2)
+    lepus (0.1.0)
+      base64
       bunny
       concurrent-ruby
       multi_json
@@ -15,6 +16,7 @@ GEM
       public_suffix (>= 2.0.2, < 7.0)
     amq-protocol (2.5.1)
     ast (2.4.2)
+    base64 (0.3.0)
     bigdecimal (3.1.8)
     bunny (2.24.0)
       amq-protocol (~> 2.3)
@@ -41,11 +43,16 @@ GEM
     parser (3.3.5.0)
       ast (~> 2.4.1)
       racc
+    prometheus_exporter (2.1.0)
+      webrick
     pry (0.14.2)
       coderay (~> 1.1)
       method_source (~> 1.0)
     public_suffix (5.1.1)
     racc (1.8.1)
+    rack (3.2.1)
+    rack-test (2.2.0)
+      rack (>= 1.3)
     rainbow (3.1.1)
     rbtree (0.4.6)
     redis (5.4.1)
@@ -112,6 +119,7 @@ GEM
       addressable (>= 2.8.0)
       crack (>= 0.3.2)
       hashdiff (>= 0.4.0, < 2.0.0)
+    webrick (1.9.2)
     zeitwerk (2.6.18)
 
 PLATFORMS
@@ -123,7 +131,10 @@ DEPENDENCIES
   de-dupe
   dotenv
   lepus!
+  prometheus_exporter (= 2.1.0)
   pry
+  rack (>= 2.2)
+  rack-test
   rspec
   rubocop
   rubocop-performance

data/README.md

@@ -1,8 +1,13 @@
 # Lepus
 
+![Lepus web dashboard](https://github.com/user-attachments/assets/a640fc43-2f53-4aa7-aede-f1464f3e6e03)
 
 Lepus is a lightweight but powerful Ruby library to help you to consume and produce messages to [RabbitMQ](https://www.rabbitmq.com/) using the [Bunny](https://github.com/ruby-amqp/bunny) gem. It's similar to the Sidekiq, Faktory, ActiveJob, SolidQueue, and other libraries, but using RabbitMQ as the message broker.
 
+## Documentation
+
+Full guides, consumer/producer recipes, middleware reference, and the web dashboard walkthrough are published at **[gems.marcosz.com.br/lepus](https://gems.marcosz.com.br/lepus/)** — part of the [marcosgz Ruby gem catalogue](https://gems.marcosz.com.br).
+
 ## Installation
 
 Add this line to your application's Gemfile:
@@ -572,6 +577,180 @@ plugin :lepus
 
 **Note**: The Puma plugin is only available if you are using Puma 6.x or higher.
 
+## Web UI Dashboard
+
+Lepus includes a built-in web dashboard that provides a real-time view of your message processing infrastructure. The dashboard allows you to monitor processes, queues, connections, and consumer performance.
+
+![Lepus web dashboard — overview of supervisors, workers, and recent activity](https://github.com/user-attachments/assets/a640fc43-2f53-4aa7-aede-f1464f3e6e03)
+
+### Starting the Web Dashboard
+
+You can start the web dashboard using the `lepus web` command:
+
+```bash
+bundle exec lepus web
+```
+
+The dashboard will be available at `http://localhost:9292` by default. You can customize the host and port:
+
+```bash
+bundle exec lepus web --port 3000 --host 127.0.0.1
+```
+
+### Web Dashboard Features
+
+The Lepus web dashboard provides:
+
+- **Process Monitoring**: View all running supervisors and workers with their PIDs, memory usage, and heartbeat status
+- **Queue Management**: Monitor queue statistics including message counts, consumer connections, and memory usage
+- **Connection Tracking**: View active RabbitMQ connections and their states
+- **Consumer Performance**: Track processed, rejected, and errored messages per consumer
+- **Real-time Updates**: Dashboard automatically refreshes to show current system state
+
+### Integrating with Rails
+
+To integrate the Lepus web dashboard into your Rails application, you can mount it as a Rack application in your routes:
+
+```ruby
+# config/routes.rb
+Rails.application.routes.draw do
+  # Your existing routes...
+
+  # Mount Lepus web dashboard (simple way)
+  mount Lepus::Web => "/lepus"
+end
+```
+
+You can also use the more explicit syntax:
+
+```ruby
+# config/routes.rb
+Rails.application.routes.draw do
+  # Your existing routes...
+
+  # Mount Lepus web dashboard (explicit way)
+  mount Lepus::Web::App.build => "/lepus"
+end
+```
+
+This will make the dashboard available at `http://your-app.com/lepus` in your Rails application.
+
+#### Process registry backend
+
+Lepus tracks running supervisors and workers in a **process registry**. Two
+backends are available:
+
+- `:file` (default for a core `require "lepus"`) — stores process data in a
+  local file under `/tmp`. Fast and dependency-free, but the file is only
+  visible to processes that share the same filesystem.
+- `:rabbitmq` — stores the same data in a dedicated RabbitMQ queue, so every
+  process connected to the same broker sees the same registry.
+
+**Requiring `lepus/web` automatically switches the default to `:rabbitmq`.**
+This is because the dashboard is almost always run in a separate process (and
+often a separate container) from the workers, and the `:file` backend cannot
+bridge that gap — you'd see an empty dashboard even with workers running. The
+dashboard still needs the RabbitMQ Management API for queue/connection data,
+but the registry is what lets it discover your workers.
+
+If you really want the file backend even with the dashboard loaded, set it
+explicitly after your `require`:
+
+```ruby
+# config/initializers/lepus.rb
+Lepus.configure do |config|
+  config.process_registry_backend = :file
+end
+```
+
+`Lepus::Web` is a plain Rack app, so authentication is applied by wrapping it
+in standard Rack middleware or by gating the mount with a real auth helper.
+Rails routing `constraints:` is **not** an authentication mechanism — a falsy
+constraint returns 404 and never prompts for credentials.
+
+HTTP Basic Auth (wrap the Rack app):
+
+```ruby
+# config/routes.rb
+require "rack/auth/basic"
+
+lepus_web = Rack::Builder.new do
+  use Rack::Auth::Basic, "Lepus Dashboard" do |username, password|
+    ActiveSupport::SecurityUtils.secure_compare(username, ENV.fetch("LEPUS_USER")) &
+      ActiveSupport::SecurityUtils.secure_compare(password, ENV.fetch("LEPUS_PASSWORD"))
+  end
+  run Lepus::Web
+end
+
+Rails.application.routes.draw do
+  mount lepus_web => "/lepus"
+end
+```
+
+Devise (only admins can see the dashboard):
+
+```ruby
+# config/routes.rb
+Rails.application.routes.draw do
+  authenticate :user, ->(u) { u.admin? } do
+    mount Lepus::Web => "/lepus"
+  end
+end
+```
+
+## Prometheus metrics (optional)
+
+Lepus ships an optional integration with
+[`prometheus_exporter`](https://github.com/discourse/prometheus_exporter). It is
+not a required dependency and is not auto-loaded — add the gem to your `Gemfile`
+and require `lepus/prometheus` explicitly from the Lepus process you want to
+instrument.
+
+```ruby
+# Gemfile
+gem "prometheus_exporter"
+```
+
+```ruby
+# e.g. config/initializers/lepus.rb, or at the top of your consumer boot script
+require "lepus/prometheus"
+
+# Optional: poll the RabbitMQ Management API for queue-level gauges
+# from a single process (typically the supervisor).
+Lepus::Prometheus.watch_queues(interval: 30)
+```
+
+Requiring `lepus/prometheus` installs the necessary hooks into
+`Lepus::Consumers::Handler` (delivery counters and latency) and
+`Lepus::Consumers::Worker` (process RSS gauge), and subscribes to
+`publish.lepus` notifications (publish counters). Metrics are sent over TCP to
+the `PrometheusExporter::Client.default` client.
+
+On the exporter side, load the bundled type collector so the server knows how
+to turn Lepus payloads into Prometheus metrics:
+
+```bash
+bundle exec prometheus_exporter -a lepus/prometheus/collector
+```
+
+Point Prometheus at the exporter (default port `9394`) and import
+[`examples/grafana-dashboard.json`](examples/grafana-dashboard.json) into
+Grafana. The dashboard covers every metric exposed by the collector.
+
+### Exposed metrics
+
+| Metric                                    | Type      | Labels                            | Source                                     |
+|-------------------------------------------|-----------|-----------------------------------|--------------------------------------------|
+| `lepus_messages_processed_total`          | counter   | `consumer`, `queue`, `result`     | `Handler#process_delivery`                 |
+| `lepus_delivery_duration_seconds`         | histogram | `consumer`, `queue`               | `Handler#process_delivery`                 |
+| `lepus_messages_published_total`          | counter   | `exchange`, `routing_key`         | `publish.lepus` notification               |
+| `lepus_publish_duration_seconds`          | histogram | `exchange`, `routing_key`         | `publish.lepus` notification               |
+| `lepus_process_rss_memory_bytes`          | gauge     | `kind`, `name`, `pid`             | `Worker#heartbeat`                         |
+| `lepus_queue_messages`                    | gauge     | `name`                            | `watch_queues` via management API          |
+| `lepus_queue_messages_ready`              | gauge     | `name`                            | `watch_queues` via management API          |
+| `lepus_queue_messages_unacknowledged`     | gauge     | `name`                            | `watch_queues` via management API          |
+| `lepus_queue_consumers`                   | gauge     | `name`                            | `watch_queues` via management API          |
+| `lepus_queue_memory_bytes`                | gauge     | `name`                            | `watch_queues` via management API          |
 
 ## Development
 

data/config.ru

@@ -0,0 +1,14 @@
+# frozen_string_literal: true
+
+$LOAD_PATH.unshift File.expand_path("lib", __dir__)
+
+require "lepus"
+require "lepus/web"
+
+# Start web services for real data
+Lepus::Web.start
+
+# Graceful shutdown
+at_exit { Lepus::Web.stop }
+
+run Lepus::Web::App.build

data/docs/README.md

@@ -0,0 +1,80 @@
+# lepus
+
+RabbitMQ-backed producer/consumer framework for Ruby — with a supervisor, middleware chains, a CLI, and a live web dashboard.
+
+Think Sidekiq or SolidQueue, but on top of RabbitMQ rather than Redis or a database. Lepus handles the operational concerns (process supervision, graceful shutdown, connection pooling, signal handling, per-worker pools) so your application code can stay focused on "what does this message do".
+
+## Contents
+
+- [Getting started](getting-started.md) — install, define your first consumer and producer, run them
+- [Configuration](configuration.md) — the full `Lepus.configure` DSL
+- [Consumers](consumers.md) — queue bindings, lifecycle, result codes, retries
+- [Producers](producers.md) — exchanges, publishing, enable/disable hooks
+- [Middleware](middleware.md) — built-in middlewares and how to write your own
+- [CLI](cli.md) — `lepus start`, `lepus web`
+- [Supervisor](supervisor.md) — process model, signals, graceful shutdown
+- [Web dashboard](web.md) — the monitoring UI
+- [Testing](testing.md) — testing consumers and producers
+- [Rails integration](rails.md) — Railtie, executor wrapping, Puma plugin
+
+## Install
+
+```ruby
+# Gemfile
+gem 'lepus'
+```
+
+## One-minute tour
+
+```ruby
+# config/initializers/lepus.rb
+Lepus.configure do |config|
+  config.rabbitmq_url   = ENV.fetch('RABBITMQ_URL', 'amqp://guest:guest@localhost:5672')
+  config.connection_name = 'my-service'
+end
+
+# app/consumers/orders_consumer.rb
+class OrdersConsumer < Lepus::Consumer
+  configure(
+    queue:       'orders',
+    exchange:    { name: 'orders', type: :topic, durable: true },
+    routing_key: ['order.*']
+  )
+
+  use :json, symbolize_keys: true
+  use :max_retry, retries: 5
+
+  def perform(message)
+    Order.create!(message.payload)
+    :ack
+  end
+end
+
+# app/producers/orders_producer.rb
+class OrdersProducer < Lepus::Producer
+  configure(exchange: { name: 'orders', type: :topic, durable: true })
+  use :json
+  use :correlation_id
+end
+```
+
+Run the consumer:
+
+```bash
+bundle exec lepus start OrdersConsumer
+```
+
+Publish from anywhere in your app:
+
+```ruby
+OrdersProducer.publish({ order_id: 42, total: 99.99 }, routing_key: 'order.created')
+```
+
+## Version & dependencies
+
+- Ruby: `>= 2.7.0`
+- Runtime: `bunny`, `thor`, `zeitwerk`, `concurrent-ruby`, `multi_json`
+
+## License
+
+MIT.

data/docs/cli.md

@@ -0,0 +1,108 @@
+# CLI
+
+The `lepus` executable is the main operational entry point. It's a Thor-based CLI with two subcommands.
+
+```bash
+bundle exec lepus <command> [args] [options]
+```
+
+## `lepus start`
+
+Boot the supervisor and run the specified consumers.
+
+```bash
+bundle exec lepus start [CONSUMER_CLASSES...] [options]
+```
+
+If no classes are specified **and** `config.consumers_directory` is set, every consumer class discovered in that directory is started.
+
+### Options
+
+| Flag | Default | Purpose |
+|------|---------|---------|
+| `--debug` | `false` | Set logger level to DEBUG. |
+| `--logfile PATH` | stdout | Write logs to a file. |
+| `--pidfile PATH` | none | Write the supervisor PID to a file. |
+| `--require_file PATH`, `-r PATH` | none | Require this file before starting (e.g. `config/environment.rb`). |
+
+### Examples
+
+```bash
+# One consumer, no framework auto-load
+bundle exec lepus start OrdersConsumer
+
+# Multiple consumers, Rails env loaded first
+bundle exec lepus start OrdersConsumer PaymentsConsumer \
+  --require_file config/environment.rb
+
+# Auto-load everything under config.consumers_directory
+bundle exec lepus start
+
+# Debug mode + pidfile (typical for a systemd unit)
+bundle exec lepus start OrdersConsumer \
+  --require_file config/environment.rb \
+  --pidfile /var/run/lepus.pid \
+  --logfile /var/log/lepus.log \
+  --debug
+```
+
+### What happens at start
+
+1. Load `--require_file` if given.
+2. Resolve consumer classes (CLI args or auto-discovered).
+3. Group consumers by their `process.name` (defaulting to `:default`).
+4. Fork one worker subprocess per group. Parent becomes the supervisor.
+5. Each worker opens a Bunny channel, declares queues/exchanges/bindings, and starts consuming with its configured thread pool.
+6. Supervisor monitors children via pipes and restarts any that crash.
+
+See [supervisor.md](supervisor.md) for the full lifecycle.
+
+## `lepus web`
+
+Run the web dashboard.
+
+```bash
+bundle exec lepus web [options]
+```
+
+### Options
+
+| Flag | Default | Purpose |
+|------|---------|---------|
+| `--port PORT`, `-p PORT` | `9292` | Port to bind. |
+| `--host HOST`, `-o HOST` | `0.0.0.0` | Host to bind. |
+
+### Example
+
+```bash
+bundle exec lepus web --port 9292 --host 0.0.0.0
+```
+
+Visit http://localhost:9292.
+
+The web UI reads from the process registry backend (`config.process_registry_backend`). For multi-host visibility, set the backend to `:rabbitmq`. See [web.md](web.md).
+
+### Mounting in Rails
+
+Instead of running the CLI, you can mount the web app directly:
+
+```ruby
+# config/routes.rb
+require 'lepus/web'
+mount Lepus::Web::App, at: '/lepus'
+```
+
+See [rails.md](rails.md) for authorization patterns.
+
+## Exit codes
+
+- `0` — graceful shutdown (SIGTERM / SIGINT).
+- Non-zero — unrecoverable error at boot (config invalid, RabbitMQ unreachable, etc.).
+
+Workers that crash after boot are restarted by the supervisor; the supervisor itself only exits on an unrecoverable event or a requested shutdown.
+
+## Environment variables
+
+- `RABBITMQ_URL` — fallback for `config.rabbitmq_url` if not explicitly set.
+
+Everything else is configured via `Lepus.configure`. See [configuration.md](configuration.md).

data/docs/configuration.md

@@ -0,0 +1,171 @@
+# Configuration
+
+Lepus is configured via a single block:
+
+```ruby
+Lepus.configure do |config|
+  # ... see below ...
+end
+```
+
+The configuration is mutable at boot and should be set once. Values are read lazily — most only matter at worker-start time.
+
+## Connection
+
+```ruby
+Lepus.configure do |config|
+  config.rabbitmq_url                    = ENV.fetch('RABBITMQ_URL', 'amqp://guest:guest@localhost:5672')
+  config.connection_name                 = 'my-service'
+  config.recovery_attempts               = 10          # nil = infinite
+  config.recover_from_connection_close   = true
+end
+```
+
+| Key | Default | Purpose |
+|-----|---------|---------|
+| `rabbitmq_url` | `ENV['RABBITMQ_URL']` or `amqp://guest:guest@localhost:5672` | Connection string |
+| `connection_name` | gem-generated | Shown in RabbitMQ management UI — set to your service name |
+| `recovery_attempts` | `10` | Max automatic reconnects; `nil` for infinite |
+| `recover_from_connection_close` | `true` | Auto-recover after a clean close |
+
+## Application metadata
+
+```ruby
+config.application_name       = 'orders-service'
+config.management_api_url     = 'http://rabbitmq:15672'   # only for :rabbitmq registry backend
+```
+
+Shown in the web dashboard. `management_api_url` is only required if `process_registry_backend = :rabbitmq`.
+
+## Consumer discovery
+
+```ruby
+config.consumers_directory = 'app/consumers'
+```
+
+When present, `lepus start` with no class arguments auto-loads every class under this directory and runs all configured consumers.
+
+## Worker pools
+
+A **worker** is a subprocess the supervisor forks. Consumers are grouped into workers by their `process.name`.
+
+```ruby
+config.worker(:default) do |w|
+  w.pool_size    = 5       # threads per worker
+  w.pool_timeout = 10.0    # seconds before yielding to another task
+
+  w.before_fork { ActiveRecord::Base.connection_handler.clear_all_connections! }
+  w.after_fork  { ActiveRecord::Base.establish_connection }
+end
+
+config.worker(:high_priority, pool_size: 10)
+```
+
+| Key | Default | Purpose |
+|-----|---------|---------|
+| `pool_size` | `2` | Max threads per worker process |
+| `pool_timeout` | `10.0` | Seconds threads wait on queue checkout |
+| `before_fork` | no-op | Block run in parent before fork — close sockets, drop DB connections, etc. |
+| `after_fork` | no-op | Block run in child after fork — reconnect, reseed RNG, etc. |
+
+Assign consumers to a named worker via their own `configure(process: { name: :high_priority })` (see [consumers.md](consumers.md)).
+
+## Producer pool
+
+Producers share a connection pool:
+
+```ruby
+config.producer do |p|
+  p.pool_size    = 5
+  p.pool_timeout = 5.0
+end
+```
+
+## Global middleware
+
+Middleware chains run around every message published or consumed, in addition to any per-producer or per-consumer `use` calls.
+
+```ruby
+config.producer_middlewares do |chain|
+  chain.use :instrumentation
+  chain.use :correlation_id
+end
+
+config.consumer_middlewares do |chain|
+  chain.use :json, symbolize_keys: true
+end
+```
+
+See [middleware.md](middleware.md).
+
+## Process registry
+
+The supervisor keeps a registry of running processes, visible to the web dashboard.
+
+```ruby
+config.process_registry_backend     = :file      # or :rabbitmq
+config.process_heartbeat_interval   = 60         # seconds
+config.process_alive_threshold      = 5 * 60     # seconds
+```
+
+- `:file` — metadata in local files under `/tmp/lepus/...`. Single-host.
+- `:rabbitmq` — metadata in RabbitMQ itself; multi-host visibility in the web dashboard.
+
+## Rails integration
+
+When Rails is loaded, a Railtie wires:
+
+- `config.app_executor = Rails.application.executor` automatically
+- `config.logger = Rails.logger`
+
+You can override either:
+
+```ruby
+config.app_executor = nil             # disable executor wrapping
+config.logger       = MyLogger.new
+```
+
+See [rails.md](rails.md).
+
+## Threading error handler
+
+```ruby
+config.on_thread_error = ->(exception) {
+  Rails.error.report(exception)
+  Honeybadger.notify(exception)
+}
+```
+
+Called when a worker thread raises. Does not stop the worker.
+
+## Logger
+
+```ruby
+config.logger = Logger.new($stdout)
+```
+
+Default: `Logger.new($stdout)` (or `Rails.logger` if Rails is present). The `--debug` CLI flag sets the level to DEBUG.
+
+## Full example
+
+```ruby
+Lepus.configure do |config|
+  config.rabbitmq_url    = ENV.fetch('RABBITMQ_URL')
+  config.connection_name = 'orders-service'
+  config.application_name = 'orders-service'
+  config.consumers_directory = 'app/consumers'
+
+  config.worker(:default) do |w|
+    w.pool_size = 5
+    w.before_fork { ActiveRecord::Base.connection_handler.clear_all_connections! }
+    w.after_fork  { ActiveRecord::Base.establish_connection }
+  end
+
+  config.producer_middlewares do |chain|
+    chain.use :instrumentation
+    chain.use :correlation_id
+  end
+
+  config.on_thread_error = ->(exc) { Honeybadger.notify(exc) }
+end
+```