cosmonats 0.6.0 → 0.8.0

data/README.md

@@ -10,7 +10,7 @@ Background jobs + real-time event streaming for Ruby — unified, in one gem, ba
 [![Gem Version](https://badge.fury.io/rb/cosmonats.svg)](https://rubygems.org/gems/cosmonats)
 [![Downloads](https://img.shields.io/gem/dt/cosmonats.svg)](https://rubygems.org/gems/cosmonats)
 [![Ruby](https://img.shields.io/badge/ruby-%3E%3D%203.1-red)](https://www.ruby-lang.org)
-[![License: LGPL v3](https://img.shields.io/badge/License-LGPL%20v3-blue.svg)](LICENSE.txt)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE.txt)
 [![Build Status](https://github.com/bitsbeam/cosmonats/actions/workflows/ci.yml/badge.svg)](https://github.com/bitsbeam/cosmonats/actions)
 
 *Battle-tested in production. Tens of millions of jobs processed and counting.*
@@ -77,6 +77,7 @@ bundle exec cosmo -C config/cosmo.yml -c 20 streams # Streams only
   - [Concurrency Limiting](#concurrency-limiting)
   - [Custom Serializers](#custom-serializers)
   - [Error Handling](#error-handling)
+  - [Middleware](#middleware)
   - [Testing](#testing)
   - [Integrations](#integrations)
 - [CLI Reference](#-cli-reference)
@@ -166,7 +167,20 @@ nothing else to run.
 gem "cosmonats"
 ```
 
-**Requirements:** Ruby ≥ 3.1, NATS Server ([install guide](https://docs.nats.io/running-a-nats-service/introduction/installation))
+**Requirements:** Ruby ≥ 3.1, NATS Server ≥ 2.14 with JetStream enabled ([install guide](https://docs.nats.io/running-a-nats-service/introduction/installation))
+
+**Optional gems** — not installed with `cosmonats`, add them only for the features you use:
+
+| Feature                                       | Gems              |
+|-----------------------------------------------|-------------------|
+| Monitoring UI (`Cosmo::Web`)                  | `rack`            |
+| Health check HTTP server (`/health`, `/ping`) | `rack`, `webrick` |
+
+```ruby
+# Gemfile
+gem "rack", "~> 3.0"
+gem "webrick", "~> 1.9"
+```
 
 Spin up NATS instantly with Docker — one command, that's it:
 ```bash
@@ -195,33 +209,39 @@ mount Cosmo::Web => "/cosmo"
 map "/cosmo" { run Cosmo::Web }
 ```
 
+> [!WARNING]
+> `Cosmo::Web` ships with **no authentication and no CSRF protection**, yet it exposes destructive actions
+> (retrying and deleting jobs, pausing streams, running and deleting crons). Never expose it unprotected.
 
-## 🚀 Quick Start
+```ruby
+# Rails, with Devise
+authenticate :user, ->(user) { user.admin? } do
+  mount Cosmo::Web => "/cosmo"
+end
 
-### 1. Create `config/cosmo.yml`
+# Any Rack app (config.ru), with HTTP Basic auth
+map "/cosmo" do
+  use Rack::Auth::Basic, "Cosmo" do |user, password|
+    Rack::Utils.secure_compare(user, ENV.fetch("COSMO_WEB_USER")) &
+      Rack::Utils.secure_compare(password, ENV.fetch("COSMO_WEB_PASSWORD"))
+  end
+  run Cosmo::Web
+end
+```
 
-```yaml
-concurrency: 5                     # Number of worker threads
 
-consumers:                         # Declare consumer groups for streams, things that pull messages and process them
-  jobs:                            # Consumer configs for jobs (or streams)
-    default:                       # Stream name
-      ack_policy: explicit         # Acknowledgment required for each message can be explicit, none, or all
-      max_deliver: 30              # Max retry attempts before sending to a dead stream. Safety ceiling only, keep it above every job class's own retry
-      max_ack_pending: 10          # Max messages waiting for ack, if exceeded, the server will stop delivering new messages until some are acked
-      ack_wait: 15                 # Seconds to wait for ack before redelivering
-      subject: jobs.%{name}.>      # Subject pattern for this consumer, %{name} replaced with stream name, becomes `jobs.default.>`
+## 🚀 Quick Start
 
-setup:                             # Initial stream creation only `cosmo -S`
-  jobs:                            # Stream configs for jobs (or streams)
-    default:                       # Stream name
-      storage: file                # Storage type (file or memory)
-      retention: workqueue         # Retention policy (limits, interest, workqueue). workqueue - deletes acked/nacked, limits - append only
-      subjects: ["jobs.%{name}.>"] # Subject pattern for this stream, %{name} replaced with stream name
-      allow_direct: true           # Allow direct messages to stream (required for web UI)
+### 1. Configure (optional)
+
+Nothing is required: without `config/cosmo.yml`, Cosmo runs one `default` job stream plus its own service streams.
+To change anything, write Cosmo's defaults into your project and edit them:
+
+```bash
+bundle exec cosmo --init # writes config/cosmo.yml
 ```
 
-### 2. Create streams in NATS (one-time), grabs config from setup section of `config/cosmo.yml`
+### 2. Create streams in NATS (one-time, idempotent)
 
 ```bash
 bundle exec cosmo -S
@@ -280,6 +300,16 @@ ReportJob.perform_at(Time.parse("2026-01-25 10:00"), 42) # Scheduled
 ReportJob.perform_sync(42)                               # Inline, no NATS (great for tests)
 ```
 
+Inside `perform`, the job instance knows about the message it came from:
+
+| Accessor       | Value                                                                                   |
+|----------------|-----------------------------------------------------------------------------------------|
+| `jid`          | Job id, returned by `perform_async`                                                     |
+| `attempt`      | 1-based delivery attempt                                                                |
+| `enqueued_at`  | `Time` the job was enqueued; for a cron job, when NATS fired the schedule               |
+| `scheduled_by` | Subject of the cron schedule that fired the job, `nil` for jobs enqueued any other way  |
+| `batch_id`     | Id of the [batch](#batches) the job belongs to, `nil` outside a batch                   |
+
 ### Streams
 
 ```ruby
@@ -325,97 +355,87 @@ message.term                         # Permanent failure, no retry
 **NATS subjects** follow a dot-separated hierarchy (`events.clicks.homepage`).
 The `>` wildcard matches everything after that prefix. Think of subjects as topic names — flexible routing with no extra configuration.
 
-**Full `config/cosmo.yml` example:**
+**`config/cosmo.yml` is optional.** Cosmo ships its defaults in
+[`lib/cosmo/config/cosmo.yml`](lib/cosmo/config/cosmo.yml) (`cosmo --init` copies it into your project), and your
+file is deep-merged over them: keep only what you change. The one list that is not merged is your job streams —
+listing any under `setup.jobs` replaces the built-in `default` stream and its consumer:
 ```yaml
-timeout: 25                 # Shutdown timeout in seconds
-concurrency: &concurrency 1 # Number of worker threads
-max_retries: &max_retries 3 # Default max retries
-batch_expiry: 259200         # Seconds before a Batch's tracking data expires (default: 3 days)
+concurrency: 10
 
 stream_config: &stream_config
-  storage: file         # storage type (file or memory)
-  retention: workqueue  # retention policy (limits, interest, workqueue)
-  duplicate_window: 120 # time window for duplicate message detection in seconds
-  discard: old          # discard new messages when stream is full (discard new or old)
-  allow_direct: true    # allow direct messages to stream, required for web UI
-  subjects:
-    - jobs.%{name}.>    # subject pattern for stream, %{name} will be replaced with stream name
+  storage: file
+  retention: workqueue
+  duplicate_window: 120 # seconds
+  discard: new
+  allow_direct: true    # required by the web UI
+  subjects: ["jobs.%{name}.>"]
 
 consumer_config: &consumer_config
-  ack_policy: explicit    # ack policy (explicit, none, all), each individual message must be acknowledged
-  max_deliver: 30         # maximum number of times a message will be delivered before it's considered failed. keep it above every job class's own retry; a job that still exceeds it is capped and dead-lettered early with a warning
-  max_ack_pending: 20     # maximum number of messages with pending ack for this consumer
-  ack_wait: 60            # time in seconds to wait for an ack before redelivering the message
-  subject: jobs.%{name}.> # subject pattern for consumer, %{name} will be replaced with stream name
-
-consumers:
-  jobs:
-    critical:
-      <<: *consumer_config
-      priority: 50
-    high:
-      <<: *consumer_config
-      priority: 30
-    default:
-      <<: *consumer_config
-      priority: 15
-    low:
-      <<: *consumer_config
-      priority: 5
-    scheduled:
-      <<: *consumer_config
-      max_deliver: 5
-      max_ack_pending: 100
-      ack_wait: 10
+  ack_policy: explicit
+  max_deliver: 30       # keep it above every job class's retry
+  max_ack_pending: 20
+  ack_wait: 60          # seconds
+  subject: jobs.%{name}.>
 
 setup:
   jobs:
-    critical:
-      <<: *stream_config
-      description: Very critical priority jobs
-    high:
-      <<: *stream_config
-      description: Higher priority jobs
-    default:
-      <<: *stream_config
-      description: Default priority jobs
-    low:
-      <<: *stream_config
-      description: Lower priority jobs
-    scheduled:
-      <<: *stream_config
-      description: Scheduled jobs
-    dead:
-      <<: *stream_config
+    critical: { <<: *stream_config }
+    default: { <<: *stream_config }
+  streams:              # Cosmo::Stream processors are always yours to declare
+    events:
+      storage: file
       retention: limits
-      max_msgs: 10000
-      max_age: 604800 # 7d
-      description: Broken jobs (DLQ)
-
-development:
-  verbose: false
-  concurrency: *concurrency
+      max_age: 86400
+      subjects: ["events.>"]
 
-staging:
-  verbose: true
-  concurrency: 3
+consumers:
+  jobs:
+    critical: { <<: *consumer_config, priority: 50 }
+    default: { <<: *consumer_config, priority: 15 }
+```
+`max_age`, `duplicate_window`, and `ack_wait` are in seconds. Without `setup.jobs`, `consumers.jobs.default` still
+tunes the built-in consumer (e.g. just `ack_wait: 300`).
 
-production:
-  concurrency: 3
+**Service streams** — the `scheduled` stream (delayed jobs, crons), the `dead` stream (dead letters), and the stats
+counters and KV buckets are Cosmo's own: they are not part of `cosmo.yml`, and `cosmo --setup` creates them next to
+yours. Tune them in `Cosmo.configure`:
+```ruby
+Cosmo.configure do |config|
+  config.replicas = 3                # every service stream and bucket, for a NATS cluster (default 1)
+  config.dead.max_age = "14d"        # dead-letter retention: max_age, max_msgs (10_000), max_bytes (-1); default 7d
+  config.dead.enabled = false        # drop jobs that give up instead of parking them
+  config.scheduled.enabled = false   # no scheduled stream or scheduler: perform_in/perform_at and crons raise
+  config.batches.expiry = "1d"       # how long batch tracking data lives (default 3d)
+  config.metrics.retention = "90d"   # how long the Metrics tab's per-day numbers are kept (default 30d)
+  config.metrics.enabled = false     # record nothing and hide the Metrics tab
+end
 ```
+Turning `scheduled` or `dead` off also hides their pages in the web UI. Stats can't be turned off — the web UI
+depends on them.
 
-**Programmatic:**
+**Programmatic:** `cosmo.yml` holds streams, consumers, and process settings; `Cosmo.configure` holds what YAML can't
+express (loggers, middleware) and Cosmo's service settings. Command-line flags (`-c`, `-t`, `-p`) win over the file.
 ```ruby
-Cosmo::Config.set(:concurrency, 20)
-Cosmo::Config.set(:setup, :streams, :custom, { storage: "file", subjects: ["custom.>"] })
+Cosmo.configure do |config|
+  config.logger = Rails.logger # any ::Logger; defaults to stdout
+  config.log_level = :debug    # COSMO_LOG_LEVEL, when set, wins
+end
 ```
+Cosmo leaves a custom logger's formatter alone, so its lines lack Cosmo's `jid`/`elapsed` context unless you set
+`logger.formatter = Cosmo::Logger::SimpleFormatter.new`. Trace output is dropped for loggers without a `trace` level.
 
 **Environment variables:**
-```bash
-export NATS_URL=nats://localhost:4222
-export COSMO_JOBS_FETCH_TIMEOUT=0.1
-export COSMO_STREAMS_FETCH_TIMEOUT=0.1
-```
+
+| Variable                   | Default                              | Description                                                                         |
+|----------------------------|--------------------------------------|-------------------------------------------------------------------------------------|
+| `NATS_URL`                 | `nats://localhost:4222`              | NATS server(s) to connect to                                                        |
+| `COSMO_LOG_LEVEL`          | `info`                               | `trace`, `debug`, `info`, `warn`, `error` or `fatal`                                |
+| `COSMO_CLIENT_NAME`        | `cosmo-<program>-<host>-<pid>`       | NATS connection name, shown in `nats server report connections` and `/connz`        |
+| `COSMO_JS_TIMEOUT`         | `5`                                  | JetStream API timeout in seconds; raise it for the web UI on a busy server          |
+| `COSMO_CONNECT_TIMEOUT`    | `2`                                  | NATS TCP connect timeout in seconds                                                 |
+| `COSMO_JOBS_STREAMS`       | all configured                       | Comma-separated job streams to subscribe to, when `--streams`/`--stream` isn't given |
+| `COSMO_JOBS_FETCH_TIMEOUT` | `0.1`                                | Seconds a jobs worker waits on each fetch                                           |
+| `COSMO_WEB_POLL_INTERVAL`  | `5`                                  | Web UI auto-refresh interval in seconds                                             |
 
 
 ## 🔧 Advanced Usage
@@ -423,10 +443,16 @@ export COSMO_STREAMS_FETCH_TIMEOUT=0.1
 ### Cron
 
 Recurring jobs, without a separate scheduler process. A schedule is just a message parked in the
-job's own NATS stream (requires NATS Server 2.14+) — NATS fires it on the cron expression, and it
-lands back in the stream as a regular job. Deploy it once; whatever's in NATS is exactly what runs
-and exactly what shows up in the web UI's **Crons** tab, where each entry can be inspected, run
-immediately, or deleted.
+`scheduled` stream (requires NATS Server 2.14+) — NATS fires it on the cron expression, and the
+worker's scheduler dispatches the job on to the stream that runs it, the same path a delayed job
+takes. Deploy it once; whatever's in NATS is exactly what runs and exactly what shows up in the web
+UI's **Crons** tab, where each entry can be inspected, run immediately, or deleted.
+
+Every schedule lives in `scheduled` because NATS only lets a schedule fire at a subject its own
+stream covers, and refuses `discard: new` on any stream with scheduling enabled. Keeping schedules
+in one stream leaves every stream that actually runs jobs free to choose its own discard policy.
+The trade is that cron firings are dispatched by the scheduler, so at least one worker has to be
+running without `--no-scheduler`.
 
 Declare schedules right in `config/cosmo.yml`:
 
@@ -516,7 +542,7 @@ end
   end
   ```
 - Batch tracking data (pending counts, callbacks, results) expires automatically after
-  `Config[:batch_expiry]` seconds (default: 3 days).
+  `config.batches.expiry` (default: 3 days), set in `Cosmo.configure`.
 - Open and finished batches — with pending/succeeded/failed counts — are listed live in the web UI's **Batches** tab.
 - Only `Cosmo::Job`-based jobs are tracked; the ActiveJob adapter doesn't currently participate in
   batches (see [`docs/active_job.md`](docs/active_job.md)).
@@ -592,6 +618,69 @@ If the proc returns something non-numeric/non-positive, or raises, the default b
 instead. Note: if this job class also sets `limit: { concurrency: ... }` (see above), `count`
 includes deliveries that were turned away for lack of a free slot, not just failed attempts.
 
+**Error handlers** receive every error Cosmo rescues — each failed job attempt, a failing stream batch, fetch and
+scheduler errors, and rejected messages — so one hook reports them all:
+```ruby
+Cosmo.configure do |config|
+  config.error_handlers << ->(error, context) { Honeybadger.notify(error, context: context) }
+end
+```
+`context` is a Hash with a `:source` (`:job`, `:stream`, `:fetch`, `:scheduler`, `:reject`, `:retry_in`, `:limit`)
+and what is known there: for a job its payload (`jid`, `class`, `args`, …), `stream`, `subject`, and `attempt`; for
+a stream batch its `stream`, `processor`, and `size`. A handler that raises is logged and skipped.
+
+### Middleware
+
+Every job execution on a worker runs through the server middleware chain. A middleware gets the
+job instance, its parsed payload, and the NATS message, and yields to run the rest of the chain:
+```ruby
+class TimingMiddleware
+  def initialize(threshold:)
+    @threshold = threshold
+  end
+
+  def call(job, data, message)
+    started = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+    yield
+  ensure
+    elapsed = Process.clock_gettime(Process::CLOCK_MONOTONIC) - started
+    job.logger.warn("#{data[:class]} took #{elapsed.round(2)}s on #{message.metadata.stream}") if elapsed > @threshold
+  end
+end
+
+Cosmo.configure do |config|
+  config.server_middleware do |chain|
+    chain.add TimingMiddleware, threshold: 5
+  end
+end
+```
+Register middleware at boot, before workers start (e.g. in a Rails initializer; loading `cosmo.yml` afterwards keeps
+it). The chain starts with Cosmo's required middleware: `Cosmo::Middleware::Limit` (a job class's `limit:`
+option), `Cosmo::Middleware::Busy` (the Web UI's Busy page), `Cosmo::Middleware::Totals` (processed/failed counts,
+one per execution), and `Cosmo::Middleware::Metrics` (the Metrics tab). `add` appends innermost, after them; `prepend`, `insert_before`, `insert_after`, and `remove`
+reorder the chain, so moving or removing the built-ins is at your own risk.
+A middleware that raises sends the job through the usual retry/DLQ path, and one that doesn't yield skips the job
+and acks it. To put a job back for later instead, raise `Cosmo::Job::Requeue.new(delay_in_seconds)`: the job is
+redelivered without failing, though the redelivery still counts toward its `retry` and `max_deliver`.
+A fresh instance is built for every job. `perform_sync` doesn't run the chain.
+
+The **client** chain runs where a job is enqueued (`perform_async`, `perform_in`, `perform_at`, and the ActiveJob
+adapter), around the publish. It starts empty, and a middleware gets the job class name, the payload Hash that will be
+published, and the stream the job runs on:
+```ruby
+class RequestIdMiddleware
+  def call(_job_class, payload, _stream)
+    payload[:request_id] = Current.request_id
+    yield
+  end
+end
+
+Cosmo.configure do |config|
+  config.client_middleware { |chain| chain.add RequestIdMiddleware }
+end
+```
+Not yielding stops the publish: `perform_async` returns `nil`, and a batch the job was joining doesn't wait for it.
+
 ### Testing
 
 ```ruby
@@ -631,29 +720,113 @@ ActiveJob::Base.queue_adapter = Cosmo::ActiveJobAdapter::Adapter.new
 
 **Sentry:**
 ```ruby
-require "cosmo/sentry/auto"
+require "cosmo/middleware/sentry"
+
+Cosmo.configure do |config|
+  config.server_middleware { |c| c.add Cosmo::Middleware::Sentry }
+  config.error_handlers << Cosmo::Middleware::Sentry::ERROR_HANDLER
+end
 ```
-Wraps every job execution in a Sentry transaction (`queue.cosmonats`) and captures unhandled
-exceptions with the job's id, stream, subject, and retry count attached as context — no other
-setup beyond having `sentry-ruby` initialized.
+The [middleware](#middleware) wraps every job execution in a Sentry transaction (`queue.cosmonats`), and the
+[error handler](#error-handling) captures every error Cosmo rescues — failed jobs, stream batches, fetches, the
+scheduler — with its context under `cosmonats`. No other setup beyond having `sentry-ruby` initialized.
 
 
 ## 🖥️ CLI Reference
 
 ```bash
-cosmo -C config/cosmo.yml --setup                  # Create streams in NATS (idempotent)
-cosmo -C config/cosmo.yml -c 20 -r ./app/jobs jobs # Jobs only
-cosmo -C config/cosmo.yml -c 20 streams            # Streams only
-cosmo -C config/cosmo.yml -c 20                    # Both
+cosmo -C config/cosmo.yml --setup                     # Create streams in NATS (idempotent)
+cosmo -C config/cosmo.yml -c 20 -r ./app/jobs jobs    # Jobs only
+cosmo -C config/cosmo.yml -c 20 streams               # Streams only
+cosmo -C config/cosmo.yml -c 20                       # Both
+cosmo -C config/cosmo.yml jobs --streams default,high # Jobs, limited to these streams
+cosmo -C config/cosmo.yml jobs --stream critical      # Jobs, limited to one stream
+cosmo -C config/cosmo.yml jobs --no-scheduler         # Jobs, without dispatching scheduled jobs
+COSMO_JOBS_STREAMS=default,high cosmo jobs            # Same filter, without touching the command line
+```
+
+Each selected stream gets its own durable consumer, named `consumer-<stream>`. Every process that subscribes to a stream
+pulls from that same consumer, so running a dedicated fleet per stream splits the work instead of duplicating it.
+The `scheduled` stream is a service stream: it is always dispatched, and naming it in `--streams` selects nothing, use
+`--no-scheduler` on workers that should not dispatch scheduled jobs. Unknown stream names abort the process.
+
+**Global flags** (before the command):
+
+| Flag                    | Description                          | Example               |
+|-------------------------|---------------------------------------|-----------------------|
+| `-C, --config PATH`     | Config file path                      | `-C config/cosmo.yml` |
+| `-c, --concurrency INT` | Worker threads                        | `-c 20`               |
+| `-r, --require PATH`    | Auto-require directory                | `-r ./app/jobs`       |
+| `-t, --timeout NUM`     | Shutdown timeout (sec)                | `-t 60`               |
+| `-p, --http-port INT`   | Serve `/health` and `/ping` on a port | `-p 9090`             |
+| `-S, --setup`           | Setup streams & sync cron, then exit  | `--setup`             |
+| `--init`                | Write `config/cosmo.yml` with the defaults, then exit | `--init`  |
+| `-v, --version`         | Print version and exit                | `--version`           |
+| `-h, --help`            | Show help and exit                    | `--help`               |
+
+**`jobs` command options:**
+
+| Flag                    | Description                                                       | Example                        |
+|-------------------------|--------------------------------------------------------------------|---------------------------------|
+| `--streams NAMES`       | Only subscribe to these job streams, instead of all configured ones | `--streams default,high`      |
+| `--stream NAME`         | Same as `--streams`, for a single stream                           | `--stream default`             |
+| `--[no-]scheduler`      | Dispatch scheduled jobs (default: yes)                             | `--no-scheduler`               |
+| `--subject NAME`        | Job's subject                                                      | `--subject jobs.default.foo`   |
+
+`--streams`/`--stream` also read from `COSMO_JOBS_STREAMS` (comma-separated) when neither flag is given.
+
+**`streams` command options** (`--stream`/`--subject`/`--consumer_name`/`--batch_size` apply only when running a single processor; ignored once `--processors` selects more than one):
+
+| Flag                     | Description                          | Example                         |
+|--------------------------|---------------------------------------|---------------------------------|
+| `--processors NAMES`     | Only run these stream processor classes | `--processors OrderProcessor` |
+| `--stream NAME`          | Stream name                          | `--stream orders`              |
+| `--subject NAME`         | Subject name                         | `--subject orders.created`     |
+| `--consumer_name NAME`   | Consumer name                        | `--consumer_name orders-consumer` |
+| `--batch_size NUM`       | Messages per fetch batch             | `--batch_size 50`               |
+
+
+### Signals
+
+| Signal           | Effect                                                                                     |
+|------------------|--------------------------------------------------------------------------------------------|
+| `INT`, `TERM`    | Graceful shutdown: stop fetching, wait up to `--timeout` seconds for in-flight work, exit  |
+| `TSTP`           | Quiet: stop fetching new work, keep running in-flight work                                 |
+| `CONT`           | Resume fetching after `TSTP`                                                               |
+| `USR1`           | Quiet, then exit once in-flight work drains                                                |
+| `TTIN`           | Log every thread's backtrace at `WARN`, tagged with the same `tid` as regular log lines     |
+
+```bash
+kill -TSTP <pid>  # before a deploy: let the old worker finish what it has
+kill -TTIN <pid>  # a worker looks stuck: see what each thread is doing
 ```
 
-| Flag                    | Description            | Example               |
-|-------------------------|------------------------|-----------------------|
-| `-C, --config PATH`     | Config file path       | `-C config/cosmo.yml` |
-| `-c, --concurrency INT` | Worker threads         | `-c 20`               |
-| `-r, --require PATH`    | Auto-require directory | `-r ./app/jobs`       |
-| `-t, --timeout NUM`     | Shutdown timeout (sec) | `-t 60`               |
-| `-S, --setup`           | Setup streams & exit   | `--setup`             |
+The worker's state (`running`, `quiet`, `stopping`) is shown on the web UI's Processes list.
+
+### Health check
+
+Pass `-p/--http-port` (or set `http.port` in the config) to start a small WEBrick HTTP server alongside the workers.
+It needs the optional `rack` and `webrick` gems (see [Installation](#-installation)); without a port they're never loaded.
+
+```yaml
+http:
+  port: 9090
+  host: 0.0.0.0 # default
+```
+
+`GET /health` returns `200` when processors are running and NATS is connected, `503` otherwise:
+
+```bash
+curl localhost:9090/health
+# {"status":"ok","checks":{"engine":true,"nats":true}}
+```
+
+`GET /ping` always returns `200 pong` without checking anything, handy as a liveness probe (use `/health` for readiness):
+
+```bash
+curl localhost:9090/ping
+# pong
+```
 
 
 ## 🚢 Deployment
@@ -725,9 +898,16 @@ sudo systemctl enable cosmo && sudo systemctl start cosmo
 
 **Web UI** — mount `Cosmo::Web` (see [Installation](#-installation)) for a live, htmx-powered dashboard:
 - **Jobs** — enqueued, scheduled, busy, and dead views, with per-job retry and delete
+- **Processes** — the Jobs dashboard lists every live worker (idle ones included): host, pid, IP, NATS connection
+  name, command line, subscribed streams, busy/concurrency, running/quiet/stopping state, memory, NATS RTT and
+  reconnects. Each worker beats into the `_cosmoprocesses` KV bucket every 10s, and a crashed worker drops out within 60s
 - **Streams** — per-stream state (messages, bytes, consumers) with pause/resume
 - **Crons** — every schedule deployed in NATS, with run-now and delete
 - **Batches** — open and finished batches with pending/succeeded/failed counts
+- **Metrics** — per job class over today, 7, or 30 days: runs, failures, average execution time (successful runs), and
+  average wait from enqueue to first delivery, plus a per-day chart of runs and average execution time. Recorded by
+  `Cosmo::Middleware::Metrics`, flushed by every worker's heartbeat into the internal `_cosmometrics` counters stream;
+  turn it off with `config.metrics.enabled = false`
 - Summary counters (processed / failed / busy / enqueued / retries / scheduled / dead) backed by a NATS KV counter, no separate metrics store needed
 
 **Structured logs:**

data/lib/cosmo/active_job/adapter.rb

@@ -32,8 +32,7 @@ module Cosmo
         options    = { stream: stream }.merge(cosmo_opts)
         options[:at] = timestamp if timestamp
 
-        data = Job::Data.new(Executor.name, [job.serialize], options)
-        Publisher.publish_job(data)
+        Job::Enqueuer.enqueue(Executor.name, [job.serialize], options)
       end
 
       # Returns Cosmo-specific options declared on the job class via

data/lib/cosmo/api/counter.rb

@@ -2,25 +2,28 @@
 
 module Cosmo
   module API
+    # Base for NATS counters (+Nats-Incr+) kept in a stream of their own. A subclass names it with +STREAM_NAME+ and
+    # +DESCRIPTION+ and may extend {.stream_config}; every instance counts under its namespace in that stream.
     class Counter
-      STREAM_NAME = "_cosmostats"
+      # Creates the counters stream, or brings an existing one up to {.stream_config}.
+      #
+      # @return [void]
+      def self.setup!
+        Client.instance.setup_stream(self::STREAM_NAME, stream_config)
+      end
 
-      def self.instance
-        @instance ||= new("jobs")
+      # Every stored counter message carries the running total, so only the last one per subject is kept.
+      #
+      # @return [Hash]
+      def self.stream_config
+        { subjects: ["#{self::STREAM_NAME}.>"], allow_msg_counter: true, allow_direct: true, max_msgs_per_subject: 1,
+          num_replicas: Config.replicas, description: self::DESCRIPTION }
       end
 
       def initialize(namespace)
         @namespace = namespace
       end
 
-      def with
-        result = yield
-        increment(:processed) if result == true
-        increment(:failed) if result == false
-      rescue Exception # rubocop:disable Lint/RescueException
-        increment(:failed)
-      end
-
       def increment(key, by: 1, msg_id: nil)
         publish(key, "+#{by}", msg_id: msg_id)
       end
@@ -32,12 +35,12 @@ module Cosmo
       alias decr decrement
 
       def reset(key)
-        client.purge(STREAM_NAME, subject(key))
+        client.purge(self.class::STREAM_NAME, subject(key))
       end
       alias purge reset
 
       def get(key)
-        raw = client.get_message(STREAM_NAME, direct: true, subject: subject(key))
+        raw = client.get_message(self.class::STREAM_NAME, direct: true, subject: subject(key))
         Utils::Json.parse(raw.data, default: { "val" => 0 })[:val].to_i
       rescue NATS::JetStream::Error::NotFound, NATS::JetStream::Error::ServiceUnavailable, NATS::IO::Timeout
         0
@@ -64,13 +67,13 @@ module Cosmo
           raise if rescued
 
           rescued = true
-          client.create_stream(STREAM_NAME, subjects: ["#{STREAM_NAME}.>"], allow_msg_counter: true, allow_direct: true, description: "Cosmo statistics")
+          client.create_stream(self.class::STREAM_NAME, self.class.stream_config)
           retry
         end
       end
 
       def subject(key)
-        "#{STREAM_NAME}.#{@namespace}.#{key}"
+        "#{self.class::STREAM_NAME}.#{@namespace}.#{key}"
       end
 
       def client

data/lib/cosmo/api/cron/entry.rb

@@ -54,12 +54,31 @@ module Cosmo
           parts.join(".")
         end
 
-        # Subject where NATS fires the generated job message.
-        # Must be a subject covered by the same stream.
+        # Subject where NATS fires the generated job message. A schedule may only target a
+        # subject its own stream covers, so every schedule fires into the scheduled stream and
+        # the scheduler forwards the job on to {#dispatch_subject}.
         def target_subject
+          "jobs.#{::Cosmo::Job::StreamFilter::SCHEDULED}.#{job_name}"
+        end
+
+        # Subject the scheduler dispatches the fired job to, on the stream that runs it.
+        def dispatch_subject
           "jobs.#{@stream}.#{job_name}"
         end
 
+        # Headers stored on the schedule template. NATS copies the +X-+ ones onto every message
+        # it fires, which is how the scheduler learns where the job belongs.
+        def schedule_headers
+          headers = {
+            "Nats-Schedule" => @expression,
+            "Nats-Schedule-Target" => target_subject,
+            "X-Stream" => @stream,
+            "X-Subject" => dispatch_subject
+          }
+          headers["Nats-Schedule-Time-Zone"] = @timezone if @timezone
+          headers
+        end
+
         def job_name
           @job_name ||= Utils::String.underscore(@class_name)
         end
@@ -73,7 +92,8 @@ module Cosmo
             args: @args,
             name: @name,
             schedule_subject: schedule_subject,
-            target_subject: target_subject
+            target_subject: target_subject,
+            dispatch_subject: dispatch_subject
           }.compact
         end