cosmonats 0.7.0 → 0.8.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 96e901f7365784bf2fb7087311112efb263f295eddbe25a96ff32de4ebfe0652
-  data.tar.gz: b1f7327b126f85e50222af5adf2172b8fab84aa051209e9c0e9b478254bcb8a8
+  metadata.gz: 685ae4ff6735c88c3605b58f86168ce4acc2509d7dd66c7335cdeab50fc9ffa4
+  data.tar.gz: b508d08c685c6d726c1e3c56c79432d4afc952384d04ae210d0d6d673caf6c01
 SHA512:
-  metadata.gz: 1ab5c6cc101edf8dbf8909f544ed9a9fd8603c76f073dad60c5251e086195518e07ec5a6815bcf382b05c7b80f67fb6994a4615be933b33c29103ea3082990af
-  data.tar.gz: 85b15998b6286a331558e847835f6a94cf0936adae88e70a18035335b2414096d583e4619b7b2e5f5c7e579117c0d3626dbb99e189084fc3cf7aedc25dfc4fba
+  metadata.gz: 074bed30d8889561ecf77fd90d5d599e3f3cc9f487df781e08d88135b71b8ab31826b77dad8420b830e1a00632f047c46fee139d6a4e9817c0ca414795e46575
+  data.tar.gz: 99d1169938570c778981447df1385d177ad5a432e355690498c8ddac8a1e4eef6f9eedeb593b1541dc2dd51472906827e80032dcd2b0e4484435d7dce1e67771

data/README.md

@@ -77,6 +77,7 @@ bundle exec cosmo -C config/cosmo.yml -c 20 streams # Streams only
   - [Concurrency Limiting](#concurrency-limiting)
   - [Custom Serializers](#custom-serializers)
   - [Error Handling](#error-handling)
+  - [Middleware](#middleware)
   - [Testing](#testing)
   - [Integrations](#integrations)
 - [CLI Reference](#-cli-reference)
@@ -166,7 +167,7 @@ nothing else to run.
 gem "cosmonats"
 ```
 
-**Requirements:** Ruby ≥ 3.1, NATS Server ([install guide](https://docs.nats.io/running-a-nats-service/introduction/installation))
+**Requirements:** Ruby ≥ 3.1, NATS Server ≥ 2.14 with JetStream enabled ([install guide](https://docs.nats.io/running-a-nats-service/introduction/installation))
 
 **Optional gems** — not installed with `cosmonats`, add them only for the features you use:
 
@@ -208,33 +209,39 @@ mount Cosmo::Web => "/cosmo"
 map "/cosmo" { run Cosmo::Web }
 ```
 
+> [!WARNING]
+> `Cosmo::Web` ships with **no authentication and no CSRF protection**, yet it exposes destructive actions
+> (retrying and deleting jobs, pausing streams, running and deleting crons). Never expose it unprotected.
 
-## 🚀 Quick Start
+```ruby
+# Rails, with Devise
+authenticate :user, ->(user) { user.admin? } do
+  mount Cosmo::Web => "/cosmo"
+end
 
-### 1. Create `config/cosmo.yml`
+# Any Rack app (config.ru), with HTTP Basic auth
+map "/cosmo" do
+  use Rack::Auth::Basic, "Cosmo" do |user, password|
+    Rack::Utils.secure_compare(user, ENV.fetch("COSMO_WEB_USER")) &
+      Rack::Utils.secure_compare(password, ENV.fetch("COSMO_WEB_PASSWORD"))
+  end
+  run Cosmo::Web
+end
+```
 
-```yaml
-concurrency: 5                     # Number of worker threads
 
-consumers:                         # Declare consumer groups for streams, things that pull messages and process them
-  jobs:                            # Consumer configs for jobs (or streams)
-    default:                       # Stream name
-      ack_policy: explicit         # Acknowledgment required for each message can be explicit, none, or all
-      max_deliver: 30              # Max retry attempts before sending to a dead stream. Safety ceiling only, keep it above every job class's own retry
-      max_ack_pending: 10          # Max messages waiting for ack, if exceeded, the server will stop delivering new messages until some are acked
-      ack_wait: 15                 # Seconds to wait for ack before redelivering
-      subject: jobs.%{name}.>      # Subject pattern for this consumer, %{name} replaced with stream name, becomes `jobs.default.>`
+## 🚀 Quick Start
+
+### 1. Configure (optional)
 
-setup:                             # Initial stream creation only `cosmo -S`
-  jobs:                            # Stream configs for jobs (or streams)
-    default:                       # Stream name
-      storage: file                # Storage type (file or memory)
-      retention: workqueue         # Retention policy (limits, interest, workqueue). workqueue - deletes acked/nacked, limits - append only
-      subjects: ["jobs.%{name}.>"] # Subject pattern for this stream, %{name} replaced with stream name
-      allow_direct: true           # Allow direct messages to stream (required for web UI)
+Nothing is required: without `config/cosmo.yml`, Cosmo runs one `default` job stream plus its own service streams.
+To change anything, write Cosmo's defaults into your project and edit them:
+
+```bash
+bundle exec cosmo --init # writes config/cosmo.yml
 ```
 
-### 2. Create streams in NATS (one-time), grabs config from setup section of `config/cosmo.yml`
+### 2. Create streams in NATS (one-time, idempotent)
 
 ```bash
 bundle exec cosmo -S
@@ -293,6 +300,16 @@ ReportJob.perform_at(Time.parse("2026-01-25 10:00"), 42) # Scheduled
 ReportJob.perform_sync(42)                               # Inline, no NATS (great for tests)
 ```
 
+Inside `perform`, the job instance knows about the message it came from:
+
+| Accessor       | Value                                                                                   |
+|----------------|-----------------------------------------------------------------------------------------|
+| `jid`          | Job id, returned by `perform_async`                                                     |
+| `attempt`      | 1-based delivery attempt                                                                |
+| `enqueued_at`  | `Time` the job was enqueued; for a cron job, when NATS fired the schedule               |
+| `scheduled_by` | Subject of the cron schedule that fired the job, `nil` for jobs enqueued any other way  |
+| `batch_id`     | Id of the [batch](#batches) the job belongs to, `nil` outside a batch                   |
+
 ### Streams
 
 ```ruby
@@ -338,98 +355,87 @@ message.term                         # Permanent failure, no retry
 **NATS subjects** follow a dot-separated hierarchy (`events.clicks.homepage`).
 The `>` wildcard matches everything after that prefix. Think of subjects as topic names — flexible routing with no extra configuration.
 
-**Full `config/cosmo.yml` example:**
+**`config/cosmo.yml` is optional.** Cosmo ships its defaults in
+[`lib/cosmo/config/cosmo.yml`](lib/cosmo/config/cosmo.yml) (`cosmo --init` copies it into your project), and your
+file is deep-merged over them: keep only what you change. The one list that is not merged is your job streams —
+listing any under `setup.jobs` replaces the built-in `default` stream and its consumer:
 ```yaml
-timeout: 25                 # Shutdown timeout in seconds
-concurrency: &concurrency 1 # Number of worker threads
-max_retries: &max_retries 3 # Default max retries
-batch_expiry: 259200         # Seconds before a Batch's tracking data expires (default: 3 days)
+concurrency: 10
 
 stream_config: &stream_config
-  storage: file         # storage type (file or memory)
-  retention: workqueue  # retention policy (limits, interest, workqueue)
-  duplicate_window: 120 # time window for duplicate message detection in seconds
-  discard: old          # what to drop when the stream is full (new rejects publishes, old evicts the oldest)
-  allow_direct: true    # allow direct messages to stream, required for web UI
-  subjects:
-    - jobs.%{name}.>    # subject pattern for stream, %{name} will be replaced with stream name
+  storage: file
+  retention: workqueue
+  duplicate_window: 120 # seconds
+  discard: new
+  allow_direct: true    # required by the web UI
+  subjects: ["jobs.%{name}.>"]
 
 consumer_config: &consumer_config
-  ack_policy: explicit    # ack policy (explicit, none, all), each individual message must be acknowledged
-  max_deliver: 30         # maximum number of times a message will be delivered before it's considered failed. keep it above every job class's own retry; a job that still exceeds it is capped and dead-lettered early with a warning
-  max_ack_pending: 20     # maximum number of messages with pending ack for this consumer
-  ack_wait: 60            # time in seconds to wait for an ack before redelivering the message
-  subject: jobs.%{name}.> # subject pattern for consumer, %{name} will be replaced with stream name
-
-consumers:
-  jobs:
-    critical:
-      <<: *consumer_config
-      priority: 50
-    high:
-      <<: *consumer_config
-      priority: 30
-    default:
-      <<: *consumer_config
-      priority: 15
-    low:
-      <<: *consumer_config
-      priority: 5
-    scheduled:
-      <<: *consumer_config
-      max_deliver: 5
-      max_ack_pending: 100
-      ack_wait: 10
+  ack_policy: explicit
+  max_deliver: 30       # keep it above every job class's retry
+  max_ack_pending: 20
+  ack_wait: 60          # seconds
+  subject: jobs.%{name}.>
 
 setup:
   jobs:
-    critical:
-      <<: *stream_config
-      description: Very critical priority jobs
-    high:
-      <<: *stream_config
-      description: Higher priority jobs
-    default:
-      <<: *stream_config
-      description: Default priority jobs
-    low:
-      <<: *stream_config
-      description: Lower priority jobs
-    scheduled:
-      <<: *stream_config
-      discard: old            # required here: NATS refuses `discard: new` on the stream holding cron schedules
-      description: Scheduled jobs
-    dead:
-      <<: *stream_config
+    critical: { <<: *stream_config }
+    default: { <<: *stream_config }
+  streams:              # Cosmo::Stream processors are always yours to declare
+    events:
+      storage: file
       retention: limits
-      max_msgs: 10000
-      max_age: 604800 # 7d
-      description: Broken jobs (DLQ)
+      max_age: 86400
+      subjects: ["events.>"]
 
-development:
-  verbose: false
-  concurrency: *concurrency
-
-staging:
-  verbose: true
-  concurrency: 3
+consumers:
+  jobs:
+    critical: { <<: *consumer_config, priority: 50 }
+    default: { <<: *consumer_config, priority: 15 }
+```
+`max_age`, `duplicate_window`, and `ack_wait` are in seconds. Without `setup.jobs`, `consumers.jobs.default` still
+tunes the built-in consumer (e.g. just `ack_wait: 300`).
 
-production:
-  concurrency: 3
+**Service streams** — the `scheduled` stream (delayed jobs, crons), the `dead` stream (dead letters), and the stats
+counters and KV buckets are Cosmo's own: they are not part of `cosmo.yml`, and `cosmo --setup` creates them next to
+yours. Tune them in `Cosmo.configure`:
+```ruby
+Cosmo.configure do |config|
+  config.replicas = 3                # every service stream and bucket, for a NATS cluster (default 1)
+  config.dead.max_age = "14d"        # dead-letter retention: max_age, max_msgs (10_000), max_bytes (-1); default 7d
+  config.dead.enabled = false        # drop jobs that give up instead of parking them
+  config.scheduled.enabled = false   # no scheduled stream or scheduler: perform_in/perform_at and crons raise
+  config.batches.expiry = "1d"       # how long batch tracking data lives (default 3d)
+  config.metrics.retention = "90d"   # how long the Metrics tab's per-day numbers are kept (default 30d)
+  config.metrics.enabled = false     # record nothing and hide the Metrics tab
+end
 ```
+Turning `scheduled` or `dead` off also hides their pages in the web UI. Stats can't be turned off — the web UI
+depends on them.
 
-**Programmatic:**
+**Programmatic:** `cosmo.yml` holds streams, consumers, and process settings; `Cosmo.configure` holds what YAML can't
+express (loggers, middleware) and Cosmo's service settings. Command-line flags (`-c`, `-t`, `-p`) win over the file.
 ```ruby
-Cosmo::Config.set(:concurrency, 20)
-Cosmo::Config.set(:setup, :streams, :custom, { storage: "file", subjects: ["custom.>"] })
+Cosmo.configure do |config|
+  config.logger = Rails.logger # any ::Logger; defaults to stdout
+  config.log_level = :debug    # COSMO_LOG_LEVEL, when set, wins
+end
 ```
+Cosmo leaves a custom logger's formatter alone, so its lines lack Cosmo's `jid`/`elapsed` context unless you set
+`logger.formatter = Cosmo::Logger::SimpleFormatter.new`. Trace output is dropped for loggers without a `trace` level.
 
 **Environment variables:**
-```bash
-export NATS_URL=nats://localhost:4222
-export COSMO_JOBS_FETCH_TIMEOUT=0.1
-export COSMO_STREAMS_FETCH_TIMEOUT=0.1
-```
+
+| Variable                   | Default                              | Description                                                                         |
+|----------------------------|--------------------------------------|-------------------------------------------------------------------------------------|
+| `NATS_URL`                 | `nats://localhost:4222`              | NATS server(s) to connect to                                                        |
+| `COSMO_LOG_LEVEL`          | `info`                               | `trace`, `debug`, `info`, `warn`, `error` or `fatal`                                |
+| `COSMO_CLIENT_NAME`        | `cosmo-<program>-<host>-<pid>`       | NATS connection name, shown in `nats server report connections` and `/connz`        |
+| `COSMO_JS_TIMEOUT`         | `5`                                  | JetStream API timeout in seconds; raise it for the web UI on a busy server          |
+| `COSMO_CONNECT_TIMEOUT`    | `2`                                  | NATS TCP connect timeout in seconds                                                 |
+| `COSMO_JOBS_STREAMS`       | all configured                       | Comma-separated job streams to subscribe to, when `--streams`/`--stream` isn't given |
+| `COSMO_JOBS_FETCH_TIMEOUT` | `0.1`                                | Seconds a jobs worker waits on each fetch                                           |
+| `COSMO_WEB_POLL_INTERVAL`  | `5`                                  | Web UI auto-refresh interval in seconds                                             |
 
 
 ## 🔧 Advanced Usage
@@ -536,7 +542,7 @@ end
   end
   ```
 - Batch tracking data (pending counts, callbacks, results) expires automatically after
-  `Config[:batch_expiry]` seconds (default: 3 days).
+  `config.batches.expiry` (default: 3 days), set in `Cosmo.configure`.
 - Open and finished batches — with pending/succeeded/failed counts — are listed live in the web UI's **Batches** tab.
 - Only `Cosmo::Job`-based jobs are tracked; the ActiveJob adapter doesn't currently participate in
   batches (see [`docs/active_job.md`](docs/active_job.md)).
@@ -612,6 +618,69 @@ If the proc returns something non-numeric/non-positive, or raises, the default b
 instead. Note: if this job class also sets `limit: { concurrency: ... }` (see above), `count`
 includes deliveries that were turned away for lack of a free slot, not just failed attempts.
 
+**Error handlers** receive every error Cosmo rescues — each failed job attempt, a failing stream batch, fetch and
+scheduler errors, and rejected messages — so one hook reports them all:
+```ruby
+Cosmo.configure do |config|
+  config.error_handlers << ->(error, context) { Honeybadger.notify(error, context: context) }
+end
+```
+`context` is a Hash with a `:source` (`:job`, `:stream`, `:fetch`, `:scheduler`, `:reject`, `:retry_in`, `:limit`)
+and what is known there: for a job its payload (`jid`, `class`, `args`, …), `stream`, `subject`, and `attempt`; for
+a stream batch its `stream`, `processor`, and `size`. A handler that raises is logged and skipped.
+
+### Middleware
+
+Every job execution on a worker runs through the server middleware chain. A middleware gets the
+job instance, its parsed payload, and the NATS message, and yields to run the rest of the chain:
+```ruby
+class TimingMiddleware
+  def initialize(threshold:)
+    @threshold = threshold
+  end
+
+  def call(job, data, message)
+    started = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+    yield
+  ensure
+    elapsed = Process.clock_gettime(Process::CLOCK_MONOTONIC) - started
+    job.logger.warn("#{data[:class]} took #{elapsed.round(2)}s on #{message.metadata.stream}") if elapsed > @threshold
+  end
+end
+
+Cosmo.configure do |config|
+  config.server_middleware do |chain|
+    chain.add TimingMiddleware, threshold: 5
+  end
+end
+```
+Register middleware at boot, before workers start (e.g. in a Rails initializer; loading `cosmo.yml` afterwards keeps
+it). The chain starts with Cosmo's required middleware: `Cosmo::Middleware::Limit` (a job class's `limit:`
+option), `Cosmo::Middleware::Busy` (the Web UI's Busy page), `Cosmo::Middleware::Totals` (processed/failed counts,
+one per execution), and `Cosmo::Middleware::Metrics` (the Metrics tab). `add` appends innermost, after them; `prepend`, `insert_before`, `insert_after`, and `remove`
+reorder the chain, so moving or removing the built-ins is at your own risk.
+A middleware that raises sends the job through the usual retry/DLQ path, and one that doesn't yield skips the job
+and acks it. To put a job back for later instead, raise `Cosmo::Job::Requeue.new(delay_in_seconds)`: the job is
+redelivered without failing, though the redelivery still counts toward its `retry` and `max_deliver`.
+A fresh instance is built for every job. `perform_sync` doesn't run the chain.
+
+The **client** chain runs where a job is enqueued (`perform_async`, `perform_in`, `perform_at`, and the ActiveJob
+adapter), around the publish. It starts empty, and a middleware gets the job class name, the payload Hash that will be
+published, and the stream the job runs on:
+```ruby
+class RequestIdMiddleware
+  def call(_job_class, payload, _stream)
+    payload[:request_id] = Current.request_id
+    yield
+  end
+end
+
+Cosmo.configure do |config|
+  config.client_middleware { |chain| chain.add RequestIdMiddleware }
+end
+```
+Not yielding stops the publish: `perform_async` returns `nil`, and a batch the job was joining doesn't wait for it.
+
 ### Testing
 
 ```ruby
@@ -651,11 +720,16 @@ ActiveJob::Base.queue_adapter = Cosmo::ActiveJobAdapter::Adapter.new
 
 **Sentry:**
 ```ruby
-require "cosmo/sentry/auto"
+require "cosmo/middleware/sentry"
+
+Cosmo.configure do |config|
+  config.server_middleware { |c| c.add Cosmo::Middleware::Sentry }
+  config.error_handlers << Cosmo::Middleware::Sentry::ERROR_HANDLER
+end
 ```
-Wraps every job execution in a Sentry transaction (`queue.cosmonats`) and captures unhandled
-exceptions with the job's id, stream, subject, and retry count attached as context — no other
-setup beyond having `sentry-ruby` initialized.
+The [middleware](#middleware) wraps every job execution in a Sentry transaction (`queue.cosmonats`), and the
+[error handler](#error-handling) captures every error Cosmo rescues — failed jobs, stream batches, fetches, the
+scheduler — with its context under `cosmonats`. No other setup beyond having `sentry-ruby` initialized.
 
 
 ## 🖥️ CLI Reference
@@ -684,7 +758,9 @@ The `scheduled` stream is a service stream: it is always dispatched, and naming
 | `-c, --concurrency INT` | Worker threads                        | `-c 20`               |
 | `-r, --require PATH`    | Auto-require directory                | `-r ./app/jobs`       |
 | `-t, --timeout NUM`     | Shutdown timeout (sec)                | `-t 60`               |
+| `-p, --http-port INT`   | Serve `/health` and `/ping` on a port | `-p 9090`             |
 | `-S, --setup`           | Setup streams & sync cron, then exit  | `--setup`             |
+| `--init`                | Write `config/cosmo.yml` with the defaults, then exit | `--init`  |
 | `-v, --version`         | Print version and exit                | `--version`           |
 | `-h, --help`            | Show help and exit                    | `--help`               |
 
@@ -710,6 +786,23 @@ The `scheduled` stream is a service stream: it is always dispatched, and naming
 | `--batch_size NUM`       | Messages per fetch batch             | `--batch_size 50`               |
 
 
+### Signals
+
+| Signal           | Effect                                                                                     |
+|------------------|--------------------------------------------------------------------------------------------|
+| `INT`, `TERM`    | Graceful shutdown: stop fetching, wait up to `--timeout` seconds for in-flight work, exit  |
+| `TSTP`           | Quiet: stop fetching new work, keep running in-flight work                                 |
+| `CONT`           | Resume fetching after `TSTP`                                                               |
+| `USR1`           | Quiet, then exit once in-flight work drains                                                |
+| `TTIN`           | Log every thread's backtrace at `WARN`, tagged with the same `tid` as regular log lines     |
+
+```bash
+kill -TSTP <pid>  # before a deploy: let the old worker finish what it has
+kill -TTIN <pid>  # a worker looks stuck: see what each thread is doing
+```
+
+The worker's state (`running`, `quiet`, `stopping`) is shown on the web UI's Processes list.
+
 ### Health check
 
 Pass `-p/--http-port` (or set `http.port` in the config) to start a small WEBrick HTTP server alongside the workers.
@@ -807,11 +900,14 @@ sudo systemctl enable cosmo && sudo systemctl start cosmo
 - **Jobs** — enqueued, scheduled, busy, and dead views, with per-job retry and delete
 - **Processes** — the Jobs dashboard lists every live worker (idle ones included): host, pid, IP, NATS connection
   name, command line, subscribed streams, busy/concurrency, running/quiet/stopping state, memory, NATS RTT and
-  reconnects. Each worker
-  beats into the `cosmo_processes` KV bucket every 10s, and a crashed worker drops out within 60s
+  reconnects. Each worker beats into the `_cosmoprocesses` KV bucket every 10s, and a crashed worker drops out within 60s
 - **Streams** — per-stream state (messages, bytes, consumers) with pause/resume
 - **Crons** — every schedule deployed in NATS, with run-now and delete
 - **Batches** — open and finished batches with pending/succeeded/failed counts
+- **Metrics** — per job class over today, 7, or 30 days: runs, failures, average execution time (successful runs), and
+  average wait from enqueue to first delivery, plus a per-day chart of runs and average execution time. Recorded by
+  `Cosmo::Middleware::Metrics`, flushed by every worker's heartbeat into the internal `_cosmometrics` counters stream;
+  turn it off with `config.metrics.enabled = false`
 - Summary counters (processed / failed / busy / enqueued / retries / scheduled / dead) backed by a NATS KV counter, no separate metrics store needed
 
 **Structured logs:**

data/lib/cosmo/active_job/adapter.rb

@@ -32,8 +32,7 @@ module Cosmo
         options    = { stream: stream }.merge(cosmo_opts)
         options[:at] = timestamp if timestamp
 
-        data = Job::Data.new(Executor.name, [job.serialize], options)
-        Publisher.publish_job(data)
+        Job::Enqueuer.enqueue(Executor.name, [job.serialize], options)
       end
 
       # Returns Cosmo-specific options declared on the job class via

data/lib/cosmo/api/counter.rb

@@ -2,22 +2,22 @@
 
 module Cosmo
   module API
+    # Base for NATS counters (+Nats-Incr+) kept in a stream of their own. A subclass names it with +STREAM_NAME+ and
+    # +DESCRIPTION+ and may extend {.stream_config}; every instance counts under its namespace in that stream.
     class Counter
-      STREAM_NAME = "_cosmostats"
-      STREAM_CONFIG = {
-        subjects: ["#{STREAM_NAME}.>"],
-        allow_msg_counter: true,
-        allow_direct: true,
-        max_msgs_per_subject: 1,
-        description: "Cosmo statistics"
-      }.freeze
-
-      # Creates the counters stream, or brings an existing one up to STREAM_CONFIG. Every stored
-      # counter message carries the running total, so only the last one per subject is kept.
+      # Creates the counters stream, or brings an existing one up to {.stream_config}.
       #
       # @return [void]
       def self.setup!
-        Client.instance.setup_stream(STREAM_NAME, STREAM_CONFIG)
+        Client.instance.setup_stream(self::STREAM_NAME, stream_config)
+      end
+
+      # Every stored counter message carries the running total, so only the last one per subject is kept.
+      #
+      # @return [Hash]
+      def self.stream_config
+        { subjects: ["#{self::STREAM_NAME}.>"], allow_msg_counter: true, allow_direct: true, max_msgs_per_subject: 1,
+          num_replicas: Config.replicas, description: self::DESCRIPTION }
       end
 
       def initialize(namespace)
@@ -35,12 +35,12 @@ module Cosmo
       alias decr decrement
 
       def reset(key)
-        client.purge(STREAM_NAME, subject(key))
+        client.purge(self.class::STREAM_NAME, subject(key))
       end
       alias purge reset
 
       def get(key)
-        raw = client.get_message(STREAM_NAME, direct: true, subject: subject(key))
+        raw = client.get_message(self.class::STREAM_NAME, direct: true, subject: subject(key))
         Utils::Json.parse(raw.data, default: { "val" => 0 })[:val].to_i
       rescue NATS::JetStream::Error::NotFound, NATS::JetStream::Error::ServiceUnavailable, NATS::IO::Timeout
         0
@@ -67,13 +67,13 @@ module Cosmo
           raise if rescued
 
           rescued = true
-          client.create_stream(STREAM_NAME, STREAM_CONFIG)
+          client.create_stream(self.class::STREAM_NAME, self.class.stream_config)
           retry
         end
       end
 
       def subject(key)
-        "#{STREAM_NAME}.#{@namespace}.#{key}"
+        "#{self.class::STREAM_NAME}.#{@namespace}.#{key}"
       end
 
       def client

data/lib/cosmo/api/cron.rb

@@ -32,7 +32,10 @@ module Cosmo
 
       # Publish (or replace) a schedule message in NATS.
       # @return [Hash, nil] the persisted schedule as a hash, or nil on failure
+      # @raise [SchedulingDisabledError] when scheduling is turned off
       def upsert!(class_name: nil, stream: nil, schedule: nil, args: [], timezone: nil, name: nil)
+        raise SchedulingDisabledError unless Config.scheduled.enabled
+
         e = Entry.new(class_name: class_name, stream: stream, expression: schedule,
                       args: args, timezone: timezone, name: name)
         client.publish(e.schedule_subject, e.job_payload, stream: STREAM, header: e.schedule_headers)

data/lib/cosmo/api/registry.rb

@@ -57,7 +57,7 @@ module Cosmo
       private
 
       def bucket
-        KV.new(self.class::BUCKET, { ttl: self.class::TTL })
+        KV.new(self.class::BUCKET, { ttl: self.class::TTL, replicas: Config.replicas })
       end
 
       def sanitize(key)

data/lib/cosmo/api/stats/busy.rb

@@ -9,7 +9,7 @@ module Cosmo
       class Busy < Registry
         TTL = 70
         HEARTBEAT = 30
-        BUCKET = "cosmo_jobs_busy"
+        BUCKET = "_cosmobusy"
 
         def initialize
           super

data/lib/cosmo/api/stats/metrics.rb

@@ -0,0 +1,109 @@
+# frozen_string_literal: true
+
+module Cosmo
+  module API
+    module Stats
+      # Per-day metrics for every job class: runs, failures, and execution and wait time, kept in their own counters
+      # stream for +config.metrics.retention+. Workers {#record} executions in memory and the heartbeat {#flush}es them,
+      # so a job costs no NATS round trip; a crashed worker loses at most one heartbeat's worth.
+      class Metrics < Counter
+        STREAM_NAME = "_cosmometrics"
+        DESCRIPTION = "Cosmo metrics"
+        DAY = 86_400
+
+        def self.instance
+          @instance ||= new
+        end
+
+        # Each day's counters expire +config.metrics.retention+ after their last update.
+        #
+        # @return [Hash]
+        def self.stream_config
+          super.merge(max_age: Config.to_ns(Utils::Duration.parse(Config.metrics.retention)))
+        end
+
+        def initialize
+          super("jobs")
+          @lock = Mutex.new
+          @buffer = buffer
+        end
+
+        # Adds one execution to the buffer {#flush} writes.
+        #
+        # @param job_class [String]
+        # @param exec_ms [Numeric, nil] how long a successful execution took
+        # @param wait_ms [Numeric, nil] time from enqueue to the job's first delivery
+        # @param failed [Boolean]
+        # @return [void]
+        def record(job_class, exec_ms: nil, wait_ms: nil, failed: false)
+          key = [Time.now.utc.strftime("%Y%m%d"), job_class.gsub("::", "-")]
+          @lock.synchronize do
+            totals = @buffer[key]
+            failed ? totals[:failed] += 1 : totals[:count] += 1
+            totals[:exec_ms] += exec_ms if exec_ms
+            next unless wait_ms
+
+            totals[:waited] += 1
+            totals[:wait_ms] += wait_ms
+          end
+        end
+
+        # Writes the buffered executions to the counters and empties the buffer.
+        #
+        # @return [void]
+        def flush
+          pending = @lock.synchronize { @buffer.tap { @buffer = buffer } }
+          pending.each do |(day, job), totals|
+            totals.each { |field, value| increment("#{day}.#{job}.#{field}", by: value.round) if value.positive? }
+          end
+        end
+
+        # @param days [Integer] today and the days before it
+        # @return [Array<Hash>] per job class +:job+, +:count+, +:failed+, +:exec_ms+ and +:wait_ms+ (averages in
+        #   milliseconds, nil without data), busiest first
+        def summary(days: 1)
+          totals = Hash.new { |hash, job| hash[job] = Hash.new(0) }
+          read(days).each { |(_, job, field), value| totals[job][field] += value }
+          totals.map { |job, sums| row(sums).merge(job: job) }.sort_by { -_1[:count] }
+        end
+
+        # @param job_class [String]
+        # @param days [Integer]
+        # @return [Array<Hash>] one +{ date:, count:, failed:, exec_ms:, wait_ms: }+ per day, oldest first
+        def daily(job_class, days: 30)
+          sums = Hash.new { |hash, day| hash[day] = Hash.new(0) }
+          read(days).each { |(day, job, field), value| sums[day][field] += value if job == job_class }
+          dates(days).map { |day| row(sums[day]).merge(date: day) }
+        end
+
+        private
+
+        def buffer
+          Hash.new { |hash, key| hash[key] = Hash.new(0) }
+        end
+
+        def read(days)
+          filters = dates(days).map { "#{STREAM_NAME}.#{@namespace}.#{_1}.>" }
+          client.last_messages(STREAM_NAME, filters).map do |subject, data|
+            day, job, field = subject.split(".").last(3)
+            [[day, job.gsub("-", "::"), field.to_sym], Utils::Json.parse(data, default: {})[:val].to_i]
+          end
+        end
+
+        def dates(days)
+          now = Time.now.utc
+          (days - 1).downto(0).map { (now - (_1 * DAY)).strftime("%Y%m%d") }
+        end
+
+        def row(sums)
+          { count: sums[:count], failed: sums[:failed], exec_ms: average(sums[:exec_ms], sums[:count]),
+            wait_ms: average(sums[:wait_ms], sums[:waited]) }
+        end
+
+        def average(total, count)
+          total.fdiv(count).round(1) if count.positive?
+        end
+      end
+    end
+  end
+end