cosmonats 0.7.0 → 0.9.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 96e901f7365784bf2fb7087311112efb263f295eddbe25a96ff32de4ebfe0652
-  data.tar.gz: b1f7327b126f85e50222af5adf2172b8fab84aa051209e9c0e9b478254bcb8a8
+  metadata.gz: 40e30ee2fff77d85db1a3deedf05e040c72dca39d64da01220517d3d5ae8048b
+  data.tar.gz: 6f0e5a50448674483f09f54938336940899ea81fe46ce66da23478eeee871bac
 SHA512:
-  metadata.gz: 1ab5c6cc101edf8dbf8909f544ed9a9fd8603c76f073dad60c5251e086195518e07ec5a6815bcf382b05c7b80f67fb6994a4615be933b33c29103ea3082990af
-  data.tar.gz: 85b15998b6286a331558e847835f6a94cf0936adae88e70a18035335b2414096d583e4619b7b2e5f5c7e579117c0d3626dbb99e189084fc3cf7aedc25dfc4fba
+  metadata.gz: 6d57b79ee594fd7a7ace94e1e3f5d70de87b9a413b85389cddb9db51f39ed09542bd0f0a7d3c4308e7009b7a2d9997b9be0c2965efa61e259bd07baf9bfa4734
+  data.tar.gz: 8122f92e84e4176a403678225ab66b753270ab5e49334568bc3e88cfa292451bc63ea6f5383bdb458ab89f8b319a6ed33b7af471f636ffe7314978c956b6acb1

data/README.md

@@ -58,8 +58,6 @@ bundle exec cosmo -C config/cosmo.yml -c 20 jobs    # Jobs only
 bundle exec cosmo -C config/cosmo.yml -c 20 streams # Streams only
 ```
 
-![webui.gif](webui.gif)
-
 ## 📖 Index
 
 - [Why?](#-why)
@@ -77,6 +75,7 @@ bundle exec cosmo -C config/cosmo.yml -c 20 streams # Streams only
   - [Concurrency Limiting](#concurrency-limiting)
   - [Custom Serializers](#custom-serializers)
   - [Error Handling](#error-handling)
+  - [Middleware](#middleware)
   - [Testing](#testing)
   - [Integrations](#integrations)
 - [CLI Reference](#-cli-reference)
@@ -166,7 +165,7 @@ nothing else to run.
 gem "cosmonats"
 ```
 
-**Requirements:** Ruby ≥ 3.1, NATS Server ([install guide](https://docs.nats.io/running-a-nats-service/introduction/installation))
+**Requirements:** Ruby ≥ 3.1, NATS Server ≥ 2.14 with JetStream enabled ([install guide](https://docs.nats.io/running-a-nats-service/introduction/installation))
 
 **Optional gems** — not installed with `cosmonats`, add them only for the features you use:
 
@@ -208,33 +207,39 @@ mount Cosmo::Web => "/cosmo"
 map "/cosmo" { run Cosmo::Web }
 ```
 
+> [!WARNING]
+> `Cosmo::Web` ships with **no authentication and no CSRF protection**, yet it exposes destructive actions
+> (retrying and deleting jobs, pausing streams, running and deleting crons). Never expose it unprotected.
 
-## 🚀 Quick Start
+```ruby
+# Rails, with Devise
+authenticate :user, ->(user) { user.admin? } do
+  mount Cosmo::Web => "/cosmo"
+end
+
+# Any Rack app (config.ru), with HTTP Basic auth
+map "/cosmo" do
+  use Rack::Auth::Basic, "Cosmo" do |user, password|
+    Rack::Utils.secure_compare(user, ENV.fetch("COSMO_WEB_USER")) &
+      Rack::Utils.secure_compare(password, ENV.fetch("COSMO_WEB_PASSWORD"))
+  end
+  run Cosmo::Web
+end
+```
 
-### 1. Create `config/cosmo.yml`
 
-```yaml
-concurrency: 5                     # Number of worker threads
+## 🚀 Quick Start
+
+### 1. Configure (optional)
 
-consumers:                         # Declare consumer groups for streams, things that pull messages and process them
-  jobs:                            # Consumer configs for jobs (or streams)
-    default:                       # Stream name
-      ack_policy: explicit         # Acknowledgment required for each message can be explicit, none, or all
-      max_deliver: 30              # Max retry attempts before sending to a dead stream. Safety ceiling only, keep it above every job class's own retry
-      max_ack_pending: 10          # Max messages waiting for ack, if exceeded, the server will stop delivering new messages until some are acked
-      ack_wait: 15                 # Seconds to wait for ack before redelivering
-      subject: jobs.%{name}.>      # Subject pattern for this consumer, %{name} replaced with stream name, becomes `jobs.default.>`
+Nothing is required: without `config/cosmo.yml`, Cosmo runs one `default` job stream plus its own service streams.
+To change anything, write Cosmo's defaults into your project and edit them:
 
-setup:                             # Initial stream creation only `cosmo -S`
-  jobs:                            # Stream configs for jobs (or streams)
-    default:                       # Stream name
-      storage: file                # Storage type (file or memory)
-      retention: workqueue         # Retention policy (limits, interest, workqueue). workqueue - deletes acked/nacked, limits - append only
-      subjects: ["jobs.%{name}.>"] # Subject pattern for this stream, %{name} replaced with stream name
-      allow_direct: true           # Allow direct messages to stream (required for web UI)
+```bash
+bundle exec cosmo --init # writes config/cosmo.yml
 ```
 
-### 2. Create streams in NATS (one-time), grabs config from setup section of `config/cosmo.yml`
+### 2. Create streams in NATS (one-time, idempotent)
 
 ```bash
 bundle exec cosmo -S
@@ -293,6 +298,16 @@ ReportJob.perform_at(Time.parse("2026-01-25 10:00"), 42) # Scheduled
 ReportJob.perform_sync(42)                               # Inline, no NATS (great for tests)
 ```
 
+Inside `perform`, the job instance knows about the message it came from:
+
+| Accessor       | Value                                                                                   |
+|----------------|-----------------------------------------------------------------------------------------|
+| `jid`          | Job id, returned by `perform_async`                                                     |
+| `attempt`      | 1-based delivery attempt                                                                |
+| `enqueued_at`  | `Time` the job was enqueued; for a cron job, when NATS fired the schedule               |
+| `scheduled_by` | Subject of the cron schedule that fired the job, `nil` for jobs enqueued any other way  |
+| `batch_id`     | Id of the [batch](#batches) the job belongs to, `nil` outside a batch                   |
+
 ### Streams
 
 ```ruby
@@ -338,98 +353,89 @@ message.term                         # Permanent failure, no retry
 **NATS subjects** follow a dot-separated hierarchy (`events.clicks.homepage`).
 The `>` wildcard matches everything after that prefix. Think of subjects as topic names — flexible routing with no extra configuration.
 
-**Full `config/cosmo.yml` example:**
+**`config/cosmo.yml` is optional.** Cosmo ships its defaults in
+[`lib/cosmo/config/cosmo.yml`](lib/cosmo/config/cosmo.yml) (`cosmo --init` copies it into your project), and your
+file is deep-merged over them: keep only what you change. The one list that is not merged is your job streams —
+listing any under `setup.jobs` replaces the built-in `default` stream and its consumer:
 ```yaml
-timeout: 25                 # Shutdown timeout in seconds
-concurrency: &concurrency 1 # Number of worker threads
-max_retries: &max_retries 3 # Default max retries
-batch_expiry: 259200         # Seconds before a Batch's tracking data expires (default: 3 days)
+concurrency: 10
 
 stream_config: &stream_config
-  storage: file         # storage type (file or memory)
-  retention: workqueue  # retention policy (limits, interest, workqueue)
-  duplicate_window: 120 # time window for duplicate message detection in seconds
-  discard: old          # what to drop when the stream is full (new rejects publishes, old evicts the oldest)
-  allow_direct: true    # allow direct messages to stream, required for web UI
-  subjects:
-    - jobs.%{name}.>    # subject pattern for stream, %{name} will be replaced with stream name
+  storage: file
+  retention: workqueue
+  duplicate_window: 120 # seconds
+  discard: new
+  allow_direct: true    # required by the web UI
+  subjects: ["jobs.%{name}.>"]
 
 consumer_config: &consumer_config
-  ack_policy: explicit    # ack policy (explicit, none, all), each individual message must be acknowledged
-  max_deliver: 30         # maximum number of times a message will be delivered before it's considered failed. keep it above every job class's own retry; a job that still exceeds it is capped and dead-lettered early with a warning
-  max_ack_pending: 20     # maximum number of messages with pending ack for this consumer
-  ack_wait: 60            # time in seconds to wait for an ack before redelivering the message
-  subject: jobs.%{name}.> # subject pattern for consumer, %{name} will be replaced with stream name
-
-consumers:
-  jobs:
-    critical:
-      <<: *consumer_config
-      priority: 50
-    high:
-      <<: *consumer_config
-      priority: 30
-    default:
-      <<: *consumer_config
-      priority: 15
-    low:
-      <<: *consumer_config
-      priority: 5
-    scheduled:
-      <<: *consumer_config
-      max_deliver: 5
-      max_ack_pending: 100
-      ack_wait: 10
+  ack_policy: explicit
+  max_deliver: 30       # keep it above every job class's retry
+  max_ack_pending: 20
+  ack_wait: 60          # seconds
+  subject: jobs.%{name}.>
 
 setup:
   jobs:
-    critical:
-      <<: *stream_config
-      description: Very critical priority jobs
-    high:
-      <<: *stream_config
-      description: Higher priority jobs
-    default:
-      <<: *stream_config
-      description: Default priority jobs
-    low:
-      <<: *stream_config
-      description: Lower priority jobs
-    scheduled:
-      <<: *stream_config
-      discard: old            # required here: NATS refuses `discard: new` on the stream holding cron schedules
-      description: Scheduled jobs
-    dead:
-      <<: *stream_config
+    critical: { <<: *stream_config }
+    default: { <<: *stream_config }
+  streams:              # Cosmo::Stream processors are always yours to declare
+    events:
+      storage: file
       retention: limits
-      max_msgs: 10000
-      max_age: 604800 # 7d
-      description: Broken jobs (DLQ)
-
-development:
-  verbose: false
-  concurrency: *concurrency
+      max_age: 86400
+      subjects: ["events.>"]
 
-staging:
-  verbose: true
-  concurrency: 3
+consumers:
+  jobs:
+    critical: { <<: *consumer_config, priority: 50 }
+    default: { <<: *consumer_config, priority: 15 }
+```
+`max_age`, `duplicate_window`, and `ack_wait` are in seconds. Without `setup.jobs`, `consumers.jobs.default` still
+tunes the built-in consumer (e.g. just `ack_wait: 300`).
 
-production:
-  concurrency: 3
+**Service streams** — the `scheduled` stream (delayed jobs, crons), the `dead` stream (dead letters), and the stats
+counters and KV buckets are Cosmo's own: they are not part of `cosmo.yml`, and `cosmo --setup` creates them next to
+yours. Tune them in `Cosmo.configure`:
+```ruby
+Cosmo.configure do |config|
+  config.replicas = 3                # every service stream and bucket, for a NATS cluster (default 1)
+  config.dead.max_age = "14d"        # dead-letter retention: max_age, max_msgs (10_000), max_bytes (-1); default 7d
+  config.dead.enabled = false        # drop jobs that give up instead of parking them
+  config.scheduled.enabled = false   # no scheduled stream or scheduler: perform_in/perform_at and crons raise
+  config.batches.expiry = "1d"       # how long batch tracking data lives (default 3d)
+  config.metrics.retention = "90d"   # how long the Metrics tab's per-day numbers are kept (default 30d)
+  config.metrics.enabled = false     # record nothing and hide the Metrics tab
+end
 ```
+Turning `scheduled` or `dead` off also hides their pages in the web UI. Stats can't be turned off — the web UI
+depends on them.
 
-**Programmatic:**
+**Programmatic:** `cosmo.yml` holds streams, consumers, and process settings; `Cosmo.configure` holds what YAML can't
+express (loggers, middleware) and Cosmo's service settings. Command-line flags (`-c`, `-t`, `-p`) win over the file.
 ```ruby
-Cosmo::Config.set(:concurrency, 20)
-Cosmo::Config.set(:setup, :streams, :custom, { storage: "file", subjects: ["custom.>"] })
+Cosmo.configure do |config|
+  config.logger = Rails.logger # any ::Logger; defaults to stdout
+  config.log_level = :debug    # COSMO_LOG_LEVEL, when set, wins
+end
 ```
+Cosmo leaves a custom logger's formatter alone, so its lines lack Cosmo's `jid`/`elapsed` context unless you set
+`logger.formatter = Cosmo::Logger::SimpleFormatter.new`. Trace output is dropped for loggers without a `trace` level.
 
 **Environment variables:**
-```bash
-export NATS_URL=nats://localhost:4222
-export COSMO_JOBS_FETCH_TIMEOUT=0.1
-export COSMO_STREAMS_FETCH_TIMEOUT=0.1
-```
+
+| Variable                   | Default                              | Description                                                                         |
+|----------------------------|--------------------------------------|-------------------------------------------------------------------------------------|
+| `NATS_URL`                 | `nats://localhost:4222`              | NATS server(s) to connect to                                                        |
+| `COSMO_LOG_LEVEL`          | `info`                               | `trace`, `debug`, `info`, `warn`, `error` or `fatal`                                |
+| `COSMO_CLIENT_NAME`        | `cosmo-<program>-<host>-<pid>`       | NATS connection name, shown in `nats server report connections` and `/connz`        |
+| `COSMO_JS_TIMEOUT`         | `5`                                  | JetStream API timeout in seconds; raise it for the web UI on a busy server          |
+| `COSMO_CONNECT_TIMEOUT`    | `2`                                  | NATS TCP connect timeout in seconds                                                 |
+| `NATS_MAX_RECONNECT_ATTEMPTS` | `-1`                              | Reconnect attempts before giving up on NATS; `-1` keeps trying through any outage   |
+| `COSMO_JOBS_STREAMS`       | all configured                       | Comma-separated job streams to subscribe to, when `--streams`/`--stream` isn't given |
+| `COSMO_JOBS_FETCH_TIMEOUT` | `0.1`                                | Seconds a jobs worker waits on each fetch                                           |
+| `COSMO_WEB_POLL_INTERVAL`  | `5`                                  | Web UI auto-refresh interval in seconds                                             |
+| `COSMO_WEB_CACHE_TTL`      | `2`                                  | Seconds the web UI shares a refresh between viewers; `0` turns sharing off          |
 
 
 ## 🔧 Advanced Usage
@@ -536,8 +542,8 @@ end
   end
   ```
 - Batch tracking data (pending counts, callbacks, results) expires automatically after
-  `Config[:batch_expiry]` seconds (default: 3 days).
-- Open and finished batches — with pending/succeeded/failed counts — are listed live in the web UI's **Batches** tab.
+  `config.batches.expiry` (default: 3 days), set in `Cosmo.configure`.
+- Open and finished batches — with pending/succeeded/failed counts — are listed live on the web UI's **Batches** page.
 - Only `Cosmo::Job`-based jobs are tracked; the ActiveJob adapter doesn't currently participate in
   batches (see [`docs/active_job.md`](docs/active_job.md)).
 
@@ -612,6 +618,69 @@ If the proc returns something non-numeric/non-positive, or raises, the default b
 instead. Note: if this job class also sets `limit: { concurrency: ... }` (see above), `count`
 includes deliveries that were turned away for lack of a free slot, not just failed attempts.
 
+**Error handlers** receive every error Cosmo rescues — each failed job attempt, a failing stream batch, fetch and
+scheduler errors, and rejected messages — so one hook reports them all:
+```ruby
+Cosmo.configure do |config|
+  config.error_handlers << ->(error, context) { Honeybadger.notify(error, context: context) }
+end
+```
+`context` is a Hash with a `:source` (`:job`, `:stream`, `:fetch`, `:scheduler`, `:reject`, `:retry_in`, `:limit`)
+and what is known there: for a job its payload (`jid`, `class`, `args`, …), `stream`, `subject`, and `attempt`; for
+a stream batch its `stream`, `processor`, and `size`. A handler that raises is logged and skipped.
+
+### Middleware
+
+Every job execution on a worker runs through the server middleware chain. A middleware gets the
+job instance, its parsed payload, and the NATS message, and yields to run the rest of the chain:
+```ruby
+class TimingMiddleware
+  def initialize(threshold:)
+    @threshold = threshold
+  end
+
+  def call(job, data, message)
+    started = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+    yield
+  ensure
+    elapsed = Process.clock_gettime(Process::CLOCK_MONOTONIC) - started
+    job.logger.warn("#{data[:class]} took #{elapsed.round(2)}s on #{message.metadata.stream}") if elapsed > @threshold
+  end
+end
+
+Cosmo.configure do |config|
+  config.server_middleware do |chain|
+    chain.add TimingMiddleware, threshold: 5
+  end
+end
+```
+Register middleware at boot, before workers start (e.g. in a Rails initializer; loading `cosmo.yml` afterwards keeps
+it). The chain starts with Cosmo's required middleware: `Cosmo::Middleware::Limit` (a job class's `limit:`
+option), `Cosmo::Middleware::Busy` (the Web UI's Busy page), `Cosmo::Middleware::Totals` (processed/failed counts,
+one per execution), and `Cosmo::Middleware::Metrics` (the Metrics tab). `add` appends innermost, after them; `prepend`, `insert_before`, `insert_after`, and `remove`
+reorder the chain, so moving or removing the built-ins is at your own risk.
+A middleware that raises sends the job through the usual retry/DLQ path, and one that doesn't yield skips the job
+and acks it. To put a job back for later instead, raise `Cosmo::Job::Requeue.new(delay_in_seconds)`: the job is
+redelivered without failing, though the redelivery still counts toward its `retry` and `max_deliver`.
+A fresh instance is built for every job. `perform_sync` doesn't run the chain.
+
+The **client** chain runs where a job is enqueued (`perform_async`, `perform_in`, `perform_at`, and the ActiveJob
+adapter), around the publish. It starts empty, and a middleware gets the job class name, the payload Hash that will be
+published, and the stream the job runs on:
+```ruby
+class RequestIdMiddleware
+  def call(_job_class, payload, _stream)
+    payload[:request_id] = Current.request_id
+    yield
+  end
+end
+
+Cosmo.configure do |config|
+  config.client_middleware { |chain| chain.add RequestIdMiddleware }
+end
+```
+Not yielding stops the publish: `perform_async` returns `nil`, and a batch the job was joining doesn't wait for it.
+
 ### Testing
 
 ```ruby
@@ -651,11 +720,16 @@ ActiveJob::Base.queue_adapter = Cosmo::ActiveJobAdapter::Adapter.new
 
 **Sentry:**
 ```ruby
-require "cosmo/sentry/auto"
+require "cosmo/middleware/sentry"
+
+Cosmo.configure do |config|
+  config.server_middleware { |c| c.add Cosmo::Middleware::Sentry }
+  config.error_handlers << Cosmo::Middleware::Sentry::ERROR_HANDLER
+end
 ```
-Wraps every job execution in a Sentry transaction (`queue.cosmonats`) and captures unhandled
-exceptions with the job's id, stream, subject, and retry count attached as context — no other
-setup beyond having `sentry-ruby` initialized.
+The [middleware](#middleware) wraps every job execution in a Sentry transaction (`queue.cosmonats`), and the
+[error handler](#error-handling) captures every error Cosmo rescues — failed jobs, stream batches, fetches, the
+scheduler — with its context under `cosmonats`. No other setup beyond having `sentry-ruby` initialized.
 
 
 ## 🖥️ CLI Reference
@@ -684,7 +758,9 @@ The `scheduled` stream is a service stream: it is always dispatched, and naming
 | `-c, --concurrency INT` | Worker threads                        | `-c 20`               |
 | `-r, --require PATH`    | Auto-require directory                | `-r ./app/jobs`       |
 | `-t, --timeout NUM`     | Shutdown timeout (sec)                | `-t 60`               |
+| `-p, --http-port INT`   | Serve `/health` and `/ping` on a port | `-p 9090`             |
 | `-S, --setup`           | Setup streams & sync cron, then exit  | `--setup`             |
+| `--init`                | Write `config/cosmo.yml` with the defaults, then exit | `--init`  |
 | `-v, --version`         | Print version and exit                | `--version`           |
 | `-h, --help`            | Show help and exit                    | `--help`               |
 
@@ -710,6 +786,23 @@ The `scheduled` stream is a service stream: it is always dispatched, and naming
 | `--batch_size NUM`       | Messages per fetch batch             | `--batch_size 50`               |
 
 
+### Signals
+
+| Signal           | Effect                                                                                     |
+|------------------|--------------------------------------------------------------------------------------------|
+| `INT`, `TERM`    | Graceful shutdown: stop fetching, wait up to `--timeout` seconds for in-flight work, exit  |
+| `TSTP`           | Quiet: stop fetching new work, keep running in-flight work                                 |
+| `CONT`           | Resume fetching after `TSTP`                                                               |
+| `USR1`           | Quiet, then exit once in-flight work drains                                                |
+| `TTIN`           | Log every thread's backtrace at `WARN`, tagged with the same `tid` as regular log lines     |
+
+```bash
+kill -TSTP <pid>  # before a deploy: let the old worker finish what it has
+kill -TTIN <pid>  # a worker looks stuck: see what each thread is doing
+```
+
+The worker's state (`running`, `quiet`, `stopping`) is shown on the web UI's Processes list.
+
 ### Health check
 
 Pass `-p/--http-port` (or set `http.port` in the config) to start a small WEBrick HTTP server alongside the workers.
@@ -803,16 +896,29 @@ sudo systemctl enable cosmo && sudo systemctl start cosmo
 
 ## 📊 Monitoring
 
-**Web UI** — mount `Cosmo::Web` (see [Installation](#-installation)) for a live, htmx-powered dashboard:
-- **Jobs** — enqueued, scheduled, busy, and dead views, with per-job retry and delete
-- **Processes** — the Jobs dashboard lists every live worker (idle ones included): host, pid, IP, NATS connection
-  name, command line, subscribed streams, busy/concurrency, running/quiet/stopping state, memory, NATS RTT and
-  reconnects. Each worker
-  beats into the `cosmo_processes` KV bucket every 10s, and a crashed worker drops out within 60s
-- **Streams** — per-stream state (messages, bytes, consumers) with pause/resume
+**Web UI** — mount `Cosmo::Web` (see [Installation](#-installation)) for a live, htmx-powered dashboard with a
+light, dark, or automatic (system) theme. Fonts and assets are bundled, so it loads nothing from a CDN:
+- **Overview** — processed, failed, busy, enqueued, retries, scheduled and dead totals; a 7- or 30-day throughput
+  chart of successful and failed runs; a grid of every worker thread, busy ones colored by stream; the queues; and the
+  jobs running right now, longest first
+- **Jobs** — each job queue with its size, busy jobs and redeliveries, paused and resumed in place; then busy,
+  enqueued (per queue), scheduled and dead jobs, with retry and delete for dead ones and their error and backtrace
+- **Processes** — every live worker (idle ones included): host, pid, IP, NATS connection name, command line,
+  subscribed streams, busy/concurrency, running/quiet/stopping state, memory, NATS RTT and reconnects. Each worker
+  beats into the `_cosmoprocesses` KV bucket every 10s, and a crashed worker drops out within 60s
+- **Streams** — every stream with its subjects, messages, size, retention and replicas, its consumers' pending,
+  ack-pending and redelivered counts, and pause/resume
 - **Crons** — every schedule deployed in NATS, with run-now and delete
-- **Batches** — open and finished batches with pending/succeeded/failed counts
-- Summary counters (processed / failed / busy / enqueued / retries / scheduled / dead) backed by a NATS KV counter, no separate metrics store needed
+- **Batches** — open and finished batches with their progress, succeeded/failed/pending counts and callbacks
+- **Metrics** — per job class over today, 7, or 30 days: runs, failures, average execution time (successful runs), and
+  average wait from enqueue to first delivery, plus a per-day chart of runs and average execution time. Recorded by
+  `Cosmo::Middleware::Metrics`, flushed by every worker's heartbeat into the internal `_cosmometrics` counters stream;
+  turn it off with `config.metrics.enabled = false`
+
+Each panel refreshes `COSMO_WEB_POLL_INTERVAL` seconds after its previous response, never on top of one still
+running, backs off while refreshes fail, and pauses in a hidden tab. Identical refreshes from several viewers share a
+single NATS read for `COSMO_WEB_CACHE_TTL` seconds, and while NATS is unreachable the UI answers at once instead of
+holding your app server's threads.
 
 **Structured logs:**
 ```

data/lib/cosmo/active_job/adapter.rb

@@ -32,8 +32,7 @@ module Cosmo
         options    = { stream: stream }.merge(cosmo_opts)
         options[:at] = timestamp if timestamp
 
-        data = Job::Data.new(Executor.name, [job.serialize], options)
-        Publisher.publish_job(data)
+        Job::Enqueuer.enqueue(Executor.name, [job.serialize], options)
       end
 
       # Returns Cosmo-specific options declared on the job class via

data/lib/cosmo/api/counter.rb

@@ -2,22 +2,22 @@
 
 module Cosmo
   module API
+    # Base for NATS counters (+Nats-Incr+) kept in a stream of their own. A subclass names it with +STREAM_NAME+ and
+    # +DESCRIPTION+ and may extend {.stream_config}; every instance counts under its namespace in that stream.
     class Counter
-      STREAM_NAME = "_cosmostats"
-      STREAM_CONFIG = {
-        subjects: ["#{STREAM_NAME}.>"],
-        allow_msg_counter: true,
-        allow_direct: true,
-        max_msgs_per_subject: 1,
-        description: "Cosmo statistics"
-      }.freeze
-
-      # Creates the counters stream, or brings an existing one up to STREAM_CONFIG. Every stored
-      # counter message carries the running total, so only the last one per subject is kept.
+      # Creates the counters stream, or brings an existing one up to {.stream_config}.
       #
       # @return [void]
       def self.setup!
-        Client.instance.setup_stream(STREAM_NAME, STREAM_CONFIG)
+        Client.instance.setup_stream(self::STREAM_NAME, stream_config)
+      end
+
+      # Every stored counter message carries the running total, so only the last one per subject is kept.
+      #
+      # @return [Hash]
+      def self.stream_config
+        { subjects: ["#{self::STREAM_NAME}.>"], allow_msg_counter: true, allow_direct: true, max_msgs_per_subject: 1,
+          num_replicas: Config.replicas, description: self::DESCRIPTION }
       end
 
       def initialize(namespace)
@@ -35,12 +35,12 @@ module Cosmo
       alias decr decrement
 
       def reset(key)
-        client.purge(STREAM_NAME, subject(key))
+        client.purge(self.class::STREAM_NAME, subject(key))
       end
       alias purge reset
 
       def get(key)
-        raw = client.get_message(STREAM_NAME, direct: true, subject: subject(key))
+        raw = client.get_message(self.class::STREAM_NAME, direct: true, subject: subject(key))
         Utils::Json.parse(raw.data, default: { "val" => 0 })[:val].to_i
       rescue NATS::JetStream::Error::NotFound, NATS::JetStream::Error::ServiceUnavailable, NATS::IO::Timeout
         0
@@ -58,7 +58,7 @@ module Cosmo
       def publish(key, value, msg_id: nil)
         rescued = nil
         headers = { "Nats-Incr" => value }
-        headers["Nats-Msg-Id"] = msg_id if msg_id
+        headers[NATS::JetStream::Header::MSG_ID] = msg_id if msg_id
 
         begin
           ack = client.publish(subject(key), "", header: headers)
@@ -67,13 +67,13 @@ module Cosmo
           raise if rescued
 
           rescued = true
-          client.create_stream(STREAM_NAME, STREAM_CONFIG)
+          client.create_stream(self.class::STREAM_NAME, self.class.stream_config)
           retry
         end
       end
 
       def subject(key)
-        "#{STREAM_NAME}.#{@namespace}.#{key}"
+        "#{self.class::STREAM_NAME}.#{@namespace}.#{key}"
       end
 
       def client

data/lib/cosmo/api/cron/entry.rb

@@ -66,17 +66,16 @@ module Cosmo
           "jobs.#{@stream}.#{job_name}"
         end
 
-        # Headers stored on the schedule template. NATS copies the +X-+ ones onto every message
-        # it fires, which is how the scheduler learns where the job belongs.
-        def schedule_headers
-          headers = {
-            "Nats-Schedule" => @expression,
-            "Nats-Schedule-Target" => target_subject,
-            "X-Stream" => @stream,
-            "X-Subject" => dispatch_subject
-          }
-          headers["Nats-Schedule-Time-Zone"] = @timezone if @timezone
-          headers
+        # The +schedule:+ option of nats-pure's JetStream#publish that makes the template a schedule.
+        # @-shortcuts (+@daily+, +@every 5m+) travel as the cron expression, which the server understands.
+        def schedule
+          { cron: @expression, target: target_subject, time_zone: @timezone }.compact
+        end
+
+        # Headers stored on the schedule template. NATS copies them onto every message it fires,
+        # which is how the scheduler learns where the job belongs.
+        def headers
+          { "X-Stream" => @stream, "X-Subject" => dispatch_subject }
         end
 
         def job_name

data/lib/cosmo/api/cron.rb

@@ -32,10 +32,13 @@ module Cosmo
 
       # Publish (or replace) a schedule message in NATS.
       # @return [Hash, nil] the persisted schedule as a hash, or nil on failure
+      # @raise [SchedulingDisabledError] when scheduling is turned off
       def upsert!(class_name: nil, stream: nil, schedule: nil, args: [], timezone: nil, name: nil)
+        raise SchedulingDisabledError unless Config.scheduled.enabled
+
         e = Entry.new(class_name: class_name, stream: stream, expression: schedule,
                       args: args, timezone: timezone, name: name)
-        client.publish(e.schedule_subject, e.job_payload, stream: STREAM, header: e.schedule_headers)
+        client.publish(e.schedule_subject, e.job_payload, stream: STREAM, header: e.headers, schedule: e.schedule)
         build_from_nats(e.schedule_subject)
       end
 
@@ -91,8 +94,8 @@ module Cosmo
         {
           class: body[:class],
           stream: headers["X-Stream"],
-          schedule: headers["Nats-Schedule"],
-          timezone: headers["Nats-Schedule-Time-Zone"],
+          schedule: headers[NATS::JetStream::Header::SCHEDULE],
+          timezone: headers[NATS::JetStream::Header::SCHEDULE_TIME_ZONE],
           args: body[:args] || [],
           name: name_from_subject(subject),
           schedule_subject: subject,

data/lib/cosmo/api/job.rb

@@ -62,7 +62,7 @@ module Cosmo
       end
 
       def timestamp
-        headers&.dig("Nats-Time-Stamp")
+        headers&.dig(NATS::JetStream::Header::TIME_STAMP)
       end
 
       # @return [Time, nil] when the message was stored in its stream, parsed from +Nats-Time-Stamp+

data/lib/cosmo/api/kv.rb

@@ -98,14 +98,14 @@ module Cosmo
         results
       end
 
-      # CAS = Compare-And-Swap: publish +value+ with a per-message Nats-TTL,
+      # CAS = Compare-And-Swap: publish +value+ with a per-message TTL of +ttl+ whole seconds (at least 1),
       # but only if the subject's current last sequence matches +last_seq+
       # (sent as the Nats-Expected-Last-Subject-Sequence header). Raises
       # NATS::KeyValue::KeyWrongLastSequenceError if it doesn't match --
       # e.g. last_seq: 0 means "only publish if nothing exists here yet".
       def publish_cas(key, value, ttl, last_seq:)
-        headers = { "Nats-Expected-Last-Subject-Sequence" => last_seq.to_s, "Nats-TTL" => "#{ttl.to_i}s" }
-        client.js.publish("$KV.#{@name}.#{key}", value, header: headers)
+        headers = { NATS::JetStream::Header::EXPECTED_LAST_SUBJECT_SEQUENCE => last_seq.to_s }
+        client.js.publish("$KV.#{@name}.#{key}", value, header: headers, ttl: [ttl.to_i, 1].max)
       rescue NATS::JetStream::Error::APIError => e
         raise NATS::KeyValue::KeyWrongLastSequenceError, e.description if e.err_code == 10_071
 

data/lib/cosmo/api/registry.rb

@@ -57,7 +57,7 @@ module Cosmo
       private
 
       def bucket
-        KV.new(self.class::BUCKET, { ttl: self.class::TTL })
+        KV.new(self.class::BUCKET, { ttl: self.class::TTL, replicas: Config.replicas })
       end
 
       def sanitize(key)