cosmonats 0.8.0 → 0.9.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 685ae4ff6735c88c3605b58f86168ce4acc2509d7dd66c7335cdeab50fc9ffa4
-  data.tar.gz: b508d08c685c6d726c1e3c56c79432d4afc952384d04ae210d0d6d673caf6c01
+  metadata.gz: 40e30ee2fff77d85db1a3deedf05e040c72dca39d64da01220517d3d5ae8048b
+  data.tar.gz: 6f0e5a50448674483f09f54938336940899ea81fe46ce66da23478eeee871bac
 SHA512:
-  metadata.gz: 074bed30d8889561ecf77fd90d5d599e3f3cc9f487df781e08d88135b71b8ab31826b77dad8420b830e1a00632f047c46fee139d6a4e9817c0ca414795e46575
-  data.tar.gz: 99d1169938570c778981447df1385d177ad5a432e355690498c8ddac8a1e4eef6f9eedeb593b1541dc2dd51472906827e80032dcd2b0e4484435d7dce1e67771
+  metadata.gz: 6d57b79ee594fd7a7ace94e1e3f5d70de87b9a413b85389cddb9db51f39ed09542bd0f0a7d3c4308e7009b7a2d9997b9be0c2965efa61e259bd07baf9bfa4734
+  data.tar.gz: 8122f92e84e4176a403678225ab66b753270ab5e49334568bc3e88cfa292451bc63ea6f5383bdb458ab89f8b319a6ed33b7af471f636ffe7314978c956b6acb1

data/README.md

@@ -58,8 +58,6 @@ bundle exec cosmo -C config/cosmo.yml -c 20 jobs    # Jobs only
 bundle exec cosmo -C config/cosmo.yml -c 20 streams # Streams only
 ```
 
-![webui.gif](webui.gif)
-
 ## 📖 Index
 
 - [Why?](#-why)
@@ -433,9 +431,11 @@ Cosmo leaves a custom logger's formatter alone, so its lines lack Cosmo's `jid`/
 | `COSMO_CLIENT_NAME`        | `cosmo-<program>-<host>-<pid>`       | NATS connection name, shown in `nats server report connections` and `/connz`        |
 | `COSMO_JS_TIMEOUT`         | `5`                                  | JetStream API timeout in seconds; raise it for the web UI on a busy server          |
 | `COSMO_CONNECT_TIMEOUT`    | `2`                                  | NATS TCP connect timeout in seconds                                                 |
+| `NATS_MAX_RECONNECT_ATTEMPTS` | `-1`                              | Reconnect attempts before giving up on NATS; `-1` keeps trying through any outage   |
 | `COSMO_JOBS_STREAMS`       | all configured                       | Comma-separated job streams to subscribe to, when `--streams`/`--stream` isn't given |
 | `COSMO_JOBS_FETCH_TIMEOUT` | `0.1`                                | Seconds a jobs worker waits on each fetch                                           |
 | `COSMO_WEB_POLL_INTERVAL`  | `5`                                  | Web UI auto-refresh interval in seconds                                             |
+| `COSMO_WEB_CACHE_TTL`      | `2`                                  | Seconds the web UI shares a refresh between viewers; `0` turns sharing off          |
 
 
 ## 🔧 Advanced Usage
@@ -543,7 +543,7 @@ end
   ```
 - Batch tracking data (pending counts, callbacks, results) expires automatically after
   `config.batches.expiry` (default: 3 days), set in `Cosmo.configure`.
-- Open and finished batches — with pending/succeeded/failed counts — are listed live in the web UI's **Batches** tab.
+- Open and finished batches — with pending/succeeded/failed counts — are listed live on the web UI's **Batches** page.
 - Only `Cosmo::Job`-based jobs are tracked; the ActiveJob adapter doesn't currently participate in
   batches (see [`docs/active_job.md`](docs/active_job.md)).
 
@@ -896,19 +896,29 @@ sudo systemctl enable cosmo && sudo systemctl start cosmo
 
 ## 📊 Monitoring
 
-**Web UI** — mount `Cosmo::Web` (see [Installation](#-installation)) for a live, htmx-powered dashboard:
-- **Jobs** — enqueued, scheduled, busy, and dead views, with per-job retry and delete
-- **Processes** — the Jobs dashboard lists every live worker (idle ones included): host, pid, IP, NATS connection
-  name, command line, subscribed streams, busy/concurrency, running/quiet/stopping state, memory, NATS RTT and
-  reconnects. Each worker beats into the `_cosmoprocesses` KV bucket every 10s, and a crashed worker drops out within 60s
-- **Streams** — per-stream state (messages, bytes, consumers) with pause/resume
+**Web UI** — mount `Cosmo::Web` (see [Installation](#-installation)) for a live, htmx-powered dashboard with a
+light, dark, or automatic (system) theme. Fonts and assets are bundled, so it loads nothing from a CDN:
+- **Overview** — processed, failed, busy, enqueued, retries, scheduled and dead totals; a 7- or 30-day throughput
+  chart of successful and failed runs; a grid of every worker thread, busy ones colored by stream; the queues; and the
+  jobs running right now, longest first
+- **Jobs** — each job queue with its size, busy jobs and redeliveries, paused and resumed in place; then busy,
+  enqueued (per queue), scheduled and dead jobs, with retry and delete for dead ones and their error and backtrace
+- **Processes** — every live worker (idle ones included): host, pid, IP, NATS connection name, command line,
+  subscribed streams, busy/concurrency, running/quiet/stopping state, memory, NATS RTT and reconnects. Each worker
+  beats into the `_cosmoprocesses` KV bucket every 10s, and a crashed worker drops out within 60s
+- **Streams** — every stream with its subjects, messages, size, retention and replicas, its consumers' pending,
+  ack-pending and redelivered counts, and pause/resume
 - **Crons** — every schedule deployed in NATS, with run-now and delete
-- **Batches** — open and finished batches with pending/succeeded/failed counts
+- **Batches** — open and finished batches with their progress, succeeded/failed/pending counts and callbacks
 - **Metrics** — per job class over today, 7, or 30 days: runs, failures, average execution time (successful runs), and
   average wait from enqueue to first delivery, plus a per-day chart of runs and average execution time. Recorded by
   `Cosmo::Middleware::Metrics`, flushed by every worker's heartbeat into the internal `_cosmometrics` counters stream;
   turn it off with `config.metrics.enabled = false`
-- Summary counters (processed / failed / busy / enqueued / retries / scheduled / dead) backed by a NATS KV counter, no separate metrics store needed
+
+Each panel refreshes `COSMO_WEB_POLL_INTERVAL` seconds after its previous response, never on top of one still
+running, backs off while refreshes fail, and pauses in a hidden tab. Identical refreshes from several viewers share a
+single NATS read for `COSMO_WEB_CACHE_TTL` seconds, and while NATS is unreachable the UI answers at once instead of
+holding your app server's threads.
 
 **Structured logs:**
 ```

data/lib/cosmo/api/counter.rb

@@ -58,7 +58,7 @@ module Cosmo
       def publish(key, value, msg_id: nil)
         rescued = nil
         headers = { "Nats-Incr" => value }
-        headers["Nats-Msg-Id"] = msg_id if msg_id
+        headers[NATS::JetStream::Header::MSG_ID] = msg_id if msg_id
 
         begin
           ack = client.publish(subject(key), "", header: headers)

data/lib/cosmo/api/cron/entry.rb

@@ -66,17 +66,16 @@ module Cosmo
           "jobs.#{@stream}.#{job_name}"
         end
 
-        # Headers stored on the schedule template. NATS copies the +X-+ ones onto every message
-        # it fires, which is how the scheduler learns where the job belongs.
-        def schedule_headers
-          headers = {
-            "Nats-Schedule" => @expression,
-            "Nats-Schedule-Target" => target_subject,
-            "X-Stream" => @stream,
-            "X-Subject" => dispatch_subject
-          }
-          headers["Nats-Schedule-Time-Zone"] = @timezone if @timezone
-          headers
+        # The +schedule:+ option of nats-pure's JetStream#publish that makes the template a schedule.
+        # @-shortcuts (+@daily+, +@every 5m+) travel as the cron expression, which the server understands.
+        def schedule
+          { cron: @expression, target: target_subject, time_zone: @timezone }.compact
+        end
+
+        # Headers stored on the schedule template. NATS copies them onto every message it fires,
+        # which is how the scheduler learns where the job belongs.
+        def headers
+          { "X-Stream" => @stream, "X-Subject" => dispatch_subject }
         end
 
         def job_name

data/lib/cosmo/api/cron.rb

@@ -38,7 +38,7 @@ module Cosmo
 
         e = Entry.new(class_name: class_name, stream: stream, expression: schedule,
                       args: args, timezone: timezone, name: name)
-        client.publish(e.schedule_subject, e.job_payload, stream: STREAM, header: e.schedule_headers)
+        client.publish(e.schedule_subject, e.job_payload, stream: STREAM, header: e.headers, schedule: e.schedule)
         build_from_nats(e.schedule_subject)
       end
 
@@ -94,8 +94,8 @@ module Cosmo
         {
           class: body[:class],
           stream: headers["X-Stream"],
-          schedule: headers["Nats-Schedule"],
-          timezone: headers["Nats-Schedule-Time-Zone"],
+          schedule: headers[NATS::JetStream::Header::SCHEDULE],
+          timezone: headers[NATS::JetStream::Header::SCHEDULE_TIME_ZONE],
           args: body[:args] || [],
           name: name_from_subject(subject),
           schedule_subject: subject,

data/lib/cosmo/api/job.rb

@@ -62,7 +62,7 @@ module Cosmo
       end
 
       def timestamp
-        headers&.dig("Nats-Time-Stamp")
+        headers&.dig(NATS::JetStream::Header::TIME_STAMP)
       end
 
       # @return [Time, nil] when the message was stored in its stream, parsed from +Nats-Time-Stamp+

data/lib/cosmo/api/kv.rb

@@ -98,14 +98,14 @@ module Cosmo
         results
       end
 
-      # CAS = Compare-And-Swap: publish +value+ with a per-message Nats-TTL,
+      # CAS = Compare-And-Swap: publish +value+ with a per-message TTL of +ttl+ whole seconds (at least 1),
       # but only if the subject's current last sequence matches +last_seq+
       # (sent as the Nats-Expected-Last-Subject-Sequence header). Raises
       # NATS::KeyValue::KeyWrongLastSequenceError if it doesn't match --
       # e.g. last_seq: 0 means "only publish if nothing exists here yet".
       def publish_cas(key, value, ttl, last_seq:)
-        headers = { "Nats-Expected-Last-Subject-Sequence" => last_seq.to_s, "Nats-TTL" => "#{ttl.to_i}s" }
-        client.js.publish("$KV.#{@name}.#{key}", value, header: headers)
+        headers = { NATS::JetStream::Header::EXPECTED_LAST_SUBJECT_SEQUENCE => last_seq.to_s }
+        client.js.publish("$KV.#{@name}.#{key}", value, header: headers, ttl: [ttl.to_i, 1].max)
       rescue NATS::JetStream::Error::APIError => e
         raise NATS::KeyValue::KeyWrongLastSequenceError, e.description if e.err_code == 10_071
 

data/lib/cosmo/api/stats/busy.rb

@@ -43,6 +43,11 @@ module Cosmo
           super.map { _1.merge(data: Utils::Json.parse(_1[:data])) }
         end
 
+        # @return [Hash{String => Integer}] the number of jobs running per stream, for the streams running any
+        def by_stream
+          all.group_by { _1[:stream] }.transform_values(&:size)
+        end
+
         private
 
         def heartbeat_loop

data/lib/cosmo/api/stats/metrics.rb

@@ -67,12 +67,12 @@ module Cosmo
           totals.map { |job, sums| row(sums).merge(job: job) }.sort_by { -_1[:count] }
         end
 
-        # @param job_class [String]
+        # @param job_class [String, nil] nil sums every job class
         # @param days [Integer]
         # @return [Array<Hash>] one +{ date:, count:, failed:, exec_ms:, wait_ms: }+ per day, oldest first
-        def daily(job_class, days: 30)
+        def daily(job_class = nil, days: 30) # rubocop:disable Metrics/AbcSize
           sums = Hash.new { |hash, day| hash[day] = Hash.new(0) }
-          read(days).each { |(day, job, field), value| sums[day][field] += value if job == job_class }
+          read(days).each { |(day, job, field), value| sums[day][field] += value if job_class.nil? || job == job_class }
           dates(days).map { |day| row(sums[day]).merge(date: day) }
         end
 

data/lib/cosmo/api/stats.rb

@@ -41,6 +41,21 @@ module Cosmo
       def dead
         Config.dead.enabled ? Stream.new("dead").size : 0
       end
+
+      # @return [Array<Hash>] a {.queue} row per job stream
+      def queues
+        busy = Busy.instance.by_stream
+        Stream.jobs.map { queue(_1, busy) }
+      end
+
+      # @param stream [Stream] a job stream
+      # @param busy [Hash{String => Integer}] running jobs per stream, as {Busy#by_stream} counts them
+      # @return [Hash] the stream's +:name+, waiting messages as +:size+, running jobs as +:busy+, redeliveries as
+      #   +:retries+, and whether it is +:paused+
+      def queue(stream, busy = Busy.instance.by_stream)
+        { name: stream.name, size: stream.size, busy: busy.fetch(stream.name, 0), retries: stream.retries,
+          paused: stream.paused? }
+      end
     end
   end
 end

data/lib/cosmo/api/stream.rb

@@ -44,9 +44,14 @@ module Cosmo
       alias size total
 
       def retries
-        client.list_consumers(name).sum { _1["num_redelivered"].to_i }
+        consumers.sum { _1["num_redelivered"].to_i }
+      end
+
+      # @return [Array<Hash>] the stream's consumers as JetStream reports them, with their config and delivery state
+      def consumers
+        client.list_consumers(name)
       rescue NATS::Error
-        0
+        []
       end
 
       def each

data/lib/cosmo/client.rb

@@ -2,7 +2,6 @@
 
 require "socket"
 require "nats/client"
-require "cosmo/utils/overrides"
 
 module Cosmo
   class Client
@@ -10,6 +9,9 @@ module Cosmo
     JS_TIMEOUT = 5
     # Seconds to wait for the TCP connect+handshake. Override with COSMO_CONNECT_TIMEOUT.
     CONNECT_TIMEOUT = 2
+    # Reconnect attempts before the connection closes for good; -1 never gives up, so a process outlives a NATS
+    # outage. nats-pure lets NATS_MAX_RECONNECT_ATTEMPTS override it.
+    MAX_RECONNECT_ATTEMPTS = -1
 
     def self.instance
       @instance ||= Client.new
@@ -41,7 +43,8 @@ module Cosmo
     def initialize(nats_url: ENV.fetch("NATS_URL", "nats://localhost:4222"), name: self.class.default_name)
       @name = name
       Logger.debug "Connecting to NATS server at #{nats_url} as #{@name}..."
-      @nc = NATS.connect(nats_url, name: @name, connect_timeout: self.class.connect_timeout)
+      @nc = NATS.connect(nats_url, name: @name, connect_timeout: self.class.connect_timeout,
+                                   max_reconnect_attempts: MAX_RECONNECT_ATTEMPTS)
       Logger.debug "Connection established"
       @js = @nc.jetstream(timeout: self.class.js_timeout)
     end
@@ -157,7 +160,7 @@ module Cosmo
       messages = {}
       loop do
         msg = sub.next_msg(timeout: self.class.js_timeout)
-        subject = msg.header&.dig("Nats-Subject")
+        subject = msg.header&.dig(NATS::JetStream::Header::SUBJECT)
         break unless subject
 
         messages[subject] = msg.data
@@ -195,8 +198,9 @@ module Cosmo
 
     private
 
-    # NOTE: nats-pure's #pull_subscribe has no path to a true ephemeral pull consumer.
-    # Its rescue branch unconditionally sets `config[:durable_name] = durable` (jetstream.rb).
+    # NOTE: nats-pure's #pull_subscribe can't create a true ephemeral pull consumer: its rescue branch
+    # unconditionally sets `config[:durable_name] = durable` (jetstream.rb). So the consumer is created
+    # here, and #pull_subscribe only binds to it by name.
     def ephemeral_subscribe(subject, config) # rubocop:disable Metrics/AbcSize, Metrics/CyclomaticComplexity, Metrics/PerceivedComplexity
       config = config.dup
       stream = config.delete(:stream) or raise ArgumentError, "stream required for ephemeral consumers"
@@ -211,14 +215,7 @@ module Cosmo
       end
 
       info = js.add_consumer(stream, consumer_config)
-
-      sub = nc.subscribe(nc.new_inbox)
-      sub.extend(NATS::JetStream.const_get(:PullSubscription))
-      sub.jsi = NATS::JetStream.const_get(:JS)::Sub.new(
-        js: js, stream: stream, consumer: info.name,
-        nms: "#{js.prefix}.CONSUMER.MSG.NEXT.#{stream}.#{info.name}"
-      )
-      sub
+      js.pull_subscribe(subject, nil, stream: stream, name: info.name)
     end
 
     # NOTE: KV manager in nats-pure hardcodes the fields it copies into StreamConfig,

data/lib/cosmo/job/data.rb

@@ -44,7 +44,7 @@ module Cosmo
 
       # @return [Hash{String => String, Integer}] dedup id, plus where to dispatch a delayed job once it is due
       def headers
-        headers = { "Nats-Msg-Id" => jid }
+        headers = { NATS::JetStream::Header::MSG_ID => jid }
         return headers unless @at
 
         target = stream(target: true)

data/lib/cosmo/job/processor.rb

@@ -8,8 +8,8 @@ module Cosmo
       # Headers the scheduler consumes rather than forwards. The +X-+ ones address the dispatch
       # itself, and NATS sets +Nats-Scheduler+/+Nats-Schedule-Next+ on a message it fires and
       # rejects a publish that carries them back.
-      DISPATCH_HEADERS = %w[X-Stream X-Subject X-Execute-At Nats-Expected-Stream
-                            Nats-Scheduler Nats-Schedule-Next].freeze
+      DISPATCH_HEADERS = ["X-Stream", "X-Subject", "X-Execute-At", NATS::JetStream::Header::EXPECTED_STREAM,
+                          NATS::JetStream::Header::SCHEDULER, NATS::JetStream::Header::SCHEDULE_NEXT].freeze
 
       # @raise [UnknownJobStreamError] when --streams/--stream or COSMO_JOBS_STREAMS names an unconfigured stream
       def self.validate_options!(options)
@@ -71,8 +71,7 @@ module Cosmo
           messages&.each do |message|
             headers = message.header.except(*DISPATCH_HEADERS)
             stream, subject, execute_at = message.header.values_at("X-Stream", "X-Subject", "X-Execute-At")
-            headers["Nats-Expected-Stream"] = stream
-            scheduler = message.header["Nats-Scheduler"]
+            scheduler = message.header[NATS::JetStream::Header::SCHEDULER]
             if scheduler
               headers["X-Scheduled-By"] = scheduler
               headers["X-Enqueued-At"] = message.metadata.timestamp.to_f.to_s
@@ -80,7 +79,7 @@ module Cosmo
             execute_at = execute_at.to_i
 
             if now >= execute_at
-              client.publish(subject, message.data, header: headers)
+              client.publish(subject, message.data, header: headers, stream:)
               message.ack
             else
               message.nak(delay: Config.to_ns(execute_at - now))
@@ -262,7 +261,7 @@ module Cosmo
       def scheduled_by(message)
         header = message.header or return
 
-        header["X-Scheduled-By"] || header["Nats-Scheduler"]
+        header["X-Scheduled-By"] || header[NATS::JetStream::Header::SCHEDULER]
       end
 
       # A cron message is enqueued when NATS fires it, not when the scheduler dispatches it on, so

data/lib/cosmo/utils.rb

@@ -4,7 +4,6 @@ require "cosmo/utils/hash"
 require "cosmo/utils/json"
 require "cosmo/utils/string"
 require "cosmo/utils/signal"
-require "cosmo/utils/warnings"
 require "cosmo/utils/duration"
 require "cosmo/utils/stopwatch"
 require "cosmo/utils/thread_pool"

data/lib/cosmo/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Cosmo
-  VERSION = "0.8.0"
+  VERSION = "0.9.0"
 end