protobuf-nats 0.13.1 → 0.13.2.pre2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: aeb96c5f90852f8dfdf32450fe5290325684cec82dbb88bd7b7e393db09cb6bb
-  data.tar.gz: b577ae715c8fc9669b815510f0e2032b9454e9aef63c58731590c2fe2c3152c0
+  metadata.gz: 192c0200a6f76e8efb9e165fa8fd4f9ac11240cf096edf866516b604d591d932
+  data.tar.gz: 97742cf64382a4f1286140c05da7dac353fa97c9f026310f6702fb5d2acbbf55
 SHA512:
-  metadata.gz: 81fe350143450675623252856ce7fe30253e3204699c9a2a6dbe8c7c270ea381567d55d3b7bde2b2c17078c64e46700a5f35a75f660bcc7b6f3e81b5fc4ea5b9
-  data.tar.gz: f8c98030a76607518ecfdce5c47a9f2cba2d2ab52ffddfd46a75e0d173e5d62203b122fe747aacd805245815ae4901c1625f7147d5f4228f0497f94ea2f41aec
+  metadata.gz: 43143595525a958c16eccdf4eaed2ed4132f4d805924411a5fecbcd08c979b850a887fa4ddc52c5cce31261199dc26d63058f213767bb7180f631f8e1066d39f
+  data.tar.gz: c34ee341de5f07d4cc5bcbec49a860b9024b62b138e25af96fe962e84fa45886fc38f0cea5b78eed8c6b1e4c103f3c6bca91024a4d4847cb41767606cc31bf9a

data/CHANGELOG.md

@@ -1,5 +1,17 @@
 ## Changelog
 
+### 0.13.2
+Bounds the RPC transport's in-memory buffering to prevent the JVM-heap OOM introduced by the JNats → nats-pure migration. Both the client response muxer and the server intake queue are now capped by message count **and** total bytes, dropping (with client retry) rather than buffering unbounded protobuf payloads on the heap.
+
+#### Client: response-muxer heap bound
+- The shared response "firehose" is bounded by both a message count (`PB_NATS_RESPONSE_MUXER_QUEUE_SIZE`, default `1024`) and a byte ceiling (`PB_NATS_RESPONSE_MUXER_QUEUE_BYTES`, default 64 MiB); nats-pure drops (`SlowConsumer`) on whichever trips first and the RPC retries. Previously only nats-pure's 65,536-message count applied with the byte limit disabled, so a burst of large responses could hold gigabytes of Ruby objects on the JVM heap.
+- The muxer now decrements the subscription's `pending_size` after each pop, keeping a *finite* byte limit accurate — instead of disabling it as before. If a subscription can't support that accounting (no `#synchronize`), `start` raises `IncompatibleSubscription` (a tripwire for a breaking nats-pure change) rather than silently degrading.
+- New gauges: `response_muxer.pending_queue_size` and `response_muxer.pending_queue_peak` (high-water mark between the ~60s samples).
+
+#### Server: intake heap bound
+- The shared intake queue is now bounded by bytes as well as count: new `PB_NATS_SERVER_INTAKE_QUEUE_BYTES` (default 128 MiB), enforced by a `ByteBoundedQueue` with a shared byte counter. A request that would exceed the ceiling is dropped (the client retries) and emits `server.intake_bytes_dropped`; new gauge `server.pending_intake_queue_bytes`. nats-pure's per-subscription byte limit stays disabled — the shared queue counter owns byte bounding, since many subscriptions funnel into one queue.
+- Fixed a slow leak of orphaned `@overdue_flagged` entries caused by a handler-completion race; the periodic monitor now reaps them.
+
 ### 0.13.1
 Fixes regressions from the JNats → nats-pure migration (0.13.0) plus a full reliability, performance, and security hardening pass. Highlights: the client reconnects and retries correctly through dropped connections, failing nodes, and terminal closes; the server survives overload and connection loss instead of going silently deaf; TLS actually verifies the server certificate.
 

data/README.md

@@ -80,6 +80,8 @@ default is used, instead of silently becoming `0`.
 | `PB_NATS_CLIENT_RECONNECT_DELAY` | ACK timeout | Seconds to sleep before retrying after a transient transport error — see [Resilience](#resilience). |
 | `PB_NATS_CLIENT_RECONNECT_DELAY_SPLAY_LIMIT` | `1000` | Random jitter (ms, `0..limit`) added to the reconnect delay so a fleet doesn't retry in lockstep. `0` disables. |
 | `PB_NATS_RESPONSE_MUXER_DISPATCHERS` | CPUs on JRuby, `1` on CRuby | Threads draining the shared response subscription (min 1). |
+| `PB_NATS_RESPONSE_MUXER_QUEUE_SIZE` | `1024` | Message-count cap for the shared response subscription. Dispatchers drain it to ~0, so this is burst headroom, not a working set: each in-flight request holds only ~2 messages (ACK + response). Beyond it nats-pure drops (`SlowConsumer`) and the RPC retries, rather than buffering unbounded response objects on the heap. Set it to your app's request-thread-pool size if that exceeds the default (min 1). |
+| `PB_NATS_RESPONSE_MUXER_QUEUE_BYTES` | `67108864` (64 MiB) | Byte cap for the shared response subscription — the true heap ceiling. The count cap alone says nothing about size (1024 large payloads can still be gigabytes), so the firehose is bounded by whichever trips first: `PB_NATS_RESPONSE_MUXER_QUEUE_SIZE` messages or this many bytes. Matches the NATS ecosystem's per-subscription byte default (nats-pure / nats.go both use 64 MiB). Raise it if you have large payloads and heap to spare; lower it to tighten the ceiling (min 1). |
 
 #### Server
 
@@ -87,7 +89,8 @@ default is used, instead of silently becoming `0`.
 | --- | --- | --- |
 | `PB_NATS_SERVER_MAX_QUEUE_SIZE` | thread count | Queue in front of the handler thread pool; requests beyond it are NACKed. |
 | `PB_NATS_SERVER_SUBSCRIPTION_HANDLERS` | CPUs on JRuby, `1` on CRuby | Threads draining the shared intake queue and publishing ACK/NACKs (min 1). Consumer parallelism only — does not change queue-group delivery. |
-| `PB_NATS_SERVER_INTAKE_QUEUE_SIZE` | `65536` | Capacity of the shared intake queue. Smaller turns overload into prompt drops-and-retries instead of a deep stale backlog; tune down alongside `PB_NATS_SERVER_STALE_REQUEST_MS`. |
+| `PB_NATS_SERVER_INTAKE_QUEUE_SIZE` | `65536` | Message-count capacity of the shared intake queue. Smaller turns overload into prompt drops-and-retries instead of a deep stale backlog; tune down alongside `PB_NATS_SERVER_STALE_REQUEST_MS`. |
+| `PB_NATS_SERVER_INTAKE_QUEUE_BYTES` | `134217728` (128 MiB) | Byte capacity of the shared intake queue — the aggregate-heap bound the count alone can't give (65,536 large requests is a lot of heap). A request that would exceed it is dropped (the client retries), never buffered. Bounds bytes across all subscriptions; higher than the client muxer's 64 MiB since the server's count cap is higher too (min 1). |
 | `PB_NATS_SERVER_SUBSCRIPTIONS_PER_RPC_ENDPOINT` | `10` | Subscriptions created per endpoint (lets JVM servers warm up gradually). Queue groups still deliver each request to exactly one consumer. |
 | `PB_NATS_SERVER_SLOW_START_DELAY` | `10` | Seconds between slow-start subscription rounds. |
 | `PB_NATS_SERVER_PAUSE_FILE_PATH` | `nil` | While this file exists the server unsubscribes from all services; it resubscribes (with slow start) when the file is removed. |
@@ -166,7 +169,11 @@ NATS server certificate, or the connection will be rejected. Hostname (SAN/CN) v
   threads, so a slow ACK publish can't head-of-line block other subjects. Handlers self-heal like the muxer.
 - **Observability** — thread-pool gauges plus in-flight handler metrics (`server.inflight_count`,
   `server.inflight_oldest_age_ms`, `server.overdue_handler_count`, `server.pending_intake_queue_size`,
-  `server.thread_pool_saturated`). Error callbacks run on a bounded background executor; drops are counted
+  `server.pending_intake_queue_bytes`, `server.thread_pool_saturated`). A request dropped because it would exceed the
+  intake byte ceiling emits `server.intake_bytes_dropped` (see `PB_NATS_SERVER_INTAKE_QUEUE_BYTES`). The client muxer
+  gauges its response firehose (`response_muxer.pending_queue_size` and, so a burst between the ~60s samples isn't
+  missed, `response_muxer.pending_queue_peak`), plus `response_muxer.stale_tokens_cleaned`, `client.unexpected_message`,
+  and `client.invalid_message`. Error callbacks run on a bounded background executor; drops are counted
   (`Protobuf::Nats.error_callback_drop_count`) and emit `error_callback_dropped`.
 
 ## Resilience

data/lib/protobuf/nats/byte_bounded_queue.rb

@@ -0,0 +1,72 @@
+require "concurrent"
+
+module Protobuf
+  module Nats
+    # A SizedQueue that additionally bounds the total *bytes* of its contents,
+    # not just the message count. The server funnels every subscription into one
+    # shared intake queue, so a per-subscription byte limit (nats-pure's
+    # pending_bytes_limit) can't bound the aggregate heap -- this shared counter
+    # can. Count is still bounded by the SizedQueue capacity it inherits.
+    #
+    # When a push would exceed the byte ceiling we DROP the message rather than
+    # block: pushes happen on nats-pure's read thread (Subscription#dispatch), and
+    # blocking it would stall PING/PONG and every other subject. A drop mirrors
+    # nats-pure's own SlowConsumer behaviour. Non-message items (the :shutdown
+    # poison pill) carry zero bytes, so they are never dropped by the byte gate.
+    #
+    # A drop invokes the optional +on_drop+ callback with the dropped byte count,
+    # so the caller owns any (context-specific) instrumentation rather than this
+    # generic queue class hard-coding it.
+    class ByteBoundedQueue < ::SizedQueue
+      def initialize(max_msgs, max_bytes, on_drop: nil)
+        super(max_msgs)
+        @max_bytes = max_bytes
+        @on_drop = on_drop
+        @bytes = ::Concurrent::AtomicFixnum.new(0)
+      end
+
+      # Enqueue unless it would exceed the byte ceiling. The check-then-add races
+      # only concurrent pops (which lower @bytes), so the ceiling can be exceeded
+      # by at most one in-flight message -- a soft limit, like nats-pure's own
+      # byte accounting. Returns self (SizedQueue#push contract). Raises
+      # ThreadError from super on a non_block push into a count-full queue, before
+      # any bytes are counted.
+      def push(obj, non_block = false)
+        bytes = byte_size(obj)
+        if bytes > 0 && (@bytes.value + bytes) > @max_bytes
+          @on_drop&.call(bytes)
+          return self
+        end
+        super(obj, non_block)
+        @bytes.increment(bytes)
+        self
+      end
+      alias_method :<<, :push
+
+      def pop(non_block = false)
+        obj = super
+        # nil == closed/empty non_block; nothing dequeued, nothing to subtract.
+        @bytes.update { |value| [value - byte_size(obj), 0].max } if obj
+        obj
+      end
+
+      def clear
+        super
+        @bytes.value = 0
+      end
+
+      # Current resident byte total (gauge for observability).
+      def bytesize
+        @bytes.value
+      end
+
+      private
+
+      # Bytes attributable to a queued item. NATS::Msg carries #data; the
+      # :shutdown poison pill (and any other non-message sentinel) counts as 0.
+      def byte_size(obj)
+        obj.respond_to?(:data) && obj.data ? obj.data.bytesize : 0
+      end
+    end
+  end
+end

data/lib/protobuf/nats/errors.rb

@@ -13,6 +13,22 @@ module Protobuf
       class ResponseMuxer < ClientError
       end
 
+      # Raised by ResponseMuxer#start when the response subscription can't support
+      # the muxer's pending_size byte accounting (it doesn't respond to
+      # #synchronize). nats-pure's Subscription always includes MonitorMixin, so
+      # in practice this never fires against a real connection -- it's a tripwire
+      # for a significant change in nats-pure's internals (or a non-standard
+      # injected client). We take @resp_sub.synchronize on every pop to decrement
+      # pending_size and keep the finite byte cap accurate; without it that counter
+      # would only grow and eventually false-trip the limit, silently dropping
+      # every response. Failing loudly at start beats degrading silently at runtime.
+      #
+      # Deliberately NOT in RETRYABLE_TRANSPORT_ERRORS: retrying can't fix a
+      # structural mismatch. NOTE: intentionally undocumented in the README -- it's
+      # an internal invariant/tripwire, not a user-facing knob or metric.
+      class IncompatibleSubscription < ClientError
+      end
+
       class MriIOException < ::StandardError
       end
 

data/lib/protobuf/nats/response_muxer.rb

@@ -13,6 +13,30 @@ module Protobuf
       MAX_RESPONSES_PER_TOKEN = 10
       TOKEN_TTL_SECONDS = 600 # 10 minutes
 
+      # The shared response subscription is bounded by BOTH a message count and a
+      # byte ceiling; nats-pure drops (SlowConsumer) on whichever trips first, so
+      # the firehose is capped at min(count, bytes) instead of buffering unbounded
+      # protobuf payloads on the JVM heap (the 0.13.2 OOM). Dispatchers drain it to
+      # ~0, so these are burst headroom, not a working set.
+      #
+      # The count is deliberately tighter than the ecosystem's per-subscription
+      # defaults (nats-pure 65,536; nats.go 500,000): those bound off-heap buffers,
+      # this is Ruby objects on the heap, and the byte cap is the real ceiling. The
+      # byte default stays aligned at 64 MiB (nats-pure/nats.go both use it).
+      # Override via PB_NATS_RESPONSE_MUXER_QUEUE_SIZE / _QUEUE_BYTES.
+      DEFAULT_RESPONSE_QUEUE_SIZE = 1024
+      DEFAULT_RESPONSE_QUEUE_BYTES = 64 * 1024 * 1024 # 64MiB
+
+      # Sentinel pushed onto a token's queue to wake a waiter blocked in
+      # next_message. We cannot rely on Queue#close alone: on JRuby, close does
+      # NOT wake a pop() that is blocked with a timeout: -- neither the native
+      # Queue (Ruby >= 3.2 / JRuby 10) nor concurrent-ruby's RubyTimeoutQueue
+      # (Ruby < 3.2 / JRuby 9.4, whose timed pop only wakes on push) signals a
+      # timed waiter on close. CRuby's Queue#close does wake it, which is why
+      # this only ever bit JRuby. Pushing an explicit sentinel wakes the waiter
+      # immediately on every engine; next_message treats it as a timeout.
+      QUEUE_WAKE = ::Object.new
+
       def initialize
         # Per-token response queues for lock-free message delivery. @resp_map is a
         # Concurrent::Map so request threads and dispatcher threads can insert,
@@ -42,6 +66,12 @@ module Protobuf
         # run_dispatch_loop), so a later transient crash restarts the backoff
         # from 1s instead of staying pinned at the cap.
         @crash_count = ::Concurrent::AtomicFixnum.new(0)
+
+        # High-water mark of the response queue depth since the last cleanup
+        # cycle. Sampled in the dispatch loop, emitted+reset by the cleanup thread
+        # (response_muxer.pending_queue_peak) so a burst between gauge samples is
+        # still visible.
+        @pending_queue_peak = ::Concurrent::AtomicFixnum.new(0)
       end
 
       def logger
@@ -65,10 +95,42 @@ module Protobuf
         end
       end
 
+      # Message-count and byte caps for the shared response subscription (see
+      # DEFAULT_RESPONSE_QUEUE_SIZE / _BYTES). Read once each, in #start, so no
+      # memoization is needed.
+      def response_queue_size
+        ::Protobuf::Nats.env_int("PB_NATS_RESPONSE_MUXER_QUEUE_SIZE", DEFAULT_RESPONSE_QUEUE_SIZE, :min => 1)
+      end
+
+      def response_queue_bytes
+        ::Protobuf::Nats.env_int("PB_NATS_RESPONSE_MUXER_QUEUE_BYTES", DEFAULT_RESPONSE_QUEUE_BYTES, :min => 1)
+      end
+
+      # Current depth of the shared firehose; 0 before the muxer starts. Gauge for
+      # observability -- mirrors SuperSubscriptionManager#pending_queue_size.
+      def pending_queue_size
+        @resp_sub&.pending_queue&.size || 0
+      end
+
       def cleanup(token)
-        # Atomic remove-and-return; close the queue to wake any waiting threads.
+        # Atomic remove-and-return; wake+close the queue to release any waiter.
         entry = @resp_map.delete(token)
-        entry[:queue]&.close if entry
+        wake_and_close_queue(entry[:queue]) if entry
+      end
+
+      # Wake any waiter blocked in next_message on this queue, then close it.
+      # Pushing QUEUE_WAKE is what actually wakes a timed pop on JRuby (see the
+      # QUEUE_WAKE comment); close alone is insufficient there. Safe to call on
+      # an already-closed queue.
+      def wake_and_close_queue(queue)
+        return unless queue
+        begin
+          queue.push(QUEUE_WAKE)
+        rescue ::ClosedQueueError, ::ThreadError
+          # Already closed by another path; a plain (untimed) waiter, if any,
+          # was already woken by that close. Nothing more to do.
+        end
+        queue.close
       end
 
       def next_message(token, timeout)
@@ -101,7 +163,10 @@ module Protobuf
           # Queue.pop returns nil when:
           # 1. The queue is closed
           # 2. The timeout expires
-          unless msg
+          # QUEUE_WAKE is the sentinel pushed by wake_and_close_queue to wake a
+          # timed pop on JRuby (where close alone does not); treat it as a
+          # timeout so the caller fails over instead of returning garbage.
+          if msg.nil? || msg.equal?(QUEUE_WAKE)
             logger.warn "Queue closed or timeout for token #{token} during next_message"
             raise ::NATS::Timeout
           end
@@ -227,9 +292,24 @@ module Protobuf
           begin
             @resp_inbox_prefix = nats.new_inbox
 
-            # Subscribe to our per-instance inbox
+            # Subscribe to our per-instance inbox.
             @resp_sub = nats.subscribe("#{@resp_inbox_prefix}.*")
-            ::Protobuf::Nats.disable_subscription_byte_limit!(@resp_sub)
+
+            # The dispatch loop takes @resp_sub.synchronize to decrement
+            # pending_size after each pop, which keeps the finite byte cap accurate.
+            # nats-pure's Subscription includes MonitorMixin, so this always holds;
+            # if it ever doesn't, nats-pure's internals changed in a way that would
+            # break byte accounting (a growing counter that false-trips the limit
+            # and drops every response). Fail loudly rather than degrade silently.
+            unless @resp_sub.respond_to?(:synchronize)
+              raise ::Protobuf::Nats::Errors::IncompatibleSubscription,
+                "NATS subscription does not respond to #synchronize; cannot maintain pending_size byte accounting (nats-pure internals changed?)"
+            end
+
+            # Bound the firehose by both message count and bytes (see
+            # DEFAULT_RESPONSE_QUEUE_SIZE / _BYTES).
+            @resp_sub.pending_msgs_limit = response_queue_size
+            @resp_sub.pending_bytes_limit = response_queue_bytes
             @subscribed_nats.set(nats)
             @started = true
           rescue => e
@@ -293,13 +373,26 @@ module Protobuf
           next unless data
           stale_count += 1
           logger.warn "Cleaning up stale token #{token} created at #{data[:created_at]}"
-          # Close the queue to wake any waiting threads
-          data[:queue]&.close
+          # Wake any waiting thread, then close the queue.
+          wake_and_close_queue(data[:queue])
         end
 
         if stale_count > 0
           ::Protobuf::Nats.instrument "response_muxer.stale_tokens_cleaned", stale_count
         end
+
+        # Gauge the shared response firehose so a climbing backlog is visible
+        # before it turns into timeouts/SlowConsumer drops. current == depth at
+        # sample time; peak == high-water since the last cycle (reset here).
+        ::Protobuf::Nats.instrument "response_muxer.pending_queue_size", pending_queue_size
+        # Atomic read-and-reset of the high-water mark (AtomicFixnum has no
+        # get_and_set): capture the prior value inside the update block.
+        peak = 0
+        @pending_queue_peak.update do |current_value|
+          peak = current_value
+          0 # set to 0
+        end
+        ::Protobuf::Nats.instrument "response_muxer.pending_queue_peak", peak
       end
 
       # Stop the cleanup thread
@@ -348,7 +441,7 @@ module Protobuf
       # Must be called while holding LOCK (only from drop_subscription_locked).
       def fail_inflight_requests
         @resp_map.each_pair do |_token, entry|
-          entry[:queue]&.close
+          wake_and_close_queue(entry[:queue])
         end
       end
 
@@ -419,6 +512,19 @@ module Protobuf
               next
             end
 
+            # Drop the popped message's bytes from pending_size. nats-pure only
+            # decrements it in #process, which we bypass by popping pending_queue
+            # directly; without this the counter climbs monotonically and would
+            # false-trip the finite pending_bytes_limit, dropping every later
+            # response. Take the same monitor nats-pure's read thread uses.
+            # (#start guarantees the subscription responds to #synchronize.)
+            sub.synchronize { sub.pending_size -= msg.data.size }
+
+            # Sample post-pop depth into the high-water mark so a burst that fills
+            # and drains between the 60s gauge samples is still visible.
+            depth = sub.pending_queue.size
+            @pending_queue_peak.update { |current_value| [depth, current_value].max }
+
             dispatch_message(msg)
 
             # A processed message means this dispatcher is healthy: let the

data/lib/protobuf/nats/server.rb

@@ -182,9 +182,19 @@ module Protobuf
         end
 
         ::Protobuf::Nats.instrument("server.pending_intake_queue_size", subscription_manager.pending_queue_size)
+        ::Protobuf::Nats.instrument("server.pending_intake_queue_bytes", subscription_manager.pending_queue_bytes)
         ::Protobuf::Nats.instrument("server.inflight_count", count)
         ::Protobuf::Nats.instrument("server.inflight_oldest_age_ms", oldest_age_ms)
         ::Protobuf::Nats.instrument("server.overdue_handler_count", overdue)
+
+        # Reap orphaned overdue flags. The handler's ensure normally deletes
+        # @overdue_flagged[id], but the flag set above can race a completing
+        # handler: we read id from @inflight, the ensure deletes both maps, then
+        # we set @overdue_flagged[id] -- an entry nothing else will ever remove.
+        # A flag whose id is no longer in-flight is by definition orphaned.
+        @overdue_flagged.each_key do |id|
+          @overdue_flagged.delete(id) unless @inflight.key?(id)
+        end
       end
 
       # Defaults to #threads (not the raw option) so a server built with no

data/lib/protobuf/nats/super_subscription_manager.rb

@@ -5,13 +5,19 @@ require "timeout"
 require "protobuf/rpc/server"
 require "protobuf/rpc/service"
 require "protobuf/nats/thread_pool"
+require "protobuf/nats/byte_bounded_queue"
 
 module Protobuf
   module Nats
     class SuperSubscriptionManager
       def initialize(nats, &cb)
-        # Central queue used by all subscriptions
-        @pending_queue = ::SizedQueue.new(intake_queue_size)
+        # Central queue used by all subscriptions, bounded by both message count
+        # and total bytes (see intake_queue_size / intake_queue_bytes). A byte-cap
+        # drop is surfaced here as server.intake_bytes_dropped.
+        @pending_queue = ::Protobuf::Nats::ByteBoundedQueue.new(
+          intake_queue_size, intake_queue_bytes,
+          :on_drop => lambda { |bytes| ::Protobuf::Nats.instrument("server.intake_bytes_dropped", bytes) }
+        )
         @subscriptions = []
         @subscriptions_mutex = ::Mutex.new
         @nats = nats
@@ -52,6 +58,19 @@ module Protobuf
         @intake_queue_size ||= ::Protobuf::Nats.env_int("PB_NATS_SERVER_INTAKE_QUEUE_SIZE", ::NATS::IO::DEFAULT_SUB_PENDING_MSGS_LIMIT, :min => 1)
       end
 
+      # Byte ceiling for the shared intake queue -- the aggregate-heap bound the
+      # message count alone can't give (65,536 large requests is a lot of heap).
+      # Bounds resident bytes across ALL subscriptions; the ByteBoundedQueue drops
+      # a message that would exceed it rather than block nats-pure's read thread.
+      # Default 128 MiB: higher than the client muxer's 64 MiB because the server
+      # fans requests across many handler threads and its count cap is higher too.
+      DEFAULT_INTAKE_QUEUE_BYTES = 128 * 1024 * 1024 # 128MiB
+
+      # Read once, in #initialize, so no memoization is needed.
+      def intake_queue_bytes
+        ::Protobuf::Nats.env_int("PB_NATS_SERVER_INTAKE_QUEUE_BYTES", DEFAULT_INTAKE_QUEUE_BYTES, :min => 1)
+      end
+
       def queue_subscribe(name)
         logger.debug { "queue_subscribe(#{name})" }
         sub = @nats.subscribe(name, :queue => name)
@@ -89,14 +108,12 @@ module Protobuf
             break
           end
 
-          # Non-blocking push with timeout
-          begin
-            Timeout.timeout(1) do
-              @pending_queue << msg
-            end
+          # Push with a deadline (see push_with_deadline: no Timeout.timeout,
+          # which corrupts the SizedQueue mutex on JRuby).
+          if push_with_deadline(msg, 1)
             migrated_count += 1
             logger.warn "Migrated message #{migrated_count} from old queue to central queue"
-          rescue Timeout::Error
+          else
             logger.error "Failed to migrate message to central queue (queue full), dropping message"
             break
           end
@@ -117,15 +134,15 @@ module Protobuf
 
         # Wake every handler with its own poison pill.
         handlers.size.times do
-          begin
-            # Clear some space if the queue is full so the shutdown signal fits.
-            if @pending_queue.num_waiting.zero? && @pending_queue.size >= @pending_queue.max
-              logger.warn "Queue full during shutdown, clearing to make room for shutdown signal"
-              @pending_queue.clear rescue nil
-            end
+          # Clear some space if the queue is full so the shutdown signal fits.
+          if @pending_queue.num_waiting.zero? && @pending_queue.size >= @pending_queue.max
+            logger.warn "Queue full during shutdown, clearing to make room for shutdown signal"
+            @pending_queue.clear rescue nil
+          end
 
-            Timeout.timeout(1) { @pending_queue << :shutdown }
-          rescue Timeout::Error
+          # Push with a deadline (see push_with_deadline: no Timeout.timeout,
+          # which corrupts the SizedQueue mutex on JRuby).
+          unless push_with_deadline(:shutdown, 1)
             logger.error "Failed to send shutdown signal (queue blocked); will force-kill remaining handlers"
             break
           end
@@ -154,6 +171,11 @@ module Protobuf
         @pending_queue.size
       end
 
+      # Resident bytes in the shared intake queue = heap backpressure (gauge).
+      def pending_queue_bytes
+        @pending_queue.bytesize
+      end
+
       def unsubscribe_all
         # Take ownership and clear: pause/resume cycles re-subscribe from
         # scratch, so keeping the old entries only grew the array without bound
@@ -178,6 +200,32 @@ module Protobuf
         ::Protobuf::Nats.monotonic_time
       end
 
+      # Push onto the shared SizedQueue with a deadline, WITHOUT Timeout.timeout.
+      # Timeout uses an asynchronous Thread#raise, which is unsafe around the
+      # mutex SizedQueue#push takes internally: on JRuby (10.x in particular) a
+      # timeout firing mid-push unwinds through the held mutex and raises
+      # "ThreadError: Attempt to unlock a mutex which is locked by another
+      # thread/fiber" instead of Timeout::Error -- so the rescue :Timeout::Error
+      # never fires and shutdown/migration blow up. CRuby happens to unwind
+      # cleanly, which is why this only bit JRuby. Poll a non-blocking push
+      # against a monotonic deadline instead: no async raise, safe on every
+      # engine. Returns true if pushed, false if the deadline passed (queue
+      # still full) or the queue was closed.
+      def push_with_deadline(obj, timeout)
+        deadline = monotonic + timeout
+        loop do
+          begin
+            @pending_queue.push(obj, true) # non_block: raises ThreadError when full
+            return true
+          rescue ::ClosedQueueError
+            return false
+          rescue ::ThreadError
+            return false if monotonic >= deadline
+            sleep 0.01
+          end
+        end
+      end
+
       # Spawn one intake handler. Each thread owns its own crash_count so the
       # self-healing exponential backoff is correct under true parallelism (a
       # shared counter would lose updates across handlers on JRuby). The counter

data/lib/protobuf/nats/version.rb

@@ -1,5 +1,5 @@
 module Protobuf
   module Nats
-    VERSION = "0.13.1"
+    VERSION = "0.13.2.pre2"
   end
 end

data/lib/protobuf/nats.rb

@@ -235,14 +235,17 @@ module Protobuf
 
     # nats-pure increments a subscription's pending_size (bytes) for every
     # inbound message and only decrements it in its own consumption paths
-    # (next_msg / the sub's message thread). Both the client muxer and the
-    # server intake pop pending_queue directly and never run those paths, so
-    # pending_size grows monotonically and the byte-based slow-consumer limit
-    # would eventually trip on *cumulative* traffic -- silently dropping every
-    # later message on that subscription. Disable the byte limit; the
-    # message-count limit (pending_queue depth, tracked accurately for free)
-    # still bounds a genuinely slow consumer. Guarded so a non-standard/faked
-    # subscription is a no-op.
+    # (next_msg / the sub's message thread). The server intake pops
+    # pending_queue directly and never runs those paths, so pending_size grows
+    # monotonically and the byte-based slow-consumer limit would eventually trip
+    # on *cumulative* traffic -- silently dropping every later message on that
+    # subscription. Disable the byte limit; the message-count limit
+    # (pending_queue depth, tracked accurately for free) still bounds a genuinely
+    # slow consumer. Guarded so a non-standard/faked subscription is a no-op.
+    #
+    # NOTE: only the server uses this now. The client muxer instead decrements
+    # pending_size itself after each pop (ResponseMuxer#run_dispatch_loop), which
+    # keeps the counter accurate and lets it enforce a finite byte ceiling.
     def self.disable_subscription_byte_limit!(sub)
       sub.pending_bytes_limit = ::Float::INFINITY if sub.respond_to?(:pending_bytes_limit=)
     end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: protobuf-nats
 version: !ruby/object:Gem::Version
-  version: 0.13.1
+  version: 0.13.2.pre2
 platform: ruby
 authors:
 - Brandon Dewitt
@@ -183,9 +183,7 @@ extensions: []
 extra_rdoc_files: []
 files:
 - ".circleci/config.yml"
-- ".github/workflows/ci.yml"
 - ".gitignore"
-- ".travis.yml"
 - CHANGELOG.md
 - Gemfile
 - LICENSE.txt
@@ -215,6 +213,7 @@ files:
 - ext/jars/slf4j-api-1.7.25.jar
 - ext/jars/slf4j-simple-1.7.25.jar
 - lib/protobuf/nats.rb
+- lib/protobuf/nats/byte_bounded_queue.rb
 - lib/protobuf/nats/client.rb
 - lib/protobuf/nats/config.rb
 - lib/protobuf/nats/errors.rb

data/.github/workflows/ci.yml

@@ -1,64 +0,0 @@
-name: CI
-
-on:
-  push:
-    branches: [master]
-  pull_request:
-
-jobs:
-  test:
-    runs-on: ubuntu-latest
-    strategy:
-      fail-fast: false
-      matrix:
-        ruby:
-          - "3.1"
-          - "3.4"
-          - "jruby-9.4"
-          - "jruby-10.0"
-
-    # Real NATS server for the integration specs (spec/integration/); they
-    # auto-detect it on localhost:4222 (see spec/spec_helper.rb).
-    services:
-      nats:
-        image: nats:2.14-linux
-        ports:
-          - 4222:4222
-
-    env:
-      JRUBY_OPTS: "-J-Xmx1024m"
-      RAILS_ENV: test
-
-    steps:
-      - uses: actions/checkout@v4
-
-      - uses: ruby/setup-ruby@v1
-        with:
-          ruby-version: ${{ matrix.ruby }}
-          bundler-cache: true
-
-      # The cluster failover spec (spec/integration/failover_spec.rb) spawns
-      # its own two-node cluster, so it needs the nats-server binary itself --
-      # the service container above only covers the single-node specs.
-      - name: Install nats-server binary (cluster failover spec)
-        run: |
-          NATS_VERSION=v2.14.3
-          curl -sfL "https://github.com/nats-io/nats-server/releases/download/${NATS_VERSION}/nats-server-${NATS_VERSION}-linux-amd64.tar.gz" | tar xz
-          sudo mv "nats-server-${NATS_VERSION}-linux-amd64/nats-server" /usr/local/bin/
-          nats-server --version
-
-      - name: Wait for NATS
-        run: |
-          echo "Waiting for NATS to start..."
-          for i in $(seq 1 30); do
-            if nc -z localhost 4222; then
-              echo "NATS is ready!"
-              exit 0
-            fi
-            sleep 1
-          done
-          echo "NATS did not become ready" >&2
-          exit 1
-
-      - name: Run tests
-        run: bundle exec rspec

data/.travis.yml

@@ -1,16 +0,0 @@
-sudo: false
-language: ruby
-jdk:
- - openjdk8
-rvm:
-  - 2.3.0
-  - 2.7.0
-  - jruby-9.1.7.0
-  - jruby-9.2.13.0
-before_install:
-  # Install and start gnatsd
-  - ./scripts/install_gnatsd.sh
-  - $HOME/nats-server/nats-server &
-  # Install deps for project
-  - gem install bundler
-  - gem update --system