protobuf-nats 0.13.2.pre1 → 0.13.2.pre2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +12 -0
- data/README.md +9 -2
- data/lib/protobuf/nats/byte_bounded_queue.rb +72 -0
- data/lib/protobuf/nats/errors.rb +16 -0
- data/lib/protobuf/nats/response_muxer.rb +80 -2
- data/lib/protobuf/nats/server.rb +10 -0
- data/lib/protobuf/nats/super_subscription_manager.rb +26 -2
- data/lib/protobuf/nats/version.rb +1 -1
- data/lib/protobuf/nats.rb +11 -8
- metadata +2 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 192c0200a6f76e8efb9e165fa8fd4f9ac11240cf096edf866516b604d591d932
|
|
4
|
+
data.tar.gz: 97742cf64382a4f1286140c05da7dac353fa97c9f026310f6702fb5d2acbbf55
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 43143595525a958c16eccdf4eaed2ed4132f4d805924411a5fecbcd08c979b850a887fa4ddc52c5cce31261199dc26d63058f213767bb7180f631f8e1066d39f
|
|
7
|
+
data.tar.gz: c34ee341de5f07d4cc5bcbec49a860b9024b62b138e25af96fe962e84fa45886fc38f0cea5b78eed8c6b1e4c103f3c6bca91024a4d4847cb41767606cc31bf9a
|
data/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,17 @@
|
|
|
1
1
|
## Changelog
|
|
2
2
|
|
|
3
|
+
### 0.13.2
|
|
4
|
+
Bounds the RPC transport's in-memory buffering to prevent the JVM-heap OOM introduced by the JNats → nats-pure migration. Both the client response muxer and the server intake queue are now capped by message count **and** total bytes, dropping (with client retry) rather than buffering unbounded protobuf payloads on the heap.
|
|
5
|
+
|
|
6
|
+
#### Client: response-muxer heap bound
|
|
7
|
+
- The shared response "firehose" is bounded by both a message count (`PB_NATS_RESPONSE_MUXER_QUEUE_SIZE`, default `1024`) and a byte ceiling (`PB_NATS_RESPONSE_MUXER_QUEUE_BYTES`, default 64 MiB); nats-pure drops (`SlowConsumer`) on whichever trips first and the RPC retries. Previously only nats-pure's 65,536-message count applied with the byte limit disabled, so a burst of large responses could hold gigabytes of Ruby objects on the JVM heap.
|
|
8
|
+
- The muxer now decrements the subscription's `pending_size` after each pop, keeping a *finite* byte limit accurate — instead of disabling it as before. If a subscription can't support that accounting (no `#synchronize`), `start` raises `IncompatibleSubscription` (a tripwire for a breaking nats-pure change) rather than silently degrading.
|
|
9
|
+
- New gauges: `response_muxer.pending_queue_size` and `response_muxer.pending_queue_peak` (high-water mark between the ~60s samples).
|
|
10
|
+
|
|
11
|
+
#### Server: intake heap bound
|
|
12
|
+
- The shared intake queue is now bounded by bytes as well as count: new `PB_NATS_SERVER_INTAKE_QUEUE_BYTES` (default 128 MiB), enforced by a `ByteBoundedQueue` with a shared byte counter. A request that would exceed the ceiling is dropped (the client retries) and emits `server.intake_bytes_dropped`; new gauge `server.pending_intake_queue_bytes`. nats-pure's per-subscription byte limit stays disabled — the shared queue counter owns byte bounding, since many subscriptions funnel into one queue.
|
|
13
|
+
- Fixed a slow leak of orphaned `@overdue_flagged` entries caused by a handler-completion race; the periodic monitor now reaps them.
|
|
14
|
+
|
|
3
15
|
### 0.13.1
|
|
4
16
|
Fixes regressions from the JNats → nats-pure migration (0.13.0) plus a full reliability, performance, and security hardening pass. Highlights: the client reconnects and retries correctly through dropped connections, failing nodes, and terminal closes; the server survives overload and connection loss instead of going silently deaf; TLS actually verifies the server certificate.
|
|
5
17
|
|
data/README.md
CHANGED
|
@@ -80,6 +80,8 @@ default is used, instead of silently becoming `0`.
|
|
|
80
80
|
| `PB_NATS_CLIENT_RECONNECT_DELAY` | ACK timeout | Seconds to sleep before retrying after a transient transport error — see [Resilience](#resilience). |
|
|
81
81
|
| `PB_NATS_CLIENT_RECONNECT_DELAY_SPLAY_LIMIT` | `1000` | Random jitter (ms, `0..limit`) added to the reconnect delay so a fleet doesn't retry in lockstep. `0` disables. |
|
|
82
82
|
| `PB_NATS_RESPONSE_MUXER_DISPATCHERS` | CPUs on JRuby, `1` on CRuby | Threads draining the shared response subscription (min 1). |
|
|
83
|
+
| `PB_NATS_RESPONSE_MUXER_QUEUE_SIZE` | `1024` | Message-count cap for the shared response subscription. Dispatchers drain it to ~0, so this is burst headroom, not a working set: each in-flight request holds only ~2 messages (ACK + response). Beyond it nats-pure drops (`SlowConsumer`) and the RPC retries, rather than buffering unbounded response objects on the heap. Set it to your app's request-thread-pool size if that exceeds the default (min 1). |
|
|
84
|
+
| `PB_NATS_RESPONSE_MUXER_QUEUE_BYTES` | `67108864` (64 MiB) | Byte cap for the shared response subscription — the true heap ceiling. The count cap alone says nothing about size (1024 large payloads can still be gigabytes), so the firehose is bounded by whichever trips first: `PB_NATS_RESPONSE_MUXER_QUEUE_SIZE` messages or this many bytes. Matches the NATS ecosystem's per-subscription byte default (nats-pure / nats.go both use 64 MiB). Raise it if you have large payloads and heap to spare; lower it to tighten the ceiling (min 1). |
|
|
83
85
|
|
|
84
86
|
#### Server
|
|
85
87
|
|
|
@@ -87,7 +89,8 @@ default is used, instead of silently becoming `0`.
|
|
|
87
89
|
| --- | --- | --- |
|
|
88
90
|
| `PB_NATS_SERVER_MAX_QUEUE_SIZE` | thread count | Queue in front of the handler thread pool; requests beyond it are NACKed. |
|
|
89
91
|
| `PB_NATS_SERVER_SUBSCRIPTION_HANDLERS` | CPUs on JRuby, `1` on CRuby | Threads draining the shared intake queue and publishing ACK/NACKs (min 1). Consumer parallelism only — does not change queue-group delivery. |
|
|
90
|
-
| `PB_NATS_SERVER_INTAKE_QUEUE_SIZE` | `65536` |
|
|
92
|
+
| `PB_NATS_SERVER_INTAKE_QUEUE_SIZE` | `65536` | Message-count capacity of the shared intake queue. Smaller turns overload into prompt drops-and-retries instead of a deep stale backlog; tune down alongside `PB_NATS_SERVER_STALE_REQUEST_MS`. |
|
|
93
|
+
| `PB_NATS_SERVER_INTAKE_QUEUE_BYTES` | `134217728` (128 MiB) | Byte capacity of the shared intake queue — the aggregate-heap bound the count alone can't give (65,536 large requests is a lot of heap). A request that would exceed it is dropped (the client retries), never buffered. Bounds bytes across all subscriptions; higher than the client muxer's 64 MiB since the server's count cap is higher too (min 1). |
|
|
91
94
|
| `PB_NATS_SERVER_SUBSCRIPTIONS_PER_RPC_ENDPOINT` | `10` | Subscriptions created per endpoint (lets JVM servers warm up gradually). Queue groups still deliver each request to exactly one consumer. |
|
|
92
95
|
| `PB_NATS_SERVER_SLOW_START_DELAY` | `10` | Seconds between slow-start subscription rounds. |
|
|
93
96
|
| `PB_NATS_SERVER_PAUSE_FILE_PATH` | `nil` | While this file exists the server unsubscribes from all services; it resubscribes (with slow start) when the file is removed. |
|
|
@@ -166,7 +169,11 @@ NATS server certificate, or the connection will be rejected. Hostname (SAN/CN) v
|
|
|
166
169
|
threads, so a slow ACK publish can't head-of-line block other subjects. Handlers self-heal like the muxer.
|
|
167
170
|
- **Observability** — thread-pool gauges plus in-flight handler metrics (`server.inflight_count`,
|
|
168
171
|
`server.inflight_oldest_age_ms`, `server.overdue_handler_count`, `server.pending_intake_queue_size`,
|
|
169
|
-
`server.thread_pool_saturated`).
|
|
172
|
+
`server.pending_intake_queue_bytes`, `server.thread_pool_saturated`). A request dropped because it would exceed the
|
|
173
|
+
intake byte ceiling emits `server.intake_bytes_dropped` (see `PB_NATS_SERVER_INTAKE_QUEUE_BYTES`). The client muxer
|
|
174
|
+
gauges its response firehose (`response_muxer.pending_queue_size` and, so a burst between the ~60s samples isn't
|
|
175
|
+
missed, `response_muxer.pending_queue_peak`), plus `response_muxer.stale_tokens_cleaned`, `client.unexpected_message`,
|
|
176
|
+
and `client.invalid_message`. Error callbacks run on a bounded background executor; drops are counted
|
|
170
177
|
(`Protobuf::Nats.error_callback_drop_count`) and emit `error_callback_dropped`.
|
|
171
178
|
|
|
172
179
|
## Resilience
|
|
@@ -0,0 +1,72 @@
|
|
|
1
|
+
require "concurrent"
|
|
2
|
+
|
|
3
|
+
module Protobuf
|
|
4
|
+
module Nats
|
|
5
|
+
# A SizedQueue that additionally bounds the total *bytes* of its contents,
|
|
6
|
+
# not just the message count. The server funnels every subscription into one
|
|
7
|
+
# shared intake queue, so a per-subscription byte limit (nats-pure's
|
|
8
|
+
# pending_bytes_limit) can't bound the aggregate heap -- this shared counter
|
|
9
|
+
# can. Count is still bounded by the SizedQueue capacity it inherits.
|
|
10
|
+
#
|
|
11
|
+
# When a push would exceed the byte ceiling we DROP the message rather than
|
|
12
|
+
# block: pushes happen on nats-pure's read thread (Subscription#dispatch), and
|
|
13
|
+
# blocking it would stall PING/PONG and every other subject. A drop mirrors
|
|
14
|
+
# nats-pure's own SlowConsumer behaviour. Non-message items (the :shutdown
|
|
15
|
+
# poison pill) carry zero bytes, so they are never dropped by the byte gate.
|
|
16
|
+
#
|
|
17
|
+
# A drop invokes the optional +on_drop+ callback with the dropped byte count,
|
|
18
|
+
# so the caller owns any (context-specific) instrumentation rather than this
|
|
19
|
+
# generic queue class hard-coding it.
|
|
20
|
+
class ByteBoundedQueue < ::SizedQueue
|
|
21
|
+
def initialize(max_msgs, max_bytes, on_drop: nil)
|
|
22
|
+
super(max_msgs)
|
|
23
|
+
@max_bytes = max_bytes
|
|
24
|
+
@on_drop = on_drop
|
|
25
|
+
@bytes = ::Concurrent::AtomicFixnum.new(0)
|
|
26
|
+
end
|
|
27
|
+
|
|
28
|
+
# Enqueue unless it would exceed the byte ceiling. The check-then-add races
|
|
29
|
+
# only concurrent pops (which lower @bytes), so the ceiling can be exceeded
|
|
30
|
+
# by at most one in-flight message -- a soft limit, like nats-pure's own
|
|
31
|
+
# byte accounting. Returns self (SizedQueue#push contract). Raises
|
|
32
|
+
# ThreadError from super on a non_block push into a count-full queue, before
|
|
33
|
+
# any bytes are counted.
|
|
34
|
+
def push(obj, non_block = false)
|
|
35
|
+
bytes = byte_size(obj)
|
|
36
|
+
if bytes > 0 && (@bytes.value + bytes) > @max_bytes
|
|
37
|
+
@on_drop&.call(bytes)
|
|
38
|
+
return self
|
|
39
|
+
end
|
|
40
|
+
super(obj, non_block)
|
|
41
|
+
@bytes.increment(bytes)
|
|
42
|
+
self
|
|
43
|
+
end
|
|
44
|
+
alias_method :<<, :push
|
|
45
|
+
|
|
46
|
+
def pop(non_block = false)
|
|
47
|
+
obj = super
|
|
48
|
+
# nil == closed/empty non_block; nothing dequeued, nothing to subtract.
|
|
49
|
+
@bytes.update { |value| [value - byte_size(obj), 0].max } if obj
|
|
50
|
+
obj
|
|
51
|
+
end
|
|
52
|
+
|
|
53
|
+
def clear
|
|
54
|
+
super
|
|
55
|
+
@bytes.value = 0
|
|
56
|
+
end
|
|
57
|
+
|
|
58
|
+
# Current resident byte total (gauge for observability).
|
|
59
|
+
def bytesize
|
|
60
|
+
@bytes.value
|
|
61
|
+
end
|
|
62
|
+
|
|
63
|
+
private
|
|
64
|
+
|
|
65
|
+
# Bytes attributable to a queued item. NATS::Msg carries #data; the
|
|
66
|
+
# :shutdown poison pill (and any other non-message sentinel) counts as 0.
|
|
67
|
+
def byte_size(obj)
|
|
68
|
+
obj.respond_to?(:data) && obj.data ? obj.data.bytesize : 0
|
|
69
|
+
end
|
|
70
|
+
end
|
|
71
|
+
end
|
|
72
|
+
end
|
data/lib/protobuf/nats/errors.rb
CHANGED
|
@@ -13,6 +13,22 @@ module Protobuf
|
|
|
13
13
|
class ResponseMuxer < ClientError
|
|
14
14
|
end
|
|
15
15
|
|
|
16
|
+
# Raised by ResponseMuxer#start when the response subscription can't support
|
|
17
|
+
# the muxer's pending_size byte accounting (it doesn't respond to
|
|
18
|
+
# #synchronize). nats-pure's Subscription always includes MonitorMixin, so
|
|
19
|
+
# in practice this never fires against a real connection -- it's a tripwire
|
|
20
|
+
# for a significant change in nats-pure's internals (or a non-standard
|
|
21
|
+
# injected client). We take @resp_sub.synchronize on every pop to decrement
|
|
22
|
+
# pending_size and keep the finite byte cap accurate; without it that counter
|
|
23
|
+
# would only grow and eventually false-trip the limit, silently dropping
|
|
24
|
+
# every response. Failing loudly at start beats degrading silently at runtime.
|
|
25
|
+
#
|
|
26
|
+
# Deliberately NOT in RETRYABLE_TRANSPORT_ERRORS: retrying can't fix a
|
|
27
|
+
# structural mismatch. NOTE: intentionally undocumented in the README -- it's
|
|
28
|
+
# an internal invariant/tripwire, not a user-facing knob or metric.
|
|
29
|
+
class IncompatibleSubscription < ClientError
|
|
30
|
+
end
|
|
31
|
+
|
|
16
32
|
class MriIOException < ::StandardError
|
|
17
33
|
end
|
|
18
34
|
|
|
@@ -13,6 +13,20 @@ module Protobuf
|
|
|
13
13
|
MAX_RESPONSES_PER_TOKEN = 10
|
|
14
14
|
TOKEN_TTL_SECONDS = 600 # 10 minutes
|
|
15
15
|
|
|
16
|
+
# The shared response subscription is bounded by BOTH a message count and a
|
|
17
|
+
# byte ceiling; nats-pure drops (SlowConsumer) on whichever trips first, so
|
|
18
|
+
# the firehose is capped at min(count, bytes) instead of buffering unbounded
|
|
19
|
+
# protobuf payloads on the JVM heap (the 0.13.2 OOM). Dispatchers drain it to
|
|
20
|
+
# ~0, so these are burst headroom, not a working set.
|
|
21
|
+
#
|
|
22
|
+
# The count is deliberately tighter than the ecosystem's per-subscription
|
|
23
|
+
# defaults (nats-pure 65,536; nats.go 500,000): those bound off-heap buffers,
|
|
24
|
+
# this is Ruby objects on the heap, and the byte cap is the real ceiling. The
|
|
25
|
+
# byte default stays aligned at 64 MiB (nats-pure/nats.go both use it).
|
|
26
|
+
# Override via PB_NATS_RESPONSE_MUXER_QUEUE_SIZE / _QUEUE_BYTES.
|
|
27
|
+
DEFAULT_RESPONSE_QUEUE_SIZE = 1024
|
|
28
|
+
DEFAULT_RESPONSE_QUEUE_BYTES = 64 * 1024 * 1024 # 64MiB
|
|
29
|
+
|
|
16
30
|
# Sentinel pushed onto a token's queue to wake a waiter blocked in
|
|
17
31
|
# next_message. We cannot rely on Queue#close alone: on JRuby, close does
|
|
18
32
|
# NOT wake a pop() that is blocked with a timeout: -- neither the native
|
|
@@ -52,6 +66,12 @@ module Protobuf
|
|
|
52
66
|
# run_dispatch_loop), so a later transient crash restarts the backoff
|
|
53
67
|
# from 1s instead of staying pinned at the cap.
|
|
54
68
|
@crash_count = ::Concurrent::AtomicFixnum.new(0)
|
|
69
|
+
|
|
70
|
+
# High-water mark of the response queue depth since the last cleanup
|
|
71
|
+
# cycle. Sampled in the dispatch loop, emitted+reset by the cleanup thread
|
|
72
|
+
# (response_muxer.pending_queue_peak) so a burst between gauge samples is
|
|
73
|
+
# still visible.
|
|
74
|
+
@pending_queue_peak = ::Concurrent::AtomicFixnum.new(0)
|
|
55
75
|
end
|
|
56
76
|
|
|
57
77
|
def logger
|
|
@@ -75,6 +95,23 @@ module Protobuf
|
|
|
75
95
|
end
|
|
76
96
|
end
|
|
77
97
|
|
|
98
|
+
# Message-count and byte caps for the shared response subscription (see
|
|
99
|
+
# DEFAULT_RESPONSE_QUEUE_SIZE / _BYTES). Read once each, in #start, so no
|
|
100
|
+
# memoization is needed.
|
|
101
|
+
def response_queue_size
|
|
102
|
+
::Protobuf::Nats.env_int("PB_NATS_RESPONSE_MUXER_QUEUE_SIZE", DEFAULT_RESPONSE_QUEUE_SIZE, :min => 1)
|
|
103
|
+
end
|
|
104
|
+
|
|
105
|
+
def response_queue_bytes
|
|
106
|
+
::Protobuf::Nats.env_int("PB_NATS_RESPONSE_MUXER_QUEUE_BYTES", DEFAULT_RESPONSE_QUEUE_BYTES, :min => 1)
|
|
107
|
+
end
|
|
108
|
+
|
|
109
|
+
# Current depth of the shared firehose; 0 before the muxer starts. Gauge for
|
|
110
|
+
# observability -- mirrors SuperSubscriptionManager#pending_queue_size.
|
|
111
|
+
def pending_queue_size
|
|
112
|
+
@resp_sub&.pending_queue&.size || 0
|
|
113
|
+
end
|
|
114
|
+
|
|
78
115
|
def cleanup(token)
|
|
79
116
|
# Atomic remove-and-return; wake+close the queue to release any waiter.
|
|
80
117
|
entry = @resp_map.delete(token)
|
|
@@ -255,9 +292,24 @@ module Protobuf
|
|
|
255
292
|
begin
|
|
256
293
|
@resp_inbox_prefix = nats.new_inbox
|
|
257
294
|
|
|
258
|
-
# Subscribe to our per-instance inbox
|
|
295
|
+
# Subscribe to our per-instance inbox.
|
|
259
296
|
@resp_sub = nats.subscribe("#{@resp_inbox_prefix}.*")
|
|
260
|
-
|
|
297
|
+
|
|
298
|
+
# The dispatch loop takes @resp_sub.synchronize to decrement
|
|
299
|
+
# pending_size after each pop, which keeps the finite byte cap accurate.
|
|
300
|
+
# nats-pure's Subscription includes MonitorMixin, so this always holds;
|
|
301
|
+
# if it ever doesn't, nats-pure's internals changed in a way that would
|
|
302
|
+
# break byte accounting (a growing counter that false-trips the limit
|
|
303
|
+
# and drops every response). Fail loudly rather than degrade silently.
|
|
304
|
+
unless @resp_sub.respond_to?(:synchronize)
|
|
305
|
+
raise ::Protobuf::Nats::Errors::IncompatibleSubscription,
|
|
306
|
+
"NATS subscription does not respond to #synchronize; cannot maintain pending_size byte accounting (nats-pure internals changed?)"
|
|
307
|
+
end
|
|
308
|
+
|
|
309
|
+
# Bound the firehose by both message count and bytes (see
|
|
310
|
+
# DEFAULT_RESPONSE_QUEUE_SIZE / _BYTES).
|
|
311
|
+
@resp_sub.pending_msgs_limit = response_queue_size
|
|
312
|
+
@resp_sub.pending_bytes_limit = response_queue_bytes
|
|
261
313
|
@subscribed_nats.set(nats)
|
|
262
314
|
@started = true
|
|
263
315
|
rescue => e
|
|
@@ -328,6 +380,19 @@ module Protobuf
|
|
|
328
380
|
if stale_count > 0
|
|
329
381
|
::Protobuf::Nats.instrument "response_muxer.stale_tokens_cleaned", stale_count
|
|
330
382
|
end
|
|
383
|
+
|
|
384
|
+
# Gauge the shared response firehose so a climbing backlog is visible
|
|
385
|
+
# before it turns into timeouts/SlowConsumer drops. current == depth at
|
|
386
|
+
# sample time; peak == high-water since the last cycle (reset here).
|
|
387
|
+
::Protobuf::Nats.instrument "response_muxer.pending_queue_size", pending_queue_size
|
|
388
|
+
# Atomic read-and-reset of the high-water mark (AtomicFixnum has no
|
|
389
|
+
# get_and_set): capture the prior value inside the update block.
|
|
390
|
+
peak = 0
|
|
391
|
+
@pending_queue_peak.update do |current_value|
|
|
392
|
+
peak = current_value
|
|
393
|
+
0 # set to 0
|
|
394
|
+
end
|
|
395
|
+
::Protobuf::Nats.instrument "response_muxer.pending_queue_peak", peak
|
|
331
396
|
end
|
|
332
397
|
|
|
333
398
|
# Stop the cleanup thread
|
|
@@ -447,6 +512,19 @@ module Protobuf
|
|
|
447
512
|
next
|
|
448
513
|
end
|
|
449
514
|
|
|
515
|
+
# Drop the popped message's bytes from pending_size. nats-pure only
|
|
516
|
+
# decrements it in #process, which we bypass by popping pending_queue
|
|
517
|
+
# directly; without this the counter climbs monotonically and would
|
|
518
|
+
# false-trip the finite pending_bytes_limit, dropping every later
|
|
519
|
+
# response. Take the same monitor nats-pure's read thread uses.
|
|
520
|
+
# (#start guarantees the subscription responds to #synchronize.)
|
|
521
|
+
sub.synchronize { sub.pending_size -= msg.data.size }
|
|
522
|
+
|
|
523
|
+
# Sample post-pop depth into the high-water mark so a burst that fills
|
|
524
|
+
# and drains between the 60s gauge samples is still visible.
|
|
525
|
+
depth = sub.pending_queue.size
|
|
526
|
+
@pending_queue_peak.update { |current_value| [depth, current_value].max }
|
|
527
|
+
|
|
450
528
|
dispatch_message(msg)
|
|
451
529
|
|
|
452
530
|
# A processed message means this dispatcher is healthy: let the
|
data/lib/protobuf/nats/server.rb
CHANGED
|
@@ -182,9 +182,19 @@ module Protobuf
|
|
|
182
182
|
end
|
|
183
183
|
|
|
184
184
|
::Protobuf::Nats.instrument("server.pending_intake_queue_size", subscription_manager.pending_queue_size)
|
|
185
|
+
::Protobuf::Nats.instrument("server.pending_intake_queue_bytes", subscription_manager.pending_queue_bytes)
|
|
185
186
|
::Protobuf::Nats.instrument("server.inflight_count", count)
|
|
186
187
|
::Protobuf::Nats.instrument("server.inflight_oldest_age_ms", oldest_age_ms)
|
|
187
188
|
::Protobuf::Nats.instrument("server.overdue_handler_count", overdue)
|
|
189
|
+
|
|
190
|
+
# Reap orphaned overdue flags. The handler's ensure normally deletes
|
|
191
|
+
# @overdue_flagged[id], but the flag set above can race a completing
|
|
192
|
+
# handler: we read id from @inflight, the ensure deletes both maps, then
|
|
193
|
+
# we set @overdue_flagged[id] -- an entry nothing else will ever remove.
|
|
194
|
+
# A flag whose id is no longer in-flight is by definition orphaned.
|
|
195
|
+
@overdue_flagged.each_key do |id|
|
|
196
|
+
@overdue_flagged.delete(id) unless @inflight.key?(id)
|
|
197
|
+
end
|
|
188
198
|
end
|
|
189
199
|
|
|
190
200
|
# Defaults to #threads (not the raw option) so a server built with no
|
|
@@ -5,13 +5,19 @@ require "timeout"
|
|
|
5
5
|
require "protobuf/rpc/server"
|
|
6
6
|
require "protobuf/rpc/service"
|
|
7
7
|
require "protobuf/nats/thread_pool"
|
|
8
|
+
require "protobuf/nats/byte_bounded_queue"
|
|
8
9
|
|
|
9
10
|
module Protobuf
|
|
10
11
|
module Nats
|
|
11
12
|
class SuperSubscriptionManager
|
|
12
13
|
def initialize(nats, &cb)
|
|
13
|
-
# Central queue used by all subscriptions
|
|
14
|
-
|
|
14
|
+
# Central queue used by all subscriptions, bounded by both message count
|
|
15
|
+
# and total bytes (see intake_queue_size / intake_queue_bytes). A byte-cap
|
|
16
|
+
# drop is surfaced here as server.intake_bytes_dropped.
|
|
17
|
+
@pending_queue = ::Protobuf::Nats::ByteBoundedQueue.new(
|
|
18
|
+
intake_queue_size, intake_queue_bytes,
|
|
19
|
+
:on_drop => lambda { |bytes| ::Protobuf::Nats.instrument("server.intake_bytes_dropped", bytes) }
|
|
20
|
+
)
|
|
15
21
|
@subscriptions = []
|
|
16
22
|
@subscriptions_mutex = ::Mutex.new
|
|
17
23
|
@nats = nats
|
|
@@ -52,6 +58,19 @@ module Protobuf
|
|
|
52
58
|
@intake_queue_size ||= ::Protobuf::Nats.env_int("PB_NATS_SERVER_INTAKE_QUEUE_SIZE", ::NATS::IO::DEFAULT_SUB_PENDING_MSGS_LIMIT, :min => 1)
|
|
53
59
|
end
|
|
54
60
|
|
|
61
|
+
# Byte ceiling for the shared intake queue -- the aggregate-heap bound the
|
|
62
|
+
# message count alone can't give (65,536 large requests is a lot of heap).
|
|
63
|
+
# Bounds resident bytes across ALL subscriptions; the ByteBoundedQueue drops
|
|
64
|
+
# a message that would exceed it rather than block nats-pure's read thread.
|
|
65
|
+
# Default 128 MiB: higher than the client muxer's 64 MiB because the server
|
|
66
|
+
# fans requests across many handler threads and its count cap is higher too.
|
|
67
|
+
DEFAULT_INTAKE_QUEUE_BYTES = 128 * 1024 * 1024 # 128MiB
|
|
68
|
+
|
|
69
|
+
# Read once, in #initialize, so no memoization is needed.
|
|
70
|
+
def intake_queue_bytes
|
|
71
|
+
::Protobuf::Nats.env_int("PB_NATS_SERVER_INTAKE_QUEUE_BYTES", DEFAULT_INTAKE_QUEUE_BYTES, :min => 1)
|
|
72
|
+
end
|
|
73
|
+
|
|
55
74
|
def queue_subscribe(name)
|
|
56
75
|
logger.debug { "queue_subscribe(#{name})" }
|
|
57
76
|
sub = @nats.subscribe(name, :queue => name)
|
|
@@ -152,6 +171,11 @@ module Protobuf
|
|
|
152
171
|
@pending_queue.size
|
|
153
172
|
end
|
|
154
173
|
|
|
174
|
+
# Resident bytes in the shared intake queue = heap backpressure (gauge).
|
|
175
|
+
def pending_queue_bytes
|
|
176
|
+
@pending_queue.bytesize
|
|
177
|
+
end
|
|
178
|
+
|
|
155
179
|
def unsubscribe_all
|
|
156
180
|
# Take ownership and clear: pause/resume cycles re-subscribe from
|
|
157
181
|
# scratch, so keeping the old entries only grew the array without bound
|
data/lib/protobuf/nats.rb
CHANGED
|
@@ -235,14 +235,17 @@ module Protobuf
|
|
|
235
235
|
|
|
236
236
|
# nats-pure increments a subscription's pending_size (bytes) for every
|
|
237
237
|
# inbound message and only decrements it in its own consumption paths
|
|
238
|
-
# (next_msg / the sub's message thread).
|
|
239
|
-
#
|
|
240
|
-
#
|
|
241
|
-
#
|
|
242
|
-
#
|
|
243
|
-
#
|
|
244
|
-
#
|
|
245
|
-
#
|
|
238
|
+
# (next_msg / the sub's message thread). The server intake pops
|
|
239
|
+
# pending_queue directly and never runs those paths, so pending_size grows
|
|
240
|
+
# monotonically and the byte-based slow-consumer limit would eventually trip
|
|
241
|
+
# on *cumulative* traffic -- silently dropping every later message on that
|
|
242
|
+
# subscription. Disable the byte limit; the message-count limit
|
|
243
|
+
# (pending_queue depth, tracked accurately for free) still bounds a genuinely
|
|
244
|
+
# slow consumer. Guarded so a non-standard/faked subscription is a no-op.
|
|
245
|
+
#
|
|
246
|
+
# NOTE: only the server uses this now. The client muxer instead decrements
|
|
247
|
+
# pending_size itself after each pop (ResponseMuxer#run_dispatch_loop), which
|
|
248
|
+
# keeps the counter accurate and lets it enforce a finite byte ceiling.
|
|
246
249
|
def self.disable_subscription_byte_limit!(sub)
|
|
247
250
|
sub.pending_bytes_limit = ::Float::INFINITY if sub.respond_to?(:pending_bytes_limit=)
|
|
248
251
|
end
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: protobuf-nats
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.13.2.
|
|
4
|
+
version: 0.13.2.pre2
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Brandon Dewitt
|
|
@@ -213,6 +213,7 @@ files:
|
|
|
213
213
|
- ext/jars/slf4j-api-1.7.25.jar
|
|
214
214
|
- ext/jars/slf4j-simple-1.7.25.jar
|
|
215
215
|
- lib/protobuf/nats.rb
|
|
216
|
+
- lib/protobuf/nats/byte_bounded_queue.rb
|
|
216
217
|
- lib/protobuf/nats/client.rb
|
|
217
218
|
- lib/protobuf/nats/config.rb
|
|
218
219
|
- lib/protobuf/nats/errors.rb
|