protobuf-nats 0.13.1.pre3 → 0.13.1.pre4

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: b128192ffd9cfa77982eee74b65d2c1fc47d7718549cd96926f69e957ea46aa8
-  data.tar.gz: 40d7f971ee6686355c4f342ebd34066577ce863291538695f6cf1569b74b21be
+  metadata.gz: 58402417b6b024ac4c6f5f69827c02f9e130bd05d67cca6c74ed6f1c73086b36
+  data.tar.gz: a7b7d90363cd6ed871de6f28a6359cb1baa485a9926f8cca60fdda758a75e7f1
 SHA512:
-  metadata.gz: ab180b3ad24455f710daf101a8bc51413e77e6bf5e3c928132d88cd299e6b024b212e6d01187d4d6c90b53f8e38baa7bebe5ab920f52a91aeeb76663b496b76b
-  data.tar.gz: 71d4d11af63fb10d4fd061294f2ae64ed778582bae3ccc5762dda15f4644f843ca09361398508436c80d1c9f3b834d42af957b752387917f44675102ff45975e
+  metadata.gz: f53ccc62e55dd7a648183e5255559d7b2c5b87f4318218a3a882fa83bede0ac26d51a856b3efadd83e8fea165fbbf2b1ed8938912a9e4efa1bb5448a7e0c0ec2
+  data.tar.gz: ac82d95e2e9b047c3edfec4b516d10580b825cb4075068f704ffe3f9039e41aaee050658fa94125b054ad063f9a313490fc327d9250c9ca9c5cfdbb5a314ac01

data/.github/workflows/ci.yml

@@ -0,0 +1,64 @@
+name: CI
+
+on:
+  push:
+    branches: [master]
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        ruby:
+          - "3.1"
+          - "3.4"
+          - "jruby-9.4"
+          - "jruby-10.0"
+
+    # Real NATS server for the integration specs (spec/integration/); they
+    # auto-detect it on localhost:4222 (see spec/spec_helper.rb).
+    services:
+      nats:
+        image: nats:2.14-linux
+        ports:
+          - 4222:4222
+
+    env:
+      JRUBY_OPTS: "-J-Xmx1024m"
+      RAILS_ENV: test
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: ruby/setup-ruby@v1
+        with:
+          ruby-version: ${{ matrix.ruby }}
+          bundler-cache: true
+
+      # The cluster failover spec (spec/integration/failover_spec.rb) spawns
+      # its own two-node cluster, so it needs the nats-server binary itself --
+      # the service container above only covers the single-node specs.
+      - name: Install nats-server binary (cluster failover spec)
+        run: |
+          NATS_VERSION=v2.14.3
+          curl -sfL "https://github.com/nats-io/nats-server/releases/download/${NATS_VERSION}/nats-server-${NATS_VERSION}-linux-amd64.tar.gz" | tar xz
+          sudo mv "nats-server-${NATS_VERSION}-linux-amd64/nats-server" /usr/local/bin/
+          nats-server --version
+
+      - name: Wait for NATS
+        run: |
+          echo "Waiting for NATS to start..."
+          for i in $(seq 1 30); do
+            if nc -z localhost 4222; then
+              echo "NATS is ready!"
+              exit 0
+            fi
+            sleep 1
+          done
+          echo "NATS did not become ready" >&2
+          exit 1
+
+      - name: Run tests
+        run: bundle exec rspec

data/CHANGELOG.md

@@ -1,38 +1,48 @@
 ## Changelog
 
-### 0.13.1.pre2
-Additional edge-case fixes found while reviewing the 0.13.1 changes:
+### 0.13.1
+Fixes regressions from the JNats → nats-pure migration (0.13.0) plus a full reliability, performance, and security hardening pass. Highlights: the client reconnects and retries correctly through dropped connections, failing nodes, and terminal closes; the server survives overload and connection loss instead of going silently deaf; TLS actually verifies the server certificate.
 
-- **ResponseMuxer self-heal could drop to zero dispatchers.** When the sole dispatcher crashed fatally, its self-healing `start` counted the still-alive (but exiting) crashing thread, so it spawned no replacement — leaving zero dispatchers on CRuby (`dispatcher_count == 1`) and the muxer silently delivering no responses. The crashing thread now removes itself from the handler pool before re-topping it up.
-- **Client connection is rebuilt after a terminal close.** `@client_nats_connection` was memoized once and never reset, so once nats-pure gave up and fired `on_close` every later request reused a dead client forever. `on_close` now drops the cached connection so the next request rebuilds.
-- **Dropped error callbacks are now observable.** The bounded `notify_error_callbacks_async` executor silently discarded callbacks when saturated (exactly during an error flood). Drops now bump `Protobuf::Nats.error_callback_drop_count` and emit `error_callback_dropped`, without formatting/logging on the read thread.
-- **Server no longer double-publishes on a response-publish failure.** A transport error while publishing a *successful* response fell into the handler rescue and emitted a second (error) response for the same request. The handler and the success-response publish are now in separate rescue scopes. The handler-failure error response also now sends a generic message ("Internal server error") instead of the raw `error.message`, so internal handler details aren't leaked to clients (the real error is still logged server-side).
-- **Opt-in reclaim of overdue handlers.** Handlers are still never aborted by default. `PB_NATS_SERVER_RECLAIM_OVERDUE_HANDLERS=true` lets operators reclaim a pool slot held by an orphaned handler (one that outlived the client's `response_timeout`) by raising `Errors::HandlerOverdue` into it; emits `server.handler_reclaimed`.
-- **TLS now verifies the NATS server certificate.** Previously, supplying a prepared `:tls` context made nats-pure skip its own `set_params`, leaving the OpenSSL default `VERIFY_NONE` in force — any certificate was accepted (MITM exposure) — and `tls_ca_cert` was configured but read nowhere. `Config#new_tls_context` now sets `verify_mode = VERIFY_PEER` and trusts the configured `tls_ca_cert` (falling back to the system trust store when none is set). **Breaking for misconfigured deployments:** a server whose certificate does not chain to the trusted CA, which previously connected insecurely, will now be rejected. (Hostname/SAN verification is still not enabled — see known gaps.)
+#### Client: reconnect & retry
+- Restored dropped-connection retries (the retry rescue matched an error nothing raised). The client now retries the transport errors nats-pure actually raises: `EOFError`, `IOError`, `Errno::ECONNRESET`/`EPIPE`/`ECONNREFUSED`/`ECONNABORTED`/`ETIMEDOUT`/`EHOSTUNREACH`/`ENETUNREACH`, `NATS::IO::ConnectionClosedError`, and Java `IOException` on JRuby.
+- A terminally closed connection self-heals: `on_close` drops the cached connection, the next request (or in-flight retry) rebuilds it, and the response muxer detects the swap and re-subscribes on the live connection. Previously every RPC timed out until the process restarted.
+- A muxer restart wakes in-flight waiters immediately instead of leaving them to burn the full timeout on responses that can never arrive.
+- Retries are bounded and jittered (`PB_NATS_CLIENT_MAX_RETRIES`, `PB_NATS_CLIENT_RECONNECT_DELAY_SPLAY_LIMIT`), and the final failed attempt raises immediately instead of sleeping first.
+- The muxer token TTL stretches with a response timeout configured beyond 600s, so long waits aren't cleaned up mid-request.
 
-#### Known gaps noted (not changed here)
-- **TLS hostname (SAN/CN) is not verified.** Chain verification is now on, but nats-pure only sets the SSLSocket hostname for a context it builds itself, and a single static hostname would be wrong for a multi-server cluster that reconnects across hosts. Plumbing per-connection hostname verification is tracked separately.
-- **`TLS1_3_VERSION` is assumed defined.** Fine on the JRuby targets; an old MRI/OpenSSL build without the constant would raise `NameError`.
+#### Server: reliability
+- Subscriptions no longer go permanently deaf under cumulative traffic: the byte-based slow-consumer limit (which nats-pure never decrements on our consumption path) is disabled on both client and server subscriptions; the accurate message-count limit still applies.
+- Tuning `PB_NATS_SERVER_INTAKE_QUEUE_SIZE` down is safe: the slow-consumer limit is kept aligned with the queue capacity, so overload drops promptly instead of blocking nats-pure's read thread (which froze PING/PONG and every subject).
+- A terminally closed connection stops the server (logs + `server.connection_closed`) so a supervisor restarts it, instead of idling forever subscribed to nothing.
+- Failed handlers publish an RPC error response so the client fails fast instead of hanging until its response timeout; a failed success-publish no longer emits a duplicate error response; client-facing error messages are generic (details stay in server logs).
+- Pause/resume no longer leaks subscriptions; the thread-pool counter no longer goes negative at shutdown; dispatch/intake threads park instead of busy-spinning on a closed queue; self-healing always respawns a replacement dispatcher, with thread-safe backoff that decays when healthy.
+- Opt-in stale-request shedding (`PB_NATS_SERVER_STALE_REQUEST_MS`) and opt-in overdue-handler reclaim (`PB_NATS_SERVER_RECLAIM_OVERDUE_HANDLERS`). Handlers are still never aborted by default, and shutdown drains in-flight handlers before closing.
+- Lifecycle callbacks (client and server) register before `connect`, so handshake-window events are observed; a failed handshake closes the half-open client instead of leaking its reader/flusher threads.
 
-### 0.13.1
-Fixes a production regression and a set of related issues, all of the same class: assumptions left over from the JNats → nats-pure migration in 0.13.0 that became silently wrong.
-
-- **Dropped-connection retries were silently disabled.** Dropping JNats collapsed `Errors::IOException` to the never-raised `MriIOException`, so the client's reconnect/retry `rescue` became dead code. A dropped NATS connection then escaped immediately as an `RPC_ERROR` (surfacing as a 500) instead of being retried. The client now rescues the transport errors `nats-pure` and the socket layer actually raise (`EOFError`, `IOError`, `Errno::ECONNRESET`/`EPIPE`/`ECONNREFUSED`/`ETIMEDOUT`, `NATS::IO::ConnectionClosedError`, and `java.io.IOException` on JRuby) via `Errors::RETRYABLE_TRANSPORT_ERRORS` and rides them out with the existing `reconnect_delay` retry loop.
-- **Response muxer `pending_size` drift (could silently drop all responses).** nats-pure increments a subscription's `pending_size` (synchronized) for every inbound message and uses it to enforce the slow-consumer byte limit; for a callback-less subscription it never decrements it, so the muxer would be the sole consumer. Rather than mirror that accounting with a lock on every message, the muxer now **disables the byte-based limit** on its response subscription and relies on the message-count limit (the `SizedQueue` depth, tracked accurately for free). This removes the per-message lock from the dispatch hot path (~**2.7× faster** per message on JRuby — see `bench/muxer_resilience_bench.rb`) and eliminates the drift bug entirely.
-- **Dispatcher no longer busy-spins during a restart window.** If `@resp_sub` was briefly `nil` while the muxer restarted, the dispatch loop raised `NoMethodError` every iteration — busy-spinning and emitting a logged error + error-callback per spin. It now parks briefly (~0.2% of the old wasted work, zero errors).
-- **Self-healing backoff counter is now thread-safe.** The shared dispatcher crash counter was a plain `Integer` mutated by multiple dispatcher threads (it lost ~45% of updates under true parallelism on JRuby, corrupting the exponential backoff). It is now a `Concurrent::AtomicFixnum` that decays once a dispatcher is healthy.
-- **Client connection lifecycle hardening.** Connection callbacks (`on_disconnect`/`on_reconnect`/`on_close`/`on_error`) are now registered before `connect`, so handshake-time events are observed; and a failed handshake closes the half-open client so nats-pure's reader/flusher threads aren't leaked.
-- **Removed the dead `:disable_reconnect_buffer` connect option.** nats-pure has no such option (it was a JNats concept), so it was silently ignored. Transient disconnects are now handled by the client's transport-error retry path and `ack_timeout`.
-- **Server no longer leaves clients hanging on handler/publish failure.** If processing a request fails after the ACK was sent, the server now publishes an encoded `RPC_ERROR` response so the client fails fast instead of blocking until `response_timeout` (60s).
-- **Config no longer crashes when the YAML file has no section for the current environment** (or is empty); it falls back to defaults.
-- **TLS now floors at 1.2 and ceilings at 1.3** (replacing the deprecated `ssl_version = :TLSv1_2` hard pin), so TLS 1.3 is used when the server supports it and a TLS-1.2-only transport still negotiates down to 1.2. Verified on JRuby 9.4 and 10.0.
-- **Server request intake is now parallelized.** `SuperSubscriptionManager` drained the shared intake queue with a single thread that also published every ACK/NACK, so on JRuby intake was pinned to one core and one slow publish (e.g. nats-pure's buffer during a reconnect) head-of-line blocked *every* subject. Intake now fans out to `PB_NATS_SERVER_SUBSCRIPTION_HANDLERS` threads (default `processor_count` on JRuby, 1 on CRuby) with per-thread self-healing backoff. NATS queue-group semantics and subscription counts are unchanged — each request is still delivered to exactly one consumer. Measured **~8.5× intake throughput** and head-of-line stall **~505ms → ~0.4ms** at 8 handlers (`bench/server_intake_bench.rb`).
-- **Client retry is bounded and jittered.** `PB_NATS_CLIENT_MAX_RETRIES` (default 3) and `PB_NATS_CLIENT_RECONNECT_DELAY_SPLAY_LIMIT` (default 1000ms) make retries configurable, and the reconnect sleep now adds random jitter so a fleet hitting the same outage doesn't reconnect in lockstep.
-- **More transient errors are retried.** `ConnectionPool::TimeoutError` (subscription-pool exhaustion during a reconnect) is now treated as transient instead of surfacing as an `RPC_ERROR`.
-- **`connection_options` only forwards nats-pure-recognized keys** (servers, max_reconnect_attempts, connect_timeout, tls); app-level settings are read via their own accessors and no longer leak into `nats.connect`. YAML config now uses `safe_load`.
-- **Thread-pool robustness.** `wait_for_termination` prunes under its mutex and returns a real drained/timed-out result; a new `replenish` (called each server tick) respawns a worker killed by a non-StandardError. On shutdown the drain timeout tracks `handler_overdue_ms` so a legitimate long handler isn't killed mid-flight, and abandoned in-flight handlers are logged/instrumented.
-- **Error callbacks run off the read loop.** The nats `on_error` hooks dispatch via a bounded executor (`notify_error_callbacks_async`) so a slow user callback can't stall message processing for every subject.
-- **Server handler observability (long operations are first-class).** Handlers are never aborted — long-running operations (up to and beyond a minute) are allowed. The server now tracks in-flight handlers and emits `server.inflight_count`, `server.inflight_oldest_age_ms`, `server.overdue_handler_count`, `server.handler_overdue`, `server.pending_intake_queue_size`, `server.slow_handler` (opt-in via `PB_NATS_SERVER_SLOW_HANDLER_THRESHOLD_MS`), and `server.thread_pool_saturated`. A handler is only flagged "overdue" once it outlives the client's `response_timeout` (`PB_NATS_SERVER_HANDLER_OVERDUE_MS`, default 65s), so normal long ops are not mislabeled. Server duration metrics now use a monotonic clock.
+#### Performance
+- Server intake fans out across `PB_NATS_SERVER_SUBSCRIPTION_HANDLERS` threads (default `processor_count` on JRuby, 1 on CRuby): ~8.5× intake throughput, head-of-line stalls ~505ms → ~0.4ms (`bench/server_intake_bench.rb`).
+- Muxer dispatch dropped its per-message lock (~2.7× faster on JRuby, `bench/muxer_resilience_bench.rb`) and extracts reply tokens without `split` allocations.
+- `ThreadPool#push` no longer supervises the worker pool per request (the server's 1s `replenish` tick is the sole respawn path), and `ResponseMuxer#start`'s once-per-RPC check is a lock-free atomic read.
+- User error callbacks run on a bounded executor off nats-pure's read thread; drops are counted (`error_callback_drop_count`) and instrumented.
+
+#### Failover & configuration
+- New yaml keys `reconnect_time_wait`, `ping_interval`, and `max_outstanding_pings` are forwarded to nats-pure for faster dead-node detection (defaults unchanged); `max_reconnect_attempts: -1` reconnects forever.
+- Numeric env vars parse strictly: malformed values (`"5s"`, `"fast,slow"`) log and fall back to defaults instead of silently becoming `0`; `PB_NATS_SERVER_MAX_QUEUE_SIZE` defaults to the resolved thread count.
+- `connection_options` forwards only nats-pure-recognized keys (the dead JNats-era `:disable_reconnect_buffer` option is gone), and connections are named (`PB_NATS_CONNECTION_NAME` > yaml `connection_name` > hostname) for NATS monitoring.
+- A yaml config that is empty or has no section for the current environment falls back to defaults instead of crashing at boot.
+- New in-flight handler observability: `server.inflight_count`, `server.inflight_oldest_age_ms`, `server.overdue_handler_count`, `server.pending_intake_queue_size`, `server.slow_handler` (opt-in), `server.thread_pool_saturated`; server durations use a monotonic clock.
+
+#### Security
+- TLS now verifies the NATS server certificate chain (`VERIFY_PEER`, trusting `tls_ca_cert` or the system store). **Breaking for misconfigured deployments** whose certificates don't chain to the trusted CA — they previously connected unverified.
+- TLS negotiates 1.2–1.3 (replacing the deprecated 1.2 hard pin); OpenSSL builds without TLS 1.3 degrade to a 1.2 ceiling instead of raising.
+- YAML config uses `safe_load` (aliases allowed, arbitrary object deserialization rejected). TLS client keys may be any key type (`OpenSSL::PKey.read`).
+- Known gap: TLS hostname (SAN/CN) verification remains off — it needs per-connection plumbing in nats-pure; tracked separately.
+
+#### Testing, CI, dependencies
+- Real-NATS integration specs (auto-detected on `localhost:4222`): full RPC round trip, concurrency with real NACK backpressure, terminal-close self-heal, and a two-node cluster failover spec that spawns its own cluster and kills the node the client is connected to (gated on the `nats-server` binary). GitHub Actions runs the suite on CRuby 3.1/3.4 and JRuby 9.4/10.0.
+- nats-pure pinned to `>= 2.5, < 3`: the gem relies on nats-pure internals (pending-queue swap, slow-consumer semantics, subscription replay, infinite-reconnect flag) verified against 2.5.
+- Removed the unused `connection_pool` dependency and the dead client subscription-pool code; `require "timeout"` is explicit where used.
+- Soak-tested with chaos runs (nats-server killed twice mid-run): 99.4% success on CRuby 3.4, 100% on JRuby 10.0.
 
 ### 0.13.0
 This is a large overhaul of the client and server internals.
@@ -55,4 +65,3 @@ This is a large overhaul of the client and server internals.
 - Bumped `activesupport` to `>= 6.1` (from `>= 3.2`).
 - Added `concurrent-ruby` (`~> 1.3.6`, pinned so `logger` is included) and `uuid7` runtime dependencies.
 - Pinned `i18n` to `< 1.15.0` in the Gemfile (workaround for ruby-i18n/i18n#735).
-

data/README.md

@@ -16,122 +16,9 @@ Add this line to your application's Gemfile:
 gem 'protobuf-nats'
 ```
 
-And then execute:
-
-    $ bundle
-
-Or install it yourself as:
-
-    $ gem install protobuf-nats
-
-## Configuring
-
-### Environment Variables
-
-You can also use the following environment variables to tune parameters:
-
-`PB_NATS_SERVER_MAX_QUEUE_SIZE` - The size of the queue in front of your thread pool (default: thread count passed to CLI).
-
-`PB_NATS_SERVER_PAUSE_FILE_PATH` - If this file exists, the server will pause by unsubscribing all services. When the
-file is removed it will resubscribe and restart slow start (default: `nil`).
-
-`PB_NATS_SERVER_SLOW_START_DELAY` - Seconds to wait before adding another round of subscriptions (default 10).
-
-`PB_NATS_SERVER_SUBSCRIPTIONS_PER_RPC_ENDPOINT` - Number of subscriptions to create for each rpc endpoint. This number is
-used to allow JVM based servers to warm-up slowly to prevent jolts in runtime performance across your RPC network
-(default: 10). Each subscription joins the NATS queue group for its endpoint, so every request is still delivered to
-exactly one consumer — this knob controls subscription/interest count, not duplicate delivery.
-
-`PB_NATS_SERVER_SUBSCRIPTION_HANDLERS` - Number of threads that drain the shared intake queue and publish ACK/NACKs
-(see [How it works](#how-it-works)). Defaults to `Concurrent.processor_count` on JRuby and `1` on CRuby. This is the
-*consumer* parallelism for messages this server has already received; it does not change how many topics are subscribed
-to or the queue-group delivery semantics. Minimum of 1.
-
-`PB_NATS_SERVER_SLOW_HANDLER_THRESHOLD_MS` - If set (> 0), emit `server.slow_handler` when a handler runs longer than this
-many milliseconds. Informational/SLA only — handlers are never aborted (default: 0, off).
-
-`PB_NATS_SERVER_HANDLER_OVERDUE_MS` - A handler still running past this many milliseconds is reported as "overdue"
-(`server.handler_overdue` + `server.overdue_handler_count`) — i.e. the client has already given up (`response_timeout`)
-so the work is orphaned. Defaults above the client's 60s response timeout so legitimate long operations are not flagged
-(default: 65000). **This should track your clients' `PB_NATS_CLIENT_RESPONSE_TIMEOUT`** — set it to roughly that value (plus
-a small grace). If clients use a longer response timeout, raise this so handlers aren't flagged overdue while a client is
-still waiting; if shorter, lower it so orphaned work is surfaced promptly.
-
-`PB_NATS_SERVER_RECLAIM_OVERDUE_HANDLERS` - When `"true"`, actively reclaim the thread-pool slot held by an overdue
-handler (one past `PB_NATS_SERVER_HANDLER_OVERDUE_MS`, whose client has already given up) by raising
-`Errors::HandlerOverdue` into the worker; emits `server.handler_reclaimed`. **Off by default** — the documented contract
-is that handlers are never aborted, since killing a thread mid-handler can corrupt state. Enable only when orphaned work
-is saturating the pool and the server is NACKing healthy traffic (default: false).
-
-`PB_NATS_CLIENT_ACK_TIMEOUT` - Seconds to wait for an ACK from the rpc server (default: 5 seconds).
-
-`PB_NATS_CLIENT_NACK_BACKOFF_INTERVALS` - Array of milliseconds to wait between NACK retries (default: "0,1,3,5,10").
-
-`PB_NATS_CLIENT_NACK_BACKOFF_SPLAY_LIMIT` - Milliseconds to add to the NACK backoff timeout to avoid bursting retries
-(default: 10 milliseconds).
-
-`PB_NATS_CLIENT_RESPONSE_TIMEOUT` - Seconds to wait for a non-ACK response from the rpc server (default: 60 seconds).
-
-`PB_NATS_CLIENT_RECONNECT_DELAY` - When a request hits a transient transport error (e.g. the NATS connection drops or is reset), the client sleeps this many seconds before retrying to give the connection time to re-establish (default: the ACK timeout). See [Resilience](#resilience).
-
-`PB_NATS_CLIENT_RECONNECT_DELAY_SPLAY_LIMIT` - Random jitter (milliseconds, `0..limit`) added to the reconnect delay so a fleet hitting the same NATS outage does not reconnect in lockstep (default: 1000). Set to 0 to disable jitter.
-
-`PB_NATS_CLIENT_MAX_RETRIES` - Number of attempts for ack-timeouts and transient transport errors before raising (default: 3).
-
-`PB_NATS_CLIENT_SUBSCRIPTION_POOL_SIZE` - If subscription pooling is desired for the request/response cycle then the pool size maximum should be set; the pool is lazy and therefore will only start new subscriptions as necessary (default: 0)
-
-`PB_NATS_RESPONSE_MUXER_DISPATCHERS` - Number of dispatcher threads draining the shared response subscription (see [ResponseMuxer](#how-it-works)). Defaults to `Concurrent.processor_count` on JRuby (true parallelism) and `1` on CRuby (the GVL makes extra dispatchers pointless). Minimum of 1.
-
-`PB_NATS_CONNECTION_NAME` - A friendly name for the NATS connection, surfaced in NATS server monitoring, error
-reporting, and debugging. Shared by both the client and server connections. Precedence: this env var > the
-`connection_name` yaml key > the hostname (`Socket.gethostname`), so the connection is never nameless (default: hostname).
-
-`PROTOBUF_NATS_CONFIG_PATH` - Custom path to the config yaml (default: "config/protobuf_nats.yml").
-
-### YAML Config
-
-The client and server are configured via environment variables defined in the `nats-pure` gem. However, there are a
-few params which cannot be set: `servers`, `uses_tls`, `subscription_key_replacements`, and `connect_timeout`, so those must be defined in a yml file.
-
-The library will automatically look for a file with a relative path of `config/protobuf_nats.yml`, but you may override
-this by specifying a different file via the `PROTOBUF_NATS_CONFIG_PATH` env variable.
-
-The `subscription_key_replacements` feature is something we have found useful for local testing, but it is subject to breaking changes.
-
-An example config looks like this:
-```
-# Stored at config/protobuf_nats.yml
----
-  production:
-    servers:
-      - "nats://127.0.0.1:4222"
-      - "nats://127.0.0.1:4223"
-      - "nats://127.0.0.1:4224"
-    max_reconnect_attempts: 500
-    connection_name: "my-service"
-    uses_tls: true
-    tls_client_cert: "/path/to/client-cert.pem"
-    tls_client_key: "/path/to/client-key.pem"
-    tls_ca_cert: "/path/to/ca.pem"
-    connect_timeout: 2
-    server_subscription_key_only_subscribe_to_when_includes_any_of:
-      - "search"
-      - "create"
-    server_subscription_key_do_not_subscribe_to_when_includes_any_of:
-      - "old_search"
-      - "old_create"
-    subscription_key_replacements:
-      - "original_service": "replacement_service"
-```
-
-When `uses_tls` is set, the client negotiates TLS with a floor of 1.2 and a ceiling of 1.3: it uses TLS 1.3 where the NATS server supports it and falls back to 1.2 otherwise (verified on JRuby 9.4 and 10.0).
-
-The client **verifies the NATS server's certificate chain** (`verify_mode = VERIFY_PEER`). When `tls_ca_cert` is set, only certificates chaining to that CA are trusted (the private-CA case); otherwise the system trust store is used. **Note:** a server whose certificate does not chain to the configured CA will be rejected — if you are upgrading from a release that did not verify (`< 0.13.1`), make sure `tls_ca_cert` points at the CA that signed your NATS server certificate. Hostname (SAN/CN) verification is not yet enabled (chain verification still ensures the certificate is signed by your trusted CA).
-
 ## Usage
 
-This library is designed to be an alternative transport implementation used by the `protobuf` gem. In order to make
-`protobuf` use this library, you need to set the following env variable:
+This library is an alternative transport for the `protobuf` gem. Point `protobuf` at it with:
 
 ```
 PB_SERVER_TYPE="protobuf/nats/runner"
@@ -140,11 +27,10 @@ PB_CLIENT_TYPE="protobuf/nats/client"
 
 ## Example
 
-NOTE: For a more detailed example, look at the `warehouse` app in the `examples` directory of this project.
-
-Here's a tl;dr example. You might have a protobuf definition and implementation like this:
+For a more detailed example, see the `warehouse` app in the `examples` directory.
 
 ```ruby
+# app.rb
 require "protobuf/nats"
 
 class User < ::Protobuf::Message
@@ -161,103 +47,165 @@ class UserService < ::Protobuf::Rpc::Service
 end
 ```
 
-Let's assume we saved this in a file called `app.rb`
-
-We can now start an rpc server using the protobuf-nats runner and client:
+Start a server, then call it from a client:
 
 ```
 $ export PB_SERVER_TYPE="protobuf/nats/runner"
 $ export PB_CLIENT_TYPE="protobuf/nats/client"
 $ bundle exec rpc_server start ./app.rb
-...
-I, [2017-03-24T12:16:02.539930 #12512]  INFO -- : Creating subscriptions:
-I, [2017-03-24T12:16:02.543927 #12512]  INFO -- :   - rpc.user_service.create
-...
-```
 
-And we can start a client and begin communicating:
-
-```
-$ export PB_SERVER_TYPE="protobuf/nats/runner"
-$ export PB_CLIENT_TYPE="protobuf/nats/client"
 $ bundle exec irb -r ./app
-irb(main):001:0> UserService.client.create(User.new(:username => "testing 123"))
+irb> UserService.client.create(User.new(:username => "testing 123"))
 => #<User id=123 username="testing 123">
 ```
 
-And we can see the message was sent to the server and the server replied with a user which now has an `id`.
+An rpc without a matching instance method (e.g. an unimplemented `search`) is simply not subscribed to.
 
-If we were to add another service endpoint called `search` to the `UserService` but fail to define an instance method
-`search`, then `protobuf-nats` will not subscribe to that route.
+## Configuring
+
+### Environment variables
+
+Numeric variables are parsed strictly: a malformed value (e.g. `PB_NATS_CLIENT_ACK_TIMEOUT=5s`) is logged and the
+default is used, instead of silently becoming `0`.
+
+#### Client
+
+| Variable | Default | Description |
+| --- | --- | --- |
+| `PB_NATS_CLIENT_ACK_TIMEOUT` | `5` | Seconds to wait for the server's ACK. |
+| `PB_NATS_CLIENT_RESPONSE_TIMEOUT` | `60` | Seconds to wait for the RPC response. |
+| `PB_NATS_CLIENT_MAX_RETRIES` | `3` | Attempts for ACK timeouts and transient transport errors. Retries re-send the request — see [Delivery semantics](#delivery-semantics-at-least-once). |
+| `PB_NATS_CLIENT_NACK_BACKOFF_INTERVALS` | `0,1,3,5,10` | Milliseconds to wait between NACK retries (one attempt per interval). |
+| `PB_NATS_CLIENT_NACK_BACKOFF_SPLAY_LIMIT` | `10` | Random jitter (ms) added to each NACK backoff. |
+| `PB_NATS_CLIENT_RECONNECT_DELAY` | ACK timeout | Seconds to sleep before retrying after a transient transport error — see [Resilience](#resilience). |
+| `PB_NATS_CLIENT_RECONNECT_DELAY_SPLAY_LIMIT` | `1000` | Random jitter (ms, `0..limit`) added to the reconnect delay so a fleet doesn't retry in lockstep. `0` disables. |
+| `PB_NATS_RESPONSE_MUXER_DISPATCHERS` | CPUs on JRuby, `1` on CRuby | Threads draining the shared response subscription (min 1). |
+
+#### Server
+
+| Variable | Default | Description |
+| --- | --- | --- |
+| `PB_NATS_SERVER_MAX_QUEUE_SIZE` | thread count | Queue in front of the handler thread pool; requests beyond it are NACKed. |
+| `PB_NATS_SERVER_SUBSCRIPTION_HANDLERS` | CPUs on JRuby, `1` on CRuby | Threads draining the shared intake queue and publishing ACK/NACKs (min 1). Consumer parallelism only — does not change queue-group delivery. |
+| `PB_NATS_SERVER_INTAKE_QUEUE_SIZE` | `65536` | Capacity of the shared intake queue. Smaller turns overload into prompt drops-and-retries instead of a deep stale backlog; tune down alongside `PB_NATS_SERVER_STALE_REQUEST_MS`. |
+| `PB_NATS_SERVER_SUBSCRIPTIONS_PER_RPC_ENDPOINT` | `10` | Subscriptions created per endpoint (lets JVM servers warm up gradually). Queue groups still deliver each request to exactly one consumer. |
+| `PB_NATS_SERVER_SLOW_START_DELAY` | `10` | Seconds between slow-start subscription rounds. |
+| `PB_NATS_SERVER_PAUSE_FILE_PATH` | `nil` | While this file exists the server unsubscribes from all services; it resubscribes (with slow start) when the file is removed. |
+| `PB_NATS_SERVER_SLOW_HANDLER_THRESHOLD_MS` | `0` (off) | Emit `server.slow_handler` when a handler runs longer than this. Informational only. |
+| `PB_NATS_SERVER_HANDLER_OVERDUE_MS` | `65000` | Age at which a still-running handler is reported overdue (its client has already given up). Keep ≈ your clients' response timeout plus a small grace. |
+| `PB_NATS_SERVER_RECLAIM_OVERDUE_HANDLERS` | `false` | `"true"` reclaims an overdue handler's pool slot by aborting it — see note below. |
+| `PB_NATS_SERVER_STALE_REQUEST_MS` | `0` (off) | Drop requests older than this at intake (the client has already retried or given up) — see note below. |
+| `PB_NATS_SERVER_SHUTDOWN_DRAIN_TIMEOUT` | overdue ms / 1000 + 5 | Seconds to let in-flight handlers finish during shutdown before abandoning them. |
+
+#### Shared
+
+| Variable | Default | Description |
+| --- | --- | --- |
+| `PB_NATS_CONNECTION_NAME` | hostname | Connection name shown in NATS server monitoring. Precedence: env var > yaml `connection_name` > hostname. |
+| `PROTOBUF_NATS_CONFIG_PATH` | `config/protobuf_nats.yml` | Path to the yaml config. |
+
+**Overdue handlers are never aborted by default** — killing a thread mid-handler can corrupt state. Enable
+`PB_NATS_SERVER_RECLAIM_OVERDUE_HANDLERS` only when orphaned work (whose clients already gave up) is saturating the
+pool and healthy traffic is being NACKed.
+
+**Stale-request shedding trusts client clocks.** The request age comes from the UUIDv7 reply token, which encodes the
+*client's* wall-clock time — enable only with sane NTP across hosts, and keep the threshold comfortably above the
+client ACK timeout. Requests without a UUIDv7 token (foreign clients) are never shed.
+
+### YAML config
+
+Connection-level settings live in a yaml file, keyed by environment (`RAILS_ENV` / `RACK_ENV` / `APP_ENV`, default
+`development`). The default path is `config/protobuf_nats.yml`; override with `PROTOBUF_NATS_CONFIG_PATH`.
+
+```yaml
+# config/protobuf_nats.yml
+---
+  production:
+    servers:
+      - "nats://127.0.0.1:4222"
+      - "nats://127.0.0.1:4223"
+      - "nats://127.0.0.1:4224"
+    max_reconnect_attempts: 500 # -1 reconnects forever
+    reconnect_time_wait: 2      # seconds between reconnect attempts (nats-pure default: 2)
+    ping_interval: 120          # seconds between health-check PINGs (nats-pure default: 120)
+    max_outstanding_pings: 2    # missed PINGs before the connection is declared dead (nats-pure default: 2)
+    connection_name: "my-service"
+    connect_timeout: 2
+    uses_tls: true
+    tls_client_cert: "/path/to/client-cert.pem"
+    tls_client_key: "/path/to/client-key.pem"
+    tls_ca_cert: "/path/to/ca.pem"
+    server_subscription_key_only_subscribe_to_when_includes_any_of:
+      - "search"
+      - "create"
+    server_subscription_key_do_not_subscribe_to_when_includes_any_of:
+      - "old_search"
+      - "old_create"
+    subscription_key_replacements:
+      - "original_service": "replacement_service"
+```
+
+`subscription_key_replacements` is useful for local testing but subject to breaking changes.
+
+### TLS
+
+With `uses_tls`, the client negotiates TLS 1.2–1.3 and **verifies the NATS server's certificate chain**
+(`VERIFY_PEER`): certificates must chain to `tls_ca_cert` when set, or to the system trust store otherwise. If you are
+upgrading from a release that did not verify (`< 0.13.1`), make sure `tls_ca_cert` points at the CA that signed your
+NATS server certificate, or the connection will be rejected. Hostname (SAN/CN) verification is not yet enabled.
 
 ## How it works
 
-`protobuf-nats` uses a single NATS client implementation (`NATS::IO::Client` from `nats-pure`) on both CRuby and JRuby.
-
-- **ResponseMuxer** (`lib/protobuf/nats/response_muxer.rb`) — the client uses a single wildcard subscription to multiplex
-  all RPC responses (similar to the Golang NATS client) instead of subscribing/unsubscribing per request. One or more
-  dispatcher threads drain the shared subscription and route each reply to the waiting caller via a `Concurrent::Map`,
-  keyed by a UUIDv7 request token. Tune the dispatcher count with `PB_NATS_RESPONSE_MUXER_DISPATCHERS`. Slow-consumer
-  protection on the response subscription is by message count (the queue depth); the dispatch hot path does no per-message
-  locking. Dispatcher threads self-heal: a crashed dispatcher is restarted with exponential backoff (capped at 60s) that
-  decays once healthy.
-- **SuperSubscriptionManager** (`lib/protobuf/nats/super_subscription_manager.rb`) — the server manages the lifecycle of
-  RPC endpoint subscriptions (NATS queue groups, so each request is delivered to one consumer), including slow start,
-  pausing, and resubscription. All subscriptions feed one shared intake queue drained by `PB_NATS_SERVER_SUBSCRIPTION_HANDLERS`
-  handler threads, so a slow ACK publish on one message can't head-of-line block every other subject. Handler threads
-  self-heal with exponential backoff.
-- **Server observability** — beyond the thread-pool gauges, the server emits in-flight handler metrics
-  (`server.inflight_count`, `server.inflight_oldest_age_ms`, `server.overdue_handler_count`, `server.handler_overdue`,
-  `server.pending_intake_queue_size`, `server.thread_pool_saturated`). Long-running handlers are allowed and never aborted
-  by default; a handler is only "overdue" once it outlives the client's `response_timeout` (see
-  `PB_NATS_SERVER_HANDLER_OVERDUE_MS`). Overdue handlers can optionally be reclaimed (emitting `server.handler_reclaimed`)
-  via `PB_NATS_SERVER_RECLAIM_OVERDUE_HANDLERS`.
-- **Error-callback observability** — `on_error` callbacks run on a bounded background executor so a slow callback can't
-  stall message processing. If that executor saturates under an error flood, dropped callbacks are counted
-  (`Protobuf::Nats.error_callback_drop_count`) and emit `error_callback_dropped` rather than being lost silently.
+`protobuf-nats` uses `nats-pure`'s `NATS::IO::Client` on both CRuby and JRuby.
+
+- **ResponseMuxer** (client) — one wildcard subscription multiplexes all RPC responses instead of subscribing per
+  request. Dispatcher threads route each reply to its waiting caller by UUIDv7 token, lock-free on the hot path, and
+  self-heal with decaying exponential backoff if they crash.
+- **SuperSubscriptionManager** (server) — manages the endpoint subscriptions (NATS queue groups: one consumer per
+  request) including slow start and pause/resume. All subscriptions feed one shared intake queue drained by handler
+  threads, so a slow ACK publish can't head-of-line block other subjects. Handlers self-heal like the muxer.
+- **Observability** — thread-pool gauges plus in-flight handler metrics (`server.inflight_count`,
+  `server.inflight_oldest_age_ms`, `server.overdue_handler_count`, `server.pending_intake_queue_size`,
+  `server.thread_pool_saturated`). Error callbacks run on a bounded background executor; drops are counted
+  (`Protobuf::Nats.error_callback_drop_count`) and emit `error_callback_dropped`.
 
 ## Resilience
 
-The client is built to ride out transient NATS hiccups rather than surface them as request failures:
-
-- **Transient transport errors are retried.** If a request hits a dropped/reset/closed connection (`EOFError`,
-  `IOError`, `Errno::ECONNRESET`/`EPIPE`/`ECONNREFUSED`/`ETIMEDOUT`, `NATS::IO::ConnectionClosedError`, or a Java
-  `IOException` on JRuby — see `Errors::RETRYABLE_TRANSPORT_ERRORS`), the client sleeps `PB_NATS_CLIENT_RECONNECT_DELAY`
-  and retries (up to 3 attempts) while `nats-pure` re-establishes the connection in the background.
-- **Missing ACKs and NACKs are retried** with their own timeouts/backoff (`PB_NATS_CLIENT_ACK_TIMEOUT`,
-  `PB_NATS_CLIENT_NACK_BACKOFF_INTERVALS`).
-- **Server-side failures fail the caller fast.** If the server cannot process a request after it has ACKed, it publishes
-  an encoded RPC error response (a generic message; the real error is logged server-side) so the client raises immediately
-  instead of blocking until `PB_NATS_CLIENT_RESPONSE_TIMEOUT`. A transport failure while publishing a successful response
-  no longer triggers a duplicate error response.
-- **The client connection self-heals after a terminal close.** A permanently closed NATS connection is dropped and
-  rebuilt on the next request instead of being reused.
-- **The response dispatcher self-heals.** A crashed muxer dispatcher restarts with exponential backoff, and a brief
-  subscription-restart window won't busy-spin the dispatch loop.
-
-See `bench/muxer_resilience_bench.rb` for microbenchmarks of the dispatch hot path and these resilience paths.
+The client rides out transient NATS hiccups rather than surfacing them as request failures:
+
+- **Transient transport errors are retried** (`Errors::RETRYABLE_TRANSPORT_ERRORS`): the client sleeps
+  `PB_NATS_CLIENT_RECONNECT_DELAY` (plus jitter) and retries up to `PB_NATS_CLIENT_MAX_RETRIES`, rebuilding a
+  terminally closed connection — and moving the muxer's subscription onto it — before each retry.
+- **A failing NATS node fails over automatically.** `nats-pure` reconnects through the whole `servers` pool and replays
+  all subscriptions on the new node. A node that dies *silently* (no FIN/RST) is only detected by the PING health
+  check — up to `ping_interval * max_outstanding_pings` (240s at defaults); lower those yaml keys for faster failover.
+- **Missing ACKs and NACKs are retried** with their own timeouts/backoff. Every retry re-sends the request — see
+  [Delivery semantics](#delivery-semantics-at-least-once).
+- **Server-side failures fail the caller fast.** A failed handler publishes a generic RPC error response (details stay
+  in server logs) so the client raises immediately instead of waiting out its response timeout.
+- **The server exits rather than running deaf.** If its connection is terminally closed (reconnects exhausted), the run
+  loop stops — emitting `server.connection_closed` — so a supervisor restarts the process. Use
+  `max_reconnect_attempts: -1` to prefer in-process reconnects forever.
 
 ## Delivery semantics (at-least-once)
 
-**Current design choice:** RPC delivery is **at-least-once**, and the gem does **not** deduplicate requests. The resilience features above are the reason: when the client retries on an ACK/response timeout or a transient transport error, the server may have *already received and processed* the original request, so a single client call can run a handler **more than once**. (NATS queue groups guarantee each *delivered* message goes to one consumer, but they do not prevent the client from re-sending after a timeout.)
-
-The gem deliberately favors at-least-once over at-most-once: dropping work on a transient blip is usually worse than occasionally repeating it. Making this safe is therefore the **service author's responsibility** — handlers that have side effects should be written to be idempotent:
+RPC delivery is **at-least-once** and the gem does **not** deduplicate: a client retry can re-send a request the server
+already processed, so a single call can run a handler more than once. This is deliberate — dropping work on a transient
+blip is usually worse than occasionally repeating it.
 
-- Key writes on a natural/business id or a client-supplied idempotency token (upsert / `find_or_create`) rather than blind inserts.
-- Make external side effects (charges, emails, downstream RPCs) safe to repeat, or guard them with your own dedup keyed on a request id you put in the message.
-- Naturally idempotent operations (reads, idempotent upserts) need no special handling.
+Making that safe is the service author's responsibility: key writes on a natural id or idempotency token
+(upsert / `find_or_create`), and make external side effects (charges, emails, downstream RPCs) safe to repeat or guard
+them with your own dedup. An opt-in per-RPC dedup with a pluggable store may be added later; it will not be the default.
 
-**Why no built-in dedup (yet):** correct dedup across a horizontally-scaled service requires a *shared* store (a retry can land on a different server instance), a tuned TTL, and a cached response to replay on duplicates — and it only helps RPCs that aren't already idempotent. A future, **opt-in per-RPC** dedup with a pluggable store may be added; it will not be the default. Until then, treat handlers as potentially re-run.
+## Future improvements
 
-## Future Improvements (locked behind ruby version)
 - Migrate from the `uuid7` gem to native `Random#uuid_v7` once the minimum Ruby version supports it (see `UUIDv7Helper`).
 
 ## Benchmarks
 
-Microbenchmarks live in `bench/` and measure both the old and new behavior in one process (no NATS server required). See `bench/bench.md` for details. Highlights on JRuby:
-
-- `bench/muxer_resilience_bench.rb` — response-muxer dispatch hot path (~2.5× faster per message with the per-message lock removed), restart-window resilience, and crash-counter accuracy.
-- `bench/server_intake_bench.rb` — server intake fan-out (~8× throughput, head-of-line stall ~505ms → ~2ms) and the handler-exhaustion observability.
+Microbenchmarks live in `bench/` and need no NATS server; see `bench/bench.md`. Highlights on JRuby: muxer dispatch
+~2.5× faster with the per-message lock removed (`bench/muxer_resilience_bench.rb`), server intake fan-out ~8×
+throughput with head-of-line stalls ~505ms → ~2ms (`bench/server_intake_bench.rb`).
 
 ```
 bundle exec ruby -Ilib bench/server_intake_bench.rb
@@ -266,16 +214,17 @@ bundle exec ruby -Ilib bench/muxer_resilience_bench.rb
 
 ## Development
 
-After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake test` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
-
-To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake test` to run the tests. You can
+also run `bin/console` for an interactive prompt that will allow you to experiment.
 
+To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the
+version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version,
+push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
 
 ## Contributing
 
 Bug reports and pull requests are welcome on GitHub at https://github.com/mxenabled/protobuf-nats.
 
-
 ## License
 
 The gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).