nats-pure 2.5.0 → 2.7.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 3dbc568f99c60cbe6a338d5aca7d9311e22624400cdb16e35b06d14c029815d8
-  data.tar.gz: 0f5108f668ccf94894e827896dc0cdd7c3b3c0026130e2ed05c7808d933b931a
+  metadata.gz: 18d1e6a53bcd6c3dc5c3fe1aa9f03c5b1d17a68fdf6381bae8f7ef3c6a6995b3
+  data.tar.gz: ce0f4719c5902ad9b7bec424f341cab7f39a980de31452fe445fbe6733a401d0
 SHA512:
-  metadata.gz: bb75b0dba73b2f0bf83c591e6416d67bc8722f583a1170bf83fc40eb536b598439e701c7b32335d3d1050efbbe3f34f865e4a8d7259bcd35a9f5247348453281
-  data.tar.gz: 8362a52a4309761638939795d66ef007f58856d19f3971110488ef3ee7b08f7e14077eaf237749571a744fa5c8eef0b36bb822e1a7ac8f9440503a9e6b245010
+  metadata.gz: 48591880c765700baac0ab6762fc437ef1da21cd728c43f64886547da8607208f8b68599cb75802ca3bc82d1e10e6c3875c9eac9c86e9ec955f035049d4e25db
+  data.tar.gz: 6de527742e1cb8a380f2806dae0f365612f7a6dbc622ebda50bc9cb46d3a4221dc296308c4c82b9c72c108dc0ff5462138f31aaf8898429d1028e1b7b2c50573

data/CHANGELOG.md

@@ -1,3 +1,93 @@
 # Change log
 
 ## main
+
+## v2.7.0
+
+This release adds the JetStream features of nats-server 2.10 to 2.14 that the
+client lacked, such as per-message TTLs, message schedules, and consumers that
+can be paused, reset and serve priority groups. `fetch` now pulls once and
+waits, like `Fetch` of the nats.go `jetstream` package, and a subscription read
+with `next_msg` no longer drops every message after its first 64 MiB. Code that
+passed stream or KV settings the client used to drop now sends them; see Changed.
+
+### Added
+
+- JetStream: `fetch(batch, no_wait: true)` takes what the server delivers at once, without waiting for more messages, like `FetchNoWait` of nats.go and `fetch` of nats.rs: an empty Array when none are pending, or when other pulls wait for more than are pending. While the consumer may not deliver more until some are acked (`max_ack_pending`), the server holds the pull back, and the fetch returns what it got once its timeout is up, 1 second by default. (#195)
+- JetStream: `fetch` takes a block, to which it passes each delivery as it comes, in the calling thread, so that the message can be acked before the `ack_wait` of the consumer is up: `psub.fetch(100, timeout: 60) { |msg| msg.ack }`. It still returns the messages. (#195)
+- JetStream: stream configs take the settings added in nats-server 2.10: `compression`, `first_seq`, `subject_transform` and `consumer_limits` (whose `inactive_threshold` is in nanoseconds, like the other stream durations). They used to be dropped, so a stream could not be created with them, and sending a fetched config back as an update left them out. (#191)
+- JetStream: stream configs take the settings added in nats-server 2.11 to 2.14: `allow_msg_ttl` and `subject_delete_marker_ttl` (2.11; the latter in nanoseconds, like the other stream durations), `allow_msg_counter`, `allow_atomic`, `allow_msg_schedules` and `persist_mode` (2.12), and `allow_batched` (2.14). Like nats.go, the client does not send them at their defaults, such as `false`, as servers since 2.12, and 2.11 ones in strict mode, refuse settings they do not know. (#194)
+- JetStream: `NATS::JetStream::Header` names the JetStream headers: those for deduplication and optimistic concurrency, such as `MSG_ID` and `EXPECTED_LAST_SUBJECT_SEQUENCE`, and those of nats-server 2.11 to 2.14: `EXPECTED_LAST_SUBJECT_SEQUENCE_SUBJECT`, `MSG_TTL` and `MARKER_REASON` (2.11), the atomic batch headers (2.12; `BATCH_COMMIT` `"eob"` needs 2.14), the message schedule headers (2.12 and 2.14), and `REQUIRED_API_LEVEL` (2.12), which JetStream API requests send as a `header:` option, as in `js.stream_info(name, header: {...})`. It also names the headers that the server sets on the messages it republishes, sources or returns from direct gets, such as `STREAM` and `SEQUENCE`. (#194)
+- JetStream: `js.publish` takes `ttl:`, the whole seconds after which the stream removes the message, or `:never` to keep it past the stream's `max_age`, for streams with `allow_msg_ttl` (nats-server 2.11). Other values raise `ArgumentError`. (#194)
+- JetStream: `js.publish` takes `schedule:`, which makes the message a schedule that publishes it to its `target:` subject: once, `at:` a Time (nats-server 2.12), `every:` so many seconds, or on a `cron:` expression with seconds, in UTC or a `time_zone:` (2.14). The schedule can give the messages it publishes a `ttl:` (2.12), and publish the last message of a `source:` subject instead and roll up the target (`rollup: true`) (2.14; 2.12 ignores both). The stream needs `allow_msg_schedules`. An invalid schedule raises `ArgumentError`, before anything is sent; `NATS::JetStream::Header::SCHEDULE_NEXT` and `SCHEDULER` cancel a schedule. (#194)
+- JetStream: `stream_info` reports the stream's `mirror` and `sources`, including their subject transforms, and both `stream_info` and `consumer_info` report when the server reported the info, as `ts`. (#191)
+- JetStream: `create_consumer` and `update_consumer` use the create and update actions of nats-server 2.10. Creating a consumer that exists with a different config raises `NATS::JetStream::Error::ConsumerAlreadyExists`, and updating one that does not exist raises `NATS::JetStream::Error::ConsumerDoesNotExist`; both are `BadRequest` errors. An update replaces the whole config, so fields left out take their defaults. `add_consumer` still creates or updates. (#191)
+- JetStream: `msg.term(reason: "...")` passes the reason on to the server's advisory for the terminated message. Servers before 2.10.4 do not terminate a message given a reason. (#191)
+- JetStream: consumers can be paused (nats-server 2.11): created paused with `pause_until` in their config, as a `Time` or an RFC 3339 String, or paused and resumed with `pause_consumer` and `resume_consumer`; updating a consumer keeps its pause. `consumer_info` reports whether a consumer is `paused`, and the seconds until it resumes, as `pause_remaining`. Servers before 2.11 create consumers unpaused, and let `pause_consumer` and `resume_consumer` time out. (#193)
+- JetStream: pull consumers can serve priority groups (nats-server 2.11). Consumer configs take `priority_policy`, `priority_groups` and `priority_timeout` (in seconds, like the other consumer durations), and `consumer_info` reports the state of the groups as `priority_groups`. `fetch` takes the `group` to pull from, `min_pending` and `min_ack_pending` for the `"overflow"` policy, and `priority` for the `"prioritized"` policy of nats-server 2.12. (#193)
+- JetStream: with the `"pinned_client"` priority policy, a pull subscription keeps the pin id that the server delivers with its messages, and sends it with its pulls. When the server says that the subscription is no longer pinned, as its pin expired or it was unpinned with the new `unpin_consumer`, a fetch that got no messages raises `NATS::JetStream::Error::PinIdMismatch`, and the next fetch can be pinned again. The server counts the consumer's `priority_timeout` from each pull, not while a pull waits, so keep the fetch timeout, plus the time between fetches, below it, or the subscription loses its pin. (#193, #195)
+- JetStream: `reset_consumer` makes a consumer deliver again from after its ack floor, or from the stream sequence given as `seq` (nats-server 2.14). A sequence that the deliver policy of the consumer does not let it be reset to raises `NATS::JetStream::Error::ConsumerInvalidReset`, a `BadRequest` error, and one that is not an integer of 0 or more raises `ArgumentError`. For a stream or consumer that does not exist, and with servers before 2.14, the request times out. (#193)
+- JetStream: `stream_info` and `consumer_info` report the `cluster` of the stream or consumer as a Hash, including when its leader was elected, as `leader_since` (nats-server 2.12). `consumer_info` used to drop it. (#193)
+- KV: buckets can be created with `compression: true` and with `metadata`, which `status.compressed?` and `status.metadata` report. (#191)
+- KV: `watch` takes an Array of keys, watched by a single consumer (nats-server 2.10). An empty Array watches all keys. (#191)
+
+### Changed
+
+- JetStream: `fetch` pulls once, for the whole batch, and waits: it returns as soon as it has the batch, and otherwise the messages that came when its timeout is up, like `Fetch` of the nats.go `jetstream` package and `batch` of nats.rs. A fetch of more than one message used to ask first, without waiting, for the messages that were pending. When it got some, it then waited out its timeout without pulling for more, so it neither took the messages that came meanwhile nor returned sooner. When other pulls of the consumer waited for more messages than were pending, the server turned that first pull away (408 Requests Pending), and `fetch` raised `NATS::IO::Timeout` at once. (#195)
+- JetStream: a pull subscription subscribes to `<inbox>.*`, which `subject` now returns, and each pull gets a reply subject of its own under it, as ADR-13 describes, so that a fetch tells the status that ended its own pull from those that ended the pulls of other fetches. A `fetch` with `no_wait: true` pulls with a subscription of its own, which it drops when done, so that its pull does not linger when the server holds it, as while the consumer may not deliver more messages until some are acked (`max_ack_pending`): the fetch then returns what it got once its timeout is up, and leaves what comes later to the next fetch. A server with leafnodes or gateways delivers to a pull until it is 2 seconds old even once the subscription is gone, so the subscription stays until then. (#195)
+- JetStream: `fetch` raises `ArgumentError` for a `timeout` that is not a finite positive number, such as 0, with which it used to raise `NATS::IO::Timeout` at once. It leaves the options it is given unchanged; it used to set their `:timeout`. (#195)
+- JetStream: as `fetch` always pulls with an expiry of its timeout, a consumer whose `max_expires` is shorter than the timeout turns every fetch away, with 409 "Exceeded MaxRequestExpires", raised as `NATS::JetStream::API::Error`. A fetch of more than one message used to get the messages that were pending first, with a pull that had no expiry. Pass a `timeout` no longer than the consumer's `max_expires`. (#195)
+- JetStream: `fetch` returns when the server ends its pull, so it takes its timeout plus the round trip to the server. When no end comes, it waits up to 1 second longer, like nats.go, before it returns what it got or raises `NATS::IO::Timeout`. (#195)
+- JetStream and KV: settings that used to be dropped are now sent. Code that already passed `compression`, `first_seq`, `subject_transform` or `consumer_limits` to `add_stream`, or `compression` or `metadata` to `create_key_value`, created its streams and buckets without them, so calling it again for an existing stream or bucket now fails with "stream name already in use with a different configuration" (err_code 10058). Apply the setting with `update_stream`, or stop passing it. (#191)
+- JetStream: the same goes for the settings of nats-server 2.11 to 2.14: code that passed them to `add_stream` created its streams without them, so calling it again for an existing stream now fails with err_code 10058. `update_stream` applies them, except `allow_msg_counter` and `persist_mode`, which a stream cannot change. (#194)
+- JetStream: consumer configs now send `pause_until`, `priority_policy`, `priority_groups` and `priority_timeout`, which used to be dropped. As `add_consumer` creates or updates, code that already passed the priority settings to it now gives an existing consumer priority groups, after which the server turns away pulls that name no group (400 "Priority Group missing"). (#193)
+- KV: `create_key_value` takes only `true` or `false` for `compression`; anything else, such as `"s2"`, which used to be ignored, raises `ArgumentError`. (#191)
+- JetStream: `add_consumer` leaves the `ConsumerConfig` it is given unchanged. It used to set its `name` and a default `ack_policy`, and convert its durations to nanoseconds, so passing the same config again sent durations a billion times too long. (#191)
+- JetStream: two `stream_info` or `consumer_info` results are no longer `==` unless they were reported at the same time, as they now carry `ts`, and the `cluster`, `mirror` and `sources`, whose lag and activity change all the time, even with servers that do not report `ts`. (#191, #193)
+- KV: `watch` no longer looks up the bucket's stream before creating its consumer. (#191)
+
+### Fixed
+
+- JetStream: `update_stream` with the config from `stream_info` failed with err_code 10052, such as "message TTL status can not be disabled", for streams with per-message TTLs, subject delete markers, counters, message schedules or async persistence, and turned off atomic and fast batches, as `StreamConfig` dropped the settings of nats-server 2.11 to 2.14. It now keeps them. (#194)
+- JetStream: `StreamConfig` dropped `discard_new_per_subject`, so `add_stream` created streams without it, and `update_stream` with the config from `stream_info` turned it off: a subject that held `max_msgs_per_subject` messages then discarded its oldest message instead of refusing the new one. It now keeps it. (#199)
+- JetStream: `js.publish` with `stream:` added `Nats-Expected-Stream` to the header Hash it was given, so a caller that reused the Hash sent it with its later publishes. (#194)
+- JetStream: `js.publish` raised `ArgumentError: unknown keywords` after the server had stored the message, when publishing to a stream with counters or committing an atomic batch, as the acks of nats-server 2.12 carry `val`, `batch` and `count`. `PubAck` now has these fields and ignores any it does not know, so fields added to acks later cannot break publish. (#192, #177, thanks @jnowakplacewise)
+- JetStream: `add_consumer`, `consumer_info`, `subscribe` and `pull_subscribe` raised `NoMethodError` for consumers that do not ack (`ack_policy: "none"`) created without an `ack_wait`, after the consumer had been created. The server omits `ack_wait` for them, and `config.ack_wait` is now `nil`. (#192)
+- JetStream: `fetch` took a status that an earlier pull had left in the subscription, such as the 408 of a fetch that timed out, for the reply to its own pull, and raised `NATS::IO::Timeout` at once, even with messages pending: a fetch of more than one message with any such status, a fetch of one message with a second one. `fetch` now first takes what earlier pulls left, and pulls for the rest only. And a fetch that got messages before an error status, such as 409 Consumer Deleted, raised the error and dropped them; like nats.go, it now returns them. (#193, #195)
+- JetStream: when threads fetched from the same subscription at once, a fetch could take the 408 that ended the pull of another for the end of its own, and raise `NATS::IO::Timeout` before its timeout. (#195)
+- JetStream: `fetch` pulled with an expiry a little short of its timeout only after a first pull, which did not wait, came back empty, so its pull outlived it by the time that first pull took: a message delivered meanwhile waited in the subscription for the next fetch, and, if none came, was delivered again once the `ack_wait` was up. A fetch now waits for the server to end its pull, which expires with its timeout. (#195)
+- JetStream: a `fetch` that waited longer than the `ack_wait` of the consumer returned a message twice when the server delivered it again meanwhile. It now returns each message once, as last delivered. (#195)
+- JetStream: `fetch` took a message of the stream with a `Status` header for a status of the server, and raised an error or ended without it. Statuses are now only the messages with a `Status` header that the server sends without a reply. (#195)
+- JetStream: a `fetch` of one message raised `TypeError: nil can't be coerced into Float` when threads fetched from the same subscription at once: woken up for a message that another thread took, it did not wait on. It now waits until its timeout. (#193, #181, thanks @route)
+- JetStream: a `fetch` of one message raised `NATS::IO::Timeout` ("nats: fetch request timeout") before its timeout when a status that ended an earlier pull, such as the 408 of a fetch that had just timed out, came while it waited, and left its own message for the next fetch. Like nats.go, it now waits on for its message until its timeout. (#193)
+- JetStream: `fetch` could miss the reply to its pull when it came before the fetch waited for it, as when its thread was preempted: a fetch of one message then waited out its timeout and raised `NATS::IO::Timeout`, leaving its message to be redelivered after the ack wait, and a fetch of more messages returned only the first one, after its timeout. `fetch` now takes what is there before it waits. (#193)
+- KV: after its consumer was recreated, as after a server restart, a watcher yielded the entries of every key in the bucket instead of the watched keys. The recreated consumer now keeps the keys filter. (#192)
+- KV: after its consumer was recreated, a watcher yielded its last entry again, and a watcher with no entries yet, such as one on an empty bucket, never got its consumer back (`err_code` 10094) and stayed silent. The recreated consumer now starts after the last entry. (#192, #175, thanks @Xayc73)
+- A subscription without a callback, read with `next_msg`, dropped every message and reported `NATS::IO::SlowConsumer` once it had received its `pending_bytes_limit` in all, 64 MiB by default, however fast it was read, as `next_msg` did not take the messages it returned off the pending bytes. (#196)
+
+## v2.6.0
+
+This release prepares nats-pure for nats-server 2.16, whose new JetStream
+acknowledgement format breaks JetStream consumers and KV watchers in earlier
+versions, and fixes a deadlock that could hang `Client#close`.
+
+### Changed
+
+- With `reconnect: false`, losing the connection now closes the client, as nats.go does: the status becomes `CLOSED`, the close callback is called, and `publish` raises `NATS::IO::ConnectionClosedError`. It used to stay `DISCONNECTED` and keep accepting publishes that were never sent, because the close, started from the read loop, killed the read loop part way through. (#189)
+
+### Fixed
+
+- JetStream: accept the 11-token acknowledgement subjects that nats-server 2.16 sends by default (the v2 format of ADR-15). Earlier versions fail to read the metadata of every JetStream message from such a server, which breaks pull and push consumers and KV watchers. (#185)
+- JetStream: an unrecognized acknowledgement subject raises `NATS::JetStream::Error::NotJSMessage` instead of `NameError`, and subjects whose numeric fields do not parse are rejected. (#185)
+- JetStream: message metadata timestamps keep their full nanosecond precision. (#185)
+- KV: watchers track their position from message metadata, so they resume after a server restart with either acknowledgement format. (#185)
+- `Client#close` could hang forever. It stopped the client's background threads with `Thread#exit`, and killing a thread that is waiting for the client lock can lose Ruby's mutex wakeup, leaving `close` asleep on an unlocked lock. The threads are now asked to stop and joined, on close and on reconnect. (#183, #189)
+- A close while a reconnect was under way did not stick: the reconnect carried on, connected again and restarted the client, subscriptions included. (#189)
+- A subscription with a concurrency limit could run its callbacks in parallel, and so out of order: an empty pending queue released the concurrency permit twice. (#186)
+- A subscription could stop processing messages for good after a reconnect: shutting down the executor during dispatch killed the read loop and leaked the concurrency permit. (#182)
+- WebSocket: protocol lines sharing a frame, such as the `PONG` and `INFO` that nats-server 2.12.2 sends together, were misparsed (#174, thanks @artyomb), and bytes that arrived with the reply to the connect handshake were lost. (#182, #186)
+- JRuby: a refused connection raises `Errno::ECONNREFUSED`, as on MRI. (#182, #186)
+
+### Improved
+
+- Tested against nats-server 2.14, 2.15 and the upcoming 2.16 (`main`), on Ruby 3.0 to 3.4 and JRuby. (#182, #188)

data/README.md

@@ -102,6 +102,89 @@ loop do
 end
 ```
 
+`fetch` returns as soon as it has the batch, or once its timeout is up, the
+messages that came. With `no_wait: true`, it takes what the server delivers at
+once, without waiting for more. Given a block, it passes each message to it as it comes,
+so that the message can be acked before the `ack_wait` of the consumer is up:
+
+```ruby
+psub.fetch(100, timeout: 60) do |msg|
+  puts msg.data
+  msg.ack
+end
+```
+
+Messages can expire on their own, in streams that allow it (nats-server 2.11):
+
+```ruby
+js.add_stream(name: "sessions", subjects: ["sessions.>"], allow_msg_ttl: true)
+
+# Removed after an hour. With ttl: :never, kept past the max_age of the stream.
+js.publish("sessions.42", "token", ttl: 3600)
+```
+
+Messages can also be scheduled (nats-server 2.12; `every:` and `cron:` need 2.14):
+
+```ruby
+js.add_stream(name: "reminders", subjects: ["reminders.>", "due.>"], allow_msg_schedules: true)
+
+# Published to due.42 in an hour. Another schedule on reminders.42 replaces it.
+js.publish("reminders.42", "call back", schedule: {at: Time.now + 3600, target: "due.42"})
+
+# Published to due.standup at 9:00 on weekdays, Warsaw time.
+js.publish("reminders.standup", "stand-up",
+  schedule: {cron: "0 0 9 * * 1-5", time_zone: "Europe/Warsaw", target: "due.standup"})
+
+# Cancels the schedule on reminders.42.
+js.publish("reminders.cancelled", "", header: {
+  NATS::JetStream::Header::SCHEDULE_NEXT => "purge",
+  NATS::JetStream::Header::SCHEDULER => "reminders.42"
+})
+```
+
+Streams and consumers can also be managed directly:
+
+```ruby
+# Compressed, and no consumer may have more than 1000 unacknowledged messages.
+js.add_stream(name: "orders", subjects: ["orders.>"], compression: "s2",
+  consumer_limits: {max_ack_pending: 1000})
+
+config = {durable_name: "billing", filter_subject: "orders.paid", max_ack_pending: 100}
+
+# Fails with NATS::JetStream::Error::ConsumerAlreadyExists if "billing"
+# exists with a different config.
+js.create_consumer("orders", config)
+
+# An update replaces the whole config, so send all of it again.
+# Fails with NATS::JetStream::Error::ConsumerDoesNotExist if there is no "billing".
+js.update_consumer("orders", config.merge(max_ack_pending: 200))
+
+# Deliver nothing for an hour, or until resumed.
+js.pause_consumer("orders", "billing", Time.now + 3600)
+js.resume_consumer("orders", "billing")
+```
+
+Pull consumers can serve priority groups. With the `"pinned_client"` policy,
+the server delivers to one subscription at a time, and to another once it
+stops pulling for `priority_timeout` seconds. The server counts it from
+each pull, not while a pull waits: keep the fetch timeout, 5 seconds by
+default, plus the time between fetches, below `priority_timeout`, or the
+subscription can lose its pin:
+
+```ruby
+js.create_consumer("orders", durable_name: "shipping", filter_subject: "orders.paid",
+  priority_policy: "pinned_client", priority_groups: ["workers"], priority_timeout: 60)
+
+psub = js.pull_subscribe("orders.paid", "shipping", stream: "orders")
+begin
+  psub.fetch(5, group: "workers").each(&:ack)
+rescue NATS::JetStream::Error::PinIdMismatch
+  # This subscription is no longer pinned; the next fetch may be pinned again.
+rescue NATS::IO::Timeout
+  # Nothing to do, or another subscription is pinned.
+end
+```
+
 ## Service API
 
 The service API allows you to easily [build NATS services](docs/service_api.md).

data/lib/nats/io/client.rb

@@ -221,6 +221,10 @@ module NATS
       # Track whether connect has been already been called.
       @connect_called = false
 
+      # Bumped by every close, so that a reconnect started earlier can tell
+      # that the connection was closed meanwhile.
+      @close_generation = 0
+
       # New style request/response implementation.
       @resp_sub = nil
       @resp_map = nil
@@ -1293,6 +1297,11 @@ module NATS
     # Handles errors from reading, parsing the protocol or stale connection.
     # the lock should not be held entering this function.
     def process_op_error(e)
+      # A background thread being stopped by close or a reconnect sees
+      # errors from that; it must not start another close or reconnect,
+      # which would wait for the thread that is stopping it.
+      return if stopping?
+
       should_bail = synchronize do
         connecting? || closed? || reconnecting?
       end
@@ -1320,23 +1329,29 @@ module NATS
 
     def initiate_reconnect
       @status = RECONNECTING
-      @io&.close
+      old_io = @io
       @io = nil
 
+      # Wake a read loop blocked on the socket by shutting it down, and
+      # close the socket only once the loop has stopped: on JRuby a close
+      # right after the shutdown can leave the read blocked for good.
+      if @read_loop_thread&.alive? && @read_loop_thread != Thread.current
+        old_io&.shutdown_read
+      end
+
       # TODO: Reconnecting pending buffer?
 
+      generation = @close_generation
+
       # Do reconnect under a different thread than the one
       # in which we got the error.
       Thread.new do
-        # Abort currently running reads in case they're around
-        # FIXME: There might be more graceful way here...
-        @read_loop_thread.exit if @read_loop_thread.alive?
-        @flusher_thread.exit if @flusher_thread.alive?
-        @ping_interval_thread.exit if @ping_interval_thread.alive?
+        stop_threads!
+        old_io&.close
 
         @subscription_executor.shutdown
 
-        attempt_reconnect
+        attempt_reconnect(generation)
       rescue NATS::IO::NoServersError => e
         @last_err = e
         close
@@ -1346,6 +1361,8 @@ module NATS
     # Gathers data from the socket and sends it to the parser.
     def read_loop
       loop do
+        return if stopping?
+
         should_bail = synchronize do
           # FIXME: In case of reconnect as well?
           @status == CLOSED or @status == RECONNECTING
@@ -1361,6 +1378,9 @@ module NATS
         # FIXME: We do not really need a timeout here...
         retry
       rescue => e
+        # Asked to stop: the error comes from the socket being shut down.
+        return if stopping?
+
         # In case of reading/parser errors, trigger
         # reconnection logic in case desired.
         process_op_error(e)
@@ -1373,6 +1393,7 @@ module NATS
       loop do
         # Blocks waiting for the flusher to be kicked...
         @flush_queue.pop
+        return if stopping?
 
         should_bail = synchronize do
           (@status != CONNECTED && !draining?) || @status == CONNECTING
@@ -1411,9 +1432,9 @@ module NATS
       end
     end
 
-    def ping_interval_loop
+    def ping_interval_loop(stop)
       loop do
-        sleep @options[:ping_interval]
+        return if stop.wait(@options[:ping_interval])
 
         # Skip ping interval until connected
         next if !connected?
@@ -1486,8 +1507,11 @@ module NATS
       end
     end
 
-    # Reconnect logic, this is done while holding the lock.
-    def attempt_reconnect
+    # Reconnect logic. generation is @close_generation from when the
+    # reconnect started; a close since then cancels the reconnect.
+    def attempt_reconnect(generation)
+      return if closed_since?(generation)
+
       @disconnect_cb&.call(@last_err)
 
       # Clear sticky error
@@ -1496,6 +1520,8 @@ module NATS
       # Do reconnect
       srv = nil
       begin
+        return if closed_since?(generation)
+
         srv = select_next_server
 
         # Set hostname to use for TLS hostname verification
@@ -1537,37 +1563,49 @@ module NATS
         retry
       end
 
-      # Clear pending flush calls and reset state before restarting loops
-      @flush_queue.clear
-      @pings_outstanding = 0
-      @pongs_received = 0
+      # Under the lock, so that a concurrent close either cancels this or
+      # finds the new connection in place and closes it.
+      synchronize do
+        if @close_generation != generation
+          @io.close
+          @io = nil
+          return
+        end
 
-      # Replay all subscriptions
-      @subs.each_pair do |sid, sub|
-        @io.write("SUB #{sub.subject} #{sub.queue} #{sid}#{CR_LF}")
-      end
+        # Clear pending flush calls and reset state before restarting loops
+        @flush_queue.clear
+        @pings_outstanding = 0
+        @pongs_received = 0
 
-      # Flush anything which was left pending, in case of errors during flush
-      # then we should raise error then retry the reconnect logic
-      cmds = []
-      cmds << @pending_queue.pop until @pending_queue.empty?
-      @io.write(cmds.join) unless cmds.empty?
-      @status = CONNECTED
-      @pending_size = 0
+        # Replay all subscriptions
+        @subs.each_pair do |sid, sub|
+          @io.write("SUB #{sub.subject} #{sub.queue} #{sid}#{CR_LF}")
+        end
 
-      # Reset parser state here to avoid unknown protocol errors
-      # on reconnect...
-      @parser.reset!
+        # Flush anything which was left pending, in case of errors during flush
+        # then we should raise error then retry the reconnect logic
+        cmds = []
+        cmds << @pending_queue.pop until @pending_queue.empty?
+        @io.write(cmds.join) unless cmds.empty?
+        @status = CONNECTED
+        @pending_size = 0
 
-      # Now connected to NATS, and we can restart parser loop, flusher
-      # and ping interval
-      start_threads!
+        # Reset parser state here to avoid unknown protocol errors
+        # on reconnect...
+        @parser.reset!
+
+        # Now connected to NATS, and we can restart parser loop, flusher
+        # and ping interval
+        start_threads!
+      end
 
-      # Dispatch the reconnected callback while holding lock
-      # which we should have already
       @reconnect_cb&.call
     end
 
+    def closed_since?(generation)
+      synchronize { @close_generation != generation }
+    end
+
     def close_connection(conn_status, do_cbs = true)
       synchronize do
         @connect_called = false
@@ -1575,25 +1613,10 @@ module NATS
           @status = conn_status
           return
         end
+        @close_generation += 1
       end
 
-      # Kick the flusher so it bails due to closed state
-      @flush_queue << :fallout if @flush_queue
-      Thread.pass
-
-      # FIXME: More graceful way of handling the following?
-      # Ensure ping interval and flusher are not running anymore
-      if @ping_interval_thread&.alive?
-        @ping_interval_thread.exit
-      end
-
-      if @flusher_thread&.alive?
-        @flusher_thread.exit
-      end
-
-      if @read_loop_thread&.alive?
-        @read_loop_thread.exit
-      end
+      stop_threads!
 
       @subscription_executor&.shutdown
       @subscription_executor&.wait_for_termination(options[:close_timeout])
@@ -1642,21 +1665,58 @@ module NATS
       end
     end
 
+    # Asks the read loop, flusher and ping threads to stop, wakes them from
+    # wherever they block, and waits for them to finish.
+    #
+    # They are never killed: Thread#exit on a thread that is waiting for
+    # the client lock can swallow the lock's wakeup, so that the next
+    # thread waiting for the lock sleeps forever on an unlocked lock (#183).
+    def stop_threads!(timeout = NATS::IO::THREADS_STOP_TIMEOUT)
+      threads = [@read_loop_thread, @flusher_thread, @ping_interval_thread].compact
+      threads.each { |t| t[:nats_stop]&.stop! } # also wakes the ping thread
+
+      begin
+        @flush_queue&.push(:fallout, true)
+      rescue ThreadError
+        # Queue is full, so the flusher is awake already.
+      end
+      # Wake the read loop if it is blocked on the socket. Only while it
+      # runs: shutdown acts on the socket itself, which a forked child
+      # (whose threads are gone) shares with the parent.
+      if @read_loop_thread&.alive? && @read_loop_thread != Thread.current
+        @io&.shutdown_read
+      end
+
+      deadline = Process.clock_gettime(Process::CLOCK_MONOTONIC) + timeout
+      threads.each do |t|
+        next if t == Thread.current
+
+        t.join([deadline - Process.clock_gettime(Process::CLOCK_MONOTONIC), 0].max)
+      end
+    end
+
+    def stopping?
+      Thread.current[:nats_stop]&.stopped?
+    end
+
+    def start_thread(name, stop, &block)
+      thread = Thread.new(&block)
+      thread[:nats_stop] = stop
+      thread.name = name
+      thread.abort_on_exception = true
+      thread
+    end
+
     def start_threads!
       # Reading loop for gathering data
-      @read_loop_thread = Thread.new { read_loop }
-      @read_loop_thread.name = "nats:read_loop"
-      @read_loop_thread.abort_on_exception = true
+      @read_loop_thread = start_thread("nats:read_loop", NATS::IO::StopSignal.new) { read_loop }
 
       # Flusher loop for sending commands
-      @flusher_thread = Thread.new { flusher_loop }
-      @flusher_thread.name = "nats:flusher_loop"
-      @flusher_thread.abort_on_exception = true
+      @flusher_thread = start_thread("nats:flusher_loop", NATS::IO::StopSignal.new) { flusher_loop }
 
       # Ping interval handling for keeping alive the connection
-      @ping_interval_thread = Thread.new { ping_interval_loop }
-      @ping_interval_thread.name = "nats:ping_loop"
-      @ping_interval_thread.abort_on_exception = true
+      ping_stop = NATS::IO::StopSignal.new
+      @ping_interval_thread = start_thread("nats:ping_loop", ping_stop) { ping_interval_loop(ping_stop) }
 
       # Subscription handling thread pool
       @subscription_executor = Concurrent::ThreadPoolExecutor.new(
@@ -1889,6 +1949,44 @@ module NATS
     DEFAULT_DRAIN_TIMEOUT = 30
     DEFAULT_CLOSE_TIMEOUT = 30
 
+    # How long to wait for the read loop, flusher and ping threads to
+    # stop on close or reconnect.
+    THREADS_STOP_TIMEOUT = 5
+
+    # Asks a background thread to stop, and lets it sleep until then.
+    class StopSignal
+      def initialize
+        @mutex = Mutex.new
+        @cond = ConditionVariable.new
+        @stopped = false
+      end
+
+      def stop!
+        @mutex.synchronize do
+          @stopped = true
+          @cond.broadcast
+        end
+      end
+
+      def stopped?
+        @stopped
+      end
+
+      # Sleeps up to timeout seconds; returns whether stop! was called.
+      def wait(timeout)
+        deadline = Process.clock_gettime(Process::CLOCK_MONOTONIC) + timeout
+        @mutex.synchronize do
+          until @stopped
+            left = deadline - Process.clock_gettime(Process::CLOCK_MONOTONIC)
+            break if left <= 0
+
+            @cond.wait(@mutex, left)
+          end
+          @stopped
+        end
+      end
+    end
+
     # Default Pending Limits
     DEFAULT_SUB_PENDING_MSGS_LIMIT = 65536
     DEFAULT_SUB_PENDING_BYTES_LIMIT = 65536 * 1024
@@ -2004,6 +2102,15 @@ module NATS
         @socket.close
       end
 
+      # Makes a read blocked on the socket in another thread return,
+      # while writes keep working.
+      def shutdown_read
+        io = @socket.respond_to?(:to_io) ? @socket.to_io : @socket
+        io.shutdown(::Socket::SHUT_RD)
+      rescue IOError, SystemCallError
+        # Already closed or not connected.
+      end
+
       def closed?
         @socket.closed?
       end
@@ -2030,6 +2137,12 @@ module NATS
         end
 
         sock
+      rescue IOError => e
+        # JRuby raises a plain IOError for a refused connection.
+        if (RUBY_ENGINE == "jruby") && e.message.include?("Connection refused")
+          raise Errno::ECONNREFUSED
+        end
+        raise
       end
     end
   end