dalli 5.2.6 → 5.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: db76345a60e108cc8a1aa4f39adae0e657b200b4e3f17f7196702776c1ff8bc4
-  data.tar.gz: eb643908bb2aa3654f2d7c82bcce21fa82159931e162cdf17bd79d9fe40b2ed0
+  metadata.gz: 2bf7ed15c1eb033940d2d1fb04d6577f9676d12bb30b547da704a56b282f0cdf
+  data.tar.gz: 65208448d43b257b0cd4f3af9b6f4d01e13c509f13482a2f641d6d01a1573bd7
 SHA512:
-  metadata.gz: e5d227f5b4e7fda6e35907f0d94b74fcd528d0c203ce6a7c5b01ab33e2908fe78147c70df8661eedbc2b82f1024b4c89bb890fc0418745150c379805f87afea2
-  data.tar.gz: dbb2e3d7bb7b1c852a8c0d8b359f1621d0aa0f3ef32dbe1ca449e384325a929e0db6ba893357195e284c84162ce3ae4a1ce39e9f86b5b1bfde149a3725ae3a45
+  metadata.gz: ee59bbf0d4795a614a143f00594f0f90fcb17592ffb7439bf72a824dbb19d135718e18b5101706819f01039999b4d0c6c7f59c9aea98f09d78e7d6bfee4176ea
+  data.tar.gz: ffe4b0170ad6a31071ba8fe5275a4479b83efd31fe83bb97c790b19b86bf44d19ed6fc5bb67d2c2862a5e2d82068106d114e06a566c71c92d780e714762f5f26

data/CHANGELOG.md

@@ -4,6 +4,54 @@ Dalli Changelog
 Unreleased
 ==========
 
+5.3.0
+==========
+
+New features:
+
+- `enforce_read_timeout: true` makes JRuby and TruffleRuby give up waiting for a reply after `socket_timeout`, as CRuby does. Without it, a server that accepted a request but never replied blocked the caller indefinitely on those engines, including in `get` and `get_multi`. A server that stops partway through a reply can still block. It's off by default because it costs about 8% of single-key throughput on TruffleRuby and 20-29% on JRuby, and it has no effect on CRuby or TLS connections
+
+Bug fixes:
+
+- A `vivify_ttl` (or `fetch_with_lock`'s `lock_ttl`) over 30 days now protects against a thundering herd. memcached reads a TTL over 30 days as a Unix timestamp, so the stub for a miss expired at once and every caller won the recache race. `vivify_ttl` is now converted like other TTLs
+- `set_cas`, `replace_cas` and `delete_cas` dropped a CAS that couldn't be parsed, such as `'abc'`, and sent an unconditional write, so the value was overwritten or deleted whatever its CAS. They now raise `ArgumentError`
+- `flush('abc')` flushed immediately. It now raises `ArgumentError`
+- A TTL String that isn't a whole number, such as `'abc'`, became 0, which means no expiry. It now raises `ArgumentError`, including for `expires_in` when the client is created
+- A TTL, CAS, `incr`/`decr` amount or initial value outside the range memcached accepts got a `CLIENT_ERROR` reply, which a `quiet` block or `set_multi` discarded, so the write silently did nothing. These now raise `ArgumentError`
+- An invalid `vivify_ttl` or `recache_ttl` raised from deep in the request and closed the connection
+- `tombstone_ttl` read a String with a leading zero as octal (`'010'` was 8) and accepted hex such as `'0x10'`. It's now parsed in base 10 like every other TTL, and `Float::NAN` raises `ArgumentError` rather than `FloatDomainError`
+- Multi-gets now raise `Dalli::UnmarshalError` for a value that can't be deserialized, as `get` does, and only after reading the rest of the replies
+  - `get_multi` and `get_multi_cas` raised partway through reading a server's replies, closing the connection, and `get_multi_with_metadata` on more than one server silently dropped every key on that server. A large `get_multi` could also drop that server's keys without raising when it read the bad value while still sending requests (seen mostly on JRuby and TruffleRuby)
+  - In block form the other values are still yielded, and the connections stay open. `get` and the other single-key reads also keep the connection open after this error now
+- `get_multi`, `get_multi_cas` and `get_multi_with_metadata` now return each value under the key the caller passed. They rebuilt the key from the one memcached returned, so:
+  - With a namespace, a key that isn't valid UTF-8 (such as `"\xff".b`) raised `ArgumentError` for the whole call
+  - A key too long to send as is (over 250 bytes, or once base64-encoded) came back under its digested form
+  - A binary key with non-ASCII bytes came back tagged UTF-8, so `result.key?(key)` was false
+  - A key given more than once is now requested once, and a block is yielded it once
+- On TruffleRuby, a process spawned while Dalli had a connection open (with `system`, `spawn` or backticks) inherited the socket and kept the connection open on memcached's side until it exited. Dalli's sockets are now close-on-exec, as CRuby makes them by default
+- A request that failed after it was sent, such as one whose reply timed out, was sent again even though memcached may have applied it. `incr`, `decr`, `append` and `prepend` could be applied twice, and `add` and writes with a CAS could report failure for a write that had succeeded. Once some of a request has been sent, Dalli now retries only requests that are safe to apply twice (see Other changes)
+- When a server stopped while connected, the multi-key operations (`get_multi`, `get_multi_cas`, `get_multi_with_metadata`, `set_multi` and `delete_multi`) returned its keys as misses, or left them unwritten, instead of raising `Dalli::NetworkError` as documented. The retry found the server down and skipped it as if it had been down from the start. Without `failover` they now raise. With `failover` the keys still go to another server, as they do for `get` and `set`, when the retry can't connect to the failed server. A server that accepts the retry's connection and fails again is marked down during the operation, which raises `Dalli::NetworkError` either way (#1263, @jaideeppyne)
+- `Rack::Session::Dalli`: a stored session that couldn't be decoded (for example after changing the serializer, or removing a class that sessions stored) left the user on a session that was never saved, with no new cookie, so they couldn't log in until the cookie was cleared or the session expired. It's now treated as missing: the request gets a new session and cookie, and a warning is logged
+- A reply value is now checked for the CRLF that should follow it. Values were read by their declared size alone, so a size that didn't match the value (a broken proxy, or an injected reply) was returned as a hit with a `nil` value, or the rest of the value was read as the next reply. These now raise `Dalli::DalliError` and close the connection, as a value size out of range does
+- `get_multi_cas`, and `get_multi` on more than one server, returned a server's keys as misses when it sent nothing within `socket_timeout`, without retrying or counting a failure. They now treat it like any read timeout: the batch is retried, and `Dalli::NetworkError` is raised once the server is marked down, as `get`, `get_multi_with_metadata` and single-server `get_multi` already did. `socket_timeout` also used to cover the whole batch, so a large reply that was still arriving could be cut off, with its remaining keys returned as misses. The wait now starts again whenever data arrives (#1285)
+- A reply key marked base64 that isn't valid base64 raised `ArgumentError` from `get_multi`, `get_multi_cas` and `get_multi_with_metadata`, which callers rescuing `Dalli::DalliError` didn't catch. It now raises `Dalli::DalliError`. Found by the new fuzz targets
+- `set_multi` and `delete_multi` on more than one server retried forever, hanging the caller, when a server accepted their writes and then closed the connection before replying. Each quiet write reset the server's failure count, so it never reached `socket_max_failures`. Only a request whose reply is read resets it now, and the server is marked down and `Dalli::NetworkError` raised, as for `get`
+- Since 5.2.0, on JRuby with TLS, Dalli could wait `socket_timeout` for part of a reply that had already arrived, because jruby-openssl can hold it where neither `IO.select` nor OpenSSL's `pending` reports it. With the multi-get change above (#1285), that wait would have made `get_multi` raise `Dalli::NetworkError`. Dalli now reads on JRuby until the TLS socket would block
+
+Other changes:
+
+- `Rack::Session::Dalli` docs: without `expire_after`, sessions are stored with no expiry and stay in memcached until deleted or evicted
+- Numeric arguments are checked before anything is written, so a bad one raises `ArgumentError` without affecting the connection or earlier writes in a `quiet` block. Integers and Strings of decimal digits (such as `'300'`) are accepted throughout
+  - A TTL (`ttl`, `expires_in`, `vivify_ttl`, `lock_ttl`, `tombstone_ttl`) must fit memcached's signed 32-bit expiration time once a TTL over 30 days is converted to a Unix time. A negative TTL is still accepted and expires the item. A Float or `ActiveSupport::Duration` TTL is still converted with `to_i`
+  - A CAS, an `incr`/`decr` amount or initial value, and a `flush` delay must be from 0 to 2**64 - 1. A Float now raises; `incr(key, 0.9)` used to truncate the amount to 0. A String amount such as `'5'` now works
+  - `fetch`, `fetch_with_lock`, `cas` and `cas!` check `ttl` before running the block, so a bad one raises before any side effects. `fetch` checks it on a hit too
+  - `Dalli::Protocol::TtlSanitizer.sanitize` raises `ArgumentError` for a String that isn't a whole number instead of returning 0
+- Usernames and passwords from `memcached://` URIs and the `:username`/`:password` options are no longer kept in the client's or server's options, so `#inspect` can't show them. Dalli 5 has never used them, and the warning is unchanged
+- A TTL over 30 days is converted to a Unix time once per request, not twice, and the debug message for that conversion is built only when debug logging is on
+- `incr`, `decr`, `append`, `prepend`, `add`, `set_cas`, `replace_cas`, `delete_cas`, `cas`, `cas!`, `fetch_with_lock`, and `get_with_metadata` with `vivify_ttl` or `recache_ttl` now raise `Dalli::NetworkError` when the connection fails after the request was sent, where they used to retry it. Reads, `touch`, `set`, `replace` and `delete` without a CAS, and the multi-key operations are still retried, as is any request that failed before it was sent
+  - `fetch` (which stores a missing value with `add`) and `Rack::Session::Dalli` (which creates sessions with `add`) can see this error too. `Rack::Session::Dalli` handles it like any other memcached error
+  - Before a request on a connection unused for over a second, Dalli checks whether the other end closed it (memcached's `idle_timeout`, a load balancer or a firewall), and if so reconnects before sending. Without this, the first `incr` or `add` after an idle spell raised this error, and any request counted a failure toward `socket_max_failures`
+
 5.2.6
 ==========
 

data/README.md

@@ -96,6 +96,28 @@ end
 
 `drain_deferred_responses` only sends a `noop` to servers that were sent quiet requests since they were last drained, so calling it when nothing is pending costs nothing.
 
+## Network Errors and Retries
+
+When a request fails with a network error (a refused connection, a reset, or a read that times out), Dalli reconnects and retries it. A server that fails `socket_max_failures` times in a row is marked down, and requests to it raise `Dalli::NetworkError` (or, with `failover`, go to another server).
+
+On CRuby, a read that waits longer than `socket_timeout` raises a network error, which is retried and counted like any other. JRuby and TruffleRuby don't time out a blocked socket read, so on those engines a server that accepts a request and never replies blocks the caller indefinitely. Pass `enforce_read_timeout: true` to apply `socket_timeout` there too:
+
+```ruby
+Dalli::Client.new('localhost:11211', socket_timeout: 0.5, enforce_read_timeout: true)
+```
+
+It limits how long Dalli waits for a reply to start, which covers a server that stops answering. A server that stops partway through a reply can still block. It's off by default because the check costs throughput, noticeably on JRuby. It has no effect on CRuby, which doesn't need it, or on TLS connections.
+
+`get_multi` and `get_multi_cas` wait up to `socket_timeout` for any server to send more of its replies, on every engine. The wait starts again each time data arrives, so a large batch can take longer than `socket_timeout` in total. A server that sends nothing for `socket_timeout` counts as a failed read, as it would for `get`.
+
+A request that failed after some of it was sent may have been applied by memcached even though its reply never arrived. Dalli only sends such a request again when applying it twice is harmless: reads, `touch`, `set`, `replace` and `delete` without a CAS, and the multi-key operations. For `incr`, `decr`, `append`, `prepend`, `add`, writes with a CAS (`set_cas`, `replace_cas`, `delete_cas`, `cas`, `cas!`), and `get_with_metadata` with `vivify_ttl` or `recache_ttl` (which `fetch_with_lock` uses), it raises `Dalli::NetworkError` instead, since the write may or may not have happened. A retried `delete` can return false if the first attempt deleted the key.
+
+A connection that sat unused for over a second is checked before the next request. If memcached (with `idle_timeout`), a load balancer or a firewall closed it meanwhile, Dalli reconnects before sending, so the request isn't lost and doesn't count as a failure. A connection dropped without the other end closing it can't be detected this way, and a request on it fails as above.
+
+The multi-key operations (`get_multi`, `get_multi_cas`, `get_multi_with_metadata`, `set_multi` and `delete_multi`) skip a server that is already down when they start: its keys are missing from the result, or go to another server with `failover`. A server that fails during the operation is retried. If the retry can't connect to it, then without `failover` the operation raises `Dalli::NetworkError`, rather than returning that server's keys as misses or leaving them unwritten, and with `failover` its keys go to another server, or are left out when no server is left. If the retry connects but the server fails again, it's marked down partway through the operation, which raises `Dalli::NetworkError` with or without `failover`.
+
+A retried `get_multi` or `get_multi_cas` starts the batch again, so a block passed to either can be yielded a key it was already yielded before the error. Make the block safe to call twice for a key, or call the method without a block and use the Hash it returns. `get_multi_with_metadata` yields only after every server has replied, so its block sees each key once.
+
 ## Security Note
 
 By default, Dalli uses Ruby's Marshal for serialization. Deserializing untrusted data with Marshal can lead to remote code execution. If you cache user-controlled data, consider using a safer serializer:
@@ -152,6 +174,14 @@ A failed TLS handshake counts as a failed connection attempt, like a refused con
 
 The app has already run by then, so whatever it did (database writes, emails, and so on) has happened, and only its response is lost. If a request needs a different outcome, check that the session still exists before making the changes.
 
+Set `expire_after` to give sessions an expiry. It sets both the cookie's expiry and the TTL of the session in memcached. Without it, sessions are stored with no expiry: a session stays in memcached until it's deleted (for example by a logout) or evicted, and a stolen session cookie keeps working until then.
+
+```ruby
+use Rack::Session::Dalli, memcache_server: 'mc.example.com:11211', expire_after: 2 * 60 * 60 # two hours
+```
+
+A stored session that can't be decoded, for example after changing the serializer or removing a class that sessions stored, is treated as missing: the request gets a new, empty session and a new cookie, and Dalli logs a warning.
+
 ## OpenTelemetry Tracing
 
 Dalli automatically instruments operations with [OpenTelemetry](https://opentelemetry.io/) when the SDK is present. No configuration is required - just add the OpenTelemetry gems to your application:

data/lib/dalli/client.rb

@@ -77,8 +77,8 @@ module Dalli
     #
     def initialize(servers = nil, options = {})
       @normalized_servers = ::Dalli::ServersArgNormalizer.normalize_servers(servers)
+      warn_removed_options(options)
       @options = normalize_options(options)
-      warn_removed_options(@options)
       @key_manager = ::Dalli::KeyManager.new(@options)
       @ring = nil
     end
@@ -134,6 +134,10 @@ module Dalli
     #   - :return_last_access [Boolean] return seconds since last access
     #   - :return_ttl_remaining [Boolean] return seconds of TTL remaining (-1 if no TTL)
     #   - :skip_lru_bump [Boolean] don't bump LRU or update access stats
+    #   - :vivify_ttl [Integer] on a miss, create a stub that lives this long
+    #     (sanitized like any other TTL) and report won_recache
+    #   - :recache_ttl [Integer] report won_recache when the item's remaining
+    #     TTL is below this
     #
     # @return [Hash] containing:
     #   - :value - the cached value (or nil on miss)
@@ -160,6 +164,8 @@ module Dalli
     #
     def get_with_metadata(key, options = {})
       validate_routing_tokens!(options)
+      validate_ttl!(:vivify_ttl, options[:vivify_ttl])
+      validate_integer!(:recache_ttl, options[:recache_ttl])
       # A retry re-runs this method with its original arguments, so `key` must
       # stay as the caller gave it; reassigning it would add the namespace again.
       validated_key = @key_manager.validate_key(key.to_s)
@@ -200,11 +206,17 @@ module Dalli
     #
     # `req_options` accepts :p_token/:l_token, applied to every key in the batch.
     #
-    # A transient network error is retried automatically. If a server remains
-    # unreachable after retrying, raises Dalli::NetworkError rather than
-    # silently omitting that server's keys from the result.
+    # A transient network error is retried automatically. If a server stops
+    # during the call and is still unreachable after retrying, raises
+    # Dalli::NetworkError only with `failover: false`; with failover (the
+    # default) its keys go to another server, or are left out when no server
+    # is left. See "Network Errors and Retries" in the README.
     #
-    # @raise [Dalli::NetworkError] if a server is unreachable after retrying
+    # A retry starts the batch again, so a block can be yielded a key it was
+    # already yielded before the error.
+    #
+    # @raise [Dalli::NetworkError] if a server stops during the call and is
+    #   still unreachable after retrying, without failover
     # rubocop:disable-next Style/ExplicitBlockArgument
     def get_multi(*keys, req_options: nil)
       keys.flatten!
@@ -242,14 +254,18 @@ module Dalli
     #
     # `req_options` accepts :p_token/:l_token, applied to every key in the batch.
     #
-    # A transient network error is retried automatically. If a server remains
-    # unreachable after retrying, raises Dalli::NetworkError rather than
-    # returning its keys as misses or yielding a partial result.
+    # A transient network error is retried automatically. If a server stops
+    # during the call and is still unreachable after retrying, raises
+    # Dalli::NetworkError only with `failover: false`; with failover (the
+    # default) its keys go to another server, or are left out when no server
+    # is left. See "Network Errors and Retries" in the README. The block is
+    # called only once every server has replied, so never twice for a key.
     #
     # @param keys [Array<String>] the keys to fetch
     # @param req_options [Hash, nil] routing-token options
     # @return [Hash] key => { value:, cas:, stale:, miss: }
-    # @raise [Dalli::NetworkError] if a server is unreachable after retrying
+    # @raise [Dalli::NetworkError] if a server stops during the call and is
+    #   still unreachable after retrying, without failover
     def get_multi_with_metadata(*keys, req_options: nil, &block)
       keys.flatten!
       keys.compact!
@@ -306,6 +322,9 @@ module Dalli
     #   { 'key' => [value, cas_id] }
     #
     # `req_options` accepts :p_token/:l_token, applied to every key in the batch.
+    #
+    # Network errors are handled as in #get_multi, and as there a retry can
+    # yield a key a second time.
     def get_multi_cas(*keys, req_options: nil)
       validate_routing_tokens!(req_options)
 
@@ -328,11 +347,12 @@ module Dalli
     # written to the cache and returned.
     def fetch(key, ttl = nil, req_options = nil)
       req_options = req_options.nil? ? CACHE_NILS : req_options.merge(CACHE_NILS) if cache_nils
+      ttl = ttl_or_default(ttl)
       val = get(key, req_options)
       return val unless block_given? && not_found?(val)
 
       new_val = yield
-      add(key, new_val, ttl_or_default(ttl), req_options)
+      add(key, new_val, ttl, req_options)
       new_val
     end
 
@@ -369,7 +389,10 @@ module Dalli
     def fetch_with_lock(key, ttl: nil, lock_ttl: 30, recache_threshold: nil, req_options: nil, &block)
       raise ArgumentError, 'Block is required for fetch_with_lock' unless block_given?
 
-      validate_integer!(:lock_ttl, lock_ttl)
+      # ttl is only used once the block has run; checked now so a bad one
+      # raises before the lock is taken
+      ttl_or_default(ttl)
+      validate_ttl!(:lock_ttl, lock_ttl)
       validate_integer!(:recache_threshold, recache_threshold)
       validate_routing_tokens!(req_options)
       # Left unchanged for the retry below; see get_with_metadata
@@ -463,8 +486,10 @@ module Dalli
       nil
     end
 
+    # set_cas without its CAS check: the CAS here is always 0
     def set(key, value, ttl = nil, req_options = nil)
-      set_cas(key, value, 0, ttl, req_options)
+      validate_routing_tokens!(req_options)
+      perform(:set, key, value, ttl_or_default(ttl), 0, req_options)
     end
 
     ##
@@ -472,16 +497,20 @@ module Dalli
     # This method is more efficient than calling set() in a loop because
     # it batches requests by server and uses quiet mode.
     #
-    # A transient network error is retried automatically. If a server remains
-    # unreachable after retrying, raises Dalli::NetworkError; keys already
-    # sent to other servers before the error are not rolled back.
+    # A transient network error is retried automatically. If a server stops
+    # during the call and is still unreachable after retrying, raises
+    # Dalli::NetworkError only with `failover: false`; with failover (the
+    # default) its keys go to another server, or are left out when no server
+    # is left. See "Network Errors and Retries" in the README. Keys already
+    # written to other servers before an error are not rolled back.
     #
     # @param hash [Hash] key-value pairs to set
     # @param ttl [Integer] time-to-live in seconds (optional, uses default if not provided)
     # @param req_options [Hash] options passed to each set operation; accepts
     #   :p_token/:l_token, applied to every key in the batch
     # @return [void]
-    # @raise [Dalli::NetworkError] if a server is unreachable after retrying
+    # @raise [Dalli::NetworkError] if a server stops during the call and is
+    #   still unreachable after retrying, without failover
     #
     # Example:
     #   client.set_multi({ 'key1' => 'value1', 'key2' => 'value2' }, 300)
@@ -502,7 +531,11 @@ module Dalli
     ##
     # Set the key-value pair, verifying existing CAS.
     # Returns the resulting CAS value if succeeded, and falsy otherwise.
+    # The CAS must be an Integer (or a String of decimal digits) between 0 and
+    # 2**64 - 1, or nil; anything else raises ArgumentError. A CAS of 0 or nil
+    # means an unconditional set.
     def set_cas(key, value, cas, ttl = nil, req_options = nil)
+      validate_uint64!(:cas, cas)
       validate_routing_tokens!(req_options)
       perform(:set, key, value, ttl_or_default(ttl), cas, req_options)
     end
@@ -525,18 +558,22 @@ module Dalli
     ##
     # Conditionally add a key/value pair, verifying existing CAS, only if the
     # key already exists on the server.  Returns the new CAS value if the
-    # operation succeeded, or falsy otherwise.
+    # operation succeeded, or falsy otherwise.  The CAS is validated as in
+    # #set_cas.
     def replace_cas(key, value, cas, ttl = nil, req_options = nil)
+      validate_uint64!(:cas, cas)
       validate_routing_tokens!(req_options)
       perform(:replace, key, value, ttl_or_default(ttl), cas, req_options)
     end
 
     # Delete a key/value pair, verifying existing CAS.
     # Returns true if succeeded, and falsy otherwise.
-    # Delete a key, optionally with a CAS check.
+    # Delete a key, optionally with a CAS check.  The CAS is validated as in
+    # #set_cas.
     #
     # `req_options` accepts the same meta-delete options as #delete.
     def delete_cas(key, cas = 0, req_options = nil)
+      validate_uint64!(:cas, cas)
       validate_delete_options!(req_options)
       validate_routing_tokens!(req_options)
       perform(:delete, key, cas, req_options)
@@ -588,9 +625,10 @@ module Dalli
     #   is whichever one the caller asked for. This is best-effort: a transient
     #   network error is retried automatically, and keys handled before the
     #   error are not recounted, so the result may under-report when a retry
-    #   occurs. If a server remains unreachable after retrying, raises
-    #   Dalli::NetworkError.
-    # @raise [Dalli::NetworkError] if a server is unreachable after retrying
+    #   occurs. A server that stops during the call is handled as in
+    #   #set_multi.
+    # @raise [Dalli::NetworkError] if a server stops during the call and is
+    #   still unreachable after retrying, without failover
     #
     # Example:
     #   client.delete_multi(['key1', 'key2', 'key3'])
@@ -628,9 +666,8 @@ module Dalli
 
     ##
     # Incr adds the given amount to the counter on the memcached server.
-    # Amt must be a positive integer value.
-    # Default, if given, must be an Integer (or a String of decimal digits);
-    # anything else raises ArgumentError.
+    # Amt, and default if given, must be Integers (or Strings of decimal
+    # digits) between 0 and 2**64 - 1; anything else raises ArgumentError.
     #
     # If default is nil, the counter must already exist or the operation
     # will fail and will return nil.  Otherwise this method will return
@@ -642,18 +679,17 @@ module Dalli
     #
     # If the value already exists, it must have been set with raw: true
     def incr(key, amt = 1, ttl = nil, default = nil, req_options = nil)
-      check_positive!(amt)
-      validate_integer!(:default, default)
+      amt = validate_uint64!(:amt, amt)
+      default = validate_uint64!(:default, default)
       validate_routing_tokens!(req_options)
 
-      perform(:incr, key, amt.to_i, ttl_or_default(ttl), default, req_options)
+      perform(:incr, key, amt, ttl_or_default(ttl), default, req_options)
     end
 
     ##
     # Decr subtracts the given amount from the counter on the memcached server.
-    # Amt must be a positive integer value.
-    # Default, if given, must be an Integer (or a String of decimal digits);
-    # anything else raises ArgumentError.
+    # Amt, and default if given, must be Integers (or Strings of decimal
+    # digits) between 0 and 2**64 - 1; anything else raises ArgumentError.
     #
     # memcached counters are unsigned and cannot hold negative values.  Calling
     # decr on a counter which is 0 will just return 0.
@@ -668,20 +704,23 @@ module Dalli
     #
     # If the value already exists, it must have been set with raw: true
     def decr(key, amt = 1, ttl = nil, default = nil, req_options = nil)
-      check_positive!(amt)
-      validate_integer!(:default, default)
+      amt = validate_uint64!(:amt, amt)
+      default = validate_uint64!(:default, default)
       validate_routing_tokens!(req_options)
 
-      perform(:decr, key, amt.to_i, ttl_or_default(ttl), default, req_options)
+      perform(:decr, key, amt, ttl_or_default(ttl), default, req_options)
     end
 
     ##
     # Flush the memcached server, at 'delay' seconds in the future.
     # Delay defaults to zero seconds, which means an immediate flush.
+    # Delay must be a non-negative Integer (or a String of decimal digits);
+    # anything else raises ArgumentError.
     # Every server is attempted. If any of them fails, the first error is
     # raised once all the others have been flushed.
     ##
     def flush(delay = 0)
+      validate_uint64!(:delay, delay)
       first_error = nil
       results = ring.servers.map do |s|
         s.request(:flush, delay)
@@ -774,15 +813,7 @@ module Dalli
 
       raise ArgumentError, 'tombstone_ttl requires invalidate: true' unless req_options[:invalidate]
 
-      # tombstone_kwargs coerces this with Integer(), deep inside the request
-      # path; validated here first so a bad value raises cleanly instead of
-      # unwinding through Protocol::Base#request, which would close the
-      # connection on the ArgumentError Integer() raises.
-      begin
-        Integer(tombstone_ttl)
-      rescue ArgumentError, TypeError
-        raise ArgumentError, "tombstone_ttl must be an integer, got #{tombstone_ttl.inspect}"
-      end
+      validate_ttl!(:tombstone_ttl, tombstone_ttl)
     end
 
     def record_hit_miss_metrics(span, key_count, hit_count)
@@ -818,9 +849,20 @@ module Dalli
       end
     end
 
-    def single_server
+    # retried is true once a request to the server has failed with a
+    # RetryableNetworkError. Without failover, finding the server down then
+    # means that request failed, which raises, rather than the server being
+    # down before the operation began, which returns nil. With failover it
+    # returns nil either way, as the multi-server path leaves keys out when
+    # no server is left.
+    def single_server(retried: false)
       server = ring.servers.first
-      server if server&.alive?
+      return server if server&.alive?
+      if retried && !ring.failover?
+        raise Dalli::NetworkError, "#{server&.name} failed during this request and is still unreachable"
+      end
+
+      nil
     end
 
     # The three single_server_* fast-path methods below share one contract,
@@ -829,20 +871,22 @@ module Dalli
     # socket_max_failures, same as the pipelined path's retry), and a hard
     # NetworkError -- the server genuinely unreachable, not just blipping --
     # propagates to the caller rather than being swallowed into a silently
-    # wrong result. Only server_for_key/single_server finding no live server
-    # to route to at all is still silent, matching Ring#keys_grouped_by_server
-    # dropping a key it can't route on both the single- and multi-server paths.
+    # wrong result, including (without failover) the server going down before
+    # the retry, which single_server raises for. Otherwise finding no live
+    # server is still silent, matching Ring#keys_grouped_by_server dropping a
+    # key it can't route on both the single- and multi-server paths. Each
+    # method sets retried before it retries; a local keeps its value across
+    # a retry.
     def single_server_get_multi(keys, req_options = nil)
-      # A new array, so the retry below starts again from the caller's keys
-      validated_keys = keys.map { |k| @key_manager.validate_key(k.to_s) }
-      return {} unless (server = single_server)
+      retried ||= false
+      caller_keys = @key_manager.caller_keys_by_wire_key(keys)
+      return {} unless (server = single_server(retried: retried))
 
-      result = server.request(:read_multi_req, validated_keys, req_options)
-      result.transform_keys! { |k| @key_manager.key_without_namespace(k) }
-      result
+      results_for_caller_keys(server.request(:read_multi_req, caller_keys.keys, req_options), caller_keys)
     rescue Dalli::RetryableNetworkError => e
       Dalli.logger.debug { e.inspect }
       Dalli.logger.debug { 'retrying single-server get_multi because of network error' }
+      retried = true
       retry
     rescue Dalli::UnexpectedReplyKeyError => e
       # The connection is closed and the keys are treated as missing, as the
@@ -852,40 +896,54 @@ module Dalli
     end
 
     def single_server_get_multi_with_metadata(keys, req_options = nil)
-      validated_keys = keys.map { |k| @key_manager.validate_key(k.to_s) }
-      return {} unless (server = single_server)
+      retried ||= false
+      caller_keys = @key_manager.caller_keys_by_wire_key(keys)
+      return {} unless (server = single_server(retried: retried))
 
-      result = server.request(:read_multi_with_metadata_req, validated_keys, req_options)
-      result.transform_keys! { |k| @key_manager.key_without_namespace(k) }
-      result
+      results_for_caller_keys(server.request(:read_multi_with_metadata_req, caller_keys.keys, req_options),
+                              caller_keys)
     rescue Dalli::RetryableNetworkError => e
       Dalli.logger.debug { e.inspect }
       Dalli.logger.debug { 'retrying single-server get_multi_with_metadata because of network error' }
+      retried = true
       retry
     rescue Dalli::UnexpectedReplyKeyError => e
       Dalli.logger.warn { e.message }
       {}
     end
 
+    # Replies name the key as sent; returns them under the caller's keys,
+    # skipping any key that wasn't requested
+    def results_for_caller_keys(results, caller_keys)
+      results.each_with_object({}) do |(key, value), hash|
+        caller_key = @key_manager.caller_key(caller_keys, key)
+        hash[caller_key] = value if caller_key
+      end
+    end
+
     def single_server_set_multi(hash, ttl, req_options)
+      retried ||= false
       pairs = hash.transform_keys { |k| @key_manager.validate_key(k.to_s) }
-      return unless (server = single_server)
+      return unless (server = single_server(retried: retried))
 
       server.request(:write_multi_req, pairs, ttl, req_options)
     rescue Dalli::RetryableNetworkError => e
       Dalli.logger.debug { e.inspect }
       Dalli.logger.debug { 'retrying single-server set_multi because of network error' }
+      retried = true
       retry
     end
 
     def single_server_delete_multi(keys, req_options = nil)
+      retried ||= false
       validated_keys = keys.map { |k| @key_manager.validate_key(k.to_s) }
-      return 0 unless (server = single_server)
+      return 0 unless (server = single_server(retried: retried))
 
       server.request(:delete_multi_req, validated_keys, req_options)
     rescue Dalli::RetryableNetworkError => e
       Dalli.logger.debug { e.inspect }
       Dalli.logger.debug { 'retrying single-server delete_multi because of network error' }
+      retried = true
       retry
     end
 
@@ -917,10 +975,6 @@ module Dalli
       attrs
     end
 
-    def check_positive!(amt)
-      raise ArgumentError, "Positive values only: #{amt}" if amt.negative?
-    end
-
     # Numeric arguments that become meta protocol flags (GHSA-6wmv-xq9m-fmp7).
     # RequestFormatter converts them to Integer as the wire-level backstop;
     # checking here too gives the caller a clean ArgumentError for the same
@@ -933,6 +987,27 @@ module Dalli
       raise ArgumentError, "#{name} must be an Integer, got #{value.inspect}"
     end
 
+    MAX_UINT64 = (2**64) - 1
+    private_constant :MAX_UINT64
+
+    # CAS values, incr/decr amounts and initial values, and the flush delay
+    # are read by memcached as unsigned 64-bit integers. RequestFormatter
+    # raises on anything else as the wire-level backstop; checking here first
+    # keeps the connection open, for the reason given for routing tokens
+    # below. Accepts an Integer or a String of decimal digits, and returns it
+    # as an Integer.
+    def validate_uint64!(name, value)
+      return if value.nil?
+
+      int = case value
+            when Integer then value
+            when String then Integer(value, 10, exception: false)
+            end
+      return int if int && int >= 0 && int <= MAX_UINT64
+
+      raise ArgumentError, "#{name} must be an Integer between 0 and #{MAX_UINT64}, got #{value.inspect}"
+    end
+
     # Validated here, before the request reaches Protocol::Base#request, rather
     # than only at the RequestFormatter level. Reaching only the formatter's
     # check means unwinding through Protocol::Base#request, which logs the
@@ -961,15 +1036,16 @@ module Dalli
 
     def cas_core(key, always_set, ttl = nil, req_options = nil)
       validate_routing_tokens!(req_options)
+      ttl = ttl_or_default(ttl)
       (value, cas) = perform(:cas, key, req_options)
       return if !always_set && cas_miss?(value, cas)
 
       newvalue = yield(value)
       # A CAS of 0 means the key was missing. A set with CAS 0 is unconditional,
       # so add instead: it fails if another client created the key meanwhile.
-      return perform(:add, key, newvalue, ttl_or_default(ttl), req_options) if cas.zero?
+      return perform(:add, key, newvalue, ttl, req_options) if cas.zero?
 
-      perform(:set, key, newvalue, ttl_or_default(ttl), cas, req_options)
+      perform(:set, key, newvalue, ttl, cas, req_options)
     end
 
     # The CAS lookup returns [nil, 0] on a miss rather than NOT_FOUND, and a
@@ -998,12 +1074,47 @@ module Dalli
 
     ##
     # Uses the argument TTL or the client-wide default.  Ensures
-    # that the value is an integer
+    # that the value is an integer memcached accepts as an expiration
     ##
     def ttl_or_default(ttl)
-      (ttl || @options[:expires_in]).to_i
-    rescue NoMethodError
-      raise ArgumentError, "Cannot convert ttl (#{ttl}) to an integer"
+      ttl ||= @options[:expires_in] || 0
+      # validate_ttl!'s fast path, without the call
+      return ttl if ttl.is_a?(Integer) && ttl >= Protocol::TtlSanitizer::MIN_EXPIRATION_TIME &&
+                    ttl <= Protocol::TtlSanitizer::MAX_ACCEPTABLE_EXPIRATION_INTERVAL
+
+      validate_ttl!(:ttl, ttl)
+    end
+
+    # Returns the TTL as an Integer, sanitized (see TtlSanitizer), raising
+    # ArgumentError unless memcached can use it. An interval over 30 days comes
+    # back as the Unix time it becomes, so the request path doesn't convert it
+    # again. A String must be a whole number in decimal, as for the other
+    # numeric arguments; anything else is converted with to_i, so a Float or
+    # an ActiveSupport::Duration still works. Once sanitized it must fit memcached's signed 32-bit expiration time. A
+    # negative TTL is allowed: memcached stores the item as already expired.
+    # memcached rejects a TTL outside that with CLIENT_ERROR, which a quiet
+    # block or set_multi silently discards, so it is checked here, before the
+    # request is written.
+    def validate_ttl!(name, value)
+      return if value.nil?
+      # The usual case: an Integer TTL that TtlSanitizer leaves unchanged
+      return value if value.is_a?(Integer) && value >= Protocol::TtlSanitizer::MIN_EXPIRATION_TIME &&
+                      value <= Protocol::TtlSanitizer::MAX_ACCEPTABLE_EXPIRATION_INTERVAL
+
+      ttl = ttl_to_i(name, value)
+      min = Protocol::TtlSanitizer::MIN_EXPIRATION_TIME
+      max = Protocol::TtlSanitizer::MAX_EXPIRATION_TIME
+      ttl = Protocol::TtlSanitizer.sanitize(ttl)
+      return ttl if ttl.between?(min, max)
+
+      raise ArgumentError, "#{name} must be a number of seconds no smaller than #{min} or a Unix time " \
+                           "no later than #{max}, got #{value.inspect}"
+    end
+
+    def ttl_to_i(name, value)
+      value.is_a?(String) ? Integer(value, 10) : value.to_i
+    rescue ArgumentError, NoMethodError, FloatDomainError
+      raise ArgumentError, "#{name} must be an Integer, got #{value.inspect}"
     end
 
     def ring
@@ -1018,7 +1129,10 @@ module Dalli
     #
     # This method also forces retries on network errors - when
     # a particular memcached instance becomes unreachable, or the
-    # operation times out.
+    # operation times out. Once some of the request has been sent, only
+    # one that is safe to apply twice is retried; for the others
+    # (incr, decr, append, prepend, add, replace and CAS writes)
+    # Protocol::Base#request raises a NetworkError, which isn't retried.
     ##
     # rubocop:disable Naming/MethodParameterName
     def perform(op, key, ...)
@@ -1047,13 +1161,12 @@ module Dalli
     end
 
     # Works on a copy so the caller's hash is neither modified here nor able
-    # to change this client's configuration later.
+    # to change this client's configuration later. Credentials are dropped:
+    # Dalli 5 doesn't use them, and kept they could show up in #inspect.
     def normalize_options(options)
-      opts = options.dup
-      opts[:expires_in] = opts[:expires_in].to_i if opts[:expires_in]
+      opts = options.except(:username, :password)
+      opts[:expires_in] = ttl_to_i(:expires_in, opts[:expires_in]) if opts[:expires_in]
       opts
-    rescue NoMethodError
-      raise ArgumentError, "cannot convert :expires_in => #{opts[:expires_in].inspect} to an integer"
     end
 
     REMOVED_OPTIONS = {