dalli 5.0.6 → 5.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 54f211dbcca14abc18e7e9ffc7e9fe5c992c1e95afab2b6a64e46b6ec32efa8a
-  data.tar.gz: f655e7c63cc622876ae47400afd59139eea3b8e19f6585af223fafccad94e659
+  metadata.gz: e11a3903c6fa786ac8f175bbb95130760f3e5886dc24fdc0825abc84d743006c
+  data.tar.gz: b9b376166e61c2899f23ba4bd5367360cc14c9e10a5d21fdb3ac34558c4b7303
 SHA512:
-  metadata.gz: de1470d516e61e2b970c0cfaa0de3afe572124081b300ce643405813c3ba232be08da32fbc3748fbc86b39e2aaea0d6e5803d34cd9722f50c3fc47b325970c67
-  data.tar.gz: 1835dc76b3cfebe33a4a8eca38d37dcfa8af84495e0d5af523402ddd4ffa1e3a2a8c808ce4c6e2605325f8d6eded6faa4a574022680528016f091e330e987a3e
+  metadata.gz: bfc8526fce280bcb5003b81128b9e005975ffff54c1a7c82ce9d7b1104954c4dc169507108df70e4c5d90f444c01001640af9fccc038d180954e81f7a7459a55
+  data.tar.gz: 3c083001313d3dbdbc91cb122a76f8bd0af1fd5de75cddb89d2985ec67b40c313124ba56447818c6000d05633c80c4fae34cbaa560515a9947ca4a0fbe542134

data/CHANGELOG.md

@@ -4,6 +4,63 @@ Dalli Changelog
 Unreleased
 ==========
 
+5.1.0
+==========
+
+Features:
+
+- Add opaque routing tokens: `:p_token` and `:l_token` request options (#1147, #1154)
+  - `get`, `gat`, `get_cas`, `get_with_metadata`, `fetch_with_lock`, `set`/`add`/`replace`/`set_cas`/`replace_cas`, `append`/`prepend`, `incr`/`decr`, `cas`/`cas!`, `delete`/`delete_cas`, and the bulk operations (`get_multi`, `get_multi_cas`, `get_multi_with_metadata`, `set_multi`, `delete_multi`) all accept per-request `:p_token`/`:l_token` options, appended to the wire protocol as `P<token>`/`L<token>` -- applied to every key on the bulk methods
+  - memcached itself ignores these tokens; per the meta protocol spec they exist as hints for a proxy or router sitting between the client and memcached
+  - CRLF and NUL bytes raise `ArgumentError` before the request reaches the socket, both in `Dalli::Client` and in `RequestFormatter`, so a bad token can't be used for wire-protocol injection and can't close the connection out from under the caller the way a formatter-only check would
+  - The bulk methods and `delete`/`delete_cas` were deferred out of #1147 to avoid racing other in-flight PRs touching the same method signatures; #1154 completes them, including both the single-server fast path and the multi-server pipelined path for each
+  - Extracted from #1130; thanks to Nick Herson for the original idea and Jianbin Chen for porting it forward
+
+- Support tombstone (mark-stale) deletes on `delete`, `delete_cas`, and `delete_multi` (#1145, #1153)
+  - `:invalidate` marks the item stale instead of removing it, so `#get_with_metadata` / `#get_multi_with_metadata` report `stale: true` and a reader can tell "another process is repopulating this" apart from "this was never here" -- a tombstoned key is not a miss
+  - `:tombstone_ttl` controls how long the stale marker lives; requires `:invalidate`, since memcached only honors the TTL on a delete when it accompanies the invalidate flag
+  - `:drop_value` removes the item's value but leaves the item; on its own it is not a tombstone -- reads are an ordinary hit with an empty value
+  - `delete_multi` applies the same options to every key in the batch, on both the single-server and pipelined paths; its return value keeps counting keys the server found and acted on, so under `:invalidate` it reports how many keys were tombstoned rather than removed
+  - Extracted from #1130; thanks to Jianbin Chen for this contribution
+
+- Add `:miss` and `:return_ttl_remaining` to `get_with_metadata` (#1143)
+  - `:miss` is now always present in the returned Hash, distinguishing a true miss from a stored `nil` under `cache_nils` or a tombstoned, stale hit -- neither of which a `nil` `:value` alone can tell apart
+  - `:return_ttl_remaining` exposes the meta protocol's `t` flag as `:ttl_remaining` (seconds remaining, or `-1` for an item with no expiry), following the same opt-in shape as `:return_hit_status` / `:return_last_access`
+  - Extracted from #1130; thanks to Jianbin Chen for this contribution
+
+- Add `get_multi_with_metadata` for stale-aware bulk reads (#1144)
+  - Returns `{ key => { value:, cas:, stale:, miss: } }` for the keys that were found; genuine misses are omitted, matching `get_multi` / `get_multi_cas` -- a tombstoned item is a hit at the protocol level, so it is still returned, with `stale: true`
+  - Routes to the same single-server fast path / pipelined-getter split as `get_multi`
+  - Extracted from #1130; thanks to Jianbin Chen for this contribution
+
+Other changes:
+
+- Raise the documented minimum supported memcached version to 1.6.27 (#1140)
+  - Groundwork for the features above: `drop_value` tombstone deletes require 1.6.27, the highest floor of anything landing from #1130
+  - Not enforced at runtime -- `MIN_SUPPORTED_MEMCACHED_VERSION` only gates the test harness and the README's support statement, so this changes no running client's behavior
+
+Bug Fixes:
+
+- Retry a transient network error during a liveness check instead of treating it as terminal (#1150)
+  - `Dalli::Protocol::Base#alive?` caught any `NetworkError` -- including `RetryableNetworkError`, a subclass -- and unconditionally converted it to `false`, with no retry. This didn't match the retry-then-raise contract every other network-facing path in Dalli follows: a single transient connection hiccup during the liveness check itself (as opposed to an actual request) permanently reported a healthy server as down for that check
+  - Now retries once on `RetryableNetworkError`, letting `error_on_request!`'s own fail-count threshold decide when to actually give up (the same mechanism the rest of the codebase relies on): it keeps retrying until `socket_max_failures` is reached, then raises a terminal `NetworkError`, which is still converted to `false` as before
+  - **Behavior change:** a server that was previously marked "down" (engaging the `down_retry_delay` cooldown) only via an actual request could now also reach that state via a liveness check (`alive?`, and anything that calls it -- `Dalli::Client#stats`, `#reset_stats`, and `Ring`'s own server selection) exhausting its retries. Previously, a solitary transient failure during a liveness check was silently forgotten rather than tracked, so the cooldown was inconsistently applied depending on which code path first observed the failure
+  - Likely a contributing cause of the same intermittently failing `get_multi` failover integration test noted in #1149: that fix addressed the send/receive phase, but the liveness-check retry it introduced can itself force a fresh `connect()` mid-retry, giving this separate, pre-existing gap in `alive?` more chances to fire
+  - Also fixed a test (`test_ring.rb`, "detect when a dead server is up again") that had been unknowingly relying on the old behavior: it never engaged the `down_retry_delay` cooldown from a single transient failure, so its 0.5s delay never actually gated anything. Updated to use a 0s delay, since the test's intent is to verify reconnection is detected, not to test cooldown timing
+
+- Base64-encode keys containing control characters, not just NUL (#1148)
+  - `KeyRegularizer.required?` decided whether a key needed base64 encoding using `/\s/`, which matches most whitespace but none of the C0 control range (0x00-0x1F) or DEL (0x7F) -- a key that was otherwise ASCII-only and contained no whitespace (e.g. `"foo\x00bar"` or a key with an embedded ESC byte) went out on the wire unencoded
+  - Not a command-injection risk: the text protocol splits commands on CRLF, not other control bytes. The risk is key confusion -- anything downstream that treats one of these bytes specially (a C string terminating at NUL, a terminal or log line interpreting an escape byte) could silently act on a different key than Dalli believes it sent
+  - A raw control byte in a key was never protocol-compliant in the first place: memcached's own spec (`protocol.txt`) states a key "must not include control characters or whitespace." The only sanctioned way to carry such content in a key is the meta protocol's base64 (`b` flag) path -- the one whitespace and non-ASCII keys already used, and the one these keys now use too. The check is now `/[\p{Cntrl}\s]/`, matching that rule directly rather than special-casing NUL
+  - **Behavior change:** a key containing a control character now produces different bytes on the wire (base64-encoded, per the meta protocol's `b` flag) than before. Existing cache entries stored under the old, unencoded form of such a key will read as a miss once every reader has upgraded. **During a rolling deploy, old and new Dalli versions disagree about which physical key such a logical key maps to** -- not just a one-time cutover, but ongoing inconsistency between the old-version and new-version server pools for the duration of the rollout. Harmless for an ordinary cached value (worst case, extra cache misses); worth accounting for if such a key ever backs something stateful, like a lock or counter. Expected to be rare in practice: embedding a raw control byte in a cache key is unusual, and doing so was already outside what the protocol permits
+  - Found while auditing `request_formatter.rb` during the routing-token work in #1130 / #1147; unrelated to that change and predates it
+
+- Retry transient network errors in `get_multi`, `set_multi` and `delete_multi` instead of silently swallowing them (#1149)
+  - All three methods group keys by server and issue one request per server. Each per-server rescue clause caught `DalliError` and `NetworkError` together and swallowed both, just debug-logging -- since `RetryableNetworkError < NetworkError`, this also silently swallowed transient, retryable failures, dropping that server's keys from the result instead of the whole operation retrying (`get_multi`/`set_multi`'s single-server fast path did not even attempt a retry, on any failure)
+  - Six rescue sites across `PipelinedGetter`, `PipelinedSetter`, `PipelinedDeleter` and the `single_server_*` fast paths now retry a transient `RetryableNetworkError`, matching sibling rescue sites in the same files that already did this correctly
+  - **Behavior change:** if a server remains unreachable after retrying (not just a transient blip), these three methods now raise `Dalli::NetworkError` instead of silently returning an incomplete or empty result. This matches how every other Dalli::Client method already behaves on a hard network failure, and is the retry-then-raise behavior `delete_multi`'s own docs already described; it was just not reliably true for this failure path before. Code that calls these methods without rescuing `Dalli::NetworkError` should account for this if it can reach a fully unreachable server
+  - Likely the cause of an intermittently failing `get_multi` failover integration test that recurred across multiple PRs, though this could not be directly confirmed: the failure (an empty result with no exception in the logs) never reproduced locally despite repeated attempts, consistent with needing a genuine transient network hiccup that is far more likely on a loaded CI runner than an idle dev machine
+
 5.0.6
 ==========
 

data/README.md

@@ -17,7 +17,12 @@ The name is a variant of Salvador Dali for his famous painting [The Persistence
 ## Requirements
 
 * Ruby 3.3 or later (JRuby also supported)
-* memcached 1.6 or later
+* memcached 1.6.27 or later
+
+Dalli is tested against both the minimum supported memcached version and the
+latest release. Earlier 1.6.x servers are not supported: the meta protocol
+rejects unknown flags outright, so features added after a server's release fail
+with `CLIENT_ERROR invalid flag` rather than degrading.
 
 ## Configuration Options
 

data/lib/dalli/client.rb

@@ -68,6 +68,7 @@ module Dalli
     # Get the value associated with the key.
     # If a value is not found, then +nil+ is returned.
     def get(key, req_options = nil)
+      validate_routing_tokens!(req_options)
       perform(:get, key, req_options)
     end
 
@@ -75,8 +76,9 @@ module Dalli
     # Gat (get and touch) fetch an item and simultaneously update its expiration time.
     #
     # If a value is not found, then +nil+ is returned.
-    def gat(key, ttl = nil)
-      perform(:gat, key, ttl_or_default(ttl))
+    def gat(key, ttl = nil, req_options = nil)
+      validate_routing_tokens!(req_options)
+      perform(:gat, key, ttl_or_default(ttl), req_options)
     end
 
     ##
@@ -91,8 +93,9 @@ module Dalli
     ##
     # Get the value and CAS ID associated with the key.  If a block is provided,
     # value and CAS will be passed to the block.
-    def get_cas(key)
-      (value, cas) = perform(:cas, key)
+    def get_cas(key, req_options = nil)
+      validate_routing_tokens!(req_options)
+      (value, cas) = perform(:cas, key, req_options)
       return [value, cas] unless block_given?
 
       yield value, cas
@@ -106,17 +109,23 @@ module Dalli
     #   - :return_cas [Boolean] return the CAS value (default: true)
     #   - :return_hit_status [Boolean] return whether item was previously accessed
     #   - :return_last_access [Boolean] return seconds since last access
+    #   - :return_ttl_remaining [Boolean] return seconds of TTL remaining (-1 if no TTL)
     #   - :skip_lru_bump [Boolean] don't bump LRU or update access stats
     #
     # @return [Hash] containing:
     #   - :value - the cached value (or nil on miss)
     #   - :cas - the CAS value
+    #   - :miss - true when the key does not exist.  Always present.  Prefer it
+    #     over a nil :value, which cannot distinguish a miss from a stored nil
+    #     under cache_nils
     #   - :hit_before - true/false if previously accessed (only if return_hit_status: true)
     #   - :last_access - seconds since last access (only if return_last_access: true)
+    #   - :ttl_remaining - seconds of TTL remaining, -1 when the item has no
+    #     expiry (only if return_ttl_remaining: true)
     #
     # @example Get with hit status
     #   result = client.get_with_metadata('key', return_hit_status: true)
-    #   # => { value: "data", cas: 123, hit_before: true }
+    #   # => { value: "data", cas: 123, miss: false, hit_before: true }
     #
     # @example Get with all metadata without affecting LRU
     #   result = client.get_with_metadata('key',
@@ -127,6 +136,7 @@ module Dalli
     #   # => { value: "data", cas: 123, hit_before: true, last_access: 42 }
     #
     def get_with_metadata(key, options = {})
+      validate_routing_tokens!(options)
       key = key.to_s
       key = @key_manager.validate_key(key)
 
@@ -144,32 +154,98 @@ module Dalli
     # Fetch multiple keys efficiently.
     # If a block is given, yields key/value pairs one at a time.
     # Otherwise returns a hash of { 'key' => 'value', 'key2' => 'value1' }
+    #
+    # `req_options` accepts :p_token/:l_token, applied to every key in the batch.
+    #
+    # A transient network error is retried automatically. If a server remains
+    # unreachable after retrying, raises Dalli::NetworkError rather than
+    # silently omitting that server's keys from the result.
+    #
+    # @raise [Dalli::NetworkError] if a server is unreachable after retrying
     # rubocop:disable Style/ExplicitBlockArgument
-    def get_multi(*keys)
+    def get_multi(*keys, req_options: nil)
       keys.flatten!
       keys.compact!
       return {} if keys.empty?
 
+      validate_routing_tokens!(req_options)
+
       if block_given?
-        get_multi_yielding(keys) { |k, v| yield k, v }
+        get_multi_yielding(keys, req_options) { |k, v| yield k, v }
       else
-        get_multi_hash(keys)
+        get_multi_hash(keys, req_options)
       end
     end
     # rubocop:enable Style/ExplicitBlockArgument
 
+    ##
+    # Fetch multiple keys efficiently, returning a stale-aware metadata Hash per
+    # key.  If a block is given, yields key/metadata pairs one at a time.
+    #
+    # Like #get_multi and #get_multi_cas, keys that were not found are omitted
+    # from the result -- absence is the miss.  A tombstoned item (see #delete
+    # with the meta protocol's invalidate flag) is *not* a miss: it is returned
+    # with stale: true, possibly with an empty value, which is the distinction
+    # stale-aware callers need.
+    #
+    #   client.get_multi_with_metadata('a', 'b', 'absent')
+    #   # => { 'a' => { value: 'v', cas: 12, stale: false, miss: false },
+    #   #      'b' => { value: '',  cas: 13, stale: true,  miss: false } }
+    #
+    # Missing keys are `requested - result.keys`.
+    #
+    # Result key order matches request order only when every key lands on the
+    # same server; across multiple servers it follows per-server response
+    # order instead, the same as #get_multi.
+    #
+    # `req_options` accepts :p_token/:l_token, applied to every key in the batch.
+    #
+    # @param keys [Array<String>] the keys to fetch
+    # @param req_options [Hash, nil] routing-token options
+    # @return [Hash] key => { value:, cas:, stale:, miss: }
+    def get_multi_with_metadata(*keys, req_options: nil, &block)
+      keys.flatten!
+      keys.compact!
+      return {} if keys.empty?
+
+      validate_routing_tokens!(req_options)
+
+      results = Instrumentation.trace('get_multi_with_metadata',
+                                      multi_trace_attrs('get_multi_with_metadata', keys.size, keys)) do
+        if ring.servers.size == 1
+          single_server_get_multi_with_metadata(keys, req_options)
+        else
+          pipelined_getter.process_with_metadata(keys, req_options)
+        end
+      end
+
+      if block
+        results.each(&block)
+        # Matches get_multi/get_multi_cas: nil when a block is given, so
+        # callers can't come to depend on a return value that block-form
+        # get_multi never provided.
+        return nil
+      end
+
+      results
+    end
+
     ##
     # Fetch multiple keys efficiently, including available metadata such as CAS.
     # If a block is given, yields key/data pairs one a time.  Data is an array:
     # [value, cas_id]
     # If no block is given, returns a hash of
     #   { 'key' => [value, cas_id] }
-    def get_multi_cas(*keys)
+    #
+    # `req_options` accepts :p_token/:l_token, applied to every key in the batch.
+    def get_multi_cas(*keys, req_options: nil)
+      validate_routing_tokens!(req_options)
+
       if block_given?
-        pipelined_getter.process(keys) { |*args| yield(*args) }
+        pipelined_getter.process(keys, req_options) { |*args| yield(*args) }
       else
         {}.tap do |hash|
-          pipelined_getter.process(keys) { |k, data| hash[k] = data }
+          pipelined_getter.process(keys, req_options) { |k, data| hash[k] = data }
         end
       end
     end
@@ -225,6 +301,7 @@ module Dalli
     def fetch_with_lock(key, ttl: nil, lock_ttl: 30, recache_threshold: nil, req_options: nil, &block)
       raise ArgumentError, 'Block is required for fetch_with_lock' unless block_given?
 
+      validate_routing_tokens!(req_options)
       key = key.to_s
       key = @key_manager.validate_key(key)
 
@@ -300,16 +377,24 @@ module Dalli
     # This method is more efficient than calling set() in a loop because
     # it batches requests by server and uses quiet mode.
     #
+    # A transient network error is retried automatically. If a server remains
+    # unreachable after retrying, raises Dalli::NetworkError; keys already
+    # sent to other servers before the error are not rolled back.
+    #
     # @param hash [Hash] key-value pairs to set
     # @param ttl [Integer] time-to-live in seconds (optional, uses default if not provided)
-    # @param req_options [Hash] options passed to each set operation
+    # @param req_options [Hash] options passed to each set operation; accepts
+    #   :p_token/:l_token, applied to every key in the batch
     # @return [void]
+    # @raise [Dalli::NetworkError] if a server is unreachable after retrying
     #
     # Example:
     #   client.set_multi({ 'key1' => 'value1', 'key2' => 'value2' }, 300)
     def set_multi(hash, ttl = nil, req_options = nil)
       return if hash.empty?
 
+      validate_routing_tokens!(req_options)
+
       Instrumentation.trace('set_multi', multi_trace_attrs('set_multi', hash.size, hash.keys)) do
         if ring.servers.size == 1
           single_server_set_multi(hash, ttl_or_default(ttl), req_options)
@@ -323,6 +408,7 @@ module Dalli
     # Set the key-value pair, verifying existing CAS.
     # Returns the resulting CAS value if succeeded, and falsy otherwise.
     def set_cas(key, value, cas, ttl = nil, req_options = nil)
+      validate_routing_tokens!(req_options)
       perform(:set, key, value, ttl_or_default(ttl), cas, req_options)
     end
 
@@ -330,6 +416,7 @@ module Dalli
     # Conditionally add a key/value pair, if the key does not already exist
     # on the server.  Returns truthy if the operation succeeded.
     def add(key, value, ttl = nil, req_options = nil)
+      validate_routing_tokens!(req_options)
       perform(:add, key, value, ttl_or_default(ttl), req_options)
     end
 
@@ -345,17 +432,49 @@ module Dalli
     # key already exists on the server.  Returns the new CAS value if the
     # operation succeeded, or falsy otherwise.
     def replace_cas(key, value, cas, ttl = nil, req_options = nil)
+      validate_routing_tokens!(req_options)
       perform(:replace, key, value, ttl_or_default(ttl), cas, req_options)
     end
 
     # Delete a key/value pair, verifying existing CAS.
     # Returns true if succeeded, and falsy otherwise.
-    def delete_cas(key, cas = 0)
-      perform(:delete, key, cas)
+    # Delete a key, optionally with a CAS check.
+    #
+    # `req_options` accepts the same meta-delete options as #delete.
+    def delete_cas(key, cas = 0, req_options = nil)
+      validate_delete_options!(req_options)
+      validate_routing_tokens!(req_options)
+      perform(:delete, key, cas, req_options)
     end
 
-    def delete(key)
-      delete_cas(key, 0)
+    ##
+    # Delete a key.
+    #
+    # `req_options` may include the memcached meta-delete options:
+    #
+    # - `:invalidate` (Boolean) — mark the item stale instead of removing it.
+    #   This is the tombstone: readers see `stale: true` from
+    #   #get_with_metadata and #get_multi_with_metadata, and the existing value
+    #   is still readable unless `:drop_value` is also set. A tombstoned key is
+    #   *not* a miss, which lets a reader tell "another process is repopulating
+    #   this" apart from "this was never here".
+    # - `:tombstone_ttl` (Integer seconds) — how long the stale marker lives.
+    #   Requires `:invalidate`; memcached only honors the TTL on a delete when
+    #   it accompanies the invalidate flag, so passing it alone raises
+    #   ArgumentError rather than sending a request the server would treat
+    #   differently than intended. Once it elapses, reads see a miss.
+    # - `:drop_value` (Boolean) — remove the item's value but leave the item, so
+    #   a tombstone need not retain the old payload. On its own it is not a
+    #   tombstone: reads are an ordinary hit with an empty value.
+    # - `:p_token`/`:l_token` (String) — opaque routing tokens for an
+    #   intermediate proxy or router; see #get.
+    #
+    #   dc.delete('key', invalidate: true, tombstone_ttl: 30, drop_value: true)
+    #
+    # @param key [String] the key to delete
+    # @param req_options [Hash, nil] meta-delete options
+    def delete(key, req_options = nil)
+      delete_cas(key, 0, req_options)
     end
 
     ##
@@ -363,22 +482,35 @@ module Dalli
     # This method is more efficient than calling delete() in a loop because
     # it batches requests by server and uses quiet mode.
     #
+    # `req_options` accepts the same meta-delete options as #delete and applies
+    # them to every key in the batch.
+    #
     # @param keys [Array<String>] keys to delete
-    # @return [Integer] the number of keys that were found and deleted. This is
-    #   best-effort: on a network error the operation is retried, and keys
-    #   deleted before the error are not recounted, so the result may
-    #   under-report the number actually removed when a failure occurs.
+    # @param req_options [Hash, nil] meta-delete options
+    # @return [Integer] the number of keys the server found and acted on. Only a
+    #   key that did not exist decrements this count, so with `:invalidate` it
+    #   reports how many keys were tombstoned rather than removed -- the action
+    #   is whichever one the caller asked for. This is best-effort: a transient
+    #   network error is retried automatically, and keys handled before the
+    #   error are not recounted, so the result may under-report when a retry
+    #   occurs. If a server remains unreachable after retrying, raises
+    #   Dalli::NetworkError.
+    # @raise [Dalli::NetworkError] if a server is unreachable after retrying
     #
     # Example:
     #   client.delete_multi(['key1', 'key2', 'key3'])
-    def delete_multi(keys)
+    #   client.delete_multi(%w[key1 key2], invalidate: true, tombstone_ttl: 30)
+    def delete_multi(keys, req_options = nil)
       return 0 if keys.empty?
 
+      validate_delete_options!(req_options)
+      validate_routing_tokens!(req_options)
+
       Instrumentation.trace('delete_multi', multi_trace_attrs('delete_multi', keys.size, keys)) do
         if ring.servers.size == 1
-          single_server_delete_multi(keys)
+          single_server_delete_multi(keys, req_options)
         else
-          pipelined_deleter.process(keys)
+          pipelined_deleter.process(keys, req_options)
         end
       end
     end
@@ -386,15 +518,17 @@ module Dalli
     ##
     # Append value to the value already stored on the server for 'key'.
     # Appending only works for values stored with :raw => true.
-    def append(key, value)
-      perform(:append, key, value.to_s)
+    def append(key, value, req_options = nil)
+      validate_routing_tokens!(req_options)
+      perform(:append, key, value.to_s, req_options)
     end
 
     ##
     # Prepend value to the value already stored on the server for 'key'.
     # Prepending only works for values stored with :raw => true.
-    def prepend(key, value)
-      perform(:prepend, key, value.to_s)
+    def prepend(key, value, req_options = nil)
+      validate_routing_tokens!(req_options)
+      perform(:prepend, key, value.to_s, req_options)
     end
 
     ##
@@ -410,10 +544,11 @@ module Dalli
     # #cas.
     #
     # If the value already exists, it must have been set with raw: true
-    def incr(key, amt = 1, ttl = nil, default = nil)
+    def incr(key, amt = 1, ttl = nil, default = nil, req_options = nil)
       check_positive!(amt)
+      validate_routing_tokens!(req_options)
 
-      perform(:incr, key, amt.to_i, ttl_or_default(ttl), default)
+      perform(:incr, key, amt.to_i, ttl_or_default(ttl), default, req_options)
     end
 
     ##
@@ -432,10 +567,11 @@ module Dalli
     # #cas.
     #
     # If the value already exists, it must have been set with raw: true
-    def decr(key, amt = 1, ttl = nil, default = nil)
+    def decr(key, amt = 1, ttl = nil, default = nil, req_options = nil)
       check_positive!(amt)
+      validate_routing_tokens!(req_options)
 
-      perform(:decr, key, amt.to_i, ttl_or_default(ttl), default)
+      perform(:decr, key, amt.to_i, ttl_or_default(ttl), default, req_options)
     end
 
     ##
@@ -512,6 +648,30 @@ module Dalli
 
     private
 
+    # Raised before the request reaches a server: RequestFormatter enforces the
+    # same rule, but reaching it means unwinding through Protocol::Base#request,
+    # which logs the failure as unexpected and closes the connection.  A caller
+    # passing the wrong options should get a clean ArgumentError and keep its
+    # connection.
+    def validate_delete_options!(req_options)
+      return unless req_options.is_a?(Hash)
+
+      tombstone_ttl = req_options[:tombstone_ttl]
+      return unless tombstone_ttl
+
+      raise ArgumentError, 'tombstone_ttl requires invalidate: true' unless req_options[:invalidate]
+
+      # tombstone_kwargs coerces this with Integer(), deep inside the request
+      # path; validated here first so a bad value raises cleanly instead of
+      # unwinding through Protocol::Base#request, which would close the
+      # connection on the ArgumentError Integer() raises.
+      begin
+        Integer(tombstone_ttl)
+      rescue ArgumentError, TypeError
+        raise ArgumentError, "tombstone_ttl must be an integer, got #{tombstone_ttl.inspect}"
+      end
+    end
+
     def record_hit_miss_metrics(span, key_count, hit_count)
       return unless span
 
@@ -519,10 +679,10 @@ module Dalli
                           'db.memcached.miss_count' => key_count - hit_count)
     end
 
-    def get_multi_yielding(keys)
+    def get_multi_yielding(keys, req_options = nil)
       Instrumentation.trace_with_result('get_multi', get_multi_attributes(keys)) do |span|
         hit_count = 0
-        pipelined_getter.process(keys) do |k, data|
+        pipelined_getter.process(keys, req_options) do |k, data|
           hit_count += 1
           yield k, data.first
         end
@@ -531,13 +691,13 @@ module Dalli
       end
     end
 
-    def get_multi_hash(keys)
+    def get_multi_hash(keys, req_options = nil)
       Instrumentation.trace_with_result('get_multi', get_multi_attributes(keys)) do |span|
         hash = if ring.servers.size == 1
-                 single_server_get_multi(keys)
+                 single_server_get_multi(keys, req_options)
                else
                  {}.tap do |h|
-                   pipelined_getter.process(keys) { |k, data| h[k] = data.first }
+                   pipelined_getter.process(keys, req_options) { |k, data| h[k] = data.first }
                  end
                end
         record_hit_miss_metrics(span, keys.size, hash.size)
@@ -550,11 +710,33 @@ module Dalli
       server if server&.alive?
     end
 
-    def single_server_get_multi(keys)
+    # The three single_server_* fast-path methods below share one contract,
+    # matching the pipelined multi-server path they stand in for: a transient
+    # RetryableNetworkError is retried (bounded implicitly by the server's own
+    # socket_max_failures, same as the pipelined path's retry), and a hard
+    # NetworkError -- the server genuinely unreachable, not just blipping --
+    # propagates to the caller rather than being swallowed into a silently
+    # wrong result. Only server_for_key/single_server finding no live server
+    # to route to at all is still silent, matching Ring#keys_grouped_by_server
+    # dropping a key it can't route on both the single- and multi-server paths.
+    def single_server_get_multi(keys, req_options = nil)
+      keys.map! { |k| @key_manager.validate_key(k.to_s) }
+      return {} unless (server = single_server)
+
+      result = server.request(:read_multi_req, keys, req_options)
+      result.transform_keys! { |k| @key_manager.key_without_namespace(k) }
+      result
+    rescue Dalli::RetryableNetworkError => e
+      Dalli.logger.debug { e.inspect }
+      Dalli.logger.debug { 'retrying single-server get_multi because of network error' }
+      retry
+    end
+
+    def single_server_get_multi_with_metadata(keys, req_options = nil)
       keys.map! { |k| @key_manager.validate_key(k.to_s) }
       return {} unless (server = single_server)
 
-      result = server.request(:read_multi_req, keys)
+      result = server.request(:read_multi_with_metadata_req, keys, req_options)
       result.transform_keys! { |k| @key_manager.key_without_namespace(k) }
       result
     rescue Dalli::NetworkError
@@ -566,24 +748,21 @@ module Dalli
       return unless (server = single_server)
 
       server.request(:write_multi_req, pairs, ttl, req_options)
-    rescue Dalli::NetworkError
-      nil
+    rescue Dalli::RetryableNetworkError => e
+      Dalli.logger.debug { e.inspect }
+      Dalli.logger.debug { 'retrying single-server set_multi because of network error' }
+      retry
     end
 
-    def single_server_delete_multi(keys)
+    def single_server_delete_multi(keys, req_options = nil)
       validated_keys = keys.map { |k| @key_manager.validate_key(k.to_s) }
       return 0 unless (server = single_server)
 
-      server.request(:delete_multi_req, validated_keys)
+      server.request(:delete_multi_req, validated_keys, req_options)
     rescue Dalli::RetryableNetworkError => e
-      # Mirror the pipelined path: retry transient errors so a momentary blip
-      # still yields a real count. Bounded by the server's socket_max_failures,
-      # after which a hard NetworkError is raised and handled below.
       Dalli.logger.debug { e.inspect }
       Dalli.logger.debug { 'retrying single-server delete_multi because of network error' }
       retry
-    rescue Dalli::NetworkError
-      0
     end
 
     def get_multi_attributes(keys)
@@ -617,8 +796,33 @@ module Dalli
       raise ArgumentError, "Positive values only: #{amt}" if amt.negative?
     end
 
+    # Validated here, before the request reaches Protocol::Base#request, rather
+    # than only at the RequestFormatter level. Reaching only the formatter's
+    # check means unwinding through Protocol::Base#request, which logs the
+    # failure as unexpected and closes the connection -- a caller passing a
+    # bad token should get a clean ArgumentError and keep its connection.
+    ROUTING_TOKEN_FORBIDDEN = /[\r\n\0]/
+    private_constant :ROUTING_TOKEN_FORBIDDEN
+
+    def validate_routing_tokens!(req_options)
+      return unless req_options.is_a?(Hash)
+
+      validate_routing_token!(:p_token, req_options[:p_token])
+      validate_routing_token!(:l_token, req_options[:l_token])
+    end
+
+    def validate_routing_token!(name, value)
+      # Only an empty *String* is a no-op; see the matching comment in
+      # RequestFormatter#routing_tokens for why respond_to?(:empty?) is wrong
+      # here (it would also excuse [] / {} from the type check below).
+      return if value.nil? || (value.is_a?(String) && value.empty?)
+      raise ArgumentError, "#{name} must be a String, got #{value.class}" unless value.is_a?(String)
+      raise ArgumentError, "#{name} must not contain CRLF or null bytes" if value.match?(ROUTING_TOKEN_FORBIDDEN)
+    end
+
     def cas_core(key, always_set, ttl = nil, req_options = nil)
-      (value, cas) = perform(:cas, key)
+      validate_routing_tokens!(req_options)
+      (value, cas) = perform(:cas, key, req_options)
       return if value.nil? && !always_set
 
       newvalue = yield(value)
@@ -627,7 +831,13 @@ module Dalli
 
     def fetch_with_lock_request(key, ttl, lock_ttl, recache_threshold, req_options)
       server = ring.server_for_key(key)
-      result = server.request(:meta_get, key, { vivify_ttl: lock_ttl, recache_ttl: recache_threshold })
+      # req_options is the base, not the override: fetch_with_lock's own
+      # lock_ttl/recache_threshold parameters must always win, even if a
+      # caller's req_options happened to contain :vivify_ttl/:recache_ttl.
+      meta_options = req_options.is_a?(Hash) ? req_options.dup : {}
+      meta_options[:vivify_ttl] = lock_ttl
+      meta_options[:recache_ttl] = recache_threshold
+      result = server.request(:meta_get, key, meta_options)
 
       return result[:value] unless result[:won_recache]