dalli 5.0.7 → 5.1.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 10a8b21fed4728b6f98929e983262dc1fc32c59711fce78a84b0df82b0348ca1
-  data.tar.gz: 0253d48293d3b482e8362830acd9323873302e202cb230bca3fc1b1e09b1b457
+  metadata.gz: 1cb62e15948a7d649e178bcfc9aa03de998be618f4fa7d08c988f874006c941f
+  data.tar.gz: 0b2d3d9d1a7a73276a6fa22e2015805a306bab7c0e53616a9531eccff7979c33
 SHA512:
-  metadata.gz: bda1d416cf1f88514d0181ed2e7f7b53e1b935fb07901398cb91183f63e42d0ab7c9e54c177264cf77fc793ead7b8183d79cf3c335d8dee11aae5049a6f0fff9
-  data.tar.gz: d81bec09db2c23cdc95204b0bec5c491fbcb60e12823245b9d9d338502c98b2e6af5f6e58a12b11c081889ea5d63cb079b6e8ef2fb869db147f67219db86ac55
+  metadata.gz: eaab609ed368a7f8f2fa129eab6177894cfc7548ea25002b30bae4ee13d22daaa6470241f97ba6e01a5b457820587e5c08e4426b948f889f6921c556dbdd8438
+  data.tar.gz: 78d2faea9492b9f18bf7a0f5a0459a785319d0261c81f32c9fd218bb837b73550b79af8bd94373abc17f6e94db8b049a515308de2475905fda004a66de4cacc6

data/CHANGELOG.md

@@ -4,7 +4,7 @@ Dalli Changelog
 Unreleased
 ==========
 
-5.0.7
+5.1.1
 ==========
 
 Security:
@@ -13,8 +13,87 @@ Security:
   - The `default` argument of `incr`/`decr`, and `fetch_with_lock`'s `lock_ttl` and `recache_threshold`, were written into the meta protocol command without conversion, so a String containing CRLF injected additional memcached commands (e.g. `set`, `flush_all`) on the connection
   - These arguments must now be Integers, or Strings of decimal digits; anything else raises `ArgumentError` before a request is sent
   - As defense in depth, `RequestFormatter` now converts every numeric flag it writes (`D`, `J`, `N`, `R`, `T`) to an Integer
+  - Affects 3.2.0 and later (3.2.x only with `protocol: :meta`); fixed in 5.1.1 and 4.3.4
   - Thanks to oss-security-shop for the report
 
+Performance:
+
+- Reduce Ruby overhead on the single-key `get` path by about 28% (#1160)
+  - A plain `get` builds its `mg` request with one string interpolation instead of going through `meta_get`'s keyword arguments, and skips option handling when called without options
+  - A `VA <size> f<flags>` hit line is parsed in place instead of being split into tokens
+  - The key check for control characters and whitespace uses a byte class that matches the same ASCII bytes as `[\p{Cntrl}\s]`, about 5x faster; this applies to every operation that sends a key
+  - Allocations per `get` hit drop from 23 to 16
+  - Thanks to Julian Richard Contreras for this contribution
+- Speed up multi-server `get_multi` by about 30% (4 servers, 100 keys), and bring small batches in line with 2.7.11 (#1161)
+  - Each server's queries and terminating noop are sent before the next server's are built, so memcached answers earlier servers while later ones are prepared
+  - A server's queries are built in one pass with `RequestFormatter.multi_meta_get`, and plain `get_multi` no longer requests the CAS value it discards (`get_multi_cas` still does)
+  - Pipelined replies are parsed in one pass over the returned flags
+  - Routing many keys checks each server's `alive?` once per call instead of twice per key, and the ring's binary search runs over plain integers
+  - Thanks to Julian Richard Contreras for this contribution
+
+Development:
+
+- Fix offenses reported by RuboCop 1.91 and require `rubocop >= 1.91` (#1162)
+  - RuboCop 1.91 adds `Style/DirectiveScope`; single-statement `disable`/`enable` pairs become `disable-next` directives, which older RuboCop versions do not recognize
+  - Removes a misplaced `# encoding: ascii` comment in `client.rb` that Ruby had always ignored
+
+5.1.0
+==========
+
+Features:
+
+- Add opaque routing tokens: `:p_token` and `:l_token` request options (#1147, #1154)
+  - `get`, `gat`, `get_cas`, `get_with_metadata`, `fetch_with_lock`, `set`/`add`/`replace`/`set_cas`/`replace_cas`, `append`/`prepend`, `incr`/`decr`, `cas`/`cas!`, `delete`/`delete_cas`, and the bulk operations (`get_multi`, `get_multi_cas`, `get_multi_with_metadata`, `set_multi`, `delete_multi`) all accept per-request `:p_token`/`:l_token` options, appended to the wire protocol as `P<token>`/`L<token>` -- applied to every key on the bulk methods
+  - memcached itself ignores these tokens; per the meta protocol spec they exist as hints for a proxy or router sitting between the client and memcached
+  - CRLF and NUL bytes raise `ArgumentError` before the request reaches the socket, both in `Dalli::Client` and in `RequestFormatter`, so a bad token can't be used for wire-protocol injection and can't close the connection out from under the caller the way a formatter-only check would
+  - The bulk methods and `delete`/`delete_cas` were deferred out of #1147 to avoid racing other in-flight PRs touching the same method signatures; #1154 completes them, including both the single-server fast path and the multi-server pipelined path for each
+  - Extracted from #1130; thanks to Nick Herson for the original idea and Jianbin Chen for porting it forward
+
+- Support tombstone (mark-stale) deletes on `delete`, `delete_cas`, and `delete_multi` (#1145, #1153)
+  - `:invalidate` marks the item stale instead of removing it, so `#get_with_metadata` / `#get_multi_with_metadata` report `stale: true` and a reader can tell "another process is repopulating this" apart from "this was never here" -- a tombstoned key is not a miss
+  - `:tombstone_ttl` controls how long the stale marker lives; requires `:invalidate`, since memcached only honors the TTL on a delete when it accompanies the invalidate flag
+  - `:drop_value` removes the item's value but leaves the item; on its own it is not a tombstone -- reads are an ordinary hit with an empty value
+  - `delete_multi` applies the same options to every key in the batch, on both the single-server and pipelined paths; its return value keeps counting keys the server found and acted on, so under `:invalidate` it reports how many keys were tombstoned rather than removed
+  - Extracted from #1130; thanks to Jianbin Chen for this contribution
+
+- Add `:miss` and `:return_ttl_remaining` to `get_with_metadata` (#1143)
+  - `:miss` is now always present in the returned Hash, distinguishing a true miss from a stored `nil` under `cache_nils` or a tombstoned, stale hit -- neither of which a `nil` `:value` alone can tell apart
+  - `:return_ttl_remaining` exposes the meta protocol's `t` flag as `:ttl_remaining` (seconds remaining, or `-1` for an item with no expiry), following the same opt-in shape as `:return_hit_status` / `:return_last_access`
+  - Extracted from #1130; thanks to Jianbin Chen for this contribution
+
+- Add `get_multi_with_metadata` for stale-aware bulk reads (#1144)
+  - Returns `{ key => { value:, cas:, stale:, miss: } }` for the keys that were found; genuine misses are omitted, matching `get_multi` / `get_multi_cas` -- a tombstoned item is a hit at the protocol level, so it is still returned, with `stale: true`
+  - Routes to the same single-server fast path / pipelined-getter split as `get_multi`
+  - Extracted from #1130; thanks to Jianbin Chen for this contribution
+
+Other changes:
+
+- Raise the documented minimum supported memcached version to 1.6.27 (#1140)
+  - Groundwork for the features above: `drop_value` tombstone deletes require 1.6.27, the highest floor of anything landing from #1130
+  - Not enforced at runtime -- `MIN_SUPPORTED_MEMCACHED_VERSION` only gates the test harness and the README's support statement, so this changes no running client's behavior
+
+Bug Fixes:
+
+- Retry a transient network error during a liveness check instead of treating it as terminal (#1150)
+  - `Dalli::Protocol::Base#alive?` caught any `NetworkError` -- including `RetryableNetworkError`, a subclass -- and unconditionally converted it to `false`, with no retry. This didn't match the retry-then-raise contract every other network-facing path in Dalli follows: a single transient connection hiccup during the liveness check itself (as opposed to an actual request) permanently reported a healthy server as down for that check
+  - Now retries once on `RetryableNetworkError`, letting `error_on_request!`'s own fail-count threshold decide when to actually give up (the same mechanism the rest of the codebase relies on): it keeps retrying until `socket_max_failures` is reached, then raises a terminal `NetworkError`, which is still converted to `false` as before
+  - **Behavior change:** a server that was previously marked "down" (engaging the `down_retry_delay` cooldown) only via an actual request could now also reach that state via a liveness check (`alive?`, and anything that calls it -- `Dalli::Client#stats`, `#reset_stats`, and `Ring`'s own server selection) exhausting its retries. Previously, a solitary transient failure during a liveness check was silently forgotten rather than tracked, so the cooldown was inconsistently applied depending on which code path first observed the failure
+  - Likely a contributing cause of the same intermittently failing `get_multi` failover integration test noted in #1149: that fix addressed the send/receive phase, but the liveness-check retry it introduced can itself force a fresh `connect()` mid-retry, giving this separate, pre-existing gap in `alive?` more chances to fire
+  - Also fixed a test (`test_ring.rb`, "detect when a dead server is up again") that had been unknowingly relying on the old behavior: it never engaged the `down_retry_delay` cooldown from a single transient failure, so its 0.5s delay never actually gated anything. Updated to use a 0s delay, since the test's intent is to verify reconnection is detected, not to test cooldown timing
+
+- Base64-encode keys containing control characters, not just NUL (#1148)
+  - `KeyRegularizer.required?` decided whether a key needed base64 encoding using `/\s/`, which matches most whitespace but none of the C0 control range (0x00-0x1F) or DEL (0x7F) -- a key that was otherwise ASCII-only and contained no whitespace (e.g. `"foo\x00bar"` or a key with an embedded ESC byte) went out on the wire unencoded
+  - Not a command-injection risk: the text protocol splits commands on CRLF, not other control bytes. The risk is key confusion -- anything downstream that treats one of these bytes specially (a C string terminating at NUL, a terminal or log line interpreting an escape byte) could silently act on a different key than Dalli believes it sent
+  - A raw control byte in a key was never protocol-compliant in the first place: memcached's own spec (`protocol.txt`) states a key "must not include control characters or whitespace." The only sanctioned way to carry such content in a key is the meta protocol's base64 (`b` flag) path -- the one whitespace and non-ASCII keys already used, and the one these keys now use too. The check is now `/[\p{Cntrl}\s]/`, matching that rule directly rather than special-casing NUL
+  - **Behavior change:** a key containing a control character now produces different bytes on the wire (base64-encoded, per the meta protocol's `b` flag) than before. Existing cache entries stored under the old, unencoded form of such a key will read as a miss once every reader has upgraded. **During a rolling deploy, old and new Dalli versions disagree about which physical key such a logical key maps to** -- not just a one-time cutover, but ongoing inconsistency between the old-version and new-version server pools for the duration of the rollout. Harmless for an ordinary cached value (worst case, extra cache misses); worth accounting for if such a key ever backs something stateful, like a lock or counter. Expected to be rare in practice: embedding a raw control byte in a cache key is unusual, and doing so was already outside what the protocol permits
+  - Found while auditing `request_formatter.rb` during the routing-token work in #1130 / #1147; unrelated to that change and predates it
+
+- Retry transient network errors in `get_multi`, `set_multi` and `delete_multi` instead of silently swallowing them (#1149)
+  - All three methods group keys by server and issue one request per server. Each per-server rescue clause caught `DalliError` and `NetworkError` together and swallowed both, just debug-logging -- since `RetryableNetworkError < NetworkError`, this also silently swallowed transient, retryable failures, dropping that server's keys from the result instead of the whole operation retrying (`get_multi`/`set_multi`'s single-server fast path did not even attempt a retry, on any failure)
+  - Six rescue sites across `PipelinedGetter`, `PipelinedSetter`, `PipelinedDeleter` and the `single_server_*` fast paths now retry a transient `RetryableNetworkError`, matching sibling rescue sites in the same files that already did this correctly
+  - **Behavior change:** if a server remains unreachable after retrying (not just a transient blip), these three methods now raise `Dalli::NetworkError` instead of silently returning an incomplete or empty result. This matches how every other Dalli::Client method already behaves on a hard network failure, and is the retry-then-raise behavior `delete_multi`'s own docs already described; it was just not reliably true for this failure path before. Code that calls these methods without rescuing `Dalli::NetworkError` should account for this if it can reach a fully unreachable server
+  - Likely the cause of an intermittently failing `get_multi` failover integration test that recurred across multiple PRs, though this could not be directly confirmed: the failure (an empty result with no exception in the logs) never reproduced locally despite repeated attempts, consistent with needing a genuine transient network hiccup that is far more likely on a loaded CI runner than an idle dev machine
+
 5.0.6
 ==========
 

data/Gemfile

@@ -18,7 +18,7 @@ group :development, :test do
   gem 'rack', '~> 3'
   gem 'rack-session'
   gem 'rake', '~> 13.0'
-  gem 'rubocop'
+  gem 'rubocop', '>= 1.91' # disable-next directives require 1.91+
   gem 'rubocop-minitest'
   gem 'rubocop-performance'
   gem 'rubocop-rake'

data/README.md

@@ -17,7 +17,12 @@ The name is a variant of Salvador Dali for his famous painting [The Persistence
 ## Requirements
 
 * Ruby 3.3 or later (JRuby also supported)
-* memcached 1.6 or later
+* memcached 1.6.27 or later
+
+Dalli is tested against both the minimum supported memcached version and the
+latest release. Earlier 1.6.x servers are not supported: the meta protocol
+rejects unknown flags outright, so features added after a server's release fail
+with `CLIENT_ERROR invalid flag` rather than degrading.
 
 ## Configuration Options