dalli 5.0.10 → 5.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 64bc18f470e5b97d87b6531a968a9252c4ff46ba715d927c74b66e52f7549683
-  data.tar.gz: 70fc9db7148b82932b9f3323ac74a2c59dd7ed8b72bd9543dc160c5026d33880
+  metadata.gz: e11a3903c6fa786ac8f175bbb95130760f3e5886dc24fdc0825abc84d743006c
+  data.tar.gz: b9b376166e61c2899f23ba4bd5367360cc14c9e10a5d21fdb3ac34558c4b7303
 SHA512:
-  metadata.gz: a2c3def53ce7c5c4bf4d929bc45c5d1ff7f000db6b59abea3ac9ef3eb81e7f706ab8d0b01021b7002ff2b2153014b17c018c5060935e022f867b4981318afc06
-  data.tar.gz: f63a580cc08950f6da91213f0c4b9ecd2cb75961fe4467e06c11ecb434d2b13ff5950fceb31d43f8b8c42d8997fec3fd1bd40eebc1aa59dda41b2c1092a5e253
+  metadata.gz: bfc8526fce280bcb5003b81128b9e005975ffff54c1a7c82ce9d7b1104954c4dc169507108df70e4c5d90f444c01001640af9fccc038d180954e81f7a7459a55
+  data.tar.gz: 3c083001313d3dbdbc91cb122a76f8bd0af1fd5de75cddb89d2985ec67b40c313124ba56447818c6000d05633c80c4fae34cbaa560515a9947ca4a0fbe542134

data/CHANGELOG.md

@@ -4,95 +4,62 @@ Dalli Changelog
 Unreleased
 ==========
 
-5.0.10
+5.1.0
 ==========
 
-Security:
-
-- Keep the caller's key when retrying a request (GHSA-m252-9cgf-vx2w)
-  - With a `namespace`, a request retried after a transient network error (a timeout, or a connection closed by memcached or a proxy) applied the namespace a second time, so a retried read could return a different key's value and a retried write could overwrite a different key
-  - Affects single-key operations since 5.0.3, and `get_with_metadata` and `fetch_with_lock` since 4.1.0; fixed in 5.2.2, 5.1.4, 5.0.10 and 4.3.7. Clients without a namespace aren't affected
-- Complete the fix for per-request `raw: true` on reads (GHSA-wr87-m4jw-29x5)
-  - `cas`, `cas!` and `fetch_with_lock` accepted `raw: true` but didn't pass it to the read, so the value was still deserialized. They now honor it
-  - A read made with `raw: true` doesn't ask for flags, but a reply carrying them anyway (from a proxy or a hostile server) still had its value deserialized. Raw reads now ignore flags in the reply
-  - Affects 5.0.9 and earlier; fixed in 5.2.2, 5.1.4, 5.0.10, 4.3.7 and 3.2.13
-
-Bug fixes:
-
-- `get_with_metadata` and `fetch_with_lock` retried the final error raised when a server is marked down, so with `down_retry_delay: 0` they retried an unresponsive server forever. They now retry only retryable errors, like other operations
-- Fix a regression in 5.0.9: a `get_multi` that didn't finish within `socket_timeout` counted toward `socket_max_failures`, so two slow `get_multi` calls in a row marked a healthy server down. It now just closes the connection
-- Fix a regression in 5.0.9: over TLS, every request buffered in a `quiet` block was sent as its own TLS record and system call (2.5 times slower for a block of 2000 deletes). Each flush is one write again
-- Handle malformed replies from a broken or hostile server or proxy: a negative value size, a hit with no key, or a hit with no `s` flag could leave the connection out of step or raise a non-Dalli error
-- With `decompressed_max_bytes` in effect, data after the end of a compressed value's stream was returned as part of the value. It's now ignored, as without the limit
-- With a `digest_class` whose digests aren't short hex strings, shortening a long key could loop forever. It now raises `ArgumentError`
-
-Development:
-
-- Run the Tests, RuboCop and Profiles workflows on pushes to `main` and the `*-stable` branches only, so a pull request's branch isn't tested twice (backport of #1198)
-
-5.0.9
-==========
-
-Security:
-
-- Fix pipelined `get_multi` returning one key's value for another after an error reply (GHSA-p6pm-ch9v-44vx)
-  - The pipelined reply parser took any reply without a body for the end of the batch. An error reply for one key (`CLIENT_ERROR`, `SERVER_ERROR`, or `EN` from a proxy) ended it early, and the replies still on the connection were read as the replies to later commands, so a `get` could return another key's value
-  - Keys are now measured in bytes as sent, after base64 encoding when the key needs it. A key under 250 characters but over 250 bytes on the wire (for example, one with many non-ASCII characters) was the easiest way to cause that error reply. Such keys are now truncated like other long keys; keys that worked before are unchanged
-  - Affects the meta protocol since 3.2.0 (the default since 5.0.0); fixed in 5.2.1, 5.1.3, 5.0.9, 4.3.6 and 3.2.12
-- Stop retrying an unresponsive server forever (GHSA-4qp6-2jcr-596v)
-  - A server that accepted connections but never answered (or dropped the connection on a request) was retried forever, hanging the caller, because each successful reconnect reset the failure count. Requests now fail after `socket_max_failures` attempts and the server is marked down, as when it can't be reached at all; it gets a full set of attempts again after `down_retry_delay`
-  - Affects all versions; fixed in 5.2.1, 5.1.3, 5.0.9, 4.3.6 and 3.2.12
-- Limit decompressed and reply value sizes (GHSA-3553-vcg5-72jw)
-  - Values flagged as compressed were inflated without limit, so a small stored value could expand to gigabytes on read, whatever the client's `compress` or `serializer` settings. The new `decompressed_max_bytes` option (default 128 MiB; `nil` disables it) makes a read that would pass it raise `Dalli::UnmarshalError`. Custom compressors whose `decompress` takes only the data keep working, without the limit
-  - The value size in a reply was used to read or buffer that many bytes, so a malicious server could make the client allocate gigabytes. Sizes over 1 GiB, memcached's largest item, now raise `Dalli::DalliError` before reading
-  - Affects all versions; fixed in 5.2.1, 5.1.3, 5.0.9, 4.3.6 and 3.2.12
-- Honor per-request `raw: true` on reads, and add `Dalli::JSONSerializer` (GHSA-wr87-m4jw-29x5)
-  - `get_with_metadata` ignored a per-request `raw: true` and deserialized the value according to its stored flags, so a caller who asked for raw bytes could still have `Marshal.load` run on data someone else wrote to memcached. They now return the stored bytes
-  - The `raw` part affects `get_with_metadata` since 4.2.0
-  - `serializer: JSON` reads values with `JSON.load`, which on json gem versions before 3.0 creates an object of the class named in a stored `json_class` key when the json additions are loaded. The new `Dalli::JSONSerializer` reads with `JSON.parse` and only returns plain JSON types. The README and the Marshal security warning now recommend it; `serializer: JSON` itself is unchanged
-  - The `serializer: JSON` part affects any version used with the json gem before 3.0
-- Keep a forked child from writing to or closing the parent's connection (GHSA-w39f-xq2m-4g8x)
-  - A child forked while the parent had requests buffered but not yet sent, such as quiet writes inside a `quiet` block, sent them again when it closed the client or reconnected: quiet writes ran twice, and the parent's later replies could be read as the replies to other requests, returning one key's value for another. Dalli now buffers requests itself and a forked child discards them
-  - Affects 4.2.0 and later; fixed in 5.2.1, 5.1.3, 5.0.9 and 4.3.6
-  - With TLS, a forked child that closed the client (or reconnected after detecting the fork) sent a TLS close on the connection the parent was still using, ending the parent's session. A forked child now closes only its own copy of the socket
-  - Affects all versions with TLS; fixed in 5.2.1, 5.1.3, 5.0.9, 4.3.6 and 3.2.12
-
-Bug fixes:
-
-- An exception raised by a `get_multi` block (such as `Timeout::Error`) was treated as a network failure: swallowed, with the whole `get_multi` retried and keys yielded twice. It now reaches the caller
-
-Development:
-
-- Add property tests for request building and fuzz tests for the reply parsers
-- Fix three flaky tests: the `KeyManager` namespace key-length test, the Rack session freshness test, and `MemcachedManager`'s start and stop handling (backport of #1141, #1181)
-- Fix flaky failover tests: move their ports out of Linux's ephemeral port range, and wait for memcached to accept connections after starting it (backport of #1184)
-
-5.0.8
-==========
+Features:
 
-Bug fixes:
+- Add opaque routing tokens: `:p_token` and `:l_token` request options (#1147, #1154)
+  - `get`, `gat`, `get_cas`, `get_with_metadata`, `fetch_with_lock`, `set`/`add`/`replace`/`set_cas`/`replace_cas`, `append`/`prepend`, `incr`/`decr`, `cas`/`cas!`, `delete`/`delete_cas`, and the bulk operations (`get_multi`, `get_multi_cas`, `get_multi_with_metadata`, `set_multi`, `delete_multi`) all accept per-request `:p_token`/`:l_token` options, appended to the wire protocol as `P<token>`/`L<token>` -- applied to every key on the bulk methods
+  - memcached itself ignores these tokens; per the meta protocol spec they exist as hints for a proxy or router sitting between the client and memcached
+  - CRLF and NUL bytes raise `ArgumentError` before the request reaches the socket, both in `Dalli::Client` and in `RequestFormatter`, so a bad token can't be used for wire-protocol injection and can't close the connection out from under the caller the way a formatter-only check would
+  - The bulk methods and `delete`/`delete_cas` were deferred out of #1147 to avoid racing other in-flight PRs touching the same method signatures; #1154 completes them, including both the single-server fast path and the multi-server pipelined path for each
+  - Extracted from #1130; thanks to Nick Herson for the original idea and Jianbin Chen for porting it forward
+
+- Support tombstone (mark-stale) deletes on `delete`, `delete_cas`, and `delete_multi` (#1145, #1153)
+  - `:invalidate` marks the item stale instead of removing it, so `#get_with_metadata` / `#get_multi_with_metadata` report `stale: true` and a reader can tell "another process is repopulating this" apart from "this was never here" -- a tombstoned key is not a miss
+  - `:tombstone_ttl` controls how long the stale marker lives; requires `:invalidate`, since memcached only honors the TTL on a delete when it accompanies the invalidate flag
+  - `:drop_value` removes the item's value but leaves the item; on its own it is not a tombstone -- reads are an ordinary hit with an empty value
+  - `delete_multi` applies the same options to every key in the batch, on both the single-server and pipelined paths; its return value keeps counting keys the server found and acted on, so under `:invalidate` it reports how many keys were tombstoned rather than removed
+  - Extracted from #1130; thanks to Jianbin Chen for this contribution
 
-- Fix multi-server `get_multi` stopping at an empty value (#1170)
-  - The pipelined reply parser took a hit on a zero-length value (`VA 0`) for the terminating `MN`, so that server's remaining keys were silently missing from the result
-  - Its value terminator and any replies not yet parsed stayed on the connection and were read as the replies to later commands on it. Those commands could fail with `Dalli::DalliError: Response error`, or a single-key `get` could silently return another key's value
-  - An empty value is now returned as `''`, like a single-key `get` and single-server `get_multi`
-  - Affects the meta protocol since 3.2.0 (the default since 5.0.0)
-  - Thanks to Julian Richard Contreras for this contribution
+- Add `:miss` and `:return_ttl_remaining` to `get_with_metadata` (#1143)
+  - `:miss` is now always present in the returned Hash, distinguishing a true miss from a stored `nil` under `cache_nils` or a tombstoned, stale hit -- neither of which a `nil` `:value` alone can tell apart
+  - `:return_ttl_remaining` exposes the meta protocol's `t` flag as `:ttl_remaining` (seconds remaining, or `-1` for an item with no expiry), following the same opt-in shape as `:return_hit_status` / `:return_last_access`
+  - Extracted from #1130; thanks to Jianbin Chen for this contribution
 
-Development:
+- Add `get_multi_with_metadata` for stale-aware bulk reads (#1144)
+  - Returns `{ key => { value:, cas:, stale:, miss: } }` for the keys that were found; genuine misses are omitted, matching `get_multi` / `get_multi_cas` -- a tombstoned item is a hit at the protocol level, so it is still returned, with `stale: true`
+  - Routes to the same single-server fast path / pipelined-getter split as `get_multi`
+  - Extracted from #1130; thanks to Jianbin Chen for this contribution
 
-- Fix offenses reported by RuboCop 1.91 and require `rubocop >= 1.91` (backport of #1162)
+Other changes:
 
-5.0.7
-==========
+- Raise the documented minimum supported memcached version to 1.6.27 (#1140)
+  - Groundwork for the features above: `drop_value` tombstone deletes require 1.6.27, the highest floor of anything landing from #1130
+  - Not enforced at runtime -- `MIN_SUPPORTED_MEMCACHED_VERSION` only gates the test harness and the README's support statement, so this changes no running client's behavior
 
-Security:
+Bug Fixes:
 
-- Fix memcached command injection through numeric arguments (GHSA-6wmv-xq9m-fmp7)
-  - The `default` argument of `incr`/`decr`, and `fetch_with_lock`'s `lock_ttl` and `recache_threshold`, were written into the meta protocol command without conversion, so a String containing CRLF injected additional memcached commands (e.g. `set`, `flush_all`) on the connection
-  - These arguments must now be Integers, or Strings of decimal digits; anything else raises `ArgumentError` before a request is sent
-  - As defense in depth, `RequestFormatter` now converts every numeric flag it writes (`D`, `J`, `N`, `R`, `T`) to an Integer
-  - Thanks to oss-security-shop for the report
+- Retry a transient network error during a liveness check instead of treating it as terminal (#1150)
+  - `Dalli::Protocol::Base#alive?` caught any `NetworkError` -- including `RetryableNetworkError`, a subclass -- and unconditionally converted it to `false`, with no retry. This didn't match the retry-then-raise contract every other network-facing path in Dalli follows: a single transient connection hiccup during the liveness check itself (as opposed to an actual request) permanently reported a healthy server as down for that check
+  - Now retries once on `RetryableNetworkError`, letting `error_on_request!`'s own fail-count threshold decide when to actually give up (the same mechanism the rest of the codebase relies on): it keeps retrying until `socket_max_failures` is reached, then raises a terminal `NetworkError`, which is still converted to `false` as before
+  - **Behavior change:** a server that was previously marked "down" (engaging the `down_retry_delay` cooldown) only via an actual request could now also reach that state via a liveness check (`alive?`, and anything that calls it -- `Dalli::Client#stats`, `#reset_stats`, and `Ring`'s own server selection) exhausting its retries. Previously, a solitary transient failure during a liveness check was silently forgotten rather than tracked, so the cooldown was inconsistently applied depending on which code path first observed the failure
+  - Likely a contributing cause of the same intermittently failing `get_multi` failover integration test noted in #1149: that fix addressed the send/receive phase, but the liveness-check retry it introduced can itself force a fresh `connect()` mid-retry, giving this separate, pre-existing gap in `alive?` more chances to fire
+  - Also fixed a test (`test_ring.rb`, "detect when a dead server is up again") that had been unknowingly relying on the old behavior: it never engaged the `down_retry_delay` cooldown from a single transient failure, so its 0.5s delay never actually gated anything. Updated to use a 0s delay, since the test's intent is to verify reconnection is detected, not to test cooldown timing
+
+- Base64-encode keys containing control characters, not just NUL (#1148)
+  - `KeyRegularizer.required?` decided whether a key needed base64 encoding using `/\s/`, which matches most whitespace but none of the C0 control range (0x00-0x1F) or DEL (0x7F) -- a key that was otherwise ASCII-only and contained no whitespace (e.g. `"foo\x00bar"` or a key with an embedded ESC byte) went out on the wire unencoded
+  - Not a command-injection risk: the text protocol splits commands on CRLF, not other control bytes. The risk is key confusion -- anything downstream that treats one of these bytes specially (a C string terminating at NUL, a terminal or log line interpreting an escape byte) could silently act on a different key than Dalli believes it sent
+  - A raw control byte in a key was never protocol-compliant in the first place: memcached's own spec (`protocol.txt`) states a key "must not include control characters or whitespace." The only sanctioned way to carry such content in a key is the meta protocol's base64 (`b` flag) path -- the one whitespace and non-ASCII keys already used, and the one these keys now use too. The check is now `/[\p{Cntrl}\s]/`, matching that rule directly rather than special-casing NUL
+  - **Behavior change:** a key containing a control character now produces different bytes on the wire (base64-encoded, per the meta protocol's `b` flag) than before. Existing cache entries stored under the old, unencoded form of such a key will read as a miss once every reader has upgraded. **During a rolling deploy, old and new Dalli versions disagree about which physical key such a logical key maps to** -- not just a one-time cutover, but ongoing inconsistency between the old-version and new-version server pools for the duration of the rollout. Harmless for an ordinary cached value (worst case, extra cache misses); worth accounting for if such a key ever backs something stateful, like a lock or counter. Expected to be rare in practice: embedding a raw control byte in a cache key is unusual, and doing so was already outside what the protocol permits
+  - Found while auditing `request_formatter.rb` during the routing-token work in #1130 / #1147; unrelated to that change and predates it
+
+- Retry transient network errors in `get_multi`, `set_multi` and `delete_multi` instead of silently swallowing them (#1149)
+  - All three methods group keys by server and issue one request per server. Each per-server rescue clause caught `DalliError` and `NetworkError` together and swallowed both, just debug-logging -- since `RetryableNetworkError < NetworkError`, this also silently swallowed transient, retryable failures, dropping that server's keys from the result instead of the whole operation retrying (`get_multi`/`set_multi`'s single-server fast path did not even attempt a retry, on any failure)
+  - Six rescue sites across `PipelinedGetter`, `PipelinedSetter`, `PipelinedDeleter` and the `single_server_*` fast paths now retry a transient `RetryableNetworkError`, matching sibling rescue sites in the same files that already did this correctly
+  - **Behavior change:** if a server remains unreachable after retrying (not just a transient blip), these three methods now raise `Dalli::NetworkError` instead of silently returning an incomplete or empty result. This matches how every other Dalli::Client method already behaves on a hard network failure, and is the retry-then-raise behavior `delete_multi`'s own docs already described; it was just not reliably true for this failure path before. Code that calls these methods without rescuing `Dalli::NetworkError` should account for this if it can reach a fully unreachable server
+  - Likely the cause of an intermittently failing `get_multi` failover integration test that recurred across multiple PRs, though this could not be directly confirmed: the failure (an empty result with no exception in the logs) never reproduced locally despite repeated attempts, consistent with needing a genuine transient network hiccup that is far more likely on a loaded CI runner than an idle dev machine
 
 5.0.6
 ==========

data/Gemfile

@@ -18,7 +18,7 @@ group :development, :test do
   gem 'rack', '~> 3'
   gem 'rack-session'
   gem 'rake', '~> 13.0'
-  gem 'rubocop', '>= 1.91' # disable-next directives require 1.91+
+  gem 'rubocop'
   gem 'rubocop-minitest'
   gem 'rubocop-performance'
   gem 'rubocop-rake'

data/README.md

@@ -17,7 +17,12 @@ The name is a variant of Salvador Dali for his famous painting [The Persistence
 ## Requirements
 
 * Ruby 3.3 or later (JRuby also supported)
-* memcached 1.6 or later
+* memcached 1.6.27 or later
+
+Dalli is tested against both the minimum supported memcached version and the
+latest release. Earlier 1.6.x servers are not supported: the meta protocol
+rejects unknown flags outright, so features added after a server's release fail
+with `CLIENT_ERROR invalid flag` rather than degrading.
 
 ## Configuration Options
 
@@ -49,13 +54,9 @@ The separator must be a single non-alphanumeric character. Valid examples: `:`,
 By default, Dalli uses Ruby's Marshal for serialization. Deserializing untrusted data with Marshal can lead to remote code execution. If you cache user-controlled data, consider using a safer serializer:
 
 ```ruby
-Dalli::Client.new('localhost:11211', serializer: Dalli::JSONSerializer)
+Dalli::Client.new('localhost:11211', serializer: JSON)
 ```
 
-`Dalli::JSONSerializer` reads values with `JSON.parse`, so it only ever returns plain hashes, arrays, strings, numbers, booleans and `nil`. Passing the `JSON` module itself (`serializer: JSON`) reads values with `JSON.load`, which can create objects of other classes from a stored `json_class` key and is not recommended for data an attacker might write.
-
-Per-request `raw: true` returns the stored bytes without deserializing or decompressing them, on every read method that accepts it (`get`, `get_with_metadata` and `fetch`).
-
 See the [5.0-Upgrade.md](5.0-Upgrade.md) guide for upgrade information.
 
 ## OpenTelemetry Tracing