dalli 5.2.6 → 5.3.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: db76345a60e108cc8a1aa4f39adae0e657b200b4e3f17f7196702776c1ff8bc4
-  data.tar.gz: eb643908bb2aa3654f2d7c82bcce21fa82159931e162cdf17bd79d9fe40b2ed0
+  metadata.gz: ecf53cb1683b72cd89df9074f442a040de892003a6f39254e4958654a9f430a7
+  data.tar.gz: 46b400e8a5cb63261af72340de184bf7f1188a7b5fc2454a052af13e8ff62167
 SHA512:
-  metadata.gz: e5d227f5b4e7fda6e35907f0d94b74fcd528d0c203ce6a7c5b01ab33e2908fe78147c70df8661eedbc2b82f1024b4c89bb890fc0418745150c379805f87afea2
-  data.tar.gz: dbb2e3d7bb7b1c852a8c0d8b359f1621d0aa0f3ef32dbe1ca449e384325a929e0db6ba893357195e284c84162ce3ae4a1ce39e9f86b5b1bfde149a3725ae3a45
+  metadata.gz: 518451bbab10db4e47140cd9b716c1845ca1b1be08f95e5e5996d4f6dc655f7a9bfd2a40e51c968cdaa45eaa3355518d41a0e7c36a37367d2c387bc9dcb18bed
+  data.tar.gz: 84c9fa710a82fb7f444f50b84820e43c7697050fd70376117d6d9daa30f905d159bd1639e29762c027391b8521361888075b47df476f6cfb59ddcb0e76496f78

data/CHANGELOG.md

@@ -4,6 +4,61 @@ Dalli Changelog
 Unreleased
 ==========
 
+5.3.1
+==========
+
+Bug fixes:
+
+- `Rack::Session::Dalli`'s 401 for a session deleted mid-request called `Hash#slice` on the session options. A Rails session store puts an `ActionDispatch::Request::Session::Options` there, so the 401 raised `NoMethodError` instead of expiring the cookie (#1291, @seballot)
+
+5.3.0
+==========
+
+New features:
+
+- `enforce_read_timeout: true` makes JRuby and TruffleRuby give up waiting for a reply after `socket_timeout`, as CRuby does. Without it, a server that accepted a request but never replied blocked the caller indefinitely on those engines, including in `get` and `get_multi`. A server that stops partway through a reply can still block. It's off by default because it costs about 8% of single-key throughput on TruffleRuby and 20-29% on JRuby, and it has no effect on CRuby or TLS connections
+
+Bug fixes:
+
+- A `vivify_ttl` (or `fetch_with_lock`'s `lock_ttl`) over 30 days now protects against a thundering herd. memcached reads a TTL over 30 days as a Unix timestamp, so the stub for a miss expired at once and every caller won the recache race. `vivify_ttl` is now converted like other TTLs
+- `set_cas`, `replace_cas` and `delete_cas` dropped a CAS that couldn't be parsed, such as `'abc'`, and sent an unconditional write, so the value was overwritten or deleted whatever its CAS. They now raise `ArgumentError`
+- `flush('abc')` flushed immediately. It now raises `ArgumentError`
+- A TTL String that isn't a whole number, such as `'abc'`, became 0, which means no expiry. It now raises `ArgumentError`, including for `expires_in` when the client is created
+- A TTL, CAS, `incr`/`decr` amount or initial value outside the range memcached accepts got a `CLIENT_ERROR` reply, which a `quiet` block or `set_multi` discarded, so the write silently did nothing. These now raise `ArgumentError`
+- An invalid `vivify_ttl` or `recache_ttl` raised from deep in the request and closed the connection
+- `tombstone_ttl` read a String with a leading zero as octal (`'010'` was 8) and accepted hex such as `'0x10'`. It's now parsed in base 10 like every other TTL, and `Float::NAN` raises `ArgumentError` rather than `FloatDomainError`
+- Multi-gets now raise `Dalli::UnmarshalError` for a value that can't be deserialized, as `get` does, and only after reading the rest of the replies
+  - `get_multi` and `get_multi_cas` raised partway through reading a server's replies, closing the connection, and `get_multi_with_metadata` on more than one server silently dropped every key on that server. A large `get_multi` could also drop that server's keys without raising when it read the bad value while still sending requests (seen mostly on JRuby and TruffleRuby)
+  - In block form the other values are still yielded, and the connections stay open. `get` and the other single-key reads also keep the connection open after this error now
+- `get_multi`, `get_multi_cas` and `get_multi_with_metadata` now return each value under the key the caller passed. They rebuilt the key from the one memcached returned, so:
+  - With a namespace, a key that isn't valid UTF-8 (such as `"\xff".b`) raised `ArgumentError` for the whole call
+  - A key too long to send as is (over 250 bytes, or once base64-encoded) came back under its digested form
+  - A binary key with non-ASCII bytes came back tagged UTF-8, so `result.key?(key)` was false
+  - A key given more than once is now requested once, and a block is yielded it once
+- On TruffleRuby, a process spawned while Dalli had a connection open (with `system`, `spawn` or backticks) inherited the socket and kept the connection open on memcached's side until it exited. Dalli's sockets are now close-on-exec, as CRuby makes them by default
+- A request that failed after it was sent, such as one whose reply timed out, was sent again even though memcached may have applied it. `incr`, `decr`, `append` and `prepend` could be applied twice, and `add` and writes with a CAS could report failure for a write that had succeeded. Once some of a request has been sent, Dalli now retries only requests that are safe to apply twice (see Other changes)
+- When a server stopped while connected, the multi-key operations (`get_multi`, `get_multi_cas`, `get_multi_with_metadata`, `set_multi` and `delete_multi`) returned its keys as misses, or left them unwritten, instead of raising `Dalli::NetworkError` as documented. The retry found the server down and skipped it as if it had been down from the start. Without `failover` they now raise. With `failover` the keys still go to another server, as they do for `get` and `set`, when the retry can't connect to the failed server. A server that accepts the retry's connection and fails again is marked down during the operation, which raises `Dalli::NetworkError` either way (#1263, @jaideeppyne)
+- `Rack::Session::Dalli`: a stored session that couldn't be decoded (for example after changing the serializer, or removing a class that sessions stored) left the user on a session that was never saved, with no new cookie, so they couldn't log in until the cookie was cleared or the session expired. It's now treated as missing: the request gets a new session and cookie, and a warning is logged
+- A reply value is now checked for the CRLF that should follow it. Values were read by their declared size alone, so a size that didn't match the value (a broken proxy, or an injected reply) was returned as a hit with a `nil` value, or the rest of the value was read as the next reply. These now raise `Dalli::DalliError` and close the connection, as a value size out of range does
+- `get_multi_cas`, and `get_multi` on more than one server, returned a server's keys as misses when it sent nothing within `socket_timeout`, without retrying or counting a failure. They now treat it like any read timeout: the batch is retried, and `Dalli::NetworkError` is raised once the server is marked down, as `get`, `get_multi_with_metadata` and single-server `get_multi` already did. `socket_timeout` also used to cover the whole batch, so a large reply that was still arriving could be cut off, with its remaining keys returned as misses. The wait now starts again whenever data arrives (#1285)
+- A reply key marked base64 that isn't valid base64 raised `ArgumentError` from `get_multi`, `get_multi_cas` and `get_multi_with_metadata`, which callers rescuing `Dalli::DalliError` didn't catch. It now raises `Dalli::DalliError`. Found by the new fuzz targets
+- `set_multi` and `delete_multi` on more than one server retried forever, hanging the caller, when a server accepted their writes and then closed the connection before replying. Each quiet write reset the server's failure count, so it never reached `socket_max_failures`. Only a request whose reply is read resets it now, and the server is marked down and `Dalli::NetworkError` raised, as for `get`
+- Since 5.2.0, on JRuby with TLS, Dalli could wait `socket_timeout` for part of a reply that had already arrived, because jruby-openssl can hold it where neither `IO.select` nor OpenSSL's `pending` reports it. With the multi-get change above (#1285), that wait would have made `get_multi` raise `Dalli::NetworkError`. Dalli now reads on JRuby until the TLS socket would block
+
+Other changes:
+
+- `Rack::Session::Dalli` docs: without `expire_after`, sessions are stored with no expiry and stay in memcached until deleted or evicted
+- Numeric arguments are checked before anything is written, so a bad one raises `ArgumentError` without affecting the connection or earlier writes in a `quiet` block. Integers and Strings of decimal digits (such as `'300'`) are accepted throughout
+  - A TTL (`ttl`, `expires_in`, `vivify_ttl`, `lock_ttl`, `tombstone_ttl`) must fit memcached's signed 32-bit expiration time once a TTL over 30 days is converted to a Unix time. A negative TTL is still accepted and expires the item. A Float or `ActiveSupport::Duration` TTL is still converted with `to_i`
+  - A CAS, an `incr`/`decr` amount or initial value, and a `flush` delay must be from 0 to 2**64 - 1. A Float now raises; `incr(key, 0.9)` used to truncate the amount to 0. A String amount such as `'5'` now works
+  - `fetch`, `fetch_with_lock`, `cas` and `cas!` check `ttl` before running the block, so a bad one raises before any side effects. `fetch` checks it on a hit too
+  - `Dalli::Protocol::TtlSanitizer.sanitize` raises `ArgumentError` for a String that isn't a whole number instead of returning 0
+- Usernames and passwords from `memcached://` URIs and the `:username`/`:password` options are no longer kept in the client's or server's options, so `#inspect` can't show them. Dalli 5 has never used them, and the warning is unchanged
+- A TTL over 30 days is converted to a Unix time once per request, not twice, and the debug message for that conversion is built only when debug logging is on
+- `incr`, `decr`, `append`, `prepend`, `add`, `set_cas`, `replace_cas`, `delete_cas`, `cas`, `cas!`, `fetch_with_lock`, and `get_with_metadata` with `vivify_ttl` or `recache_ttl` now raise `Dalli::NetworkError` when the connection fails after the request was sent, where they used to retry it. Reads, `touch`, `set`, `replace` and `delete` without a CAS, and the multi-key operations are still retried, as is any request that failed before it was sent
+  - `fetch` (which stores a missing value with `add`) and `Rack::Session::Dalli` (which creates sessions with `add`) can see this error too. `Rack::Session::Dalli` handles it like any other memcached error
+  - Before a request on a connection unused for over a second, Dalli checks whether the other end closed it (memcached's `idle_timeout`, a load balancer or a firewall), and if so reconnects before sending. Without this, the first `incr` or `add` after an idle spell raised this error, and any request counted a failure toward `socket_max_failures`
+
 5.2.6
 ==========
 

data/README.md

@@ -96,6 +96,28 @@ end
 
 `drain_deferred_responses` only sends a `noop` to servers that were sent quiet requests since they were last drained, so calling it when nothing is pending costs nothing.
 
+## Network Errors and Retries
+
+When a request fails with a network error (a refused connection, a reset, or a read that times out), Dalli reconnects and retries it. A server that fails `socket_max_failures` times in a row is marked down, and requests to it raise `Dalli::NetworkError` (or, with `failover`, go to another server).
+
+On CRuby, a read that waits longer than `socket_timeout` raises a network error, which is retried and counted like any other. JRuby and TruffleRuby don't time out a blocked socket read, so on those engines a server that accepts a request and never replies blocks the caller indefinitely. Pass `enforce_read_timeout: true` to apply `socket_timeout` there too:
+
+```ruby
+Dalli::Client.new('localhost:11211', socket_timeout: 0.5, enforce_read_timeout: true)
+```
+
+It limits how long Dalli waits for a reply to start, which covers a server that stops answering. A server that stops partway through a reply can still block. It's off by default because the check costs throughput, noticeably on JRuby. It has no effect on CRuby, which doesn't need it, or on TLS connections.
+
+`get_multi` and `get_multi_cas` wait up to `socket_timeout` for any server to send more of its replies, on every engine. The wait starts again each time data arrives, so a large batch can take longer than `socket_timeout` in total. A server that sends nothing for `socket_timeout` counts as a failed read, as it would for `get`.
+
+A request that failed after some of it was sent may have been applied by memcached even though its reply never arrived. Dalli only sends such a request again when applying it twice is harmless: reads, `touch`, `set`, `replace` and `delete` without a CAS, and the multi-key operations. For `incr`, `decr`, `append`, `prepend`, `add`, writes with a CAS (`set_cas`, `replace_cas`, `delete_cas`, `cas`, `cas!`), and `get_with_metadata` with `vivify_ttl` or `recache_ttl` (which `fetch_with_lock` uses), it raises `Dalli::NetworkError` instead, since the write may or may not have happened. A retried `delete` can return false if the first attempt deleted the key.
+
+A connection that sat unused for over a second is checked before the next request. If memcached (with `idle_timeout`), a load balancer or a firewall closed it meanwhile, Dalli reconnects before sending, so the request isn't lost and doesn't count as a failure. A connection dropped without the other end closing it can't be detected this way, and a request on it fails as above.
+
+The multi-key operations (`get_multi`, `get_multi_cas`, `get_multi_with_metadata`, `set_multi` and `delete_multi`) skip a server that is already down when they start: its keys are missing from the result, or go to another server with `failover`. A server that fails during the operation is retried. If the retry can't connect to it, then without `failover` the operation raises `Dalli::NetworkError`, rather than returning that server's keys as misses or leaving them unwritten, and with `failover` its keys go to another server, or are left out when no server is left. If the retry connects but the server fails again, it's marked down partway through the operation, which raises `Dalli::NetworkError` with or without `failover`.
+
+A retried `get_multi` or `get_multi_cas` starts the batch again, so a block passed to either can be yielded a key it was already yielded before the error. Make the block safe to call twice for a key, or call the method without a block and use the Hash it returns. `get_multi_with_metadata` yields only after every server has replied, so its block sees each key once.
+
 ## Security Note
 
 By default, Dalli uses Ruby's Marshal for serialization. Deserializing untrusted data with Marshal can lead to remote code execution. If you cache user-controlled data, consider using a safer serializer:
@@ -152,6 +174,14 @@ A failed TLS handshake counts as a failed connection attempt, like a refused con
 
 The app has already run by then, so whatever it did (database writes, emails, and so on) has happened, and only its response is lost. If a request needs a different outcome, check that the session still exists before making the changes.
 
+Set `expire_after` to give sessions an expiry. It sets both the cookie's expiry and the TTL of the session in memcached. Without it, sessions are stored with no expiry: a session stays in memcached until it's deleted (for example by a logout) or evicted, and a stolen session cookie keeps working until then.
+
+```ruby
+use Rack::Session::Dalli, memcache_server: 'mc.example.com:11211', expire_after: 2 * 60 * 60 # two hours
+```
+
+A stored session that can't be decoded, for example after changing the serializer or removing a class that sessions stored, is treated as missing: the request gets a new, empty session and a new cookie, and Dalli logs a warning.
+
 ## OpenTelemetry Tracing
 
 Dalli automatically instruments operations with [OpenTelemetry](https://opentelemetry.io/) when the SDK is present. No configuration is required - just add the OpenTelemetry gems to your application: