dalli 3.2.13 → 4.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +24 -76
- data/Gemfile +2 -0
- data/README.md +0 -12
- data/lib/dalli/client.rb +5 -20
- data/lib/dalli/compressor.rb +5 -53
- data/lib/dalli/key_manager.rb +4 -30
- data/lib/dalli/protocol/base.rb +9 -16
- data/lib/dalli/protocol/binary/response_processor.rb +4 -20
- data/lib/dalli/protocol/binary.rb +5 -4
- data/lib/dalli/protocol/connection_manager.rb +9 -43
- data/lib/dalli/protocol/meta/key_regularizer.rb +1 -1
- data/lib/dalli/protocol/meta/request_formatter.rb +11 -25
- data/lib/dalli/protocol/meta/response_processor.rb +20 -49
- data/lib/dalli/protocol/meta.rb +8 -4
- data/lib/dalli/protocol/server_config_parser.rb +1 -1
- data/lib/dalli/protocol/ttl_sanitizer.rb +1 -1
- data/lib/dalli/protocol/value_compressor.rb +4 -40
- data/lib/dalli/protocol/value_marshaller.rb +1 -1
- data/lib/dalli/protocol/value_serializer.rb +49 -40
- data/lib/dalli/socket.rb +29 -6
- data/lib/dalli/version.rb +1 -1
- data/lib/dalli.rb +3 -3
- data/lib/rack/session/dalli.rb +42 -7
- metadata +3 -5
- data/lib/dalli/json_serializer.rb +0 -28
- data/lib/dalli/server.rb +0 -6
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 86e39b2869b7c916472e88e6cbbfb6aa1f6533aa35c9e1f1c704cffcbc680f8a
|
|
4
|
+
data.tar.gz: bf2e56610c6bae187d561ccf09105e6f5b4a29eed1308d089745ef22cf5b673b
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: e2faab814abb4b25a53d87048f7e4e81e49c609bd64725cb18607670110da3d5b8038544598c44575831b9038500f9d8220a56ec28c16b9517a21d442212d12b
|
|
7
|
+
data.tar.gz: 3e5a7c326c908d72d2d8ce2211b27f67acebbbf97c279c5faf3ba2914ec7409df7f259a9082d82903cc9ae6b6716a137994aba849321546b8d3d3a35f8f7f5af
|
data/CHANGELOG.md
CHANGED
|
@@ -1,82 +1,30 @@
|
|
|
1
1
|
Dalli Changelog
|
|
2
2
|
=====================
|
|
3
3
|
|
|
4
|
-
|
|
5
|
-
==========
|
|
6
|
-
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
-
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
-
|
|
18
|
-
-
|
|
19
|
-
-
|
|
20
|
-
-
|
|
21
|
-
|
|
22
|
-
-
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
-
|
|
28
|
-
- The pipelined reply parser took any reply without a body for the end of the batch. An error reply for one key (`CLIENT_ERROR`, `SERVER_ERROR`, or `EN` from a proxy) ended it early, and the replies still on the connection were read as the replies to later commands, so a `get` could return another key's value
|
|
29
|
-
- Keys are now measured in bytes as sent (after base64 encoding when the meta protocol needs it). A key under 250 characters but over 250 bytes, for example one with many non-ASCII characters, passed validation:
|
|
30
|
-
- with the meta protocol, memcached rejected it with the error reply above
|
|
31
|
-
- with the binary protocol, memcached dropped the connection, and `get` or `get_multi` with that key retried forever
|
|
32
|
-
- Such keys are now truncated like other long keys; keys that worked before are unchanged
|
|
33
|
-
- The reply parser affects `protocol: :meta` since 3.2.0; the key length affects both protocols; fixed in 5.2.1, 5.1.3, 5.0.9, 4.3.6 and 3.2.12
|
|
34
|
-
- Stop retrying an unresponsive server forever (GHSA-4qp6-2jcr-596v)
|
|
35
|
-
- A server that accepted connections but never answered (or dropped the connection on a request) was retried forever, hanging the caller, because each successful reconnect reset the failure count. Requests now fail after `socket_max_failures` attempts and the server is marked down, as when it can't be reached at all; it gets a full set of attempts again after `down_retry_delay`
|
|
36
|
-
- Affects all versions; fixed in 5.2.1, 5.1.3, 5.0.9, 4.3.6 and 3.2.12
|
|
37
|
-
- Limit decompressed and reply value sizes (GHSA-3553-vcg5-72jw)
|
|
38
|
-
- Values flagged as compressed were inflated without limit, so a small stored value could expand to gigabytes on read, whatever the client's `compress` or `serializer` settings. The new `decompressed_max_bytes` option (default 128 MiB; `nil` disables it) makes a read that would pass it raise `Dalli::UnmarshalError`. Custom compressors whose `decompress` takes only the data keep working, without the limit
|
|
39
|
-
- The value size in a reply (a meta `VA` size or a binary body length) was used to read or buffer that many bytes, so a malicious server could make the client allocate gigabytes. Sizes over 1 GiB, memcached's largest item, now raise `Dalli::DalliError` before reading
|
|
40
|
-
- Affects all versions; fixed in 5.2.1, 5.1.3, 5.0.9, 4.3.6 and 3.2.12
|
|
41
|
-
- Honor per-request `raw: true` on reads, and add `Dalli::JSONSerializer` (GHSA-wr87-m4jw-29x5)
|
|
42
|
-
- `get`, `gat` and `fetch` ignored a per-request `raw: true` and deserialized the value according to its stored flags, so a caller who asked for raw bytes could still have `Marshal.load` run on data someone else wrote to memcached. They now return the stored bytes
|
|
43
|
-
- The `raw` part affects all versions
|
|
44
|
-
- `serializer: JSON` reads values with `JSON.load`, which on json gem versions before 3.0 creates an object of the class named in a stored `json_class` key when the json additions are loaded. The new `Dalli::JSONSerializer` reads with `JSON.parse` and only returns plain JSON types. The README now recommends it; `serializer: JSON` itself is unchanged
|
|
45
|
-
- The `serializer: JSON` part affects any version used with the json gem before 3.0
|
|
46
|
-
- Keep a forked child from closing the parent's TLS session (GHSA-w39f-xq2m-4g8x)
|
|
47
|
-
- With TLS, a forked child that closed the client (or reconnected after detecting the fork) sent a TLS close on the connection the parent was still using, ending the parent's session. A forked child now closes only its own copy of the socket
|
|
48
|
-
- Affects all versions with TLS; fixed in 5.2.1, 5.1.3, 5.0.9, 4.3.6 and 3.2.12
|
|
49
|
-
|
|
50
|
-
- Fix three flaky tests: the `KeyManager` namespace key-length test, the Rack session freshness test, and `MemcachedManager`'s start and stop handling (backport of #1141, #1183)
|
|
51
|
-
- Fix flaky failover tests: move their ports out of Linux's ephemeral port range, and wait for memcached to accept connections after starting it (backport of #1184)
|
|
52
|
-
|
|
53
|
-
3.2.11
|
|
54
|
-
==========
|
|
55
|
-
|
|
56
|
-
- Fix multi-server `get_multi` stopping at an empty value (#1170)
|
|
57
|
-
- The pipelined reply parser took a hit on a zero-length value (`VA 0`) for the terminating `MN`, so that server's remaining keys were silently missing from the result
|
|
58
|
-
- Its value terminator and any replies not yet parsed stayed on the connection and were read as the replies to later commands on it. Those commands could fail with `Dalli::DalliError: Response error`, or a single-key `get` could silently return another key's value
|
|
59
|
-
- An empty value is now returned as `''`, like a single-key `get` and single-server `get_multi`
|
|
60
|
-
- Affects only clients using `protocol: :meta`; the default binary protocol is not affected
|
|
61
|
-
- Thanks to Julian Richard Contreras for this contribution
|
|
62
|
-
|
|
63
|
-
3.2.10
|
|
64
|
-
==========
|
|
65
|
-
|
|
66
|
-
- Fix `Rack::Session::Dalli` with connection_pool 3.x, which only accepts keyword arguments (backport of #1051)
|
|
67
|
-
- Add `logger` as a runtime dependency, since it is no longer a default gem as of Ruby 4.0 (backport of #1010, olleolleolle)
|
|
68
|
-
- CI: patch memcached 1.5.22 so it builds on ubuntu-22.04, and run tests there (backport of #1033)
|
|
69
|
-
|
|
70
|
-
3.2.9
|
|
71
|
-
==========
|
|
72
|
-
|
|
73
|
-
Security:
|
|
74
|
-
|
|
75
|
-
- Fix memcached command injection through the `incr`/`decr` default with the meta protocol (GHSA-6wmv-xq9m-fmp7)
|
|
76
|
-
- With `protocol: :meta`, the `default` argument of `incr`/`decr` was written into the command without conversion, so a String containing CRLF injected additional memcached commands (e.g. `set`, `flush_all`) on the connection. The default binary protocol is not affected
|
|
77
|
-
- `default` must now be an Integer, or a String of decimal digits; anything else raises `ArgumentError` before a request is sent
|
|
78
|
-
- As defense in depth, the meta `RequestFormatter` now converts every numeric flag it writes (`D`, `J`, `N`, `T`) to an Integer
|
|
79
|
-
- Thanks to oss-security-shop for the report
|
|
4
|
+
4.0.0
|
|
5
|
+
==========
|
|
6
|
+
|
|
7
|
+
BREAKING CHANGES:
|
|
8
|
+
|
|
9
|
+
- Require Ruby 3.1+ (dropped support for Ruby 2.6, 2.7, and 3.0)
|
|
10
|
+
- Removed `Dalli::Server` deprecated alias - use `Dalli::Protocol::Binary` instead
|
|
11
|
+
- Removed `:compression` option - use `:compress` instead
|
|
12
|
+
- Removed `close_on_fork` method - use `reconnect_on_fork` instead
|
|
13
|
+
|
|
14
|
+
Other changes:
|
|
15
|
+
|
|
16
|
+
- Add security warning when using default Marshal serializer (silence with `silence_marshal_warning: true`)
|
|
17
|
+
- Add defense-in-depth input validation for stats command arguments
|
|
18
|
+
- Add `string_fastpath` option to skip serialization for simple strings (byroot)
|
|
19
|
+
- Meta protocol set performance improvement (danmayer)
|
|
20
|
+
- Fix connection_pool 3.0 compatibility for Rack session store
|
|
21
|
+
- Fix session recovery after deletion (stengineering0)
|
|
22
|
+
- Fix cannot read response data included terminator `\r\n` when use meta protocol (matsubara0507)
|
|
23
|
+
- Support SERVER_ERROR response from Memcached as per the [memcached spec](https://github.com/memcached/memcached/blob/e43364402195c8e822bb8f88755a60ab8bbed62a/doc/protocol.txt#L172) (grcooper)
|
|
24
|
+
- Update Socket timeout handling to use Socket#timeout= when available (nickamorim)
|
|
25
|
+
- Serializer: reraise all .load errors as UnmarshalError (olleolleolle)
|
|
26
|
+
- Reconnect gracefully when a fork is detected instead of crashing (PatrickTulskie)
|
|
27
|
+
- Update CI to test against memcached 1.6.40
|
|
80
28
|
|
|
81
29
|
3.2.8
|
|
82
30
|
==========
|
data/Gemfile
CHANGED
data/README.md
CHANGED
|
@@ -25,18 +25,6 @@ The name is a variant of Salvador Dali for his famous painting [The Persistence
|
|
|
25
25
|
* [Forum](https://github.com/petergoldstein/dalli/discussions/categories/q-a) - If you have questions about Dalli, please post them here.
|
|
26
26
|
* [Client API](https://www.rubydoc.info/gems/dalli) - Ruby documentation for the `Dalli::Client` API
|
|
27
27
|
|
|
28
|
-
## Security Note
|
|
29
|
-
|
|
30
|
-
By default, Dalli uses Ruby's Marshal for serialization. Deserializing untrusted data with Marshal can lead to remote code execution. If you cache user-controlled data, or your memcached server could be written to by someone else, consider using a safer serializer:
|
|
31
|
-
|
|
32
|
-
```ruby
|
|
33
|
-
Dalli::Client.new('localhost:11211', serializer: Dalli::JSONSerializer)
|
|
34
|
-
```
|
|
35
|
-
|
|
36
|
-
`Dalli::JSONSerializer` reads values with `JSON.parse`, so it only ever returns plain hashes, arrays, strings, numbers, booleans and `nil`. Passing the `JSON` module itself (`serializer: JSON`) reads values with `JSON.load`, which can create objects of other classes from a stored `json_class` key and is not recommended for data an attacker might write.
|
|
37
|
-
|
|
38
|
-
Per-request `raw: true` returns the stored bytes without deserializing or decompressing them, on every read method that accepts it (`get` and `fetch`).
|
|
39
|
-
|
|
40
28
|
## Development
|
|
41
29
|
|
|
42
30
|
After checking out the repo, run `bin/setup` to install dependencies. You can run `bin/console` for an interactive prompt that will allow you to experiment.
|
data/lib/dalli/client.rb
CHANGED
|
@@ -155,8 +155,8 @@ module Dalli
|
|
|
155
155
|
# - nil if the key did not exist.
|
|
156
156
|
# - false if the value was changed by someone else.
|
|
157
157
|
# - true if the value was successfully updated.
|
|
158
|
-
def cas(key, ttl = nil, req_options = nil, &
|
|
159
|
-
cas_core(key, false, ttl, req_options, &
|
|
158
|
+
def cas(key, ttl = nil, req_options = nil, &)
|
|
159
|
+
cas_core(key, false, ttl, req_options, &)
|
|
160
160
|
end
|
|
161
161
|
|
|
162
162
|
##
|
|
@@ -166,8 +166,8 @@ module Dalli
|
|
|
166
166
|
# Returns:
|
|
167
167
|
# - false if the value was changed by someone else.
|
|
168
168
|
# - true if the value was successfully updated.
|
|
169
|
-
def cas!(key, ttl = nil, req_options = nil, &
|
|
170
|
-
cas_core(key, true, ttl, req_options, &
|
|
169
|
+
def cas!(key, ttl = nil, req_options = nil, &)
|
|
170
|
+
cas_core(key, true, ttl, req_options, &)
|
|
171
171
|
end
|
|
172
172
|
|
|
173
173
|
##
|
|
@@ -257,8 +257,6 @@ module Dalli
|
|
|
257
257
|
##
|
|
258
258
|
# Incr adds the given amount to the counter on the memcached server.
|
|
259
259
|
# Amt must be a positive integer value.
|
|
260
|
-
# Default, if given, must be an Integer (or a String of decimal digits);
|
|
261
|
-
# anything else raises ArgumentError.
|
|
262
260
|
#
|
|
263
261
|
# If default is nil, the counter must already exist or the operation
|
|
264
262
|
# will fail and will return nil. Otherwise this method will return
|
|
@@ -271,7 +269,6 @@ module Dalli
|
|
|
271
269
|
# If the value already exists, it must have been set with raw: true
|
|
272
270
|
def incr(key, amt = 1, ttl = nil, default = nil)
|
|
273
271
|
check_positive!(amt)
|
|
274
|
-
validate_integer!(:default, default)
|
|
275
272
|
|
|
276
273
|
perform(:incr, key, amt.to_i, ttl_or_default(ttl), default)
|
|
277
274
|
end
|
|
@@ -279,8 +276,6 @@ module Dalli
|
|
|
279
276
|
##
|
|
280
277
|
# Decr subtracts the given amount from the counter on the memcached server.
|
|
281
278
|
# Amt must be a positive integer value.
|
|
282
|
-
# Default, if given, must be an Integer (or a String of decimal digits);
|
|
283
|
-
# anything else raises ArgumentError.
|
|
284
279
|
#
|
|
285
280
|
# memcached counters are unsigned and cannot hold negative values. Calling
|
|
286
281
|
# decr on a counter which is 0 will just return 0.
|
|
@@ -296,7 +291,6 @@ module Dalli
|
|
|
296
291
|
# If the value already exists, it must have been set with raw: true
|
|
297
292
|
def decr(key, amt = 1, ttl = nil, default = nil)
|
|
298
293
|
check_positive!(amt)
|
|
299
|
-
validate_integer!(:default, default)
|
|
300
294
|
|
|
301
295
|
perform(:decr, key, amt.to_i, ttl_or_default(ttl), default)
|
|
302
296
|
end
|
|
@@ -379,17 +373,8 @@ module Dalli
|
|
|
379
373
|
raise ArgumentError, "Positive values only: #{amt}" if amt.negative?
|
|
380
374
|
end
|
|
381
375
|
|
|
382
|
-
# Numeric arguments that become meta protocol flags (GHSA-6wmv-xq9m-fmp7).
|
|
383
|
-
# Checked before the request starts so a bad value raises a clean
|
|
384
|
-
# ArgumentError, instead of Protocol::Base#request logging it as
|
|
385
|
-
# unexpected and closing the connection.
|
|
386
|
-
def validate_integer!(name, value)
|
|
387
|
-
Dalli::Protocol::Meta::RequestFormatter.integer_flag(name, value) unless value.nil?
|
|
388
|
-
end
|
|
389
|
-
|
|
390
376
|
def cas_core(key, always_set, ttl = nil, req_options = nil)
|
|
391
|
-
|
|
392
|
-
(value, cas) = perform(:cas, key, req_options)
|
|
377
|
+
(value, cas) = perform(:cas, key)
|
|
393
378
|
return if value.nil? && !always_set
|
|
394
379
|
|
|
395
380
|
newvalue = yield(value)
|
data/lib/dalli/compressor.rb
CHANGED
|
@@ -13,54 +13,8 @@ module Dalli
|
|
|
13
13
|
Zlib::Deflate.deflate(data)
|
|
14
14
|
end
|
|
15
15
|
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
# Compressed input is fed to zlib this many bytes at a time. MRI's zlib
|
|
19
|
-
# yields the output in small chunks; one that returns it instead still
|
|
20
|
-
# produces at most MAX_EXPANSION times this per step.
|
|
21
|
-
INFLATE_SLICE_BYTES = 64 * 1024
|
|
22
|
-
|
|
23
|
-
# max_bytes caps the decompressed size; inflating stops and raises as soon
|
|
24
|
-
# as the output passes it, so a small compressed value can't expand into
|
|
25
|
-
# gigabytes of memory.
|
|
26
|
-
def self.decompress(data, max_bytes: nil)
|
|
27
|
-
return Zlib::Inflate.inflate(data) unless may_exceed?(data, max_bytes)
|
|
28
|
-
|
|
29
|
-
inflate_within_limit(Zlib::Inflate.new, data, max_bytes)
|
|
30
|
-
end
|
|
31
|
-
|
|
32
|
-
# False when even the largest possible expansion stays within max_bytes,
|
|
33
|
-
# so the value can be inflated in one step
|
|
34
|
-
def self.may_exceed?(data, max_bytes)
|
|
35
|
-
max_bytes ? data.bytesize * MAX_EXPANSION > max_bytes : false
|
|
36
|
-
end
|
|
37
|
-
|
|
38
|
-
def self.inflate_within_limit(inflater, data, max_bytes)
|
|
39
|
-
out = String.new(encoding: Encoding::BINARY)
|
|
40
|
-
0.step(data.bytesize - 1, INFLATE_SLICE_BYTES) do |pos|
|
|
41
|
-
inflate_slice(inflater, data.byteslice(pos, INFLATE_SLICE_BYTES), out, max_bytes)
|
|
42
|
-
# Anything after the end of the stream isn't part of the value, as
|
|
43
|
-
# with Zlib::Inflate.inflate; finish would return it as output
|
|
44
|
-
break if inflater.finished?
|
|
45
|
-
end
|
|
46
|
-
return out if inflater.finished?
|
|
47
|
-
|
|
48
|
-
append_within_limit(out, inflater.finish, max_bytes)
|
|
49
|
-
ensure
|
|
50
|
-
inflater.close
|
|
51
|
-
end
|
|
52
|
-
|
|
53
|
-
# Both yielded and returned output count toward the limit
|
|
54
|
-
def self.inflate_slice(inflater, slice, out, max_bytes)
|
|
55
|
-
rest = inflater.inflate(slice) { |chunk| append_within_limit(out, chunk, max_bytes) }
|
|
56
|
-
append_within_limit(out, rest, max_bytes)
|
|
57
|
-
end
|
|
58
|
-
|
|
59
|
-
def self.append_within_limit(out, chunk, max_bytes)
|
|
60
|
-
out << chunk if chunk
|
|
61
|
-
return out if out.bytesize <= max_bytes
|
|
62
|
-
|
|
63
|
-
raise Dalli::UnmarshalError, "Decompressed value exceeds #{max_bytes} bytes (decompressed_max_bytes)"
|
|
16
|
+
def self.decompress(data)
|
|
17
|
+
Zlib::Inflate.inflate(data)
|
|
64
18
|
end
|
|
65
19
|
end
|
|
66
20
|
|
|
@@ -78,11 +32,9 @@ module Dalli
|
|
|
78
32
|
io.string
|
|
79
33
|
end
|
|
80
34
|
|
|
81
|
-
def self.decompress(data
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
# MAX_WBITS + 16 makes zlib read the gzip format (and check its CRC)
|
|
85
|
-
Compressor.inflate_within_limit(Zlib::Inflate.new(Zlib::MAX_WBITS + 16), data, max_bytes)
|
|
35
|
+
def self.decompress(data)
|
|
36
|
+
io = StringIO.new(data, 'rb')
|
|
37
|
+
Zlib::GzipReader.new(io).read
|
|
86
38
|
end
|
|
87
39
|
end
|
|
88
40
|
end
|
data/lib/dalli/key_manager.rb
CHANGED
|
@@ -30,13 +30,10 @@ module Dalli
|
|
|
30
30
|
|
|
31
31
|
def initialize(client_options)
|
|
32
32
|
@key_options =
|
|
33
|
-
DEFAULTS.merge(client_options.
|
|
33
|
+
DEFAULTS.merge(client_options.slice(*OPTIONS))
|
|
34
34
|
validate_digest_class_option(@key_options)
|
|
35
35
|
|
|
36
36
|
@namespace = namespace_from_options
|
|
37
|
-
# The meta protocol base64-encodes keys with whitespace or non-ASCII
|
|
38
|
-
# characters, so their length on the wire differs from their length here
|
|
39
|
-
@meta_protocol = client_options[:protocol]&.to_s == 'meta'
|
|
40
37
|
end
|
|
41
38
|
|
|
42
39
|
##
|
|
@@ -54,7 +51,7 @@ module Dalli
|
|
|
54
51
|
raise ArgumentError, 'key cannot be blank' unless key&.length&.positive?
|
|
55
52
|
|
|
56
53
|
key = key_with_namespace(key)
|
|
57
|
-
|
|
54
|
+
key.length > MAX_KEY_LENGTH ? truncated_key(key) : key
|
|
58
55
|
end
|
|
59
56
|
|
|
60
57
|
##
|
|
@@ -80,7 +77,7 @@ module Dalli
|
|
|
80
77
|
def namespace_regexp
|
|
81
78
|
return /\A#{Regexp.escape(evaluate_namespace)}:/ if namespace.is_a?(Proc)
|
|
82
79
|
|
|
83
|
-
@namespace_regexp ||= /\A#{Regexp.escape(namespace)}
|
|
80
|
+
@namespace_regexp ||= /\A#{Regexp.escape(namespace)}:/ unless namespace.nil?
|
|
84
81
|
end
|
|
85
82
|
|
|
86
83
|
def validate_digest_class_option(opts)
|
|
@@ -110,30 +107,7 @@ module Dalli
|
|
|
110
107
|
##
|
|
111
108
|
def truncated_key(key)
|
|
112
109
|
digest = digest_class.hexdigest(key)
|
|
113
|
-
|
|
114
|
-
truncated = "#{prefix}#{TRUNCATED_KEY_SEPARATOR}#{digest}"
|
|
115
|
-
# A prefix with whitespace or non-ASCII characters sends the whole key
|
|
116
|
-
# base64-encoded, which can still be too long on the wire; shorten it.
|
|
117
|
-
while wire_length(truncated) > MAX_KEY_LENGTH && !prefix.empty?
|
|
118
|
-
prefix = prefix[0...-1]
|
|
119
|
-
truncated = "#{prefix}#{TRUNCATED_KEY_SEPARATOR}#{digest}"
|
|
120
|
-
end
|
|
121
|
-
return truncated if wire_length(truncated) <= MAX_KEY_LENGTH
|
|
122
|
-
|
|
123
|
-
# Only possible with a digest_class whose digests aren't short hex strings
|
|
124
|
-
raise ArgumentError, "#{digest_class} digests are too long to fit in a truncated key"
|
|
125
|
-
end
|
|
126
|
-
|
|
127
|
-
# memcached's key limit applies to the key as sent: in bytes, and, with
|
|
128
|
-
# the meta protocol, after base64 encoding when the key needs it. A key
|
|
129
|
-
# within the limit in characters can still be too long. With the meta
|
|
130
|
-
# protocol memcached rejects it with a CLIENT_ERROR that must never reach
|
|
131
|
-
# a pipelined get; with the binary protocol it drops the connection.
|
|
132
|
-
def wire_length(key)
|
|
133
|
-
return key.bytesize unless @meta_protocol
|
|
134
|
-
|
|
135
|
-
encoded, = Dalli::Protocol::Meta::KeyRegularizer.encode(key)
|
|
136
|
-
encoded.bytesize
|
|
110
|
+
"#{key[0, prefix_length(digest)]}#{TRUNCATED_KEY_SEPARATOR}#{digest}"
|
|
137
111
|
end
|
|
138
112
|
|
|
139
113
|
def prefix_length(digest)
|
data/lib/dalli/protocol/base.rb
CHANGED
|
@@ -27,12 +27,6 @@ module Dalli
|
|
|
27
27
|
@connection_manager = ConnectionManager.new(hostname, port, socket_type, @options)
|
|
28
28
|
end
|
|
29
29
|
|
|
30
|
-
# True when this request passed raw: true. A stored value is then
|
|
31
|
-
# returned as stored, never deserialized or decompressed.
|
|
32
|
-
def raw_request?(options)
|
|
33
|
-
options.is_a?(Hash) && options[:raw] ? true : false
|
|
34
|
-
end
|
|
35
|
-
|
|
36
30
|
# Chokepoint method for error handling and ensuring liveness
|
|
37
31
|
def request(opkey, *args)
|
|
38
32
|
verify_state(opkey)
|
|
@@ -124,16 +118,13 @@ module Dalli
|
|
|
124
118
|
def pipeline_abort
|
|
125
119
|
response_buffer.clear
|
|
126
120
|
@connection_manager.abort_request!
|
|
127
|
-
return unless connected?
|
|
128
|
-
|
|
129
|
-
# Closes the connection, which ensures that our connection
|
|
130
|
-
# clean state for future requests
|
|
131
|
-
|
|
132
|
-
|
|
133
|
-
|
|
134
|
-
Dalli.logger.warn { "#{name} didn't finish a get_multi within socket_timeout; closing the connection" }
|
|
135
|
-
@connection_manager.close
|
|
136
|
-
nil
|
|
121
|
+
return true unless connected?
|
|
122
|
+
|
|
123
|
+
# Closes the connection, which ensures that our connection
|
|
124
|
+
# is in a clean state for future requests
|
|
125
|
+
@connection_manager.error_on_request!('External timeout')
|
|
126
|
+
rescue NetworkError
|
|
127
|
+
true
|
|
137
128
|
end
|
|
138
129
|
|
|
139
130
|
# Did the last call to #pipeline_response_setup complete successfully?
|
|
@@ -163,6 +154,8 @@ module Dalli
|
|
|
163
154
|
private
|
|
164
155
|
|
|
165
156
|
ALLOWED_QUIET_OPS = %i[add replace set delete incr decr append prepend flush noop].freeze
|
|
157
|
+
private_constant :ALLOWED_QUIET_OPS
|
|
158
|
+
|
|
166
159
|
def verify_allowed_quiet!(opkey)
|
|
167
160
|
return if ALLOWED_QUIET_OPS.include?(opkey)
|
|
168
161
|
|
|
@@ -66,9 +66,7 @@ module Dalli
|
|
|
66
66
|
raise Dalli::DalliError, "Response error #{resp_header.status}: #{RESPONSE_CODES[resp_header.status]}"
|
|
67
67
|
end
|
|
68
68
|
|
|
69
|
-
|
|
70
|
-
# decompressing it, whatever flags it was stored with
|
|
71
|
-
def get(cache_nils: false, raw: false)
|
|
69
|
+
def get(cache_nils: false)
|
|
72
70
|
resp_header, body = read_response
|
|
73
71
|
|
|
74
72
|
return false if resp_header.not_stored? # Not stored, normal status for add operation
|
|
@@ -77,7 +75,7 @@ module Dalli
|
|
|
77
75
|
raise_on_not_ok!(resp_header)
|
|
78
76
|
return true unless body
|
|
79
77
|
|
|
80
|
-
unpack_response_body(resp_header, body,
|
|
78
|
+
unpack_response_body(resp_header, body, true).last
|
|
81
79
|
end
|
|
82
80
|
|
|
83
81
|
##
|
|
@@ -102,8 +100,7 @@ module Dalli
|
|
|
102
100
|
true
|
|
103
101
|
end
|
|
104
102
|
|
|
105
|
-
|
|
106
|
-
def data_cas_response(raw: false)
|
|
103
|
+
def data_cas_response
|
|
107
104
|
resp_header, body = read_response
|
|
108
105
|
return [nil, resp_header.cas] if resp_header.not_found?
|
|
109
106
|
return [nil, false] if resp_header.not_stored?
|
|
@@ -111,7 +108,7 @@ module Dalli
|
|
|
111
108
|
raise_on_not_ok!(resp_header)
|
|
112
109
|
return [nil, resp_header.cas] unless body
|
|
113
110
|
|
|
114
|
-
[unpack_response_body(resp_header, body,
|
|
111
|
+
[unpack_response_body(resp_header, body, true).last, resp_header.cas]
|
|
115
112
|
end
|
|
116
113
|
|
|
117
114
|
# Returns the new value for the key, if found and updated
|
|
@@ -212,17 +209,6 @@ module Dalli
|
|
|
212
209
|
# The remaining three values in the array are the ResponseHeader,
|
|
213
210
|
# key, and value.
|
|
214
211
|
##
|
|
215
|
-
# memcached can't store an item larger than 1 GiB (its -I maximum). A
|
|
216
|
-
# pipelined reply claiming an impossible size would otherwise have the
|
|
217
|
-
# buffer wait for (and accumulate) that many bytes.
|
|
218
|
-
MAX_BODY_BYTES = 1024 * 1024 * 1024
|
|
219
|
-
|
|
220
|
-
def check_body_size!(size)
|
|
221
|
-
return if size <= MAX_BODY_BYTES
|
|
222
|
-
|
|
223
|
-
raise Dalli::DalliError, "Reply body size #{size} is out of range"
|
|
224
|
-
end
|
|
225
|
-
|
|
226
212
|
def getk_response_from_buffer(buf)
|
|
227
213
|
# There's no header in the buffer, so don't advance
|
|
228
214
|
return [0, nil, nil, nil, nil] unless contains_header?(buf)
|
|
@@ -236,8 +222,6 @@ module Dalli
|
|
|
236
222
|
# error response that needs to be discarded.
|
|
237
223
|
return [ResponseHeader::SIZE, resp_header.ok?, resp_header.cas, nil, nil] if body_len.zero?
|
|
238
224
|
|
|
239
|
-
check_body_size!(body_len)
|
|
240
|
-
|
|
241
225
|
resp_size = ResponseHeader::SIZE + body_len
|
|
242
226
|
# The header is in the buffer, but the body is not. As we don't have
|
|
243
227
|
# a complete response, don't advance the buffer
|
|
@@ -22,7 +22,7 @@ module Dalli
|
|
|
22
22
|
def get(key, options = nil)
|
|
23
23
|
req = RequestFormatter.standard_request(opkey: :get, key: key)
|
|
24
24
|
write(req)
|
|
25
|
-
response_processor.get(cache_nils: cache_nils?(options)
|
|
25
|
+
response_processor.get(cache_nils: cache_nils?(options))
|
|
26
26
|
end
|
|
27
27
|
|
|
28
28
|
def quiet_get_request(key)
|
|
@@ -33,7 +33,7 @@ module Dalli
|
|
|
33
33
|
ttl = TtlSanitizer.sanitize(ttl)
|
|
34
34
|
req = RequestFormatter.standard_request(opkey: :gat, key: key, ttl: ttl)
|
|
35
35
|
write(req)
|
|
36
|
-
response_processor.get(cache_nils: cache_nils?(options)
|
|
36
|
+
response_processor.get(cache_nils: cache_nils?(options))
|
|
37
37
|
end
|
|
38
38
|
|
|
39
39
|
def touch(key, ttl)
|
|
@@ -44,10 +44,10 @@ module Dalli
|
|
|
44
44
|
|
|
45
45
|
# TODO: This is confusing, as there's a cas command in memcached
|
|
46
46
|
# and this isn't it. Maybe rename? Maybe eliminate?
|
|
47
|
-
def cas(key
|
|
47
|
+
def cas(key)
|
|
48
48
|
req = RequestFormatter.standard_request(opkey: :get, key: key)
|
|
49
49
|
write(req)
|
|
50
|
-
response_processor.data_cas_response
|
|
50
|
+
response_processor.data_cas_response
|
|
51
51
|
end
|
|
52
52
|
|
|
53
53
|
# Storage Commands
|
|
@@ -119,6 +119,7 @@ module Dalli
|
|
|
119
119
|
# if the key doesn't already exist, rather than
|
|
120
120
|
# setting the initial value
|
|
121
121
|
NOT_FOUND_EXPIRY = 0xFFFFFFFF
|
|
122
|
+
private_constant :NOT_FOUND_EXPIRY
|
|
122
123
|
|
|
123
124
|
def decr_incr(opkey, key, count, ttl, initial)
|
|
124
125
|
expiry = initial ? TtlSanitizer.sanitize(ttl) : NOT_FOUND_EXPIRY
|
|
@@ -38,7 +38,6 @@ module Dalli
|
|
|
38
38
|
@sock = nil
|
|
39
39
|
@pid = nil
|
|
40
40
|
|
|
41
|
-
@fail_count = 0
|
|
42
41
|
reset_down_info
|
|
43
42
|
end
|
|
44
43
|
|
|
@@ -54,7 +53,7 @@ module Dalli
|
|
|
54
53
|
Dalli.logger.debug { "Dalli::Server#connect #{name}" }
|
|
55
54
|
|
|
56
55
|
@sock = memcached_socket
|
|
57
|
-
@pid =
|
|
56
|
+
@pid = PIDCache.pid
|
|
58
57
|
@request_in_progress = false
|
|
59
58
|
rescue SystemCallError, *TIMEOUT_ERRORS, EOFError, SocketError => e
|
|
60
59
|
# SocketError = DNS resolution failure
|
|
@@ -85,8 +84,6 @@ module Dalli
|
|
|
85
84
|
def down!
|
|
86
85
|
close
|
|
87
86
|
log_down_detected
|
|
88
|
-
# Once down_retry_delay passes, the server gets a full set of attempts
|
|
89
|
-
@fail_count = 0
|
|
90
87
|
|
|
91
88
|
@error = $ERROR_INFO&.class&.name
|
|
92
89
|
@msg ||= $ERROR_INFO&.message
|
|
@@ -103,20 +100,20 @@ module Dalli
|
|
|
103
100
|
|
|
104
101
|
def confirm_ready!
|
|
105
102
|
close if request_in_progress?
|
|
106
|
-
|
|
103
|
+
reconnect_on_fork if fork_detected?
|
|
107
104
|
end
|
|
108
105
|
|
|
109
106
|
def confirm_in_progress!
|
|
110
107
|
raise '[Dalli] No request in progress. This may be a bug in Dalli.' unless request_in_progress?
|
|
111
108
|
|
|
112
|
-
|
|
109
|
+
reconnect_on_fork if fork_detected?
|
|
113
110
|
end
|
|
114
111
|
|
|
115
112
|
def close
|
|
116
113
|
return unless @sock
|
|
117
114
|
|
|
118
115
|
begin
|
|
119
|
-
|
|
116
|
+
@sock.close
|
|
120
117
|
rescue StandardError
|
|
121
118
|
nil
|
|
122
119
|
end
|
|
@@ -125,13 +122,6 @@ module Dalli
|
|
|
125
122
|
abort_request!
|
|
126
123
|
end
|
|
127
124
|
|
|
128
|
-
# A forked child shares the parent's connection, so it closes only its
|
|
129
|
-
# own file descriptor. Closing the TLS socket itself would send
|
|
130
|
-
# close_notify and end the parent's TLS session.
|
|
131
|
-
def close_socket
|
|
132
|
-
fork_detected? ? @sock.to_io.close : @sock.close
|
|
133
|
-
end
|
|
134
|
-
|
|
135
125
|
def connected?
|
|
136
126
|
!@sock.nil?
|
|
137
127
|
end
|
|
@@ -150,11 +140,6 @@ module Dalli
|
|
|
150
140
|
raise '[Dalli] No request in progress. This may be a bug in Dalli.' unless @request_in_progress
|
|
151
141
|
|
|
152
142
|
@request_in_progress = false
|
|
153
|
-
# A completed request is what proves the server healthy again, so the
|
|
154
|
-
# failure count resets here rather than on reconnect: a server that
|
|
155
|
-
# accepts connections but never answers would otherwise reset it on
|
|
156
|
-
# every retry and never reach socket_max_failures.
|
|
157
|
-
@fail_count = 0
|
|
158
143
|
end
|
|
159
144
|
|
|
160
145
|
def abort_request!
|
|
@@ -169,19 +154,7 @@ module Dalli
|
|
|
169
154
|
error_on_request!(e)
|
|
170
155
|
end
|
|
171
156
|
|
|
172
|
-
# memcached can't store an item larger than 1 GiB (its -I maximum), so a
|
|
173
|
-
# reply claiming more (or a negative size) is malformed or hostile.
|
|
174
|
-
# Reading allocates the full count up front, so check before reading.
|
|
175
|
-
MAX_READ_BYTES = (1024 * 1024 * 1024) + 2 # plus the value's trailing "\r\n"
|
|
176
|
-
|
|
177
|
-
def check_read_size!(count)
|
|
178
|
-
return if count.between?(0, MAX_READ_BYTES)
|
|
179
|
-
|
|
180
|
-
raise Dalli::DalliError, "Reply size #{count} from #{name} is out of range"
|
|
181
|
-
end
|
|
182
|
-
|
|
183
157
|
def read(count)
|
|
184
|
-
check_read_size!(count)
|
|
185
158
|
@sock.readfull(count)
|
|
186
159
|
rescue SystemCallError, *TIMEOUT_ERRORS, EOFError => e
|
|
187
160
|
error_on_request!(e)
|
|
@@ -222,9 +195,8 @@ module Dalli
|
|
|
222
195
|
raise Dalli::NetworkError, message
|
|
223
196
|
end
|
|
224
197
|
|
|
225
|
-
# Called on connect. Deliberately leaves @fail_count alone; see
|
|
226
|
-
# finish_request!.
|
|
227
198
|
def reset_down_info
|
|
199
|
+
@fail_count = 0
|
|
228
200
|
@down_at = nil
|
|
229
201
|
@last_down_at = nil
|
|
230
202
|
@msg = nil
|
|
@@ -240,28 +212,22 @@ module Dalli
|
|
|
240
212
|
end
|
|
241
213
|
|
|
242
214
|
def log_warn_message(err_or_string)
|
|
243
|
-
detail = err_or_string.is_a?(String) ? err_or_string : "#{err_or_string.class}: #{err_or_string.message}"
|
|
244
215
|
Dalli.logger.warn do
|
|
245
216
|
detail = err_or_string.is_a?(String) ? err_or_string : "#{err_or_string.class}: #{err_or_string.message}"
|
|
246
217
|
"#{name} failed (count: #{@fail_count}) #{detail}"
|
|
247
218
|
end
|
|
248
219
|
end
|
|
249
220
|
|
|
250
|
-
def
|
|
221
|
+
def reconnect_on_fork
|
|
251
222
|
message = 'Fork detected, re-connecting child process...'
|
|
252
223
|
Dalli.logger.info { message }
|
|
253
|
-
#
|
|
254
|
-
# connection, setting us up for reconnect on next request.
|
|
224
|
+
# Close socket on a fork and reconnect immediately
|
|
255
225
|
close
|
|
256
|
-
|
|
226
|
+
establish_connection
|
|
257
227
|
end
|
|
258
228
|
|
|
259
229
|
def fork_detected?
|
|
260
|
-
|
|
261
|
-
# Process._fork hook, and another library's fork hook (such as
|
|
262
|
-
# connection_pool closing its connections) can run in the child before
|
|
263
|
-
# it, which would close the parent's TLS session as if it were ours.
|
|
264
|
-
@pid && @pid != Process.pid
|
|
230
|
+
@pid && @pid != PIDCache.pid
|
|
265
231
|
end
|
|
266
232
|
|
|
267
233
|
def log_down_detected
|
|
@@ -10,7 +10,7 @@ module Dalli
|
|
|
10
10
|
# memcached supports the use of base64 hashes for keys containing
|
|
11
11
|
# whitespace or non-ASCII characters, provided the 'b' flag is included in the request.
|
|
12
12
|
class KeyRegularizer
|
|
13
|
-
WHITESPACE = /\s
|
|
13
|
+
WHITESPACE = /\s/
|
|
14
14
|
|
|
15
15
|
def self.encode(key)
|
|
16
16
|
return [key, false] if key.ascii_only? && !WHITESPACE.match(key)
|