dalli 4.3.5 → 4.3.7
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +78 -0
- data/README.md +17 -1
- data/lib/dalli/client.rb +27 -14
- data/lib/dalli/compressor.rb +53 -5
- data/lib/dalli/json_serializer.rb +28 -0
- data/lib/dalli/key_manager.rb +28 -2
- data/lib/dalli/protocol/base.rb +23 -7
- data/lib/dalli/protocol/binary/response_processor.rb +20 -4
- data/lib/dalli/protocol/binary.rb +8 -8
- data/lib/dalli/protocol/connection_manager.rb +98 -10
- data/lib/dalli/protocol/meta/response_processor.rb +53 -16
- data/lib/dalli/protocol/meta.rb +8 -7
- data/lib/dalli/protocol/value_compressor.rb +30 -3
- data/lib/dalli/protocol/value_serializer.rb +1 -1
- data/lib/dalli/version.rb +1 -1
- data/lib/dalli.rb +1 -0
- metadata +2 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 4b1a68fbdf9ea89024d6e9bc5803f94d3342c2f7b8bd7cfea3a97f7a873f7b76
|
|
4
|
+
data.tar.gz: 93d1c2fcb71fa739e6a00808cddb11567ad9fed3caa4d0751c924576bc04bfe0
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: bc42cde8b6f61ba60768bf43a9ebf8b1088dc8b2efbb4e32b5ed1abedd3c3be81c66aa02dc1a2558fd319debcc8074e61471996ad8825afed32d4fd741f8a312
|
|
7
|
+
data.tar.gz: 3a8320685a39edef196feaa387ef954caea41650b540c7f37d32d4821b5804f41631e5346e14b21912de0963f147c7d55e8c1ad08e24d2f7d25073975ba64d72
|
data/CHANGELOG.md
CHANGED
|
@@ -4,6 +4,84 @@ Dalli Changelog
|
|
|
4
4
|
Unreleased
|
|
5
5
|
==========
|
|
6
6
|
|
|
7
|
+
4.3.7
|
|
8
|
+
==========
|
|
9
|
+
|
|
10
|
+
Security:
|
|
11
|
+
|
|
12
|
+
- Keep the caller's key when retrying a request (GHSA-m252-9cgf-vx2w)
|
|
13
|
+
- With a `namespace`, a request retried after a transient network error (a timeout, or a connection closed by memcached or a proxy) applied the namespace a second time, so a retried read could return a different key's value and a retried write could overwrite a different key
|
|
14
|
+
- Affects `get_with_metadata` and `fetch_with_lock` since 4.1.0 (single-key operations on 4.x aren't affected); fixed in 5.2.2, 5.1.4, 5.0.10 and 4.3.7. Clients without a namespace aren't affected
|
|
15
|
+
- Complete the fix for per-request `raw: true` on reads (GHSA-wr87-m4jw-29x5)
|
|
16
|
+
- `cas` and `cas!` (both protocols) and `fetch_with_lock` (meta protocol) accepted `raw: true` but didn't pass it to the read, so the value was still deserialized. They now honor it
|
|
17
|
+
- A read made with `raw: true` doesn't ask for flags, but a reply carrying them anyway (from a proxy or a hostile server) still had its value deserialized. Meta protocol raw reads now ignore flags in the reply
|
|
18
|
+
- Affects 4.3.6 and earlier; fixed in 5.2.2, 5.1.4, 5.0.10, 4.3.7 and 3.2.13
|
|
19
|
+
- Complete the fork-safety fix (GHSA-w39f-xq2m-4g8x)
|
|
20
|
+
- Fork detection used a pid cache refreshed by a `Process._fork` hook. When another library's fork hook ran first in the child (connection_pool, loaded before Dalli as in Rails, closes its connections there), Dalli didn't know it was in a forked child and ended the parent's TLS session. It now checks `Process.pid`
|
|
21
|
+
- 4.3.6's write buffer kept a reference to the caller's value string with the meta protocol, so changing the string before the buffer was sent (for example, reusing one buffer inside a `quiet` block) changed the bytes sent, losing values or adding commands. The buffer now holds a copy
|
|
22
|
+
- Fixed in 4.3.7 and 3.2.13
|
|
23
|
+
|
|
24
|
+
Bug fixes:
|
|
25
|
+
|
|
26
|
+
- `get_with_metadata` and `fetch_with_lock` retried the final error raised when a server is marked down, so with `down_retry_delay: 0` they retried an unresponsive server forever. They now retry only retryable errors, like other operations
|
|
27
|
+
- Fix a regression in 4.3.6: a `get_multi` that didn't finish within `socket_timeout` counted toward `socket_max_failures`, so two slow `get_multi` calls in a row marked a healthy server down. It now just closes the connection
|
|
28
|
+
- Fix a regression in 4.3.6: over TLS, every request buffered in a `quiet` block was sent as its own TLS record and system call (2.5 times slower for a block of 2000 deletes). Each flush is one write again
|
|
29
|
+
- Handle malformed replies from a broken or hostile server or proxy: a negative value size, a hit with no key, or a hit with no `s` flag could leave the connection out of step or raise a non-Dalli error
|
|
30
|
+
- With `decompressed_max_bytes` in effect, data after the end of a compressed value's stream was returned as part of the value. It's now ignored, as without the limit
|
|
31
|
+
- With a `digest_class` whose digests aren't short hex strings, shortening a long key could loop forever. It now raises `ArgumentError`
|
|
32
|
+
|
|
33
|
+
Development:
|
|
34
|
+
|
|
35
|
+
- Run the Tests, RuboCop and Profiles workflows on pushes to `main` and the `*-stable` branches only, so a pull request's branch isn't tested twice (backport of #1198)
|
|
36
|
+
|
|
37
|
+
4.3.6
|
|
38
|
+
==========
|
|
39
|
+
|
|
40
|
+
Security:
|
|
41
|
+
|
|
42
|
+
- Fix pipelined `get_multi` returning one key's value for another after an error reply (GHSA-p6pm-ch9v-44vx)
|
|
43
|
+
- The pipelined reply parser took any reply without a body for the end of the batch. An error reply for one key (`CLIENT_ERROR`, `SERVER_ERROR`, or `EN` from a proxy) ended it early, and the replies still on the connection were read as the replies to later commands, so a `get` could return another key's value
|
|
44
|
+
- Keys are now measured in bytes as sent (after base64 encoding when the meta protocol needs it). A key under 250 characters but over 250 bytes, for example one with many non-ASCII characters, passed validation:
|
|
45
|
+
- with the meta protocol, memcached rejected it with the error reply above
|
|
46
|
+
- with the binary protocol, memcached dropped the connection, and `get` or `get_multi` with that key retried forever
|
|
47
|
+
- Such keys are now truncated like other long keys; keys that worked before are unchanged
|
|
48
|
+
- The reply parser affects `protocol: :meta` since 3.2.0; the key length affects both protocols; fixed in 5.2.1, 5.1.3, 5.0.9, 4.3.6 and 3.2.12
|
|
49
|
+
- Stop retrying an unresponsive server forever (GHSA-4qp6-2jcr-596v)
|
|
50
|
+
- A server that accepted connections but never answered (or dropped the connection on a request) was retried forever, hanging the caller, because each successful reconnect reset the failure count. Requests now fail after `socket_max_failures` attempts and the server is marked down, as when it can't be reached at all; it gets a full set of attempts again after `down_retry_delay`
|
|
51
|
+
- Affects all versions; fixed in 5.2.1, 5.1.3, 5.0.9, 4.3.6 and 3.2.12
|
|
52
|
+
- Limit decompressed and reply value sizes (GHSA-3553-vcg5-72jw)
|
|
53
|
+
- Values flagged as compressed were inflated without limit, so a small stored value could expand to gigabytes on read, whatever the client's `compress` or `serializer` settings. The new `decompressed_max_bytes` option (default 128 MiB; `nil` disables it) makes a read that would pass it raise `Dalli::UnmarshalError`. Custom compressors whose `decompress` takes only the data keep working, without the limit
|
|
54
|
+
- The value size in a reply (a meta `VA` size or a binary body length) was used to read or buffer that many bytes, so a malicious server could make the client allocate gigabytes. Sizes over 1 GiB, memcached's largest item, now raise `Dalli::DalliError` before reading
|
|
55
|
+
- Affects all versions; fixed in 5.2.1, 5.1.3, 5.0.9, 4.3.6 and 3.2.12
|
|
56
|
+
- Honor per-request `raw: true` on reads, and add `Dalli::JSONSerializer` (GHSA-wr87-m4jw-29x5)
|
|
57
|
+
- With the binary protocol, `get`, `gat` and `fetch`, and with the meta protocol, `get_with_metadata`, ignored a per-request `raw: true` and deserialized the value according to its stored flags, so a caller who asked for raw bytes could still have `Marshal.load` run on data someone else wrote to memcached. They now return the stored bytes
|
|
58
|
+
- The `raw` part affects binary `get` and `gat` in all versions, and `get_with_metadata` since 4.2.0
|
|
59
|
+
- `serializer: JSON` reads values with `JSON.load`, which on json gem versions before 3.0 creates an object of the class named in a stored `json_class` key when the json additions are loaded. The new `Dalli::JSONSerializer` reads with `JSON.parse` and only returns plain JSON types. The README and the Marshal security warning now recommend it; `serializer: JSON` itself is unchanged
|
|
60
|
+
- The `serializer: JSON` part affects any version used with the json gem before 3.0
|
|
61
|
+
- Keep a forked child from writing to or closing the parent's connection (GHSA-w39f-xq2m-4g8x)
|
|
62
|
+
- A child forked while the parent had requests buffered but not yet sent, such as quiet writes inside a `quiet` block, sent them again when it closed the client or reconnected: quiet writes ran twice, and the parent's later replies could be read as the replies to other requests, returning one key's value for another. Dalli now buffers requests itself and a forked child discards them
|
|
63
|
+
- Affects 4.2.0 and later; fixed in 5.2.1, 5.1.3, 5.0.9 and 4.3.6
|
|
64
|
+
- With TLS, a forked child that closed the client (or reconnected after detecting the fork) sent a TLS close on the connection the parent was still using, ending the parent's session. A forked child now closes only its own copy of the socket
|
|
65
|
+
- Affects all versions with TLS; fixed in 5.2.1, 5.1.3, 5.0.9, 4.3.6 and 3.2.12
|
|
66
|
+
|
|
67
|
+
Bug fixes:
|
|
68
|
+
|
|
69
|
+
- An exception raised by a `get_multi` block (such as `Timeout::Error`) was treated as a network failure: swallowed, with the whole `get_multi` retried and keys yielded twice. It now reaches the caller
|
|
70
|
+
- Fix `set_multi` with the binary protocol
|
|
71
|
+
- Each quiet `setq` waited for a reply memcached never sends for a successful write, so `set_multi` timed out on its first key without storing anything (and, with retries, hung or marked the server down). Affects 4.2.0 and later
|
|
72
|
+
|
|
73
|
+
Notes:
|
|
74
|
+
|
|
75
|
+
- Document [Ruby bug 21195](https://bugs.ruby-lang.org/issues/21195), a VM crash (`[BUG] rb_sys_fail_path_in(io_fillbuf, ...) - errno == 0`) when a socket read using `IO#timeout` is interrupted by a signal, in the README (#1188)
|
|
76
|
+
- Affects Ruby 3.2.x, 3.3.0–3.3.7 and 3.4.0–3.4.2. Fixed in 3.3.8 and 3.4.3; Ruby 3.2 reached end of life without the fix
|
|
77
|
+
- Ruby 3.1 doesn't use `IO#timeout` and isn't affected
|
|
78
|
+
|
|
79
|
+
Development:
|
|
80
|
+
|
|
81
|
+
- Drop Ruby 3.2 from the CI matrix, since that bug intermittently crashed its test runs (#1188)
|
|
82
|
+
- Fix three flaky tests: the `KeyManager` namespace key-length test, the Rack session freshness test, and `MemcachedManager`'s start and stop handling (backport of #1141, #1182)
|
|
83
|
+
- Fix flaky failover tests: move their ports out of Linux's ephemeral port range, and wait for memcached to accept connections after starting it (backport of #1184)
|
|
84
|
+
|
|
7
85
|
4.3.5
|
|
8
86
|
==========
|
|
9
87
|
|
data/README.md
CHANGED
|
@@ -20,6 +20,18 @@ The name is a variant of Salvador Dali for his famous painting [The Persistence
|
|
|
20
20
|
* Ruby 3.1 or later
|
|
21
21
|
* memcached 1.4 or later (1.6+ recommended for meta protocol support)
|
|
22
22
|
|
|
23
|
+
### Known Ruby issue: crash on an interrupted socket read
|
|
24
|
+
|
|
25
|
+
On Ruby 3.2 and later, Dalli sets socket timeouts with `IO#timeout`. Ruby 3.2.x, 3.3.0–3.3.7 and 3.4.0–3.4.2 have a bug ([Ruby #21195](https://bugs.ruby-lang.org/issues/21195)) where a timed read interrupted by a signal aborts the whole process:
|
|
26
|
+
|
|
27
|
+
```
|
|
28
|
+
[BUG] rb_sys_fail_path_in(io_fillbuf, fd:N ) - errno == 0
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
The crash is inside Ruby itself, so it can't be rescued. It was fixed in Ruby 3.3.8 and 3.4.3. Ruby 3.2 reached end of life without the fix. If you see this crash, upgrade to Ruby 3.3.8+ or 3.4.3+. Ruby 3.1 doesn't use `IO#timeout` and isn't affected.
|
|
32
|
+
|
|
33
|
+
Ruby 3.2 is no longer part of this branch's CI matrix for the same reason.
|
|
34
|
+
|
|
23
35
|
## Protocol Options
|
|
24
36
|
|
|
25
37
|
Dalli supports two protocols for communicating with memcached:
|
|
@@ -61,9 +73,13 @@ The separator must be a single non-alphanumeric character. Valid examples: `:`,
|
|
|
61
73
|
By default, Dalli uses Ruby's Marshal for serialization. Deserializing untrusted data with Marshal can lead to remote code execution. If you cache user-controlled data, consider using a safer serializer:
|
|
62
74
|
|
|
63
75
|
```ruby
|
|
64
|
-
Dalli::Client.new('localhost:11211', serializer:
|
|
76
|
+
Dalli::Client.new('localhost:11211', serializer: Dalli::JSONSerializer)
|
|
65
77
|
```
|
|
66
78
|
|
|
79
|
+
`Dalli::JSONSerializer` reads values with `JSON.parse`, so it only ever returns plain hashes, arrays, strings, numbers, booleans and `nil`. Passing the `JSON` module itself (`serializer: JSON`) reads values with `JSON.load`, which can create objects of other classes from a stored `json_class` key and is not recommended for data an attacker might write.
|
|
80
|
+
|
|
81
|
+
Per-request `raw: true` returns the stored bytes without deserializing or decompressing them, on every read method that accepts it (`get`, `get_with_metadata` and `fetch`).
|
|
82
|
+
|
|
67
83
|
See the [4.0-Upgrade.md](4.0-Upgrade.md) guide for more information.
|
|
68
84
|
|
|
69
85
|
## OpenTelemetry Tracing
|
data/lib/dalli/client.rb
CHANGED
|
@@ -134,14 +134,18 @@ module Dalli
|
|
|
134
134
|
def get_with_metadata(key, options = {})
|
|
135
135
|
raise_unless_meta_protocol!
|
|
136
136
|
|
|
137
|
-
|
|
138
|
-
|
|
137
|
+
# A retry re-runs this method with its original arguments, so `key` must
|
|
138
|
+
# stay as the caller gave it; reassigning it would add the namespace again.
|
|
139
|
+
validated_key = @key_manager.validate_key(key.to_s)
|
|
139
140
|
|
|
140
|
-
server = ring.server_for_key(
|
|
141
|
-
Instrumentation.trace('get_with_metadata', trace_attrs('get_with_metadata',
|
|
142
|
-
server.request(:meta_get,
|
|
141
|
+
server = ring.server_for_key(validated_key)
|
|
142
|
+
Instrumentation.trace('get_with_metadata', trace_attrs('get_with_metadata', validated_key, server)) do
|
|
143
|
+
server.request(:meta_get, validated_key, options)
|
|
143
144
|
end
|
|
144
|
-
|
|
145
|
+
# Only a retryable error is retried. The NetworkError raised when a server
|
|
146
|
+
# is marked down is final, as in #perform; retrying it looped forever once
|
|
147
|
+
# down_retry_delay let the same server be tried again.
|
|
148
|
+
rescue RetryableNetworkError => e
|
|
145
149
|
Dalli.logger.debug { e.inspect }
|
|
146
150
|
Dalli.logger.debug { 'retrying get_with_metadata with new server' }
|
|
147
151
|
retry
|
|
@@ -238,14 +242,17 @@ module Dalli
|
|
|
238
242
|
validate_integer!(:lock_ttl, lock_ttl)
|
|
239
243
|
validate_integer!(:recache_threshold, recache_threshold)
|
|
240
244
|
|
|
241
|
-
|
|
242
|
-
|
|
245
|
+
# Left unchanged for the retry below; see get_with_metadata
|
|
246
|
+
validated_key = @key_manager.validate_key(key.to_s)
|
|
243
247
|
|
|
244
|
-
server = ring.server_for_key(
|
|
245
|
-
Instrumentation.trace('fetch_with_lock', trace_attrs('fetch_with_lock',
|
|
246
|
-
fetch_with_lock_request(
|
|
248
|
+
server = ring.server_for_key(validated_key)
|
|
249
|
+
Instrumentation.trace('fetch_with_lock', trace_attrs('fetch_with_lock', validated_key, server)) do
|
|
250
|
+
fetch_with_lock_request(validated_key, ttl, lock_ttl, recache_threshold, req_options, &block)
|
|
247
251
|
end
|
|
248
|
-
|
|
252
|
+
# Only a retryable error is retried. The NetworkError raised when a server
|
|
253
|
+
# is marked down is final, as in #perform; retrying it looped forever once
|
|
254
|
+
# down_retry_delay let the same server be tried again.
|
|
255
|
+
rescue RetryableNetworkError => e
|
|
249
256
|
Dalli.logger.debug { e.inspect }
|
|
250
257
|
Dalli.logger.debug { 'retrying fetch_with_lock with new server' }
|
|
251
258
|
retry
|
|
@@ -592,7 +599,8 @@ module Dalli
|
|
|
592
599
|
end
|
|
593
600
|
|
|
594
601
|
def cas_core(key, always_set, ttl = nil, req_options = nil)
|
|
595
|
-
|
|
602
|
+
# req_options reaches the read too, so raw: true returns the stored bytes
|
|
603
|
+
(value, cas) = perform(:cas, key, req_options)
|
|
596
604
|
return if value.nil? && !always_set
|
|
597
605
|
|
|
598
606
|
newvalue = yield(value)
|
|
@@ -601,7 +609,12 @@ module Dalli
|
|
|
601
609
|
|
|
602
610
|
def fetch_with_lock_request(key, ttl, lock_ttl, recache_threshold, req_options)
|
|
603
611
|
server = ring.server_for_key(key)
|
|
604
|
-
|
|
612
|
+
# req_options (e.g. raw: true) reaches the read too. It's the base, not
|
|
613
|
+
# the override: fetch_with_lock's own lock_ttl/recache_threshold win.
|
|
614
|
+
meta_options = req_options.is_a?(Hash) ? req_options.dup : {}
|
|
615
|
+
meta_options[:vivify_ttl] = lock_ttl
|
|
616
|
+
meta_options[:recache_ttl] = recache_threshold
|
|
617
|
+
result = server.request(:meta_get, key, meta_options)
|
|
605
618
|
|
|
606
619
|
return result[:value] unless result[:won_recache]
|
|
607
620
|
|
data/lib/dalli/compressor.rb
CHANGED
|
@@ -13,8 +13,54 @@ module Dalli
|
|
|
13
13
|
Zlib::Deflate.deflate(data)
|
|
14
14
|
end
|
|
15
15
|
|
|
16
|
-
|
|
17
|
-
|
|
16
|
+
# DEFLATE can't expand data by more than about 1032:1
|
|
17
|
+
MAX_EXPANSION = 1032
|
|
18
|
+
# Compressed input is fed to zlib this many bytes at a time. MRI's zlib
|
|
19
|
+
# yields the output in small chunks; one that returns it instead still
|
|
20
|
+
# produces at most MAX_EXPANSION times this per step.
|
|
21
|
+
INFLATE_SLICE_BYTES = 64 * 1024
|
|
22
|
+
|
|
23
|
+
# max_bytes caps the decompressed size; inflating stops and raises as soon
|
|
24
|
+
# as the output passes it, so a small compressed value can't expand into
|
|
25
|
+
# gigabytes of memory.
|
|
26
|
+
def self.decompress(data, max_bytes: nil)
|
|
27
|
+
return Zlib::Inflate.inflate(data) unless may_exceed?(data, max_bytes)
|
|
28
|
+
|
|
29
|
+
inflate_within_limit(Zlib::Inflate.new, data, max_bytes)
|
|
30
|
+
end
|
|
31
|
+
|
|
32
|
+
# False when even the largest possible expansion stays within max_bytes,
|
|
33
|
+
# so the value can be inflated in one step
|
|
34
|
+
def self.may_exceed?(data, max_bytes)
|
|
35
|
+
max_bytes ? data.bytesize * MAX_EXPANSION > max_bytes : false
|
|
36
|
+
end
|
|
37
|
+
|
|
38
|
+
def self.inflate_within_limit(inflater, data, max_bytes)
|
|
39
|
+
out = String.new(encoding: Encoding::BINARY)
|
|
40
|
+
0.step(data.bytesize - 1, INFLATE_SLICE_BYTES) do |pos|
|
|
41
|
+
inflate_slice(inflater, data.byteslice(pos, INFLATE_SLICE_BYTES), out, max_bytes)
|
|
42
|
+
# Anything after the end of the stream isn't part of the value, as
|
|
43
|
+
# with Zlib::Inflate.inflate; finish would return it as output
|
|
44
|
+
break if inflater.finished?
|
|
45
|
+
end
|
|
46
|
+
return out if inflater.finished?
|
|
47
|
+
|
|
48
|
+
append_within_limit(out, inflater.finish, max_bytes)
|
|
49
|
+
ensure
|
|
50
|
+
inflater.close
|
|
51
|
+
end
|
|
52
|
+
|
|
53
|
+
# Both yielded and returned output count toward the limit
|
|
54
|
+
def self.inflate_slice(inflater, slice, out, max_bytes)
|
|
55
|
+
rest = inflater.inflate(slice) { |chunk| append_within_limit(out, chunk, max_bytes) }
|
|
56
|
+
append_within_limit(out, rest, max_bytes)
|
|
57
|
+
end
|
|
58
|
+
|
|
59
|
+
def self.append_within_limit(out, chunk, max_bytes)
|
|
60
|
+
out << chunk if chunk
|
|
61
|
+
return out if out.bytesize <= max_bytes
|
|
62
|
+
|
|
63
|
+
raise Dalli::UnmarshalError, "Decompressed value exceeds #{max_bytes} bytes (decompressed_max_bytes)"
|
|
18
64
|
end
|
|
19
65
|
end
|
|
20
66
|
|
|
@@ -32,9 +78,11 @@ module Dalli
|
|
|
32
78
|
io.string
|
|
33
79
|
end
|
|
34
80
|
|
|
35
|
-
def self.decompress(data)
|
|
36
|
-
|
|
37
|
-
|
|
81
|
+
def self.decompress(data, max_bytes: nil)
|
|
82
|
+
return Zlib::GzipReader.new(StringIO.new(data, 'rb')).read unless Compressor.may_exceed?(data, max_bytes)
|
|
83
|
+
|
|
84
|
+
# MAX_WBITS + 16 makes zlib read the gzip format (and check its CRC)
|
|
85
|
+
Compressor.inflate_within_limit(Zlib::Inflate.new(Zlib::MAX_WBITS + 16), data, max_bytes)
|
|
38
86
|
end
|
|
39
87
|
end
|
|
40
88
|
end
|
|
@@ -0,0 +1,28 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require 'json'
|
|
4
|
+
|
|
5
|
+
module Dalli
|
|
6
|
+
##
|
|
7
|
+
# A serializer that stores values as JSON and never creates Ruby objects
|
|
8
|
+
# from what it reads.
|
|
9
|
+
#
|
|
10
|
+
# Passing the JSON module itself as the serializer (serializer: JSON) reads
|
|
11
|
+
# values back with JSON.load, which on json gem versions before 3.0 honors
|
|
12
|
+
# "json_class" and instantiates other classes when the json gem's additions
|
|
13
|
+
# are loaded. This serializer reads with JSON.parse, so a value written to
|
|
14
|
+
# memcached by someone else can only come back as plain hashes, arrays,
|
|
15
|
+
# strings, numbers, booleans and nil.
|
|
16
|
+
#
|
|
17
|
+
# Dalli::Client.new(servers, serializer: Dalli::JSONSerializer)
|
|
18
|
+
##
|
|
19
|
+
module JSONSerializer
|
|
20
|
+
def self.dump(value)
|
|
21
|
+
JSON.generate(value)
|
|
22
|
+
end
|
|
23
|
+
|
|
24
|
+
def self.load(data)
|
|
25
|
+
JSON.parse(data)
|
|
26
|
+
end
|
|
27
|
+
end
|
|
28
|
+
end
|
data/lib/dalli/key_manager.rb
CHANGED
|
@@ -41,6 +41,9 @@ module Dalli
|
|
|
41
41
|
|
|
42
42
|
@namespace = namespace_from_options
|
|
43
43
|
@namespace_separator = @key_options[:namespace_separator]
|
|
44
|
+
# The meta protocol base64-encodes keys with whitespace or non-ASCII
|
|
45
|
+
# characters, so their length on the wire differs from their length here
|
|
46
|
+
@meta_protocol = client_options[:protocol]&.to_s == 'meta'
|
|
44
47
|
end
|
|
45
48
|
|
|
46
49
|
##
|
|
@@ -58,7 +61,7 @@ module Dalli
|
|
|
58
61
|
raise ArgumentError, 'key cannot be blank' unless key&.length&.positive?
|
|
59
62
|
|
|
60
63
|
key = key_with_namespace(key)
|
|
61
|
-
key
|
|
64
|
+
wire_length(key) > MAX_KEY_LENGTH ? truncated_key(key) : key
|
|
62
65
|
end
|
|
63
66
|
|
|
64
67
|
##
|
|
@@ -122,7 +125,30 @@ module Dalli
|
|
|
122
125
|
##
|
|
123
126
|
def truncated_key(key)
|
|
124
127
|
digest = digest_class.hexdigest(key)
|
|
125
|
-
|
|
128
|
+
prefix = key[0, [prefix_length(digest), 0].max]
|
|
129
|
+
truncated = "#{prefix}#{TRUNCATED_KEY_SEPARATOR}#{digest}"
|
|
130
|
+
# A prefix with whitespace or non-ASCII characters sends the whole key
|
|
131
|
+
# base64-encoded, which can still be too long on the wire; shorten it.
|
|
132
|
+
while wire_length(truncated) > MAX_KEY_LENGTH && !prefix.empty?
|
|
133
|
+
prefix = prefix[0...-1]
|
|
134
|
+
truncated = "#{prefix}#{TRUNCATED_KEY_SEPARATOR}#{digest}"
|
|
135
|
+
end
|
|
136
|
+
return truncated if wire_length(truncated) <= MAX_KEY_LENGTH
|
|
137
|
+
|
|
138
|
+
# Only possible with a digest_class whose digests aren't short hex strings
|
|
139
|
+
raise ArgumentError, "#{digest_class} digests are too long to fit in a truncated key"
|
|
140
|
+
end
|
|
141
|
+
|
|
142
|
+
# memcached's key limit applies to the key as sent: in bytes, and, with
|
|
143
|
+
# the meta protocol, after base64 encoding when the key needs it. A key
|
|
144
|
+
# within the limit in characters can still be too long. With the meta
|
|
145
|
+
# protocol memcached rejects it with a CLIENT_ERROR that must never reach
|
|
146
|
+
# a pipelined get; with the binary protocol it drops the connection.
|
|
147
|
+
def wire_length(key)
|
|
148
|
+
return key.bytesize unless @meta_protocol
|
|
149
|
+
|
|
150
|
+
encoded, = Dalli::Protocol::Meta::KeyRegularizer.encode(key)
|
|
151
|
+
encoded.bytesize
|
|
126
152
|
end
|
|
127
153
|
|
|
128
154
|
def prefix_length(digest)
|
data/lib/dalli/protocol/base.rb
CHANGED
|
@@ -34,6 +34,13 @@ module Dalli
|
|
|
34
34
|
@raw_mode
|
|
35
35
|
end
|
|
36
36
|
|
|
37
|
+
# True when values should come back as stored: the client is in raw mode,
|
|
38
|
+
# or this request passed raw: true. A stored value is then never
|
|
39
|
+
# deserialized or decompressed for a caller who asked for raw bytes.
|
|
40
|
+
def raw_request?(options)
|
|
41
|
+
raw_mode? || (options.is_a?(Hash) && options[:raw]) ? true : false
|
|
42
|
+
end
|
|
43
|
+
|
|
37
44
|
# Chokepoint method for error handling and ensuring liveness
|
|
38
45
|
def request(opkey, *args)
|
|
39
46
|
verify_state(opkey)
|
|
@@ -99,7 +106,15 @@ module Dalli
|
|
|
99
106
|
reconnect_on_pipeline_complete!
|
|
100
107
|
values = nil
|
|
101
108
|
|
|
102
|
-
|
|
109
|
+
# Only the read is socket I/O. The caller's block runs below, outside
|
|
110
|
+
# this rescue, so an Errno or Timeout::Error it raises reaches the
|
|
111
|
+
# caller instead of being mistaken for a network failure (which closed
|
|
112
|
+
# the connection and retried the whole get_multi, yielding keys twice).
|
|
113
|
+
begin
|
|
114
|
+
response_buffer.read
|
|
115
|
+
rescue SystemCallError, *TIMEOUT_ERRORS, *SSL_ERRORS, EOFError => e
|
|
116
|
+
@connection_manager.error_on_request!(e)
|
|
117
|
+
end
|
|
103
118
|
|
|
104
119
|
status, cas, key, value = response_buffer.process_single_getk_response
|
|
105
120
|
# status is not nil only if we have a full response to parse
|
|
@@ -125,8 +140,6 @@ module Dalli
|
|
|
125
140
|
end
|
|
126
141
|
|
|
127
142
|
values || {}
|
|
128
|
-
rescue SystemCallError, *TIMEOUT_ERRORS, *SSL_ERRORS, EOFError => e
|
|
129
|
-
@connection_manager.error_on_request!(e)
|
|
130
143
|
end
|
|
131
144
|
|
|
132
145
|
# Abort current pipelined get. Generally used to signal an external
|
|
@@ -139,10 +152,13 @@ module Dalli
|
|
|
139
152
|
@connection_manager.abort_request!
|
|
140
153
|
return true unless connected?
|
|
141
154
|
|
|
142
|
-
# Closes the connection, which ensures that our connection
|
|
143
|
-
#
|
|
144
|
-
|
|
145
|
-
|
|
155
|
+
# Closes the connection, which ensures that our connection is in a
|
|
156
|
+
# clean state for future requests. The get_multi ran out of time
|
|
157
|
+
# overall rather than hitting a socket error, so it doesn't count
|
|
158
|
+
# toward socket_max_failures: two slow get_multis in a row would
|
|
159
|
+
# otherwise mark a healthy server down.
|
|
160
|
+
Dalli.logger.warn { "#{name} didn't finish a get_multi within socket_timeout; closing the connection" }
|
|
161
|
+
@connection_manager.close
|
|
146
162
|
true
|
|
147
163
|
end
|
|
148
164
|
|
|
@@ -66,7 +66,9 @@ module Dalli
|
|
|
66
66
|
raise Dalli::DalliError, "Response error #{resp_header.status}: #{RESPONSE_CODES[resp_header.status]}"
|
|
67
67
|
end
|
|
68
68
|
|
|
69
|
-
|
|
69
|
+
# raw: true returns the value as stored, without deserializing or
|
|
70
|
+
# decompressing it, whatever flags it was stored with
|
|
71
|
+
def get(cache_nils: false, raw: false)
|
|
70
72
|
resp_header, body = read_response
|
|
71
73
|
|
|
72
74
|
return false if resp_header.not_stored? # Not stored, normal status for add operation
|
|
@@ -75,7 +77,7 @@ module Dalli
|
|
|
75
77
|
raise_on_not_ok!(resp_header)
|
|
76
78
|
return true unless body
|
|
77
79
|
|
|
78
|
-
unpack_response_body(resp_header, body,
|
|
80
|
+
unpack_response_body(resp_header, body, !raw).last
|
|
79
81
|
end
|
|
80
82
|
|
|
81
83
|
##
|
|
@@ -100,7 +102,8 @@ module Dalli
|
|
|
100
102
|
true
|
|
101
103
|
end
|
|
102
104
|
|
|
103
|
-
|
|
105
|
+
# raw: true returns the value as stored; see #get
|
|
106
|
+
def data_cas_response(raw: false)
|
|
104
107
|
resp_header, body = read_response
|
|
105
108
|
return [nil, resp_header.cas] if resp_header.not_found?
|
|
106
109
|
return [nil, false] if resp_header.not_stored?
|
|
@@ -108,7 +111,7 @@ module Dalli
|
|
|
108
111
|
raise_on_not_ok!(resp_header)
|
|
109
112
|
return [nil, resp_header.cas] unless body
|
|
110
113
|
|
|
111
|
-
[unpack_response_body(resp_header, body,
|
|
114
|
+
[unpack_response_body(resp_header, body, !raw).last, resp_header.cas]
|
|
112
115
|
end
|
|
113
116
|
|
|
114
117
|
# Returns the new value for the key, if found and updated
|
|
@@ -199,6 +202,17 @@ module Dalli
|
|
|
199
202
|
# The remaining three values in the array are the ResponseHeader,
|
|
200
203
|
# key, and value.
|
|
201
204
|
##
|
|
205
|
+
# memcached can't store an item larger than 1 GiB (its -I maximum). A
|
|
206
|
+
# pipelined reply claiming an impossible size would otherwise have the
|
|
207
|
+
# buffer wait for (and accumulate) that many bytes.
|
|
208
|
+
MAX_BODY_BYTES = 1024 * 1024 * 1024
|
|
209
|
+
|
|
210
|
+
def check_body_size!(size)
|
|
211
|
+
return if size <= MAX_BODY_BYTES
|
|
212
|
+
|
|
213
|
+
raise Dalli::DalliError, "Reply body size #{size} is out of range"
|
|
214
|
+
end
|
|
215
|
+
|
|
202
216
|
def getk_response_from_buffer(buf, offset = 0)
|
|
203
217
|
# There's no header in the buffer, so don't advance
|
|
204
218
|
return [0, nil, nil, nil, nil] unless buf && buf.bytesize >= offset + ResponseHeader::SIZE
|
|
@@ -212,6 +226,8 @@ module Dalli
|
|
|
212
226
|
# error response that needs to be discarded.
|
|
213
227
|
return [ResponseHeader::SIZE, resp_header.ok?, resp_header.cas, nil, nil] if body_len.zero?
|
|
214
228
|
|
|
229
|
+
check_body_size!(body_len)
|
|
230
|
+
|
|
215
231
|
resp_size = ResponseHeader::SIZE + body_len
|
|
216
232
|
# The header is in the buffer, but the body is not. As we don't have
|
|
217
233
|
# a complete response, don't advance the buffer
|
|
@@ -23,7 +23,7 @@ module Dalli
|
|
|
23
23
|
req = RequestFormatter.standard_request(opkey: :get, key: key)
|
|
24
24
|
write(req)
|
|
25
25
|
@connection_manager.flush
|
|
26
|
-
response_processor.get(cache_nils: cache_nils?(options))
|
|
26
|
+
response_processor.get(cache_nils: cache_nils?(options), raw: raw_request?(options))
|
|
27
27
|
end
|
|
28
28
|
|
|
29
29
|
def quiet_get_request(key)
|
|
@@ -35,7 +35,7 @@ module Dalli
|
|
|
35
35
|
req = RequestFormatter.standard_request(opkey: :gat, key: key, ttl: ttl)
|
|
36
36
|
write(req)
|
|
37
37
|
@connection_manager.flush
|
|
38
|
-
response_processor.get(cache_nils: cache_nils?(options))
|
|
38
|
+
response_processor.get(cache_nils: cache_nils?(options), raw: raw_request?(options))
|
|
39
39
|
end
|
|
40
40
|
|
|
41
41
|
def touch(key, ttl)
|
|
@@ -47,11 +47,11 @@ module Dalli
|
|
|
47
47
|
|
|
48
48
|
# TODO: This is confusing, as there's a cas command in memcached
|
|
49
49
|
# and this isn't it. Maybe rename? Maybe eliminate?
|
|
50
|
-
def cas(key)
|
|
50
|
+
def cas(key, options = nil)
|
|
51
51
|
req = RequestFormatter.standard_request(opkey: :get, key: key)
|
|
52
52
|
write(req)
|
|
53
53
|
@connection_manager.flush
|
|
54
|
-
response_processor.data_cas_response
|
|
54
|
+
response_processor.data_cas_response(raw: raw_request?(options))
|
|
55
55
|
end
|
|
56
56
|
|
|
57
57
|
# Storage Commands
|
|
@@ -63,7 +63,7 @@ module Dalli
|
|
|
63
63
|
# Pipelined set - writes a quiet set request without reading response.
|
|
64
64
|
# Used by PipelinedSetter for bulk operations.
|
|
65
65
|
def pipelined_set(key, value, ttl, options)
|
|
66
|
-
storage_req(:setq, key, value, ttl, 0, options)
|
|
66
|
+
storage_req(:setq, key, value, ttl, 0, options, quiet: true)
|
|
67
67
|
end
|
|
68
68
|
|
|
69
69
|
def add(key, value, ttl, options)
|
|
@@ -77,7 +77,7 @@ module Dalli
|
|
|
77
77
|
end
|
|
78
78
|
|
|
79
79
|
# rubocop:disable-next Metrics/ParameterLists
|
|
80
|
-
def storage_req(opkey, key, value, ttl, cas, options)
|
|
80
|
+
def storage_req(opkey, key, value, ttl, cas, options, quiet: quiet?)
|
|
81
81
|
(value, bitflags) = @value_marshaller.store(key, value, options)
|
|
82
82
|
ttl = TtlSanitizer.sanitize(ttl)
|
|
83
83
|
|
|
@@ -85,8 +85,8 @@ module Dalli
|
|
|
85
85
|
value: value, bitflags: bitflags,
|
|
86
86
|
ttl: ttl, cas: cas)
|
|
87
87
|
write(req)
|
|
88
|
-
@connection_manager.flush unless quiet
|
|
89
|
-
response_processor.storage_response unless quiet
|
|
88
|
+
@connection_manager.flush unless quiet
|
|
89
|
+
response_processor.storage_response unless quiet
|
|
90
90
|
end
|
|
91
91
|
|
|
92
92
|
def append(key, value)
|
|
@@ -37,7 +37,10 @@ module Dalli
|
|
|
37
37
|
@request_in_progress = false
|
|
38
38
|
@sock = nil
|
|
39
39
|
@pid = nil
|
|
40
|
+
@write_buffer = []
|
|
41
|
+
@write_buffer_bytes = 0
|
|
40
42
|
|
|
43
|
+
@fail_count = 0
|
|
41
44
|
reset_down_info
|
|
42
45
|
end
|
|
43
46
|
|
|
@@ -53,8 +56,9 @@ module Dalli
|
|
|
53
56
|
Dalli.logger.debug { "Dalli::Server#connect #{name}" }
|
|
54
57
|
|
|
55
58
|
@sock = memcached_socket
|
|
56
|
-
|
|
57
|
-
@
|
|
59
|
+
# Writes are buffered in @write_buffer instead; see WRITE_BUFFER_FLUSH_BYTES
|
|
60
|
+
@sock.sync = true
|
|
61
|
+
@pid = Process.pid
|
|
58
62
|
@request_in_progress = false
|
|
59
63
|
rescue SystemCallError, *TIMEOUT_ERRORS, EOFError, SocketError => e
|
|
60
64
|
# SocketError = DNS resolution failure
|
|
@@ -85,6 +89,8 @@ module Dalli
|
|
|
85
89
|
def down!
|
|
86
90
|
close
|
|
87
91
|
log_down_detected
|
|
92
|
+
# Once down_retry_delay passes, the server gets a full set of attempts
|
|
93
|
+
@fail_count = 0
|
|
88
94
|
|
|
89
95
|
@error = $ERROR_INFO&.class&.name
|
|
90
96
|
@msg ||= $ERROR_INFO&.message
|
|
@@ -114,15 +120,23 @@ module Dalli
|
|
|
114
120
|
return unless @sock
|
|
115
121
|
|
|
116
122
|
begin
|
|
117
|
-
|
|
123
|
+
close_socket
|
|
118
124
|
rescue StandardError
|
|
119
125
|
nil
|
|
120
126
|
end
|
|
121
127
|
@sock = nil
|
|
122
128
|
@pid = nil
|
|
129
|
+
discard_write_buffer
|
|
123
130
|
abort_request!
|
|
124
131
|
end
|
|
125
132
|
|
|
133
|
+
# A forked child shares the parent's connection, so it closes only its
|
|
134
|
+
# own file descriptor. Closing the TLS socket itself would send
|
|
135
|
+
# close_notify and end the parent's TLS session.
|
|
136
|
+
def close_socket
|
|
137
|
+
fork_detected? ? @sock.to_io.close : @sock.close
|
|
138
|
+
end
|
|
139
|
+
|
|
126
140
|
def connected?
|
|
127
141
|
!@sock.nil?
|
|
128
142
|
end
|
|
@@ -141,6 +155,11 @@ module Dalli
|
|
|
141
155
|
raise '[Dalli] No request in progress. This may be a bug in Dalli.' unless @request_in_progress
|
|
142
156
|
|
|
143
157
|
@request_in_progress = false
|
|
158
|
+
# A completed request is what proves the server healthy again, so the
|
|
159
|
+
# failure count resets here rather than on reconnect: a server that
|
|
160
|
+
# accepts connections but never answers would otherwise reset it on
|
|
161
|
+
# every retry and never reach socket_max_failures.
|
|
162
|
+
@fail_count = 0
|
|
144
163
|
end
|
|
145
164
|
|
|
146
165
|
def abort_request!
|
|
@@ -148,6 +167,7 @@ module Dalli
|
|
|
148
167
|
end
|
|
149
168
|
|
|
150
169
|
def read_line
|
|
170
|
+
flush_write_buffer
|
|
151
171
|
data = @sock.gets("\r\n")
|
|
152
172
|
error_on_request!('EOF in read_line') if data.nil?
|
|
153
173
|
data
|
|
@@ -155,27 +175,56 @@ module Dalli
|
|
|
155
175
|
error_on_request!(e)
|
|
156
176
|
end
|
|
157
177
|
|
|
178
|
+
# memcached can't store an item larger than 1 GiB (its -I maximum), so a
|
|
179
|
+
# reply claiming more (or a negative size) is malformed or hostile.
|
|
180
|
+
# Reading allocates the full count up front, so check before reading.
|
|
181
|
+
MAX_READ_BYTES = (1024 * 1024 * 1024) + 2 # plus the value's trailing "\r\n"
|
|
182
|
+
|
|
183
|
+
def check_read_size!(count)
|
|
184
|
+
return if count.between?(0, MAX_READ_BYTES)
|
|
185
|
+
|
|
186
|
+
raise Dalli::DalliError, "Reply size #{count} from #{name} is out of range"
|
|
187
|
+
end
|
|
188
|
+
|
|
158
189
|
def read(count)
|
|
190
|
+
check_read_size!(count)
|
|
191
|
+
flush_write_buffer
|
|
159
192
|
@sock.readfull(count)
|
|
160
193
|
rescue SystemCallError, *TIMEOUT_ERRORS, *SSL_ERRORS, EOFError => e
|
|
161
194
|
error_on_request!(e)
|
|
162
195
|
end
|
|
163
196
|
|
|
197
|
+
# Requests are buffered here rather than in the socket's own IO buffer
|
|
198
|
+
# (the socket is sync). Ruby flushes an IO's write buffer when any process
|
|
199
|
+
# holding it closes or finalizes the IO, so after a fork a child could
|
|
200
|
+
# send bytes the parent had buffered but not sent yet: a quiet write would
|
|
201
|
+
# run twice, or the parent's replies would shift onto the wrong requests.
|
|
202
|
+
# A forked child discards this buffer instead. Like IO's own buffer, it
|
|
203
|
+
# is sent once it grows past this size, or when a reply is read.
|
|
204
|
+
WRITE_BUFFER_FLUSH_BYTES = 64 * 1024
|
|
205
|
+
|
|
164
206
|
def write(bytes)
|
|
165
|
-
|
|
166
|
-
|
|
207
|
+
# A copy: the caller may change its string before the buffer is sent
|
|
208
|
+
# (the meta protocol writes a value string as its own part), as Ruby's
|
|
209
|
+
# IO buffer copied on write
|
|
210
|
+
@write_buffer << (bytes.frozen? ? bytes : bytes.dup)
|
|
211
|
+
@write_buffer_bytes += bytes.bytesize
|
|
212
|
+
flush_write_buffer if @write_buffer_bytes >= WRITE_BUFFER_FLUSH_BYTES
|
|
213
|
+
bytes.bytesize
|
|
214
|
+
rescue SystemCallError, *TIMEOUT_ERRORS, *SSL_ERRORS, IOError => e
|
|
167
215
|
error_on_request!(e)
|
|
168
216
|
end
|
|
169
217
|
|
|
170
218
|
def flush
|
|
171
|
-
|
|
172
|
-
rescue SystemCallError, *TIMEOUT_ERRORS, *SSL_ERRORS => e
|
|
219
|
+
flush_write_buffer
|
|
220
|
+
rescue SystemCallError, *TIMEOUT_ERRORS, *SSL_ERRORS, IOError => e
|
|
173
221
|
error_on_request!(e)
|
|
174
222
|
end
|
|
175
223
|
|
|
176
224
|
# Non-blocking read. Here to support the operation
|
|
177
225
|
# of the get_multi operation
|
|
178
226
|
def read_nonblock
|
|
227
|
+
flush_write_buffer
|
|
179
228
|
@sock.read_available
|
|
180
229
|
end
|
|
181
230
|
|
|
@@ -202,8 +251,9 @@ module Dalli
|
|
|
202
251
|
raise Dalli::RetryableNetworkError, message
|
|
203
252
|
end
|
|
204
253
|
|
|
254
|
+
# Called on connect. Deliberately leaves @fail_count alone; see
|
|
255
|
+
# finish_request!.
|
|
205
256
|
def reset_down_info
|
|
206
|
-
@fail_count = 0
|
|
207
257
|
@down_at = nil
|
|
208
258
|
@last_down_at = nil
|
|
209
259
|
@msg = nil
|
|
@@ -228,13 +278,18 @@ module Dalli
|
|
|
228
278
|
def reconnect_on_fork
|
|
229
279
|
message = 'Fork detected, re-connecting child process...'
|
|
230
280
|
Dalli.logger.info { message }
|
|
231
|
-
#
|
|
281
|
+
# Drops anything the parent had buffered, closes the inherited socket
|
|
282
|
+
# without touching the parent's connection, and reconnects immediately
|
|
232
283
|
close
|
|
233
284
|
establish_connection
|
|
234
285
|
end
|
|
235
286
|
|
|
236
287
|
def fork_detected?
|
|
237
|
-
|
|
288
|
+
# Process.pid rather than PIDCache: PIDCache is refreshed by a
|
|
289
|
+
# Process._fork hook, and another library's fork hook (such as
|
|
290
|
+
# connection_pool closing its connections) can run in the child before
|
|
291
|
+
# it, which would close the parent's TLS session as if it were ours.
|
|
292
|
+
@pid && @pid != Process.pid
|
|
238
293
|
end
|
|
239
294
|
|
|
240
295
|
def log_down_detected
|
|
@@ -255,6 +310,39 @@ module Dalli
|
|
|
255
310
|
time = Time.now - @down_at
|
|
256
311
|
Dalli.logger.warn { format('%<name>s is back (downtime was %<time>.3f seconds)', name: name, time: time) }
|
|
257
312
|
end
|
|
313
|
+
|
|
314
|
+
private
|
|
315
|
+
|
|
316
|
+
# One write per flush. A TLS socket sends each argument of
|
|
317
|
+
# write(*parts) as its own record and system call, so the parts are
|
|
318
|
+
# joined first.
|
|
319
|
+
def flush_write_buffer
|
|
320
|
+
return if @write_buffer.empty?
|
|
321
|
+
|
|
322
|
+
@sock.write(@write_buffer.size == 1 ? @write_buffer.first : joined_write_buffer)
|
|
323
|
+
discard_write_buffer
|
|
324
|
+
end
|
|
325
|
+
|
|
326
|
+
# Joined as bytes: requests can mix UTF-8 and binary strings, which
|
|
327
|
+
# String#<< refuses to combine when both hold non-ASCII bytes
|
|
328
|
+
if String.method_defined?(:append_as_bytes)
|
|
329
|
+
def joined_write_buffer
|
|
330
|
+
out = String.new(capacity: @write_buffer_bytes, encoding: Encoding::BINARY)
|
|
331
|
+
@write_buffer.each { |part| out.append_as_bytes(part) }
|
|
332
|
+
out
|
|
333
|
+
end
|
|
334
|
+
else
|
|
335
|
+
def joined_write_buffer
|
|
336
|
+
out = String.new(capacity: @write_buffer_bytes, encoding: Encoding::BINARY)
|
|
337
|
+
@write_buffer.each { |part| out << part.b }
|
|
338
|
+
out
|
|
339
|
+
end
|
|
340
|
+
end
|
|
341
|
+
|
|
342
|
+
def discard_write_buffer
|
|
343
|
+
@write_buffer.clear
|
|
344
|
+
@write_buffer_bytes = 0
|
|
345
|
+
end
|
|
258
346
|
end
|
|
259
347
|
end
|
|
260
348
|
end
|
|
@@ -28,22 +28,25 @@ module Dalli
|
|
|
28
28
|
@value_marshaller = value_marshaller
|
|
29
29
|
end
|
|
30
30
|
|
|
31
|
-
|
|
31
|
+
# raw: true (a request made with raw: true) returns the value as stored,
|
|
32
|
+
# ignoring any flags in the reply. The request didn't ask for flags, so
|
|
33
|
+
# a reply carrying them anyway must not get the value deserialized.
|
|
34
|
+
def meta_get_with_value(cache_nils: false, raw: false)
|
|
32
35
|
tokens = error_on_unexpected!([VA, EN, HD])
|
|
33
36
|
return cache_nils ? ::Dalli::NOT_FOUND : nil if tokens.first == EN
|
|
34
37
|
return true unless tokens.first == VA
|
|
35
38
|
|
|
36
|
-
|
|
39
|
+
retrieve(read_data(tokens[1].to_i), raw ? 0 : bitflags_from_tokens(tokens), raw)
|
|
37
40
|
end
|
|
38
41
|
|
|
39
|
-
def meta_get_with_value_and_cas
|
|
42
|
+
def meta_get_with_value_and_cas(raw: false)
|
|
40
43
|
tokens = error_on_unexpected!([VA, EN, HD])
|
|
41
44
|
return [nil, 0] if tokens.first == EN
|
|
42
45
|
|
|
43
46
|
cas = cas_from_tokens(tokens)
|
|
44
47
|
return [nil, cas] unless tokens.first == VA
|
|
45
48
|
|
|
46
|
-
[
|
|
49
|
+
[retrieve(read_data(tokens[1].to_i), raw ? 0 : bitflags_from_tokens(tokens), raw), cas]
|
|
47
50
|
end
|
|
48
51
|
|
|
49
52
|
def meta_get_without_value
|
|
@@ -62,12 +65,13 @@ module Dalli
|
|
|
62
65
|
#
|
|
63
66
|
# Used by meta_get for comprehensive metadata retrieval.
|
|
64
67
|
# Supports thundering herd protection (N/R flags) and metadata flags (h/l/u).
|
|
65
|
-
def meta_get_with_metadata(cache_nils: false, return_hit_status: false, return_last_access: false
|
|
68
|
+
def meta_get_with_metadata(cache_nils: false, return_hit_status: false, return_last_access: false,
|
|
69
|
+
raw: false)
|
|
66
70
|
tokens = error_on_unexpected!([VA, EN, HD])
|
|
67
71
|
result = build_metadata_result(tokens)
|
|
68
72
|
result[:hit_before] = hit_status_from_tokens(tokens) if return_hit_status
|
|
69
73
|
result[:last_access] = last_access_from_tokens(tokens) if return_last_access
|
|
70
|
-
result[:value] = parse_value_from_tokens(tokens, cache_nils)
|
|
74
|
+
result[:value] = parse_value_from_tokens(tokens, cache_nils, raw)
|
|
71
75
|
result
|
|
72
76
|
end
|
|
73
77
|
|
|
@@ -79,11 +83,16 @@ module Dalli
|
|
|
79
83
|
}
|
|
80
84
|
end
|
|
81
85
|
|
|
82
|
-
def parse_value_from_tokens(tokens, cache_nils)
|
|
86
|
+
def parse_value_from_tokens(tokens, cache_nils, raw = false) # rubocop:disable Style/OptionalBooleanParameter
|
|
83
87
|
return cache_nils ? ::Dalli::NOT_FOUND : nil if tokens.first == EN
|
|
84
88
|
return unless tokens.first == VA
|
|
85
89
|
|
|
86
|
-
|
|
90
|
+
retrieve(read_data(tokens[1].to_i), raw ? 0 : bitflags_from_tokens(tokens), raw)
|
|
91
|
+
end
|
|
92
|
+
|
|
93
|
+
# Values from a raw request are returned as stored
|
|
94
|
+
def retrieve(value, bitflags, raw)
|
|
95
|
+
raw ? value : @value_marshaller.retrieve(value, bitflags)
|
|
87
96
|
end
|
|
88
97
|
|
|
89
98
|
def meta_set_with_cas
|
|
@@ -148,6 +157,10 @@ module Dalli
|
|
|
148
157
|
end
|
|
149
158
|
|
|
150
159
|
def full_response_from_buffer(tokens, body, resp_size)
|
|
160
|
+
# A hit with no key can't be matched to a request: skip it (false
|
|
161
|
+
# status) rather than yield it, or take it for the end of the pipeline
|
|
162
|
+
return [resp_size, false, nil, nil, nil] unless tokens.any? { |t| t.start_with?('k') }
|
|
163
|
+
|
|
151
164
|
value = @value_marshaller.retrieve(body, bitflags_from_tokens(tokens))
|
|
152
165
|
[resp_size, tokens.first == VA, cas_from_tokens(tokens), key_from_tokens(tokens), value]
|
|
153
166
|
end
|
|
@@ -172,13 +185,14 @@ module Dalli
|
|
|
172
185
|
header_len = header.bytesize + TERMINATOR.length
|
|
173
186
|
body_len = body_len_from_tokens(tokens)
|
|
174
187
|
|
|
175
|
-
#
|
|
176
|
-
#
|
|
177
|
-
#
|
|
178
|
-
#
|
|
179
|
-
#
|
|
180
|
-
#
|
|
181
|
-
|
|
188
|
+
# A complete response with no body. Only the terminating noop's MN
|
|
189
|
+
# ends the pipeline. Any other bodyless reply (CLIENT_ERROR,
|
|
190
|
+
# SERVER_ERROR, ...) answers one key's request: report it with a
|
|
191
|
+
# false status so the caller skips it and keeps reading. Treating it
|
|
192
|
+
# as the end would leave the remaining replies on the connection for
|
|
193
|
+
# later commands to read as their own. A hit on an empty value (VA 0)
|
|
194
|
+
# still has a body -- just its terminator -- so it's parsed below.
|
|
195
|
+
return [header_len, tokens.first == MN, nil, nil, nil] if no_body?(tokens, body_len)
|
|
182
196
|
|
|
183
197
|
resp_size = header_len + body_len + TERMINATOR.length
|
|
184
198
|
# The header is in the buffer, but the body is not. As we don't have
|
|
@@ -191,6 +205,22 @@ module Dalli
|
|
|
191
205
|
full_response_from_buffer(tokens, body, resp_size)
|
|
192
206
|
end
|
|
193
207
|
|
|
208
|
+
# memcached can't store an item larger than 1 GiB (its -I maximum). A
|
|
209
|
+
# pipelined reply claiming an impossible size would otherwise have the
|
|
210
|
+
# buffer wait for (and accumulate) that many bytes.
|
|
211
|
+
MAX_VALUE_BYTES = 1024 * 1024 * 1024
|
|
212
|
+
|
|
213
|
+
# The size given right after "VA", or 0 when it isn't a number
|
|
214
|
+
def positional_size(tokens)
|
|
215
|
+
tokens.first == VA && tokens[1]&.match?(/\A\d+\z/) ? tokens[1].to_i : 0
|
|
216
|
+
end
|
|
217
|
+
|
|
218
|
+
def check_value_size!(size)
|
|
219
|
+
return if size.between?(0, MAX_VALUE_BYTES)
|
|
220
|
+
|
|
221
|
+
raise Dalli::DalliError, "Reply value size #{size} is out of range"
|
|
222
|
+
end
|
|
223
|
+
|
|
194
224
|
# A zero-size reply has no body unless it's a VA (a hit on an empty value)
|
|
195
225
|
def no_body?(tokens, body_len)
|
|
196
226
|
body_len.zero? && tokens.first != VA
|
|
@@ -236,7 +266,11 @@ module Dalli
|
|
|
236
266
|
end
|
|
237
267
|
|
|
238
268
|
def body_len_from_tokens(tokens)
|
|
239
|
-
|
|
269
|
+
# memcached also gives the size positionally ("VA <size> ..."); use it
|
|
270
|
+
# when there's no s flag rather than reading the value as replies
|
|
271
|
+
size = tokens.any? { |t| t.start_with?('s') } ? value_from_tokens(tokens, 's')&.to_i : positional_size(tokens)
|
|
272
|
+
check_value_size!(size) if size
|
|
273
|
+
size
|
|
240
274
|
end
|
|
241
275
|
|
|
242
276
|
def value_from_tokens(tokens, flag)
|
|
@@ -256,6 +290,9 @@ module Dalli
|
|
|
256
290
|
end
|
|
257
291
|
|
|
258
292
|
def read_data(data_size)
|
|
293
|
+
# Checked before the terminator is added: -1 or -2 would otherwise
|
|
294
|
+
# read 1 or 0 bytes and leave the connection out of step
|
|
295
|
+
check_value_size!(data_size)
|
|
259
296
|
@io_source.read(data_size + TERMINATOR.bytesize)&.chomp!(TERMINATOR)
|
|
260
297
|
end
|
|
261
298
|
end
|
data/lib/dalli/protocol/meta.rb
CHANGED
|
@@ -30,7 +30,7 @@ module Dalli
|
|
|
30
30
|
req = RequestFormatter.meta_get(key: encoded_key, base64: base64, skip_flags: skip_flags)
|
|
31
31
|
write(req)
|
|
32
32
|
@connection_manager.flush
|
|
33
|
-
response_processor.meta_get_with_value(cache_nils: cache_nils?(options))
|
|
33
|
+
response_processor.meta_get_with_value(cache_nils: cache_nils?(options), raw: raw_request?(options))
|
|
34
34
|
end
|
|
35
35
|
|
|
36
36
|
def quiet_get_request(key)
|
|
@@ -47,7 +47,7 @@ module Dalli
|
|
|
47
47
|
req = RequestFormatter.meta_get(key: encoded_key, ttl: ttl, base64: base64, skip_flags: skip_flags)
|
|
48
48
|
write(req)
|
|
49
49
|
@connection_manager.flush
|
|
50
|
-
response_processor.meta_get_with_value(cache_nils: cache_nils?(options))
|
|
50
|
+
response_processor.meta_get_with_value(cache_nils: cache_nils?(options), raw: raw_request?(options))
|
|
51
51
|
end
|
|
52
52
|
|
|
53
53
|
def touch(key, ttl)
|
|
@@ -61,12 +61,13 @@ module Dalli
|
|
|
61
61
|
|
|
62
62
|
# TODO: This is confusing, as there's a cas command in memcached
|
|
63
63
|
# and this isn't it. Maybe rename? Maybe eliminate?
|
|
64
|
-
def cas(key)
|
|
64
|
+
def cas(key, options = nil)
|
|
65
65
|
encoded_key, base64 = KeyRegularizer.encode(key)
|
|
66
|
-
req = RequestFormatter.meta_get(key: encoded_key, value: true, return_cas: true, base64: base64
|
|
66
|
+
req = RequestFormatter.meta_get(key: encoded_key, value: true, return_cas: true, base64: base64,
|
|
67
|
+
skip_flags: raw_request?(options))
|
|
67
68
|
write(req)
|
|
68
69
|
@connection_manager.flush
|
|
69
|
-
response_processor.meta_get_with_value_and_cas
|
|
70
|
+
response_processor.meta_get_with_value_and_cas(raw: raw_request?(options))
|
|
70
71
|
end
|
|
71
72
|
|
|
72
73
|
# Comprehensive meta get with support for all metadata flags.
|
|
@@ -96,7 +97,7 @@ module Dalli
|
|
|
96
97
|
def meta_get(key, options = {})
|
|
97
98
|
encoded_key, base64 = KeyRegularizer.encode(key)
|
|
98
99
|
req = RequestFormatter.meta_get(
|
|
99
|
-
key: encoded_key, value: true, return_cas: true, base64: base64,
|
|
100
|
+
key: encoded_key, value: true, return_cas: true, base64: base64, skip_flags: raw_request?(options),
|
|
100
101
|
vivify_ttl: options[:vivify_ttl], recache_ttl: options[:recache_ttl],
|
|
101
102
|
return_hit_status: options[:return_hit_status],
|
|
102
103
|
return_last_access: options[:return_last_access], skip_lru_bump: options[:skip_lru_bump]
|
|
@@ -105,7 +106,7 @@ module Dalli
|
|
|
105
106
|
@connection_manager.flush
|
|
106
107
|
response_processor.meta_get_with_metadata(
|
|
107
108
|
cache_nils: cache_nils?(options), return_hit_status: options[:return_hit_status],
|
|
108
|
-
return_last_access: options[:return_last_access]
|
|
109
|
+
return_last_access: options[:return_last_access], raw: raw_request?(options)
|
|
109
110
|
)
|
|
110
111
|
end
|
|
111
112
|
|
|
@@ -15,11 +15,17 @@ module Dalli
|
|
|
15
15
|
compress: true,
|
|
16
16
|
compressor: ::Dalli::Compressor,
|
|
17
17
|
# min byte size to attempt compression
|
|
18
|
-
compression_min_size: 4 * 1024 # 4K
|
|
18
|
+
compression_min_size: 4 * 1024, # 4K
|
|
19
|
+
# max size a stored value may decompress to (nil: no limit). Guards
|
|
20
|
+
# against a small compressed value expanding into gigabytes on read.
|
|
21
|
+
decompressed_max_bytes: 128 * 1024 * 1024 # 128 MiB
|
|
19
22
|
}.freeze
|
|
20
23
|
|
|
21
24
|
OPTIONS = DEFAULTS.keys.freeze
|
|
22
25
|
|
|
26
|
+
KEYWORD_PARAMETER_TYPES = %i[key keyreq].freeze
|
|
27
|
+
private_constant :KEYWORD_PARAMETER_TYPES
|
|
28
|
+
|
|
23
29
|
# https://www.hjp.at/zettel/m/memcached_flags.rxml
|
|
24
30
|
# Looks like most clients use bit 1 to indicate gzip compression.
|
|
25
31
|
FLAG_COMPRESSED = 0x2
|
|
@@ -38,8 +44,16 @@ module Dalli
|
|
|
38
44
|
end
|
|
39
45
|
|
|
40
46
|
def retrieve(value, bitflags)
|
|
41
|
-
|
|
42
|
-
|
|
47
|
+
return value unless bitflags.anybits?(FLAG_COMPRESSED)
|
|
48
|
+
|
|
49
|
+
max_bytes = @compression_options[:decompressed_max_bytes]
|
|
50
|
+
# Custom compressors that only define decompress(data) keep working,
|
|
51
|
+
# without the limit
|
|
52
|
+
if max_bytes && compressor_accepts_max_bytes?
|
|
53
|
+
compressor.decompress(value, max_bytes: max_bytes)
|
|
54
|
+
else
|
|
55
|
+
compressor.decompress(value)
|
|
56
|
+
end
|
|
43
57
|
|
|
44
58
|
# TODO: We likely want to move this rescue into the Dalli::Compressor / Dalli::GzipCompressor
|
|
45
59
|
# itself, since not all compressors necessarily use Zlib. For now keep it here, so the behavior
|
|
@@ -48,6 +62,19 @@ module Dalli
|
|
|
48
62
|
raise UnmarshalError, "Unable to uncompress value: #{$ERROR_INFO.message}"
|
|
49
63
|
end
|
|
50
64
|
|
|
65
|
+
def compressor_accepts_max_bytes?
|
|
66
|
+
return @compressor_accepts_max_bytes if defined?(@compressor_accepts_max_bytes)
|
|
67
|
+
|
|
68
|
+
@compressor_accepts_max_bytes =
|
|
69
|
+
begin
|
|
70
|
+
compressor.method(:decompress).parameters.any? do |type, name|
|
|
71
|
+
name == :max_bytes && KEYWORD_PARAMETER_TYPES.include?(type)
|
|
72
|
+
end
|
|
73
|
+
rescue NameError # an object without a reflectable decompress method
|
|
74
|
+
false
|
|
75
|
+
end
|
|
76
|
+
end
|
|
77
|
+
|
|
51
78
|
def compress_by_default?
|
|
52
79
|
@compression_options[:compress]
|
|
53
80
|
end
|
|
@@ -102,7 +102,7 @@ module Dalli
|
|
|
102
102
|
Dalli.logger.warn 'SECURITY WARNING: Dalli is using Marshal for serialization. ' \
|
|
103
103
|
'Marshal can execute arbitrary code during deserialization. ' \
|
|
104
104
|
'If your memcached server could be compromised, consider using ' \
|
|
105
|
-
'a safer serializer
|
|
105
|
+
'a safer serializer: Dalli::Client.new(servers, serializer: Dalli::JSONSerializer)'
|
|
106
106
|
@@marshal_warning_logged = true # rubocop:disable Style/ClassVars
|
|
107
107
|
end
|
|
108
108
|
end
|
data/lib/dalli/version.rb
CHANGED
data/lib/dalli.rb
CHANGED
|
@@ -62,6 +62,7 @@ require_relative 'dalli/version'
|
|
|
62
62
|
require_relative 'dalli/instrumentation'
|
|
63
63
|
|
|
64
64
|
require_relative 'dalli/compressor'
|
|
65
|
+
require_relative 'dalli/json_serializer'
|
|
65
66
|
require_relative 'dalli/protocol_deprecations'
|
|
66
67
|
require_relative 'dalli/client'
|
|
67
68
|
require_relative 'dalli/key_manager'
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: dalli
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 4.3.
|
|
4
|
+
version: 4.3.7
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Peter M. Goldstein
|
|
@@ -41,6 +41,7 @@ files:
|
|
|
41
41
|
- lib/dalli/client.rb
|
|
42
42
|
- lib/dalli/compressor.rb
|
|
43
43
|
- lib/dalli/instrumentation.rb
|
|
44
|
+
- lib/dalli/json_serializer.rb
|
|
44
45
|
- lib/dalli/key_manager.rb
|
|
45
46
|
- lib/dalli/options.rb
|
|
46
47
|
- lib/dalli/pid_cache.rb
|