dalli 3.2.11 → 3.2.12

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 17d814f4043cf5882166c1519dd436fb4a7d0caef32c86789dd5da3c5e51e52c
4
- data.tar.gz: 27ba5ff384558a7ab08c7dcb3e6f2cf43af4c04dddc1ce2e5ca4ab1d35b01001
3
+ metadata.gz: f8f2b8a277c11f36013ba0a3997f962e38a9ce19a4dbe05dfc42a647016d03e0
4
+ data.tar.gz: 19945129c63f04cf6dc704063eb31bd243998a987b8f8b729345d5b554d2ca36
5
5
  SHA512:
6
- metadata.gz: 63914cbb89b5a5d517dfd11f72b6bd56b22c9cb0700d9d53a037e12c7d9872150038f5a50b20f959bda47160644a30a433720e99da54ad048b15fee03ddb9baf
7
- data.tar.gz: 11ede1a2e918d269cd084f171c97e5bad156b96fe0384e366cd531cc68a543bd1061b34ac437a7c2794e27ad6f63a3661e035c01bb4e8172d6753cfbca839fd1
6
+ metadata.gz: 3cc736edcbbcb13054fba7ace600d0e88b4b6e323b84ee11446dd57071b6f9a8db525050b0a7b3c8166c943efb042d791dc62a38f930bed7a5b818ad96fc9152
7
+ data.tar.gz: db46135255631a70767867449b0ed61b14d59ce2a8e8b65fe809a17f660a8d4f46a5c62197fa8d408371f7697711d4fae92031721e13809519b52a94427dd380
data/CHANGELOG.md CHANGED
@@ -4,6 +4,35 @@ Dalli Changelog
4
4
  Unreleased
5
5
  ==========
6
6
 
7
+ 3.2.12
8
+ ==========
9
+
10
+ - Fix pipelined `get_multi` returning one key's value for another after an error reply (GHSA-p6pm-ch9v-44vx)
11
+ - The pipelined reply parser took any reply without a body for the end of the batch. An error reply for one key (`CLIENT_ERROR`, `SERVER_ERROR`, or `EN` from a proxy) ended it early, and the replies still on the connection were read as the replies to later commands, so a `get` could return another key's value
12
+ - Keys are now measured in bytes as sent (after base64 encoding when the meta protocol needs it). A key under 250 characters but over 250 bytes, for example one with many non-ASCII characters, passed validation:
13
+ - with the meta protocol, memcached rejected it with the error reply above
14
+ - with the binary protocol, memcached dropped the connection, and `get` or `get_multi` with that key retried forever
15
+ - Such keys are now truncated like other long keys; keys that worked before are unchanged
16
+ - The reply parser affects `protocol: :meta` since 3.2.0; the key length affects both protocols; fixed in 5.2.1, 5.1.3, 5.0.9, 4.3.6 and 3.2.12
17
+ - Stop retrying an unresponsive server forever (GHSA-4qp6-2jcr-596v)
18
+ - A server that accepted connections but never answered (or dropped the connection on a request) was retried forever, hanging the caller, because each successful reconnect reset the failure count. Requests now fail after `socket_max_failures` attempts and the server is marked down, as when it can't be reached at all; it gets a full set of attempts again after `down_retry_delay`
19
+ - Affects all versions; fixed in 5.2.1, 5.1.3, 5.0.9, 4.3.6 and 3.2.12
20
+ - Limit decompressed and reply value sizes (GHSA-3553-vcg5-72jw)
21
+ - Values flagged as compressed were inflated without limit, so a small stored value could expand to gigabytes on read, whatever the client's `compress` or `serializer` settings. The new `decompressed_max_bytes` option (default 128 MiB; `nil` disables it) makes a read that would pass it raise `Dalli::UnmarshalError`. Custom compressors whose `decompress` takes only the data keep working, without the limit
22
+ - The value size in a reply (a meta `VA` size or a binary body length) was used to read or buffer that many bytes, so a malicious server could make the client allocate gigabytes. Sizes over 1 GiB, memcached's largest item, now raise `Dalli::DalliError` before reading
23
+ - Affects all versions; fixed in 5.2.1, 5.1.3, 5.0.9, 4.3.6 and 3.2.12
24
+ - Honor per-request `raw: true` on reads, and add `Dalli::JSONSerializer` (GHSA-wr87-m4jw-29x5)
25
+ - `get`, `gat` and `fetch` ignored a per-request `raw: true` and deserialized the value according to its stored flags, so a caller who asked for raw bytes could still have `Marshal.load` run on data someone else wrote to memcached. They now return the stored bytes
26
+ - The `raw` part affects all versions
27
+ - `serializer: JSON` reads values with `JSON.load`, which on json gem versions before 3.0 creates an object of the class named in a stored `json_class` key when the json additions are loaded. The new `Dalli::JSONSerializer` reads with `JSON.parse` and only returns plain JSON types. The README now recommends it; `serializer: JSON` itself is unchanged
28
+ - The `serializer: JSON` part affects any version used with the json gem before 3.0
29
+ - Keep a forked child from closing the parent's TLS session (GHSA-w39f-xq2m-4g8x)
30
+ - With TLS, a forked child that closed the client (or reconnected after detecting the fork) sent a TLS close on the connection the parent was still using, ending the parent's session. A forked child now closes only its own copy of the socket
31
+ - Affects all versions with TLS; fixed in 5.2.1, 5.1.3, 5.0.9, 4.3.6 and 3.2.12
32
+
33
+ - Fix three flaky tests: the `KeyManager` namespace key-length test, the Rack session freshness test, and `MemcachedManager`'s start and stop handling (backport of #1141, #1183)
34
+ - Fix flaky failover tests: move their ports out of Linux's ephemeral port range, and wait for memcached to accept connections after starting it (backport of #1184)
35
+
7
36
  3.2.11
8
37
  ==========
9
38
 
data/README.md CHANGED
@@ -25,6 +25,18 @@ The name is a variant of Salvador Dali for his famous painting [The Persistence
25
25
  * [Forum](https://github.com/petergoldstein/dalli/discussions/categories/q-a) - If you have questions about Dalli, please post them here.
26
26
  * [Client API](https://www.rubydoc.info/gems/dalli) - Ruby documentation for the `Dalli::Client` API
27
27
 
28
+ ## Security Note
29
+
30
+ By default, Dalli uses Ruby's Marshal for serialization. Deserializing untrusted data with Marshal can lead to remote code execution. If you cache user-controlled data, or your memcached server could be written to by someone else, consider using a safer serializer:
31
+
32
+ ```ruby
33
+ Dalli::Client.new('localhost:11211', serializer: Dalli::JSONSerializer)
34
+ ```
35
+
36
+ `Dalli::JSONSerializer` reads values with `JSON.parse`, so it only ever returns plain hashes, arrays, strings, numbers, booleans and `nil`. Passing the `JSON` module itself (`serializer: JSON`) reads values with `JSON.load`, which can create objects of other classes from a stored `json_class` key and is not recommended for data an attacker might write.
37
+
38
+ Per-request `raw: true` returns the stored bytes without deserializing or decompressing them, on every read method that accepts it (`get` and `fetch`).
39
+
28
40
  ## Development
29
41
 
30
42
  After checking out the repo, run `bin/setup` to install dependencies. You can run `bin/console` for an interactive prompt that will allow you to experiment.
@@ -13,8 +13,49 @@ module Dalli
13
13
  Zlib::Deflate.deflate(data)
14
14
  end
15
15
 
16
- def self.decompress(data)
17
- Zlib::Inflate.inflate(data)
16
+ # DEFLATE can't expand data by more than about 1032:1
17
+ MAX_EXPANSION = 1032
18
+ # Compressed input is fed to zlib this many bytes at a time. MRI's zlib
19
+ # yields the output in small chunks; one that returns it instead still
20
+ # produces at most MAX_EXPANSION times this per step.
21
+ INFLATE_SLICE_BYTES = 64 * 1024
22
+
23
+ # max_bytes caps the decompressed size; inflating stops and raises as soon
24
+ # as the output passes it, so a small compressed value can't expand into
25
+ # gigabytes of memory.
26
+ def self.decompress(data, max_bytes: nil)
27
+ return Zlib::Inflate.inflate(data) unless may_exceed?(data, max_bytes)
28
+
29
+ inflate_within_limit(Zlib::Inflate.new, data, max_bytes)
30
+ end
31
+
32
+ # False when even the largest possible expansion stays within max_bytes,
33
+ # so the value can be inflated in one step
34
+ def self.may_exceed?(data, max_bytes)
35
+ max_bytes ? data.bytesize * MAX_EXPANSION > max_bytes : false
36
+ end
37
+
38
+ def self.inflate_within_limit(inflater, data, max_bytes)
39
+ out = String.new(encoding: Encoding::BINARY)
40
+ 0.step(data.bytesize - 1, INFLATE_SLICE_BYTES) do |pos|
41
+ inflate_slice(inflater, data.byteslice(pos, INFLATE_SLICE_BYTES), out, max_bytes)
42
+ end
43
+ append_within_limit(out, inflater.finish, max_bytes)
44
+ ensure
45
+ inflater.close
46
+ end
47
+
48
+ # Both yielded and returned output count toward the limit
49
+ def self.inflate_slice(inflater, slice, out, max_bytes)
50
+ rest = inflater.inflate(slice) { |chunk| append_within_limit(out, chunk, max_bytes) }
51
+ append_within_limit(out, rest, max_bytes)
52
+ end
53
+
54
+ def self.append_within_limit(out, chunk, max_bytes)
55
+ out << chunk if chunk
56
+ return out if out.bytesize <= max_bytes
57
+
58
+ raise Dalli::UnmarshalError, "Decompressed value exceeds #{max_bytes} bytes (decompressed_max_bytes)"
18
59
  end
19
60
  end
20
61
 
@@ -32,9 +73,11 @@ module Dalli
32
73
  io.string
33
74
  end
34
75
 
35
- def self.decompress(data)
36
- io = StringIO.new(data, 'rb')
37
- Zlib::GzipReader.new(io).read
76
+ def self.decompress(data, max_bytes: nil)
77
+ return Zlib::GzipReader.new(StringIO.new(data, 'rb')).read unless Compressor.may_exceed?(data, max_bytes)
78
+
79
+ # MAX_WBITS + 16 makes zlib read the gzip format (and check its CRC)
80
+ Compressor.inflate_within_limit(Zlib::Inflate.new(Zlib::MAX_WBITS + 16), data, max_bytes)
38
81
  end
39
82
  end
40
83
  end
@@ -0,0 +1,28 @@
1
+ # frozen_string_literal: true
2
+
3
+ require 'json'
4
+
5
+ module Dalli
6
+ ##
7
+ # A serializer that stores values as JSON and never creates Ruby objects
8
+ # from what it reads.
9
+ #
10
+ # Passing the JSON module itself as the serializer (serializer: JSON) reads
11
+ # values back with JSON.load, which on json gem versions before 3.0 honors
12
+ # "json_class" and instantiates other classes when the json gem's additions
13
+ # are loaded. This serializer reads with JSON.parse, so a value written to
14
+ # memcached by someone else can only come back as plain hashes, arrays,
15
+ # strings, numbers, booleans and nil.
16
+ #
17
+ # Dalli::Client.new(servers, serializer: Dalli::JSONSerializer)
18
+ ##
19
+ module JSONSerializer
20
+ def self.dump(value)
21
+ JSON.generate(value)
22
+ end
23
+
24
+ def self.load(data)
25
+ JSON.parse(data)
26
+ end
27
+ end
28
+ end
@@ -34,6 +34,9 @@ module Dalli
34
34
  validate_digest_class_option(@key_options)
35
35
 
36
36
  @namespace = namespace_from_options
37
+ # The meta protocol base64-encodes keys with whitespace or non-ASCII
38
+ # characters, so their length on the wire differs from their length here
39
+ @meta_protocol = client_options[:protocol]&.to_s == 'meta'
37
40
  end
38
41
 
39
42
  ##
@@ -51,7 +54,7 @@ module Dalli
51
54
  raise ArgumentError, 'key cannot be blank' unless key&.length&.positive?
52
55
 
53
56
  key = key_with_namespace(key)
54
- key.length > MAX_KEY_LENGTH ? truncated_key(key) : key
57
+ wire_length(key) > MAX_KEY_LENGTH ? truncated_key(key) : key
55
58
  end
56
59
 
57
60
  ##
@@ -107,7 +110,27 @@ module Dalli
107
110
  ##
108
111
  def truncated_key(key)
109
112
  digest = digest_class.hexdigest(key)
110
- "#{key[0, prefix_length(digest)]}#{TRUNCATED_KEY_SEPARATOR}#{digest}"
113
+ prefix = key[0, prefix_length(digest)]
114
+ truncated = "#{prefix}#{TRUNCATED_KEY_SEPARATOR}#{digest}"
115
+ # A prefix with whitespace or non-ASCII characters sends the whole key
116
+ # base64-encoded, which can still be too long on the wire; shorten it.
117
+ while wire_length(truncated) > MAX_KEY_LENGTH
118
+ prefix = prefix[0...-1]
119
+ truncated = "#{prefix}#{TRUNCATED_KEY_SEPARATOR}#{digest}"
120
+ end
121
+ truncated
122
+ end
123
+
124
+ # memcached's key limit applies to the key as sent: in bytes, and, with
125
+ # the meta protocol, after base64 encoding when the key needs it. A key
126
+ # within the limit in characters can still be too long. With the meta
127
+ # protocol memcached rejects it with a CLIENT_ERROR that must never reach
128
+ # a pipelined get; with the binary protocol it drops the connection.
129
+ def wire_length(key)
130
+ return key.bytesize unless @meta_protocol
131
+
132
+ encoded, = Dalli::Protocol::Meta::KeyRegularizer.encode(key)
133
+ encoded.bytesize
111
134
  end
112
135
 
113
136
  def prefix_length(digest)
@@ -27,6 +27,12 @@ module Dalli
27
27
  @connection_manager = ConnectionManager.new(hostname, port, socket_type, @options)
28
28
  end
29
29
 
30
+ # True when this request passed raw: true. A stored value is then
31
+ # returned as stored, never deserialized or decompressed.
32
+ def raw_request?(options)
33
+ options.is_a?(Hash) && options[:raw] ? true : false
34
+ end
35
+
30
36
  # Chokepoint method for error handling and ensuring liveness
31
37
  def request(opkey, *args)
32
38
  verify_state(opkey)
@@ -66,7 +66,9 @@ module Dalli
66
66
  raise Dalli::DalliError, "Response error #{resp_header.status}: #{RESPONSE_CODES[resp_header.status]}"
67
67
  end
68
68
 
69
- def get(cache_nils: false)
69
+ # raw: true returns the value as stored, without deserializing or
70
+ # decompressing it, whatever flags it was stored with
71
+ def get(cache_nils: false, raw: false)
70
72
  resp_header, body = read_response
71
73
 
72
74
  return false if resp_header.not_stored? # Not stored, normal status for add operation
@@ -75,7 +77,7 @@ module Dalli
75
77
  raise_on_not_ok!(resp_header)
76
78
  return true unless body
77
79
 
78
- unpack_response_body(resp_header, body, true).last
80
+ unpack_response_body(resp_header, body, !raw).last
79
81
  end
80
82
 
81
83
  ##
@@ -209,6 +211,17 @@ module Dalli
209
211
  # The remaining three values in the array are the ResponseHeader,
210
212
  # key, and value.
211
213
  ##
214
+ # memcached can't store an item larger than 1 GiB (its -I maximum). A
215
+ # pipelined reply claiming an impossible size would otherwise have the
216
+ # buffer wait for (and accumulate) that many bytes.
217
+ MAX_BODY_BYTES = 1024 * 1024 * 1024
218
+
219
+ def check_body_size!(size)
220
+ return if size <= MAX_BODY_BYTES
221
+
222
+ raise Dalli::DalliError, "Reply body size #{size} is out of range"
223
+ end
224
+
212
225
  def getk_response_from_buffer(buf)
213
226
  # There's no header in the buffer, so don't advance
214
227
  return [0, nil, nil, nil, nil] unless contains_header?(buf)
@@ -222,6 +235,8 @@ module Dalli
222
235
  # error response that needs to be discarded.
223
236
  return [ResponseHeader::SIZE, resp_header.ok?, resp_header.cas, nil, nil] if body_len.zero?
224
237
 
238
+ check_body_size!(body_len)
239
+
225
240
  resp_size = ResponseHeader::SIZE + body_len
226
241
  # The header is in the buffer, but the body is not. As we don't have
227
242
  # a complete response, don't advance the buffer
@@ -22,7 +22,7 @@ module Dalli
22
22
  def get(key, options = nil)
23
23
  req = RequestFormatter.standard_request(opkey: :get, key: key)
24
24
  write(req)
25
- response_processor.get(cache_nils: cache_nils?(options))
25
+ response_processor.get(cache_nils: cache_nils?(options), raw: raw_request?(options))
26
26
  end
27
27
 
28
28
  def quiet_get_request(key)
@@ -33,7 +33,7 @@ module Dalli
33
33
  ttl = TtlSanitizer.sanitize(ttl)
34
34
  req = RequestFormatter.standard_request(opkey: :gat, key: key, ttl: ttl)
35
35
  write(req)
36
- response_processor.get(cache_nils: cache_nils?(options))
36
+ response_processor.get(cache_nils: cache_nils?(options), raw: raw_request?(options))
37
37
  end
38
38
 
39
39
  def touch(key, ttl)
@@ -38,6 +38,7 @@ module Dalli
38
38
  @sock = nil
39
39
  @pid = nil
40
40
 
41
+ @fail_count = 0
41
42
  reset_down_info
42
43
  end
43
44
 
@@ -84,6 +85,8 @@ module Dalli
84
85
  def down!
85
86
  close
86
87
  log_down_detected
88
+ # Once down_retry_delay passes, the server gets a full set of attempts
89
+ @fail_count = 0
87
90
 
88
91
  @error = $ERROR_INFO&.class&.name
89
92
  @msg ||= $ERROR_INFO&.message
@@ -113,7 +116,7 @@ module Dalli
113
116
  return unless @sock
114
117
 
115
118
  begin
116
- @sock.close
119
+ close_socket
117
120
  rescue StandardError
118
121
  nil
119
122
  end
@@ -122,6 +125,13 @@ module Dalli
122
125
  abort_request!
123
126
  end
124
127
 
128
+ # A forked child shares the parent's connection, so it closes only its
129
+ # own file descriptor. Closing the TLS socket itself would send
130
+ # close_notify and end the parent's TLS session.
131
+ def close_socket
132
+ fork_detected? ? @sock.to_io.close : @sock.close
133
+ end
134
+
125
135
  def connected?
126
136
  !@sock.nil?
127
137
  end
@@ -140,6 +150,11 @@ module Dalli
140
150
  raise '[Dalli] No request in progress. This may be a bug in Dalli.' unless @request_in_progress
141
151
 
142
152
  @request_in_progress = false
153
+ # A completed request is what proves the server healthy again, so the
154
+ # failure count resets here rather than on reconnect: a server that
155
+ # accepts connections but never answers would otherwise reset it on
156
+ # every retry and never reach socket_max_failures.
157
+ @fail_count = 0
143
158
  end
144
159
 
145
160
  def abort_request!
@@ -154,7 +169,19 @@ module Dalli
154
169
  error_on_request!(e)
155
170
  end
156
171
 
172
+ # memcached can't store an item larger than 1 GiB (its -I maximum), so a
173
+ # reply claiming more (or a negative size) is malformed or hostile.
174
+ # Reading allocates the full count up front, so check before reading.
175
+ MAX_READ_BYTES = (1024 * 1024 * 1024) + 2 # plus the value's trailing "\r\n"
176
+
177
+ def check_read_size!(count)
178
+ return if count.between?(0, MAX_READ_BYTES)
179
+
180
+ raise Dalli::DalliError, "Reply size #{count} from #{name} is out of range"
181
+ end
182
+
157
183
  def read(count)
184
+ check_read_size!(count)
158
185
  @sock.readfull(count)
159
186
  rescue SystemCallError, *TIMEOUT_ERRORS, EOFError => e
160
187
  error_on_request!(e)
@@ -195,8 +222,9 @@ module Dalli
195
222
  raise Dalli::NetworkError, message
196
223
  end
197
224
 
225
+ # Called on connect. Deliberately leaves @fail_count alone; see
226
+ # finish_request!.
198
227
  def reset_down_info
199
- @fail_count = 0
200
228
  @down_at = nil
201
229
  @last_down_at = nil
202
230
  @msg = nil
@@ -222,8 +250,8 @@ module Dalli
222
250
  def close_on_fork
223
251
  message = 'Fork detected, re-connecting child process...'
224
252
  Dalli.logger.info { message }
225
- # Close socket on a fork, setting us up for reconnect
226
- # on next request.
253
+ # Closes the inherited socket without touching the parent's
254
+ # connection, setting us up for reconnect on next request.
227
255
  close
228
256
  raise Dalli::NetworkError, message
229
257
  end
@@ -27,12 +27,15 @@ module Dalli
27
27
  @value_marshaller = value_marshaller
28
28
  end
29
29
 
30
- def meta_get_with_value(cache_nils: false)
30
+ # raw: true returns the value as stored, without deserializing or
31
+ # decompressing it, whatever flags it was stored with
32
+ def meta_get_with_value(cache_nils: false, raw: false)
31
33
  tokens = error_on_unexpected!([VA, EN, HD])
32
34
  return cache_nils ? ::Dalli::NOT_FOUND : nil if tokens.first == EN
33
35
  return true unless tokens.first == VA
34
36
 
35
- @value_marshaller.retrieve(read_line, bitflags_from_tokens(tokens))
37
+ value = read_line
38
+ raw ? value : @value_marshaller.retrieve(value, bitflags_from_tokens(tokens))
36
39
  end
37
40
 
38
41
  def meta_get_with_value_and_cas
@@ -140,13 +143,14 @@ module Dalli
140
143
 
141
144
  tokens, header_len, body_len = tokens_from_header_buffer(buf)
142
145
 
143
- # We have a complete response that has no body.
144
- # This is either the response to the terminating
145
- # noop or, if the status is not MN, an intermediate
146
- # error response that needs to be discarded. A hit
147
- # on an empty value (VA 0) still has a body -- just
148
- # its terminator -- so it's parsed below as a value.
149
- return [header_len, true, nil, nil, nil] if no_body?(tokens, body_len)
146
+ # A complete response with no body. Only the terminating noop's MN
147
+ # ends the pipeline. Any other bodyless reply (CLIENT_ERROR,
148
+ # SERVER_ERROR, ...) answers one key's request: report it with a
149
+ # false status so the caller skips it and keeps reading. Treating it
150
+ # as the end would leave the remaining replies on the connection for
151
+ # later commands to read as their own. A hit on an empty value (VA 0)
152
+ # still has a body -- just its terminator -- so it's parsed below.
153
+ return [header_len, tokens.first == MN, nil, nil, nil] if no_body?(tokens, body_len)
150
154
 
151
155
  resp_size = header_len + body_len + TERMINATOR.length
152
156
  # The header is in the buffer, but the body is not. As we don't have
@@ -159,6 +163,17 @@ module Dalli
159
163
  full_response_from_buffer(tokens, body, resp_size)
160
164
  end
161
165
 
166
+ # memcached can't store an item larger than 1 GiB (its -I maximum). A
167
+ # pipelined reply claiming an impossible size would otherwise have the
168
+ # buffer wait for (and accumulate) that many bytes.
169
+ MAX_VALUE_BYTES = 1024 * 1024 * 1024
170
+
171
+ def check_value_size!(size)
172
+ return if size.between?(0, MAX_VALUE_BYTES)
173
+
174
+ raise Dalli::DalliError, "Reply value size #{size} is out of range"
175
+ end
176
+
162
177
  # A zero-size reply has no body unless it's a VA (a hit on an empty value)
163
178
  def no_body?(tokens, body_len)
164
179
  body_len.zero? && tokens.first != VA
@@ -194,7 +209,9 @@ module Dalli
194
209
  end
195
210
 
196
211
  def body_len_from_tokens(tokens)
197
- value_from_tokens(tokens, 's')&.to_i
212
+ size = value_from_tokens(tokens, 's')&.to_i
213
+ check_value_size!(size) if size
214
+ size
198
215
  end
199
216
 
200
217
  def value_from_tokens(tokens, flag)
@@ -27,7 +27,7 @@ module Dalli
27
27
  encoded_key, base64 = KeyRegularizer.encode(key)
28
28
  req = RequestFormatter.meta_get(key: encoded_key, base64: base64)
29
29
  write(req)
30
- response_processor.meta_get_with_value(cache_nils: cache_nils?(options))
30
+ response_processor.meta_get_with_value(cache_nils: cache_nils?(options), raw: raw_request?(options))
31
31
  end
32
32
 
33
33
  def quiet_get_request(key)
@@ -40,7 +40,7 @@ module Dalli
40
40
  encoded_key, base64 = KeyRegularizer.encode(key)
41
41
  req = RequestFormatter.meta_get(key: encoded_key, ttl: ttl, base64: base64)
42
42
  write(req)
43
- response_processor.meta_get_with_value(cache_nils: cache_nils?(options))
43
+ response_processor.meta_get_with_value(cache_nils: cache_nils?(options), raw: raw_request?(options))
44
44
  end
45
45
 
46
46
  def touch(key, ttl)
@@ -15,11 +15,17 @@ module Dalli
15
15
  compress: true,
16
16
  compressor: ::Dalli::Compressor,
17
17
  # min byte size to attempt compression
18
- compression_min_size: 4 * 1024 # 4K
18
+ compression_min_size: 4 * 1024, # 4K
19
+ # max size a stored value may decompress to (nil: no limit). Guards
20
+ # against a small compressed value expanding into gigabytes on read.
21
+ decompressed_max_bytes: 128 * 1024 * 1024 # 128 MiB
19
22
  }.freeze
20
23
 
21
24
  OPTIONS = DEFAULTS.keys.freeze
22
25
 
26
+ KEYWORD_PARAMETER_TYPES = %i[key keyreq].freeze
27
+ private_constant :KEYWORD_PARAMETER_TYPES
28
+
23
29
  # https://www.hjp.at/zettel/m/memcached_flags.rxml
24
30
  # Looks like most clients use bit 1 to indicate gzip compression.
25
31
  FLAG_COMPRESSED = 0x2
@@ -47,8 +53,16 @@ module Dalli
47
53
  end
48
54
 
49
55
  def retrieve(value, bitflags)
50
- compressed = (bitflags & FLAG_COMPRESSED) != 0
51
- compressed ? compressor.decompress(value) : value
56
+ return value unless bitflags.anybits?(FLAG_COMPRESSED)
57
+
58
+ max_bytes = @compression_options[:decompressed_max_bytes]
59
+ # Custom compressors that only define decompress(data) keep working,
60
+ # without the limit
61
+ if max_bytes && compressor_accepts_max_bytes?
62
+ compressor.decompress(value, max_bytes: max_bytes)
63
+ else
64
+ compressor.decompress(value)
65
+ end
52
66
 
53
67
  # TODO: We likely want to move this rescue into the Dalli::Compressor / Dalli::GzipCompressor
54
68
  # itself, since not all compressors necessarily use Zlib. For now keep it here, so the behavior
@@ -57,6 +71,19 @@ module Dalli
57
71
  raise UnmarshalError, "Unable to uncompress value: #{$ERROR_INFO.message}"
58
72
  end
59
73
 
74
+ def compressor_accepts_max_bytes?
75
+ return @compressor_accepts_max_bytes if defined?(@compressor_accepts_max_bytes)
76
+
77
+ @compressor_accepts_max_bytes =
78
+ begin
79
+ compressor.method(:decompress).parameters.any? do |type, name|
80
+ name == :max_bytes && KEYWORD_PARAMETER_TYPES.include?(type)
81
+ end
82
+ rescue NameError # an object without a reflectable decompress method
83
+ false
84
+ end
85
+ end
86
+
60
87
  def compress_by_default?
61
88
  @compression_options[:compress]
62
89
  end
data/lib/dalli/version.rb CHANGED
@@ -1,7 +1,7 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module Dalli
4
- VERSION = '3.2.11'
4
+ VERSION = '3.2.12'
5
5
 
6
6
  MIN_SUPPORTED_MEMCACHED_VERSION = '1.4'
7
7
  end
data/lib/dalli.rb CHANGED
@@ -57,6 +57,7 @@ end
57
57
  require_relative 'dalli/version'
58
58
 
59
59
  require_relative 'dalli/compressor'
60
+ require_relative 'dalli/json_serializer'
60
61
  require_relative 'dalli/client'
61
62
  require_relative 'dalli/key_manager'
62
63
  require_relative 'dalli/pipelined_getter'
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: dalli
3
3
  version: !ruby/object:Gem::Version
4
- version: 3.2.11
4
+ version: 3.2.12
5
5
  platform: ruby
6
6
  authors:
7
7
  - Peter M. Goldstein
@@ -40,6 +40,7 @@ files:
40
40
  - lib/dalli/cas/client.rb
41
41
  - lib/dalli/client.rb
42
42
  - lib/dalli/compressor.rb
43
+ - lib/dalli/json_serializer.rb
43
44
  - lib/dalli/key_manager.rb
44
45
  - lib/dalli/options.rb
45
46
  - lib/dalli/pid_cache.rb