dalli 5.2.1 → 5.2.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +29 -0
- data/README.md +13 -0
- data/lib/dalli/client.rb +30 -20
- data/lib/dalli/compressor.rb +5 -0
- data/lib/dalli/key_manager.rb +6 -3
- data/lib/dalli/protocol/base.rb +9 -4
- data/lib/dalli/protocol/connection_manager.rb +20 -1
- data/lib/dalli/protocol/meta.rb +10 -6
- data/lib/dalli/protocol/response_buffer.rb +4 -1
- data/lib/dalli/protocol/response_processor.rb +45 -18
- data/lib/dalli/version.rb +1 -1
- metadata +1 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: cd910ad2e0c6e5aa47b8591e9ba8c5806b6208fbab0ae0852637aecc48939051
|
|
4
|
+
data.tar.gz: bd0db96211060011acb2135feb337d8acc88638b08c57b3fa875445695a09b59
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 82da7aebd50661e1ab9af25dab7d66e5f70cb15260ff9ca4b3476e63eeb60cbb2ad1299718e56312f9a6a61d7a2b14d394fcc470325329f1152db42ac921ee6d
|
|
7
|
+
data.tar.gz: a036421fac121aa342adc3beced711ce64a1bd241673a9314f1f1454a951515890732c4c9651e52baf65c94cd01c4b5cbb655c8dcca4fa4db03d656c3cfc4abe
|
data/CHANGELOG.md
CHANGED
|
@@ -4,6 +4,35 @@ Dalli Changelog
|
|
|
4
4
|
Unreleased
|
|
5
5
|
==========
|
|
6
6
|
|
|
7
|
+
5.2.2
|
|
8
|
+
==========
|
|
9
|
+
|
|
10
|
+
Security:
|
|
11
|
+
|
|
12
|
+
- Keep the caller's key when retrying a request (GHSA-m252-9cgf-vx2w)
|
|
13
|
+
- With a `namespace`, a request retried after a transient network error (a timeout, or a connection closed by memcached or a proxy) applied the namespace a second time, so a retried read could return a different key's value and a retried write could overwrite a different key
|
|
14
|
+
- Affects single-key operations since 5.0.3, single-server `get_multi` since 5.1.0, and `get_with_metadata` and `fetch_with_lock` since 4.1.0; fixed in 5.2.2, 5.1.4, 5.0.10 and 4.3.7. Clients without a namespace aren't affected
|
|
15
|
+
- Complete the fix for per-request `raw: true` on reads (GHSA-wr87-m4jw-29x5)
|
|
16
|
+
- A read made with `raw: true` doesn't ask for flags, but a reply carrying them anyway (from a proxy or a hostile server) still had its value deserialized. Raw reads now ignore flags in the reply
|
|
17
|
+
- Affects 5.2.1 and earlier; fixed in 5.2.2, 5.1.4, 5.0.10, 4.3.7 and 3.2.13
|
|
18
|
+
|
|
19
|
+
Bug fixes:
|
|
20
|
+
|
|
21
|
+
- `get_with_metadata` and `fetch_with_lock` retried the final error raised when a server is marked down, so with `down_retry_delay: 0` they retried an unresponsive server forever. They now retry only retryable errors, like other operations
|
|
22
|
+
- Fix a regression in 5.2.1: a `get_multi` that didn't finish within `socket_timeout` counted toward `socket_max_failures`, so two slow `get_multi` calls in a row marked a healthy server down. It now just closes the connection
|
|
23
|
+
- Fix a regression in 5.2.1: over TLS, every request buffered in a `quiet` block was sent as its own TLS record and system call (2.5 times slower for a block of 2000 deletes). Each flush is one write again
|
|
24
|
+
- Handle malformed replies from a broken or hostile server or proxy: a negative value size, a hit with no key, or a hit with no `s` flag could leave the connection out of step or raise a non-Dalli error
|
|
25
|
+
- With `decompressed_max_bytes` in effect, data after the end of a compressed value's stream was returned as part of the value. It's now ignored, as without the limit
|
|
26
|
+
- With a `digest_class` whose digests aren't short hex strings, shortening a long key could loop forever. It now raises `ArgumentError`
|
|
27
|
+
|
|
28
|
+
Notes:
|
|
29
|
+
|
|
30
|
+
- Document how long keys are shortened, and that a short key written in the same form names the same item, in the README's security note (#1200)
|
|
31
|
+
|
|
32
|
+
Development:
|
|
33
|
+
|
|
34
|
+
- Run the Tests, RuboCop and Profiles workflows on pushes to `main` and the `*-stable` branches only, so a pull request's branch isn't tested twice (once for the push and once for the pull request) (#1198)
|
|
35
|
+
|
|
7
36
|
5.2.1
|
|
8
37
|
==========
|
|
9
38
|
|
data/README.md
CHANGED
|
@@ -97,6 +97,19 @@ Dalli::Client.new('localhost:11211', serializer: Dalli::JSONSerializer)
|
|
|
97
97
|
|
|
98
98
|
Per-request `raw: true` returns the stored bytes without deserializing or decompressing them, on every read method (`get`, `get_multi`, `get_cas`, `get_with_metadata` and `fetch`).
|
|
99
99
|
|
|
100
|
+
### Long keys
|
|
101
|
+
|
|
102
|
+
memcached limits keys to 250 bytes. Dalli shortens a longer key (counting its namespace) to the start of the key, followed by `:md5:` and a hex digest of the whole key:
|
|
103
|
+
|
|
104
|
+
```ruby
|
|
105
|
+
dc.set("report:#{'x' * 300}", 'data')
|
|
106
|
+
# stored under "report:xxx...xxx:md5:<32 hex characters>"
|
|
107
|
+
```
|
|
108
|
+
|
|
109
|
+
The shortened key is an ordinary memcached key, so a short key written in exactly that form names the same item, and reading or writing it reads or overwrites the long key's value. This only matters if untrusted input can choose an entire cache key. If it can, hash the untrusted part yourself so the key never needs shortening, or reject keys that contain `:md5:`.
|
|
110
|
+
|
|
111
|
+
The digest is MD5 by default. `digest_class:` takes any object that responds to `hexdigest`, such as `Digest::SHA256`. Changing it renames every shortened key, so existing entries for long keys are missed once.
|
|
112
|
+
|
|
100
113
|
See the [5.0-Upgrade.md](5.0-Upgrade.md) guide for upgrade information.
|
|
101
114
|
|
|
102
115
|
## OpenTelemetry Tracing
|
data/lib/dalli/client.rb
CHANGED
|
@@ -144,14 +144,18 @@ module Dalli
|
|
|
144
144
|
#
|
|
145
145
|
def get_with_metadata(key, options = {})
|
|
146
146
|
validate_routing_tokens!(options)
|
|
147
|
-
|
|
148
|
-
|
|
147
|
+
# A retry re-runs this method with its original arguments, so `key` must
|
|
148
|
+
# stay as the caller gave it; reassigning it would add the namespace again.
|
|
149
|
+
validated_key = @key_manager.validate_key(key.to_s)
|
|
149
150
|
|
|
150
|
-
server = ring.server_for_key(
|
|
151
|
-
Instrumentation.trace('get_with_metadata', trace_attrs('get_with_metadata',
|
|
152
|
-
server.request(:meta_get,
|
|
151
|
+
server = ring.server_for_key(validated_key)
|
|
152
|
+
Instrumentation.trace('get_with_metadata', trace_attrs('get_with_metadata', validated_key, server)) do
|
|
153
|
+
server.request(:meta_get, validated_key, options)
|
|
153
154
|
end
|
|
154
|
-
|
|
155
|
+
# Only a retryable error is retried. The NetworkError raised when a server
|
|
156
|
+
# is marked down is final, as in #perform; retrying it looped forever once
|
|
157
|
+
# down_retry_delay let the same server be tried again.
|
|
158
|
+
rescue RetryableNetworkError => e
|
|
155
159
|
Dalli.logger.debug { e.inspect }
|
|
156
160
|
Dalli.logger.debug { 'retrying get_with_metadata with new server' }
|
|
157
161
|
retry
|
|
@@ -347,14 +351,17 @@ module Dalli
|
|
|
347
351
|
validate_integer!(:lock_ttl, lock_ttl)
|
|
348
352
|
validate_integer!(:recache_threshold, recache_threshold)
|
|
349
353
|
validate_routing_tokens!(req_options)
|
|
350
|
-
|
|
351
|
-
|
|
354
|
+
# Left unchanged for the retry below; see get_with_metadata
|
|
355
|
+
validated_key = @key_manager.validate_key(key.to_s)
|
|
352
356
|
|
|
353
|
-
server = ring.server_for_key(
|
|
354
|
-
Instrumentation.trace('fetch_with_lock', trace_attrs('fetch_with_lock',
|
|
355
|
-
fetch_with_lock_request(
|
|
357
|
+
server = ring.server_for_key(validated_key)
|
|
358
|
+
Instrumentation.trace('fetch_with_lock', trace_attrs('fetch_with_lock', validated_key, server)) do
|
|
359
|
+
fetch_with_lock_request(validated_key, ttl, lock_ttl, recache_threshold, req_options, &block)
|
|
356
360
|
end
|
|
357
|
-
|
|
361
|
+
# Only a retryable error is retried. The NetworkError raised when a server
|
|
362
|
+
# is marked down is final, as in #perform; retrying it looped forever once
|
|
363
|
+
# down_retry_delay let the same server be tried again.
|
|
364
|
+
rescue RetryableNetworkError => e
|
|
358
365
|
Dalli.logger.debug { e.inspect }
|
|
359
366
|
Dalli.logger.debug { 'retrying fetch_with_lock with new server' }
|
|
360
367
|
retry
|
|
@@ -787,10 +794,11 @@ module Dalli
|
|
|
787
794
|
# to route to at all is still silent, matching Ring#keys_grouped_by_server
|
|
788
795
|
# dropping a key it can't route on both the single- and multi-server paths.
|
|
789
796
|
def single_server_get_multi(keys, req_options = nil)
|
|
790
|
-
|
|
797
|
+
# A new array, so the retry below starts again from the caller's keys
|
|
798
|
+
validated_keys = keys.map { |k| @key_manager.validate_key(k.to_s) }
|
|
791
799
|
return {} unless (server = single_server)
|
|
792
800
|
|
|
793
|
-
result = server.request(:read_multi_req,
|
|
801
|
+
result = server.request(:read_multi_req, validated_keys, req_options)
|
|
794
802
|
result.transform_keys! { |k| @key_manager.key_without_namespace(k) }
|
|
795
803
|
result
|
|
796
804
|
rescue Dalli::RetryableNetworkError => e
|
|
@@ -957,18 +965,20 @@ module Dalli
|
|
|
957
965
|
# rubocop:enable Naming/MethodParameterName
|
|
958
966
|
return yield if block_given?
|
|
959
967
|
|
|
960
|
-
|
|
961
|
-
|
|
968
|
+
# `retry` below re-runs this method with its original arguments, so `key`
|
|
969
|
+
# must stay as the caller gave it: reassigning it would apply the
|
|
970
|
+
# namespace a second time and send the retry to a different key.
|
|
971
|
+
validated_key = @key_manager.validate_key(key.to_s)
|
|
962
972
|
|
|
963
|
-
server = ring.server_for_key(
|
|
973
|
+
server = ring.server_for_key(validated_key)
|
|
964
974
|
|
|
965
975
|
if Instrumentation.enabled?
|
|
966
976
|
op_name = op.name
|
|
967
|
-
Instrumentation.trace(op_name, trace_attrs(op_name,
|
|
968
|
-
server.request(op,
|
|
977
|
+
Instrumentation.trace(op_name, trace_attrs(op_name, validated_key, server)) do
|
|
978
|
+
server.request(op, validated_key, ...)
|
|
969
979
|
end
|
|
970
980
|
else
|
|
971
|
-
server.request(op,
|
|
981
|
+
server.request(op, validated_key, ...)
|
|
972
982
|
end
|
|
973
983
|
rescue RetryableNetworkError => e
|
|
974
984
|
Dalli.logger.debug { e.inspect }
|
data/lib/dalli/compressor.rb
CHANGED
|
@@ -39,7 +39,12 @@ module Dalli
|
|
|
39
39
|
out = String.new(encoding: Encoding::BINARY)
|
|
40
40
|
0.step(data.bytesize - 1, INFLATE_SLICE_BYTES) do |pos|
|
|
41
41
|
inflate_slice(inflater, data.byteslice(pos, INFLATE_SLICE_BYTES), out, max_bytes)
|
|
42
|
+
# Anything after the end of the stream isn't part of the value, as
|
|
43
|
+
# with Zlib::Inflate.inflate; finish would return it as output
|
|
44
|
+
break if inflater.finished?
|
|
42
45
|
end
|
|
46
|
+
return out if inflater.finished?
|
|
47
|
+
|
|
43
48
|
append_within_limit(out, inflater.finish, max_bytes)
|
|
44
49
|
ensure
|
|
45
50
|
inflater.close
|
data/lib/dalli/key_manager.rb
CHANGED
|
@@ -122,15 +122,18 @@ module Dalli
|
|
|
122
122
|
##
|
|
123
123
|
def truncated_key(key)
|
|
124
124
|
digest = digest_class.hexdigest(key)
|
|
125
|
-
prefix = key[0, prefix_length(digest)]
|
|
125
|
+
prefix = key[0, [prefix_length(digest), 0].max]
|
|
126
126
|
truncated = "#{prefix}#{TRUNCATED_KEY_SEPARATOR}#{digest}"
|
|
127
127
|
# A prefix with whitespace or non-ASCII characters sends the whole key
|
|
128
128
|
# base64-encoded, which can still be too long on the wire; shorten it.
|
|
129
|
-
while wire_length(truncated) > MAX_KEY_LENGTH
|
|
129
|
+
while wire_length(truncated) > MAX_KEY_LENGTH && !prefix.empty?
|
|
130
130
|
prefix = prefix[0...-1]
|
|
131
131
|
truncated = "#{prefix}#{TRUNCATED_KEY_SEPARATOR}#{digest}"
|
|
132
132
|
end
|
|
133
|
-
truncated
|
|
133
|
+
return truncated if wire_length(truncated) <= MAX_KEY_LENGTH
|
|
134
|
+
|
|
135
|
+
# Only possible with a digest_class whose digests aren't short hex strings
|
|
136
|
+
raise ArgumentError, "#{digest_class} digests are too long to fit in a truncated key"
|
|
134
137
|
end
|
|
135
138
|
|
|
136
139
|
# memcached's key limit applies to the key as sent: in bytes, and after
|
data/lib/dalli/protocol/base.rb
CHANGED
|
@@ -203,10 +203,13 @@ module Dalli
|
|
|
203
203
|
@connection_manager.abort_request!
|
|
204
204
|
return true unless connected?
|
|
205
205
|
|
|
206
|
-
# Closes the connection, which ensures that our connection
|
|
207
|
-
#
|
|
208
|
-
|
|
209
|
-
|
|
206
|
+
# Closes the connection, which ensures that our connection is in a
|
|
207
|
+
# clean state for future requests. The get_multi ran out of time
|
|
208
|
+
# overall rather than hitting a socket error, so it doesn't count
|
|
209
|
+
# toward socket_max_failures: two slow get_multis in a row would
|
|
210
|
+
# otherwise mark a healthy server down.
|
|
211
|
+
Dalli.logger.warn { "#{name} didn't finish a get_multi within socket_timeout; closing the connection" }
|
|
212
|
+
@connection_manager.close
|
|
210
213
|
true
|
|
211
214
|
end
|
|
212
215
|
|
|
@@ -342,6 +345,7 @@ module Dalli
|
|
|
342
345
|
# Use clear (not reset) to keep pipeline_complete? = true, which is
|
|
343
346
|
# the expected state before pipeline_response_setup is called.
|
|
344
347
|
response_buffer.clear
|
|
348
|
+
response_buffer.raw = raw_request?(options)
|
|
345
349
|
|
|
346
350
|
# The terminating noop is sent by pipeline_response_setup
|
|
347
351
|
write(quiet_get_requests(keys, options, return_cas: return_cas))
|
|
@@ -353,6 +357,7 @@ module Dalli
|
|
|
353
357
|
def pipelined_get_interleaved(keys, chunk_size, results, options = nil)
|
|
354
358
|
# Initialize the response buffer for draining during send phase
|
|
355
359
|
response_buffer.ensure_ready
|
|
360
|
+
response_buffer.raw = raw_request?(options)
|
|
356
361
|
|
|
357
362
|
keys.each_slice(chunk_size) do |chunk|
|
|
358
363
|
# Build and write this chunk of requests
|
|
@@ -352,13 +352,32 @@ module Dalli
|
|
|
352
352
|
|
|
353
353
|
private
|
|
354
354
|
|
|
355
|
+
# One write per flush. A TLS socket sends each argument of
|
|
356
|
+
# write(*parts) as its own record and system call, so the parts are
|
|
357
|
+
# joined first.
|
|
355
358
|
def flush_write_buffer
|
|
356
359
|
return if @write_buffer.empty?
|
|
357
360
|
|
|
358
|
-
@sock.write(
|
|
361
|
+
@sock.write(@write_buffer.size == 1 ? @write_buffer.first : joined_write_buffer)
|
|
359
362
|
discard_write_buffer
|
|
360
363
|
end
|
|
361
364
|
|
|
365
|
+
# Joined as bytes: requests can mix UTF-8 and binary strings, which
|
|
366
|
+
# String#<< refuses to combine when both hold non-ASCII bytes
|
|
367
|
+
if String.method_defined?(:append_as_bytes)
|
|
368
|
+
def joined_write_buffer
|
|
369
|
+
out = String.new(capacity: @write_buffer_bytes, encoding: Encoding::BINARY)
|
|
370
|
+
@write_buffer.each { |part| out.append_as_bytes(part) }
|
|
371
|
+
out
|
|
372
|
+
end
|
|
373
|
+
else
|
|
374
|
+
def joined_write_buffer
|
|
375
|
+
out = String.new(capacity: @write_buffer_bytes, encoding: Encoding::BINARY)
|
|
376
|
+
@write_buffer.each { |part| out << part.b }
|
|
377
|
+
out
|
|
378
|
+
end
|
|
379
|
+
end
|
|
380
|
+
|
|
362
381
|
def discard_write_buffer
|
|
363
382
|
@write_buffer.clear
|
|
364
383
|
@write_buffer_bytes = 0
|
data/lib/dalli/protocol/meta.rb
CHANGED
|
@@ -38,7 +38,7 @@ module Dalli
|
|
|
38
38
|
RequestFormatter.plain_meta_get(key, skip_flags)
|
|
39
39
|
end
|
|
40
40
|
flushed_write(req)
|
|
41
|
-
response_processor.meta_get_with_value(cache_nils: cache_nils?(options))
|
|
41
|
+
response_processor.meta_get_with_value(cache_nils: cache_nils?(options), raw: raw_request?(options))
|
|
42
42
|
end
|
|
43
43
|
|
|
44
44
|
def quiet_get_request(key, options = nil)
|
|
@@ -59,7 +59,7 @@ module Dalli
|
|
|
59
59
|
skip_flags = raw_mode? || (options && options[:raw])
|
|
60
60
|
req = RequestFormatter.meta_get(key: key, ttl: ttl, skip_flags: skip_flags, **routing_token_kwargs(options))
|
|
61
61
|
flushed_write(req)
|
|
62
|
-
response_processor.meta_get_with_value(cache_nils: cache_nils?(options))
|
|
62
|
+
response_processor.meta_get_with_value(cache_nils: cache_nils?(options), raw: raw_request?(options))
|
|
63
63
|
end
|
|
64
64
|
|
|
65
65
|
def touch(key, ttl)
|
|
@@ -75,7 +75,7 @@ module Dalli
|
|
|
75
75
|
req = RequestFormatter.meta_get(key: key, value: true, return_cas: true, skip_flags: raw_request?(options),
|
|
76
76
|
**routing_token_kwargs(options))
|
|
77
77
|
flushed_write(req)
|
|
78
|
-
response_processor.meta_get_with_value_and_cas
|
|
78
|
+
response_processor.meta_get_with_value_and_cas(raw: raw_request?(options))
|
|
79
79
|
end
|
|
80
80
|
|
|
81
81
|
# Comprehensive meta get with support for all metadata flags.
|
|
@@ -116,7 +116,7 @@ module Dalli
|
|
|
116
116
|
response_processor.meta_get_with_metadata(
|
|
117
117
|
cache_nils: cache_nils?(options), return_hit_status: options[:return_hit_status],
|
|
118
118
|
return_last_access: options[:return_last_access],
|
|
119
|
-
return_ttl_remaining: options[:return_ttl_remaining]
|
|
119
|
+
return_ttl_remaining: options[:return_ttl_remaining], raw: raw_request?(options)
|
|
120
120
|
)
|
|
121
121
|
end
|
|
122
122
|
|
|
@@ -322,7 +322,9 @@ module Dalli
|
|
|
322
322
|
next unless line.start_with?('VA ')
|
|
323
323
|
|
|
324
324
|
tokens = line.chomp!(TERMINATOR).split
|
|
325
|
-
|
|
325
|
+
size = tokens[1].to_i
|
|
326
|
+
response_processor.check_value_size!(size)
|
|
327
|
+
value = @connection_manager.read(size + TERMINATOR.bytesize)&.chomp!(TERMINATOR)
|
|
326
328
|
stale = response_processor.stale_from_tokens(tokens)
|
|
327
329
|
cas = response_processor.cas_from_tokens(tokens)
|
|
328
330
|
bitflags = is_raw ? 0 : response_processor.bitflags_from_tokens(tokens)
|
|
@@ -350,7 +352,9 @@ module Dalli
|
|
|
350
352
|
# splitting it into tokens
|
|
351
353
|
def parse_multi_get_value(line, is_raw)
|
|
352
354
|
processor = response_processor
|
|
353
|
-
|
|
355
|
+
size = processor.size_from_va_line(line)
|
|
356
|
+
processor.check_value_size!(size)
|
|
357
|
+
value = @connection_manager.read(size + TERMINATOR.bytesize)&.chomp!(TERMINATOR)
|
|
354
358
|
key = processor.key_from_va_line(line)
|
|
355
359
|
return unless key
|
|
356
360
|
|
|
@@ -15,6 +15,9 @@ module Dalli
|
|
|
15
15
|
# when advancing through parsed responses.
|
|
16
16
|
##
|
|
17
17
|
class ResponseBuffer
|
|
18
|
+
# Whether the current pipeline was requested with raw: true
|
|
19
|
+
attr_writer :raw
|
|
20
|
+
|
|
18
21
|
def initialize(io_source, response_processor)
|
|
19
22
|
@io_source = io_source
|
|
20
23
|
@response_processor = response_processor
|
|
@@ -39,7 +42,7 @@ module Dalli
|
|
|
39
42
|
# Attempts to process a single response from the buffer,
|
|
40
43
|
# advancing the offset past the consumed bytes.
|
|
41
44
|
def process_single_getk_response
|
|
42
|
-
response = @response_processor.getk_response_from_buffer(@buffer, @offset)
|
|
45
|
+
response = @response_processor.getk_response_from_buffer(@buffer, @offset, raw: @raw || false)
|
|
43
46
|
@offset += response.pop
|
|
44
47
|
response
|
|
45
48
|
end
|
|
@@ -51,12 +51,15 @@ module Dalli
|
|
|
51
51
|
@value_marshaller = value_marshaller
|
|
52
52
|
end
|
|
53
53
|
|
|
54
|
-
|
|
54
|
+
# raw: true (a request made with raw: true) returns the value as stored,
|
|
55
|
+
# ignoring any flags in the reply. The request didn't ask for flags, so
|
|
56
|
+
# a reply carrying them anyway must not get the value deserialized.
|
|
57
|
+
def meta_get_with_value(cache_nils: false, raw: false)
|
|
55
58
|
line = read_line
|
|
56
59
|
# A hit ("VA <size> f<flags>") is parsed in place rather than split
|
|
57
60
|
# into tokens, which saves several allocations on the hottest path.
|
|
58
61
|
if line&.start_with?(VA_PREFIX)
|
|
59
|
-
return
|
|
62
|
+
return retrieve(read_data(size_from_va_line(line)), raw ? 0 : bitflags_from_va_line(line), raw)
|
|
60
63
|
end
|
|
61
64
|
|
|
62
65
|
tokens = line&.split || []
|
|
@@ -64,7 +67,7 @@ module Dalli
|
|
|
64
67
|
when EN
|
|
65
68
|
cache_nils ? ::Dalli::NOT_FOUND : nil
|
|
66
69
|
when VA # only reached for an unusually formatted hit line
|
|
67
|
-
|
|
70
|
+
retrieve(read_data(tokens[1].to_i), raw ? 0 : bitflags_from_tokens(tokens), raw)
|
|
68
71
|
when HD
|
|
69
72
|
true
|
|
70
73
|
else
|
|
@@ -72,14 +75,14 @@ module Dalli
|
|
|
72
75
|
end
|
|
73
76
|
end
|
|
74
77
|
|
|
75
|
-
def meta_get_with_value_and_cas
|
|
78
|
+
def meta_get_with_value_and_cas(raw: false)
|
|
76
79
|
tokens = error_on_unexpected!(T_VA_EN_HD)
|
|
77
80
|
return [nil, 0] if tokens.first == EN
|
|
78
81
|
|
|
79
82
|
cas = cas_from_tokens(tokens)
|
|
80
83
|
return [nil, cas] unless tokens.first == VA
|
|
81
84
|
|
|
82
|
-
[
|
|
85
|
+
[retrieve(read_data(tokens[1].to_i), raw ? 0 : bitflags_from_tokens(tokens), raw), cas]
|
|
83
86
|
end
|
|
84
87
|
|
|
85
88
|
def meta_get_without_value
|
|
@@ -99,13 +102,13 @@ module Dalli
|
|
|
99
102
|
# Used by meta_get for comprehensive metadata retrieval.
|
|
100
103
|
# Supports thundering herd protection (N/R flags) and metadata flags (h/l/u).
|
|
101
104
|
def meta_get_with_metadata(cache_nils: false, return_hit_status: false, return_last_access: false,
|
|
102
|
-
return_ttl_remaining: false)
|
|
105
|
+
return_ttl_remaining: false, raw: false)
|
|
103
106
|
tokens = error_on_unexpected!(T_VA_EN_HD)
|
|
104
107
|
result = build_metadata_result(tokens)
|
|
105
108
|
result[:hit_before] = hit_status_from_tokens(tokens) if return_hit_status
|
|
106
109
|
result[:last_access] = last_access_from_tokens(tokens) if return_last_access
|
|
107
110
|
result[:ttl_remaining] = ttl_remaining_from_tokens(tokens) if return_ttl_remaining
|
|
108
|
-
result[:value] = parse_value_from_tokens(tokens, cache_nils)
|
|
111
|
+
result[:value] = parse_value_from_tokens(tokens, cache_nils, raw)
|
|
109
112
|
result
|
|
110
113
|
end
|
|
111
114
|
|
|
@@ -122,11 +125,11 @@ module Dalli
|
|
|
122
125
|
}
|
|
123
126
|
end
|
|
124
127
|
|
|
125
|
-
def parse_value_from_tokens(tokens, cache_nils)
|
|
128
|
+
def parse_value_from_tokens(tokens, cache_nils, raw = false) # rubocop:disable Style/OptionalBooleanParameter
|
|
126
129
|
return cache_nils ? ::Dalli::NOT_FOUND : nil if tokens.first == EN
|
|
127
130
|
return unless tokens.first == VA
|
|
128
131
|
|
|
129
|
-
|
|
132
|
+
retrieve(read_data(tokens[1].to_i), raw ? 0 : bitflags_from_tokens(tokens), raw)
|
|
130
133
|
end
|
|
131
134
|
|
|
132
135
|
def meta_set_with_cas
|
|
@@ -229,13 +232,15 @@ module Dalli
|
|
|
229
232
|
# The remaining three values in the array are the ResponseHeader,
|
|
230
233
|
# key, and value.
|
|
231
234
|
##
|
|
232
|
-
|
|
235
|
+
# raw: true returns values as stored, ignoring flags in the replies;
|
|
236
|
+
# see meta_get_with_value.
|
|
237
|
+
def getk_response_from_buffer(buf, offset = 0, raw: false)
|
|
233
238
|
# Find the header terminator starting from offset
|
|
234
239
|
term_idx = buf.byteindex(TERMINATOR, offset)
|
|
235
240
|
return [0] unless term_idx
|
|
236
241
|
|
|
237
242
|
if buf.byteslice(offset, VA_PREFIX.bytesize) == VA_PREFIX
|
|
238
|
-
response = va_response_from_buffer(buf, offset, term_idx)
|
|
243
|
+
response = va_response_from_buffer(buf, offset, term_idx, raw)
|
|
239
244
|
return response if response
|
|
240
245
|
end
|
|
241
246
|
|
|
@@ -256,7 +261,9 @@ module Dalli
|
|
|
256
261
|
when BYTE_B then base64 ||= token == 'b'
|
|
257
262
|
end
|
|
258
263
|
end
|
|
259
|
-
|
|
264
|
+
# memcached also gives the size positionally ("VA <size> ..."); use it
|
|
265
|
+
# when there's no s flag rather than reading the value as replies
|
|
266
|
+
body_len = size ? flag_int(size, 's') : positional_size(tokens)
|
|
260
267
|
|
|
261
268
|
# A complete response with no body. Only the terminating noop's MN
|
|
262
269
|
# ends the pipeline. Any other bodyless reply (CLIENT_ERROR,
|
|
@@ -276,8 +283,12 @@ module Dalli
|
|
|
276
283
|
# The full response is in our buffer, so parse it and return
|
|
277
284
|
# the values
|
|
278
285
|
body = buf.byteslice(offset + header_len, body_len)
|
|
279
|
-
value =
|
|
280
|
-
|
|
286
|
+
value = retrieve(body, raw ? 0 : flag_int(bitflags, 'f'), raw)
|
|
287
|
+
# A hit with no key can't be matched to a request: skip it (false
|
|
288
|
+
# status) rather than yield it, or take it for the end of the pipeline
|
|
289
|
+
return [false, resp_size] unless key
|
|
290
|
+
|
|
291
|
+
key = key.delete_prefix!('k')
|
|
281
292
|
key = KeyRegularizer.decode(key) if base64
|
|
282
293
|
[tokens.first == VA, flag_int(cas, 'c'), key, value, resp_size]
|
|
283
294
|
end
|
|
@@ -286,7 +297,7 @@ module Dalli
|
|
|
286
297
|
# without allocating the header, its tokens or the flag strings.
|
|
287
298
|
# Gives the same result as the token path. Returns nil to fall back to
|
|
288
299
|
# it when the header has no s flag or a zero size.
|
|
289
|
-
def va_response_from_buffer(buf, offset, term_idx)
|
|
300
|
+
def va_response_from_buffer(buf, offset, term_idx, raw = false) # rubocop:disable Style/OptionalBooleanParameter
|
|
290
301
|
size = bitflags = cas = key = nil
|
|
291
302
|
base64 = false
|
|
292
303
|
pos = buf.byteindex(' ', offset + VA_PREFIX.bytesize)
|
|
@@ -309,9 +320,17 @@ module Dalli
|
|
|
309
320
|
resp_size = header_len + size + TERMINATOR.length
|
|
310
321
|
return [0] unless buf.bytesize >= offset + resp_size
|
|
311
322
|
|
|
312
|
-
value =
|
|
313
|
-
|
|
314
|
-
[
|
|
323
|
+
value = retrieve(buf.byteslice(offset + header_len, size), raw ? 0 : bitflags || 0, raw)
|
|
324
|
+
# Skipped like the token path does; see getk_response_from_buffer
|
|
325
|
+
return [false, resp_size] unless key
|
|
326
|
+
|
|
327
|
+
key = KeyRegularizer.decode(key) if base64
|
|
328
|
+
[true, cas || 0, key, value, resp_size]
|
|
329
|
+
end
|
|
330
|
+
|
|
331
|
+
# Values from a raw request are returned as stored
|
|
332
|
+
def retrieve(value, bitflags, raw)
|
|
333
|
+
raw ? value : @value_marshaller.retrieve(value, bitflags)
|
|
315
334
|
end
|
|
316
335
|
|
|
317
336
|
# Integer value of a flag's token in buf, after its one-byte prefix,
|
|
@@ -431,6 +450,11 @@ module Dalli
|
|
|
431
450
|
|
|
432
451
|
# A pipelined reply claiming an impossible size would otherwise have the
|
|
433
452
|
# buffer wait for (and accumulate) that many bytes
|
|
453
|
+
# The size given right after "VA", or 0 when it isn't a number
|
|
454
|
+
def positional_size(tokens)
|
|
455
|
+
tokens.first == VA && tokens[1]&.match?(/\A\d+\z/) ? tokens[1].to_i : 0
|
|
456
|
+
end
|
|
457
|
+
|
|
434
458
|
def check_value_size!(size)
|
|
435
459
|
return if size.between?(0, MAX_VALUE_BYTES)
|
|
436
460
|
|
|
@@ -447,6 +471,9 @@ module Dalli
|
|
|
447
471
|
end
|
|
448
472
|
|
|
449
473
|
def read_data(data_size)
|
|
474
|
+
# Checked before the terminator is added: -1 or -2 would otherwise
|
|
475
|
+
# read 1 or 0 bytes and leave the connection out of step
|
|
476
|
+
check_value_size!(data_size)
|
|
450
477
|
@io_source.read(data_size + TERMINATOR.bytesize)&.chomp!(TERMINATOR)
|
|
451
478
|
end
|
|
452
479
|
end
|
data/lib/dalli/version.rb
CHANGED