dalli 5.2.2 → 5.2.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +52 -0
- data/README.md +40 -0
- data/lib/dalli/client.rb +74 -16
- data/lib/dalli/pipelined_getter.rb +5 -2
- data/lib/dalli/pipelined_setter.rb +9 -0
- data/lib/dalli/protocol/base.rb +65 -2
- data/lib/dalli/protocol/connection_manager.rb +15 -3
- data/lib/dalli/protocol/meta.rb +65 -9
- data/lib/dalli/protocol/server_config_parser.rb +10 -5
- data/lib/dalli/protocol/string_marshaller.rb +3 -5
- data/lib/dalli/protocol/value_compressor.rb +6 -4
- data/lib/dalli/protocol/value_marshaller.rb +21 -5
- data/lib/dalli/protocol.rb +3 -3
- data/lib/dalli/socket.rb +40 -3
- data/lib/dalli/version.rb +1 -1
- data/lib/dalli.rb +4 -0
- data/lib/rack/session/dalli.rb +37 -10
- metadata +2 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 1a31a4159ee5cd637dca6676f343595f653798864bef28b109fbc329debe2aff
|
|
4
|
+
data.tar.gz: efba0a1a4bb20eff2fd4f6b26c15359351484401502a798fa398b5bf1301d046
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 38f6d3ba18790b94875f18bf6239a520f57144ba763826063471bea2047aa6def2cc528d37de35f29037a0c33ca9bb8b46d7ed0ddc5dccebf8914888c71dafd7
|
|
7
|
+
data.tar.gz: 9f091240e20313b7cbb9250e900cbf3aa02f9aedbffceabeedc7355bc780cd3895e231c77bb091ab97680dbdac85e093635e3cfbca904fd7f095512181e4a59d
|
data/CHANGELOG.md
CHANGED
|
@@ -4,6 +4,58 @@ Dalli Changelog
|
|
|
4
4
|
Unreleased
|
|
5
5
|
==========
|
|
6
6
|
|
|
7
|
+
5.2.3
|
|
8
|
+
==========
|
|
9
|
+
|
|
10
|
+
Security:
|
|
11
|
+
|
|
12
|
+
- Don't rely on memcached's auto-create for keys that need base64 encoding (GHSA-r47x-6p6r-mvcp)
|
|
13
|
+
- memcached stores an item made by auto-create (`ma` with `N`/`J`, or `mg` with `N`) for a base64-encoded key without its binary-key flag, and later replies name the item by its raw key bytes. A key containing spaces or CR/LF could then make a multi-get read the replies after it as different ones
|
|
14
|
+
- `incr` and `decr` with an initial value now create such counters with an add instead. Return values are unchanged
|
|
15
|
+
- The new `verify_reply_keys` option makes `get_multi`, `get_multi_cas` and `get_multi_with_metadata` ignore replies for keys they didn't request, which guards against items created by older Dalli versions or other clients until memcached is upgraded. It's off by default, since it costs a multi-get about 15%
|
|
16
|
+
- Affects the meta protocol since 3.2.0; fixed in 5.2.3, 5.1.5, 5.0.11, 4.3.8 and 3.2.14. memcached 1.6.45 and earlier still name these items by their raw key bytes
|
|
17
|
+
|
|
18
|
+
Bug fixes:
|
|
19
|
+
|
|
20
|
+
- `get_multi_with_metadata` and its `CacheResult` adapter now retry transient network errors and raise `Dalli::NetworkError` on terminal failures instead of returning misses or partial results (#1222, @jaideeppyne)
|
|
21
|
+
- A per-request `raw: true` write no longer compresses the value (#1213)
|
|
22
|
+
- Values at or above the compression threshold (4 KB by default) were compressed and flagged even with `raw: true`. Raw reads ignore flags, so reading them back with `raw: true` returned the compressed bytes
|
|
23
|
+
- Raw values are now stored exactly as given. Callers that pass `raw: true` without `compress: false` store large values uncompressed; Rails' `MemCacheStore` passes `compress: false` and isn't affected
|
|
24
|
+
- Fix `cas` and `cas!` edge cases, and make `flush` attempt every server (#1214)
|
|
25
|
+
- `cas!` on a missing key used an unconditional set, so a value another client created while the block ran was overwritten. It now adds the key and returns `false` if another client got there first
|
|
26
|
+
- With `cache_nils: true`, `cas` treated a stored `nil` as a miss and didn't call the block
|
|
27
|
+
- The docs now say what `cas` and `cas!` return on success: the new CAS value, not `true`. The return values are unchanged
|
|
28
|
+
- `flush` stopped at the first server that raised, so later servers weren't flushed. It now flushes every server it can reach, then raises the first error
|
|
29
|
+
- Handle failed TLS handshakes like other connection failures, and warn when TLS verification is off (#1215)
|
|
30
|
+
- A failed or timed-out handshake left the TCP socket open until GC and raised `OpenSSL::SSL::SSLError` directly, so the server was never marked down. The socket is now closed, and the error goes through the usual retry and `socket_max_failures` handling, raising `Dalli::NetworkError` (or `Dalli::RingError` once no server is available)
|
|
31
|
+
- Certificate and hostname checks come only from the `ssl_context` you pass, and a bare `OpenSSL::SSL::SSLContext.new` checks neither. Dalli now logs a warning once per process when `verify_mode` doesn't include `VERIFY_PEER` or `verify_hostname` is false; see the new TLS section in the README
|
|
32
|
+
- Connection and configuration fixes (#1216)
|
|
33
|
+
- `memcached://` URIs with a bracketed IPv6 host (`memcached://[::1]:11211`) now connect
|
|
34
|
+
- A server weight below 1 raises `ArgumentError`. Before, a ring where every weight was 0 raised `FloatDomainError`
|
|
35
|
+
- A reply line longer than 8 KiB without a CRLF, or one cut short by the connection closing, is now a network error that closes the connection. Before, the line had no length limit
|
|
36
|
+
- On CRuby, DNS resolution is now limited by `socket_timeout` (`resolv_timeout:`) as well as the connect
|
|
37
|
+
- `Dalli::Client.new` no longer modifies the options hash it's given, and accepts a frozen one (#1217)
|
|
38
|
+
- `value_max_bytes` now counts the key and memcached's per-item overhead (#1217)
|
|
39
|
+
- memcached's `-I` limit covers the whole item, not only the value. A value of exactly 1 MB passed Dalli's check and then failed in memcached with `Dalli::ServerError`, which `quiet` blocks and `set_multi` silently discarded
|
|
40
|
+
- Dalli now checks value + key + 63 bytes, the overhead measured on memcached 1.6, and raises `Dalli::ValueOverMaxSize` before sending. A custom `value_max_bytes` is now that much stricter
|
|
41
|
+
- A String `value_max_bytes` is converted to an Integer instead of raising on every set
|
|
42
|
+
- `Rack::Session::Dalli` fixes (#1217)
|
|
43
|
+
- It no longer requires the `connection_pool` gem unless `pool_size` or `pool_timeout` is given
|
|
44
|
+
- `pool_size` and `pool_timeout` are no longer passed to `Dalli::Client`, and the options hash isn't modified
|
|
45
|
+
- A failed session delete is logged as a warning, not only when `$VERBOSE` is set
|
|
46
|
+
- One value that can't be stored no longer discards other buffered writes (#1218)
|
|
47
|
+
- A value that raised `Dalli::MarshalError` or `Dalli::ValueOverMaxSize` closed the connection, which dropped writes still buffered from the same `quiet` block or `set_multi`. Multi-server `set_multi` did this without raising
|
|
48
|
+
- Both `set_multi` paths now store every other key, skip the bad one, and then raise. Single-server `set_multi` used to store nothing
|
|
49
|
+
- When single-server `set_multi` failed to marshal a value, the error log included every value in the batch. It now lists only the keys
|
|
50
|
+
- `Rack::Session::Dalli`'s 401 for a session deleted mid-request is now a valid Rack response (#1220)
|
|
51
|
+
- When a session is deleted while a request is using it (for example, by a logout in another request), the store still returns 401 instead of writing the session back. The 401 now has a `content-type` and expires the session cookie, so the browser stops sending the dead session ID
|
|
52
|
+
- The app's response body is now closed, as the Rack spec requires. Before, it was dropped without `close`
|
|
53
|
+
- The app has already run when the 401 is returned, so its side effects have happened. This is documented in the README
|
|
54
|
+
|
|
55
|
+
Improvements:
|
|
56
|
+
|
|
57
|
+
- Add a `connect_timeout` option to set a separate timeout for DNS resolution and TCP/UNIX socket connects (including reconnects); it defaults to `socket_timeout`. TLS handshakes continue to use `socket_timeout` (#1221, @underwoo16)
|
|
58
|
+
|
|
7
59
|
5.2.2
|
|
8
60
|
==========
|
|
9
61
|
|
data/README.md
CHANGED
|
@@ -59,6 +59,17 @@ Dalli::Client.new('localhost:11211', namespace: 'myapp', namespace_separator: '/
|
|
|
59
59
|
|
|
60
60
|
The separator must be a single non-alphanumeric character. Valid examples: `:`, `/`, `|`, `.`, `-`, `_`, `#`
|
|
61
61
|
|
|
62
|
+
### Maximum Item Size
|
|
63
|
+
|
|
64
|
+
memcached rejects items larger than its `-I` setting (1MB by default). Set `value_max_bytes` to the same value if you change it:
|
|
65
|
+
|
|
66
|
+
```ruby
|
|
67
|
+
# memcached started with -I 2m
|
|
68
|
+
Dalli::Client.new('localhost:11211', value_max_bytes: 2 * 1024 * 1024)
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
memcached's limit covers the whole item, so Dalli checks the stored (serialized and compressed) value plus the key plus 63 bytes of per-item overhead against `value_max_bytes`, and raises `Dalli::ValueOverMaxSize` before sending anything larger. With the defaults, the largest value you can store under a 10-byte key is 1,048,576 - 10 - 63 = 1,048,503 bytes.
|
|
72
|
+
|
|
62
73
|
### Deferred Draining
|
|
63
74
|
|
|
64
75
|
By default, a `quiet` (or `multi`) block ends by waiting for the replies memcached still sends for its quiet requests, one round trip for each server the block wrote to. With `defer_drain: true`, the block returns without waiting:
|
|
@@ -112,6 +123,35 @@ The digest is MD5 by default. `digest_class:` takes any object that responds to
|
|
|
112
123
|
|
|
113
124
|
See the [5.0-Upgrade.md](5.0-Upgrade.md) guide for upgrade information.
|
|
114
125
|
|
|
126
|
+
### Counters and verify_reply_keys
|
|
127
|
+
|
|
128
|
+
memcached names an item made by auto-create for a base64-encoded key (one with whitespace, control or non-ASCII characters) by its raw key bytes in later replies. Since 5.2.3, Dalli creates counters for such keys with an add rather than auto-create. Items created before then, or by other clients, can still be named that way until memcached releases a fix (1.6.45 and earlier behave this way). If keys passed to `incr` or `decr` with an initial value can contain untrusted input, set `verify_reply_keys: true` until then: multi-gets then ignore replies for keys they didn't request, at a cost of about 15% per multi-get.
|
|
129
|
+
|
|
130
|
+
## TLS
|
|
131
|
+
|
|
132
|
+
Pass an `OpenSSL::SSL::SSLContext` as `ssl_context:` to connect over TLS. Certificate and hostname verification come only from that context: Dalli adds no checks of its own. A bare `OpenSSL::SSL::SSLContext.new` verifies nothing, so it accepts any certificate from any server. Configure the context to verify both:
|
|
133
|
+
|
|
134
|
+
```ruby
|
|
135
|
+
ssl_context = OpenSSL::SSL::SSLContext.new
|
|
136
|
+
ssl_context.set_params(
|
|
137
|
+
verify_mode: OpenSSL::SSL::VERIFY_PEER,
|
|
138
|
+
verify_hostname: true,
|
|
139
|
+
ca_file: '/path/to/ca.crt' # omit to use the system's trusted certificates
|
|
140
|
+
)
|
|
141
|
+
|
|
142
|
+
Dalli::Client.new('memcached.example.com:11211', ssl_context: ssl_context)
|
|
143
|
+
```
|
|
144
|
+
|
|
145
|
+
Dalli logs a warning once per process when the context's `verify_mode` is `VERIFY_NONE`, or when it is `VERIFY_PEER` but `verify_hostname` is false.
|
|
146
|
+
|
|
147
|
+
A failed TLS handshake counts as a failed connection attempt, like a refused connection: once `socket_max_failures` attempts have failed, the server is marked down and requests fail over to the other servers.
|
|
148
|
+
|
|
149
|
+
## Rack Sessions
|
|
150
|
+
|
|
151
|
+
`Rack::Session::Dalli` stores Rack sessions in memcached. If a session is deleted while a request is using it (for example, when the user logs out in another tab during a slow request), the store doesn't write the session back, so a logged-out session can't come back. It replaces the app's response with a `401 Wrong session ID` that expires the session cookie.
|
|
152
|
+
|
|
153
|
+
The app has already run by then, so whatever it did (database writes, emails, and so on) has happened, and only its response is lost. If a request needs a different outcome, check that the session still exists before making the changes.
|
|
154
|
+
|
|
115
155
|
## OpenTelemetry Tracing
|
|
116
156
|
|
|
117
157
|
Dalli automatically instruments operations with [OpenTelemetry](https://opentelemetry.io/) when the SDK is present. No configuration is required - just add the OpenTelemetry gems to your application:
|
data/lib/dalli/client.rb
CHANGED
|
@@ -35,15 +35,20 @@ module Dalli
|
|
|
35
35
|
# to memcached. Default: true.
|
|
36
36
|
# - :compression_min_size - the minimum size (in bytes) for which Dalli will compress values sent to Memcached.
|
|
37
37
|
# Defaults to 4K.
|
|
38
|
+
# - :value_max_bytes - the largest item memcached will accept, i.e. its -I setting. Defaults to 1MB. Like
|
|
39
|
+
# memcached, Dalli counts the key and 63 bytes of item overhead against this limit along
|
|
40
|
+
# with the (serialized, compressed) value, so the largest value that can be stored is
|
|
41
|
+
# value_max_bytes - key bytesize - 63. Larger values raise Dalli::ValueOverMaxSize.
|
|
38
42
|
# - :serializer - defaults to Marshal
|
|
39
43
|
# - :compressor - defaults to Dalli::Compressor, a Zlib-based implementation
|
|
40
44
|
# - :cache_nils - defaults to false, if true Dalli will not treat cached nil values as 'not found' for
|
|
41
45
|
# #fetch operations.
|
|
42
46
|
# - :raw - If set, disables serialization and compression entirely at the client level.
|
|
43
47
|
# Only String values are supported. This is useful when the caller handles its own
|
|
44
|
-
# serialization (e.g., Rails' ActiveSupport::Cache).
|
|
45
|
-
#
|
|
46
|
-
#
|
|
48
|
+
# serialization (e.g., Rails' ActiveSupport::Cache). The per-request :raw option
|
|
49
|
+
# (e.g. set(key, value, ttl, raw: true)) does the same for a single call: String
|
|
50
|
+
# values are stored as-is, uncompressed, and non-String values raise
|
|
51
|
+
# Dalli::MarshalError.
|
|
47
52
|
# - :digest_class - defaults to Digest::MD5, allows you to pass in an object that responds to the hexdigest method,
|
|
48
53
|
# useful for injecting a FIPS compliant hash object.
|
|
49
54
|
# - :otel_db_statement - controls the +db.query.text+ span attribute when OpenTelemetry is loaded.
|
|
@@ -58,6 +63,17 @@ module Dalli
|
|
|
58
63
|
# quiet requests (NF, NS, errors) wait on the connection until then, so a client that only makes
|
|
59
64
|
# quiet requests should call #drain_deferred_responses periodically. See "Deferred Draining" in
|
|
60
65
|
# the README.
|
|
66
|
+
# - :socket_timeout - timeout in seconds for reads and writes on a connection, and for the TLS
|
|
67
|
+
# handshake. Also used for connecting unless :connect_timeout is set.
|
|
68
|
+
# Default: 1.
|
|
69
|
+
# - :connect_timeout - timeout in seconds for DNS resolution and TCP/UNIX socket connects,
|
|
70
|
+
# including reconnects after a failure. Defaults to :socket_timeout.
|
|
71
|
+
# TLS handshakes use :socket_timeout.
|
|
72
|
+
# - :verify_reply_keys - defaults to false. When true, get_multi, get_multi_cas and get_multi_with_metadata only
|
|
73
|
+
# accept replies for the keys they requested; after a reply naming another key, the rest of
|
|
74
|
+
# that server's replies are treated as misses. memcached can name an item by its raw key
|
|
75
|
+
# bytes when the item was created by auto-create from a base64-encoded key, which Dalli no
|
|
76
|
+
# longer does. Costs a multi-get about 15%.
|
|
61
77
|
#
|
|
62
78
|
def initialize(servers = nil, options = {})
|
|
63
79
|
@normalized_servers = ::Dalli::ServersArgNormalizer.normalize_servers(servers)
|
|
@@ -226,9 +242,14 @@ module Dalli
|
|
|
226
242
|
#
|
|
227
243
|
# `req_options` accepts :p_token/:l_token, applied to every key in the batch.
|
|
228
244
|
#
|
|
245
|
+
# A transient network error is retried automatically. If a server remains
|
|
246
|
+
# unreachable after retrying, raises Dalli::NetworkError rather than
|
|
247
|
+
# returning its keys as misses or yielding a partial result.
|
|
248
|
+
#
|
|
229
249
|
# @param keys [Array<String>] the keys to fetch
|
|
230
250
|
# @param req_options [Hash, nil] routing-token options
|
|
231
251
|
# @return [Hash] key => { value:, cas:, stale:, miss: }
|
|
252
|
+
# @raise [Dalli::NetworkError] if a server is unreachable after retrying
|
|
232
253
|
def get_multi_with_metadata(*keys, req_options: nil, &block)
|
|
233
254
|
keys.flatten!
|
|
234
255
|
keys.compact!
|
|
@@ -371,24 +392,26 @@ module Dalli
|
|
|
371
392
|
# compare and swap values using optimistic locking.
|
|
372
393
|
# Fetch the existing value for key.
|
|
373
394
|
# If it exists, yield the value to the block.
|
|
374
|
-
#
|
|
375
|
-
#
|
|
395
|
+
# Store the block's return value as the new value for the key.
|
|
396
|
+
# The store will fail if someone else changed the value.
|
|
376
397
|
#
|
|
377
398
|
# Returns:
|
|
378
|
-
# - nil if the key did not exist
|
|
399
|
+
# - nil if the key did not exist (or held nil and :cache_nils is false);
|
|
400
|
+
# the block is not called.
|
|
379
401
|
# - false if the value was changed by someone else.
|
|
380
|
-
# -
|
|
402
|
+
# - the new CAS value (a positive Integer) if the value was successfully updated.
|
|
381
403
|
def cas(key, ttl = nil, req_options = nil, &)
|
|
382
404
|
cas_core(key, false, ttl, req_options, &)
|
|
383
405
|
end
|
|
384
406
|
|
|
385
407
|
##
|
|
386
408
|
# like #cas, but will yield to the block whether or not the value
|
|
387
|
-
# already exists.
|
|
409
|
+
# already exists. If the key did not exist, the block receives nil.
|
|
388
410
|
#
|
|
389
411
|
# Returns:
|
|
390
|
-
# - false if the value was changed by someone else
|
|
391
|
-
#
|
|
412
|
+
# - false if the value was changed by someone else, including another
|
|
413
|
+
# client creating a missing key while the block ran.
|
|
414
|
+
# - the new CAS value (a positive Integer) if the value was successfully updated.
|
|
392
415
|
def cas!(key, ttl = nil, req_options = nil, &)
|
|
393
416
|
cas_core(key, true, ttl, req_options, &)
|
|
394
417
|
end
|
|
@@ -650,9 +673,20 @@ module Dalli
|
|
|
650
673
|
##
|
|
651
674
|
# Flush the memcached server, at 'delay' seconds in the future.
|
|
652
675
|
# Delay defaults to zero seconds, which means an immediate flush.
|
|
676
|
+
# Every server is attempted. If any of them fails, the first error is
|
|
677
|
+
# raised once all the others have been flushed.
|
|
653
678
|
##
|
|
654
679
|
def flush(delay = 0)
|
|
655
|
-
|
|
680
|
+
first_error = nil
|
|
681
|
+
results = ring.servers.map do |s|
|
|
682
|
+
s.request(:flush, delay)
|
|
683
|
+
rescue StandardError => e
|
|
684
|
+
first_error ||= e
|
|
685
|
+
nil
|
|
686
|
+
end
|
|
687
|
+
raise first_error if first_error
|
|
688
|
+
|
|
689
|
+
results
|
|
656
690
|
end
|
|
657
691
|
alias flush_all flush
|
|
658
692
|
|
|
@@ -805,16 +839,26 @@ module Dalli
|
|
|
805
839
|
Dalli.logger.debug { e.inspect }
|
|
806
840
|
Dalli.logger.debug { 'retrying single-server get_multi because of network error' }
|
|
807
841
|
retry
|
|
842
|
+
rescue Dalli::UnexpectedReplyKeyError => e
|
|
843
|
+
# The connection is closed and the keys are treated as missing, as the
|
|
844
|
+
# pipelined get_multi does for a server whose replies are out of step
|
|
845
|
+
Dalli.logger.warn { e.message }
|
|
846
|
+
{}
|
|
808
847
|
end
|
|
809
848
|
|
|
810
849
|
def single_server_get_multi_with_metadata(keys, req_options = nil)
|
|
811
|
-
keys.map
|
|
850
|
+
validated_keys = keys.map { |k| @key_manager.validate_key(k.to_s) }
|
|
812
851
|
return {} unless (server = single_server)
|
|
813
852
|
|
|
814
|
-
result = server.request(:read_multi_with_metadata_req,
|
|
853
|
+
result = server.request(:read_multi_with_metadata_req, validated_keys, req_options)
|
|
815
854
|
result.transform_keys! { |k| @key_manager.key_without_namespace(k) }
|
|
816
855
|
result
|
|
817
|
-
rescue Dalli::
|
|
856
|
+
rescue Dalli::RetryableNetworkError => e
|
|
857
|
+
Dalli.logger.debug { e.inspect }
|
|
858
|
+
Dalli.logger.debug { 'retrying single-server get_multi_with_metadata because of network error' }
|
|
859
|
+
retry
|
|
860
|
+
rescue Dalli::UnexpectedReplyKeyError => e
|
|
861
|
+
Dalli.logger.warn { e.message }
|
|
818
862
|
{}
|
|
819
863
|
end
|
|
820
864
|
|
|
@@ -913,12 +957,23 @@ module Dalli
|
|
|
913
957
|
def cas_core(key, always_set, ttl = nil, req_options = nil)
|
|
914
958
|
validate_routing_tokens!(req_options)
|
|
915
959
|
(value, cas) = perform(:cas, key, req_options)
|
|
916
|
-
return if
|
|
960
|
+
return if !always_set && cas_miss?(value, cas)
|
|
917
961
|
|
|
918
962
|
newvalue = yield(value)
|
|
963
|
+
# A CAS of 0 means the key was missing. A set with CAS 0 is unconditional,
|
|
964
|
+
# so add instead: it fails if another client created the key meanwhile.
|
|
965
|
+
return perform(:add, key, newvalue, ttl_or_default(ttl), req_options) if cas.zero?
|
|
966
|
+
|
|
919
967
|
perform(:set, key, newvalue, ttl_or_default(ttl), cas, req_options)
|
|
920
968
|
end
|
|
921
969
|
|
|
970
|
+
# The CAS lookup returns [nil, 0] on a miss rather than NOT_FOUND, and a
|
|
971
|
+
# stored item always has a non-zero CAS. So under cache_nils a zero CAS
|
|
972
|
+
# marks the miss, and a stored nil is a hit, as in #not_found?.
|
|
973
|
+
def cas_miss?(value, cas)
|
|
974
|
+
cache_nils ? cas.zero? : value.nil?
|
|
975
|
+
end
|
|
976
|
+
|
|
922
977
|
def fetch_with_lock_request(key, ttl, lock_ttl, recache_threshold, req_options)
|
|
923
978
|
server = ring.server_for_key(key)
|
|
924
979
|
# req_options is the base, not the override: fetch_with_lock's own
|
|
@@ -986,7 +1041,10 @@ module Dalli
|
|
|
986
1041
|
retry
|
|
987
1042
|
end
|
|
988
1043
|
|
|
989
|
-
|
|
1044
|
+
# Works on a copy so the caller's hash is neither modified here nor able
|
|
1045
|
+
# to change this client's configuration later.
|
|
1046
|
+
def normalize_options(options)
|
|
1047
|
+
opts = options.dup
|
|
990
1048
|
opts[:expires_in] = opts[:expires_in].to_i if opts[:expires_in]
|
|
991
1049
|
opts
|
|
992
1050
|
rescue NoMethodError
|
|
@@ -60,6 +60,9 @@ module Dalli
|
|
|
60
60
|
#
|
|
61
61
|
# `req_options` accepts :p_token/:l_token, applied to every key in the batch.
|
|
62
62
|
#
|
|
63
|
+
# A transient network error is retried automatically. A terminal
|
|
64
|
+
# Dalli::NetworkError propagates instead of returning partial metadata.
|
|
65
|
+
#
|
|
63
66
|
def process_with_metadata(keys, req_options = nil)
|
|
64
67
|
return {} if keys.empty?
|
|
65
68
|
|
|
@@ -67,9 +70,9 @@ module Dalli
|
|
|
67
70
|
results = {}
|
|
68
71
|
groups_for_keys(keys).each do |server, keys_for_server|
|
|
69
72
|
results.merge!(server.request(:read_multi_with_metadata_req, keys_for_server, req_options))
|
|
70
|
-
rescue Dalli::
|
|
73
|
+
rescue Dalli::NetworkError
|
|
71
74
|
raise
|
|
72
|
-
rescue DalliError
|
|
75
|
+
rescue DalliError => e
|
|
73
76
|
Dalli.logger.debug { e.inspect }
|
|
74
77
|
Dalli.logger.debug { "unable to get keys for server #{server.name}" }
|
|
75
78
|
end
|
|
@@ -18,6 +18,9 @@ module Dalli
|
|
|
18
18
|
# A transient network error is retried automatically. If a server remains
|
|
19
19
|
# unreachable after retrying, raises Dalli::NetworkError.
|
|
20
20
|
#
|
|
21
|
+
# A value that can't be marshalled or is over value_max_bytes is skipped;
|
|
22
|
+
# the other keys are still stored, then the first such error is raised.
|
|
23
|
+
#
|
|
21
24
|
# @param hash [Hash] key-value pairs to set
|
|
22
25
|
# @param ttl [Integer] time-to-live in seconds
|
|
23
26
|
# @param req_options [Hash] options passed to each set operation
|
|
@@ -26,10 +29,12 @@ module Dalli
|
|
|
26
29
|
def process(hash, ttl, req_options)
|
|
27
30
|
return if hash.empty?
|
|
28
31
|
|
|
32
|
+
@value_error = nil
|
|
29
33
|
@ring.lock do
|
|
30
34
|
servers = setup_requests(hash, ttl, req_options)
|
|
31
35
|
finish_requests(servers)
|
|
32
36
|
end
|
|
37
|
+
raise @value_error if @value_error
|
|
33
38
|
rescue Dalli::RetryableNetworkError => e
|
|
34
39
|
Dalli.logger.debug { e.inspect }
|
|
35
40
|
Dalli.logger.debug { 'retrying pipelined sets because of network error' }
|
|
@@ -62,6 +67,10 @@ module Dalli
|
|
|
62
67
|
server.request(:pipelined_set, key, value, ttl, req_options)
|
|
63
68
|
rescue Dalli::NetworkError
|
|
64
69
|
raise
|
|
70
|
+
rescue Dalli::MarshalError, Dalli::ValueOverMaxSize => e
|
|
71
|
+
# Nothing was written for this key, so the server's other keys are
|
|
72
|
+
# unaffected; raised by #process once the batch is complete
|
|
73
|
+
@value_error ||= e
|
|
65
74
|
rescue DalliError => e
|
|
66
75
|
Dalli.logger.debug { e.inspect }
|
|
67
76
|
Dalli.logger.debug { "unable to set key #{key} for server #{server.name}" }
|
data/lib/dalli/protocol/base.rb
CHANGED
|
@@ -43,6 +43,7 @@ module Dalli
|
|
|
43
43
|
@options = client_options.merge(user_creds)
|
|
44
44
|
@raw_mode = client_options[:raw]
|
|
45
45
|
@defer_drain = client_options[:defer_drain] ? true : false
|
|
46
|
+
@verify_reply_keys = client_options[:verify_reply_keys] ? true : false
|
|
46
47
|
@value_marshaller = @raw_mode ? StringMarshaller.new(@options) : ValueMarshaller.new(@options)
|
|
47
48
|
@connection_manager = ConnectionManager.new(hostname, port, socket_type, @options)
|
|
48
49
|
end
|
|
@@ -79,8 +80,14 @@ module Dalli
|
|
|
79
80
|
|
|
80
81
|
request_completed = true
|
|
81
82
|
response
|
|
82
|
-
rescue Dalli::MarshalError => e
|
|
83
|
-
log_marshal_err(args.first, e)
|
|
83
|
+
rescue Dalli::MarshalError, Dalli::ValueOverMaxSize => e
|
|
84
|
+
log_marshal_err(args.first, e) if e.is_a?(Dalli::MarshalError)
|
|
85
|
+
# Raised only while the connection is clean: a value is marshalled
|
|
86
|
+
# before any of its request is written, and write_multi_req raises
|
|
87
|
+
# only after reading every reply. Closing the connection would
|
|
88
|
+
# discard earlier quiet writes still waiting in the write buffer.
|
|
89
|
+
@connection_manager.abort_request!
|
|
90
|
+
request_completed = true
|
|
84
91
|
raise
|
|
85
92
|
rescue Dalli::DalliError
|
|
86
93
|
raise
|
|
@@ -153,6 +160,8 @@ module Dalli
|
|
|
153
160
|
# avoiding intermediate Hash allocation. Returns nil.
|
|
154
161
|
# Without a block, returns a Hash of { key => [value, cas] }.
|
|
155
162
|
def pipeline_next_responses(&block)
|
|
163
|
+
return abandon_pipeline if @pipeline_out_of_step
|
|
164
|
+
|
|
156
165
|
reconnect_on_pipeline_complete!
|
|
157
166
|
values = nil
|
|
158
167
|
|
|
@@ -174,6 +183,13 @@ module Dalli
|
|
|
174
183
|
# to the noop at the end of the pipeline
|
|
175
184
|
finish_pipeline && break if status && key.nil?
|
|
176
185
|
|
|
186
|
+
# A key that wasn't requested means the replies are out of step
|
|
187
|
+
# (see expect_reply_keys): keep what was read so far, drop the rest
|
|
188
|
+
if unexpected_reply_key?(key)
|
|
189
|
+
abandon_pipeline
|
|
190
|
+
break
|
|
191
|
+
end
|
|
192
|
+
|
|
177
193
|
# If the status is ok and the key is not nil, then this is a
|
|
178
194
|
# getkq response with a value that we want to set in the response hash
|
|
179
195
|
unless key.nil?
|
|
@@ -346,6 +362,7 @@ module Dalli
|
|
|
346
362
|
# the expected state before pipeline_response_setup is called.
|
|
347
363
|
response_buffer.clear
|
|
348
364
|
response_buffer.raw = raw_request?(options)
|
|
365
|
+
expect_reply_keys(keys)
|
|
349
366
|
|
|
350
367
|
# The terminating noop is sent by pipeline_response_setup
|
|
351
368
|
write(quiet_get_requests(keys, options, return_cas: return_cas))
|
|
@@ -358,6 +375,7 @@ module Dalli
|
|
|
358
375
|
# Initialize the response buffer for draining during send phase
|
|
359
376
|
response_buffer.ensure_ready
|
|
360
377
|
response_buffer.raw = raw_request?(options)
|
|
378
|
+
expect_reply_keys(keys)
|
|
361
379
|
|
|
362
380
|
keys.each_slice(chunk_size) do |chunk|
|
|
363
381
|
# Build and write this chunk of requests
|
|
@@ -390,6 +408,12 @@ module Dalli
|
|
|
390
408
|
status, cas, key, value = response_buffer.process_single_getk_response
|
|
391
409
|
break if status.nil? # No complete response available
|
|
392
410
|
|
|
411
|
+
# Handled by the next pipeline_next_responses, once sending is done
|
|
412
|
+
if unexpected_reply_key?(key)
|
|
413
|
+
@pipeline_out_of_step = true
|
|
414
|
+
break
|
|
415
|
+
end
|
|
416
|
+
|
|
393
417
|
results[key] = [value, cas] unless key.nil?
|
|
394
418
|
end
|
|
395
419
|
rescue SystemCallError, Dalli::NetworkError
|
|
@@ -401,6 +425,42 @@ module Dalli
|
|
|
401
425
|
@response_buffer ||= ResponseBuffer.new(@connection_manager, response_processor)
|
|
402
426
|
end
|
|
403
427
|
|
|
428
|
+
# A multi-get only accepts replies for the keys it asked for. memcached
|
|
429
|
+
# names an item created by auto-create (ma or mg with N) for a
|
|
430
|
+
# base64-encoded key by its raw bytes, without the b flag; such a key
|
|
431
|
+
# can contain spaces or CR/LF, which split the reply header and make the
|
|
432
|
+
# replies after it parse as different ones. A key that wasn't requested
|
|
433
|
+
# is how that shows up.
|
|
434
|
+
# Only with the verify_reply_keys option, since building the set costs a
|
|
435
|
+
# multi-get about 15%. Dalli itself no longer creates such items (see
|
|
436
|
+
# Meta#decr_incr); the check guards against ones that older versions or
|
|
437
|
+
# other clients created, on memcached versions with the behavior.
|
|
438
|
+
def expect_reply_keys(keys)
|
|
439
|
+
@pipeline_out_of_step = false
|
|
440
|
+
@expected_reply_keys = (keys.to_set { |k| reply_key_form(k) } if @verify_reply_keys)
|
|
441
|
+
end
|
|
442
|
+
|
|
443
|
+
def unexpected_reply_key?(key)
|
|
444
|
+
!key.nil? && @expected_reply_keys && !@expected_reply_keys.include?(reply_key_form(key))
|
|
445
|
+
end
|
|
446
|
+
|
|
447
|
+
# Keys compared as bytes; ASCII keys compare equal in any encoding as is
|
|
448
|
+
def reply_key_form(key)
|
|
449
|
+
key.ascii_only? ? key : key.b
|
|
450
|
+
end
|
|
451
|
+
|
|
452
|
+
# Keeps what was read, drops the rest of this server's replies, and
|
|
453
|
+
# closes the connection so nothing unread is left on it. Not a socket
|
|
454
|
+
# failure, so it doesn't count toward socket_max_failures; the keys not
|
|
455
|
+
# read are simply missing from the result.
|
|
456
|
+
def abandon_pipeline
|
|
457
|
+
Dalli.logger.warn { "#{name} replied to a get_multi with a key that wasn't requested; closing the connection" }
|
|
458
|
+
@pipeline_out_of_step = false
|
|
459
|
+
response_buffer.clear
|
|
460
|
+
@connection_manager.close
|
|
461
|
+
{}
|
|
462
|
+
end
|
|
463
|
+
|
|
404
464
|
# Called after the noop response is received at the end of a set
|
|
405
465
|
# of pipelined gets
|
|
406
466
|
def finish_pipeline
|
|
@@ -414,7 +474,10 @@ module Dalli
|
|
|
414
474
|
@connection_manager.reconnect! 'pipelined get has completed' if pipeline_complete?
|
|
415
475
|
end
|
|
416
476
|
|
|
477
|
+
# The single-server set_multi path passes its whole pairs hash as the
|
|
478
|
+
# first argument; log only its keys, never the values
|
|
417
479
|
def log_marshal_err(key, err)
|
|
480
|
+
key = key.keys.join("', '") if key.is_a?(Hash)
|
|
418
481
|
Dalli.logger.error "Marshalling error for key '#{key}': #{err.message}"
|
|
419
482
|
Dalli.logger.error 'You are trying to cache a Ruby object which cannot be serialized to memcached.'
|
|
420
483
|
end
|
|
@@ -60,8 +60,8 @@ module Dalli
|
|
|
60
60
|
@pid = Process.pid
|
|
61
61
|
@request_in_progress = false
|
|
62
62
|
@quiet_responses_pending = false
|
|
63
|
-
rescue SystemCallError, *TIMEOUT_ERRORS, EOFError, SocketError => e
|
|
64
|
-
# SocketError = DNS resolution failure
|
|
63
|
+
rescue SystemCallError, *TIMEOUT_ERRORS, *SSL_ERRORS, EOFError, SocketError => e
|
|
64
|
+
# SocketError = DNS resolution failure; SSL_ERRORS = failed TLS handshake
|
|
65
65
|
error_on_request!(e)
|
|
66
66
|
end
|
|
67
67
|
|
|
@@ -188,10 +188,22 @@ module Dalli
|
|
|
188
188
|
@request_in_progress = false
|
|
189
189
|
end
|
|
190
190
|
|
|
191
|
+
# The longest reply line Dalli reads, in bytes, CRLF included. A meta
|
|
192
|
+
# reply header is at most about 500 bytes: "VA <size>" plus its flags,
|
|
193
|
+
# the largest being a 250-byte key base64-encoded to 336 bytes, a
|
|
194
|
+
# 20-digit CAS, a 32-byte opaque and a few numeric fields. The longest
|
|
195
|
+
# stats line (stats settings) carries a file path such as ext_path, up
|
|
196
|
+
# to PATH_MAX (4096). 8 KiB leaves room for both, while a server that
|
|
197
|
+
# never sends CRLF can't grow the read buffer without bound.
|
|
198
|
+
MAX_LINE_BYTES = 8 * 1024
|
|
199
|
+
|
|
191
200
|
def read_line
|
|
192
201
|
flush_write_buffer
|
|
193
|
-
data = @sock.gets("\r\n")
|
|
202
|
+
data = @sock.gets("\r\n", MAX_LINE_BYTES)
|
|
194
203
|
error_on_request!('EOF in read_line') if data.nil?
|
|
204
|
+
# Without its CRLF the line either ran past MAX_LINE_BYTES or was cut
|
|
205
|
+
# short by EOF; either way the connection is out of step
|
|
206
|
+
error_on_request!("Reply line without CRLF in the first #{MAX_LINE_BYTES} bytes") unless data.end_with?("\r\n")
|
|
195
207
|
data
|
|
196
208
|
rescue SystemCallError, *TIMEOUT_ERRORS, *SSL_ERRORS, EOFError => e
|
|
197
209
|
error_on_request!(e)
|
data/lib/dalli/protocol/meta.rb
CHANGED
|
@@ -234,12 +234,45 @@ module Dalli
|
|
|
234
234
|
# rubocop:disable-next Metrics/ParameterLists
|
|
235
235
|
def decr_incr(incr, key, delta, ttl, initial, options = nil)
|
|
236
236
|
ttl = initial ? TtlSanitizer.sanitize(ttl) : nil # Only set a TTL if we want to set a value on miss
|
|
237
|
+
# memcached stores an item created by ma's auto-create (N/J) for a
|
|
238
|
+
# base64-encoded key without its binary-key flag, and later replies
|
|
239
|
+
# name it with the raw key bytes. Create such counters with an add.
|
|
240
|
+
return decr_incr_with_add(incr, key, delta, ttl, initial, options) if initial && KeyRegularizer.required?(key)
|
|
241
|
+
|
|
237
242
|
write(RequestFormatter.meta_arithmetic(key: key, delta: delta, initial: initial, incr: incr, ttl: ttl,
|
|
238
243
|
quiet: quiet?, **routing_token_kwargs(options)))
|
|
239
244
|
finish_write(quiet?)
|
|
240
245
|
response_processor.decr_incr unless quiet?
|
|
241
246
|
end
|
|
242
247
|
|
|
248
|
+
# Like decr_incr with an initial value, without memcached's auto-create:
|
|
249
|
+
# on a miss the counter is added with the initial value (which is
|
|
250
|
+
# returned, as auto-create would), and the add losing a race to another
|
|
251
|
+
# client means the counter now exists, so the incr is tried again. In a
|
|
252
|
+
# quiet block both requests are sent and whichever applies takes effect.
|
|
253
|
+
# rubocop:disable-next Metrics/ParameterLists
|
|
254
|
+
def decr_incr_with_add(incr, key, delta, ttl, initial, options)
|
|
255
|
+
add_options = (options || {}).merge(raw: true) # counters are stored raw
|
|
256
|
+
return quiet_decr_incr_with_add(incr, key, delta, ttl, initial, add_options) if quiet?
|
|
257
|
+
|
|
258
|
+
2.times do
|
|
259
|
+
value = decr_incr(incr, key, delta, nil, nil, options)
|
|
260
|
+
return value unless value.nil?
|
|
261
|
+
|
|
262
|
+
write_storage_req(:add, key, initial.to_s, ttl, nil, add_options, quiet: false)
|
|
263
|
+
return initial if response_processor.meta_set_with_cas
|
|
264
|
+
end
|
|
265
|
+
decr_incr(incr, key, delta, nil, nil, options)
|
|
266
|
+
end
|
|
267
|
+
|
|
268
|
+
# rubocop:disable-next Metrics/ParameterLists
|
|
269
|
+
def quiet_decr_incr_with_add(incr, key, delta, ttl, initial, add_options)
|
|
270
|
+
write(RequestFormatter.meta_arithmetic(key: key, delta: delta, initial: nil, incr: incr, ttl: nil,
|
|
271
|
+
quiet: true, **routing_token_kwargs(add_options)))
|
|
272
|
+
write_storage_req(:add, key, initial.to_s, ttl, nil, add_options, quiet: true)
|
|
273
|
+
finish_write(true)
|
|
274
|
+
end
|
|
275
|
+
|
|
243
276
|
# Other Commands
|
|
244
277
|
def flush(delay = 0)
|
|
245
278
|
write(RequestFormatter.flush(delay: delay))
|
|
@@ -293,7 +326,7 @@ module Dalli
|
|
|
293
326
|
buffer = RequestFormatter.multi_meta_get(keys, skip_flags: is_raw, **routing_token_kwargs(options))
|
|
294
327
|
flushed_write(buffer)
|
|
295
328
|
buffer.clear
|
|
296
|
-
read_multi_get_responses(is_raw)
|
|
329
|
+
read_multi_get_responses(is_raw, keys)
|
|
297
330
|
end
|
|
298
331
|
|
|
299
332
|
# Stale-aware bulk get. Returns { key => { value:, cas:, stale:, miss: } }
|
|
@@ -310,12 +343,13 @@ module Dalli
|
|
|
310
343
|
**routing_token_kwargs(options))
|
|
311
344
|
flushed_write(buffer)
|
|
312
345
|
buffer.clear
|
|
313
|
-
read_multi_metadata_responses(is_raw)
|
|
346
|
+
read_multi_metadata_responses(is_raw, keys)
|
|
314
347
|
end
|
|
315
348
|
|
|
316
349
|
# Unlike read_multi_get_responses this locates tokens by flag rather than
|
|
317
350
|
# by position, because the c flag shifts the key's index.
|
|
318
|
-
def read_multi_metadata_responses(is_raw)
|
|
351
|
+
def read_multi_metadata_responses(is_raw, keys)
|
|
352
|
+
expect_reply_keys(keys)
|
|
319
353
|
hash = {}
|
|
320
354
|
while (line = @connection_manager.read_line)
|
|
321
355
|
break if line.start_with?('MN')
|
|
@@ -331,23 +365,34 @@ module Dalli
|
|
|
331
365
|
key = response_processor.key_from_tokens(tokens)
|
|
332
366
|
next if key.nil?
|
|
333
367
|
|
|
368
|
+
raise_unexpected_reply_key(key) if unexpected_reply_key?(key)
|
|
369
|
+
|
|
334
370
|
hash[key] = { value: @value_marshaller.retrieve(value, bitflags), cas: cas, stale: stale, miss: false }
|
|
335
371
|
end
|
|
336
372
|
hash
|
|
337
373
|
end
|
|
338
374
|
|
|
339
|
-
def read_multi_get_responses(is_raw)
|
|
375
|
+
def read_multi_get_responses(is_raw, keys)
|
|
376
|
+
expect_reply_keys(keys)
|
|
340
377
|
hash = {}
|
|
341
378
|
while (line = @connection_manager.read_line)
|
|
342
379
|
break if line.start_with?('MN')
|
|
343
380
|
next unless line.start_with?('VA ')
|
|
344
381
|
|
|
345
382
|
key, value = parse_multi_get_value(line, is_raw)
|
|
383
|
+
raise_unexpected_reply_key(key) if unexpected_reply_key?(key)
|
|
346
384
|
hash[key] = value if key
|
|
347
385
|
end
|
|
348
386
|
hash
|
|
349
387
|
end
|
|
350
388
|
|
|
389
|
+
# The replies after a key that wasn't requested can't be trusted (see
|
|
390
|
+
# Base#expect_reply_keys). Raising ends the request, which closes the
|
|
391
|
+
# connection; the caller treats this server's keys as missing.
|
|
392
|
+
def raise_unexpected_reply_key(key)
|
|
393
|
+
raise Dalli::UnexpectedReplyKeyError, "#{name} replied with a key that wasn't requested: #{key.inspect}"
|
|
394
|
+
end
|
|
395
|
+
|
|
351
396
|
# Reads the "VA <size> [f<flags>] k<key> ..." line in place rather than
|
|
352
397
|
# splitting it into tokens
|
|
353
398
|
def parse_multi_get_value(line, is_raw)
|
|
@@ -364,16 +409,27 @@ module Dalli
|
|
|
364
409
|
|
|
365
410
|
# Single-server fast path for set_multi. Inlines request formatting to
|
|
366
411
|
# minimize per-key overhead. Avoids PipelinedSetter server grouping.
|
|
412
|
+
#
|
|
413
|
+
# A value that can't be marshalled or is over value_max_bytes is skipped;
|
|
414
|
+
# the other keys are still stored, then the first such error is raised
|
|
415
|
+
# (as PipelinedSetter does on the multi-server path).
|
|
367
416
|
def write_multi_req(pairs, ttl, req_options)
|
|
368
417
|
ttl = TtlSanitizer.sanitize(ttl) if ttl
|
|
369
|
-
|
|
418
|
+
value_error = nil
|
|
419
|
+
entries = pairs.filter_map do |key, raw_value|
|
|
370
420
|
[key, @value_marshaller.store(key, raw_value, req_options)]
|
|
421
|
+
rescue Dalli::MarshalError, Dalli::ValueOverMaxSize => e
|
|
422
|
+
value_error ||= e
|
|
423
|
+
nil
|
|
371
424
|
end
|
|
372
425
|
|
|
373
|
-
|
|
374
|
-
|
|
375
|
-
|
|
376
|
-
|
|
426
|
+
unless entries.empty?
|
|
427
|
+
buffer = RequestFormatter.multi_meta_set(entries, ttl: ttl, **routing_token_kwargs(req_options))
|
|
428
|
+
flushed_write(buffer)
|
|
429
|
+
buffer.clear
|
|
430
|
+
response_processor.consume_all_responses_until_mn
|
|
431
|
+
end
|
|
432
|
+
raise value_error if value_error
|
|
377
433
|
end
|
|
378
434
|
|
|
379
435
|
# Single-server fast path for delete_multi. Writes all quiet delete requests
|
|
@@ -33,7 +33,7 @@ module Dalli
|
|
|
33
33
|
username: uri.user,
|
|
34
34
|
password: uri.password
|
|
35
35
|
}
|
|
36
|
-
[uri.
|
|
36
|
+
[uri.hostname, normalize_port(uri.port), :tcp, DEFAULT_WEIGHT, auth_details]
|
|
37
37
|
end
|
|
38
38
|
|
|
39
39
|
def self.parse_non_uri(str)
|
|
@@ -61,11 +61,11 @@ module Dalli
|
|
|
61
61
|
# in case of unix socket, allow only setting of weight, not port
|
|
62
62
|
raise Dalli::DalliError, "Could not parse hostname #{res[0]}" if res[4]
|
|
63
63
|
|
|
64
|
-
[nil, normalize_weight(res[3])]
|
|
64
|
+
[nil, normalize_weight(res[3], res[0])]
|
|
65
65
|
end
|
|
66
66
|
|
|
67
67
|
def self.attributes_for_tcp_socket(res)
|
|
68
|
-
[normalize_port(res[3]), normalize_weight(res[4])]
|
|
68
|
+
[normalize_port(res[3]), normalize_weight(res[4], res[0])]
|
|
69
69
|
end
|
|
70
70
|
|
|
71
71
|
def self.normalize_host_from_match(str, res)
|
|
@@ -78,8 +78,13 @@ module Dalli
|
|
|
78
78
|
Integer(port || DEFAULT_PORT)
|
|
79
79
|
end
|
|
80
80
|
|
|
81
|
-
|
|
82
|
-
|
|
81
|
+
# A weight below 1 gives the server no points on the ring, and if every
|
|
82
|
+
# server has one the ring divides by a zero total weight.
|
|
83
|
+
def self.normalize_weight(weight, str)
|
|
84
|
+
value = Integer(weight || DEFAULT_WEIGHT)
|
|
85
|
+
raise ArgumentError, "Server weight must be at least 1, got #{value} in #{str}" if value < 1
|
|
86
|
+
|
|
87
|
+
value
|
|
83
88
|
end
|
|
84
89
|
end
|
|
85
90
|
end
|
|
@@ -10,7 +10,8 @@ module Dalli
|
|
|
10
10
|
##
|
|
11
11
|
class StringMarshaller
|
|
12
12
|
DEFAULTS = {
|
|
13
|
-
# max
|
|
13
|
+
# max memcached item size in bytes: value + key + ITEM_OVERHEAD_BYTES. Set it to
|
|
14
|
+
# memcached's -I value (default 1 MB).
|
|
14
15
|
value_max_bytes: 1024 * 1024
|
|
15
16
|
}.freeze
|
|
16
17
|
|
|
@@ -55,10 +56,7 @@ module Dalli
|
|
|
55
56
|
private
|
|
56
57
|
|
|
57
58
|
def error_if_over_max_value_bytes(key, value)
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
message = "Value for #{key} over max size: #{value_max_bytes} <= #{value.bytesize}"
|
|
61
|
-
raise Dalli::ValueOverMaxSize, message
|
|
59
|
+
ValueMarshaller.error_if_over_max_value_bytes(key, value, value_max_bytes)
|
|
62
60
|
end
|
|
63
61
|
end
|
|
64
62
|
end
|
|
@@ -84,11 +84,13 @@ module Dalli
|
|
|
84
84
|
end
|
|
85
85
|
|
|
86
86
|
# Checks whether we should apply compression when serializing a value
|
|
87
|
-
# based on the specified options. Returns false
|
|
88
|
-
#
|
|
89
|
-
#
|
|
90
|
-
#
|
|
87
|
+
# based on the specified options. Returns false for a raw request
|
|
88
|
+
# (raw reads ignore bitflags, so the value must be stored as given) or
|
|
89
|
+
# unless the value is greater than the minimum compression size.
|
|
90
|
+
# Otherwise returns based on a method-level option if specified,
|
|
91
|
+
# falling back to the server default.
|
|
91
92
|
def compress_value?(value, req_options)
|
|
93
|
+
return false if req_options&.dig(:raw)
|
|
92
94
|
return false unless value.bytesize >= compression_min_size
|
|
93
95
|
return compress_by_default? unless req_options && !req_options[:compress].nil?
|
|
94
96
|
|
|
@@ -13,12 +13,30 @@ module Dalli
|
|
|
13
13
|
extend Forwardable
|
|
14
14
|
|
|
15
15
|
DEFAULTS = {
|
|
16
|
-
# max
|
|
16
|
+
# max memcached item size in bytes: value + key + ITEM_OVERHEAD_BYTES. Set it to
|
|
17
|
+
# memcached's -I value (default 1 MB).
|
|
17
18
|
value_max_bytes: 1024 * 1024
|
|
18
19
|
}.freeze
|
|
19
20
|
|
|
20
21
|
OPTIONS = DEFAULTS.keys.freeze
|
|
21
22
|
|
|
23
|
+
# memcached's -I limit applies to the whole item, not just the value. On
|
|
24
|
+
# 64-bit memcached 1.6 an item takes the value, the key, and 63 bytes of
|
|
25
|
+
# overhead: a 48-byte header, 8 bytes of CAS, 4 bytes of client flags,
|
|
26
|
+
# the key's NUL terminator and the value's trailing CRLF. (Measured
|
|
27
|
+
# against memcached 1.6.45 with -I 512k, 1m, 2m and 5m; an item stored
|
|
28
|
+
# with zero flags or with CAS disabled needs a few bytes less.)
|
|
29
|
+
ITEM_OVERHEAD_BYTES = 63
|
|
30
|
+
|
|
31
|
+
def self.error_if_over_max_value_bytes(key, value, value_max_bytes)
|
|
32
|
+
item_bytes = value.bytesize + key.bytesize + ITEM_OVERHEAD_BYTES
|
|
33
|
+
return if item_bytes <= value_max_bytes
|
|
34
|
+
|
|
35
|
+
message = "Value for #{key} over max size: #{value_max_bytes} <= #{item_bytes} " \
|
|
36
|
+
"(#{value.bytesize} value bytes + #{key.bytesize} key bytes + #{ITEM_OVERHEAD_BYTES} item overhead)"
|
|
37
|
+
raise Dalli::ValueOverMaxSize, message
|
|
38
|
+
end
|
|
39
|
+
|
|
22
40
|
def_delegators :@value_serializer, :serializer
|
|
23
41
|
def_delegators :@value_compressor, :compressor, :compression_min_size, :compress_by_default?
|
|
24
42
|
|
|
@@ -28,6 +46,7 @@ module Dalli
|
|
|
28
46
|
|
|
29
47
|
@marshal_options =
|
|
30
48
|
DEFAULTS.merge(client_options.slice(*OPTIONS))
|
|
49
|
+
@marshal_options[:value_max_bytes] = @marshal_options[:value_max_bytes].to_i
|
|
31
50
|
end
|
|
32
51
|
|
|
33
52
|
def store(key, value, options = nil)
|
|
@@ -49,10 +68,7 @@ module Dalli
|
|
|
49
68
|
end
|
|
50
69
|
|
|
51
70
|
def error_if_over_max_value_bytes(key, value)
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
message = "Value for #{key} over max size: #{value_max_bytes} <= #{value.bytesize}"
|
|
55
|
-
raise Dalli::ValueOverMaxSize, message
|
|
71
|
+
self.class.error_if_over_max_value_bytes(key, value, value_max_bytes)
|
|
56
72
|
end
|
|
57
73
|
end
|
|
58
74
|
end
|
data/lib/dalli/protocol.rb
CHANGED
|
@@ -16,9 +16,9 @@ module Dalli
|
|
|
16
16
|
[Timeout::Error]
|
|
17
17
|
end
|
|
18
18
|
|
|
19
|
-
# SSL errors
|
|
20
|
-
#
|
|
21
|
-
#
|
|
19
|
+
# SSL errors during the handshake or during read/write operations count as
|
|
20
|
+
# a failed request: they trigger reconnection and, after
|
|
21
|
+
# socket_max_failures, mark the server down.
|
|
22
22
|
SSL_ERRORS =
|
|
23
23
|
if defined?(OpenSSL::SSL::SSLError)
|
|
24
24
|
[OpenSSL::SSL::SSLError]
|
data/lib/dalli/socket.rb
CHANGED
|
@@ -8,6 +8,12 @@ module Dalli
|
|
|
8
8
|
# Various socket implementations used by Dalli.
|
|
9
9
|
##
|
|
10
10
|
module Socket
|
|
11
|
+
# Timeout for establishing a connection: connect_timeout if set,
|
|
12
|
+
# otherwise socket_timeout.
|
|
13
|
+
def self.connect_timeout_for(options)
|
|
14
|
+
options[:connect_timeout] || options[:socket_timeout]
|
|
15
|
+
end
|
|
16
|
+
|
|
11
17
|
##
|
|
12
18
|
# Common methods for all socket implementations.
|
|
13
19
|
##
|
|
@@ -162,11 +168,13 @@ module Dalli
|
|
|
162
168
|
end
|
|
163
169
|
|
|
164
170
|
def self.create_socket_with_timeout(host, port, options)
|
|
171
|
+
timeout = Dalli::Socket.connect_timeout_for(options)
|
|
165
172
|
if supports_connect_timeout?
|
|
166
|
-
|
|
173
|
+
# resolv_timeout bounds the DNS lookup, which connect_timeout doesn't cover
|
|
174
|
+
sock = new(host, port, connect_timeout: timeout, resolv_timeout: timeout)
|
|
167
175
|
yield(sock)
|
|
168
176
|
else
|
|
169
|
-
Timeout.timeout(
|
|
177
|
+
Timeout.timeout(timeout) do
|
|
170
178
|
sock = new(host, port)
|
|
171
179
|
yield(sock)
|
|
172
180
|
end
|
|
@@ -228,12 +236,41 @@ module Dalli
|
|
|
228
236
|
[seconds, microseconds].pack(timeval_pack_format(sock))
|
|
229
237
|
end
|
|
230
238
|
|
|
239
|
+
# Closes the TCP socket if the handshake fails or times out, so a failed
|
|
240
|
+
# attempt doesn't hold a file descriptor until GC.
|
|
231
241
|
def self.wrapping_ssl_socket(tcp_socket, host, ssl_context)
|
|
242
|
+
warn_if_unverified(ssl_context)
|
|
243
|
+
connected = false
|
|
232
244
|
ssl_socket = Dalli::Socket::SSLSocket.new(tcp_socket, ssl_context)
|
|
233
245
|
ssl_socket.hostname = host
|
|
234
246
|
ssl_socket.sync_close = true
|
|
235
247
|
ssl_socket.connect
|
|
248
|
+
connected = true
|
|
236
249
|
ssl_socket
|
|
250
|
+
ensure
|
|
251
|
+
tcp_socket.close unless connected
|
|
252
|
+
end
|
|
253
|
+
|
|
254
|
+
# Class variable to track whether the TLS verification warning has been logged
|
|
255
|
+
@@ssl_verification_warning_logged = false # rubocop:disable Style/ClassVars
|
|
256
|
+
|
|
257
|
+
# Verification comes entirely from the caller's ssl_context, and a bare
|
|
258
|
+
# SSLContext.new verifies nothing, so warn (once per process) when the
|
|
259
|
+
# certificate or the hostname won't be checked.
|
|
260
|
+
def self.warn_if_unverified(ssl_context)
|
|
261
|
+
return if @@ssl_verification_warning_logged
|
|
262
|
+
|
|
263
|
+
problem = if !ssl_context.verify_mode.to_i.anybits?(::OpenSSL::SSL::VERIFY_PEER)
|
|
264
|
+
'verify_mode is VERIFY_NONE, so the server certificate is not checked'
|
|
265
|
+
elsif !ssl_context.verify_hostname
|
|
266
|
+
'verify_hostname is false, so the certificate is not checked against the hostname'
|
|
267
|
+
end
|
|
268
|
+
return unless problem
|
|
269
|
+
|
|
270
|
+
Dalli.logger.warn "SECURITY WARNING: Dalli's ssl_context #{problem}. " \
|
|
271
|
+
'Use ssl_context.set_params(verify_mode: OpenSSL::SSL::VERIFY_PEER, ' \
|
|
272
|
+
'verify_hostname: true) to verify the server.'
|
|
273
|
+
@@ssl_verification_warning_logged = true # rubocop:disable Style/ClassVars
|
|
237
274
|
end
|
|
238
275
|
end
|
|
239
276
|
|
|
@@ -261,7 +298,7 @@ module Dalli
|
|
|
261
298
|
attr_accessor :options
|
|
262
299
|
|
|
263
300
|
def self.open(path, options = {})
|
|
264
|
-
Timeout.timeout(options
|
|
301
|
+
Timeout.timeout(Dalli::Socket.connect_timeout_for(options)) do
|
|
265
302
|
sock = new(path)
|
|
266
303
|
sock.options = { path: path }.merge(options)
|
|
267
304
|
init_socket_options(sock, options)
|
data/lib/dalli/version.rb
CHANGED
data/lib/dalli.rb
CHANGED
|
@@ -31,6 +31,10 @@ module Dalli
|
|
|
31
31
|
# socket/server communication error that can be retried
|
|
32
32
|
class RetryableNetworkError < NetworkError; end
|
|
33
33
|
|
|
34
|
+
# a multi-get reply named a key that wasn't requested, so the rest of that
|
|
35
|
+
# server's replies can't be trusted
|
|
36
|
+
class UnexpectedReplyKeyError < DalliError; end
|
|
37
|
+
|
|
34
38
|
# Implements the NullObject pattern to store an application-defined value for 'Key not found' responses.
|
|
35
39
|
class NilObject; end # rubocop:disable Lint/EmptyClass
|
|
36
40
|
NOT_FOUND = NilObject.new
|
data/lib/rack/session/dalli.rb
CHANGED
|
@@ -2,7 +2,6 @@
|
|
|
2
2
|
|
|
3
3
|
require 'rack/session/abstract/id'
|
|
4
4
|
require 'dalli'
|
|
5
|
-
require 'connection_pool'
|
|
6
5
|
require 'English'
|
|
7
6
|
|
|
8
7
|
module Rack
|
|
@@ -21,6 +20,11 @@ module Rack
|
|
|
21
20
|
namespace: 'rack:session'
|
|
22
21
|
}
|
|
23
22
|
|
|
23
|
+
# Our pool options and the ConnectionPool options they set. These are
|
|
24
|
+
# not passed on to Dalli::Client.
|
|
25
|
+
POOL_OPTIONS = { pool_size: :size, pool_timeout: :timeout }.freeze
|
|
26
|
+
private_constant :POOL_OPTIONS
|
|
27
|
+
|
|
24
28
|
# Brings in a new Rack::Session::Dalli middleware with the given
|
|
25
29
|
# `:memcache_server`. The server is either a hostname, or a
|
|
26
30
|
# host-with-port string in the form of "host_name:port", or an array of
|
|
@@ -73,10 +77,18 @@ module Rack
|
|
|
73
77
|
@data = build_data_source(options)
|
|
74
78
|
end
|
|
75
79
|
|
|
76
|
-
|
|
77
|
-
|
|
80
|
+
# If the session is deleted from memcached while a request is running
|
|
81
|
+
# (for example, by a logout in another request), the session is not
|
|
82
|
+
# written back, so a logged-out session can't come back. The app's
|
|
83
|
+
# response is replaced with a 401 that expires the session cookie.
|
|
84
|
+
# The app has already run by then, so any side effects it had (database
|
|
85
|
+
# writes, emails, and so on) have happened.
|
|
86
|
+
def context(env, app = @app)
|
|
87
|
+
body = nil
|
|
88
|
+
super(env, ->(app_env) { app.call(app_env).tap { |response| body = response[2] } })
|
|
78
89
|
rescue MissingSessionError
|
|
79
|
-
|
|
90
|
+
body.close if body.respond_to?(:close)
|
|
91
|
+
missing_session_response(env)
|
|
80
92
|
end
|
|
81
93
|
|
|
82
94
|
def find_session(req, sid)
|
|
@@ -111,13 +123,23 @@ module Rack
|
|
|
111
123
|
def delete_session(_req, sid, options)
|
|
112
124
|
with_dalli_client do |dc|
|
|
113
125
|
key = memcached_key_from_sid(sid)
|
|
114
|
-
dc
|
|
126
|
+
delete_session_key(dc, key) if key
|
|
115
127
|
generate_sid_with(dc) unless options[:drop]
|
|
116
128
|
end
|
|
117
129
|
end
|
|
118
130
|
|
|
119
131
|
private
|
|
120
132
|
|
|
133
|
+
# A failed delete leaves the old session readable in memcached (e.g.
|
|
134
|
+
# after logout), so always log it, even though with_dalli_client then
|
|
135
|
+
# handles the error as before.
|
|
136
|
+
def delete_session_key(client, key)
|
|
137
|
+
client.delete(key)
|
|
138
|
+
rescue ::Dalli::DalliError, Errno::ECONNREFUSED => e
|
|
139
|
+
::Dalli.logger.warn("#{self.class} failed to delete a session from memcached: #{e.class}: #{e.message}")
|
|
140
|
+
raise
|
|
141
|
+
end
|
|
142
|
+
|
|
121
143
|
def memcached_key_from_sid(sid)
|
|
122
144
|
sid.private_id if sid.respond_to?(:private_id)
|
|
123
145
|
end
|
|
@@ -162,6 +184,12 @@ module Rack
|
|
|
162
184
|
end
|
|
163
185
|
end
|
|
164
186
|
|
|
187
|
+
def missing_session_response(env)
|
|
188
|
+
options = env[Rack::RACK_SESSION_OPTIONS] || @default_options
|
|
189
|
+
cookie = Rack::Utils.delete_set_cookie_header(key, options.slice(:path, :domain))
|
|
190
|
+
[401, { 'content-type' => 'text/plain', 'set-cookie' => cookie }, ['Wrong session ID']]
|
|
191
|
+
end
|
|
192
|
+
|
|
165
193
|
def write_session_safely!(dalli_client, sid, persisted_data, write_args:)
|
|
166
194
|
if persisted_data && persisted_data[:id] == sid # That means that we update the existing session
|
|
167
195
|
# Override the session only if it still exists in the store!
|
|
@@ -181,15 +209,14 @@ module Rack
|
|
|
181
209
|
end
|
|
182
210
|
|
|
183
211
|
def retrieve_client_options(options)
|
|
184
|
-
# Filter out Rack::Session-specific options and apply our defaults
|
|
185
|
-
filtered_opts = options.reject { |k, _| DEFAULT_OPTIONS.key? k }
|
|
212
|
+
# Filter out Rack::Session-specific and pool options and apply our defaults
|
|
213
|
+
filtered_opts = options.reject { |k, _| DEFAULT_OPTIONS.key?(k) || POOL_OPTIONS.key?(k) }
|
|
186
214
|
DEFAULT_DALLI_OPTIONS.merge(filtered_opts)
|
|
187
215
|
end
|
|
188
216
|
|
|
189
217
|
def retrieve_pool_options(options)
|
|
190
|
-
{}
|
|
191
|
-
pool_options[
|
|
192
|
-
pool_options[:timeout] = options.delete(:pool_timeout) if options[:pool_timeout]
|
|
218
|
+
POOL_OPTIONS.each_with_object({}) do |(option, pool_option), pool_options|
|
|
219
|
+
pool_options[pool_option] = options[option] if options[option]
|
|
193
220
|
end
|
|
194
221
|
end
|
|
195
222
|
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: dalli
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 5.2.
|
|
4
|
+
version: 5.2.3
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Peter M. Goldstein
|
|
@@ -89,7 +89,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
|
89
89
|
- !ruby/object:Gem::Version
|
|
90
90
|
version: '0'
|
|
91
91
|
requirements: []
|
|
92
|
-
rubygems_version: 4.0.
|
|
92
|
+
rubygems_version: 4.0.23
|
|
93
93
|
specification_version: 4
|
|
94
94
|
summary: High performance memcached client for Ruby
|
|
95
95
|
test_files: []
|