dalli 5.0.5 → 5.0.7
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +126 -0
- data/lib/dalli/client.rb +44 -7
- data/lib/dalli/options.rb +1 -1
- data/lib/dalli/pipelined_deleter.rb +17 -10
- data/lib/dalli/protocol/base.rb +14 -1
- data/lib/dalli/protocol/connection_manager.rb +34 -10
- data/lib/dalli/protocol/key_regularizer.rb +8 -9
- data/lib/dalli/protocol/meta.rb +28 -71
- data/lib/dalli/protocol/request_formatter.rb +95 -34
- data/lib/dalli/protocol/response_buffer.rb +21 -22
- data/lib/dalli/protocol/response_processor.rb +41 -15
- data/lib/dalli/socket.rb +47 -32
- data/lib/dalli/version.rb +1 -1
- metadata +2 -3
- data/lib/dalli/pid_cache.rb +0 -40
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 10a8b21fed4728b6f98929e983262dc1fc32c59711fce78a84b0df82b0348ca1
|
|
4
|
+
data.tar.gz: 0253d48293d3b482e8362830acd9323873302e202cb230bca3fc1b1e09b1b457
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: bda1d416cf1f88514d0181ed2e7f7b53e1b935fb07901398cb91183f63e42d0ab7c9e54c177264cf77fc793ead7b8183d79cf3c335d8dee11aae5049a6f0fff9
|
|
7
|
+
data.tar.gz: d81bec09db2c23cdc95204b0bec5c491fbcb60e12823245b9d9d338502c98b2e6af5f6e58a12b11c081889ea5d63cb079b6e8ef2fb869db147f67219db86ac55
|
data/CHANGELOG.md
CHANGED
|
@@ -4,6 +4,132 @@ Dalli Changelog
|
|
|
4
4
|
Unreleased
|
|
5
5
|
==========
|
|
6
6
|
|
|
7
|
+
5.0.7
|
|
8
|
+
==========
|
|
9
|
+
|
|
10
|
+
Security:
|
|
11
|
+
|
|
12
|
+
- Fix memcached command injection through numeric arguments (GHSA-6wmv-xq9m-fmp7)
|
|
13
|
+
- The `default` argument of `incr`/`decr`, and `fetch_with_lock`'s `lock_ttl` and `recache_threshold`, were written into the meta protocol command without conversion, so a String containing CRLF injected additional memcached commands (e.g. `set`, `flush_all`) on the connection
|
|
14
|
+
- These arguments must now be Integers, or Strings of decimal digits; anything else raises `ArgumentError` before a request is sent
|
|
15
|
+
- As defense in depth, `RequestFormatter` now converts every numeric flag it writes (`D`, `J`, `N`, `R`, `T`) to an Integer
|
|
16
|
+
- Thanks to oss-security-shop for the report
|
|
17
|
+
|
|
18
|
+
5.0.6
|
|
19
|
+
==========
|
|
20
|
+
|
|
21
|
+
Performance:
|
|
22
|
+
|
|
23
|
+
- Skip the cas-return flag on quiet `meta_set` requests (#1131)
|
|
24
|
+
- In quiet mode memcached suppresses the `ms` response entirely, so the CAS requested by the `c` flag can never be read; sending it only added two bytes to every request
|
|
25
|
+
- Applies to the bulk-write paths, where quiet sets are emitted: `Dalli::Client#multi` blocks and the pipelined setter
|
|
26
|
+
- Extracted from #1130; thanks to Jianbin Chen for this contribution
|
|
27
|
+
|
|
28
|
+
- Reduce allocations in `KeyRegularizer` and multi-key request paths (#1120)
|
|
29
|
+
- Decomposed `KeyRegularizer#encode` into separate `needs_encoding?` and `encode` calls so the common happy path avoids allocating an intermediate array for the two-element return value
|
|
30
|
+
- Refactored `multi_get`/`multi_set`/`multi_delete` command generation into `RequestFormatter` to share its key-encoding helpers
|
|
31
|
+
- Thanks to Jean Boussier for this contribution
|
|
32
|
+
|
|
33
|
+
- Reduce allocations in `ResponseBuffer` pipelined getk parsing (#1117)
|
|
34
|
+
- `process_single_getk_response` was building a fresh array to return results alongside the updated offset; refactored to store the offset as the last element of the existing tokens array and pop it, saving one allocation per response
|
|
35
|
+
- Also skips trailing nils in the token array
|
|
36
|
+
- Thanks to Jean Boussier for this contribution
|
|
37
|
+
|
|
38
|
+
- Enable frozen string literals in `RequestFormatter` (#1118)
|
|
39
|
+
- Frozen string literals had been inadvertently disabled; re-enabling reduces allocations by ~300,000 objects in a 10,000-iteration `get_multi_cas` benchmark (562 MB → 550 MB total allocated)
|
|
40
|
+
- Thanks to Jean Boussier for this contribution
|
|
41
|
+
|
|
42
|
+
- Reduce allocations in `ResponseProcessor#value_from_tokens` (#1113)
|
|
43
|
+
- `token[1..].to_i` was allocating a new string for every token parsed; replaced with in-place `slice!` followed by a token reset to avoid poisoning subsequent token comparisons
|
|
44
|
+
- Saves 4 allocations per entry in `get_multi_cas` workloads (a hotspot for IdentityCache)
|
|
45
|
+
- Thanks to Jean Boussier for this contribution
|
|
46
|
+
|
|
47
|
+
- Reduce allocations in common operation paths (#1111)
|
|
48
|
+
- Use `Symbol#name` over `Symbol#to_s` to return a frozen string without allocation
|
|
49
|
+
- Skip trace attribute hash construction when OpenTelemetry instrumentation is disabled
|
|
50
|
+
- Use argument forwarding (`...`) in `Client#perform` and `Threadsafe#request` to avoid splat array allocation
|
|
51
|
+
- Use `match?` in `KeyRegularizer#encode` to avoid `MatchData` object allocation
|
|
52
|
+
- Reduces objects allocated by ~26% and memory by ~6% for a simple `get` workload
|
|
53
|
+
- Thanks to Jean Boussier for this contribution
|
|
54
|
+
|
|
55
|
+
- Fix pathological memory behavior in `ResponseBuffer` (#1114)
|
|
56
|
+
- `compact_if_needed` was intended to reclaim memory by slicing off consumed bytes, but `buffer.byteslice(@offset..)` on an unfrozen string causes Ruby to allocate a hidden third string as the copy-on-write owner rather than freeing the original
|
|
57
|
+
- Redesigns buffer management to pass reusable buffer objects directly to `read`/`read_nonblock`, avoiding reallocation on each response read
|
|
58
|
+
- Reduces allocations from ~2.38 GB to ~649 MB in a `get_multi_cas` benchmark over 10,000 iterations
|
|
59
|
+
- Accompanied by new unit tests for `ResponseBuffer` (#1115)
|
|
60
|
+
- Thanks to Jean Boussier for this contribution
|
|
61
|
+
|
|
62
|
+
Features:
|
|
63
|
+
|
|
64
|
+
- `delete_multi` now returns the number of keys found and deleted (#1126)
|
|
65
|
+
- Previously the return value was unspecified; callers (e.g. Rails, see rails/rails#58071) had no way to tell how many keys were actually removed
|
|
66
|
+
- The count is derived from the meta protocol's quiet-mode delete responses with no extra round-trips: successful deletes are suppressed while misses report `NF`, so any response received before the terminator is a key that was not deleted
|
|
67
|
+
- The single-server fast path now shares the pipelined path's bounded retry on transient (`RetryableNetworkError`) network errors, so both paths behave consistently; the returned count is best-effort and may under-report if a network error triggers a retry, since keys deleted before the error are not recounted
|
|
68
|
+
- Thanks to Iliana Hadzhiatanasova for this contribution
|
|
69
|
+
|
|
70
|
+
Bug Fixes:
|
|
71
|
+
|
|
72
|
+
- Raise instead of returning a truncated value when the peer closes mid-response (#1135)
|
|
73
|
+
- `IO#read(count)` on a blocking socket accumulates across TCP chunks and hands back a shorter buffer (or `nil`) in only one case: the stream hit EOF. That short buffer was passed through as the response body, so a memcached restart, proxy drop, or load balancer timeout partway through a response could surface a truncated but still decodable value to the caller, indistinguishable from a real one
|
|
74
|
+
- A short read is now treated as the premature EOF it is, raising and closing the dirty socket so the request is retried on a fresh connection
|
|
75
|
+
- CRuby only; the JRuby path already used `Socket#readfull`, which enforces the same contract
|
|
76
|
+
- Extracted from #1130; thanks to Ian Ker-Seymer for the original fix and Jianbin Chen for the port
|
|
77
|
+
|
|
78
|
+
- Tear down the connection when a non-`StandardError` aborts a request (#1136)
|
|
79
|
+
- `Async::Stop` and `Thread#kill` descend from `Exception` rather than `StandardError`, so the rescue clauses in `Protocol::Base#request` never saw them; a scheduler cancelling a fiber parked on a response read skipped `close` entirely, leaving the connection marked as having a request in progress with partial response bytes still unread on the wire, and returning that half-used client to the pool under `connection_pool`
|
|
80
|
+
- `Protocol::Base#request` now closes in an `ensure` unless the request ran to completion, and `ConnectionManager#close` performs its state cleanup in an `ensure` so a second cancellation landing inside `@sock.close` cannot leave the socket non-nil with the request still marked in progress
|
|
81
|
+
- `Dalli::DalliError` and `Dalli::MarshalError` now close the connection at the point of failure rather than at the start of the next request; those paths already left the request in progress and `ConnectionManager#confirm_ready!` closed on the next call, so this changes when the close happens rather than adding one
|
|
82
|
+
- Extracted from #1130; thanks to Dan Mayer for the original fix and Jianbin Chen for the port
|
|
83
|
+
|
|
84
|
+
- Fix `ResponseBuffer` compaction logic (#1119)
|
|
85
|
+
- `COMPACT_THRESHOLD` was removed in #1116 as apparently unused, but the constant was referenced by the compaction guard; its absence silently disabled buffer compaction
|
|
86
|
+
- Restored the constant, corrected the compaction condition, and improved the buffer-shrinking implementation to use `String#bytesplice` (backed by `memmove`) for true in-place compaction
|
|
87
|
+
- Adds targeted tests covering the compaction threshold and shrink behavior
|
|
88
|
+
- Thanks to Jean Boussier for this contribution
|
|
89
|
+
|
|
90
|
+
Maintenance:
|
|
91
|
+
|
|
92
|
+
- Scope `StrictWarnings` to Dalli's own source (#1134)
|
|
93
|
+
- The test suite runs under `-w` and prepends a hook to `Warning.singleton_class` that turns warnings into failures, but that hook is global: a warning emitted while loading any third-party gem aborted the whole suite before a single test ran
|
|
94
|
+
- `json` 2.21.2's pure-Ruby generator (used on JRuby, where the C extension is unavailable) warns `method redefined; discarding old to_hash` at require time, which took the `jruby-10` CI job red with no change to Dalli
|
|
95
|
+
- Warnings are now attributed to a source file and only raise for `lib/` and `test/`; attribution prefers the location Ruby embeds in the message, since the stack at that point describes the require chain rather than the offending code
|
|
96
|
+
- Portable attribution of `Kernel#warn` callers also required walking the stack rather than indexing it (Ruby 3.3/3.4 push an `<internal:warning>` frame that 4.0 does not), skipping RubyGems' `Kernel#warn` shim (active on JRuby but not CRuby), and resolving relative backtrace paths
|
|
97
|
+
|
|
98
|
+
- Make raw and namespace fast path tests actually use those options (#1129)
|
|
99
|
+
- Followup to #1127: the `raw` and `namespace` variants passed those options to the helper that starts memcached, which configures the client the tests then discarded, so neither option was ever exercised
|
|
100
|
+
- Passes the options to the client under test and adds assertions that fail if they are absent
|
|
101
|
+
- Thanks to Iliana Hadzhiatanasova for this contribution
|
|
102
|
+
|
|
103
|
+
- Benchmark `set_multi` and add a `delete_multi` target (#1132)
|
|
104
|
+
- Enables the two `set_multi` reports that were commented out pending the arrival of `set_multi`, resolving the accompanying TODO
|
|
105
|
+
- Adds a `delete_multi` target comparing the pipelined path against N single deletes
|
|
106
|
+
- Extracted from #1130; thanks to Jianbin Chen for this contribution
|
|
107
|
+
|
|
108
|
+
- Bump CI memcached to 1.6.41 and run benchmarks on pull requests (#1133)
|
|
109
|
+
- The tests workflow moves from 1.6.40 to 1.6.41; the benchmarks and profile workflows had drifted back on 1.6.23
|
|
110
|
+
- Extracted from #1130; thanks to Jianbin Chen for this contribution
|
|
111
|
+
|
|
112
|
+
- Disable RuboCop metrics cops (#1128)
|
|
113
|
+
- Thanks to Jean Boussier for this contribution
|
|
114
|
+
|
|
115
|
+
- Remove `PIDCache` module (#1125)
|
|
116
|
+
- `Process.pid` is cached natively by Ruby 3.3+ (via https://bugs.ruby-lang.org/issues/19443), making the manual cache unnecessary now that Dalli requires Ruby 3.3+
|
|
117
|
+
- Thanks to Jean Boussier for this contribution
|
|
118
|
+
|
|
119
|
+
- Remove unused `COMPACT_THRESHOLD` constant from `ResponseBuffer` (#1116)
|
|
120
|
+
- Followup cleanup after the buffer management redesign in #1114
|
|
121
|
+
- Note: subsequently found to be in use; restored and corrected in #1119
|
|
122
|
+
- Thanks to Jean Boussier for this contribution
|
|
123
|
+
|
|
124
|
+
- Use `String#byteindex` instead of `String#index` when searching for the response terminator in `getk_response_from_buffer` (#1112)
|
|
125
|
+
- The result feeds directly into `byteslice`; `byteindex` makes the intent explicit, though both return the same value since the buffer encoding is always `BINARY`
|
|
126
|
+
- Thanks to Jean Boussier for this contribution
|
|
127
|
+
|
|
128
|
+
- Make single-server fast path tests actually exercise the fast path (#1127)
|
|
129
|
+
- The batch-operation tests built clients through a helper that registers two address aliases for the same memcached process, so every client had a 2-server ring and the tests always ran through the pipelined path instead of the single-server fast path
|
|
130
|
+
- Adds a `single_server_client` test helper that builds a client with a single address, and uses it in the affected `get_multi`, `set_multi`, and `delete_multi` tests
|
|
131
|
+
- Thanks to Iliana Hadzhiatanasova for this contribution
|
|
132
|
+
|
|
7
133
|
5.0.5
|
|
8
134
|
==========
|
|
9
135
|
|
data/lib/dalli/client.rb
CHANGED
|
@@ -225,6 +225,8 @@ module Dalli
|
|
|
225
225
|
def fetch_with_lock(key, ttl: nil, lock_ttl: 30, recache_threshold: nil, req_options: nil, &block)
|
|
226
226
|
raise ArgumentError, 'Block is required for fetch_with_lock' unless block_given?
|
|
227
227
|
|
|
228
|
+
validate_integer!(:lock_ttl, lock_ttl)
|
|
229
|
+
validate_integer!(:recache_threshold, recache_threshold)
|
|
228
230
|
key = key.to_s
|
|
229
231
|
key = @key_manager.validate_key(key)
|
|
230
232
|
|
|
@@ -364,12 +366,15 @@ module Dalli
|
|
|
364
366
|
# it batches requests by server and uses quiet mode.
|
|
365
367
|
#
|
|
366
368
|
# @param keys [Array<String>] keys to delete
|
|
367
|
-
# @return [
|
|
369
|
+
# @return [Integer] the number of keys that were found and deleted. This is
|
|
370
|
+
# best-effort: on a network error the operation is retried, and keys
|
|
371
|
+
# deleted before the error are not recounted, so the result may
|
|
372
|
+
# under-report the number actually removed when a failure occurs.
|
|
368
373
|
#
|
|
369
374
|
# Example:
|
|
370
375
|
# client.delete_multi(['key1', 'key2', 'key3'])
|
|
371
376
|
def delete_multi(keys)
|
|
372
|
-
return if keys.empty?
|
|
377
|
+
return 0 if keys.empty?
|
|
373
378
|
|
|
374
379
|
Instrumentation.trace('delete_multi', multi_trace_attrs('delete_multi', keys.size, keys)) do
|
|
375
380
|
if ring.servers.size == 1
|
|
@@ -397,6 +402,8 @@ module Dalli
|
|
|
397
402
|
##
|
|
398
403
|
# Incr adds the given amount to the counter on the memcached server.
|
|
399
404
|
# Amt must be a positive integer value.
|
|
405
|
+
# Default, if given, must be an Integer (or a String of decimal digits);
|
|
406
|
+
# anything else raises ArgumentError.
|
|
400
407
|
#
|
|
401
408
|
# If default is nil, the counter must already exist or the operation
|
|
402
409
|
# will fail and will return nil. Otherwise this method will return
|
|
@@ -409,6 +416,7 @@ module Dalli
|
|
|
409
416
|
# If the value already exists, it must have been set with raw: true
|
|
410
417
|
def incr(key, amt = 1, ttl = nil, default = nil)
|
|
411
418
|
check_positive!(amt)
|
|
419
|
+
validate_integer!(:default, default)
|
|
412
420
|
|
|
413
421
|
perform(:incr, key, amt.to_i, ttl_or_default(ttl), default)
|
|
414
422
|
end
|
|
@@ -416,6 +424,8 @@ module Dalli
|
|
|
416
424
|
##
|
|
417
425
|
# Decr subtracts the given amount from the counter on the memcached server.
|
|
418
426
|
# Amt must be a positive integer value.
|
|
427
|
+
# Default, if given, must be an Integer (or a String of decimal digits);
|
|
428
|
+
# anything else raises ArgumentError.
|
|
419
429
|
#
|
|
420
430
|
# memcached counters are unsigned and cannot hold negative values. Calling
|
|
421
431
|
# decr on a counter which is 0 will just return 0.
|
|
@@ -431,6 +441,7 @@ module Dalli
|
|
|
431
441
|
# If the value already exists, it must have been set with raw: true
|
|
432
442
|
def decr(key, amt = 1, ttl = nil, default = nil)
|
|
433
443
|
check_positive!(amt)
|
|
444
|
+
validate_integer!(:default, default)
|
|
434
445
|
|
|
435
446
|
perform(:decr, key, amt.to_i, ttl_or_default(ttl), default)
|
|
436
447
|
end
|
|
@@ -569,11 +580,18 @@ module Dalli
|
|
|
569
580
|
|
|
570
581
|
def single_server_delete_multi(keys)
|
|
571
582
|
validated_keys = keys.map { |k| @key_manager.validate_key(k.to_s) }
|
|
572
|
-
return unless (server = single_server)
|
|
583
|
+
return 0 unless (server = single_server)
|
|
573
584
|
|
|
574
585
|
server.request(:delete_multi_req, validated_keys)
|
|
586
|
+
rescue Dalli::RetryableNetworkError => e
|
|
587
|
+
# Mirror the pipelined path: retry transient errors so a momentary blip
|
|
588
|
+
# still yields a real count. Bounded by the server's socket_max_failures,
|
|
589
|
+
# after which a hard NetworkError is raised and handled below.
|
|
590
|
+
Dalli.logger.debug { e.inspect }
|
|
591
|
+
Dalli.logger.debug { 'retrying single-server delete_multi because of network error' }
|
|
592
|
+
retry
|
|
575
593
|
rescue Dalli::NetworkError
|
|
576
|
-
|
|
594
|
+
0
|
|
577
595
|
end
|
|
578
596
|
|
|
579
597
|
def get_multi_attributes(keys)
|
|
@@ -607,6 +625,19 @@ module Dalli
|
|
|
607
625
|
raise ArgumentError, "Positive values only: #{amt}" if amt.negative?
|
|
608
626
|
end
|
|
609
627
|
|
|
628
|
+
# Numeric arguments that become meta protocol flags (GHSA-6wmv-xq9m-fmp7).
|
|
629
|
+
# RequestFormatter converts them to Integer as the wire-level backstop;
|
|
630
|
+
# checking here too raises a clean ArgumentError before the request
|
|
631
|
+
# starts, instead of Protocol::Base#request logging it as unexpected and
|
|
632
|
+
# closing the connection.
|
|
633
|
+
def validate_integer!(name, value)
|
|
634
|
+
return if value.nil?
|
|
635
|
+
|
|
636
|
+
value.is_a?(String) ? Integer(value, 10) : Integer(value)
|
|
637
|
+
rescue ArgumentError, TypeError, FloatDomainError
|
|
638
|
+
raise ArgumentError, "#{name} must be an Integer, got #{value.inspect}"
|
|
639
|
+
end
|
|
640
|
+
|
|
610
641
|
def cas_core(key, always_set, ttl = nil, req_options = nil)
|
|
611
642
|
(value, cas) = perform(:cas, key)
|
|
612
643
|
return if value.nil? && !always_set
|
|
@@ -651,7 +682,7 @@ module Dalli
|
|
|
651
682
|
# operation times out.
|
|
652
683
|
##
|
|
653
684
|
# rubocop:disable Naming/MethodParameterName
|
|
654
|
-
def perform(op, key,
|
|
685
|
+
def perform(op, key, ...)
|
|
655
686
|
# rubocop:enable Naming/MethodParameterName
|
|
656
687
|
return yield if block_given?
|
|
657
688
|
|
|
@@ -659,8 +690,14 @@ module Dalli
|
|
|
659
690
|
key = @key_manager.validate_key(key)
|
|
660
691
|
|
|
661
692
|
server = ring.server_for_key(key)
|
|
662
|
-
|
|
663
|
-
|
|
693
|
+
|
|
694
|
+
if Instrumentation.enabled?
|
|
695
|
+
op_name = op.name
|
|
696
|
+
Instrumentation.trace(op_name, trace_attrs(op_name, key, server)) do
|
|
697
|
+
server.request(op, key, ...)
|
|
698
|
+
end
|
|
699
|
+
else
|
|
700
|
+
server.request(op, key, ...)
|
|
664
701
|
end
|
|
665
702
|
rescue RetryableNetworkError => e
|
|
666
703
|
Dalli.logger.debug { e.inspect }
|
data/lib/dalli/options.rb
CHANGED
|
@@ -16,16 +16,19 @@ module Dalli
|
|
|
16
16
|
# Deletes multiple keys from memcached.
|
|
17
17
|
#
|
|
18
18
|
# @param keys [Array<String>] keys to delete
|
|
19
|
-
# @return [
|
|
19
|
+
# @return [Integer] the number of keys that were deleted. This is
|
|
20
|
+
# best-effort: on a network error the operation is retried, and keys
|
|
21
|
+
# deleted before the error are not recounted, so the result may
|
|
22
|
+
# under-report the number actually removed when a failure occurs.
|
|
20
23
|
##
|
|
21
24
|
def process(keys)
|
|
22
|
-
return if keys.empty?
|
|
25
|
+
return 0 if keys.empty?
|
|
23
26
|
|
|
24
27
|
@ring.lock do
|
|
25
|
-
|
|
26
|
-
finish_requests(
|
|
28
|
+
groups = setup_requests(keys)
|
|
29
|
+
finish_requests(groups)
|
|
27
30
|
end
|
|
28
|
-
rescue
|
|
31
|
+
rescue Dalli::RetryableNetworkError => e
|
|
29
32
|
Dalli.logger.debug { e.inspect }
|
|
30
33
|
Dalli.logger.debug { 'retrying pipelined deletes because of network error' }
|
|
31
34
|
retry
|
|
@@ -36,7 +39,7 @@ module Dalli
|
|
|
36
39
|
def setup_requests(keys)
|
|
37
40
|
groups = groups_for_keys(keys)
|
|
38
41
|
make_delete_requests(groups)
|
|
39
|
-
groups
|
|
42
|
+
groups
|
|
40
43
|
end
|
|
41
44
|
|
|
42
45
|
##
|
|
@@ -45,24 +48,28 @@ module Dalli
|
|
|
45
48
|
##
|
|
46
49
|
def make_delete_requests(groups)
|
|
47
50
|
groups.each do |server, keys_for_server|
|
|
48
|
-
keys_for_server.
|
|
51
|
+
keys_for_server.select! do |key|
|
|
49
52
|
server.request(:pipelined_delete, key)
|
|
53
|
+
true
|
|
50
54
|
rescue DalliError, NetworkError => e
|
|
51
55
|
Dalli.logger.debug { e.inspect }
|
|
52
56
|
Dalli.logger.debug { "unable to delete key #{key} for server #{server.name}" }
|
|
57
|
+
false
|
|
53
58
|
end
|
|
54
59
|
end
|
|
55
60
|
end
|
|
56
61
|
|
|
57
62
|
##
|
|
58
63
|
# Sends noop to each server to flush responses and ensure all deletes complete.
|
|
64
|
+
# Returns the total successful deletes across servers.
|
|
59
65
|
##
|
|
60
|
-
def finish_requests(
|
|
61
|
-
|
|
62
|
-
server.request(:
|
|
66
|
+
def finish_requests(groups)
|
|
67
|
+
groups.sum do |server, keys_for_server|
|
|
68
|
+
server.request(:finish_pipelined_delete, keys_for_server.size)
|
|
63
69
|
rescue DalliError, NetworkError => e
|
|
64
70
|
Dalli.logger.debug { e.inspect }
|
|
65
71
|
Dalli.logger.debug { "unable to complete pipelined delete on server #{server.name}" }
|
|
72
|
+
0
|
|
66
73
|
end
|
|
67
74
|
end
|
|
68
75
|
|
data/lib/dalli/protocol/base.rb
CHANGED
|
@@ -40,12 +40,14 @@ module Dalli
|
|
|
40
40
|
verify_state(opkey)
|
|
41
41
|
|
|
42
42
|
begin
|
|
43
|
+
request_completed = false
|
|
43
44
|
@connection_manager.start_request!
|
|
44
45
|
response = send(opkey, *args)
|
|
45
46
|
|
|
46
47
|
# pipelined_get/pipelined_get_interleaved emit query but don't read the response(s)
|
|
47
48
|
@connection_manager.finish_request! unless %i[pipelined_get pipelined_get_interleaved].include?(opkey)
|
|
48
49
|
|
|
50
|
+
request_completed = true
|
|
49
51
|
response
|
|
50
52
|
rescue Dalli::MarshalError => e
|
|
51
53
|
log_marshal_err(args.first, e)
|
|
@@ -56,6 +58,18 @@ module Dalli
|
|
|
56
58
|
log_unexpected_err(e)
|
|
57
59
|
close
|
|
58
60
|
raise
|
|
61
|
+
ensure
|
|
62
|
+
# If the begin block didn't complete -- any exception, including a
|
|
63
|
+
# non-StandardError such as Async::Stop or Thread#kill, which the
|
|
64
|
+
# rescue clauses above never see -- tear down the connection so a
|
|
65
|
+
# half-used client isn't returned to the pool.
|
|
66
|
+
#
|
|
67
|
+
# The local flag is deliberate: reading $ERROR_INFO here would see
|
|
68
|
+
# the *outer* exception when `request` is called from inside a
|
|
69
|
+
# rescue clause (a common cache-fallback shape), and would then
|
|
70
|
+
# falsely tear down the pipelined_get happy path, which leaves
|
|
71
|
+
# @request_in_progress true on purpose until the caller drains.
|
|
72
|
+
close unless request_completed
|
|
59
73
|
end
|
|
60
74
|
end
|
|
61
75
|
|
|
@@ -95,7 +109,6 @@ module Dalli
|
|
|
95
109
|
# When a block is given, yields (key, value, cas) for each response,
|
|
96
110
|
# avoiding intermediate Hash allocation. Returns nil.
|
|
97
111
|
# Without a block, returns a Hash of { key => [value, cas] }.
|
|
98
|
-
# rubocop:disable Metrics/AbcSize, Metrics/CyclomaticComplexity, Metrics/PerceivedComplexity
|
|
99
112
|
def pipeline_next_responses(&block)
|
|
100
113
|
reconnect_on_pipeline_complete!
|
|
101
114
|
values = nil
|
|
@@ -4,8 +4,6 @@ require 'English'
|
|
|
4
4
|
require 'socket'
|
|
5
5
|
require 'timeout'
|
|
6
6
|
|
|
7
|
-
require 'dalli/pid_cache'
|
|
8
|
-
|
|
9
7
|
module Dalli
|
|
10
8
|
module Protocol
|
|
11
9
|
##
|
|
@@ -54,7 +52,7 @@ module Dalli
|
|
|
54
52
|
|
|
55
53
|
@sock = memcached_socket
|
|
56
54
|
@sock.sync = false # Enable buffered I/O for better performance
|
|
57
|
-
@pid =
|
|
55
|
+
@pid = Process.pid
|
|
58
56
|
@request_in_progress = false
|
|
59
57
|
rescue SystemCallError, *TIMEOUT_ERRORS, EOFError, SocketError => e
|
|
60
58
|
# SocketError = DNS resolution failure
|
|
@@ -117,10 +115,16 @@ module Dalli
|
|
|
117
115
|
@sock.close
|
|
118
116
|
rescue StandardError
|
|
119
117
|
nil
|
|
118
|
+
ensure
|
|
119
|
+
# A non-StandardError (e.g. a second Async::Stop fired into the
|
|
120
|
+
# fiber while it is already inside this cleanup) can escape
|
|
121
|
+
# @sock.close; run the state cleanup unconditionally so the client
|
|
122
|
+
# isn't returned to the pool with a half-closed socket and
|
|
123
|
+
# @request_in_progress == true.
|
|
124
|
+
@sock = nil
|
|
125
|
+
@pid = nil
|
|
126
|
+
abort_request!
|
|
120
127
|
end
|
|
121
|
-
@sock = nil
|
|
122
|
-
@pid = nil
|
|
123
|
-
abort_request!
|
|
124
128
|
end
|
|
125
129
|
|
|
126
130
|
def connected?
|
|
@@ -165,12 +169,16 @@ module Dalli
|
|
|
165
169
|
end
|
|
166
170
|
else
|
|
167
171
|
def read(count)
|
|
168
|
-
|
|
172
|
+
read_bytes(count)
|
|
169
173
|
rescue SystemCallError, *TIMEOUT_ERRORS, *SSL_ERRORS, EOFError => e
|
|
170
174
|
error_on_request!(e)
|
|
171
175
|
end
|
|
172
176
|
end
|
|
173
177
|
|
|
178
|
+
# Alias for callers that want to make the exact-length contract explicit
|
|
179
|
+
# at the call site.
|
|
180
|
+
alias read_exact read
|
|
181
|
+
|
|
174
182
|
def write(bytes)
|
|
175
183
|
@sock.write(bytes)
|
|
176
184
|
rescue SystemCallError, *TIMEOUT_ERRORS, *SSL_ERRORS, IOError => e
|
|
@@ -193,8 +201,8 @@ module Dalli
|
|
|
193
201
|
|
|
194
202
|
# Non-blocking read. Here to support the operation
|
|
195
203
|
# of the get_multi operation
|
|
196
|
-
def
|
|
197
|
-
@sock.read_available
|
|
204
|
+
def read_available(...)
|
|
205
|
+
@sock.read_available(...)
|
|
198
206
|
end
|
|
199
207
|
|
|
200
208
|
def max_allowed_failures
|
|
@@ -252,7 +260,7 @@ module Dalli
|
|
|
252
260
|
end
|
|
253
261
|
|
|
254
262
|
def fork_detected?
|
|
255
|
-
@pid && @pid !=
|
|
263
|
+
@pid && @pid != Process.pid
|
|
256
264
|
end
|
|
257
265
|
|
|
258
266
|
def log_down_detected
|
|
@@ -273,6 +281,22 @@ module Dalli
|
|
|
273
281
|
time = Time.now - @down_at
|
|
274
282
|
Dalli.logger.warn { format('%<name>s is back (downtime was %<time>.3f seconds)', name: name, time: time) }
|
|
275
283
|
end
|
|
284
|
+
|
|
285
|
+
private
|
|
286
|
+
|
|
287
|
+
# Reads exactly `count` bytes. IO#read(count) on a blocking socket blocks
|
|
288
|
+
# until it has `count` bytes, accumulating across TCP chunks internally,
|
|
289
|
+
# and only hands back a shorter (or nil) buffer when the stream hits EOF.
|
|
290
|
+
# So a short read means the peer closed mid-response: raise EOFError and
|
|
291
|
+
# let read's existing `rescue EOFError` route it through
|
|
292
|
+
# error_on_request!, which closes the dirty socket for a retry on a fresh
|
|
293
|
+
# connection (and preserves the $ERROR_INFO context down! relies on).
|
|
294
|
+
def read_bytes(count)
|
|
295
|
+
buffer = @sock.read(count)
|
|
296
|
+
return buffer if buffer && buffer.bytesize == count
|
|
297
|
+
|
|
298
|
+
raise EOFError, "EOF reading #{count} bytes; received #{buffer ? buffer.bytesize : 0}"
|
|
299
|
+
end
|
|
276
300
|
end
|
|
277
301
|
end
|
|
278
302
|
end
|
|
@@ -9,19 +9,18 @@ module Dalli
|
|
|
9
9
|
# allowed.
|
|
10
10
|
# memcached supports the use of base64 hashes for keys containing
|
|
11
11
|
# whitespace or non-ASCII characters, provided the 'b' flag is included in the request.
|
|
12
|
-
|
|
13
|
-
|
|
12
|
+
module KeyRegularizer
|
|
13
|
+
module_function
|
|
14
14
|
|
|
15
|
-
def
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
strict_base64_encoded = [key].pack('m0')
|
|
19
|
-
[strict_base64_encoded, true]
|
|
15
|
+
def required?(key)
|
|
16
|
+
!key.ascii_only? || /\s/.match?(key)
|
|
20
17
|
end
|
|
21
18
|
|
|
22
|
-
def
|
|
23
|
-
|
|
19
|
+
def encode(key)
|
|
20
|
+
[key].pack('m0')
|
|
21
|
+
end
|
|
24
22
|
|
|
23
|
+
def decode(encoded_key)
|
|
25
24
|
strict_base64_decoded = encoded_key.unpack1('m0')
|
|
26
25
|
strict_base64_decoded.force_encoding(Encoding::UTF_8)
|
|
27
26
|
end
|